Resolved -
This incident has been resolved. We have successfully mitigated the impact caused by the ThreatDown update and are observing stability across our fleet.
If you have ThreatDown installed and continue to experience issues—such as your server being unresponsive or down—please reboot your server via the management portal at my.liquidweb.com and reach out to our support team for further assistance.
Feb 3, 13:56 EST
Update -
We are continuing to see stability across the fleet and have not observed any recent alerts related to this issue. Our team has been actively working on this incident to apply the fix across the affected servers, and we have made solid progress so far, with completion on the impacted list.
We are actively monitoring the situation and will keep this space updated as soon as we have further information available.
If you believe your system may be affected or require assistance, please chat with us or submit a ticket at my.liquidweb.com.
Thank you for your understanding and patience!
Feb 1, 23:52 EST
Update -
We are making steady progress by implementing necessary fixes and working closely with ThreatDown to find a lasting solution. The systems are stabilizing, and we are in good shape to manage the situation.
We will continue to monitor the situation and share further updates as they become available. Please contact support if you experience any issues. Thank you for your patience.
Feb 1, 00:17 EST
Update -
At this time, we have downgraded the severity of this incident. Our teams remain actively engaged and continue to monitor the affected systems. Any new issues will be addressed by applying the mblinux-updater-fix, followed by a server reboot as needed. We are also continuing to work closely with ThreatDown support toward a permanent resolution.
If you experience any service interruptions, please don’t hesitate to reach out via chat or submit a support ticket through your customer portal at my.liquidweb.com. We appreciate your patience and understanding as our teams work through this, and we’ll provide further updates as they become available.
Jan 30, 19:41 EST
Update -
Our team is actively working to resolve the issue by running the mblinux-updater-fix, which has proven effective when combined with a server reboot. Based on ongoing service alerts, we are applying this fix where required and closely monitoring affected systems to ensure stability.
We are actively working with ThreatDown to identify the best permanent resolution and outline next steps. If you believe your system may still be affected or you experience any service interruption, please contact us via chat or submit a support ticket at my.liquidweb.com.
Further updates will be provided as they become available.
Jan 30, 08:57 EST
Monitoring -
Our team has made progress by applying the fix to the affected servers. We have been working closely with the ThreatDown team to obtain the latest updates from their side in order to address this issue. We will continue to monitor the situation and take action where necessary.
If you believe your system may be affected or you are experiencing a service interruption, please contact us via chat or submit a support ticket at my.liquidweb.com.
We will provide further updates as they become available.
Thank you for your patience and support.
Jan 29, 23:40 EST
Update -
We have identified the issue and confirmed it impacts a subset of services with the ThreatDown (TD) add-on enabled. WE have additional resources engaged with Threat Stack. Our team is working through and applying manual fixes to servers that we have been able to identify as affected by this incident.
We’ll provide additional updates as they become available.
If you believe your system is impacted or are experiencing a service interruption, please chat with us or submit a ticket at my.liquidweb.com.
Thank you for your patience and support.
Jan 29, 20:18 EST
Update -
Our team is actively addressing impacted systems and working to restore service as quickly as possible.
If you believe your system is affected or need assistance, please chat with us or submit a ticket at my.liquidweb.com.
Thank you for your patience and support.
Jan 29, 17:07 EST
Update -
We’ve identified the issue and confirmed it impacts a subset of services with the ThreatDown (TD) add-on enabled. Our team is actively identifying affected systems and deploying a fix as quickly as possible.
We’ll provide additional updates as they become available.
If you believe your system is impacted or are experiencing a service interruption, please chat with us or submit a ticket at my.liquidweb.com.
Thank you for your patience and support.
Jan 29, 11:25 EST
Update -
The issue has been identified and is affecting only services with the ThreatDown (TD) add-on enabled. Our Support team is actively working to implement a fix as quickly as possible.
We appreciate your patience and understanding. Further updates will be shared here as they become available.
If you need assistance, please chat with us or submit a ticket at my.liquidweb.com. Thank you for your patience and understanding. We appreciate your support.
Jan 29, 10:27 EST
Identified -
The issue has been identified, and our Support team is actively working to implement the fix as quickly as possible.
We appreciate your patience and understanding. Further updates will be shared here as they become available.
Jan 29, 00:08 EST
Investigating -
We are currently investigating the POP3 and SSH service down alerts on a subset of servers. The issue appears to have occurred after an endpoint update on the Mbdaemon associated with the Threatdown add-on service.
Our engineering team is actively working to identify the root cause and resolve the issue.
If you need assistance, please chat with us or submit a ticket at my.liquidweb.com. Thank you for your patience and understanding. We appreciate your support.
Jan 28, 23:06 EST
Completed -
The scheduled maintenance has been completed.
Jan 31, 04:00 EST
In progress -
Scheduled maintenance is currently in progress. We will provide updates as necessary.
Jan 31, 01:00 EST
Scheduled -
Liquid Web engineers will be performing scheduled network maintenance to upgrade network hardware for improved performance and stability.
Maintenance Window: Saturday, January 31st, 1:00 AM – 4:00 AM ET
During this maintenance window, services under the location where maintenance is being performed may experience brief packet loss, latency, or re-connection. Case email notifications have been sent to the affected clients.
You can subscribe to receive further updates regarding this maintenance above.
Our support team is on hand should you need any assistance or have any questions or concerns. You can reach us through the following channels:
Resolved -
The outgoing email issue with the Premium Business Email service affecting Yahoo and Microsoft has been completely resolved. We have been monitoring the platform and have not observed any further problems.
Thank you for your patience during this time.
Jan 28, 21:15 EST
Identified -
Our email vendor continues to monitor email delivery to Yahoo and Microsoft recipients. Yahoo is still deferring some emails, and the vendor is actively working to clear the queue and address the senders causing these deferrals.
For Microsoft, the vendor is coordinating directly with Microsoft to resolve the delivery issues. We will provide updates here as soon as new information becomes available.
We appreciate your patience as we work to fully restore email delivery.
Jan 28, 17:21 EST
Investigating -
We have identified that users of the Premium Business Email services are experiencing difficulties sending emails to Microsoft and Yahoo-related addresses. We are actively working with Microsoft's and Yahoo's teams to resolve this issue.
We appreciate your patience and understanding.
If you need assistance, please chat with us or create a case at my.liquidweb.com.
Jan 27, 23:06 EST
Resolved -
This incident has been resolved.
Jan 27, 22:18 EST
Monitoring -
A fix has been implemented to address the issue, and services are stabilizing. We are actively monitoring the updates to ensure continued recovery. We apologize for the inconvenience and will provide additional updates if needed.
Investigating -
We are currently tracking a network degradation issue affecting the Chicago, IL area. Our upstream provider, Cloudflare, is working to resolve the matter.
We are monitoring the situation closely and will provide updates as they become available.
Resolved -
This incident has been resolved.
Jan 23, 04:32 EST
Monitoring -
A fix has been implemented and we are monitoring the results.
Jan 23, 01:56 EST
Identified -
Microsoft is currently experiencing widespread service degradation. This issue specifically impacts users attempting to send or receive emails to Microsoft-hosted accounts (including Outlook, Hotmail, and Office 365).
Affected users may experience limited functionality across multiple Microsoft 365 services or receive delivery errors such as "451 4.3.2 Temporary server" or "Host lookup failed."
Our team is monitoring the situation closely and will provide further updates as more information becomes available. We appreciate your patience and understanding.
If you need any help, please chat with us or create a ticket via my.liquidweb.com.
Jan 23, 00:50 EST
Completed -
The scheduled maintenance has been completed.
Jan 20, 21:00 EST
In progress -
Scheduled maintenance is currently in progress. We will provide updates as necessary.
Jan 20, 20:00 EST
Scheduled -
We will be conducting scheduled maintenance on my.liquidweb.com remote console access. This work is part of our ongoing efforts to ensure reliability and stability across our infrastructure.
Expected Impact: During the maintenance window, some users may experience temporary disconnections from remote console pages (lasting a few minutes).
If disconnected, users can click “Reconnect” or refresh the page to restore their session. The console will resume exactly where it left off. This behavior is similar to the existing 1-hour idle timeout already in place for console sessions.
Our engineers will monitor the maintenance closely to minimize any potential disruption.
We appreciate your patience and understanding as we perform this important maintenance.
Internal System Maintenance Reference: [CC-13116]
Jan 16, 16:10 EST