Service disruption at the Phoenix Data Center
Incident
Report for Liquid Web - Services
Subscribe to Updates
Update
The vast majority of impacted servers have been brought online and are functional at this time. We continue to monitor the stability of the network to ensure normal operations.
Our engineers are still working on restoring functionality to the Cloud Sites Platform but are making progress. The root cause has been identified and progress is being made. Updates and more information regarding those efforts can be found on our Cloud Sites Status Page here: https://status.websitesettings.com/incidents/0l8hyyhr6v93
Our engineers are still working on restoring functionality to the Cloud Sites Platform but are making progress. The root cause has been identified and progress is being made. Updates and more information regarding those efforts can be found on our Cloud Sites Status Page here: https://status.websitesettings.com/incidents/0l8hyyhr6v93
Update
Our Nexcess Engineering teams have brought the majority of systems and services in our Phoenix, Arizona data center back online, and work remains actively underway to restore the remaining pending services.
Over the past several hours, we have observed continued stability across all restored systems with no red flags identified. Out of an abundance of caution, our teams are continuing to perform ongoing checks and closely monitor facility environmental conditions alongside overall infrastructure health.
We will provide further updates in this space as restore and verification efforts progress.
Over the past several hours, we have observed continued stability across all restored systems with no red flags identified. Out of an abundance of caution, our teams are continuing to perform ongoing checks and closely monitor facility environmental conditions alongside overall infrastructure health.
We will provide further updates in this space as restore and verification efforts progress.
Update
Our Nexcess Engineering teams have successfully brought the majority of systems and services in our Phoenix, Arizona data center back online, and work continues actively to restore the remaining pending services.
For all restored systems, our teams are conducting thorough checks across the infrastructure to confirm complete stability and ensure there is no ongoing operational impact. Out of an abundance of caution, we continue to closely monitor facility environmental conditions and overall infrastructure health.
We will provide further updates in this space as restoration and verification efforts progress.
For all restored systems, our teams are conducting thorough checks across the infrastructure to confirm complete stability and ensure there is no ongoing operational impact. Out of an abundance of caution, we continue to closely monitor facility environmental conditions and overall infrastructure health.
We will provide further updates in this space as restoration and verification efforts progress.
Update
Our Nexcess Engineering teams continue working to restore all remaining systems and services in our Phoenix, Arizona data center. While significant progress has been made and many systems have returned to service, our teams are continuing to bring remaining infrastructure back online in a controlled and careful manner.
As systems and network services are restored, our Engineering and Operations teams are monitoring them very closely to confirm they remain stable and continue operating as expected. We are also actively watching environmental conditions and infrastructure health throughout the facility to reduce the risk of additional service disruption, hardware damage, or data loss.
Work will continue until all affected systems and services have been fully restored. Our Engineering and Leadership teams remain actively engaged, and we will continue to provide updates as restoration efforts progress and overall service stability is confirmed.
As systems and network services are restored, our Engineering and Operations teams are monitoring them very closely to confirm they remain stable and continue operating as expected. We are also actively watching environmental conditions and infrastructure health throughout the facility to reduce the risk of additional service disruption, hardware damage, or data loss.
Work will continue until all affected systems and services have been fully restored. Our Engineering and Leadership teams remain actively engaged, and we will continue to provide updates as restoration efforts progress and overall service stability is confirmed.
Monitoring
Internal ambient temperatures in our Phoenix, Arizona data center have continued to moderate and onsite Nexcess Engineers have begun to restore service to networking equipment and other services. Our Senior Engineering and Leadership teams are actively monitoring the situation and will continue to protect customer infrastructure and reduce the risk of hardware damage or data loss where necessary, however at this time customers should begin to see network service restoring to their environments.
Update
Dear Valued Customer,
We are continuing to work closely with our provider as they address the elevated temperatures at the Phoenix facility. Progress is being made, and temperatures are continuing to come down as cooling systems are restored and additional mitigation efforts are put in place.
We apologize for the disruption and appreciate your patience as we work to protect your infrastructure and restore normal operations. We will continue to provide updates as the situation progresses.
For the latest status and updates from the facility, please visit:
https://status.phoenixnap.com/incidents/dnjp5pfv2mmh
Sincerely,
Nexcess Support Team
We are continuing to work closely with our provider as they address the elevated temperatures at the Phoenix facility. Progress is being made, and temperatures are continuing to come down as cooling systems are restored and additional mitigation efforts are put in place.
We apologize for the disruption and appreciate your patience as we work to protect your infrastructure and restore normal operations. We will continue to provide updates as the situation progresses.
For the latest status and updates from the facility, please visit:
https://status.phoenixnap.com/incidents/dnjp5pfv2mmh
Sincerely,
Nexcess Support Team
Update
Dear Valued Customer,
Update: Our teams are continuing to work closely with our data center provider at the Phoenix location. Our data center teams and on-site vendors continue working to restore adequate cooling.
As part of our ongoing mitigation efforts, we are gracefully shutting down impacted systems where necessary to help protect customer data and reduce the risk of data corruption or hardware-related issues while temperatures remain elevated.
We apologize for the disruption and appreciate your patience as we work to protect your infrastructure and restore normal operations.
We will continue to provide updates as the situation progresses.
Sincerely,
Nexcess Support Team
Update: Our teams are continuing to work closely with our data center provider at the Phoenix location. Our data center teams and on-site vendors continue working to restore adequate cooling.
As part of our ongoing mitigation efforts, we are gracefully shutting down impacted systems where necessary to help protect customer data and reduce the risk of data corruption or hardware-related issues while temperatures remain elevated.
We apologize for the disruption and appreciate your patience as we work to protect your infrastructure and restore normal operations.
We will continue to provide updates as the situation progresses.
Sincerely,
Nexcess Support Team
Identified
Dear Valued Customer,
We are currently responding to a critical facilities issue affecting our Phoenix, Arizona data center.
A major chiller equipment failure at the facility has significantly reduced cooling capacity, resulting in elevated temperatures and an increased risk of service disruption and potential hardware impact.
The data center provider is actively working to restore normal cooling operations. Replacement work is underway, with current estimates indicating repairs may take 10–16 hours or longer.
Additional information regarding the facility incident is available on the data center provider's status page:
https://status.phoenixnap.com/incidents/dnjp5pfv2mmh
Our teams are closely monitoring conditions and working directly with the data center provider to minimize customer impact. To protect customer infrastructure and reduce the risk of hardware damage or data loss, we may need to perform controlled shutdowns of affected systems if facility temperatures reach unsafe levels. Where necessary, we will prioritize orderly shutdown procedures to protect the underlying infrastructure.
Customers with services hosted within the affected Phoenix facility should prepare for potential service interruptions while remediation efforts continue.
We understand the seriousness of this situation and the impact an extended disruption may have on your business. We will continue to provide updates as conditions change and additional information becomes available. We sincerely apologize for the disruption and appreciate your patience as we work to protect your infrastructure and restore normal operations.
Sincerely,
Nexcess Support Team
We are currently responding to a critical facilities issue affecting our Phoenix, Arizona data center.
A major chiller equipment failure at the facility has significantly reduced cooling capacity, resulting in elevated temperatures and an increased risk of service disruption and potential hardware impact.
The data center provider is actively working to restore normal cooling operations. Replacement work is underway, with current estimates indicating repairs may take 10–16 hours or longer.
Additional information regarding the facility incident is available on the data center provider's status page:
https://status.phoenixnap.com/incidents/dnjp5pfv2mmh
Our teams are closely monitoring conditions and working directly with the data center provider to minimize customer impact. To protect customer infrastructure and reduce the risk of hardware damage or data loss, we may need to perform controlled shutdowns of affected systems if facility temperatures reach unsafe levels. Where necessary, we will prioritize orderly shutdown procedures to protect the underlying infrastructure.
Customers with services hosted within the affected Phoenix facility should prepare for potential service interruptions while remediation efforts continue.
We understand the seriousness of this situation and the impact an extended disruption may have on your business. We will continue to provide updates as conditions change and additional information becomes available. We sincerely apologize for the disruption and appreciate your patience as we work to protect your infrastructure and restore normal operations.
Sincerely,
Nexcess Support Team
Investigating
We are currently investigating network issues affecting servers in our Phoenix Data Center. During this time, you may experience connectivity disruptions or degraded performance.
Our engineers are actively working to identify the root cause and restore service stability as quickly as possible.
If you have any questions or require assistance, please contact us via chat or open a support case through portal.liquidweb.com
We appreciate your patience and understanding while we work to resolve this issue. Thank you.
Our engineers are actively working to identify the root cause and restore service stability as quickly as possible.
If you have any questions or require assistance, please contact us via chat or open a support case through portal.liquidweb.com
We appreciate your patience and understanding while we work to resolve this issue. Thank you.
This incident affects: Network (PHX Network).