Cloud Block Storage Service Degradation
Incident
Report for Liquid Web - Services
Resolved
This incident has been resolved.
Update
Our teams are continuing to work re-enabling CBS volumes where they were previously disabled. There are minimal ongoing effects at this time. Please reach out if you notice any ongoing issues, our Support teams are here to help.
Your understanding and patience are greatly appreciated as we finalize this process. We will continue to provide updates and remain grateful for your trust.
Your understanding and patience are greatly appreciated as we finalize this process. We will continue to provide updates and remain grateful for your trust.
Monitoring
The backfill for our CEPH storage system upgrade has completed at this time. The precise ETA on full resolution still uncertain, however we are working currently on re-enabling CBS volumes where they were previously disabled.
Your understanding and patience are greatly appreciated as we finalize this process. We will continue to provide updates and remain grateful for your trust.
Thank you for your ongoing patience as we work diligently to resolve this matter.
Your understanding and patience are greatly appreciated as we finalize this process. We will continue to provide updates and remain grateful for your trust.
Thank you for your ongoing patience as we work diligently to resolve this matter.
Update
We're making steady progress toward completing our ongoing CEPH storage system upgrade. The vast majority of issues have been successfully resolved, with only a small percentage of backfill remaining.
Please note that the final stages are progressing at a slower rate. Although we're close to completion, the precise ETA is still uncertain due to the complexities involved. Rest assured, however, that resolution is on the horizon.
If you're experiencing heavy loads and your CBS drive is connected, we recommend disconnecting it and keeping it offline until the backfill operation is complete. Our Support team stands ready to assist with this or any other concerns you may have.
Your understanding and patience are greatly appreciated as we finalize this process. We will continue to provide updates and remain grateful for your trust.
Thank you for your ongoing patience.
Please note that the final stages are progressing at a slower rate. Although we're close to completion, the precise ETA is still uncertain due to the complexities involved. Rest assured, however, that resolution is on the horizon.
If you're experiencing heavy loads and your CBS drive is connected, we recommend disconnecting it and keeping it offline until the backfill operation is complete. Our Support team stands ready to assist with this or any other concerns you may have.
Your understanding and patience are greatly appreciated as we finalize this process. We will continue to provide updates and remain grateful for your trust.
Thank you for your ongoing patience.
Update
We're making significant progress on our ongoing CEPH storage system upgrade. Currently, we've completed approximately 97.1% of the backfill operation. We're aggressively pushing to complete the remaining 2.9% as quickly as possible. However, we are unable to provide an exact ETA at this moment.
While this issue only affects a small proportion of our clients, it may cause intermittent heavy load on the affected servers as the maintenance proceeds. If you're experiencing high loads and your CBS drive is connected, we recommend disconnecting it and leaving it offline until the backfill operation is completed and our engineering team finalizes the ongoing maintenance.
If you're facing this scenario, don't hesitate to contact our Support team so we can assist you with your CBS drive. We appreciate your understanding and patience as we work towards fully resolving this situation.
Thanks again for your trust in us. We'll continue to provide updates as we make further progress.
While this issue only affects a small proportion of our clients, it may cause intermittent heavy load on the affected servers as the maintenance proceeds. If you're experiencing high loads and your CBS drive is connected, we recommend disconnecting it and leaving it offline until the backfill operation is completed and our engineering team finalizes the ongoing maintenance.
If you're facing this scenario, don't hesitate to contact our Support team so we can assist you with your CBS drive. We appreciate your understanding and patience as we work towards fully resolving this situation.
Thanks again for your trust in us. We'll continue to provide updates as we make further progress.
Update
We continue to experience disruption due to issues with our ongoing CEPH storage systems upgrade, which has impacted the performance of some Cloud Block Storage (CBS) devices. This resulted in some instances becoming unresponsive and temporary inaccessibility of cPanel/Plesk backup drives.
We've resolved most issues and actively monitor CEPH to ensure smooth operation. Measures are underway to prevent future occurrences, including accelerating the backfill operation, investigating known bugs, engaging the CEPH community for solutions, and rectifying configuration inconsistencies on 3 of our 72 storage nodes.
Our commitment to your service quality and reliability remains strong. We appreciate your patience and understanding and will provide further updates as we progress. Thanks for your trust and patience.
We've resolved most issues and actively monitor CEPH to ensure smooth operation. Measures are underway to prevent future occurrences, including accelerating the backfill operation, investigating known bugs, engaging the CEPH community for solutions, and rectifying configuration inconsistencies on 3 of our 72 storage nodes.
Our commitment to your service quality and reliability remains strong. We appreciate your patience and understanding and will provide further updates as we progress. Thanks for your trust and patience.
Update
Our Engineering teams are continuing to work towards resolving the remaining issues and improving overall performance.
Thank you for your patience as we work to resolve this issue. We will update this status with any additional details.
Thank you for your patience as we work to resolve this issue. We will update this status with any additional details.
Update
Our Engineering team continues to work on resolving the remaining issues. Additional updates will be provided as new information becomes available. Thank you for your continued patience.
Identified
Our Engineers have implemented a fix for Cloud Block Storage. Unfortunately at this time there are still a few customers still being affected.
If you have any questions, please contact a member of our support team via live-chat, ticket, or by phone at (800)-580-4985, (517)-322-0434 (international).
If you have any questions, please contact a member of our support team via live-chat, ticket, or by phone at (800)-580-4985, (517)-322-0434 (international).
Monitoring
A fix has been implemented and service has been restored to our Cloud Block Storage. At this time we are currently monitoring the results.
Update
We are continuing to investigate this issue.
Update
We are continuing to investigate this issue.
Investigating
At this time our team is investigating an issue with our Cloud Block Storage service. Additional information will be provided as soon as it is available.
If you have any questions, please contact a member of our support team via live-chat, ticket, or by phone at (800)-580-4985, (517)-322-0434 (international).
If you have any questions, please contact a member of our support team via live-chat, ticket, or by phone at (800)-580-4985, (517)-322-0434 (international).