Degraded Performance on NetApp based products and services
Incident
Report for Liquid Web - Services
Postmortem
If you would like further information please contact a member of our support team via live-chat, ticket, or by phone at (800)-580-4985, (517)-322-0434 (international).
Resolved
This incident is resolved.
Update
Thank you for your patience with us as we have been working to bring our Cloud VPN service back online. This is now available again for use and includes an added 2FA option that can be enabled per user. If you are still seeing any issues please let our support team know.
Update
We are continuing to monitor for any further issues.
Update
Our engineers continue to work to re-establish our Cloud VPN functionality. We recognize that the extended outage is impacting your business, and we sincerely apologize. At this time, we expect to have our VPN functional by Wednesday, June 26th. If you need an alternative solution before this timeline, we are happy to meet with you to discuss more immediate options.
Following the NetApp storage outage, we have completed a thorough evaluation of our infrastructure. As part of our evaluation, we are hardening our systems and implementing industry best practices in a number of areas. We recognize that this doesn’t take away the challenges that you have experienced this past week, but know that we are committed to aggressive strategies to prevent future incidents. In conjunction with our forthcoming root cause analysis (RCA), we are also taking steps to make our environments more stable, performant, and secure.
If you require assistance with changes prior to when our Cloud VPN service is re-enabled, please reach out to our 24/7 support team. Thank you for your continued patience.
Following the NetApp storage outage, we have completed a thorough evaluation of our infrastructure. As part of our evaluation, we are hardening our systems and implementing industry best practices in a number of areas. We recognize that this doesn’t take away the challenges that you have experienced this past week, but know that we are committed to aggressive strategies to prevent future incidents. In conjunction with our forthcoming root cause analysis (RCA), we are also taking steps to make our environments more stable, performant, and secure.
If you require assistance with changes prior to when our Cloud VPN service is re-enabled, please reach out to our 24/7 support team. Thank you for your continued patience.
Update
On Monday, June 10, 2024, there was a service-impacting outage affecting a limited set of customers utilizing our NetApp service, which includes VMware, CloudSites, Cloud Load Balancers, Cloud VPN, and SAN-based services. Upon discovering the NetApp storage array(s) were inaccessible, our team launched an investigation and began working on a swift resolution. We immediately engaged the NetApp North America leadership and their engineering teams to assist us. By Monday evening, we restored service to an initial set of customers, with the majority up and running by the next morning.
This week, our focus has been to ensure all customer issues are resolved and services fully restored. While we’ve made significant progress, we are still addressing some remaining issues and intermittent disruptions, and we remain vigilant to keep systems online and stable.
Our investigation into the root cause of the outage is ongoing. While this comprehensive analysis will take time, it is crucial for developing strategies to prevent incidents in the future. We expect to conclude the investigation in the coming weeks.
This product is known for its performance and stability, which continues to be one of the main reasons customers choose this platform with us. In fact, we have experienced very high NetApp performance since the inception, with very limited and sparse outages. Our confidence continues to be high in our ability to provide continued high performance and stability for this product. Our teams are working diligently to deliver exactly that for our customers.
We sincerely apologize for the inconvenience this disruption has caused and thank you for your patience and understanding as we work to regain your trust. Your business is important to us, and we are committed to ensuring you have the stable and reliable environment you deserve.
If you are still experiencing any further issues with your services, please do not hesitate to contact a member of our support team via live-chat, ticket, email (support@liquidweb.com), or by phone at (800)-580-4985, (517)-322-0434 (international).
This week, our focus has been to ensure all customer issues are resolved and services fully restored. While we’ve made significant progress, we are still addressing some remaining issues and intermittent disruptions, and we remain vigilant to keep systems online and stable.
Our investigation into the root cause of the outage is ongoing. While this comprehensive analysis will take time, it is crucial for developing strategies to prevent incidents in the future. We expect to conclude the investigation in the coming weeks.
This product is known for its performance and stability, which continues to be one of the main reasons customers choose this platform with us. In fact, we have experienced very high NetApp performance since the inception, with very limited and sparse outages. Our confidence continues to be high in our ability to provide continued high performance and stability for this product. Our teams are working diligently to deliver exactly that for our customers.
We sincerely apologize for the inconvenience this disruption has caused and thank you for your patience and understanding as we work to regain your trust. Your business is important to us, and we are committed to ensuring you have the stable and reliable environment you deserve.
If you are still experiencing any further issues with your services, please do not hesitate to contact a member of our support team via live-chat, ticket, email (support@liquidweb.com), or by phone at (800)-580-4985, (517)-322-0434 (international).
Monitoring
Services have been restored for most of our impacted customers; however, we are seeing intermittent issues as we continue to address restoration activities.
We continue to monitor for any further problems. If you are still experiencing issues, please contact our support team via live chat, ticket, or by phone at (800)-580-4985 or (517)-322-0434 (international).
We value your business, and apologize for the problems that have occurred. We recognize the serious nature of this incident and are working diligently to mitigate future issues.
We continue to monitor for any further problems. If you are still experiencing issues, please contact our support team via live chat, ticket, or by phone at (800)-580-4985 or (517)-322-0434 (international).
We value your business, and apologize for the problems that have occurred. We recognize the serious nature of this incident and are working diligently to mitigate future issues.
Update
Our team is actively collaborating with the NetApp North America leadership and senior engineers to address the current issues with our NetApp-based products and services. We expect to be nearing a resolution path by mid-morning. We will keep you informed with updates on our progress and aim to provide a more detailed update at that time.
Update
Our team is continuing to work with the NetApp North America leadership team and senior engineers to address the current situation involving our NetApp-based products and services. We will continue to provide updates on any progress we have made and aim to provide a more detailed update in the morning.
Update
We continue to work closely with the NetApp North America leadership team and senior engineers to address the current situation involving our NetApp-based products and services.
We'll continue to provide updates throughout the evening and aim to provide a more detailed update tomorrow morning.
We'll continue to provide updates throughout the evening and aim to provide a more detailed update tomorrow morning.
Update
Earlier today, we began investigating degraded performance and were able to isolate the issue to some areas of our NetApp-based products and services, which include VMware, CloudSites, and SAN-based services. We have engaged the NetApp North America leadership team and have multiple active bridges with senior NetApp engineering teams working towards a resolution.
We will continue to post updates throughout the evening. At this point, we aim to provide a more detailed update tomorrow morning. Please know that our team is dedicated to resolving this issue and will be working nonstop until it is resolved.
We will continue to post updates throughout the evening. At this point, we aim to provide a more detailed update tomorrow morning. Please know that our team is dedicated to resolving this issue and will be working nonstop until it is resolved.
Update
We are diligently working to resolve the issue affecting VMware and SAN-based services. Rest assured, our team is fully committed to finding a resolution as soon as possible.
At this time, we do not have an estimated restoration time for the affected services, but we will provide updates as soon as more information becomes available.
If you are experiencing a separate issue, please note that there may be delays, and we will address requests as soon as possible.
At this time, we do not have an estimated restoration time for the affected services, but we will provide updates as soon as more information becomes available.
If you are experiencing a separate issue, please note that there may be delays, and we will address requests as soon as possible.
Update
We are continuing our work on resolving the issue affecting VMware, Shared Load Balancer, and SAN-based services. Our team is making progress in identifying and pursuing a resolution path, investigating all potential solutions to ensure our customers are back online and running smoothly.
We will keep you updated on our progress as soon as information becomes available. Please be assured that bringing our customers back online is our number one priority.
If you are experiencing a separate issue, please note that we are experiencing significant delays and will address requests as soon as possible.
We will keep you updated on our progress as soon as information becomes available. Please be assured that bringing our customers back online is our number one priority.
If you are experiencing a separate issue, please note that we are experiencing significant delays and will address requests as soon as possible.
Update
We are actively working on resolving the issue affecting VMware, Shared Load Balancer, and SAN-based services. Our team is making progress in identifying and pursuing a resolution path, investigating all potential solutions to ensure our customers are back online and running smoothly.
We will keep you updated on our progress as soon as information becomes available. Please be assured that bringing our customers back online is our number one priority.
If you are experiencing a separate issue, please note that we are experiencing significant delays and will address requests as soon as possible.
We will keep you updated on our progress as soon as information becomes available. Please be assured that bringing our customers back online is our number one priority.
If you are experiencing a separate issue, please note that we are experiencing significant delays and will address requests as soon as possible.
Update
Systems and Network Engineering Teams are continuing to work through this issue. Your understanding and cooperation during this period are greatly appreciated as the teams work diligently to restore normal service. Should you have any inquiries, please do not hesitate to reach out to a member of our support team via live-chat, ticket, or by phone at (800)-580-4985, (517)-322-0434 (international).
Update
Systems and Network Engineering Teams are still working to resolve this issue. Thank you for your continued patience. If you have any questions, please contact a member of our support team via live-chat, ticket, or by phone at (800)-580-4985, (517)-322-0434 (international).
Update
Systems and Network Engineering Teams continue to work on implementing a fix for this issue. Impacted services will remain interrupted until the fix is fully implemented. The scope of this incident has been redefined to include all NetApp based products and services, including VMWare, Shared Load Balancers, and SAN based services.
Thank you for your continued patience while our teams work to resolve this issue. If you have any questions, please contact a member of our support team via live-chat, ticket, or by phone at (800)-580-4985, (517)-322-0434 (international).
Thank you for your continued patience while our teams work to resolve this issue. If you have any questions, please contact a member of our support team via live-chat, ticket, or by phone at (800)-580-4985, (517)-322-0434 (international).
Identified
Systems and Network Engineering Teams have identified the source of the service impact and are working to implement a fix. Impacted services will remain interrupted until the fix is fully implemented. We will provide further updates as soon as they become available.
Thank you for your continued patience while our teams work to resolve this issue. If you have any questions, please contact a member of our support team via live-chat, ticket, or by phone at (800)-580-4985, (517)-322-0434 (international).
Thank you for your continued patience while our teams work to resolve this issue. If you have any questions, please contact a member of our support team via live-chat, ticket, or by phone at (800)-580-4985, (517)-322-0434 (international).
Update
We are continuing to investigate this.
Update
We are continuing to investigate this issue.
Update
We are continuing to investigate this issue.
Update
We are continuing to investigate this issue.
Update
We are continuing to investigate this issue.
Investigating
We are currently investigating a problem affecting VMware, CloudSites and the load balancers in our environment. Our team is examining the problem, and we will provide updates as soon as we make progress.
Thank you for your patience while we work to resolve this problem. If you have any questions, please contact a member of our support team via live-chat, ticket, or by phone at (800)-580-4985, (517)-322-0434 (international).
Thank you for your patience while we work to resolve this problem. If you have any questions, please contact a member of our support team via live-chat, ticket, or by phone at (800)-580-4985, (517)-322-0434 (international).
This incident affected: Load Balancers (Shared Load Balancers - DC2, Shared Load Balancers - DC3, Shared Load Balancers - DC4, Shared Load Balancers - DC5) and VMware Cloud, VMware Private Cloud.