Following the Creator outage on March 14th and 15th, we’ve launched an investigation into the root causes and a company-wide effort to improve processes, specifically around how we communicate issues to our users. We’re aware of the impact this outage and the lack of timely communications has had on our customers. We’ve not experienced this level of outage in over 7 years. However, in this particular case we didn’t uphold our own standards of how we want to treat our customers. We’re sorry and we will do better. We will post a more detailed follow-up with our learnings and plans to improve in the next few weeks. Here is the summary of what we know so far.
At 08:30 CET on Monday, Mar 14, 2022 our internal monitoring systems alerted us to an outage that was affecting our ability to serve web content with Creator. This coincided with a distributed denial of service (DDoS) against our web delivery services, which is a fairly common occurrence. Usually, those DDoS attacks are handled automatically by our systems, on some occasions we have to intervene manually and follow mitigation procedures. In this instance, however, using our normal mitigation efforts to handle the DDoS actually caused the situation to worsen. There was no impact on the security of customers’ data or websites at any time.
We spent the better part of the day and night trying to understand the cause and eliminate dependencies. Finally, on Tuesday, March 15th around 11:00 CET, we found one of our logging and monitoring vendors’ software was not functioning correctly and prevented new servers from being provisioned. We worked closely with the vendor to mitigate the issue and our systems began to return to an operational state. We closely monitored the situation until we had full confidence the issue was resolved and no further impact would occur. We are proceeding to investigate with our vendor to address the root cause of the software issue.
Thank you for giving us the space to do our best to continue supporting your businesses to thrive.