Service Interruption - 2022/07/25
Summary: The interruption is solved and everything is good now. The list of actions are described below. If you have any questions, feel free to reach us.
Dear customers and users,
Carbone experienced a partial network issue this week-end (23th to 25th of July).
Everything is back on track now since 6:25am UTC, and we apologize for any inconvenience
Saturday 23th July 2022 at 05:00pm UTC-0: network issue impacting our gateway in SBG region.
Monday 25th July 2022 at 06:09am UTC-0: one client raised an issue on Carbone support
Monday 25th July 2022 at 06:27am UTC-0: everything is fixed for Carbone clients - network traffic redirected to other healthy region
Monday 25th July 2022 at 09:01am UTC-0: our cloud provider OVHCloud identified the issue https://public-cloud.status-ovhcloud.com/incidents/zz1wqhp70wk8
Monday 25th July 2022 at 10:00am UTC-0: we fixed the network traffic of SBG region using another temporary gateway.
Monday 25th July 2022 at 12:26pm UTC-0: our cloud provider OVHCloud fixed the global network issue
Monday 25th July 2022 at 01:20pm UTC-0: we restored our default architecture configuration and restored the network traffic to SBG region
Carbone Cloud services were impacted (API and Studio):
Some reports were generated with an empty image if they contain dynamic image injected with an URL
Some attempts to add/remove/delete a template was returning error 500
A network issue on our cloud provider OVHCloud in SBG region.
Our alerting system was not triggered because >95% of the service was running normally.
An evolution of our cloud architecture in planned in august to resolve automatically those interruptions.
And we will improve our alerting system using what we've learned.
We will update the advancement of actions on Twitter, Linkedin and through our newsletters (You can subscribe to our newsletter by Login to your account > click on Account tab > Check Subscribe to the Carbone Newsletter).
Feel free to reach us on the chat, the team is available to answer your questions.
Enjoy creating with Carbone, and Cheers! - The Carbone Team
Dear customers and users,
Carbone experienced a partial network issue this week-end (23th to 25th of July).
Everything is back on track now since 6:25am UTC, and we apologize for any inconvenience
Chronological events
Saturday 23th July 2022 at 05:00pm UTC-0: network issue impacting our gateway in SBG region.
Monday 25th July 2022 at 06:09am UTC-0: one client raised an issue on Carbone support
Monday 25th July 2022 at 06:27am UTC-0: everything is fixed for Carbone clients - network traffic redirected to other healthy region
Monday 25th July 2022 at 09:01am UTC-0: our cloud provider OVHCloud identified the issue https://public-cloud.status-ovhcloud.com/incidents/zz1wqhp70wk8
Monday 25th July 2022 at 10:00am UTC-0: we fixed the network traffic of SBG region using another temporary gateway.
Monday 25th July 2022 at 12:26pm UTC-0: our cloud provider OVHCloud fixed the global network issue
Monday 25th July 2022 at 01:20pm UTC-0: we restored our default architecture configuration and restored the network traffic to SBG region
Impact
Carbone Cloud services were impacted (API and Studio):
Some reports were generated with an empty image if they contain dynamic image injected with an URL
Some attempts to add/remove/delete a template was returning error 500
Root cause
A network issue on our cloud provider OVHCloud in SBG region.
Our alerting system was not triggered because >95% of the service was running normally.
Improvements
An evolution of our cloud architecture in planned in august to resolve automatically those interruptions.
And we will improve our alerting system using what we've learned.
We will update the advancement of actions on Twitter, Linkedin and through our newsletters (You can subscribe to our newsletter by Login to your account > click on Account tab > Check Subscribe to the Carbone Newsletter).
Feel free to reach us on the chat, the team is available to answer your questions.
Enjoy creating with Carbone, and Cheers! - The Carbone Team
Updated on: 07/26/2022
Thank you!