It was the longest downtime we've ever experienced. We've completely resolved the issue and made the changes in the infrastructure that would prevent such events from happening in the future.
All the users of the live stream music recognition API who was affected by the issue were given 2 additional days of subscription.
About 8:30 the network of the server with the accounting DB was down. We’ve got notifications that the endpoint is down in a couple of seconds after it happened, about 8:30 AM UTC. About 8:48 we’ve identified the issue: it was caused by an ISP outage. The ISP promised to resolve the issue until 11 AM.
We've been able to launch the api.audd.io front with a snapshot of all the tokens on a separate server. During this time, the bot and the billing haven't worked, but api.audd.io did work and accepted requests.
About 11:20 AM, the internet was restored, and we've been able to restore everything.
We made changes in the infrastructure to ensure that a complete outage of any datacenter won't affect our service that much.