api.audd.io unavailable
Incident Report for AudD
Postmortem

It was the longest downtime we've ever experienced. We've completely resolved the issue and made the changes in the infrastructure that would prevent such events from happening in the future.

All the users of the live stream music recognition API who was affected by the issue were given 2 additional days of subscription.

About 8:30 the network of the server with the accounting DB was down. We’ve got notifications that the endpoint is down in a couple of seconds after it happened, about 8:30 AM UTC. About 8:48 we’ve identified the issue: it was caused by an ISP outage. The ISP promised to resolve the issue until 11 AM.

We've been able to launch the api.audd.io front with a snapshot of all the tokens on a separate server. During this time, the bot and the billing haven't worked, but api.audd.io did work and accepted requests.

About 11:20 AM, the internet was restored, and we've been able to restore everything.

We made changes in the infrastructure to ensure that a complete outage of any datacenter won't affect our service that much.

Posted Jul 21, 2020 - 14:26 UTC

Resolved
We made changes in the infrastructure that would prevent such events from happening.
Posted Jul 21, 2020 - 12:16 UTC
Update
The outage partially affected the streams recognition callbacks.
Posted Jul 21, 2020 - 09:48 UTC
Identified
The issue has been identified. One of our ISPs has an outage. They promised to resolve the issue until 11 AM UTC.
We've been able to run the api.audd.io on a different server. The bot is still not working.
Posted Jul 21, 2020 - 09:29 UTC
Investigating
We are currently investigating this issue.
Posted Jul 21, 2020 - 08:30 UTC
This incident affected: Audio streams, Website and API (api.audd.io).