mastodon.social - Notice history

Early morning Feb 24 we were indirectly made aware of a misconfiguration on our object storage domain (files.mastodon.social) that allowed anyone to see the list of all uploaded files. Within 30 minutes this mistake was corrected. However, we have reasons to believe that the issue has existed since Feb 2, when we began upgrading our infrastructure. Normally Mastodon relies on long, randomly generated file names with high entropy to ensure that certain files are accessed only by those who know the link. However, that misconfiguration allowed that measure to be bypassed. Most files in our object storage are public in nature–profile pictures, custom emojis, images and videos attached to public posts. But there is a type of file that should never be accessed by anyone but its owner, and it’s the user’s archive takeout. When a user requests an archive takeout of their account, all of their posts, favorites and bookmarks are put into an archive which remains accessible for 7 days after. At the time of the incident, 5000 such user archive takeouts were available. We have immediately deleted them to prevent anyone from downloading them. We have notified the affected users and we are changing the Mastodon software to not rely on high entropy links for access control to user archive takeouts any longer, as well as adding an automated check into the admin dashboard to detect similar misconfigurations and notify other server operators about them. Security is important to us and we are continuously improving our processes as we scale our organization from one employee to multiple to ensure that mistakes like this do not happen in the future.

There is an issue in our Kubernetes cluster that we are working to resolve.

We are currently investigating this incident.

The website is currently experiencing abnormal error rates.

We are identified the issue and are working on resolving it.

mastodon.social is currently unavailable.

We are currently investigating this incident.

We've identified a delay in processing incoming posts (i.e. posts from other servers in the fediverse).

Additional compute power is being deployed to process them and get back to normal.

We are currently investigating large bursts of traffic that cause our systems to slow down.

We've identified a delay in processing incoming posts (i.e. posts from other servers in the fediverse) and are working on a mitigation.