mastodon.social - Notice history

All systems operational

Website & API - Operational

99.94% uptime
Nov 2024 · 100.0%Dec · 100.0%Jan 2025 · 99.83%

Previous page

Next page

Nov 2024
Dec 2024
Jan 2025

Background queues - Operational

99.95% uptime
Nov 2024 · 100.0%Dec · 100.0%Jan 2025 · 99.86%

Previous page

Next page

Nov 2024
Dec 2024
Jan 2025

Media storage - Operational

99.95% uptime
Nov 2024 · 100.0%Dec · 100.0%Jan 2025 · 99.86%

Previous page

Next page

Nov 2024
Dec 2024
Jan 2025

Streaming API - Operational

99.95% uptime
Nov 2024 · 100.0%Dec · 100.0%Jan 2025 · 99.86%

Previous page

Next page

Nov 2024
Dec 2024
Jan 2025

Notice history

View current status

Jan 2025

Degraded performance
ResolvedDegraded performance284 hours 41 minutes
  • Resolved
    UTC
    Resolved

    Data has been restored, and performance issues have been mitigated.

  • Identified
    UTC
    Identified

    We've experienced an issue with our recent cache storage upgrade, causing some ephermal data loss (e.g. home feeds losing scrollback or showing the number of active users as "0"). We are working on fixing the underlying issue and restoring/regenerating the lost data.

  • Investigating
    UTC
    Investigating
    We are currently investigating this incident.
Timelines are not updating with some clients
Resolved193 hours 44 minutes
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Monitoring
    UTC
    Monitoring

    The database software developer has been very responsive about this issue. They helped us troubleshoot it and provided us with a new version that solved the problem.

    We are currently monitoring the system closely, but everything seems to work fine now.

  • Identified
    UTC
    Identified

    Last week, we switched our Redis database to another software. This led to user reports of timelines not updating anymore with using some clients, like Ivory or Pachli.

    After investigating the issue, we discovered that this new software returns invalid data in some specific cases, which are triggered by the API those Mastodon clients use.

    We have reached out to the software developers to troubleshoot this and are hopeful it will get fixed soon. In the meantime, we are also working on a way to roll back to our previous software in case we can not get this fixed quickly.

    We will post more details on this incident when possible.

Website inaccessible
ResolvedPartial outage30 minutes
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Investigating
    UTC
    Investigating
    We are currently investigating this incident.
JAN21
Database upgrade
CompletedMaintenance2 hours 44 minutes
  • Completed
    January 21, 2025 at 11:44 AMUTC
    Completed
    January 21, 2025 at 11:44 AMUTC

    Database is successfully upgraded, all configuration errors have been cleaned up. Thank you to everyone for your patience.

  • Update
    January 21, 2025 at 11:00 AMUTC
    Update
    January 21, 2025 at 11:00 AMUTC

    The upgrade was successful and mastodon.social is now running on the newer database servers.

    There is a small delay in processing due to the backlog of request accumulated during our downtime, but everything should be back to real-time soon.

    We are closely monitoring the servers and will update this status once everything is back to normal.

  • Update
    January 21, 2025 at 9:57 AMUTC
    Update
    January 21, 2025 at 9:57 AMUTC

    We encountered a configuration issue while performing the update. We are working on a fix and the service should be back online soon.

  • Update
    January 21, 2025 at 9:31 AMUTC
    Update
    January 21, 2025 at 9:31 AMUTC
    Maintenance is now in progress.
  • Update
    January 21, 2025 at 9:00 AMUTC
    Update
    January 21, 2025 at 9:00 AMUTC
    Maintenance is now in progress
  • In progress
    January 21, 2025 at 9:00 AMUTC
    In progress
    January 21, 2025 at 9:00 AMUTC

    We will be upgrading the database systems to new hardware, as well as a new major version of PostgreSQL. The upgrade should be very quick, but some small downtime will still occur for around a minute.

Backend DB connection issues
ResolvedPartial outage2 hours 55 minutes
  • Resolved
    UTC
    Resolved

    No acknowledgement from Hetzner, but everything remains stable. Considering this closed.

  • Monitoring
    UTC
    Monitoring

    DB connections are stable again. Current suspect is a Hetzner networking issue somewhere in the backend. Currently monitoring.

  • Investigating
    UTC
    Investigating

    There appears to be an issue with our nodes connecting to the DB reliably. We are currently investigating the issue.

Dec 2024

No notices reported this month

Nov 2024

Server autoscaling/capacity issues
ResolvedDegraded performance18 hours 30 minutes
  • Resolved
    UTC
    Resolved

    Capacity issues have been resolved, and the autoscaler has been adjusted.

  • Identified
    UTC
    Identified

    There was an issue with our cluster autoscaler where it seems to have scaled down the number of nodes we have for our cluster, resulting in strained resources. We are trying to scale back up, but Hetzner is having availability issues in the region we are deployed in. We are working to resolve the issue as quickly as possible.

Previous

Nov 2024 to Jan 2025

Next