mastodon.social - Timelines are not updating with some clients – Incident details

All systems operational

Timelines are not updating with some clients

Resolved
Operational
Started over 1 year agoLasted 193 hours 44 minutes

Affected

Website & API

Degraded performance from 12:55 PM to 5:06 PM, Operational from 5:06 PM to 2:39 PM

Updates
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Monitoring
    UTC
    Monitoring

    The database software developer has been very responsive about this issue. They helped us troubleshoot it and provided us with a new version that solved the problem.

    We are currently monitoring the system closely, but everything seems to work fine now.

  • Identified
    UTC
    Identified

    Last week, we switched our Redis database to another software. This led to user reports of timelines not updating anymore with using some clients, like Ivory or Pachli.

    After investigating the issue, we discovered that this new software returns invalid data in some specific cases, which are triggered by the API those Mastodon clients use.

    We have reached out to the software developers to troubleshoot this and are hopeful it will get fixed soon. In the meantime, we are also working on a way to roll back to our previous software in case we can not get this fixed quickly.

    We will post more details on this incident when possible.