mastodon.social - Connectivity/timeout issues with sidekiq – Incident details

All systems operational

Connectivity/timeout issues with sidekiq

Monitoring
Operational
Started about 2 months ago

Affected

Website & API

Degraded performance from 2:00 PM to 6:42 AM, Operational from 6:42 AM to 12:00 AM

Background queues

Degraded performance from 2:00 PM to 6:42 AM, Operational from 6:42 AM to 12:00 AM

Updates
  • Monitoring
    Monitoring

    We found out the cause for this issue (within a dependency) and we deployed a fix 8 hours ago. So far, it seems to have completely fixed the issue.

  • Investigating
    Investigating

    There appears to be an issue with connection timeouts to our Sidekiq runners, causing some operations to take longer than expected, or require multiple tries. We are currently investigating this issue.