mastodon.social - Elasticsearch issues – Incident details

All systems operational

Elasticsearch issues

Resolved
Major outage
Started 2 months agoLasted about 1 hour

Affected

Website & API

Major outage from 9:24 PM to 9:46 PM, Degraded performance from 9:46 PM to 3:07 PM

Background queues

Major outage from 9:24 PM to 9:46 PM, Degraded performance from 9:46 PM to 10:26 PM

Updates
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Monitoring
    UTC
    Monitoring

    Search functionality has been restored and volume timeouts seem to have been fixed. We will continue to monitor the situation.

  • Identified
    UTC
    Identified

    The issue seems to be incredibly high timeouts on Hetzner volumes, causing components with high-throughput volumes like Elasticsearch to degrade and fail. Search has temporarily been disabled while we investigate this issue and attempt to work around it.

  • Investigating
    UTC
    Investigating

    There appears to be an issue with our Elasticsearch cluster that is causing the instance to fail. We are currently investigating the issue.