Issues on DISROOT

Maintenance Announcement

Sat, 24 Feb 2024 10:35:00 +0000

We will shut down our network at midnight UTC on Feb 24. This does not affect the current status.

[Resolved] New Pipeline Rollout

Sat, 05 Oct 2019 16:24:00 +0000

There may be disruptions in the rollout.

Testing New cState Features

Fri, 04 Oct 2019 18:05:00 +0000

There is a new feature in cState version 4 that lets you make what are called informational posts. The main difference is that there will be no Unresolved or Resolved in under a minute text on the pages.

This is essentially a page with a date and title that shows up in the incident history.

[Resolved] US East Connection Issues

Wed, 25 Apr 2018 04:13:00 +0000

Resolved - We believe all users experiencing issues have been able to connect at this time. (05:54 UTC — May 25)

Monitoring - We believe the connectivity issues are being caused by an isolated ISP issue. We’ve had reports that swapping to Google DNS servers (see here; https://developers.google.com/speed/public-dns/docs/using) resolves the problem for users. (04:40 UTC — May 25)

Investigating - We’re aware of reports that users are experiencing connection issues on the East coast of the United States. We’re currently investigating these issues, and apologize for any inconvenience it may be causing you. (04:13 UTC — May 25)

[Resolved] Unavailable Guilds & Connection Issues

Fri, 13 Apr 2018 15:54:00 +0000

Post-mortem

At approximately 14:01, a Redis instance acting as the primary for a highly-available cluster used by our API services was migrated automatically by Google’s Cloud Platform. This migration caused the node to incorrectly drop offline, forcing the cluster to rebalance and trigger known issues with the way our API instances handle Redis failover. After resolving this partial outage, unnoticed issues on other services caused a cascading failure through Example Chat App’s real time system. These issues caused enough critical impact that Example Chat App’s engineering team was forced to fully restart the service, reconnecting millions of clients over a period of 20 minutes.

Update - A fix has been implemented and we are monitoring the results. Looks like this has been fixed. (17:30 UTC — Apr 13)

Monitoring - After hitting the ole reboot button Example Chat App is now recovering. We’re going to continue to monitor as everyone reconnects. (16:50 UTC — Apr 13)

Investigating - We’re aware of users experiencing unavailable guilds and issues when attempting to connect. We’re currently investigating. (15:54 UTC — Apr 13)

[Resolved] Issues Sending DMs

Sun, 17 Dec 2017 16:24:00 +0000

Update - This incident has been resolved.

Investigating - We’re aware of an issue affecting sending DMs and viewing online friends. We’re online and working on a resolution.