The issue was traced to a partial outage in AWS. The root cause was a malfunction within a single AZ, which caused a handful of instances to erroneously be replaced by the ASG. This manifested in Websolr as indices suddenly returning HTTP 500-level errors. Users without replication additionally experienced some data loss. Further questions should be directed to firstname.lastname@example.org.
Posted 11 days ago. Mar 07, 2019 - 17:11 UTC
We are investigating some issues affecting servers in a single availability zone. We are responding to issues as they arise, and are actively investigating the root cause. Automatic recovery is happening slowly, but steadily.
Posted 11 days ago. Mar 07, 2019 - 16:03 UTC
This incident affected: Region Health: AWS US East (Virginia).