Resolved -
This incident is resolved.
Jul 10, 20:09 UTC
Update -
We are continuing to monitor for any further issues.
Jul 10, 20:08 UTC
Monitoring -
Request latency has recovered and we are continuing to monitor the situation
Jul 10, 16:27 UTC
Investigating -
We are aware of the issue, have identified the root cause, and are actively mitigating impact during spikes while working toward a resolution.
Jul 10, 15:17 UTC
Resolved -
All async jobs are now processing normally
Jul 6, 11:53 UTC
Monitoring -
Between approximately 3:00am and 4:15am ET on July 6th, a percentage of in-flight async jobs failed due to an error in our job throttling system. We mitigated the issue by disabling throttling at ~4:15am ET, and no further failures have occurred since. Failed jobs will need to be resubmitted. Job processing may be delayed over the next hour as the system catches up. Throttling will remain disabled until the underlying issue is identified and fixed. We'll post a follow-up once that work is complete.
Jul 6, 08:32 UTC