Increased platform error rates
Incident Report for Living Security Platform
Resolved
This incident has been resolved.
Posted Dec 08, 2021 - 04:18 UTC
Update
We are continuing to monitor for any further issues.
Posted Dec 07, 2021 - 23:39 UTC
Update
The root cause of the outage has been mitigated and we are seeing operational performance return to normal. At this time there may be an increased delay of events delivery within the system but should not materially impact end-users.
Posted Dec 07, 2021 - 22:59 UTC
Update
We are still experiencing infrastructure issues which are impacting site availability. We continue to monitor the situation with Amazon Web Services and will provide more information as it becomes available.
Posted Dec 07, 2021 - 22:35 UTC
Update
The root cause of this issue is an impairment of several network devices in the US-EAST-1 Region of AWS. Mitigation efforts continue to be underway.
Posted Dec 07, 2021 - 19:38 UTC
Update
At this time Amazon Web Services indicates they are actively working to restore service, but do not have an ETA of restoration. We are continuing to monitor the situation.

In lieu of new information, we will provide our next update in 1 hour. approx 1:30PM CT
Posted Dec 07, 2021 - 18:29 UTC
Update
We continue to experience an outage due to an outage across Amazon Web Services. At this time Amazon is reporting degraded ability to provide updates to customers.

We are continuing to monitor the situation and will update you in 30 minutes at approx. 12:30PM CT
Posted Dec 07, 2021 - 17:56 UTC
Update
Since our last update, the scope of the Amazon Web Service outage has increased and we continue to experience an outage in both Teams and Training products.

We have added a popup on the sites to notify end users that we are currently experiencing an outage.

We are continuing to monitor the situation and will provide updates again in 30 minutes approx. 11:45 AM CT.
Posted Dec 07, 2021 - 17:19 UTC
Monitoring
Service appears to be restoring, however the issue has not officially been resolved by Amazon Web Services, so we continue to maintain our status of partial outage at this time. We will provide updates again in 30 minutes. Approx. 11:15 CT
Posted Dec 07, 2021 - 16:46 UTC
Update
Amazon Web Services is reporting increased error rates. Subsequently our partner vendors are also experiencing issues. At this time we are currently experiencing an outage in both our Teams and Training products.

We are continuing to investigate and will provide our next update in 15 minutes approx 11:45 CT.
Posted Dec 07, 2021 - 16:33 UTC
Investigating
We are currently investigating elevated error rates within the platform. We will update every 15 minutes as we learn more information.
Posted Dec 07, 2021 - 16:14 UTC
This incident affected: Video Delivery (Video Player, HLS Media Delivery), Content Delivery (Assets CDN, Content Delivery API), Infrastructure (API Infrastructure, Core Storage), Authentication (User Authentication, User Provisioning), Notifications (SMTP Delivery, Mail API, Notification Service), and User Management, Campaign Management, SCIM API.