Performance Degradation

Minor incident Web Platform Template Editor Report Writer PDF Report Online Scheduler Metrics HTML Report Contacts Automation
2021-01-27 13:50 MDT · 20 hours, 7 minutes, 24 seconds

Updates

Resolved

We have identified the issue impacting system performance and have resolved it. We are still investigating for the root cause, but we’re happy to report our platform is stable once again!

January 28, 2021 · 09:58 MDT
Investigating

We are continuing to investigate the issue. Some details is you’re curious:

Beginning yesterday we began seeing memory spikes on our web servers without any changes prompted from our end. This results in failed requests and erratic behavior from retried processes (like template section double-ups).

This could be the result of external sources and/or bad actors, given our investigation so far and what we know. We are taking steps to alleviate the issue, spinning up additional servers and changing performance variables to run at lower memory thresholds. These will hopefully keep the system operational until we can unearth the root cause and add countermeasures to prevent it.

January 28, 2021 · 08:03 MDT
Investigating

We are seeing additional server performance issues this morning. We are currently investigating.

January 28, 2021 · 06:37 MDT
Monitoring

Systems are now back to normal. We are monitoring the situation.

January 27, 2021 · 13:57 MDT
Investigating

Currently experiencing some issues with our web servers failing to respond to some requests. We are investigating the cause.

January 27, 2021 · 13:52 MDT

← Back