A fix has been implemented and we are monitoring the results.
Posted May 27, 2022 - 12:10 CEST
Update
We are currently recovering built-up tasks and are processing the backlog of tasks succesfully.
Posted May 27, 2022 - 10:20 CEST
Update
We have rebuild our queue servers, and are switching new tasks over to the new infrastructure.
Posted May 27, 2022 - 09:57 CEST
Identified
The issue has been identified and a fix is being implemented.
Posted May 27, 2022 - 08:24 CEST
Monitoring
We are still monitoring, due to our throughput being below normal levels.
Posted May 26, 2022 - 14:32 CEST
Update
We are continuing to work on this issue. Synchronous generation and high-priority/user facing tasks are fine at the moment; bulk tasks are being processed succesfully albeit on a lower rate (resulting in longer timings) than usual.
Posted May 26, 2022 - 14:03 CEST
Identified
We have started short-term mitigations by removing the load peak to recover to a stable operational level.
Posted May 26, 2022 - 13:22 CEST
Investigating
Our rendering and processing infrastructure is running behind due to high load, which is causing delay in content delivering. No data has been lost and we are trying to adapt the infrastructure.
Posted May 26, 2022 - 12:48 CEST
This incident affected: NLG Cloud (Text Engine, NEXT (v3)).