Incident: Degraded Genius Performance During Black Friday
Incident Date: November 29, 2019 12:15 EST - 19:15 EST
Affected Services: Genius Transactions, Store and Forward
Due to increased volume on Black Friday, starting around 12:15 EST, all Genius merchants may have experienced intermittent increases in their transaction processing times, as well as an increased number of transactions being processed “offline” by Store and Forward.
During the incident, all Genius merchants may have experienced intermittent increases in their transaction processing times. The issue caused a larger than average number of Store and Forward enabled workstations to go into “offline mode” temporarily as a protective measure.
The team increased overall transactional capacity in each data center by adding additional application and web servers. The team is also performing an investigation into the performance of key areas within the Genius services while making significant improvements to the tools and models for all future stress and performance tests preventing future incidents.