Gateway

Degraded Genius Performance During Black Friday

Resolvido
Avaliado

Incident Summary

Incident: Degraded Genius Performance During Black Friday

Incident Date: November 29, 2019 12:15 EST - 19:15 EST

Affected Services: Genius Transactions, Store and Forward


Incident Description

Due to increased volume on Black Friday, starting around 12:15 EST, all Genius merchants may have experienced intermittent increases in their transaction processing times, as well as an increased number of transactions being processed “offline” by Store and Forward.

Client Impact

During the incident, all Genius merchants may have experienced intermittent increases in their transaction processing times. The issue caused a larger than average number of Store and Forward enabled workstations to go into “offline mode” temporarily as a protective measure.


Resolution and Next Steps

The team increased overall transactional capacity in each data center by adding additional application and web servers. The team is also performing an investigation into the performance of key areas within the Genius services while making significant improvements to the tools and models for all future stress and performance tests preventing future incidents.