Gateway

Increased decline rate for P2PE-enabled Genius terminals

Löst
Bedömd

Incident Summary

Incident: Increased decline rate for P2PE-enabled Genius terminals

Incident Date: November 29, 2019 11:56 EDT - 19:15 EDT

Affected Services: Genius P2PE Transactions


Incident Description

Starting around 11:56 EST on November 29th, some P2PE transactions serviced by the Genius Chicago data center experienced intermittent issues connecting to our HSMs. This continued every 10 to 15 minutes until 19:15 EST. Transactions serviced by the Boston data center were unaffected.

Client Impact

The incident impacted approximately 8000 transactions and 1000 merchants. Only merchants using the P2PE version of Genius were affected. Of the 8000 affected transactions, approximately 3500 transactions received otherwise unexpected “declined” responses from their terminals, while 4500 transactions experienced delays in their ability to report on the transactions.

Timeline

  • 11:56: An increase in P2PE volume began affecting the HSM in the Chicago data center.
  • 12:43: SEV-3 incident raised to the on-call team.
  • 13:18: The NOC began investigation & remediation activities.
  • 19:15: P2PE traffic was stabilized.

Resolution and Next Steps

  • As of Wednesday, December 4, the team has doubled the number of HSMs in the Chicago data center, bringing it to parity with the Boston data center.
  • The team is continuing to increasing the number of available HSMs in both data centers to increase the available throughput and redundancy to ensure the issue does not arise in the future.
  • The work to add additional capacity is planned for completion on December 6, 2019.