Payment Gateway Services DNS Issue

Incident Report for TokenEx

Resolved

TokenEx leverages a DNS resolution service provided by our IaaS provider. This service started experiencing intermittent outages causing the app servers supporting the Payment Services API to experience DNS resolution failures. Because of the intermittent nature of the outage, the threshold to alert the TokenEx Operations team was not reached at the onset of the issue.

The intermittent DNS issues causing the errors stopped before any remedial action was taken by TokenEx.
Timeline of events:
• 1646 First error was logged
• 1715 The number of errors reached the predefined threshold to generate an alert
• 1715 Service restored by downstream DNS provider

TokenEx is in the process of implementing automatic failover to a secondary DNS provider should the primary provider experience service-impacting issues.

Posted Apr 01, 2021 - 16:30 CDT