For 51 minutes on Friday morning, no shopper could pay

On Friday, February 5, nobody could pay at any Farmgate shop from 9:12 to 10:03 a.m. Eastern. The security certificate for our payment pages expired because its automatic renewal had been failing for 30 nights, and the warnings went to a mailbox no one reads. We renewed it by hand, have added the two alarms that would have caught this in January, and list the rest of the fixes below.

By Deborah Achterberg, head of engineering, with the on-call engineer and the support lead on shift. This review is blameless: it describes systems and decisions, and looks for no one to fault.

51 minwith payments down, 9:12 to 10:03 a.m.

1,280checkouts failed at the payment step

448of those orders were never completed (35%)

412shops with a failed checkout, of 1,930 active (21%)

Shoppers came back and paid for 832 of the 1,280 orders by Sunday night. The 448 that stayed lost were worth about $20,800 to our shops, at an average order of $46.50. No card details were exposed and nobody was charged twice. Support answered 94 tickets.

Timeline

Friday, February 5. All times a.m., Eastern.

9:12The certificate for pay.farmgate.example expires. Browsers refuse to open the payment step and show a security warning instead.
9:14Completed payments fall to zero. No alarm sounds.
9:21First support ticket: a shop owner says customers see a warning where the card form should be.
9:26Six tickets in five minutes describe the same warning. Support pages the on-call engineer.
9:29On-call acknowledges and opens an incident channel.
9:34Expired certificate confirmed in a browser.
9:37Renewal run by hand. It fails: the job can’t add the DNS record that proves we own the domain.
9:41Status page changed from “All systems operational” to “Payments unavailable”.
9:48Reason found: the renewal job holds a key for the DNS provider we left on December 15.
9:55Key for the current DNS provider created and given to the job.
9:58Renewal succeeds.
10:03New certificate live on all four load balancers. The first payment goes through.
10:09Payment success back to its usual 97%. Status page set to resolved.
10:40Email sent to all 412 affected shops, each with its own list of interrupted orders.

Where the 51 minutes went

14 minbefore anyone was paged
8 minto find the expired certificate
29 minto renew it

Root cause

A certificate is what lets a browser trust that pay.farmgate.example is really ours. Each one lasts 90 days. A nightly job renews it once 30 days are left, and to do that the job must prove we control the domain by adding a temporary record to our DNS.

On December 15 we moved our DNS to a new provider. The renewal job wasn’t on the migration checklist, so it kept the old provider’s key. The payments certificate reached its 30-day mark on January 6. The job failed that night and the 29 nights after it, until the certificate ran out.

Each failure was emailed to a shared mailbox set up years ago that no team reads. Nothing else was watching the expiry date.

Why it took 14 minutes to notice

Our payments alarm counts requests that fail. With an expired certificate the browser gives up before it sends anything, so our servers saw no failures. They saw no payments at all, and nothing alarms on silence.

What else was exposed

Two more certificates were on the same path, due to expire on February 19 and March 2. Both were renewed by hand on the day.

What went well

Support saw one pattern in six tickets and paged five minutes after the first, without waiting for an alarm.
On-call acknowledged in three minutes and had the expired certificate confirmed five minutes later.
Carts were kept, so the 832 shoppers who came back didn’t start over.
Affected shops had a plain account and their order lists 37 minutes after payments returned.

What didn’t

Thirty warnings were sent and none reached a person.
Nothing checked our certificates from outside, the way a browser does.
The payments alarm can’t tell “no failures” from “no payments”.
The DNS migration checklist covered what reads our DNS and missed what writes to it.
The status page said “All systems operational” for the first 29 minutes.

Action items

No.	Action	Owner	Due	Status
1	Check every public certificate from outside our network each hour, and page when one has fewer than 21 days left.	Dele OkaforPlatform	Fri, Feb 5	Done
2	Send renewal failures to the on-call pager and close the shared mailbox.	Dele OkaforPlatform	Mon, Feb 8	Done
3	Let the support lead on shift change the status page without asking engineering.	Aroha NgataSupport	Wed, Feb 10	In progress
4	Alarm when no payment succeeds for five minutes between 7 a.m. and 10 p.m. Eastern.	Mirela PopescuPayments	Fri, Feb 12	In progress
5	Add “what writes to this system?” to the migration checklist, and go back over the December DNS move with it.	Jonas BergströmInfrastructure	Fri, Feb 19	Planned
6	List every certificate and scheduled job, each with a team that owns it.	Dele OkaforPlatform	Fri, Feb 26	Planned

We will go through this list again on Monday, March 1, and say in the engineering update which items are closed. Corrections to this review are welcome: tell the Platform team.