Why Your Customer Got Charged Twice.

By Muhammad Umar·December 5, 2025·6 min read·Issue #11

A customer emails to say they have been charged twice. You check, and they have. Same amount, same card, eleven seconds apart.

Nothing in your logs looks wrong. There is no exception, no failed job, no obvious bug. One request went out, timed out, and was retried, which is exactly what your code was written to do.

The retry is not the bug. The retry is correct behaviour built on top of a gap in what a network is able to tell you, and until you see the gap the fix will keep looking like a judgement call about how many times to retry.

The problem: a timeout is not a failure

When your service calls a payment API and gets no response, you know precisely one thing: nothing came back.

You do not know whether the request arrived. You do not know whether it was processed. You do not know whether a reply was sent and lost on the way home. All three produce the same silence.

Both worlds look identical from where you are standing, and no amount of waiting separates them. That is not a limitation of your monitoring. It is a property of sending messages over an unreliable channel, and it is the reason two parties cannot reach guaranteed agreement over one.¹

So your retry logic is being asked to make a decision with information that does not exist. Retry and you risk charging twice. Do not retry and you risk a customer who paid for nothing.

You cannot make the ambiguity go away. You can only decide which side of it you would rather be on, and then remove the consequences.

Pick your poison, then fix it

There are exactly two things you can do when you get silence, and each has a name.

At-most-once means you never retry. Every request is attempted a single time. You will never double charge, and you will sometimes lose a payment entirely with no record that anything was meant to happen.

At-least-once means you retry until you get a definite answer. Nothing is ever silently lost, and duplicates become routine rather than exceptional.

You may have heard of a third option. Exactly-once delivery does not exist, and it is worth being precise about why, because the phrase appears in a lot of product documentation.

For a message to be delivered exactly once, the sender would have to know whether the previous attempt arrived, which is the thing it cannot know. What systems that advertise exactly-once actually provide is at-least-once delivery combined with duplicate suppression at the receiver. The duplicates still happen. They are absorbed somewhere you cannot see.

That is not a criticism. It is the correct design, and it is also the design you should copy. Retry freely, and make the second attempt harmless.

The fix: idempotency

An operation is idempotent when doing it twice has the same effect as doing it once. Setting a value is idempotent. Adding to a value is not.

Some HTTP methods are defined this way. PUT and DELETE are specified as idempotent, POST is not, which is why the problem shows up in exactly the places it does.²

The mechanism for making a payment idempotent is a key that the client generates and reuses across retries of the same logical operation.

curl https://api.example.com/charges \
  -H "Idempotency-Key: 8f14e45f-ea8d-4a1b-9b52-3c7d1e0a9f21" \
  -d amount=4000 \
  -d currency=gbp

The server records the key with the result. If the same key arrives again it returns the stored response instead of performing the work a second time. Stripe documents this behaviour and stores keys for 24 hours.³

Two details decide whether this actually works, and both are easy to get wrong.

The key must come from the client, and must not change on retry

Generate it when the operation is first attempted, not inside the retry loop. A key generated per attempt is a different key every time, which means the server sees three unrelated charges and processes all three.

It also must not be derived from the request contents. Two genuinely separate purchases of the same item for the same amount would hash identically, and the second one would be silently swallowed as a duplicate. That failure is harder to notice than a double charge, because nobody complains about a payment that never happened.

The guarantee belongs in the database, not the application

The obvious implementation is to look up the key, and insert if it is absent. Under concurrency that is a race: two retries can both check, both find nothing, and both proceed.

Timeouts and retries arrive in bursts, so this is not a rare interleaving. It is the common case.

CREATE TABLE charges (
  idempotency_key text PRIMARY KEY,
  customer_id     bigint NOT NULL,
  amount_pence    bigint NOT NULL,
  response        jsonb,
  created_at      timestamptz NOT NULL DEFAULT now()
);

With the key as a primary key, the second insert fails on the constraint no matter how the attempts interleave. Catch that failure and return the stored response.

This is the part worth taking away even if you forget everything else. A uniqueness rule enforced by the database holds under concurrency. The same rule enforced by your code does not.

The approaches compared

Where this bites beyond payments

Payments are where it gets noticed, because the customer notices. The same gap is everywhere else and usually goes unreported.

• Webhook receivers, which every provider will eventually deliver twice, and which are documented as doing so.
• Queue consumers, where a job that times out mid-processing gets redelivered to another worker.
• Any job runner that retries on failure, including the one that sends email.
• Form submissions, where the user is the retry mechanism and the double click is the second attempt.

The email case is a good illustration of how invisible this is. A duplicate receipt email is mildly embarrassing and generates no ticket, so a system can send thousands of them before anyone mentions it.

When this is the wrong advice

When the operation is already idempotent. Setting a status to cancelled twice changes nothing the second time. Adding keys and a deduplication table there is machinery guarding against an outcome that cannot occur.

When a duplicate is genuinely harmless and rare. A metrics counter that occasionally double counts does not need a distributed correctness story. Weigh the cost of the machinery against the cost of the error, rather than applying it everywhere on principle.

When you are storing the key but not the response. This is the partial implementation that looks finished. Rejecting the duplicate is only half the job: the retrying client still needs to learn what happened, and returning an error to a retry of a request that succeeded leaves the caller believing the payment failed.

The takeaway

The instinct after a double charge is to tune the retry logic. Fewer attempts, longer timeouts, more caution. None of that closes the gap, because the gap is in what the network can tell you and not in your configuration.

Assume the request will arrive more than once. Give it a name the server can recognise, and let the database enforce the rule. Then a retry becomes what it should always have been: a question asked twice, answered once.