Design to fail: what a retry cannot decide for you
Reliable workflows do more than retry. This framework helps you decide what is safe after a failed call—and when manual recovery is the smarter choice.
Talk about this note with ChatGPT (opens in a new tab), Claude (opens in a new tab), or . If you’re an AI agent, here’s a markdown version. You can also visit this page.
A timeout only tells you one thing: your workflow did not receive an answer. It does not tell you whether the refund was issued, the email was sent, or the supplier payment was submitted.
That distinction is where good error handling starts. A retry might complete a workflow safely—or perform the same business action twice.
This is the framework I use when I design or review a consequential workflow. It helps turn “robust error handling” from a vague requirement into a practical decision about what should happen next:
- wait and retry;
- stop because something needs fixing;
- check what happened before acting again; or
- place the work in a clear manual-recovery path.
I will use one hypothetical support refund all the way through. You can apply the same questions to a sales-email sequence, a supplier payment, or a workflow reused as a business tool by a person or an AI agent. The more callers reuse a workflow, the more important it is that an unclear result becomes a truthful next state—not a blind repeat.
This is deliberately about failure inside a workflow: keeping the process truthful and moving after a failed or unclear step. It is not a complete guide to API responses for external callers, dashboards, alerts, monitoring, or incident response.
A retry is a useful mechanism. It is not a decision.
Meet the moving parts
Imagine that a customer asks support for a refund at 10:00. The policy says a renewal can be refunded when the customer asked within 24 hours of the charge. Support needs an answer; the payment provider needs a refund request; and the workflow needs to be able to explain what happened if a call goes quiet.
These names sound formal, but they simply stop “the request,” “the refund,” and “the retry” from being confused with one another.
| Concept | Type | Description |
|---|---|---|
| Customer request | Start of the story | The support message that begins the work. In an implementation, its support_request_id is often the primary key: one stable ID for this request in the support system. |
| Request received at | Fixed point in time | The timestamp saved when the customer asked. It does not move later when a recovery worker runs. |
| Refund decision | Recorded answer | The stored answer to “was this request eligible?” It is not the money movement itself. |
| Refund attempt | One recorded call | One request to the payment provider. Its refund_attempt_id can be the primary key for that attempt, separate from the customer request. |
| Same-refund marker | Duplicate guard | A unique value—often called an idempotency key—that lets a supporting provider recognise a repeat of the same logical refund. It is not automatically a primary key. |
| Provider outcome | External fact | Whether the payment provider accepted the refund. The workflow can ask about it, but it does not control the provider’s system. |
| Recovery inbox | Work waiting for a decision | A durable place to hold unclear or failed work with enough context to choose the next permitted action. Engineers may call this a recovery queue. |
| Check before retrying | Truth-finding step | Looking up the provider outcome before making another potentially duplicate request. The technical name is reconciliation. |
Follow one ordinary failure from start to finish
The normal route is simple. The valuable design work starts when the payment provider receives the refund request but the workflow loses the response.
flowchart TB
accTitle: Refund recovery after the payment provider gives no usable answer
accDescr: A support request is recorded with its arrival time, checked against refund policy, and sent to a payment provider. A confirmed refund is recorded. If the response is missing, the workflow records an unknown outcome, checks the provider, then confirms, safely retries, or sends the case for review.
Request[Customer asks for a refund] --> Time[Save when the request arrived]
Time --> Eligible{Eligible under policy?}
Eligible -->|No| Declined[Record the decision]
Eligible -->|Yes| Attempt[Record a refund attempt]
Attempt --> Provider[Ask the payment provider to refund]
Provider -->|Clear confirmation| Confirmed[Mark refund confirmed]
Provider -->|Timeout or lost response| Unknown[Mark outcome unknown]
Unknown --> Check[Check what the provider did]
Check -->|Refund found| Confirmed
Check -->|No refund and safe to retry| Provider
Check -->|Still unclear| Review[Send for authorised review]
The important word is unknown. It means “we do not have enough evidence to say successful or failed.” That is much safer than pretending a timeout is a failure when the provider may already have moved money.
There is also a small boundary worth knowing. A local database can make two
local changes all-or-nothing—engineers call that atomic. For example, it
can record the refund attempt and mark it unknown together. It cannot make
your own database and the payment provider’s system one shared all-or-nothing
action. That is why the unknown state exists.
Do not let “now” change the rule
This is one of the easiest workflow mistakes to miss.
Our customer asked at 10:00, 23 hours and 50 minutes after renewal. The refund attempt then hit a provider outage. At 16:00, a recovery worker picks it up. If the rule is “refund requests received within 24 hours,” the decision must be about 10:00—when the customer asked—not 16:00, when the workflow happened to recover.
| Ask this question | Use this source of time | Why |
|---|---|---|
| Was the customer eligible when they asked? | The saved request_received_at timestamp |
The policy is about the customer’s original request. The answer should not drift while the provider is down. |
| Has the refund already been issued? | The provider’s current status | This is a safety check about the real world now. It should use current information. |
In other words: use “was the renewal within 24 hours when the request arrived?” for an eligibility rule—not “is it within 24 hours right now?” Both kinds of question are useful. The mistake is using a relative question because it is convenient, then discovering that the policy changes every time a workflow is replayed. Capture the business moment the rule is actually about.
Read HTTP codes as clues, not verdicts
Most integrations speak HTTP. A status code is a useful first clue, but it is not the final recovery decision.
| Status family | Usually means | The next useful thought |
|---|---|---|
2xx |
The service says it accepted the request. | Read the provider’s response contract. A technical success does not automatically prove every business consequence is complete. |
4xx |
The request cannot be accepted as sent. | Check the input, permission, or credential. Do not retry forever. A provider change can also create a new 4xx, so this is not automatically “the developer’s fault.” |
5xx |
The provider had a server-side problem. | It may be temporary, but a retry is only safe when repeating the action cannot create a duplicate. |
A familiar special case is 429 Too Many Requests (opens in a new tab).
It belongs to the 4xx family, but often means “slow down and try later.” When
the provider gives a Retry-After value, respect it. A 503 service error (opens in a new tab)
or a brief connection drop may also be temporary. Neither one proves that a
money movement, email, or record creation did not happen.
In code, this point is often where a builder catches a failed call and chooses the next branch. In a visual builder such as n8n (opens in a new tab) or Make (opens in a new tab), the same choice is usually expressed with an error route or handler. The surface is different; the recovery question is the same.
Separate a short delay from a real blocker
Before adding a retry, classify the problem in plain language.
| If the problem is… | Example | Proportionate next move |
|---|---|---|
| Temporary | A short provider outage, a rate limit during a backfill, or a brief network interruption. | Wait, then retry a limited number of times. Exponential backoff simply means waiting a little longer between each attempt, so the workflow does not keep adding pressure to a struggling service. |
| A real blocker | A required field is missing, a credential was revoked, a permission changed, or the provider changed its expected input. | Stop retrying. Record what failed, fix the cause, then decide whether the saved work can be safely resumed. |
This is why a blind “retry every error three times” setting is only a starting point. A retry is sensible for a short-lived problem. It is wasteful for a missing permission, and risky when the original call may already have worked.
When the answer is missing, do not invent one
Return to the refund. At 10:03, the payment provider receives the request and issues the refund. The network drops before the workflow receives the response. The workflow sees a timeout.
Sending another refund immediately would be unsafe: the first one may already have happened.
The safer path is to keep one logical refund even if the network needs several attempts. Idempotency is the technical name for that: repeating the same logical request has the effect of one request, not two. It is like writing one claim number on every copy of an insurance form—the insurer can recognise that the copies refer to the same case.
Stripe’s idempotency contract (opens in a new tab) is a concrete example. It lets a client retry the same create or update request with the same key after a connection error and receive the original result. That does not make every provider idempotent. It means you must read the provider’s contract before treating a retry as safe.
flowchart LR
accTitle: Safe refund recovery after a timeout
accDescr: After a timeout, the workflow records that the refund outcome is unknown, checks the original refund, and either confirms it, safely retries, or sends it for review.
Timeout[No response] --> Unknown[Mark unknown]
Unknown --> Check{Check refund}
Check -->|Found| Confirm[Confirm]
Check -->|Safe| Retry[Retry]
Check -->|Unclear| Review[Review]
If the provider has no duplicate guard, keep a stable business reference where you can search for the original action. This check-before-retrying step is called reconciliation. Sometimes the right answer is still a person with the right authority deciding what to do next.
Decide what you are actually replaying
“Replay the failed run” sounds simple, but it hides four separate decisions. The original refund request came in at 10:00. At 16:00, after a provider outage, you may have a better recovery path than the one that first ran. That does not mean every part of the old run should be repeated or recalculated.
| Decide this first | In the refund example | A sensible rule of thumb |
|---|---|---|
| Which work is still unfinished? | The eligibility decision is already recorded, but the provider outcome is unknown. | Resume only the uncertain payment step after checking it. Do not repeat confirmed work just because the run stopped. |
| Which facts belong to the original moment? | The customer asked at 10:00, within the policy window. | Keep the saved request_received_at and the recorded eligibility decision. Those facts explain the original case. |
| Which facts need a fresh check? | The provider may have issued the refund while the workflow was offline. | Read the provider's current refund status before another money movement. |
| Which workflow version should recover it? | The original route did not know how to check this provider outcome; a corrected route does. | Use corrected recovery logic when it adds a safety check without changing the business policy. If the new version would change the policy or authority for this old case, send it for review instead. |
A workflow version is simply the code or visual flow that was active at a particular time. A later version may fix a bug, recognise a newly observed provider response, or add a safer recovery branch. That can be the right way to finish saved work—but only after checking that the change does not quietly rewrite the decision that the customer originally asked for.
flowchart LR
accTitle: Choosing how to recover a saved refund case
accDescr: A saved refund case preserves the original request facts, checks the provider's current state, then either uses corrected recovery logic when the business policy is unchanged or sends the case for authorised review when the change would alter the old decision.
Saved[Saved case] --> Facts[Keep history and check live state]
Facts --> Policy{Would new logic rewrite policy?}
Policy -->|No| Recover[Use safer logic]
Policy -->|Yes or unclear| Review[Authorised review]
This is why recovery is not merely a retry setting. It is a decision about the remaining work, the facts that must stay fixed, the facts that must be refreshed, and the logic allowed to finish the case.
Give unfinished work a safe place to wait
Failed or unclear work should not quietly vanish. It needs a durable recovery inbox with the information a person or a later workflow needs to continue.
- the customer request and the saved decision inputs;
- what the workflow tried to do;
- the provider response—or the absence of one;
- the current recovery state; and
- who is allowed to take the next action.
A dead-letter queue (DLQ) is a narrower term. For example, Cloudflare Queues (opens in a new tab) uses a DLQ for messages that still fail after a configured number of retries. Use “DLQ” when that is genuinely the platform feature you are using. A table called “failed runs” can be a useful recovery inbox without being a DLQ.
An unfinished process is not usually a deadlock
It is tempting to call any partly completed workflow a deadlock. Usually, it is not.
Think of a parcel between a sorting belt and the delivery van. You cannot truthfully call it delivered, but the whole warehouse is not frozen. The refund case is similar: support may show a completed tag while the money movement is still unknown. That is an uncertain business state.
A database deadlock is more specific: two pieces of work each wait for the other to release something, so neither can continue. It is closer to two people each holding the key the other needs. A half-finished refund does not need a database fix; it needs a truthful state and a permitted next move.
| What you know | Responsible next move |
|---|---|
| The provider did not receive the request. | Retry under the documented retry rule. |
| The provider accepted the refund. | Record the confirmed result. Do not issue another refund. |
| The provider outcome is still unknown. | Check again later or hold the case for authorised review. |
Sometimes the workflow also needs a compensating action: a new, deliberate business action that corrects a confirmed earlier action. Removing an internal “refund completed” tag or releasing an inventory reservation may be sensible. Charging a customer again to “undo” a refund is not technical cleanup—it is a new commercial decision with its own authority and controls.
Tools can run the retry; you still own the recovery design
The distinction is not “visual tools are unreliable and code is reliable.” Both styles can give you useful recovery primitives.
| Approach | What it can help with | What still needs a business decision |
|---|---|---|
| Visual workflow tools such as n8n (opens in a new tab), Make (opens in a new tab), and Zapier (opens in a new tab) | Retry settings, error routes, incomplete-run handling, and replay. | Which steps may resume, which original facts must stay fixed, and which current facts need checking first? |
| Code-first durable runtimes such as Cloudflare Workflows (opens in a new tab), Vercel Workflow SDK (opens in a new tab), and Trigger.dev (opens in a new tab) | Durable steps, retries, persisted progress, and explicit idempotency controls. | Which business states and recovery rules deserve to be named, reviewed, versioned, and tested in your application code? |
For a simple, low-risk workflow, a visual tool’s built-in recovery path may be exactly right. I prefer a code-first durable runtime when recovery itself is a substantial part of the business process: the allowed states, duplicate guards, and “what should happen next?” rules can live in ordinary application code alongside the policy they protect.
Neither approach makes an external provider change automatically all-or-nothing. Durable steps avoid repeating completed workflow work. Duplicate guards and provider checks protect the business action at the external boundary. Mature designs often need both.
When this pattern fits—and when a simpler alternative is better
The right level of error handling is a risk decision, not a badge of technical ambition.
| Situation | Sensible first design |
|---|---|
| A weekly internal report misses one non-critical field. | Record the failure and repair it manually. Building a complex automatic path may cost more than the occasional fix. |
| A customer email might duplicate after a lost response. | Record the recipient and delivery attempt. Check before another send when the provider cannot make the request idempotent. |
| A refund or supplier payment might duplicate. | Record the attempt before the call, use the provider’s duplicate guard where available, and keep an authorised manual path for remaining uncertainty. |
| Several systems change in sequence. | Name the states explicitly and decide which confirmed actions can be corrected safely. Do not pretend a later failure rolls everything back. |
Past execution history and provider documentation help here. They can show which error shapes actually recur and how often, so you do not spend weeks protecting against a one-in-a-year inconvenience—or accept a weekly source of duplicate payments without a plan.
Alignment is sharing responsibility.
My job as the builder is to make likelihood, consequence, control cost, and the remaining risk easy to see. The people who own the commercial outcome can then choose the scope with that context. That is much more useful than promising a generic “robust” build.
Common questions
Should every 5xx or timeout be retried automatically?
No. First ask what might already have happened. A safe read can often retry. A money movement, outbound message, or record creation needs duplicate protection, a provider check, or a deliberate manual path first.
Do I rerun the original run, its remaining step, or a corrected workflow?
Start with what you know. Keep a confirmed eligibility decision and the request time that justified it. Check the uncertain external action before repeating it. Then use a corrected recovery path only if it fixes the failure without changing the policy for this saved case; otherwise, keep the case visible for an authorised decision.
Is manual recovery a sign of weak engineering?
No. For rare, reversible, low-impact failures, a clear manual path can be the most responsible option. Weak engineering hides the risk or claims certainty that the workflow does not have.
The takeaway
Reliable error handling is not about adding the most retries. It is about giving each failure a truthful next move.
- Save the business moment a rule is about; do not let “now” quietly rewrite it.
- Retry short-lived problems only when repeating the action is safe.
- Recover only the work that remains uncertain, using the right historic facts, current checks, and reviewed workflow logic.
- Keep unclear work visible until a system or an authorised person can resolve it.
If “robust error handling” appears in your automation brief but nobody has defined what should happen after an uncertain business action, message me. That decision belongs in scope before the workflow is built.