Building Scalable Self-Healing AI Systems

Every AI system fails eventually. A model returns malformed JSON. A vector store times out mid-retrieval. An agent gets stuck in a loop, calling the same tool five times with no new information. A downstream API rate-limits you halfway through a multi-step workflow that already cost you three LLM calls.

The usual response to this is more try/except blocks and a Slack alert. That works fine when you are demoing a prototype to five users. It falls apart the moment you have hundreds of concurrent agent runs in production, each one a long-lived, multi-step, non-deterministic process where failure can happen at any node in the chain. A naive retry in this setting can also duplicate a side effect. An email gets sent twice. A refund gets issued twice. A webhook fires twice and a downstream system reacts twice.

This post is about the architecture that prevents that. Not a single library or framework, but a way of thinking about state, retries, verification, and observability so that failure becomes something your system absorbs and recovers from, instead of something that takes it down or corrupts its data.

To keep things concrete, we will follow one running example through the whole post: an AI support agent that handles customer refund requests. It reads an incoming support ticket, retrieves relevant policy documents and order history using retrieval augmented generation (RAG), drafts a response, decides whether the refund is valid, and if so, calls a payments API to actually issue the refund. This is a good example because it touches almost every failure mode we care about: retrieval, generation, structured decision making, and a real, irreversible side effect at the end.

Why AI Systems Fail Differently From Regular Software

Traditional distributed systems fail in fairly well understood ways. A server crashes. A network link drops packets. A request times out because a downstream service is overloaded. Most backend engineers already have good instincts and good tools for these problems: retries with backoff, health checks, load balancers, database transactions.

AI systems inherit every one of those failure modes, because under the hood they are still distributed systems making network calls. But they also introduce a second, less familiar category of failure that lives at the semantic level, not the transport level. The request can succeed completely as far as your network stack is concerned, and still be wrong.

Let's go through what this looks like using our refund agent.

Non-deterministic output failures. The refund agent asks the model to return a decision as JSON: {"eligible": true, "refund_amount": 42.00, "reason": "..."}. Most of the time this works. Occasionally the model wraps it in a markdown code fence, or adds a sentence of commentary before the JSON, or returns "refund_amount": "forty two dollars" as a string instead of a number. The HTTP call to the model provider returned a 200. Nothing "failed" in the traditional sense. But your code that expects a clean JSON object will crash or silently misbehave.

Cascading agent failures. Say the RAG step retrieves the wrong policy document, perhaps an outdated refund policy that was superseded three months ago. The generation step then correctly follows the instructions it was given, and confidently drafts a refund approval based on stale rules. Every individual step "worked." The pipeline as a whole produced a wrong and expensive outcome, because an early error propagated forward instead of staying contained.

Silent degradation. Nothing crashes. The refund agent keeps approving and rejecting refunds every day. But over a few weeks, your embedding model provider quietly updates their model, or your vector index grows and retrieval quality drops, or a new class of tickets starts coming in that your prompts were never tuned for. Approval accuracy slowly drifts from 95 percent to 80 percent. No alert fires because nothing threw an exception. You only find out when finance notices a spike in incorrect refunds a month later.

Expensive retries. If a database query fails, retrying it costs a few milliseconds and effectively nothing. If your refund agent's LLM call fails or produces a bad output, retrying it costs real tokens, real latency, and if you are not careful, a real risk of double-issuing a refund. This means the retry strategy itself has to be a first class design decision, not an afterthought.

The core design problem across all four of these is the same: your system needs to be able to tell the difference between "this failed and a simple retry will fix it," "this failed and needs to be routed differently," and "this looks completely fine on the surface but is actually degrading underneath," and then respond to each one differently, automatically, without a human watching every single run.

The Four Pillars of a Self-Healing System

Self-healing is not one feature. It emerges from four layers working together: how you manage state, how you verify output, how you handle partial outages, and how you observe the whole thing. Here is the shape of the system we are building toward.

Each layer solves a different problem, and none of them alone is enough. Let's walk through each one in depth, using the refund agent as our example throughout.

Pillar 1: Durable Execution as the Foundation

The single highest leverage decision you can make in this whole architecture is separating workflow state from process state.

Here is what that distinction means in practice. Imagine the refund agent is implemented as a plain function that runs inside a request handler: it retrieves context, calls the model, validates the output, and calls the payments API, all as local variables inside one function call. This works fine until the process handling that request crashes. Maybe it is an out of memory error, maybe it is a routine deploy that restarts your servers, maybe it is a pod getting rescheduled by Kubernetes. The moment that process dies, every local variable dies with it. If the crash happened right after the model call but before the payments API call, you have no record that a decision was even made. The customer's ticket is now stuck, and nobody knows it.

This is exactly the problem that durable execution engines solve. A tool like Temporal treats your workflow as a sequence of individually persisted steps, called activities. Every activity's input and its result get written to durable storage automatically, as the workflow runs. If the worker process handling the workflow dies for any reason, a new worker picks up the exact same workflow, replays its event history, and resumes precisely where it left off, without re-running any activity that already completed. The refund decision that was already made does not get recomputed. Only the step that was interrupted gets retried.

This matters more for AI pipelines than for typical CRUD backends, for a subtle but important reason: LLM calls are not idempotent in the way a database read is. If a workflow engine naively re-ran the entire pipeline from scratch after a crash, it might call the model again and get a different, non-deterministic answer the second time, one that no longer matches decisions already made downstream. Durable execution avoids this entirely by never re-running a step that already finished.

Here is the refund workflow expressed as a diagram:

Every box here is a durably checkpointed step. If the process crashes right after step G, the payments API call, but before step I, the system does not need to guess what happened. It can see from the persisted history that the refund was already issued, and it resumes at step I instead of calling the payments API a second time. This is what turns "a crash corrupted our refund pipeline" into "a crash caused a thirty second delay."

A rough sketch of what this looks like in code, using a Temporal style workflow definition, makes the idea concrete:

export async function refundWorkflow(ticket: SupportTicket) {
  const context = await activities.retrieveContext(ticket);

  let decision = await activities.draftDecision(ticket, context);
  let attempts = 0;

  while (!isValidSchema(decision) && attempts < 3) {
    decision = await activities.repairDecision(ticket, context, decision);
    attempts++;
  }

  const judged = await activities.judgeDecision(decision);

  if (judged.approved) {
    const result = await activities.issueRefund(decision);
    await activities.notifyCustomer(ticket, result);
    return result;
  } else {
    await activities.routeToHumanReview(ticket, decision, judged.reason);
  }
}

Notice that each await activities.x(...) call is a durable checkpoint. Everything above the payments API call is safe to replay as many times as needed. The payments API call itself is the one step that must never accidentally run twice, which brings us to the next pillar.

Pillar 2: Structured Verification Loops, Not Blind Retries

A retry that just resends the exact same prompt and hopes for a different answer is close to a coin flip. A genuinely self-healing system treats a failure as information to act on, not just a reason to try again unchanged.

There are three layers of verification worth building, in increasing order of cost.

Schema validation as a hard gate. Every structured output from the model, in our example the JSON refund decision, passes through a strict validator before anything downstream is allowed to touch it. Tools like Pydantic in Python or Zod in TypeScript are a natural fit here. When validation fails, the important detail is what you do next: instead of simply resending the same prompt, you feed the specific validation error back into the next attempt. "Your last response had refund_amount as a string, it must be a number" is a far more useful instruction than a generic retry, and it dramatically increases the odds the next attempt actually succeeds.

LLM as judge verification. Schema validity tells you the shape of the output is correct. It says nothing about whether the output is actually right. For our refund agent, a response can be perfectly valid JSON and still recommend approving a refund that clearly violates policy. This is where a second, independent model call comes in: given the ticket, the retrieved policy, and the drafted decision, a judge model scores whether the decision looks reasonable. Keeping this as a separate step from generation matters, because it means you can use a cheaper or more strictly prompted model purely for judging, and you can update your judging criteria without touching your generation prompts at all.

Bounded correction loops with a real escalation path. Retry with feedback needs a hard ceiling. In practice two or three attempts is usually the right number. Past that ceiling, the system needs a defined fallback, not an infinite loop that quietly burns tokens. For the refund agent, that fallback is the human review queue you saw in the diagram above: if the model cannot produce a valid, judge approved decision within three attempts, the ticket gets routed to a person instead of looping forever.

Here is what that bounded loop looks like as a sequence over time:

The key idea across all three layers is the same: a failure at attempt N should make attempt N plus one more informed, not identical to attempt N.

Pillar 3: Circuit Breakers and Graceful Degradation

Scale exposes a failure mode that almost never shows up while you are developing locally: the partial outage. Your vector database is technically up, but responding three times slower than usual. Your model provider has not gone fully down, but is rate limiting a fraction of your requests. In these situations, the worst possible thing your system can do is keep hitting the struggling dependency at full volume, because that just makes the outage worse and produces a wall of timeouts for every in-flight request, including ones that had nothing to do with the original problem.

The classic pattern for this is the circuit breaker, and it is worth understanding as an actual state machine, not just a buzzword.

In the closed state, everything is normal, and requests flow through to the dependency as usual, with failures simply counted. Once failures cross a threshold within a time window, say more than 30 percent of calls to the payments API fail within one minute, the breaker trips to open. While open, new requests do not even attempt to reach the dependency. They get routed instantly to a fallback, without waiting for a timeout that would take several seconds each. For our refund agent, an open circuit on the payments API might mean queuing approved refunds to be issued once the API recovers, rather than blocking the whole ticket pipeline on it. After a cooldown period, the breaker moves to half open and lets a single trial request through. If it succeeds, the breaker closes again and traffic resumes normally. If it fails, it goes straight back to open and waits longer before trying again.

Circuit breakers are most useful when applied per dependency, not globally. The refund agent talks to at least three external systems: the vector store for retrieval, the model provider for generation, and the payments API for issuing refunds. Each deserves its own breaker, because they fail independently and recovering one should never be blocked on another.

Two more pieces complete this pillar. Backpressure means your orchestration layer tracks how many activities are currently in flight against each dependency, and deliberately slows down new work before you get rate limited, instead of only reacting after the fact. Fallback routing means deciding in advance which tasks can tolerate a lower capability model when the primary one is degraded. Drafting a first pass refund decision might be fine on a smaller, cheaper model during a partial outage, as long as the judge step still holds the line on quality before anything gets approved.

Pillar 4: Observability That Detects Degradation, Not Just Crashes

You cannot fix what you cannot see, and the semantic failures we described earlier are, by definition, invisible to standard uptime and error rate monitoring. The refund agent's HTTP calls can all return 200 while the actual decisions being made get steadily worse. This is the layer that closes that gap.

At minimum, four things need to be instrumented per step and per workflow run.

Full input and output tracing. Every LLM call and every tool call should be logged with its full input and output, not just a summary. When a refund gets incorrectly approved and someone asks why three days later, you need to be able to trace back through the exact retrieved context, the exact prompt, and the exact model response that led to that decision. Without this, debugging a bad outcome becomes guesswork.

Cost and latency as health signals, not just billing line items. A sudden spike in cost per ticket is very often the earliest visible sign of a retry loop or a routing bug, and it typically shows up well before any hard error does. If your refund agent's average cost per ticket doubles overnight, that is worth investigating immediately, even if error rates look completely normal.

Output quality drift metrics tracked over time. Validation failure rate, the distribution of judge scores, and the retry rate trend should all be tracked as time series, not just point in time numbers. This is what turns degradation into a visible slope on a graph, something you can catch weeks before it becomes a real incident, instead of only discovering it after a threshold is breached.

Correlated dashboards that tie workflow state to output quality. A dashboard in a tool like Grafana that shows "workflow X is currently stuck at the judge step, and judge approval scores for this ticket category have dropped 20 percent over the last hour" as a single connected signal is far more actionable than two separate alerts that an on-call engineer has to manually connect in their head at 2 a.m.

This observability layer also feeds directly back into pillar 2. A system that logs, in a structured way, exactly why an output failed validation or why a judge rejected a decision, is a system that can eventually notice patterns and route around that specific failure class automatically, instead of relying on the same generic retry every single time.

Putting It All Together

Stepping back, the refund agent's full architecture looks like this:

The orchestration layer owns the workflow's state machine and checkpointing. This is what lets the system survive a crash and resume exactly where it left off, instead of losing track of a refund decision that was already made.

The generation layer wraps every LLM and tool call with schema validation and a bounded retry with feedback loop. This is what turns "the model gave a wrong answer" into "the model gave a corrected answer," instead of just repeating the same mistake.

The resilience layer sits between the orchestration layer and every external dependency, using circuit breakers, backpressure, and fallback routing to keep a partial outage in one dependency from turning into a full outage of the whole pipeline.

The observability layer instruments every hop across all three of the layers above. This is what turns invisible, slow degradation into a concrete, actionable signal, and it is what tells you, over time, which failure classes are common enough to deserve a dedicated repair path instead of a generic retry.

None of these four pillars is sufficient on its own. Durable execution without verification just retries a bad decision reliably, over and over, instead of correcting it. Verification without observability cannot tell you when your judge model itself has drifted and started approving things it should not. The real resilience comes from how tightly these layers feed into each other: a validation failure should update your drift metrics, a circuit breaker trip should appear on the same dashboard as your retry rate trend, and a workflow resuming from a checkpoint should carry forward the context of exactly why it failed the first time, not just a blank slate.

Common Pitfalls Worth Naming

A few mistakes show up repeatedly when teams build systems like this, and they are worth calling out directly.

Retrying without changing anything. The most common anti-pattern is a retry loop that resends the identical prompt on failure. This wastes tokens and rarely fixes anything, because the model has no new information to correct its mistake with. Always feed the specific failure reason back into the next attempt.

No ceiling on retries. A bounded loop that is not actually bounded in the code, perhaps because the escalation path was never implemented, can quietly burn through budget on a single stuck ticket for hours. Always implement the fallback path, not just the retry path.

Treating side effects as safely retriable. Calling the payments API is fundamentally different from calling the model. It has a real, hard to reverse consequence. Any step with an external side effect needs either a durable, deduplicated execution guarantee, or an idempotency key so that a retry after a crash cannot double charge or double refund.

Monitoring uptime instead of output quality. A dashboard full of green checkmarks for HTTP status codes tells you almost nothing about whether your refund agent is making good decisions. The semantic failure modes described earlier will not show up there at all.

Building the resilience layer before the observability layer. It is tempting to jump straight to circuit breakers and fallback models, because they feel like the "advanced" part of the system. In practice, you cannot tune sensible thresholds for a circuit breaker without first knowing, from real trace data, how each dependency actually fails and how often.

Where to Start If You Are Retrofitting This

Very few teams get to build this from a blank slate. Most of the time, you already have a working pipeline and are trying to make it more resilient without a full rewrite. In that situation, this is the order that tends to pay off fastest.

1. Add durable checkpointing first. It is the foundation everything else depends on, and on its own it already eliminates the single worst failure mode: losing track of work that was already partially done.
2. Add schema validation gates on every structured output. This is cheap to add, usually just a validation library and a few lines of glue code, and it immediately catches a large class of silent failures that were previously invisible.
3. Add tracing before you add alerting. You need to actually see the shape of your failures, in real data, before you can decide what thresholds are worth paging a human over. Alerting built on guesses tends to either miss real problems or drown people in false positives.
4. Add circuit breakers last, once tracing tells you which dependencies actually fail often enough to justify one. Building a circuit breaker for a dependency that almost never fails is wasted complexity. Let the data tell you where to spend that effort.

Self-healing systems are not the ones that never fail. Failure is guaranteed the moment you are operating at real scale with real, non-deterministic models in the loop. The systems that actually hold up in production are the ones where failure is a normal, instrumented, recoverable state, handled the same way every time, instead of an exception that only gets handled well when someone happens to be watching.