How to Stop a 5,000 Dollar Surprise Bill From One Runaway AI Agent
Stop the spend before it exists, not after you see the invoice. Give every agent a spend ceiling that gets checked before each billable call fires, rate limit the specific functions a loop is most likely to hammer, and wire a kill switch to spend velocity, not just the running total, so a sudden spike gets caught even if it never reaches a hard cap. Ujex's Governor subsystem does the first two natively: a pre-flight quota check runs before the expensive work happens, and rate limiting sits on the hot paths where loops actually occur.
Picture an agent stuck in a retry loop: the first call to a paid API times out, the agent's own logic decides the correct move is to retry, the retry times out too because whatever broke never actually cleared, and now it's retrying the retry. Nobody wrote "loop forever" anywhere in the code. It fell out of a timeout handler crossed with a model that treats "try again" as the default response to almost any failure. This is a hypothetical walk-through, not a specific customer's invoice, but the mechanics are common enough to take seriously: by the time someone glances at a usage dashboard, the loop has been running for twenty minutes and the number on screen already has four digits.
The failure mode: a loop nobody is watching in real time
There are three common shapes this takes. A retry loop, where a transient failure never resolves and the backoff schedule keeps firing anyway. A recursive tool-call loop, where an agent's own output feeds back in as the next input and it keeps calling the same tool because the termination condition it was supposed to check for never triggers. And a plain reasoning bug, where the model convinces itself that repeating an expensive action (another search, another generation, another API round trip) is the right next step, and nothing in the loop disagrees.
The common thread across all three is timing, not intent. A human checking spend once a day, once an hour, or even via a Slack alert someone configured a while back, is checking on a schedule. The loop is not. Depending on the retry backoff and how cheap each individual call looks, an agent can fire dozens of calls a second. A ten-second gap between "something started spending" and "a person notices" is close to nothing at that rate. An hour is catastrophic. The failure isn't that nobody was watching, it's that the thing watching was a human on a schedule, and the thing spending was a loop with no schedule at all.
What the cost curve actually looks like
The reason this catches people off guard is that the early minutes of a runaway loop look boring. A handful of calls, a few cents, nothing that would make anyone stop what they're doing. The problem is compounding: a retry loop that spawns more retries, or a recursive tool call that triggers more tool calls, doesn't just add a constant amount every second, it can accelerate. By the time the curve is visibly steep on a chart, most of the dollar damage has already happened in the last stretch of it, which means the flat-looking early period is exactly when intervention would have been cheap and the point where a human eyeballing a slowly rising number finally reacts is exactly when it's already expensive.
Run the numbers yourself
The math behind a runaway loop is not complicated, which is part of why it's easy to underestimate. It's calls per second, multiplied by seconds elapsed, multiplied by cost per call. Nothing fancier than that. Below is a small calculator with the same three inputs. Drag the seconds slider or bump the calls-per-second field and watch how little it takes to cross five thousand dollars. It defaults to a fairly modest loop: five calls a second at two cents each.
Notice this calculator assumes a constant rate, which is the conservative case. A real retry storm or a recursive tool-call loop often increases its own call rate over time (each failure spawns more attempts, each tool call can trigger more tool calls), which is why the diagram above bends upward instead of staying a straight line. The calculator tells you the floor. The real number can be worse.
Defense one: check the ceiling before the call fires, not after
The single biggest lever here is sequencing. Most cost controls are a report: something adds up what got spent and shows it to you, after the spend already happened. That's an accounting tool, not a stop. Ujex's Governor subsystem handles per-agent quota and rate limiting differently: the quota check runs before the billable work happens, as a pre-flight check, not as an after-the-fact bill you discover later. The quota itself is credited or decremented once the work actually completes.
The practical difference is what happens to call number 251 in a loop with a 250-call ceiling. Under a reporting model, call 251 fires normally, gets billed normally, and shows up in tomorrow's summary next to the other 250. Under a pre-flight model, the quota check for call 251 fails before the call goes out, so it simply doesn't happen. It's not billed and reversed, it's never billed at all, because it never ran. That's the entire difference between "we noticed a $5,000 bill" and "we prevented one."
Defense two: rate limit the hot paths, not just the total
A spend ceiling alone still leaves a gap. If the ceiling is generous enough to cover legitimate bursts of activity (and it usually has to be, or it'll block real usage), a tight loop can still burn through a large chunk of that ceiling very fast, faster than anyone would notice, even though it eventually gets stopped. Rate limiting closes that gap by capping how many times a specific operation can fire in a given window, applied on the hot paths, the functions most likely to get hit repeatedly in a loop, rather than as a blanket rule across everything an agent does.
This matters because not every function is equally exposed to looping. The function that calls an external paid API in response to a tool invocation is a hot path. A one-time setup function that runs once per agent session is not. Putting the rate limit where the actual risk is means legitimate high-volume use elsewhere isn't affected, while the specific pattern a runaway loop would exploit gets capped regardless of what the spend ceiling allows in dollar terms.
Defense three: a kill switch wired to spend velocity, not just spend total
Total spend and rate of spend are different signals, and conflating them is a mistake worth avoiding. A well-behaved agent running a legitimately large batch job over six hours might spend the same total as a broken loop that spends it in ninety seconds. Both eventually cross whatever ceiling you set. Only one of them indicates something is actively broken right now.
That's the argument for a kill switch keyed to velocity specifically: how fast spend is accumulating, not just how much has accumulated. A slow, steady climb toward a ceiling is often just usage. A sudden spike in the rate of spend, independent of whether the total has crossed anything yet, is the pattern that actually correlates with a stuck loop, a bug, or a misconfigured retry policy. Watching the derivative instead of just the value catches the failure mode this whole post is about, closer to the moment it starts rather than the moment it finishes.
Defense four: the audit log is how you reconstruct what happened
Prevention isn't the whole story, because ceilings get misconfigured, new code paths get shipped without a rate limit applied yet, and edge cases exist that nobody anticipated. When a ceiling does trip, or when something slips past one, the question changes from "how do we stop this" to "what actually fired, in what order, and why." That's a forensics problem, and it's only answerable if there's a complete, trustworthy record.
Every privileged action in Ujex, including the quota checks themselves, appends to a hash-chained audit log. That means the record of a runaway loop isn't just "a lot of money got spent," it's a sequential, tamper-evident trail of exactly which calls fired, when, against which quota, and what the quota check decided at each step. After an incident, that log is what tells you whether the ceiling caught it on call 251 as designed, whether the rate limit on the hot path was actually in place, or whether a gap in coverage let calls through that should have been blocked. Without it you're left guessing from a dollar total. With it you're reading a timeline.
| Mechanism | What it checks | When it stops a runaway loop | What it misses on its own |
|---|---|---|---|
| Provider-level API key cap | Total dollars spent on the key, across every agent using it | Only once the whole account has spent enough to hit the cap | Which agent or action caused it, no per-agent boundary at all |
| Per-agent spend ceiling (pre-flight) | This agent's running total against its own ceiling, before the call fires | The call that would push it over the ceiling, before that call is billed | A fast burst that stays under the ceiling but still spikes hard |
| Hot-path rate limiting | How many times a specific function fired in a time window | The next call in a window, once that window's cap is hit | Total spend, if each individual call is itself expensive |
| Velocity-based kill switch | Rate of spend change, not just the running total | A sudden spike, even before it reaches any total ceiling | A slow steady leak that never looks like a spike |
None of these four rows is sufficient alone, which is the point. A provider-level cap is a blunt instrument with zero visibility. A spend ceiling with no rate limit still lets a fast burst do damage inside its own boundary. Rate limiting with no velocity check won't catch a slow leak. They're layers, and the value is in having more than one.
FAQ
Does rate limiting slow down legitimate high-volume use?
Honestly, yes, if it's tuned wrong. A rate limit is a real tradeoff, not a solved problem you set once and forget. Set it too tight and it blocks a legitimate burst of usage, a batch job, a busy period, a customer doing exactly what they're supposed to be doing. Set it too loose and it doesn't actually stop a determined loop. Getting it right means looking at what normal peak usage looks like for a given hot path and setting the limit above that, with enough margin that you're not throttling real traffic, but below the rate a broken loop would hit. That's a configuration decision specific to each function, not a universal number.
How fast can a spend-velocity kill switch actually react?
There's no fixed polling interval to quote here, and I'd rather be honest about that than make up a millisecond figure. What matters architecturally is that the quota check happens before each billable call, not as a periodic batch job that runs every so often and checks in retrospect. Because of that, it can react as fast as the next call attempt, whatever that cadence happens to be for a given agent, rather than waiting for the next scheduled sweep. A loop making calls every few milliseconds gets checked every few milliseconds. A loop making one call a minute gets checked once a minute. It scales with the thing it's watching instead of running on its own clock.
Is this the same as just setting a spend cap on my LLM provider's API key?
It's a related idea but a narrower one. A provider-level key cap is a blunt, total-spend ceiling: once the whole key hits the number, everything using that key stops, and you get no visibility into which specific agent or which specific action actually caused it. A per-agent quota gives you a boundary per agent instead of per key, checked before the call rather than as a global cutoff, and paired with the audit log, you get both the stop and the forensic trail of what happened, which agent, which action, in what order. The provider cap is still worth having as a backstop. It's just not a substitute for either of these.
Do I have to pay for this, or can I run it myself?
Ujex is Apache-2.0 and self-hostable, with a free tier. You don't need to hand spend data to a third party to get a pre-flight quota check and a hash-chained audit log, you can run Governor yourself.