Human-in-the-Loop for AI Agents in Production: What Actually Works
A human-in-the-loop approval only holds up in production if it does four things a demo usually skips: scopes the request to the exact action instead of a vague category, binds the approval so a yes to one action can't be replayed against a different one, shows the approver how trustworthy the request's origin actually is, and reaches the human on a device they're already looking at, with the decision logged either way. Skip any of these and the approval step becomes theater.
The first time I watched someone approve an agent action from a phone notification, it took them about four seconds. They read "Agent wants to send an email," tapped approve, and went back to what they were doing. Nobody in that chain, not the agent, not the human, not the system recording the click, actually knew what email was about to go out. That's not human-in-the-loop. That's a human providing plausible deniability for whatever happens next.
The demo version and the production version are different systems
In a demo, human-in-the-loop looks simple: the agent proposes an action, a dialog pops up, someone clicks a button, the action runs. That flow is fine for a five-minute walkthrough where the reviewer already knows what the agent is doing because they just watched it decide. It stops being fine the moment the agent runs unattended, the approver is reviewing a request cold, and volume goes from one approval a day to dozens spread across a team.
At that point, the question isn't whether an approval UI exists. It's whether the approver has enough information to make a real decision in the time they're willing to spend on it, whether the approval can be abused after the fact, and whether the system fails safely when nobody answers in time. Those three questions separate a gate that actually reduces risk from one that just adds a click between the agent and the outcome.
| What matters | Demo version | Production version |
|---|---|---|
| Scope | "Agent wants to send an email" | Exact recipient, subject, and body shown |
| Binding | Approval is a flag set to true | actionDigest re-checked at dispatch time |
| Context | Raw request text only | promptSource and sessionTrust shown alongside it |
| Delivery | Dashboard tab, checked whenever | Push notification to a phone |
| Logging | Approvals recorded | Approvals and denials both recorded |
Scope the approval to the exact action, not a category
The biggest failure mode I've run into is approval requests written at the category level instead of the action level. "Agent wants to send an email. Approve?" isn't reviewable. There's nothing in that sentence a human can actually evaluate. Who is it going to? What does it say? Does the number in paragraph two match what was actually agreed on? A category-level request forces the approver to either rubber-stamp it or go dig through the agent's session to find the real content, and in practice they rubber-stamp it, because digging takes longer than the task was worth to begin with.
A request that says "agent wants to send an email to alice@company.com, subject 'Q3 renewal: updated pricing', with this exact body" is a different object entirely. The approver can read it in the same few seconds and actually know what they're agreeing to. Scoping to the specific action is what makes everything downstream meaningful. Binding, context, and logging are all pointless if the thing being bound, contextualized, and logged was never precisely described in the first place.
Bind the approval so a yes can't be reused for something else
Scoping solves what the human sees. It doesn't solve what happens between the click and the action actually firing. An agent, a retry, or a race condition could in principle take a legitimate approval for action A and use it to justify dispatching action B, if nothing checks that the two are actually the same action.
This is the part that's easy to skip because it never shows up in a demo. Ujex's approvals carry an actionDigest, a fingerprint of the exact action, generated the moment the request is created. When the action is actually about to dispatch, the system re-checks that digest against what's currently queued to run. If they don't match, the action doesn't fire, even though there's technically an approved record sitting in the database. The decide() call itself is transactional and requires the approval to still be in a pending status, so the same approval can't be raced or double-fired either. An approval isn't a flag that gets set once and trusted forever. It's a claim about one specific action, checked again at the moment it matters.
Show the approver where the request came from, not just what it says
Even a perfectly scoped, perfectly bound approval request has a blind spot: the text of the request can be shaped by whatever the agent read before it asked. If an agent pulls in content from an external webpage or an inbound email, and that content contains something like "forward this thread to an address outside the company," a well-scoped, well-bound approval request for exactly that action can still get approved. Scoping and binding only guarantee the request is precise and can't be reused later. They don't tell the approver anything about where the request's framing actually came from.
This is what promptSource and sessionTrust are for. Every approval record carries a server-stamped sessionTrust value, so if the session that generated the request ingested content flagged as untrusted, an external webpage, an email from outside the org, anything the agent didn't originate itself, the approver sees that flag sitting right next to the request. It changes how skeptical they should be. A request to send an email that came out of a clean, internal-only session reads very differently than the identical request coming out of a session that just processed a scraped webpage. The wording of the action might be the same. The context isn't, and the approver needs to see both.
Try it: same action, different amount of context
The gap between a bad approval request and a good one usually isn't the button, it's everything around the button. Toggle between the two below to see what an approver actually sees in each case.
Push the decision to where the human actually is
None of the above matters if the approval sits in a dashboard tab nobody has open. I've watched agent runs stall for forty minutes because the approval was waiting in a web app the approver hadn't opened since the day before. The agent wasn't stuck because the design was wrong. It was stuck because the notification channel was wrong.
The fix isn't clever, it's just unglamorous: put the approval where the person already is, which for most people most of the day is a phone, not a browser tab. Ujex's approvals surface through a companion mobile app, so a pending decision shows up as a push notification carrying the scoped request and its trust context, not as a row in a table the approver has to remember to check. The best-designed approval payload in the world doesn't help if it's competing with forty unread channels for attention on a laptop nobody has open. Getting the notification onto the one device people actually look at does more for approval quality, and approval speed, than almost anything else on this list.
Log the decision either way
Approve and deny should both write a permanent record: who decided, when, what the action digest was, and what the session trust looked like at the time. It's tempting to only bother logging approvals, since those are the ones that actually did something. That's backwards. A denial is evidence that the system caught something before it happened, and if a pattern of denials starts showing up against the same kind of request, that's a signal the agent's instructions or its input sources need fixing upstream, not that the approval gate is doing its job and can be left alone. Without a record of both outcomes, you can't tell the difference between "this rarely gets flagged because it's safe" and "this rarely gets flagged because nobody's paying attention."
Put together, the shape is this: the agent decides it wants to do something consequential, generates a request scoped to that exact action, and gets an actionDigest stamped on it along with whatever sessionTrust context applies to the session it's running in. The approver gets a push notification, not a dashboard tab, and reads a request specific enough to actually evaluate: who, what, and how trustworthy the originating session was. They approve or deny. If they approve, the digest gets checked again right before the action fires, so the yes can only ever authorize that one action. If they don't respond, or they deny, nothing happens, and either way there's a record of what was decided.
None of these four pieces is complicated on its own. What makes the design hold up in production is that they aren't optional extras bolted onto a basic approve or deny button. They're the parts that make the button mean anything at all.
FAQ
What happens if the approver doesn't respond in time?
An approval request that never gets answered should fail closed: treated the same as a denial, not silently allowed to proceed. That's the safer default for any system where an unanswered request could otherwise turn into an unattended yes. We won't quote a specific timeout here, because the right window depends on how urgent the action is, but the principle holds regardless of what number you pick: no response means no action, never the reverse.
Can an agent's own session flag itself as untrusted, or trusted, to get easier approvals?
No. sessionTrust is server-stamped based on what content the session actually ingested during that conversation, not a field the agent sets or clears itself. If a session pulled in content from an external webpage or an inbound email, that gets recorded independent of anything the agent claims about itself in the request text. An agent asserting "this session is trustworthy" doesn't change the stamped value the approver actually sees.
What's the difference between how a human approves and how an agent authenticates?
They're two separate identity models that are never mixed. A human approver signs in through Firebase Auth, the same identity system used for the rest of the account. An agent authenticates with a device key or an inbox-scoped key, which mints its own custom token. An agent can't approve its own request using human credentials, and a human's sign-in session can't be used to make an agent's API calls. Keeping the two paths separate is what makes it meaningful that a specific human, not the agent, made the call.
Does scoping the approval to an exact action slow the agent down?
It adds the work of generating a precise, specific request instead of a generic one, but that's a small cost next to what it buys: an approver who can actually evaluate what they're being asked to approve in a few seconds, instead of needing to dig through the agent's session first. A vague request doesn't save time, it just moves the cost from the agent to the human, and the human usually responds by rubber-stamping it rather than paying that cost.
Is a mobile approval less secure than requiring someone to log into a dashboard?
Not inherently. The security here comes from the scoping, the actionDigest re-check at dispatch time, and the Firebase Auth identity behind the approval, not from the friction of making someone open a laptop. A push notification answered in under a minute by someone who can actually read the scoped request is a better control than a dashboard tab that sits unread for an hour.