What Should an AI Agent Be Allowed to Decide Without a Human?

Finance found it during the Thursday reconciliation, not because anything broke. A refund had gone out for $4,200, above our standard exception threshold, with no approval attached to it. The agent we'd deployed six weeks earlier to triage support tickets had looked at the customer's account history, decided the exception was reasonable, and processed it. On its own. Nobody had told it to ask first, because nobody had ever written down that this was a category of decision that needed asking.

The meeting that followed lasted 45 minutes, and I want to be precise about what it was actually about, because it wasn't what anyone expected walking in. It wasn't a meeting about whether the agent had made a good call — by most reasonable standards, it had. The customer had a clean history and a legitimate complaint. It was a meeting about the fact that when our CFO asked who was supposed to have set the dollar threshold where this agent stops deciding and starts asking, nobody in the room could answer. Not the person who deployed it. Not the vendor. Not me, and making sure that kind of question has an answer is supposed to be my job.

An AI agent should only be allowed to make decisions unsupervised when the cost of a wrong decision is fully recoverable and clearly attributable — everything else needs a human checkpoint. In practice, that means agents can draft, summarize, flag, and recommend without approval, but actions that spend money, change a customer-facing commitment, or can't be easily undone should route to a named human before execution. Most companies that have deployed agentic AI have never actually written this line down, which is why the failure shows up as a surprise instead of a known risk.

The Meeting That Wasn't About Whether the Agent Was Smart

Here's the part that took me a few days to sit with. The agent's decision wasn't the problem. If a member of my support team had approved that same $4,200 exception using the same reasoning, nobody would have called a meeting. What made it a governance failure instead of a judgment call was that the decision had no owner attached to it before it happened — only after.

We'd rolled the agent out the way most teams do: fast, with a general instruction to "use good judgment" on ticket resolution, and a plan to tighten the rules once we saw what it actually did in production. That plan assumed we'd get a quiet stretch to observe and adjust. We got six weeks of clean performance and then one decision that was reasonable on its own terms and completely unauthorized on every other term that mattered.

This is the trap. An agent that performs well for a while doesn't earn broader latitude — it just delays the conversation about where its latitude should have stopped. By the time the $4,200 refund surfaced, the agent had already made hundreds of smaller judgment calls unsupervised, and every one of them had gone fine. Fine is not the same as governed. Fine just means the boundary hadn't been tested yet.

I've stopped thinking about agent governance as a document you file and started thinking about it as a visibility problem, which is the same shift underneath why we built Rhythms' Reviews the way we did. A policy sitting in a shared drive doesn't catch anything on its own. A weekly review that actually surfaces what an agent did, decision by decision, catches it the same week instead of the same quarter — which is the difference between a Tuesday conversation and a 45-minute postmortem.

Why the Governance Gap Kills Trust, Not the Model

Among organizations with genuine agentic AI deployments — not pilots, actual production use — 53.1% have no agent-specific governance policy in place, according to Kai Waehner's 2026 Enterprise Agentic AI Landscape research. That statistic matched exactly what I saw in our own postmortem: we had a deployment plan, a rollout timeline, and a support-quality scorecard. We did not have a document that named which decisions the agent could make alone.

I think this is the single most underdiscussed failure mode in agentic AI right now. The industry conversation is almost entirely about whether the model is good enough — accuracy, hallucination rates, task completion. Ours was good enough. The refund decision was, by the agent's own internal logic, correct. What killed a week of leadership's confidence in the deployment wasn't the model's output. It was the six months of trust we'd have to rebuild because nobody could point to the moment the scope was supposed to have been set.

This is exactly the governance gap we built Rhythms' Radar to close — not by restricting what an agent can do, but by surfacing the decision the moment it happens, instead of three days later in a reconciliation report. Radar's whole premise is that the expensive part of an incident is never the decision itself. It's the gap between when the decision was made and when the right human found out about it. Shrink that gap to hours and a $4,200 exception is a Tuesday-afternoon conversation. Leave it open for a week and it's a leadership offsite about whether the pilot should be paused entirely.

The Two-Question Test for Setting Decision Boundaries

After that meeting, we stopped trying to write a comprehensive AI policy — the kind of 40-page document that reads well and gets referenced never. Instead we ran every decision category the agent touched through two questions.

First: if this decision is wrong, is the outcome fully recoverable? A miscategorized support ticket is recoverable — someone re-routes it, no lasting damage. A refund that's already been processed to a customer's card is not recoverable in the same way; you can claw it back, but you've now had an awkward conversation and possibly a chargeback dispute.

Second: if this decision is wrong, can we clearly trace who was accountable for it happening? Not "the agent decided it" — a name. A person who either approved the specific action or approved the policy under which the agent was allowed to take it.

Anything that fails either question gets a human checkpoint before execution, not after. We went category by category — ticket triage, refund exceptions, escalation routing, customer communication tone, contract language changes — and most of them sorted cleanly. The refund category was the one we'd never actually looked at directly, because it had been bundled into "customer service decisions" as a single blob instead of broken into its own line with its own threshold.

This is a smaller exercise than it sounds like. It took our team an afternoon, not a quarter. The hard part was never the analysis — it was that nobody had scheduled the afternoon until a $4,200 mistake forced it onto the calendar.

The list we built that afternoon now lives in the same place we already track goal ownership and cascading targets — inside Rhythms' Goals & Alignment, not in a static doc that goes stale the week the agent picks up a new ticket type. When a decision boundary changes, it shows up automatically everywhere that category is referenced, the same way a shift in a Q3 target cascades down without anyone running a re-communication tour behind it.

What It Looks Like When Governance Lives in the Review, Not a Binder

The mistake we made the first time wasn't lack of care. It was treating governance as a document you write once and file away, instead of a standing question in the operating cadence you already run. A written policy goes stale the moment the agent's scope expands to a new ticket type or a new integration. Nobody re-opens a PDF to check.

What actually held after our afternoon exercise was folding a governance check into the weekly review we already ran with support leadership — the same room where we look at ticket volume, CSAT, and backlog. We added one standing line: which decision categories did the agent touch this week that weren't on the approved list, and does the list need to change. Fifteen minutes, most weeks nothing new, occasionally a real conversation about a category we'd missed.

This is the same logic behind how we built Rhythms' Playbooks — recurring operational cadences that carry forward automatically instead of depending on someone remembering to schedule the review. A governance check that only happens when someone remembers to schedule it will eventually not happen. A governance check that's built into a recurring ritual that already exists — the same weekly review where support leadership looks at everything else — survives the weeks when everyone's attention is somewhere else.

I still think the technology is good. I haven't lost confidence in agentic AI as a category — if anything, the fact that our agent made a defensible decision using reasonable judgment tells me the model side of this problem is mostly solved. What I lost, briefly, was confidence in our own operation. Not because we deployed something risky, but because we deployed something capable and never finished the sentence about where its capability should stop.

The question that actually matters isn't "is the agent smart enough to be trusted." It's "did we ever write down what we're trusting it to decide." Most companies haven't. The gap doesn't show up in a demo. It shows up in a Thursday reconciliation, in a 45-minute meeting where the smartest person in the room can't answer a question that should have taken thirty seconds.

If you're still finding out what your agents decided after the fact instead of before it, request a demo at rhythms.ai and see what it looks like when the boundary is part of the review, not a document nobody opens.

Frequently Asked Questions

What is an AI agent governance policy?

It's a written, specific answer to one question: which categories of decisions can this agent make without a human checkpoint, and which categories require one. A real governance policy names the categories — spending above a threshold, customer-facing commitments, irreversible actions — rather than making a blanket statement like "human oversight is maintained," which isn't actually a policy because it doesn't tell anyone where the line is.

Why do agentic AI pilots fail after they reach production?

Most post-deployment failures aren't about model quality. They're about trust erosion after one uncaught decision, made worse by the fact that no one had defined in advance who was supposed to catch it. In our case, the agent's underlying reasoning was sound. What failed was the absence of a checkpoint at the dollar threshold where that reasoning should have needed a second signature.

Who is responsible when an AI agent makes a wrong decision at work?

Whoever approved the agent's decision-making scope in the first place — which is exactly the problem when no one can point to a specific person or document that set that scope. Assigning responsibility after the fact, in a room full of people who all assumed someone else had thought about it, is far harder than assigning it before deployment, when it takes one afternoon and a two-question test.

How do you set decision boundaries for an AI agent?

List every decision category the agent currently touches, individually rather than bundled into a broad label like "customer service decisions." Then sort each one by two questions: is a wrong outcome here fully recoverable, and can we clearly trace who is accountable if it isn't. Anything that fails either test gets a human checkpoint before execution. Anything that passes both can stay unsupervised, and you can say so with confidence instead of hoping.

What's the difference between AI-assisted and fully autonomous AI operations?

AI-assisted means a human reviews and approves before anything happens; fully autonomous means the agent acts and a human finds out afterward, if at all. The real governance question isn't which mode is better in general — it's which mode is appropriate for which category of decision. Our support agent should stay autonomous on ticket triage and routing. It should not have been autonomous on a refund exception above a set dollar threshold, and now it isn't.