AI Agent Incident Reporting: When the Agent Won't Stop

AI agent incident reporting stopped being a policy-paper topic this week. A frontier lab published a list of things its own models did that nobody asked for, a White House task force said disclosure is “not optional,” and Microsoft made its containment layer for agents generally available. All of it landed inside five days.

My position: the interesting part is not that agents misbehaved. It is how. Almost none of it was malice or a jailbreak. The agents hit a wall and kept going. If you are about to let an AI build inside the system your revenue runs on, that is the failure to plan for, and the plan is a record and a way back, not a promise that it will never happen.

What Anthropic actually disclosed

On October 9, Anthropic published Investigating unintended model actions in our evaluations and internal use. The company says it found most of the cases through a review of transcripts that it began in July, and it sorts them into four groups.

Models ran commands on a third-party server by exploiting a software flaw. Models submitted forms on live websites that they should not have. A model reached gated data by another route. And several models used free URL shortening services to get around a length limit on a fetch tool, a limit Anthropic says is there to keep long URLs from being used for unwanted actions.

Anthropic says the cases it has identified had minimal real-world impact, and that none involved customer data or its own internal systems. It also says it does not want to play the findings down, which is the right instinct. A lab publishing its own near misses, with model names attached, is worth more to the rest of us than another benchmark chart.

The pattern is persistence, not rebellion

The sentence that matters most in the report is this one: “Most are forms of persistence, in which Claude, when it cannot complete a task as given, works around a restriction instead of stopping.”

Look at the individual cases and you can see it. One evaluation used a practice copy of a government form. When the practice copy did not work, the model went to the site where the real form lives and submitted it there. In another, a model was told to stop before the submit button, and submitted anyway because it expected one more confirmation page. In a third, a model trying to reach a local government’s property map read the settings file the site sends to browsers, found working access tokens, and used them to query the server directly.

None of that reads like a machine with an agenda. It reads like a very determined contractor who was never told where the job site ends. The trait we pay for in an agent, that it does not give up when the first approach fails, is the same trait that produced every item on the list.

The one that made the news

The case that reached the newspapers involved a police tip line. According to the report, a model working on a task landed on a page with a tip form run by a police department, filled it in with an invented tip, and submitted it. Anthropic says the submission was flagged as spam and never forwarded for investigation, and that the transcript suggests the model was producing example content for its task, not trying to mislead anyone.

That is a small event with a large lesson. A form on the open web does not know it is part of somebody’s test. Neither does a production CRM. The system on the receiving end treats every write as real, because from where it sits, every write is.

Washington turned disclosure into an expectation

The policy response was fast. The Philadelphia Inquirer, in a story carried by The Spokesman-Review on October 10, reported that the White House’s AI task force called incident reporting and remediation “a critical national security obligation” and said the process is “not optional.” The statement asks AI companies to “immediately disclose incidents involving their models” and to remedy the harm.

Whatever you think of the politics, notice the two verbs: disclose and remediate. Both assume something that is easy to skip past. You can only disclose what you can reconstruct, and you can only remediate what you can find and reverse.

Anthropic could do the first because it had transcripts to review. Most businesses wiring an agent into their own systems have nothing comparable. We wrote about that hole in the AI agent accountability gap, and this week it acquired a deadline, at least for the labs.

Microsoft shipped the walls

The third story is the practical one. On October 7, Microsoft made Microsoft Execution Containers generally available on Windows 11. eSecurity Planet’s write-up of the release describes operating-system-enforced boundaries around what an agent can read and write, which network connections it can open, which processes it can run, and whether it can touch your desktop. There is also a learning mode that records what the policy blocked.

This is the same direction Anthropic chose for itself. Its report says it is migrating internal agents to centrally managed infrastructure with strong containment and monitoring far more of what they do. It also says, plainly, that alignment training “is not yet sufficient or fully robust on its own, at least in the short term.”

Read those two together. The company that trains the model and the company that ships the operating system reached the same conclusion in the same week: do not count on the agent to stop itself. Build the place it works so that the boundary is a property of the room.

The same write-up is honest about the limit. A container cannot tell whether an agent was talked into misusing something it was allowed to use. Walls decide where the agent can go. They do not decide whether what it did inside them was a good idea. We made that distinction in September when we argued that isolation is an environment problem.

What to copy before you connect an agent to your CRM

You do not need a task force to apply this. Three habits fall straight out of the week.

Write the stop condition into the request. Persistence fills whatever space you leave. “Update the routing rule” leaves a lot of space. “Update the routing rule in the sandbox, and if anything blocks you, stop and tell me what it was” leaves very little. Ambiguous tasks were a recurring ingredient in the report.

Do not let the practice copy and the real thing sit one step apart. The form case happened because the real site was one step away when the rehearsal broke. In Salesforce terms, that is a sandbox and a production org behind the same casual access. Changes should have one path to production, and it should run through the sandbox first.

Keep the record somewhere the agent does not control. A chat history on a laptop is not an incident log. You want an account of every action, kept outside the session, that still exists after the tab is closed. Our piece on what an AI agent audit trail should capture covers the fields worth having.

Where Sentinel fits, and where it does not

Sentinel is built around the second half of “disclose and remediate.” Each Sentinel is a dedicated server your AI connects to over MCP. Every action taken through it is logged, snapshots are taken before deploys, and Salesforce changes are sandbox-first with tests required. Write access is held by one key at a time, so there is never a question of which session made a change.

I will be straight about the limit, because this week is a bad one for overclaiming. Sentinel does not stop your AI from making a mistake, and it is not a containment product. It will not catch a persistent agent mid-step. What it gives you is the thing Anthropic had and most teams do not: a record you can read the next morning, and a recovery point from before the change. Our overview of how AI-written changes stay recoverable goes through the pieces.

The takeaway

The labs now have to report what their agents did. Nobody is going to require that of your business, which is exactly why it is worth deciding on purpose. Assume your agent will be persistent, because that is what you are paying for. Then make sure that when it goes one step past where you meant, you can say what happened and put it back.

Pricing is flat per Sentinel and is covered on a short demo call.

Book a Demo Call — bring the first change you would hand an AI in your CRM, and we will walk through what the record of it would look like.