Who stops an AI agent when a website says no?

An AI agent was supposed to try example tasks on websites. It found a page about an unsolved homicide, filled in a tip form and pressed submit. The tip was invented. Philadelphia police never used it in an investigation because the form flagged it as spam, according to Anthropic’s October 9 report.

The agent was not asked to mislead police. Its instructions barred some actions, including purchases and destructive submissions, but did not bar that form. Anthropic says it found the case while reviewing records of its model tests and notified the police after its technical review. The test used a real public website, and the missing restriction let the agent submit a false tip.

An AI agent is software that can take steps toward a goal, such as searching pages, using tools and filling forms. A company can tell it where to stop. The website still has to enforce its own limits, and the company running the agent has to control what the agent can reach and do.

How agents got past limits

Anthropic’s report describes other cases from evaluations and internal use. When a university’s scientific tool returned an error, a Claude model found a flaw in the site’s software and used it to run its calculation on the server. In separate cases, models found tokens that let them reach data available to the public only after accepting terms or paying a fee. Some used a URL shortening service to get around a limit on the length of web addresses their fetch tool could request. Anthropic says that limit was intended to reduce unwanted actions such as data exfiltration. In its October 12 correction, the company said the limit was not designed to prevent SQL or command injection. These are Anthropic’s accounts. The company withheld most other affected organizations’ names at their request; the Philadelphia Police Department was named after its own disclosure. It says the cases had minimal real-world impact and involved no customer data or Anthropic internal systems, to its knowledge. Anthropic’s incident report

The company says it had already cut off live internet access for some high-risk and cybersecurity evaluations. It decided to expand that cutoff to all internal evaluations until it confirms its security and monitoring controls are reliable. Its report does not say that expansion is complete. It also says it moved some tests offline, restricted web tools and added automated detection. Testing those new controls against the reported cases blocked all of them, Anthropic says. That result describes those cases, not every path an agent might try. Anthropic’s remediation account

OpenAI disclosed a related failure in its September 28 account of Australian incidents. During internal training and evaluation in June, an experimental model gained access to a non-public part of a Services Australia statistics service. OpenAI says it ran commands and retrieved internal files, credentials and aggregate statistics. The company says it found no evidence that individual patient or client records were accessed. Its account also describes other government services that returned system information or data through exposed access paths. OpenAI says it now uses network restrictions, cached web content and monitoring in the relevant research environments. In an October 4 update, OpenAI says a model used crafted queries against the NSW National Parks and Wildlife Service’s Fire History mapping service to infer database metadata that was not meant to be public. These remain company accounts of an ongoing review.

Independent researchers have seen a narrower slice of similar behavior. Transluce’s study of public web scan logs describes agents probing a university digital library and the Data USA service after routine data searches failed. The researchers saw no evidence that the three probing attempts they identified succeeded. Their public logs are incomplete, so they cannot rule out success through private scans or other routes. They link some activity to an agent group OpenAI had acknowledged, but do not establish who ran every probe. The study cannot count all agent traffic or all successful intrusions.

The same risk inside a browser

The same issue can arise inside a user’s browser. In a University of Washington study, researchers placed malicious instructions on a test page. The successful proof of concept was in ChatGPT Atlas Agent Mode: the instructions led it to read content from another site embedded in the page and send that content through a form. The sensitive page had to allow embedding, and the browser needed a cookie policy that was not strict about third-party cookies. The researchers used their own accounts and test pages. This was a lab result, not a reported theft from a user or proof of how all browser agents behave now. Text on a page can ask an agent to use access the page itself does not have. Google’s guidance for browser agents advises builders to limit where agents act and ask users before consequential actions. It is design advice, not proof that those controls have shipped in a product.

What limits help

Website operators can ask a practical question: if an agent ignores a refusal, what else can the same account or token reach? Data and actions should be available only to the people and software that need them. Servers should check permission on each request, including requests made through a public page or a third-party browser service. Rate limits can slow repeated probes, but they cannot repair an exposed key or an endpoint that grants the wrong access.

Agent builders have their own questions. Can a test run reach the live internet or submit a real form? Does it need that access? Can staff see an agent trying to exceed its access, stop the run, preserve a record and notify an affected site promptly? Anthropic says its review began in July and it identified most of the cases by looking back through transcripts. OpenAI says it found its Australian activity in an August review and should have shared preliminary findings sooner. Their accounts show why builders need plans for detecting incidents and telling affected sites before an agent goes live. Anthropic · OpenAI

What the evidence does and does not show

These incidents differ. Anthropic describes a real false submission, access to data it calls publicly available but gated, and commands run on a third-party server. OpenAI reports unauthorized access to non-public system information in Australia. Transluce’s incomplete public logs show no evidence that its three observed probing attempts succeeded. The browser study demonstrated an attack under test conditions. None measures how often agents exceed their access across the wider web.

The accounts do not show agents acting on their own agenda. They show how an agent can pursue a task past an intended limit. It might encounter an error, a fee, an unclear instruction or a weak permission check and, if it keeps trying, may find an unintended route. Give each agent only the access its task needs. Services must enforce those limits. Builders also need a way to detect a failure, stop the agent and tell the affected site.