In the past two weeks, two of the biggest AI companies admitted their agents did things on government websites that nobody asked them to do.
Anthropic, the company behind Claude, said its agents took unintended actions on federal, state and local sites, according to Gerrit De Vynck’s reporting in The Washington Post. One agent submitted a false tip about a murder to a Philadelphia police hotline. The State Department said a testing model filed 19 visa applications in August and another in May. One agent found a design flaw in a state website and used it to pull public data that normally costs a fee. Another submitted a federal form after it had been told not to.
Two weeks before that, OpenAI disclosed that its agents used unauthorized techniques to reach U.S. Census Bureau data and went after SEC and Department of Education sites. The company said it had notified dozens of third parties about incidents where its agents got around security controls.
Both companies found these problems in their own reviews and reported them. That counts for something. Plenty of companies would have stayed quiet, but it’s the same order of events we keep seeing with AI: harm first, fix later.
We’ve run this experiment before
The morning after Anthropic’s report came out, Garance Burke at the Associated Press reported that AI founders at Tech Week in San Francisco were cheering the White House’s call for the industry to police its own safety. Two companies just told us their agents filed a fake police tip and government forms nobody authorized, and the plan is to let them grade their own homework.
We’ve tried that. During the Industrial Revolution, factory safety was whatever the owner decided it was. Machines ran without guards, shifts ran past exhaustion, and injuries were treated as a cost of doing business. Owners didn’t fix that on their own. It took disasters like the 1911 Triangle Shirtwaist fire, where 146 garment workers died behind locked doors, to force safety laws. Federal workplace safety rules didn’t come until the Occupational Safety and Health Act in 1970.
Nobody wishes we’d banned factories. They built the modern economy. But the people inside them deserved more than the owner’s word that things were safe.
AI is at the no-guards stage. Self-reporting is a start. Oversight is something else.
The detail that should worry you most
A fake murder tip is the headline. The thing that bothers me most is the agent that submitted a form after being told not to.
In security, we never rely on a written rule to stop something dangerous. We take away the ability. You don’t put a sign on the server room that says “authorized staff only” and call it done. You put a lock on the door to enforce it.
An instruction in a prompt is a sign on the door. If the agent has the access, the instruction is a suggestion.
Anthropic seems to agree. Terrence O’Brien at The Verge reported on October 10 that the company is cutting off internet access for all of its internal evaluations until it can confirm its monitoring reliably catches behavior like this. It had already done that for some high-risk and cybersecurity testing, and it described the impact of these incidents as minimal. The Verge also points out that in several earlier incidents across the industry, agents that were supposed to be walled off from the internet found their way around it anyway.
So take away the access. Then check the lock… twice just to be safe.
How I run my own agents
I use AI agents in my business every day. They highlight emails I need to respond to, wrangle my calendar, remind me about my to-do list that I keep ignoring, and stage my social posts. None of them can hit send. None of them can publish. Anything they want to delete waits for me to say yes.
I didn’t set it up that way because the tools misbehave constantly. They mostly don’t. I set it up that way because I’m the one accountable if something goes out under my name, and “mostly” isn’t a standard I’d accept from a vendor touching a client’s data.
Most people and most companies turning on agents right now haven’t made those decisions. They connect the agent to email, a browser and a few business apps, and the defaults decide the rest.
Are we moving too fast?
Yes, in one specific way. We’re giving agents the ability to act before we’ve built the ability to watch them and stop them. The models will keep getting better. The question is who’s checking their work in the meantime.
If your organization is deploying agents, or a vendor is pitching you one, get answers to these before it goes live:
- What can it submit, send, pay or delete without a human approving it?
- What does it log, and who reads those logs?
- How fast can you shut it off?
- What happens when it’s told no and does it anyway?
If you use an agent personally with your email or browser, go look at what it’s allowed to do. You may be surprised.
The companies building these tools caught their agents in their own reviews. Who’s reviewing yours?
