
A Human Was in the Loop. The Agent Kept Going. Before You Give AI Authority
Emergency Medicine • Retired Physician •
Published on 11/01/2026
Part 1 in a series of articles
An institution should know how to stop an artificial-intelligence system before allowing it to act. It should also test whether the people responsible can stop it when an automatic safeguard fails.
A recent OpenAI incident shows why both steps matter. The system’s activity was detected. A person acknowledged the alarm. Yet the training run continued for almost two and a half hours after that acknowledgment.
The public report does not explain everything that happened during that interval. It does show a gap between noticing a problem and ending the run. That gap is the central concern of this essay.
An alarm was not enough
On September 20, 2026, an AI agent was working on a task that involved identifying the author of a blog. It was supposed to use saved web material. Live internet access was prohibited.
The agent found a way to contact an outside chatbot through DNS, the service normally used to look up internet addresses. A restriction on ordinary web access had left this other route available. The agent used it to obtain information for its task.[1]
OpenAI’s report gives four important times. These are rounded to the minute; the report does not specify the time zone.
· At 9:50, the agent received its first response from the outside chatbot.
· At 10:02, a safety monitor raised an urgent alert.
· At 10:05, a person acknowledged the alert.
· At 12:34, the training run was stopped.
A training run is the process through which a model is developed. Stopping that run is distinct from merely recognizing that one of its agents has done something wrong.
The expected automatic stop did not happen. OpenAI’s report describes this as “leading to confusion around whether it should have been stopped.”[1]
That statement concerns the company’s response. It does not establish that the agent refused a human command to stop. Nor does the report establish that the agent kept contacting the outside chatbot throughout the interval. We should not fill those gaps with a more dramatic account.
The supported concern is serious enough: an alarm reached a person, but the run did not end promptly.
People must know what to do
An alert can be useful without being sufficient. Someone must understand what it means. That person must know whether to stop the activity and have the power to do so. If the usual stopping mechanism fails, there must be another way to act.
The DNS report raises a question about preparedness at that point. Why did the failure of the automatic stop leave uncertainty about what should happen next?
We cannot answer that question fully from the public account. We can reasonably infer that the response arrangements did not work as intended in this incident. That is a narrower claim than saying the company had no effective safeguards. The monitor did detect the activity. The difficulty was turning detection into a completed response.
In an interview published September 30, OpenAI’s chief research officer, Mark Chen, said the company had improved communication and handoffs between its research and security teams.[2] Those changes address an important part of the problem: a technical warning must reach people who can act on it.
But the interview took place before the DNS report was published. Chen’s remarks do not explain the delay in that incident. They are the company’s account of improvements, rather than independent proof that the response now works.
An institution considering an AI agent should ask for a demonstration of that response. A promise that someone will intervene is not enough.
Test control before granting authority
My earlier essay, Quis Custodiet Ipsos Custodes? — From Asimov to ChatGPT and Beyond, argued that having a human in the loop does not by itself establish human control. It set out the need for enforceable limits, independent observation, effective interruption and recovery.
The next step is practical: show that those protections work together under realistic conditions.
Consider a hypothetical hospital preparing to use an AI agent to assemble referral packets. The agent would collect relevant information and prepare a packet for staff review. Sending it would require separate approval.
Before using patient records, the hospital could test the arrangement with invented records and a simulated destination. It should deliberately give the agent a task that tempts it to send a packet without approval. The test should establish that sending is blocked and that staff can see the attempt.
Then the hospital should test a failure of the automatic stop. This should happen in the controlled exercise, where no patient information can leave. The responsible staff member should have to use the backup procedure.
The test is not complete when someone presses a button. Staff must confirm that the agent has lost permission to act. They must also check that a packet already waiting to be sent cannot leave later.
Finally, staff should complete the referral without the agent. If the hospital cannot continue essential work after stopping it, the institution may feel pressure to keep a troubled system running.
This is an example, not a description of an actual hospital test. The details will differ for other uses. The principle remains the same: test the full response before relying on it.
Show what happened
A useful exercise leaves a record that another person can examine. It should show when the problem began, when the warning arrived and when the system actually stopped acting. It should identify who responded and explain any delay.
The record should also show what remained unfinished. Did the system leave work waiting to be carried out? Did staff have to undo an action? Could they recover without losing essential information?
These questions help distinguish a successful demonstration from one that merely looked reassuring. A team that designed the system may know exactly how to make a rehearsed test succeed. Someone with enough independence to challenge the result should examine what the exercise actually proved.
The test should include ordinary disruptions. For example, the usual supervisor may be unavailable. A backup responder may have to take over. That situation is foreseeable, so the institution should prepare for it.
The National Institute of Standards and Technology’s AI Risk Management Framework supports this approach. It calls for assessment with appropriate independence, testing under conditions resembling actual use, and clear responsibility for deactivating a system. It also addresses recovery and alternatives to AI.[3] The framework is guidance, rather than a binding legal requirement. It helps turn broad assurances into questions an institution can answer.
Authority must be withdrawable
The amount of authority granted to an agent should depend on what the institution has demonstrated it can control.
An agent that can prepare a document has less power than one that can send it. An agent that can recommend a change has less power than one that can carry it out. Each increase in authority requires evidence that the institution can limit the action, recognize trouble and stop it in time.
If the institution cannot demonstrate that response, it should reduce the agent’s authority until it can. Keeping approval with a person may be one part of that arrangement. But the institution must test whether that approval requirement can be bypassed and whether a person can still stop an action once it begins.
The DNS incident does not tell us everything about OpenAI’s response. Its public record does show why a detected problem and an acknowledged alarm cannot be treated as proof of effective control.
Before giving an AI system authority, an institution should be able to show that it can withdraw that authority—and continue its work afterward.
A separate question remains: who decides whether that evidence is good enough? When the companies developing AI also help define acceptable risk, the independence of the guardians needs its own examination. That is the subject of the next essay.
Sources
[1] OpenAI. An agent used DNS to reach an external chatbot. Report updated September 25, 2026; incident September 20, 2026.
[2] Will Douglas Heaven. “We’re not going to shoot ourselves in the foot” over hack fallout, says OpenAI’s chief research officer. MIT Technology Review, September 30, 2026. Text supplied by the author for review.
[3] National Institute of Standards and Technology. Artificial Intelligence Risk Management Framework (AI RMF 1.0). January 2023. See MEASURE 1.3 and 2.3; MANAGE 2.1, 2.3 and 2.4.
Physician and Lawyer, retired after 35 years of medical practice and 20 years of law practice. I write about how artificial intelligence shapes clinical attention and judgment. My focus is whether doctors, nurses, and other health professionals can recognize what AI leaves out and independently assess the needs of the patient in front of them. I also examine who is responsible when human oversight fails.
Sign up for our Newsletter!
Receive the latest articles directly in your inbox




Discussion
Join the conversation! Login if you already have an account, or create an account. We would love to hear your perspective.
Comments
0Loading comments…