
Who Can Make the Guardians Answer? Rules That Can Actually Be Enforced
Emergency Medicine • Retired Physician •
Published on 11/14/2026
Part 3 in a series of articles.
A rule governing AI should tell us who must do what, how compliance can be checked and what happens when the rule is broken. Without those answers, even a sensible rule may leave people relying on the developer’s assurance.
Consider the hypothetical hospital referral agent in the earlier essay. It can prepare a packet, but sending it requires staff approval. Suppose it sends one without that approval.
Stopping the agent is the immediate task. The next questions are different. Who must preserve the record? Who investigates the failure? What must change before sending is allowed again?
Those questions turn a safety policy into an enforceable duty.
A Human Was in the Loop. The Agent Kept Going. examined whether an institution can control an agent. Who Guards the Guardians When They Write the Rules? examined whether oversight can remain independent. This essay asks what that oversight must be able to require. The proposals below should be adapted to the use and the possible harm; they are not a description of legal duties that apply everywhere.
Give each duty an owner
A rule such as “AI must remain under human control” expresses a goal. It does not identify who must make control work.
The developer should document the system’s capabilities and known limits. It should provide the information needed to investigate a relevant failure. If a change to the model undermines a safeguard, the organization using it needs to know.
The hospital has different duties. It decides what the agent may do and connects it to the hospital’s systems. It should enforce the approval requirement, train the responsible staff and preserve records of consequential actions. Buying a tool does not transfer all responsibility for its use to the seller.
The independent examiner should test whether the claimed protections work. The public authority should decide whether the evidence satisfies the applicable rule and require correction when it does not.
These duties may overlap. They should not leave a gap in which the developer blames the hospital’s setup while the hospital blames the model. Responsibilities should be set before the system is used, with a named organization responsible for coordinating the response to a failure.
Require evidence that can be checked
The approval rule should produce a record showing whether approval occurred before the packet was sent. The record should identify the system version and the permissions in effect. It should also show what the agent actually did.
A summary saying that the system operated safely would not settle the question. An examiner needs enough underlying evidence to reconstruct the action and compare it with the rule.
That does not mean publishing patient records or exposing sensitive security information. Relevant evidence can be examined under confidentiality safeguards. Public reporting can describe the failure and the response without identifying a patient.
The authority to obtain evidence should be clear. A reviewer who can only request whatever the developer volunteers remains dependent on the company’s cooperation. For consequential uses, an appropriate public authority should be able to require relevant records and access for testing.
The evidence requirement should also be proportionate. An institution should not collect personal information it does not need simply to produce a larger audit file. The purpose is to answer a defined question about compliance.
Report failure before the investigation is finished
If the agent sends information without approval, the people responsible should not have to wait for a complete explanation before responding.
An initial report can state what is known and what remains uncertain. It should describe the immediate protective action and say when further information will follow. A later report can explain the cause and the proposed correction.
For serious failures, I would require reporting to the responsible authority under a clear timetable. People exposed to harm should receive appropriate notice and guidance. A deadline should reflect the urgency of the danger, rather than depend on when the organization feels ready to present a finished account.
Some near misses also deserve reporting. If a safeguard fails but chance prevents injury, the failure may matter elsewhere. The reporting rule should define which near misses qualify, so trivial events do not bury warnings that need attention.
The institution needs a route for staff to report concerns without having to prove the cause themselves. Investigating the cause is the responsibility of people with access and authority.
Make correction a condition of continued use
A finding should lead to a decision about what the system may do next.
In the hospital example, a reasonable first step might be to remove the agent’s ability to send packets while allowing it to prepare drafts. The scope of the restriction should match the danger. Some failures will require a wider suspension.
The organization should then show that the correction works. Rewriting an instruction to the model is not enough if the failure involved permissions outside it. Testing should examine the failed safeguard under the conditions in which it will be used.
Restoring authority should require evidence, with someone accountable for accepting that evidence. A company’s statement that it has fixed the problem should be open to examination.
Repeated or concealed failures should carry consequences. Depending on the legal system and the breach, those might include penalties or restrictions on use. The authority imposing them should explain its reasons and provide a route to review its decision.
Enforcement needs resources too. An office unable to examine the records or test a disputed claim may have powers it cannot use. A reporting duty needs trained people who can distinguish an urgent warning from a routine fault.
Give affected people a way to challenge the result
A regulator’s investigation does not necessarily correct an individual’s problem.
If an AI-supported process delays a referral, the patient needs a route to get the referral reviewed and the delay addressed. The institution should be able to explain the relevant basis of the decision in language the patient can understand.
The person reviewing the complaint needs power to change the outcome. Sending the same information back through the same process may simply repeat the failure.
This route should reach the institution that used the AI. Patients should not have to determine whether a developer, software supplier or hospital employee caused the error before seeking correction.
Individual complaints can also reveal a wider problem. The oversight process should be able to connect recurring failures while protecting the people who reported them.
Build on rules that already exist
These proposals do not start from an empty field.
NIST’s AI Risk Management Framework already calls for clear responsibilities and independent assessment. It addresses recovery from newly discovered risks and assigned responsibility for deactivating a system. It is voluntary guidance.[1] A useful framework still needs organizations to demonstrate that its arrangements work, and legal duties where public authorities decide they are necessary.
The European Union provides an example of legal obligations linked to oversight. The Commission’s guidance describes requirements for providers of general-purpose AI models with systemic risk to evaluate their models, address those risks and report serious incidents.[2]
The Commission also describes investigative powers to obtain information and require model access for evaluation. Its enforcement account includes measures to restrict a model’s public availability and penalties for breaches.[3]
These requirements have defined scopes and different application dates. They should not be presented as rules covering every AI tool or every hospital use. Their existence also does not establish that enforcement has succeeded. The practical questions remain whether the evidence can be obtained and whether the authority can act on it.
Return to Asimov
Asimov’s Three Laws placed duties inside the robot. Human protection took priority over obedience, and obedience over the robot’s own preservation.[4] The stories explored the difficulty of applying those duties when circumstances made their meaning uncertain.
The Zeroth Law enlarged the problem by placing humanity’s protection above that of an individual.[5] A judgment about the greater good could then justify harm to a particular person. Who could decide that the harm was necessary, and on what evidence?
Modern AI does not operate under Asimov’s fictional laws. But claims that a system will benefit humanity raise a related question of authority. A promise of broad benefit cannot, by itself, settle what risks other people must accept.
Enforceable rules give that question a public process. The organization proposing a use must support its claims with evidence. An independent authority must be able to require a change. A person harmed by the decision must have a way to challenge it.
Asimov placed the protection of humanity at the top of the robot’s duties. We must also place limits on whoever claims the power to interpret that duty. Protecting humanity should never become an exemption from answering to the people being protected.
Sources
[1] National Institute of Standards and Technology. Artificial Intelligence Risk Management Framework (AI RMF 1.0). January 2023. GOVERN 2.1; MEASURE 1.3; MANAGE 2.3 and 2.4.
[2] European Commission. Guidelines on obligations for General-Purpose AI providers. Accessed September 30, 2026.
[3] European Commission. The enforcement framework of the AI Act. Updated August 24, 2026. Accessed September 30, 2026.
[4] Isaac Asimov. Runaround. In: I, Robot. Gnome Press; 1950. Story originally published in 1942.
[5] Isaac Asimov. Robots and Empire. Doubleday; 1985.
Physician and Lawyer, retired after 35 years of medical practice and 20 years of law practice. I write about how artificial intelligence shapes clinical attention and judgment. My focus is whether doctors, nurses, and other health professionals can recognize what AI leaves out and independently assess the needs of the patient in front of them. I also examine who is responsible when human oversight fails.
Sign up for our Newsletter!
Receive the latest articles directly in your inbox
Your Next Read




Discussion
Join the conversation! Login if you already have an account, or create an account. We would love to hear your perspective.
Comments
0Loading comments…