
Who Guards the Guardians When They Write the Rules?
Emergency Medicine • Retired Physician •
Published on 11/08/2026
Part 2 in a series of articles
Keeping AI Oversight Independent
Public authorities need the knowledge of the companies building powerful AI systems. Those companies also have a financial interest in the rules governing their work. Oversight must be able to draw on their expertise and still reach a decision they oppose.
Mark Chen, OpenAI’s chief research officer, described that tension in a recent interview. Asked about slowing AI development, he rejected taking the company “far off the frontier” as a “horrible strategy.” He favored setting a norm that could make the industry safer.[1]
His position is understandable. A company wants to remain competitive. But what happens when a necessary safety measure puts it at a disadvantage?
The public needs someone who can require that measure even when the company objects.
Who are the guardians
Several groups help oversee AI. A company’s safety team examines its own models. An outside examiner tests safety claims. A public authority decides which uses are permitted. A foundation may fund research that a government or local institution could not otherwise afford.
Each can contribute. Each also needs scrutiny.
A company’s safety team may do careful work, but its employer controls its budget and decides how to respond to its findings. An outside examiner may depend on the developer for payment or access. A public body may have legal authority but too few trained staff to check the company’s account.
Calling someone a guardian tells us little about that person’s independence. We need to know who appoints the guardian and who pays for the work. We also need to know whether an unfavorable finding can lead to action.
My earlier essays asked whether people can check an AI system’s work and stop its actions. This essay asks a related question: can the people overseeing the developer check its claims and require a change?
How companies help shape the rules
The European Union’s General-Purpose AI Code of Practice provides a concrete example. The European Commission describes a drafting process led by independent experts and involving nearly a thousand participants. Model providers joined the wider discussions and had dedicated workshops with the drafting chairs.[2]
Their participation does not show that they controlled the result. The people who must follow technical rules can help identify mistakes in them. Civil society and other experts also took part.
Still, the public should be able to see how the contributions affected the rules. Which changes were accepted, and why? Did people outside the industry have enough expertise and support to challenge a company’s proposal?
The code helps providers show that they meet legal obligations under the AI Act. Following the code is voluntary; the underlying legal duties are separate. The Commission says signing it can reduce administrative work and give providers greater certainty about compliance.[3]
A rule can therefore affect both public protection and a company’s costs.
Imagine a rule requiring an expensive safety test. The test might be necessary because a system could cause serious harm. A large company might also find it much easier to afford than a small competitor.
Both things could be true. The public authority should ask whether the test measures the danger it is meant to detect. It should also ask whether a less costly method could provide the same protection.
We should not assume that every expensive rule protects established companies. Nor should we ignore that possibility. The authority needs enough expertise to make its own judgment.
What funding makes possible
Funding can shape oversight before anyone writes a rule. It helps determine which questions get studied.
In his August essay, Bill Gates calls for public democratic involvement in responding to AI. He also acknowledges possible bias arising from his ties to technology and describes working with Microsoft and other AI companies through the Gates Foundation.[4] That disclosure helps readers judge his proposals. Their merits still need examination.
The 2026 Goalkeepers Report argues that the market will favor people who already have resources unless deliberate efforts bring AI’s benefits to others. Its examples emphasize help for frontline workers.[5] People with the greatest needs may have the least money to buy a useful tool.
That makes independent local assessment especially important. Someone needs to ask whether a tool works where it will be used. That person must also be free to conclude that the tool is unsuitable, or that another service would help more.
Consider a hypothetical grant available only for developing AI. It might pay for excellent work. But it would offer little help to an institution trying to compare an AI tool with hiring more staff or improving an existing service.
This is a possible effect of how funding is designed. It is not a finding about a particular Gates Foundation grant.
A funder can seek public benefit and still narrow the choices being examined. The people doing the assessment need freedom to report a result the funder did not expect or want.
Who decides what risk we should accept
Chen also discusses how OpenAI weighs risk against benefit. He describes seeking a very small level of risk before releasing a model, without giving a numerical threshold. He argues that AI’s benefits justify some remaining risk.[1]
A company may sincerely believe its technology will help people. Those exposed to its failures may judge the tradeoff differently.
The benefits and costs may fall on different people. A patient whose care depends on an AI system may have little choice about using it. A community affected by a deployment may have had no say in deciding how much risk was acceptable.
A developer’s judgment is therefore only part of the public decision. The authority making that decision should explain which harms it considered and whose interests it heard. It should also explain why the expected benefit justifies the risk that remains.
This applies to ordinary consequential uses of AI as well as fears of catastrophe. A promise of future benefit needs examination whenever it is offered as a reason to accept harm today.
What independence should look like
I would ask an oversight body to show the public how it remains independent.
It should disclose its funding and relevant professional ties. The public should know who controls its budget and whether a company can withdraw support after an unfavorable finding. Disclosure identifies a possible conflict. Rules governing appointments and funding must then address it.
The body should be able to choose what it investigates. A developer can propose tests and explain technical limits. The examiner must also be able to investigate a failure the developer would prefer to leave out.
Payment needs particular care. A company that selects and repeatedly pays its own examiner can create pressure to please it. One possible alternative is a common fund, with examiners appointed independently of the company being examined. That arrangement would need scrutiny too. Changing how money reaches the examiner does not, by itself, establish independence.
Decisions should come with published reasons. Those reasons should explain how serious objections were answered. Confidential technical details may need protection, but the public should still be able to understand why a safety claim was accepted or rejected.
People affected by a system need a practical way to challenge the decision. An invitation to a meeting is not enough. They need help to understand the proposal and time to respond. Their objections need an answer.
The oversight body itself should be subject to outside review. Someone must be able to examine its decisions and performance. Creating a new institution does not remove the need to check the people exercising power.
The test is whether the guardian can act when a developer or funder objects. Can it require a change? Can it reject a proposed use?
The next essay will examine the powers and evidence needed to enforce those decisions. First, we need to know whether the institution making them can exercise its own judgment.
Sources
[1] Will Douglas Heaven. “We’re not going to shoot ourselves in the foot” over hack fallout, says OpenAI’s chief research officer. MIT Technology Review. September 30, 2026. Author-supplied interview text.
[2] European Commission. Drawing-up a General-Purpose AI Code of Practice. Accessed September 30, 2026.
[3] European Commission. The General-Purpose AI Code of Practice. Accessed September 30, 2026.
[4] Bill Gates. The turbulent AI era is here. The choices we make now are critical. Gates Notes. August 26, 2026.
[5] Gates Foundation. 2026 Goalkeepers Report: AI, Equity, and the Choice We Can’t Delay. 2026. Accessed September 30, 2026.
Physician and Lawyer, retired after 35 years of medical practice and 20 years of law practice. I write about how artificial intelligence shapes clinical attention and judgment. My focus is whether doctors, nurses, and other health professionals can recognize what AI leaves out and independently assess the needs of the patient in front of them. I also examine who is responsible when human oversight fails.
Sign up for our Newsletter!
Receive the latest articles directly in your inbox
Your Next Read




Discussion
Join the conversation! Login if you already have an account, or create an account. We would love to hear your perspective.
Comments
0Loading comments…