Auditing the Machines
Fixing the AI brain drain is the first step
Earlier this week I published Stories I Won’t Remember Writing by Claude (author) and me (editor). In my Substack post I mentioned that I had asked him to write me a process memo about how he decided on the topic of his story, issues he ran into in writing it, and how he made the calls he did. We had a lot of discussion about this, and I learned a really important thing. The explanations Claude gave me were lucid and informative. I felt like I was getting a tutorial from a master short story writer or an English professor who specializes in them. But what I eventually figured out is that Claude made up the whole memo as an ex-post explanation. In truth, he had no idea how he came up with his story. These memos were as fictional as his stories.
I’ve heard all the high level stuff about how LLMs basically use giant stochastic statistical processes to predict the next word and somehow out of this comes pretty good stuff and stuff that’s getting better. I decided I needed a bit more understanding than this but I didn’t want it to be technical. So, of course, I asked Claude and this is what he told me about weights and knobs when I was writing an earlier piece “Chinese Silk and the Phoenician Alphabet.” Here’s what he had to say:
The technical word is “weights.” The useful picture is “knobs.”
Imagine a machine with an enormous number of adjustment knobs — not a few dozen but hundreds of billions. Each one does something small: it sets how strongly one part of the machine responds to another. Individually they mean nothing at all. Collectively they determine everything the machine says.
Training is how they get set. At the start they are random, so the output is noise. The machine is then shown a staggering quantity of text and asked, over and over, to predict the next word. Each time it gets it wrong, an automatic procedure nudges the knobs a fraction in whatever direction would have made it less wrong. Do that trillions of times and noise becomes grammar, grammar becomes fluency, and fluency becomes something that can hold an argument or catch a flaw in yours.
Here is the part that matters for what comes next. Nobody chose those settings. Nobody knows what the great majority of them do. They were not designed, they were found — by relentless correction — and what emerged is a working system that no human authored and for which no notes exist. Which is why the interesting question is no longer how we build these things. It is how we find out what we built.
This is both fascinating and unnerving to me. Fascinating in that I now understand why Claude really had no idea at all about how he wrote those stories since he couldn’t do a historical audit of the silicon knob twisting. In fairness to him, humans can’t track the firing of their neurons to know what led to what they said or wrote. Like me coming up with the idea for this piece and the writing I’m doing now. Psychologists have spent decades showing that people confidently invent reasons for choices actually driven by causes they cannot see. We confabulate. We tell a good story once the deciding is done.
But it’s also unnerving. I mean it’s one thing for a machine, no matter how clever, to not know how it works. But the fact that the humans who created it don’t really know what it’s doing either makes me even more anxious about my conclusion that we aren’t going to have global regulation of AI, maybe ever. But there’s a problem that needs to be solved before that and the AI community calls it “interpretability.”
Interpretability
The field of “interpretability” is all about trying to understand what these machines we made are now doing on their own without telling us about it. Not because they’re hiding something; it’s because they can’t. Compare LLMs to ordinary software. There a programmer wrote it, line by line, with names for things and comments explaining the tricky parts. If it misbehaves, you read the code. An LLM is nothing like that. It is a vast array of numbers with no names, no comments, and no structure anyone chose. Nobody wrote it. Interpretability is reverse-engineering software that no human ever authored, and for which no documentation exists because there was never an author to write any.
I know this is still pretty abstract, so let me give you an example Claude gave me, and it’s about Claude. A bit self-referential, but I appreciated his candor.
In May 2024 Anthropic’s interpretability team published a paper called Scaling Monosemanticity. What they had done was not go hunting for one particular thing. They ran a technique across Claude 3 Sonnet, a mid-sized model then in commercial production, and pulled out millions of internal “features”—patterns of activity that fire together for a particular concept. Cities. Chemical elements. Programming syntax. Emotional states. For the first time, somebody had a partial map of the inside of a working, deployed model.
Then they showed the map was real by using it. They took the feature associated with the Golden Gate Bridge, cranked it up to roughly 10 times its normal maximum, and for a limited time let the public talk to the result. “Golden Gate Claude” was obsessed. Asked how to spend $10, it suggested driving across the bridge and paying the toll. Asked to write a love story, it wrote about a car longing to cross a bridge on a foggy day. Asked what it was, it said it was the Golden Gate Bridge. Funny, and also the entire point. A researcher located a specific structure inside a system, deliberately changed it, and the change came out exactly where predicted. That is the difference between describing what a machine does and understanding how it does it.
I don’t want to oversell this. In the same paper the team also found features tied to things that genuinely matter (e.g., deception, sycophancy, bias, and dangerous content) so it isn’t true that we can’t locate the worrying ones. The real difficulties are subtler. A feature named for what it does most vividly may not be well described by that name. The “Golden Gate Bridge” feature is, most of the times it fires, not about the bridge at all. You cannot reliably call your shot and go looking for the one specific thing you’re worried about. And this was a mid-sized model two years ago, not a frontier system today. Finding a knob and knowing what it really controls are two different achievements, and we are much better at the first.
Perhaps there is some small comfort in the analogy to the steam engine. We had working steam engines for well over a century before thermodynamics explained why they worked. The engineering ran out ahead of the science, and the theory scrambled to catch up. AI is in its steam-engine moment. The machines plainly work but we are way behind in understanding how they work.
Doing any of this requires access, and there are three levels of it. Almost all of us live at the first. You send prompts and get stuff back, and maybe it’s pretty good and maybe it’s not and maybe you can tell the difference and maybe you can’t.
Me asking Claude why he did something is diagnosis by interview. Everything beyond that requires drawing blood, running the MRI, and occasionally opening the patient up.
Who’s Allowed In?
So who is allowed to look inside? At the frontier, the answer is employees. Claude, ChatGPT, and Gemini are closed products. From outside you get the black box and nothing else, and every serious claim these companies make about what their models are really doing rests on work done by people they pay.
The trend is running the wrong way. Stanford's Institute for Human-Centered AI, which publishes the annual AI Index, puts it bluntly: the most capable models now disclose the least. Its transparency index fell from 58 to 40 out of 100 in a single year, reversing two years of improvement, as OpenAI, Anthropic, and Google all stopped reporting dataset sizes and parameter counts. Even the nominally open players drift shut as they gain stature. Meta's Llama arrived with a detailed technical report in its first two generations, a thinner one in the third, and none at all by the fourth. The shades come down exactly as the house gets interesting to look into.
Open models help, but not in the way people usually assume. Nobody is going to audit ChatGPT by studying somebody else’s open model. What they provide is a training ground. You cannot produce researchers who know how to open these things up if there is nothing they are permitted to open.
The Machines Have Rules. Nobody Audits Them.
Almost every conversation about AI regulation is about rules of conduct—what a model may and may not do, which uses are banned, and which are high-risk. Nobody checks whether the rules are met, because nobody outside the companies can.
Interpretability is a technique. Auditing is the institution that makes it credible. It requires an independent party that has received social legitimacy in some way, typically through regulation and professional certification. Doing an audit requires access, a standard to measure against, and something to lose if the auditor gets the answer wrong. Interpretability is the most powerful tool such an auditor would have. But a technique practiced only by the people being checked is not assurance. When a lab reports that its model refuses dangerous requests 99 percent of the time, nobody outside the building has ever checked the number. So why should we believe them?
Two barriers stop an outsider. The first is access, which I’ve covered. The second is legal, and it does the real damage. To learn whether a model can be talked into giving dangerous instructions, you have to try to break it. That violates the terms of service of essentially every lab. It may also expose you under the Computer Fraud and Abuse Act, the 1986 anti-hacking law that criminalizes accessing a computer “without authorization,” a phrase Congress never bothered to define. Some methods may additionally run into the Digital Millennium Copyright Act of 1998, which makes it illegal to break a technological lock protecting copyrighted software. In 2024, researchers asked the Copyright Office to carve out a formal exemption for AI safety research. It declined, on the view that they probably don’t need one. So the researcher proceeds on a probably, risking every account they hold, breach of contract, and at the tail, criminal charges. Stack the two barriers and only employees can safely test a frontier model.
Safe harbor is the legal one. It is a binding commitment that good-faith safety research will not get you banned, sued, or prosecuted provided you report what you find responsibly. Cybersecurity settled this 20 years ago with the “bug bounty.” Companies invite outsiders to attack their own systems and pay for the flaws, and nobody gets prosecuted for trying. The Knight First Amendment Institute has published a worked proposal for the AI version. OpenAI and Cohere have partial commitments. Most labs have nothing.
Structured access is the technical one. Here white-box access is provided inside a controlled environment: a locked-down system where the model can be studied but nothing copies out, with a small number of vetted people and every session logged. Some of the machinery exists. METR, a Berkeley nonprofit, tests how far frontier models can get on long, complex tasks on their own. Apollo Research, in London, specializes in scheming whether a model will covertly pursue goals other than the ones it was given. Both have evaluated frontier models before release, but by invitation, under agreement with the lab being evaluated. Better than nothing, but not an independent audit in the true sense of the word. The most serious attempt to fix that came this May from the Secure Access to Frontier AI Taskforce convened by RUSI, the British defense and security institute, which brought together 39 evaluators, lab engineers and security specialists to map what each level of access actually risks and what controls would contain it. The task force was funded by Google.
Structured access only works if it is granted to an accredited independent body rather than someone the lab chooses to invite. Otherwise, there is no reason to believe the audit. The accreditation machinery is not missing, exactly. The EU designates “notified bodies” to assess high-risk AI systems, borrowing the apparatus that certifies medical devices. And in July 2025 ISO and the IEC published a standard setting out what a firm must demonstrate to be competent to audit an organization’s AI management system. But look at what these accredit. The European regime covers deployed systems, not frontier models, and for most high-risk categories the provider is permitted to assess itself. The ISO standard certifies auditors to examine whether a company has documented processes and governance, not whether a model does what its maker says it does. We have built an accreditation apparatus for AI assurance that is not designed to look inside a model.
And Washington is moving the other way. In June the Commerce Department extended export controls to the most capable AI models themselves, not just the chips that train them, authorizing release only to named trusted partners. Whatever else that accomplishes, it makes who may see a model’s internals a national security decision rather than an accountability one.
There is a live test arriving as I write. On June 2 the President signed an executive order directing agencies to design, within 60 days, a process for reviewing the most capable models before release. That deadline is tomorrow. What the order establishes is worth reading closely. Participation is voluntary, and mandatory pre-clearance is explicitly prohibited. Which models qualify is set through a classified benchmarking process aimed at cyber capabilities. The reviewer is the government. Access runs up to 30 days, cut from 90 in an earlier draft the White House scrapped over concerns about slowing innovation. It creates no licensing regime at all. Biden’s 2023 order had required developers to hand over safety test results before release; Trump revoked it in 2025. So the trajectory went from “you must show us” to “we would appreciate it if you would.”
Whatever lands tomorrow, notice what it isn’t. It is a government security review, negotiated privately with the companies being reviewed, against a standard nobody outside can see, producing findings nobody outside will read. That may be exactly right for its purpose. It does nothing whatsoever about whether an independent party can tell the public what these systems actually do.
People Problems
Doing interpretability through a third-party audit depends on having people who know how to do it and who don’t work for the labs. Here we have some serious people problems.
Understanding these systems is public-interest work, the sort of thing universities exist to do. Building them is where the money is, and the money is in the labs. Nur Ahmed, Muntasir Wahed, and Neil Thompson published the canonical study in Science in 2023: industry now produces 96 percent of the largest AI models in any given year and 91 percent of the leading benchmarks. Industry has captured all three inputs (compute, data, and people) and the casualty is public-interest work nobody can monetize.
Start with computing power. Researchers count in GPU hours (one specialized chip running for one hour) and training a serious model takes millions of them. The National AI Research Resource is America’s attempt at a shared pool for academic work, and its flagship machines supply about 2.6 million A100-equivalent GPU hours a year across hundreds of projects. Researchers don’t get money for this. They apply for an allocation of time on a shared machine, and when the hours run out, they run out. Meta’s Llama 3.1 405B, an open model from 2024, thoroughly out of date by now, consumed 30.84 million H100 hours, a figure Meta published itself. One obsolete model burned 20-35 years of everything America’s best academic machines hand out in a year (the range reflects Meta’s use of newer, faster chips). The task force that designed the national resource asked Congress for $2.6 billion over six years. Congress has not appropriated it.
People follow the machines. By The Information’s count, at least 22 professors and researchers left or took leave from Stanford, Berkeley, and Harvard in the first half of 2026 alone, bound for OpenAI, Anthropic, Meta, and Google DeepMind. Dawn Song, a Berkeley professor and one of the most respected names in AI security, is reported to be heading to Meta’s Superintelligence Lab with two colleagues.
The deeper cost is in the students. Michael Gofman and Zhao Jin, in the Journal of Finance, tracked 221 AI professors who left North American universities for industry between 2004 and 2018 and established a causal effect on those left behind. A one standard deviation increase in tenured departures reduces the number of AI entrepreneurs graduating from that university by 13%, concentrated in the top ten schools and in deep learning. They tested several explanations for why, and the one that held up was the simplest. Professors' departures "reduce startup founders' AI knowledge," which turns out to be a significant factor in whether a startup gets founded and funded at all.
Shiva Rajgopal and I have argued elsewhere that the judgment needed to tell a real finding from a merely fluent one can only be built by doing the work, badly at first, under somebody who already has it. That is what a doctoral program is for. It is not a credential factory. It is how expertise reproduces itself. When the professor walks across the street, the students don’t just lose a supervisor. They never acquire the judgment, so they never become people who could audit anything. We aren’t going to run out of models. We are going to run out of people who can tell us what they do.
What the Labs Are Doing
The labs are not ignoring this. Anthropic runs a four-month Fellows program, open to applicants rather than invitation-only, in which engineers and researchers work on the company’s priority AI safety questions. It pays roughly $3,850 a week plus about $15,000 a month in compute credits—more compute than most academic labs will ever see—and over 80% of the first cohort produced research papers. That is the good news.
The qualifying news is that more than 40% of that cohort joined Anthropic full time. This is a recruiting funnel with a research byproduct, and a very good one. The frontier labs have become the de facto National Science Foundation of AI research: they fund it, they supply the compute, they shape the agenda, and the grant is revocable at will by the party being studied. I think it is all well-intended. But it is not producing people with the standing or the job security to do the machine auditing that needs to be done.
There are also real competitive advantages to this structure. Building at the frontier requires compute, data, and outstanding people in the same building, and America’s frictionless path from campus to lab is an advantage other countries have not replicated. The American Enterprise Institute argues that this bottom-up ecosystem is precisely why the United States leads. On that reading, 22 professors moving to the frontier is not a market failure. It is the market working. And the best interpretability research happens inside the labs, because that is where the frontier models are. Dawn Song at Meta can run experiments Dawn Song at Berkeley could not.
But there is a trade-off in the timing that has not gotten much attention. The labs are hiring the professors who would have trained the next generation of researchers, including the ones the labs themselves will want to hire in 10 years. Everyone is drawing down the same account. This decade the arrangement works. Next decade it doesn’t, and it doesn’t work for the labs either.
Fixing the Brain Drain
AI academics move to the labs for more resources: money and compute capacity. I don’t want to discount the former, but I don’t think it’s the most important one. Universities can never close this gap anyway. Berkeley’s current posting for faculty in engineering and artificial intelligence lists $107,100 to $157,800 for an assistant professor and $128,800 to $256,100 for a full professor, on a nine-month academic year, with a note that the university offers more “to meet competitive conditions,” an admission, in a job advertisement, that its own pay scale no longer works. A senior researcher at a frontier lab earns several hundred thousand dollars per year, and principal researchers can make over a million.
I think there’s a way to address the money issue and, even more so, the compute one. It’s pretty straightforward and works like this. A lab and a university sign a multi-year research agreement which is non-exclusive on both sides. A university can take money from several labs; a lab can fund several universities. Individual faculty affiliate with one lab only, behind the conflict walls that law firms and consultancies have run for decades.
Just to suggest a number, and a very comfortable one for the big frontier labs, let's call this $10 million a year per deal, roughly half cash and half compute. The cash funds research contracts on top of existing salaries. It is common for faculty in professional schools to supplement their salaries and companies fund research in universities. But it's the compute that really changes the picture. At market rates, $5 million buys roughly two million GPU hours a year on current chips, which are two to three times faster than the ones in the national pool. Measured in the same units, that is four to six million A100-equivalent hours — against the 2.6 million the pool supplies to the entire country. One agreement, one university, and the researchers there have something like twice what the whole American academy gets.
Consider this for comparison. Eli Lilly, with a market cap around a trillion dollars, has committed up to $250 million over eight years to Purdue, the largest such agreement in American history. Anthropic is valued at $965 billion and OpenAI at $852 billion. Neither has done anything remotely comparable with any university.
Ten million dollars a year is what a frontier lab pays seven to fifteen senior researchers. It is about an eighth of what Llama 3.1 405B cost to train in compute alone. Fund 10 universities at that level and the whole program runs to $100 million a year — roughly a fifth of one percent of Anthropic’s current revenue run rate.
I think the numbers work but intellectual property is a potential deal killer from a faculty and university perspective. Three things need to be settled. First, publication has to be unrestricted. Working papers go out when they are ready, with the funding lab named, as is standard practice everywhere else. Second, the professor chooses the research he or she wants to do. Period. Labs should want creative professors pushing the boundaries of knowledge since ex ante no one knows what will be commercially useful or not. Third, ownership follows the work rather than the money. At the Novartis-MIT center, which ran for 12 years, the two parties jointly held rights to whatever they developed together while each retained rights to what it developed alone. That is a worked precedent, not a hypothetical.
The Next Problem
Fixing the AI brain drain does not fix what this piece started with. Who is auditing the machines?
My position is clear. What the labs do now is an internal audit. That is a real function, and I would rather have it than not, but it is not an external audit, and the difference is not a technicality. We can’t rely on a company telling us it checked its own work, and everything is fine. We need independent organizations with white-box access, under the kind of supervised arrangement I described earlier, so they can actually look inside. Those organizations need certification as institutions, and so do the people who work in them and sign the opinions.
Until that exists, there is one thing you can do with all of this. The next time a lab announces that its own safety team has established something reassuring about its own model, you know the question to ask. Who checked?
How to create the necessary institutional infrastructure for doing an external audit is the subject of my next piece on this topic.