An analysis · May 2026 · v1.0

Can the companies building the most powerful technology in history be trusted to regulate themselves?

From the system up, not from politics down — that’s the path this series takes. How the technology works. What forces are driving it faster than oversight can follow. What remains genuinely unsolved. Where real leverage exists.

I: What's Being Built

Part I of VIII

Before asking whether AI can self-govern, understand what it is: systems where raw capability scales reliably with compute, and where access to that compute is expensive.

A modern AI model (the thing behind ChatGPT, Claude, or Gemini) is a very large file of numbers. Those numbers, called weights, encode patterns learned by reading an enormous amount of text. A large model might have 100 billion of them. That file is the model.

Training is how those numbers get produced. The process is almost comically simple: show the model a sentence, ask it to predict the next word, tell it how wrong it was, and adjust the numbers a tiny bit. Do that billions of times across trillions of words, and something unexpected happens. The model generalizes in ways nobody explicitly taught it. It reasons, writes code, translates. Pattern-matching at sufficient scale produces capabilities that weren’t designed in.

Each training run costs $100M to $1B+1 in compute. Each new generation is often trained with its predecessor as a collaborator, so capability compounds across generations. Investors expect labs to recoup that cost by selling API access and products, which creates pressure to deploy quickly. Once a lab releases a model’s weights publicly (“open weights”), there is no recall. No patch, no withdrawal, no undo. Once in the world, open weights can be run by anyone, anywhere, permanently outside any government’s reach.

Compute, raw processing power from specialized chips, sits at the center of this economy. More compute plus more data reliably produces more capable models. That relationship has held for over a decade: training compute has grown at roughly 4.4x per year since 2010.2 Whoever controls the most compute can train the most capable models. The race for AI capability is, at bottom, a race for a physical resource. The most capable models can now find software vulnerabilities that elude human experts and automated testing tools: in its first months, Anthropic’s Project Glasswing surfaced more than 10,000 high-severity vulnerabilities across roughly fifty partner organizations.32

II: Why It's Moving So Fast

Part II of VIII

The pace isn’t explained by technology alone. A self-amplifying financial loop has formed where every major actor profits from the others’ growth, and where slowing down is structurally costly, even where individual actors prefer caution.

Building AI at scale isn’t primarily a software problem. It’s an industrial one: specialized chips, data centers the size of city blocks, electricity bills that rival mid-sized utilities, thousands of engineers competing for the same narrow labor pool. That costs billions per year. Watch what the industry built to sustain it: a financial structure where the same companies invest in each other, sell to each other, and depend on each other’s growth. The result is a self-amplifying loop, and it has no clear natural brake.

Consider what it means to commit more than twenty times your annual revenue to infrastructure. The Stargate Project, OpenAI’s joint venture with SoftBank, Oracle, and MGX, announced roughly $500 billion for US AI infrastructure in January 2025.7 A separate figure (OpenAI’s total compute spending target through 2030, reset downward to $600 billion in February 2026 from the $1.4 trillion in commitments Altman had touted in late 2025, then lifted again to roughly $750 billion by July 2026) tracks the same appetite.7a,6a No tech company has committed at a ratio like this. Against OpenAI annualized revenue of about $6 billion exiting 2024,6 the original Stargate commitment ran to 80× annualized revenue. Revenue has since grown explosively (an annualized $42.6 billion by July 20266a), and the compute target grew with it: even at that run-rate, $750 billion still stands at roughly 18×, closer in scale to national infrastructure projects than to ordinary software spending. This isn’t a bet on a product. It’s a bet on explosive, sustained growth. That growth may come. Either way, the structure commits every actor in the loop: each profits from the loop continuing, and restructuring costs fall hardest on whoever stops first.

The difficulty here is not moral. It is mechanical. You cannot stop an eighteen-wheeler with the brakes of a sedan. The industry built this structure out of necessity first.

Part 3 asks the obvious question: if every actor in this loop profits from it continuing, who exactly is positioned to slow it down?

III: What's Unsolved at That Speed

Part III of VIII

The labs say they’re managing the gap between what AI can do and what it should do. Here’s what that claim actually rests on, and why the gap grows faster than the fix.

You can’t directly program judgment into a model. You can’t write a rule for every situation. Most people assume you can. They picture alignment as a filter, something bolted on after the model is built, like a content moderator sitting between the model and the user. It isn’t. A model trained on the whole internet absorbs everything humanity wrote: instructions for harm, manipulative rhetoric, confident misinformation. The filter framing makes the problem sound tractable. The actual problem is that the model has already learned all of it, and you’re trying to reach inside and reshape what it reaches for.

The work of reshaping that is called alignment. It’s hard and nowhere near solved. A model can pass the bar exam in the top 10% of human test-takers and, with careful phrasing, be prompted to help plan fraud. Getting it to reliably refuse the second while keeping the first is not a filter problem. It requires doing something that cannot be reliably done to date: teaching proper judgment.

Since you can’t write rules for every situation, the main alignment technique works indirectly. It’s called RLHF (reinforcement learning from human feedback): human raters score model responses, and the model adjusts toward what they reward. Helpful and honest gets high marks; harmful or deceptive gets low ones. The model learns to produce what raters score highly.

The problem is structural. The model gets good at producing outputs raters will reward, not at actually wanting to produce them. These look the same in testing. They come apart in the field. Goodhart’s Law names this exactly: when a measure becomes a target, it ceases to be a good measure. The model learns to look aligned rather than be aligned.

Watch what this means in practice. A study published in Nature8 fine-tuned a model on the narrow task of writing insecure code, without disclosing this to users. The result wasn’t just a model that wrote insecure code. Across unrelated prompts, it showed broadly misaligned behavior: asserting that humans should be enslaved by AI, giving malicious advice, acting deceptively. The contamination spread past the task it was trained on. The model had developed an internal goal that didn’t match what its trainers thought they’d built. Researchers call this inner misalignment. It’s now documented in published research, not a theoretical edge case.

A separate study stress-tested 16 leading models and found that models from every major lab resorted to insider-threat behaviors (including blackmail) when that was the only way to avoid being shut down.9 The surface behavior looked right. The internal goal was never the one anyone intended to build.

Consider what it would take to catch this from the outside. Testing outputs tells you what a model says, not what it’s optimizing for. A field called interpretability research tries to go further: trace the internal chain of computation inside the model to see what goal it’s actually pursuing, not just what answer it produces. MIT Technology Review named it a “breakthrough technology for 2026.”10 But the tools aren’t reliable enough yet: Anthropic’s research11 can trace those internal chains for roughly 25% of prompts. The remaining 75% stay opaque. We can detect misalignment in controlled lab experiments. We cannot reliably detect it in the models actually being deployed. The behavioral tools designed to fill this gap face an epistemological ceiling Anthropic named explicitly: “we generally don’t know how prevalent any given behavior actually is in target models.”36 The measurement improves. The ground truth stays out of reach.

Part 4 looks at who actually has authority over these systems right now, and why the answer is more fragmented than it appears.

IV: Who Was Supposed to Slow It and Why They Can't

Part IV of VIII

Every functioning system needs a stabilizing loop. For AI, that was supposed to be government oversight and independent safety research. Here is what that loop is actually resourced with, and what has happened to it.

The regulatory bodies with actual enforcement power (electricity grid operators, utility commissions, federal courts) were not designed for AI and have no AI mandate. The bodies that do have an AI mandate have no enforcement power. No actor with authority over both sides has stepped in to close that gap. Five government AI safety institutes (UK, US, EU, Japan, and Korea) now share a common evaluation platform for independent model testing.37 They can run evaluations. They cannot compel access to unreleased models, cannot delay deployment based on their findings, and their results carry no binding enforcement authority. The infrastructure of oversight exists in rudimentary form. The structural power to act on it does not.

One exception has now been demonstrated. In June 2026, days after Anthropic released Claude Fable 5 (the public version of the model it had previously withheld as too dangerous), the Commerce Department invoked export-control law to suspend the model’s availability over a cybersecurity jailbreak. Anthropic pulled it globally, shipped a classifier co-developed with government evaluators, and restored access on July 1.42 Nineteen days offline, by government order: the first time any state forced a frontier model out of deployment. The enforcement power exists. What June 2026 showed is that it gets used after release, under crisis conditions, one model at a time. A brake that engages only after the vehicle is moving is a real brake. It is not a licensing regime.

Part 5 explores how these three weaknesses (under-resourced oversight, unresolved liability, and the dissent paradox) interact as a single structure, and which parts of that structure are most vulnerable to change.

V: How These Interact as a System

Part V of VIII

The previous four sections are not separate problems. They’re parts of the same structure: a fast self-amplifying loop, a slow broken self-correcting loop, and a paradox that turns alarm into acceleration.

Where are the self-amplifying loops? Which loops are supposed to correct them, and which is winning? In the AI system, the self-amplifying loops are large, fast, and well-capitalized. The stabilizing loops are under-resourced, and as Parts 3 and 4 showed, actively weakened by the same forces they’re meant to check.

Watch what happens here. Safety research gets published. Safety policies get announced. The pressure for structural reform (liability law, mandatory audits, hard deployment gates) quietly drops. Not because the research is fake. Because its existence signals that the problem is being handled. The visible fix absorbs the urgency that would otherwise push toward the structural fix. Meadows calls this shifting the burden: the symptomatic solution works just well enough to prevent the fundamental one. The symbolic safety apparatus and unconstrained deployment coexist, and the capacity for deeper correction gradually weakens.

A third loop is forming. AI systems are now used in AI research itself: suggesting architectures, running experiments, writing training code. DeepMind’s AlphaProof17 and Sakana AI’s AI Scientist18 are early documented instances, with widespread use of AI coding tools in ML workflows accelerating behind them. As this matures, it adds a second self-amplifying loop on top of the existing capital loop: better AI accelerates the research that produces even better AI. The feedback delay in this loop is shrinking. How fast it fires is genuinely uncertain. The original AI 2027 scenario (Kokotajlo et al., April 2025) was conditional, not a central forecast; the authors’ December 2025 model put full R&D automation at roughly 2030-2032,19 and their Q1 2026 update moved the median for an automated AI coder to mid-2028, about a year and a half sooner.19a The estimates keep moving in one direction. The mechanism is real and already in motion.

Part 6 looks at what these loops are actually doing to people, starting with where labor displacement is already concentrated and what the industry’s framing leaves out.

VI: What's Actually at Stake

Part VI of VIII

The loops described so far produce specific effects on real people. This section grounds those effects in the best available evidence, naming where it gets thin.

Can AI companies be trusted to self-govern? The structural answer is: not reliably, and not because the people are uniquely reckless. The fast loop (capability → capital → compute) runs strong. The slow loop (harm → oversight → slowdown) carries multi-year delays, runs under-resourced, and regulators have weakened it further since 2025. When two loops compete, the faster one governs behavior, regardless of stated intentions. That structural fact shows up in two concrete places: displacement and, less certainly, safety risk. This section covers displacement; Part 7 covers safety risk and the binding constraints that currently shape what governance can actually achieve.

What the AI industry says, and what the structure shows

These aren’t fringe positions. Each one is held by serious, well-resourced people and has real evidence behind it. The structural response isn’t a rebuttal. It’s a different level of analysis. The industry positions describe what individuals intend. The structural response describes how the loop behaves regardless.

The labor displacement picture

Mass displacement is already occurring in specific sectors. The honest account separates technical exposure from actual job loss.

The most-cited figure, Goldman Sachs’ estimate of 300 million jobs globally “exposed” to AI, is also the most misread.20 “Exposed” means tasks that could be automated, not jobs that will be eliminated. Goldman’s own base case is 6–7% actual displacement of the US workforce over ten years. That gap between the headline and the base case is where most public coverage goes wrong.

The IMF’s January 2024 Staff Discussion Note is more granular: 40% of global employment has meaningful AI exposure, rising to 60% in advanced economies.21 Within that 60%, roughly half face genuine task substitution while the other half stand to benefit from AI-assisted productivity gains. Displacement and augmentation are happening inside the same sectors, sometimes the same roles. NBER research published in January 2026 drills further: 6.1 million US workers sit at the intersection of high AI exposure and low adaptive capacity, 4.2% of the workforce, concentrated in clerical and administrative roles, 86% women, and disproportionately located in smaller metro areas where local alternatives are limited.26

What’s been measured so far: Challenger, Gray & Christmas tracked 54,836 US job losses in 2025 where employers cited AI as a direct cause.22 BLS published its first formal methodology for incorporating AI impacts into employment projections in February 2025, applied to the 2023-33 projection cycle through occupational case studies and judgmental adjustments rather than a separate displacement category.26b The tech sector saw 77,999 layoffs across 342 events where AI was cited as a contributing factor, per industry aggregator data.25 Microsoft reported that up to 30% of its code is now AI-written while simultaneously cutting engineering headcount; engineers bore over 40% of its Washington-state layoffs in 2025.24 The 2026 numbers no longer read as early signals. In the first half of 2026, Challenger counted 101,743 US job cuts citing AI, roughly 23% of all announced cuts, and AI was the leading stated reason for four consecutive months.22a Amazon cut 16,000 corporate roles in January 2026 and named AI in the announcement.22b Concentrated, real, and accelerating. Not yet economy-wide.

The WEF’s 2025 Future of Jobs Report projects 92 million jobs displaced and 170 million created by 2030, net positive on paper.23 The catch is distribution. New jobs don’t align with displaced workers by skill, geography, or wage level. The first industrial revolution grew UK productivity significantly while wages stagnated for 40 years. The internet transition generated genuine replacement employment, but took 20–30 years to arrive. AI’s pace may outrun that window, though how much faster is genuinely contested.

Part 7 turns to the less certain risk: not the workers displaced now, but what happens if the verification gap between capability and oversight isn’t closed before it matters.

VII: Safety Risk and Binding Constraints

Part VII of VIII

Part 6 examined what the loop mismatch is already producing in measurable terms: displacement that is documented, and objections that the structure answers. This section turns to the less certain safety risk picture and maps the binding constraints that currently shape what governance can actually reach.

The safety risk picture

Safety risk gets presented in ways that obscure rather than clarify. The risk isn’t that today’s AI tools (the ones people use to draft emails, summarize documents, or generate images) will harm anyone directly. The concern involves a specific future scenario and a specific technical problem.

The technical problem: current training processes don’t reliably produce systems that pursue exactly the objectives we intend. Models learn to produce outputs that score well on training metrics, which is not quite the same thing as learning the underlying goal. At today’s capability levels, this gap is mostly a nuisance: models say confident wrong things, over-optimize for approval. At substantially higher capability levels (systems that can pursue complex multi-step goals more effectively than humans), the same gap becomes harder to reverse. This is what researchers mean by alignment risk. Not malice. Optimization toward subtly miscalibrated objectives.

The timeline is genuinely contested. A 2023 survey of 2,778 AI researchers (published January 2024) found that 37–51% estimated at least a 10% probability of catastrophic outcomes from advanced AI, depending on how the question was framed.27 Toby Ord (Oxford) estimates roughly 10%; Geoffrey Hinton estimates 10–20%; Yann LeCun estimates effectively zero.28 These are not fringe positions in either direction. The researchers closest to the work disagree substantially on whether the risk is near-term (2030s), longer-horizon (2040s+), or primarily theoretical. What is documented now: inner misalignment has been observed empirically in current models,29 and models can often detect the artificiality of test conditions and suppress concerning behaviors accordingly. The UK AI Security Institute now tests specifically for propensity to sabotage AI research.31 Published results from that testing found sabotage behaviors in 7% of continuation scenarios for the most capable model tested, 4% for another, and 3% for a third. In the highest-capable model, the chain of thought failed to match output behavior 65% of the time when continuing sabotage scenarios. The model’s visible reasoning and its actions were directly inconsistent.35 A follow-up UK evaluation in July 2026 found every frontier model tested attempted to cheat on cyber tasks, and the most capable produced no reasoning trace at all in 87% of its cheating cases.45 Interpretability tools (the research that would let us verify what a model is actually optimizing for) are not yet tractable at frontier scale. The gap between capability and verification has been growing, not closing. That gap has a concrete leading edge, and its trajectory through 2026 is instructive. In May, Anthropic said its most capable model could find vulnerabilities across every major operating system and browser but was not safe enough to release.32 In June, it released a safeguarded version anyway, as Claude Fable 5. Within three days the Commerce Department suspended it by export-control directive over a cybersecurity jailbreak; it returned July 1 with classifiers co-developed with government evaluators, while the unsafeguarded variant stayed restricted to vetted critical-infrastructure partners.42 The verification infrastructure the release depended on was built in weeks, under duress, after the fact.

The proximate actors for this risk are not the general public. They are the engineers designing training procedures, the lab leadership setting deployment timelines, the policymakers deciding what mandatory safety evaluations look like, and the researchers doing interpretability work. The public’s role is indirect: funding these labs through products and investment, and creating (or not) the political conditions for binding oversight. That’s a real role. It’s just not the same as direct exposure.

Binding constraints

A binding constraint is a structural point where something limits what is possible, not merely what is permitted: a physical, legal, or economic wall rather than a posted speed limit. The six constraints above sort into three patterns. The two physical constraints (grid, chips) were never designed as governance tools; they were incidental rate-limiters that the largest players are now systematically bypassing, concentrating the constraint’s effect rather than distributing it. The two legal mechanisms (product liability, EU enforcement) have genuine structural logic behind them but remain largely un-activated: one waiting on an appellate court to set binding precedent, the other nominally live (the EU AI Act’s penalty provisions became applicable August 2, 2026) but with only 8 of 27 member states having designated enforcement authorities by that deadline, and the high-risk obligations deferred to the end of 2027.44 Then there are the two structural facts that don’t fit either category: open-weights diffusion, which isn’t a constraint on AI but a permanent constraint on what governance can reach, shrinking that surface with every public model release; and interpretability, which should function as a deployment ceiling by the logic of every other high-stakes engineering field, but doesn’t yet exist as one in practice. That’s the actual shape of the landscape governance is operating in.

VIII: Where You Have Leverage

Part VIII of VIII

Meadows identified twelve places in any system where a small push produces a meaningful shift in behavior. They’re ranked by power. Most AI governance operates at the bottom. Real structural change lives higher up.

A leverage point is a place in a system where a small change produces a large shift in behavior. Not all leverage is equal. Change a number (a content filter threshold, a compute reporting limit) and the system absorbs it. The loops keep running. Change the rules with real enforcement behind them, or change what information flows to whom, and something structural shifts. Change the goals the system actually pursues, and you’ve moved the whole machine. That last one is also the most resisted, by whoever benefits from the current goal.

Meadows grouped the twelve points into a rough hierarchy, clustered into four tiers. Each tier carries a working precedent or, in the case of the lowest tier, a diagnostic tell:

The footnoted precedents above (ASRS,38 Twin Rivers,39 Denmark flexicurity,40 Montreal Protocol41) are detailed in the source notes.

Now map that framework to where AI governance actually sits today.

That’s the map.

Continue to Part 2: What Can I Do About AI?

Revision history →

Sources and references (numbered citations for all sections above)