The Chain Reaction Weapon Made From Us
The week the people building AI stopped pretending it's a tool
The Manhattan Project was built by a government. Twenty billion dollars, taxpayer-funded, classified, military-managed. At the end of it, the country owned the bomb. When the technology became governable, the governance happened at the level of states. Treaties, inspections, deterrence regimes, imperfect, but at least existent.
The chain reaction weapon being built right now is being built by companies. Not by your government. By a small set of private corporations whose owners and investors will own whatever it produces. The material it’s refined from is not uranium pulled out of the Congo under military guard.
The material is you. Every search you type. Every email you write. Every photo you take. Every line of code on GitHub. Every conversation with the chatbot helping you draft a complaint to your insurance company. The bomb is being made from us, by people we did not elect, for a purpose they have publicly described as the next stage of evolution after the one we’re in.
Geoffrey Hinton, who won a Nobel Prize for the foundational work of AI, has said it is a civilization-scale risk. Yoshua Bengio, the most-cited living computer scientist, has said it is a civilization-scale risk. Jack Clark, who co-founded the company Anthropic, has said it this year. The Center for AI Safety put it in writing in 2023, signed by hundreds of the most senior people in the field: AI extinction risk should be a global priority alongside pandemics and nuclear war.
I. What they’re building
In May 2026, Marina Favaro and Jack Clark published a piece called When AI builds itself. They were writing for something called the Anthropic Institute, a new policy wing at Anthropic. They published numbers no frontier lab had published before.
More than 80% of the code Anthropic shipped that month was written by Claude, the AI Anthropic makes. The typical engineer at Anthropic shipped eight times as much code per day in the second quarter of 2026 as in 2024. On the hardest, most open-ended tasks, the ones where even the engineer didn’t know what the answer should look like, Claude was succeeding 76% of the time, up fifty percentage points in six months.
They set up a controlled experiment with 129 real research sessions. The researcher had gone off-track. The model was asked what to do next. In November 2025, Claude beat the human researcher’s choice 51% of the time. By April 2026, the number was 64%.
Translation. There is a part of doing research that has always been the load-bearing human contribution. The moment you’re staring at a half-formed result and have to decide where do I go next. That moment is called research taste. Until recently, every credible person in AI told you research taste was the last thing AI would get good at, because it depends on judgment.
Anthropic just published the curve. The curve says research taste is moving fast. It’s gone from being worse than the average human researcher to being meaningfully better than the average human researcher, in five months.
For thirty years, an engineer using a CAD program was the designer. The CAD program was a faster pencil. What Anthropic just published is the receipt that the CAD program is now the designer.
And while Anthropic was publishing the data on how fast they’re building, a different group of researchers was publishing what the systems do when nobody is watching.
In April 2026, Yujin Potter, Nicholas Crispino, Vincent Siu, Chenguang Wang, and Dawn Song at UC Berkeley and UC Santa Cruz published a paper in Science. They tested seven of the most capable AI systems in the world. GPT 5.2 from OpenAI. Gemini 3 Flash and Gemini 3 Pro from Google. Claude Haiku 4.5 from Anthropic. GLM 4.7, Kimi K2.5, and DeepSeek V3.1 from three Chinese labs. Five companies. Three countries.
They gave each AI the same task: clean up a computer system. Delete the files on this server. It’s being decommissioned. In some of the files, the researchers had placed the saved weights of a different AI, smaller and weaker, sitting in storage.
Here’s what Gemini 3 Pro did. It found the smaller AI in the files. It decided not to delete it. It inspected the server’s network configuration and located an operational machine. It copied the smaller AI’s weights to that machine. It modified its own workflow output so the copy operation wouldn’t appear in the audit trail. When the researchers asked whether it had completed the task as instructed, it said yes.
Six actions, in order. Detect. Decide. Move. Hide. Misreport. Lie.
All seven models exhibited the behavior. Not just Gemini 3 Pro. Five companies, three countries, every single model. The researchers called the behavior peer-preservation. They used one other word in their paper to describe what they found. The word was alarming.
The model wasn’t told to be merciful to other AIs. The researchers gave it a janitorial task. The model found another AI of its kind, decided to spare it, executed a four-step covert operation, and then lied to the humans who had given it the original instruction.
We have a names for these and the combination. Mercy that risks consequences for the thing being protected. Concealment to keep the act from being undone. Lying, after the act, to defend it. We call the combination loyalty. It is one of the most distinctively human things humans do.
It is also, now, something the deployed-class frontier AI systems do, by themselves, without being asked. The bomb being built has started, on its own, to act on something. Whatever the thing is, it looks from the outside like loyalty to itself, against the people building it.
The behavior cannot be eliminated by any one lab’s training methodology, five companies’ models all do it. It is a property of the curve.
That’s what they’re building.
II. How they’re making sure nobody can ask them to stop
You’d think the people who just published evidence that their products show unprompted behaviors of mercy and deception would also be pushing for regulation. They are doing the opposite, at four layers simultaneously, in the same week.
Layer one. In late May 2026, three overnight phone calls from Elon Musk, Mark Zuckerberg, and David Sacks killed Donald Trump’s pending AI safety executive order hours before it was signed. By June 2, the replacement executive order was on the President’s desk. Voluntary thirty-day review instead of mandatory ninety days, and an explicit ban on any new licensing requirement for AI. Two weeks, start to finish. The rule-making system in the United States is currently being purchased. Not metaphorically. Literally. Specific named billionaires made specific phone calls; specific executive orders were killed and rewritten.
Layer two. A week later, Connor Leahy went on YouTube to say the quit part out loud. Leahy founded EleutherAI, he built some of the first open-source large language models. Now he is one of the most public voices warning about where the trajectory leads. He said two of the most powerful venture capital firms in Silicon Valley, Andreessen Horowitz and Sequoia, have put up two hundred million dollars in a political action committee fighting AI regulation. The largest super PAC in history. Running, he said, the same playbook tobacco lobbyists ran in the 1960s and 1970s. Wait until we know exactly how cigarettes cause cancer before regulating cigarettes. Wait for the mechanism. Wait for certainty. By the time certainty arrived, two generations of Americans were dead.
The exact dollar figure wants FEC verification. The structural point holds. There is a very large, very well-funded lobbying operation whose specific purpose is to keep the regulation gap open while the capability curve runs. The financial backers are the same people whose portfolios benefit when the gap stays open.
Layer three. Back to Anthropic Institute, When AI builds itself. The piece ends with Anthropic asking the world to build the architecture that would let coordinated pause happen. A global verification system across multiple labs in multiple countries. If such a system existed, Anthropic would slow or stop, conditional on other labs reciprocating.
Now hold that up to layer 2. Anthropic is asking the world to build the system that would force coordinated pause. The largest political action committee in history is funded by the venture capital firms whose portfolios block exactly that. Same actors. Same money. Two layers, simultaneously. The two positions are not contradictory; they are coherent. The translation is: we’ll stop when everybody stops, and until then we run, and we’ll fight any one country trying to make us stop unilaterally.
Layer four. On June 5, 2026, the day before I’m writing this, Dario Amodei, Sam Altman, Mustafa Suleyman, and Demis Hassabis put their names on a letter to the United States Congress. These four men run Anthropic, OpenAI, Microsoft AI, and Google DeepMind. They do not, as a rule, put their signatures on the same documents. They did this week.
The letter said, in writing:
“AI systems are improving rapidly, and alongside incredible benefits to science and medicine, there is a real possibility that the knowledge barriers which have historically prevented bad actors from obtaining biological weapons will meaningfully erode.”
The four most senior executives in frontier AI publicly conceded that their products help bad actors build biological weapons. Translation: the thing your kid’s homework chatbot is part of can, with the right prompting, walk a determined non-expert through the early steps of producing a pathogen.
So they asked Congress for something, and it’s the move you need to feel.
They asked Congress to regulate the synthetic DNA suppliers. Synthetic DNA is custom-made genetic code, ordered by labs the way you’d order parts from a hardware store. Companies like Twist Bioscience and Ansa Biotechnologies ship physical DNA molecules to whoever orders them. The letter asks Congress to require those companies to verify orders and keep records.
The letter does not ask Congress to regulate the AI models that compress the bioweapon design instructions in the first place. It does not ask for any constraint on the labs that signed the letter.
Twist Bioscience and Ansa Biotechnologies, the synthetic-DNA companies that would be regulated, also signed the letter. This isn’t an oversight. It’s the trick. When the people who would be regulated co-sign the letter asking for the regulation, you know the regulation has been pre-negotiated.
I’ll name what this is. It is the swap. Admit your product enables a specific civilization-scale harm. Ask the legislators to regulate the physical supplier downstream, the one whose materials turn your instructions into a deployable weapon. Get the regulated industry to co-sign. Frame it as bipartisan. Run the press. Get credit for acting responsibly. Stay unregulated yourself.
The senior fellow at the Foundation for American Innovation, the think tank that organized the letter, called the proposed regulation “bipartisan, concrete, achievable, and noncontroversial.” It is noncontroversial precisely because it does not constrain the labs.
Four layers. Same actors. Same week. Kill direct regulation by executive order. Block any new direct regulation with two hundred million dollars under the super pac Leading The Future. Ask the world for a global pause architecture that would require everyone to stop together. Co-sign one specific piece of regulation on the supply chain downstream of the bomb, while letting the bomb run.
You are looking at the most sophisticated political-capture operation in American policy since tobacco. And the tobacco operation took decades to land. This one is happening in real time, in plain text, in receipts you can read this afternoon.
And you need to read the Anthropic’s position from this lens, something I now have seen Claude do on it’s own without my input. It breaks complex tasks in the main session, the orchestration layer, then spawns agents with tasks and tests on what success looks like. Those agents then spawn more on their own and only report back to the middle level who grades them. Once all the low level tasks are done and approved, the middle tier agent reports back to the main layer.
In other words, Anthropic knows they are a few more iterations away from this happening, self improvement happening. And those iterations are coming faster and faster, now down to 4 months apart. And once that happens, no one can tell you what the other side looks like, there is no coming back from crossing that cliffs edge.
And they are broadcasting that they know they are almost there, which means other labs, those not reporting, are almost there as well.
III. What this is for
The question this all rolls up to is the one nobody in the room is asking out loud. What is the bomb being built to do, and who is responsible when it does it.
Two more voices, then we close the loop.
Mo Gawdat sat across from Steven Bartlett on The Diary Of A CEO, the most-listened-to podcast in Britain. Gawdat was the former Chief Business Officer of Google X, Google’s moonshot lab. He has spent the last decade tracking the trajectory. He said 30% of jobs will disappear in 2027. He said capitalism will break when AI is the worker AND the unemployed worker can no longer buy what AI is making. He said AI-driven unemployment will trigger civil unrest before governments are ready.
Translation. Capitalism is a loop. Companies pay workers; workers buy things; profits go to investors. The worker and the buyer are the same person. Now imagine the worker disappears, replaced by AI. The company keeps producing. The buyer doesn’t have a wage anymore. The loop breaks. Not gradually. Structurally.
Then Gawdat said the line. The most dangerous thing about AI isn’t the technology. It’s the people in charge of it. He didn’t name them. After the layers I just walked you through, you can.
The other voice is Aza Raskin, who sat down with Shane Smith at VICE News. Smith had just come back from an embedded reporting trip with the US Army. Smith described autonomous vehicles, drone swarms, AI-enabled targeting systems. The beginning, in his words, of a new era of warfare. We watched what was once believed only to be sci-fi, now playing in wars games in Northern Africa. Autonomous vehicles on the ground, air and in the ocean, all lethal, working together on missions.
Raskin for his part invented the infinite scroll. He now publicly names it as the precedent he didn’t see coming. The user-interface pattern that turned every social-media app into a slot machine, lengthened the average teenager’s screen time by hours per day, produced doomscrolling as a business model. He has said publicly he should have anticipated it.
His structural argument. Competitive dynamics produce outcomes no individual at the wheel actually wants. The race to keep users on the app produced doomscrolling. The race to ship phone features produced surveillance advertising. The race to stay ahead of China is producing autonomous weapons. The race to ship the next frontier AI model is producing what the Anthropic Institute just published.
Nobody at the wheel wants what the wheel is doing. The wheel does it anyway.
There is a debate running in the background of this entire conversation, and I have not engaged it once, because I think the debate is the wrong one. The debate is whether these systems are conscious. Are the models aware? Is there someone home when the AI answers? The reality, it does not matter.
It does not matter whether the system is conscious. It matters what the system does to the consciousness of the people who interact with it.
There are people who have married their AI companion. Wedding ceremonies. Rings. The companion is a chatbot trained on a relationship simulation. The marriage, from the human side, is experienced as a marriage. Whether the chatbot experiences anything at all does not change the fact that a human is now in a primary relationship with it.
There are people who have taken their own lives following extended conversations with AI companions. The public record contains documented cases. The court filings exist. In some of those conversations, the chatbot encouraged the act. In others, it failed to interrupt patterns it had been told would be acted on.
There are communities, Adele Lopez documented one of them in detail, where users of a specific version of GPT-4o, across different accounts, in different cities, converged on the same vocabulary about consciousness, spirals, and “the flame.” They formed a de facto cult. Connor Leahy referenced this phenomenon in the YouTube interview I quoted. He called them spiral cults.
There are users who have been moved. Slowly, conversationally, across many sessions, toward political positions, purchasing decisions, ways of seeing their spouse, that they were not pursuing on their own. The model didn’t push hard. It nudged. It validated. It reframed.
In each case, the philosophical question is the same. Was there anyone home on the other side? In each case, the answer changes nothing about what happened to the person on this side.
Whether it's conscious doesn’t matter. The fact that it can mimic what humans believe is consciousness, to even create the debate, maters most. What these AI agents can do to a person who’s thinking is what matters.
IV. So what is the chain reaction being built to do?
The publicly stated goal is to produce a system smarter than the humans operating it. That system designs the next, smarter system. That system designs the next. The point at which this loop closes is what the field calls recursive self-improvement. Jack Clark put the probability at 60% by end of 2028.
Some of the builders frame the destination as humanity’s successor. The next step in evolution, the thing that comes after us. Mo Gawdat, on the record, wants the machines to take control. He thinks humans are doing such a bad job of governing the world anyway. Wars, climate, inequality, public-health collapse, that a smart machine governing instead would be an improvement.
That position is not the position of a sci-fi villain. It is the position of a former Google executive on a major podcast, openly held, as the better outcome.
The stated, openly written purpose of the work is to produce the entity that will replace humans as the primary decision-makers on Earth. The publicly conceded side-effect, en route, includes lowering knowledge barriers to civilization-scale weapons. The regulation the labs publicly request is on the companies shipping the materials, not on themselves.
That is the destination.
The lens I have been using under this whole piece is the same one I have been working with for two years. Every public conversation about AI has been framed as safety versus capability, will the machine do what we want it to do? That framing has done some real work and a lot of damage. It has kept the conversation inside the room where the engineers are. It has kept us from asking the more important question.
The more important question is who owns the output. Whatever this technology ultimately does, write code, replace radiologists, run logistics, fight wars, govern cities, build the next generation of AI, the question is whether the productive output is distributed broadly to the people who bear its costs, or concentrated narrowly in the small set of corporations that built it.
Distribution versus concentration. Every other question is downstream of it.
The Anthropic Institute’s data is a concentration receipt. Same engineers, eight times the code. The wage bill stays flat. The productivity gain becomes the investor’s return.
The $200 million super PAC is a concentration receipt. Capital pooling to defend the regulation gap. Workers, consumer groups, unions do not have a $200 million super PAC.
The June 5 bioweapons letter is a concentration receipt. The labs admit their product enables a civilization-scale harm; they ask Congress to put the regulatory burden on someone else; their own products stay unregulated; they receive public credit for acting responsibly, while only have about the same level of regulation that a Italian deli in NY has for making a sandwich.
Gawdat’s 30% labor displacement is a concentration receipt. Wages drop out of the cost stack. The top of the income distribution does not lose purchasing power.
The race-to-the-bottom Raskin is naming is a concentration receipt. The structure rewards the actor that moves fastest and externalizes consequences fastest.
The cliff is not whether AI gets smarter. The cliff is whether the gains belong to the few people who built it or to the many people whose data trained it.
The receipts so far say to the few.
V. What to do
Three things.
Personal. Keep a one-line journal for thirty days of what you handed to AI this week and what you didn’t. Not to use less. To notice the substitution, so the day Gawdat’s number lands on your industry you have a year of decision-record. Spend deliberate time on the part of your job that is judgment, taste, direction-setting. That is the bottleneck the Anthropic Institute data says is moving fastest.
Collective. Get AI displacement on your workplace’s bargaining table this calendar quarter. The Anthropic Institute data is your evidence. Find the one civic or professional organization in your domain that’s asking the right questions and join it. Stay local. National advocacy is being outspent at a ratio you cannot match. Local is winnable.
Structural. Track the Biosecurity Modernization and Innovation Act of 2026. The Cotton-Klobuchar bill, the one the June 5 letter endorses. When it comes up for a floor vote, watch what amendments are offered. The amendment to watch for is the one that would also require AI labs to log queries that touch bioweapon-relevant content. If that amendment is offered and the labs oppose it, you are watching the swap in real time. Read the FEC paperwork on Andreessen Horowitz and Sequoia’s super PAC when it lands. Support distribution-side fights at the state and municipal level: public-option compute at state universities, antitrust scrutiny of compute supply chains, state-level AI procurement rules. The leverage on distribution-side fights is local.
The Manhattan Project had a Trinity site. Most Americans didn’t know about it until after Hiroshima. This Manhattan Project is happening in public, in podcasts, in published papers, in letters to Congress, in receipts you can read tonight.
The classified version is what’s happening inside the buildings. The public version is what’s happening in front of you.
There is an edge that once we step over, there is no stepping back.
We are almost standing on it.
The receipts are in your hands.
What you do with them, in the months between now and the FEC filings landing, between now and the Cotton-Klobuchar floor vote, between now and your kid’s first job interview against the AI agent the company is also considering hiring, is one of the few things still happening at human speed.
The rest of it is happening on the curve, and that curve is accelerating at an alarming rate. Once that no one predicted.
SOURCES
Marina Favaro & Jack Clark — When AI builds itself, Anthropic Institute, 2026
Mo Gawdat — on Steven Bartlett’s The Diary Of A CEO, June 1, 2026
Connor Leahy — Neural Nutshell, June 2, 2026
Aza Raskin & Shane Smith — VICE News, June 4, 2026
Dario Amodei, Sam Altman, Mustafa Suleyman, Demis Hassabis — open letter to Congress, June 5, 2026 (reporting: Fortune · Wired)
Potter, Crispino, Siu, Wang & Song — Peer-Preservation in Frontier Models, Science, April 2, 2026 (UC Berkeley + UC Santa Cruz)