Nuclear weapons
Article
Nuclear weapons is a recurring concept in the Astral Codex Ten archive, appearing 3 times across 3 issues between August 06, 2021 and September 10, 2024. The archive places it in contexts such as “Nuclear weapons do exist, they can blow up entire cities”; “We remember the race for nuclear weapons because they’re a binary technology”; “predict things like nuclear weapons, global warming, or the singularity”. It most often appears alongside AI, Eliezer Yudkowsky, 10,000 AD.
Metadata
- Category: Concepts
- Mention count: 3
- Issue count: 3
- First seen: August 06, 2021
- Last seen: September 10, 2024
Appears In
- Highlights From The Comments On Acemoglu And AI
- Most Technologies Aren’t Races
- Contra DeBoer On Temporal Copernicanism
Related Pages
-
- AI (2 shared issues)
-
- Eliezer Yudkowsky (2 shared issues)
-
- 10,000 AD (1 shared issues)
-
- AGI (1 shared issues)
-
- Agricultural Revolution (1 shared issues)
-
- Agricultural Revolution (1 shared issues)
-
- AI Impacts (1 shared issues)
-
- AI risk (1 shared issues)
-
- Alan Turing (1 shared issues)
-
- algorithmic bias (1 shared issues)
-
- alignment (1 shared issues)
-
- AlphaGo (1 shared issues)
External Links
Source Context
Recovered passages from the original issue text. When the raw archive preserved outbound links inside the source passage, they are listed directly under the quote.
We know in hindsight that this engineer's concerns weren't entirely wrong. Nuclear weapons do exist, they can blow up entire cities, and a nuclear war could plausibly end civilization. But nevertheless, anything the Gunpowder Safety Committee does is bound to be completely and utterly useless. Uranium had not yet been discovered. Lise Meitner and Otto Frisch wouldn't be born for another 500 years. Nobody knew what an isotope was, and their conception of atoms was as different from real atoms as nuclear bombs are from handgonnes. Rockets existed, but one that could deliver tons of payload to a target thousands of miles away was purely in the realm of fantasy. Even though the Roman military engineer detected a real trend--the improvement of weapons--and even though he extrapolated with some accuracy to foretell a real existential threat, he couldn't possibly forecast the timeline or the nature of the threat, and therefore couldn't possibly do anything useful to inform nuclear policy in the 20th century.
In conclusion, AI is like a caveman fighting a three-headed dog in Constantinople. The dog is trying to summon a demon, and the demon is going to unleash a genie. The caveman could fight the demon if he had nuclear weapons, but all he has is an antique musket, and also, just yesterday an eminent physicist told him that nuclear fission was “the merest moonshine”. He could escape the genie if he had a Mars rocket, but nobody can solve the rocket alignment problem, and also Mars might already be overpopulated. If only there had been some kind of fire alarm that could have warned him of this!
Inline links: summon a demon, unleash a genie, “the merest moonshine”, the rocket alignment problem, Mars might already be overpopulated, fire alarm
A late Byzantine shouldn’t have worried that cutesy fireworks were going to immediately lead to nukes. But instead of worrying that the fireworks would keep him up at night or explode in his face, he should have worried about giant cannons and the urgent need to remodel Constantinople’s defenses accordingly. If near-term AI risks are like worrying about fireworks keeping you awake at night and long-term AI risks like worrying about the Ottomans having cannons, then long-term AI worries are right. But if near-term AI risks are like worrying about the Ottomans having cannons and long-term AI risks are like worrying about nukes, then long-term AI worries are wrong.
The most consequential “races” have been for specific military technologies during wars; most famously, the US won the “race” for nuclear weapons. America’s enemies got nukes soon afterwards, but the brief moment of dominance was enough to win World War II. Maybe in some sense the British won a “race” for radar, although it wasn’t a “race” in the sense that the Axis knew about it and was competing to get it first. Maybe in some sense countries “race” to get better fighter jets, tanks, satellites, etc than their rivals. But ordinary mortals don’t concern themselves with such things. No part of US automobile policy is based on “winning the car race” against China, in some sense where consumer car R&D will affect tanks and our military risks being left behind.
We remember the race for nuclear weapons because they’re a binary technology - either you have them, or you don’t. When the US invented stealth bombers, its enemies had slightly worse planes that were slightly less stealthy. But when the US invented nukes, its enemies were stuck with normal bombs; there is no slightly-worse-nuke that can only destroy half a city. Everywhere outside the most extreme transhumanist scenarios, AI is more like the stealth bomber. You may have GPT-3, GPT-4, some future GPT-5, but a two year gap means you have slightly worse AIs, not that you have no AI at all. The only case where there’s a single critical point - where you either have the transformative AI or nothing - is in the hard-takeoff scenario where at a certain threshold AI recursively self-improves to infinity. If someone reaches this threshold before you do, then you’ve lost a race!2
Inline links: 2
Third, those uniformly distributed across techno-economic advances. You’d use this to answer questions like “how likely is it that the most important discovery/invention in history thus far happens during my lifetime?” This seems like the right way to predict things like nuclear weapons, global warming, or the singularity. But it’s harder to measure than the previous two.