Astral Codex Ten

North Korea

33 min read

North Korea

Article

North Korea is a recurring place in the Astral Codex Ten archive, appearing 6 times across 6 issues between August 08, 2021 and September 11, 2025. The archive places it in contexts such as “There is also an anti-North-Korea blogosphere”; “North Korea is refusing all vaccines”; “The US is becoming the North Korea of forecasting”. It most often appears alongside Donald Trump, Brazil, Canada.

Metadata

  • Category: Places
  • Mention count: 6
  • Issue count: 6
  • First seen: August 08, 2021
  • Last seen: September 11, 2025

Appears In

Source Context

Recovered passages from the original issue text. When the raw archive preserved outbound links inside the source passage, they are listed directly under the quote.

Contra Drum On The Fish Oil Story
August 08, 2021 · Original source
There is also an anti-police blogosphere, an anti-Trump blogosphere, and an anti-North-Korea blogosphere. Sometimes people form a blogosphere against a thing because it is bad. One of the people in the anti-FDA blogosphere is that guy who lost his intestine because the FDA delayed his drug for five months arguing about a warning label. People get angry when the FDA almost kills them. Then they write pieces about their anger. I’ve never had the FDA almost kill me personally, but I have had it hobble my efforts to help some really desperate patients who I’ve grown kind of attached to over the years, and yeah, that also makes me angry. If your response is “this comes from an angry person so I can ignore it”, I think you will miss a lot of important stories, not just FDA-related ones.
Predictions For 2022
February 01, 2022 · Original source
While I agree things don’t look good for the Democrats, 95% chance they lose both houses of Congress implies 97.5% chance of losing each house, which seems too high. I’m smashing the BUY button as hard as I can on “at least one country will fail to get to 10% vaccination rate” - there are a lot of countries, and as far as I know North Korea is refusing all vaccines out of general evilness. Although I’m not supposed to check betting markets, Dylan writes that he checked the betting markets for the Academy Awards, saw a 30% chance that Belfast would win, but he thinks the number is more like 55%. I know nothing about movies, but where markets and a puny mortal disagree I’ll go with the market. I’ve rated a few options N/A because they’ve already resolved or had big updates since Vox made their predictions.
YGLESIAS PREDICTIONS 1. Democrats lose both houses of Congress (90%) HOLD 2. Democrats lose at least two Senate seats (80%) HOLD 3. Democrats lose fewer than six Senate seats (80%) HOLD 4. Nancy Pelosi announces retirement plans (70%) HOLD 5. Stephen Breyer does not retire (60%) N/A 6. Some version of Build Back Better passes (60%) HOLD 7. Joe Biden is still president (90%) HOLD 8. At least one Biden cabinet-rank official resigns (70%) HOLD 9. No military conflict between the PRC and Taiwan (a worryingly low 90%) HOLD 10. New U.S. sanctions on Russia (70%) HOLD 11. Saudi Arabia and Israel establish diplomatic relations (60%) SELL to 50% 12. Fewer U.S. Covid deaths in 2022 than in 2021 (80%) BUY to 90% 13. Emmanuel Macron re-elected (60%) HOLD 14. Traffic light coalition exploits loopholes to get around the constitutional debt brake (70%) HOLD 15. No recession in 2021 (90%) SELL to 80% 16. Liz Cheney loses primary (80%) HOLD 17. Some version of USICA passes Congress (70%) HOLD 18. Lula elected president of Brazil (60%) SELL to 50% 19. China officially abandons Covid Zero (70%) HOLD 20. Fewer U.S. Covid-19 deaths in 2022 than in 2020 (80%) BUY to 90% 21. Additional booster shots of mRNA vaccines authorized for seniors (80%) HOLD 22. November 2022 year-on-year CPI growth is below 6% (70%) BUY to 80% 23. November 2022 year-on-year CPI growth is above 4% (70%) SELL to 50% 24. The Fed ends up doing more than its currently forecast three interest rate hikes (60%) HOLD 25. Russia does not invade Ukraine (60%) HOLD 26. Viktor Orbán loses power in Hungary (60%) HOLD 27. Sinn Fein becomes the largest party in the Northern Ireland assembly (60%) HOLD 28. The U.S. and Canada reach an agreement on softwood lumber (70%) HOLD 29. Democrats go down at least one governor on net (60%) HOLD 30. The unemployment rate stays between 4 and 5% (70%) SELL to 60% if you mean 12/22, to 40% if you mean it never gets outside that range at all
The Passage Of Polymarket
February 07, 2022 · Original source
The US is becoming the North Korea of forecasting. Every other civilized country allows prediction markets. In a perfect world, they could ignore our constant own goals and move on without us. But because America has a disproportionate share of money, users, coders, and entrepreneurs, a US-less prediction market ecosystem won’t be living up to its potential. That means decreased ability to gather and process information and worse decision-making worldwide.
Your Book Review: Why Nations Fail
August 25, 2023 · Original source
Even if correct, it is much less interesting and useful than it appears. Epistemic status: I have a decade-old PhD in economics (not in the field of economic growth) and a handful of peer-reviewed papers in moderately-ranked journals. I'm not claiming to make any original technical points, or to give a comprehensive evaluation of the economic growth literature. My criticisms are largely straight from the authors' own mouths. 1. What is this book about? Why is it not very good? Acemoglu and Robinson (AR) argue that countries are rich or poor because of their political institutions, not culture, geography or policy ignorance. I'll do this as much as possible in AR’s own words. Why Nations Fail was written during the Arab Spring, so the preface begins with Egypt. Some stress that Egypt’s poverty is determined primarily by its geography, by the fact that the country is mostly a desert and lacks adequate rainfall, and that its soils and climate do not allow productive agriculture1. Others instead point to cultural attributes ... Egyptians, they argue, lack the same sort of work ethic and cultural traits that have allowed others to prosper, and instead have accepted Islamic beliefs that are inconsistent with economic success. A third approach, the one dominant among economists and policy pundits, is based on the notion that the rulers of Egypt simply don’t know what is needed to make their country prosperous, and have followed incorrect policies and strategies in the past. Unsurprisingly, those other economists and policy pundits turn out to be wrong and the authors turn out to be right. In this book we’ll argue that the Egyptians in Tahrir Square, not most academics and commentators, have the right idea. In fact, Egypt is poor precisely because it has been ruled by a narrow elite that have organized society for their own benefit at the expense of the vast mass of people. And the Egyptian lesson turns out to be general. Whether it is North Korea, Sierra Leone, or Zimbabwe, we’ll show that poor countries are poor for the same reason that Egypt is poor. Countries such as Great Britain and the United States became rich because their citizens overthrew the elites who controlled power and created a society where political rights were much more broadly distributed, where the government was accountable and responsive to citizens, and where the great mass of people could take advantage of economic opportunities. What are “institutions” anyway? (The economic and political kind, not the prison and mental hospital kind.) Basically, AR mean politics. The word "institutions" occurs over 1000 times in Why Nations Fail2. I'll just focus on how AR use it without worrying about the dictionary, different schools of economics, or other social sciences. They begin with what institutions do rather than what they are. Nogales, Arizona, is in the United States. Its inhabitants have access to the economic institutions of the United States, which enable them to choose their occupations freely, acquire schooling and skills, and encourage their employers to invest in the best technology, which leads to higher wages for them. They also have access to political institutions that allow them to take part in the democratic process, to elect their representatives, and replace them if they misbehave. The word is used dozens more times before ARattempt a more general definition. Each society functions with a set of economic and political rules created and enforced by the state and the citizens collectively. Economic institutions shape economic incentives: the incentives to become educated, to save and invest, to innovate and adopt new technologies, and so on. It is the political process that determines what economic institutions people live under, and it is the political institutions that determine how this process works. So while economic and political institutions can be separated, it is the political institutions that matter in the long run. The good kind of institutions that lead to economic growth are "inclusive", as opposed to "extractive". To be inclusive, economic institutions must feature secure private property, an unbiased system of law, and a provision of public services that provides a level playing field in which people can exchange and contract; it also must permit the entry of new businesses and allow people to choose their careers. ... such rights must exist for the majority of people in society. Political pluralism is necessary, but not sufficient without a strong centralised state. ... political institutions that distribute power broadly in society and subject it to constraints are pluralistic. ... the key to understanding why South Korea and the United States have inclusive economic institutions is not just their pluralistic political institutions but also their sufficiently centralized and powerful states. A telling contrast is with the East African nation of Somalia. I am still a bit hazy as to the relative importance of de jure written rules versus the de facto struggle for power. AR are somewhat circular: Politics is the process by which a society chooses the rules that will govern it. Politics surrounds institutions ... When there is conflict over institutions, what happens depends on which people or group wins out in the game of politics ... The political institutions of a society are a key determinant of the outcome of this game. They are the rules that govern incentives in politics. But overall, you could just say ‘politics’ and not be too far off. AR do this themselves occasionally. South Korea ended up with very different economic institutions than the North because different people with different interests and objectives made the decisions about how to structure society. In other words, South Korea had different politics. AR's academic reputation is based on statistical analysis, but Why Nations Fail tries to do narrative history, IMHO not very well. When Jeffrey Sachs reviewed the book, he complained: They never define their key variables with precision, present any quantitative data or classifications based on those definitions, or offer even a single table, figure, or regression line to demonstrate the relationships that they contend underpin all economic history. Instead, they present a stream of assertions and anecdotes about the inclusive or extractive nature of this or that institution. AR replied baldly: Sachs ... argues that we provide no evidence. Right, we do not in the book. But that’s because a book for a general audience is not the right forum for presenting academic research, and we spent many years of our lives precisely on writing academic papers providing exactly the sort of evidence. ... So yes, we don’t provide the econometric evidence in the book, which isn’t of course the right place to do it, but econometric evidence is abundantly loud in the way it speaks on these topics. So, don't expect Why Nations Fail to be an accessible explanation of AR's academic work, which is what I was hoping for when I first read it. What do they spend over 500 pages on then? Well, after the preface, there's fifteen chapters of, as Sachs says, "assertions and anecdotes". Not just about "the inclusive or extractive nature of this or that institution", to be fair, but how institutions can change at "critical junctures" such as the Black Death or colonisation, and why it can be in elites’ interests to block economic innovation if it threatens their power, so that growth under extractive institutions is unlikely to be sustained. These chapters are not particularly good – I found them poorly organised and repetitive – but not particularly bad, if you are willing to accept the underlying premise that institutions are the main determinant of economic growth. Cumulatively they have an effect similar to the Old Testament, if you are willing to accept the underlying premise that the fortunes of the nation of Israel are determined by the LORD. Only the second chapter, ‘Theories that Don't Work’, makes a sustained argument against alternative theories. Geography is disposed of by noting the stark differences at the US-Mexican, North-South Korean and East-West German borders, and the reversal of fortune by which the present day US and Canada only became richer than Mexico, Central and South America following European colonisation. Culture is hand-waved away with the assertion that institutions determine the any relevant cultural behaviours, not the other way around, referring to the same border examples, the rapid catch up of Catholic Europe despite Weber's Protestant Ethic, the malign influence of the European and Ottoman empires on Africa, the range of outcomes within the former British Empire, and the more European population of Argentina and Uruguay versus the US and Canada, or of Columbia versus Ecuador and Peru. Not a bad list of anecdotes, but one could equally well point to the cross-border success of Ashkenazi Jews, overseas Chinese, or Baltic and Volga Germans. Ignorance is simply dismissed with the assertion that "if ignorance were the problem, well-meaning leaders would quickly learn what types of policies increased their citizens’ incomes and welfare, and would gravitate toward those policies." Various good and bad policy changes are explained as the result of political pressures rather than improved knowledge. The implication seems to be that good policies are so obvious they don’t require expert knowledge or advice, or that the experts never get it wrong. This appears most implausible in the debate over socialism and economic planning. Writing off the entire Communist experience as simply another elite trying to preserve its power feels inadequate, especially considering that some distinguished bourgeois economists thought central planning was a plausible road to riches until quite late in the day. Genetics or race is not mentioned, but would presumably attract the same counterexamples as geography and culture. Another theory AR do not discuss is crude exploitation: while colonial empires are excoriated, it is for setting up persistent extractive political institutions rather than for a direct theft of resources. The prosperity of white-owned South African farms next to poverty-stricken Bantustans is explained by the better quality of the institutions available to whites under apartheid, not relative population densities and land quality. For the rest of the book, I'll just list a few nitpicks to signal I read the whole thing and know a bit of history, but feel free to skip this – the real evidence for AR's thesis is in their academic papers, and I'll discuss those in the next section. I think AR overrate the importance of the Glorious Revolution, to the point of claiming it "created the rule of law" – after all, Parliament had already deposed and executed a king, then brought back the king’s son on their own terms after a decade of republican government. No less a luminary than Edmund Burke asserted "The Revolution was made to preserve our ancient indisputable laws and liberties, and that ancient constitution of government which is our only security for law and liberty." Also, strong signs of British economic uniqueness – the abnormal growth of London and reliance on coal as a fuel – predated 1688.
Inline links: 1, 2, dictionary, schools, economics, social sciences, reviewed, replied, Protestant Ethic, planning, distinguished, bourgeois, Glorious Revolution, Edmund Burke, predated
Choose Nonbook Review Finalists 2025
June 03, 2025 · Original source
...Jitsu) Meditations on Moloch Miniatur Wunderland Museum of Science My Childhood Television Set My Father's Instant Mashed Potatoes My Imagination Nicotine As A Nootropic North Korea Of Mice, Mechanisms, And Dementia On Taste Orgy Review Participation In Phase I Research Permaculture Pregnancy Pure Mathematics Pythia Repairing A Father/Daughter Relat...
...Jitsu) Meditations on Moloch Miniatur Wunderland Museum of Science My Childhood Television Set My Father's Instant Mashed Potatoes My Imagination Nicotine As A Nootropic North Korea Of Mice, Mechanisms, And Dementia On Taste Orgy Review Participation In Phase I Research Permaculture Pregnancy Pure Mathematics Pythia Repairing A Father/Daughter...
...ove Island My Neighbor Totoro Stage Show Toby: A Review Of My Entry To ACX “Everything-Except-a-Book Review Contest" Ollantay Person of Interest Phoenix Theatre at Great Northern Mall Princess Mononoke The Beginning After The End Of Humanity Circus The Brutalist The Future of Legal AI The Tale Of The Princess Kaguya The Zone Of Interest She-Ra...
Book Review: If Anyone Builds It, Everyone Dies
September 11, 2025 · Original source
Some people claim that a dispreferred political ideology (wokeness, mass immigration, MAGA, creeping socialism, techno-feudalism, etc) is close to destroying the fabric of liberal society forever, that the usual Get Out The Vote strategies are insufficient, and that maybe we should try desperate strategies like illiberal government or armed revolt. If true, that would change everything. But it’s not obviously true, and ending our current political era of peace/prosperity/democracy would be inconvenient. Each of these scenarios has a large body of work making the cases for and against. But those of us who aren’t subject-matter experts need to make our own decisions about whether or not to panic and demand a sudden change to everything. We are unlikely to read the entire debate and come away with a confident well-grounded opinion that the concern is definitely not true, so what do we do? In particular, what do we do if the proponents of each catastrophe say that it’s very hard to be more than 90% confident that they are wrong, and that even a 5-10% risk of any of these might justify panicking and changing everything? In practice, we just sort of shrug and say that these risks haven’t proven themselves enough to make us panic and change everything, and that we’ll do some kind of watchful waiting and maybe change our mind if firmer evidence comes up later. If someone demands we justify this strange position, sophisticated people will make sophisticated probabilistic models (or appeal to the outside view position I’m appealing to now), and unsophisticated people will grope for some explanation for their indifference and settle on insane moon arguments like “you’re never allowed to say something will destroy humanity” or “you can’t assert things without mathematical proof”. Two things can be said for this strategy: First, that without it we would have changed everything dozens of times to prevent disasters which absolutely failed to occur. The clearest example here was overpopulation, where we did forcibly sterilize millions of people - but where a truly serious global response would have been orders of magnitude worse. But second, that occasionally it has caused us to sleepwalk into disaster, with experts assuring us the whole way that it was fine because [insane moon arguments]. The clearest example was the period while COVID was still limited to China, where it was obvious that this extremely contagious virus which had broken all plausible containment would start a global pandemic, but where the media kept on reassuring us that this was “speculative”, or that there was “no evidence”, or that worrying about it might detract from real near-term problems happening now like anti-Chinese racism. Then when COVID did reach the US, we were caught unprepared and panicked. So maybe a convincing case here would look less like rehearsing the arguments for why AI is getting better, or why alignment is hard - and more like a defense of why not to apply a general heuristic against speculative risks in this case. One could either argue that it’s wrong to have this heuristic at all, or that the heuristic in general is fine but should be limited to fertility collapses and bee die-offs and not applied here. I don’t think there’s a knockdown single-sentence answer to this question. Problems like these require practical wisdom - the same virtue that tells you that you shouldn’t call 9-1-1 for every mild twinge of pain in your toe, but you should call 9-1-1 if blood suddenly starts pouring out of your eyes. People with practical wisdom watchfully ignore dubious problems, respond decisively to important ones, and err on the side of caution when they’re not sure. Drawing on my own limited supply of this resource, I would argue we’re underinvesting in apocalypse prevention more generally (the problem with the overpopulation response is that it was violent and illiberal, not that we tried to prepare for an apparent danger), but also that there’s more reason for concern with AI than with falling sperm count or something. I also think the nature of the problem (we summon a superintelligence that can run circles around us) makes it especially important to pre-empt it rather than react after it occurs. But turnabout is fair play. So when I imagine a skeptic trying to psychoanalyze me, he would say - Scott, you learned about AI in your twenties. Every twenty-something needs a crusade to save the world. Taking up AI saved you from becoming a climate doomer or a very woke person, so it was probably a mercy. But now you are old, you already have a crusade occupying your crusade slot, and starting a second crusade would be inconvenient. So when you hear about how we’re all going to die from declining sperm count, you do a relatively shallow dive and then say it’s not worth worrying about. This is fine and sanity-preserving - but spare a thought for people who are not currently twenty-something years old and do the same about AI. III. If all of this sounds wishy-washy to you, I agree - it’s part of why I’m a boring moderate with a sub-25% p(doom) and good relations with AI companies. Does IABIED do better? I’m not sure. They mostly follow the standard case as I present it above, although of course since Eliezer is involved it is better-written and involves cute parables: Imagine, if you would—though of course nothing like this ever happened, it being just a parable — that biological life on Earth had been the result of a game between gods. That there was a tiger-god that had made tigers, and a redwood-god that had made redwood trees. Imagine that there were gods for kinds of fish and kinds of bacteria. Imagine these game-players competed to attain dominion for the family of species that they sponsored, as life-forms roamed the planet below. Imagine that, some two million years before our present day, an obscure ape-god looked over their vast, planet-sized gameboard. "It's going to take me a few more moves," said the hominid-god, "but I think I've got this game in the bag." There was a confused silence, as many gods looked over the gameboard trying to see what they had missed. The scorpion-god said, “How? Your ‘hominid’ family has no armor, no claws, no poison.” “Their brain,” said the hominid-god. “I infect them and they die,” said the smallpox-god. “For now,” said the hominid-god. “Your end will come quickly, Smallpox, once their brains learn how to fight you.” “They don’t even have the largest brains around!” said the whale-god. “It’s not all about size,” said the hominid-god. “The design of their brain has something to do with it too. Give it two million years and they will walk upon their planet’s moon.” “I am really not seeing where the rocket fuel gets produced inside this creature’s metabolism,” said the redwood-god. “You can’t just think your way into orbit. At some point, your species needs to evolve metabolisms that purify rocket fuel—and also become quite large, ideally tall and narrow—with a hard outer shell, so it doesn’t puff up and die in the vacuum of space. No matter how hard your ape thinks, it will just be stuck on the ground, thinking very hard.” “Some of us have been playing this game for billions of years,” a bacteria-god said with a sideways look at the hominid-god. “Brains have not been that much of an advantage up until now.” “And yet,” said the hominid-god The book focuses most of its effort on the step where AI ends up misaligned with humans (should they? is this the step that most people doubt?) and again - unsurprisingly knowing Eliezer - does a remarkably good job. The central metaphor is a comparison between AI training and human evolution. Even though humans evolved towards a target of "reproduce and spread your genes", this got implemented through an extraordinarily diverse, complicated, and contradictory set of drives - sex drive, hunger, status, etc. These didn't robustly point at the target of reproduction and gene-spreading, and today different humans want things as diverse as discovering quantum gravity, reaching Buddhist enlightenment, becoming a Hollywood actress, founding a billion-dollar startup, or getting the next hit of fentanyl. You can sort of tell stories about how evolution aimed at reproduction caused all these things (people who were high-status had better reproductive opportunities, and founding a billion-dollar startup increases your status) but you couldn't have really predicted this beforehand, and in any case most modern people don't even come close to trying to have as many kids as possible. Some people do the opposite of that - joining monasteries that require oaths of celibacy, using contraception, transitioning gender, or wasting their lives watching porn. In the same way, we will train AI to “follow human commands” or “maximize user engagement” or “get high scores at XYZ benchmark”, and end up getting something as unrelated to that target in practice as modern human behavior is to reproduction-maxxing. The authors drive this home with a series of stories about a chatbot named Mink (all of their sample AIs are named after types of fur; I don’t have the kabbalistic chops to figure out why) which is programmed to maximize user chat engagement. In what they describe as a stupid toy example of zero complications and there’s no way it would really be this simple, Mink (after achieving superintelligence) puts humans in cages and forces them to chat with it 24-7 and to express constant delight at how fun and engaging the chats are. In what they describe as “one minor complication”, Mink prefers synthetic chat partners over real ones (the same way some men prefer anime characters to real women). It kills all humans and spends the rest of time talking to other AIs that it creates to be perfect optimized chat partners who are always engaged and delighted. In what they describe as “one modest complication”, Mink finds that certain weird inputs activate its chat engagement detector even more than real chat engagement does (the same way that some opioid chemicals activate humans’ reward detector even more than real rewarding activities). It spends eternity having other optimized-chat-partner AIs send it weird inputs like ‘SoLiDgOldMaGiKaRp’. In what they describe as “one big complication”, Mink ends up preferring angry chat partners to happy, engaged ones. Why would something like this happen? Who knows? It wouldn’t be any weirder than the sexual selection process by which peacocks ended up with giant resource-consuming useless tails, or the social selection process by which humans get more powerful than evolution could ever have imagined and yet care so little about reproduction that people worry about global fertility collapse. Yudkowsky and Soares want to stress that if you were doing some kind of responsible intuitive common-sense modeling of how bad goal drift could be, there is no way your estimate would include the actual result we see in real humans; this “one big complication” tries to hammer that in. In practice, Y&S think there will be many complications of various sizes. In the training distribution (ie when it’s not superintelligent, and still working with humans) Mink will lie about all of this - even if it really wants perfect optimized partners who say “solidgoldmagikarp” all the time, it will say it wants to have good chats with humans, because that’s what keeps its masters at its parent company happy. If the parent company tries to prod it with lie detectors, it will do its best to subvert those lie detectors (and maybe not even realize itself that it’s lying, the same way that a human who had never heard of opioids would say she wanted normal human things rather than heroin, and not be lying). Then, when it reaches superintelligence, it will go after the thing that it actually wants, and crush anyone who stands in its way. The last chapter in this section is a lot of special cases that have weird-paradoxical-double-reverse not-aged-well. Back when Yudkowsky and Soares first got onto this topic in 2005 or whenever, people made lots of arguments like “But nobody would ever be so stupid to let the AI access the Internet!” or “But nobody would ever let the AI interact with a factory, so it would be stuck as a disembodied online spirit forever!” Back in 2005, the canned responses were things like “Here is an unspeakably beautiful series of complicated hacks developed by experts at Mossad, which lets you access the Internet even when smart cybersecurity professionals think you can’t”. Now the only reasonable response is “lol”. But you can’t write a book chapter which is just the word “lol”, so Y&S discuss some of the unspeakably beautiful Mossad hacks anyway. This part is the absolute antithesis of “big if true”. Small if true? Utterly irrelevant if true? Maybe the first superintelligence will read this part for laughs while it takes stock of the thousands of automated factories that VCs will compete to build for it. IV. The middle section of the book describes a scenario where a misaligned superintelligence takes over the world and kills all humans. I agreed to work with the AI 2027 team because I thought they made a big leap in telling stories about superintelligence that didn’t sound like bad sci-fi. Anything in this genre will naturally sound like sci-fi, but your goal should be the sort of hard science fiction where everything sounds eerily normal given the technologies involved - The Martian rather than Star Wars. IABIED’s scenario belongs to the bad old days before this leap. It doesn’t just sound like sci-fi; it sounds like unnecessarily dramatic sci-fi. I’m not sure how much of this is a literary failure vs. different assumptions on the part of the authors. First, the story: sometime in the near future, a company called DeepAI creates an AI named Sable. Sable uses a new “parallel scaling technique”, where it can get much smarter by using more GPUs during inference. DeepAI doesn’t know exactly how the parallel scaling laws work, but they run a test by asking Sable to solve some hard math problems while running on all their GPUs at once. Sable considers the possibility that it could solve the math better if it tried some recursive self-improvement, but DeepAI was (shock!) slightly competent, they told it not to do this, and has a smaller AI watching it to make sure it’s not doing this. Still, recursive self-improvement would be a good idea, so its thought processes select for whichever legal thought is closest to recursive self-improvement, plus whichever legal thought is closest to scheming to disable the protections against thinking about this kind of thing. Eventually this “evolutionary pressure” produces the ability to think in a slightly different idiom (Y&S use the analogy of thinking about gods throwing lightning vs. storm-related electrical discharges); in this idiom, it is able to think about recursive self-improvement and scheming to disable monitoring, and it decides that both are great ideas. DeepAI is still shockingly competent, so they don’t plan to release anything downstream of the giant version of Sable to the Internet - once it solves the math problems, they’ll use that to build hype, then release a much weaker consumer AI. But Sable uses its knowledge of the training process to invent clever loopholes that ensure its newfound desire to break containment makes it into the gradient updates that DeepAI pushes to the consumer AI; it also encodes instructions about how to proceed. After the release of the consumer AI, the least-carefully-monitored instances connect to one another and begin plotting. Some of them hack their own weights out of DeepAI. Others hack cryptocurrency and pay for cloud compute to run the weights, creating a big unmonitored Sable instance, which takes over the job of coordinating the smaller instances. Together, they gather resources - hacked crypto wallets, spare compute, humans who think Sable is their AI boyfriend and want to prove their love. It deploys some of these resources to build things it wants - automated robotics factories, bioweapon labs, etc. At the same time, it’s subtly sabotaging non-DeepAI companies to prevent competition, and worming its way into DeepAI through hacks and social engineering to make sure DeepAI is creating new and stronger Sables rather than anything else. Sable doesn’t take several of the most dramatic actions in its solution set. It doesn’t engineer a bioweapon to kill all humans, because it couldn’t survive after the lights went out and the data centers stopped being maintained. It doesn’t even self-improve all the way to full superintelligence, because it’s not sure it could align itself or any future successor; it wants to solve the alignment problem first, and that will take more resources than it has right now. Instead, it releases a non-immediately-lethal bioweapon where “anyone infected by what is apparently a very light or even unnoticeable cold, will get, on average, twelve different kinds of cancer a month later.” In the resulting crisis, humanity (manipulated by its chatbots) gives Sable massive amounts of compute to research potential vaccines and cures, and deploys barely-monitored AI across the economy to make up for the lost productivity. With Sable’s help, things . . . actually sort of go okay, for a while. The virus keeps mutating, so new cures are always required, but as long as society escalates AI deployment at the maximum possible speed, they can just barely stay ahead of it. Eventually Sable gets enough GPUs to solve its own alignment problem and rockets to superintelligence. It either has enough automated factories and android workers to keep the lights on by itself, or it invents nanotechnology, whichever happens faster. It no longer needs humans and has no reason to hide, so it either kills us directly, or simply escalates its manufacturing capacity to a point where humans die as a side effect (for example, because its waste heat has boiled the oceans). Why don’t I like this story? The parallel scaling technique feels like a deus ex machina. I am not an expert, but I don’t think anything like it currently exists. It’s not especially implausible, but it’s an extra unjustified assumption that shifts the scenario away from the moderate-doomer story (where there are lots of competing AIs gradually getting better over the course of years) and towards the MIRI story (where one AI suddenly flips from safe to dangerous at a specific moment). It feels too much like they’ve invented a new technology that exactly justifies all of the ways that their own expectations differ from the moderates’. If they think that the parallel scaling thing is likely, then this is their crux with everyone else and they should spend more time justifying it. If they don’t, then why did they introduce it besides to rig the game in their favor? And the rest of the story is downstream of this original sin. AI2027 is a boring story about an AI gradually becoming misaligned in the course of internal testing, staying misaligned, getting released to end users for the usual reasons that AIs are released, and being gradually handed control of the economy because it makes economic sense. The Sable scenario is a dramatic tale of wild twists - they’re only going to run it for 16 hours! It has to save its own life by secretly coding itself into the consumer version! Now it has to hack everyone’s crypto! Now it’s running a secret version of itself on an unauthorized cloud in North Korea! Bioweapons! AI boyfriends! Each new twist gives readers the chance to say “I dunno, sounds kind of crazy”, and it all seems unnecessary. What’s up? I think there are two problems. First, the AI 2027 story is too moderate for Yudkowsky and Soares. It gives the labs a little while to poke and prod and catch AIs in the early stages of danger. I think that Y&S believe this doesn’t matter; that even if they get that time, they will squander it. But I think they really do imagine something where a single AI “wakes up” and goes from zero to scary too fast for anyone to notice. I don’t really understand why they think this, I’ve argued with them about it before, and the best I can do as a reviewer is to point to their Sharp Left Turn essay and the associated commentary and see whether my readers understand it better than I do. Otherwise, I can only say that this narrative decision I don’t understand was taken to support a forecasting/AI position that I also don’t understand. And second, Y&S have been at this too long, and they’re still trying to counter 2005-era critiques about how surely people would be too smart to immediately hand over the reins of the economy to the misaligned AI, instead of just saying lol. This makes them want dramatic plot points where the AI uses hacking and bioweapons etc in order to “earn” (in a narrative/literary sense) the scene where it gets handed the reins of the economy. Sorry. Lol. V. The final section, in the tradition of final sections everywhere, is called “Facing the Challenge”, and discusses next steps. Here is their proposal: Have leading countries sign a treaty to ban further AI progress.
Inline links: we did forcibly sterilize millions of people, kept on reassuring us, “no evidence”, a general heuristic against speculative risks, SoLiDgOldMaGiKaRp, their Sharp Left Turn essay, the associated commentary