Astral Codex Ten

Nuclear Regulatory Commission

8 min read

Nuclear Regulatory Commission

Article

Nuclear Regulatory Commission is a recurring organization in the Astral Codex Ten archive, appearing 2 times across 2 issues between July 01, 2023 and October 05, 2023. The archive places it in contexts such as “Nuclear Regulatory Commission training and assessment specialists”; “Once the NRC chose to implement Probabilistic Risk Assessment”; “Since the creation of the Nuclear Regulatory Commission in 1975, there has never been a major nuclear accident in the US”. It most often appears alongside United States, 1960 Valdivia earthquake, AEC.

Metadata

  • Category: Organizations
  • Mention count: 2
  • Issue count: 2
  • First seen: July 01, 2023
  • Last seen: October 05, 2023

Appears In

Source Context

Recovered passages from the original issue text. When the raw archive preserved outbound links inside the source passage, they are listed directly under the quote.

Your Book Review: Safe Enough?
July 01, 2023 · Original source
The quotes are from the book Safe Enough? A History of Nuclear Power and Accident Risk, by Thomas Wellock. In his day job, Wellock is the official historian of the U.S. Nuclear Regulatory Commission (NRC), an organization whose official responsibilities include screaming ‘Yes!’ to anyone who broaches this question. A coarsely cynical reader might thus expect Wellock to sidestep damning details of nuclear risk at the behest of his employer. This cynicism does a disservice to Wellock’s ambition.
Inline links: Safe Enough? A History of Nuclear Power and Accident Risk
Yet "Safe Enough?" is less of a history of events than a biography of an idea, the birth of "Probabilistic Risk Assessment" as the guiding principle for understanding and mitigating risks in complex systems. The heroes of Wellock's book are not nuclear plant night shift assistant supervisors, or the Nuclear Regulatory Commission training and assessment specialists, though they each make important cameos. The city of Toledo, Ohio is not safeguarded by watchful superheroes. It is protected by a methodology.
"Safe Enough?" was not written as a defense of the Nuclear Regulatory Commission's regimented style. But as an outsider reading about the math for the first time, it became clear to me that once the NRC chose to implement Probabilistic Risk Assessment, an intrusive bureaucracy became its destiny.
Pause For Thought: The AI Pause Debate
October 05, 2023 · Original source
HOW LONG TO PAUSE. The biggest disadvantage of pausing for a long time is that it gives bad actors (eg China)1 a chance to catch up. Suppose the West is right on the verge of creating dangerous AI, and China is two years away. It seems like the right length of pause is 1.9999 years, so that we get the benefit of maximum extra alignment research and social prep time, but the West still beats China. Obviously the problem with the Surgical Pause is that we might not know when we’re on the verge of dangerous AI, and we might not know how much of a lead “the good guys” have. Surgical Pause proponents suggest being very conservative with both free variables. This is less of a well-thought-out plan and more saying “come on guys, let’s at least try to be strategic here”. At the limit, it suggests we probably shouldn’t pause for six months, starting right now. Since this involves leading labs burning their lead time for safety, in theory it could be done unilaterally by the single leading lab, without international, governmental, or even inter-lab coordination. But you could buy more time if you got those things too. Some leading labs have promised to do this when the time is right - for example OpenAI and (a previous iteration of) DeepMind - with varying levels of believability. AnonResearcherAtMajorAILab discussed some of the strategy here in Aim For Conditional AI Pauses, and this Less Wrong post is also very good. Regulatory Pause: If one benefit of the Simple Pause is to use the time to prepare for AI socially and politically, maybe we should just pause until we’ve completed social and political preparations. David Manheim suggests a monitoring agency like the FDA. It would “fast-track” small AIs and trivial re-applications of existing AIs, but carefully monitor new “frontier models” for signs of danger. Regulators might look for dangerous capabilities by asking AIs to hack computers or spread copies of themselves, or test whether they’ve been programmed against bias/misinformation/etc. We could pause only until we’ve set up the regulatory agency, and take hostile actions (like restrict chip exports) only to other countries that don’t cooperate with our regulators or set up domestic regulators of their own. Many people in tech are regulation-skeptical libertarians, but proponents point out that regulation fails in a predictable direction: it usually does successfully prevent bad things, it just also prevents good things too. Since the creation of the Nuclear Regulatory Commission in 1975, there has never been a major nuclear accident in the US. And sure, this is because the NRC prevented any nuclear plants from being built in the United States at all from 1975 to 2023 (one was finally built in July). Still, they technically achieved their mandate. Likewise, most medications in the US are safe and relatively effective, at the cost of an FDA approval process being so expensive that we only get a tiny trickle of new medications each year and hundreds of thousands of people die from unnecessary delays. But medications are safe and effective. Or: San Francisco housing regulators almost never approve new housing, so housing costs millions of dollars and thousands of San Franciscans are homeless - but certainly there’s no epidemic of bad houses getting approved and then ruining someone’s view or something. If we extrapolate this track record to AI, AI regulators will be overcautious, progress will slow by orders of magnitude or stop completely - but AIs will be safe. This is a depressing prospect if you think the problems from advanced AI would be limited to more spam or something. But if you worry about AI destroying the world, maybe you should accept a San-Francisco-housing-level of impediment and frustration. A regulatory pause could be better than a total stop if you think it will be more stable (lots of industries stay heavily regulated forever, and only a few libertarians complain), or if you think maybe the regulator will occasionally let a tiny amount of safe AI progress happen. But it could be worse than a total stop if you expect continued progress will eventually produce unsafe AIs regardless of regulation. You might expect this if you’re worried about deceptive alignment, eg superintelligent AIs that deliberately trick regulators into thinking they’re safe. Or you might think AIs will eventually be so powerful that they can endanger humanity from a walled-off test environment even before official approval. The classic Bostrom/Yudkowsky model of alignment implies both of these things. David Manheim and Thomas Larsen set out their preferred versions of this strategy in What’s In A Pause? and Policy Ideas For Mitigating AI Risk. Total Stop: If you expect AIs to exhibit deceptive alignment capable of fooling regulators, or to be so dangerous that even testing them on a regulator’s computer could be apocalyptic, maybe the only option is a total stop. It’s tough to imagine a total stop that works for more than a few years. You have at least three problems: NON-PARTICIPANTS. As with any pause proposal, unfriendly countries (eg China) can keep working on AI. You can refuse to export chips to them, which will slow them down a little, but their own chips will eventually be up to the task. You will either need a diplomatic miracle, or willingness to resort to less diplomatic forms of coercion. This doesn’t have to be immediate war: Israel has come up with “creative” ways to slow Iran’s nuclear program, and countries trying to frustrate China’s chip industry could do the same. But great powers playing these kinds of games against each other risks wider conflict.
Inline links: 1, OpenAI, DeepMind, Aim For Conditional AI Pauses, this Less Wrong post, look for dangerous capabilities, finally built, What’s In A Pause?, Policy Ideas For Mitigating AI Risk