Astral Codex Ten

AI Policy Institute

8 min read

AI Policy Institute

Article

AI Policy Institute is a recurring organization in the Astral Codex Ten archive, appearing 2 times across 2 issues between October 05, 2023 and December 01, 2023. The archive places it in contexts such as “Source: AI Policy Institute and YouGov, h/t Holly”; “The poll was funded by the AI Policy Institute”. It most often appears alongside Bostrom, China, YouGov.

Metadata

  • Category: Organizations
  • Mention count: 2
  • Issue count: 2
  • First seen: October 05, 2023
  • Last seen: December 01, 2023

Appears In

Source Context

Recovered passages from the original issue text. When the raw archive preserved outbound links inside the source passage, they are listed directly under the quote.

Pause For Thought: The AI Pause Debate
October 05, 2023 · Original source
For every word like "trust" or "worried", assume I mean "...enough to outweigh other considerations" Along with this overall arc, the debate included a few other points: Holly Elmore argued in The Case For AI Advocacy To The Public that pro-pause activists should be more willing to take their case to the public. EA has a long history of trying to work with companies and regulators, and has been less confident in its ability to execute protests, ads, and campaigns. But in most Western countries, the public hates AI and wants to stop it. If you also want to stop it, the democratic system provides fertile soil. Holly is putting her money where her mouth is and leading anti-AI protests at the Meta office in San Francisco; the first one was last month, but there might be more later. Source: AI Policy Institute and YouGov, h/t Holly Matthew Barnett said in The Possibility Of An Indefinite AI Pause that it might be hard to control the length of a pause once started, and might drag on longer than people who expected a well-planned surgical pause might like. He points to supposedly temporary moratoria that later became permanent (eg aboveground nuclear test ban, various bans on genetic engineering) and regulatory agencies that became so strict they caused the subject of their regulation to essentially cease to happen (eg nuclear plant construction for several decades). Such an indefinite pause would either collapse in a disastrous actualization of compute overhang, or require increasingly draconian international pressure to sustain. He thinks of this as a strong argument against most forms of pause, although he is willing to consider a “licensing” system that looks sort of like regulation. Quintin Pope said in AI Is Centralizing By Default, Let’s Not Make It Worse that the biggest threat from AI is centralizing power, either to dictators or corporations. AIs are potentially more loyal flunkies than humans, and let people convert power (including political power and money) into intelligence more efficiently than the usual methods. His interest is mostly in limiting the damage, putting him skew to most of the other people in this debate. He would support regulation that makes it easier for small labs to catch up to big ones, or that limits the power-centralizing uses of AI, but oppose regulation focused on centralizing AI power into a few big, supposedly-safer corporations. Percent of population in each country saying AI has more benefits than drawbacks. Pope uses this table to suggest AI regulation would be decentralizing, since the furthest-ahead countries are the most eager to regulate. Source: Ipsos; h/t Quintin II. For a “debate”, this lacked much inter-participant engagement. Most people posted their manifesto and went home. The exception was the comments section of Nora’s post, AI Pause Will Likely Backfire. As usual, a lot of the discussion was just clarifying what everyone was fighting about, but there were also a few real fights: Gerald Monroe thought that the history of nuclear weapons suggested pauses like this were impossible (because many countries did build nuclear weapons). David Manheim thought it suggested pauses like this could work (because there were some successful arms limitation treaties, and less nuclear proliferation than would have happened without international cooperation). Manheim also brought up the successful bans on ozone-destroying CFCs and on human cloning.
Inline links: The Case For AI Advocacy To The Public, leading anti-AI protests at the Meta office in San Francisco, https://substackcdn.com/image/fetch/$s_!vUmB!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4ebc8f9a-6c5a-4bd8-8542-358bd4fd0181_512x410.webp, The Possibility Of An Indefinite AI Pause, AI Is Centralizing By Default, Let’s Not Make It Worse, https://substackcdn.com/image/fetch/$s_!0Ae8!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F165f17cd-ca63-4f54-95e4-29f53b890578_1204x1126.webp, AI Pause Will Likely Backfire, thought
Source: AI Policy Institute and YouGov, h/t Holly Matthew Barnett said in The Possibility Of An Indefinite AI Pause that it might be hard to control the length of a pause once started, and might drag on longer than people who expected a well-planned surgical pause might like. He points to supposedly temporary moratoria that later became permanent (eg aboveground nuclear test ban, various bans on genetic engineering) and regulatory agencies that became so strict they caused the subject of their regulation to essentially cease to happen (eg nuclear plant construction for several decades). Such an indefinite pause would either collapse in a disastrous actualization of compute overhang, or require increasingly draconian international pressure to sustain. He thinks of this as a strong argument against most forms of pause, although he is willing to consider a “licensing” system that looks sort of like regulation. Quintin Pope said in AI Is Centralizing By Default, Let’s Not Make It Worse that the biggest threat from AI is centralizing power, either to dictators or corporations. AIs are potentially more loyal flunkies than humans, and let people convert power (including political power and money) into intelligence more efficiently than the usual methods. His interest is mostly in limiting the damage, putting him skew to most of the other people in this debate. He would support regulation that makes it easier for small labs to catch up to big ones, or that limits the power-centralizing uses of AI, but oppose regulation focused on centralizing AI power into a few big, supposedly-safer corporations. Percent of population in each country saying AI has more benefits than drawbacks. Pope uses this table to suggest AI regulation would be decentralizing, since the furthest-ahead countries are the most eager to regulate. Source: Ipsos; h/t Quintin II. For a “debate”, this lacked much inter-participant engagement. Most people posted their manifesto and went home. The exception was the comments section of Nora’s post, AI Pause Will Likely Backfire. As usual, a lot of the discussion was just clarifying what everyone was fighting about, but there were also a few real fights: Gerald Monroe thought that the history of nuclear weapons suggested pauses like this were impossible (because many countries did build nuclear weapons). David Manheim thought it suggested pauses like this could work (because there were some successful arms limitation treaties, and less nuclear proliferation than would have happened without international cooperation). Manheim also brought up the successful bans on ozone-destroying CFCs and on human cloning.
Inline links: The Possibility Of An Indefinite AI Pause, AI Is Centralizing By Default, Let’s Not Make It Worse, https://substackcdn.com/image/fetch/$s_!0Ae8!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F165f17cd-ca63-4f54-95e4-29f53b890578_1204x1126.webp, AI Pause Will Likely Backfire, thought
Links For November 2023
December 01, 2023 · Original source
I admit I’m surprised by this! The poll was funded by the AI Policy Institute, but it seems to have been conducted through YouGov, and the results are stable to various different framings of the question. Maybe people don’t actually think about extinction in real life, but if the pollsters bring it up, people will agree that it sounds like a bad thing? Or who knows, maybe they’re really worried.
Inline links: through YouGov, various different framings of the question