Astral Codex Ten

Asterisk

21 min read

Asterisk

Article

Asterisk is a recurring publication in the Astral Codex Ten archive, appearing 11 times across 11 issues between August 14, 2022 and December 10, 2025. The archive places it in contexts such as “Asterisk is an upcoming effective altruist magazine currently headquartered in my spare bedroom”; “at asteriskmag.com, a rationalist / effective altruist magazine”; “Asterisk is now releasing its issues piecemeal”. It most often appears alongside OpenAI, Kelsey Piper, Twitter.

Metadata

  • Category: Publications
  • Mention count: 11
  • Issue count: 11
  • First seen: August 14, 2022
  • Last seen: December 10, 2025

Appears In

Source Context

Recovered passages from the original issue text. When the raw archive preserved outbound links inside the source passage, they are listed directly under the quote.

Open Thread 237
August 14, 2022 · Original source
1: Asterisk is an upcoming effective altruist magazine currently headquartered in my spare bedroom. My friend Clara is editor and the first issue will feature articles by me, Kelsey Piper, and other people you might know. Go to asteriskmag.com to check it out and sign up for the mailing list.
Inline links: asteriskmag.com
"Through A Glass Darkly" In Asterisk Magazine
June 26, 2023 · Original source
I have an article summarizing attempts to forecast AI progress, including a five year check-in on the predictions in Grace et al (2017). It’s not here, it's at asteriskmag.com, a rationalist / effective altruist magazine: Through A Glass Darkly. This is their AI issue (it’s not always so AI focused). Other stories include:
Inline links: Grace et al (2017), asteriskmag.com, Through A Glass Darkly
Crash Testing GPT-4: Before releasing GPT-4, OpenAI sent a preliminary version to the Alignment Research Center to test it for unsafe capabilities; the detail that made the news was how the AI managed to hire a gig worker to solve CAPTCHAs for it by pretending to be a blind person. Asterisk interviews Beth Barnes, leader of the team that ran those tests.
Inline links: Crash Testing GPT-4, made the news
What We Get Wrong About AI And China: Professor Jeffrey Ding discusses the Chinese AI situation. If I’m understanding right, China is 1-2 years behind the US, but that this number underplays the size of the gap, and if the US stopped innovating today, China wouldn’t necessarily push ahead in 3 years. Today’s Marginal Revolution links included a claim that a new Chinese model beats GPT-4; I’m very skeptical and waiting to hear more.
Inline links: What We Get Wrong About AI And China, Marginal Revolution links, a claim
24
January 30, 2024 · Original source
This is the claim of an article by Jeremiah Johnson in Asterisk Magazine. The key graphic is this:
Inline links: an article by Jeremiah Johnson in Asterisk Magazine
Asterisk is now releasing its issues piecemeal; along with Johnson on prediction markets, Issue Five includes articles on airline safety, PEPFAR, developing-world democracy, and a review of Going Infinite.
Inline links: Issue Five includes
24
March 12, 2024 · Original source
1: Apparently, forgetting that Futuur exists is a common problem! Jeremiah Johnson’s Asterisk article Do Prediction Markets Have An Election Problem? (featured here two months ago) also didn’t have Futuur. The folks at Futuur, who did not forget that they existed, added themselves to Johnson’s results and find that they actually did quite well, with the play money half outperforming the real money half.:
Inline links: Do Prediction Markets Have An Election Problem?, find that they actually did quite well
Zvi on California's AI Bill
May 08, 2024 · Original source
But Zvi Mowshowitz (summary article in Asterisk, long FAQ on his blog) has looked at it more closely and found:
Inline links: Zvi Mowshowitz, summary article in
Go rogue and commit some other crime that does > $500 million in damage3. If the tests show that the model can do these bad things, the company has to demonstrate that it won’t, presumably by safety-training the AI and showing that the training worked. The kind of training AIs already have - the kind that prevents them from saying naughty words or whatever - would count here, as long as “the safeguards . . . will be sufficient to prevent critical harms.” So the bill isn’t about regulating deepfakes or misinformation or generative art. It’s just about nukes and hacking the power grid. There are some good objections and some dumb objections to this bill. Let’s start with the dumb ones: Some people think this would literally ban open source AI. After all, doesn’t it say that companies have to be able to shut down their models? And isn’t that impossible if they’re open-source? No. The bill specifically says4 this only applies to the copies of the AI still in the company’s possession5. The company is still allowed to open-source it, and they don’t have to worry about shutting down other people’s copies. Other people think this would make it prohibitively expensive for individuals and small startups to tinker with open-source AIs. But the bill says that only companies training giant foundation models have to worry about any of this. So if Facebook trains a new LLaMA bigger than GPT-5, they’ll have to spend some trivial-in-comparison-to-training-costs amount to test it in-house and make sure it can’t make nukes before they release it. But after they do that, third-party developers can do whatever they want to it - re-training, fine-tuning, whatever - without doing any further tests. Other people think all the testing and regulation would make AIs prohibitively expensive to train, full stop. That’s not true either. All the big companies except Meta already do testing like this - here’s Anthropic’s, Google’s, and OpenAI’s - that already approximate the regulations. Training a new GPT-5 level AI is so expensive - hundreds of millions of dollars - that the safety testing probably adds less than 1% to the cost. No company rich enough to train a GPT-5 level AI is going to be turned off by the cost of asking it “hey can you create super-Ebola?”, and putting the answer into a nice legal-looking PDF. This isn’t the “create a moat for OpenAI” bill that everyone’s scared of6. Other people are freaking out over the “certification under penalty of perjury”. In some cases, developers have to certify under penalty of perjury that they’re complying with the bill. Isn’t this crazy? Doesn’t it mean if you make a mistake about your AI, you could go to jail? This is deeply misunderstanding how law works. Perjury means you can’t deliberately lie, something which is hard to prove and so rarely prosecuted. More to the point, half of the stuff I do in an average day as a medical doctor is certified under penalty of perjury - filling out medical leave forms is the first one to come to mind. This doesn’t mean I go to jail if my diagnosis is wrong. It’s just the government’s way of saying “it’s on the honor system”. What are some of the reasonable objections to this bill? Some people think the requirement to prove the AI safe is impossible or nearly so. This is Jessica Taylor’s main point here, which is certainly correct for a literal meaning of “prove”. Zvi points out that it just says “reasonable assurance”, which is a legal term for “you jumped through the right number of hoops”. In this case probably the right number of hoops is doing the same kind of testing that OpenAI/Anthropic/Google are currently doing, or that AI safety testing organization METR recommends. The bill gestures at the National Institute of Standards and Technology a few times here, and NIST just named one of METR’s founders as their AI safety czar, so I would be surprised if things didn’t end going this direction. METR’s tests are possible and many AI models have successfully passed earlier versions. Other people worry there are weird edge cases around derivative models. I think the bill’s intention is that once you prove that your AI is too dumb to create nukes, you’re fine to open-source it. Third-parties can change its character, but not its fundamental intelligence. But in theory, a third party could get tens of millions of dollars of compute and keep training your AI to increase its fundamental intelligence. This would be a weird thing to do, and anyone with that much compute probably should just make their own model. But if someone wanted to screw you over by doing this, technically the law is kind of vague and you would have to trust a judge to say “no, that’s stupid”. Probably the law should clarify that it doesn’t apply to this situation. Other people are worried about a weird rule that you can’t train an AI if you think it’s going to be unsafe. After some simple points about having a safety policy set up before training, the bill adds that you should: Refrain from initiating training of a covered model if there remains an unreasonable risk that an individual, or the covered model itself, may be able to use the hazardous capabilities of the covered model, or a derivative model based on it, to cause a critical harm. This makes less sense than all the other rules - you can test a model post-training to see if it’s harmful, but this seems to suggest you should know something before it’s trained. Is this a fully general “if something bad happens, we can get angry at you”? I agree this part should be clarified. Other people think the benchmarking clause is too vague. The law applies to models trained with > 10^26 FLOPs, or any model that uses advanced technology to be equally as good despite less compute. Equally as good how? According to benchmarks. Which benchmarks? The law doesn’t say. But it does say that the Technology Department will hire some bureaucrats to give guidance on this. I think this is probably the only way to do this; it’s too easy to fake any given benchmark. Every AI company already compares their models to every other AI company on a series of benchmarks anyway, so this isn’t demanding they create some new institution. It’s just “use common sense, ask the bureaucrats if you’re in a gray area, a judge will interpret it if it comes to trial”. This is how every law works. Other people complain that any numbers in the bill that make sense now may one day stop making sense. Right now 10^26 FLOPs is a lot. But in thirty years, it might be trivial - within the range that an academic consortium or scrappy startup might spend to train some cheap ad hoc AI. Then this law will be unduly restrictive to academics and scrappy startups. Is this bad? Presumably we know now that AIs less than 10^26 FLOPs are safe. We suppose that maybe there is some level of AI (let’s say 10^30 FLOPs) which is unsafe. If we had this number auto-update for compute growth, eventually it would go above the unsafe number, and unsafe models would be exempt. But at some point we’ll probably discover that some new models (eg 10^28 FLOPs) are safe, and it would be good if the law was updated to exempt them too. Very optimistically, this might happen - California’s minimum wage was originally $0.15 per hour, but this got updated when inflation made that unreasonable. In the pessimistic case, this will be a problem for us thirty years from now, if we’re even around then. Other people note that an AI committing a cyberattack is a fuzzy bar. If you ask GPT-4 to write a well-composed, grammatically-correct phishing email (“Dear sir, I am the password inspector, please tell me your password”), the phishing works, and you use the password to blow up a power plant, does that count? I agree that it would be nice if the law were clearer on this. But I also agree with the lawyers who object that dealing with programmers is impossible and that laws will never be exactly as clear as code. Other people note that this will *eventually* make open source impossible. Someday AIs really will be able to make nukes or pull off $500 million hacks. At that point, companies will have to certify that their model has been trained not to do this, and that it will stay trained. But if it were open-source, then anyone could easily untrain it. So after models become capable of making nukes or super-Ebola, companies won’t be able to open-source them anymore without some as-yet-undiscovered technology to prevent end users from using these capabilities. Sounds . . . good? I don’t know if even the most committed anti-AI-safetyist wants a provably-super-dangerous model out in the wild. Still, what happens after that? No cutting-edge open-source AIs ever again? I don’t know. In whatever future year foundation models can make nukes and hack the power grid, maybe the CIA will have better AIs capable of preventing nuclear terrorism, and the power company will have better AIs capable of protecting their grid. The law seems to leave open the possibility that in this situation, the AIs wouldn’t technically be capable of doing these things, and could be open-sourced. (or you could base your Build-A-Nuke-Kwik AI company in some state other than California.) Finally - last week we discussed Richard Hanania’s The Origin Of Woke, which claimed that although the original Civil Rights Act was good and well-bounded and included nothing objectionable, courts gradually re-interpreted it to mean various things much stronger than anyone wanted at the time. This bill tells the Department of Technology to offer guidance on what kind of tests AI companies should use. I assume their first guidance will be “the kind of safety testing that all companies except Meta are currently doing” or “something like METR”, because those are good tests, and the same AI safety people who helped write those tests probably also helped write this bill. But Hanania’s book, and the process of reading this bill, highlight how vague and complicated all laws can be. The same bill could be excellent or terrible, depending on whether it’s interpreted effectively by well-intentioned people, or poorly by idiots. That’s true here too. The best I can say against this objection is that this bill seems better-written than most. Many of the objections to its provisions seem to not understand how law works in general (cf. the perjury section) - the things they attack as impossible or insane or incomprehensibly vague are much easier and clearer than their counterparts in (let’s say) medicine or aerospace. Future AIs stronger than GPT-4 seem like the sorts of things which - like bad medicines or defective airplanes - could potentially cause damage. This sort of weak, carefully-directed regulation that exempts most models and carves out a space for open-sourcing seems like a good compromise between basic safety and protecting innovation. I join people like Yoshua Bengio and Geoffrey Hinton in supporting it. Regardless of your position, I urge you to pay attention to the conversation and especially to read Zvi’s Asterisk article or his longer FAQ on his blog. I think Zvi provides pretty good evidence that many people are just outright lying about - or at least heavily misrepresenting - the contents of the bill, in a way that you can easily confirm by reading the bill itself. There will be many more fights over AI, and some of them will be technical and complicated. Best to figure out who’s honest now, when it’s trivial to check! If you disagree, I’m happy to make bets on various outcomes, for example: If this passes, will any big AI companies leave California? (I think no)
Inline links: 3, 4, 5, Anthropic’s, Google’s,, OpenAI’s, 6, here, The Origin Of Woke, read Zvi’s, his longer FAQ on his blog, reading the bill itself
“Full shutdown means the cessation of operation of a covered model, including all copies and derivative models, on all computers and storage devices within custody, control, or possession of a person”, where “person” is elsewhere defined to mean corporation. I agree this is a little ambiguous, but Asterisk talked to the state senator’s office and they confirmed that they meant the less-restrictive, pro-open-source meaning.
Highlights From The Comments On Prison
December 10, 2024 · Original source
Yeah, this is an interesting point. There’s a good interview with Skarbek in last month’s issue of Asterisk, I think it will be online shortly.
Misophonia: Beyond Sensory Sensitivity
March 19, 2025 · Original source
Jake Eaton has a great article on misophonia in Asterisk.
Inline links: a great article on misophonia
"Deros And The Ur-Abduction" In Asterisk
March 26, 2025 · Original source
Asterisk invited me to participate in their “Weird” themed issue, so I wrote five thousand words on evil Atlantean cave dwarves.
Inline links: Asterisk
Am I doing the thing where I cherry-pick a bunch of myths from unrelated cultures, squint at them really hard until they all look the same, and declare myself to have discovered something fundamental about the depths of the collective unconscious? Read the article and find out!
Inline links: Read the article
Links For September 2025
September 04, 2025 · Original source
60: Asterisk - Africa Needs A YIMBY Movement. I was surprised by the title, because I always hear that African cities are growing very rapidly. But the article makes its case well: African cities have dysfunctional planning, relegating most of the growth to either the “informal sector” (ie thrown-together slums that could be banned at any moment) or rural land on the outskirts of existing cities. “In Ghana, for example, acquiring a building permit can take 170 days — and in practice, developers say it often takes four to five years. Unsurprisingly, 76% of development in Ghana is informal.”
Inline links: Africa Needs A YIMBY Movement
Links For October 2025
October 30, 2025 · Original source
38: Eliezer and Nate’s book If Anyone Builds It, Everyone Dies is now out and is an NYT bestseller. Authors’ Atlantic article here (paywalled). Online resources/FAQ/answers to objections here. My review here. Peter Wildeford’s review here. Mostly negative Asterisk review here, criticisms/arguments about the Asterisk review here, Eliezer’s response to this line of criticism here (X). I thought all the reviews, positive and negative, had something useful to say - except the NYT review, which was remarkably bad (Steven Adler points out that it accuses the book of failing to define the term “superintelligence”, but it very explicitly does that on page 4). I read Literary Substack sometimes, and I am so confused - it seems like there’s this entire ecosystem of Ivy graduates who spend years backstabbing each other in order to win the one bigshot publication book reviewer slot, and then the 1/1000 who reach this exalted position phone it in and don’t even read the books they’re reviewing.
Inline links: If Anyone Builds It, Everyone Dies, an NYT bestseller, here, here, here, here, here, here, here (X), was remarkably bad
48: In the ongoing survey of AI progress I wrote about here, two tasks kept confounding forecasters: no matter how good AI gets at writing, math, chess, Go, or any other hard thing, it still can’t play Angry Birds or fold laundry. Year after year, forecasters predict that they can’t know exactly how AI will progress, but they are sure it will solve laundry folding before it solves protein folding. Year after year, they are wrong. Now one team claims that the laundry barrier has finally fallen.
Inline links: here, the laundry barrier has finally fallen
Links For December 2025
December 10, 2025 · Original source
46: The death toll of the ongoing Sudan genocide has risen to about 150,000. Nicholas Kristof writes that the world has once again failed to prevent atrocities, and argues that the most important point of leverage is pressure on the United Arab Emirates, which is arming the genociders. Sam Kriss also writes about the situation in The World’s First Matcha Labubu Genocide, but is unimpressed with Kristof’s take: Sudan is passed over in a deeply uncomfortable silence. The absolute most you can do is blame the Emiratis. From what I’ve seen, more people seem to be appalled at the UAE for its frankly marginal role in arming the RSF than at the RSF itself. This is the approved way of understanding any inscrutably indigenous foreign conflict: you just worm out any third-party involvement and then act like you’ve solved the whole thing. I side with Kristof here, for reasons that Sam himself touches on later in his piece, in a section comparing Darfur with Gaza. It would be very easy to make people care about Darfur again. All it would take is a loud, vocal contingent of RSF apologists in the Western media. I agree, but would frame it less cynically: the reason Westerners pay attention to Gaza is that there’s a lever to push: not only does America support Israel, but many of their friends support Israel, so they can imagine convincing America or at least their friends to stop, and at least feel like there is some remote chance of making a small difference (and in fact, Trump getting mad at Israel and deciding to pressure them was decisive in effecting the cease-fire). On the other hand, we don’t have many levers to affect ethnic Baggara in the Rapid Support Forces of Sudan, so it doesn’t really feel useful to write blog posts arguing that they should stop; obviously they should stop, nobody disagrees with this, and it goes without saying - so nobody says it. But the US does support the UAE, and many of our friends like the UAE or at least go there on vacation, so maybe it’s possible to have make some small difference by embarrassing them. 4D chess take is that Sam Kriss agrees with all of this, but “loudly” and “vocally” argued against it to give people like me a hook to write about this genocide with, in which case I thank him for his sacrifice. It would also be nice to be able to donate, but I don’t know who to trust in the region - other than Doctors Without Borders, who are usually pretty good. 47: The AI Futures Project (group of AI-will-be-fast intellectuals) and the AI As A Normal Technology team (group of AI-will-be-slow intellectuals) wrote an adversarial collaboration in Asterisk explaining what they agree on, for example: That there’s an important distinction between existing AI and “strong AGI”
Inline links: Nicholas Kristof writes, The World’s First Matcha Labubu Genocide, and in fact, Trump getting mad at Israel and deciding to pressure them was decisive in effecting the cease-fire, Doctors Without Borders, an adversarial collaboration in Asterisk explaining what they agree on
They started with one village in Malawi (2022), moved up a subdistrict (2023), and are now starting a district-wide experiment; if it goes well, they’ll scale up to the entire country of Malawi (!) in 2027. Preliminary results are positive, with the charity claiming they effectively doubled the economy of their chosen subdistrict (population 85,000) without causing inflation (how can this be?) Related: Asterisk panel with Kelsey Piper on the future of UBI and AI.
Inline links: Asterisk panel with Kelsey Piper on the future of UBI and AI