Astral Codex Ten

Meditations on Moloch

18 min read

Meditations on Moloch

Article

Meditations on Moloch is a recurring publication in the Astral Codex Ten archive, appearing 3 times across 3 issues between August 06, 2021 and July 26, 2025. The archive places it in contexts such as “See eg Meditations on Moloch”; “your “Meditations on Moloch” essay”; “a bunch of people became regular readers of the blog because they read Meditations on Moloch”. It most often appears alongside Deiseach, Scott, 4chan.

Metadata

  • Category: Publications
  • Mention count: 3
  • Issue count: 3
  • First seen: August 06, 2021
  • Last seen: July 26, 2025

Appears In

Source Context

Recovered passages from the original issue text. When the raw archive preserved outbound links inside the source passage, they are listed directly under the quote.

Highlights From The Comments On Acemoglu And AI
August 06, 2021 · Original source
My personal estimates are more like 75% chance, 25% chance, and a distribution that peaks about 20 years later than this one. I think the Metaculus position is consistent with all of “this probably won’t happen”, “THIS IS SUPER-TERRIFYING”, “this is most likely far away”, and “BUT FOR ALL WE KNOW IT COULD BE TOMORROW!” I realize this is an annoying way for things to be. ————————————————— CraigMichael writes: >But all the AI regulation in the world won’t help us unless we humans resist the urge to spread misinformation to maximize clicks. Was with you up to this point. There are several solutions to this other than willpower (resisting the urge). The basic idea - change incentives so that while spreading misinformation is possible but substantially less desirable/lucrative than other options for online behaviors. This isn’t so hard to imagine. Say there’s a lot of incentives to earn money online doing creative or useful things. Like Mechanical Turk, but less route behavior and more performing a service or matching needs. Like I wish I had a help desk for English questions where the answers were good and not people posturing to look good to other people on the English Stack Exchange, for example. I would pay them per call or per minute or whatever. Totally unexplored market AFAIK because technology hasn’t been developed yet. Another idea - Give people more options to pay at an article-level for information that’s useful to them or to have related questions answered or something like that without needing a subscription or a bundle. Say there’s some article about anything and I want to contact the author and be like “hey, here’s a related question, I’m willing to offer you X dollars to answer.” The person says “I’ll do it for x+10 dollars.” One site used to unlock articles to the public after a threshold of Bitcoin have been donated on a PPV basis. It both incentives the author and had a positive externality. Everyone is so invested in ads that they don’t work on technology and ideas to create new markets. To paraphrase Jaron Lanier we need to make technology so good it seduces away from destroying ourselves. Partly I want to complain that obviously I was using the quoted sentence as a rhetorical device. But I guess the whole point of that sentence and its paragraph was to argue against saying false things as a rhetorical device, so - hoist on my own petard, I guess. I’m less optimistic than Craig is about this solution, because it seems to me that socially virtuous technology will always be less fun/addictive than nonvirtuous technology, simply because the virtuous technology has to hit two targets (virtuous, fun/addictive), the nonvirtuous technology only has to hit one target, and it’s easier to optimize for a target with zero other constraints than with one other constraint. See eg Meditations on Moloch. ————————————————— Souf asks: Is there a convincing argument that AGI is possible within any reasonable timeframe (like... 50 years), other than the intuitions of esteemed AI researchers? Do they have any way to back up their estimates (of some tens of percent), and why they shouldn't be millionths of a percent? It is, as another poster said, an "extraordinary claim." I'd like to see some extraordinary support of those particular numbers. If I had to answer this question, I would point to the sorts of work AI Impacts does, where they try to estimate how capable computers were in 1980, 1990, etc, draw a line to represent the speed at which computers are becoming more capable, figure out where humans are at the same metric, and check the time when that line crosses however capable you’ve decided humans are. This is obviously really hard because you have to operationalize some definition of “capable” or “intelligent” or some other word that is hard to operationalize, but when you do it you usually get sometime in the mid-21st century. You’re going to point out that this argument doesn’t really qualify as “convincing”. I admit it doesn’t meet trial-by-jury standards of evidence. So I guess my real answer would be “it’s the #$@&ing prior”. Like, you certainly don’t have knock-down evidence that it’s impossible, I don’t have a knock-down evidence that it’s certain, so it might happen and it might not. How “might” are we talking? I don’t know, it would seem weird if this quickly-advancing technology being researched by incredibly smart people with billions of dollars in research funding from lots of megacorporations just reached some point and then stopped. Okay, fine, maybe it will keep advancing at the same rate, how fast is that in terms of time-to-AGI? Now we’re back at AI Impacts drawing lines again. The stupidest possible prior is always 50-50. We would have to be very stupid people to use the stupidest possible prior. But here we are. I wouldn’t want to give a 50-50 chance of us inventing FTL travel by 2100, because FTL travel seems physically impossible. I wouldn’t want to give a 50-50 chance of us inventing slower-than-light-but-still-pretty-good starships by 2100, because, I dunno, space travel isn’t advancing that fast and nobody is really working on it that hard. For AI, I don’t know, I kinda want to say 50-50. If I were going to try to update away from 50-50, I would want to look at AI Impacts style line graphs, expert opinion, and prediction markets. All of those seem to make me update up instead of down, so I don’t think I would go lower than 50-50. But there’s enough Knightian uncertainty to make an entire Round Table here, so who knows? Hardly a “convincing” argument, but I’m just trying to avoid the McAfee Fallacy: ————————————————— Souf continues: The argument that we are "in the middle of a period of extremely rapid progress in AI research, when barrier after barrier is being breached" makes it seem like all AI "progress" is on some sort of line that ends in AGI. That feels like sleight-of-hand. Even Scott himself refers to AGI here as a "new class of actor," so I'm failing to see how current lines of "progress" will indubitably result the emergence of something completely novel and different? Lots of smart people disagree with me on this one, but I think the path from here to AGI is pretty straight. I mean, it will take thousands of people who are all much smarter than I am to do it, but it’ll happen. My argument is something like - human brains are remarkably similar to rat brains, only much bigger. They’re still a little similar to insect brains. It looks like if you have a basic functioning brain, and you scale it up, it gets human intelligence. Existing AIs like AlphaGo or GPT seem to be basically a blob of learning-ability, a plan for pointing the blob at a specific problem, and lots and lots of training data. I think the past five years have shown that this basic model generalizes really well. OpenAI’s programs can now write essays, compose music, and generate pictures, not because they had three parallel amazing teams working on writing/music/art AIs, but because they took a blob of learning ability and figured out how to direct it at writing/music/art, and they were able to get giant digital corpuses of text / music / pictures to train it. DeepMind is finding that it can win lots of games, from Go to StarCraft to obstacle courses in simulated environments, by pointing a blob of learning-ability at the game and making it play against itself a zillion times (ie generate its own training data). My impression is that human/rat/insect brains are a blob of learning-ability which the rest of the nervous system successfully points at the world, and especially at aspects of the world that the organism needs to pay attention to (eg food sources, sex, etc). This isn’t exactly right, there are a few genetically-encoded programs, but not that many and it’s pretty hard. Right now I think our main advantages over AI systems are something like: our nervous system is pretty good at pointing us at the world and extracting training data from it. If you wanted an AI that learned being-in-the-world skills as well as we do, it would have to have an amazing robot body, and right now robot bodies aren’t that amazing.
Inline links: writes, Meditations on Moloch, Souf, the sorts of work AI Impacts does, https://substackcdn.com/image/fetch/$s_!3MgL!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2F7db78f49-9ccb-4b6e-ac18-cfb79f52cb04_584x232.png, not that many and it’s pretty hard
Highlights From The Comments On POSIWID
April 15, 2025 · Original source
Biden supports Xi’s campaign to prevent Chinese people from getting addicted to video games, and wants to keep video-game-enabling GPUs out of the country. …and design experiments to distinguish between these, or wait for more chip sanctions to see how they pan out. But in real life, we can be very sure some of these (like 2 and 7) weren’t intended, and others (like 4) were. Why? Some combination of trusting Biden’s stated goals, psychoanalyzing Biden’s plausible goals, checking who lobbied Biden to do this, and reading enough international relations journals to get a sense of what policymakers are thinking about. I think it’s fine to do black box systems analysis, just like it’s fine to do behaviorism. But we should view these as methodological commitments for a specific group, rather than good strategies for normal people. Jared Peterson (blog) writes: This originally struck me as rather silly and as an obvious misinterpretation of an idea that has nothing to do with human intentions...then I read the comments and saw many people claiming exactly that! Donella Meadows is an important figure in the field of Systems Thinking, and says by definition (whether human designed or not), systems have a purpose. "A system’s function or purpose is not necessarily spoken, written, or expressed explicitly, except through the operation of the system. The best way to deduce the system’s purpose is to watch for a while to see how the system behaves. Purposes are deduced from behavior, not from rhetoric or stated goals” One way to think about this is that Meadows would be OK talking about Molochs purpose as something coherent. Is changing the climate the purpose of modern capitalism? In one sense, no. But simultaneously, it is perfectly coherent to talk about the system as having that exact purpose because the system seems to work towards that goal. Even if you push against the system, the system seems to adapt and continue with that goal anyways. There is something almost intelligent about systems where they seem to work towards goals that no one ever intended. But the phrase isn't about human goals at all! Oh! I agree this makes sense if you need to talk about the “purpose” of an un-designed system with no humans in it. Moonshadow writes: This sentiment is grasping towards the same sort of place as your "Meditations on Moloch" essay. No-one involved in the system wants what the system actually ends up doing. But whatever their individual intents, /the system as a whole/, if allowed to grow naturally, inevitably ends up doing what Moloch wants. Of course the purpose we intended for the system isn't really that, any more than Moloch really exists. But you can't begin the meta level fight - of designing the system's high level organisational structures and incentives to try to reduce this effect, instead of letting it emerge organically like it always does - unless you first admit the problem. I agree this is a useful thing to talk about, I just don’t think “purpose” is the right word for it. I’m not even sure “system” is the right word for it. A good example of Moloch would be two countries having a nuclear arms race. But how is this POSIWID? The purpose of the . . . system of two countries . . . is to . . . have a nuclear arms race? This is pretty different from how I usually hear it used. But here is a dissenting voice. Ajb writes: POSIWID was not originally an antagonistic political snark. It's perfectly sensible to notice that a system may be fulfilling other purposes than it does officially, and this is not incompatible with it operating in good faith. You can think of it as a bit like Chesterton's fence: * to reform a system you should understand what purposes it fulfills, not just what it is officially supposed to do * These additonal or alternative purposes may in fact be desirable ones that you should avoid breaking. Cybernetics (where the phrase originated) drew a lot of inspiration from biology, and there obviously nothing has an 'official purpose' at all. But it nevertheless has organisation and is functional. Rob writes: The problem with quoting aphorisms like this is that it misses the context - specifically the context of a management consultant (viz. Stafford Beer) who spends his entire life being told about systems his clients have put in place, with some stated purpose in mind. Those systems do not achieve their stated purposes, but can be continually defended against change by re-stating the purpose - this shouldn't work, but in practice it often does, because most people aren't great at decoupling intent from outcome. "The purpose of a system is what it does" is a good rhetorical counter, because it acknowledges that, in practice, any continuation of a system with known outcomes is a tacit acceptance of those outcomes as the system's real purpose. You don't get to claim some other "real" purpose once you know what the outcomes are. My interpretation has always been in the spirit of this tweet: https://x.com/primawesome/status/1178671690261286918?lang=en > My neighbor told me coyotes keep eating his outdoor cats so I asked how many cats he has and he said he just goes to the shelter and gets a new cat afterwards so I said it sounds like he’s just feeding shelter cats to coyotes and then his daughter started crying. I agree this makes more sense in the context of some supposed person claiming that “the system has good intentions” means they should never have to change the system. I don’t think I really see this failure mode. I bet a lot of you are going to yell at me and say that, I don’t know, homelessness or something is like this. But defenders of the current homelessness system never say you can’t change it because it had good intentions when it started. I predict they would say that their own group is doing good work, and it’s everyone else who needs to change. Or that the current system works a little and just needs to be funded more. Or that the current system is better than nothing, and your proposed attempt to “change” it is secretly a plan to gut it and leave homeless people without help. I definitely don’t think they’d say “Yes, your proposed change would improve the system, but you’re not allowed to make it because the people who designed the current system had good intentions”. Leah Libresco Sargeant writes: I think the Catholic principle of double effect is helpful here. This often comes up in the case of eg delivering a baby pre-viability because the mom has an infection that will progress to sepsis and death if she and the baby aren’t separated. The three criteria are: the nature of the act is itself good, or at least morally neutral;
Inline links: blog, writes, https://substackcdn.com/image/fetch/$s_!Dy01!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6929f8eb-51df-403a-8666-268f9f1f034f_529x185.png, writes, writes, https://x.com/primawesome/status/1178671690261286918?lang=en, writes
Your Review: The Astral Codex Ten Commentariat (“Why Do We Suck?”)
July 26, 2025 · Original source
However, this pie chart only considers ACX vs SSC, not pre-2016 SSC vs post-2016-SSC. It is therefore still maybe consistent with Scott’s writing getting worse in April 2016 and never recovering. This could straightforwardly explain the drop in Commentariat quality in 2016 (but not 2021), but the evidence for a decline in writing quality centred on this period is anyway very mixed. April 2016 has some great posts (including the ‘classic’ The Ideology is Not the Movement), but there were a lot of good posts around that time - the very start of May 2016 includes another ‘classic’ in the form of Be Nice, At Least Until you can Coordinate Meanness. Nor can it be that readers somehow intuit that Scott has nothing more valuable to say on any topic going forward, because 2017 contains classics like Guided by the Beauty of our Weapons, or my personal favourite SSC-era post, Considerations on Cost Disease. Not to mention, of course, there are some cracking ACX-era posts which are nearly a decade away at this point. In my head, the cleanest story is that a bunch of people became regular readers of the blog because they read Meditations on Moloch or another of the universally-loved posts that were linked everywhere and then left when they realised the median post was ‘merely’ as good as The Ideology is Not the Movement, but this story doesn’t make sense – you could certainly argue the toss about when ‘peak’ SSC was, but if you believe it exists you’d surely have to put it centred somewhere around 2014. This would mean that the group of people who are disappointed by Scott’s output would have to get interested in the blog in 2014, stick around through the whole of 2015, and then leave en masse in April 2016 despite 2016 (in my subjective opinion) being better than 2015 for ‘important’ posts. Another point to consider is that the ‘Scott’s writing sucks now’ hypothesis needs not only to explain why engagement fell off in 2016, but also why multisyllabic words and type/token ratio also peaked around that time. I think you can maybe tell a story where Scott’s writing gets worse in 2016 so people engage less with the comments (producing less comment depth and more zero-length comment chains) but it is very difficult to imagine how Scott’s writing getting worse produces more multisyllabic words. If Scott’s writing drives the disengagement, you have to start loading up the ‘evaporative cooling’ hypothesis with a lot of weird epicycles in order for everything to all make sense at once. In summary, I’m agnostic on the question of whether Scott’s writing has got worse. I personally don’t think it has (although the frequency of ‘hits’ was remarkable in 2014) but perhaps it has changed a bit over time. However, I’m reasonably certain that nothing Scott writes is the reason for the dropoff in engagement around 2016, because there’s no coherent story you can tell that fits that hypothesis. I think this is an unproductive sidetrack to consider in a review of the Commentariat specifically. The user experience of the blog got worse
Inline links: The Ideology is Not the Movement, Be Nice, At Least Until you can Coordinate Meanness, Guided by the Beauty of our Weapons, Considerations on Cost Disease, Meditations on Moloch