Astral Codex Ten

LessWrong

43 min read

LessWrong

Article

LessWrong is a recurring publication in the Astral Codex Ten archive, appearing 23 times across 23 issues between March 21, 2021 and April 01, 2026. The archive places it in contexts such as “Steve2152 on LessWrong reminds me that acetylcholine levels”; “Read https://www.lesswrong.com/posts/WAkvnzxvNfeTJL4BT/funds-are-available-to-support-lesswrong-groups-among-others”; “posts on LessWrong/The Alignment Forum”. It most often appears alongside ACX, Scott, Daniel.

Metadata

  • Category: Publications
  • Mention count: 23
  • Issue count: 23
  • First seen: March 21, 2021
  • Last seen: April 01, 2026

Appears In

Source Context

Recovered passages from the original issue text. When the raw archive preserved outbound links inside the source passage, they are listed directly under the quote.

Open Thread 164
March 21, 2021 · Original source
5: In response to recent posts on depression, several people asked if there was a parameter of the brain corresponding to learning rate. Steve2152 on LessWrong reminds me that acetylcholine levels sort of correspond to this - see also my previous post on this subject. Although everything is implicated in depression in one way or another, acetylcholine is implicated less than most other things and doesn’t show a lot of promise as a target for depression treatments - so either we’re missing something, or all of you who thought this was crucial are wrong.
Inline links: acetylcholine levels sort of correspond to this, my previous post on this subject
Open Thread 182
July 25, 2021 · Original source
2: My friends Claire and Buck are trying to distribute money earmarked for rationalist/EA "outreach", and they've decided that the ACX community is close enough to count. If you're doing any ACX community work - for example, running a local meetup group, a podcast, a mailing list, etc - and you think you could do a better job with more money, consider getting in touch with them. It doesn't have to be ACX-branded in particular, it can also just be about related topics. Read https://www.lesswrong.com/posts/WAkvnzxvNfeTJL4BT/funds-are-available-to-support-lesswrong-groups-among-others for more information.
Inline links: https://www.lesswrong.com/posts/WAkvnzxvNfeTJL4BT/funds-are-available-to-support-lesswrong-groups-among-others
ACX Grants ++: The Second Half
February 10, 2022 · Original source
#74: Apply Constructor Theory To AI Constructor theory is a framework developed by the physicist David Deutsch which seeks to express scientific theories as claims about which physical transformations are possible and which are impossible. This is in contrast to the standard framework which describes physical systems in terms of their initial conditions and laws of evolution. It is hoped that this framework will solve fundamental problems in physics and other fields. I believe that there is an analogy between the problems in the natural sciences which constructor theory was developed to solve and the AI alignment problem. I would like to spend a couple of months thinking about this and fleshing out my ideas as posts on LessWrong/The Alignment Forum and opening them up for discussion. I am currently in the final few months of a PhD in theoretical physics during which I have published two papers. After my PhD finishes, I would like to spend some time (two or three months) researching this problem and will need some funding to do this full time during this period. If you would like to fund this work or discuss the idea further, please send an email to AlfredSPH@protonmail.com .
#120: Tool To Develop Arguments In Parallel I've been working on a tool that facilitates an argument where two competing theories are developed in parallel in an iterative manner. The goal of the process is: (1) to produce a pair of coherent arguments that stand on their own instead of a long chain of correspondence which can be difficult to follow; (2) to ensure that all relevant counterarguments are addressed, or in the case their not, to make it easy for the reader to notice this; (3) to provide the debaters an opponent to spar with from the start which should result in sounder arguments; and finally (4) to be more feasible than adversarial collaboration since the elusive goal of converging views need not be met. I can't seek funding via Grants++ for legal reasons. But if you're otherwise interested, check out the GitHub repo (tinyurl.com/2p8w4jbe) or the LessWrong post (tinyurl.com/2s3z7ct8) and feel free to contact me (mat5n@outlook.com).
Highlights From The Comments On Motivated Reasoning And Reinforcement Learning
February 11, 2022 · Original source
One of 5 or so places in the brain that can get a dopamine burst when a bad thing happens (opposite of the usual) is closely tied to inferotemporal cortex (IT). I talked about it in "Example 2C" here - https://www.lesswrong.com/posts/jrewt3rLFiKWrKuyZ/big-picture-of-phasic-dopamine#Example_2C__Visual_attention Basically, as far as I can tell, IT is "making decisions" about what to attend to within the visual scene, and it's being rewarded NOT for "things are going well in life", but rather for "something scary or exciting is happening". So from IT's own narrow perspective, noticing the lion is very rewarding. (Amusingly, "noticing a lion" was the example in my blog post too!)
Inline links: https://www.lesswrong.com/posts/jrewt3rLFiKWrKuyZ/big-picture-of-phasic-dopamine#Example_2C__Visual_attention
You should penalize theories really heavily for every piece of information that has to travel from the genome to the brain. It certainly should be true that people try to spin things in self-serving ways: this is Trivers’ theory of self-deception and consciousness as public relations agent. But that requires communicating an entire new philosophy of information processing from genome to brain. Unless you could do it with reinforcement learning, which you’ve already got.
Inline links: Trivers’ theory of self-deception and consciousness as public relations agent
Spring Meetups In Seventy Cities
April 10, 2022 · Original source
BANGALORE, INDIA Contact: Faiz (faiz_abbas@protonmail.com) Date: April 24 Time: 4:00 PM Coordinates: https://plus.codes/7J4VXJF4+PR Location: Matteo Coffea, Church Street, near MG road Group info: Bangalore SSC has been meeting monthly since 2018
Inline links: https://plus.codes/7J4VXJF4+PR, Bangalore SSC
JAKARTA, INDONESIA Contact: Jati (indonesiarationalist@gmail.com) Date: May 8 Time: 3:30 PM Coordinates: https://plus.codes/6P58RR8G+J4Q Location: Kawisari Cafe & Eatery in Menteng, Central Jakarta. The nearest train station is Gondangdia (15 minutes walk or just take an online moto-taxi). Feel free to bring whatever you think could be fun or exciting! The organizer will be there from 15.00 WIB. Notes: Please RSVP on LessWrong or send an E-mail to the above address. Group info: Jakarta has a rationality-adjacent group that meets occasionally, so some members of that group will come to this ACX meetup.
Inline links: https://plus.codes/6P58RR8G+J4Q
ATHENS, GREECE Contact: Elias (minus42cgn@gmail.com) Date: April 27 Time: 7:00 PM Coordinates: https://plus.codes/8G95WMRV+Q4F Location: Stavros Niarchos Park, Great Lawn, 37.941753, 23.692632 Notes: Please RSVP on LessWrong
Inline links: https://plus.codes/8G95WMRV+Q4F, RSVP on LessWrong
Meetups Everywhere 2022: Times & Places
August 26, 2022 · Original source
You can see a map of all the events on the LessWrong community page.
Inline links: the LessWrong community page
Extra Info For Meetup Organizers: 1. If you’re the host, bring a sign that says “ACX MEETUP” and prop it up somewhere (or otherwise be identifiable). 2. Bring blank labels and pens for nametags. 3. Have people type their name and email address in a spreadsheet or in a Google Form (accessed via a bit.ly link or QR code), so you can start a mailing list to make organizing future meetups easier. 4. If it’s the first meetup, people are probably just going to want to talk, and if you try to organize some kind of “fun” “event” it’ll probably just be annoying. 5. It’s easier to schedule a followup meetup while you’re having the first, compared to trying to do it later on by email. 6. In case people want to get to know each other better outside the meetup, you might want to mention reciprocity.io, the rationalist friend-finder/dating site. 7. If you didn’t make a LessWrong event for your meetup, the LessWrong team did it for you using the email address you gave here. To claim your event, log into LW (or create an account) using that email address, or message the LW team on Intercom (chat button in the bottom right corner of lesswrong.com).
Inline links: reciprocity.io
BAGHDAD, IRAQ Contact: MA, tofiahmed117[at]gmail[dot]com, Discord: WolframSigma#1532, Telegram Time: Friday, September 2, 11:00 AM Location: Grinders Coffeeshop Coordinates: 8H568FG6+73 Event link(s): LessWrong JERUSALEM, ISRAEL Contact: Zvi Schreiber, zvi[at]zvi[dot]net, WhatsApp +972 54 569 1100 Time: Wednesday, October 19, 6:00 PM Location: Malcha technology park garden Coordinates: 8G3QP5XP+PP Event link(s): LessWrong REHOVOT, ISRAEL Contact: David Manheim, David[at]alter[dot]org[dot]il Time: Sunday, September 11, 8:00 PM Location: Outside porch of Aroma Coffee, הרצל 218, רחובות Coordinates: 8G3PWR25+MP Event link(s): LessWrong, Facebook event Notes: Please RSVP on Facebook so we can give updates if needed TEL AVIV, ISRAEL Contact: Adam & inbar M, projectscentrum[at]gmail[dot]com, inbar192[at]gmail[dot]com, Whatsapp +46762791415 (Adam) Time: Sunday, September 4, 7:00 PM Location: Hamenia industrial loft at Beit Alfa 7 (רחוב בית אלפא 7). Look for a door with ACX sign. Two floors up. Coordinates: 8G4P3Q8Q+85 Event link(s): LessWrong Group info: We've just made a Facebook group and are planning to organize monthly meetings going forward Notes: For questions contact Adam on email or WhatsApp. Feel free to bring a snack or a bottle of white wine. AMMAN, JORDAN Contact: Daniel, dnledvs[at]gmail[dot]com Time: Tuesday, September 20, 6:30 PM Location: Rustic, Jabal al Weibdeh Coordinates: 8G3QXW49+WG Event link(s): LessWrong Notes: We're hoping to grow the group, so feel free to come even if you've only read a few posts! +1s are also welcome. CAPE TOWN, SOUTH AFRICA Contact: Mark Chimes, chimes[dot]mark[at]gmail[dot]com, WhatsApp 0826568573 Time: Saturday, September 17, 11:00 AM Location: Truth Coffee Roasting, 36 Buitenkant St, Cape Town City Centre - we'll put a sign on the table Coordinates: 4FRW3CFF+3M Event link(s): LessWrong Group info: We met up pre-Covid and pre-ACX as an SSC group. Now we're getting back in the swing of things. We eat lunch and chat about philosophy, politics, and sometimes SSC/ACX blog posts. Notes: We're planning on having another meetup on the 8th October if you can't make the first. DAR ES SALAAM, TANZANIA Contact: Arno, arnorohwedder[at]gmail[dot]com, +255763998637 Time: Thursday, September 29, 7:30 PM Location: The Deck, Masaki Coordinates: 6G5X776J+X6 Event link(s): LessWrong Notes: Seeing if there are any interested people in Dar, look forward to meeting, if you are coming please send me a whatsapp. DUBAI, UAE Contact: RS, xyxyxz[at]gmail[dot]com, +971552726281 (WhatsApp) Time: Friday, September 30, 7:30 PM Location: Starbucks, Garhoud Coordinates: 7HQQ68VR+94 Event link(s): LessWrong Group info: Met once before Notes: Please RSVP on LessWrong, or message me on WhatsApp
Inline links: Telegram, 8H568FG6+73, LessWrong, 8G3QP5XP+PP, LessWrong, 8G3PWR25+MP, LessWrong, Facebook event, 8G4P3Q8Q+85, LessWrong, Facebook group, 8G3QXW49+WG, LessWrong, 4FRW3CFF+3M, LessWrong, 6G5X776J+X6, LessWrong, 7HQQ68VR+94, LessWrong
MR Tries The Safe Uncertainty Fallacy
March 30, 2023 · Original source
Therefore, it’ll be fine. You’re not missing anything. It’s not supposed to make sense; that’s why it’s a fallacy. For years, people used the Safe Uncertainty Fallacy on AI timelines: Eliezer didn’t realize that at our level, you can just name fallacies. Since 2017, AI has moved faster than most people expected; GPT-4 sort of qualifies as an AGI, the kind of AI most people were saying was decades away. When you have ABSOLUTELY NO IDEA when something will happen, sometimes the answer turns out to be “soon”. Now Tyler Cowen of Marginal Revolution tries his hand at this argument. We have absolutely no idea how AI will go, it’s radically uncertain: No matter how positive or negative the overall calculus of cost and benefit, AI is very likely to overturn most of our apple carts, most of all for the so-called chattering classes. The reality is that no one at the beginning of the printing press had any real idea of the changes it would bring. No one at the beginning of the fossil fuel era had much of an idea of the changes it would bring. No one is good at predicting the longer-term or even medium-term outcomes of these radical technological changes (we can do the short term, albeit imperfectly). No one. Not you, not Eliezer, not Sam Altman, and not your next door neighbor. How well did people predict the final impacts of the printing press? How well did people predict the final impacts of fire? We even have an expression “playing with fire.” Yet it is, on net, a good thing we proceeded with the deployment of fire (“Fire? You can’t do that! Everything will burn! You can kill people with fire! All of them! What if someone yells “fire” in a crowded theater!?”). Therefore, it’ll be fine: I am a bit distressed each time I read an account of a person “arguing himself” or “arguing herself” into existential risk from AI being a major concern. No one can foresee those futures! Once you keep up the arguing, you also are talking yourself into an illusion of predictability. Since it is easier to destroy than create, once you start considering the future in a tabula rasa way, the longer you talk about it, the more pessimistic you will become. It will be harder and harder to see how everything hangs together, whereas the argument that destruction is imminent is easy by comparison. The case for destruction is so much more readily articulable — “boom!” Yet at some point your inner Hayekian (Popperian?) has to take over and pull you away from those concerns. (Especially when you hear a nine-part argument based upon eight new conceptual categories that were first discussed on LessWrong eleven years ago.) Existential risk from AI is indeed a distant possibility, just like every other future you might be trying to imagine. All the possibilities are distant, I cannot stress that enough. The mere fact that AGI risk can be put on a par with those other also distant possibilities simply should not impress you very much. So we should take the plunge. If someone is obsessively arguing about the details of AI technology today, and the arguments on LessWrong from eleven years ago, they won’t see this. Don’t be suckered into taking their bait. Look. It may well be fine. I said before my chance of existential risk from AI is 33%; that means I think there’s a 66% chance it won’t happen. In most futures, we get through okay, and Tyler gently ribs me for being silly. Don’t let him. Even if AI is the best thing that ever happens and never does anything wrong and from this point forward never even shows racial bias or hallucinates another citation ever again, I will stick to my position that the Safe Uncertainty Fallacy is a bad argument. Normally this would be the point where I try to steelman Tyler and explain in more detail why the strongest version of his case is wrong. But I’m having trouble figuring out what the strong version is. Here are three possibilities: 1) The base rate for things killing humanity is very low, so we would need a strong affirmative argument to shift our estimate away from that base rate. Since there’s so much uncertainty, we don’t have strong affirmative arguments, and we should stick with our base rate of “very low”. Suppose astronomers spotted a 100-mile long alien starship approaching Earth. Surely this counts as a radically uncertain situation if anything does; we have absolutely no idea what could happen. Therefore - the alien starship definitely won’t kill us and it’s not worth worrying? Seems wrong. What’s the base rate for alien starships approaching Earth killing humanity? We don’t have a base rate, because we’ve never been in this situation before. What is the base rate for developing above-human-level AI killing humanity? We don’t . . . you get the picture. You can try to fish for something sort of like a base rate: “There have been a hundred major inventions since agriculture, and none of them killed humanity, so the base rate for major inventions killing everyone is about 0%”. But I can counterargue: “There have been about a dozen times a sapient species has created a more intelligent successor species: australopithecus → homo habilis, homo habilis → homo erectus, etc - and in each case, the successor species has wiped out its predecessor. So the base rate for more intelligent successor species killing everyone is about 100%”. The Less Wrongers call this game “reference class tennis”, and insist that the only winning move is not to play. Thinking about this question in terms of base rates is just as hard as thinking of it any other way, and would require arguments for why one base rate is better than another. Tyler hasn’t made any. 2) There are so many different possibilities - let’s say 100! - and dying is only one of them, so there’s only a 1% chance that we’ll die. This is sort of how I interpret: Existential risk from AI is indeed a distant possibility, just like every other future you might be trying to imagine. All the possibilities are distant, I cannot stress that enough. The mere fact that AGI risk can be put on a par with those other also distant possibilities simply should not impress you very much. Alien time again! Here are some possible ways the hundred-mile long starship situation could end: The aliens are peaceful and want to share their advanced technology
Inline links: https://substackcdn.com/image/fetch/$s_!jYyh!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F672bd8f5-484b-48ff-a80f-539d7b77b66c_599x238.png, sort of qualifies as an AGI, tries his hand at this argument
Spring Meetups Everywhere 2023
April 10, 2023 · Original source
There should very shortly be a map of these meetups on the LessWrong community page.
Inline links: the LessWrong community page
Extra Info For Meetup Organizers: 1. If you’re the host, bring a sign that says “ACX MEETUP” and prop it up somewhere (or otherwise be identifiable). 2. Bring blank labels and pens for nametags. 3. Have people type their name and email address in a spreadsheet or in a Google Form (accessed via a bit.ly link or QR code), so you can start a mailing list to make organizing future meetups easier. 4. If it’s the first meetup, people are probably just going to want to talk, and if you try to organize some kind of “fun” “event” it’ll probably just be annoying. 5. It’s easier to schedule a followup meetup while you’re having the first, compared to trying to do it later on by email. 6. In case people want to get to know each other better outside the meetup, you might want to mention reciprocity.io, the rationalist friend-finder/dating site. 7. If you didn’t make a LessWrong event for your meetup, the LessWrong team did it for you using the email address you gave here. To claim your event, log into LW (or create an account) using that email address, or message the LW team on Intercom (chat button in the bottom right corner of lesswrong.com).
Inline links: reciprocity.io
CAPE TOWN, SOUTH AFRICA Contact: Yaseen Mowzer Contact Info: yaseen [at] mowzer [dot] co [dot] za Time: Saturday, 27 May 2023, 11:00 AM. Location: Truth Coffee Roasting, 36 Buitenkant St, Cape Town City Centre, Cape Town, South Africa Coordinates: https://plus.codes/4FRW3CCF+P3 Event Link: https://www.lesswrong.com/events/AAPnyjpNBwtBD6hix/cape-town-south-africa-acx-meetups-everywhere-spring-2023 Notes: Whatsapp: +27 79 813 5144
Inline links: https://plus.codes/4FRW3CCF+P3, https://www.lesswrong.com/events/AAPnyjpNBwtBD6hix/cape-town-south-africa-acx-meetups-everywhere-spring-2023
The Extinction Tournament
July 20, 2023 · Original source
There are centuries’ worth of data on non-genetically-engineered plagues to give us base rates; these give us a base rate of ~25% per century = 20% between now and 2100. But we have better epidemiology and medicine than most of the centuries in our dataset. The experts said 8% chance and the superforecasters said 4% chance, and both of those seem like reasonable interpretations of the historical data to me. The “WHO declares emergency” question is even easier - just look at how often it’s done that in the past and extrapolate forward. Both superforecasters and experts mostly did that. Likewise, lots of scientists have put a lot of work into modeling the climate, there aren’t many surprises there, and everyone basically agreed on the extent of global warming: Wherever there was clear past data, both superforecasters and experts were able to use it correctly and get similar results. It was only when they started talking about things that had never happened before - global nuclear war, bioengineered pandemics, and AI - that they started disagreeing. Were the participants out of their depth? Peter McCluskey, one of the more-AI-concerned superforecasters in the tournament, wrote about his experience on Less Wrong. Quoting liberally: I signed up as a superforecaster. My impression was that I knew as much about AI risk as any of the subject matter experts with whom I interacted (the tournament was divided up so that I was only aware of a small fraction of the 169 participants). I didn't notice anyone with substantial expertise in machine learning. Experts were apparently chosen based on having some sort of respectable publication related to AI, nuclear, climate, or biological catastrophic risks. Those experts were more competent, in one of those fields, than news media pundits or politicians. I.e. they're likely to be more accurate than random guesses. But maybe not by a large margin […] The persuasion seemed to be spread too thinly over 59 questions. In hindsight, I would have preferred to focus on core cruxes, such as when AGI would become dangerous if not aligned, and how suddenly AGI would transition from human levels to superhuman levels. That would have required ignoring the vast majority of those 59 questions during the persuasion stages. But the organizers asked us to focus on at least 15 questions that we were each assigned, and encouraged us to spread our attention to even more of the questions […] Many superforecasters suspected that recent progress in AI was the same kind of hype that led to prior disappointments with AI. I didn't find a way to get them to look closely enough to understand why I disagreed. My main success in that area was with someone who thought there was a big mystery about how an AI could understand causality. I pointed him to Pearl, which led him to imagine that problem might be solvable. But he likely had other similar cruxes which he didn't get around to describing. That left us with large disagreements about whether AI will have a big impact this century. I'm guessing that something like half of that was due to a large disagreement about how powerful AI will be this century. I find it easy to understand how someone who gets their information about AI from news headlines, or from laymen-oriented academic reports, would see a fair steady pattern of AI being overhyped for 75 years, with it always looking like AI was about 30 years in the future. It's unusual for an industry to quickly switch from decades of overstating progress, to underhyping progress. Yet that's what I'm saying has happened. I've been spending enough time on LessWrong that I mostly forgot the existence of smart people who thought recent AI advances were mostly hype. I was unprepared to explain why I thought AI was underhyped in 2022. Today, I can point to evidence that OpenAI is devoting almost as much effort into suppressing abilities (e.g. napalm recipes and privacy violations) as it devotes to making AIs powerful. But in 2022, I had much less evidence that I could reasonably articulate. What I wanted was a way to quantify what fraction of human cognition has been superseded by the most general-purpose AI at any given time. My impression is that that has risen from under 1% a decade ago, to somewhere around 10% in 2022, with a growth rate that looks faster than linear. I've failed so far at translating those impressions into solid evidence. Skeptics pointed to memories of other technologies that had less impact (e.g. on GDP growth) than predicted (the internet). That generates a presumption that the people who predict the biggest effects from a new technology tend to be wrong. > Superforecasters' doubts about AI risk relative to the experts isn't primarily driven by an expectation of another "AI winter" where technical progress slows. ... That said, views on the likelihood of artificial general intelligence (AGI) do seem important: in the postmortem survey, conducted in the months following the tournament, we asked several conditional forecasting questions. The median superforecaster's unconditional forecast of AI-driven extinction by 2100 was 0.38%. When we asked them to forecast again, conditional on AGI coming into existence by 2070, that figure rose to 1%. There was also little or no separation between the groups on the three questions about 2030 performance on AI benchmarks (MATH, Massive Multitask Language Understanding, QuALITY). This suggests that a good deal of the disagreement is over whether measures of progress represent optimization for narrow tasks, versus symptoms of more general intelligence. The “won’t understand causality” and “what if it’s all hype” objections really don’t impress me. Many of the people in this tournament hadn’t really encountered arguments about AI extinction before (potentially including the “AI experts” if they were just eg people who make robot arms or something), and a couple of months of back and forth discussion in the middle of a dozen other questions probably isn’t enough for even a smart person to wrap their brain around the topic. Was this tournament done so long ago that it has been outpaced by recent events? The tournament was conducted in summer 2022. This was before ChatGPT, let alone GPT-4. The conversation around AI noticeably changed pitch after these two releases. Maybe that affected the results? In fact, the participants have already been caught flat-footed on one question: A recent leak suggested that the cost of training GPT-4 was $63 million, which is already higher than the superforecasters’ median estimate of $35 million by 2024 has already been proven incorrect. I don’t know how many petaFLOP-days were involved in GPT-4, but maybe that one is already off also. There was another question on when an AI would pass a Turing Test. The superforecasters guessed 2060, the domain experts 2045. GPT-4 hasn’t quite passed the exact Turing Test described in the study, but it seems very close, so much so that we seem on track to pass it by the 2030s. Once again the experts look better than the superforecasters. So is it possible that we, in 2023, now have so much better insight into AI than the 2022 forecasters that we can throw out their results? We could investigate this by looking at Metaculus, a forecasting site that’s probably comparably advanced to this tournament. They have a question suspiciously similar to XPT’s global catastrophe framing: In summer 2022, the Metaculus estimate was 30%, compared to the XPT superforecasters’ 9% (why the difference? maybe because Metaculus is especially popular with x-risk-pilled rationalists). Since then it’s gone up to 38%. Over the same period, Metaculus estimates of AI catastrophe risk went from 6% to 15%. If the XPT superforecasters’ probabilities rose linearly by the same factor as Metaculus forecasters’, they might be willing to update total global catastrophe risk to 11% and AI catastrophe risk to 5%. But the main thing we’ve updated on since 2022 is that AI might be sooner. But most people in the tournament already agreed we would get AGI by 2100. The main disagreement was over whether it would cause a catastrophe once we got it. You could argue that getting it sooner increases that risk, since we’ll have less time to work on alignment. But I would be surprised if the kind of people saying the risk of AI extinction is 0.4% are thinking about arguments like that. So maybe we shouldn’t expect much change. FRI called back a few XPT forecasters in May 2023 to see if any of them wanted to change their minds, but they mostly didn’t. Overall I don’t think this was just a problem of the incentives being bad or the forecasters being stupid. This is a real, strong disagreement. We may be able to slightly increase their forecast based on recent events, but this would only change the estimate a little. Breaking Down The AI Estimate How did the forecasters arrive at their AI estimate? What were the cruxes between the people who thought AI was very dangerous, and the people who thought it wasn’t? You can think of AI extinction as happening in a series of steps: We get human-level AI by 2100.
Inline links: https://substackcdn.com/image/fetch/$s_!KJ84!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb1f1c4fd-5981-458c-959f-bf9a19ff28da_801x129.png, wrote about his experience, Pearl, https://substackcdn.com/image/fetch/$s_!CfZT!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2362d361-ae0a-4e4f-ad97-cbeb1fcbe827_817x351.png, the cost of training GPT-4 was $63 million, Metaculus, a question, https://substackcdn.com/image/fetch/$s_!k5Ep!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F09cb5518-f8d1-4a98-8c44-97158857dbd8_772x364.png
Your Book Review: The Laws of Trading
July 21, 2023 · Original source
Tales of Icarus flying too close to the sun, where readers revel in schadenfreude, e.g., When Genius Failed. With The Laws of Trading, Agustin Lebron has written something different: part love letter to trading, part philosophical treatise on epistemology and modeling the world around us, and part guide to applied decision-making. Lebron’s Laws are Laws of the Jungle, not Laws of Nature. He views financial markets as the most competitive Darwinian environment on Earth, where participants must adapt or die. According to Lebron, the book is for people working in finance and trading, as well as anyone in the business of making rational decisions. This explicitly rationalist bent is similar to Julia Galef’s The Scout Mindset or Annie Duke’s Thinking in Bets. Where The Laws of Trading sets itself apart is with the best description of financial market dynamics that I’ve ever seen while diving deep into philosophical concepts. Why trust Lebron? He is an engineer, worked as a quantitative trader and researcher at Jane Street, and has a deep understanding of trading. He has what Taleb would describe as skin in the game. You and I may read Astral Codex Ten in our spare time, post on LessWrong, and navel gaze about our epistemic certainty, but at the end of the day most of us are pursuing rationality for fun, as a hobby. Traders like Lebron pursue rationality as a profession: Their livelihood depends on having a better model of the world than their competition. There are lessons to learn from them that apply to our daily lives. 1: Motivation Know why you are doing a trade before you trade. “What is trading about? Fundamentally, it’s about the relationship between you and the rest of the world.” Right now, you’re making a trade. You’re trading your time to read this book review. You have a cost: you could be spending time with your loved ones, exercising, working, sleeping. You might be hoping to learn something, to take away lessons that you can apply to your life, or simply to entertain yourself. Here, off the bat, are two key insights: We are all making trades, all of the time.
Inline links: The Laws of Trading
In Defense Of Describable Dating Preferences
August 16, 2023 · Original source
My take is that 'date me' docs can't work for their stated purpose of pre-filtering for increased compatibility above & beyond the obvious-to-everyone stuff because compatibility can't be predicted empirically using exactly the sort of questions/criteria which fill up these date docs/spreadsheets: https://www.lesswrong.com/posts/6yiayg5QWtWme4JN8/anatomy-of-a-dating-document?commentId=ctD7rHdPpwdSW8jMt Dating appears to be a numbers game, where you want as many as possible to find the 'magic' match or settle for the best you can get eg https://www.thisamericanlife.org/791/transcript
Inline links: https://www.lesswrong.com/posts/6yiayg5QWtWme4JN8/anatomy-of-a-dating-document?commentId=ctD7rHdPpwdSW8jMt, https://www.thisamericanlife.org/791/transcript
Gwern lists some of them here. I won’t go too much into any individual study, except to note that Sparks (2020) is a great name for someone researching the causes of romantic attraction, and Wood & Furr sounds like a children’s cartoon about adorable animals. I’ll separate them, plus some related work, into a few designs:
Inline links: here, Sparks (2020), Wood & Furr
Meetups Everywhere 2023: Times & Places
August 25, 2023 · Original source
You can see a map of all the events on the LessWrong community page. You can also see a searchable sheet at this Airtable link.
Inline links: the LessWrong community page, Airtable link
Extra Info For Meetup Organizers: 1. If you’re the host, bring a sign that says “ACX MEETUP” and prop it up somewhere (or otherwise be identifiable). 2. Bring blank labels and pens for nametags. 3. Have people type their name and email address in a spreadsheet or in a Google Form (accessed via a bit.ly link or QR code), so you can start a mailing list to make organizing future meetups easier. 4. If it’s the first meetup, people are probably just going to want to talk, and if you try to organize some kind of “fun” “event” it’ll probably just be annoying. 5. It’s easier to schedule a followup meetup while you’re having the first, compared to trying to do it later on by email. 6. In case people want to get to know each other better outside the meetup, you might want to mention reciprocity.io, the rationalist friend-finder/dating site. 7. If you didn’t make a LessWrong event for your meetup (or if you did but Skyler didn’t know about it) the LessWrong team did it for you using the email address you gave here. To claim your event, log into LW (or create an account) using that email address, or message the LW team on Intercom (chat button in the bottom right corner of lesswrong.com).
Inline links: reciprocity.io
DUBAI, UAE Contact: RS Contact Info: xyxyxz[at]gmail[dot]com Time: Sunday, September 24th, 7:00 PM Location: Unwind Boardgame Cafe - Zabeel Coordinates: https://plus.codes/7HQQ67MV+HV Notes: Please RSVP on LessWrong or send an email
Inline links: https://plus.codes/7HQQ67MV+HV
Spring Meetups Everywhere 2024 - Call For Organizers
March 08, 2024 · Original source
You give me the information, and on March 29th (or so), I’ll post it on ACX. An event will also be created on LessWrong’s Community page.
Inline links: LessWrong’s Community
Spring Meetups Everywhere 2024
March 30, 2024 · Original source
There should very shortly be a map of these meetups on the LessWrong community page.
Inline links: the LessWrong community page
Extra Info For Meetup Organizers: 1. If you’re the host, bring a sign that says “ACX MEETUP” and prop it up somewhere (or otherwise be identifiable). 2. Bring blank labels and markers for nametags. 3. Have people type their name and email address in a spreadsheet or in a Google Form (accessed via a bit.ly link or QR code), so you can start a mailing list to make organizing future meetups easier. 4. If it’s the first meetup, people are probably just going to want to talk, and if you try to organize some kind of “fun” “event” it’ll probably just be annoying. 5. It’s easier to schedule a followup meetup while you’re having the first, compared to trying to do it later on by email. 6. In case people want to get to know each other better outside the meetup, you might want to mention reciprocity.io, the rationalist friend-finder/dating site. 7. If you didn’t make a LessWrong event for your meetup, the LessWrong team did it for you using the email address you gave here. To claim your event, log into LW (or create an account) using that email address, or message the LW team on Intercom (chat button in the bottom right corner of lesswrong.com).
Inline links: reciprocity.io
ABUJA, NIGERIA Contact: Olaoluwa Contact Info: akinloluwa[dot]olaoluwa[at]gmail[dot]com Time: Saturday, April 20th, 11:00 AM Location: The 'High Table' at Habil Cafe, No 3 Atapkme Street, Wuse II, Abuja. There will be a small sign saying 'Abuja ACX Meetup' Coordinates: https://plus.codes/6FX93F9H+J9 Notes: RSVP on LessWrong will be nice. Ended up eating all the food last time ):
Inline links: https://plus.codes/6FX93F9H+J9
Meetups Everywhere 2024: Times & Places
August 29, 2024 · Original source
You can see a map of all the events on the LessWrong community page. (Or possibly you will be able to soon.)
Inline links: the LessWrong community page
Extra Info For Meetup Organizers: 1. If you’re the host, bring a sign that says “ACX MEETUP” and prop it up somewhere (or otherwise be identifiable). 2. Bring blank labels and pens for nametags. 3. Have people type their name and email address in a spreadsheet or in a Google Form (accessed via a bit.ly link or QR code), so you can start a mailing list to make organizing future meetups easier. 4. If you’re having trouble thinking of something to talk about, the attendees probably also read ACX. Talk about a recent post or book review that you liked. 5. If it’s the first meetup, people are probably just going to want to talk, and you shouldn’t try to organize some kind of planned workshop or anything like that. 6. It’s easier to schedule a followup meetup while you’re having the first, compared to trying to do it later on by email. 7. If you didn’t make a LessWrong event for your meetup (or if you did but Skyler didn’t know about it) the LessWrong team did it for you using the username or email address you gave on the form. To claim your event, log into LW (or create an account) using that email address, or message the LW team on Intercom (chat button in the bottom right corner of lesswrong.com).
Contact: Yaseen Mowzer Contact Info: yaseen[a t]mowzer[dot]co[d ot]za Time: Saturday, September 14th, 06:00 PM Location: Truth Coffee Roasting, 36 Buitenkant St, Cape Town City Centre - we'll put a sign on the table Coordinates: https://plus.codes/4FRW3CCF+P3 Notes: Please RSVP using LessWrong or email or WhatsApp (+27 79 813 5144), so book I big enough table.
Inline links: https://plus.codes/4FRW3CCF+P3
24
September 17, 2024 · Original source
A Twitter user pointed out (and I confirmed) that upon being asked “What is the probability that Joe Biden is still President in October 2025?”, it goes through a lot of reasoning about his age and dementia and finally concludes 55% because he’s not that demented. I originally thought this might be due to the knowledge cutoff (it doesn’t know Biden dropped out in favor of Harris), but if I ask the AI about October 2029, then it says that Joe Biden has dropped out in favor of Harris (even though in that question it doesn’t matter). So now I think it’s more like ChatGPT’s tendency to round anything that sounds vaguely like the surgeon riddle off to the surgeon riddle - in the same way, FiveThirtyNine rounds off anything that sounds vaguely like the popular question “is Biden too old and demented to stay president?” into that question, even though there are much stronger non-dementia-related reasons he can’t be president next year. The FutureSearch team wrote a LessWrong post generalizing these kinds of observations, Contra Papers Claiming Superhuman AI Forecasting. They examine four claims, including the one above, and find similar problems with all of them. Sometimes the teams involved missed potential data contamination (ie their LLM wasn’t forecasting, it just already knew the answers). Other times the LLM failed but - in the spirit of technologists everywhere - the researchers invented finicky definitions of “above human level” by which even mediocre AIs qualified. They conclude: Today's autonomous AI forecasting can be better than average, or even experienced, human forecasters…but it's very unlikely that any autonomous AI forecaster yet built is close to the accuracy of a top 2% Metaculus forecaster, or the crowd. Still, FiveThirtyNine is a big advance in at least one way: as far as I know, it’s the first high-quality AI forecaster which is free to the general public. Try it out! This means there’s still time to use this joke when they invent the actually good one! r/MarkMyWords This is a subreddit for people who want to record bold predictions. There’s nothing formal - nobody gives probabilities, and some of them don’t even have end dates. It’s just people going out on a limb to say they’re sure something will happen. …most of them are “mark my words, time will prove Democrats right about everything, and reveal Republicans to be disgusting criminal hypocrites”. …so much so that it kind of fails as a potentially interesting institution and becomes just another monument to how sad the Internet’s gotten. Still, it might be fun to keep going until you find an old post where the prediction has already “resolved”, and see what happens. Here are some of the highest-upvoted posts from at least a year ago (minus pop culture and dumb in-jokes): MMW: It will turn out the Notre Dame fire was actually arson, and not an “accident” as the Paris police initially claimed.
Inline links: round anything that sounds vaguely like the surgeon riddle off to the surgeon riddle, Contra Papers Claiming Superhuman AI Forecasting, Try it out!, https://substackcdn.com/image/fetch/$s_!-I28!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0ca5e763-6951-41f1-899d-d033d9729bd0_592x116.png, This is a subreddit, https://substackcdn.com/image/fetch/$s_!5XR0!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F07c2b745-404d-4cc4-90ac-3dbbfd7eb2c9_1274x549.png, It will turn out the Notre Dame fire was actually arson, and not an “accident” as the Paris police initially claimed
Book Review Contest 2024 Winners
October 11, 2024 · Original source
2nd: Nine Lives, reviewed by David Matolcsi. David is an AI safety researcher from Hungary, currently living in Berkeley. He doesn't have much publicly available writing yet, but plans to publish some new blog posts on LessWrong in the coming months
Inline links: Nine Lives
Meetups Everywhere Spring 2025: Times & Places
March 25, 2025 · Original source
There should very shortly be a map of these meetups on the LessWrong community page.
Inline links: the LessWrong community page
Extra Info For Meetup Organizers: 1. If you’re the host, bring a sign that says “ACX MEETUP” and prop it up somewhere (or otherwise be identifiable). 2. Bring blank labels and pens for nametags. 3. If you’re having trouble thinking of something to talk about, the attendees probably also read ACX. Ask people about a recent post or book review that they liked. 4. If it’s the first meetup, people are probably just going to want to talk, and you shouldn’t try to organize some kind of planned workshop or anything like that. 5. Have people type their name and email address in a spreadsheet or in a Google Form (accessed via a bit.ly link or QR code), so you can start a mailing list to make organizing future meetups easier. 6. It’s easier to schedule a followup meetup while you’re having the first, compared to trying to do it later on by email. 7. If you didn’t make a LessWrong event for your meetup (or if you did but Skyler didn’t know about it) the LessWrong team did it for you using the username or email address you gave on the form. To claim your event, log into LW (or create an account) using that email address, or message the LW team on Intercom (chat button in the bottom right corner of lesswrong.com).berkel
Contact: Ozge Contact Info: ozgeco[a t]yahoo[period]com Time: Saturday, May 3rd, 1:00 PM Location: Cafe Modern at Galataport, Istanbul Modern Museum Entrance Floor Coordinates: https://plus.codes/8GHC2XGM+94 Notes: I organize this meeting with the EA Istanbul Group. ACX readers, AI Safety and EA people, all of you are warmly welcomed. If possible, let me know that you will be attending by dropping an email or replying on LessWrong. I will be sitting outside of the cafe - weather permitting- with a ACX Meeting sign on the table. Looking forward to meeting old friends and new ones!
Inline links: https://plus.codes/8GHC2XGM+94
My Takeaways From AI 2027
April 08, 2025 · Original source
Title comes from this LessWrong post, but it was the impression I got from AI 2027 too. If things go this fast, there won’t be time for a grassroots-level campaign for safety, or even for safety-related legislation. Whether or not the AI is safe will depend on company insiders. First, the CEO/board/leadership and how much they choose to prioritize safety. Second, the alignment team, and how skilled they are. Third, the rank-and-file employees, and how much they grumble/revolt if their company seems to be acting irresponsibly.
Inline links: this LessWrong post
Bayes For Everyone
May 30, 2025 · Original source
Over the years quite a few folks have attempted to explain it clearly. Eliezer wrote his famous essay back in 2003 (which Khalid Azad helpfully summarized in 2007), Scott’s written about it a number of times, Steven Pinker takes a whack at it in Rationality, Julia Galef speaks about it on BigThink, and so on and so forth. Recently, there’s even been a book explaining Bayes to babies. Bayesianism has become quite a racket!
Inline links: his famous essay back in 2003, helpfully summarized in 2007, http://www.amazon.com/Rationality-What-Seems-Scarce-Matters/dp/0525561994, speaks about it on BigThink, a book explaining Bayes to babies
Images can do this. We shouldn’t be surprised: our pre-vertebrate ancestors evolved to see stuff, and to this day a substantial portion of our cortices is devoted to processing what our eyes take in. If we can find a way to represent this visually, we can sidestep this problem. And indeed on LessWrong, there’s an explanation that uses blobs, triangles, and pentagons; another person has built one out of Venn diagrams. My tastes, though, skew toward the simple, and I prefer the visualization created by the YouTuber 3Blue1Brown in a video that sets a new standard for elegance in explanation:
Inline links: an explanation that uses blobs, triangles, and pentagons, out of Venn diagrams, in a video that sets a new standard for elegance in explanation
Not at all — mathematically, they’re equivalent! In working through the problem visually, you do every single step that the equation does. There may be practical or aesthetic reasons someone might prefer working with the equation, but don’t forget what the purpose of the sword is.
Inline links: what the purpose of the sword is
Highlights From The Comments On Missing Heritability
July 03, 2025 · Original source
Ooh, I have an extensive discussion of this in a recent post: https://www.lesswrong.com/posts/xXtDCeYLBR88QWebJ/heritability-five-battles Relevant excerpts follow:
Inline links: https://www.lesswrong.com/posts/xXtDCeYLBR88QWebJ/heritability-five-battles
Meetups Everywhere 2025: Times and Places
August 29, 2025 · Original source
There should shortly be a map of these meetups on the LessWrong community page.
Inline links: the LessWrong community page
Extra Info For Meetup Organizers: 1. If you’re the host, bring a sign that says “ACX MEETUP” and prop it up somewhere (or otherwise be identifiable). 2. Bring blank labels and pens for nametags. 3. If you’re having trouble thinking of something to talk about, the attendees probably also read ACX. Ask people about a recent post or book review that they liked. 4. If it’s the first meetup, people are probably just going to want to talk, and you shouldn’t try to organize some kind of planned workshop or anything like that. 5. Have people type their name and email address in a spreadsheet or in a Google Form (accessed via a bit.ly link or QR code), so you can start a mailing list to make organizing future meetups easier. 6. It’s easier to schedule a followup meetup while you’re having the first, compared to trying to do it later on by email. 7. If you didn’t make a LessWrong event for your meetup (or if you did but Skyler didn’t know about it) the LessWrong team did it for you using the username or email address you gave on the form. To claim your event, log into LW (or create an account) using that email address, or message the LW team on Intercom (chat button in the bottom right corner of lesswrong.com).
Contact: Nihal Contact Info: propwash[a t]duck[period]com Time: Sunday, October 5th, 4:00 PM Location: Matteo Coffea, Church Street Coordinates: https://plus.codes/7J4VXJF4+PR Group Link: https://www.lesswrong.com/groups/i5vLw9xnG9iwXNQZZ Notes: Check the lesswrong group page for the announcement, and RSVP there.
Inline links: https://plus.codes/7J4VXJF4+PR, https://www.lesswrong.com/groups/i5vLw9xnG9iwXNQZZ
Meetups Everywhere Spring 2026: Times & Places
April 01, 2026 · Original source
There should shortly be a map of these meetups on the LessWrong community page.
Inline links: the LessWrong community page
Extra Info For Meetup Organizers: 1. If you’re the host, bring a sign that says “ACX MEETUP” and prop it up somewhere (or otherwise be identifiable). 2. Bring blank labels and pens for nametags. 3. If you’re having trouble thinking of something to talk about, the attendees probably also read ACX. Ask people about a recent post or book review that they liked. 4. If it’s the first meetup, people are probably just going to want to talk, and you shouldn’t try to organize some kind of planned workshop or anything like that. 5. Have people type their name and email address in a spreadsheet or in a Google Form (accessed via a bit.ly link or QR code), so you can start a mailing list to make organizing future meetups easier. 6. It’s easier to schedule a followup meetup while you’re having the first, compared to trying to do it later on by email. 7. If you didn’t make a LessWrong event for your meetup (or if you did but Skyler didn’t know about it) the LessWrong team did it for you using the username or email address you gave on the form. To claim your event, log into LW (or create an account) using that email address, or message the LW team on Intercom (chat button in the bottom right corner of lesswrong.com).
Contact: Faiz Contact Info: faiz_abbas[@]protonmail[.]com Time: Sunday, April 5th, 4:00 PM Location: Matteo Coffea, 2, Church Street, Brigade Rd. Get inside Matteo Coffea and walk to the backside seating area. By backside, I mean all the way back - there is a second section to the cafe past the restrooms. You can find us seated on the right or left side with an ACX Meet-ups sign pointing the way Coordinates: https://plus.codes/7J4VXJF4+PR Group Link: https://www.lesswrong.com/groups/i5vLw9xnG9iwXNQZZ Notes: Please RSVP on LessWrong so I know how many people are coming: https://www.lesswrong.com/events/3yEAjBzu5D7HgManA/bengaluru-acx-meetups-everywhere-spring-26
Inline links: https://plus.codes/7J4VXJF4+PR, https://www.lesswrong.com/groups/i5vLw9xnG9iwXNQZZ, https://www.lesswrong.com/events/3yEAjBzu5D7HgManA/bengaluru-acx-meetups-everywhere-spring-26