Astral Codex Ten

Samotsvety Forecasting

13 min read

Samotsvety Forecasting

Article

Samotsvety Forecasting is a recurring organization in the Astral Codex Ten archive, appearing 2 times across 2 issues between October 18, 2022 and March 21, 2024. The archive places it in contexts such as “Samotsvety Forecasting is a team made of top prediction market players”; “Samotsvety Forecasting is a team of some of the top forecasters in the world”. It most often appears alongside Manifold, Sam Altman, 2024 election.

Metadata

  • Category: Organizations
  • Mention count: 2
  • Issue count: 2
  • First seen: October 18, 2022
  • Last seen: March 21, 2024

Appears In

Source Context

Recovered passages from the original issue text. When the raw archive preserved outbound links inside the source passage, they are listed directly under the quote.

22
October 18, 2022 · Original source
Several additional plaintiffs I can’t find good information about You can find the complaint here. The plaintiffs write: The [CFTC’s action], without explanation or other indication of reasoned decisionmaking, without “written notice of the facts or conduct which may warrant” the Revocation, and without providing anyone “an opportunity to demonstrate or achieve compliance” with the terms of No-Action Relief or other requirements, violates the Administrative Procedure Act. 5 U.S.C. §§ 558, 706. Among other things, the Revocation is “arbitrary, capricious, an abuse of discretion, [and/or] otherwise not in accordance with law” and occurred “without observance of procedure required by law.” The Court should “hold unlawful and set aside” the Revocation, including its command that contracts that would otherwise turn on events occurring after February 2023 be prematurely liquidated. 5 U.S.C. § 706. The Court also should enter a preliminary and then permanent injunction against the prescriptions in the Revocation requiring the liquidation of contracts by February 2023, including contracts that concern the 2024 elections, well before they would ordinarily mature. I am not a lawyer, but it sounds kind of like they’re saying “the decision was bad, and the Administrative Procedure Act says regulators shouldn’t do bad things”. I am split between the part of me which hates government regulators doing bad things, and the part of me which feels like this is how you get a cover-your-ass-ocracy that never does anything at all without fifteen layers of paperwork and ten trillion dollars per action. Whatever. At least this time it’s in my favor. Of course there are prediction markets about it: Source: Insight Prediction Nuclear Warcasting, Part 2 Samotsvety Forecasting is a team made of top prediction market players and tournament winners, vaguely affiliated with effective altruism, who make predictions in the public interest. Earlier this year, they got attention for forecasting the risk of nuclear war - in particular, they said there was an a 0.01% per month chance of London getting nuked this spring. Since then, most of the fear has crystallized into a specific scenario. Suppose Russia is losing very badly in Ukraine. Putin, fearing a coup or revolution at home if he gives up, decides to use a tactical nuclear weapon, ie a “small” nuke more suited to winning battles than destroying cities. He nukes a Ukrainian battlefield position. The West is enraged at this violation of the nuclear taboo and feels like it needs to respond decisively - maybe by nuking something on Russia’s side, or through some other act of extreme escalation. Then Russia feels like they need to respond, and eventually it escalates to strikes on major cities and global nuclear war. There are reasons for doubt. Tactical nukes wouldn’t really be useful in Ukraine; the battle lines are too spread out and there’s no single place where a nuclear explosion could take out a substantial portion of Ukraine’s forces. In the past, nuclear powers have accepted lost wars gracefully rather than turning to nukes. And the Russians deny it, and saying this is all just Western propaganda intended to scare people. Amid this uncertainty, Samotsvety has published an update: now they are at 16% chance that “Russia uses any type of nuclear weapon in Ukraine in the next year”, and 0.02% per month of a strike on London. Although they didn’t mention it this time, they previously said the risk of a strike on San Francisco was a little over half that of London; I don’t know if that’s changed. See also Dan Keys’ comment here for some skepticism of Samotsvety’s process. Swift Centre is a lot like Samotsvety; they’re a collection of top forecasters brought together by EA to make important predictions. They also took a swing at the nuclear question, and said 9.1% chance of a hostile nuclear detonation in Europe in the next six months. They didn’t calculate the risk that this would spread to global war, but they did discuss how different scenarios would bring the risk up or down: One of my hopes for forecasting is that it eventually becomes so well-validated that decision-makers can take these kinds of considerations into account: “Should we sent ATACMS missiles to Ukraine? It would have such-and-such benefits, but also increase the risk of nuclear escalation by 3.6%, is it worth it?” We can’t directly compare Samotsvety and Swift because they’re predicting over different time periods. But assuming that there’s more risk in the next six months than in the six months after that, I think Samotsvety is a little higher but they’re not embarrassingly far off. Metaculus is a bit more optimistic than either, believing there’s only a 4% chance of detonation in Ukraine in 2023 and a 7% chance of any use in the next ~year. Max Tegmark is going much higher than anyone else and says 16% chance of global nuclear war. Kalshi Applies For Election Markets Kalshi is a regulated and fully-legal prediction market with good lobbyists and a compliance team. This means the CFTC probably won’t randomly shut them down one day. But it also means they can only create new markets with CFTC permission. In July, Kalshi asked the CFTC for permission to make midterm election prediction markets - specifically, which party will win control of the House and Senate. The CFTC has said they will make a decision by October 28 (which doesn’t leave much time for predicting to happen before the November 8 election, but I guess it sets a precedent). September was the Request For Comment period, when the CFTC solicited comments from stakeholders about what they should do. Kalshi tried really hard to get lots of people to send in positive assessments - I know this because of how many people asked me “why is the CEO of Kalshi emailing me about this thing?” Their strategy seems to have worked; among the people who wrote to the CFTC in support were: A managing director at JP Morgan
Inline links: additional plaintiffs, here, https://substackcdn.com/image/fetch/$s_!LaYa!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2F0aaec309-35d6-430d-aeaa-b7cab022f4d6_548x215.png, Insight Prediction, Samotsvety Forecasting, Earlier this year, accepted lost wars gracefully, published an update, comment here, Swift Centre, They also took a swing at the nuclear question, https://substackcdn.com/image/fetch/$s_!TczM!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2F702c1236-47d9-4c60-9b99-ab9b2ca320e7_737x377.png, a 4% chance, a 7% chance, 16% chance, In July, they will make a decision by October 28, A managing director at JP Morgan
One of my hopes for forecasting is that it eventually becomes so well-validated that decision-makers can take these kinds of considerations into account: “Should we sent ATACMS missiles to Ukraine? It would have such-and-such benefits, but also increase the risk of nuclear escalation by 3.6%, is it worth it?” We can’t directly compare Samotsvety and Swift because they’re predicting over different time periods. But assuming that there’s more risk in the next six months than in the six months after that, I think Samotsvety is a little higher but they’re not embarrassingly far off. Metaculus is a bit more optimistic than either, believing there’s only a 4% chance of detonation in Ukraine in 2023 and a 7% chance of any use in the next ~year. Max Tegmark is going much higher than anyone else and says 16% chance of global nuclear war. Kalshi Applies For Election Markets Kalshi is a regulated and fully-legal prediction market with good lobbyists and a compliance team. This means the CFTC probably won’t randomly shut them down one day. But it also means they can only create new markets with CFTC permission. In July, Kalshi asked the CFTC for permission to make midterm election prediction markets - specifically, which party will win control of the House and Senate. The CFTC has said they will make a decision by October 28 (which doesn’t leave much time for predicting to happen before the November 8 election, but I guess it sets a precedent). September was the Request For Comment period, when the CFTC solicited comments from stakeholders about what they should do. Kalshi tried really hard to get lots of people to send in positive assessments - I know this because of how many people asked me “why is the CEO of Kalshi emailing me about this thing?” Their strategy seems to have worked; among the people who wrote to the CFTC in support were: A managing director at JP Morgan
Inline links: a 4% chance, a 7% chance, 16% chance, In July, they will make a decision by October 28, A managing director at JP Morgan
In Continued Defense Of Non-Frequentist Probabilities
March 21, 2024 · Original source
Consider some object or process which might or might not be a coin - perhaps it’s a dice, or a roulette wheel, or a US presidential election. We divide its outcomes into two possible bins - evens vs. odds, reds vs. blacks, Democrats vs. Republicans - one of which I have arbitrarily designated “heads” and the other “tails” (you don’t get to know which side is which). It may or may not be fair. What’s the probability it comes out heads? The answer to all of these is exactly the same - 50% - even though you have wildly different amounts of knowledge about each. This is because 50% isn’t a description of how much knowledge you have, it’s a description of the balance between different outcomes. Is it bad that one term can mean both perfect information (as in 1) and total lack of information (as in 3)? No. This is no different from how we discuss things when we’re not using probability. Do vaccines cause autism? No. Does drinking monkey blood cause autism? Also no. My evidence on the vaccines question is dozens of excellent studies, conducted so effectively that we’re as sure about this as we are about anything in biology. My evidence on the monkey blood question is that nobody’s ever proposed this and it would be weird if it were true. Still, it’s perfectly fine to say the single-word answer “no” to both of them to describe where I currently stand. If someone wants to know how much evidence/certainty is behind my “no”, they can ask, and I’ll tell them. Likewise, is there a God? Maybe you ask the world’s top philosopher of religion, who has spent his entire life thinking about this question, and he says “I’m not sure”. Then you ask a random teenager who has given it two seconds’ thought, and she also says “I’m not sure”. Neither of these people has done anything wrong. Their identical answers conceal a vastly different amount of thought that’s gone into the question. But it’s your job to ask each person how much thought they put in, not the job of the English language to design a way of saying the words “I’m not sure” that communicates level of effort and expertise. Likewise, if I answer there’s a 0.001% chance vaccines cause autism, and a 0.001% chance monkey blood causes autism, it’s not the job of probability theory to tell you how much effort went into that assessment and how much of an expert I am. If you care about that, you can ask me! 3. What Is Samotsvety Better Than You At? Samotsvety Forecasting is a team of some of the top forecasters in the world. Their job is to assign probabilities to future events. They seem very good at it. They win forecasting contests. They make lots of money on prediction markets. They get featured in media articles. Sometimes people hire them as consultants when they have some forecasting question relevant to their business. Sometimes some client will ask Samotsvety for a prediction relative to their business, for example whether Joe Biden will get impeached, and they will give a number like “it’s 17% likely that this thing will happen”. This number has some valuable properties: It’s well-calibrated. Things that they assign 17% probability to will happen about 17% of the time. If you randomly change this number (eg round it to 20%, or invert it to 83%) you will be less well-calibrated.
Inline links: get featured in media articles
If you were someone whose job involved adjusting for situations like these - for example, an oil trader who thought that Joe Biden getting impeached would change oil prices - you would do better at your job, over the long run, by treating it as 17% likely that Joe Biden would be impeached then by guessing randomly, or consistently taking some number 5% higher than Samotsvety’s number, or using some other method. If there were two oil companies, and one consistently used Samotsvety’s numbers to make decisions, and the other consistently took a random number 1-100 and made decisions with that, the one that used Samotsvety’s numbers would do better.
In other words, there’s something special about the number 17% on this question. It has properties that other numbers like 38% or 99.9999% don’t have. If someone asked you (rather than Samotsvety) for this number, you would give a less good number that didn’t have these special properties. If by some chance you actually were better at finding these kinds of numbers than Samotsvety, you could probably get a job as a forecasting consultant. Or you could make lots of play money on Manifold, or lots of real money on the stock market, or help your preferred political party as a campaign strategist.