Samotsvety
Article
Samotsvety is a recurring organization in the Astral Codex Ten archive, appearing 9 times across 9 issues between March 14, 2022 and July 01, 2025. The archive places it in contexts such as “Thanks to Samotsvety and their friends for making this happen!”; “The first group like this to catch my attention was Samotsvety , who describe themselves as ‘a group of forecasters’”; “Samotsvety has published an update”. It most often appears alongside Metaculus, Polymarket, Manifold.
Metadata
- Category: Organizations
- Mention count: 9
- Issue count: 9
- First seen: March 14, 2022
- Last seen: July 01, 2025
Appears In
Related Pages
-
- Metaculus (7 shared issues)
-
- Polymarket (6 shared issues)
-
- Manifold (5 shared issues)
-
- Twitter (5 shared issues)
-
- Ukraine (5 shared issues)
-
- Eliezer Yudkowsky (4 shared issues)
-
- Kalshi (4 shared issues)
-
- Manifold Markets (4 shared issues)
-
- OpenAI (4 shared issues)
-
- Augur (3 shared issues)
-
- California (3 shared issues)
-
- Democrats (3 shared issues)
External Links
Source Context
Recovered passages from the original issue text. When the raw archive preserved outbound links inside the source passage, they are listed directly under the quote.
Enter Samotsvety Forecasts. This is a team of some of the best superforecasters in the world. They won the CSET-Foretell forecasting competition by an absolutely obscene margin, “around twice as good as the next-best team in terms of the relative Brier score”. If the point of forecasting tournaments is to figure out who you can trust, the science has spoken, and the answer is “these guys”.
Inline links: an absolutely obscene margin
You can read more about their methodology and reasoning on the post on Effective Altruism Forum, but I found this table helpful:
Inline links: Effective Altruism Forum
Along with reassuring me I made the right choice not to run and hide, this is a new landmark in translating forecasting results to the real world. The whole stack of technologies came together: tournaments to determine who the best predictors are, methods for aggregating probabilities, and a real-world question that lots of people care about. Thanks to Samotsvety and their friends for making this happen!
The first group like this to catch my attention was Samotsvety, who describe themselves as “a group of forecasters with a great track record working on applying forecasting to impactful questions”; they released a nuclear risk forecast in March which sparked some good debate.
Several additional plaintiffs I can’t find good information about You can find the complaint here. The plaintiffs write: The [CFTC’s action], without explanation or other indication of reasoned decisionmaking, without “written notice of the facts or conduct which may warrant” the Revocation, and without providing anyone “an opportunity to demonstrate or achieve compliance” with the terms of No-Action Relief or other requirements, violates the Administrative Procedure Act. 5 U.S.C. §§ 558, 706. Among other things, the Revocation is “arbitrary, capricious, an abuse of discretion, [and/or] otherwise not in accordance with law” and occurred “without observance of procedure required by law.” The Court should “hold unlawful and set aside” the Revocation, including its command that contracts that would otherwise turn on events occurring after February 2023 be prematurely liquidated. 5 U.S.C. § 706. The Court also should enter a preliminary and then permanent injunction against the prescriptions in the Revocation requiring the liquidation of contracts by February 2023, including contracts that concern the 2024 elections, well before they would ordinarily mature. I am not a lawyer, but it sounds kind of like they’re saying “the decision was bad, and the Administrative Procedure Act says regulators shouldn’t do bad things”. I am split between the part of me which hates government regulators doing bad things, and the part of me which feels like this is how you get a cover-your-ass-ocracy that never does anything at all without fifteen layers of paperwork and ten trillion dollars per action. Whatever. At least this time it’s in my favor. Of course there are prediction markets about it: Source: Insight Prediction Nuclear Warcasting, Part 2 Samotsvety Forecasting is a team made of top prediction market players and tournament winners, vaguely affiliated with effective altruism, who make predictions in the public interest. Earlier this year, they got attention for forecasting the risk of nuclear war - in particular, they said there was an a 0.01% per month chance of London getting nuked this spring. Since then, most of the fear has crystallized into a specific scenario. Suppose Russia is losing very badly in Ukraine. Putin, fearing a coup or revolution at home if he gives up, decides to use a tactical nuclear weapon, ie a “small” nuke more suited to winning battles than destroying cities. He nukes a Ukrainian battlefield position. The West is enraged at this violation of the nuclear taboo and feels like it needs to respond decisively - maybe by nuking something on Russia’s side, or through some other act of extreme escalation. Then Russia feels like they need to respond, and eventually it escalates to strikes on major cities and global nuclear war. There are reasons for doubt. Tactical nukes wouldn’t really be useful in Ukraine; the battle lines are too spread out and there’s no single place where a nuclear explosion could take out a substantial portion of Ukraine’s forces. In the past, nuclear powers have accepted lost wars gracefully rather than turning to nukes. And the Russians deny it, and saying this is all just Western propaganda intended to scare people. Amid this uncertainty, Samotsvety has published an update: now they are at 16% chance that “Russia uses any type of nuclear weapon in Ukraine in the next year”, and 0.02% per month of a strike on London. Although they didn’t mention it this time, they previously said the risk of a strike on San Francisco was a little over half that of London; I don’t know if that’s changed. See also Dan Keys’ comment here for some skepticism of Samotsvety’s process. Swift Centre is a lot like Samotsvety; they’re a collection of top forecasters brought together by EA to make important predictions. They also took a swing at the nuclear question, and said 9.1% chance of a hostile nuclear detonation in Europe in the next six months. They didn’t calculate the risk that this would spread to global war, but they did discuss how different scenarios would bring the risk up or down: One of my hopes for forecasting is that it eventually becomes so well-validated that decision-makers can take these kinds of considerations into account: “Should we sent ATACMS missiles to Ukraine? It would have such-and-such benefits, but also increase the risk of nuclear escalation by 3.6%, is it worth it?” We can’t directly compare Samotsvety and Swift because they’re predicting over different time periods. But assuming that there’s more risk in the next six months than in the six months after that, I think Samotsvety is a little higher but they’re not embarrassingly far off. Metaculus is a bit more optimistic than either, believing there’s only a 4% chance of detonation in Ukraine in 2023 and a 7% chance of any use in the next ~year. Max Tegmark is going much higher than anyone else and says 16% chance of global nuclear war. Kalshi Applies For Election Markets Kalshi is a regulated and fully-legal prediction market with good lobbyists and a compliance team. This means the CFTC probably won’t randomly shut them down one day. But it also means they can only create new markets with CFTC permission. In July, Kalshi asked the CFTC for permission to make midterm election prediction markets - specifically, which party will win control of the House and Senate. The CFTC has said they will make a decision by October 28 (which doesn’t leave much time for predicting to happen before the November 8 election, but I guess it sets a precedent). September was the Request For Comment period, when the CFTC solicited comments from stakeholders about what they should do. Kalshi tried really hard to get lots of people to send in positive assessments - I know this because of how many people asked me “why is the CEO of Kalshi emailing me about this thing?” Their strategy seems to have worked; among the people who wrote to the CFTC in support were: A managing director at JP Morgan
Inline links: additional plaintiffs, here, https://substackcdn.com/image/fetch/$s_!LaYa!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2F0aaec309-35d6-430d-aeaa-b7cab022f4d6_548x215.png, Insight Prediction, Samotsvety Forecasting, Earlier this year, accepted lost wars gracefully, published an update, comment here, Swift Centre, They also took a swing at the nuclear question, https://substackcdn.com/image/fetch/$s_!TczM!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2F702c1236-47d9-4c60-9b99-ab9b2ca320e7_737x377.png, a 4% chance, a 7% chance, 16% chance, In July, they will make a decision by October 28, A managing director at JP Morgan
One of my hopes for forecasting is that it eventually becomes so well-validated that decision-makers can take these kinds of considerations into account: “Should we sent ATACMS missiles to Ukraine? It would have such-and-such benefits, but also increase the risk of nuclear escalation by 3.6%, is it worth it?” We can’t directly compare Samotsvety and Swift because they’re predicting over different time periods. But assuming that there’s more risk in the next six months than in the six months after that, I think Samotsvety is a little higher but they’re not embarrassingly far off. Metaculus is a bit more optimistic than either, believing there’s only a 4% chance of detonation in Ukraine in 2023 and a 7% chance of any use in the next ~year. Max Tegmark is going much higher than anyone else and says 16% chance of global nuclear war. Kalshi Applies For Election Markets Kalshi is a regulated and fully-legal prediction market with good lobbyists and a compliance team. This means the CFTC probably won’t randomly shut them down one day. But it also means they can only create new markets with CFTC permission. In July, Kalshi asked the CFTC for permission to make midterm election prediction markets - specifically, which party will win control of the House and Senate. The CFTC has said they will make a decision by October 28 (which doesn’t leave much time for predicting to happen before the November 8 election, but I guess it sets a precedent). September was the Request For Comment period, when the CFTC solicited comments from stakeholders about what they should do. Kalshi tried really hard to get lots of people to send in positive assessments - I know this because of how many people asked me “why is the CEO of Kalshi emailing me about this thing?” Their strategy seems to have worked; among the people who wrote to the CFTC in support were: A managing director at JP Morgan
If we submit Question #2 to the Supreme Court, will they rule in favor? …and so on. A few randomly selected questions went before the Supreme Court each year, and the corresponding markets were resolved normally. In every other case, the market’s verdict was final, and bettors got their money refunded. 5.5: Asking people who are good at prediction markets to do other things Prediction markets have many limitations. They can’t predict far future events. They have trouble predicting things that involve money becoming worthless (eg human extinction). There are complicated biases in predicting conditionals. One way around these is to see who keeps winning prediction markets, assume those people are smart and good at predicting things, and ask them to predict these issues directly. This would lose the well-aligned-incentives and canonicity of prediction markets. But it might be better than asking random experts in related fields who have never shown any particular skill at prediction to do this. Or the prediction market experts could work with the domain-relevant experts to come up with a joint statement. Samotsvety is a group of some of the world’s top forecasters. They have released predictions on various risks and issues that markets/tournaments aren’t a good match for, using the same methodology that they use when predicting in markets and tournaments. I trust them less than I would trust a well-functioning market or tournament, but still quite a substantial amount. 5.6: Education Even if nobody uses them for anything important, prediction markets’ existence has a salutary effect on the discourse. There’s something about framing a question as “at what odds would you bet on this in a prediction market?” that seems to make people smarter. Some lifelong Democrat will be going on about how the Democrats can’t possibly lose the next election and then you ask “at what odds would you bet on this in a prediction market?” and they suddenly backtrack. Even the idea that we can think of events as occurring at some specific probability seems nonintuitive to a lot of people, and encountering prediction markets is the best way I know of to make those people understand this. Prediction markets also do really well at communicating a sense of how you’re not always right. A lot of people list off all the things they and their political side have been right about, cleverly eliding or making exceptions for every time they were wrong. Thinking more in terms of “How much money would you have been able to make on a prediction market using your skill of magically always being right?” seems to sometimes snap these people out of it, and help them avoid overconfidence. I would love if everyone on all sides of a debate had some prediction market experience, even if we weren’t able to use prediction markets to settle the debate directly. 6. What’s the current status of prediction markets? The United States currently classifies prediction markets as gambling, so real-money prediction markets are mostly illegal. This has forced markets to pursue one of three strategies: Operate outside the United States, closed to US citizens. Polymarket fills this niche effectively.
Inline links: Samotsvety, Polymarket
1: Vox: How A Ragtag Group Of Friends Became The Best At Forecasting World Events. Profile of Samotsvety, the world’s leading forecasting team.
Samotsvety: Samotsvety is a well-known forecasting team that usually wins these kinds of things. You can read more about them here. They scored 98th percentile, better than the aggregate of all other superforecasters. There’s are a few asterisks on this result: first, it wasn’t exactly a team effort - one of their forecasters did the work and “ran it by” everyone else without getting any objections. Second, for complicated legal reasons that they explained and which satisfied me, they couldn’t enter the contest proper and had to send me their guesses later, so I had to take it on trust that they were made in January along with everyone else’s.
Inline links: You can read more about them here
My main takeaway is that Metaculus beats prediction markets, superforecasters, wisdom of crowds, and (probably, most of the time) Samotsvety. Based on the performance of last year’s winners, most people who outperform Metaculus do so by luck and will regress to the mean next year. This contest leaves open the possibility that a small number of people (maybe including Ezra Karger) might be able to consistently get super-Metaculus performance - it just takes more than one contest to identify them.
Thanks to everyone who participated in this contest. Extra thanks to Christian Williams from Metaculus and the Manifold team for getting their respective sites involved, to Jonathan Mann and Samotsvety for willingly submitting to testing, and to Eric Neyman for calculating the scores.
Halawi and Tetlock’s AIs did between slightly-worse-than and equivalent-to the participant aggregate, so let’s say 90-95th percentile. FutureSearch claims to equal a 98th percentile forecaster, but they got this number through totally different and slightly suspicious methodology, so I don’t know if it’s actually any better. Still, we see that Samotsvety is capable of 98%ile performance (likely real and repeatable) and Metaculus of 99.5th. So there’s still a long way to go before we exhaust the limits of what’s possible to predict given the available amount of information! Towards Rationality Engines An interlude, before we get to other interesting prediction news. Forecasting AIs are pretty cool. I wouldn’t have expected them to work as well as they do. They are already superforecaster-level, and given the amount of low-hanging fruit that gets picked every day here, I can see them equalling or exceeding the top human forecasters in the next few years But they can’t answer many of the questions we care about most - questions that aren’t about prediction. Do masks prevent COVID transmission? Was OJ guilty? Did global warming contribute to the California superdrought? What caused the opioid crisis? Is social media bad for children? I see two interesting challenges ahead here: Making an AI that can do this.
I know many other superforecasters (conservatively: 10, but probably many more) who are very concerned about AI risk (example: Samotsvety). Is this just cherry-picking? Could we find 10+ superforecasters who believe anything? Or is this question unusual in its bimodal distribution of superforecaster opinions?
Inline links: Samotsvety
2: Sentinel is a foresight and emergency response team seeking to detect and react fast to large-scale global risks (eg a pandemic). The foresight team is run by some members of Samotsvety, the forecasting team that won the CSET Foretell tournaments several years in a row, and they publish some weekly minutes that tracks ongoing threats. They're seeking a larger audience for their reporting, collaborators, and funding; if you want to reach out you can do so at hello@sentinel-team.org.
32: Superforecaster Nuno Sempere (maybe with the rest of the Samotsvety team?) has launched askaforecaster.com. You can get a “quick take” for $20, a “thoughtful response” for $150, or an “in-depth look” for $1000.
Inline links: askaforecaster.com