PALM
Article
PALM is a recurring brand in the Astral Codex Ten archive, appearing 2 times across 2 issues between April 18, 2022 and March 12, 2024. The archive places it in contexts such as “Second, PALM, a new and very impressive language AI:”; “he gets twelve LLMs - including Bard, GPT, Claude, Mistral, PaLM, LLaMa”; “twelve LLMs - including … PaLM”. It most often appears alongside Eliezer Yudkowsky, Metaculus, Asterisk.
Metadata
- Category: Brands
- Mention count: 2
- Issue count: 2
- First seen: April 18, 2022
- Last seen: March 12, 2024
Appears In
Related Pages
-
- Eliezer Yudkowsky (2 shared issues)
-
- Metaculus (2 shared issues)
-
- Asterisk (1 shared issues)
-
- Bard (1 shared issues)
-
- Berkeley (1 shared issues)
-
- Bitcoin (1 shared issues)
-
- China (1 shared issues)
-
- Chinchilla (1 shared issues)
-
- Christiano (1 shared issues)
-
- Claude (1 shared issues)
-
- COVID-19 (1 shared issues)
-
- DALL-E2 (1 shared issues)
External Links
Source Context
Recovered passages from the original issue text. When the raw archive preserved outbound links inside the source passage, they are listed directly under the quote.
Second, PALM, a new and very impressive language AI:
Inline links: PALM
PALM explaining jokes (source) Third, Chinchilla, a paper and associated model suggesting that people have been training AIs inefficiently all this time, and that probably a small tweak to the process could produce better results with the same computational power.
Inline links: source, Chinchilla
Are these the data I’ve been trying to get for years - which forecasting platforms beat which others? I don’t think so - Metaculus’ good Briar score only means it performs well on Metaculus’ questions, which might be easier or harder than some other platform’s questions. Can we use the Halawi et al AI as a fixed comparison point, since it’s always the same skill level? I’m not sure - it trained on each of these markets for the style of question that’s in each market, so it might be biased. Still, these numbers are all about where I would expect them to be, except maybe Polymarket, which does better than I would have expected. But the crowd still beats the AI, right? Halawi et al object that humans can forecast only when they feel like it - you can bet on a prediction market question you feel confident on, and avoid one you don’t. When they let their AI forecast only on those questions where it’s most likely to do well (eg those with lots of relevant news articles), it very slightly outperforms the human crowd. As AI gets better, will it naturally beat humans in forecasting? Halawi et al say this won’t be trivial. They find a version of their system based off GPT-3.5 is only very slightly worse than the final version built off GPT-4. This suggests a forecasting AI built off GPT-5 or 6 might get only small improvements. The second team is Tetlock et al. They start from the same place as Halawi - out-of-the-box LLMs aren’t good at forecasting. They’re more scathing about this than Halawi was - they argue that out-of-the-box models do worse than predicting 50% for everything (this was close to true of human forecasters in the ACX tournament). Instead of increasing quality, Tetlock increases quantity. He wants to do wisdom of crowds, where the crowd is a bunch of different LLMs. So he gets twelve LLMs - including Bard, GPT, Claude, Mistral, PaLM, LLaMa, some Chinese models I’d never heard of, and a couple of variations on these bases - asks them to predict questions, and averages the results. Remember, you gotta prompt your model with “you are a smart person”, or else it won’t be smart! The results: Next, we compare the LLM crowd performance to that of the human crowd for our second hypothesis, directly putting the two crowd-aggregation mechanisms head-to-head. To do this, we use the same LLM crowd average as before (taking the median LLM prediction on each question and averaging up the Brier scores across questions). We compare this to the average of median human predictions on the same questions. In our preregistered analysis, we fail to find statistically significant differences between the LLM crowd’s mean Brier score of M=0.20 (SD=0.12) and that of the human crowd, M=0.19 (SD=0.19), t(60) = 0.19, p = 0.850 Their study was much smaller than Halawi’s (31 questions vs. 3,672), so I don’t think this result (nonsignificant small difference) should be considered different from Halawi’s (significant small difference). Still, it’s weird, isn’t it? Halawi used a really complicated tower of prompts and APIs and fine-tunings, and Tetlock just got more LLMs, and they both did about the same. I have two questions after reading these results: Did they actually do the same, or is this just a function of the small sample size in Tetlock and the non-head-to-head comparison?