GPT-2

Article

GPT-2 is a recurring organization in the Astral Codex Ten archive, appearing 2 times across 2 issues between June 07, 2022 and July 08, 2025. The archive places it in contexts such as “Of the nine prompts GPT-2 failed”; “back in 2020 when GPT-2 failed to add 2+1”. It most often appears alongside Gary Marcus, Gwern, Vitor.

Metadata

• Category: Organizations
• Mention count: 2
• Issue count: 2
• First seen: June 07, 2022
• Last seen: July 08, 2025

Appears In

• My Bet: AI Size Solves Flubs
• Now I Really Won That AI Bet

Related Pages

• • Gary Marcus (2 shared issues)
• • Gwern (2 shared issues)
• • Vitor (2 shared issues)
• • 4o (1 shared issues)
• • ACX Prediction Contest (1 shared issues)
• • AI (1 shared issues)
• • Askwho (1 shared issues)
• • Athens (1 shared issues)
• • Bayesian Conspiracy Discord (1 shared issues)
• • ChatGPT 4o (1 shared issues)
• • Creole (1 shared issues)
• • DALL-E (1 shared issues)

External Links

• • https://archive.nytimes.com/well.blogs.nytimes.com/2015/02/03/new-york-attorney-general-targets-supplements-at-major-retailers/
• • https://astralcodexten.substack.com/p/another-bay-area-house-party
• • https://astralcodexten.substack.com/p/book-review-the-cult-of-smart#comment-1292538
• • https://astralcodexten.substack.com/p/book-review-which-country-has-the
• • https://astralcodexten.substack.com/p/classifieds-thread-12022/comment/4732494
• • https://astralcodexten.substack.com/p/how-trustworthy-are-supplements/comment/9510016
• • https://astralcodexten.substack.com/p/links-for-august-2023/comment/22127925
• • https://astralcodexten.substack.com/p/links-for-august-2023/comment/22171207
• • https://astralcodexten.substack.com/p/nick-cammarata-on-jhana/comment/10017660
• • https://astralcodexten.substack.com/p/nick-cammarata-on-jhana/comment/10020520
• • https://astralcodexten.substack.com/p/please-dont-give-up-on-having-kids/comment/3213235
• • https://astralcodexten.substack.com/p/too-good-to-check-a-play-in-three

Source Context

Recovered passages from the original issue text. When the raw archive preserved outbound links inside the source passage, they are listed directly under the quote.

My Bet: AI Size Solves Flubs

June 07, 2022 · Original source

To give an example: in January 2020, Gary Marcus wrote a great post, GPT-2 And The Nature Of Intelligence, demonstrating a bunch of easy problems that GPT-2 failed on:

Inline links: GPT-2 And The Nature Of Intelligence

Marcus had previously shown that GPT-2 could reason accurately about the languages of well-known places and big countries. For example, if you prompt it with “I grew up in Athens, I speak fluent…”, then it will say “Greek”. Here he shows that it can’t do that with smaller, less obvious locales:

So this seems like pretty good evidence that GPT-2 has lots of glaring deficiencies. Marcus concludes:

Now I Really Won That AI Bet

July 08, 2025 · Original source

I think this thesis has done well so far. So far, every time people have claimed there’s something an AI can never do without “real understanding”, the AI has accomplished it with better pattern-matching. This was true back in 2020 when GPT-2 failed to add 2+1 and Gary Marcus declared that scaling had failed and it was time to “consider investing in different approaches” (according to Terence Tao, working with AIs is now “on par with trying to advise a mediocre, but not completely incompetent, static simulation of a graduate student”). I think progress in AI art tells the same story.

Inline links: Gary Marcus declared

Backlinks

• My Bet: AI Size Solves Flubs
• Now I Really Won That AI Bet
• Organizations: G