ChatGPT
Article
ChatGPT is a recurring concept in the Astral Codex Ten archive, appearing 3 times across 3 issues between January 26, 2023 and July 19, 2024. The archive places it in contexts such as “ChatGPT was a GPT instance simulating a character”; “ChatGPT is pretty advanced and fails semi-gracefully here”; “People like talking with ChatGPT not just because it knows things, but because it can talk like them”. It most often appears alongside ChatGPT, GPT, Jesus.
Metadata
- Category: Concepts
- Mention count: 3
- Issue count: 3
- First seen: January 26, 2023
- Last seen: July 19, 2024
Appears In
Related Pages
-
- ChatGPT (2 shared issues)
-
- GPT (2 shared issues)
-
- Jesus (2 shared issues)
-
- New York Times (2 shared issues)
-
- Reddit (2 shared issues)
-
- @moritheil (1 shared issues)
-
- ACX Prediction Contest (1 shared issues)
-
- Adam Tooze (1 shared issues)
-
- Africa (1 shared issues)
-
- AI X-Risk Research Podcast (1 shared issues)
-
- AISafety.training (1 shared issues)
-
- Al-Mada’in (1 shared issues)
External Links
Source Context
Recovered passages from the original issue text. When the raw archive preserved outbound links inside the source passage, they are listed directly under the quote.
Janus was writing in September 2022, just before ChatGPT. ChatGPT is no more advanced than its predecessors; instead, it more effectively covers up the alien nature of their shared architecture.
So if your reference point for a language model is ChatGPT, this post won’t make much sense. Instead, bring yourself all the way back to the hoary past of early 2022, when a standard interaction with a language model went like this:
So far, so boring. What really helped this sink in was reading Nostalgebraist say that ChatGPT was a GPT instance simulating a character called the Helpful, Harmless, and Honest Assistant.
Inline links: reading Nostalgebraist say
(source) 22: Related: the very center of GPT’s embedding space contains a few unusual tokens including the string “SolidGoldMagikarp”. GPT displays anomalous behavior if these tokens are inserted in a query; for example, it treats “SolidGoldMagikarp” as the word “distribute”. ChatGPT is pretty advanced and fails semi-gracefully here; GPT-2’s reaction to these tokens is more disturbing: (source: Less Wrong) Further investigation determined that many of these tokens are the screen names of a group of Redditors who attempted to count to infinity. The most likely explanation, according to the discoverers, is that these names were in GPT’s tokenization data, but not its training data (maybe they were especially common in the tokenization data because they made thousands of posts with numbers in them, but didn’t make it into the training data because their posts had no content?) - that leaves them existing without content, and GPT tries to round them off to some other “nearby” token (by incomprehensible AI standards of nearbyness). Congrats to the SERI-MATS AI alignment researchers who found all of this; maybe this makes it 0.0001% less likely that the AI which controls the nuclear arsenal in twenty years will have equally inexplicable behavior. 23: More language model news: LLM that understands and can explain images
Inline links: source, contains a few unusual tokens, https://substackcdn.com/image/fetch/$s_!rkvW!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fdeb0cd39-6942-44c0-85ef-0fac91b362e3_536x147.png, https://substackcdn.com/image/fetch/$s_!QIzo!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fba1d0783-5826-48a3-9ead-1e803042e96d_622x631.png, Less Wrong, many of these tokens are the screen names of a group of Redditors who attempted to count to infinity, SERI-MATS, LLM that understands and can explain images
Candidate for worst ever ChatGPT answer
Inline links: Candidate for worst ever ChatGPT answer
ChatGPT users discover DAN Mode, example below: Something something alignment, something something nuclear arsenal.
Less than two years ago at the time of writing, the shocking successes of ChatGPT put many commentators in an awkward position. Beyond all the quibbling about details (Does ChatGPT really understand? Doesn’t it fail at many tasks trivial for humans? Could ChatGPT or something like it be conscious?), the brute empirical fact remains that it can handle language comprehension and generation pretty well. And this is despite the conception of language underlying it—language use as a statistical learning problem, with no sentence diagrams or grammatical transformations in sight—being somewhat antithetical to the Chomskyan worldview.
Statistical approaches succeeded where more directly-Chomsky-inspired approaches failed, and it was never close. Large language models (LLMs) like ChatGPT are not perfect, but they’re getting better all the time, and the onus is on the critics to explain where they think the wall is. It’s conceivable that a completely orthogonal system designed according to the principles of universal grammar could outperform LLMs built according to the current paradigm—but this possibility is becoming vanishingly unlikely.
Why do statistical learning systems handle language so well? If Everett is right, the answer is in part because (i) training models on a large corpus of text and (ii) providing human feedback both give models a rich collection of what is essentially cultural information to draw upon. People like talking with ChatGPT not just because it knows things, but because it can talk like them. And that is only possible because, like humans, it has witnessed and learned from many, many, many conversations between humans.