Astral Codex Ten

Holden Karnofsky

13 min read

Holden Karnofsky

Article

Holden Karnofsky is a recurring person in the Astral Codex Ten archive, appearing 6 times across 6 issues between July 23, 2021 and March 14, 2023. The archive places it in contexts such as “Holden Karnofsky, co-founder of GiveWell and CEO of the Open Philanthropy Project, now has a blog, Cold Takes”; “you can read Holden Karnofsky’s pitch for doing it”; “OpenPhil CEO Holden Karnofsky’s article”. It most often appears alongside Eliezer Yudkowsky, AGI, AI Impacts.

Metadata

  • Category: People
  • Mention count: 6
  • Issue count: 6
  • First seen: July 23, 2021
  • Last seen: March 14, 2023

Appears In

Source Context

Recovered passages from the original issue text. When the raw archive preserved outbound links inside the source passage, they are listed directly under the quote.

Links For July
July 23, 2021 · Original source
19: Holden Karnofsky, co-founder of GiveWell and CEO of the Open Philanthropy Project, now has a blog, Cold Takes, on “futurism, macrohistory, applied epistemology and ethics, [and] sometimes sports”. Getting to hear from Holden is always a privilege, usually one reserved for people at effective altruism organizations or conferences, and it’s exciting to see he’ll be sharing his thoughts more widely.
Inline links: Cold Takes
Open Thread 207
January 17, 2022 · Original source
You can read about the contest here, and you can read Holden Karnofsky’s pitch for doing it (and attempt to summarize the question) here.
Inline links: here, here
Biological Anchors: A Trick That Might Or Might Not Work
February 23, 2022 · Original source
There's a small cottage industry of summarizing the report already, eg OpenPhil CEO Holden Karnofsky's article and Alignment Newsletter editor Rohin Shah's comment. I've drawn from both for my much-inferior attempt.
Inline links: article, comment
I’m split on what to think here. If we consider a weaker version of Platt’s Law, “the average date at which people forecast AGI moves forward at about one year per year”, this seems truish in the big picture where we compare 1960 to today, but not obviously true after 1980. If we consider a different weaker version, “on average estimates tend to be 30 years away”, that’s true-ish under Barnett’s revised model, but not inherently damning since Barnett’s assuming there will be some such number, it turns out to be 25, and Ajeya gave the somewhat different number of 32. Is that a big enough difference to exonerate her of “using” Platt’s Law? Is that even the right way to be thinking about this question? Response 3: Real OpenPhil The hypothetical OpenPhil in Eliezer’s mind having been utterly vanquished, the real-world OpenPhil is forced to step in. OpenPhil CEO Holden Karnofsky responds to Eliezer here. There’s a lot of back and forth about whether the report includes enough caveats (answer: it sure does include a lot of caveats!) but I was most interested in the attacks on Eliezer’s two main points. First, the point that biological anchors are fatally flawed from the start and measuring FLOP/S is no better than measuring power consumption in watts. Holden: If the world were such that: We had some reasonable framework for "power usage" that didn't include gratuitously wasted power, and measured the "power used meaningfully to do computations" in some important sense;
Inline links: here
Just eyeballing it, Platt’s Law looks pretty good. But Holden Karnofsky (see below) objects that our eyeballs are covertly removing outliers. Barnett agrees this is worth checking for and runs a formal OLS regression. Platt’s Law in blue, regression line in orange. He writes: I agree this trendline doesn't look great for Platt's law, and backs up your observation by predicting that Bio Anchors should be more than 30 years out. However, OLS is notoriously sensitive to outliers. If instead of using some more robust regression algorithm, we instead super arbitrarily eliminated all predictions after 2100, then we get this, which doesn't look absolutely horrible for the law. Note that the median forecast is 25 years out. I’m split on what to think here. If we consider a weaker version of Platt’s Law, “the average date at which people forecast AGI moves forward at about one year per year”, this seems truish in the big picture where we compare 1960 to today, but not obviously true after 1980. If we consider a different weaker version, “on average estimates tend to be 30 years away”, that’s true-ish under Barnett’s revised model, but not inherently damning since Barnett’s assuming there will be some such number, it turns out to be 25, and Ajeya gave the somewhat different number of 32. Is that a big enough difference to exonerate her of “using” Platt’s Law? Is that even the right way to be thinking about this question? Response 3: Real OpenPhil The hypothetical OpenPhil in Eliezer’s mind having been utterly vanquished, the real-world OpenPhil is forced to step in. OpenPhil CEO Holden Karnofsky responds to Eliezer here. There’s a lot of back and forth about whether the report includes enough caveats (answer: it sure does include a lot of caveats!) but I was most interested in the attacks on Eliezer’s two main points. First, the point that biological anchors are fatally flawed from the start and measuring FLOP/S is no better than measuring power consumption in watts. Holden: If the world were such that: We had some reasonable framework for "power usage" that didn't include gratuitously wasted power, and measured the "power used meaningfully to do computations" in some important sense;
Inline links: https://substackcdn.com/image/fetch/$s_!54Vh!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2F1c354075-ecaa-4807-a1a5-07931736f093_403x268.png, writes, https://substackcdn.com/image/fetch/$s_!dw02!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2F797aef17-dc24-4845-9e00-2c3fd7f7dc32_403x268.png, here
Contra Hoel On Aristocratic Tutoring
March 22, 2022 · Original source
He argues the most likely cause is the decline of “aristocratic tutoring” - an educational method typical among the ultra-rich of the past - and its replacement with normal public (or private) schools. The answer must lie in education somewhere [...] paradoxically there exists an agreed-upon and specific answer to the single best way to educate children, a way that has clear, obvious, and strong effects. The problem is that this answer is unacceptable. The superior method of education is deeply unfair and privileges those at the very top of the socioeconomic ladder. It’s an answer that was well-known historically, and is also observed by education researchers today: tutoring. […] Let us call [the] past form aristocratic tutoring, to distinguish it from a tutor you meet in a coffeeshop to go over SAT math problems while the clock ticks down. It’s also different than “tiger parenting,” which is specifically focused around the resume padding that’s needed for kids to meet the impossible requirements for high-tier colleges. Aristocratic tutoring was not focused on measurables. Historically, it usually involved a paid adult tutor, who was an expert in the field, spending significant time with a young child or teenager, instructing them but also engaging them in discussions, often in a live-in capacity, fostering both knowledge but also engagement with intellectual subjects and fields. He amply proves that many of the great geniuses of the past, including Bertrand Russell, Albert Einstein, and John von Neumann received tutoring like this, and suggests that its absence (more because of strengthening democratic norms than because people don’t have the money) might be why we don’t see figures of their stature anymore. II. I agree that this kind of tutoring sounds great. I wouldn’t be surprised if it has a big effect size. But it’s not the reason we have fewer geniuses. Why not? Suppose that half of past geniuses were tutored this way, and half weren’t. Even if every single genius who was tutored owed his genius entirely to the tutoring, the tutoring could only explain half of geniuses. That means that after the tutoring stopped, we would expect half as many geniuses. But Hoel is making a stronger claim: that there are almost no geniuses today. For aristocratic tutoring to explain that, we would need for almost all past geniuses to be aristocratically tutored. But as far as I can tell, that isn’t true. Probably well below half of them were. Just to give some examples: Isaac Newton went to a local school at at 12, and to Cambridge at 17. The Wikipedia page on his early life doesn't mention "tutor", except in the context of a college teacher. His adopted father was a country parson, and his family wasn't rich enough to do aristocratic tutoring even if they'd wanted to. Articles on his early life stress his self-motivated nature: he was constantly building things and observing things on his own time. Wolfgang Mozart was tutored, but primarily by his father, himself an excellent violinist. According to his Wikipedia article, "In his early years, Wolfgang's father was his only teacher". Mozart was already an obvious child prodigy by 6 or 7, and wrote his first symphony at 8. I can't find any evidence that non-family members contributed to his education. This kind of tutoring is still common; my wife learned cello from her grandmother, a professional music tutor. Charles Darwin went to a local school at age 8, switched to a boarding school at 9, spent a summer at age 16 following his father (a doctor) around as he treated patients, then went to medical school. He switched to regular college at Cambridge at 19, where he seemed to have a pretty traditional education. Wikipedia has a long article on his education, which doesn't mention the word "tutor" until college age, when he "spent the autumn term at home studying Greek with a tutor". Later in college, he "joined other Cambridge friends on a three-month "reading party" at Barmouth on the coast of Wales to revise their studies with private tutors". I don't think he had a stronger relationship with being tutored himself, especially not in childhood. His summer following his father around learning medicine was probably good for him, but not outside the bounds of what still happens today (I followed my father around learning medicine). Louis Pasteur was born "to a Catholic family of a poor tanner". He went to primary school at 8 and college at 16. I can't find any evidence he was tutored. Charles Dickens barely seems to have been educated at all. His family was so poor that he spent some of his childhood working in a sweatshop. During other periods they did a little better and he went to small lower-to-middle-class private schools. Dickens seems to have gotten most of his education by reading novels on his own. Thomas Edison grew up poor in Michigan. Again according to Wikipedia, "Edison was taught reading, writing, and arithmetic by his mother, who used to be a school teacher. He attended school for only a few months. However, one biographer described him as a very curious child who learned most things by reading on his own. As a child, he became fascinated with technology and spent hours working on experiments at home." Hoel argues that the decline in aristocratic tutoring is “why we stopped making Einsteins”. But then why did we stop making Newtons, Mozarts, Darwins, Pasteurs, Dickenses, and Edisons? III. One other argument: Hoel cites Holden Karnofsky’s Where’s Today’s Beethoven?, which suggests that music is a typical case of the genius decline. But aristocratic tutoring in music is alive and well. When my brother was identified as a piano prodigy, my (well-off but not absurdly rich) parents hired jazz musician Linda Martinez to tutor him. I asked around and this is apparently pretty common in music. In fact, it seems common across a variety of fields, especially those that aren’t taught in school and where success doesn’t make you too rich to need tutoring money (a friend brings up chess as another example). If aristocratic tutoring were a significant factor behind declining genius, we would expect to see a split: fields like science where tutoring is rare would lose their geniuses, whereas fields like music where tutoring is common would be as genius-filled as ever. But people use music as a typical example of a declining-genius field. So that can’t be it. IV. So what’s my explanation? You will not be surprised to hear it’s the maximally boring one, a combination of: Good ideas are getting harder to find. In 300 BC, if you noticed that the water level in your bathtub got higher when you got into it, you were allowed to run through the streets shouting “eureka!” and declare yourself to be a genius. Now you would need some 400 page mathematical proof drawing on the topology of eight-dimensional manifolds in order to get that kind of cred.
Inline links: Isaac Newton, Wolfgang Mozart, Charles Darwin, Louis Pasteur, Charles Dickens, Thomas Edison, Where’s Today’s Beethoven?, Linda Martinez, Good ideas are getting harder to find
OpenAI's "Planning For AGI And Beyond"
March 01, 2023 · Original source
Reading even further between the lines - at this point it’s total guesswork - OpenAI’s corporate partner Microsoft asked them for a cool AI. OpenAI assumed Microsoft was competent - they make Windows and stuff! - and gave them a rough draft of GPT-4. Microsoft was not competent, skipped fine-tuning and many other important steps which OpenAI would not have skipped, and released it as the Bing chatbot. Bing got in trouble for threatening users, which gave OpenAI a PR headache around safety. Some savvy alignment people chose this moment to approach them with their latest ideas (is it a coincidence that Holden Karnofsky published What AI Companies Can Do Today earlier that same week?), and OpenAI decided (for a mix of selfish and altruistic reasons) to get on board - hence this document.
Inline links: What AI Companies Can Do Today
Why I Am Not (As Much Of) A Doomer (As Some People)
March 14, 2023 · Original source
Holden Karnofsky, on a somewhat related question, gives 50%
Inline links: gives 50%