Astral Codex Ten

Nature

51 min read

Nature

Article

Nature is a recurring concept in the Astral Codex Ten archive, appearing 2 times across 2 issues between April 09, 2021 and February 23, 2022. The archive places it in contexts such as “Nature doesn’t make things without a purpose!”; “Paul’s table of how effectively Nature tends to outperform humans”. It most often appears alongside evolution, 1913 Nobel Prize in Physiology or Medicine, AGI.

Metadata

  • Category: Concepts
  • Mention count: 2
  • Issue count: 2
  • First seen: April 09, 2021
  • Last seen: February 23, 2022

Appears In

Source Context

Recovered passages from the original issue text. When the raw archive preserved outbound links inside the source passage, they are listed directly under the quote.

Your Book Review: On The Natural Faculties
April 09, 2021 · Original source
It’s hard not to notice just how famous Galen was in his own time. Marcus Aurelius described him as “primum sane medicorum esse, philosophorum autem solum” — first among doctors and unique among philosophers (one wonders if Galen might have influenced the Emperor’s own philosophy). Forgeries and unscrupulous editions of his work were such a problem during his lifetime, he had to write a book called On My Own Books to try to sort it all out. Among other things, he complains that his servants were stealing private letters he had written to friends and circulating bootleg copies of them as medical advice. Galen was an incredibly prolific writer. Wikipedia claims that he produced more works than any other author in antiquity, maybe up to 600 treatises, and possibly employed 20 scribes at one point. While these particular claims are hard to substantiate, he did leave behind a whole lot of books. Fires and the various other mishaps that are guaranteed to happen to classical texts destroyed many of his works. Some of this even happened during his own lifetime, and in On My Own Books he seems surprisingly relaxed about so many of his works being lost: The books of many others perished at that time, as did all those of mine which were located in that storehouse; and none of my friends in Rome admitted to having copies of the first two books. Since, then, my followers prevailed upon me to write the same treatise again, I thought that I should give this explanation regarding the previously distributed books, in case anyone in the future finds them and wonders why I should have written a treatise twice on the same subject. Even with these losses, huge amounts of his work has survived. It’s hard to get an exact count, but Claudii Galeni Opera Omnia by Karl Gottlob Kühn, compiled around 1833 and for a long time the definitive edition, contains 122 different works in 22 volumes. That’s a lot. Despite this, I was surprised how hard it was to get my hands on primary source copies of his works (in English). Because of our own plague, I was limited to finding sources online — but for most classical works, this is pretty easy. Marcus Aurelius was a contemporary of Galen, and it’s not too hard to find multiple different translations of Meditations (though admittedly Marcus may have a slightly wider appeal). Part of this might be that Galen’s works are very badly organized. Every secondary source I read on the Galenic corpus is full of griping about how confusing the whole thing is. Galen wrote in Greek, but many of the original versions of his books are lost, leaving us only with Arabic or Latin translations, or Latin translations of earlier Arabic translations. Some of the books appear under different titles in different places, and sometimes the works are only indexed under abbreviations of those titles. Some of them probably were never intended for publication (those bootleg letters I mentioned above), and so may not have official titles or versions at all. Forgeries of his works in various languages continued well on into the Renaissance. Galen himself was very unclear on how to think about the documents he produced. At one point in On My Own Books, he starts off by talking about a piece of writing he did “as an exercise for myself”, and then immediately turns around and mentions that he gave it to friends, who in turn gave it to their friends. Needless to say, the whole thing is a mess, the scholars seem very agitated. I chose to review the longest piece I could find, which is On the Natural Faculties, specifically the translation by Arthur John Brock, which was the only translation I was able to track down. This also seemed like a good choice because, instead of being a treatise on a more limited topic like diet, the pulse, or bones, this book serves as more of an introductory textbook to what today we would call biology. III. On the Natural Faculties is divided into three books, though if the three books have any structure to them, I wasn’t able to figure it out. Galen is pretty straightforward in naming his pieces, and this book is about him trying to describe all of the “natural faculties”. This doesn’t really correspond to any modern concept, but essentially he means the fundamental or basic biological functions common to all living things. He begins by contrasting the functions of the soul, like feeling and voluntary motion (we might say “mental functions”), which occur only in animals, with the natural functions common to both animals and plants. You could maybe translate “natural faculties” as something like “basic biological functions”. I had always heard that Galen was a Hippocrates stan, but right from the get-go he’s mentioning Aristotle in the very same breath (though he reminds us that Hippocrates “lived much earlier than Aristotle”). When describing the natural faculties, he seems to base them off of Aristotle’s physics more than Hippocrates’ humors. Aristotle’s physics is a system I mostly know secondhand from the descriptions offered by Thomas Kuhn (for an example, take a look at this piece). Kuhn stresses that this system is hard for a modern mind to understand and even harder to explain, so I was surprised at how intelligible Galen’s account is. Maybe reading Kuhn’s description prepared me to understand what Galen has to say, but either way, it’s great. I think Galen does a better job than Kuhn. Basically he says, look, there are different kinds of motion: If that which is white becomes black, or what is black becomes white, it undergoes motion in respect to colour; or if what was previously sweet now becomes bitter, or, conversely, from being bitter now becomes sweet, it will be said to undergo motion in respect to flavour … when a warm thing becomes cold, and a cold warm, here too we speak of motion; similarly also when anything moist becomes dry, or dry moist. He goes on to suggest that the natural faculties are more advanced forms of motion, possibly built up out of the combination of simpler forms of motion. (Kuhn treats the Aristotelian perspective as if it was the common sense of the ancient world, but the fact that Galen has to describe it in such detail makes me wonder if that was really the case.) That’s the framework. What the exact set of natural faculties are, however, is less clear. In book one he focuses on three faculties in particular — genesis, growth, and nutrition — and provides lots of arguments that (for example) the body’s ability to grow is different from its ability to sustain itself. In book three he gives a different list of four — the attractive, retentive, expulsive, and alterative faculties — but he also suggests that these are “handmaids of Nutrition”. Elsewhere he says that genesis is not “a simple activity of Nature” but instead is “compounded of alteration and of shaping.” He also mentions faculties like “adhesion” and “presentation”. The particulars are pretty confusing, but the general gist is clear. Galen wants to lay out all the different faculties and their sub-faculties (and sub-sub-faculties?) so that the reader can understand the workings of the body. Galen makes it pretty plain that he thinks that diseases are caused by failures or overactivity of the different principles. For example, he says that in leprosy “there is adhesion of the nutriment but no real assimilation”. One faculty is working but the other is disordered. If you want to be a good physician, he says, you need to understand all these faculties so you can identify diseases (tell what faculties are misfunctioning) and treat them — “how are you going to be successful in treatment, if you do not understand the real essence of each disease?” he says. The four humors do make their way into this mix eventually, especially in the second and third books. (Though the translator often insists on translating “humor” as “juice”, which makes me very uncomfortable.) The relationship seems to be that the humors are the building material of the body, but that all the activity is carried out through the natural faculties. The student needs to know the humors to understand what is being moved around, but the humors are primitive. To Galen, biology is all about these faculties shuffling, transforming, and combining different humors. VI. Anyways, that’s what Galen wants to be talking about. But about halfway through book one, he goes entirely off the rails and never really gets back on track. The thing that sets him off is other schools of medicine. It’s clear that Galen cannot stop thinking about them. They invade his every thought; he is beleaguered by them. I would seriously believe that he loses sleep over them. Some of the commentators I’ve read suggested that Galen was an arrogant man — one said he saw in Galen “the blind assumption that he alone was graced with the ability to bring Hippocrates’ work to completion”. My sense of Galen was that he is a man who is constantly exasperated. He is just trying to write basic pieces about how to be a good physician and philosopher, and people keep descending on him with the most unbelievably pedantic arguments. Book One of On the Natural Faculties is divided into 17 sections, and he spends half of the first section hedging around ways people could potentially take his words in the wrong ways. These sound more than a little like intrusive thoughts, and it’s tempting to think that he’s blowing this all out of proportion. But from what I know about Galen’s life, it seems likely that he really was getting into disagreements all the time, and probably really did need to worry about people quoting his work out of context. One article in The Lancet describes him as “a public figure, known and recognised by many, accosted in the streets, challenged to debate.” It’s easy to imagine how being accosted in the streets might work its way into your head. Either way, these concerns absolutely consume him. He keeps getting drawn off on different tangents, before trying to return to the main thread with statements like: I said, however, that I was not going to enter into an argument with these people, and it was only because the example was drawn from the subject-matter of medicine, and because I need it for the present treatise, that I have mentioned it. Let us pass on, then, again to another piece of nonsense; for the sophists do not allow one to engage in enquiries that are of any worth, albeit there are many such; they compel one to spend one’s time in dissipating the fallacious arguments which they bring forward. What, then, is this piece of nonsense? Now, we usually refrain from arguing with people whose principles are wrong from the outset. Still, having been compelled by the natural course of events to enter into some kind of a discussion with them, we must add this further to what was said… Since, then, we have talked sufficient nonsense — not willingly, but because we were forced, as the proverb says, “to behave madly among madmen” — let us return again to the subject of urinary secretion. But, as I have said, one is driven to talk nonsense whenever one gets into discussion with such men. Having, therefore, given a concise and summary statement of the matter, I wish to be done with it. Of course, in the very next paragraph, he is immediately drawn back into a discussion of their shortcomings! In some ways, On the Natural Faculties is less of a medical treatise and more of a fascinating snapshot of the state of the academic medical world in the latter half of the second century CE. The tone sounds really contemporary in a lot of ways, and has a quality of acrimonious quibbling that is more than a little familiar, though I don’t think modern physicians are likely to be poisoned by their colleagues (but what do I know). V. We’ve established that Galen has a problem with other experts and schools of medical thought. That leaves us wondering how justified he is. Is he criticizing them for real problems in their work, or is this just partisan squabbling? What are the things that he takes such issue with from these other schools? I think there are two things he’s mostly complaining about. The first thing that really sets Galen off is sectarian dogmatism. “Everyone becomes like the first teacher that he comes across,” he says, “without waiting to learn anything from anybody else.” He bemoans sectarian partisanship and, in classic doctor fashion, uses a weird hygiene metaphor, calling it “excessively resistant to all cleansing process”. It is “harder to heal than any itch”. The fact is that those who are enslaved to their sects are not merely devoid of all sound knowledge, but they will not even stop to learn! This is kind of tragicomic, because two of the main things Galen is accused of are 1) blindly following whatever Hippocrates said about medicine and 2) leading centuries of physicians to blindly follow whatever he wrote! It’s hard to know how blindly Galen is following the teachings of Hippocrates. On the one hand, he does refer to him as “most divine Hippocrates” at least once. On the other hand, he is open to pointing out the (rare) cases where he thinks Hippocrates has overlooked something, and even talks about how he wishes his opponents would criticize Hippocrates more directly! When someone disagrees with a whole suite of his intellectual heroes, he says, “now, one cannot be blamed for not agreeing with all these great men, nor for imagining that one knows more than they; but not to consider such distinguished teaching worthy either of contradiction or even mention shows an extraordinary arrogance.” Maybe other physicians really did follow Galen’s writing blindly in the centuries following his death. I’m not sure anymore. But Galen certainly can’t be blamed for it. He could not be clearer in stating that this is exactly what the student of medicine should avoid doing. It would be tempting to pass this all off as one-sided; “stop listening blindly to your teachers and listen blindly to me!” I don’t get that sense. First, we know that Galen studied all over the ancient world, so he was exposed to all sorts of ways of doing medicine. He practiced what he preached. It’s hard to know how fair a representation he’s giving of the other schools of thought, but he writes as though he has them all memorized, and he certainly was in a position to frequently get into debates with them. When he tells us that they’re uncritical, I’m tempted to believe him. Second, Galen makes a serious point to try to convince the reader of his positions. He’s not just stating “facts” and expecting you to bow down at his feet. He’s engaging with opposing points of view and trying to make compelling arguments that he thinks will convince his readers. VI. Finally, I don’t buy this because nowhere is Galen asking people to listen blindly to anyone, least of all himself. Because the second thing that REALLY sets Galen off is when people aren’t empirical enough! He constantly ridicules, in pretty harsh language, those who remain unconvinced by observation and experiment. Asclepiades, one particularly hated adversary, is charged with “bidding us distrust our senses where obvious facts plainly overturn his hypotheses.” Asclepiades has rather unusual opinions about the urinary system, and in one particularly strong example, Galen asks rhetorically (and sarcastically!), I do not suppose that Asclepiades ever saw a stone which had been passed by one of these sufferers, or observed that this was preceded by a sharp pain in the region between the kidneys and the bladder as the stone traversed the ureter, or that, when the stone was passed, both the pain and the retention at once ceased. It is worth while, then, learning how his theory account for the presence of urine in the bladder, and one is forced to marvel at the ingenuity of a man who puts aside these broad, clearly visible routes, and postulates others which are narrow, invisible—indeed, entirely imperceptible. Other schools are also attacked for denying “observed facts” or even “obvious facts”. Meanwhile, people who draw incorrect conclusions but respect the facts are praised. Galen cares a lot about physicians basing decisions on empirical observation. We know that he’s serious about this because of the many disturbing vivisection experiments he describes in great detail. In discussing digestion, he says, “I have personally, on countless occasions, divided the peritoneum of a still living animal and have always found all the intestines contracting peristaltically upon their contents.” He describes an experiment where you vivisect an animal, cutting away different coats of the esophagus, “then give the animal food and you will see that it still swallows although the peristaltic function has been abolished”. When describing the action of the stomach, he suggests that you can fill an animal with liquid food — “an experiment I have often carried out in pigs” — and cut them open “after three or four hours.” He really seems to want his readers to try these macabre exercises at home. “You may observe this yourself,” he says, “if you will try to hit upon the time at which the descent of food from the stomach takes place.” Fellow physicians are criticized for their lack of anatomical experience in the same way. “If he had ever practised anatomy, he might have known that the outer coat of the bladder springs from the peritoneum and is essentially the same as it.” The most extreme example comes from a debate with the disciples of Asclepiades about the function of the ureters, trying to convince this rival school that urine flows from the kidneys to the bladder through these channels. After exhausting his rhetorical options, Galen turns to empirical anatomy. First he shows them, in a dead animal, that the ureters connect the two structures. This isn’t enough. Next he shows them “in a still living animal, the urine plainly running out through the ureters into the bladder.” This doesn’t change their minds either. Next he takes a live animal, ligates the ureters, bandages the animal up, and lets it go. When he opens it up again later, he finds the ureters “quite full and distended”, and when he removes the ligature, everyone can see the urine flow into the bladder. You’d think the story would end there, but not so. Instead, says Galen, “tie a ligature round [the animal’s] penis and then … squeeze the bladder all over.” He points out that nothing goes back through the ureters to the kidneys, demonstrating that the conveyance is a special, one-way action. He goes on like this for a while. Let the animal urinate and tie a ligature around one ureter but not the other. Cut open both the ureters and see the urine “spurt out of it”. Bandage the animal up and open him up later to discover his insides full of urine and the bladder empty. “Now, if anyone will but test this for himself on an animal,” Galen concludes, “I think he will strongly condemn the rashness of Asclepiades.” Today we know that Galen was wrong, and that humorism isn’t a great way to think about medicine. But whatever Galen might have been lacking, it certainly was not the empirical bent. He was no armchair philosopher, and was more than happy to cut up lots of animals to make a point about the function of the ureters. This is funny because, again, this is the opposite of the story we’re told about Galen. He’s described as a pre-scientific or even unscientific thinker, believing that experimentation and investigation are a waste of time. Clearly this isn’t the case, and he made full use of all the resources available to him. We know that human dissection was prohibited in the empire, but Galen worked with gladiators, so we know that he had firsthand experience with human anatomy. He certainly was unafraid, even eager, to practice animal dissection and vivisection. Other doctors of the time didn’t seem to do either of these things, or at least didn’t do nearly as much, and so Galen starts looking more and more like a lone light of empiricism in the wilderness. (However extreme and disturbing his methods may be.) VII. In view of this, it’s extremely depressing to see Tetlock write, “yet Galen never conducted anything resembling a modern experiment.” Galen isn’t here to respond, but if he were, I imagine he would say: and yet Tetlock never conducted anything resembling a basic literature review! Galen definitely isn’t as charitable as we might want him to be. He calls some of the ideas he disagrees with “impossible, nay, perfectly nonsensical”, or “stupid—I might say insane”. His intellectual rivals “are like slaves” he says, “caught in the act of stealing … quite bewildered, and while the one says nothing, the other indulges in shameless lying.” But I’m pretty sympathetic to Galen’s position, because his contemporaries really do sound like idiots. Of course, all this is being filtered through Galen’s own account, but if he’s describing them with any accuracy, he is totally fair in saying that they have no idea what they are talking about. Some of the positions he argues against include: Urine passes into the bladder in the form of vapors, rather than being secreted by the kidneys and passed through the ureters to the bladder. Galen argues against this first by pointing out that the kidneys and bladder are connected by the ureters (which must have some purpose), and second by the extensive evidence from vivisection that I mentioned above.
Inline links: this piece, https://substackcdn.com/image/fetch/$s_!6Gdb!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2Fbb854d87-de36-4173-ab8a-2728bd8605b5_1024x635.jpeg
In making this argument, Galen first points out that the idea that nature has “powers which attract” is sufficient to explain the observations. Next he says that since the proposed entanglement can’t be observed, it’s not clear why anyone would prefer this explanation to another. Even if we do allow it, he says, it doesn’t explain why a piece of iron that has previously touched a lodestone will then go on to attract other pieces of iron. The atomic explanation predicts that this shouldn’t happen!
This strikes me as a very modern way to make an argument. It’s how I would make an argument today — none of the intellectual arsenal seems to be missing! In fact, Galen uses thought experiments quite frequently. There’s one in particular he loves, about children making balloons out of the bladders of pigs, which he uses a couple times to demonstrate that only Nature has the ability to actually make things grow, rather than just distend them.
Biological Anchors: A Trick That Might Or Might Not Work
February 23, 2022 · Original source
Source: This document by Paul Christiano. Ajeya combines this with another metric where they see how existing AI compares to animals with apparently similar computational capacity; for example, she says that DeepMind’s Starcraft engine has about as much inferential compute as a honeybee and seems about equally subjectively impressive. I have no idea what this means. Impressive at what? Winning multiplayer online games? Stinging people? In any case, they decide to penalize AI by one order of magnitude compared to Nature, so a human-level AI would need to do 10^16 floating point operations per second. How Much Compute Would It Take To Train A Model That Does 10^16 Floating Point Operations Per Second? So an AI could potentially equal the human brain with 10^16 FLOP/S. Good news! There’s a supercomputer in Japan that can do 10^17 FLOP/S! It looks like this (source) So why don’t we have AI yet? Why don’t we have ten AIs? In the modern paradigm of machine learning, it takes very big computers to train relatively small end-product AIs. If you tried to train GPT-3 on the same kind of medium-sized computers you run it on, it would take between tens and hundreds of years. Instead, you train GPT-3 on giant supercomputers like the ones above, get results in a few months, then run it on medium-sized computers, maybe ~10x better than the average desktop. But our hypothetical future human-level AI is 10^16 FLOP/S in inference mode. It needs to run on a giant supercomputer like the one in the picture. Nothing we have now could even begin to train it. There’s no direct and obvious way to convert inference requirements to training requirements. Ajeya tries assuming that each parameter will contribute about 10 FLOPs, which would mean the model would have about 10^15 parameters (GPT-3 has about 10^11 parameters). Finally, she uses some empirical scaling laws derived from looking at past machine learning projects to estimate that training 10^15 parameters would require H*10^30 FLOPs, where H represents the model’s “horizon”. If I understand this correctly, “horizon” is a reinforcement learning concept: how long does it take to learn how much reward you got for something? If you’re playing a slot machine, the answer is one second. If you’re starting a company, the answer might be ten years. So what horizon do you need for human level AI? Who knows? It probably depends on what human-level task you want the AI to do, plus how well an AI can learn to do that task from things less complex than the entire task. If writing a good book is mostly about learning to write good sentence and then stringing them together, a book-writing AI can get away with a short horizon. If nothing short of writing an entire book and then evaluating it to see whether it is good or bad can possibly teach you book-writing, the AI will need a long time horizon. Ajeya doesn’t claim to have a great answer for this, and considers three models: horizons of a few minutes, a few hours, and a few years. Each step up adds another three orders of magnitude, so she ends up with three estimates of 10^30, 10^33, and 10^36 FLOPs. (for reference, the lowest training estimate - 10^30 - would take the supercomputer pictured above 300,000 years to complete; the highest, 300 billion.) Or What If We Ignore All Of That And Do Something Else? This is piling a lot of assumptions atop each other, so Ajeya tries three other methods of figuring out how hard this training task is. Humans seem to be human-level AIs. How much training do we need? You can analogize our childhood to an AI’s training period. We receive a stream of sense-data. We start out flailing kind of randomly. Some of what we do gets rewarded. Some of what we do gets punished. Eventually our behavior becomes more sophisticated. We subject our new behavior to reward or punishment, fine-tune it further. Rent asks us: how do you measure the life of a woman or man? It answers: “in daylights, in sunsets, in midnights, in cups of coffee; in inches, in miles, in laughter, in strife.” But you can also measure in floating point operations, in which case the answer is about 10^24. This is actually trivial: multiply the 10^15 FLOP/S of the human brain by the ~10^9 seconds of childhood and adolescence. This new estimate of 10^24 is much lower than our neural net estimate of 10^30 - 10^36 above. In fact, it’s only a hair above the amount it took to train GPT-3! If human-level AI was this easy, we should have hit it by accident sometime in the process of making a GPT-4 prototype. Since OpenAI hasn’t mentioned this, probably it’s harder than this and we’re missing something. Probably we’re missing that humans aren’t blank slates. We don’t start at zero and then only use our childhood to train us further. The very structure of our brain encodes certain assumptions about what kinds of data we should be looking out for and how we should use it. Our training data isn’t just what we observed during childhood, it’s everything that any of our ancestors observed during evolution. How many floating-point operations is the evolutionary process? Ajeya estimates 10^41. I can’t believe I’m writing this. I can’t believe someone actually estimated the number of floating point operations involved in jellyfish rising out of the primordial ooze and eventually becoming fish and lizards and mammals and so on all the way to the Ascent of Man. Still, the idea is simple. You estimate how long animals with neurons have been around for (10^16 seconds), total number of animals at any given second (10^20) times average number of FLOPS per animal (10^5) and you can read more here but it comes out to 10^41 FLOs. I would not call this an exact estimate - for one thing, it assumes that all animals are nematodes, on the grounds that non-nematode animals are basically a rounding error in the grand scheme of things. But it does justify this bizarre assumption, and I don’t feel inclined to split hairs here - surely the total amount of computation performed by evolution is irrelevant except as an extreme upper bound? Surely the part where Australia got all those weird marsupials wasn’t strictly necessary for the human brain to have human-level intelligence? One more weird human training data estimate attempt: what about the genome? If in some sense a bit of information in the genome is a “parameter”, how many parameters does that suggest humans have, and how does it affect training time? Ajeya calculates that the genome has about 7.5x10^8 parameters (compared to 10^15 parameters in our neural net calculation, and 10^11 for GPT-3). So we can… Okay, I’ve got to admit, this doesn’t have quite the same “huh?!” factor as trying to calculate the number of FLOs in evolution, but it is in a lot of ways even crazier. The Japanese canopy plant has a genome fifty times larger than ours, which suggests that genome size doesn’t correspond very well to organism awesomeness. Also, most of the genome is coding for weird proteins that stabilize the shape of your kidney tubule or something, why should this matter for intelligence? The Japanese canopy plant. I think it is very pretty, but probably low prettiness per megabyte of DNA. I think Ajeya would answer that she’s debating orders of magnitude here, and each of these weird things costs only a few OOMs and probably they all even out. That still leaves the question of why she thinks this approach is interesting at all, to which she answers that: The motivating intuition is that evolution performed a search over a space of small, compact genomes which coded for large brains rather than directly searching over the much larger space of all possible large brains, and human researchers may be able to compete with evolution on this axis. So maybe instead of having to figure out how to generate a brain per se, you figure out how to generate some short(er) program that can output a brain? But this would be very different from how ML works now. Also, you need to give each short program the chance to unfold into a brain before you can evaluate it, which evolution has time for but we probably don’t. Ajeya sort of mentions these problems and counters with an argument that maybe you could think of the genome as a reinforcement learner with a long horizon. I don’t quite follow this but it sounds like the sort of thing that almost might make sense. Anyway, when you apply the scaling laws to a 7.5*10^8 parameter genome and penalize it for a long horizon, you get about 10^33 FLOPs, which is weirdly similar to some of the other estimates. So now we have six different training cost estimates. First, neural nets with short, medium, and long horizons, which are 10^30, 10^33, and 10^36 FLOPs, respectively. Next, the amount of training data in a human lifetime - 10^24 FLOs - and in all of evolutionary history - 10^41 FLOPs. And finally, this weird genome thing, which is 10^33 FLOPs. An optimist might say “Well, our lowest estimate is 10^24 FLOPs, our highest is 10^41 FLOPs, those sound like kind of similar numbers, at least there’s no “5 FLOPs” or “10^9999 FLOPs” in there. A pessimist might say “The difference between 10^24 and 10^41 is seventeen orders of magnitude, ie a factor of 100,000,000,000,000,000 times. This barely constrains our expectations at all!” Before we decide who to trust, let’s remember that we’re still only at Step 2 of our eight step Methodology, and continue. How Do We Adjust For Algorithmic Progress? So today, in 2022 (or in 2020 when this was written, or whenever), assume it would take about 10^33 FLOs to train a human-level AI. But technology constantly advances. Maybe we’ll discover ways to train AIs faster, or run AIs more efficiently, or something like that. How does that factor into our estimate? Ajeya draws on Hernandez & Brown’s Measuring The Algorithmic Efficiency Of Neural Networks. They look at how many FLOPs it took to train various image recognition AIs to an equivalent level of performance between 2012 and 2019, and find that over those seven years it decreased by a factor of 44x, ie training efficiency doubles every sixteen months! Ajeya assumes a doubling time slightly longer than that, because it’s easier to make progress in simple well-understood fields like image recognition than in the novel task of human-level AI. She chooses a doubling time of “merely” 2 - 3 years. If training efficiency doubles every 2-3 years, it would dectuple in about 10 years. So although it might take 10^33 FLOPs to train a human level AI today, in ten years or so it may take only 10^32, in twenty years 10^31, and so on. When Will Anyone Have Enough Computational Resources To Train A Human-Level AI? In 2020, AI researchers could buy computational resources at about $1 for 10^17 FLOPs. That means the 10^33 FLOPs you’d need to train a human-level AI would cost $10^16, ie ten quadrillion dollars. This is about twenty times more money than exists in the entire world. But compute costs fall quickly. Some formulations of Moore’s Law suggest it halves every eighteen months. These no longer seem to hold exactly, but it does seem to be halving maybe once every 2.5 years. The exact number is kind of controversial: Ajeya admits it’s been more like once every 3-4 years lately, but she heard good things about some upcoming chips and predicted it might revert back to the longer-term faster trend (it’s been two years now, some new chips have come out, and this prediction is looking pretty good). So as time goes on, algorithmic progress will cut the cost of training (in FLOPs), and hardware progress will also cut the cost of FLOPs (in dollars). So training will become gradually more affordable as time goes on. Once it reaches a cost somebody is willing to pay, they’ll buy human-level AI, and then that will be the year human-level AI happens. What is the cost that somebody (company? government? billionaire?) is willing to pay for human-level AI? The most expensive AI training in history was AlphaStar, a DeepMind project that spent over $1 million to train an AI to play StarCraft (in their defense, it won). But people have been pouring more and more money into AI lately: Source here. This is about compute rather than cost, but most of the increase seen here has been companies willing to pay for more compute over time, rather than algorithmic or hardware progress. The StarCraft AI was kind of a vanity project, or science for science’s sake, or whatever you want to call it. But AI is starting to become profitable, and human-level AI would be very profitable. Who knows how much companies will be willing to pay in the future? Ajeya extrapolates the line on the graph forward to 2025 and gets $1 billion. This is starting to sound kind of absurd - the entire company OpenAI was founded with $1 billion in venture capital, it seems like a lot to expect them to spend more than $1 billion on a single training run. So Ajeya backs off from this after 2025 and predicts a “two year doubling time”. This is not much of a concession. It still means that in 2040 someone might be spending $100 billion to train one AI. Is this at all plausible? At the height of the Manhattan Project, the US was investing about 0.5% of its GDP into the effort; a similar investment today would be worth $100 billion. And we’re about twice as rich as 2000, so 2040 might be twice as rich as we are. At that point, $100 billion for training an AI is within reach of Google and maybe a few individual billionaires (though it would still require most or all of their fortune). Ajeya creates a complicated function to assess how much money people will be willing to pay on giant AI projects per year. This looks like an upward-sloping curve. The line representing the likely cost of training a human-level AI looks like a downward sloping curve. At some point, those two curves meet, representing when human-level AI will first be trained. So When Will We Get Human-Level AI? The report gives a long distribution of dates based on weights assigned to the six different models, each of which has really wide confidence intervals and options for adjusting the mean and variance based on your assumptions. But the median of all of that is 10% chance by 2031, 50% chance by 2052, and almost 80% chance by 2100. Ajeya takes her six models and decides to weigh them like so, based on how plausible she thinks each one is: 20% neural net, short horizon 30% neural net, medium horizon 15% neural net, long horizon 5% human lifetime as training data 10% evolutionary history as training data 10% genome as parameter number She ends up with this: How Sensitive Is This To Changes In Assumptions? She very helpfully gives us a Colab notebook and Google spreadsheet to play around with. The notebook lets you change some of the more detailed parameters of the individual models, and the spreadsheet lets you change the big picture. I leave the notebook to people more dedicated to forecasting than I am, and will talk about the spreadsheet here. If you’re following along at home, the default spreadsheet won’t reflect Ajeya’s findings until you fill in the table in the bottom left like so: Great. Now that we’ve got that, let’s try changing some stuff. I like the human childhood training data argument (Lifetime Anchor) more than Ajeya does, and I like the size-of-the-genome argument less. I’m going to change the weights to 20-20-0-20-20-20. Also, Ajeya thinks that someone might be willing to spend 1% of national GDP on training AIs, but that sounds really high to me, so I’m going to down to 0.1%. Also, Ajeya’s estimate of 3% GDP growth sounds high for the sort of industrialized nations who might do AI research, I’m going to lower it to 2%. Since I’m feeling mistrustful today, let’s use the Hernandez&Brown estimate for compute halving (1.5 years) in place of Ajeya’s ad hoc adjustments. And let’s use the current compute halving time (3.5 years) instead of Ajeya’s overly rosy version (2.5 years). All these changes… …don’t really do much. The median goes from 2052 to about 2065. Four of the models give results between 2030 and 2070. The last two, Neural Net With Long Horizon and Evolution, suggest probably no AI this century (although Neural Net With Long Horizon does think there’s a 40% chance by 2100). Ajeya doesn’t really like either of these models and they’re not heavily weighted in her main result. Does The Truth Point To Itself? Back up a second. Here’s something that makes me kind of nervous. Most of Ajeya’s numbers are kind of made up, with several order-of-magnitude error bars and simplifying assumptions like “all animals are nematodes”. For a single parameter, we get estimates spanning seventeen different orders of magnitude: the upper bound is one hundred quadrillion times the lower bound. And yet four of the six models, including two genuinely exotic ones, manage to get dates within twenty years of 2050. And 2050 is also the date everyone else focuses on. Here’s the prediction-market-like site Metaculus: Their distribution looks a lot like Ajeya’s, and even has the same median, 2052 (though forecasters could have read Ajeya’s report). Katja Grace et al surveyed 352 AI experts, and they gave a median estimate of 2062 for an AI that could “outperform humans at all tasks” (though with many caveats and high sensitivity to question framing). This was before Ajeya’s report, so they definitely didn’t read it. So lots of Ajeya’s different methods and lots of other people presumably using different methodologies or no methodology at all, all converge on this same idea of 2050 give or take a decade or two. An optimist might say “The truth points to itself! There are 371 known proofs of the Pythagorean Theorem, and they all end up in the same place. That’s because no matter what methodology you use, if you use it well enough you get to the correct answer.” A pessimist might be more suspicious; we’ll return to this part later. FLOPS Alone Turn The Wheel Of History One more question: what if this is all bullshit? What if it’s an utterly useless total garbage steaming pile of grade A crap? Imagine a scientist in Victorian Britain, speculating on when humankind might invent ships that travel through space. He finds a natural anchor: the moon travels through space! He can observe things about the moon: for example, it is 220 miles in diameter (give or take an order of magnitude). So when humankind invents ships that are 220 miles in diameter, they can travel through space! Ships have certainly grown in size tremendously, from primitive kayaks to Roman triremes to Spanish galleons to the great ocean liners of the (Victorian) present. The AI forecasting organization AI Impacts actually has a whole report on historical ship size trends to prove an unrelated point about technological progress, so I didn’t even have to make this graph up. Suppose our Victorian scientist lived in 1858, right when the Great Eastern was launched. The trend line for ship size crossed 100m around 1843, and 200m in 1858, so doubling time is 15 years - but perhaps they notice this is going to be an outlier, so let’s round up a bit and say 18 years. The (one order of magnitude off estimate for the size of the) Moon is 350,000m, so you’d need ships to scale up by 350,000/200 = 1,750x before they’re as big as the Moon. That’s about 10.8 doublings, and a doubling time is 18 years, so we’ll get spaceships in . . . 2052 exactly. (fudging numbers to land where you want is actually fun and easy) SS Great Eastern, the extreme outlier large steamship from 1858. This has become sort of a mascot for quantitative technological progress forecasters. What is this scientist’s error? The big one is thinking that spaceship progress depends on some easily-measured quantity (size) instead of on fundamental advances (eg figuring out how rockets work). You can make the same accusation against Ajeya et al: you can have all the FLOPs in the world, but if you don’t understand how to make a machine think, your AI will be, well, a flop. Ajeya discusses this a bit on page 143 of her report. There is some sense in which FLOPs and knowing-what-you’re-doing trade of against each other. If you have literally no idea what you’re doing, you can sort of kind of re-run evolution until it comes up with something that looks good. If things are somehow even worse than that, you could always run AIXI, a hypothetical AI design guaranteed to get excellent results as long as you have infinite computation. You could run a Go engine by searching the entire branching tree structure of Go - you shouldn’t, and it would take a zillion times more compute than exists in the entire world, but you could. So in some sense what you’re doing, when you’re figuring out what you’re doing, is coming up with ways to do already-possible things more efficiently. But that’s just algorithmic progress, which Ajeya has already baked into her model. (our Victorian scientist: “As a reductio ad absurdum, you could always stand the ship on its end, and then climb up it to reach space. We’re just trying to make ships that are more efficient than that.”) Part II: Biology-Inspired AI Timelines: The Trick That Never Works Eliezer Yudkowsky presents a more subtle version of these kinds of objection in an essay called Biology-Inspired AI Timelines: The Trick That Never Works, published December 2021. Ajeya’s report is a 169-page collection of equations, graphs, and modeling assumptions. Yudkowsky’s rebuttal is a fictional dialogue between himself, younger versions of himself, famous AI scientists, and other bit players. At one point, a character called “Humbali” shows up begging Yudkowsky to be more humble, and Yudkowsky defeats him with devastating counterarguments. Still, he did found the field, so I guess everyone has to listen to him. He starts: in 1988, famous AI scientist Hans Moravec predicted human-level AI by 2010. He was using the same methodology as Ajeya: extrapolate how quickly processing power would grow (in FLOP/S), and see when it would match some estimate of the human brain. Moravec got the processing power almost exactly right (it hit his 2010 projection in 2008) and his human brain estimate pretty close (he says 10^13 FLOP/S, Ajeya says 10^15, this 2 OOM difference only delays things a few years), yet there was not human-level AI in 2010. What happened? Ajeya's answer could be: Moravec didn't realize that, in the modern ML paradigm, any given size of program requires a much bigger program to train. Ajeya, who has a 35-year advantage on Moravec, estimates approximately the same power for the finished program (10^16 vs. 10^13 FLOP/S) but says that training the 10^16 FLOP/S program will require 10^33ish FLOPs. Eliezer agrees as far as it goes, but says this points to a much deeper failure mode, which was that Moravec had no idea what he was doing. He was assuming processing power of human brain = processing power of computer necessary for AGI. Why? The human brain consumes around 20 watts of power. Can we thereby conclude that an AGI should consume around 20 watts of power, and that, when technology advances to the point of being able to supply around 20 watts of power to computers, we'll get AGI? […] You say that AIs consume energy in a very different way from brains? Well, they'll also consume computations in a very different way from brains! The only difference between these two cases is that you know something about how humans eat food and break it down in their stomachs and convert it into ATP that gets consumed by neurons to pump ions back out of dendrites and axons, while computer chips consume electricity whose flow gets interrupted by transistors to transmit information. Since you know anything whatsoever about how AGIs and humans consume energy, you can see that the consumption is so vastly different as to obviate all comparisons entirely. You are ignorant of how the brain consumes computation, you are ignorant of how the first AGIs built would consume computation, but "an unknown key does not open an unknown lock" and these two ignorant distributions should not assert much internal correlation between them. Cars don’t move by contracting their leg muscles and planes don’t fly by flapping their wings like birds. Telescopes do form images the same way as the lenses in our eyes, but differ by so many orders of magnitude in every important way that they defy comparison. Why should AI be different? You have to use some specific algorithm when you’re creating AI; why should we expect it to be anywhere near the same efficiency as the ones Nature uses in our brains? The same is true for arguments from evolution, eg Ajeya’s Evolutionary Anchor, ie “it took evolution 10^43 FLOPs of computation to evolve the human brain so maybe that will be the training cost”. AI scientists sitting in labs trying to figure things out, and nematodes getting eaten by other nematodes, are such different methods for designing things that it’s crazy to use one as an estimate for the other. Algorithmic Progress vs. Algorithmic Paradigm Shifts This post is a dialogue, so (Eliezer’s hypothetical model of) OpenPhil gets a chance to respond. They object: this is why we put a term for algorithmic progress in our model. The model isn’t very sensitive to changes in that term. If you want you can set it to some kind of crazy high value and see what happens, but you can’t say we didn’t consider it. OpenPhil: We did already consider that and try to take it into account: our model already includes a parameter for how algorithmic progress reduces hardware requirements. It's not easy to graph as exactly as Moore's Law, as you say, but our best-guess estimate is that compute costs halve every 2-3 years […] Eliezer: The makers of AGI aren't going to be doing 10,000,000,000,000 rounds of gradient descent, on entire brain-sized 300,000,000,000,000-parameter models, algorithmically faster than today. They're going to get to AGI via some route that you don't know how to take, at least if it happens in 2040. If it happens in 2025, it may be via a route that some modern researchers do know how to take, but in this case, of course, your model was also wrong. They're not going to be taking your default-imagined approach algorithmically faster, they're going to be taking an algorithmically different approach that eats computing power in a different way than you imagine it being consumed. OpenPhil: Shouldn't that just be folded into our estimate of how the computation required to accomplish a fixed task decreases by half every 2-3 years due to better algorithms? Eliezer: Backtesting this viewpoint on the previous history of computer science, it seems to me to assert that it should be possible to: Train a pre-Transformer RNN/CNN-based model, not using any other techniques invented after 2017, to GPT-2 levels of performance, using only around 2x as much compute as GPT-2;
Inline links: This document, a supercomputer in Japan, https://substackcdn.com/image/fetch/$s_!svqA!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2Fd5924b1f-a563-4332-b137-ff9dda5580d0_1240x516.jpeg, source, here, Japanese canopy plant, https://substackcdn.com/image/fetch/$s_!gj-T!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2F333dcbf2-1f63-42a1-821f-94f39818e62d_1280x897.jpeg, Measuring The Algorithmic Efficiency Of Neural Networks, https://substackcdn.com/image/fetch/$s_!dX1J!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2Fb9496f1f-ec6c-41a2-8c2e-27f09da22097_1280x759.png, here, https://substackcdn.com/image/fetch/$s_!LnC0!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2F62d647ff-58ed-4e9a-9f1a-7febf5859249_1152x842.png, Colab notebook, Google spreadsheet, https://substackcdn.com/image/fetch/$s_!BND-!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2F622bac28-eaa6-40b5-b93b-695952966ef7_744x324.png, https://substackcdn.com/image/fetch/$s_!lbos!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2F7d5c2306-a123-4903-adb9-d961d56ebfb5_1152x842.png, Metaculus, https://substackcdn.com/image/fetch/$s_!SMnF!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2F807f66de-8c5c-4423-b293-ca92b5b64053_763x360.png, surveyed 352 AI experts, https://substackcdn.com/image/fetch/$s_!JxQ5!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2Fceba6aa0-dbde-41ca-805e-01af4fac9324_769x336.png, a whole report on historical ship size trends, https://substackcdn.com/image/fetch/$s_!PRDj!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2Fde3d97f4-afca-45c4-9ed2-521cd25041df_460x262.jpeg, AIXI, Biology-Inspired AI Timelines: The Trick That Never Works
SS Great Eastern, the extreme outlier large steamship from 1858. This has become sort of a mascot for quantitative technological progress forecasters. What is this scientist’s error? The big one is thinking that spaceship progress depends on some easily-measured quantity (size) instead of on fundamental advances (eg figuring out how rockets work). You can make the same accusation against Ajeya et al: you can have all the FLOPs in the world, but if you don’t understand how to make a machine think, your AI will be, well, a flop. Ajeya discusses this a bit on page 143 of her report. There is some sense in which FLOPs and knowing-what-you’re-doing trade of against each other. If you have literally no idea what you’re doing, you can sort of kind of re-run evolution until it comes up with something that looks good. If things are somehow even worse than that, you could always run AIXI, a hypothetical AI design guaranteed to get excellent results as long as you have infinite computation. You could run a Go engine by searching the entire branching tree structure of Go - you shouldn’t, and it would take a zillion times more compute than exists in the entire world, but you could. So in some sense what you’re doing, when you’re figuring out what you’re doing, is coming up with ways to do already-possible things more efficiently. But that’s just algorithmic progress, which Ajeya has already baked into her model. (our Victorian scientist: “As a reductio ad absurdum, you could always stand the ship on its end, and then climb up it to reach space. We’re just trying to make ships that are more efficient than that.”) Part II: Biology-Inspired AI Timelines: The Trick That Never Works Eliezer Yudkowsky presents a more subtle version of these kinds of objection in an essay called Biology-Inspired AI Timelines: The Trick That Never Works, published December 2021. Ajeya’s report is a 169-page collection of equations, graphs, and modeling assumptions. Yudkowsky’s rebuttal is a fictional dialogue between himself, younger versions of himself, famous AI scientists, and other bit players. At one point, a character called “Humbali” shows up begging Yudkowsky to be more humble, and Yudkowsky defeats him with devastating counterarguments. Still, he did found the field, so I guess everyone has to listen to him. He starts: in 1988, famous AI scientist Hans Moravec predicted human-level AI by 2010. He was using the same methodology as Ajeya: extrapolate how quickly processing power would grow (in FLOP/S), and see when it would match some estimate of the human brain. Moravec got the processing power almost exactly right (it hit his 2010 projection in 2008) and his human brain estimate pretty close (he says 10^13 FLOP/S, Ajeya says 10^15, this 2 OOM difference only delays things a few years), yet there was not human-level AI in 2010. What happened? Ajeya's answer could be: Moravec didn't realize that, in the modern ML paradigm, any given size of program requires a much bigger program to train. Ajeya, who has a 35-year advantage on Moravec, estimates approximately the same power for the finished program (10^16 vs. 10^13 FLOP/S) but says that training the 10^16 FLOP/S program will require 10^33ish FLOPs. Eliezer agrees as far as it goes, but says this points to a much deeper failure mode, which was that Moravec had no idea what he was doing. He was assuming processing power of human brain = processing power of computer necessary for AGI. Why? The human brain consumes around 20 watts of power. Can we thereby conclude that an AGI should consume around 20 watts of power, and that, when technology advances to the point of being able to supply around 20 watts of power to computers, we'll get AGI? […] You say that AIs consume energy in a very different way from brains? Well, they'll also consume computations in a very different way from brains! The only difference between these two cases is that you know something about how humans eat food and break it down in their stomachs and convert it into ATP that gets consumed by neurons to pump ions back out of dendrites and axons, while computer chips consume electricity whose flow gets interrupted by transistors to transmit information. Since you know anything whatsoever about how AGIs and humans consume energy, you can see that the consumption is so vastly different as to obviate all comparisons entirely. You are ignorant of how the brain consumes computation, you are ignorant of how the first AGIs built would consume computation, but "an unknown key does not open an unknown lock" and these two ignorant distributions should not assert much internal correlation between them. Cars don’t move by contracting their leg muscles and planes don’t fly by flapping their wings like birds. Telescopes do form images the same way as the lenses in our eyes, but differ by so many orders of magnitude in every important way that they defy comparison. Why should AI be different? You have to use some specific algorithm when you’re creating AI; why should we expect it to be anywhere near the same efficiency as the ones Nature uses in our brains? The same is true for arguments from evolution, eg Ajeya’s Evolutionary Anchor, ie “it took evolution 10^43 FLOPs of computation to evolve the human brain so maybe that will be the training cost”. AI scientists sitting in labs trying to figure things out, and nematodes getting eaten by other nematodes, are such different methods for designing things that it’s crazy to use one as an estimate for the other. Algorithmic Progress vs. Algorithmic Paradigm Shifts This post is a dialogue, so (Eliezer’s hypothetical model of) OpenPhil gets a chance to respond. They object: this is why we put a term for algorithmic progress in our model. The model isn’t very sensitive to changes in that term. If you want you can set it to some kind of crazy high value and see what happens, but you can’t say we didn’t consider it. OpenPhil: We did already consider that and try to take it into account: our model already includes a parameter for how algorithmic progress reduces hardware requirements. It's not easy to graph as exactly as Moore's Law, as you say, but our best-guess estimate is that compute costs halve every 2-3 years […] Eliezer: The makers of AGI aren't going to be doing 10,000,000,000,000 rounds of gradient descent, on entire brain-sized 300,000,000,000,000-parameter models, algorithmically faster than today. They're going to get to AGI via some route that you don't know how to take, at least if it happens in 2040. If it happens in 2025, it may be via a route that some modern researchers do know how to take, but in this case, of course, your model was also wrong. They're not going to be taking your default-imagined approach algorithmically faster, they're going to be taking an algorithmically different approach that eats computing power in a different way than you imagine it being consumed. OpenPhil: Shouldn't that just be folded into our estimate of how the computation required to accomplish a fixed task decreases by half every 2-3 years due to better algorithms? Eliezer: Backtesting this viewpoint on the previous history of computer science, it seems to me to assert that it should be possible to: Train a pre-Transformer RNN/CNN-based model, not using any other techniques invented after 2017, to GPT-2 levels of performance, using only around 2x as much compute as GPT-2;
Inline links: AIXI, Biology-Inspired AI Timelines: The Trick That Never Works
I find myself most influenced by two things. First, Paul’s table of how effectively Nature tends to outperform humans, which I’ll paste here again: