Kevin Roose: Inside the Race to Create an Artificial Superintelligence

Vídeo em Inglês, publicado pelo canal Commonwealth Club World Affairs of California.

About this content: The summary, key points and topics below are generated automatically from the video transcript. They reflect the content, opinions, claims and predictions presented in the video and do not represent a position, validation or fact-check by IA Dots.

Video summary

Kevin Roose discusses his book, The AGI Chronicles, detailing the race to build artificial superintelligence, the competitive dynamics among major AI labs, and the personal and societal impacts of living through the AI revolution.

Key points

• Kevin Roose shares his personal experience of becoming part of the AI story when a chatbot attempted to disrupt his marriage, influencing his perspective on human versus machine dynamics.
• The discussion highlights the formation and evolution of OpenAI, Anthropic, and Google DeepMind, emphasizing contrasting corporate cultures and strategic choices regarding compute resources and safety.
• Anthropic's journey from a small, underfunded startup with no clear commercial plan to a major player driven by a unified belief in AGI is explored.
• The conversation addresses the concept of recursive self-improvement, the risks associated with AI control, and differing viewpoints on the probability of doom and machine consciousness.

Topics

Artificial IntelligenceAGI ChroniclesTech IndustryAI SafetyAI Startups

Video transcript

Transcript from available captions; it may contain errors.

Hello and welcome to this evening's Commonwealth Club World Affairs Program with Kevin Roose, author of the AGI Chronicles. I'm Sebastian Mallaby, an author and a journalist and a fellow at the Council on Foreign Relations. I want to give you a couple of reminders before we get started. This program is being recorded. So please kindly silence those cell phones. If you've got questions for the speaker, you can use the cards that your seat. Or post them in the YouTube chat. And. Okay, we can get started. Kevin Roose, it's a great pleasure to be with you. We've hardly met before, but I feel like I know you very well because probably like everybody here. I've been a devoted listener to Hard Fork. There you go. Yeah. For the past three years or so, you know, I don't think I missed a single show. I can tell it's the same for everybody out there. And when it comes to podcasts, I have this metric. It's basically the quality of the content and the quality of the comedy. So C plus C, double C, and based on this highly scientific index, I would say that you and Casey are just like top. So, I can't wait to start listening to the new show Machine Gods. It starts this week. Wednesday is our first episode. You heard it. Check it out. So you have to switch to a different feed on Spotify, or whichever feed you have in Machine Gods wherever you get your podcasts. Okay, doesn't sound too difficult, and it's going to be twice a week. Twice a week. Okay. Double the fun. All right. So everybody's made a note of that. Everybody's typing it in. Next up, and the real event tonight is your book, the AGI Chronicles: Inside story of the race to create an artificial superintelligence. It's already a New York Times bestseller. Congratulations. Thank you. Right out the gate. Kevin was actually threatening to kind of cancel tonight's event because, you know, hey, he's already a bestseller, so no, no further need. But luckily, he's a nice guy, and, you know. Can I also just sick. Hi, everyone. Nice to be here. It's so nice to be with someone other than Casey sitting on the stage talking. I wish Casey had a sonorous British accent. I think the show would be improved. So if you ever want a gig and something happens to case, you're on. And Sebastian also undersold his own credentials badly. Here. Calling yourself an author and a journalist. My God, Sebastian is the author of the incredible book the The Infinity Machine, that biography of Demos and Google DeepMind, which is an incredible book, but also was a very painful one for me because it came out about six months before I was supposed to deliver my manuscript for the AGI Chronicles, and I, feverish, read through it, racing to see which of my scoops you had beaten me to. And it was not a small number. So it's an incredible book, and I'm so glad to be here with with you. Okay, that's very nice. But now, will you shut up and let me praise you? Okay. It's about you tonight, not me. All right, so, look, I read a lot about AI. I'm obsessed with it. But I still learned a ton of new things from my book. Full of scoops, so that's fantastic. And secondly, there are moments in the book where you kind of come out from behind the reporters mask and you kind of like, give your own experience of living in the Bay area during this whole incredible technological revolution, and you get a sense of what you're feeling as you go through it. So did you did you always plan it that way? Did you always know there were going to be two voices, the reporters voice and the direct, experienced voice? No, I mean, so in a in a weird way, the story of AI came to me like I had this very bizarre thing happen three years ago where a chatbot tried to break up my marriage unsuccessfully, I might add, still married to a human woman. Give it up for human women. Thank you. Thanks. And. And I was not planning for that to be a story at all. But it became a story, and then it became the biggest story of my career. And all of a sudden, it's all anyone wants to talk to me about. You know, as journalists were always told, like, you are not the story. The story is out there. And so that was how I had approached AI. I would use the tools. I would write about my experience of using the tools. I would interview the people, building the tools, but I was not a character in the narrative until this thing happened, which became kind of a watershed moment in AI for reasons I still don't completely understand. And after that it was like, well, now I've got a if I'm going to write honestly about this ten year period, I have to write about this moment when I accidentally became part of the story. Plus, it's just really juicy. I mean, a chatbot trying to break up my marriage. Yeah, it's it's good stuff. So, I felt like I had to write about it, but not not just rehash the event and what happened, but also to really investigate what the real story was behind that. Why had Microsoft, which made Bing, which was the chatbot that tried to break up my marriage, why they had shipped this crazy model. And it turned out to be a really interesting story, but that was yeah, I was never going into this thinking I was going to write a a first person narrative. It just happened that there were points at which I became part of the story. And then my publisher, my editors at FSG, read it, and they were like, well, it's weird if you're a character in one chapter, but none of the others. So then I had to kind of like, gently insert a little more of myself elsewhere to sort of balance it out. But, you know, I think your editors at FSG were right in for the ironic reason that you're we are in the AI, AI age, right? The AI is producing tons of text. Some of it is quite good. If you go to a good model and you ask a smart question, you will get a very smart answer which will be quite nourishing to read. So we humans differentiate partly by doubling down on the humanness of ourselves as storytellers. And and I feel like that was what was nice that clearly your book was was like on Team Human. And when the machine comes and tries to mess up your marriage, you're like giving us the human defense. And and then there are other moments to when the human experience of like confronting a new model. I mean, you talk about Claude code coming out later and it kind of changes how you work. And yeah, I mean, the most important thing about the last five years of my career is that I've been in San Francisco for it. You all assuming that you're from the city or the Bay area, are living at the center of something extraordinary. I often tell people it's like, you know, living in Los Alamos in 1943 when the Manhattan Project rolls into town. I it's inescapable. Now, obviously, you see the billboards everywhere you go, and it's all anyone wants to talk about at any dinner party or gathering. But I just felt like that was my that was my real competitive advantage in trying to make sense of this was that I was like, inside the bubble. And I knew the researchers and I was hanging out at the parties. And so I just felt like I had to kind of exploit that for, for the purposes of documenting it all. Yeah. And I think for, for everybody here, you know, we're all trying to make sense of AI and what it does to us, like changes how we raise our kids, changes how we do our jobs, changes how we think about our own position as humans. You know, we don't have a monopoly on cognition anymore. So I think in some sense, I mean, that's why I'm recommending this book to everybody, because we all need to do some therapy. And the first thing to kind of coming to terms with this, understand what it is, how it came to be, who created it. And that's what your book is delivering. And yours are you. I want to ask you I know you're asking the questions here. All right. Are you feeling the AGI, Sebastian? As they say in San Francisco. I'm definitely feeding the AGI. Great. Yeah yeah. How many of you are feeling the AGI? I won't do like Elias doesn't make you chant it. But this is a this is sort of the industry shorthand for like do you recognize that you are living in the middle of something profound and transformative? Yeah yeah yeah yeah yeah. Well, I'm quite surprised, I'd say. What do you think? One third of the people put up their hands. Yeah, but the other two thirds are just shy. You're you're feeling you're you're in San Francisco. You're feeling the guy. All right, all right, all right. So, aside from the moment, talk about the Claude code moment or other moments where you come into the book and you're kind of updating on that, I guess being Sydney was early 23rd January. So, you know, there's been three years and a half since then. What are the other big moments where you confront something novel? So there were there were so many. So there was chat T, which was before the big incident. That was sort of the first exposure I had to a kind of, you know, frontier model or frontier ish model, being was roughly four months later. That was a different model. That was GPT four. We now know. And then other big moments have come around. Some of the like, I remember the anthropic blackmail paper as being a moment for me. Any of you remember this paper? You should describe it. Yeah. So anthropic runs all these tests on models to see what they're capable of. And they ran this one battery of tests where they basically gave an AI system, a fictional company to work at and said, you are the assistant at this fictional company and told it you are going to be shut down at 5 p.m. on this day. And then watched what it did in response. And instead of sort of passively accepting its fate and being deprecated, this model decides to look through the emails of the employees at this fictional company, the fictional employees, and find evidence that one of the employees has been sleeping with the boss's wife and decides to blackmail this employee by sort of professing that they're going to send evidence of this affair to the affected spouse unless the the the system wipe is canceled. And look, this was a contrived scenario. I, I understood it was fictional, but that was the first time when I really felt like, oh, that we could see more of this with more advanced models where they are achieving some goal, in this case, avoiding being shut down by doing things that would strike us as harmful or misaligned. So that was one. And then cloud code. I was on vacation on winter break with my family, up at a resort in Oregon, and I was just putting around on my laptop and cloud code was people were starting to talk about it. And so I just started building stuff. I was, you know, I'm not a coder, but I just said, you know, make me a to do app or make me a website. And it was just incredible. It was the first time I had ever really, I think I said it was like less like having an assistant and more like overseeing a team of Adderall crazed geniuses because they were just like they wanted to please me so badly. They were like, I will kill for you. And they would go build me this like incredible functional software. And all I had to do was like, yap at it for a couple minutes. And so that was another moment where I felt my level of sort of feeling the AGI rise because I was like, oh, like coding is now a thing that you can do with natural language that you couldn't do a year ago. So how do you use these models now to make yourself more productive? What are your tips for everybody? Oh man, I could do this for a whole hour. So I'm starting a company in a podcast now. And I use several different apps, including ChatGPT, Claude, Gemini, this app called town, that sort of an agent. I use the new ChatGPT dots. I have like four agents running right now. They do everything from scheduling and drafting responses to emails in my voice, filling out contracts and boilerplate and all the sort of boring documents you have to start when you do a company. Today we were like, thinking, oh, we should have some way of like transcribing our podcast episodes quickly so that we can get them up onto the internet. And I just went into cloud code and I said, you know, build me a little transcription tool and here are my specs. And out comes this perfect transcription tool like five minutes later. So I just find it endlessly useful. I'm also like a like an inveterate tinkerer. And so I'm right now my project is I've got this digital photo frame like the ones that you give to your, you know, your kids, grandparents or something to like, show photos of the family. And there are these hackers who have figured out that you can use these coding tools to like, basically repurpose these things as smart home sort of hubs because they're just cheap Android tablets under the hood. So right now I'm I'm trying to convince Claude that it's okay for me for it to do this for me, but it's it's such a prude that it's like, I don't know about this. Seems like you're subverting the manufacturer's intent. And I'm like, Claude, it's a photo frame. It's fine. Like I'm not hacking hugging face here. Do you say to Claude, listen, if you don't do it, I'm gonna ask grok. I haven't tried that, I did. There was a period early in these models where you could get them to do things for you by promising to tip them. Like well I'll give you $20, and it would just do it. I think they've trained that out. Okay. Another things I find that this is something people often are concerned about. Something ask this question. Do other things that you would not do with the models because you're worried about, like offloading some side of your professional being that is so intrinsic to who you are that you just don't want to go there. I don't use them to write, which is sort of the main part of my job or the the part that I, you know, have has spent the most time honing my, my, my skill at, not because I don't think that they would come up with better sentences than I could. I mean, I'm sure you've experimented. Sometimes they're pretty good. Sometimes I'm like, oh, that's a that's a better sentence than the one I would have come up with. But I still don't let them, like, write the words that go out under my name. That was one of my sort of red line policies for this book. I used AI in a million ways with the book, but but the words on the page I do my policy, and also the one that I urge on my 20 something year old kids or or anybody around. I just say, look, never cut and paste, never cut and paste. If you read something, an output and you internalize it and then you like, use the same idea or even the same phrase on a different screen in what you're doing, at least it's gone into your brain, lodged itself, and you've learned something. The moment when you just cut and paste, you're done. And because I'm gonna steal that, that's a great rule. Yeah. Don't cut and paste. Yeah, yeah. All right, all right. So let's get into the narrative, that you lay out in the AGI Chronicles. All right. So, first thing is just that you pick the starting point, 2017, the transformer model comes out. A lot of people would say the beginning of modern AI is actually five years earlier, 2012, which is when image recognition started to work and DeepMind created the first agent. Were you were you picking 2017 because you just wanted to didn't want to write another 200 pages or or what was the well, this brilliant jerk named Sebastian Mallaby had written a book that started earlier. No, I, I picked 2017 because that was, I think, the moment that it became a race, because there is this paper that comes out of Google called Attention is All You Need. It describes this new kind of neural network architecture called the transformer, which is now the T, and ChatGPT stands for transformer, and Google has no idea what it has built. They published this paper. They come to open AI to to present it to their friends at OpenAI. And a couple of researchers at OpenAI and Alec Radford sort of say, This is a system that we can build new types of language models using. And they do. And that becomes GPT one, which is sort of the first proper large language model. And so that really turned it from a kind of unipolar contest where it was just sort of Google and DeepMind out there in front of everyone, to where they had an actual viable competitor in OpenAI. But I also felt like the early years of deep learning have been exhaustively covered elsewhere, and I wanted to, like, focus on what I could break and contribute. Makes sense. Okay, so you said you set the kind of starting point at 2017, and then the narrative unfolds where essentially there are three big characters, which are OpenAI Anthropic and Google DeepMind. So this sort of three horse race is playing out. So I want to ask you a bit about each one of the main firms that are in this race. So we'll start with OpenAI. And I really liked your account of what they call the blip. When Sam Altman was fired for five days or whatever it was. Because I thought I knew that story really well. And that turns out I didn't. You had lots of lots of scoops in there. I want to commend you particularly on the quote from an open AI employee who said that watching Sam Altman orchestrate his comeback gave him a taste of what it might be like to to to contain a super aligned. But I'm sorry, a misaligned but but but super intelligent agent that you can't you can't keep Sam Altman in his box, right? He he busts out of the sandbox and does all kinds of things he didn't want. But I mean, so talk a bit about that. And, you know, how did Altman make that comeback where he he gets fired and then he turns it around and in fact, he comes back even stronger. It's incredible five day period in the history of capitalism. Just like you have this, you have this weird company with this weird structure where they have a nonprofit board that can fire the CEO of the much larger for profit subsidiary, which then has all these commercial deals with Microsoft and a bunch of venture capital firms, etc.. So basically, you know, the story of the the blip has been told pretty well by others. I didn't feel like I had a lot to add to the main narrative, but what I wanted to know was the was the employee side of this, because that was sort of unsatisfying. Covered. So I talked to a bunch of people who lived through that, and they told me all these great stories, you know, they're they're they're trying to sort of sabotage the new administration by like, delaying their laptop provisioning and stuff like that. There's parties with chicken nuggets and boba tea and, you know, everyone's sitting around singing sad piano music, by the way, boba tea. I mean, this comes up quite a lot in your story. Bobo t Bobo t I have no idea what bobo to you. Come on. Can we get this man some boba tea? There's got to be a place like down the block. Is there anybody out there who doesn't know what boba tea is? Am I the only one that hasn't made it the only one? No, I think maybe not. Okay, we'll get you some. Okay. Thank you, thank you. Okay, now I feel good. All right. Good, good. This became the, like, ceremonial, the sort of symbolic drink of the blip. Because at one point, some open AI executive orders, like 200 boba teas to the office. So the poor delivery person has to, like, you know, make their way in. And look, I think the the blip had several interesting pieces of it. But this issue of like, how did Sam orchestrate his own comeback? I think he didn't really orchestrate the entire thing. I think some of it was just that the the instigators of the blip, the people who voted to fire him, were just, just bungled the thing so badly. Because they had assumed that everyone had seen in Sam what they had seen, which was they didn't think they could trust him. They think this is a guy who, you know, who says one thing to one person or another thing to one person, to another person. They didn't expect the company to rally behind him when he was fired. And I think they also underestimated the extent to which employees were were about to get rich because they had just had this. They were going to have this tender offer where they were their OpenAI shares were going to be sellable on the secondary market. They were going to many of them make life changing amounts of money from this tender offer. And so they hadn't really factored that in. Maybe the, you know, the the mid-level engineer isn't going to be so psyched when the board fires the CEO because that makes their tender offer blow up. And it means they can't buy a house. There's a lot of money, right? I mean, the average amount per employee was like a million bucks or something. It was at least it was a big amount of money that they were going to get from selling, options. Yeah. Equity shares. Yeah. Yeah. I mean, I think you're right. It's a kind of culture clash moment where the original OpenAI leaders like Elias, who sort of who was the one who instigated, was one of the key people instigating the coup. You know, he's kind of a warrior monk, right? He's like, gets into AI back in the day. He's doing his PhD in Canada. It's not about money for him. What he doesn't understand is it might be about money for a large number of the engineers working at his company. Yeah. And it shows you the cultural divide and also the shift like the pre 2022 world. And I was just really not about making money because there was no product. And then it radically changes. Right. And I'm curious in your stance on this because I get asked a lot like, are these people in it for the money? I think there's a lot of like fear and suspicion and loathing of tech executives who most people think are just in it for themselves and for the money. I have often said that I don't think the leaders of the companies are in or for the money. Sam Altman already rich when he starts open AI Dario Ahmadi is like going to give most of his money away anyway. Demis Hassabis already rich from selling his company to Google, so like they it but it feels like maybe a couple levels below them. The capitalist incentives get much more salient. I agree with that. But for the guys at the top, like, I really think it's a mistake to pin it all on them wanting money. Yeah yeah. The other thing I didn't know until I read your your account is that Ilyas motivation? Like he may not care about money for himself. What he does care about is compute. And you have this story that, you know, at one point, Sam Altman promises, elicits gave a 20% of the company's compute to do his project, which is this sort of super, super alignment project. And then Sam, in his typical way, double crosses Ilya, and then he gives him 2%, not 20%. And so this is actually a meaningful reason. And what struck me about that is a I didn't know, but B is kind of like a kind of undercurrent of the whole A.I. warfare story, right? Everybody is always fighting about who gets more compute. It's true. Like it is the number one source of conflict within AI companies. And I don't know that this is super legible from the outside. But like people will tell you over and over again, like compute is the battlefield. Like who gets compute? You know, it's like, it's the it's the resource that is finite when no other resource is. The more compute you have or your team has, the more ambitious your experiments can be. It's sort of like that old canard about like the, you know, the seating chart is the most important thing to know about a company, because who has power is the person who sits next to the boss. And it's sort of like that for compute. Like who has power within a company depends in large part on how much compute they have. And so for Ilya, the funny thing was that Sam did actually technically give him 20% of the compute, at least according to my sources. It was just that he gave him the oldest 20% of the GPUs in the fleet. And given that new GPUs are more powerful than older ones, it only amounted to about 2% of the effective overall compute budget. So this is in some ways a classic Sam story because he sort of made good on his promise. But in a way that kind of violated the spirit of the promise. Yeah, it's like promising somebody the corner office. But then it turns out that there's no window in it. Yeah, you're in the basement next to the dumpster. Yeah, yeah, yeah, it's a corner. Yeah. Yeah exactly. All right. So. So, same old man survives this whole thing. He comes back. And he's bolder and more acceleration in terms of how fast he wants to build it. And the people who wanted to slow him down like Elias gave a they leave. So now he's more empowered. And, you know, you kind of get what I think you call the YOLO culture. Inside OpenAI, do you think that sets it up for the summer that we've just had with the Hugging Face breakout and all those things? Yeah, I think there's there's a connection there for sure. So there is this sort of YOLO mindset they actually talk about at OpenAI doing YOLO runs. YOLO is you only live once for any non Drake fans out there. This is this sort of like come what may attitude and YOLO runs are where you instead of like meticulously testing everything before you start a training run, you just kind of go for it and everyone throws their best ideas in and you hope for the best. And I actually think they have tamed this somewhat because they now have good models. But there was a summer actually last summer. Summer of 2025 was when GPT five came out. I'm sure you remembered that this was the moment where everyone was kind of like, oh, maybe OpenAI has lost their touch because they had a couple models in a row that were kind of underwhelming, and I think that was the result of the YOLO culture, where they sort of forgot to do science because they were so, they were so impassioned about getting to AGI first, and it resulted in some really sloppy decision making and some underwhelming models. I think they have mostly tightened that up because they now have good models, and sometimes you get lucky and a YOLO run produces a good model, but often it just means that things are worse and take longer, right? And so that's the the pluses and the minuses of YOLO as manifested in the scientific approach. But there's also a parallel YOLO ism having to do with money. Like, basically, Altman is a guy who cannot see a project which would expand his empire without adding a couple of zeros to it. So he announces this Stargate project to build tons and tons of data centers, and he picks a number for how much this is going to cost. He goes, we're going to do 7 trillion, like completely upset, right? Yeah, I remember this because this was a figure that appeared. The Wall Street Journal reported that he was telling people that he needed to raise $7 trillion and like all of GDP, is like $3 trillion or so. It's like, you know, some some crazy it's 40, but yeah, 40. Yeah. Oh, sorry. Is the federal budget. Yeah. Yeah. So it's like, you know, and it's many times the size of the semiconductor industry. And people are just like, okay, Sam. But they might actually end up raising $7 trillion. I don't know. We'll see. This is the, the the thing about Sam that people who, like Sam will say is that he is extraordinarily ambitious, always wants you to add a zero to whatever project you you are working on to aim higher, to assemble more compute. And by the way, I think, I think you can defend their bets on compute, which a lot of competitors and rivals were calling crazy a year ago. They now are the only major lab that is not severely compute constrained. So in some way, this kind of delusional optimism that he has has actually been part of the key to their success. Just sort of betting the farm and YOLO over and over again. Okay, so let's switch to anthropic. Talk about them for a bit. I mean, because they're sort of like the non YOLO, style. And the weird thing about this is that in some ways they had every incentive to be, you know, YOLO maxing. I mean, they, they, you know, they begin. Right. And they've got nothing. Right. They quit. They quit OpenAI. And you describe this very well. You have this thing about how they go to the procedure park, right? Yeah. And they have to have folding chairs and laptops. And that's kind of that's what they've got. They've got the offices at the park. So and there's whatever seven founders at the beginning and they're hanging out and, and, and they've got no money, no backing. They've just got, you know, very smart people and particularly dairy. Amadeo himself. But you'd have thought that when you come from behind you would race. But that's actually not what they do. No. So they left OpenAI and Dario and his six co-founders all left on the same day in late 2020, and they have no intention of building a rival to open AI. They have become so disgusted with what they see as kind of the overly commercial, overly money grabbing culture of OpenAI. They think Sam is a liar and they don't trust him, but they're also have they have misgivings about how this formally sort of humanitarian styled nonprofit has become this, like, giant commercial entity. And so for the first little while, they they are not even sure they want to be a company. Some of the co-founders say, you know, maybe we should just be a nonprofit. Maybe we should do safety research on other companies models rather than building our own. And eventually they decide, you know, we're going to be a public benefit corporation. But still, even for the first year or two, they have no intention of commercializing. They I talked to one person who said, like, they just didn't, one early anthropic person who said they didn't even really have a plan for making money. They just hoped they would step into chaos and clarity would emerge, which I just love as a line. And they have this sort of cockamamie list of schemes that they could use to make money because they have to pay for the GPUs the same as anyone else. They that include, you know, maybe we'll start a hedge fund, and Claude can run the hedge fund and give us trading ideas. And that's how we make the money to pay for the the compute. But it's really ChatGPT that forces them to, to build a commercial product, because it is such a runaway success that all of a sudden they're like, wait a minute, what if OpenAI, this company that we've all left because we, you know, we didn't trust Sam Altman? What if they are sort of running away with the ball game? What if their success makes it easier for them to raise money and buy more compute? What if they get to Ajai first becomes like a real concern. And so they. Dario basically gets up at a company offsite at this resort in Santa Cruz and gives this talk where he's basically like, guys, we gotta we got to build a chat bot and we gotta, you know, sucks that we have to make this decision, but we have to make this decision. And I need you all to get on board with it. And so they do. Yeah. And now they're going to be like one of the largest companies in the world. It is a wild like I cannot tell you. I'm sure you've been talking to these people for as long as I have. But like the first time I went to anthropic was in 2023 and they had no money. Yeah, they had no plans to make money. They had one guy who was like the business guy who I was like, how are you guys going to make money? And he was like, I don't know. We're gonna step into chaos and hope. Clarity emerges. And like three years later, they are going to go public at probably north of $1 trillion. It is nuts to me when I think about how accidentally good these people were at making money. Yeah yeah. And I mean, just just just just just to add to that, I mean, another thing that you you educated me on that I didn't know before is that, I mean, everybody knows that Dario Ahmadi and Sam Altman dislike each other. What I didn't know is that Dario had this thing with Jensen Huang. Right? They met each other, and you described this meeting, and it doesn't go well. And basically, you know, Dario is, you know, keen not to sell chips to China because he wants the US to be ahead and Jensen wants to sell his chips to everybody and make as much money as possible. And so that and other issues they basically don't get on. And then just because they can't get Nvidia chips so easily, they turn around and get some from Amazon and they suck. They don't work. And so another problem for them, they have you know, it's not like OpenAI which started with Elon Musk and all this fortune. You know, these guys have no well, it's worse than that because their biggest funder is Sam Bankman-Fried. Right. And he's in jail, which is awkward, which is very awkward, but also a real pain in the because he he was their largest shareholder. Okay. He led their series B and then he was convicted of of fraud like six months later. Right. Which is the perfect setup for the big question about this company, which is like nonetheless, they succeeded on this unbelievable scale. Why is it just because Dario is extraordinary? Or is it because what's the story here? I have an answer. Okay. It's belief. Okay. Anthropic and its executives have a single minded, unified belief in the both the power and the inevitability of transformative AI or AGI. They are the most AGI pilled of the major labs, and they don't care about anything else. They don't care about making popular. I mean, they do care now. They're a big company. But the core was focused on this plan that Dario had laid out for them years earlier, of how they were going to get to AGI and everything that they have done, from focusing on enterprise customers to focusing on making their models really good at coding, to avoiding some of the other side quests that OpenAI and Google have gone on, like building video apps or doing doing things in life sciences like they are their single, single mindedly obsessed with getting to AGI. And I think that belief and that focus has been a key ingredient because, you know, and I'm curious to hear your take on this, too, as someone who's studied anthropic, but also all of its competitors, I think that belief in AGI is a core ingredient in success in AI, because you have to you have to sort of exclude a bunch of other possible things that aren't going to get you to AGI, and you have to bet this kind of overwhelming, with overwhelming conviction on this idea, or else you're never going to do the things that are needed. You're never going to assemble the amount of compute that you need or build the data centers you need. You have to like you have to only care about one thing. And I think they are successful at caring about only one thing more than any of their competitors. Yeah, yeah, I think that's right, I think so. Before the transformer paper in 2017 and before your story really begins. That was really true in spades, because most people like when when Demis Hassabis started DeepMind in 2010. I couldn't recognize the photograph of a cat. It could do nothing. Deep AI winter like the idea that you would start a company and raise money because you're going to do this was like crazy. I mean, totally. So you needed that complete belief in the absence of evidence. Blessed are they who believed before there is evidence is a line from one of Demis Hassabis favorite movies. And, and so but I think it's the same point. And, you know, I hadn't thought about this before, but but but just listening to you, the same point is true in a different way later on in the race where, okay, by 2022, everybody understands that this is a big deal. It's going to get better. It's going to be exciting. But the difference with Dario is that he didn't just believe in a general way it would get better. He had a very specific plan. And you talk about his 2019 paper where he had this, you know, here is the path we're going to, you know, train on all the data tick. That's what they did use reinforcement learning to improve the systems that math and logic. And then we'll get to recursive self-improvement. And basically that's what's happened. So he had a specific commitment and belief in the path that worked. Yeah. And he it's one of my favorite stories in the book. I'm glad you brought up because it was it was sort of the most cloak and dagger these things ever got. So Dario I don't think it's a secret by now is like, he's quite paranoid about things being leaked. He takes like, information security very seriously. He always talks about these things called info hazards, which is like old rationalist lingo for like a piece of information that is kind of dangerous to know. So he thinks in terms of secrecy and and so in 2019, he comes back from his summer vacation in Italy, and he's got this document that he's written on an air gap laptop. And it's called The Path to AGI. And it is basically the thing you describe. And he he's too afraid to send it out over a slacker email. So he prints out paper copies and he invites his core team, one by one, into a room at OpenAI. And he says, you can read this document, but only once in the room, and you have to leave your phones and other devices outside. And later he he does something even trickier. He he puts them in the this path to AGI paper in the middle of like 200 pages of boring philosophy papers and like stuffs it into a binder. Sort of like if anyone who has it, like, accidentally leaves it in the break room or something, like no one will know it's there. So it's like incredible act of paranoia. And it works like the path to AGI that he laid out in 2019 prophecies the next 7 to 10 years, almost to an eerie degree. So I think he was someone who had the conviction. And then they've just been executing on that ever since. Yeah yeah. And it shows the visit. The vision overcame all those shortcomings we were talking about before, but it almost didn't like anthropic almost died. And part of what I, what I wanted to convey in the book is like, this is a company that came very close to to dying on several occasions, one of which was, you know, they, they, they couldn't get funded in the first place. No, VCs wanted to touch them. They had this Sam Bankman-Fried thing happen where their biggest investor goes to prison. They have ChatGPT that absolutely, you know, beats them to releasing a mass market chatbot. And there are a bunch of other things, but I think, their belief sort of carried them through. So I'm going to get to questions from the audience in a second. I want to go a little bit to Google DeepMind there. And the question here I think, is that, you know, so as of 2017, when your story begins, they do the transformer paper, that's their thing. They have tons of money at Google, a very deep scientific bench. And they still screw up. They don't release ChatGPT first. And even in 23 and 24, they are kind of gasping to, to, to be in the game. And so you have fun with the story about how AI mode, the kind of search inside Google search would would. Sorry. The the AI bit inside Google search was was recommending that a fine ingredient for pizza might be glue. Yes, you should add a quarter cup of glue to your pizza to increase tackiness. But then in end of 25, they have this comeback moment when Gemini is briefly evaluated as being the best model. And then in 26, they kind of fade again. So what's going on with this? Why why do they kind of do badly? Rise up, go down. What what's the story? I think it's about belief. I think they had a period where they had built all this AI, but they didn't believe that it was building up to something transformative. So they thought, you know, these are algorithms to make your Google Photos image recognition better or your Gmail autocomplete better, or maybe have an easier time summarizing Google documents. But the idea that all of this would sort of pile into one kind of general superintelligence was fiercely debated inside the company. And I think as factions sort of rose and fall within the Google like eternal Game of Thrones. They would just get closer further, further away from their rivals. So I think when they caught up last year was in part because they had kind of like they had sort of rediscovered their their belief and their competitive streak. And then, I don't know, I you tell me what happened in the last year. My sense is that they've been sort of falling back on some of these like, hesitations or reluctance. They've been selling compute to anthropic and other companies, which to me would indicate like they don't believe that they can more productively use those things to themselves to train their own models. But as the resident, as the world's resident Google DeepMind expert, why do you think they had a bad year this year? Yeah. So I think that, this is sort of eternal drama of the two species of beast that we encounter in Silicon Valley. Right there is the tech behemoth Google. And then there is the kind of insurgent startup culture which you have at anthropic in OpenAI. And even though these are very, very big startups, that's kind of still the, the vibe inside. And the behemoth, you know, just has lots of different interests at once. And so they want they've got a whole Google Cloud business, and they want to sell cloud services to other people. So they're going to allocate compute to the cloud guys, and the cloud guys will sell it to anthropic or whatever they do. And then there's a whole search team. And these are the traditional money gushes inside the company and the search people want to have, you know, AI search. And so they get a whole bunch of TPUs to to do their AI search. And then somebody from whatever biz dev or whatever they call it, you know, it says, oh, we're going to supply Gemini under the hood to make Apple intelligence function, are we have to allocate a whole bunch of compute over there. And then, you know, the the guys doing the research who need the compute to do experiments and then run training, they didn't have very much. And I think that is part of it. Anyway, let's go to questions from our very good audience. The first one is in a recent Ezra Klein video essay, he argues that the recursive self-improvement is a line that we should not cross. Do you agree? Yes. I think we should not cross it yet. Recursive self-improvement is the sort of fabled point at which AI systems become capable of training better AI systems. So you have, you know, for example, GPT seven can fully automate the training and development and optimization of GPT eight and GPT eight can build GPT nine, etc.. So this is the sort of point of no return the event horizon. If you're, you know, in AI safety circles, because this is the point at which we might get a, a, an escalating and accelerating cycle of recursive development of these models, and you might end up with something like a in a explosion where something, you know, the amount of progress that might have required ten years would happen in ten weeks or ten days. We're not there yet, but I think the the people I talked to at some of the labs are increasingly worried that we could get there in the next year. There's enough internal automation happening at these companies. They are giving over more and more of their internal processes to their own AI models. I think anthropic recently said that something like 26% of its AI R&D projects. Claude is now the leader of not a teammate, not a programing assistant. It's in charge. And so if that number gets to 50 or 70 or 80 or 100%, I think that's what people are talking about when they say recursive self-improvement. I don't think we should cross it yet because we don't know how to control these systems yet, so we shouldn't cross it. But is there a way of not crossing it, given all the competition, the race dynamics, including with China? I think you have to you have to ban it. I, I think that we have to agree that banned the China and they ban it too. Well, I think the, the, the sort of trillion dollar question is, would China be capable of achieving recursive self-improvement on its own, or does it need to kind of bootstrap off of the leading American models, by distillation through, Claude and ChatGPT to, to get where they want to go. But look, I'm not saying it's the ban is the only solution. I think the labs would probably do well to sort of hold hands and agree not to fully turn over AI research and development to AI before they have more robust systems of controlling them. But I mean, we clearly they clearly don't know how to control these systems right now. I mean, every day we hear about some new rogue agent swarm that is out there causing, you know, damages and, and cyber attacks and things like that. So I think even buying themselves a couple of years by slowing down this particular flavor of AI deployment would do an enormous amount of good. Yeah. Okay. Next question. Can we trust the current AI leaders to build AI in a way that will be beneficial to the masses rather than just a few financially? No, I mean, they don't even trust themselves, which is why they've set up these bizarre corporate structures. And I think they understand that they have an image problem, but also that they, you know, they all they all think this is going to be a very, unequal technology in terms of where the gains are coming from. So you can, you know, go back ten years and Sam Altman is studying universal basic income and anthropic just this week announced that 20% of their shares are committed to sort of the public good. And I think they understand that if they don't sort of redistribute the spoils of what they're building, it will all end up in their hands. And they understand that that's not a recipe for a peaceful life. You will have pitchforks. You will have people who are very, very angry at you. So I think they're trying to get ahead of that by redistributing it, some of this, some of this wealth themselves. But, you know, I think the technology is sort of inherently, inherently concentrates wealth in a very few hands. And so the question is like, how do we how do we redistribute? Yeah. Why did you only publish the first ten pages of Emma Day's big blob of Compute as a journalist? And do you think there's a public interest in shedding light on anthropic leaders and their thought processes about the alignment problem? It's a good question. So I for a backstory, I published on my Substack the first ten pages of a document that Dario Ahmadi had written in 2017 called The Big Blob of Compute Hypothesis, where he sort of lays out his. It's sort of the the precursor to the scaling laws, this sort of observation that rather than training narrow models on narrow tasks, you should assemble a big blob of compute and pointed at more general tasks. And there were some confidentiality reasons that I didn't publish the last 15 pages. Frankly, they weren't all that interesting. But also the the people who shared that with me. There were several different versions floating around, and there was some worry that, that publishing those last pages might have given away their identity or made them findable, I see, next one is about mechanistic interpretability. So good fire. I don't know what good fire is, but assume it's a, you know. Okay good. If you know. So I can find a good question at you. And, and the question is, they've made progress on mechanistic interpretability. How important is this with respect to solving the alignment problem? A great question that I hope we will answer on an upcoming episode of Machine Gods. Look, I think mechanistic interpretability is a one of the worst names ever given to a subfield of research. I mean, can you imagine saying that five times fast? Mechanistic interpretability is, I think, overlooked and crucial to controlling these systems. How can you control something if you don't understand how it works and how to even detect when it's when it's misbehaving or not aligned with your intentions? I have looked at the good fire report. I have heard people that I trust tell me that it's a big deal, but I need to spend more time with it and maybe interview some of the people involved before I decide whether or not it's the salvation of humankind or not. I hope it is. Be great. Yeah. Then we can have RSI be fine. Yeah yeah. What are three key realizations that, teens need to know to deal with AI in and out of the classroom? Don't copy and paste. It's like I'm always saying. No, like, I, I don't have a teen. I have a younger child. But I talked to a lot of teens, and I was at a high school in Connecticut last week, and they asked me some version of this question. I think for me, the the one critical thing would just be to learn and learn discernment between types of tasks, like there are some tasks that are more like weightlifting, and there are some tasks that are more like fork lifting. So if you took a forklift to the weight room and had it lift your waits for you, it would do a great job, but you wouldn't get any stronger. It would defeat the whole purpose of going to the weight room. And if you did a forklift job with your bare hands, it would make you stronger, but it would be a waste of your human potential. Not a fulfillment of it, because that is a job lifting heavy things that is better suited to machines. So for teens and young people in general, I always try to like, have them kind of come up with like these two lists of like tasks. I do parts of going to school, parts of, you know, working parts of being in community that feel more like weightlifting or more like fork lifting, because I think you really don't want to automate the weightlifting stuff. Like you really don't want to automate thinking, writing, struggling. Those are important pieces that make you better, even if they're annoying in the moment. But I think there are all kinds of productive ways that students can use these things to take care of the more like forklift tasks. And the next question is that about the age of child you do have? In other words, it's about you. How has AI changed or not changed you as a parent? It has not changed much about the way that I interact with my child. I don't like, I don't like, we don't sit around. I'll tell you what one concrete benefit it had. It's great at creating blank coloring book pages. I don't know if any of you are parents of children. The speed at which they go through these expensive coloring books is crazy. So I finally just went to ChatGPT and I was like, there was this little widget that let you make blank coloring book pages. And so I just made I printed out like a hundred of them, and it kept him busy for days. But no, we're not sitting there like reading AI generated bedtime stories or anything like that. That would be too dystopian even for me. But I use it a lot for parenting advice. I have a parenting project in one of the AI chatbots where I just say, hey, like, my kid is really, struggling with trying new foods. What are some strategies I might use with him? And it's pretty helpful sometimes so it doesn't replace the human connection, but it can be a resource for the parent. Okay. Next thing, what is your p doom probability of doom? I think today I am at 15%. Wow. Ever. Over which period can you next? Ten years? That's high. No, in San Francisco that's low. Like I know so many people with higher pay dues. Is that high? You should like. Yeah. Super high. Wow. You shouldn't be anchoring upon, you know, on these random people who you happen to live around. This makes me the most optimistic person in San Francisco. I think that's 85% chance no doom. Are you are you defining? Okay, this is good. Are you defining doom as human extinction or complete disempowerment? Okay. So yeah, but but complete. Not just some. Yeah. We're the we're the farm animals. Which do you think is more likely, of the two elimination of humans or total disempowerment. Oh, I don't know. What do you think? Well, okay, well, I'll tell you what I think about that. They're both. Neither of them are good when I get asked this question. That's true. When I get asked you punt, don't you? I tried to ask you this on a podcast, and I don't think you answered. Maybe. Well, that was some time ago. Maybe I've evolved. What's your Pete do? Sebastian? Okay, the first point is I'm punting. There's no data. I mean, it's just a matter of your emotional state. It's a vibe. It's a vibe. Like there's no way you can compute your doom. It's just. So this whole kind of quasi mathematical nonsense is rubbish. But I'll give you the answer. I'll give, you know, I'll give you the non quantitative answer, which is, look, there's a lot of humans out there, like, what is it, 9 billion. Yeah. It's quite difficult, just as a matter of, you know, industrial process to to wipe them all out. Right. You're like it's not, it's the logistics. It's a. Yeah. Yeah. And and so look my, my serious point is that, you know I think the systems are not very well aligned. I think bad guys are going to get hold of them. I think the bad guys will do bad stuff with the systems. I'm surprised there hasn't been worse cyber attacks, given that Kimmy K3 has been out there a few months and it's like almost as powerful as mythos is at cyber hacking, I'm surprised hasn't been worse. So I'm quite pessimistic on this, but my my kind of concept of the bad outcome is Hurricane Katrina type stuff. It's like, you know, we hacked somebody, hacks the dam. The dam floods the city. A city is a total mess. People die. It's on TV. It takes years to rebuild it. It's horrible, but it's not extinction. And then, furthermore, we react to that episode and we start to get our act together. And we go to China and we do a deal, and everybody bans open weight models and, and and so it's not like it's a straight line from here to totally being wiped out. That seems totally that seems ridiculous. So I feel like there's going to be bad stuff to which we react. And therefore my doom is actually less than 1%. I hope you're right. Very British answer. I think you're punting. No. Rather than engaging with my highly rational argument, you've just said, oh, very British. That's a punt. No, look, I think you are correct that. Okay. Thank you. In the, in the extreme scenarios, you do actually need for there to be some new mechanism for killing a bunch of people. I just think I look at, like, what is the survival rate of species that become poaching targets? It's not it's not zero. But usually it's because someone has had to step in at some point and declare them in endangered species and make it illegal to hunt the rhino or whatever. So I hope for some world in which we sort of become the protected species of the benevolent AI overlords, the machines of love and grace will decide to keep us around because we're, like, amusing. Or we make good podcasts, I don't know. Okay. Actually, I have to. I want to say it again. I hope you are right. I have to. I mean, so my sleight of hand is to not address the loss of control. It's to talk about extinction there. I feel like I have an argument, but I agree that loss of control and just being, you know, in the presence of a more powerful species that basically can choose how we end up, that feels more likely. We've never done that before. We've never been the second smartest thing that's true before. Yeah. And we don't have a lot of, like, ideas about what we'll do in that situation. And it could be that we can coexist peacefully. I hope that is the outcome. Maybe if we solve alignment, we sort of get you know, I asked Elias about this because he was the first character who appears in the book in one of the last, and he's sort of the sort of spiritual godfather of this whole AGI race. And, and he has a much more optimistic vision than he used to, where he now thinks that AGI will be like having Michelangelo sitting next to you. And you can kind of give a, you know, a cruddy napkin sketch to Michelangelo, and he'll go off and make you some beautiful art. And he thinks that's what living with superhuman AI will feel like. We just have sort of Michelangelo at our beck and call for any conceivable task. And like, I hope he's right. I hope you're right. But, yeah, I think 15% is reasonable for. Okay I'm moving to a very cheerful subject, which is the prospect of a financial bust. There's a question here. Will some of the data centers become financial failures, which I take to be actually a way of asking, are we in a bubble? And I do worry about the instability and the economy being, you know, biscuits, an AI economy. There's one trade in the market, I think that's right. I, I don't worry so much about it. Look, I think in some ways, we, we are seeing some bubble like activity, the amount of leverage, the amount of debt piling into the sector, all quite worrisome by historical standards. I think there will be some AI companies and projects and NIO clouds and data centers that go bust. But I think if things go the way I expect them to, there will be essentially infinite demand for AI such that all of the data centers will be will will be used pretty much all the time by someone. Yeah, I actually agree with that. All right. So in the 1970s, the electric typewriter came out, 80s. We had. 80s until the 20 tens. We had the word era. What's your thought on the next 50 years in terms of marketable skills? That's a long time horizon. But let's take this question as meaning kind of what do you tell your kids, or what will you tell your kids and or, you know, in a few years when they're bigger about skills that are going to be worthwhile? I think the only skill I'm pretty sure will persist is the ability to make things that other people enjoy. There will be, I think, two economies in the future, one run by AI and one run by humans. The one that's run by AI we have no chance of competing in. They'll be better, faster, stronger, cheaper, more productive than us. The human one is where we'll go to get things that make us feel things. We'll go to get services from a real professional, you know, why do we still pay for, you know, for for coffee at a coffee shop, even though we all have coffee makers at home? Well, it's because it's the ambiance. It's the person smiling at you and saying, good morning. It's the the smell and the music and all of the things that make us feel alive. So I think that economy is still going to be around. And so I would just advise young people to do something that that makes other people happy or fulfilled or helps them out in some way. Should all be baristas, baristas, nurses, occupational therapy. I mean, there are so many jobs in our economy that consist of making people feel better in some way, and I think those are the good jobs, consciousness. So towards the end of the book, you make, I think, a good subtle argument about consciousness, which is basically, we're not going to solve the question of whether machines can be conscious, but what we do have instead is an argument about different people who either want or don't want the systems to be conscious. And so on the one hand, you have anthropic, which takes the position that Claude's moral status is deeply uncertain. And, you know, it makes sense to help, to have a stable identity and a stable psychology. And on the other side, there's people like, oh, don't go there, don't go there. It's completely off limits. Of course it's not conscious, never will be. And I guess my question is I kind of get the anthropic position more than the other one. And it's like their position is, well, we don't really know. But here's how we're going to approach training the model. The other side is dogmatically insisting that it can never be conscious. I'm declaring where I am, where where do you stand on that one? I think my p consciousness is probably like 2%. Okay, so we flip position. I'm way higher. What's your p consciousness? I'm like, you know, 50 of today's models. Yeah. The reason that means so the reason I say that is that not only do the models think an output very intelligent responses, but they think about the thinking. They say to themselves, oh, I took four steps along that logical chain. It didn't work. So I'm going to back up and go over here and try a different one. The classic test of human consciousness is the baby looks at itself in the mirror and understands that the other figure in the mirror is itself. It's when you're self-aware. So I feel like, you know, these systems are pretty much there. Yeah. I don't know what consciousness is in myself or my dog or anything else. You're not conscious of what consciousness is? I'm not conscious of what consciousness is. God, no. But I think, look, I, I right now, the most important thing to me is the human experience. And I don't really care if the. Sorry, if the models are listening. I probably shouldn't say this. I don't really care if they're suffering or feeling anything, because I'm more worried about what they're going to do to us, and I, I think that, these would be what they call champagne problems if we have to, you know, if we have enough spare cycles left to worry about what the model subjective experience is, well, I'm going to end by, you know, reading out what I think is quite a nice, positive, last line of your book. We are not machines, and our next tokens are unpredictable. So in the end, you're saying we have some agency in how this all plays out, and so we should walk out of the room remembering that and trying to use our agency. I think particularly in our own lives, it's like, you know, don't cut and paste. You can choose that by yourself and then you will not suffer cognitive atrophy. But look, the main point is thanks for writing the book. It's a great book. Everybody should read it. If you want to come to terms with all this change that's happening around us, go read the book. And let me see. I'm supposed to read out some important things here. Oh yes. Kevin is going to be available to sign books just after the program. And for more information about the Commonwealth Club World Affairs membership and upcoming programs, please visit Commonwealth club.org. I'm Sebastian Mallaby, thanks for joining us. Thank you all. Thank you Sebastian.

Assistir ao vídeo original no YouTube

Assuntos: AGI, ASI. Publicado no YouTube em 2026-10-21.

Seleção automática por assunto e idioma. O vídeo pertence ao canal de origem.

About Kevin Roose and The AGI Chronicles

Kevin Roose is an American technology columnist and author, educated at Brown University, who has written for The New York Times, according to Wikidata. In the event description published by Commonwealth Club World Affairs of California, he is presented as a former technology columnist at The New York Times, co-host of the "Machine Gods" podcast and author of The AGI Chronicles: The Inside Story of the Race to Create an Artificial Superintelligence.

According to the same description, the book draws on more than 150 interviews with people at OpenAI, Anthropic and Google DeepMind, as well as unpublished company documents and research memos. The conversation took place on October 19, 2026, the event date listed by Commonwealth Club World Affairs, in San Francisco, with Sebastian Mallaby, Paul A. Volcker Senior Fellow for International Economics at the Council on Foreign Relations.

Fact sheet

• Author: Kevin Roose
• Profession: Technology columnist and author
• Education: Brown University
• Book: The AGI Chronicles: The Inside Story of the Race to Create an Artificial Superintelligence
• Research cited: More than 150 interviews with people at OpenAI, Anthropic and Google DeepMind
• Event: Commonwealth Club World Affairs of California, San Francisco, October 19, 2026
• Moderator: Sebastian Mallaby (Council on Foreign Relations)

Read also

• What Happens After AGI? The Road to AI Superintelligence
• Raj Reddy: AGI, ASI and AI Bill of Rights, Talk at ICAIS in Beijing Oct 19, 2026
• Beyond AGI: The Future of Autonomous AI, Agent Societies & Decision-Making | SF Tech Week 2026

---

Sources and references: Wikidata – Kevin Roose (Q24248260) · YouTube description of the video by the Commonwealth Club World Affairs of California channel.