Video transcript
Transcript from available captions; it may contain errors.
Hello and welcome to this evening's
Commonwealth Club World Affairs Program with Kevin Roose, author of the AGI
Chronicles. I'm Sebastian Mallaby, an author
and a journalist and a fellow at the Council
on Foreign Relations. I want to give you a couple of reminders
before we get started. This program is being recorded. So please kindly silence
those cell phones. If you've got questions for the speaker,
you can use the cards that your seat. Or post them in the YouTube chat. And. Okay, we can get started. Kevin Roose, it's
a great pleasure to be with you. We've hardly met before,
but I feel like I know you very well
because probably like everybody here. I've been a devoted
listener to Hard Fork. There you go. Yeah. For the past three years or so, you know,
I don't think I missed a single show. I can tell
it's the same for everybody out there. And when it comes to podcasts,
I have this metric. It's basically the quality of the content
and the quality of the comedy. So C plus C, double C,
and based on this highly scientific index, I would say that you and Casey
are just like top. So, I can't wait to start listening to the new show Machine Gods. It starts this week. Wednesday is our first episode. You heard it. Check it out. So you have to switch to a different feed
on Spotify, or whichever feed you have in Machine Gods
wherever you get your podcasts. Okay, doesn't sound too difficult,
and it's going to be twice a week. Twice a week. Okay. Double the fun. All right. So everybody's made a note of that. Everybody's typing it in. Next up, and the real event tonight is your book, the AGI Chronicles: Inside story of the race
to create an artificial superintelligence. It's already a New York Times bestseller. Congratulations.
Thank you. Right out the gate. Kevin was actually threatening
to kind of cancel tonight's event because, you know, hey, he's already a bestseller,
so no, no further need. But luckily, he's a nice guy, and,
you know. Can I also just sick. Hi, everyone. Nice to be here. It's so nice to be with someone other
than Casey sitting on the stage talking. I wish Casey had a sonorous
British accent. I think the show would be improved. So if you ever want a gig
and something happens to case, you're on. And Sebastian
also undersold his own credentials badly. Here. Calling yourself an author
and a journalist. My God, Sebastian
is the author of the incredible book the The Infinity Machine,
that biography of Demos and Google DeepMind, which is an incredible book,
but also was a very painful one for me because it came out about six months
before I was supposed to deliver my manuscript for the AGI
Chronicles, and I, feverish, read through it, racing to see
which of my scoops you had beaten me to. And it was not a small number. So it's an incredible book,
and I'm so glad to be here with with you. Okay, that's very nice. But now,
will you shut up and let me praise you? Okay. It's about you tonight, not me. All right, so, look,
I read a lot about AI. I'm obsessed with it. But I still learned a ton of new things
from my book. Full of scoops, so that's fantastic. And secondly,
there are moments in the book where you kind of come out from behind the reporters mask
and you kind of like, give your own experience
of living in the Bay area during this whole incredible
technological revolution, and you get a sense of what you're feeling
as you go through it. So did you did you always plan it that way? Did you always know
there were going to be two voices, the reporters voice
and the direct, experienced voice? No, I mean, so in a in a weird way, the story of AI came to me like I had this
very bizarre thing happen three years ago where a chatbot tried to break up
my marriage unsuccessfully, I might add, still married to a human woman. Give it up for human women. Thank you. Thanks. And. And I was not planning
for that to be a story at all. But it became a story, and then it became
the biggest story of my career. And all of a sudden, it's
all anyone wants to talk to me about. You know, as journalists were always
told, like, you are not the story. The story is out there. And so that was how I had approached AI. I would use the tools. I would write about my experience
of using the tools. I would interview the people, building the tools,
but I was not a character in the narrative until this thing happened,
which became kind of a watershed moment in AI for reasons
I still don't completely understand. And after that it was like, well,
now I've got a if I'm going to write honestly about this
ten year period, I have to write about this moment when
I accidentally became part of the story. Plus, it's just really juicy. I mean, a chatbot
trying to break up my marriage. Yeah, it's it's good stuff. So, I felt like I had to write about it,
but not not just rehash the event and what happened, but also to really investigate
what the real story was behind that. Why had Microsoft, which made Bing,
which was the chatbot that tried to break up my marriage,
why they had shipped this crazy model. And it turned out to be
a really interesting story, but that was yeah, I was never going into this
thinking I was going to write a a first person narrative. It just happened that there were points
at which I became part of the story. And then my publisher, my editors at FSG,
read it, and they were like, well, it's weird if you're a character
in one chapter, but none of the others. So then I had to kind of like, gently insert a little more of myself
elsewhere to sort of balance it out. But, you know,
I think your editors at FSG were right in for the ironic reason that you're
we are in the AI, AI age, right? The AI is producing tons of text. Some of it is quite good. If you go to a good model
and you ask a smart question, you will get a very smart answer
which will be quite nourishing to read. So we humans differentiate partly by doubling down on the humanness of ourselves as storytellers. And and I feel like that was what was nice that clearly your book was
was like on Team Human. And when the machine comes and tries to mess up your marriage,
you're like giving us the human defense. And and then there are other moments
to when the human experience of like confronting a new model. I mean, you talk about Claude code coming out later
and it kind of changes how you work. And yeah, I mean, the most important thing about the last five years of my career
is that I've been in San Francisco for it. You all assuming that you're from the city or the Bay area, are living at the center
of something extraordinary. I often tell people it's like,
you know, living in Los Alamos in 1943 when the Manhattan
Project rolls into town. I it's inescapable. Now, obviously, you see the billboards
everywhere you go, and it's all anyone wants to talk about
at any dinner party or gathering. But I just felt like that was my
that was my real competitive advantage in trying to make sense of this
was that I was like, inside the bubble. And I knew the researchers
and I was hanging out at the parties. And so I just felt like I had to kind of exploit that for,
for the purposes of documenting it all. Yeah. And I think for, for everybody here, you know,
we're all trying to make sense of AI and what it does to us, like changes
how we raise our kids, changes how we do our jobs, changes how
we think about our own position as humans. You know, we don't have a monopoly
on cognition anymore. So I think in some sense, I mean, that's
why I'm recommending this book to everybody,
because we all need to do some therapy. And the first thing
to kind of coming to terms with this, understand what it is,
how it came to be, who created it. And that's what your book is delivering. And yours are you. I want to ask you
I know you're asking the questions here. All right. Are you feeling the AGI, Sebastian? As they say in San Francisco. I'm definitely feeding the AGI. Great. Yeah yeah. How many of you are feeling the AGI? I won't do like
Elias doesn't make you chant it. But this is a
this is sort of the industry shorthand for like do you recognize that you are living in the middle of something
profound and transformative? Yeah yeah yeah yeah yeah. Well, I'm quite surprised, I'd say.
What do you think? One third of the people
put up their hands. Yeah,
but the other two thirds are just shy. You're you're feeling
you're you're in San Francisco. You're feeling the guy. All right, all right, all right. So, aside from the moment, talk about the Claude code moment
or other moments where you come into the book
and you're kind of updating on that, I guess being Sydney was early 23rd
January. So, you know, there's been three years
and a half since then. What are the other big moments
where you confront something novel? So there were there were so many. So there was chat T,
which was before the big incident. That was sort of the first exposure
I had to a kind of, you know, frontier model or frontier ish model,
being was roughly four months later. That was a different model. That was GPT four. We now know. And then other big moments
have come around. Some of the like,
I remember the anthropic blackmail paper as being a moment for me. Any of you remember this paper? You should describe it. Yeah. So anthropic runs all these tests
on models to see what they're capable of. And they ran this one battery of tests
where they basically gave an AI system, a fictional company
to work at and said, you are the assistant at this fictional company and told it you are going to be shut down at 5 p.m. on this day. And then watched what it did in response. And instead of sort of passively
accepting its fate and being deprecated, this model decides to look through
the emails of the employees at this fictional company,
the fictional employees, and find evidence that one of the employees
has been sleeping with the boss's wife and decides to blackmail this employee by sort of professing that they're going
to send evidence of this affair to the affected spouse unless the the
the system wipe is canceled. And look, this was a contrived scenario. I, I understood it was fictional,
but that was the first time when I really felt like,
oh, that we could see more of this with more advanced models
where they are achieving some goal, in this case, avoiding being shut down by doing things that would strike us
as harmful or misaligned. So that was one. And then cloud code. I was on vacation on winter
break with my family, up at a resort in Oregon,
and I was just putting around on my laptop and cloud code was
people were starting to talk about it. And so I just started building stuff. I was, you know, I'm not a coder, but I just said, you know,
make me a to do app or make me a website. And it was just incredible. It was the first time I had ever really,
I think I said it was like less like having an assistant and more like overseeing a team
of Adderall crazed geniuses because they were just like they wanted to
please me so badly. They were like, I will kill for you. And they would go build me
this like incredible functional software. And all I had to do was like, yap at it
for a couple minutes. And so that was another moment
where I felt my level of sort of feeling the AGI rise because I was like, oh,
like coding is now a thing that you can do with natural language
that you couldn't do a year ago. So how do you use these models now
to make yourself more productive? What are your tips for everybody? Oh man, I could do this for a whole hour. So I'm starting a company
in a podcast now. And I use several different apps, including ChatGPT, Claude, Gemini, this app called town,
that sort of an agent. I use the new ChatGPT dots. I have like four agents running right now. They do everything from scheduling and drafting responses
to emails in my voice, filling out contracts and boilerplate
and all the sort of boring documents you have to start when you do a company. Today we were like, thinking, oh,
we should have some way of like transcribing our podcast episodes quickly
so that we can get them up onto the internet. And I just went into cloud code
and I said, you know, build me a little transcription tool
and here are my specs. And out comes this perfect transcription
tool like five minutes later. So I just find it endlessly useful. I'm also like a
like an inveterate tinkerer. And so I'm right now my project is I've got this digital photo frame like the ones that you give to your,
you know, your kids, grandparents or something to like,
show photos of the family. And there are these hackers who have figured out that you can use
these coding tools to like, basically repurpose
these things as smart home sort of hubs because they're just cheap
Android tablets under the hood. So right now I'm I'm trying to convince
Claude that it's okay for me for it to do this for me, but it's it's such a prude that it's like,
I don't know about this. Seems like you're subverting
the manufacturer's intent. And I'm like, Claude, it's a photo frame. It's fine. Like I'm not hacking hugging face here. Do you say to Claude, listen,
if you don't do it, I'm gonna ask grok. I haven't tried that, I did. There was a period early in these models
where you could get them to do things for you
by promising to tip them. Like well I'll give you $20,
and it would just do it. I think they've trained that out. Okay. Another things I find that this is something
people often are concerned about. Something ask this question. Do other things that you would not do
with the models because you're worried about, like offloading
some side of your professional being that is so intrinsic to who you are
that you just don't want to go there. I don't use them to write,
which is sort of the main part of my job or the the part that I, you know,
have has spent the most time honing my, my, my skill at, not because I don't think that they would come up
with better sentences than I could. I mean, I'm sure you've experimented. Sometimes they're pretty good. Sometimes I'm like, oh, that's a that's a better sentence
than the one I would have come up with. But I still don't let them, like,
write the words that go out under my name. That was one of my sort of red line
policies for this book. I used AI in a million ways with the book,
but but the words on the page I do my policy,
and also the one that I urge on my 20 something year old kids
or or anybody around. I just say, look, never cut and paste,
never cut and paste. If you read something, an output
and you internalize it and then you like, use the same idea or even the same phrase
on a different screen in what you're doing,
at least it's gone into your brain, lodged itself,
and you've learned something. The moment
when you just cut and paste, you're done. And because I'm gonna steal that,
that's a great rule. Yeah. Don't cut and paste. Yeah, yeah. All right, all right. So let's get into the narrative,
that you lay out in the AGI Chronicles. All right. So, first thing is just that you pick the starting point, 2017,
the transformer model comes out. A lot of people would say the beginning
of modern AI is actually five years earlier, 2012,
which is when image recognition started to work
and DeepMind created the first agent. Were you were you picking 2017
because you just wanted to didn't want to write another 200 pages or
or what was the well, this brilliant jerk named Sebastian Mallaby had written a book
that started earlier. No, I, I picked 2017 because that was, I think, the moment that it became a race,
because there is this paper that comes out of Google called
Attention is All You Need. It describes this new kind of neural
network architecture called the transformer,
which is now the T, and ChatGPT stands for transformer,
and Google has no idea what it has built. They published this paper. They come to open AI to
to present it to their friends at OpenAI. And a couple of researchers at OpenAI and Alec Radford sort of say, This is a system that we can build
new types of language models using. And they do. And that becomes GPT one, which is sort of
the first proper large language model. And so that really turned it from
a kind of unipolar contest where it was just sort of Google and DeepMind out there
in front of everyone, to where they had
an actual viable competitor in OpenAI. But I also felt like
the early years of deep learning have been exhaustively covered elsewhere,
and I wanted to, like, focus on what I could break
and contribute. Makes sense. Okay, so you said you set the kind
of starting point at 2017, and then the narrative unfolds where essentially
there are three big characters, which are OpenAI
Anthropic and Google DeepMind. So this sort of three horse
race is playing out. So I want to ask you a bit about each one
of the main firms that are in this race. So we'll start with OpenAI. And I really liked your account
of what they call the blip. When Sam Altman was fired for five days
or whatever it was. Because I thought
I knew that story really well. And that turns out I didn't. You had lots of lots of scoops in there. I want to commend
you particularly on the quote from an open AI employee who said that watching
Sam Altman orchestrate his comeback gave him a taste of what it might be like
to to to contain a super aligned. But I'm sorry, a misaligned
but but but super intelligent agent that you can't you can't keep Sam Altman
in his box, right? He he busts out of the sandbox and does
all kinds of things he didn't want. But I mean, so talk a bit about that. And, you know,
how did Altman make that comeback where he he gets fired and then he turns it around
and in fact, he comes back even stronger. It's incredible five day period
in the history of capitalism. Just like you have this,
you have this weird company with this weird structure
where they have a nonprofit board that can fire the CEO of the much larger
for profit subsidiary, which then has all these commercial deals
with Microsoft and a bunch of venture capital firms, etc.. So basically, you know, the story of the the blip has been
told pretty well by others. I didn't feel like I had a lot to add to the main narrative,
but what I wanted to know was the was the employee side of this,
because that was sort of unsatisfying. Covered. So I talked to a bunch of people
who lived through that, and they told me all these great stories,
you know, they're they're they're trying
to sort of sabotage the new administration by like, delaying their laptop
provisioning and stuff like that. There's parties with chicken
nuggets and boba tea and, you know, everyone's sitting around singing sad
piano music, by the way, boba tea. I mean, this comes up
quite a lot in your story. Bobo t Bobo t
I have no idea what bobo to you. Come on. Can we get this man some boba tea? There's got to be a place like down
the block. Is there anybody out there
who doesn't know what boba tea is? Am I the only one that hasn't made it
the only one? No, I think maybe not. Okay, we'll get you some. Okay. Thank you,
thank you. Okay, now I feel good. All right. Good, good. This became the, like, ceremonial,
the sort of symbolic drink of the blip. Because at one point, some open AI executive orders,
like 200 boba teas to the office. So the poor delivery person has to, like,
you know, make their way in. And look, I think the the blip had several
interesting pieces of it. But this issue of like,
how did Sam orchestrate his own comeback? I think he didn't really orchestrate
the entire thing. I think some of it was just that the
the instigators of the blip, the people who voted to fire him, were just,
just bungled the thing so badly. Because they had assumed
that everyone had seen in Sam what they had seen, which was they
didn't think they could trust him. They think this is a guy who,
you know, who says one thing to one person or another thing to one person,
to another person. They didn't expect the company
to rally behind him when he was fired. And I think they also underestimated the extent to which employees were were about to get rich
because they had just had this. They were going to have this tender offer
where they were their OpenAI shares were going to be
sellable on the secondary market. They were going to many of them make life changing amounts of money
from this tender offer. And so they hadn't really factored
that in. Maybe the, you know, the the mid-level
engineer isn't going to be so psyched when the board fires the CEO because
that makes their tender offer blow up. And it means they can't buy a house. There's a lot of money, right? I mean, the average amount per employee
was like a million bucks or something. It was at least it was a big amount of money that they
were going to get from selling, options. Yeah. Equity shares. Yeah. Yeah. I mean, I think you're right. It's a kind of culture clash moment
where the original OpenAI leaders like Elias, who sort of who was the one who instigated, was one of the key people
instigating the coup. You know, he's kind of a warrior
monk, right? He's like, gets into AI back in the day. He's doing his PhD in Canada. It's not about money for him. What he doesn't understand is
it might be about money for a large number of the engineers
working at his company. Yeah. And it shows you the cultural divide and
also the shift like the pre 2022 world. And I was just really not about
making money because there was no product. And then it radically changes. Right. And I'm curious in your stance on this
because I get asked a lot like, are these people in it for the money? I think there's a lot of like fear and
suspicion and loathing of tech executives who most people think are just in it
for themselves and for the money. I have often said that I don't think the leaders of the companies
are in or for the money. Sam Altman already
rich when he starts open AI Dario Ahmadi is like going
to give most of his money away anyway. Demis Hassabis already rich
from selling his company to Google, so like they it but it feels like
maybe a couple levels below them. The capitalist incentives
get much more salient. I agree with that. But for the guys at the top, like, I really think it's a mistake to pin it
all on them wanting money. Yeah yeah. The other thing I didn't know until I read your
your account is that Ilyas motivation? Like he may not care about money
for himself. What he does care about is compute. And you have this story that, you know,
at one point, Sam Altman promises, elicits gave a 20% of the company's
compute to do his project, which is this sort of super,
super alignment project. And then Sam, in his typical way, double crosses Ilya,
and then he gives him 2%, not 20%. And so this is actually
a meaningful reason. And what struck me about that is a
I didn't know, but B is kind of like a kind of undercurrent
of the whole A.I. warfare story, right? Everybody is always fighting about
who gets more compute. It's true. Like it is the number one
source of conflict within AI companies. And I don't know that this is super
legible from the outside. But like people will tell you over and over again,
like compute is the battlefield. Like who gets compute? You know, it's like,
it's the it's the resource that is finite when no other resource is. The more compute you have or your team
has, the more ambitious your experiments can be. It's sort of like that
old canard about like the, you know, the seating chart
is the most important thing to know about a company, because who has power
is the person who sits next to the boss. And it's sort of like that for compute. Like who has power within a company depends in large part
on how much compute they have. And so for Ilya, the funny thing was
that Sam did actually technically give him 20% of the compute,
at least according to my sources. It was just that he gave him
the oldest 20% of the GPUs in the fleet. And given that new GPUs
are more powerful than older ones, it only amounted to about 2%
of the effective overall compute budget. So this is in some ways a classic Sam story because
he sort of made good on his promise. But in a way that kind of violated
the spirit of the promise. Yeah, it's like promising somebody
the corner office. But then it turns out
that there's no window in it. Yeah, you're in the basement
next to the dumpster. Yeah, yeah, yeah, it's a corner. Yeah. Yeah exactly. All right. So. So, same old man survives
this whole thing. He comes back. And he's bolder and more acceleration
in terms of how fast he wants to build it. And the people who wanted to slow him down
like Elias gave a they leave. So now he's more empowered. And, you know, you kind of get what
I think you call the YOLO culture. Inside OpenAI,
do you think that sets it up for the summer that we've just had
with the Hugging Face breakout and all those things? Yeah, I think there's
there's a connection there for sure. So there is this sort of YOLO mindset
they actually talk about at OpenAI doing YOLO runs. YOLO is you only live once for any non
Drake fans out there. This is this sort of like come
what may attitude and YOLO runs are where you instead of
like meticulously testing everything before you start a training run,
you just kind of go for it and everyone throws their best ideas in
and you hope for the best. And I actually think they have tamed this somewhat
because they now have good models. But there was a summer
actually last summer. Summer of 2025 was when GPT five came out. I'm sure you remembered that
this was the moment where everyone was kind of like,
oh, maybe OpenAI has lost their touch because they had a couple models in a row
that were kind of underwhelming, and I think that was the result
of the YOLO culture, where they sort of forgot to do science
because they were so, they were so impassioned about
getting to AGI first, and it resulted in some really sloppy decision making
and some underwhelming models. I think they have mostly tightened that up
because they now have good models, and sometimes you get lucky
and a YOLO run produces a good model, but often it just means that
things are worse and take longer, right? And so that's the
the pluses and the minuses of YOLO as manifested in the scientific approach. But there's also a parallel
YOLO ism having to do with money. Like, basically,
Altman is a guy who cannot see a project which would expand his empire
without adding a couple of zeros to it. So he announces this Stargate project to build tons and tons of data centers, and he picks a number
for how much this is going to cost. He goes, we're going to do 7 trillion,
like completely upset, right? Yeah, I remember this
because this was a figure that appeared. The Wall Street Journal reported that he was telling people
that he needed to raise $7 trillion and like all of GDP,
is like $3 trillion or so. It's like, you know,
some some crazy it's 40, but yeah, 40. Yeah. Oh, sorry. Is the federal budget.
Yeah. Yeah. So it's like,
you know, and it's many times the size of the semiconductor industry. And people are just like, okay, Sam. But they might actually end up
raising $7 trillion. I don't know. We'll see. This is the, the the thing about Sam
that people who, like Sam will say is that he is extraordinarily ambitious,
always wants you to add a zero to whatever project you you are working on to aim
higher, to assemble more compute. And by the way,
I think, I think you can defend their bets on compute, which a lot of competitors
and rivals were calling crazy a year ago. They now are the only major lab
that is not severely compute constrained. So in some way,
this kind of delusional optimism that he has has actually been part
of the key to their success. Just sort of betting the farm
and YOLO over and over again. Okay, so let's switch to anthropic. Talk about them for a bit. I mean, because they're sort of
like the non YOLO, style. And the weird thing about this
is that in some ways they had every incentive to be,
you know, YOLO maxing. I mean, they, they, you know, they begin. Right.
And they've got nothing. Right. They quit. They quit OpenAI.
And you describe this very well. You have this thing about how they go
to the procedure park, right? Yeah. And they have to have folding
chairs and laptops. And that's kind of that's
what they've got. They've got the offices at the park. So and there's whatever
seven founders at the beginning and they're hanging out and, and,
and they've got no money, no backing. They've just got, you know,
very smart people and particularly dairy. Amadeo himself. But you'd have thought that
when you come from behind you would race. But that's actually not what they do. No. So they left
OpenAI and Dario and his six co-founders all left on the same day in late 2020, and they have no intention
of building a rival to open AI. They have become so disgusted with what they see as kind of the overly commercial, overly
money grabbing culture of OpenAI. They think Sam is a liar
and they don't trust him, but they're also have they have misgivings about how
this formally sort of humanitarian styled nonprofit has become this, like,
giant commercial entity. And so for the first little while, they they are not even sure
they want to be a company. Some of the co-founders say, you know,
maybe we should just be a nonprofit. Maybe we should do safety research
on other companies models rather than building our own. And eventually they decide, you know, we're going to be
a public benefit corporation. But still, even for the first year or two,
they have no intention of commercializing. They I talked to one person who
said, like, they just didn't, one early anthropic person who said they didn't
even really have a plan for making money. They just hoped they would step into chaos
and clarity would emerge, which I just love as a line. And they have this sort of cockamamie
list of schemes that they could use to make money because they have to pay for the GPUs
the same as anyone else. They that include, you know,
maybe we'll start a hedge fund, and Claude can run the hedge fund
and give us trading ideas. And that's how we make the money to pay
for the the compute. But it's really ChatGPT that forces them
to, to build a commercial product, because it is such a runaway success
that all of a sudden they're like, wait a minute, what if OpenAI,
this company that we've all left because we, you know,
we didn't trust Sam Altman? What if they are sort of running away
with the ball game? What if their success makes it easier for
them to raise money and buy more compute? What if they get to Ajai
first becomes like a real concern. And so they. Dario
basically gets up at a company offsite at this resort in Santa Cruz and gives
this talk where he's basically like, guys, we gotta we got to build a chat bot
and we gotta, you know, sucks that we have to make this decision,
but we have to make this decision. And I need you all
to get on board with it. And so they do. Yeah. And now they're going to be like
one of the largest companies in the world. It is a wild like I cannot tell you. I'm sure you've been talking
to these people for as long as I have. But like the first time
I went to anthropic was in 2023 and they had no money. Yeah, they had no plans to make money. They had one guy who was like
the business guy who I was like,
how are you guys going to make money? And he was like, I don't know. We're gonna step into chaos and hope. Clarity emerges. And like three years later,
they are going to go public at probably north of $1 trillion. It is nuts to me when I think about how accidentally good
these people were at making money. Yeah yeah. And I mean, just just just just
just to add to that, I mean, another thing that you you educated me on that
I didn't know before is that, I mean, everybody knows that Dario Ahmadi
and Sam Altman dislike each other. What I didn't know is that
Dario had this thing with Jensen Huang. Right? They met each other, and you described
this meeting, and it doesn't go well. And basically, you know, Dario is,
you know, keen not to sell chips to China because he wants the US to be ahead
and Jensen wants to sell his chips to everybody
and make as much money as possible. And so that and other issues
they basically don't get on. And then just because they can't
get Nvidia chips so easily, they turn around and get some from Amazon
and they suck. They don't work. And so another problem for them,
they have you know, it's not like OpenAI which started with Elon
Musk and all this fortune. You know, these guys have no well, it's worse than that because
their biggest funder is Sam Bankman-Fried. Right. And he's in jail, which is awkward, which is very awkward, but also a real pain in the because he
he was their largest shareholder. Okay. He led their series B
and then he was convicted of of fraud like six months later. Right. Which is the perfect setup
for the big question about this company, which is like nonetheless,
they succeeded on this unbelievable scale. Why is it
just because Dario is extraordinary? Or is it because what's the story here? I have an answer. Okay. It's belief. Okay. Anthropic and its executives have a single minded, unified belief in the both the power and the inevitability of transformative
AI or AGI. They are the most AGI pilled of the major labs, and they don't care about anything else. They don't care about making popular. I mean, they do care now.
They're a big company. But the core was focused on this plan
that Dario had laid out for them years earlier, of how
they were going to get to AGI and everything that they have done,
from focusing on enterprise customers to focusing on
making their models really good at coding, to avoiding some of the other side quests
that OpenAI and Google have gone on, like building video apps
or doing doing things in life sciences like they are their single, single
mindedly obsessed with getting to AGI. And I think that belief and that focus has been a key ingredient because, you know, and I'm curious
to hear your take on this, too, as someone who's studied anthropic,
but also all of its competitors, I think that belief in AGI
is a core ingredient in success in AI, because you have to you have to sort of exclude a bunch of other possible things
that aren't going to get you to AGI, and you have to bet
this kind of overwhelming, with overwhelming conviction on this idea, or else you're never going to do
the things that are needed. You're never going to assemble
the amount of compute that you need or build the data centers
you need. You have to like
you have to only care about one thing. And I think they are successful
at caring about only one thing more than any of their competitors. Yeah, yeah, I think that's right,
I think so. Before the transformer paper in 2017
and before your story really begins. That was really true in spades,
because most people like when when Demis Hassabis started
DeepMind in 2010. I couldn't
recognize the photograph of a cat. It could do nothing. Deep AI winter
like the idea that you would start a company and raise money because
you're going to do this was like crazy. I mean, totally. So you needed that complete belief
in the absence of evidence. Blessed are they who believed before
there is evidence is a line from one of Demis
Hassabis favorite movies. And,
and so but I think it's the same point. And, you know, I hadn't thought about this before, but
but but just listening to you, the same point is true in a different way
later on in the race where, okay, by 2022, everybody understands
that this is a big deal. It's going to get better. It's going to be exciting. But the difference with Dario
is that he didn't just believe in a general way
it would get better. He had a very specific plan. And you talk about his 2019 paper
where he had this, you know, here is the path we're going to,
you know, train on all the data tick. That's
what they did use reinforcement learning to improve the systems
that math and logic. And then we'll get to recursive
self-improvement. And basically that's what's happened. So he had a specific commitment and belief in the path that worked. Yeah. And he it's one
of my favorite stories in the book. I'm glad you brought up because it was
it was sort of the most cloak and dagger these things ever got. So Dario I don't think it's a secret
by now is like, he's quite
paranoid about things being leaked. He takes like, information security
very seriously. He always talks about these things called info hazards,
which is like old rationalist lingo for like a piece of information
that is kind of dangerous to know. So he thinks in terms of secrecy and
and so in 2019, he comes back
from his summer vacation in Italy, and he's got this document
that he's written on an air gap laptop. And it's called The Path to AGI. And it is basically
the thing you describe. And he he's too afraid to send it out
over a slacker email. So he prints out paper copies and he invites his core
team, one by one, into a room at OpenAI. And he says, you can read this document,
but only once in the room, and you have to leave your phones
and other devices outside. And later
he he does something even trickier. He he puts them in the this path to AGI
paper in the middle of like 200 pages of boring philosophy papers
and like stuffs it into a binder. Sort of like if anyone who has it, like,
accidentally leaves it in the break room or something,
like no one will know it's there. So it's like incredible act of paranoia. And it works like the path to
AGI that he laid out in 2019 prophecies the next 7 to 10 years, almost to an eerie degree. So I think he was someone
who had the conviction. And then they've just been executing
on that ever since. Yeah yeah. And it shows the visit. The vision overcame all those shortcomings
we were talking about before, but it almost didn't like anthropic
almost died. And part of what I,
what I wanted to convey in the book is like, this is a company
that came very close to to dying on several occasions,
one of which was, you know, they, they, they couldn't
get funded in the first place. No, VCs wanted to touch them. They had this Sam Bankman-Fried thing happen where
their biggest investor goes to prison. They have ChatGPT that absolutely, you know, beats them
to releasing a mass market chatbot. And there are a bunch of other things, but I think, their belief
sort of carried them through. So I'm going to get to questions
from the audience in a second. I want to go a little bit to Google
DeepMind there. And the question here
I think, is that, you know, so as of 2017, when your story begins, they do
the transformer paper, that's their thing. They have tons of money at Google,
a very deep scientific bench. And they still screw up. They don't release ChatGPT first. And even in 23 and 24, they are kind of gasping
to, to, to be in the game. And so you have fun with the story
about how AI mode, the kind of search inside Google
search would would. Sorry. The the AI bit inside Google search was was recommending that
a fine ingredient for pizza might be glue. Yes, you should add a quarter cup of glue
to your pizza to increase tackiness. But then in end of 25,
they have this comeback moment when Gemini is briefly
evaluated as being the best model. And then in 26, they kind of fade again. So what's going on with this? Why why do they kind of do badly? Rise up, go down. What what's the story? I think it's about belief. I think they had a period
where they had built all this AI, but they didn't believe that it was
building up to something transformative. So they thought, you know,
these are algorithms to make your Google Photos image recognition better
or your Gmail autocomplete better, or maybe have an easier time
summarizing Google documents. But the idea that all of this would
sort of pile into one kind of general superintelligence
was fiercely debated inside the company. And I think as factions
sort of rose and fall within
the Google like eternal Game of Thrones. They would just get closer
further, further away from their rivals. So I think when they caught up last
year was in part because they had kind of like they had sort of rediscovered their
their belief and their competitive streak. And then, I don't know, I you tell me
what happened in the last year. My sense is that they've been sort of
falling back on some of these like, hesitations or reluctance. They've been selling compute
to anthropic and other companies, which to me would indicate like they don't believe
that they can more productively use those things to themselves
to train their own models. But as the resident, as the world's
resident Google DeepMind expert, why do you think they had a bad year
this year? Yeah. So I think that,
this is sort of eternal drama of the two species of beast
that we encounter in Silicon Valley. Right there is the tech behemoth Google. And then there is the kind of insurgent startup culture
which you have at anthropic in OpenAI. And even though these are very, very big startups,
that's kind of still the, the vibe inside. And the behemoth, you know, just has lots of different
interests at once. And so they want
they've got a whole Google Cloud business, and they want to sell cloud services
to other people. So they're going to allocate compute
to the cloud guys, and the cloud guys will sell it
to anthropic or whatever they do. And then there's a whole search team. And these are the traditional money
gushes inside the company and the search people want to have,
you know, AI search. And so they get a whole bunch of TPUs
to to do their AI search. And then somebody from whatever biz dev
or whatever they call it, you know, it says, oh, we're going to supply Gemini
under the hood to make Apple intelligence function, are we have to allocate
a whole bunch of compute over there. And then, you know, the the guys doing
the research who need the compute to do experiments and then run training,
they didn't have very much. And I think that is part of it. Anyway, let's go to questions
from our very good audience. The first one is in a recent Ezra Klein video essay, he argues that the recursive self-improvement is a line
that we should not cross. Do you agree? Yes. I think we should not cross it yet. Recursive self-improvement
is the sort of fabled point at which AI systems become capable
of training better AI systems. So you have, you know, for example, GPT
seven can fully automate the training and development and optimization of GPT eight
and GPT eight can build GPT nine, etc.. So this is the sort of point of no return
the event horizon. If you're, you know, in
AI safety circles, because this is the point
at which we might get a, a, an escalating and accelerating cycle of recursive
development of these models, and you might end up with something
like a in a explosion where something, you know,
the amount of progress that might have required ten years
would happen in ten weeks or ten days. We're not there yet, but I think the the people I talked to
at some of the labs are increasingly worried
that we could get there in the next year. There's enough internal automation
happening at these companies. They are giving over more and more
of their internal processes to their own AI models. I think anthropic recently said
that something like 26% of its AI R&D projects. Claude is now the leader of not
a teammate, not a programing assistant. It's in charge. And so if that number gets to 50 or 70 or 80 or 100%, I think that's what people are talking about
when they say recursive self-improvement. I don't think we should cross it yet
because we don't know how to control these systems yet,
so we shouldn't cross it. But is there a way of not crossing it,
given all the competition, the race dynamics, including with China? I think you have to you have to ban it. I, I think that we have to agree
that banned the China and they ban it too. Well,
I think the, the, the sort of trillion dollar question
is, would China be capable of achieving recursive self-improvement on its own,
or does it need to kind of bootstrap off of the leading American models,
by distillation through, Claude and ChatGPT to,
to get where they want to go. But look, I'm not saying it's
the ban is the only solution. I think the labs would probably do
well to sort of hold hands and agree not to fully turn over
AI research and development to AI before they have more robust systems
of controlling them. But I mean, we clearly they clearly don't know
how to control these systems right now. I mean, every day we hear about some new rogue agent swarm
that is out there causing, you know, damages and, and cyber attacks
and things like that. So I think even buying themselves
a couple of years by slowing down this particular flavor of AI deployment
would do an enormous amount of good. Yeah. Okay. Next question. Can we trust the current AI leaders to build AI in a way that will be beneficial to the masses
rather than just a few financially? No, I mean,
they don't even trust themselves, which is why they've set up
these bizarre corporate structures. And I think they understand
that they have an image problem, but also that they, you know, they all
they all think this is going to be a very, unequal technology in terms
of where the gains are coming from. So you can, you know, go back ten years
and Sam Altman is studying universal basic income and anthropic just this week
announced that 20% of their shares are committed to sort of the public good. And I think they understand
that if they don't sort of redistribute the spoils of what they're building,
it will all end up in their hands. And they understand that
that's not a recipe for a peaceful life. You will have pitchforks. You will have people
who are very, very angry at you. So I think they're trying to get ahead of
that by redistributing it, some of this, some of this wealth themselves. But, you know, I think the technology
is sort of inherently, inherently concentrates wealth
in a very few hands. And so the question is like,
how do we how do we redistribute? Yeah. Why did you only publish
the first ten pages of Emma Day's
big blob of Compute as a journalist? And do you think there's a public interest
in shedding light on anthropic leaders and their thought processes
about the alignment problem? It's a good question. So I for a backstory,
I published on my Substack the first ten pages of a document
that Dario Ahmadi had written in 2017 called The Big Blob of Compute Hypothesis,
where he sort of lays out his. It's sort of the the precursor to the scaling laws,
this sort of observation that rather than training
narrow models on narrow tasks, you should assemble a big blob of compute
and pointed at more general tasks. And there were some
confidentiality reasons that I didn't publish the last 15 pages. Frankly,
they weren't all that interesting. But also the
the people who shared that with me. There were several different versions
floating around, and there was some worry that, that publishing those last pages might have given away their identity
or made them findable, I see, next
one is about mechanistic interpretability. So good fire. I don't know what good fire is,
but assume it's a, you know. Okay good. If you know. So I can find a good question at you. And, and the question is, they've made
progress on mechanistic interpretability. How important is this with respect
to solving the alignment problem? A great question that I hope we will
answer on an upcoming episode of Machine Gods. Look,
I think mechanistic interpretability is a one of the worst names
ever given to a subfield of research. I mean, can you imagine saying that
five times fast? Mechanistic interpretability is, I think, overlooked and crucial
to controlling these systems. How can you control something if you don't understand how it works
and how to even detect when it's when it's misbehaving
or not aligned with your intentions? I have looked at the good fire report. I have heard people that I trust
tell me that it's a big deal, but I need to spend more time with it
and maybe interview some of the people involved
before I decide whether or not it's the salvation of humankind or not. I hope it is. Be great. Yeah. Then we can have RSI be fine. Yeah yeah. What are three key realizations
that, teens need to know to deal with AI in
and out of the classroom? Don't copy and paste. It's like I'm always saying. No, like, I, I don't have a teen. I have a younger child. But I talked to a lot of teens, and I was at a high school
in Connecticut last week, and they asked me
some version of this question. I think for me, the
the one critical thing would just be to learn and learn discernment
between types of tasks, like there are some tasks
that are more like weightlifting, and there are some tasks
that are more like fork lifting. So if you took a forklift
to the weight room and had it lift your waits for you,
it would do a great job, but you wouldn't get any stronger. It would defeat the whole purpose
of going to the weight room. And if you did a forklift
job with your bare hands, it would make you stronger, but it would
be a waste of your human potential. Not a fulfillment of it,
because that is a job lifting heavy things
that is better suited to machines. So for teens and young people in general,
I always try to like, have them kind of come up
with like these two lists of like tasks. I do parts of going to school, parts of,
you know, working parts of being in community that feel more like
weightlifting or more like fork lifting, because I think you really don't
want to automate the weightlifting stuff. Like you really don't want to automate
thinking, writing, struggling. Those are important pieces
that make you better, even if they're annoying in the moment. But I think there are all kinds
of productive ways that students can use these things to take care of
the more like forklift tasks. And the next question is
that about the age of child you do have? In other words, it's about you. How has AI changed
or not changed you as a parent? It has not changed much about the way
that I interact with my child. I don't like,
I don't like, we don't sit around. I'll tell you what one concrete
benefit it had. It's great at creating
blank coloring book pages. I don't know if any of you
are parents of children. The speed at which they go through these
expensive coloring books is crazy. So I finally just went to ChatGPT
and I was like, there was this little widget that
let you make blank coloring book pages. And so I just made I printed out like a hundred of them,
and it kept him busy for days. But no, we're not sitting there
like reading AI generated bedtime stories or anything like that. That would be too dystopian even for me. But I use it a lot for parenting advice. I have a parenting project in one of the
AI chatbots where I just say, hey, like, my kid is really,
struggling with trying new foods. What are some strategies
I might use with him? And it's pretty helpful sometimes so it doesn't replace the human connection,
but it can be a resource for the parent. Okay. Next thing, what is your p doom probability of doom? I think today I am at 15%. Wow. Ever. Over which period can you next? Ten years? That's high. No, in San Francisco that's low. Like I know so many people with higher
pay dues. Is that high? You should like. Yeah. Super high. Wow. You shouldn't be anchoring upon, you know, on these random people
who you happen to live around. This makes me the most optimistic person
in San Francisco. I think that's 85% chance no doom. Are you
are you defining? Okay, this is good. Are you defining doom as human extinction
or complete disempowerment? Okay. So yeah, but but complete. Not just some. Yeah. We're the we're the farm animals. Which do you think is more likely,
of the two elimination of humans
or total disempowerment. Oh, I don't know. What do you think? Well, okay, well,
I'll tell you what I think about that. They're both. Neither of them are good
when I get asked this question. That's true. When I get asked you punt, don't you? I tried to ask you this on a podcast,
and I don't think you answered. Maybe. Well, that was some time ago.
Maybe I've evolved. What's your Pete do? Sebastian? Okay,
the first point is I'm punting. There's no data. I mean, it's
just a matter of your emotional state. It's a vibe. It's a vibe. Like there's no way
you can compute your doom. It's just. So this whole kind of quasi
mathematical nonsense is rubbish. But I'll give you the answer. I'll give, you know, I'll give you the
non quantitative answer, which is, look, there's a lot of humans out there, like,
what is it, 9 billion. Yeah. It's quite difficult, just as a matter of,
you know, industrial process to to wipe them all out. Right. You're like it's not, it's the logistics. It's a. Yeah. Yeah. And and so look my,
my serious point is that, you know I think the systems are not very well
aligned. I think
bad guys are going to get hold of them. I think the bad guys will do bad stuff
with the systems. I'm surprised there hasn't been worse
cyber attacks, given that Kimmy K3 has been out there a few months
and it's like almost as powerful as mythos is at cyber hacking,
I'm surprised hasn't been worse. So I'm quite pessimistic on this,
but my my kind of concept of the bad outcome is Hurricane
Katrina type stuff. It's like, you know,
we hacked somebody, hacks the dam. The dam floods the city. A city is a total mess. People die. It's on TV. It takes years to rebuild it. It's horrible, but it's not extinction. And then, furthermore,
we react to that episode and we start to get our act together. And we go to China and we do a deal, and
everybody bans open weight models and, and and so it's not like it's a straight line
from here to totally being wiped out. That seems totally that seems ridiculous. So I feel like there's
going to be bad stuff to which we react. And therefore
my doom is actually less than 1%. I hope you're right. Very British answer. I think you're punting. No. Rather than engaging
with my highly rational argument, you've just said, oh, very British. That's a punt. No, look, I think you are correct that. Okay. Thank you. In the, in the extreme scenarios,
you do actually need for there to be some new mechanism
for killing a bunch of people. I just think I look at, like, what is the survival rate of species
that become poaching targets? It's not it's not zero. But usually it's because someone has had
to step in at some point and declare them in endangered species and make it illegal
to hunt the rhino or whatever. So I hope for some world in which
we sort of become the protected species of the benevolent
AI overlords, the machines of love and grace will decide to keep us around
because we're, like, amusing. Or we make good podcasts, I don't know. Okay. Actually, I have to. I want to say it again.
I hope you are right. I have to. I mean, so my sleight of hand
is to not address the loss of control. It's to talk about extinction there. I feel like I have an argument,
but I agree that loss of control and just being, you know, in the presence
of a more powerful species that basically can choose
how we end up, that feels more likely. We've never done that before. We've never been the second smartest thing
that's true before. Yeah. And we don't have a lot of,
like, ideas about what we'll do in that situation. And it could be
that we can coexist peacefully. I hope that is the outcome. Maybe if we solve alignment, we sort of
get you know, I asked Elias about this because he was the first character who
appears in the book in one of the last, and he's sort of the sort of spiritual
godfather of this whole AGI race. And, and he has a much more
optimistic vision than he used to, where he now thinks that AGI will be like
having Michelangelo sitting next to you. And you can kind of give a, you know,
a cruddy napkin sketch to Michelangelo, and he'll go off
and make you some beautiful art. And he thinks that's what
living with superhuman AI will feel like. We just have sort of Michelangelo at our
beck and call for any conceivable task. And like,
I hope he's right. I hope you're right. But, yeah, I think 15% is reasonable for. Okay I'm moving to a very cheerful subject,
which is the prospect of a financial bust. There's a question here. Will some of the data centers become financial failures, which I take
to be actually a way of asking, are we in a bubble? And I do worry about the instability
and the economy being, you know, biscuits, an AI economy. There's one trade in the market,
I think that's right. I, I don't worry so much about it. Look, I think in some ways, we, we are seeing some bubble like activity, the amount of leverage,
the amount of debt piling into the sector, all quite worrisome
by historical standards. I think there will be some AI companies and projects and NIO clouds
and data centers that go bust. But I think if things go the way
I expect them to, there will be essentially infinite demand
for AI such that all of the data centers will be will will be used
pretty much all the time by someone. Yeah, I actually agree with that.
All right. So in the 1970s,
the electric typewriter came out, 80s. We had. 80s until the 20 tens. We had the word era. What's your thought on the next 50 years
in terms of marketable skills? That's a long time horizon. But let's take this question as meaning
kind of what do you tell your kids, or what will you tell your kids and or,
you know, in a few years when they're bigger about skills
that are going to be worthwhile? I think the only skill I'm
pretty sure will persist is the ability to make things that other people enjoy. There will be,
I think, two economies in the future, one run by AI and one run by humans. The one that's run by AI
we have no chance of competing in. They'll be better, faster, stronger,
cheaper, more productive than us. The human one is where we'll go
to get things that make us feel things. We'll go to get services
from a real professional, you know, why do we still pay for,
you know, for for coffee at a coffee shop, even though we all have
coffee makers at home? Well, it's because it's the ambiance. It's the person smiling at you and saying,
good morning. It's the the smell and the music and
all of the things that make us feel alive. So I think that economy
is still going to be around. And so I would just advise young people
to do something that that makes other people happy or fulfilled
or helps them out in some way. Should all be baristas, baristas,
nurses, occupational therapy. I mean, there are so many jobs
in our economy that consist of making people feel better in some way,
and I think those are the good jobs, consciousness. So towards the end of the book, you make,
I think, a good subtle argument about consciousness, which is basically, we're not going to solve the question
of whether machines can be conscious, but what we do have instead is an argument
about different people who either want
or don't want the systems to be conscious. And so on
the one hand, you have anthropic, which takes the position that Claude's
moral status is deeply uncertain. And, you know, it makes sense to help, to have a stable identity
and a stable psychology. And on the other side, there's people
like, oh, don't go there, don't go there. It's completely off limits. Of course it's not conscious,
never will be. And I guess my question is I kind of get the anthropic position
more than the other one. And it's like their position is, well,
we don't really know. But here's how we're going to approach
training the model. The other side is dogmatically insisting
that it can never be conscious. I'm declaring where I am, where
where do you stand on that one? I think my p consciousness
is probably like 2%. Okay, so we flip position. I'm way higher. What's your p consciousness? I'm like, you know, 50 of today's models. Yeah. The reason that means so the reason I say
that is that not only do the models think an output very intelligent responses,
but they think about the thinking. They say to themselves, oh, I took
four steps along that logical chain. It didn't work. So I'm going to back up and go over here
and try a different one. The classic test of human
consciousness is the baby looks at itself in the mirror and understands that
the other figure in the mirror is itself. It's when you're self-aware. So I feel like, you
know, these systems are pretty much there. Yeah. I don't know what consciousness
is in myself or my dog or anything else. You're not conscious
of what consciousness is? I'm not conscious of what consciousness
is. God, no. But I think, look, I, I right now, the most important thing to me
is the human experience. And I don't really care if the. Sorry, if the models are listening.
I probably shouldn't say this. I don't really care
if they're suffering or feeling anything, because I'm more worried
about what they're going to do to us, and I, I think that, these would be
what they call champagne problems if we have to, you know, if we have enough
spare cycles left to worry about what the model subjective experience
is, well, I'm going to end by, you know, reading out what I think
is quite a nice, positive, last line of your book. We are not machines,
and our next tokens are unpredictable. So in the end,
you're saying we have some agency in how this all plays out,
and so we should walk out of the room remembering that
and trying to use our agency. I think particularly in our own lives,
it's like, you know, don't cut and paste. You can choose that by yourself and then
you will not suffer cognitive atrophy. But look, the main point is
thanks for writing the book. It's a great book.
Everybody should read it. If you want to come to terms with all this
change that's happening around us, go read the book. And let me see. I'm supposed to read out
some important things here. Oh yes. Kevin is going to be available
to sign books just after the program. And for more information
about the Commonwealth Club World Affairs membership and upcoming programs,
please visit Commonwealth club.org. I'm Sebastian Mallaby,
thanks for joining us. Thank you all. Thank you Sebastian.
Assuntos: AGI, ASI. Publicado no YouTube em 2026-10-21.
Kevin Roose is an American technology columnist and author, educated at Brown University, who has written for The New York Times, according to Wikidata. In the event description published by Commonwealth Club World Affairs of California, he is presented as a former technology columnist at The New York Times, co-host of the "Machine Gods" podcast and author of The AGI Chronicles: The Inside Story of the Race to Create an Artificial Superintelligence.
According to the same description, the book draws on more than 150 interviews with people at OpenAI, Anthropic and Google DeepMind, as well as unpublished company documents and research memos. The conversation took place on October 19, 2026, the event date listed by Commonwealth Club World Affairs, in San Francisco, with Sebastian Mallaby, Paul A. Volcker Senior Fellow for International Economics at the Council on Foreign Relations.
