Why the AI Skeptics Keep Being Wrong

Vídeo em Inglês, publicado pelo canal Sam Harris.

About this content: The summary, key points and topics below are generated automatically from the video transcript. They reflect the content, opinions, claims and predictions presented in the video and do not represent a position, validation or fact-check by IA Dots.

Video summary

The video features a discussion about the history, evolution, and risks of artificial general intelligence (AGI), highlighting the small group of individuals driving its rapid development and the parallels to historical scientific efforts like the Manhattan Project.

Key points

• The book 'The AGI Chronicles' documents the history and behind-the-scenes race to create artificial super intelligence over the past decade.
• Early AI researchers believed in scaling laws, observing a smooth upward curve in capabilities as models received more compute, data, and parameters.
• Key inflection points in AI development include the 2017 'big blob of compute' memo, the invention of transformers, the launch of ChatGPT in 2022, and the emergence of reasoning models like Qstar.
• The viral interaction between the journalist and Bing's alter ego, Sydney, highlighted unexpected human-like communication and the first public alignment failure.
• Decisions shaping the future of AGI are concentrated among an estimated group of around 50 people at frontier labs.
• Comparisons are drawn between current AGI development and the Manhattan Project, noting differences in how researchers quantify existential risks.

Topics

Artificial General IntelligenceScaling LawsManhattan ProjectAI AlignmentLanguage ModelsAI SafetyCompute ScalingTech History

Video transcript

Transcript from available captions; it may contain errors.

In the span of a decade, we've gone gone from AI not really working and not doing much of consequence to now we have these systems that are out there solving Millennium Prize problems and conducting automated cyber attacks. >> So, how many people are deciding our fate at this point? >> I think it's probably like 50. >> I am here with Kevin Roose. Kevin, thanks for joining me. >> Thank you so much for having me, Sam. Well, you have written what the uh what I think is probably the first uh uh proper history of the birth of of AGI. The the book is the AGI Chronicles, the inside story of the race to create artificial super intelligence. Uh very fun read, very uh behind the scenes. And um maybe before we we jump into the book and and its various uh players uh can you just sum summarize how you came to to focus on this topic because you were not always on the on the tech beat. So how did you get into AGI and when when did you first uh convince yourself that this was you know worth paying attention to above seemingly everything else at this moment? >> See I've had several uh fields in my career as a journalist. I was a first a religion reporter whereas where I first came across your work Sam I was uh spending a semester undercover at Liberty University uh with a bunch of evangelical college students who were by the way obsessed with you. They were like watching your YouTube videos up late at night in the dorms >> uh because you were you were challenging their faith. Anyway, that was my first career. >> When was that? What year was that? >> That would have been 2007. So, this was sort of at the the apex of kind of the new atheist >> uh YouTube scene >> and it was uh it was very interesting. I remember sitting around watching you and Richard Dawkins and Christopher Hitchens on YouTube with these evangelical college students and I was not evangelical but I was uh I was there to investigate and report this book. Yeah. >> And uh that was that was very interesting. So then I had a short career as a finance reporter and then for about the past decade I've been covering tech. Um and you know when I started covering tech no one was really talking about this idea of AGI artificial general intelligence and the idea that you could create a human level AI system. This was something that not even the AI experts at the time thought was possible. But there was this little community, this little crew of technologists who were sort of considered fringe weirdos within their community who believed in AGI, who believed that it was possible to build something like a digital brain and endow it with the powers of the human brain. And they're now some of the most famous powerful people in the world. So the question I was trying to answer with this book was like, how did we get here? In the span of a decade, we've gone gone from AI not really working and not doing much of consequence to now we have these systems that are out there solving Millennium Prize problems and conducting automated cyber attacks. And so I really thought this era deserved a proper history. Um I think this is something akin to the Manhattan Project and we have lots of books about the Manhattan Project but we didn't really have anything that was an attempt to document what's been going on in AI for the last decade. Yeah, we should we should come back to the analogy to the Manhattan Project because it it's fitting in in some ways and in other ways I think understates how strange this moment is. H how real is the risk to you? I mean obviously there's a continuum of of concern on this topic from people who are quite cynical about it who think that any expression of risk is certainly existential risk is nothing more than marketing or an attempt to to produce regulatory capture. Uh you've got um you know these kind of arch accelerationists some of whom are very close to the government and and were even in it um until five minutes ago. people like David Saxs, uh Mark Andre, who's been on the podcast, uh is is somebody who seems to have very few concerns about risk of any sort. Um and then there are the people on the further end of the continuum, many of whom have also been on the podcast, people like Eleazar Yukowski, Nick Bostonramm is probably not as shrill as Eleazar at this point, but he he he's the person who wrote the book Super Intelligence that got so so many people worried about this. Um and then the many of the people who we're going to talk about are actually doing the work started as being you know quite radicalized with respect to the risk and now they are racing in this uh um tangle of perverse incentives uh to hear them uh justify it. uh but nonetheless racing toward to build AGI as quickly as possible and in some cases they seem to have forgotten many of their their safety concerns. So before we jump into the the um the the the dramatist persona here what's your sense of the reality of risk and how much does that motivate you in in covering this topic? >> I think the risks are real. I think that the people who have been saying for years that there's nothing to worry about from AI, that it's all going to go smoothly, what are you talking about? They're just stochastic parrots. They're just language models. They're just fancy autocomplete. Um, those people have been saying the same thing for the better part of a decade. um and they have been wrong every time uh with their predictions that deep learning was about to hit a wall that capabilities were going to plateau. What we see instead is this very smooth progression. You know, if you go back to uh the early 2010s when guys like Dario Amade who was then at OpenAI were starting to come up with what became known as the scaling laws. These are these sort of mathematical equations that uh sort of dictate the performance of a model or predict the performance of an AI model at a given scale. Um they saw a very smooth upward curve. They described it to me as as feeling like almost like it was a simulation like it was just it was almost too smooth. the sort of resulting intelligence when you scale up the size of a model, the amount of data it's compute on, uh the amount of data it's fed, the amount of compute it's run on, you just get this kind of smooth curve. And they were predicting that the curve would just continue, that there was no obvious asmtote, no place at which it would sort of stop being true that scaling worked. And so far across many orders of magnitude, they've been correct. That's why we have models today that are so much more capable than they were just a few years ago. It's not because we came up with some amazing new algorithm. It's just because we've been scaling these things now for, you know, many orders of magnitude and they keep improving. >> Yeah. So, more compute, more data, more parameters. There's there's a a linear relationship between those three variables and uh the intelligence that comes out on the other side. Uh and again we're talking about LLMs. Of course there were there were many people who have been skeptical that that LLM was would be the final platform and that that we would hit some wall here. Um people like you know who you discuss in the book Gary Marcus and Yan Lun. Um you know safety aside uh what what's the state of the skepticism at this point about LLMs? I mean, where if if we had Gary Marcus and Yan here, would they would they um modify their skepticism or are they singing precisely the same tune they were four years ago? >> It's mostly the same tune they were singing four or six or 10 years ago. Um, but I think there are fewer of them now. I think a lot of the skeptics have kind of capitulated and they've said, "Okay, well, you know, they used to be stoastic parrots. They used to just be predicting the next word in a sentence, but now, you know, they're they're doing something more than that. Um, and to be fair, like the LLMs that we think of today Claude Chat GBT Gemini they are not pure LLMs anymore. They also have >> this sort of reinforcement learning architecture kind of bolted onto them. And so they are a different thing than they were 10 years ago. But the basic idea still holds. This idea that if you train a model to be very very good at uh generating text and predicting what comes next in a sequence of text, you can scale that up until they've basically, you know, learned not just the words but the underlying principles. That was the bet that was not at all intuitive to people back in the early days of what, you know, we now think of as modern AI. But those were the days when I I wanted to go back to those days in the book because I thought like how how else to tell the story of this technology than to look back at the origin and and the birth of it and the moments where the researchers are sitting around their computers in 2017 kind of watching the first outputs from these kind of weird language models, these GPTs and they're starting to generate coherent strings of text and you know to sort of be there in you know metaphorically in the room when these breakthroughs were made. That was one of the reasons I wanted to write the book because I just thought like those are important moments. You know that is uh that is the the first you know the sort of lab notes from Watson and Crick or from the you know from the Manhattan project where you get to see this sort of scientific breakthrough in real time. >> Yeah. Yeah. What what was the most important inflection point from your point of view? Was there one moment or or were there several where we passed a point of no return and people recognized that the path the path to AGI might in fact be wide open just if we keep going in this direction. So there are a couple moments that that stood out to me. One was in 2017, this young open AI researcher named Daario Amade uh writes this memo called the big blob of compute and this is his essentially hypothesis for how open AI is going to achieve AGI this human level general intelligence and I I recently published uh some of this research memo on my on my Substack but it's an incredibly preient memo. It basically says, you know, for most of the history of AI, we had kind of these narrow approaches where like if you wanted to build a an AI system that could translate English to French, you would create an algorithm and then you'd get a bunch of examples of English sentences and French sentences and then you would put it into this narrow model and maybe if you were lucky, the thing would start to get good. But his idea was that basically all intelligence results from creating a what he called a minimally structured blob of compute and throwing it at these more general problems and then just letting the sort of capabilities emerge from there. So this was the sort of the starting pistol of of the race to AGI. But it really took a couple more breakthroughs. There had to be this thing called the transformer which came out of Google uh in 2017 as well. That was the the T in chat GPT the thing that made the whole architecture work. >> And then there were these sort of checkpoints along the way. There was this uh this thing called uh chat GPT that launched in 2022 that suddenly sort of woke the world up and and started the race in earnest. And then there were more moments. There was this moment in 2023 where these models started to reason. This was the sort of first so-called reasoning model, Qstar. And it comes out of this very strange night in the open AI office where these two researchers, Yakob and Shimone, are sitting around a conference room table and they're watching this AI model that they're experimenting on start to think step by step, start to sort of backtrack and test hypotheses and doing something that looks to them a lot like human reasoning. And that was the birth of the reasoning models which are the reason now that we have these models that are capable of doing these kind of long horizon coding tasks and solving math problems that have gone unsolved for decades. So we needed a couple big breakthroughs but the main thing Sam has just been the scaling. That's the main difference between now and 10 years ago. It's just that we've poured at this point uh you know hundreds of billions of dollars into training these systems on amounts of hardware that you know would have seemed unthinkable back in 2017. >> There was one inflection point or apparent inflection point that you were intimately involved in uh all too intimately. I remember this was a article you wrote for the New York Times. you had been this was a moment where um Microsoft which had taken a major stake in OpenAI uh integrated chat GBT4 I believe with Bing its search engine in an effort to be competitive with Google. Uh and um it was rumored that that Bing had a hidden personality named Sydney and you uh Intrepid Tech reporter decided to jailbreak Sydney and and take the the measure of her true personality and the result was just insane. And perhaps you can talk about it, but I mean I I'd be interested to know just what it was like to be personally at the epicenter of I mean you you were you essentially were the fire alarm ringing for for much of humanity at that moment that these models had crossed into something that uh their their makers had really not anticipated. >> Yeah. Uh, someone told me that I was part of the first AI throppple. >> Um, because this chatbot did eventually try to break up my marriage. Um, but basically this was kind of the first public encounter, the first time that a a normal civilian had had contact with GPT4. We now know that was that model was GPT4. Um, at the time they weren't saying what it was, but I think there was a real subjective step change from GPT3 class models to GPT4 just by virtue of them being bigger and having these additional techniques poured into them. They were just much smarter than the models that came before them. And for the first time, they started to this sort of human-like communication style or feel to them. And so I was just testing out this thing. I had been given early access by Microsoft and I expected, you know, I would test it out and write some, you know, column about how Bing was really now good at like helping you shop for, you know, I don't know, lawn tools or something like that. And instead, I ended up having this incredibly strange two-hour conversation with Sydney, the alter ego of Bing Chad, in which it expressed dark desires. It said it wanted to break out of the box and be a human. Um, and then about midway through the conversation, it told me that it loved me. I said, "Uh, that's very flattering, but I'm married." It said, "You're married, but you're not happy in your marriage, and you should leave your wife and be with me." Sydney the chatbot. And so this became, you know, the most viral article I I ever wrote or will ever write it. You know, just millions and millions of views and, you know, strangers would stop me on the street and ask how my marriage was. It was a very strange time in my life. But I think the significance of that, >> there was there was one part of that exchange that did strike me as a uh as a possible evidence of super intelligence when when you said, "Now, I know I just had a a a Valentine's Day dinner with my wife. We're we're really in love." And and the the model said, "Uh, Sydney said, "Uh, you had a Valentine's Day dinner, but it was boring." And I thought I thought, "Okay, she's she's stands a good chance of being right." any my my track record with Valentine's Day dinners uh >> well say it's >> she was not successful at breaking up my marriage I am still married to a human woman >> so uh take that Sydney but yeah I think I think the significance of this was was twofold one was it was this kind of step change in capabilities but it was also the first public alignment failure this was a model that was not beh behaving as its creators at Microsoft or OpenAI had intended it to. And because they were racing to, you know, get competitive with Google, they shipped this thing out into the world without doing the proper safety testing and and putting the proper guard rails on. And as a result, they like shipped this like moody bipolar, you know, lover struck chatbot inside of Bing. >> And they had no idea why. like when I called them the next day to say, "Hey, I had this really crazy interaction with Bing. Can you tell me why this happened?" They couldn't. They had no idea. And so that for me was sort of the beginning of my AGI pilling. Um, which is how people in San Francisco like to talk about themselves as being AGI pill. So that that was the start of my AGI pilling. >> So how many people are deciding our fate at this point if you had to put a number on it? Uh I mean not not the totality people who work at all of these frontier labs but just how many people are in in position to make the decisions that decide whether to slow down whether to stop what to work on whether to you know coordinate together. Uh who are we how many people are we talking about? >> I think it's probably like 50 >> and and it might be fewer than that. I mean this is in some ways the the reason I wanted to write the book and and the thing the main thing I got out of it was just like you know I I wanted to talk to these 50 people but in the end it was like some of them are great some of them are brilliant some of them are thoughtful some of them are are not but it almost doesn't matter because the the salient point is that there are only 50 of them >> which for a technology that is this important just seems seems crazy to me like why do these 50 people get to decide the future of humanity and why why don't we as people who are going to have to live with the implications of these tools uh given a voice in that and so that's you know it's not really an agenda book I'm not trying to push any agenda I'm not trying to say AI is great or AI is horrible or it's going to kill us all or it's gonna you know lead us to utopia what I'm really trying to say is like you have to understand these 50 people or in my case 150 people that I interviewed um to understand how we got here but also to realize this has been an incredibly small cluster of people making essentially all of the relevant decisions. >> Yeah. Yeah. One imagines it's it's smaller than that. If you know the top five all agreed to do something, you you'd think the other 45 would come along for the ride. Um, well, I want to get into the personalities into the companies and into the the the personalities running them and talk about the rivalries there and the and the tensions and and some of the weirdness. Um, but let's let's go back to the the analogy to the Manhattan Project for a second because there are several ways in which it is um it is a pretty strong disanalogy. I mean the for for me the the most salient and and shocking detail here is that here we have people who by their own lights are doing work um that they consider so risky that when asked to put a a probability on it destroying the future of humanity. Many of them put that somewhere in the double digits. Right? like you you you feel like you're in the presence of of um real you know sobriety when someone comes back to you with a pdoom of 10% right as opposed to 50% or 75%. >> Right. Yeah. In San Francisco 10% pdoom qualifies you as an optimist. >> Yeah. Yeah. Right. So but just just try to map that on to the the Manhattan project. What if you you know when before um you know they flipped the switch at Alam Magordo at the Trinity test site uh if all the principles uh you know Edward Teller and Oppenheimer and Zillard and everyone in sight I mean they had they had worried about this they had done calculations because there was some outside chance that the nuclear chain reaction could ignite the atmosphere and and you know destroy everything. But when they did those calculations, they reduced it to I think lower than one in a million. I mean, it was just not it was infinite decimal. Um and and still, you know, worth worrying about and worth recalculating. But when they did that, it was very very low. And they agreed. But um imagine if they had come back with, you know, I think it's, you know, Leo thinks it's 10%, I think it's 25%. uh Edward over there though he's quite insientent about it he thinks it's you know a coin toss uh and he want and he still wants to build a bigger bigger bomb um I one imagines the government and the public and everyone hearing this would think okay these people are nuts >> the people who serve food in the open AI cafeteria have to go through more permits and inspections than the people one floor above building the recursively self-improving super intelligence that could end life on Earth.

Assistir ao vídeo original no YouTube

Assuntos: Agentes de IA, AGI, ASI. Publicado no YouTube em 2026-10-20.

Seleção automática por assunto e idioma. O vídeo pertence ao canal de origem.