191. Is It Already Too Late to Control AI? (Anthropic Co-Founder, Jack Clark)

Thanks for listening to the rest is politics sign up to the rest is politics plus to enjoy ad free listening receive a weekly newsletter join our members chat room again early access to live show tickets just go to the rest is politics comm that's the rest is politics comm the world we're in is one where AI companies are building the equivalent of nuclear power plants and we just had our first case for when we upgrade the plant a nuclear bomb also falls out of it you say oh this has some implications are we in a world in which we're essentially saying that a bunch of voluntary good guys build these amazing weapons and then they just choose not to release them but nobody can tell them what to do we're entering an era where you actually need to do serious coordination including between governments to keep having all gas no brakes it's not sustainable to keep this going we have to figure out that coordination both within the industry and maybe more importantly and more challengingly with China if everyone gets access to it you're now rolling like a whole bunch of loaded dice with with the lives of people around the planet we have to somehow force a larger conversation on what do we want to do with this technology and how do we want it to be developed this episode is presented by IG so whether it's the vetting of senior political appointments or it's costly policy u-turns the last few months of UK politics have been a master class of what happens when you don't do your homework and the consequences as we've seen can ripple out far beyond the initial decision and the same rules apply when choosing your investment platform because picking the wrong one can hit you directly and in a very measurable way some investment platforms charge hundreds of pounds more a year than others losing you money that never got the chance to grow with IG there are no annual fees meaning more of your money stays invested from day one and on top of that you pay zero commission on stocks shares and ETFs so there are no hidden charges quietly eating into your returns and even better IG has been trusted by British investors for over 50 years so you know you're in good hands search IG.com to find out more IG trade invest progress your capital is at risk and other fees may apply this episode is brought to you by Vanta and is this is not your imagination risk and regulation really are ramping up and customers now expect proof of security just to do business that's why Vanta is a game-changer so Vanta automates your compliance process and brings compliance risk and customer trust together on one AI powered platform so whether you're preparing for an SOC 2 or running an enterprise GRC program Vanta keeps you secure keeps your deals moving over 14,000 companies from startups to big enterprises trust Vanta and these companies like ramp and rioters spend 82% less time on audits with Vanta that's not just faster compliance it's more time for growth get started at Vanta.com slash politics that's Vanta.com slash politics hello this week only we're bringing you an interview with Jack Clark the co founder of anthropic which people will know from Claude one of the great AI companies as well one of the fastest growing companies in the world and one of the most significant voices and leaders in AI today Jack's a Brit now as some of you might know Matt Clifford and I have been doing a mini-series for our trip plus members on AI over the past few months we got huge feedback I still think AI is the single biggest issue in all politics anywhere in the world more important than immigration housing cost of living or anything and I am hoping to draw you into that conversation in this conversation with Jack because I think it's a great way of engaging with some of the big questions in AI safety jobs employment the future shape of the economy and how on earth people think about a technology that could transform the world in really powerful ways for good or ill so I hope you enjoy it Rory and Matt talking to Jack Clarke welcome to the rest is politics AI with me Rory Stewart me Matt Clifford and we are very lucky to have with us today Jack Clarke from anthropic thanks for having me anthropic is one of the very largest fastest growing companies in the world oriented around AI the product which many people will have engaged with is Claude but they've also released mythos they've had a huge fight with the Department of Defense stroke Department of War Jack is one of the co-founders along with Dario Amodei and we are very lucky to have him with us today and rather unusually in the world of Titans of Silicon Valley is this himself British so welcome and thank you for joining us absolutely it's great to be on a British podcast thank you can we start by just take us back to your Britishness I mean how British are you I mean where did you grow up I didn't think it was you that would be doing the purity test I think my credentials are pretty good here I mean I grew up in Brighton my mother was a nurse my dad was a grumpy guy from South Shields who had become a copywriter and you know went to state school primary secondary college and then I went to the University of East Anglia so I've been English and being rained on for my formative years and what did you study I studied literature and creative writing which was a bit of a curveball because in school I did very well on the sciences and I got D grades in English so obviously I was like you know what this is telling me I'm gonna make my fortune with creative writing and studying literature but I'd always been a huge reader and I was very interested in going and getting an education which was actually very much oriented around self-study and reading and grappling with ideas through the form of fiction which is my main hobby and always has been and when is this when when are you going to university this would be 2006 or so so let's get a sort of bit of the census world so this is the moment at which I get Twitter and Facebook are just getting off the ground the iPhones just about to be launched so you're right there and sort of going to university at a moment where those sort of things are exploding mm-hmm and at some point you become kind of interested in technology you don't become casual issue guru instead you you set off in another path so I've been reading science fiction from when I was about the age of 11 or so and I'd always been very interested in studying different parts of science from you know how and colonies works to I briefly wanted to be a town planner so reading about cities and trying to model out cities of very basic computer programs to fractals and everything else and I have this memory of when I was about 14 writing a story in which ants simulated in a supercomputer eventually tried to bootstrap their way out of a simulation by building like an AI to break them out and then I read a story by the writer Greg Egan called Crystal Knights which is about crabs in a computer building technology to break from else not like damn it he's done it too but it was because I was I was just already at a young age very obsessed with technology and what it meant what I spent my time reading and writing about at university was what kind of stories can be told grappling with the implications of technology and how do you make it interesting because technology breaks a lot of narrative you expect it because you you became a journalist straight out of that yeah you mentioned creative writing as a hobby and for those of us who read your newsletter we kind of get to experience this every every week but was that always gonna be the side hustle and the journalism was the core I had no illusions about my chance of making it as a fiction writer of short stories in the mid-2000s I wanted to be a journalist because I was obsessed with technology and studying it and writing about it I felt like the most important story in the world was what was happening even back then with things like data centers and the build out of them I became a technology journalist in 2009 and I remember you know I was reading papers about databases that were being used by Google to train machine learning models I remember talking to the people that made that that software that you click on that shows you're not a bot on Google where it says click over bicycles I interviewed him in 2009 and I said this is very interesting when are you getting bought by Google and they were bought by Google three months later so I felt like I was on the money so one of the things he did as the journalist is you interviewed another great celebrity Brit in the world of AI you interviewed Demis Hassabis yes yes and you got a bit of a scoop tell us a little bit about what what what he said to you then yeah so I moved to Silicon Valley in 2013 to write about artificial intelligence I'd become obsessed of it it was clear to me that this was where the research was happening in Silicon Valley I started attending conferences reading about neural networks reading about deep learning and pretty soon after that you run into someone like Demis Hassabis who at that time had a small startup called DeepMind that no one had heard of but shortly after DeepMind was bought by Google in I suppose 2014 or so I had gone to a conference in New York where Demis was and I introduced myself by explaining to Demis I'd read a lot of his research papers and sort of quoting them to him which I think led to him tolerating my interest as a journalist and I pitched him and I did a lengthy interview a few months later and I asked him what he thought should ultimately be done with AI technology if we succeeded in building the sorts of powerful things which were then being imagined and he said he thought it should be sort of managed by or supervised via the United Nations that's on record in a Bloomberg Businessweek story from around that time. It's interesting I was saying this just before but I think it is worth saying that arguably the two most successful Brits in the history of Silicon Valley are Jack Clark co-founder of the fastest-growing company of all time and Mike Moritz the Sequoia Capital partner and fonder of Google among many others. Oh Demis! He's not in Silicon Valley though, he's not abandoned us yet. I'm like you Jack, you know. And they're both you and Mike Moritz trained as journalists. Yep. Is that a coincidence? It's not you it's like a humanities graduate. This would be what my colleagues in Silicon Valley call cope. I like to think that having a journalist mindset means that you ask a bunch of unusual questions, you're trained to ask questions, you're trained to be skeptical. I've always found that a lot of the work that I've done which involves supervising technical teams to ask like off-the-wall questions has been heavily informed by my background as a journalist where you're just trying to figure out what are the interesting questions to be asked. You know in the early days of Anthropic I asked could AI manufacture bioweapons and we set up a team to do that and now I'm running a team of economists asking well what impact will AI or potentially very powerful AI systems have on the economy. I think just by asking those questions you might get some good answers. Did it feel weird to make the jump when you jumped into working OpenAI or was it did it feel quite natural by that point? OpenAI is the big competitor of Anthropic that created ChatGBT and that's run by a man called Sam Altman. Sorry back over to you. It felt completely bizarre. You joined OpenAI and I remember talking to Greg and Ilya on the first day and we were figuring out what the whole job was and soon after joining OpenAI I said to Greg Brockman I think I'll just start going to DC and doing policy. He was very supportive of that and so I got on a plane to DC not knowing how to do policy but figuring out that it was going to be very important. How big was OpenAI then? I was among the first you know 25 or 30 employees like Dario had joined a month or two before me if I recall correctly. He was on my interview loop when I interviewed there. And what did it feel like then? What was what was the company like? What drew you to it? I'd been reading about all of these AI research papers and when OpenAI was announced it's hard for me to describe just how unbelievably stacked that team was and how notorious they were. Not notorious to anyone who hadn't been following it. I was saying to my colleagues at Bloomberg I was like good lord they got Ilya Sutskever. Oh and Dario and Greg Brockman. No one knows who these people are. All they knew was like Sam Altman who was somewhat famous at the time. Tell us a little bit about those three people so we get a sense of them because some of them had actually established themselves doing very remarkable research as academics. Yes you know Ilya Sutskever had been on the team up in Toronto with Geoff Hinton another famous Brit who had cracked the ImageNet challenge, an image recognition challenge in 2012 by training a system using neural networks on graphical processing units made by NVIDIA which subsequently became quite important. This is something Matt told us about in the where we've been doing this AI series together where he explained this lovely moment where suddenly there's this incredible improvement in performance. I at the time had been making charts as a journalist plotting performance on machine learning benchmarks and I remember putting the charts in and being like oh my goodness it's happening I have to move to Silicon Valley right now. So Ilya notorious for that had written many foundational papers at Google already. Dario had studied some of the early scaling work on things like speech recognition initially at Baidu and then he'd worked at Google and written a paper called Concrete Problems in AI Safety Grappling with Future Safety Issues of Powerful AI Systems in 2015 and it's hard for me to express how like completely berserk it seems to be writing papers about the safety issues of powerful AI systems when all you have are computer vision systems but just barely work some of the time and then there were you know Greg Bachman who was known as just notoriously one of the best most talented engineers in the world. He'd helped found Stripe which is an amazingly like fast-growing big startup now and there were many many other names like that so I from my perspective I was like this is the team it has a mandate to do try and make powerful AI systems for the benefit of humanity. If there was ever a moment to stack all my chips on AI it was then so I resigned from Bloomberg and I got many emails that day and comments from people I'd looked up to saying I was making the worst mistake of my life. Except at another level there must be a lot of people very jealous of you because presumably Silicon Valley is also full of tens of thousands of young men and some young women who are desperate to get in on the ground floor of what's going to be the next trillion dollar company. I think even at the time it was not that well-known there were a bunch of adorable misfits and weirdos who believed in what was going to happen with AI but it was not even that high status or well-known in Silicon Valley initially and actually you were competing against DeepMind which was already very established so it was already bought by Google by that yeah how are you gonna possibly be you know build another lab when Google has DeepMind. And where are you gonna get the money from presumably becomes a question quite quickly once it becomes obvious that these things quite expensive. Exactly and you know OpenAI was announced over 1 billion dollars funding amount from commitments from Elon Musk and others and so it was serious about their capital requirements. Now then I get there and you realize there isn't a plan. You're like what's what's the plan to build artificial general intelligence and no one knows because no one has an idea of what to do at the time. You don't have language models. You were running an amazing series of experiments at the company which were basically all oriented around ambition. Can we can we train an AI system that could beat humans at a complex video game? Can we train an AI system to operate a robot hand? Can we train an AI system to end up being able to generate text? Now that last one turned out to be very important but all of the other projects were important for having an organization that was continually doing unbelievably ambitious things that seemed like outrageously bold goals and through trying to work on those goals you built bigger systems that had ever been built to work for problem. You developed infrastructure that hadn't existed before. You encountered bugs at the frontier of scale which no one had encountered and you also started to build up this information of huh every time we dump more compute into these systems on these outrageous challenges they get better. So you started to develop this important intuition that actually we could scale the performance of AI systems not just through having clever ideas as human researchers but allocating more compute to the training of them which turned out to be one of the most important insights for what subsequently has happened. And it's worth maybe just spending a minute on something you already alluded to which is that you know Dario when he went to work at OpenAI had already done this work on concrete problems in AI safety and actually at the time if we think about the context in which OpenAI was founded it was actually quite oriented around at least safety from the perspective of things like concentration of power and other risks right? I mean we've recently because of this epic court battle between Elon Musk and OpenAI seen a lot of the you know founding emails and documents and there's this big thing of we don't want Demis to control this technology and today of course Anthropic I think even now described itself as an AI safety research company. Do you want to talk a little bit about the kind of importance of this idea in the founding journey of both OpenAI and Anthropic? Taking safety of AI systems seriously requires you to to hold in your head how powerful they might become in the future and I think that this notion of the technology not being as it is today but moving very quickly to become a lot more advanced has been both important for motivating the research agendas because you build a different research agenda if you believe something is about to happen in the future some some change up in scale but also safety has ended up being extremely coupled to our ability to deploy this technology you know as the technology gets more powerful deployment into society is gated by oh well if it's good at hacking how do you make sure that you can deploy the good parts of coding but not not hacking if it's good at biology research how do you stop the proliferation of bioweapon capabilities but how do you allow useful biology to happen these are very subtle questions that drive very very complicated research agendas and so I think thinking about safety allows you to think about what I think of as the most like ambitious technological optimist version of the tech which is where you've been able to deploy it very very broadly. Can I just develop Matt's point one more stage so from the outside it feels a little bit as though OpenAI is set up because Elon Musk is scared by a conversation he has with Larry Page and David Sassavis and he thinks they aren't concerned enough about safety and he sets it up for Sam Altman. Then the story emerges that Elon Musk is concerned about what Sam Altman's doing then another story emerges that Dario Moday your friend and you in fact leave OpenAI because you're concerned about safety so now we have a situation in which it feels as though half a dozen of the most famous names in AI have gone through a journey where each one of them is pointing to another saying we think you're reckless and not safe enough. I would think of it more that the great thing about about the world is you get to run a range of different experiments. I view all of these as organizational experiments. There's an organization that was already within one of the large tech companies trying to build AI. There was an organization that started as a nonprofit and then converted itself into into sort of a for-profit for capital purposes which is OpenAI that is trying to build it and then after a few years of sitting around you know working together I think we realized that we had our own vision for how to build a safety focused organization and we viewed it very much as the time to build like and run another experiment in this domain is is now because back in 2020 you can get a sense of how expensive the whole project of building AI is about to be. OpenAI started in 2016 of a billion dollars you're sitting there saying oh it's four years later we're gonna need a ton of money to do this so you had to do it then but I wouldn't claim that it's because you're claiming on on our end oh this approach is definitely wrong it's more we have an approach that we think is going to be subtly different and we want to like run the experiment in the most pure way possible just to challenge for a second one of the things we keep hearing is safety I mean certainly it seems as though that's what is driving Elon Musk in 2016 he's worried that services systems not safe enough and then there seems to be an anxiety that OpenAI isn't safe enough I mean so somewhere here and and this is one of the things that I notice when I'm talking to the founders these companies and I'm saying why are you taking these risks or why are you traveling at this speed often the answer is well I have to get there before this other person and the other person was often one of their friends ten years ago and I've got to get there before them because they're very dangerous and unless my models they're ahead of them we're gonna be in trouble and then it moves on to and anyway the US has got their get ahead of China because they're also very dangerous so there's a sense that we're in a kind of America's Cup race where we're speeding along and the storms coming in and the winds in the sails and everyone's saying why I can fix the safety but I can't fix it in a way that's going to slow up my ship because I've got to get there ahead of them I mean I'd say there was an era where you wanted to run a range of experiments on different organizational designs different research agendas that era has happened I think we're entering an era where you actually need to do serious coordination including between governments it's great that China and the US actually recently discussed AI of a high-level summit but also you're going to have to bring companies and governments and other parts of society together to talk about this because the the era to keep having all gas no brakes is probably probably you know entering the rearview mirror right it's not it's not sustainable to keep this going and we have to somehow force a larger conversation on what do we want to do with this technology and how do we want it to be developed. So if you were writing the report card on the world for how we're doing on this like where would you get to so like for listeners Jack and I have known each other for quite a long time from before Anthropic but we start really getting to know each other better when I was working UK government Jack was obviously doing his role in Anthropic and we worked together on creating the UK AC and the first AI safety summit and these were sort of early efforts to try and dip our toes into this coordination that was nearly three years ago. What's has changed? How do you think we're doing? Well now we have the UK AC evaluating both Mythos and GPT 5.5 from two separate companies on cyber security challenges which the UK AC has built in partnership with the UK intelligence community. Explain to people what an AC is. An AC is the AI Security Institute formerly the AI Safety Institute but they changed the name. Very good okay on we go. And the AC has built tests cyber security tests which neither of our companies has seen. We our technical staff believe in the legitimacy of the tests AC has built because we trust for the talent there which is extraordinary and the AC has built a test that governments around the world can trust for cyber risks because it wasn't built by a company it is them impartially testing our systems. If you'd said to me in 2020 we're going to build an entirely new function within the UK government that does frontier testing of AI systems the most powerful AI systems that will ever be built and they will invent their own tests that won't come from the companies and these tests will be better than the ones the companies build I would have said that's impossible. There is no way the government can build that function and yet it has and that function is being replicated in countries around the world it's been replicated in the US they could do with more money in the US than we'll get from that. So obviously you were right at the core of building this thing. How do we answer Jack's challenge here? It seems a bit weird doesn't it? I mean given these companies have literally are spending hundreds of billions of dollars and are able to pay single programmers 200 million dollars a year how on earth does a government put together something that can actually run independent tests that actually do any good because I can imagine the American companies swaggering around and saying forget it you're never gonna have the money you're never gonna have the talent this is never gonna work. Well and some of them did not Jack but I remember a very well-known person in AI saying to me I just can't believe that you'll be better than us at you know machine learning evaluations. I remember saying no but I really hope we're better at bioweapons and so joking aside that is part of the answer I think and you know to Jack's credit I think he saw this very very early that both from a legitimacy perspective and a capacity perspective there are some things that only governments can do. The reason I sort of slightly messed up my life in 2023 to get involved in this is that I really believe that it was both first order good like it's just a good thing for the world that these things AI Security Institute exists and can do this work but I also think the second order effective building state capacity in the UK to have a lot of AI experts in government actually understanding what's happening would have a bunch of second order benefits for the UK. Now to your point could you know Anthropic Open AI Google you know pay such people more to absolutely but I think if you believe as I do and I think Jack does that this is gonna be the most important technology you know of our lifetimes probably ever then you need I think you need to believe from a pure democratic legitimacy point of view they can't be something that's done to governments. Governments have to start to build the skills to do that. How do the governments get the money to pay people in a competitive rate if they can all go off and work at Silicon Valley for much more money? Well two things I'd say one it's extraordinary how quickly the amounts of money have changed you know I remember securing the first hundred million for AC or what became AC in 2023 and it seemed like a colossal amount of money at the time and obviously now it's it's not but I do think and you know this is this is like slightly off-topic but I think it's worth saying I do think that to me one of the lessons is that when governments actually have a real clear mission with a degree of urgency and top-down support from the Prime Minister both Rishi Sunak and followed up by Keir Starmer actually there are a lot of people even people whose opportunity cost is earning huge amounts of sums in big tech firms that are willing to take up the challenge. So I'm not saying hiring AC has always been easy obviously don't work there anymore but I definitely think the caliber of talent that you've been able to get through the mission has been very impressive. Okay well Jack the problem that we've got is that AC is not the United Nations of which Demis Hassabis dreamt in this interview that he gave to you. It's effectively voluntary right and it's the UK government and the UK government is not the US government it's not the Chinese government. One of the things you're right that you know when you're being optimistic you say isn't it lovely that US and China discussed AI safety actually if you look at that summit what's shocking is how little Xi Jinping and Donald Trump discussed AI safety that's 50% of the world's economy those the two countries that have all the foundation models that matter in the whole world sitting in them and they're not really doing it and so it doesn't I think ultimately really matter if plucky little Britain gets on and pushes ahead with its stuff if 50% of the global economy is not playing ball. So push or miss where how do you trust a plane that has taken off in another country with a different governance system and different regulatory system to your own? It's because you have common standards on things like aerospace safety and testing authorities which exist in each of these different countries or sometimes of a regional level and planes are able to take off in countries including countries which are at war with one another sometimes and land in each other's countries because you have reciprocal technocratic standards organizations that are actually facilitating some shared notion of safety. Now the things that like the UK AC which have been replicated in the US China has its own efforts here other countries have their own is exactly the beginnings of what you need for some of this notion of what standards look like. Now is that going to cover everything? No for some of the larger risks you're going to need bilateral or multilateral agreements between countries and high-level diplomatic discuss give us examples of some of those bigger risks that would need something bigger I would say the question of what you do about proliferation of national security capabilities that touch on cyber or bio or like nuclear is exactly the kind of thing that traditionally has been the domain of nations trying to talk with one another about non-proliferation regimes, safety regimes, testing regimes that is clearly an area where we're going to need to have some agreements with teeth eventually. So let's just dig into one of them which which might raise bioweapons. Yes. Explain a little bit to the ordinary intelligent listener to our podcast what are the potential threats of bioweapons and why you would need a particular structure to deal with. The best way to think of it is that biology is inherently dual use right it's it's it's the science of what what our bodies are made of. Now the experts that can build things like vaccine candidates also have the same expertise needed to build things that could cause terrible havoc in the in the realm of realm of viruses. Why don't they? Well the world doesn't the world doesn't incentivize for this the world doesn't want this to happen and also there are things like biological weapons treaties this has been a topic of discussion among governments for a long time and also the the basic goodness of people there aren't that many people that want to just visit harm on on others there are some but not many. And those that do typically don't have the capability. Those that well the number of people this gets to the point right the number of people who want to visit harm on others in the world is relatively small for this kind of horrendous act and they typically aren't trained biologists and if they are carrying out acts of terrorism or usually doesn't requires you to not be a lone wolf that requires some amount of coordination. Now the risks of AI systems are AI systems are universal educators and if you take either an individual or small set of people that want to commit some act of bioterrorism and they have the ability to access a universal educator which is versed in every aspect of biology then suddenly those people have been accelerated and they've been accelerated without paying the coordination costs or conspiracy costs that typically allow us to find groups of maniacs in the world. Additionally at the state level states are controlled in this domain by many agreements between states to restrict this area because there is no interest in things that have the potential for vast collateral uncontrollable damage. States prefer military capabilities that can be a lot more targeted. A bioweapon is almost the definition of a thing which is very hard to target and has huge spillover effects. So for bio you need to basically solve for two challenges. One how do you make it hard for individual actors to access knowledge like street-level AI that allows them access to knowledge that would allow them to cause harm and I think that problem has been worked on by industry and government for many years now so far with some effectiveness. The harder challenge in our future though is what happens when we have capabilities that are the same as or better than the best single group of biologists in the world which is what we have right now with systems capable of cyber offense and then the question becomes okay how do you think about access at the state level or the corporate level for these capabilities which are now are now transformative at the high end as well. Should we switch to cyber then because it's very live and obviously Anthropic has been in the news on this on this topic so do you want to do 30 seconds on Mythos for the 2% of listeners who haven't seen it all over the news? Anthropic trained a model recently called called Mythos. It is a standard AI model that is you know uses standard techniques same as the ones that the other frontier labs do and it's very very good at a range of skills including aspects of cyber offense and cyber defense. And it's a general model. It also would be quite good at writing Shakespeare's sonnets. It's great at creative writing, it's great at coding, it's great at biology but it's similar to a few months ago where AI systems got good enough at coding but suddenly loads of programmers started using them. It went through some you know hard-to-predict point of being sufficiently good at cyber but it gets interesting from the point of experts. And again just to explain to you by cyber you mean the capacity to launch cyber attacks, cyber crime? I mean the capacity to find bugs in software like Firefox or Windows or your iPhone and hack into it which is then the key ingredient to cause to carry out hacks or cyber crime. Can you say a little bit because obviously we're already here in this world. This is not no longer a thing that might happen in future. It's already here and Anthropics already had to make some decisions about how to deal with this and one of them is this sort of structured access. I don't know if that's how you describe it. We are running an experiment right now into how do you how do you take systems that have this capability and you try to diffuse them into the world in a way that is is defense dominant. Because what we know now that this system exists is okay at some point in the future it will be systems like this will proliferate. Many people will train them. The kind of water level of hacking skill in the world will have risen generically as a consequence of these systems. How do you deal with that environment? Just to understand presumably there are two different things. One of them is that the thing will be much better at finding the bug or the little backdoor into your system. The second thing is that it presumably could mount many many more attacks very very quickly and respond very quickly to the defense. So instead of some dude sitting in their backyard trying something, failing and then trying again, this thing could be generating tens of thousands of these attacks and learning all the time. So there's an amazing opportunity here right? You have systems that now you can turn on to the world's most important software and ask it to find bugs in it and indeed it is finding thousands of bugs in widely used software at a rate far faster than what these organizations because we've shared it with third parties you know like JP Morgan, Microsoft, others have found in the past. Okay now again sorry to keep interrupting but I just to bring the audience along. We were told two years ago, don't worry about this because exactly the same moment as the cyber attack capacity increases, the cyber defense capacity will increase. Who told you that? Well I don't know, but I mean these are people that you know well and in fact some of them you were talking about well before we came into the room but I don't want to drop them in it but I literally was with them two and a half years ago. I raised exactly this problem and I was told very very straightly by these people, don't worry about it because cyber defense will increase at just the same speed as cyber attack. Well I mean it's the issue of say cyber attacks or say biological weapon attacks is the defender has to be right all the time, the attacker only has to be right once. So these things don't have a relationship of being like symmetric at all. They're extremely asymmetric and so a lot of what we're trying to do is figure out okay as these capabilities arrive, how do you give defenders an advantage? The main thing you can do is give them time which means finding ways to release these systems to what I think of as a you start with a small circle of organizations and then you try to learn how to expand the circle over time. We are at our small circle of organizations today with Glasswing or have access to Mythos. The goal is expand it over time such that they can use this to raise the kind of defensive posture of the world and also get intuitions for how you can use AI systems to change cyber defense. And it feels like Mythos, even if from a capability perspective it's another point on a relatively smooth curve, I think that for large organizations it crossed some threshold of relevance where I think even just in the last few weeks far fewer leaders of large organizations within the public sector in the private sector like yeah but it's tech bro hype. It feels like it's been a helpful thing for getting the world to take AI seriously. Maybe the way to think about it is your CTO has to care about coding capabilities and but your software engineers are getting accelerated but your CEO general counsel and board needs to know about cyber attacks and vulnerabilities and so Mythos raises to all of these organizations AI just got real in a domain which you all care about beyond the technologists. Okay quick break and then back for more. So can we just go back a minute and talk about you talk rightly about this sort of this capability will broadly diffuse you know right now Anthropic has it arguably OpenAI has something close to it maybe Google does but you know this is very small and as you say you've been able to the structured access program you've chosen who who gets it. As you say everything we know about AI progress to date suggests that that won't be true for very long. What's your current take? So what won't be true very long that a tiny handful of companies have models that capable you know. You're assuming that quite quickly in six months time Chinese companies will catch up or whatever. Somewhere between you know zero and twelve months I think we can expect but I'd love you to push back if you don't agree an open source equivalent capability but that's likely from China but you know possibly somewhere else. One do you think that's true and two like given what you said about you know attackers don't have to get lucky ones defense have to get lucky every time. What do you expect the real-world consequences of that to look like say a year out? Yes I think that in the order of a year another model will arrive that proliferates generally. Absent that is also a policy choice so it could be the case that government like the Chinese government says we shouldn't proliferate an open weight model that's capable of cyber hacking. There are very incentives and disincentives on this side. What happens to the world? Well you will see likely a rise in some amount of hacks. You will also see a step change in how organizations approach computer security. I think the place that you end up with is one where computer security looks more like the white blood cells in your body where you have many many AI systems running as Rory said all the time at speed patrolling your organization in the software and continually finding and fixing bugs and it will be a new more robust way to do computer security than before. Just as when our own immune system encounters a new virus and we have no defenses for it some people get sick the same will be true in the cyber environment. We will see bad hacks likely proliferate due to AI and the world will go through some period of adjustment. I do think that on the other side of this you end up with likely a more robust world from a cyber capability than we've had before. Let me be challenging for a second and unfair maybe. I guess the anxiety if you were listening to this is that's maybe a little bit too confident and optimistic. You've sort of described a scenario which is a bit bumpy but we come out the other end and things are a bit better. Somebody listening to that might say well isn't there a scenario where actually that could be pretty catastrophic that that period you've just described before the white blood cells get going and some companies fail to adapt some do might actually be a description of AI models unleashing tens of thousands of unbelievably aggressive effective cyberattacks which could do shattering damage to the global economy. Some might say that this is a scenario that you could encounter as well. Some of this is a choice. It is a choice not just on the part of companies but how seriously governments take this and how aggressively governments and companies work together to go around critical infrastructure and other providers and secure it. Okay my follow-up question. So you basically that the narrative we got is you chose you you realized you had this thing you chose not to release it thank you right but again listening to that it's a bit like well these guys kind of built a nuclear bomb and they decided that was a bit dangerous so they decided on their own volition not to let anyone have it. That's a bit worrying because that implies there isn't at the moment a government regulator or somebody who told you you can't release this. If we're relying because that now we have to gamble that it's not just you're being good guys. Yeah. Apparently the guys running Gemini have to be good guys. The guys running open AI have to be good guys. The guys running Grok have to be good guys. I mean are we in a world in which we're essentially saying that a bunch of voluntary good guys build these amazing weapons and then they just choose not to release them but nobody can tell them what to do? I think the world we're in is one where AI companies are building the equivalent of nuclear power plants and we just had our first case where when we upgrade the plant a nuclear bomb also falls out of it. You say oh this has some implications. Now as a society do we want there to be more nuclear power? Of course we do. Do we also want to like manage the risks of the nuclear power? Of course. Do you want to deal with a potential proliferation problem of nuclear power plants spitting out nuclear bombs? Absolutely. Do you need laws for that? Yes of course you do. Like I'm not I'm not sitting here saying leave it to industry. No the situation I've described is like extremely unusual and I think the thing which we're trying to do with Mythos is we're trying to tell the whole story which is hey good news it keeps being the case that as we make these AI systems better they are more capable and many of the things that we want them to be capable at like discovering vulnerabilities in code advancing science advancing our ability to to push forward health care and also it turns out that as you make them better at this dual use capabilities are now showing up at a geostrategic. This means we can't treat it like a normal technology. We were going to have to change our mindset. By sharing it with a set of companies and with organizations like the AC we've made it so that you don't need to trust the claims of the originating company. You can ask them or you can ask for AC and that has generated I think the best information for the world but this is legitimate and it gives us time to work out what we need to do and of course as part of that you should have some regulation that says you don't get to choose whether or not to release a nuclear bomb but but you also don't want to have a regulation that prevents the nuclear power part because then you're going to be where where the world found itself after overreacting to nuclear power issues in the 70s where we just stopped building it in large swaths of the world and lost out. Let's up the ante a little bit on that. So you recently wrote what I think is a brilliant essay that everyone should read on this idea of recursive self-improvement. So you'll finesse it better than this but effectively AI either can improve itself and therefore you get you should get you could get some sort of intelligence explosion we can get into whether that's the right way to describe it. Just explain it again for the audience. Why would it explode? So effectively right now you know AI progress is bottlenecked on Jack's very smart colleagues and their counterparts and you know they need to keep coming up with good ideas and they need to keep improving the machine. Jack's written a great essay about the timelines to a machine improving itself and remove at least one bottleneck. When it's improving itself it would be as though the machine had a hundred thousand of the best software engineers in the world working 24-7 and suddenly it would be able to improve much more quickly. So this you know to Jack's you know I think really productive analogy of we're building power plants that spit out nuclear bombs you know I think I think you can up the ante and say you know like that that is like the next level of that. Then you're building power plants that build themselves which also have the property of occasionally spitting out nuclear bombs. I suppose like some people would say well you know Rory said you know is there you know we want to make a choice there's regulation you say yeah we should make a choice. I suppose like one more profound worry would be do we have a choice? Not in the sense that you know I mean in some like technical sense of course we have a choice but there is this sort of you know some of my you know much more you know sort of left-leaning friends would talk about you know the sort of techno-capitalism you know that actually there is no choice here. We kind of have to do this at some level. We are just to do what? We have to allow this to happen now. Because the techno-capitalists want us to allow it. So I think there is a point of view that like that actually the choice runs counter to all our incentives. You've already brought one of the arguments that's used like if we don't do it someone else will do it. You know that's true both within countries you know between competitors and it's true across countries. Imagine that we self-deny the right to build nuclear power plants that build other nuclear power plants that spit out nuclear bombs. China will do that. Russia will do that. And so there is this sense of like I'm really interested in like where is that choice? Now I actually think that we are not just our incentives and that virtue is a thing and we can make choices but what do you say to people that say it sounds like you're describing something where actually there is no choice. We have to do this. Well there's a choice which is basically on a spectrum between like maximizing individual sovereignty and maximizing like safety or what might critically be called paternalism where you know maximal safety is no one gets access to it other than maybe just the government or like the safe organizations and maximum you know liberty is everyone gets access to it. We trust in the ability of people to like experiment with this. Now obviously both of these are ridiculous like positions like if basically no one gets access to it you get essentially none of the benefits and you centralize power into like one or two entities or a small handful. If everyone gets access to it you're now rolling like a whole bunch of loaded dice with the lives of people around the planet. There must be paths through this that look more like gradations of access to the technology that we allow to come to. You're saying essentially that we have to figure out that coordination both within the industry and maybe more importantly and to Rory's previous skepticism more challengingly with China and I think I'll get it wrong I think you said you thought there was a 60% chance by the end of 2028. Yeah so we're talking you know two and a half years to coordinate you know the within industry thing is quite challenging but with China on not only that we do this but how we do it. Paradoxically I think co-ordinating with China is easier than coordinating between these bunch of people sitting in Silicon Valley. I think that I'm less worried about China than I am about how on earth you get. I think countries have the enlightened view that they're around for a long time and there's an interest in reducing chaos and I think industry has the incentives of we might be around for a very short time and we're in intense amounts of competition. So I agree with Rory but actually I think it may be easier. I think the Chinese Communist Party is relatively rational. I think if China is convinced that this thing represents some existential threat to the Chinese Communist Party to humanity to etc and they're six months behind they have every incentive actually to come up with some regulation. But it takes two to tango. The people that I find much more difficult to understand coming on board as I don't really wake up in the morning and think Elon Musk, Mark Zuckerberg, Sam Altman these are the guys that are ready to sit around. I was actually thinking of you last week when Sam Altman and Elon Musk dutifully turned up to court to give evidence in this thing and I think like one thing, Jay, I'd love to know whether you given your some of your recent experiences with you agree with this in the abstract you might want to talk not want to talk about specifics but actually one encouraging thought on that is actually American companies show up to court and they sort of broadly do what judges say and they do what their governments tell them to do and actually if anything I think what we've learned over the last six months is that governments will be very assertive what a particular the US government be very assertive it has a ton of levers at its disposal to make companies do what they want to do and the idea that the US government couldn't force coordination between these three or four private companies I think that seems very unlikely to me my big bigger worry is maybe the CCP if feeling it's dealing with someone that's coming to the table will make a deal but actually coordinating those two countries to do a deal is not a sure deal. Yeah I would say let's just look at the the general landscape all of the frontier AI companies ended up doing bioweapon classifiers all of them have ended up sharing lots of details of one another now it's not been made mandatory but I think that's almost beside the point because you can show that industry like self-coordinated on to hey let's not randomly proliferate things like bioweapon risks into the world. That feels like an even lower bar than showing up to court. We didn't proliferate bioweapons so we show up to court okay like that's two wins we'll take it now that to me is is is at least a proof that you can do this basic form of coordination it does require two to tango it requires there to be greater political will to do something tougher on on regulation we have a regime of transparency and transparency reporting right now which is almost like tell us about your manufacturing processes and details of the labels you put on your AI systems clearly it'll go further and in the same way we have aircraft automotive and food testing for safety before you ship it to consumers or I'm a recent dad lots of my kids toys get tested effectively to make sure that when my kid inevitably eats it it's not fully covered. I'm worried by this Jeff because actually the story of that regulation was terrible there were a lot of kids getting poisoned with lead in their toys before that happened and the story of all regulation has this property but we have surely we can learn but that is the fact you say that is a reason for optimism right or pessimism because unfortunately the examples you've given are much much more slow moving and the tolerance rate for failure is much higher so food safety you know humans been eating food for 150,000 years a current species right and we've learned a lot about what kills us and what doesn't and a batch kill someone here and eventually we get up for food safety but you're talking about a technology which in two and a half years can be spitting out bioweapons nuclear weapons etc so and again getting to the aviation safety is a slightly different story I was with aviation safety people this morning China is completely obsessed with planes not crashing which is one of the reasons why technological development in China is really slowed and one of the reasons why it's gonna take me eight years to build a jet engine and then another seven years to roll it out is they're terrified about planes falling out of the sky we're not talking about that kind of industry you know encouraged though Rory by the how active the interest of the US government is in this from a national security perspective I'm well okay let me come into that in a second because I think that actually raises another type of question but let me let me just sort of put out my thing right yeah okay what I worry about is that if in two years time something really catastrophic happened I don't know critical national infrastructure critical energy I mad tens of thousands of mad cyber attacks bioweapons released etc and somebody replayed these conversations that we're having they would not be that impressed because the gap between the catastrophe and the sort of reassuring stories about well the US government's take national security seriously and the companies have you know voluntarily done stuff on bioweapons and you know we all know we need to get to regulation it doesn't feel like there's quite the sense virtually I mean I had this recently with with one of the big companies saying to me after an hour and a half argument listen I agree with you we should be regulated but you've got regulators all of us together if I were you I'd be going out there and doing the regulation you can't expect us to do it because we're in a race with all the others but what I don't see on the other side is where are people generating all the details of what it is that they want to test these companies on regulate them on in fact often what I hear is the company's playing devil's advocate and saying well come on you want to regulate us tell us what you want to regulate you and you're not telling us what you want I I disagree quite quite strongly I think it is the ability of a choice of companies to do stuff beyond what is mandated by regulation we do this already today where we commit to a range of safety testing of our own of our own products and we publish it and we say we think that ultimately this should be what regulation should look like but we can generate information ourselves at all of the frontier companies there are people that care deeply about the safety of their systems I mean how could you not when you're building it it's not you know you don't get to stare directly into the heart of like the ultimate cyber hacking machine and say oh well this will be fine watched Oppenheimer right there's a moment where those guys think there is a non-trivial possibility that when they trigger the first atom bomb they could get a chain reaction that blows up the whole universe and they do it I mean we know that about human nature we know that humans can be very worried about things we know that 30% of the engineers and Google could be very anxious about these things but we also know from things that have gone wrong in the past that companies can make catastrophic mistakes notwithstanding the goodwill and the seriousness the engineers and what does the future looking back want it wants you to have some kind of mandated safety testing thing which everyone has to do and everyone has to go through and it wants you to have some notion of sharing details about these risks with society and it wants you to do something where society prepositions to get advantages or deal with risks or to say no if the risks seem intolerable relative to the gain all of these things have the shape of beginning to happen now and we as a company are in like vocal often support so vocal in fact that it sometimes causes us issues with other other other parties in this space and I believe that governments will will ultimately act it will just feel it will feel down to the wire because my sense and you you you two are experts in this but governments it takes a lot to move them to action and it takes a lot of evidence before a crisis for them to do anything before a crisis but it surely this applies um we've talked a lot about like what I would call national security type risk today for understandable reasons we for the rest of this miniseries we spoke a lot about the economic impacts obviously I think your new role anthropic or newish role at least in part is very focused on this well first maybe just headline like what do you think you know there's a lot of talk about jobs apocalypse there's a lot of talk about differential impacts within and across countries where do you think we are on that and then I'd love to dive into a little bit like what do you think government and kind of should do on that we're somewhat where we were with with AI and national security a few years ago there's an instinct that things are about to happen but are important there is no real measurement or testing infrastructure built within governments to do this and the companies have only just begun including with the the anthropic index to share information but all of those are the ingredients from which you can build a telemetry system to basically tie to make causal claims of if AI company does X why happens in the economy we absolutely have the ability to generate that data to do that across companies and government it should be regulated but companies share information about this index so the anthropic economic index looks at all of the ways that AI is being used or by by our customers in a privacy preserving way and it joins that with what are called onet job classifications which things like the Bureau of Labor Statistics use to classify changes happening in the economy and what this allows you to do is look at the economic activity happening on the AI company platform and join it with the same economic data used to reason about the economy writ large if we have any hope of being able to make strong claims about the impact of AI on the economy versus CEOs laying off people and saying it's due to AI but rather it was due to covert overhiring which is a sin many commit then we need to set up these kind of data sharing systems that absolutely can be done and is being done now I'm in here in England just meeting with people with the new AI and economics Institute in the UK government which aims to do just that but we don't know what the shape of a future economy is all I can tell you is as I can't reconcile the capabilities of these systems with the economy staying as it is today like clearly like massive changes will happen everyone but tries to predict this tends to be wrong in like ways for the comedic and outrageous in the future if you forced me to predict it I'd say clearly you get productivity multipliers on things that AI touches clearly you get the emergence of new companies that are able to do a lot more with way fewer people relative to previous generation companies and probably you were going to have some issue with early just out of school hiring because those are the people that have almost the least set of skills for the most replaceable by AI systems beyond that it's very hard for me to say what the shape of the future economy with AI is because I don't know if the productivity multipliers compounded also create new industries I don't know what the shape of like whether these new firms that are doing all of this this new business generation if they proliferate in a much larger number of a normal business formation you don't think that your vision of RSI this recursive self-improvement is incompatible with the world of human labor or you do I think that under something like RSI the economy grows so much that it's like humans sit on top of an economy that's hundreds of times larger than the one today you know a lot of economic doctrine is that what you end up doing is you end up validating and verifying the outputs of automated processes that's what happens in in large chunks of the world around us and things like manufacturing and pricing risk and figuring out as people how you make agreements based on the risk of what you do see the insurance markets see things like bond markets which essentially model risk at the country level I think that there will be ample employment for people in new jobs and specialisms we can't imagine that sit on top of this much larger economy but on route to that you're going to see like massive massive changes in the structure of the economy and in jobs but it's very hard to predict what those changes will be I think you can just bet there will be massive changes with confidence connecting those two two stories together if at three stories mythos risk and jobs one of the things that that worries me and you were talking about the US government regulating is that the US government could wake up in a couple of years time and say mythos 15 we believe for national security reasons is too dangerous to release outside the United States America could launch these horrible cyber attacks right which point these frontier a my models become proprietary within the United States Europeans can't access the latest cutting-edge frontier models and then there's a huge sucking sound as all the economic value is sucked out of Europe towards the United States where these AI native companies your trillion dollar company with three employees are set up in the US on the basis of US frontier models and then you make a lot of money and we sit around in Europe hoping that you're going to be feel that you made so much money that eventually you're going to in the way that I'm sure Donald Trump would love to share generously with the rest of the world the proceeds of your wealth let's do a deal yeah I think this is a not impossible scenario it's a very worrying one I think there's a couple of things that we need to need to work on one is it's again hard to me to reconcile the shape of this future economy with I guess current ways that we try to like tax or control corporations especially AI corporations the picture looks more to me like well these companies including us are going to have computers all over the world the computers are going to be where lots of economic activity is taking place there must be some way to more directly target that in terms of targeted forms of taxation and I'm not a tax expert I'm not claiming I have the answers I'm just saying a basic intuition is that's that's where the thing changing your economy is and it's it's it has a body and it's geographically distributed outside the US surely you can target that sounds like a good argument for investing in domestic compute it is a hugely good argument in investing in domestic compute infrastructure the second part is with with technologies that end up getting classified for you know military use or being deemed to be relevant to national security there is always this tension of sovereignty and building and other things and I have been saying to governments around the world since I started working in AI policy in 2016 or 17 you guys should build a big computer we're building big computers and it's giving us like outrageous amounts of leverage have you considered building a big computer I think the choice of building a big computer which you could do AI systems on is still there for the sort of European community plus plus plus England it is a choice and and the problem with this choice is for the numbers get more outrageous each year so the best time to start it was last year the second best time is now a bad time is next year and you see that you see the picture I think that has to be a part of this what if you learnt over the course of the years debating this stuff about what is productive and what isn't productive in these conversations one of things that worries me sometimes if I talk about safety to someone like you who's been doing this for 10 years as you've heard it all before your mind will be closed on certain kinds of arguments and you know somebody will say you know how about the bioweapon that's going to blow up the world and you're going to be like oh for goodness sake not again I've been talking about this for 10 years and here's my answer where do you begin to feel the conversation is a bit dead and inert and where do you feel the really life questions are positive questions are okay what are the exact regulations that you need to do we had a productive one here where you were sort of pushing me and saying well that doesn't really cut it what do we what is actually necessary I think it's useful to just get people to label specifics I think it's also good to actually talk about feelings this might be the fact I've lived in California for too long but I now say stuff like this but you know as as someone working on this technology of course I'm very excited by it and I work on it because fundamentally I think I think humanity has a huge range of challenges ahead of it this century and getting through them requires us to figure out smarter ways to generate power various science breakthroughs a huge range of medical treatments AI can absolutely help us do that but I'm scared of it I'm scared of the technology that I'm building and I'm scared of how it is governed less than like the toys I buy for my kids or the food I buy from the supermarket it seems like an insane situation to me and I think actually getting people that work on this to say how they feel is good because it makes us accountable for it like I'm I'm saying I'm I'm scared I'm worried about this because it makes me even more accountable to solve that you know so that for everyone else as well well maybe there's a very last one given that you mentioned your kids yeah number one question I got after doing this mini-series with her at the end of last year was I was the optimist by the way on the show you might be surprised to hear you know what do you think but what did it mean for your kids what are you doing for your kids like how you think about educating them what do you want them to do how do you think about that I mean I think that the the sort of childlike wonder and curiosity I have a three-year-old but that that children have is is remarkable and it reminded me of when I have that as a child on how school methodically beat the curiosity out of me as much as hard as it could and the thing that AI gives you is a machine that is leveraged directly by one's intuition and curiosity and basically having having areas of like huge passion and having curiosity about the world are things that are massively leveraged by AI technology and having you know rote skills or things like specific career plans or things which are almost disadvantaged by it so I'm trying to teach my kid I mean you can't really teach a three-year-old anything but in a few years encouraging this culture of curiosity encouraging obsession because the way that I've been most oriented to this technology is I have this passion project of writing newsletters and writing fiction within it and actually having a personal creative practice and hobby has been one of the best ways for me to both use AI in ways that kind of delight and excite and empower me but also one of the best ways for me to feel calibrated about how good it is because I have something I deeply understand and occasionally now I'm like oh it finally wrote a good bit of dialogue for a story okay you know it's good to get calibrated and I think it's just reminds you of the amazing excitement where you know in in a very realistic sense we have taught sand to think bizarre bizarre stuff is a fur and I think being awake to that is is is important as well thank you very much we love the teaching sand to think thank you thank you and look forward to speaking again soon we have absolutely we will be talking soon anyway in other setups but thank you thank you well we hope you enjoyed that so much to get into so many other questions we could have pushed harder and I think we need to keep pushing harder because Jack as one of the co-founders one of the biggest companies world is an example of an individual with incredible personal influence and power that could totally upend our economies our national security our public service and in fact most of the future of humanity and human society it's a very few people and we really need to make sure they're thinking clearly and honestly so we hope you enjoyed that and look forward to hearing what you made of it and don't worry I will be back next week