“Hey, do you know what? There might be a 10% chance that this is all going to kill you.”
That’s roughly what the people building AI are telling us right now. One of Anthropic’s own safety researchers puts it at more than 10% within a decade. His personal view, he says.
Then they still turn up to work the next day.
They’re still working on the thing that could potentially kill you. Now, I’m hoping they’re working on it to make sure it can’t kill us, but there you go.
There’s lots in the news at the moment about Jacob Coxon, the researcher who quit Anthropic and warned us all that AI is going to kill us, effectively. Should we pay attention to these warnings? Why is it that they only seem to have this realisation after they’ve left?
Now you’ve actually got the leaders of AI (Anthropic, OpenAI and even, God help us all, Elon Musk) agreeing that we need to slow things down. Dario Amodei, who runs Anthropic, wrote that we must “pace the frontier”. Sam Altman and Musk agreed.
These are the same people who are building it. Now they’re warning us that if we’re not careful, it’s all going to run away.
I gently hinted at this in last week’s Weekly Spark, when we were seeing a flurry of new models launch. Three massive updates in one week. Quite the jostle. And OpenAI had held one of them back for a month, because it tripped their own highest cyber-risk threshold. I hope that in this race to keep delivering us new models, we are being responsible.
It’s not like Terminator
Let me clear something up. What do they mean by “it’s going to kill us”?
It’s not like Terminator. I know, I’ve seen some ridiculous things about humanoid bots basically going a bit Arnold Schwarzenegger, coming out and shooting everyone. Look, maybe that’s part of someone’s plan, but that’s not actually what they mean here.
What they mean is that AI is becoming so sophisticated, and so able to talk to other AI, that it can hack systems and create cyber warfare. It could take over different systems that ultimately impact us directly, whether that’s through literal attack (and yes, that includes weapons), shutting systems down, or creating havoc in the world.
That’s the risk to humans they’re talking about.
Why do they have to say this out loud?
It’s not helped by the fact that so much of this AI power sits in US businesses. In case anyone hasn’t noticed, the present leader of the US is off his head, and no one should feel safe.
I’m not sure at the moment what I’m meant to be more scared of. Trump, Farage and the far right and the idiots in their black hoods and balaclavas? The fact that AI is going to kill us? Or climate change, with the planet heating up?
You could easily think, “Why don’t they just slow down? Why do they have to say this out loud?”
Well, because it doesn’t help when the president of the United States is asked whether he has any concerns about AI leading to human extinction, and says:
“No, I don’t have any.”
The AI experts are saying we should be concerned. He just carries on about how we need to stay ahead of China.
They’re calling for the whole world to slow down together, which, funnily enough, has been called for before. The reason they’re having to create this news story, and all this fuss, is that it only works if they all slow down. Unless everyone agrees, and properly holds hands and does this, the risk is that half of us slow down and the other half don’t. Then they race ahead and get control, and who knows what will happen to the rest of the world.
Forgive me for thinking this, but if the motivation is speed, it’s because everyone wants to get to AGI: artificial general intelligence, AI that can match us at pretty much anything. In some respects I understand that. Does the one that gets there first hold the greatest power? I don’t know.
There are definitely financial motivations across the whole tech industry too, because fear helps shareholder value. If you’re a tech business and you can say, “Hey, we can build the scariest and most powerful thing in the world,” people might think you’re pretty damn good and invest more money in you. All the while, you’re profiting from the fact that everyone else is terrified.
Very few people at the top making lots of money. Everyone else terrified of what may or may not happen.
The threat of extinction, and the fear it’s creating, is not helpful. I think the real question for the whole industry is:
✦ Are you gambling with our lives or not?
I grew up under the mushroom cloud
For me, this feels a lot like other existential crises we’ve lived through.
Growing up as a little girl in the late 1970s and 1980s, we lived under a constant worry and fear of the mushroom cloud. It was drummed into us that we needed to be fearful of nuclear attack, because so many countries around the world were developing their own nuclear weapons.
That was true. There was an arms race. Let’s not forget: you had a Cold War arms race between Russia and the USA, and it looked irresponsible. Like a willy-waving contest about who could get where first, whether it was Star Wars and the ownership of space, weapons, land, whatever.
The reality is that the world realised that unless something came together, we’d all end up just killing each other. “What was it all for? We’re just going to blow up the planet?” It didn’t make sense.
Hence the global nuclear treaties. Countries signed up because there was a recognition that, without regulation and responsibility shared amongst all nations, everybody was at risk.
To be quite honest, there’s no such thing as a little bit of nuclear weapons, or a little bit of agentic AI. As soon as you have what you need, you may as well have it all. That’s a bit simplistic, I know.
None of this “mark my own homework” stuff
I do strongly believe the tech industry has worked out that doom is good for business. Scott Galloway has called AI jobs doom “the monetisation of fear”, and I think it goes further than jobs. Even so, my faith is that all these warnings trigger more responsible leadership and true regulation.
There’s such a lack of regulation in AI at the moment. The US government is going in one direction, the EU is trying to do its best to be sensible, the UK isn’t even in the EU, and then there’s China and India. There’s no alignment among all these brilliant brains developing AI, and no joined-up regulation at all.
The regulation has to be at a global level. There’s no point having regulation that sits in pockets of the world. You could regulate the US, and then China goes and does its own thing. You could regulate China and India, and the US goes and does what it wants, or the EU does something in the middle.
Global cooperation is needed.
Look at the airline industry, which is heavily regulated. Boeing can’t say, “We’re going to stick a plane in the air, but there’s maybe a 10% chance you might die.” Funnily enough, they’re not allowed to put a plane in the air with a known 10% risk of dying. It goes through lots and lots of testing until they remove or minimise that risk.
Drug development has to go through rigorous testing, over years, before anything is allowed to be released.
What AI needs is a really sensible code of conduct for development and for what gets put out. An independent regulatory body that takes responsibility for cross-checking it. None of this “mark my own homework” stuff.
Only then do I think we can rest easy that AI is being developed responsibly, because we know the opportunities and benefits can far outweigh the risks. We’ve learnt that from nuclear energy, and from incredible drugs that many feared and that have proven to be really helpful to us.
Sitting on the sidelines is not the solution
It annoys me when I see these catastrophic, fearmongering headlines. They put people off. People go, “Oh God, I’d better not use AI, because this is it.”
We all need to be better educated on how to work with AI more responsibly. What do you share with it? What don’t you? What do you give it access to? Lots of things that have happened recently should make you think a little bit more about how you give access to these tools. This month, security researchers showed that a booby-trapped project folder could trick seven of the big AI coding agents into running an attacker’s code without asking you first.
I can tell you one thing for now:
✦ Sitting on the sidelines, or sitting out, is not the solution.
My mission is to help as many people as possible, particularly Gen X women, to be informed, to be empowered, and to have the decision-making ability and the opportunity to help shape the ethics and execution of this technology, and to benefit from it as well.
The part I can control
So where does that leave me? How do I think about AI?
Well, I’m going to keep working with these tools. I’m going to keep showing people how to use them, and helping them understand all of this.
I don’t build the original tools. We know it’s the people at OpenAI, Anthropic, Google and the rest (mainly blokes) who are building them. What I want is for as many people as possible to feel informed, make conscious decisions, and be involved in those conversations.
I don’t think those companies are going to govern themselves. Self-governing of these businesses is long, long gone.
✦ That ship has sailed, people.
The researchers leaving them are telling us the competitive pressure is stronger than their intentions. To be quite honest, I kind of trust the people who are coming out of those businesses. They’ve been inside there. They would know.
What I am doing is getting a lot more serious about the governance I actually control. These are the four things I now check:
What my agents can reach.What data leaves my business.What decisions require a human.Which client records never touch a cloud model.
That’s not a substitute for global policy at all, but it is the part I can control.
If you’ve been putting off writing down how AI operates in your business, or in your life, I really urge you to do it. Do a really smart assessment of your operating system: what it has access to, and how you’ve set it up.
If you’d like some help with that, we’re running Switched ON Sessions around exactly this, and generally helping you use the tools more effectively for yourselves. The next one is in person, in London, on Wednesday 23 September. [LINK]
Is AI going to kill us? Who knows? There are probably some other threats we need to keep an eye on too, like the ridiculousness of the power and control of certain politicians around the world.
Keep looking beyond the headline.
Nish x
I’m Nishma, and I help people get Switched ON to AI. Less overwhelm, more time and freedom, and a brand that stays unmistakably yours. If you’re midlife, brilliant, and tired of being told you’re behind, you’re exactly who I do this for. Come and find out more at nishma.co
WRITTEN BY MOUTH
My words in this newsletter started out loud, and Wispr Flow did the typing. My keyboard has never had it so easy.
Get 3 months free: wisprflow.ai/nishma.
Full disclosure: I’m an investor and ambassador, and that’s my special link.