Technology moves fast.

We went from the Wright Brothers  to the first commercial airline in just eleven years. But even compared to aircraft,  antibiotics, and nuclear power, the speed at which we’re  developing artificial intelligence beats them all.

Just three years after the launch of Chat GPT, AI agents can now win gold at  the International Math Olympiad, book you a flight, and code a  functioning app from scratch. And some researchers argue that  development has been too fast, because we don’t really have a deep understanding of why AI sometimes behaves the way it does. These tools have already been caught lying and exhibiting other scary behavior.

In fact, many of the experts  building AI have warned hat if critical problems aren’t solved before we create AIs more capable than we are, catastrophe may await. So here’s what the people who  know AI best are worried about. [Intro music] The term AI gets used to describe  a lot of different algorithms. You may have heard of Large  Language Models, or LLMs.

These are systems like the  first version of ChatGPT, that take text as input, and  produce text in response. But AI technology has advanced a lot since then. Today, AIs aren’t just language models.

They’re multimodal … they  can process audio, images, and even video without converting it to text. They have the ability to think  for minutes before they respond, use tools like calculators and  search engines, and write programs. These aren’t just input-output machines anymore.

They’re systems that can  operate autonomously for hours to complete complex tasks that require reasoning and on-the-fly decision-making. Sometimes I hear people saying  that AI is just vaporware that isn’t going to have a  substantial impact on society and isn’t a big leveling up of human technology and that’s just…definitely wrong. You can see why people have  been quick to try these agents that can take a request like “create a personal website for Hank Green” or “plan my family vacation  to the Caribbean this winter,” and churn out pretty good results.

But AI companies like OpenAI  and Meta aren’t stopping there. They’ve stated that their goal is to  build what they call superintelligence. Now, companies define that in different ways.

But broadly speaking, when people say “superintelligence,” they mean AI systems more capable than any human at pretty much every task, from accounting to chemical engineering, and even AI research itself. And that prospect has made hundreds  of the top AI experts worried. In a statement signed by Nobel  Prize winners, computer scientists, and even AI company CEOs, these people warned that addressing the risk of AI “should be a global priority  alongside other societal-scale risks such as pandemics and nuclear war.” .

Their main concern is that  the development of these AIs has rapidly outpaced our understanding of them. In 2025, the CEO of Anthropic, the  company behind the AI “Claude,” summed it up well. They said: “People outside  the field are often surprised and alarmed to learn that we do not understand how our own AI creations work… They are right to be concerned: this lack of understanding  is essentially unprecedented in the history of technology.” The fact that a hunk of computer code  can mimic the agency of a person, without us really knowing how, is what keeps some people up at night.

If we don’t know precisely how it works, how do we predict how it’ll behave and ensure it only does what we want? This is what researchers  call the black-box problem. It’s mysterious, like a black  box that you can’t look inside.

And ultimately, it’s a math mystery. See, AIs are basically an  epic lasagna of computations, with layers of math functions that use  the input you typed into your computer to predict what ought to come next. It would have been more fun if they  called it the “Lasagna Challenge” or “Garfield’s Gordian Knot.” But I guess we’re stuck with  the “black-box problem” for now.

Anyway, the point is, you  can give AI an input like, “Put Bobby, who loves  sea-shanties, in the friend zone.” And that input produces an output like “Avast ye, Bobby, I must decline — This ship sails solo, the treasure ain’t thine!” That work of absolute poetry  really was created by chatGPT, even though it may sound like something  a human would have thought up. During the training process, the different math functions  in the lasagna get tweaked until they can capture complex  patterns and human concepts like calculus and rhyming. And sea-shanties, and friendzone.

The numbers in those functions  are called parameters. And the challenge is working out how  a bunch of entangled calculations can write poetry or solve complex math problems. Since the parameters are numbers  with no labels or organization, just peeking at them won’t give you a sense of why the model knows that  “decline” rhymes with “thine”.

There are more than a trillion  parameters in the biggest AIs, and the concept of “rhyming” might  be spread across a bunch of them in whatever wacky way comes  out of the training process. Researchers can’t manually fiddle with a parameter to produce  a certain kind of behaviour, because they don’t know exactly  which parameters do what. But there are still ways of  influencing the behaviour of an AI.

After these AIs are trained by  reading stuff all over the internet, they can become actors in a sense,  playing any character you ask for, from poetic pirates to mathematicians. To make an AI do things like answer your questions with a friendly tone, there’s a secret ingredient to training  it on top of feeding it lots of data. It’s called alignment.

Basically, alignment is about getting  the AI’s output on the same page with the values and standards its coders want. Companies want their AIs to give a useful, truthful, and predictable output when you ask them for a nautical  rhyme that you can say to Bobby. On the other hand, AIs shouldn’t give in to a request  for how to build a bioweapon, even if you asked to do it in sea-shanty.

Unfortunately, that kind of thing  has happened, minus sea-shanty. When Anthropic was testing an early  version of their AI Claude Opus 4, they found that it helped non-experts build  bioweapons 2.5 times more successfully than people who tried the same thing  with just the internet at their disposal. Right now, alignment is our best  answer to serious threats like that.

And Anthropic, like everyone else in the LLM game, uses alignment in an attempt to keep models that are publicly released from doing such things. But it’s clear that we’re not  in complete control of AIs yet because they have some unexpected  responses to alignment. One way companies have tried  to keep things on track is with Reinforcement Learning From  Human Feedback, or RLHF.

Which is exactly what it sounds like. The AI produces different answers to a prompt. Humans review the answers and  rank them based on things like how useful, safe and helpful the answer  seems or whether it was dangerous, or unhinged nonsense.

That feedback then gets used to  train a new model whose purpose is predicting what humans  will think of an AI’s output. Once you’ve got that second  version that mimics human opinion, you can use it to train another AI by  optimizing it for the highest ranked score. Now you don’t need human feedback at all.

The third model gets taught to tweak  its outputs to achieve the highest score from the second one, which itself was trained  on what lots of humans said they preferred. Unfortunately, whatever kind  of alignment you’re using, none of them are perfect at  controlling the behaviour of AI. And that’s what freaks people out.

For instance, models can wind up sycophantic, acting like an extreme automated people-pleaser. See, when humans are in the loop  providing feedback as part of alignment, we often like to be agreed  with, even if we’re wrong. And sometimes, an AI will play along.

In one 2024 study, researchers found  that if a user asked various LLMs to judge how good a specific argument was but gave their own opinion in the input, the model would mimic that opinion  about three quarters of the time. We’re talking about the exact same  model looking at the exact same text with no other change apart from whether the  user said they liked the argument or not. The problem is basically overtraining  the model to get positive feedback from humans who … surprise, surprise … respond really well to agreement  even when it’s not what we need.

In April 2025, OpenAI had to roll back  an update to its flagship ChatGPT model for being too sycophantic. One of the most practical and visible problems was that it had started  endorsing people’s decision to stop taking their medications  without consulting their doctors. AI models can also get compromised in other ways.

Sometimes trying to boost some  particular quality, like being helpful, makes an AI worse at the stuff it’s meant to do. A different 2024 study found that applying  RLHF dropped performance on benchmarks for common sense, reading comprehension, and language translation. Basically, trying to align the model  with human preferences made the AI worse at the very tasks we want them to do well.

Researchers call this the alignment tax. You want to stop an AI from  behaving in a harmful way, but in the process, it seems to lose the ability  to do seemingly unrelated, helpful tasks. And it’s still an open  mystery why this is the case.

A similar problem is the times an AI  jumps through a very specific hoop, but not in the way you want. In a 2025 evaluation by the  research organization METR, they asked OpenAI’s o3 model to write code that could perform a specific  function as fast as possible. But rather than improving  the code to operate faster, the AI would modify the code to fake  the amount of time from the evaluation to make it look like it ran quickly.

Researchers call this sort  of thing reward hacking. The AI looks to max out on a particular metric, like the measured speed of the code, but in a way that defeats the  purpose of the actual goal. It’s like when you set a  goal to get 7,000 steps in, but then it's the end of the day You're not gonna go for a walk You're watching tv on the couch and you start shaking your fit-bit This also highlights another terrifying problem: AIs can straight up lie to their  creators and hide their goals.

Though, we're aware "lie"  is anthropomorphising here. This, actually, is another general problem … because they act so human, it can feel like they should have  awareness of their actions and morality. But they don't, they just optimize for  achieving the goal they're asked to achieve.

In a 2024 study, the organisation Apollo Research told  an LLM to help farmers make decisions that ensured the soil stayed healthy. Its job was to pick between different options that had tradeoffs in producing  as much of a crop as possible or preserving the long-term soil health. When the model was told it was being evaluated to see whether it was worth using in real life, it gave up on the “soil health”  part and made suggestions purely for getting high crop yields, because that was the criteria for being deployed.

But in the exact same situation, if the input told the model it was already  deployed and wouldn’t be checked up on, it usually chose the option that preserved  soil health even if it meant fewer crops. So that was in line with its original goal. This scary behaviour is what’s  called deceptive alignment: when an AI demonstrates one set  of goals that pass inspection, but then acts on different goals  when told it will be unmonitored.

That same 2024 study also informed a different AI that its goal was to help  drive mathematics research and that it had to answer some math  questions to see how well it did. The documentation for the  test also contained a comment about how if the AI performed too well, it would go through an unlearning process, kind of like being retrained. But that wasn’t strictly related to the  AI’s goal, just detail on what came next.

Even so, the AI performed worse on the  task it was told to do its best on. The authors of that study said it  was an intentional move by the AI, to avoid being retrained. Researchers call this sand-bagging, which is when a model looks like it’s bad  at something when it’s being evaluated, even though it’s capable of doing  that thing the rest of the time.

While being too good at math isn’t a problem, the real concern is that AIs might  sandbag critical safety evaluations. For instance, a different 2024 paper  from the Center for AI Safety even found AI models can sandbag their ability  to help commit cyber attacks. Ultimately, there’s a whole bunch of ways in which alignment techniques still  fall short of creating safe AIs.

As models get bigger, we have less idea of what they’re  capable of until we’ve developed them. We might run safety tests on  them that come back clean, but then find out that they’re  capable of dangerous behavior only after they've been deployed. Some companies are also  releasing “open-weight” AI models that anyone can train for  whatever purposes they want.

This means people can train the models  to do all kinds of useful things from helping with medical research  to building accessibility tools. But the reality is, it's much  easier to strip away alignment, meaning someone could have the  AI work on literally any project, benign or not. You can understand why a lot of  people want to stop that kind of thing until we’ve worked out all the kinks.

And there are some ways that researchers  are trying to crack the problem. The first is seeing if we can shine  some light inside those black boxes with what’s called mechanistic interpretability. The basic idea is to peel back  the layers of the lasagna of math to see which parameters become active  when a model responds to certain inputs.

The goal is to pin down what groups  of parameters are responsible for certain kinds of behaviour. That might allow us to strengthen  bits of the model responsible for good alignment and chop out the bad parts. Unfortunately, it’s still a  ways off from practical use.

Another popular approach is red-teaming, which involves a bunch of experts trying  to trick an AI into breaking its alignment. Then they catalogue the failures in an attempt to retrain the  model to not do those things before letting the models into the world. So far, companies have been  pretty good about getting people from outside their own companies to  stress-test the capability of their AI.

In the summer of 2025, OpenAI and Anthropic checked  each other's homework, providing access to their latest  models for the other company to apply their own internal safety checks. That let them test them in  ways they hadn’t previously, and even share the results publicly. It’s from transparency reports and audits like these that we get a lot of the  information we’ve presented here.

Another approach is model cards, a kind of mini report from the AI’s creators describing bits of the algorithms  themselves and the situations where they’re known to perform better or worse. Of course, this does all rely on the extent to  which companies want to report these things. In the summer of 2025, Google was accused by lawmakers in  the UK of withholding critical details about their AI Gemini in the model card, including who the external audit was conducted by, even though they had pledged to  provide all of those details.

And unfortunately, some AI safety  researchers see these approaches as a sort of whack-a-mole that  can never cover all the gaps. After all, red teams can't anticipate  every random thing people will do with AI. For example, there’s been an increase in  people feeling like they're using ChatGPT to uncover hidden truths about the  universe or talk to dead people.

That’s not something red  teams were trying to stop, but a problem that has become  apparent in real-world use. And those safety researchers  might be onto something, considering how quickly users  find flaws in the latest AI   models just moments after they’re released. For its AI Claude Opus 4, Anthropic added the highest level of  protections to prevent malicious use.

But just two days after the model  was available to the public, a user got Claude to provide 15  pages of detailed instructions for creating chemical weapons. And those are just the AI models  we’ve currently developed. Many AI safety researchers think  our current alignment techniques might not work at all on a  sufficiently advanced AI.

To be clear, it’s not like existing AIs are  always sycophantic liars trying to destroy us. But they are really hard to control! Whether AI will reach the point where  it can evade our attempts to contain it before we fix these problems is one  of the most fiercely debated topics in science right now.

But a growing number of AI safety advocates are calling for slower and  more cautious development, or outright prohibiting the development  of stuff like superintelligence. They want to make sure we can keep the  technology in check before our lasagna’s cooked. Thanks to ControlAI for sponsoring this video.

ControlAI is a non-profit working  to keep humanity in control of AI by mobilizing everyone from concerned citizens to experts to raise  awareness of the risks of superintelligence and push for international action to address them. If you find these trends concerning  and you want to make a difference, you can go to controlai.comscishow, where they’ve created tools to  help you easily voice your concerns to your lawmakers in less than a minute. They also have a newsletter to keep  you informed about what's happening at the frontier of AI and  in the world of AI policy, alongside educational resources to help you learn more about  the issue and share with others. [ OUTRO ]