You know the story: a menacing  android hunts down a group of humans in a post-apocalyptic wasteland to  … deny them healthcare coverage.

And, alright, maybe that’s not the  premise for a sci-fi blockbuster. But as far as worrying about  artificial intelligence goes, real life dangers like that are probably  more concerning than what’s in the movies.

While Hollywood often depicts the threat of AI as machines that have gained consciousness … Just what do you think you're doing, Dave? … that day may or may not ever come. But AI-powered machines already make  an alarming number of decisions for us, from driving our cars to piloting military drones and selecting targets on the battlefield. And according to the people who study it, AI doesn’t need to be self-aware to be dangerous. [♪ INTRO] If you are worried about the machine  uprising, we’ve got good news!

While AI systems are becoming  more powerful every day, there’s no evidence that  they’re becoming conscious. But I also have some bad news. There are a lot of other ways AI can go  horribly wrong, and it kind of already has.

So let’s start with what  happens when AI is fed bad data. In 2019, the healthcare company Optum  was using an AI-powered algorithm they built to predict which patients  would benefit from extra medical care. You know, the kind of thing  that might save some lives.

That algorithm selected  people with higher predicted medical expenses for extra healthcare assistance. And it was developed using  past insurance claims data, previous diagnoses and procedures, and  patient demographics like age and sex. So it sounds like a good idea to make  informed decisions based on previous data.

But this algorithm wound up  discriminating against Black patients, leading them to miss out  on healthcare they needed. A 2019 study published in the journal  Science found that Optum’s algorithm was massively under-predicting  the health risks for Black patients compared to white patients. The algorithm gave a Black person  and a white person the same predicted risk score, even though  the Black person was more likely to have worse actual health and be in greater need.

Now, before you get out your pitchforks, technically there was nothing  racist about the algorithm itself. The weird thing about this result is that race was not included in the demographics that  Optum’s algorithm ran through at all. It turned out the problem was the  data they fed their algorithm.

That data was related to how much money had previously been spent on a person’s care. Not a pure measure of their physical health. And, historically, inequality and  discrimination have led to an imbalance in how money has been spent on different groups.

Which means if you try and  predict healthcare needs based on past healthcare expenses, you’ll naturally wind up discriminating against Black patients. So the data going into their algorithm  wasn’t ever going to address their goal. This kind of problem goes beyond healthcare.

Any AI can fail if its data aren’t  representative, fair, accurate, and reliable. In cases like this, AI tools aren't entirely  to blame for the mayhem that ensues. But deciding whether you can blame  them can be a much bigger problem.

Under all the slick interfaces, AI  tools are really a big ol’ tangle of mathematical functions and parameters  that scientists call models, because they “model” how an outcome  or decision depends on the data. Even when those models seem to work, understanding why they work  is a whole other story. In the good ol’ days, statistical  models were often based on straightforward assumptions about how  different data depended on each other.

For instance, you can fit a straight  line through the middle of a bunch of data points to predict something  like “A student’s expected grade” based on their “attendance”. This kind of “linear” model  is still incredibly useful. The parameters that define the  model, like the slope of the line, are “learned” from the data.

For stats geeks, this is where the good stuff is. By looking at the parameters  like the slope of that line, you can say in plain English “this  is exactly how much we expect a student’s grade to increase  for every extra day they attend”. So in simple cases, the parameters  directly tell us about what assumptions the model is making about the relationship  between “attendance” and “grades,” and importantly, whether those  assumptions are reasonable.

But modern AI tools are developed  with way more complicated models. Maybe you’ve heard of “deep learning”,  the kind of fancy algorithms that lurk behind pretty much all modern AI. The “deep” refers to the huge number of  layers of mathematical functions whose outputs become the input for the next layer.

It’s line graphs all the way down! And in an algorithmic lasagna  with billions of parameters, there’s no air-tight way of saying  “The algorithm thinks this picture is a cat because of its cute  fuzzy face, pointy ears and tail”. The parameters are basically impossible  to understand in the same way.

Researchers often call this the black  box problem since, unlike a linear model, we can’t easily peek inside  and see what’s really going on. That’s big trouble because one of the  main reasons for creating AI is to automate complex tasks where we can’t simply program every possible response,  like driving a car for example. And if we don’t know why an AI is making  the prediction it does in those situations, it becomes much harder to hold it  accountable, rely on it, or fix it.

And that’s what the ride-sharing company, Uber, found out with their self-driving cars. They modified them with all kinds of  sophisticated sensors from cameras, to radar, to its laser-based cousin “LIDAR”. The idea was to detect every object  within a certain distance of the car, like other cars, pedestrians, and cyclists.

That way it could respond appropriately. But there weren't any concrete “rules”  the algorithm had for how it identified certain objects, just very complicated  pattern matching from the data. And when we don’t understand  what those patterns are or how they’re used inside the black  box, things can go wrong, fast.

In March 2018, while a self-driving Uber  car was moving around Arizona at night, it struck and killed Elaine Herzberg, a  pedestrian who had been crossing the road. After the event, a report by the  National Transport Safety Board found that the sensors detected Elaine in  enough time to prevent collision, but the algorithm kept changing its  prediction of what she was from “vehicle” to “bicycle” to “other object”,  making it harder for the AI to determine that the car needed to stop. So how badly did the AI mess up?

Well, humans driving on a  dark night might also struggle with identifying whether that  blob in the distance is a person. But at least a human driver  could explain their decision. Even though we can peer into every  element of an AI model down to the last parameter, it’s much harder  to understand why it failed to stop.

All of this points to the big  danger of the black box problem. When an AI behaves in odd and unpredictable ways, it’s not always possible to  understand the logic behind it. Without that sort of understanding,  it’s much harder to know when we can hand off duties to an AI safely.

Uber and other companies claim to have  improved their modeling since 2018. But a lot of the models AI  scientists evaluate still make these sorts of mistakes today. And we’re not much closer to explaining how AI models like these work than we were back then.

So we’re in a kind of limbo between making  the most sophisticated algorithms and keeping them comprehensible for the  humans who need to hold them accountable. And accountability is super important  for AI tools because they’re also prone to acting in ways that can seem completely bizarre if we don’t  know what they’re thinking. In 2017, Facebook was experimenting  with chatbots trained to negotiate with each other over virtual  items worth different points, saying things like “I want the books  and the hats and you get the ball”.

And for the most part, they did pretty well. They came up with strategies much like  humans, such as driving a hard bargain or even lying about what they really wanted. But things got weird.

Instead of using grammatically correct English, the language the models were trained in,  they occasionally started saying stuff like “I can can I I everything else.” and  “Balls have zero to me to me to me to me…”. And that sounds like the  coding bug of the century. But the kicker is, after  exchanging nonsense like this, the two AIs would sometimes  complete a successful deal!

What seems to have happened is  that despite setting a clear goal “Negotiate to get what you want  using English words”, sometimes, the approach taken by the AI looked  nothing like what we might expect… even though it did technically  achieve the outcome. And because of our arch nemesis, the  black box problem, we can’t exactly look in and work out what the made-up  language of the AI really means. Researchers call this problem goal misalignment.

You can give an AI an objective,  but with enough flexibility, the way it achieves that  objective might seem completely alien and sometimes go catastrophically wrong. For example, the securities and  exchange commission found that a 2010 stock market crash that caused a  total temporary loss of $1 trillion in market value was caused by an algorithm  tasked with executing a bunch of sales. That was in large part because  the algorithm focused only on how much of the asset it sold  without paying attention to its price or the timescales it was trying to sell the  stock for, creating chaos in the markets.

As AI gets more sophisticated and takes  over more financial trading from humans, the risk of them executing wild strategies in pursuit of their goals risks  wild tailspins in the market. And finance isn’t the only  industry affected by this problem. In 2011, a company you might  have heard of called YouTube sent out an internal memo  stating the company’s main goal was to maximize watch time.

That’s the amount of time people stick  around for when they’re watching videos, based on YouTube’s recommendation algorithm. It probably worked, since a few  years later they achieved the goal of having 1 billion hours of watch time  on the platform in a single day. But not everyone would call it a success.

A former AI engineer at Google came  out in 2018 to say that one side effect was the algorithm pushing more conspiracy  theory videos on people. Yikes! I guess nothing keeps you watching YouTube like getting convinced that  the rest of the media is lying.

Now of that is pretty terrifying. But it can be explained by  Goodhart’s Law, which says that “if you make something you’re  measuring an explicit target, it stops being a good measure.” Giving an algorithm a simple  goal like “increase watch time”, and optimizing solely for that without  taking other factors into account can conflict with important goals  like “keep the platform safe” and “don’t stoke a political fire”. Wherever AI is used in complex environments, goal misalignment and Goodhart’s  Law are always lurking.

Just telling an AI tool what  outcome you want to achieve doesn’t mean it’ll go about it in  the way that you think, or even want. If we aren’t careful about exactly  what problem we’re trying to solve, even the best data and algorithms  could destabilize our politics, the economy, and maybe even our access to food as turbocharged AI enters the world of farming. The good news is that, even if  we can’t look into the black box, these algorithms are changeable.

A 2024 study in the Proceedings  of the National Academy of Science showed that YouTube’s algorithm  can be retrained to provide better recommendations over a month of  thoughtful choices in what you watch. But that’s the thing! It  requires being thoughtful.

About how we engage with AI  and about how we evaluate it. And speaking of being  thoughtful… It’s time to consider the scariest application  of

AI: autonomous weapons. I’m talking drones mounted with guns that can identify and eliminate  targets all on their own. And by “targets,” I mean buildings and people. As one 2023 study points out, pretty  much all of the major world powers are developing these weapons.

And as autonomous drones become a warfare reality, the idea of having a swarm of drones that can coordinate an attack seems  to be on everyone’s minds. On top of the issues with data,  interpretability, and alignment, something lots of researchers worry  about is the lack of information on how well autonomous weapons systems work. Military ops are famously covert.

As a 2021 report from the United Nations  Institute for Disarmament Research points out, there isn’t a whole  lot of public data, analysis, or research on AI weapon capabilities. And unlike “YouTube views” or “driving a car,” it’s not exactly possible to go out  and collect a bunch of real world data on how easily a swarm of AI drones can distinguish between a combatant or a civilian  in live combat situations. The data used to test the system might not reflect the situations it has to face in reality.

Like, the environment could change, humans  could exploit and fool unknown flaws in the model, or an unexpected earthquake could throw off the entire navigational system. As the report puts it “autonomous system  failures are infinitely possible – in a certain sense, inevitable”. Greaaaaaaaaat… Real world evaluation is super important  when we can’t look into the mind of an AI tool and make sure it really  understands what it’s dealing with.

And if the AI isn’t “aligned”  in the way we want it to be, it’s harder to trust its behavior  when the stakes are high. This is bad for both the army  and individual soldiers alike. A 2018 study in the Journal of Military  Ethics points out that these flaws stop soldiers from relying on  these weapons on the battlefield, unlike the squad mates they’ve built trust with.

The concern is so great that a treaty  to ban lethal autonomous weapons without human oversight is being  considered by the United Nations, with more than 100 countries in support. Now I know this video has  covered some things that could keep a person up at night. But we can’t ban all AI, any more than we can ban a straight line through data on a graph.

And few people want to, because it’s  revolutionizing parts of healthcare, drug discovery, environmental monitoring,  public services, and more for the better. What many AI researchers are  calling for is more oversight. In addition to supporting treaties and regulation, they’re pushing for corporations  to prove their systems are safe before releasing them.

You know, the same thing we require  of companies in other industries, like autopilot software in aviation. In other words, they’re saying we all need to be conscious of the risks AI  poses, whether it is or not. Thanks to the Future of Life Institute  for supporting this SciShow video.

FLI is a global nonprofit that aims to steer advanced technologies towards benefiting humanity. One of those is, unsurprisingly,  artificial intelligence. If you would like to learn more,  please visit futureoflife.org/safety. [♪ OUTRO]