[♪ INTRO ].

Data. The word is everywhere these days.

Every company is dying to tell you about its big data, data analytics, data privacy, data warehouse, data lake, data data data data. At the center of the data mania is data mining—the practice of sifting through all those piles of information for insights. Data mining recently made big news with the Cambridge Analytica scandal.

The political consultancy reportedly sucked up data about millions of Facebook users without their knowledge, then used it to profile and sway voters in the US, UK, and elsewhere. And similar techniques let companies like Amazon, Facebook, and Google work out what we want to see or buy—sometimes with shocking accuracy. It’s a little creepy.

It’s not just ads and politics, either. Data mining allows airlines to predict who’s going to miss a flight; it tells big-box stores who’s pregnant; it helps doctors spot fatal infections; and it’s even enabled cell phone companies to predict massacres in the Congo. The power of data mining and the hype surrounding it can make it sound like a magic wand—one that will either save your business or sink democracy.

Of course, data mining doesn’t really involve any unicorn hair or phoenix tail feathers. It’s just applied statistics, searching lots of data points for patterns that humans might not spot. Those patterns are based not on human intuition, but on whatever the data suggests, so sometimes they can seem incredibly subtle or even alien.

But there’s no more magic in data mining than there is in a weather forecast. In fact, data mining is a lot like meteorology. Meteorologists aim for two things: first, they want to describe patterns in the weather—to boil down its massive complexity into a few numbers and equations.

And second, they want to predict Tuesday’s weather. That’s the whole point. Similarly, Spotify’s data scientists might be interested in describing medieval rock fans, recognizing them as a group distinct from nerdcore or freak folk fans — yes, that's a real sub-genre.

Ultimately, though, what’s most important to companies like Spotify is predicting what each person wants to listen to. The key with data mining is that it achieves description and prediction not through careful study by experts, but by analyzing large amounts of data. In Spotify’s case, that might mean scanning for patterns in genre labels, acoustic attributes,.

Internet reviews, and anything else about each track, plus the age, location, friend group, and other scraps of information about each user. Data mining is more about spotting patterns than explaining them. Of course, the words “pattern” and “data” can mean just about anything.

There are no clear definitions for data mining, data science, or big data, and they’re sometimes used interchangeably with each other or with machine learning. That’s why it’s so easy to slap these buzzwords onto any project for instant venture capital karma. That being said, a few types of techniques consistently earn the “data mining” label.

The most broadly applicable one is classification, where you try to categorize things. For example, Target famously realized as early as 2002 that they could guess who was pregnant and send them baby-related coupons. That’s a textbook classification problem: Target needed to assign each customer to one of two categories: either “probably pregnant” or “probably not pregnant.” Classification typically works in several stages.

First, each example, or instance, has to be broken down into a collection of numerical attributes, or features. For a store like Target, an instance might be your mom 7 months before you were born. The features would be things like “How many bottles of unscented lotion did she buy in the last three months?

How about in the quarter before that?” And likewise for zinc supplements, Asian pears, and every other product in the inventory. The store would also need labels for some chunk of the data—the ground truth about whether those customers were pregnant. Target got those labels from baby registries and due dates customers had shared.

Once the data’s all lined up, it’s time for training. That’s where the system tries to tease out patterns from all the labeled examples. Learning to classify is such a basic, common need that dozens of algorithms, the mathematical procedures computer programs follow, have been devised for it.

Which algorithm works best depends on all kinds of factors, like how many categories there are and how different features are connected to each other. But many classification algorithms are similar in that they treat each feature as a drop of evidence for one category or the other. The features get weights indicating how strongly they boost or weaken someone’s chances of falling into the “yes” category — that they are pregnant, for example.

Those weights are what the system learns during training. Basically, it’s figuring out how informative each attribute is. Finally, to classify instances the system hasn’t seen before, it puts together all the weighted contributions, and maybe stuffs the resulting number through a bit of mathematical machinery to slide it up or down.

If the result is negative, that instance goes in the “no” bucket. If it’s positive—load up the crib coupons! Each individual feature doesn’t tell you much.

In fact, many turn out to be irrelevant. But together they can be really powerful. Target’s approach worked so well that when one customer complained that his teenage daughter was getting coupons for baby clothes, he ended up apologizing to Target.

Turned out the company knew about his daughter’s pregnancy before he did! Classification is useful any time you want to tell one group of things from another. Insurance companies use it to guess which elderly patients will die soon so that they can start end-of-life counseling.

Doctors use it to check whether premature babies are developing dangerous infections, since the classifier can put together subtle disease indicators before humans would notice any signs. I could spend all day listing uses for classification, but it’s far from the only type of data mining. One close cousin is known as regression.

And no, that doesn’t mean deciding you like Limp Bizkit again. In regression, instead of predicting a category, the goal is to predict a number. Take Target again.

They wanted to know not just whether each customer was pregnant, but when to send each coupon. So they managed to estimate due dates, too. That’s a regression question—how many weeks until the customer gives birth.

Regression often depends on dozens or even thousands of variables—the features that describe each example. It finds an equation or curve to fit the data points, telling you how high you’d expect the curve to be given any arbitrary input. Or in this case, how far away you’d expect the customer’s due date to be.

Like in classification, many regression techniques give each feature a weight, then combine the positive and negative contributions from the weighted features to get an estimate. And, also like classification, regression is used everywhere. One of the better-known examples is Google Flu Trends.

In 2008, it began publishing real-time estimates of how many people had the flu based on searches for words like “fever” and “cough.” Regression is also part of predictive policing software — programs that look at historical data to guess how likely a crime is to occur in each area. The third major data mining technique is clustering. As the name suggests, the goal here is to group data points in a way that helps with the analysis.

In the marketing world, clustering emerged in the 1980s—well before data mining—with the work of a market researcher named Howard Moskowitz. He struck gold when he realized there wasn’t one best pasta sauce. Consumers showed three distinct types of preferences—and the previously unrecognized group that craved extra-chunky turned out to be worth millions.

Clustering is often used to analyze market segmentation like this, but to understand how the techniques work, let’s take a different example: eBay. On eBay, you can get millions of products, from antiques to zip ties. Even within a single category, like electronics, the selection is overwhelming.

So eBay organizes things into subcategories. But it’s a pain for humans to trawl through all the electronics, identify subcategories, and assign every product to a subcategory. Instead, the company can use clustering to automatically group the products.

Again, each product first has to be broken down into numerical features, like how many times “printer” appears in the description, or who manufactured it. The simplest clustering method is to guess how many distinct subcategories there should be. Then you randomly lump items together into that many clusters, and keep shifting items between groups to make each cluster tighter.

In the end, similar products end up settling into clusters together. But we don’t have to stop there! The blue and silver versions of the same camera don’t really deserve separate listings; they’re variants of the same product.

So in addition to subcategories, it would be nice to find listings to merge. Sites like eBay can do both simultaneously with a technique called hierarchical clustering. Rather than a single set of categories, hierarchical clustering produces a sort of taxonomic tree.

For example, it might find that cameras are much more like each other than like TVs. But within cameras, the DSLRs and point-and-shoots each get their own subgroup, albeit slightly less distinct ones. And within those are many different models, each with a few variants. (on image) Companies like Cambridge Analytica use these techniques to look for groups of voters who will respond to the same kinds of advertising, and Spotify can use them to guess who will like similar music.

The fourth staple of data mining is anomaly detection. It’s basically a special case of classification—identifying instances that are unusual or worrisome. The IRS uses anomaly detection to spot likely tax evaders, and credit card companies use it to flag transactions that don’t fit your usual buying habits.

It also helps industries with heavy-duty equipment. For instance, power companies and airlines can see when a generator or jet engine is starting to vibrate differently than usual. Some anomalies can be detected just by looking for deviations from averages.

Fancier techniques include looking for instances that don’t match any cluster, or comparing instances with the closest other examples to see if their feature values are far off. Finally, association learning reveals which birds are of a feather. The idea is to look through, say, millions of grocery store purchases to see what gets bought together and when.

A classic example is the Osco drug store chain, which once found that many customers bought beer and diapers together on Friday evenings. Contrary to popular legend, the store never acted on this profound insight, but stores regularly use observations like this to optimize their floor layouts and inventory. For instance, Walmart discovered that shoppers buy lots of Pop Tarts immediately before hurricanes, so it started to stock up.

Association learning has broader applications, too. CellTel, an African cell phone company, realized it could spot impending massacres in the Congo when everyone nearby started buying prepaid phone cards. The five strategies we’ve covered—classification, regression, clustering, anomaly detection, and association learning—form the backbone of data mining.

What makes them so powerful is that they offer standard mathematical tools you can use for everything from curating Facebook feeds to optimizing store layouts. But that ease of use can also lead people astray. Data mining is just one step in the process of extracting knowledge from data—and it’s all too easy to whip out an algorithm without carefully selecting the data, massaging it into the right form, and considering how to interpret the results.

Remember Google Flu Trends? It shut down after a few years, but not because the algorithm was broken. Search auto completion had totally thrown off the data, and engineers had given it too much leeway to interpret seasonal words like “snow” as evidence of the flu.

Then there are the queasy social implications of sharing data in the first place, and of letting companies form such an intimate understanding of our behavior. In other words … the creep factor. So as powerful as it is, the math of data mining is just the beginning.

Sometimes the hardest part is all the messy human stuff. Thanks for watching this episode of SciShow! If you’re interested in the ways companies can use psychology to learn even more about you from your data, you can check out our video about that over on the SciShow Psych channel. [ ♪OUTRO ].