This SciShow video is supported by Linode!

Go to linode.com/scishow for a $100 60-day credit on a new Linode account. At first glance, math and art  might seem like total opposites.

Math is full of rigid rules, while  art is free-flowing and creative. Math can be churned out by a  computer, while at least most art still involves a human that can love  and suffer and experience the world. But as different as the two seem,  sometimes math can be the key to revealing secrets of art, and seeing deeper into artwork than we otherwise could. [INTRO] Now, when we talk about using math to decode art, we’re not talking about the kind of algebra or calculus you can scratch out on a chalkboard.

The kind of math we’re talking  about is mostly done by computers, which can process images  in ways that humans cannot. This came in handy when it  came to solving a mystery tied to the masterpiece known  as the Ghent Altarpiece. The altarpiece is a 15th-century oil painting by the Flemish brothers Hubert and Jan van Eyck.

It’s made up of more than a  dozen different paintings, with side panels on hinges so  it can be opened or closed, which lets people display different  paintings for different occasions. When the wings are open, it’s about as wide as a two-car garage door and almost twice as tall. And it is incredibly detailed.  In the painting on one panel, there’s an open book, just centimeters  tall, with some barely visible text.

Art historians were curious if it was a real text, but the words were completely illegible. You have to understand, this  painting has been through a lot. During wars and riots, it was  taken apart, looted, recovered, and reassembled over and over again.

So it’s not surprising that the text  wasn’t exactly looking its best. But a team of mathematicians thought  they might just be able to fix that. The main problem was there were  cracks throughout the painting.

This happens naturally thanks  to changes in humidity. As different parts of a painting  absorb and release moisture, they expand and contract. This made it hard to tell what was  text and what was just a crack.

But the team thought if they could  identify the cracks in the painting, maybe they could fill them in  and make the letters clear again. Fortunately, they did not have  to start from scratch here. There are lots of digital image  processing algorithms out there that can pick out little branchy  patterns that kind of look like cracks.

Like blood vessels in medical images, or rivers and roads in satellite images. One way to do this is by using  something called convolutional filters. Generally speaking, a convolutional  filter takes a plain image and applies some operation to  all the pixels in that image.

That process creates a modified  image based on the original. If you have ever used an Instagram filter before, you’ve probably used one of these. But in addition to adding  sparkle to your Instagram posts, convolutional filters can also be used to identify certain features in images.

It works like this: The program  starts with a small grid of values called a kernel. The kernel is going  to transform the image one pixel at a time, using the pixels around  the one it focuses on to do so. As it passes over an image, it  multiplies each pixel by the value that the kernel overlays on it, then adds  up the values you generate from it.

Then, that new value is divided by the sum of the values in our kernel, like this. That new number gets placed onto a  duplicated version of the old image, over and over again until the  whole thing has been analyzed. And changing the numbers that you put into that kernel will help you  identify specific parts of images.

For instance, if you want to detect edges, you can use a kernel that has  negative values on one side and positive values on the  other side, like this one. If all the pixels in one area have the same value, when you add up all the products,  the negatives and positives will cancel each other out, and  the final result will be zero. But if one side is darker or lighter,  the pixels will have different values, like this.

And the result will be  some positive or negative number. These results help a computer  figure out where an image changes from dark to light which  is what we see as an edge. And in the end, it outputs a map  of all the edges in the image.

Now that might not seem  especially useful if your goal is to identify cracks, not edges. But a crack is essentially just  two edges right next to each other. So you can use these edge-detecting  filters to pick out cracks.

And that is what the team did  with the Ghent Altarpiece. Now once the team had identified  them, they digitally painted them in, leaving behind only the text. Which was actually legible!

At  least, to the art historians. And not only that, they could  even tell what book it came from, a 14th-century religious text. So, thanks to a little help from math, we can analyze this  masterpiece in a whole new way.

But sometimes, it’s not just  the surface of a painting that artists and historians are interested in. Because a finished painting  doesn’t always tell a full story. Today, X-ray images reveal  that many famous paintings were painted on top of other artwork.

For instance, X-rays showed us  that Picasso’s famous painting “The Old Guitarist” has a figure  of a woman painted underneath it. This kind of thing fascinates art lovers, because these hidden  paintings can reveal what else was on the artist’s mind, or  what they wanted to cover up. But even though X-rays can reveal  that a hidden painting exists, it’s really hard to actually  recreate that painting.

Features from different images get mixed up, and it’s hard to tell which  lines belong to which painting, especially if a canvas has been  painted over several times. But once again, math can help. And some of the same researchers  who worked on decoding the text in the Ghent Altarpiece were able  to develop a sort of answer key for future research into  paintings under paintings.

To be clear, the altarpiece itself isn’t hiding any older artwork underneath the paint. But it’s actually the perfect  work for training a machine how to separate two images  because of how it’s built. Like we said before, it was  created to show different scenes depending on certain holidays and functions, which means that the panels  on its shutters are two-sided, there’s a painting on each side.

So if you take an X-ray  image of any of these panels, you will see both sides at once. It’s like a puzzle with an answer key: The X-ray image showing the  two layers on top of each other is the puzzle, and the regular photograph  of each side is the answer key. You can use this to train a computer to separate a superimposed image into two different ones.

To do this, the team of  researchers used what’s called a convolutional neural network,  a type of computer model that can be trained to recognize images. You basically give it a bunch of  examples until it can pick out patterns well enough to do the task on its own. These models use different convolutional filters to create maps of features  like edges, curves, or circles.

So, they basically summarize any  image as a collection of features and then they match this collection  of features to some known object. In this case, the researchers  trained a neural network to tease apart two images in one mixed image. And to get that to work, they  sort of worked backwards.

They started with two regular photos of each side. Then they had a neural network generate two separate X-ray images based on each one. And since these were based on photographs, there was no overlapping  image from the other side.

Next, it recombined those X-ray  images to create a mixed X-ray image and compared this to the real X-ray image showing the overlapping paintings. It was programmed to repeat  this process over and over until the two images were as similar as possible. In the end, the team had a neural  network that was extremely good at pulling out two images that  combined to make an overlapping image.

But cryptic messages and hidden images aren’t the only mysteries  swirling around old paintings. One of the biggest ones is just who made it? Like, remember how we said the  Ghent Altarpiece was painted by both Hubert and Jan Van Eyck?

Well, one question that has  never been entirely answered is which brother did what. We know Hubert started the project, and Jan finished it after  Hubert died partway through. But so far, no one has been  able to sort out exactly what each brother contributed.

And all over the art world,  there are questions like this. Not only are there plenty of unsigned or otherwise anonymous paintings out there, but there are plenty of  forgeries floating around, too. So if we can use math to help  figure out who created an artwork, there are a lot of potential applications.

Now, that’s not to say that we  humans are totally in the dark when it comes to finding  out who a mystery artist is. The same way you can tell  apart people’s handwriting by recognizing patterns, an art  expert can sometimes tell you who made a painting based on patterns in the brushwork. But mathematical tools can take this up a notch.

They can summarize all of those  intangible qualities that make up art, like brushstrokes and texture, into  a bunch of numbers and statistics. One way mathematicians extract  key features of a painting is by taking a hi-res image and  blurring it, first just a little bit, and then more and more. Each time they blur it, they  subtract the blurry image from the less blurry image right before it.

The difference between the two  images is all of the information that was lost when the image was blurred. You can imagine that in some spots, like, the sky, there won’t be a big difference between the blurred version and the original, because there’s not much detail there. But in other spots, say, a garden full of flowers, there will be a bigger difference  because in the blurred version you lose a lot of information about the details of the flowers and the fruits and the leaves.

So, each time you blur and subtract, you capture some key features  of the original image. As you do this with blurrier and blurrier images, you extract more and more levels of detail. And after many iterations, you can  take all of the lost detail from each level of blurring and add it together.

What you end up with is a  summary of the original image that just contains its key features. They have a name for this, it’s  called a wavelet decomposition. Once a painting has been summarized this way, mathematicians can use a number  of different statistical tools to see how it compares to another painting.

One approach is to use  what’s called a Markov model, which is a way to summarize  patterns mathematically. Essentially, this model breaks down a pattern into a grid of little pieces, called states. And then it looks at the probability that one state will be neighboring another.

It can do this for textural patterns, too. So, it’s a way to represent the patterns of Van Gogh’s brushstrokes with math. By comparing the Markov  models of different paintings, you can tell how similar the textures are, and hopefully that gives you an  idea if you’re looking at art by the same artist or two different ones.

And a group of researchers  put this method to the test, using 101 scanned images from two art museums. Most were by van Gogh, a few were  most definitely not by van Gogh, and 13 of them were up for debate. The team used wavelet decomposition  to analyze the paintings, and while their results weren’t perfect, they were pretty successful  at separating the van Goghs from the non-van Goghs, or van fauxs, if you will.

So far, this kind of approach  hasn’t been used widely, but in theory it could help solve  the mystery of which brother painted what in the Ghent Altarpiece. Now, none of this is to say that math  is taking the place of art experts. But it can be another tool in their toolbox.

So as much as math and art  might sometimes seem like fundamentally different  ways of exploring the world, putting the two together can give us new insights and new ways to answer age-old questions. Thanks for watching this SciShow  video, supported by Linode! Linode is a cloud computing company from Akamai that provides storage space,  databases, cloud services, and award-winning support to you or your company.

And when I say “company” I’m using  that term as strictly or broadly as your situation requires,  because Linode can help out COOs at any sized corporation, teachers  at grade school or universities, and self-employed content creators going it alone or with a large team behind them. Linode provides a variety of services that can benefit each of those diverse roles. And you can use Linode to  meet the extent of your needs without wasting your money on  services that someone else might want.

Then, if you end up transitioning  from the lone ranger to a larger support team, Linode  can scale with your growth and still provide dependable access to your streaming, grading, and analytics needs. To get started with Linode you can click the link in the description down below or go to linode.com/scishow for a $100 60-day credit on a new Linode account. [OUTRO]