Face Recognition using OpenCV and Python: A Beginner's Guide.

Face Recognition using OpenCV and Python: A Beginner’s Guide

…Finding out who’s stealing my followers using Face Recognition and OpenCV…

 

Did you hear that?!

I think it’s him again… Damn, I don’t know what’s what right now!

I’m not only losing my friends… Wait, scratch that, I don’t have any friends… I’m losing my followers!

Noooo!

I'm not overreacting… My followers are everything to me!

I didn’t think it would be necessary, but when necessity stops at your door, you have to answer it. I’m bringing out the big guns. That’s right; I’m going all-out Face Recognition on this creep.

I know for a fact that there’s something out there. My Snapchat never lies.

Well, except for all the filters: dog-ears, sparkle, rainbows, etc. but let’s call that beauty enhancement.

If you followed my last post, Snapchat has been finding another person whenever I take a selfie. If you didn’t follow my previous post, then you’re dead to me!

JK! Click here to see my last post on Face Detection and catch up.

Ok, back to business!

I’ve decided to attack this creep with Facial Recognition because I am not afraid of no ghost, but I need to know who this is!

So, according to my friend @superdazzlepimpnerd, Facial Recognition will do the job for me.

But first, I have to understand what it takes to recognize a face; how the technology works and how I can use it to find out who this ghost is!

In that sense, this second part of my story will unfold like this:

  1. I'll explain the nerdy theory of OpenCV's three built-in face recognizers.
  2. Then, I'll show you the coding process I followed.
  3. Next, I'll compare all three face recognizers, with their pros and cons.
  4. Finally, I'll bust this creep once and for all!

By the end of this tutorial, hopefully, I will be able to solve this mystery using my Face Recognizer, and you will learn how to build yours and use it in whatever the heck you want. Seriously, I don't care.

Just before we start:

Can I get a little peace up in here?

I’ve been listening to random 50’s music up in the morning. Late at night. In the shower…

Is that “Love me tender” again? Am I listening to “Love me tender” for the 1,543rd time right now?!

That is just way too many times.

1. Background on Face Recognition

When you look at an apple, your mind immediately tells you: that is an apple. This process is recognition in the simplest of terms. So, what’s facial recognition? The same, but for faces, obviously.

But, the real question is:

How can a computer recognize a face?

Take a real life example:

When you meet someone for the first time, you don’t know who that person is at once, right? While he's talking to you or shaking your hand, you’re looking at his face: eyes, nose, mouth, skin tone… This process is your mind gathering data and training for face recognition.

Next, that person tells you that his name is Kirill (yes, our All-Star Data Science Mentor). So, your brain has already gotten the face data, and now it has learned that this data belongs to Kirill.

The next time you see Kirill or see a picture of his face, your mind will follow this exact process:

  1. Face Detection: Look at the picture and find a face in it.
  2. Data Gathering: Extract unique characteristics of Kirill’s face that it can use to differentiate him from another person, like eyes, mouth, nose, etc.
  3. Data Comparison: Despite variations in light or expression, it will compare those unique features to all the features of all the people you know.
  4. Face Recognition: It will determine “Hey, that’s my boy Kirill!”
face recognition process

Our mind's Face Recognition Process

 

Then, the more you meet Kirill, the more data you will collect about him, and the quicker your mind will be able to recognize him.

Or, at least you should. Whether or not you are good with names is another story.

Here is when it gets better:

Our human brains are wired to do all these things automatically. In fact, we are very good at detecting faces almost everywhere:

face detection in inanimate objects

Meet my (Loser) crew!

Computers aren’t able, yet, to do this automatically, so we need to teach them how to do it step-by-step.

But, you already knew that which is why you’re reading this article (duh).

However, you probably assumed that it’s incredibly difficult to code your computer to recognize faces, right? Well, keep reading my friend cause I am here to end with this creep and your superstitions as well.

 

2. Theory of OpenCV Face Recognizers

Thanks to OpenCV, coding facial recognition is now easier than ever. There are three easy steps to computer coding facial recognition, which are similar to the steps that our brains use for recognizing faces. These steps are:

  1. Data Gathering: Gather face data (face images in this case) of the persons you want to identify.
  2. Train the Recognizer: Feed that face data and respective names of each face to the recognizer so that it can learn.
  3. Recognition: Feed new faces of that people and see if the face recognizer you just trained recognizes them.

It's that simple!

And this is how our Face Recognizer will look once we finish coding it:

face recognition using opencv output

Hello, it’s me.

OpenCV has three built-in face recognizers and thanks to its clean coding, you can use any of them just by changing a single line of code. Here are the names of those face recognizers and their OpenCV calls:

  • EigenFaces – cv2.face.createEigenFaceRecognizer()
  • FisherFaces – cv2.face.createFisherFaceRecognizer()
  • Local Binary Patterns Histograms (LBPH) – cv2.face.createLBPHFaceRecognizer()

You might be wondering:

“Which Face Recognizer should I use and when?”

Here is a summary of each one that will answer that question. 

Let’s rock!

 

2.1 EigenFaces Face Recognizer

This algorithm considers the fact that not all parts of a face are equally important or useful for face recognition. Indeed, when you look at someone, you recognize that person by his distinct features, like the eyes, nose, cheeks or forehead; and how they vary respect to each other.

In that sense, you are focusing on the areas of maximum change. For example, from the eyes to the nose there is a significant change, and same applies from the nose to the mouth. When you look at multiple faces, you compare them by looking at these areas, because by catching the maximum variation among faces, they help you differentiate one face from the other.

In this way, is how EigenFaces recognizer works. It looks at all the training images of all the people as a whole and tries to extract the components which are relevant and useful and discards the rest. These important features are called principal components.

Note: We will use the terms: principal components, variance, areas of high change and useful features indistinctly as they all mean the same.

Below is an image showing the variance extracted from a list of faces.

OpenCV EigenFaces Recognizer Principal Components for Face Recognition

EigenFaces Face Recognizer Principal Components. Source: docs.opencv.org

You can see that the useful features represent faces which receive the name of Eigen Faces. I mean, how else do you think the algorithm got its name?

So, EigenFaces recognizer trains itself by extracting principal components, but it also keeps a record of which ones belong to which person. Thus, whenever you introduce a new image to the algorithm, it repeats the same process as follows:

  1. Extract the principal components from the new picture.
  2. Compare those features with the list of elements stored during training.
  3. Find the ones with the best match.
  4. Return the ‘person’ label associated with that best match component.

In simple words, it’s a game of matching.

However, one thing to note in above image is that EigenFaces algorithm also considers illumination as an important feature. In consequence, lights and shadows are picked up by EigenFaces, which classifies them as representing a ‘face.'

Face recognition picks up on human things, dominated by shapes and shadows: two eyes, a nose, a mouth.

So, maybe that shadow was an Eigen Face?!

 

2.2 FisherFaces Face Recognizer

This algorithm is an improved version of the last one. As we just saw, EigenFaces looks at all the training faces of all the people at once and finds principal components from all of them combined. By doing that, it doesn't focus on the features that discriminate one individual from another. Instead, it concentrates on the ones that represent all the faces of all the people in the training data, as a whole.

But here’s the kicker:

Consider the lighting changes in following images:

Illumination Variations

Since EigenFaces also finds illumination as a useful component, it will find this variation very relevant for face recognition and may discard the features of the other people's faces, considering them less useful. In the end, the variance that EigenFaces has extracted represents just one individual's facial features.

How to fix this issue?

We can do it by tunning EigenFaces so that it extracts useful features from the faces of each person separately instead of extracting them from all the faces combined. In this way, even if one person has high illumination changes, it will not affect the other people's features extraction process.

Precisely, FisherFaces face recognizer algorithm extracts principal components that differentiate one person from the others. In that sense, an individual's components do not dominate (become more useful) over the others.

Below is an image of principal components using FisherFaces algorithm.

OpenCV FisherFaces Recognizer Principal Components for Face Recognition

FisherFaces Face Recognizer Principal Components. Source: docs.opencv.org

You can see that the features represent faces which receive the name of Fisher Faces. Are you noticing a theme with the names of the algorithms?

One thing to note here is that FisherFaces only prevents features of one person from becoming dominant, but it still considers illumination changes as a useful feature. We know that light variation is not a useful feature to extract as it is not part of the actual face.

Then, how to get rid of this problem?

Here is where our next face recognizer comes in.

 

2.3 Local Binary Patterns Histograms (LBPH) Face Recognizer

I wrote a detailed explanation of Local Binary Patterns Histograms in my previous article on face detection, which I’m sure you’ve read by now.

So, here I will just give a brief overview of how it works.

We know that Eigenfaces and Fisherfaces are both affected by light and, in real life, we can't guarantee perfect light conditions. LBPH face recognizer is an improvement to overcome this drawback.

The idea with LBPH is not to look at the image as a whole, but instead, try to find its local structure by comparing each pixel to the neighboring pixels. 

The LBPH Face Recognizer Process

Take a 3×3 window and move it across one image. At each move (each local part of the picture), compare the pixel at the center, with its surrounding pixels. Denote the neighbors with intensity value less than or equal to the center pixel by 1 and the rest by 0.

After you read these 0/1 values under the 3×3 window in a clockwise order, you will have a binary pattern like 11100011 that is local to a particular area of the picture. When you finish doing this on the whole image, you will have a list of local binary patterns.

OpenCV LBPH Recognizer for Face Recognition

LBP conversion to binary. Source: López & Ruiz; Local Binary Patterns applied to Face Detection and Recognition.

Now, after you get a list of local binary patterns, you convert each one into a decimal number using binary to decimal conversion (as shown in above image) and then you make a histogram of all of those decimal values. A sample histogram looks like this:

LBPH Histogram Sample

Histogram Sample.

In the end, you will have one histogram for each face in the training data set. That means that if there were 100 images in the training data set then LBPH will extract 100 histograms after training and store them for later recognition. Remember, the algorithm also keeps track of which histogram belongs to which person.

Later during recognition, the process is as follows:

  1. Feed a new image to the recognizer for face recognition.
  2. The recognizer generates a histogram for that new picture.
  3. It then compares that histogram with the histograms it already has.
  4. Finally, it finds the best match and returns the person label associated with that best match.

Below is a group of faces and their respective local binary patterns images. You can see that the LBP faces are not affected by changes in light conditions:

OpenCV LBPH Recognizer Principal Components for Face Recognition

LBPH Face Recognizer Principal Components. Source: docs.opencv.org

 

The theory part is over, and now it is time to unmask this moron. 

Brace yourself. Here comes the coding!

Coding

This is how I code. In the zone. Much speed.

 

3. Coding Face Recognition Using Python and OpenCV

We are going to divide the Face Recognition process in this tutorial into three steps:

  1. Prepare Training Data: Read training images for each person/subject along with their labels, detect faces from each image and assign each detected face an integer label of the person it belongs.
  2. Train Face Recognizer: Train OpenCV's LBPH recognizer by feeding it the data we prepared in step 1.
  3. Prediction: Introduce some test images to face recognizer and see if it predicts them correctly.

To detect faces, I will use the code from my previous article on face detection

Before we start the actual coding, we need to install the Code Dependencies and import the Required Modules:

Code Dependencies

Install the following dependencies:
  1. OpenCV 3.2.0 
  2. Python v3.5 
  3. NumPy that makes computing in Python easy. It contains a powerful implementation of N-dimensional arrays which we will use for feeding data as input to OpenCV functions.

Required Modules

Import the following modules:
  • cv2: This is the OpenCV module for Python used for face detection and face recognition.
  • os: We will use this Python module to read our training directories and file names.
  • numpy: This module converts Python lists to numpy arrays as OpenCV face recognizer needs them for the face recognition process.

In [1]

3.1 Prepare training data.

The premise here is simple:

The more images used in training, the better.

Being thorough with this principle is important because it is the only way for training a face recognizer so it can learn the different ‘faces’ of the same person; for example: with glasses, without glasses, laughing, sad, happy, crying, with a beard, without a beard, etc.

So, our training data consists of total two people with 12 images of each one. All training data is inside the folder:training-data.

This folder contains one subfolder for every individual, named with the format: sLabel (e.g. s1, s2) where the label is the integer assigned to that person. For example, the subfolder called s1 means that it contains images for person 1.

With that in mind, the directory structure tree for training data is as follows:

training data folder structure

On the other hand, The foldertest-data contains images that we will use to test our face recognition program after we have trained it successfully.

Considering that the OpenCV face recognizer only accepts labels as integers, we need to define a mapping between integer tags and the person’s actual name.

So, below I am defining the mapping of a person’s integer labels and their respective names.

I have a sneaking suspicion that my follower thief is none other than Elvis Presley. Why else do I keep listening to 1950s rock and roll?

Note: As we have not assigned label 0 to anyone, the mapping for tag 0 is empty:

In [2]

Data Preparation for Face Recognition.

Perhaps you are thinking:

Why are are we talking about preparing data?

Well, to know which face belongs to which person, OpenCV face recognizer accepts information in a particular format. In fact, it receives two vectors:

  • One is the faces of all the people.
  • The second is the integer labels for each face.

For example, if we had two individuals and two images for each one:

data preparation 1

Then, the data preparation step will produce following face and label vectors:

In detail, we can further divide this step into the following sub-steps:

  1. Read all the sub folders names provided in the folder training-data. In this tutorial; we have folder names:s1, s2.
  2. Extract label number. Remember that all the sub folders containing images of a person following the format:sLabel where Label is an integer representing each person. So for example, folder name: s1 means that the person has label 1, s2 means the person's label is 2, and so on. We will assign the integer extracted in this step to every face detected in the next one.
  3. Read all the images of the person, and apply face detection to each one.
  4. Add each face to face vectors with the corresponding person label (extracted in above step)

Let's code this part!

As I said before, we are going to use the code of my last article on face detection.

“Wait, what article? What is this dude talking about?!”

Seriously?

Unbelievable. Here is the link again.

In [3]

As you can see, we are using OpenCV's LBP face detector. To break it down:

  • In line 4, we converted the image to grayscale because OpenCV mostly operates in gray scale.
  • Then, in line 8, we loaded LBP face detector using classcv2.CascadeClassifier.
  • After that, in line 12,  we used classcv2.CascadeClassifier‘ detectMultiScale method to detect all the faces in the image.
  • Next, in line 20, from exposed faces, we only picked the first one, because in a person's portrait it is supposed to be only one face (under the assumption that there will be only one prominent face).
  • As faces returned by the method detectMultiScaleare rectangles (x, y, width, height) and not actual faces images, we have to extract face area from the main image. So in line 23, we extracted face area from the gray picture and return both the face image area and face rectangle.

Now, you have a face detector. Also, you know the four steps to data preparation. I think we are ready to code it. 

Let's go!

In [4]

We have defined a function that takes the path where it stores training subjects' folders as a parameter. This function follows the same data preparation sub steps that we reviewed previously:

(step-1) In line 8,  we used the methodos.listdir to read names of all folders stored on the path, so they start functioning as a parameter.

In line 10-13 we defined labels and faces vectors.

(step-2) After that, we went through all subjects' folder names and from each one we extracted, in line 27, the label information. As folder names follow the sLabelnaming convention, just by removing the letters from folder name we will get the tag assigned to that subject.

(step-3) In line 34, we read all the current person's images, and in lines 39-66 we went through traverse those images one by one.

In lines 53-54, we used OpenCV's imshow(window_title, image) along with OpenCV's waitKey(interval) methods to display the current picture.

The methodwaitKey(interval) pauses the code flow for the given interval (milliseconds). So, we are using a 100ms interval so that we can view the image window for that time.

In line 57, we implemented face detection on the current picture.

(step-4) In lines 62-66, we added the detected face and label to their respective vectors.

But a function can't do anything unless we call it on some data that it has to prepare, right?

Don't worry; I have got data for two faces. I am sure you will recognize at least one of them!

I’m pretty sure that The King is the ghost, so I’ve put his image into the face recognition step. I recognize his face, and I am sure you will recognize him, too!

 

test images for face recognition

Look at these handsome faces

Let's use these images of two handsome devils to prepare data for training of our Face Recognizer. Here is a simple code to do that:

In [5]

It's time to train our face recognizer so that, once trained, it can recognize new faces of the people it's been trained on. Ready? Ok then let's get training.

3.2 Train Face Recognizer.

As we mentioned earlier, OpenCV comes equipped with three face recognizers.

  1. EigenFaces: cv2.face.createEigenFaceRecognizer()
  2. FisherFaces: cv2.face.createFisherFaceRecognizer()
  3. Local Binary Patterns Histogram (LBPH): cv2.face.LBPHFisherFaceRecognizer()

We are going to use now LBPH recognizer this time and see if my theory about the ghost of Elvis stealing my followers is right. It doesn't matter which of the OpenCV's face recognition programs you use because the code will remain the same. You just have to change one line, which is the face recognizer initialization line given below.

In [6]

Now that we have initialized our face recognizer and we also have prepared our training data, it's time to train. We will do that by calling the methodtrain(faces-vector, labels-vector) of face recognizer.

In [7]

Did you notice that instead of passing vectorlabels directly to face recognizer, we are first converting it to numpy array? The reason is that OpenCV expects labels vector to be a numpyarray.

This is my favourite part now, when we find out if the thief is the King of Rock and Roll or not.
The anticipation is killing me…
Here we go!
It’s time for the…

3.3 Prediction

This is where we get to see if our algorithm is recognizing our individual faces or not.

We’re going to take one test image of each person, use face detection and then pass those faces to our trained face recognizer. Then we find out if our face recognition is successful.

Below are some utility functions that we will use for drawing bounding box (rectangle) around the face and putting the person's name near the face bounding box.

In [8]

The first functiondraw_rectangle draws a rectangle on the image based on given coordinates. It uses OpenCV's built in functioncv2.rectangle(img, topLeftPoint, bottomRightPoint, rgbColor, lineWidth) to do so.

The second functiondraw_text uses OpenCV's built in functioncv2.putText(img, text, startPoint, font, fontSize, rgbColor, lineWidth) to draw text on the image.

Now that we have the drawing functions, we just need to call the face recognizer's predict(face) method to test our face recognizer on test images. The following function does the prediction for us:

In [9]

If we break this last code down:

  • line-6 read the test image.
  • line-7 detect the face from test image
  • line-11 recognize the face by calling face recognizer's predict(face) method. This method will return a label.
  • line-12 get the name associated with the tag.
  • line-16 draw rectangle around the detected face.
  • line-18 draw name of the predicted individual above face rectangle.

Now that we have the prediction function well defined, the next step is to call this function on our test images, display them to see if our face recognizer correctly performs face recognition. 

So, let's do it. This step is what we have been waiting for…

It’s Now or Never

In [10]

The face recognition worked! It can tell who I am and who Elvis Presley is, so it’s been trained correctly.
I had a suspicious mind…
I kept listening to random 50s music and dancing with my hips; not trying to be sexy. It just happened. I mean I didn’t remember buying those blue suede shoes but I thought they might have been left here by someone else.
I'm all shook up!

4. Endnotes

You can download the complete code and relevant files from this Github repo.

Face Recognition is fascinating on and OpenCV has made it incredibly straightforward and easy for us to code it. It just takes a few lines of code to have a fully working face recognition application.

We can switch between all three OpenCV face recognizers by changing only a single line of code. It's that simple.

Although EigenFaces, FisherFaces, and LBPH face recognizers are fine, there are even better ways to perform face recognition like using Histogram of Oriented Gradients (HOGs) and Neural Networks.

More advanced face recognition algorithms are implemented using a combination of OpenCV and Machine Learning.

I have plans to write some articles for those more advanced methods as well.

But for now, I want to figure out if the King has left the building…

Please let me know in your comments what do you think about this tutorial and my future plans (now that I have more peace in my life)

 

0

Ramiz Raja
Ramiz Raja

Ramiz Raja is a Self-Driving Car Engineer who loves automation and is passionate about autonomous vehicles. He has been working on computer vision and machine learning projects for more than 4 years. Some of his projects include Vehicle Detection and Tracking, Advanced Lane Finding, Pedestrian Detection, and Traffic Sign Detection and Classification. He loves writing about computer vision and sharing his knowledge with others.

What are you waiting for?

EMPOWER YOUR CAREER WITH SUPERDATASCIENCE

CLAIM YOUR TRIAL MEMBERSHIP NOW
as seen on: