Eye and face tracking

Learning goal

You will understand how the camera finds the face and the eye, and be able to decide when to leave this help on and when to switch it off. You will explain that recognition isn't measuring sharpness but choosing where to look; you will set the human, animal and vehicle options of subject detection according to the scene; you will choose for yourself which eye will be sharp; you will recognize in advance the situations where detection gets confused; and in dim or backlit scenes, on any skin, light or dark, you will know what to do to keep the face readable for the camera.

The question

In a park, you are photographing a dog that keeps running and stopping, turning right and then left. If you tried to move the focus point by hand, the dog would escape the point every time. But the camera draws a small frame around the dog's head in the viewfinder, then the frame narrows down to its eye, and wherever the dog runs, the frame goes with it. Without touching a single button, the dog's eye is sharp in most of the frames you take.

How does the camera find eyes? How does it tell an eye apart from a piece of fabric, a leaf or a stain on a wall, and where is the limit of this skill? In this lesson we will talk about what eye and face tracking does, what it can't do, and when you need to back it up with your own decision.

Context

In Lesson 10.1 we saw that autofocus finds sharpness by measuring the contrast at edges or the slide between two beams. This measurement answers one question: is the thing at this point sharp, and if not, which way should the lens go? But it doesn't answer the question of which point to look at. In Lesson 10.3 you were the one answering this second question: you were taking the focus point onto the eye by hand.

Eye and face tracking hands this second job over to the camera. When the camera recognizes a face, an eye, an animal or a vehicle in the scene, it places the focus point there itself and carries it along as the subject moves. The system that measures sharpness stays the same; what changes is who decides where the measurement is made. Knowing this distinction lets you understand where the problem is when tracking goes wrong.

Simple explanation

How does the camera recognize a face?

The camera's sensor keeps reading the image even before you press the shutter button; this is the live image you see in the viewfinder or on the screen. Eye and face detection is software that scans this live image many times a second. The software has been prepared, with a very large number of example images, to recognize the general shape of faces, eyes or particular animals: it looks for a pattern such as two dark points, a bump between them and a line below, and tries to catch this pattern at different angles and different sizes.

When a face is found, the camera draws a detection frame around it. On many cameras a frame appears first around the face, then around the eye, and when sharpness is found, the frame's color or shape changes. There is an important subtlety here: recognition chooses where focus will look, but sharpness is still found by the focus system that measures phase difference or contrast. So the camera can recognize a face correctly and still miss sharpness; on the other hand, even if it doesn't recognize the face at all, it can give that face perfect sharpness with single-point focus. This is why face detection, and the option many menus list as eye detection / Eye AF, are best thought of as a guide for where to measure, not as a guarantee of sharpness.

Human, animal, vehicle: which subject?

Many modern cameras can recognize not only human faces but other subjects too. Although it varies by camera, the options are often grouped as human, animal, bird and vehicle; the vehicle group can include cars, motorcycles, trains or planes. A camera that recognizes vehicles tries, for example, to focus on a racing car driver's helmet or the front of a train; a camera that recognizes animals looks for the eye of a cat, a dog or a bird. This kind of subject detection works only for the shapes it was prepared for.

On cameras where this choice is made from the menu, choosing the right group matters. In human mode a bird's eye isn't looked for; in bird mode a child's face can be pushed into second place. Some cameras also offer an automatic setting that chooses the group itself; this is useful when switching between very different subjects, but if there is both a person and a dog in the scene, which one it will pick isn't always predictable. If your subject is clear, choosing the group explicitly as well reduces the camera's margin of error.

Which eye will be sharp?

If the face isn't turned fully toward the camera, the two eyes are at different distances; with a wide aperture these two eyes may not both fit into the slice of sharpness at the same time. That is why cameras usually offer an eye selection setting: automatic, left eye or right eye. In the automatic setting the camera usually chooses the eye closer to the camera; many portrait photographers also prefer the near eye to be sharp, because the viewer's gaze usually goes to it first.

This is a habit, not an unchangeable law. If the light falls on the far eye and the near eye stays in shadow, or if your story is about a look that belongs to the far eye, you can choose the eye that will be sharp. If you want both eyes to be sharp together, stopping the aperture down a little to widen the slice of sharpness is a more reliable path than trusting detection to pick the right eye.

Where does detection get confused?

Detection works well as long as the shape of a face or an eye can be read in the image. When the shape breaks down, the camera falters. In full profile one eye and half a face are visible; sunglasses hide the eyes, and a reflection in the lens of a pair of glasses is sometimes chosen instead of the eye; the brim of a hat buries the eyes in shadow; a mask covers the lower half of the face. In a crowd, the face the camera chooses is often the face that takes up the most room in the frame, and this may not be your subject; it also happens that it jumps to the face on a poster in a shop window, a statue or a painting.

Light also decides whether the shape can be read. In a dim place, or in front of a bright window, the difference between the eyes and the skin shrinks on a backlit face. In this case the eyes of a dark-skinned face can stand out less easily against the pattern the software is looking for than the eyes of a light-skinned face; in strong noon sun, on the other hand, the detail of a light-skinned face can get lost in bright patches. Whatever the skin, the solution is the same: bring a little light onto the face, move the subject somewhere brighter than the background, or, if detection still can't find it, switch to single-point focus and put sharpness on the eye yourself.

Visual evidence

One late afternoon, in a small coffee shop in a bazaar, you are photographing a dark-skinned coffee maker with glasses, standing behind the counter over a small long-handled coffee pot. In the shop window behind them is a bright advertising poster, with a smiling face on it. With eye detection on, in the first frame you take with wide-area focus, the detection frame sticks to the face on the poster; the poster is sharp, and the coffee maker's face is a soft shadow.

You ask the coffee maker to turn the pot a little toward the light coming from the window, and you move the focus area to a zone over the coffee maker, so that detection works only in that area. This time the frame appears on the coffee maker's face, but it slides to the reflection of the window in their glasses. When they tilt their head slightly, the reflection disappears and the frame settles on their eye. In the last frame the coffee maker's eyelashes, the curve of their eyebrow and the steam rising from the pot are sharp; the poster behind is a soft patch of color. The face was there all along; what changed was the light and angle at which the camera could read it.

Common mistakes

Summary

Eye and face tracking is software that recognizes the shape of a face, an eye, an animal or a vehicle in the live image, places the focus point there and carries it along as the subject moves; sharpness, though, is still measured by the focus system. You choose the subject group and the eye that will be sharp according to the scene, and you know that detection can go wrong with reflections in glasses, profile, hats, crowds, posters or low light. Bringing light onto the face to keep it readable for the camera, or going back to single-point focus when needed, keeps the eye sharp on every skin.

Mini assignment

In this assignment you will try the strength and the limit of eye detection on the same day.

1. Photograph a moving subject, a child or a pet, with eye detection and continuous focus; count in how many frames the eye is sharp.
2. Photograph the same subject with eye detection off and single-point focus, and compare the two results.
3. Photograph someone with glasses or a hat, first with their back to a window, then with their face toward the window; look at where the detection frame settles.
4. Photograph two people of different skin tones in a dim room; observe how easily detection finds each of them, then bring a lamp closer to their faces and try again.

Check yourself

• Did you catch a frame where the detection frame appeared but the eye still stayed soft?
• Did detection work more easily on the face turned toward the window than on the one with its back to it?
• Were you able to photograph sharply, by adding light or switching to a single point, the face detection faltered on in the dim room?

Glossary

Mini quiz

1. What does eye detection mainly take over?

• A) Setting the exposure
• B) Choosing where sharpness will be searched for
• C) Setting the white balance
• D) Stopping down the aperture

2. If eye detection doesn't find a bird's eye in a bird photo, what do you look at first?

• A) The ISO value
• B) The white balance
• C) Whether the subject group is set to bird or animal
• D) The file format

3. On a face at an angle with a wide aperture, if you want both eyes to be sharp together, which is the most reliable path?

• A) Setting eye selection to the right eye
• B) Stopping the aperture down a little to widen the slice of sharpness
• C) Making the shutter speed faster
• D) Leaving detection on automatic

4. If the camera can't detect a backlit dark-skinned face in front of a bright window, what do you do?

• A) Press the shutter button again and again and wait
• B) Sharpen the photo afterward
• C) Lower the exposure compensation
• D) Bring light onto the face, or put sharpness on the eye yourself with single-point focus

Answers

1. B — Recognition chooses where the focus point goes; the system that measures sharpness is still the focus system.
2. C — A camera left in human mode doesn't look for a bird's eye; the right subject group comes first.
3. B — Detection focuses on only one eye; to cover both eyes at once, the slice of sharpness must widen.
4. D — In backlight the difference between eye and skin shrinks; either you bring the light that makes the shape readable, or you place sharpness yourself.

Summary card

• Recognition chooses where to look; the focus system still finds sharpness.
• Choose the subject group for the scene: human, animal, bird, vehicle.
• You decide which eye is sharp; for both eyes, stop the aperture down a little.
• Profile, reflections in glasses, hats, posters and crowds can confuse detection.
• In dim light or backlight, on every skin, bring light to the face or switch to a single point.

Bridge to the next lesson

Detection, wide area and single point all rely in the end on the same measurement: the camera has to find enough contrast at an edge. In a dark room, behind a shop window's glass, or on the hairs of an insect photographed from a few centimeters away, this measurement falters and the lens goes back and forth. The next lesson talks about when autofocus gets it wrong and about finding sharpness by hand in these moments; and about how focus peaking, which colors sharp edges on the screen, and magnifying the image make this job easier.