Computational photography

Learning goal

You will be able to explain what your phone does behind the scenes when you press the shutter button: how HDR blends frames of different brightness, how night mode stacks frames on top of each other in the dark, how portrait mode blurs the background after the fact. You will recognize in which scene each mode shines and in which it leaves traces; you will check default settings such as face smoothing and turn them off when needed, and learn to accept the phone's decisions knowingly rather than blindly.

The question

Think of your mother's birthday dinner. Candles are burning on the table, and behind it there is a dark window. You raise your phone, take a frame and see something surprising on the screen: the dim room is as bright as day, the city lights outside the window can be made out, your mother's face is smooth and the wall behind her is softly blurred. Yet when you looked with your own eyes, the room was dim, the window was pitch black and the wall was perfectly sharp.

What does the phone do on its own? At the moment you press the shutter button, the phone isn't actually taking a single photo; it gathers several frames, aligns them, combines them, recognizes the face and the sky in the scene and edits each one separately. This is called computational photography. The result is often impressive, but it isn't always faithful to reality or flawless. In this lesson we will see how this invisible craftsmanship works and where it stumbles.

Context

A phone's small sensor can't gather as much light in a single frame as a big camera does. To get past this limit, manufacturers turned not to physics but to computation: many frames instead of one, decisions that change with the scene instead of fixed settings, a blur produced by software instead of a real lens blur. In the previous lessons you saw some of the side effects of this processing: the ghost trails that form in combined frames and the smoothing done in the dark. This lesson opens up, one by one, the modes behind those side effects.

Knowing them is the key to using your phone better. Someone who knows when each mode comes into play, what it changes and in which conditions it gets things wrong learns to turn a mode on and off, or to set up the scene in a way that suits the mode. The phone makes its decisions silently; this lesson makes those decisions audible. You will see the detailed settings in the module's last lesson, in pro mode.

Simple explanation

How do HDR and night mode work?

HDR stands for high dynamic range; that is, being able to show both very bright and very dark areas in the same frame. Think of a neighbor drinking coffee in front of a balcony door: outside is bright white, their face is dark. With HDR, at the moment you press the shutter button the phone takes several frames; some record the dark areas correctly, some the bright areas. Then it blends them: it takes the outside from the window and the face from the room, and produces a single frame. On most phones HDR is now always on and works without you noticing.

Night mode uses a similar idea in the dark. The phone gathers frames one after another over a second or two, sometimes longer. In each frame the grain appears in random places, but the scene itself is always in the same place. When the phone aligns the frames and takes their average, the random grain shrinks while the scene grows stronger. That is why a dim room comes out as bright as day. This stacking needs both the phone and the subject to stay still until the counter on the screen runs out.

How does portrait mode blur the background?

In a big camera, the blur in the background comes from the physics of the lens and the sensor. A phone's tiny sensor can't give that softness on its own; portrait mode closes the gap with computation. The phone produces a depth map that guesses what is in front and what is behind in the scene. To do this it uses more than one camera, the special structure of the sensor, or software trained to recognize images. Then it leaves the person sharp and blurs everything behind them with software.

From a distance the result is convincing, but up close it can leave traces. The depth map gets things wrong most at complex edges: loose, fine strands of hair, the outer edge of curly hair, glasses frames, the glass rim of a cup, the gaps between fingers. In these areas the blur either spills into the person or a sharp halo stays in the background. While a real lens's blur grows slowly with distance, computed blur sometimes sits at the same strength in front and behind, and the scene looks like a cardboard set.

What does the phone do to faces?

Part of computational photography is recognizing the scene. The phone tells apart the sky, greenery and faces in the frame and applies a different edit to each: it deepens the sky, livens up the green, brightens the face. This often gives pleasing results. But on some phones, beauty settings that go beyond this, smoothing the face or making the skin silky, may come switched on by default.

These settings don't treat everyone the same. A process that tries to pull a face toward a certain brightness and smoothness can show one face lighter than it is and another grayer than it is; it can wipe out the skin's true color, freckles, moles and texture. When you photograph people of different skin tones, checking the result with your own eyes, turning off the beauty setting if needed and showing the person in their own color is the photographer's responsibility. The fact that the phone makes a decision doesn't mean that decision is right.

Visual evidence

Go back to the birthday dinner. In the frame taken with the phone's own decisions, the room is as bright as day and the city lights outside the window can be made out. But while night mode gathers its frames, the grandchildren jumping around the cake appear twice, semi-transparent like ghosts. In your mother's photo taken in portrait mode, the outer edge of her curly gray hair has blurred along with the background; a lock of hair looks as if it has been cut off and thrown away. Half of the wine glass in her hand is sharp, half blurred. Because the beauty setting is on, the lines and moles on your mother's face have been wiped away, and her skin has slid to a tone lighter and flatter than its true color.

Now imagine photographing the same dinner deliberately. You tell the grandchildren "don't move for three seconds" and lean the phone on the table until the night mode counter runs out; the faces are sharp, with no ghosts. You photograph your mother's portrait in normal mode instead of portrait mode, and a little closer; her hair shows strand by strand, and the whole glass shines like glass. You turn off the beauty setting; the smile lines on her face, her moles and her skin color are just as they are. The room is a little dimmer, but it is truly the room of that evening. The same phone, the same dinner; the difference is that you manage the phone's decisions knowingly.

Common mistakes

Summary

A phone doesn't take a single photo; in HDR it blends frames of different brightness, in night mode it stacks frames on top of each other, in portrait mode it produces a depth map and blurs the background with software, and it edits faces separately. These computations are often impressive but can leave ghost trails on moving subjects, cut-outs at complex edges and an unreal smoothness on faces; someone who knows their limits manages the phone's decisions by turning modes on and off.

Mini assignment

In this assignment you bring your phone's invisible craftsmanship into view.

1. Photograph someone standing in front of a bright window, first in normal mode, and look at how balanced the window and the face appear.
2. Photograph the same person in portrait mode, zoom in on the photo and find where the blur gets it wrong at the edges of the hair, glasses or fingers.
3. Take a frame in night mode in a dim room; have someone move, then repeat while everyone stays still.
4. Check whether there is a beauty or face-smoothing option in the camera settings; if there is, photograph the same face with it on and off and note the difference.

Check yourself

• Were you able to find at least one edge where portrait mode got it wrong?
• Did you see that movement in night mode left a ghost trail and disappeared when everyone stayed still?
• Did you check whether your phone has a face-editing setting and make your decision?

Glossary

Mini quiz

1. How does night mode make a dark room look bright?

• A) By firing the flash several times
• B) By aligning the frames it takes one after another and averaging them
• C) By turning the photo black and white
• D) By making the sensor bigger

2. Where does portrait mode's background blur come from?

• A) From the phone's lens sliding backward
• B) From the lens being dirty
• C) From being computed with software according to the depth map
• D) From the photo being saved at low resolution

3. In which areas does the depth map get things wrong most?

• A) In the middle of a plain wall
• B) In a clear sky
• C) On a sharp, single-colored floor
• D) At fine strands of hair, glasses frames and glass rims

4. Why should the phone's face beauty setting be checked?

• A) Because it can pull the face into a single mold and change skin color and texture unrealistically
• B) Because it always makes the photo blurry
• C) Because it uses up the battery
• D) Because it works only on the front camera

Answers

1. B — In each frame the grain is in random places while the scene is always in the same place; when the frames are aligned and averaged, the grain shrinks and the scene brightens.
2. C — In a tiny sensor this softness doesn't form on its own; the phone guesses what is in front and what is behind and blurs the background with software.
3. D — At complex, fine and transparent edges it is hard to separate front from back; the blur either spills into the subject or a sharp halo stays in the background.
4. A — The default beauty setting doesn't treat every face the same; it can pull the skin toward a single brightness and smoothness and wipe out its true color, moles and texture.

Summary card

• HDR: blends frames of different brightness; the window and the face show together.
• Night mode: stacks and averages frames; don't move until the counter runs out.
• Portrait mode: depth map + software blur; gets it wrong at hair, glasses and glass edges.
• The phone edits faces separately; check the beauty setting.
• Manage the phone's decisions by turning modes on and off.

Bridge to the next lesson

You have seen what your phone does with software. But when you look at the back of your phone, you see not one but two, three, sometimes four cameras; and these aren't software but real lenses. The next lesson opens with this curiosity: why does my phone have 3 cameras? There you will see that the 0.5x, 1x and 3x buttons are actually different lenses, and how switching to the right camera instead of digital zoom changes the photo.