Your brain's hidden color correction: color constancy

Short answer

The light that reaches your eye depends jointly on an object's color and the color of the light falling on it; yet you see an apple as red both under the noon sun and under an incandescent lamp. The brain does this by comparing objects with their surroundings rather than one by one, estimating the light illuminating the scene and subtracting it from the calculation. A camera, however, doesn't understand the scene, doesn't know familiar objects, and records each pixel on its own; so it can only make the same correction with simple assumptions, and it gets it wrong more often.

Why isn't the incoming light enough?

The light that reaches your eye from a surface is the product of two things: which wavelengths the surface reflects and in what proportion, and which wavelengths the light falling on it carries and how much of each. A white sheet of paper at noon reflects far more blue light than the same white paper does under an evening incandescent lamp. The light reaching your eye is markedly different in the two cases.

Yet the paper looks white in both cases. Edwin Land and John McCann open their 1971 paper with exactly this observation: color sensations show a strong relationship to reflectance, even though the amount of light reaching the eye depends on the product of reflectance and illumination. The visual system must be reaching this result by some method that doesn't measure the amount of light.

The brain's tendency to see an object's color independent of the illumination is called color constancy; KE-03, which explains why the dress photo split everyone in two, introduced this concept. Here we look at its mechanism and its limits.

Land's Mondrian experiments

Edwin Land, the founder of Polaroid, ran a long series of experiments on color vision starting in the late 1950s. The most famous were done with boards called "Mondrians," because they resembled the paintings of the artist Piet Mondrian: a surface divided into rectangles, made up of dozens of different colors of matte paper.

Land lit this board with three separate projectors; each one gave off light in only a narrow wavelength band, long, medium, or short. He first measured the energy in the three bands reaching the eye from one piece of paper on the board, say a rectangle that looked white. Then, by changing the intensity of the projectors, he readjusted the illumination so that a completely different-colored piece of paper, say a rectangle that looked green, sent the eye the exact same three energies. Physically, the light reaching the eye from the two papers was now identical. Observers, however, still saw the green paper as green. What determined the color wasn't the light coming from that point, but that point's relationship to its surroundings.

Land presented these results to a general audience in Scientific American in 1977. The explanation he proposed was retinex theory: the name combines the retina with the cortex, because Land wasn't sure where the calculation was taking place.

The core of the theory is this: for each cone channel, a surface's lightness in that channel is calculated relative to the brightest surface in the scene, through a chain of ratios formed with neighboring surfaces. Because illumination affects every surface by the same proportion, it cancels out in these ratios. So the green paper, whatever the absolute light reaching the eye, stays relatively bright compared to its neighbors in the medium-wavelength channel and relatively dark in the other channels, and so looks green.

How does the brain estimate the light?

Retinex was one explanation, but not the only one. Later research showed that the brain uses more than one clue together for its illumination estimate.

The first clue is in the retina itself. Cones that stay under warm light for a long time reduce their sensitivity to the long-wavelength light they're receiving in abundance; this partly balances out the light's effect on its own.

The second clue is comparison; as retinex says, a surface's color is evaluated relative to its neighbors. The third is clues coming from the scene itself: reflections on bright surfaces show the color of the light directly, and the color of shadows signals a second light source. The fourth is knowledge: you know that a banana is yellow, a leaf is green, and what color a familiar person's skin is, and this knowledge helps the estimate.

The power of these clues shows up when you change the surroundings. In a famous image Lotto and Purves published in 2002, two cubes made of colored squares stand side by side; one is drawn as if lit by bluish light, the other by yellowish light. One square on the top face of each cube is, in pixel value, exactly the same gray. In the blue-lit cube the gray square looks yellowish, and in the yellow-lit cube it looks bluish. The brain subtracts the light it estimates for each cube, and the same gray turns into two different colors.

How complete is constancy?

Color constancy is powerful but not complete. Researchers use a constancy index to measure it: the color an observer sets is compared with the color a perfect observer, who has fully subtracted the effect of the light, would choose.

A comprehensive review published in 2011 by David Foster of the University of Manchester brings these measurements together. In a study done with images of natural scenes, the average index ranged from 0.60 to 0.97 depending on the scene and the change in light. In another experiment, when the background around the test surface was shrunk and left only at the edge of the visual field, the measured constancy dropped from about 0.99 to 0.49. So constancy depends on how much information is in the scene.

There are also known cases where constancy collapses. If the light is in a very narrow wavelength band, as in the low-pressure sodium lamp example in KE-01, there's no color information left to compare between surfaces. If the scene is dark or the clues are weak, as in the dress photo, the brain can get caught between two different light estimates. If a surface is shown alone, on a dark background, without any surroundings, constancy is largely lost.

Why can't cameras correct for this on their own?

The answer to the question on the lesson card is that these clues the brain uses aren't present in the camera. The camera's sensor measures the light at each pixel but doesn't understand the scene. It doesn't know which surface is a white wall and which is an orange curtain; it doesn't remember the color of familiar objects; it can't draw as rich an illumination estimate from three-dimensional structure, shadows, and glare as the brain can. On top of that, a photo is later viewed in a different setting from the one it was taken in; the viewer's own constancy works according to the light of the room they're in, not the light in the photo.

This is why cameras try to imitate the brain with auto white balance: they use simple rules, like assuming the scene's average is gray, or image recognition. How these methods work and where they go wrong is the subject of FS-01; the example below shows what the same difference looks like on an ordinary evening. Even today's most advanced algorithms wrestle with essentially the same difficulty: separating light from surface by looking only at the reflected light is a problem with no definite solution on its own.

Example — the same face under two lights

Say you meet a friend first at noon by a café window, then in the evening at a restaurant lit with warm bulbs. Their skin looks the same to you in both settings. Yet the light reaching your eye is very different in the two settings: at the restaurant, the skin reflects noticeably more orange light than it does in the window light.

Your brain balances this out in several ways: it adapts to the warm light, uses the table's white cloth and the walls as comparison surfaces, and also factors in its expectation, since it already knows your friend's face. This correction works regardless of the skin's color, darkness, or undertone.

Photos you take on your phone in the same two settings, however, look different. If the camera can't fully subtract the restaurant's light, the skin skews orange. What's more, when you later look at the photo in a different setting, say on a screen in daylight, your brain is no longer adapted to the restaurant's light; the restaurant's clues are confined inside a small frame. That's why the shift in the photo stands out as a shift you didn't notice at the table.

Common misconception

Summary

1. The light reaching the eye depends jointly on the surface's reflectance and the color of the illumination; color constancy is the brain's tendency to subtract the illumination and see the surface.
2. Land's Mondrian experiments showed that two surfaces sending the eye the same light can look different colors because of their surroundings; retinex theory derives color from lightness comparisons made with neighbors in each cone channel.
3. Constancy is achieved through chromatic adaptation, comparison, scene clues, and knowledge, but it isn't complete; a camera can't make this correction on its own because it doesn't understand the scene and because a photo is viewed in a different setting.

Test yourself

1. (Easy) What two things does the light reaching our eye from a surface depend on?
2. (Easy) What does a value of 1 on the constancy index indicate?
3. (Medium) Why can't cameras make this correction on their own?
4. (Medium) What does it prove that observers in the Mondrian experiment still saw the green paper as green?
5. (Hard) Why does the same gray square look like a different color in the two cubes in Lotto and Purves's image?
6. (Hard) Why does the orange shift you didn't notice at the restaurant stand out when you look at the photo in daylight?

Answers

1. On which wavelengths the surface reflects and in what proportion, and the composition of the light falling on it; the light reaching the eye is the product of these two things ("Why isn't the incoming light enough?").
2. Perfect constancy; it shows that the observer chooses the color that an observer who has fully subtracted the illumination would choose ("How complete is constancy?").
3. The camera measures the light at each pixel but doesn't understand the scene; it doesn't know the color of familiar objects, and it can't draw as rich an illumination estimate from shadows and three-dimensional structure as the brain can. Also, the photo is viewed in a different setting, and the viewer's constancy works according to that setting's light ("Why can't cameras correct for this on their own?").
4. That color is determined not by the light coming to the eye from a point, but by that point's relationship with its surroundings; that two papers can be seen as different colors even when the light reaching the eye from them is physically identical ("Land's Mondrian experiments").
5. The two cubes are drawn as if lit by differently colored light; the brain subtracts the light it estimates for each cube. Once the blue light is subtracted, the gray is left looking yellowish; once the yellow light is subtracted, it's left looking bluish ("How does the brain estimate the light?").
6. At the restaurant, your brain had adapted to the warm light and was using the surrounding surfaces for comparison. When you look at the photo in daylight, you're no longer adapted to that light, and the restaurant's clues are confined to a small frame, so the shift becomes visible ("Example — the same face under two lights").

Sources

• Land, E. H., and McCann, J. J., "Lightness and Retinex Theory," Journal of the Optical Society of America 61(1): 1–11, 1971.
• Land, E. H., "The Retinex Theory of Color Vision," Scientific American 237(6), 1977.
• Foster, D. H., "Color constancy," Vision Research 51(7): 674–700, 2011.
• Lotto, R. B., and Purves, D., "The empirical basis of color perception," Consciousness and Cognition, 2002.

Next step

BI-14 gathers the places where this lesson's calculation goes wrong: the afterimage flag that stays in inverted colors after you look away, the same gray squares looking different on a shadowed checkerboard, and colored shadows. BI-19, meanwhile, turns perception into a measurement: it explains the CIELAB space, built with the goal of arranging colors according to the differences the eye sees, through the concepts of lightness, chroma, and hue angle.