I once spent four hours on a single strand of hair.

Not retouching it. Not painting it. Masking it. It was a portrait composite for a book cover, a woman standing against a stormy coastal sky, and the flyaway strands at the crown of her head were catching the light in exactly the way that makes a photo feel real and alive. I knew that if I lost those strands to a hard mask edge, the whole piece would collapse. The sky behind her would look pasted in. Which, technically, it was. My job was to make sure nobody could tell.

That’s what masking is, at its core: it’s the art of making a lie look like the truth.

Why Edges Lie More Than You Think

Most people think of masking as a selection problem. You want to isolate a subject, so you grab a tool, draw a line around them, and move on. But that framing misses what’s actually happening optically.

In the real world, edges are rarely clean. When a photographer captures a person standing in front of a sky, the camera sensor is responding to light that’s scattering, bouncing, and bleeding across the boundary between subject and background. Fine hair picks up color from the sky behind it. A soft jacket edge goes slightly out of focus. The transition between the subject and the world around them is a negotiation, not a hard border.

When you composite, you’re replacing one background with another. But the light information baked into your subject’s edges still belongs to the original scene. That’s the fundamental tension. The subject is carrying visual data from a world that no longer exists in your image. Your mask has to manage that transition honestly, or the eye will catch it immediately, even if the viewer can’t articulate why.

The Refine Edge Workflow That Actually Works

In Photoshop, I start almost every subject selection with Select Subject (under the Select menu), which as of recent versions uses Adobe’s AI engine and is genuinely good for roughly 80% of the work. But that last 20% is everything.

After getting my initial selection, I go to Select > Select and Mask. Inside that workspace, I set the View to “On Layers” so I can see the actual composite rather than a red overlay. I set Edge Detection radius to somewhere between 2px and 6px depending on the complexity of the edge. For hair and fur, I check Smart Radius. Then I use the Refine Edge Brush Tool (the second tool in the toolbar, keyboard shortcut R) and paint directly over the problem areas, typically the hair and any soft or translucent edges.

The output matters too. I almost always output to a New Layer with Layer Mask. Not a selection, not a new document. A layer with a mask I can come back and paint on manually. Because Refine Edge is excellent but not infallible, and I want the flexibility to go in with a soft black brush at 10-20% opacity and knock back any edge artifacts by hand.

For really complex hair, especially when the subject was shot against a light background and you’re placing them on a dark one, I’ll often use the Channels panel to build a luminosity-based mask by hand. You’re looking for the channel with the most contrast between the hair and the background, duplicating it, burning the midtones down with Curves, then loading it as a selection. It takes longer. It’s worth it.

Color Contamination: The Problem Nobody Talks About Enough

Here’s the thing that trips up a lot of people who are otherwise solid at making selections. Even a technically perfect mask will look wrong if you don’t address color contamination.

When your subject was photographed against a green screen or a bright white background, that background color has wrapped itself around the edges of your subject. It’s in the semi-transparent pixels. The fix isn’t in the mask. Inside Select and Mask, there’s a checkbox at the bottom called Decontaminate Colors. Check it and set the amount to around 50-70%. It works by replacing contaminated edge pixels with color extrapolated from the fully opaque areas nearby. It’s not magic, but it removes the halo that would otherwise appear and force you into destructive edits.

If the contamination is severe and Decontaminate Colors isn’t enough, I’ll add a new layer clipped to the subject, set to Color blend mode, and paint in the correct edge color by sampling from the interior of the subject. A 3-5px soft brush at 60-80% opacity. This is a technique I picked up from studying matte painting workflows, and it translates directly to photo compositing.

When the Mask Is Right But the Composite Still Looks Wrong

I spent six months studying how light behaves on water for a single album cover. The subject was a musician, and I needed to place her on a reflective surface so convincing that it felt like the photographer had actually been there. I got the mask perfect. Clean edges, no contamination, beautiful hair separation. And the composite still looked fake for weeks.

The problem wasn’t the mask. It was everything around it. No contact shadow. No reflection. No color bounce from the water surface onto the underside of her chin. A mask gets your subject into the scene. Integration is what makes them belong there.

This distinction is worth burning into your workflow. Masking is the cost of entry. The rest of the work is making the light believe what you’ve told it to believe.

The single most important thing I can tell you about masking is this: the goal is never a clean edge. The goal is a believable transition, and those are two completely different targets.