I once spent three hours on a single strand of hair.

Not because I’m obsessive. Because the client was a record label, the subject was a woman with curly, wind-blown hair photographed against a white studio backdrop, and the final image needed to place her inside a thunderstorm. Every strand that read wrong would pull the viewer out of the image. In compositing, the moment someone thinks “that looks edited,” you’ve lost them. The mask is usually what gives you away.

Masking is not a technical checkbox. It is the core craft of compositing. If you understand what a mask is actually doing to your image data, you will make better decisions at every step of the workflow.

What a Mask Is Actually Doing to Your Pixels

A layer mask in Photoshop is an 8-bit grayscale channel attached to a layer. White reveals. Black conceals. Gray creates partial transparency. That sounds simple, and the concept is. The execution is where most compositors lose the plot.

When you paint black on a mask with a hard brush at 100% opacity and 100% flow, you are making a binary decision: visible or not visible. That binary thinking is what makes composites look pasted. Real edges in photography are not binary. They are soft, they are motion-blurred, they are diffracted by the lens. A subject photographed at f/1.8 has edges that fall off differently than the same subject at f/8. Your mask needs to reflect the optical reality of the original capture, not the clean geometry of a selection.

The practical implication: before you build any mask, look at your source image and identify its optical characteristics. What is the depth of field? Is there motion blur on any edge? What focal length was used? A wide-angle lens at close range creates edge distortion. A telephoto at distance compresses. These details live in your edges, and your mask has to honor them.

The Selection Is Only the Starting Point

Most people treat the initial selection as the mask. It isn’t. It’s raw material.

My workflow starts with Select > Subject (in Photoshop 2024 and newer, this is significantly better than it was three years ago), followed immediately by going into Select and Mask. I work at 100% zoom on the edges I care about most. For the record label project I mentioned, the hair was the entire battle. I used the Refine Edge Brush at a radius of about 3 to 5 pixels along the hair boundary, letting Photoshop sample the transition between hair and background. On a white backdrop, this works well. On a complex background, you need a different approach, which I’ll get to.

Once I output the selection as a layer mask, I switch to a 5-pixel soft brush at 20% opacity and begin painting by hand. Not to draw a new edge, but to push the existing one. I darken areas that should fall into shadow. I soften transitions at the shoulder where the jacket fabric meets the background. The mask is not done when the selection is made. The mask is done when the edge is optically plausible.

For hair and fine detail against complex backgrounds, I use a luminosity-based approach. I duplicate the layer, desaturate it, boost the contrast with Curves until the subject separates from the background, and use that as the basis for my mask. Channels work the same way. Find the channel with the most contrast between subject and background, duplicate it, paint in corrections, and load it as a selection. This is not a new technique. It is a reliable one.

When the Background Is Working Against You

The thunderstorm composite did not have a clean studio backdrop to work from. I was pulling a second element, a silhouetted tree, from an image shot at dusk against a purple-gray sky. The tree’s dark branches against a mid-tone sky gave me almost no contrast to work with in any single channel.

Here is what I did. I duplicated the layer and applied a high-contrast black and white adjustment, pushing the blues toward white and the darks toward black using the color sliders in the Black and White panel. That gave me a workable channel I could load as a selection. I still had to paint corrections on the mask by hand for about 40 minutes, but the selective desaturation gave me an 80% accurate edge to start from instead of 20%.

The tool that has changed my complex-background workflow more than anything else in the last two years is the AI-powered masking in Photoshop’s Remove Background function combined with manual refinement. It is not perfect, but it removes the first hour of labor and leaves you with something correctable.

The Shadow Problem Nobody Talks About Enough

The mask gets the subject into the scene. The shadow keeps them there.

My first published composite, a promotional piece for a small theater company, had a shadow that traveled in the wrong direction relative to the light source in the background plate. The client approved it. It printed. Nobody in the audience said a word. I noticed the moment the files went to press and I have not made that mistake since.

Shadows created with a distorted duplicate of your subject layer, filled with a dark color and set to Multiply blend mode between 45 and 65% opacity, will almost always look more believable than a drop shadow effect. The key variable is the angle. Before I paint or warp any shadow, I find the primary light source in my background plate and I trace a line. The shadow falls opposite the light, along that line. Every time.

Feathering Is Not a Shortcut, It’s a Decision

Feathering a selection before converting it to a mask is appropriate in specific situations: elements with soft natural edges, objects in motion, subjects in shallow depth of field. It is not a substitute for building an accurate edge. A feathered mask on a crisp subject will make them look like they are floating behind frosted glass.

When I do feather, I use values between 0.5 and 2 pixels for subjects with moderate depth of field and push toward 4 to 6 pixels only for elements that were clearly shot with a very shallow focus. Those numbers are not arbitrary. They correspond to the actual pixel-level softness you see when you zoom to 200% on the original edge.

The mask is the truth of your composite. Build it like the image depends on it, because it does.