How to remove the background from an image

The shape is the easy part

Finding roughly where the subject is is close to solved. A segmentation model trained on enough photographs will outline a person, a product or a pet correctly almost every time, and has done for several years.

What separates a cutout that looks real from one that looks cut out is the edge: the two or three pixels where subject becomes background. Get those wrong and the shape being perfect does not save it.

Three things go wrong there:

A hard edge. If every pixel is either subject or background, curves come out with visible stair-steps and the whole thing reads as pasted on. Real edges are partial: a pixel on the boundary is genuinely part subject, part background, and has to be part transparent.

Colour contamination. A pixel that is 60% hair and 40% green wall has a greenish cast baked into it. Make it 60% opaque without correcting the colour and you get a green fringe: the classic sign of a bad cutout, and the reason compositors have unmixed backgrounds for decades.

Too much feathering. Overcorrecting blurs the boundary until the subject looks slightly out of focus against a sharp background.

Why hair and glass are genuinely hard

Both break the assumption that a pixel belongs to one thing.

Hair is thinner than a pixel. A strand crossing a sensor site contributes maybe 30% of that pixel's colour, so the honest answer is 30% opacity, not a yes-or-no decision. This is why a good cutout of a person is mostly a good cutout of their hair, and why tools are judged on it.

Glass, smoke, water and motion blur are transparent by nature. What is behind them is part of them. There is no correct binary mask for a wine glass, and any tool that produces one has thrown away the thing that made it look like glass.

Fine mesh and foliage fail the same way, for the same reason.

If a cutout has to be perfect on any of these, expect to touch it up. No automatic tool is reliably right, and one that claims to be is producing confident nonsense on the hard 5%.

The white-shirt-on-a-white-wall problem

A subject region that matches the background exactly is the case that breaks naive approaches. A colour threshold, or a chroma key, sees white and removes it, including the shirt.

This is why segmentation models are worth the download: they recognise a person rather than not-background-coloured pixels, so the white shirt stays because it is part of the person, not because it is a different colour.

Running it locally versus paying for it

The commercial services (you know the ones) do this well and charge per image, or give you a low-resolution result free and the full one behind a subscription.

A model small enough to run in a browser is a real download: tens of megabytes the first time, cached afterwards, and a few seconds per image on your own processor rather than instantly on theirs.

What you get for that:

• The photograph never leaves your machine. For product shots this is

commercial confidentiality; for photographs of people it is more than that.

• No per-image cost and no resolution cap. The full-size result is the

only result.

• It works offline once the weights are cached.

What you give up is a couple of seconds per image and, on the very hardest edges, some quality against the best paid models. For most work the difference is not visible; on a backlit portrait with flyaway hair it can be.

Getting a better result from the photograph

The cutout is downstream of the picture. Cheap improvements, in order of how much they help:

• Separate the subject from the background by distance. A metre of gap

puts the background out of focus, and a soft background is far easier to cut than a sharp one.

• Avoid the subject wearing the background's colour. Solvable by a model,

still the hardest case.

• Even, diffuse light. Hard shadows on the background read as edges.
• Do not shoot into the light. Backlighting produces a bright rim through

the hair that is genuinely ambiguous.

• Shoot bigger than you need. More pixels across the boundary means more

information about what is partial.

What to do with the edge afterwards

Once the mask is right, two controls carry most of the remaining quality:

Edge position: move the boundary a fraction inward to eat a contaminated fringe, or outward to keep detail a tight mask is clipping. Inward by a pixel fixes more real photographs than any other single adjustment.

Feather: how sharply the mask falls from opaque to transparent. Match it to the sharpness of the photograph: a crisp studio shot wants almost none, a handheld shot at a slow shutter wants more, because its real edges are soft too.