End-to-End Avatar Capture in the Wild

In computer vision, “ground truth” usually lives in a studio. You have perfect lights, synchronized cameras, and subjects wearing wig caps. If you have that, digital humans are a solved problem.

But building for the real world means accepting the kitchen. You get harsh ceiling lights, selfie distortion, and users who refuse to take off their glasses.

At Copresence, we found that bridging the gap between cartoonish avatars and studio realism isn’t about better cameras. It requires treating a noisy selfie not as a “bad scan” but as a sparse signal for an inference problem.

De-lighting and Accessories Removal

Our first hurdle was lighting. A raw scan captures skin plus shadows. If you project that texture directly, the avatar looks permanently lit by a ceiling bulb.

We trained models to decouple lighting from texture, predicting clean albedo (pure skin color) from noisy inputs. We do the same for accessories. If you scan a user with glasses, normally standard algorithms fuse the frames into the face. Our pipeline identifies glasses as non-face geometry and separates them.

The Hair Problem

Hair is the hardest part of digitization. It’s tough to scan with traditional methods due to its sparsity and chaotic reflection.

To solve this, we trained a proprietary AI model to predict the user’s hairstyle. The predicted style is already close to the user’s style, but we refine it even more using an iterative optimization approach that matches individual strands to the user’s photographed hairstyle.

The Hybrid Future

The output is standard 3D assets compatible with engines like Unreal or Unity. However, our method can also be used to enable volumetric or neural rendering techniques. The geometry then acts as a 3D proxy, while neural networks “paint” hyper-realistic details on top in real-time.

Try it Yourself

If you are curious about how this approach holds up in practice, you can test the current iteration of our pipeline and generate your own avatar at copresence.tech .