SelfSwapper: Self-Supervised Face Swapping via Shape Agnostic Masked AutoEncoder

Jaeseong Lee^*, Junha Hyung^*, Sohyun Jung, Jaegul Choo

ECCV 2024

KAIST

* denotes first co-author.

Paper Code (TBA)

Key Idea

(A) Comparison with Prior Works: Prior methods balance conflicting losses: reconstruction and identity. Our approach uses a self-supervised method with a clear ground truth, leading to more stable training.

(B) Ours Base vs. Ours Full: We compare our base model with our enhanced method, which adds perforation confusion and random mesh scaling. Green masks show source 3DMM, red masks target 3DMM, and orange masks their intersection.

When the source face is larger, the base model cuts off the jaw, and when smaller, it fails to fill in gaps. Ours Full solves both, generating realistic results.

Curated Videos

left: source video, right: swapped video

Comparison with Target-Oriented Methods

Comparison with Source-Oriented Methods

You may find interesting ...

We focus on our research field as portrait manipulation. So we guide you guys to the our sibling works in the field of portrait manipulation.

RobustSwap introduces a StyleGAN2-based face-swapping method with minimizing source attribute leakage.

MagiCapture introduces a diffusion-based portrait personalization method with a novel training scheme.

BibTeX

@article{lee2024selfswapper,
      title={SelfSwapper: Self-Supervised Face Swapping via Shape Agnostic Masked AutoEncoder},
      author={Lee, Jaeseong and Hyung, Junha and Jeong, Sohyun and Choo, Jaegul},
      journal={arXiv preprint arXiv:2402.07370},
      year={2024}
    }