FATE: Full-head Gaussian Avatar with Textural Editing from Monocular Video

1Nanjing University, 2OPPO
+corresponding author
Teaser Image

From a monocular portrait video input, we propose FATE to reconstruct an animatable 3D head avatar, which enables Gaussian texture editing and allows for 360$^\circ$ full-head synthesis.

Abstract

Reconstructing high-fidelity, animatable 3D head avatars from effortlessly captured monocular videos is a pivotal yet formidable challenge. Although significant progress has been made in rendering performance and manipulation capabilities, notable challenges remain, including incomplete reconstruction and inefficient Gaussian representation. To address these challenges, we introduce FATE — a novel method for reconstructing an editable full-head avatar from a single monocular video. FATE integrates a sampling-based densification strategy to ensure optimal positional distribution of points, improving rendering efficiency. A neural baking technique is introduced to convert discrete Gaussian representations into continuous attribute maps, facilitating intuitive appearance editing. Furthermore, we propose a universal completion framework to recover non-frontal appearance, culminating in a 360$^\circ$-renderable 3D head avatar. FATE outperforms previous approaches in both qualitative and quantitative evaluations, achieving state-of-the-art performance. To the best of our knowledge, FATE is the first animatable and 360$^\circ$ full-head monocular reconstruction method for a 3D head avatar.

Method

In Stage I, we perform sampling-based densification in the UV space and train a Gaussian head avatar using the preprocessed monocular video dataset. In Stage II, given the learned head avatar, we construct a continuous function $f(\mathbf{p})$ in the UV space using U-Net $\mathcal{H}$ and bilinear kernel $\mathcal{B}$, baking the Gaussian attributes into several maps. Additionally, we propose a universal framework to complete the side and rear appearance under monocular settings.

Self Reenactment

Cross Reenactment

Full-head Completion

Edited Animation

More Results

BibTeX

@misc{zhang2024fatefullheadgaussianavatar,
      title={FATE: Full-head Gaussian Avatar with Textural Editing from Monocular Video}, 
      author={Jiawei Zhang and Zijian Wu and Zhiyang Liang and Yicheng Gong and Dongfang Hu and Yao Yao and Xun Cao and Hao Zhu},
      year={2024},
      eprint={2411.15604},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2411.15604}, 
}