Abstract: We present a novel approach that enables photo-realistic re-animation ofportrait videos using only an input video. In contrast to existing approachesthat are restricted to manipulations of facial expressions only, we are thefirst to transfer the full 3D head position, head rotation, face expression,eye gaze, and eye blinking from a source actor to a portrait video of a targetactor. The core of our approach is a generative neural network with a novelspace-time architecture. The network takes as input synthetic renderings of aparametric face model, based on which it predicts photo-realistic video framesfor a given target actor. The realism in this rendering-to-video transfer isachieved by careful adversarial training, and as a result, we can createmodified target videos that mimic the behavior of the synthetically-createdinput. In order to enable source-to-target video re-animation, we render asynthetic target video with the reconstructed head animation parameters from asource video, and feed it into the trained network -- thus taking full controlof the target. With the ability to freely recombine source and targetparameters, we are able to demonstrate a large variety of video rewriteapplications without explicitly modeling hair, body or background. Forinstance, we can reenact the full head using interactive user-controlledediting, and realize high-fidelity visual dubbing. To demonstrate the highquality of our output, we conduct an extensive series of experiments andevaluations, where for instance a user study shows that our video edits arehard to detect.

References & Citations