Interactive Control over Temporal Consistency while Stylizing Video Streams

1Hasso Plattner Institute, University of Potsdam, Germany 2Digitalmasterpieces GmbH, Germany
*equal contribution
In EGSR 2023 / CGF Journal

Our blind video consistency approach allows for interactive control over temporal consistency in stylized videos and video streams.


Image stylization has seen significant advancement and widespread interest over the years, leading to the development of a multitude of techniques. Extending these stylization techniques, such as Neural Style Tranfser (NST), to videos is often achieved by applying them on a per-frame basis. However, per-frame stylization usually lacks temporal consistency expressed by undesirable flickering. Most of the existing approaches for enforcing temporal consistency suffer from one or more of the following drawbacks: They (1) are only suitable for a limited range of techniques, (2) do not support online processing as they require the complete video as input, (3) cannot provide consistency for the task of stylization, or (4) do not provide interactive consistency control. Domain-agnostic techniques to temporal consistency aim to eradicate flickering completely, but typically disregard aesthetic aspects. For stylization tasks, however, consistency control is an essential requirement as a certain amount of flickering adds to the artistic look and feel. Moreover, making this control interactive is paramount from a usability perspective. To achieve the above requirements, we propose an approach that stylizes video streams in real-time at full HD resolutions while providing interactive consistency control. We develop a lite optical-flow network that operates at 80 FPS on desktop systems with sufficient accuracy. Further, we employ an adaptive combination of local and global consistency features and enable interactive selection between them. Objective and subjective evaluations demonstrate that our method is superior to state-of-the-art video consistency approaches.

Supplementary Video


 @article {10.1111:cgf.14891,
journal = {Computer Graphics Forum},
title = {{Interactive Control over Temporal Consistency while Stylizing Video Streams}},
author = {Shekhar, Sumit and Reimann, Max and Hilscher, Moritz and Semmo, Amir and Döllner, Jürgen and Trapp, Matthias},
year = {2023},
publisher = {The Eurographics Association and John Wiley & Sons Ltd.},
ISSN = {1467-8659},
DOI = {10.1111/cgf.14891}


Our work "Interactive Control over Temporal Consistency while Stylizing Video Streams" work was partially funded by the German Federal Ministry of Education and Research (BMBF) through grants 01IS15041 – “mdViPro” and 01IS19006 – “KI-Labor ITSE” and the Research School on "Service-Oriented Systems Engineering" of the Hasso Plattner Institute.

Digital Masterpieces
BMBF: Bundesministerium für Bildung und Forschung
Universität Potsdam