Complete and temporally consistent video outpainting

We describe a novel method for video outpainting. The goal of outpainting is to fill in missing regions at the edges of video frames. Our focus lies on converting portrait (9:16) to landscape (16:9) video. In contrast, most video comple- tion research is focused on inpainting: filling a masked sec- tion within the frame based on the remaining, known pixels. Our proposed method consists of three main aspects: (1) We form a background estimation using video object seg- mentation and video inpainting methods, (2) we use optical flow to form temporal consistency, and (3) we propose im- age shifting to improve individual frame completions. Our method is able to successfully broaden the aspect ratio of a video. On most videos, we achieve realistic results. Only on videos with complex camera motion and foreground objects leaving the frame, the quality is less. In contrast to other state-of-the-art methods, our method is able to reconstruct the full frame, including unseen image parts. Moreover, it is temporally consistent. We evaluate our method on the DAVIS and YouTube-VOS datasets. The code is publicly available.

ISBN:978-1-6654-8739-9

Jaar van publicatie:2022

Toegankelijkheid:Open