SteadyFlow: Spatially Smooth Optical Flow for Video Stabilization

Shuaicheng Liu1         Lu yuan2         Ping Tan1         Jian Sun2

1. National University of Singapore               2. Microsoft Research

Abstract:

We propose a novel motion model, SteadyFlow, to represent the motion between neighboring video frames for stabilization. A SteadyFlow is a specific optical flow by enforcing strong spatial coherence, such that smoothing feature trajectories can be replaced by smoothing pixel profiles, which are motion vectors collected at the same pixel location in the SteadyFlow over time. In this way, we can avoid brittle feature tracking in a video stabilization system. Besides, SteadyFlow is a more general 2D motion model which can deal with spatially-variant motion. We initialize the SteadyFlow by optical flow and then discard discontinuous motions by a spatial-temporal analysis and fill in missing regions by motion completion. Our experiments demonstrate the effectiveness of our stabilization on real-world challenging videos.

 

Paper [PDF]

Related Projects

Shuaicheng Liu, Mingyu Li, Shuyuan Zhu, Bing Zeng: CodingFlow: Enable Video Coding for Video Stabilization. IEEE Transactions on Image Processing (TIP), vol. 26, no. 7, pp. 3291-3302, 2017. [PDF]

Shuaicheng Liu, Ping Tan, Lu Yuan, Jian Sun, Bing Zeng: MeshFlow: Minimum Laency Online Video Stabilization. European Conference on Computer Vision (ECCV). 2016. [PDF][Video][Model Code]

Shuaicheng Liu, Lu yuan, Ping Tan, Jian Sun. Bundled Camera Paths for Video Stabilization. ACM Transactions on Graphics (Proceeding of SIGGRAPH) 2013. [PDF][project page]

Shuaicheng Liu, Yinting Wang, Lu Yuan, Jiajun Bu, Ping Tan, Jian Sun: Video Stabilization with a Depth Camera. IEEE Conference on Computer Vision and Patten Recognition(CVPR) 2012 [PDF][project page]

 

Video Spotlight

 

Full Demo Video: download [64Mb]

 

Downloads:

Example 1: Rolling shutter together with large occlusion
Input   Our method  
Example 2: The synthesized example
Input   Our method Liu et al 2013
Example 3: Two rolling shutter examples
Input  Our method Baker et al.2010
Input Our method Karpenko et al 2011
Example 4: Videos contain large foreground , moving towards camera
Input   Our method  
Input   Our method  
Example 5: Video contains quick camera zooming, comparison between with and without adaptive smoothing.
Input Our method No adaptive
Example 6: Stabilize by raw optical flow  
Input   Our method Raw optical flow
Example 7: Motion completion by strong gaussian smoothing
Input Our method Motion completion by gaussian smoothing
Example 8: more examples in the paper
Input  Our method  

 

Limitations

Our spatial-temporal analysis failed to distinguish foreground and background when videos contain dominate foregrounds, (foregrounds occupy more than half area of a frame and exist for a long time).

Input Our method
Input Our method
Input Our method
Input Our method