The proposed invention represents a motion estimation algorithm for convolutional neural network (CNN) accelerators designed for live computer vision applications.
In the real-time video, every frame differs only slightly from previous frames. A generic CNN accelerator runs equally expensive computations for every frame. An alternative strategy, termed activation motion compensation (AMC), is to process the input video as a mixture of keyframes, which undergo full and precise CNN execution, and predicted frames, which use cheaper approximate execution.
During keyframes, AMC stores the network’s input and the target layer’s output for reuse during predicted frames. During predicted frames, AMC detects the motion between the last stored keyframe and the new input frame, produces a 2D vector field describing the visual displacement between the frames, and updates previously computed CNN activations.
CNN activations are linked to specific regions of the input image referred to as receptive fields. RFBME estimates the motion of entire receptive fields. The resulting displacement vector for each input receptive field maps to a corresponding displacement vector for a value in the target activation. Because these receptive fields overlap significantly there is a great deal of redundant work with simplistic block motion estimation. RFBME is a system that can exploit this redundancy.
Algorithms that require receptive field motion analysis, e.g. AMC algorithm
Name: Ryan Luebke