A research report is now available on
Purpose. Evaluation of neural computational models of motion perception currently lacks a proper methodology for benchmarking. Here, we propose an evaluation methodology for motion estimation mechanisms which is based on human visual performance, as measured in psychophysics and neurobiology. Offering proper evaluation methodology is essential to continue progress in modeling. This general idea has been very well understood and applied in computer vision where challenging benchmarks are now available for several key problems, allowing models to be compared and further improved (Baker et al. 2007). The proposed standardized tools provide a straightforward methodology in order to identify the necessary and sufficient mechanisms involved in motion processing. This allows to compare different approaches, and to challenge current models of motion processing in order to define current failures in our comprehension of visual cortical function.
Method. We built a database of image sequences to depict input test cases corresponding to displays used in psychophysical settings or in physiological experiments. The data sets are fully annotated in terms of image and stimulus size (resolution) and ground truth data concerning dynamics, direction, speed, etc. Since different kinds of models have different kinds of representation and granularity (which is also valid for experimental settings), we had to define generic outputs for each considered experiment as well as correctness evaluation tools. We propose to use output data generated by the considered model as read out that relates to observer task or functional behavior. Amplitude of pursuit or direction likelihoods (histograms) are two examples.
Initial results. We probed several models of motion perception by utilizing the proposed benchmark The employed models show very different properties and internal mechanisms, such as feedforward normalizing models of V1 and MT processing (Simoncelli & Heeger 1998) and recurrent feedback models (Bayerl & Neumann 2004, Tlapale et al. 2008). Our results demonstrate the usefulness of the approach by highlighting current properties and failures in processing. The complete database as well as detailed scoring instructions are available on our website http://www-sop.inria.fr/neuromathcomp/software/motionpsychobench together with results derived by investigating several models including the ones mentioned above. We suggest that the proposed approach provides a valuable tool to unravel the fundamental mechanisms of the visual cortex in motion perception.
