Participants: Francois Bremond, Anh Tuan Nghiem, Valery Valentin, Ruihua Ma, Monique Thonnat.
|
We led the project ETISEO (Evaluation du Traitement et de l'Interprétation de Séquences Video) campaign on performance evaluation of video surveillance. More than 16 international teams have processed the video dataset and submitted their results. Our goal was to propose a new evaluation methodology for inferring meaningful information on the evaluated algorithms. Specifically we have carried out the following works:
Refinement of the evaluation tools. Certain evaluation tools have some limits so appropriate modification is needed. We found that different participants use contradictory hypotheses. For example, some participants detect static objects and some participants do not detect the objects that have stopped moving for a certain period of time. Therefore, it is impossible to compare the results of two participants that use two different hypotheses. To overcome this problem, filters were implemented to extract the reference data on which all the participants have adopted the same hypothesis;
Analize the evaluation metrics. There are some metrics that add noise to the evaluation results. For some metrics, we have proposed to use another distance function so that these metrics become more accurate;
Proposition of a new evaluation methodology. It can generalize the evaluation results performed on selected videos to new video sequences. More precisely, we address each video processing problem separately and estimate the upper bound of algorithm capacity in solving a given problem. If this value is smaller than the difficulty level of new sequences, we can conclude that the algorithm cannot achieve acceptable performance on these sequences. To validate the new evaluation methodology, we have defined two metrics to address the problems of (1) handling weakly contrasted objects and (2) objects mixed with shadow. The preliminary results show that, with this methodology, we can extrapolate the evaluation results for new sequences.
|
Finally we organized in December 2006 a seminar on the evaluation campaign gathering more than 50 participants from international organizations. This work has been published in WMVC07 and AVSS07.
