Participants: Francois Bremond, Anh Tuan Nghiem, Valery Valentin, Ruihua Ma, Monique Thonnat.
Figure 1.
performance
evaluation depends on sensors (position, type) and scenes conditions. |
We led the project ETISEO (Evaluation du Traitement et de
l'Interprétation de Séquences Video) campaign on performance
evaluation of video surveillance. More than 16 international teams
have processed the video dataset and submitted their results. Our goal
was to propose a new evaluation methodology for inferring meaningful
information on the evaluated algorithms. Specifically we have carried
out the following works:
Refinement of the evaluation tools. Certain evaluation tools have some
limits so appropriate modification is needed. We found that different participants use
contradictory hypotheses. For example, some participants detect static objects and some participants do not detect the objects that
have stopped moving for a certain period of time. Therefore, it is impossible to compare the
results of two participants
that use two different hypotheses. To overcome this problem, filters were implemented
to extract the reference data on which all the participants have adopted the same
hypothesis;
Analize the evaluation metrics. There are some metrics that add noise to the
evaluation results. For some metrics, we have proposed to use another distance function
so that these metrics become more accurate;
Proposition of a new evaluation methodology. It can generalize the evaluation
results performed on selected videos to new video sequences. More precisely, we address
each video processing problem separately and estimate the upper bound of algorithm
capacity in solving a given problem. If this value is smaller than the difficulty
level of new sequences, we can conclude that the algorithm cannot achieve acceptable
performance on these sequences.
To validate the new evaluation methodology, we have defined two metrics to address
the problems of (1) handling weakly contrasted objects and (2) objects mixed with shadow.
The preliminary results show that, with this methodology, we can extrapolate the
evaluation results for new sequences.
Figure 2.
Selection of 6 sites for ETISEO video acquisition. |
Finally we organized in December 2006 a seminar on the evaluation campaign
gathering more than 50 participants from international
organizations. This work has been published in WMVC07 and AVSS07.