Second Evaluation Report

Executive Summary

This is the Second Evaluation Report for the ADVISOR project. It provides the second intermediate evaluation performed by the end-users associated with the project, i.e. STIB (metro of Brussels) and TMB (metro of Barcelona).

ADVISOR is a project that intends to improve management of public transport networks through the improved use of CCTV cameras. The project will undertake to integrate several developments in picture content detection and analysis with technologies for annotating video images, and presenting them for inspection by CCTV operators when trigger events occur. Metro stations will be used as experimentation sites for test beds and a demonstrator.

The goal of ADVISOR is to assist human operators by automatic selection, recording and annotation of "interesting" images such as "abnormal" crowd and individual’s behaviours.

The ADVISOR system will be made available to End-users through an appropriate Human Computer Interface (HCI) that will allow them to activate some functions and also to be alerted by the system as new interesting events or incidents are detected. This HCI will be used to let them evaluate the system three times during the project timescale.

Three main evaluation milestones have been programmed respectively after the completion of Test Bed 1, Test Bed 2 and Final Demonstration prototypes. They are conducted according to the Evaluation Plan described in Deliverable R8.1.

A second intermediate evaluation has been organised to capture the End-user feed-back when the Test Bed 2 set of developments were ready. As mentioned above, this was the second milestone where all the developments available at that point were put together and made available to all the project partners and attendees. The first difference from. the first intermediate evaluation was that, this time, the developments were integrated with each other. The second difference was that it has been possible to evaluate, at least partly, the system quantitatively.

Test Bed 2 was, by its nature, still an incomplete prototype and it was therefore not easy, especially for non-technical people, to differentiate the several software modules running together on many details, i.e. to differentiate the capture, motion detection, tracking, behaviour recognition and archiving modules. Moreover, due to the level of development achieved so far, only a part of the full evaluation process was possible (e.g. it was not possible to evaluate co-operation in a multi-user architecture with only one HCI). Nevertheless, this second intermediate evaluation has been conducted according to the Evaluation Plan.

Reactions to ADVISOR presentation have been very positive both from STIB and TMB. Some of the evaluators involved in the process were not previously aware of ADVISOR. Most of them did not work actively in the project.  This makes us quite confident that their opinions and feed-back are reasonably objective and could not be influenced by their own involvement in the work done.

Although the scope of this second evaluation was still limited, it has been useful to refine a number of issues, especially those concerning the Human Computer Interface specification. This will be reflected in producing a second release of Deliverable R5.1.

From the evaluation, we identified a number of actions to improve various aspects of the system. These were mainly to:

This demonstrated that, as expected, the second intermediate evaluation also provided feed-back to the core ADVISOR software that will be taken into consideration for the final demonstration.

Download the full report (2.074MB, Adobe Acrobat format).

 


Project Co-ordinator: Michael Naylor

Issue 5,  24 June 2003