This document describes the validation of the Annotated Digital Video for Intelligent Surveillance and Optimised Retrieval (ADVISOR) Demonstrator system [1]. The report includes the data used for the validation, the validation process employed, the results obtained and an analysis of the results.
The system was validated at the Sagrada Familia Metro station in Barcelona, where the Demonstrator was taken for evaluation, demonstration and validation purposes. The validation process involved testing the behaviour recognition, the archive search and the archive retrieval of the ADVISOR system. The validation of the behaviour recognition comprised the detection of fighting, blocking, overcrowding, jumping over the barrier and vandalism.
The evaluations, validation and demonstrations were conducted using both live and recorded data. For the validation task, the system was tested using four input channels in parallel, the four channels being composed of three recorded sequences and one live input stream from the main hall of the Sagrada Familia Metro station. The validation was conducted over four hours which, over the four channels, gave a combined total of 16 hours of validation. The three recorded data sequences were constructed using shorter sequences of the required behaviours, created using actors. The accuracy of the report was also measured as percentage of the ground-truth behaviour over which the correct report was generated by ADVISOR.
In total, out of 21 fighting incidents in all the Demonstrator sequences, 20 alarms were correctly generated, giving a very good detection rate of 95%. These twenty correctly identified alarms had an average report accuracy of 68%. Out of nine blocking incidents, seven alarms were generated, giving a detection rate of 78%. These seven alarms were found to be 60% accurate on average. Out of 42 instances of jumping over the barrier, including repeated incidents, the behaviour was detected 37 times, giving a success rate of 88%. The two sequences of vandalism were always detected with an overall accuracy of 71%, over six instances of vandalism. Finally, the two overcrowding alarms in camera C11 were consistently detected, with an overall accuracy of 80% over 7 separate instances of the alarms. The overcrowding alarms were also consistently detected in the live camera C10, with some 28 separate events being detected. In conclusion, the algorithms responded very successfully to the input data, with high detection rates and with all the reports being above approximately 70% accurate. No false alarms were generated during the playback of the recorded sequences although one false blocking alarm was generated in the live input.
An independent validation of the behaviour algorithms was performed by INRIA, outside of the ADVISOR system, and not in multi-channel real-time. The following results were obtained. Out of 17 blocking incidents, 16 incidents were correctly identified, giving a success rate of 94%. The fighting behaviour was detected in 23 out of 27 incidents, giving a detection rate of 85%. Overcrowding was detected in the two incidents of that behaviour and vandalism was detected all three times. However, one of the vandalism incidents also produced a false fighting alarm. Finally, all three jumping over the barrier sequences were detected.
The archive search and retrieval functionality was found to work as specified, through the controls provided on the HCI.
In conclusion, the ADVISOR system meets the requirements of the Demonstrator as laid out in the functional specification document. The system was easily able to cope with live input, although the threshold for overcrowding may need to be adjusted. The system works very well on the constructed sequences, but the lack of real data or real fighting events during the validation makes it difficult to assess the ADVISOR system's ability with these events. Finally, the performance of the ADVISOR Demonstrator in Barcelona, showed that it has a lot of potential to improve the security and safety of passengers on a Metro system. Furthermore, the system could have a wide-range of applications to detect and respond to human behaviours in many different settings.
Download the full report (668KB, Adobe Acrobat format).
Issue 5, 24 June 2003 |