The ADVISOR system was tested in two-stages. The first stage of validation, involved playing a sequence of footage from a CCTV camera, through the system and noting the resulting behaviour report generated by ADVISOR. If the system generated the correct behaviour report then the second-stage of validation was applied which measured the accuracy of the report. This was achieved by measuring the percentage of the sequence during which the correct report was generated.
The system was tested using various sequences involving the behaviours of blocking and fighting, as described in the functional specification. Blocking corresponds to a situation when a group of at least 2 people is stopped in a predefined zone for at least 4 seconds and can potentially block the path of other people. Fighting corresponds to a situation when a group of people (at least 2 persons) are pushing, kicking or grasping each other for at least 2 seconds.
Single camera sequences were mainly used but a combined sequence using two cameras in parallel was also tested.
It was found that out of a total of 55 single camera sequences, there were 22 (40%) true results, including the true negative results, where no events were generated for sequences that had no fighting or blocking. The blocking sequences were found to be correctly reported more often than the fighting sequences. Indeed, in all four individual blocking sequences, blocking was correctly reported and in the consecutive sequence, blocking was also correctly reported each time, until the system crashed due to an overflow failure.
Fighting was reported less successfully, with 80 % of the individual sequences being correctly reported but only 15% of the consecutive sequences being correctly identified. This is due to a problem with background files, where the system is unable to reset itself to an empty scene after fighting has occurred and will be rectified in the demonstrator system.
For the sequences that were correctly reported in the first-phase of validation, the behaviour reports were found to be very accurate under the criteria of the second-phase of validation. Overall, those sequences tested in the second-phase gave an average accuracy score of 82%.
When testing ADVISOR with two cameras in combination, no behaviour reports were generated, even though there were spurious overcrowding reports generated when the sequences were run through individually. This may indicate a problem with multi-camera use.
In conclusion, the ADVISOR system meets the requirements of Test-Bed 2 as laid out in the functional specification document. The system works very well on a few individual sequences, but does not work so well on other sequences, even when the fighting is quite exaggerated. In particular, the fighting algorithms have low success rate in recognising fighting behaviour. On the other hand, the blocking algorithms seem to work very well.
Download the full report (574KB, Adobe Acrobat format).
Issue 5, 24 June 2003 |