Participants: Francois Bremond, Etienne Corvee, Jose Luis Patino Vilchis, Monique Thonnat.
The CARETAKER (Content Analysis and REtrieval Technologies to Apply Extraction to massive Recording) project aims at studying, developing and assessing multimedia knowledge-based content analysis, knowledge extraction components and metadata management sub-systems in the context of automated situation awareness, diagnosis and decision support. More precisely, CARETAKER will focus on the extraction of structured knowledge from large multimedia collections recorded over networks of cameras and microphones deployed in real sites. The produced audio-visual streams, if stored and automatically analyzed, represent a useful source of information in urban/environment planning, resource optimization, disabled/elderly person monitoring, etc., in addition to surveillance and safety issues.
We have considered two types of content knowledge: a first layer of primitive events that can be extracted from the raw data streams, such as ambient sounds, the degree of crowding present in the scene and the routes taken by individual people. A second layer of higher semantic events is defined based on more complex relationships between the primitive events and detected from longer term analysis.
Real testbed sites inside the metro of Roma and Torino, involving more than 30 sensors each (20 cameras and 10 microphones), has been provided. Additionally, the identification of the real user needs and beneficial use-case scenarios has served as a reference point for the correct framing of the semantic description scheme i.e. the ontology, the knowledge extraction components and the interface and demonstrator optimization.
|
Within CARETAKER, the Orion project team is in charge of the long term tracking of objects of interest, the ontology-based event detection and of knowledge discovery.
Ontology
An ontology is the set of all concepts and relations between concepts shared by the community of a given domain. The ontology is particularly useful for experts of the application domain to use scene understanding systems in an autonomous way. In ORION, we have extended this year our video ontology toward two directions: (1) the ontology has been adapted for urban monitoring applications; (2) the video ontology has been combined with an ontology for audio events, to enrich the description of scenarios of interest.
Tracking
Tracking is one of the most studied topics in dynamic scene analysis. In ORION, as well as with other partners, novel algorithms available at the consortium have first been adapted to the specific environment of metro sites, and properly evaluated for the task. All the available context information is employed (e.g. rough calibration of the ground plane). In particular, one aim of CARETAKER is to increase robustness with respect to knowledge discovery: the detection and tracking of people and activity recognition should be performed on a long-term basis (more than one month).
Event recognition and knowledge discovery
The first objective in knowledge modeling and event recognition consists in developing new algorithms for the recognition of higher-level/composite events defined in the ontology from the user requirements, using the low-level primitive streams-of-data events coming from diverse sources of information (trajectories, audio/video activities), ontology-driven methods (i.e. scenario-based), or a mixture of both. For example, we are able to define and recognize when a luggage is being abandoned by a person based on two constraints: when a luggage remains in a predefined zone for a minimum predefined period of time and when it is located far enough to any passing persons, then this luggage is considered as 'abandoned luggage' as shown in figure 1. In this figure, the luggage labeled 'LUGGAGE 10' has been left apart by the person labeled 'PERSON 11' for a too long period of time and both person 11 and person 8 are located far enough from this luggage for it to be detected as abandoned.
Figure 3. A jumping over barrier event detected in Roma subway, with the video. 
The second objective in knowledge modeling and event recognition is the investigation of unsupervised techniques for the recognition of both common events and unusual events. In all cases, the overall goal is to produce indexing information to feed the user system, for both the online supervision system and the offline retrieval one. Broadly speaking, different types of events exist, which can be characterized by their occurrence frequency, their expectedness, and their relevance. Their recognition poses different challenges that have been addressed using different models and methodologies. In ORION, the first results of primitive video events and object-of-interest trajectories are being analyzed using data mining tools. For instance, trajectories in a scene have been clustered into categories according to their entering and exiting a zone in the scene.
Figure 3. Clustering of trajectories into 21 groups to match people activities in Torino subway (45min), 2052 trajectories. 
Figure 4. Analysis of the Hall zone: there is an increase of people after 6:45 . 
For more information, see the CARETAKER Project Web-Site