Learning events

Incremental Learning of Events in Video using Reliable Information

Participants: Marcos David Zúñiga Barraza, Francois Bremond, Monique Thonnat.

The goal of this thesis is to propose a general video understanding approach for learning and recognition of events occurring in videos, for real world applications. This video understanding framework is composed of four tasks:
First, at each video frame, a segmentation task detects the moving regions, represented by bounding boxes enclosing them.
Second, to each moving region, a new 3D classifier associates a object class label (e.g. person, vehicle) and a 3D parallelepiped described by its width, height, length, position, orientation, and visual reliability measures of these attributes. Next images show examples of classification results. Red bounding boxes represents the classification of a moving region as a person, while the brown bounding box the classification of a moving box as a vehicle.

Parking lot video classification result. Locked chamber video classification result.

Third, a new multi-object tracking algorithm uses these object descriptions to generate tracking hypotheses about the objects evolving in the scene. Reliability measures associated to the object features are used to perform a proper selection of valuable information.
Next image shows tracking results for a road video. Blue lines represent the trajectory of the tracked centroid of the bounding box, while the red lines represent the trajectory of the centroid of the parallelepiped base.

Road video tracking result.

Finally, a new incremental event learning algorithm aggregates on-line the attributes and reliability information of the tracked objects to learn a hierarchy of concepts describing the events occurring in the scene. Reliability measures are used to focus the learning process on the most valuable information. Simultaneously, the event learning approach recognises the events associated to the objects evolving in the scene.
The next image shows an overview of the complete approach.

Overview of the video understanding approach for event learning.

The tracking approach has been validated using video-surveillance benchmark videos publicly accessible. The complete video understanding framework has been evaluated with videos for a real elderly care application. The framework has been able to successfully learn events related to trajectory (e.g. change in 3D position an velocity), posture (e.g. standing up, crouching), and object interaction (e.g. person approaching to a table), among other events, with a minimal configuration effort.