Facial Expression Recognition for Alzheimer's disease patients

Recognizing expressions in severely demented Alzheimer's disease (AD) patients is essential, since such patients have lost a substantial amount of their cognitive capacity, and some even their verbal communication ability (e.g., aphasia).  This leaves patients dependent on clinical staff to assess their verbal and non-verbal language, in order to communicate important messages, as of the discomfort associated to potential complications of the AD.  Such assessment classically requires the patients' presence in a clinic, and time consuming examination involving medical personnel.  Thus, expression monitoring is costly and logistically inconvenient for patients and clinical staff, which hinders among others large-scale monitoring.
We have developed approaches for automated recognition of facial activities and expressions of severely demented patients, where we distinguish between four activity and expression states, namely talking, singing, neutral and smiling.
The approaches cater to the challenging setting of current medical recordings of music-therapy sessions, which include continuous pose variations, occlusions, camera-movements, camera-artifacts, as well as changing illumination.  Additionally and importantly, the (elderly) patients exhibit generally less profound facial activities and expressions in a range of intensities and predominantly occurring in combinations (e.g., talking and smiling). 



One developed approach (see above) is based on the extension of the Improved Fisher Vectors (IFV) for videos, representing a video-sequence using both, local, as well as the related spatio-temporal features.  We test our algorithm on a dataset of over 229 video sequences, acquired from 10 AD patients, with promising results, which have sparked substantial interest in the medical community.  The proposed approach can play a key role in assessment of different therapy treatments, as well as in remote large-scale healthcare-frameworks.

Related Scientific Papers

• Dantcheva, Antitza; Bilinski, Piotr; Nguyen, Thanh; Broutart, Jean Claude; Bremond, Francois Expression Recognition for Severely Demented Patients in Music Reminiscence - Therapy EUSIPCO'17, 25th European Signal Processing Conference, August 28 - September 2, 2017, Kos island, Greece.

• Dantcheva, Antitza; Bilinski, Piotr; Broutart, Jean Claude; Robert, Philippe; Bremond, Francois Emotion facial recognition by the means of automatic video analysis Gerontechnology, vol. 15, pp. 12s, September 2016.

Impact of Makeup on Automated Face Analysis Systems

This research deals with the impact of facial cosmetics on automated biometric systems:

• analysis of the impact, 
• makeup detection,
• design of algorithms that reduce this impact.

Facial makeup has the ability to alter the appearance of a person.  We established the negative impact of such alteration on the matching accuracy of automated face recognition systems (see Makeup datasets and Baseline Results), as well as automated gender and age estimation algorithms.
Moreover, we designed a method to automatically detect the presence of makeup in face images, based on shape, texture and color characteristics of a face image.
Further, we presented an adaptive pre-processing scheme that exploits knowledge of the presence or absence of facial makeup to improve the matching accuracy of a face matcher.

Makeup for Spoofing

We received the Best Paper Award (Runner Up) at the 3rd IEEE International Conference on Identity, Security and Behavior Analysis (ISBA 2017) for our work on "Spoofing Faces Using Makeup: An Investigative Study", co-authored by Cunjian Chen (Canon, China), Thomas Swearingen (Michigan State University, USA) and Arun Ross (Michigan State University, USA). In this paper we study the vulnerability of face recognition systems to makeup - where an adversary applies facial cosmetics to look like someone else. This process, known as spoofing, can undermine the integrity of a face recognition system. We systematically evaluate this vulnerability and suggest ways in which it can be counteracted.
In addition, I won the Tabula Rasa Spoofing Attack Award, which was featured in the news.

Makeup to enhance facial aesthetics

Makeup for Age Alteration

 

Related Scientific Papers

• C. Chen, A. Dantcheva, T. Swearingen, A. Ross, "Spoofing Faces Using Makeup: An Investigative Study," Proc. of 3rd IEEE International Conference on Identity, Security and Behavior Analysis (ISBA), (New Delhi, India), February 2017.
• C. Chen, A. Dantcheva, A. Ross, "An Ensemble of Patch-based Subspaces for Makeup-Robust Face Recognition," Information Fusion, vol 32, pp. 80-92, November 2016. 
• C. Chen, A. Dantcheva, A. Ross, "Impact of facial cosmetics on automatic gender and age estimation algorithms," Proc. of 9th International Conference on Computer Vision Theory and Applications (VISAPP), (Lisbon, Portugal), January 2014.
• C. Chen, A. Dantcheva, A. Ross, "Automatic Facial Makeup Detection with Application in Face Recognition," Proc. of 6th IAPR International Conference on Biometrics (ICB), (Madrid, Spain), June 2013.
• A. Dantcheva, C. Chen, A. Ross, "Can Facial Cosmetics Affect the Matching Accuracy of Face Recognition Systems?," Proc. of 5th IEEE International Conference on Biometrics: Theory, Applications and Systems (BTAS), (Washington DC, USA), September 2012.
• A. Dantcheva, C. Chen, A. Ross, "Makeup challenges automated face recognition systems," SPIE Newsroom 2013, Defense and Security. DOI: 10.1117/1.1201303.004795

Soft Biometrics

My Ph.D. work presented the concept of soft biometric systems (SBSs), as systems that employ weak semantic human traits towards human (re-)identification, database search pruning and facial aesthetics prediction. Such traits include gender, age, presence of glasses, hair and skin color. SBSs inherit the non intrusiveness and computational efficiency, which allow for fast, enrolment-free and pose-invariant biometric analysis, even in the absence of consent and cooperation of the surveillance subject.