Andreas Maroulis
VicarVision, Amsterdam, the Netherlands, andreas@vicarvision.nl Intro
Baby FaceReader (BFR) is a novel computer vision solution used to automatically code facial expressions in infants based on the Noldus FaceReader [1] Using the Viola-Jones algorithm [2], BFR detects an infant face in a video or an image, models the detected face using a 3D Active Appearance Model (AAM) [3], and then uses neural network classifiers to determine facial Action Unit (AU) activation based on Oster’s Baby Facial Action Coding System (Baby FACS) [4]. BFR applies a 3D AAM on 2D images / videos that is more robust to movement than 2D AAMs. The 3D AAM is design by mapping 3D models of faces on the equivalent 2D models of the same faces, thus backpropagading a function that can infer a 3D model for any new 2D image of face. While Baby FACS AUs objectively quantify individual movements in the face and seemingly any combinations of those movements, it has 2 main limitations:
1. Studying co-occurrence of individual action units to determine holistic global expressions remains a tedious, time-consuming tasks
2. Understanding the affective and cognitive meaning of each action unit or co-occurrence of action units is not within the scope of the coding manual
As a result, researchers often code facial expressions in more holistic terms such as positive, negative, pain, smiles and cry faces [5, 6, 7]. While these studies show high reliability scores in coding of these facial expressions, they do not investigate the morphological differences that differentiate one facial expression from another (i.e. what makes a negative facial expression negative, and is that different from a pain or cry expression?). For such a facial expressions codes to be truly reliable they must be expressed in objective terms, i.e. using combinations of AUs to define the morphology of each defined expression. This study will build on previous work [8] and will have 2 main goals:
1. To improve BFR’s AU classifiers. In [8] BFR exhibited a 0.60 agreement score against manually annotated images. The gold standard for an agreement score in facial action coding is 0.70 [9].
2. To use the improved BFR classifiers to objectively define a variety of facial expressions such as positive, negative, pain, smiles and cry faces in terms of AUs.
Method
To evaluate the BFR’s improved performance we will follow the same procedure as [8]. Using the Baby FACS dataset [4] of 74 images. Contrary to FACS, Baby FACS defines AU3 as Brow Knitting (the movement caused by a contraction of the corrugator supercili muscle that causes the eyebrows to move towards each other), and AU4 as Brown Knotting (the movement of the procerus muscle that causes the eyebrows to lower). The adult FACS manual collapses those two movements into one (AU4). Similarly, we will collapse AUs 3 and 4 into one category (3+4) to fit the current framework of FaceReader. We will do the same of AUs 26 and 27 (26+27). Finally, we will evaluate BFR AU classification results for AUs 1, 3+4, 5, 6, 7, 9, 10, 12, 15, 17, 18, 20, 25, 26+27.
To investigate the AU characteristic of holistic facial expressions we will run BFR’s improved classifiers on 3 different datasets:
The City Infant Faces Dataset [5]: A collection of 154 images of infants (age: 0 - 12 months) annotated as positive, negative and neutral facial expressions.
The COPE (Classification of Pain Expressions) Database [6]. A collection of 204 facial images of neonates annotated as rest, cry, air stimulus, friction and pain expressions.
Messinger Databrary Dataset [7]. A collection of 10 videos of 6-month olds in a Face-to-Face-Still-Face Paradigm coded for AUs 6+12 (smile faces), and 6+20 (cry faces)
We will evaluate AU activation for all the aforementioned images / videos in the datasets and provide a suggested AU coding scheme for coding each expression.
Results
Preliminary results are the following:
Using BFR, we managed to model 72% of the face in the City Infant FaceDataset. This created a dataset of model infant faces with 43 Positive facial expressions, 34 Negative Facial Expressions, and 32 Neutral Facial expressions.
Figure 1 shows box plots of the AU intensity for each of the 3 categories of expressions. Activations of of AUs 6,
156
R.A Grant et al. (Eds.): Measuring Behavior 2018, ISBN 978-1-910029-39-8 Manchester, UK, 5th-8th June 2018
12, and 25 are often associated with positive facial expressions. Any intensity of AUs 25 can also be associated to negative expressions. Any intensity of AU43 is also most often associated to negative facial expressions. Low to medium intensities of AUs 1, 4, 6, 7, 10 are also associated to negative facial expressions. Finally, low to medium intensities of AUs 1, 5, 17, 25, and 26 are also associated to neutral facial expressions. It should be noted that Baby FaceReader has reported F1 scores of at least 0.60 in most of the aforementioned AUs [8].
We also executed a BFR analysis on the COPE infant databases. Due to occlusions on most of the infant faces from hats, blankets and hands, as well as the very young age of the infants in the dataset (3-6 weeks), the current version of BFR could not model this dataset.
Finally, a preliminary analysis of the Messinger Dataset showed that BFR could model at least 70% the faces in video frames of the dataset. Detailed AU classification will be presented during MB2018.
Furthermore, improvement of BFR’s AU classifiers is ongoing and final results will presented at MB2018, together with an updated AU classification of the CITY infant faces databases. We do expect to reach an average agreement score of at least 0.7. We will also train the BFR AAM on younger infant faces with some occlusions to account for the difficulties presented in the COPE infant dataset. We expect our updated AAM and classifiers to indicate expressions categories that may overlap (e.g. cry, negative, pain) and will thus suggest more objective classification categories based on AU activation.
Figure 1: AU intensity box plots for Positive, Negative, and Neutral Facial Expressions of City Infant Faces Databases
Discussion
While coding infant expressions in terms of AUs may be objective measure to quantify facial expressions, creating categories of facial expressions objectively defined in terms of AUs will lead to faster coding procedure of facial
157
R.A Grant et al. (Eds.): Measuring Behavior 2018, ISBN 978-1-910029-39-8 Manchester, UK, 5th-8th June 2018
expressions that is easier to understand. Similarly, Ekman and Friesen have created EMFACS [10], a simplified version of the FACS manual [9] to quickly code emotional facial expressions for adults.
Acknowledgements
The research received funding from the Marie Sklodowska-Curie Actions of the Horizon 2020 Framework Program H2020-MSCA-ITN-2015. The program name is “Brainview” and the REA grant agreement n°A140857.
We would also like to thank Rebecca Webb, Susan Brahnam, and Daniel Messinger for permission to use the City Infant, COPE, and Messinger Databrary Dataset respectively for such research.
References:
1. Noldus (2016). FaceReader: Tool for automatic analysis of facial expression: Version 7.0. Wageningen, the Netherlands: Noldus Information Technology B.V.
2. Viola, P., & Jones, M. J. (2004). Robust real-time face detection. International journal of computer vision, 57(2), 137-154. Chicago.
3. Cootes, T. F., & Taylor, C. J. (2004). Statistical models of appearance for computer vision. Imaging Science and Biomedical Engineering (University of Manchester, Manchester, UK).
4. Oster, H. (2016). Baby FACS: Facial Action Coding System for infants and young children. Unpublished monograph and coding manual. New York University.
5. Webb, R., Ayers, S., & Endress, A. (2017). The City Infant Faces Database: A validated set of infant facial expressions. Behavior research methods, 1-9.
6. Nanni, L., Lumini, A., & Brahnam, S. (2010). Local binary patterns variants as texture descriptors for medical image analysis. Artificial intelligence in medicine, 49(2), 117-125.
7. Messinger, D. (2014). Facial expressions in 6-month old infants and their parents in the still face paradigm and attachment at 15 months in the Strange Situation. Databrary. Retrieved February 7, 2018 from http://doi.org/10.17910/B7059D
8. Maroulis, A. Spink, A.J., Theuws, J.J.M., Oster, H., Buitelaar, J. (2017). Sweet or Sour: Validating Baby FaceReader to Analyse Infant Responses to Food. Poster to be presented in 12th Pangborn Sensory Science Symposium, 20-24 August 2017, Providence, Rhode Island
9. Ekman, P., Friesen, W. V., & Hager, J. C. (2002). Facial action coding system: The manual on CD-ROM.
Instructor’s Guide. Network Information Research Co, Salt Lake City.
Friesen, W. V., & Ekman, P. (1983). EMFACS-7: Emotional facial action coding system. Unpublished manuscript, University of California at San Francisco, 2(36), 1.
158
R.A Grant et al. (Eds.): Measuring Behavior 2018, ISBN 978-1-910029-39-8 Manchester, UK, 5th-8th June 2018