Emotional Descriptors

Early works on automatic emotion recognition systems proposed the use of descrip-tors [62, 90, 230] to model the expression, and mostly stochastic [241] or rule-based classifiers [192] to label it. The contributions of these works are on representing emotion expressions, and most of them were based on Ekman’s assumption of universal emotions.

Most of the early works dealt with two separately streams: face expressions and speech. These works applied different feature descriptors in a way to describe an expression. Most of these feature descriptors were based on general image descriptors, when applied to face expressions, or general sound descriptors, when applied to speech. A review of the field from Chellappa et al. [44] exhibits a broad range of different methods used to recognize faces and shows that most works were still at an early stage and not close to being used in real-world scenarios.

In early research, some common methods for face expression recognition involve the use of motion descriptors [152, 89] and template matching [20]. All these works were computationally expensive, not able to deal with real-time processing, and had problems with generalization. For example, if the pictures were captured under different lighting condition or from different persons, these models did not show good performance.

Some of the works for face expression, however, were inspired by the psycho-logical studies of Ekman and Friesen and the FACS system was used as inspiration for many computational models [90, 191, 51]. These models use mostly template-matching approaches to identify the Action Units (AUs) in a person’s face and use simple rule-based classifiers [51] or even simple neural networks [257] to identify the expressions.

The works using the FACS models solved a big problem in the field: they could recognize expressions with a fair degree of generalization based on the evidence of Ekman’s research. However, these models faced a new problem: they had to identify and describe, perfectly, the AUs, otherwise the classification would not work due to the fact that similar AUs are involved in completely different emotions [191].

Several approaches proposed better ways to map the AUs, including the tempo-ral analysis of the face movement [224], geometric deformation analysis [167], and profile contour analysis [225]. These methods are strongly dependent on the AUs to describe expressions and were able to identify dynamic and static ones. Cowie

et al. [57] exhibit different solutions for identifying and describing the AUs, and discuss how effective they are for generalizing the expression for different persons.

The use of explicit descriptors, mostly related to FACS, introduced an increase of studies for automatic emotion recognition systems. However, the FACS model, and models based on common explicit descriptors show a big problem: it is difficult to represent spontaneous expressions with them [86, 307]. The purpose of FACS is to describe muscle movements and classify them into emotional concepts. How-ever, emotion expressions are spontaneous, and different persons will express the same emotional concept in different ways. The works based on FACS and explicit descriptors cannot deal with this nature, and are mostly used for basic emotion recognition, such as the six universal emotions [290].

In the past decade, the introduction of Convolutional Neural Networks (CNNs), among other deep learning models, provided an important evolution on image recognition tasks. Such models use the concept of implicit feature descriptors to classify complex images. Instead of using a set of pre-defined descriptors, CNNs learn a particular descriptor which will perform best on the classification task which it was applied. This was shown to be efficient in several different image recognition tasks [171, 151, 146].

Given the success of CNNs in several different tasks, it was largely applied for emotion recognition tasks [94, 209, 149], showing an improvement on general-ization. Different architectures were applied for different tasks. Fasel et al. [94]

propose an architecture which is head-pose invariant. Their model evaluates a CNN trained with different expressions presented with different head-poses and this model was able to generalize the learned features for different subjects.

In the approach of Matsugu et al. [209], they use a rule-based mechanism in between the convolution layers to improve the generalization of the detected features. This rule-based mechanism identifies if the learned features are related to different face structures, such as eyes or mouth, and use this information to apply a selective mask in the filters. This is used to create a face detection mechanism, firstly, and then to identify face expressions. They show that their system increases the recognition of the six basic emotions when compared to common FACS-based systems.

Karou et al. [149] use a combination of general and specific convolution chan-nels to recognize emotions in videos. Their architecture has different training steps, first based on general image descriptors, and later on specific emotion expressions.

They use a video aggregation scheme to recognize emotions in a sequence, where N-frames are aggregated in one representation by summing the frames’ own rep-resentations. They show that their network could be used in real-life emotion recognition tasks by using their model to recognize emotions in movies and series clips.

The use of CNNs to describe facial emotions showed also a big improvement for spontaneous expressions recognition [302, 186, 283]. Although using CNNs, these works rely on heavily pre-processed data [302], complex tree and rule-based descriptors identification [186] or even in different explicit feature descriptors used to complete the final facial representation [283].

3.5. Affective Computing

Although the use of CNNs to learn face expressions presented a better per-formance when compared to FACS-based systems, recent studies show a certain correlation on what the network learns and the AUs of the FACS, as presented by Khorrami et al. [154]. In their work, they discuss that the learned features of a CNN trained with facial expressions approximate some AUs, showing that the network actually learns how to detect AUs without any pre-defined rule. This shows that the FACS system could be limiting the recognition of spontaneous ex-pressions, however, describing expressions with muscular movements show to be a robust facial expression descriptor.

Similarly as facial expressions, the development of auditory emotional descrip-tors evolved in the past decades from explicit descripdescrip-tors [64] to implicit ones [1]. The FACS system does not have any auditory emotional description, which led to a wide range of different descriptors and acoustic models used in emotion recognition.

Most of the works on speech emotion recognition are based on popular auditory descriptors, which were mostly developed to describe the human voice. Early works used a combination of simple acoustic descriptors [230], such as vocal energy, pitch, speech rate, among others to identify mostly the six universal emotions. These models rely heavily on a very clean input data, mostly an expression of a word or a short sentence.

Different kinds of descriptors were developed and applied to speech recognition.

The most successful were the ones based on the Mel Frequency Cepstral Coefficients (MFCC), which proved to be suitable for speech representation [264]. MFCCs are described as the coefficients derived from the cepstral representation of an audio sequence, which converts the power spectrum of an audio clip into the Mel-scale frequency. The Mel scale was shown to be closer to human auditory system’s response than the linear frequency.

Each auditory feature descriptor carries its own information, changing the na-ture of the audio representation. For the same clip of sounds, very distinct infor-mation can be extracted for different tasks. Thus, Madsen et al. [203] use a set of three different descriptors, namely chroma features, loudness, and MFCC, to represent distinct auditory information. They also obtain different temporal/non-temporal representations for each descriptor, using sliding windows for discrete representations or Hidden Markov Models for temporal dependencies. A total of 83 feature representations is obtained. After the feature extraction, they use a hi-erarchical non-parametric Bayesian model with a Gaussian process to classify the features. They show that their approach has a certain degree of generalization, but the exhaustive search for tuning each parameter of the model for multiple feature representations and feature combinations is not a viable option.

Similar to the work of [203], several approaches [251, 147, 195] use an extensive feature representation strategy to represent the audio input: they extract several features, creating an over-sized representation to be classified. The strength of this strategy relies on redundancy. The problem is that usually, it is not clear how well each of these descriptors actually represents the data, which can lead to not capturing the essential aspects of the audio information, and decreasing the

gener-alization capability [134]. The use of over-sized auditory feature representations is the focus of heavy critics [307, 87] because of the incapability of these descriptors to describe the nuances of emotion expressions in speech.

In a similar way as facial expressions, implicit descriptors methods, mostly CNNs, were used recently to describe emotions in human speech [138, 204]. These systems use a series of pre-processing techniques to remove mostly the noise of the audio signal, and let the CNN learn how to represent the data and classify it in emotion concepts [264]. In this strategy, the CNN is able to learn how to represent the auditory information in the most efficient way for the task. Initial research was done for music [190], and speech recognition [272] and was shown to be successful in avoiding overfitting. The problem with this approach is that it needs an extensive amount of data to learn high-level audio representations, especially when applied to natural sounds, which can include speech, music, ambient sounds and sound effects [65].

Although there are models to describe systematically the emotional components in speech [56], there is no consensus on how to identify affective components in a human voice, and thus the evolution of emotional auditory descriptors is limited and much more work is necessary.

Many other models which are not based on face expressions or speech were proposed. The use of several physiological signals, coming from muscle movements, skin temperature, and electrocardiograms were investigated [236, 156, 155] and delivered good performance, however, used uncomfortable and specific sensors to obtain the signals, not suitable for be used in real-world scenarios. Mechanisms such as bracelets or localized electrodes were used.

In a similar approach, the use of Electroencephalogram (EEG) signals was used as emotion descriptors [25, 194, 197]. Using non-invasive EEG mechanisms, the brain behavior was captured when different emotion experiences were presented to a human. The initial results showed poor accuracy and generalization but open one new approach on emotion descriptors.

Another way to describe emotions is using body movement and posture. There are studies which show the relation between different body postures and movements with emotional concepts [55], and even with micro-expressions [83]. Most of the computational models in this area use the body shape [43] or common movement descriptors [40] to represent the expression and showed good accuracy and a certain level of generalization. However, the best performance used different sensors such as depth cameras or gloves to capture the movements [158].