東華大學圖書館 |

A Synthetic FACS Framework for Expanding Facial Expression Lexicons.

紀錄類型:	書目-電子資源 : Monograph/item
正題名/作者:	A Synthetic FACS Framework for Expanding Facial Expression Lexicons./
作者:	Butler, Crystal.
出版者:	Ann Arbor : ProQuest Dissertations & Theses, : 2021,
面頁冊數:	125 p.
附註:	Source: Dissertations Abstracts International, Volume: 82-10, Section: B.
Contained By:	Dissertations Abstracts International82-10B.
標題:	Computer science. -
電子資源:	https://pqdd.sinica.edu.tw/twdaoapp/servlet/advanced?query=28157617
ISBN:	9798597014760

A Synthetic FACS Framework for Expanding Facial Expression Lexicons.
Butler, Crystal.

A Synthetic FACS Framework for Expanding Facial Expression Lexicons. - Ann Arbor : ProQuest Dissertations & Theses, 2021 - 125 p.

Source: Dissertations Abstracts International, Volume: 82-10, Section: B.

Thesis (Ph.D.)--New York University Tandon School of Engineering, 2021.

This item must not be sold to any third party vendors.

Facial expressions are a key modality of nonverbal communication. However, no comprehensive reference mapping the kinematics of facial expression (FE) production to visual changes in appearance and the semantics of those changes exists. Production and recognition of a small set of 6-7 basic expressions of emotion have been thoroughly explored in the psychology literature, namely:• surprise• fear• happy• sadness• anger• disgust• contempt (optionally)Some less widely accepted additions such as pain and compound basic emotions have also been described. Defining a broader range of more nuanced, communicative FEs is important to inform applications such as healthcare diagnostics and treatment, automated facial expression recognition, animation and assistive systems that incorporate affective, intelligent virtual agents.This research project builds and validates a framework that can be leveraged to create more extensive FE lexicons. It begins with modeling FEs on a custom 3D avatar, using the Facial Action Coding System (FACS) as the foundation for synthesizing facial behavior. FACS describes individual muscle movements in the face as action units (AUs), which can be variably activated and combined to generate all behaviors that the human face can exhibit. The FACS-valid avatar can be controlled computationally to produce a nearly infinite number of weighted AU combinations. A key benefit of using synthetic FACS rather than expressions posed by human actors is that movements can be precisely parameterized and reliably replicated across diverse physiognomies. The avatar and scripts for generatively modeling facial behavior are referred to as the MiFace component of the framework.Even if AU weights are fixed and a ceiling of 12 simultaneously activated AUs is set, the number of possible combinations of 38 AUs (24 primary AUs plus their unilateral variants, where applicable)that the avatar can model exceeds four billion. Therefore, to test the lexicon construction framework, I generatively modeled two subsets of candidate FEs as the stimulus sets for proof-of concept observer judgement studies. In those studies, crowdsourced participants voted on whether each FE conveys interpretable nonverbal information, then assigned freely chosen labels to those that were deemed to be communicative. Employing crowd workers to map FEs to freely chosen natural language labels allows for more ecologically valid, fine-grained descriptions of perceived FE signal values.To analyze the resulting label sets, the framework incorporates a novel human-in-the-loop artificial intelligence pipeline to perform natural language processing (NLP). The pipeline comprises two key components: word embeddings fine-tuned to perform optimally on FE word semantic similarity scoring, and a hierarchical agglomerative clustering step that tests for adequate set coherence. The word embeddings provide real-valued scores for semantic similarity between all labels in a set; these scores act as inputs to the clustering step, and for coherent label sets, provide the basis for calculating the representative label (the set medoid). In order to evaluate word embedding performance, a set of FE word pairs with gold standard synonymy scores was developed. Scores were calculated by aggregating values chosen by a group of human raters, following a procedure similar to that used in making the SimLex-999 set of scored synonyms [51]. The synonymy-scored FE word pair benchmark is referred to as the FreeRes dataset. The NLP pipeline tuned for performance in scoring FE word pair synonymy is FreeRes-NLP.By applying FreeRes-NLP to label sets gathered in the two FE labeling observer judgement studies mentioned previously, an example FE lexicon has been produced. It is made freely available, along with other key products of the project. Chapters two through seven of this publication describe project components that have been shared for use by the broader research community, specifically:1. MiFace, an open source 3D model human capable of synthesizing FACS-validated facial muscle movements. The Maya avatar model created for testing the FE lexicon framework is made freely available for other researchers to use or modify. There are no other models available, paid or otherwise, that provide control of accurately represented individual AUs. TheMaya Embedded Language scripts used to generatively model candidate FEs are included in MiFace. Creation of the model and the method used to generate two test sets of candidate FEs are described in Chapter 2.2. Synthetic FE images with full crowdsourced label sets for each. FE images that were considered communicative are available with complete lists of associated single word labels, as given by naive observers. Label sets contain about 40 words each. The design and methods of the two observer judgement studies carried out to gather the label sets are described in Chapter 3.3. FreeRes synonym set, a set of FE word pairs with gold standard synonymy scores assigned by human raters. 156 pairs of FE words were scored for synonymy by 10 crowdsourced raters each. The averaged results are provided for use as a gold standard measure ofNLP model performance on tasks that require a real value representation of semantic similarity between FE terms. Chapter 4 describes the design and methods for the FE word synonymy study.4. An open word embeddings set fine-tuned for optimal performance on FE word pair semantic similarity scoring tasks. To understand the degree of agreement within an FE label set, a quantitative measure of synonymy between the constituents is required. Calculating the semantic similarity, or synonymy, between words is a specialized problem in NLP. Vector space models, a commonly used family of language representations that include word embeddings, are typically capable of distinguishing words based on a broader set of relatedness relationships; semantic similarity is only one of many possible dimensions. I tested a broad range of off-the-shelf and fine-tuned vector space models, including contextual embeddings distilled from a recently developed state-of-the-art transformer model. Based on testing with three scored synonym sets, word embeddings tuned on a custom-built dictionary and thesaurus corpus performed best. Details of model testing and training are given in Chapter5.5. FreeRes-NLP, an open-source NLP pipeline for determining the representative FE label from a set. To establish highly descriptive, ecologically valid FE semantics, it is desirable to have a group of untrained observers freely provide natural language labels that describe their perceptions of FEs. However, no standard procedure for analyzing label sets from free-response studies currently exists. The pipeline developed for this project implements a human-in-the-loop process for programmatically assessing the coherence of a label set using hierarchical agglomerative clustering, then selects the set member that best represents the overall set semantics, i.e., the set medoid. Example output is demonstrated in the Figure 6.1dendrogram. The pipeline incorporates calculations based on the word embeddings described in Chapter 4. Label clustering, the set coherence test and comparisons to manual analyses performed in prior free response studies are provided in Chapter 6.6. An example FE lexicon, containing the communicative FEs from two crowdsourced labeling studies. Two test sets of FE images were generated by fixing the parameters (i.e., activation weights) of a subset of AUs and generating all variants for a specified AU count on the avatar. The images went through a two-stage process in which they are first assigned binary labels denoting whether or not the FE conveys a meaningful signal by crowd workers, as described in Chapter 1. For FEs that passed the first stage, free-response labels were assigned by crowd workers and processed using the pipeline described in Chapter 5.

ISBN: 9798597014760Subjects--Topical Terms:

523869
Computer science.
Subjects--Index Terms:

Computer graphics

A Synthetic FACS Framework for Expanding Facial Expression Lexicons.
LDR:09512nmm a2200421 4500 001 2283746
005 20211115071639.5
008 220723s2021 ||||||||||||||||| ||eng d
020 $a 9798597014760
035 $a (MiAaPQ)AAI28157617
035 $a AAI28157617
040 $a MiAaPQ $c MiAaPQ
100 1 $a Butler, Crystal. $3 3558392
245 1 0 $a A Synthetic FACS Framework for Expanding Facial Expression Lexicons.
260 1 $a Ann Arbor : $b ProQuest Dissertations & Theses, $c 2021
300 $a 125 p.
500 $a Source: Dissertations Abstracts International, Volume: 82-10, Section: B.
500 $a Advisor: Perlin, Kenneth.
502 $a Thesis (Ph.D.)--New York University Tandon School of Engineering, 2021.
506 $a This item must not be sold to any third party vendors.
520 $a Facial expressions are a key modality of nonverbal communication. However, no comprehensive reference mapping the kinematics of facial expression (FE) production to visual changes in appearance and the semantics of those changes exists. Production and recognition of a small set of 6-7 basic expressions of emotion have been thoroughly explored in the psychology literature, namely:• surprise• fear• happy• sadness• anger• disgust• contempt (optionally)Some less widely accepted additions such as pain and compound basic emotions have also been described. Defining a broader range of more nuanced, communicative FEs is important to inform applications such as healthcare diagnostics and treatment, automated facial expression recognition, animation and assistive systems that incorporate affective, intelligent virtual agents.This research project builds and validates a framework that can be leveraged to create more extensive FE lexicons. It begins with modeling FEs on a custom 3D avatar, using the Facial Action Coding System (FACS) as the foundation for synthesizing facial behavior. FACS describes individual muscle movements in the face as action units (AUs), which can be variably activated and combined to generate all behaviors that the human face can exhibit. The FACS-valid avatar can be controlled computationally to produce a nearly infinite number of weighted AU combinations. A key benefit of using synthetic FACS rather than expressions posed by human actors is that movements can be precisely parameterized and reliably replicated across diverse physiognomies. The avatar and scripts for generatively modeling facial behavior are referred to as the MiFace component of the framework.Even if AU weights are fixed and a ceiling of 12 simultaneously activated AUs is set, the number of possible combinations of 38 AUs (24 primary AUs plus their unilateral variants, where applicable)that the avatar can model exceeds four billion. Therefore, to test the lexicon construction framework, I generatively modeled two subsets of candidate FEs as the stimulus sets for proof-of concept observer judgement studies. In those studies, crowdsourced participants voted on whether each FE conveys interpretable nonverbal information, then assigned freely chosen labels to those that were deemed to be communicative. Employing crowd workers to map FEs to freely chosen natural language labels allows for more ecologically valid, fine-grained descriptions of perceived FE signal values.To analyze the resulting label sets, the framework incorporates a novel human-in-the-loop artificial intelligence pipeline to perform natural language processing (NLP). The pipeline comprises two key components: word embeddings fine-tuned to perform optimally on FE word semantic similarity scoring, and a hierarchical agglomerative clustering step that tests for adequate set coherence. The word embeddings provide real-valued scores for semantic similarity between all labels in a set; these scores act as inputs to the clustering step, and for coherent label sets, provide the basis for calculating the representative label (the set medoid). In order to evaluate word embedding performance, a set of FE word pairs with gold standard synonymy scores was developed. Scores were calculated by aggregating values chosen by a group of human raters, following a procedure similar to that used in making the SimLex-999 set of scored synonyms [51]. The synonymy-scored FE word pair benchmark is referred to as the FreeRes dataset. The NLP pipeline tuned for performance in scoring FE word pair synonymy is FreeRes-NLP.By applying FreeRes-NLP to label sets gathered in the two FE labeling observer judgement studies mentioned previously, an example FE lexicon has been produced. It is made freely available, along with other key products of the project. Chapters two through seven of this publication describe project components that have been shared for use by the broader research community, specifically:1. MiFace, an open source 3D model human capable of synthesizing FACS-validated facial muscle movements. The Maya avatar model created for testing the FE lexicon framework is made freely available for other researchers to use or modify. There are no other models available, paid or otherwise, that provide control of accurately represented individual AUs. TheMaya Embedded Language scripts used to generatively model candidate FEs are included in MiFace. Creation of the model and the method used to generate two test sets of candidate FEs are described in Chapter 2.2. Synthetic FE images with full crowdsourced label sets for each. FE images that were considered communicative are available with complete lists of associated single word labels, as given by naive observers. Label sets contain about 40 words each. The design and methods of the two observer judgement studies carried out to gather the label sets are described in Chapter 3.3. FreeRes synonym set, a set of FE word pairs with gold standard synonymy scores assigned by human raters. 156 pairs of FE words were scored for synonymy by 10 crowdsourced raters each. The averaged results are provided for use as a gold standard measure ofNLP model performance on tasks that require a real value representation of semantic similarity between FE terms. Chapter 4 describes the design and methods for the FE word synonymy study.4. An open word embeddings set fine-tuned for optimal performance on FE word pair semantic similarity scoring tasks. To understand the degree of agreement within an FE label set, a quantitative measure of synonymy between the constituents is required. Calculating the semantic similarity, or synonymy, between words is a specialized problem in NLP. Vector space models, a commonly used family of language representations that include word embeddings, are typically capable of distinguishing words based on a broader set of relatedness relationships; semantic similarity is only one of many possible dimensions. I tested a broad range of off-the-shelf and fine-tuned vector space models, including contextual embeddings distilled from a recently developed state-of-the-art transformer model. Based on testing with three scored synonym sets, word embeddings tuned on a custom-built dictionary and thesaurus corpus performed best. Details of model testing and training are given in Chapter5.5. FreeRes-NLP, an open-source NLP pipeline for determining the representative FE label from a set. To establish highly descriptive, ecologically valid FE semantics, it is desirable to have a group of untrained observers freely provide natural language labels that describe their perceptions of FEs. However, no standard procedure for analyzing label sets from free-response studies currently exists. The pipeline developed for this project implements a human-in-the-loop process for programmatically assessing the coherence of a label set using hierarchical agglomerative clustering, then selects the set member that best represents the overall set semantics, i.e., the set medoid. Example output is demonstrated in the Figure 6.1dendrogram. The pipeline incorporates calculations based on the word embeddings described in Chapter 4. Label clustering, the set coherence test and comparisons to manual analyses performed in prior free response studies are provided in Chapter 6.6. An example FE lexicon, containing the communicative FEs from two crowdsourced labeling studies. Two test sets of FE images were generated by fixing the parameters (i.e., activation weights) of a subset of AUs and generating all variants for a specified AU count on the avatar. The images went through a two-stage process in which they are first assigned binary labels denoting whether or not the FE conveys a meaningful signal by crowd workers, as described in Chapter 1. For FEs that passed the first stage, free-response labels were assigned by crowd workers and processed using the pipeline described in Chapter 5.
520 $a Results of the data analysis are provided in Chapter 7 in the form of an FE lexicon, with FE images mapped to descriptive single word labels and weighted AU combinations. An example lexicon entry is shown in Figure 7.1.
590 $a School code: 1988.
650 4 $a Computer science. $3 523869
650 4 $a Psychology. $3 519075
650 4 $a Linguistics. $3 524476
653 $a Computer graphics
653 $a Emotion
653 $a Facial Expression
653 $a Generative design
653 $a Natural language processing
653 $a Social signal processing
653 $a Facial Action Coding System
653 $a Action Units
690 $a 0984
690 $a 0621
690 $a 0290
710 2 $a New York University Tandon School of Engineering. $b Computer Science and Engineering. $3 3428054
773 0 $t Dissertations Abstracts International $g 82-10B.
790 $a 1988
791 $a Ph.D.
792 $a 2021
793 $a English
856 4 0 $u https://pqdd.sinica.edu.tw/twdaoapp/servlet/advanced?query=28157617