SamanehTaghizadeh OsloMet 2023
SamanehTaghizadeh OsloMet 2023
Samaneh Taghizadeh
Autumn 2023
Benchmarking the User Experience of
Different AI Talking Head Generation
Samaneh Taghizadeh
© 2023 Samaneh Taghizadeh
http://www.oslomet.no/
Interviews of abused and maltreated children are an important and often the primary
source of information in court trials, but the children’s testimonies are too often
invalidated because they are collected in an unreliable and incorrect way. Thus,
performing such interviews must follow strict protocols, stating the questions in an
open and non-leading way, i.e., it is a sensitive and challenging task for professional
practitioners. While utilizing real cases of child abuse for training therapists might be
harmful and sophisticated, effective practice and training is essential to improve their
skills. With advancements in machine learning and generative tools in recent years,
their great potential can be exploited for educational purposes. We are researching an
AI-based avatar to mimic the behavior of maltreated children to be used for interview
training in such scenarios, here with a focus on the visual part of the avatar. In this
respect, a talking-head generation is an AI-based tool that can be utilized for training
professionals to deal with child abuse cases. By using such tools, challenging and
sensitive situations can be simulated. In this study, we will examine the potential of
different generative models for talking head avatars in the context of child abuse. We
created different videos using current state-of-the-art models such as MakeItTalk, First
Order motion, and Talking-Face PC-AVS with different styles. These styles include
Cartoony, Painted, and Original. By running a user study, the effectiveness in terms of
realism and preferences has been examined. The results show that the Talking-Face PC-
AVS model creates better user experiences. Furthermore, according to the user study,
we did not observe significant differences among the different styles of the videos. To
improve the talking head avatars in child abuse applications, choosing the appropriate
model should be considered as the first priority. We expect that more sophisticated
models regardless of the style, can contribute to educating experts for the child abuse
interviews remarkably.
i
ii
Acknowledgments
Also, I am grateful to the whole family of SimulaMet, for giving me the opportunity to
participate in the talking-head avatars project.
iii
iv
Contents
Abstract i
Acknowledgments iii
1 Introduction 1
1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Problem Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.3 Scope and Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.4 Research Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.5 Ethical Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.6 Main Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.7 Thesis Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
v
2.6.1 Audio-driven Talking-Head Generation . . . . . . . . . . . . . . . 21
2.6.2 Video-driven Talking-Head Generation . . . . . . . . . . . . . . . 22
2.7 Uncanny Valley . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
3 Research Methodology 25
3.1 Tools and Technologies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
3.1.1 Google Colaboratory . . . . . . . . . . . . . . . . . . . . . . . . . . 25
3.1.2 Python . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
3.1.3 Python Libraries . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
3.1.4 Analysis Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
3.2 Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
3.2.1 MakeItTalk Model . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
3.2.2 First Order motion model . . . . . . . . . . . . . . . . . . . . . . . 31
3.2.3 PC-AVS model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
4 Experiment 37
4.1 User Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
4.1.1 Demographic of participants . . . . . . . . . . . . . . . . . . . . . 38
4.1.2 Questionnaire . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
4.1.3 Test Materials . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
4.1.4 Data cleaning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
7 Conclusion 79
7.1 Major Takeaways . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
A Questionnaire 1 81
B Questionnaire 2 89
vi
List of Figures
3.1 The overview of the MakeItTalk model has been published in their
paper (Y. Zhou et al., 2020) . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
3.2 The overview of the FOMM model has been published in their paper
(Siarohin et al., 2019b) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
3.3 The overview of the PC-AVS model has been published in their paper
(H. Zhou et al., 2021) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
vii
5.10 Perceived realism of avatars . . . . . . . . . . . . . . . . . . . . . . . . . . 72
5.11 Overall favorite avatars . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
5.12 Preferred avatars for computer-mediated conversations . . . . . . . . . . 73
5.13 Perceived realism of avatars during speech . . . . . . . . . . . . . . . . . 73
5.14 Most believable avatars . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
viii
List of Tables
ix
x
Chapter 1
Introduction
1.1 Motivation
Child abuse is a serious problem that has far-reaching consequences for the physical,
emotional, and psychological well-being of children. According to the World Health
Organization (WHO)1 , child abuse refers to all forms of physical or emotional
maltreatment, sexual abuse, neglect or negligent treatment, and exploitation that result
in actual or potential harm to the child’s health, development or dignity (World Health
Organization, 2021). Child abuse can occur in any setting, including the home, school,
and community. It is estimated that millions of children worldwide are affected by
abuse each year, with many cases going unreported.
In society, it is crucial to prioritize the well-being of children and ensure they receive
the greatest care, to prevent any form of mistreatment. By taking proactive measures
to protect children, we can avoid the potential fundamental problems that may arise
as consequences. This perspective emphasizes the significance of creating a safe and
nurturing environment for children, highlighting the importance of collective efforts
to safeguard their rights and promote their welfare.
According to WHO, child maltreatment results in the annual death of at least 850
children under the age of 15. Approximately 71% of homicide deaths occur in low-
income and middle-income countries, where rates are 2.4 times higher compared to
high-income states. Boys make up 60% of the victims in these cases. Homicide rates
are higher among children under 4 years old compared to older children aged 5-9 and
10-14. Eastern Europe experienced a peak in child homicide rates during the economic
and political transition, and while the rates have declined, they remain higher in this
region (Sethi et al., 2013).
1 https://www.who.int/
1
The effects of child abuse are devastating and can last a lifetime. Children who
experience abuse are more likely to suffer from a range of emotional, behavioral,
and physical problems. They may experience anxiety, depression, post-traumatic
stress disorder (PTSD), and other mental health disorders. Physically, children who
experience abuse may suffer from injuries, poor growth, and developmental delays. In
addition, they may be at a higher risk of developing chronic health conditions such as
heart disease, diabetes, and obesity (Norman et al., 2012).
The negative effects of child abuse are not limited to the individual child. They can
also impact the child’s family, friends, and community. For instance, child abuse can
lead to increased healthcare costs, decreased work productivity, and increased crime
rates. As a result, it is important for healthcare providers, psychologists, and social
workers to be aware of the effects of child abuse and to provide appropriate care and
support to affected children.
The prevention of child abuse is a complex and multifaceted issue that requires the
involvement of multiple sectors, including healthcare, education, law enforcement,
and social services. Prevention efforts should focus on addressing the underlying
causes of abuse, such as poverty, social inequality, and family dysfunction. In addition,
it is important to provide support and resources to families and caregivers to help them
provide a safe and nurturing environment for their children (Lambie, 2005).
So far, we can state that child abuse is a serious problem that has far-reaching
consequences for the physical, emotional, and psychological well-being of children.
There should be serious plans for solving this important problem. Healthcare
providers, psychologists, and social workers have an important role to play in
preventing and addressing the effects of child abuse. By working together, we can
help to ensure that all children have the opportunity to live healthy and fulfilling lives.
In this case, it is crucial to communicate with the victims and inquire about the details
of the abuse. Professionals need to ask different questions to find out various aspects
of that misbehavior and gather all facts in this regard. Being aware of these details can
assist society in finding effective solutions in this regard.
Given the sensitivity and importance of the topic of child abuse, it is vital to consider
the challenges associated with working with children, particularly those who have
been affected by abuse. Such children may be more sensitive than their peers.
One of the critical steps in addressing child abuse is the ability to communicate
effectively with the affected children and conduct interviews to gather details about
the maltreatment. This procedure holds significant importance for the healthcare
system’s enhancement of more effective resolutions to this issue. However, prevailing
literature underscores diverse factors that create hindrances for children when it
2
comes to openly sharing their abuse encounters with professionals. Thus, it becomes
vital for professionals involved in handling child abuse cases to comprehend the
essential needs of children and skillfully attend to those demands during the disclosure
phase (Ettinger, 2022). Nevertheless, conducting interviews with child abuse victims
presents distinct challenges. These youngsters often encounter difficulties in placing
trust in others and might have undergone trauma inflicted by close family members,
thereby adding complexity to the interview process.
To this end, extensive training and practice are necessary before professionals begin
working with children. This is a standard process in many other professions, such as
piloting. Pilots do not fly airplanes during their initial training but instead progress
through various levels and simulations before being granted permission to fly. In
the child abuse topic, simulation is extensively different. Professionals working in
the field of child abuse face significant challenges when it comes to practice. Each
child is unique and has their own set of rights that must be protected, and even the
slightest misstep or inexperience when interacting with a child can have severe and
lasting consequences for their well-being.
3
trainers to interact with the interface as a child as well.
In conclusion, it is clear that there is a need for tools to train professionals who
work with child abuse victims, including social workers and law enforcement officers.
Virtual agents provide a valuable means for professionals to practice interviewing
techniques and improve their communication skills with child abuse victims in a
safe and controlled environment. Actually, studies have shown that computer-based
interactive learning can further improve investigative interviewing skills (Powell et al.,
2016).
This project not only tackles a vital societal challenge but also highlights the
potential of technology-driven solutions to enhance the well-being of vulnerable
individuals. By bridging the divide between technology and social welfare, it offers an
inspiring paradigm that aligns with the broader endeavor of leveraging technological
innovations to address urgent social concerns. The project’s interdisciplinary essence,
integrating the technical capabilities of computer science with the social sciences
insights into human behavior, showcases the potency of collaboration in crafting
meaningful solutions. The project’s focus on augmenting communication and
investigative skills seamlessly resonates with the broader goal of utilizing technology
for positive societal transformation (Salehi, Hassan, Lammerse et al., 2022a).
2 https://www.simulamet.no/simulamet-projects
4
1.2 Problem Statement
As previously mentioned, adequate training for professionals and psychologists in
conducting interviews with children is of paramount importance in their work with
young individuals. To create an effective platform for this purpose, practical tools are
necessary. Fortunately, there are various technological tools that can be used to create
such systems. Talking head models can serve as an excellent resource in this regard.
Talking head models, also known as virtual agents, are computer-generated characters
that simulate human-like conversation and facial expressions (Htike, 2017). These
models have been increasingly used in various applications, including child abuse
projects. During these interviews, professionals have the opportunity to practice their
interviewing skills using talking head avatars. This approach ensures that trainers
can thoroughly train without any concerns about potentially harming children while
asking necessary questions. Additionally, the virtual agent can be programmed to
provide information and support to the child, such as resources for counseling or legal
assistance (Salehi, Hassan, Lammerse et al., 2022b).
There are several talking head models (e.g MakeItTalk3 (Y. Zhou et al., 2020), PC-AVS
model 4 (H. Zhou et al., 2021), FOMM model 5 (Siarohin et al., 2019b), OneShot6 (T.-C.
Wang et al., 2021), FACIAL 7 (C. Zhang et al., 2021b) and ...) used to create avatars, each
with its own strengths and weaknesses. Some of these models are older and represent
the first generation of this technology, generating avatars using only a single picture
and audio. Although these models can be relatively simple to use, some of them are
more complex and require a large number of inputs, resulting in more realistic avatars.
In addition to models, the choice of materials used to create avatars can also
significantly impact their overall appearance. There are various options available,
including using real photographs and recordings to create an avatar, opting for
cartoony or stylized images, or using images with filters that give them a painted
look. Similarly, there is the choice between using human-made sounds or robotic-
generated ones for the audio component of the avatar. The inquiries investigated in
this study revolved around users’ interactions with talking head avatars, exploring
their experiences during communication and their emotional responses to witnessing
the avatar’s speech. The study delved into aspects such as users’ preferences, and their
perceptions of the avatar’s realism. This study examined features related to both the
3 https://github.com/yzhou359/MakeItTalk
4 https://github.com/Hangz-nju-cuhk/Talking-Face_PC-AVS
5 https://github.com/AliaksandrSiarohin/first-order-model/tree/master
6 https://github.com/zhanglonghao1992/One-Shot_Free-View_Neural_Talking_Head_Synthesis
7 https://github.com/zhangchenxu528/FACIAL
5
visual aspects of avatars and their realism while speaking. From the models presented
earlier, a subset was chosen for a user study aimed at addressing the following research
inquiries:
• RQ1: Which models can generate avatars that generally give a better feeling to
the audience?
• RQ2: What distinctions emerge in user experiences when interacting with real,
cartoony, and painted avatars? How do these differences influence viewers’
perceptions and emotional responses toward these avatars?
We believed that addressing these questions could help fill a minor gap within our
scope of work concerning generated talking head avatars. The findings from this
study have the potential to guide developers in refining various aspects of this field,
particularly in areas aligned with our study’s focus, such as exploring different styles
and models.
The study’s focus was on addressing the issue of child abuse, prompting the use of
avatars featuring children’s images. Initially, the text concerning child abuse was
selected, but considering its potential impact on participants’ emotions, the concept
shifted to something general.
6
would offer a more pronounced contrast in participants’ emotional responses. This
approach allowed us to gain valuable insights into how participants perceived each
style and how their reactions differed among the chosen avatars.
The applicability of the results and conclusions from this study might not extend to
other domains within this field. The findings within our narrow scope provide the
basis for our reliance. One of the study’s limitations arises from this constraint. For
instance, results could differ if avatars featured adult individuals or if alternative
models with superior performance in distinct aspects were chosen. We opted to
minimize the number of videos in the study to expedite participant responses in
the questionnaires, a choice that stemmed from the desire for efficient engagement.
However, it’s important to note that this decision also posed a limitation to our
study. This pragmatic approach allowed us to streamline the study process, enabling
participants to share their insights comfortably within a reasonable time frame.
7
1.5 Ethical Considerations
In our context, ethical issues have various aspects. As artificial intelligence (AI)
continues to play an increasingly significant role in various fields, the ethical
implications of its use are becoming more crucial. Deepfakes, a highly sensitive issue,
possess the potential for both malicious and socially harmful purposes. Consequently,
steps are being taken to create ethical guidelines and regulations that prohibit the
unethical use of this technology (Meskys et al., 2020). Talking heads are a similar
technology to deepfakes, with some differences, but they present similar ethical
challenges. Like deepfakes, talking heads also can be used to create fake news
or defame individuals. This can have serious consequences, such as damage to
reputations or even public safety.
8
1.6 Main Contributions
The core inquiries of this study were geared towards gaining a comprehensive
understanding of participants’ perceptions and satisfaction, with a focus on assessing
their overall experiences, the realism of speech and movements, and their comfort
levels during interactions with avatars. In the later phase, we aimed to discern
participants’ preferences among avatars with cartoony, painted, or real picture styles.
It’s noteworthy that the study also involved comparing the outcomes of different
talking head models using consistent audio, images, and videos.
To achieve our research objectives, a user study was conducted where participants
first watched videos made using a model, showing one character in a certain style.
After that, they answered six questions that ranged from bad to excellent. Participants
watched 19 videos in total, but the first one was excluded from our analysis since it
served as an anchor. The user study was carried out using Microsoft Forms and was
distributed to individuals through various platforms to collect responses.
The main focus was to understand how comfortable participants felt when they used
these avatars. Additionally, we asked more detailed questions about how well the
avatars’ lip movements matched and how natural their head movements looked. We
also wanted to know how good the avatars were overall. We looked at different kinds
of avatars, like boys and girls with cartoony, painted, and real pictures. We compared
and studied them closely.
In the upcoming paragraphs, a closer look will be taken at the research questions and
provide a simple breakdown of the findings we have uncovered in response to each
one.
• "Which models can create avatars that make people feel better?".
This question was initially raised in the problem statement section. Out of the three
models which are tested, the one that lets us control the avatar’s poses stood out.
The features considered in the judgement of these models based on six important
things: how good the whole experience felt, how the avatar’s head moved, whether
its lip movements matched the speech, how well it talked, the video quality, and how
comfortable people were with it. Interestingly, the model with pose control scored
better in all six aspects.
• "What distinctions emerge in user experiences when interacting with real, cartoony, and
painted avatars? How do these differences influence viewers’ perceptions and emotional
responses toward these avatars?".
Throughout the user study, a notable observation emerged, indicating that the
9
various styles employed did not exert a significantly influential effect on the
outcomes. Notably, among all the features assessed, videos adopting the Original style
predominantly claimed the top positions. There were instances where participants
displayed a comparable preference for the cartoony style, although this distinction did
not yield a noteworthy variance in the overall assessment.
• Chapter 1: Introduction
Crucial information has been incorporated into the initial chapter. The motivation
behind the whole study, including the significance of the child abuse topic and
essential statistics, has been explained. Furthermore, the problem statement and all the
limitations encountered during this study have been detailed. In this context, ethical
concerns, which encompass various aspects briefly discussed in this chapter as well,
hold a significant place.
In this chapter, various aspects of related research are explored, as evident from its title.
The initial section is devoted to prior studies on child abuse, while the subsequent
parts delve into technical facets such as machine learning methods and the creation
of talking head avatars. The discussion also covers topics like deepfakes and face-
swapping. The uncanny valley concept is also discussed in this chapter.
The Research Methodology chapter delves into the models utilized for video produc-
tion in the user study. An elucidation of the assorted tools employed across the study’s
duration is presented in the initial segment of this chapter. Additionally, a discussion
is offered regarding the selection rationale for these models from an array of alternat-
ives, as outlined at the outset. A summary table encapsulating the chosen models is
also provided, offering a concise overview of their key attributes.
• chapter 4: Experiment
10
In the Experiment chapter, we delve into the details of all the materials we used in
the user study. This includes the information we collected from the participants, as
well as their background details. We thoroughly examine the questionnaires that were
given to the participants during the study. Following that, we illustrate the process of
cleaning and preparing the data to obtain the final results and visual representations.
This chapter outlines the systematic approach we followed to carry out the study.
This chapter showcases numerous plots and elucidates the data derived from
participants. The results are categorized into three sections: the main effect of the
model, the main effect of the character, and the main effects of the style. The latter
part of the chapter delves deeper into the second segment of the questionnaire, which
centers around the uncanny valley concept. Detailed insights obtained from the
questionnaires are presented.
This chapter presents a comprehensive overview of the entire study along with the
conclusions drawn from the analysis of the conducted user study. While we initially
intended to conduct a second phase of the user study, various limitations hindered
its execution. As part of future work, we propose exploring the extensive potential
sub-topics within this domain, considering the scope for further investigation and
development.
11
12
Chapter 2
In order to cover the interdisciplinary nature of the topic, the related work chapter will
survey different areas. One crucial aspect of this research concerns child abuse and
maltreatment. Thus, firstly, an overview of relevant papers and statistics on this issue
will be presented. Then, a review of the academia on various machine learning models
and their applications will be provided, including the differences between audio-
driven and video-driven models, the progress of different models, and the emergence
of complex yet highly effective talking head models. The chapter will explore the
combination of these two topics, which involves using talking head avatars to develop
a tool for taking action against child abuse.
13
to evaluate and monitor the scale of the issue, particularly in measuring trends in the
most severe cases. Improved record-keeping of children, supplemented by regular
surveys, is considered crucial by concerned professionals to detect the much larger
proportion of maltreatment in the community that goes unnoticed by child protection
agencies.
However, it should be noted that these statistics pertain to countries where people
have access to information. It is important to acknowledge that there are numerous
instances of maltreatment occurring in many countries, which often go unreported due
to various challenges and barriers. Maltreatment in the community is often chronic
in nature, rather than acute. Supportive interventions for familial malfunction and
parenting assistance are required by most families, rather than retribution and blame.
A child’s healthy development depends on safe, stable, and nurturing relationships
with parents and other caregivers. Severe and recurrent maltreatment may result in
toxic stress, impact brain development in childhood, and cause cognitive impairment
and the adoption of health-risk behaviors, with adverse mental and physical health
outcomes. Post-traumatic stress disorder has been reported in as many as one-quarter
of abused children, and child maltreatment may be responsible for nearly a quarter
of the burden of mental illness, particularly when combined with other adverse
childhood experiences (Sethi et al., 2013).
A deeper investigation into the child abuse statistics and impact is out of the scope of
this study. A brief glance at the current reports suggests how important it is to improve
therapeutic and clinical practices to resolve child abuse-related issues in society. As
emphasized earlier in Chapter 1, the goal of using new technologies like AI-based tools
in this context is to provide more effective ways for training professionals. Going deep
into machine learning techniques, we can find extensive possibilities and potential for
these applications.
14
are relevant to this study.
15
main function is to produce new data points resembling the training data they were
provided. These models find extensive applications in image synthesis, natural
language processing, and data augmentation. Figure 2.1 illustrates the timeline of
progress in this field (Toshpulatov et al., 2023). In the subsequent paragraphs, we will
delve into some of the methods in greater detail.
CNN is a type of neural network used for image synthesis and talking face generation
in videos. They work on the principle of signal values arranged in a regular grid,
and the interactions between these values are limited to a local neighborhood. CNNs
have been successful in creating realistic images and synchronizing the movement of
a talking face with the audio. This involves predicting sequential pixels to generate
images and using Conditional PixelCNN to condition the model on specific vectors
(Toshpulatov et al., 2023).
CNN offers promising solutions for generating talking faces in videos. The Cascaded
Refinement Network and X2Face are two notable methods that demonstrate the
capabilities of CNNs in producing realistic images and controlling facial expressions
and poses. These advancements open up new possibilities for video face editing and
puppeteering, with potential applications in various fields of computer vision and
multimedia.
16
2.3.2 Variational Autoencoder (VAE)
Variational Autoencoders and their diverse adaptations have found extensive applic-
ation across various domains, including dialogue generation, image synthesis, and
learning disentangled representations(Shao et al., 2020). VAE is a popular generat-
ive modeling approach extensively used for unsupervised representation learning. It
consists of two interconnected components - the encoder (recognition model) and the
decoder (generative model) - working in tandem to produce meaningful latent repres-
entations and approximate posterior distributions (Kingma & Welling, 2019). In a VAE,
the input data is compressed into a compact representation called the latent space by
the encoder. This condensed representation contains essential information about the
input. Then, the decoder reconstructs the original data from the latent representation.
The VAE’s ability to disentangle different factors of variation makes it useful for vari-
ous applications. One fascinating application is talking head generation, where VAEs
can create realistic video sequences of talking faces with control over facial expressions
and speech. This makes VAEs a powerful tool in the fields of artificial intelligence and
computer graphics.
17
model (Kammoun et al., 2022). The figure presents an overview of the GAN model,
showcasing selected images from a project available on GitHub (Perarnau et al., 2016).
Figure 2.2: Baseline GANs architecture for face generation(Kammoun et al., 2022)
The versatility of GANs and their ability to create realistic data have led to a myriad of
applications in the fields of artificial intelligence and computer science. They have been
successfully applied to tasks like image generation, style transfer, super-resolution,
and data augmentation, as well as text-to-image synthesis and generating realistic
human faces. GANs have revolutionized the field of generative modeling and continue
to drive innovation in various domains, offering exciting opportunities for advancing
artificial intelligence research and applications.
One of the key advantages of NeRF is its ability to generate high-resolution, realistic
images without relying on discretized voxel grids, overcoming the storage limitations
associated with traditional methods. This continuous representation also makes the
optimization process more efficient, enabling NeRF to achieve state-of-the-art results in
view synthesis tasks. Additionally, NeRF can handle real-world scenes with complex
geometry and appearance, providing impressive results for various applications, such
as view synthesis and virtual reality. Overall, NeRF offers a powerful and versatile
18
solution for generating lifelike views of scenes, elevating the field of computer vision
and graphics to new heights.
The versatility of 3DMM extends to various domains, including human face recog-
nition, face generation, animation, lip synchronization, experimental psychology, en-
tertainment, and texture knowledge representation. To overcome the limitations of
conventional 3D reconstruction techniques in reconstructing 3D models from paint-
ings, researchers have combined 3DMM with other methods, resulting in significant
advancements in 3D human face reconstruction and landmark localization. Further-
more, deep learning techniques, particularly GANs, and CNNs, have revolutionized
face generation by synthesizing high-quality, realistic human face images. CNNs have
been instrumental in creating facial expressions and recognizing facial characteristics.
Recent studies have focused on enriching 3D human face geometry with photomet-
19
ric data using GANs, enabling the generation of high-resolution facial geometry and
reflectance maps.
20
2.5.3 Face Generation
This type of face synthesis involves generating a talking face video from a single face
image and a sequence of audio. It uses neural networks to generate facial expressions
based on the input audio. Researchers have worked on improving these face synthesis
techniques by disentangling face attributes and identities, using face segmentation,
and introducing temporal coherence to create more realistic results. While deepfake
technology has various applications, it has also raised concerns about its potential
misuse, especially in creating deceptive and misleading content. Efforts are being
made to develop countermeasures and identify deepfake videos to address these
challenges.
21
was trained on a dataset of more than 17 hours of video recordings of a person speaking
in different poses, lighting conditions, and backgrounds. The study showed that
the audio-driven model could generate realistic videos that closely matched the lip
movements and facial expressions in the audio input (Suwajanakorn et al., 2017).
Both audio-driven and video-driven models have their advantages and disadvantages
in talking head models. Audio-driven models are useful when the input video is not
available or is of low quality, while video-driven models can generate more accurate
and realistic videos but require more data and computational resources. Further
research is needed to improve the performance of both types of models and to explore
new approaches for generating talking head videos.
To avoid the uncanny valley, designers of talking head avatars must carefully balance
realism with stylization. They may also need to adjust the avatar’s facial expressions
and movements to make them more natural and engaging. Additionally, user testing
can help identify any aspects of the avatar that may trigger the uncanny valley
response. The graph 2.3 presented in (Igaue & Hayashi, 2023) clearly illustrates the
threshold of the uncanny valley.
22
Figure 2.3: Graph of Uncanny Valley Threshold represented with (Igaue & Hayashi,
2023)
Insights into the mechanisms underlying the uncanny valley and potential design
principles for bridging the gap between human likeness and comfort level are
presented in MacDorman et al.’s paper (MacDorman et al., 2009). These principles
are also relevant to talking head avatars. Adjusting the texture and level of detail
in the avatar’s skin can have an impact on its perceived eeriness and human
likeness. Additionally, unconventional facial proportions may be more unsettling in
photorealistic faces, and a discrepancy in the size and texture of the eyes and face can
contribute to an eerie appearance (MacDorman et al., 2009).
In conclusion, the design of talking head avatars can be affected by the uncanny
valley phenomenon. Realism and stylization must be balanced by designers, who
must carefully adjust facial expressions and movements to prevent the uncanny valley
response in users from being triggered. The principles outlined in (MacDorman et al.,
2009) can guide the design of talking head avatars that are both engaging and realistic.
In our user study, a cartoony and painted style was chosen to examine whether it
elicited a better response from users in terms of the uncanny valley phenomenon.
In summary, we examine some of the mentioned methods in this section and try
to propose a benchmark for the talking head generation methods based on multiple
factors in the context of realistic facial expressions. The more sophisticated models are
supposed to provide better results, while the uncanny valley phenomenon suggests
there should be a trade-off between realism and stylization.
23
24
Chapter 3
Research Methodology
Within the realm of tools, a distinction was made between those used for executing
model-related code and those employed for in-depth result analysis. The compre-
hensive elucidation of these tools will be undertaken within this section, affording a
comprehensive understanding of the procedural aspects adopted.
25
or configurations. Due to the use of various models and the requirement to execute
them, it becomes crucial to ensure compatibility between different Python libraries.
To achieve this goal, a tool was chosen that could install all the necessary libraries
from the beginning. This approach facilitated seamless integration and ensured the
smooth execution of the models. However, a drawback of this approach was that
uploaded files, such as pictures, audio, and videos, were not automatically saved. As
a result, users had to re-upload these files each time they started running the code
after a certain period. This inconvenience could be time-consuming and interrupt the
workflow during repeated executions.
3.1.2 Python
The majority of the work in executing models in this thesis relies on Python 3, the latest
version of the Python programming language, which was released in 2008. Python is a
popular and versatile programming language known for its simplicity and readability.
It is widely used in various domains, including web development, data analysis,
artificial intelligence, and automation. Python’s straightforward syntax and extensive
libraries make it accessible for beginners and efficient for experienced developers. Its
versatility and strong community support have contributed to its widespread adoption
across different industries, making Python a valuable tool for tackling a wide range of
programming tasks.
26
learn also provides features for data preprocessing, model evaluation, and model
selection, making it a complete package for machine learning projects. It is widely
used in academia and industry for various applications, and its active community
ensures that it stays up-to-date with the latest advancements in the field of machine
learning. This library has been used in the implementation of our models. The
Librosa library was utilized in both the MakeItTlk model and the PC-AVS model in
this study. It serves as a Python library designed specifically for audio and music signal
processing, offering a user-friendly interface to handle audio data and extract relevant
information from audio files. Librosa enables users to analyze, manipulate, and
extract features from audio signals, making it a versatile tool for various audio-related
tasks. Its widespread use in the field of audio processing and music research can be
attributed to its simplicity, efficiency, and extensive documentation. For developers
and researchers working on audio-related projects, Librosa proves to be an invaluable
resource, facilitating effective audio data processing and analysis within their Python
applications. PyTorch, used in the FOMM model motion model and PC-AVS model,
stands as a prominent open-source machine learning library extensively utilized in the
artificial intelligence domain. Created by Facebook’s AI Research Lab (FAIR), PyTorch
offers a versatile and efficient framework for constructing and training deep learning
models. Its unique "define-by-run" approach, featuring a dynamic computational
graph, empowers developers to construct models on-the-fly while executing the code,
leading to simplified debugging, experimentation, and implementation of intricate
architectures.
27
making processes.
3.2 Models
In order to evaluate users’ experiences with various talking head avatars, it was neces-
sary to select specific models to execute and generate results for user feedback. Mul-
tiple models were considered, encompassing both older and more recent iterations.
One of the criteria used for selecting models was to ensure that they had a functional
GitHub page with open-source codes that could be executed with Google Colab. In
the initial stage of the user study, our focus was on audio-driven models, which are
well-known in the field. We aimed to compare the results of two of these models with
a more recent model.
Initially, we gathered a long list of talking head avatars spanning different years. From
this list, we narrowed it down to nine models, including one called Facial 1 (C. Zhang
et al., 2021a). This model uses implicit attribute learning to create realistic dynamic
talking faces. It combines techniques like face reconstruction, audio feature extraction,
and face rendering to generate lifelike animations for various purposes. We attempted
to use this model, but it required a specific dataset for training. However, finding or
creating the needed dataset was time-consuming and unfeasible, leading us to exclude
this model from our selection.
The other Model in our shortlist was the "LiveSpeechPortraits2 " model that is an
implementation of a real-time photorealistic talking-head animation system (Lu et al.,
2021). This system generates personalized talking-head animations driven solely by
audio signals at a frame rate of over 30 fps. It consists of three stages: deep audio
feature extraction, learning facial dynamics and motions from the audio features,
and synthesizing photorealistic renderings with explicit control of head poses using
image-to-image translation networks. The model showcases the ability to create high-
fidelity personalized facial details and offers better performance compared to existing
techniques, as demonstrated through qualitative and quantitative evaluations. This
model offers a user-friendly web demo3 that’s quite convenient to use. However, its
limitation became apparent in our case. The demo only allowed selections from their
predefined characters and audio options, lacking the option to utilize custom audio
or video inputs. This misalignment with our project’s objectives led us to remove
this model from consideration. The subsequent option explored was the "Wav2Lip4 "
1 https://github.com/zhangchenxu528/FACIAL
2 https://github.com/YuanxunLu/LiveSpeechPortraits
3 https://replicate.com/yuanxunlu/livespeechportraits
4 https://github.com/Rudrabha/Wav2Lip
28
model, recognized for accurately syncing lip movements with speech in videos. This
model is versatile, accommodating various identities, languages, and even CGI faces.
It draws from a research paper presented at ACM Multimedia 2020 (Prajwal et al.,
2020). The Wav2Lip model effectively generates lip-sync animations, aligning lip
movements with different audio inputs to produce remarkable results. However, this
model’s applicability was limited by our dataset, which solely contained pictures of
characters without corresponding videos. Consequently, due to this constraint, we
couldn’t utilize the Wav2Lip model for the purposes of this study.
There are two other models named "One-Shot5 " and "PIRenderer6 ". The One-Shot
Free-View Neural Talking Head Synthesis(T.-C. Wang et al., 2021) has a goal to
create dynamic talking-head videos for video conferencing. It generates talking-
head animations from a single image, allowing viewing from different angles. The
repository includes training, demo capabilities, and pre-trained models for various
resolutions. It focuses on "one-shot free-view" synthesis, using rotation matrices
and post-processing like face restoration for improved results. The "PIRenderer"
model, from the ICCV2021 paper(Ren et al., 2021), introduces a unique method for
creating controllable portrait images using Semantic Neural Rendering. By untangling
3DMM parameters, the model enables intuitive control over facial motion synthesis,
useful for portrait editing, alignment, imitation, and audio-driven reenactment. The
model’s architecture intelligently manipulates facial motions based on fully untangled
3DMM parameters, offering a powerful tool for creating realistic and manageable facial
animations and advancing portrait image synthesis. However, both models lack a
Google Colab demo that was promised to be available soon but hasn’t been published
yet due to system limitations and time constraints. As a result, they were removed
from the current list and considered for future work due to challenges in executing the
code and the time required. Table 3.1 shows the summary of all unused models and
the reason of exclusion.
5 https://github.com/zhanglonghao1992/One-Shot_Free-View_Neural_Talking_Head_Synthesis
6 https://github.com/RenYurui/PIRender
29
Model Name Type Summary of Model Reason for Exclusion
Dynamic Utilizes implicit attribute learning Specific dataset needed for training
Facial
Talking for realistic talking faces. impractical to obtain.
Real-time Generates real-time personalized Web demo limitations
LiveSpeechPortraits
Animation talking-head animations based on audio signals only predefined choices allowed.
Lip-Sync Accurately syncs lip movements Dataset lacked character
Wav2Lip
Animation with speech in videos. videos for lip-syncing.
Talking Head Creates dynamic talking-head Lack of Google Colab demo
One-Shot
Synthesis videos from single images. unavailable due to constraints.
Controllable Generates controllable portrait Absence of Google Colab demo
PIRenderer
Portrait images using Semantic Neural Rendering. unavailable due to constraints.
Finally, for this study, three models were selected: MakeItTalk Model, FOMM model,
and PC-AVS model. Each model will be briefly outlined, with a little more explanation
provided for each:
To achieve this, the researchers developed a novel architecture for the deep learning
model, which combines a motion encoder that extracts facial movements from a video,
a speech encoder that converts audio into a latent representation, and a decoder that
generates the final video. The model is trained using a loss function that encourages it
to generate videos that match both the facial movements and the audio.
30
Figure 3.1: The overview of the MakeItTalk model has been published in their paper
(Y. Zhou et al., 2020)
The authors of the paper also propose a technique for voice conversion, which allows
the generated videos to have the voice of a different person than the one in the reference
images. This is achieved by training a separate neural network to convert the voice of
the input audio to the desired target voice.
They also have faced some limitations in the current approach and suggested avenues
for future research, such as incorporating more complex facial expressions and
gestures and improving the model’s ability to handle variations in lighting and pose.
Overall, this method presents a promising avenue for generating high-quality talking
head videos with a wide range of applications.
To utilize the MakeItTalk model, we must have an audio file in the *.wav format and a
single avatar picture with a resolution of 256*256 pixels. The resulting video shows the
main output in the center, the landmark video on the left, and the picture on the right.
The final step involves cropping the video to retain only the main portion. This can be
done either by modifying the code or by using video editing software, with the latter
option resulting in higher video quality. After completing these steps, the final videos
are produced. The MakeItTalk model can be found in this GitHub directory7
31
representation is then combined with the appearance of the source image using a
generator network to generate an animated sequence (Siarohin et al., 2019a).
The method uses a set of key points, which are defined as distinctive locations on
the image surface, to represent the motion between two frames. The key point
displacements are computed by comparing the locations of the key points in the two
frames. The local affine transformations are used to describe the non-rigid deformation
of the image surface between the frames.
To model occlusions in the image, the method uses a binary mask that indicates which
parts of the image should be inpainted. The mask is generated by computing the
foreground and background regions based on the difference between the source and
driving frames. The method then applies the mask to the appearance of the source
image to generate an inpainted image that is used by the generator network.
The generator network consists of a series of convolutional layers that encode the
appearance of the source image and the motion representation of the driving video.
The encoded features are then passed through a series of deconvolutional layers to
generate the animated sequence. A comprehensive understanding of their model can
be gleaned from the detailed overview presented in figure 3.2.
Figure 3.2: The overview of the FOMM model has been published in their paper
(Siarohin et al., 2019b)
The method was evaluated both quantitatively and qualitatively on several benchmark
datasets and showed superior performance compared to state-of-the-art methods. The
method demonstrated particularly impressive results on datasets with highly non-
rigid objects, such as the Tai-Chi-HD dataset, indicating the effectiveness of the local
affine transformations in modeling complex motion.
To generate a talking head avatar using the FOMM model, we require an image with
dimensions of 256*256 pixels and a video in *.mp4 format. The video should contain
32
the desired audio that will be incorporated into the final result. Additionally, the video
should not exceed in size. The model utilizes the audio and head movement from
the video to generate a synchronized and lip-synced final video on a single image. To
achieve this, we recorded a video with exaggerated lip movements and added a robotic
audio track to create the desired final version. The Link for checking more details about
this model is in the footnote 8 .
Figure 3.3: The overview of the PC-AVS model has been published in their paper
(H. Zhou et al., 2021)
The PC-AVS model uses a novel complementary learning procedure that optimizes
both lip-sync accuracy and pose control. It also introduces a contrastive loss function
that improves the synchronization between the audio and video. The model achieves
high performance on several metrics, including lip-sync accuracy, head movement
8 https://github.com/AliaksandrSiarohin/first-order-model.git
33
naturalness, and video realness. It also demonstrates great robustness under extreme
conditions, such as large poses and viewpoints.
Technical issues addressed in the paper include the design and training of the three
modules, the complementary learning procedure, the contrastive loss function, and the
evaluation metrics used to assess the performance of the model. Ablation studies were
conducted to evaluate the impact of different aspects of the model, such as the pose
code length and the design of the generator. Overall, the PC-AVS model represents a
significant advancement in the field of audio-visual synthesis and has the potential for
various applications, such as virtual assistants, video conferencing, and entertainment.
To use the PC-AVS model, certain inputs are required which are specified in a script
called "prepare-testing-files.py". This script provides great flexibility in formulating
metadata through the use of several flags. The first flag, "src-pose-path", is used to
indicate the driving pose source path, which can be either a *.mp4 file or a folder
containing frames in the form of "0*.jpg" starting from 0. The second flag, "src-audio-
path", is used to specify the audio source path, which can be a *.mp3 audio file or a
*.mp4 video file. If a video file is given, the frames are automatically saved in a folder,
and the "src-mouth-frame-path" flag is disabled. The third flag, "src-mouth-frame-
path", is used when the audio path is not a video path, and it provides the folder
containing the video frames that are synced with the source audio. The fourth flag,
"src-input-path", is the path to the input reference image, which is converted to frames
when it is a video file. Lastly, the "csv-path" flag specifies the path to the metadata
csv file to be saved. This CSV file can be manually modified or additional lines can be
added to it following the rules defined in "prepare-testing-files.py" or the data loader
"data/voxtest-dataset.py". The "misc" folder contains several demo choices, including
ones used in the provided video. Users are free to rearrange these files across folders
and are welcome to record their own audio files. We have used the "src-pose-path"
from the main code but included our own picture and video with the voice which we
wanted to have on a result video. Here 9 is the GitHub URL to check more details
about this model.
In summary, the method used in this study includes different parts such as
implementing the models to generate different videos with varying characteristics
and running a user study to evaluate the impact of different cases. In Table 3.2, an
overview of the used models is provided. This table presents important details such as
the types of models, their GitHub locations, required inputs, and notable features. This
streamlined presentation allows readers to easily grasp the models’ characteristics. The
user study is explained in detail in the next chapter.
9 https://github.com/Hangz-nju-cuhk/Talking-Face_PC-AVS.git
34
Model Name Type GitHub Required Inputs Notable Features
Audio file (in *.wav format) Realistic facial
MakeItTalk Audio-Driven Link
Avatar picture (256x256 pixels) animations
Image (256x256 pixels) Effective for highly
FOMM Image Animation Link
Video (in *.mp4 format) non-rigid objects
Driving pose video (in *.mp4 or image frames)
Lip-sync accuracy,
PC-AVS Audio-Driven Link Audio source (in *.mp3 or *.mp4 format)
free pose control
Input reference image
35
36
Chapter 4
Experiment
For this user study, a total of 37 individuals participated, each of whom was selected at
random to complete the questionnaire. The questionnaire was distributed through
37
various platforms and research groups as well as among individuals who were
believed to be interested in participating and providing feedback. It took almost 10
days to reach out to a good number of responses. However, there was a problem
with the questionnaire, as shuffling the questions was necessary to avoid bias in the
answers. Using Google Forms was not an option for this purpose, so we had to search
for different free platforms for collecting data. We found that Microsoft Forms was a
very good tool for gathering data. But we still encountered an issue as we could only
lock one period for one questionnaire and shuffle the rest. We needed to have some
initial questions at the beginning, followed by a set of shuffled questions, and then
another section for our overall assessment, which also needed to be shuffled. As a
result, we had to create two separate questionnaires.
To be able to match the two questionnaires and identify which person answered
which questionnaire, we included two unprompted questions to track the twin
questionnaires. We did not ask any personal questions, but for some analyses,
we needed to know which questionnaire belonged to which person. Overall, the
combination of crowdsourcing and passive monitoring of network traffic provided
valuable insights and feedback from a diverse group of individuals, enabling us to
evaluate the performance of communication networks effectively. The questionnaires
could be found in A and B.
As part of our study, participants were provided with optional questions that aimed
to gather information about their gender, age, profession, and experience with
talking head user studies. While participants were not required to provide personal
information, these questions were included to gain a better understanding of the
characteristics of the respondents and their opinions. Out of the total participants,
34 individuals voluntarily provided their gender, while 35 participants provided their
age. To facilitate the understanding and visualization of the study’s findings, all
diagrams related to the research will be provided below.
38
Figure 4.1: Gender Distribution
The question about the profession was answered by 30 individuals. The participants
in this user study belong to diverse professional backgrounds that can be categorized
into three main categories: medical and health professions, science and engineering
professions, and business and administration professions. The first category includes
professions related to healthcare, such as doctors and pharmacists, as well as general
interns. The second category includes professions related to science and engineering,
such as water hydraulic engineers, Ph.D. cancer biologists, and software developers.
The third category includes professions related to business and administration, such as
39
digital marketing managers, research project coordinators, and innovation managers.
It is worth noting that some professions may belong to multiple categories. A chart
4.3 can be drawn to visually represent the distribution of the participants’ professions
across the three categories.
4.1.2 Questionnaire
To provide transparency and aid in comprehending the research methodology,
the complete questionnaire will be included in the appendix of this study. The
questionnaire consisted of specific questions that aimed to gather relevant information
for the study.
Main Questionnaire
Pretest questions were administered to gather information about the answerers’ details
and to check the appropriateness of sound on their devices. Two random questions
were also included to determine which questionnaire belonged to each participant as
explained before.
The main study involved participants watching videos and rating their experience
based on a series of questions. These questions were inspired by the work of (Wilson
et al., 2018), which proposed a questionnaire for the assessment of realism in stimuli.
Some questions were also inspired by another questionnaire conducted in (Salehi,
40
Hassan, Shafiee Sabet et al., 2022). Questions about the quality and naturalness of
head movement were added to the questionnaire to assess this aspect as well.
Labels Codes
How was your overall experience with the avatar? OverallQoE
How accurately did the lips move in sync with the audio? LipSync
How do you rate the naturalness of head movement? HeadMove
How realistic the avatar was talking? Talking
How do you rate the overall quality of this video? VideoQuality
How comfortable do you feel conversing with this avatar? Comfortably
The individual questions and their respective purposes in the main test part will be
presented below. This will assist in better understanding the methodology and provide
a comprehensive overview of the study’s objectives.
First question: How was your overall experience with the avatar?
This is a broad and general question that seeks to gather feedback on the user’s
experience with the talking head avatar. This question aims to capture the user’s
overall impression of the avatar’s effectiveness as a means of communication and
whether it was engaging and useful.
Second question: How accurately did the lips move in sync with the audio?
This question aims to evaluate the lip sync accuracy in the videos, which is a crucial
aspect of communication between the user and the talking head avatar. The purpose
of including this question is to compare the users’ overall sentiment with the quality
of the lip sync and examine their relationship.
41
One important aspect of the user’s experience with the avatar is the naturalness of
head movement.
When the avatar responds naturally to the user’s head movements, it can enhance the
user’s sense of presence in the virtual environment. This sense of presence can lead to
a more immersive and engaging experience. Furthermore, the natural head movement
can also improve communication between the user and the avatar. By mimicking
natural human behavior, the avatar can convey subtle cues and emotions that are not
easily conveyed through text or voice communication.
In addition, the natural head movement can provide the user with more interaction
and feedback based on the avatar’s behavior. For example, the avatar may nod its
head in agreement or shake its head in disagreement, providing the user with instant
feedback. This feedback can enhance the user’s sense of control over the avatar and
the virtual environment, leading to a more satisfying experience.
For talking head avatars to effectively communicate with users, it is crucial that
they are capable of producing natural-sounding speech and establishing a strong
connection with the user. Given that users often rely on visual cues, such as the
avatar’s mouth movements, to determine whether the avatar is speaking accurately, it
is important to design avatars that can produce realistic mouth movements and convey
emotions through facial expressions.
Fifth question: How do you rate the overall quality of this video?
While previous models have addressed numerous features of these avatars, such as lip
sync and head movement, one crucial aspect that must not be overlooked is the quality
of the video. High-quality video is essential in facilitating accurate communication
between the user and the avatar, as it enables the user to discern details such as facial
expressions and gestures more clearly. In addition, better video quality can enhance
the user’s overall experience and increase engagement with the avatar. Therefore,
designers of talking head avatars must prioritize the quality of the video to ensure
that users can interact with the avatars effectively and enjoyably.
Sixth question: How comfortable do you feel conversing with this avatar?
In communication between users and talking head avatars, one of the most crucial
42
aspects is to ensure that users feel comfortable interacting with the avatars. This is
especially critical in fields such as child abuse, where talking head avatars can be used
to assist professionals in communicating with children. When users feel comfortable,
professionals can ask concise and effective questions, and children are more likely to
answer honestly and provide more detailed information. Thus, designers of talking
head avatars must prioritize user comfort to enhance the effectiveness and accuracy of
communication between the user and the avatar in sensitive fields such as child abuse.
Post-Test Questionnaire
Questions Labels
Q1 Generally, which avatars have the most realistic appearance for you?
Q2 Generally, which avatars do you like the most?
Q3 For a conversation (talking to a computer), which avatars do you prefer to use?
Q4 Considering the avatars’ talking, which was the most realistic?
Q5 Generally, which avatars were the most believable for you?
For our user study, we utilized a 5-point Likert scale, with scores ranging from 1 to 5.
The words "bad," "poor," "fair," "good," and "excellent" were used to represent scores 1
to 5 in our analysis. The use of the Likert scale enables us to quantitatively measure
participants’ responses and facilitates data analysis. The Likert scale was initially
developed by Dr. Rensis Likert, a sociologist at the University of Michigan, in the 1930s
43
as a scientific method of measuring psychological attitudes. Since then, the Likert scale
has been widely used in various fields, including social sciences, marketing research,
and psychology. Its ability to capture participants’ attitudes in a quantitative manner
has made it a popular tool in many research settings.
Figure 4.4: Illustration of the two distinct Characters evaluated in the user study.
Three different styles were generated for each character. The objective was to find
a code that could generate cartoonish and painted versions of the pictures. Useful
code was discovered on Github to generate results, but the quality was considered
inadequate. An attempt was made to apply Wang et al.’s proposed model for
generating animated images to our pictures (X. Wang & Yu, 2020). While the results
obtained were satisfactory, the significant difference between the original and final
videos was not easily discernible for those particular images. Therefore, they were
not utilized as the final videos. However, some codes were not able to generate high-
quality images (Yang et al., 2022), so a website was used to generate different styles for
the characters1 . Below in figure 4.5, the cartoony, painted, and original styles used in
the user study can be found.
1 https://toonme.com/
44
(a) Cartoonish (b) Painted
(c) Original
While the voices were not a primary focus of the study, we used a robotic voice
generated with the help of a Website to discuss topics related to children and schools.
The original version of the text focused on child abuse in schools. However, after
the production of the video, it was discovered that the content had the potential to
cause distress among participants. We aimed to create a positive feeling and avoid any
references to child abuse that could negatively affect participants. Furthermore, the use
of children’s voices in the video was deemed ethically inappropriate, resulting in a lack
of credibility. In addition, the text had to be read in a brief span of 10 seconds. These
issues were resolved by modifying the text and replacing the robotic child sound.
For the anchor condition video, the frequency of the audio was altered to create an
unsettling effect, while still utilizing the same audio. The final text was as follows:
"Today was a fantastic day at school. My teachers were supportive and encouraging,
and my classmates were friendly. I was introduced to a lot of new concepts that really
sparked my curiosity.". Used audio both for Donya and Kian can be found below:
45
Items Acronyms
MakeItTalk M
Models PC-AVS (Pose Controllable) P
FOMM (First Order) F
Character Donya (girl) D
Kian (boy) K
Original O
Style Cartoony C
Painted P
Finally, 19 videos ( 1 anchor condition + 18 main videos ) were generated with all the
pictures using three different models. Table 4.3 contains the necessary abbreviations
for each character, model, and style, which are required to tag the videos appropriately
for analysis. Having these tags is crucial to be able to conduct a thorough analysis of
the videos.
Table 4.4 is created to summarize all video combinations, and each video was assigned
a specific code to facilitate analysis.
All the videos were uploaded to YouTube to provide links for the questionnaire. This is
the YouTube channel link and all the videos can be found Here2 . Screenshots of some
of the videos can be seen in Figure 4.8.
2 https://www.youtube.com/channel/UCsbhzj0XIHtkP3mXfr6ovcg
46
Model Gender Style Video Name
M D P MDP
M D C MDC
M D O MDO
P D P PDP
P D C PDC
P D O PDO
F D P FDP
F D C FDC
F D O FDO
M K P MKP
M K C MKC
M K O MKO
P K P PKP
P K C PKC
P K O PKO
F K P FKP
F K C FKC
F K O FKO
47
Figure 4.8: Screenshots of some of the videos on YouTube.
In addition to the 18 videos, an anchor condition, which was the first video, was
added. An anchor condition is a technique used in research to ensure the accuracy
and reliability of collected data. It involves introducing a reference point or a known
answer that participants can use as a basis for their responses. This helps to minimize
the influence of irrelevant factors, such as random or uninformative numbers, that may
bias the responses of the participants (Chapman & Johnson, 1994).
In the user study conducted, an anchor condition was employed to validate the
48
responses collected from participants in a video questionnaire. The questionnaire
aimed to assess the participants’ perceptions of a video with poor sound quality.
By employing an anchor condition, we were able to enhance the validity and reliability
of the collected data, which is essential for drawing meaningful conclusions from
research studies. By introducing a reference point or a known answer, researchers
can minimize the influence of irrelevant factors and ensure that participants provide
honest and accurate responses.
In summary, all the details about the questionnaires, both before and after participants
answered them, have been presented in this chapter. The organization of the
questionnaires and their materials was discussed, followed by a comprehensive
elaboration of the questions and their individual purposes. The demographics of
the participants and the methods employed to collect the answers are subsequently
outlined.
49
50
Chapter 5
The data gathered from users was analyzed using SPSS (Statistical Package for the
Social Sciences), a powerful tool that enables the illustration and tabulation of data,
and provides a more comprehensive understanding of it.1
SPSS is a software package widely used in social science research and statistical
analysis. It allows researchers to manipulate and analyze large datasets with a wide
variety of statistical techniques, including descriptive statistics, correlation analysis,
regression analysis, factor analysis, and many others.Therefore, tables were generated
through SPSS to discuss the results, which will be provided below. In this study,
the GLM Repeated Measure test in SPSS was selected to examine the data, as it
offers enhanced insights based on the specified input parameters. Repeated measures
analysis in General Linear Model within SPSS involves the examination of data
where the same subjects are measured under multiple conditions. This method takes
into consideration the correlated nature of the repeated measurements within each
subject, enabling the exploration of within-subject effects and interactions between
factors. Repeated measures analysis in SPSS GLM is particularly valuable for studying
the impact of different interventions, assessing variations across related conditions,
or analyzing the effects of manipulated variables on the same subjects. It allows
for the investigation of main effects and interactions between factors, providing a
comprehensive understanding of how variables influence each other within the same
subjects across various conditions.
Initially, different parameters were defined in the SPSS to conduct the study. Six
main questions were asked about each video, each targeting an important feature
of the talking head avatars. The overall quality of experience (OverallQoE) was
measured, as well as the comfort level of users while watching the videos, which was
1 https://www.ibm.com/products/spss-statistics
51
summarized as Comfortable. The remaining features that were assessed were lipSync,
Talking, HeadMovement, and videoQuality.Table 4.1 contains the questions related
to each feature, providing a comprehensive overview of the questions asked during
the study. Following this, models, styles, and characters were defined to enable the
analysis of data in relation to each of these parameters. This step was necessary to
effectively analyze and interpret the data collected and to gain a more comprehensive
understanding of the users’ experiences with the talking head avatars. During the
user study, participants were provided with choices ranging from "Bad" to "Excellent."
However, to facilitate result analysis, it was necessary to convert these qualitative
responses into numeric values. Accordingly, we assigned a value of 1 to "Bad," 2 to
"Poor," 3 to "Fair," 4 to "Good," and 5 to "Excellent." By doing so, we were able to
observe and interpret the results using numerical representations.
Additionally, all box plots in the following pages were created using the PowerBI
software 2 , known for its strong data visualization capabilities. The software’s
powerful dashboard allows for displaying data details effectively. The data was
gathered from Microsoft Forms, provided in Excel format, and then imported into
PowerBI. By transforming the data, various plots were generated using the PowerBI
software. These box plots display the range of participants’ insights, and the black dot
inside each box represents the average answer provided by the users.
In our study, three main factors were focused on: models, characters, and styles, as
mentioned earlier. For each factor, we had six different features to look at. The results
obtained for each of these factors in relation to the other variables will be presented in
the following pages. The objective is to identify and analyze the main effects of these
factors within the context of our research.
52
user satisfaction rating. It is important to highlight that the PKO video emerged as the
winner in terms of the best overall Quality of Experience (QoE), as observed in the box
plot. On average, it received a favorable rating of 3.43 from participants. This video
was generated using the PC-AVS model and featured the character Kian (boy) with
the Original style. Other videos created using the PC-AVS model showed minimal
differences compared to the winning PKO video. The MKO video also performed
well, obtaining a respectable score of 3.14. Similarly, the FDC video, produced by the
FOMM model, achieved a decent rating of 2.95. However, the FKO video, which was
a combination of the FOMM model, Kian character, and Original style, received the
lowest score of 2.14. Furthermore, the PKO video received the highest comfortability
rating of 3.32, closely followed by the second video, PDP (PC-AVS, Donya, Painted
style), with only a slight difference between their comfortability scores. This can be
seen in part (b) of the box plot (Figure 5.1). Once again, the FKO video had the
lowest comfortability performance among all the videos, garnering a score of 2.05. The
analysis conducted using SPSS corroborates the patterns observed in the Power BI box
plots.
53
(a) OverallQoE
(b) Comfortable
54
Regarding LipSync, box plot 5.2, part (a), illustrates that the participants perceived
three videos to be almost identical. These videos were generated using the PC-AVS
model and featured Kian, regardless of the style used. Their score was 3.68 which is a
good score. However, the FKO video, which was generated using the FOMM model
and featured the Original Kian style, had very poor performance in this aspect.
In the analysis conducted on the LipSync feature data through SPSS, the assumption
of sphericity was found to be violated by Mauchly’s test (χ2 (2) = 6.274, p = 0.043),
prompting us to adjust degrees of freedom using Huynh-Feldt estimates (ε = 0.898).
A significant main impact stemming from Model emerged, with an F(1.796, 64.653)
value of 73.959, p < 0.001, and a noteworthy partial η 2 effect size of .746. Follow-
up assessments unveiled that LipSync scores were significantly higher in the PC-AVS
compared to both MakeItTalk (p < 0.001) and FOMM (p < 0.001). Moreover, significant
differences existed between ratings of MakeItTalk and FOMM (p < 0.001).
In terms of HeadMovement, the participants perceived the PDP video to have the
best performance with a score of 3.27. This video was generated using the PC-AVS
model and featured Painted Kian. This information is illustrated in box plot 5.2,
Part (b). MDO which is the video made by MakeItTalk model, Donya character,
and Original style had a very poor performance with a score of 2.14. Additionally,
A repeated-measures ANOVA was conducted with the aim of comprehending the
influence of Model on HeadMove scores. In this regard, Mauchly’s test indicated
that the assumption of sphericity was met, χ2 (2) = 2.856, p = 0.240. There was a
significant main effect of Model, F(2, 35) = 19.562, p < 0.001, partial η 2 = 0.352. Post
hoc tests showed HeadMove scores were significantly higher at PC-AVS compared to
MakeItTalk (p < 0.001) and FOMM (p < 0.001). scores at MakeItTalk and FOMM did
not significantly differ. In terms of the Talking feature, the box plot analysis revealed
that the PKO video had the most positive impact on the participants, obtaining an
impressive score of 3.5. Additionally, all the other videos created by the PC-AVS
model outperformed the videos produced by the two other models. MKO, the original
Kian made by MakeItTalk model, and FDC video which we know is Cartoony Donya
made by FOMM model also did a good job among two other models. For quality
factor Talking, the repeated-measures ANOVA yielded nearly an identical results.
Mauchly’s test indicated that the assumption of sphericity was met, χ2 (2) = 0.860, p
= 0.650 that prompting us to adjust degrees of freedom using Huynh-Feldt estimates
(ε = 1.000). There was a significant main effect of Model, F(2, 35) = 48.506, p < 0.001,
partial η 2 = 0.574. Post hoc tests showed Talking scores were significantly higher at
PC-AVS model compared to MakeItTalk model (p < 0.001) and FOMM (p < 0.001).
Scores at MakeItTalk model and FOMM did not significantly differ. The last feature
to be evaluated in this study is the quality of the videos. It was observed that PKO
55
received the highest quality rating from the participants, with an average score of
3.43. Other videos generated by the PC-AVS model also received positive feedback
from the participants, indicating good satisfaction. This can be seen in the box plot
depicted in Figure 5.3, with Parts (a) and (b) showing the distribution of quality
scores. By looking closely at analysis coming from a repeated-measures ANOVA to
evaluate the effect of Model on VideoQuality scores, Mauchly’s test indicated that the
assumption of sphericity was violated, χ2 (2) = 5.826, p = 0.054, so degrees of freedom
were corrected using Huynh-Feldt estimates (ε = 0.907). There was a significant main
effect of Model, F(1.814, 65.308) = 60.269, p < 0.001, partial η 2 = 0.626. Post hoc tests
showed VideoQuality scores were significantly higher at PC-AVS model compared to
MakeItTalk (p < .001) and FOMM (p < .001). Scores at MakeItTalk model and FOMM
model did not significantly differ. In summary, for an overarching perspective on the
videos’ performance concerning user experience, it is beneficial to refer to Table 5.1
which presents the mean and standard deviation of data across all six features.
56
(a) LipSync
57
(a) Talking
58
5.2 The main effect of the characters
As mentioned before, two characters, Kian (boy) and Doya (girl), were used in the
videos. The box Plot 5.4 (a) displays the OverallQoE of the participants towards the
videos made with each character separately. It can be observed that there was not
much difference between the two characters in terms of the final result.In the PC-
AVS model, an average score of 3.36 was achieved by Kian, while Donya obtained
a slightly lower average score of 3.30. In the MakeItTalk model, Kian exhibited a
better performance compared to Donya. However, in the FOMM model, both Kian
and Donya demonstrated identical performance levels.
Examining the results derived from the repeated measures ANOVA analysis, it also
approves that the main effect of the Character variable did not yield statistical
significance in the context of the OverallQoE feature (F(1, 36) = 2.235, p = 0.144, partial
η 2 = 0.058). Regarding comfortability, individuals tend to experience slightly more
comfort with Kian in the PC-AVS and MakeItTalk model, whereas Donya performed
better in the FOMM model. These findings can be observed in the corresponding charts
of our analysis 5.4 (b). However, in the repeated measure ANOVA these differences
were shown to not be significantly different. The main effect of Character was not
significant, F(1, 36) = 0.225, p = 0.638. This indicates there was no significant difference
in comfortability scores between the two characters overall, when collapsing across
model and style.
In terms of the lipsync feature, Kian exhibited better performance in both the PC-
AVS and MakeItTalk models, with scores of 3.68 and 2.76, respectively. Donya also
performed well, achieving a score of 3.46 in the PC-AVS model, but showed weaker
performance in the MakeItTalk model. The lowest lipsync score was 2.08, which was
attributed to Kian in the FOMM model. These findings can be observed 5.5 Box plot
part (a). However, the repeated measure ANOVA showed the main effect of Character
on quality factor LipSync was not significant, F(1, 36) = 2.814, p = 0.102, partial η 2
= 0.073. In the HeadMove feature, both Kian and Donya demonstrated nearly equal
performance in the PC-AVS and FOMM models, achieving scores of 3.10 in PC-AVS.
However, in the MakeItTalk model, Kian exhibited slightly better performance. These
findings are illustrated in Box plot (b) of Figure 5.5. By looking at the values that come
from repeated measure ANOVA, the main effect of Character was not significant in
this feature as expected ( F(1, 36) = 0.035, p = 0.853, partial η 2 = 0.001.).
59
(a) OverallQoE
(b) Comfortable
Figure 5.4: The main effect of the characters - OverallQoE and Comfortability
60
(a) LipSync
Figure 5.5: The main effect of the characters - LipSync and HeadMove
61
(a) Talking
Figure 5.6: The main effect of the characters - Talking and VideoQuality
62
Regarding the talking feature, the box plots indicate that individuals tend to have
a more comfortable experience with Kian when using the PC-AVS and MakeItTalk
models. For example, Kian achieved a score of 3.31 for the PC-AVS model and 2.77 for
the MakeItTalk model. On the other hand, Donya showed a better performance in the
FOMM model with a score of 2.5, compared to Kian’s score of 2.2. An examination of
the data using repeated measures ANOVA also sheds light on the fact that the main
effect of Character was not significant, F(1, 36) = 0.223, p = 0.639, partial η 2 = 0.006.
The final feature we will explore is the video quality. The results for this aspect were
consistent with the previous features, where the PC-AVS model emerged as the winner
for both characters. Kian, in particular, performed exceptionally well with a score of
3.31. The MakeItTalk model also demonstrated good performance in terms of video
quality, achieving a score of 2.77 for Kian. These findings are visually represented
in the corresponding box plots (a) and (b) of our analysis 5.6. Actually, the findings
obtained through the utilization of repeated measures ANOVA also demonstrate that
the main effect of Character was not significant, F(1, 36) = 3.493, p = 0.070, partial η 2 =
0.088.
In summary, the table 5.2 shows the means and standard deviations for the 6 evaluated
features across the two characters. For most features, the means are fairly similar
between the two characters. The biggest difference is for OverallQoE, where Kian has
a higher mean rating (2.93) compared to Donya (2.85). This indicates participants rated
the overall quality of experience for Kian as mildly better than for Donya. For other
features like LipSync, Comfortable, Talking, HeadMove, and VideoQuality the means
are very close between the two characters, with at most a 0.13 difference in means.
The standard deviations are also comparable between the two characters, showing a
similar amount of variance in scores for both Donya and Kian.
Donya Kian
Features Mean StandardDeviation Mean StandardDeviation
LipSync 2.70 0.81 2.82 1.05
Comfortable 2.71 0.13 2.73 0.13
Talking 2.72 0.12 2.75 0.12
HeadMove 2.70 0.13 2.69 0.12
VideoQuality 2.75 0.12 2.83 0.13
OverallQoE 2.85 0.12 2.93 0.12
63
5.3 The main effect of the Styles
In this user study, three different styles were used including Original, Painted, and
Cartoony as mentioned before. In the feature of overallQoE, it can be observed that
the PC-AVS model exhibited consistent performance across all three styles. The slight
variation in performance was observed in the original style with a score of 3.38, but
the difference is negligible. Similarly, the MakeItTalk model demonstrated uniform
performance across all styles, while the FOMM model showed lower performance in
the original style with a score of 2.23.
The outcomes emerging from the analysis conducted via repeated measures ANOVA
revealed that there was a significant main effect of Style on quality factor overallQoE,
F(2, 35) = 5.368, p = 0.010, partial η 2 = 0.130. Regarding the comfortability factor, the
results align with the findings from previous features, which are depicted in the box
plots shown in figure 5.7 parts (a) and (b). There was a significant main effect for Style,
F(2, 35) = 4.068, p = 0.026, partial η 2 = 0.189 regarding this feature.
In the lipsync factor, all three styles exhibited similar performance, achieving an
average score of approximately 3.5 in the PC-AVS model. However, in the MakeItTalk
model, the cartoony style demonstrated slightly better performance with a score of
2.61. Interestingly, the FOMM model also achieved the same score of 2.61 in the
cartoony style. However, the FOMM model performed poorly in the original style,
obtaining a score of 1.74, which was one of the lowest performances among all features,
styles, characters, and models. The repeated measure ANOVA shows a trend for the
main effect of Style on lipsync, with F(2, 35) = 3.218, p = 0.052, and a partial η 2 of 0.155.
It’s evident that the p-value closely approaches the significance threshold.
In the HeadMove feature, the cartoony style exhibited slightly better performance
across all models, although the differences were very minimal. The scores for the three
styles in the three models were very close to each other in this feature. For a better
understanding of the results, all the details are presented in the figure 5.8 part (b).
By looking at the repeated measure ANOVA analysis, there was no significant main
effect for Style, F(2, 35) = 2.215, p = 0.147, partial η 2 = .058 in term of the HeadMove
feature. In the next two features, which are talking and video quality, the PC-AVS
model demonstrated better performance. In the video quality feature, the painted
style obtained a slightly higher score compared to the other two styles, while in the
talking feature, the original style outperformed the others. The MakeItTalk model
exhibited consistent performance in both features and across all three styles. However,
the FOMM model showed weak performance in the original style for both features.
The evidence supporting this observation can be found in the box plots provided in
64
Figures 5.9 part (a) and (b). Looking at the results come from SPSS, no substantial
primary impact was observed for Style, as indicated by the results of F(2, 35) = 2.473,
p = 0.116, with a partial η 2 of 0.064, concerning the videoQuality feature. Similarly, for
the talking feature, no significant primary effect for Style was apparent, with F(2, 35) =
2.714, p = 0.091, and a partial η 2 of 0.070. The following table 5.3 presents the means
and standard deviations of various features across different styles: Cartoony, Painted,
and Original. Each cell displays the mean and standard deviation values for a specific
feature under each style.
65
(a) OverallQoE
(b) Comfortable
Figure 5.7: The main effect of the Styles - OverallQoE and Comfortability
66
(a) LipSync
Figure 5.8: The main effect of the Styles - LipSync and HeadMove
67
(a) Talking
Figure 5.9: The main effect of the Styles - Talking and VideoQuality
68
5.4 The interactions among Models, Characters, and
Styles
Following an examination of the primary effects associated with individual models,
characters, and styles, our attention turns towards comprehending the significance
encompassing the intricate interplay between these variables. This comprehensive
analysis is facilitated through the application of repeated measure ANOVA within
the SPSS framework. The insights drawn from this analysis are a culmination of
meticulous investigations performed for each distinctive attribute. Subsequently,
an intricate breakdown of each feature ensues, commencing with the evaluation of
"overallQoE." Within this context, it is imperative to highlight that the interaction
between Model and Character yielded a notable degree of significance, with an F(2,
35) = 12.487, p < 0.001, along with a partial η 2 value of 0.258. Correspondingly, the
interaction between Model and Style also exhibited significance, as indicated by an
F0(4, 33) = 5.368, p = 0.002, accompanied by a partial η 2 value of 0.157. Furthermore, the
interaction between Character and Style yielded a significant outcome, underscored
by an F(2, 35) = 6.002, p = 0.006, and a partial η 2 value of 0.255. However, it’s
paramount to recognize that the pinnacle of significance was achieved within the
three-way interaction encompassing Model, Character, and Style. This interaction was
underscored by a compelling F(4, 33) = 4.810, p = 0.004, along with a substantial partial
η 2 value of 0.368. It’s intriguing to note that while the character factor alone did not
exhibit a significant effect, its interactions with other components such as models or
styles magnified its importance and led to notable impacts on users’ experiences. This
phenomenon underscores the intricate nature of user perception, where seemingly
subtle interactions between variables can yield substantial and meaningful outcomes
in shaping the overall user experience.
Analyzing the impact of head movement in our study, the interaction between Model
and Character emerged as significant, yielding an F(2, 35) = 3.569, p = 0.040, with
a partial η 2 value of 0.090. However, the interactions between Model and Style, as
well as Character and Style, did not exhibit significance. Interestingly, the highest
level of significance was achieved through the three-way interaction involving Model,
Character, and Style. Here, the F-statistic recorded a compelling value of (4, 33) =
6.035, p = 0.001, with a substantial partial η 2 value of 0.422. These findings underscore
how the collaborative influence of these variables plays a pivotal role in shaping
users’ perceptions and engagement in relation to head movement within talking head
avatars.
Examining the Talking feature, we observed that the interaction between Model and
69
Character yielded significant results, with an F(2, 35) = 8.445, p = 0.001, and a partial
η 2 value of .190. However, the Model * Style interaction did not exhibit significance.
Notably, the Character * Style interaction displayed significance, underlined by an
F(2, 35) = 5.209, p = 0.010, and a partial η 2 value of 0.229. Conversely, the three-
way interaction involving Model, Character, and Style did not attain significance, as
indicated by an F(4, 33) = 3.747, p = 0.013, and a partial η 2 value of 0.312.
For the analysis of VideoQuality, the interaction between Model and Character yielded
remarkable significance, marked by an F(2, 35) = 11.020, p < 0.001, and a partial η 2
value of 0.234. Similarly, the Model * Style interaction displayed significance, with an
F-statistic of (4, 33) = 3.398, p = .028, and a partial η 2 value of 0.273. Furthermore,
the Character * Style interaction revealed significance, denoted by an F(2, 35) = 6.892,
p = 0.003, and a partial η 2 value of 0.283. The three-way interaction encompassing
Model, Character, and Style achieved considerable significance, recording an F(4, 33)
= 4.119, p = 0.008, and a partial η 2 value of 0.333. These findings highlight the intricate
interplay between these variables in influencing users’ perception of video quality in
talking head avatars.
Analyzing the LipSync feature, the interaction between Model and Character yielded
profound significance, with an F(2, 35) = 15.612, p < 0.001, and a substantial partial
η 2 value of 0.471. Similarly, the Model * Style interaction exhibited significance, as
indicated by an F(4, 33) = 6.012, p = 0.001, and a partial η 2 value of 0.422. The Char *
Style interaction also yielded noteworthy significance, marked by an F(2, 35) = 12.159,
p < 0.001, and a partial η 2 value of 0.410. The highest level of significance, however,
was achieved through the three-way Model * Character * Style interaction, with an
F(4, 33) = 10.126, p < 0.001, and a substantial partial η 2 value of 0.551. These findings
underscore the collaborative significance of these factors in shaping users’ perceptions
of lip sync accuracy within talking head avatars.
Finally, examining the feature of comfortability, the interaction between Model and
Character demonstrated significant influence, with an F(2, 35) = 9.627, p < 0.001, and a
noteworthy partial η 2 value of 0.355. Similarly, the Model * Style interaction exhibited
significance, marked by an F(4, 33) = 5.032, p = 0.003, and a partial η 2 value of 0.379.
Furthermore, the Char * Style interaction displayed significance, with an F(2, 35) =
5.209, p = 0.010, and a partial η 2 value of 0.229. The three-way interaction involving
Model, Character, and Style also attained significance, as indicated by an F(4, 33) =
5.610, p = 0.001, and a substantial partial η 2 value of 0.405. These findings emphasize
the collective impact of these variables on users’ perceived comfort when interacting
with talking head avatars.
This concise summary table encapsulates the findings from our analyses of various
70
avatar features. For a comprehensive overview of the significance levels, F-statistics,
p-values, and partial η 2 values related to the interactions between Models, Characters,
and Styles, look at table 5.4
The participants were asked the following question in the final questionnaire:
"Generally, which avatars did have the most realistic appearance for you?" To present
71
the responses clearly, a donut plot was utilized 5.10, revealing that a significant 61% of
the participants favored the original style, perceiving it as the most realistic compared
to the other two styles.
The second general question inquired about the participants’ overall preference re-
garding the avatars: "Generally, which avatars do you like the most?". Understanding
users’ emotional responses and preferences towards avatar styles holds significance, as
these factors can influence trust and engagement during conversations. Surprisingly,
41% of the participants expressed a distinct fondness for the cartoony avatars, show-
casing a notable departure from concerns related to the uncanny valley phenomenon.
Additionally, approximately 39% of participants indicated a strong liking towards the
original avatars, revealing a preference that closely rivalled the appeal of the cartoony
style. These findings, presented in Figure 5.11 as a donut plot.
Figure 5.10: Perceived realism of avatars Figure 5.11: Overall favorite avatars
The next question asked participants, "Considering the avatars’ talking, which were
the most realistic?" This question focused on how real the avatars felt during
72
conversations, not just their appearance. Interestingly„ 52% of participants found the
cartoony avatars to be the most realistic, while 25% preferred the original pictures, and
22% liked the painted style. The distribution of responses is shown in Figure 5.13,
giving us the insight into how participants perceived the realism of different avatar
styles during interactions.
Figure 5.12: Preferred avatars for Figure 5.13: Perceived realism of avatars
computer-mediated conversations during speech
The last question, "Generally, which avatars were the most believable for you?", held
significant importance in achieving the goals of this study. Given the application of
our research in training psychologists and professionals to effectively communicate
and interview children who have experienced trauma or injury, the believability of
avatars becomes a crucial factor. The findings revealed that 52% of the participants
found the original style to be the most believable, followed by 30% who favored the
cartoony style, and 17% who found the painted style to be more believable. Figure 5.14
illustrates the distribution of responses among the participants, shedding light on their
perceptions of avatar believability.
73
In summary, this chapter has presented all the obtained results from the questionnaires,
offering a dual perspective. One approach involved the utilization of Power BI
box plots to enhance the visualization of participant responses. However, the more
significant aspect encompassed the analysis of answers through repeated measure
ANOVA, employing the SPSS software. This analysis provided valuable insights that
will be discussed in the subsequent chapter. The key effects of the model, character,
style, and their interactions were considered. Finally, a comprehensive understanding
of the uncanny valley was achieved through the visualization of donut plots in the
latter part of the questionnaire.
74
Chapter 6
The outcomes and findings chapter of this study zoomed in on how different factors
influenced what people thought about talking head avatars. The model used to create
the avatars made a big difference videos made with the pose controllable model stood
out as more convincing and appealing to participants. Interestingly, in some of the
features, the style of the avatars also played a role in shaping the results for certain
aspects. In this chapter, we will dig into the results and findings we uncovered. We
will also compare our initial expectations and guesses with what actually happened
in the study. Additionally, we will discuss the challenges we faced along the way. It
is important to recognize that our study covers a specific part of a larger field. By
addressing the limitations we encountered and making more accurate predictions in
future work, we can build upon what we’ve learned and make our research more
comprehensive.
RQ1: Which models can generate avatars that generally give a better feeling to the audience?
RQ2: What distinctions emerge in user experiences when interacting with real, cartoony,
and painted avatars? How do these differences influence viewers’ perceptions and emotional
responses toward these avatars?
We obtained a clear answer to the first question, with the pose controllable model
outperforming others in all six features, as illustrated in the box plots in the previous
section. This model prioritized refining both LipSync and HeadMovement, channeling
emphasis into enhancing the avatar’s overall pose. This emphasis on head movements
and proficient LipSync likely contributed to improved user-avatar communication.
75
Moreover, the commendable video quality associated with this model could have
further bolstered participants’ engagement and experiences, according to the author’s
perspective.
In regard to the avatar’s style, which was our second research question, we initially
expected it to strongly influence all features, as the "uncanny valley" effect suggests
that avatars resembling humans can feel strange. Although, the study showed that
style did not make a huge difference. Notably, the painted, cartoony and original style
avatars were the favorite among participants, showing that all avatars can still be well-
received.
This finding supports the concept of the uncanny valley effect. People liked the original
style avatars almost as much as the cartoony or painted styles and not more. It was
the initial hypothesis that perhaps more realistic avatars with original pictures would
receive higher scores. However, we observed that realism wasn’t the sole determining
factor in whether people liked the avatars or not. As highlighted in the donut plots
from the previous chapter, people even experienced a sense of realism when interacting
with avatars in a cartoony style. Interestingly, it’s worth noting that the perception of
realism wasn’t exclusively tied to avatars with original pictures. Other elements, such
as lipsync and head movement, played substantial roles in creating believable avatars.
This suggests that the overall convincing quality of an avatar goes beyond its visual
attributes alone.
Moreover, the character’s gender did not seem to have a strong influence on people’s
preferences. Both boy and girl avatars were equally favored by the participants.
This outcome was expected, as the boy character, despite being generated with GAN
and not representing a real person, demonstrated remarkable performance, making it
difficult for the human eye to discern any differences between the deepfake character
and an actual individual. Therefore, our prediction proved accurate, and there were
no significant discernible differences between the boy and girl characters.
The video made by the PC-AVS model, named PKO, along with Kian’s original style,
emerged as the winner in terms of overallQoE. This feature was of utmost importance
in our evaluation. Therefore, we can confidently assert that PKO was the best video
among all. Additionally, it also outperformed others in the comfortability and lipsync
features. However, in the HeadMove feature, the video created by the PC-AVS model,
PDP, featuring Donya in a painted style, achieved significantly better results, with
PKO securing the second position in this category. Furthermore, PKO’s superior
performance in the talking feature, coupled with its commendable video quality,
solidified its position as the overall winner in this study.
Our findings lead to the conclusion that in the realm of talking head avatars designed
76
for both children and adults, the observation consistently highlights the significance of
model quality. This quality directly influences factors like lip sync performance, audio
synchronization, video quality, and, most importantly, the overall comprehensive user
experience.
Focusing on avatars created specifically for child abuse interviewing, a previous study
within the child abuse avatar project revealed that experts emphasize the substantial
impact of realism on outcomes(Hassan, Salehi, Røed et al., 2022). Aligning this
with our broader observations, it’s apparent that directing attention towards realism
entails enhancing multiple facets of the avatars. This encompasses not only the visual
appearance but also other features predominantly influenced by the models, which
significantly contribute to achieving a heightened sense of realism.
77
included there..
Ultimately, we identified models that were suitable for our benchmarking task based
on their accessibility and ease of use via Google Colab links. Despite encountering
challenges when executing these selected models, most issues were solvable. For
instance, some models were coded with outdated library commands that required
modification to function properly with newer library versions. These conflicts were
identified and resolved, ensuring the successful execution of the code. While our
journey presented numerous hurdles, our commitment to addressing and overcoming
each challenge allowed us to advance through the benchmarking process.
Another challenge revolved around the selection of characters for our study. The
decision-making process involved choosing between utilizing real individuals or
employing characters generated by GAN style which at the end, we have used both
of them. Various characters were tested in conjunction with the videos to identify
the most suitable match. A crucial aspect that contributed to achieving favorable
outcomes was the preference for images where the person was looking directly
forward. Furthermore, the task of selecting appropriate styles posed its own set of
challenges. The chosen styles needed to exhibit believability while simultaneously
exhibiting distinct differences from one another.
Initially, our attempt to source styles from a GitHub repository proved unfavorable,
as the resulting output suffered from poor quality. Ultimately, the solution emerged
in the form of generating cartoony and painted images through a dedicated website,
which we introduce in the third chapter of this thesis. This strategic shift in approach
facilitated the creation of styles that met the desired criteria and contributed positively
to the overall quality and credibility of our study’s results.
78
Chapter 7
Conclusion
The primary aim of this thesis is to thoroughly evaluate talking head models based on
user experiences and the level of realism they produce. This goal carries a critical
purpose: to enhance the realism of a platform, which is intended for use by law
enforcement and child protective service professionals to practice interview training
with children who have endured trauma or abuse. This effort constitutes a segment of
a larger project, wherein the development of talking head avatars serves as a platform
for expert training before conducting actual interviews. The importance of this training
stems from the sensitivity associated with interviewing such children. It is hoped that
the results of this study can contribute, even if in a small way, to the enhancement of
talking head avatars’ communication capabilities with users.
79
key consideration is mitigating the uncanny valley effect, even for cartoonish or
painted styles. This study highlighted the substantial influence of the uncanny
valley phenomenon. Surprisingly, the cartoon and painted styles could compete
with the more traditional original photo style by circumventing the uncanny
valley.
• One of the early resolutions formed during the brainstorming stage of this study
was to compile a list of video-driven models and subsequently compare them
with audio-driven models by conducting another user study. This comparative
analysis aimed to shed light on the differences between these two model types.
An additional intention was to conduct a separate user study to enable a
comprehensive comparison of the outcomes derived from these distinct model
groups. However, due to the time-consuming nature of these tasks, this phase
had to be omitted from the current investigation.
• In exploring this field, there are several aspects that experts could think about.
One thing to think about is adding more different models or trying out different
looks for the same character. This could be interesting for future research. Also,
our study mainly looked at how people felt when using these avatars. But there’s
another possibility we can explore: looking at how well the models work using
some extra ways to measure their performance that we have. This could give us
more insights. This idea would help us see how the models perform compared
to how users feel about them. This would help us understand more about this
area in a deeper way.
80
Appendix A
Questionnaire 1
1 https://forms.office.com/r/2b3teM8EBQ
81
Benchmarking Talking-Heads
Models:
(First Part)
Welcome!
The aim of this study is to get your opinion on different artificial faces. Imagine a scenario where
you are speaking with a robot that has an artificial face. You will watch a series of videos of these
avatars and then you are asked to answer some questions. Your responses to the questionnaires
will be used for scientific research, please treat them very seriously to contribute with valid
results. There is no right or wrong answer as we are looking for your opinion. The estimated time
duration of the study is 15-20 minutes.
0 1 2 3 4 5 6 7 8 9 10
Yes
No
7
Please write down the 5-digit number from the audio file to confirm that
you have access to a proper speaker or headphone! *
CAPTCHA
Ready
9
AC: Please answer these questions after watching the video bellow: *
Noisy
How was
your overall
experience
with the
avatar?
How
accurately did
the lips move
in sync with
the audio?
How do you
rate the
naturalness
of head
movement?
How realistic
the avatar
was talking?
was talking?
How do you
rate the
overall
quality of this
video?
How
comfortable
do you feel
conversing
with this
avatar?
Appendix B
Questionnaire 2
1 https://forms.office.com/r/0zWNMGJTVC
89
Benchmarking Talking-Heads
Models
(second part)
Post-test Questionnaire: Answering final questions and submitting the form.
* Required
0 1 2 3 4 5 6 7 8 9 10
This content is neither created nor endorsed by Microsoft. The data you submit will be sent to the form owner.
Microsoft Forms
Bibliography
Abdullah, M., Madain, A., & Jararweh, Y. (2022). Chatgpt: Fundamentals, applications
and social impacts. 2022 Ninth International Conference on Social Networks
Analysis, Management and Security (SNAMS), 1–8. https : / / doi . org / 10 . 1109 /
SNAMS58071.2022.10062688
Barman, N., Reznik, Y., & Martini, M. (2023). Datasheet for subjective and objective
quality assessment datasets. arXiv preprint arXiv:2305.02142.
Baugerud, G. A., Johnson, M. S., Klingenberg Røed, R., Lamb, M. E., Powell, M.,
Thambawita, V., Hicks, S. A., Salehi, P., Hassan, S. Z., Halvorsen, P., & Riegler,
M. A. (2021). Multimodal virtual avatars for investigative interviews with
children. Proceedings of the 2021 ACM Workshop on Intelligent Cross-Data Analysis
and Retrieval, 2–8. https://doi.org/10.1145/3463944.3469269
Bertram, D. (2007). Likert scales. Retrieved November, 2(10), 1–10.
Chapman, G. B., & Johnson, E. J. (1994). The limits of anchoring. Journal of Behavioral
Decision Making, 7(4), 223–242.
Chen, L., Cui, G., Kou, Z., Zheng, H., & Xu, C. (2020). What comprises a good talking-
head video generation?: A survey and benchmark.
Cunningham, P., Cord, M., & Delany, S. J. (2008). Supervised learning. In Machine
learning techniques for multimedia: Case studies on organization and retrieval (pp. 21–
49). Springer.
Dike, H. U., Zhou, Y., Deveerasetty, K. K., & Wu, Q. (2018). Unsupervised learning
based on artificial neural network: A review. 2018 IEEE International Conference
on Cyborg and Bionic Systems (CBS), 322–327. https://doi.org/10.1109/CBS.
2018.8612259
Eskimez, S. E., Zhang, Y., & Duan, Z. (2021). Speech driven talking face generation
from a single image and an emotion condition. IEEE Transactions on Multimedia,
24, 3480–3490.
Ettinger, T. R. (2022). Children’s needs during disclosures of abuse. SN Social Sciences,
2(7), 101. https://doi.org/10.1007/s43545-022-00397-6
Girden, E. R. (1992). Anova: Repeated measures. sage.
93
Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S.,
Courville, A., & Bengio, Y. (2014). Generative adversarial networks.
Hamzelou, J. (2017). Robots could help children give evidence in child abuse cases.
New Scientist, 233(3118), 20. https://doi.org/10.1016/S0262-4079(17)30305-3
Hassan, S. Z., Salehi, P., Riegler, M. A., Johnson, M. S., Baugerud, G. A., Halvorsen, P., &
Sabet, S. S. (2022). A virtual reality talking avatar for investigative interviews of
maltreat children. Proceedings of the 19th International Conference on Content-Based
Multimedia Indexing, 201–204. https://doi.org/10.1145/3549555.3549572
Hassan, S. Z., Salehi, P., Røed, R. K., Halvorsen, P., Baugerud, G. A., Johnson, M. S.,
Lison, P., Riegler, M., Lamb, M. E., Griwodz, C., & Sabet, S. S. (2022). Towards an
ai-driven talking avatar in virtual reality for investigative interviews of children.
Proceedings of the 2nd Workshop on Games Systems, 9–15. https : / / doi . org / 10 .
1145/3534085.3534340
Htike, K. (2017). A review on data-driven learning of a talking head model.
International Journal of Intelligent Systems Technologies and Applications, 16, 169.
https://doi.org/10.1504/IJISTA.2017.084239
Igaue, T., & Hayashi, R. (2023). Signatures of the uncanny valley effect in an artificial
neural network. Computers in Human Behavior, 146, 107811. https://doi.org/
https://doi.org/10.1016/j.chb.2023.107811
Kammoun, A., Slama, R., Tabia, H., Ouni, T., & Abid, M. (2022). Generative adversarial
networks for face generation: A survey. ACM Comput. Surv., 55(5). https://doi.
org/10.1145/3527850
Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., & Aila, T. (2020). Analyzing
and improving the image quality of stylegan. Proceedings of the IEEE/CVF
conference on computer vision and pattern recognition, 8110–8119.
Kingma, D. P., & Welling, M. (2019). An introduction to variational autoencoders.
Foundations and Trends® in Machine Learning, 12(4), 307–392. https://doi.org/
10.1561/2200000056
Labs, E. (2023). Eleven labs - text to speech api [[Online; accessed 27-March-2023]].
Lambie, G. W. (2005). Child abuse and neglect: A practical guide for professional school
counselors. Professional School Counseling, 8(3), 249–258. Retrieved March 22,
2023, from http://www.jstor.org/stable/42732466
Lu, Y., Chai, J., & Cao, X. (2021). Live Speech Portraits: Real-time photorealistic talking-
head animation. ACM Transactions on Graphics, 40(6). https://doi.org/10.1145/
3478513.3480484
MacDorman, K. F., Green, R. D., Ho, C.-C., & Koch, C. T. (2009). Too real for comfort?
uncanny responses to computer generated faces [Including the Special Issue:
Enabling elderly users to create and share self authored multimedia content].
94
Computers in Human Behavior, 25(3), 695–710. https://doi.org/https://doi.org/
10.1016/j.chb.2008.12.026
Meskys, E., Kalpokiene, J., Jurcys, P., & Liaudanskas, A. (2020). Regulating deep fakes:
Legal and ethical considerations. Journal of Intellectual Property Law & Practice,
15(1), 24–31.
Mildenhall, B., Srinivasan, P. P., Tancik, M., Barron, J. T., Ramamoorthi, R., & Ng, R.
(2021). Nerf: Representing scenes as neural radiance fields for view synthesis.
Commun. ACM, 65(1), 99–106. https://doi.org/10.1145/3503250
Mori, M., MacDorman, K. F., & Kageki, N. (2012). The uncanny valley [from the field].
IEEE Robotics & automation magazine, 19(2), 98–100.
Norman, R. E., Byambaa, M., De, R., Butchart, A., Scott, J., & Vos, T. (2012). The long-
term health consequences of child physical abuse, emotional abuse, and neglect:
A systematic review and meta-analysis. PLoS Medicine, 9.
Perarnau, G., van de Weijer, J., Raducanu, B., & Álvarez, J. M. (2016). Invertible
Conditional GANs for image editing. NIPS Workshop on Adversarial Training.
Powell, M. B., Guadagno, B., & Benson, M. (2016). Improving child investigative
interviewer performance through computer-based learning activities. Policing
and Society, 26(4), 365–374. https://doi.org/10.1080/10439463.2014.942850
Prajwal, K. R., Mukhopadhyay, R., Namboodiri, V. P., & Jawahar, C. (2020). A lip sync
expert is all you need for speech to lip generation in the wild. Proceedings of the
28th ACM International Conference on Multimedia, 484–492. https://doi.org/10.
1145/3394171.3413532
Ren, Y., Li, G., Chen, Y., Li, T. H., & Liu, S. (2021). Pirenderer: Controllable portrait
image generation via semantic neural rendering.
Salehi, P., Hassan, S. Z., Lammerse, M., Sabet, S. S., Riiser, I., Røed, R. K., Johnson,
M. S., Thambawita, V., Hicks, S. A., Powell, M., Lamb, M. E., Baugerud, G. A.,
Halvorsen, P., & Riegler, M. A. (2022a). Synthesizing a talking child avatar to
train interviewers working with maltreated children. Big Data and Cognitive
Computing, 6(2), 62. https://doi.org/10.3390/bdcc6020062
Salehi, P., Hassan, S. Z., Lammerse, M., Sabet, S. S., Riiser, I., Røed, R. K., Johnson,
M. S., Thambawita, V., Hicks, S. A., Powell, M., Lamb, M. E., Baugerud, G. A.,
Halvorsen, P., & Riegler, M. A. (2022b). Synthesizing a talking child avatar to
train interviewers working with maltreated children. Big Data and Cognitive
Computing, 6(2). https://doi.org/10.3390/bdcc6020062
Salehi, P., Hassan, S. Z., Shafiee Sabet, S., Astrid Baugerud, G., Sinkerud Johnson, M.,
Halvorsen, P., & Riegler, M. A. (2022). Is more realistic better? a comparison
of game engine and gan-based avatars for investigative interviews of children.
Proceedings of the 3rd ACM Workshop on Intelligent Cross-Data Analysis and
Retrieval, 41–49.
95
Schmidt, S., Zadtootaghaj, S., Möller, S., Metzger, F., Hirth, M., & Suznjevic, M. (2018).
Subjective evaluation methods for gaming quality (p. game).
Sethi, D., Bellis, M., Hughes, K., Gilbert, R., Mitis, F., & Galea, G. (2013). European report
on preventing child maltreatment. World Health Organization. Regional Office for
Europe.
Shao, H., Yao, S., Sun, D., Zhang, A., Liu, S., Liu, D., Wang, J., & Abdelzaher, T. (2020).
Controlvae: Controllable variational autoencoder. International Conference on
Machine Learning, 8655–8664.
Siarohin, A., Lathuilière, S., Tulyakov, S., Ricci, E., & Sebe, N. (2019a). First order mo-
tion model for image animation. In H. Wallach, H. Larochelle, A. Beygelzimer, F.
d’Alché-Buc, E. Fox & R. Garnett (Eds.), Advances in neural information processing
systems. Curran Associates, Inc. https://proceedings.neurips.cc/paper_files/
paper/2019/file/31c0b36aef265d9221af80872ceb62f9-Paper.pdf
Siarohin, A., Lathuilière, S., Tulyakov, S., Ricci, E., & Sebe, N. (2019b). First order
motion model for image animation. Advances in neural information processing
systems, 32.
Suwajanakorn, S., Seitz, S. M., & Kemelmacher-Shlizerman, I. (2017). Synthesizing
obama: Learning lip sync from audio. ACM Trans. Graph., 36(4). https://doi.
org/10.1145/3072959.3073640
Thies, J., Zollhofer, M., Stamminger, M., Theobalt, C., & Nießner, M. (2016). Face2face:
Real-time face capture and reenactment of rgb videos. Proceedings of the IEEE
conference on computer vision and pattern recognition, 2387–2395.
This Person Does Not Exist [Accessed on March 28, 2023]. (n.d.).
Toshpulatov, M., Lee, W., & Lee, S. (2023). Talking human face generation: A survey.
Expert Systems with Applications, 119678.
Veale, M., & Binns, R. (2017). Fairer machine learning in the real world: Mitigating
discrimination without collecting sensitive data. Big Data & Society, 4(2),
2053951717743530.
Wang, H.-n., Liu, N., Zhang, Y.-y., Feng, D.-w., Huang, F., Li, D.-s., & Zhang, Y.-m.
(2020). Deep reinforcement learning: A survey. Frontiers of Information Technology
& Electronic Engineering, 21(12), 1726–1744.
Wang, T.-C., Mallya, A., & Liu, M.-Y. (2021). One-shot free-view neural talking-head
synthesis for video conferencing. Proceedings of the IEEE/CVF conference on
computer vision and pattern recognition, 10039–10049.
Wang, X., & Yu, J. (2020). Learning to cartoonize using white-box cartoon representa-
tions. Proceedings of the IEEE/CVF conference on computer vision and pattern recog-
nition, 8090–8099.
Wilson, E., Hewett, D. G., Jolly, B. C., Janssens, S., & Beckmann, M. M. (2018).
Is that realistic? the development of a realism assessment questionnaire and
96
its application in appraising three simulators for a gynaecology procedure.
Advances in Simulation, 3(1), 1–7.
World Health Organization. (2021). Child maltreatment [Accessed on March 22, 2023].
Yang, S., Jiang, L., Liu, Z., & Loy, C. C. (2022). Vtoonify: Controllable high-resolution
portrait video style transfer. ACM Transactions on Graphics (TOG), 41(6), 1–15.
Zhang, C., Zhao, Y., Huang, Y., Zeng, M., Ni, S., Budagavi, M., & Guo, X. (2021a). Facial:
Synthesizing dynamic talking face with implicit attribute learning. Proceedings
of the IEEE/CVF International Conference on Computer Vision (ICCV), 3867–3876.
Zhang, C., Zhao, Y., Huang, Y., Zeng, M., Ni, S., Budagavi, M., & Guo, X. (2021b).
FACIAL: synthesizing dynamic talking face with implicit attribute learning.
CoRR, abs/2108.07938. https://arxiv.org/abs/2108.07938
Zhang, T., Deng, L., Zhang, L., & Dang, X. (2020). Deep learning in face synthesis:
A survey on deepfakes. 2020 IEEE 3rd International Conference on Computer and
Communication Engineering Technology (CCET), 67–70. https://doi.org/10.1109/
CCET50901.2020.9213159
Zhou, H., Sun, Y., Wu, W., Loy, C. C., Wang, X., & Liu, Z. (2021). Pose-controllable
talking face generation by implicitly modularized audio-visual representation.
Proceedings of the IEEE/CVF conference on computer vision and pattern recognition,
4176–4186.
Zhou, Y., Han, X., Shechtman, E., Echevarria, J., Kalogerakis, E., & Li, D. (2020). Makelt-
talk: Speaker-aware talking-head animation. ACM Transactions On Graphics
(TOG), 39(6), 1–15.
97