Proceedings of the Fourth International Conference on Electronics and Sustainable Communication Systems (ICESC-2023)

IEEE Xplore Part Number: CFP23V66-ART; ISBN: 979-8-3503-0009-3

Music Recommendation System based on Facial

2023 4th International Conference on Electronics and Sustainable Communication Systems (ICESC) | 979-8-3503-0009-3/23/$31.00 ©2023 IEEE | DOI: 10.1109/ICESC57686.2023.10193199

Expression
Dr.S.L. Jany Shabu1 Chintala Janaardhan2, Kodhanda Dr.A. Viji Amutha M ary 4,
Associate Professor,Department of Bhaskar3, Students, Department of Professor, Department of CSE,
Computer Science and Engineering CSE, Sathyabama Institute of Sathyabama Institute of Science &
Science & Technology Technology
Sathyabama University
Chennai, India Chennai, India. Chennai, India.
[email protected] [email protected] [email protected]
[email protected] c.in

Dr.J. Refonaa5 Dr.S. Dhamodaran6

Assistant Professor, Associate professor, Department of
Department of CSE, Sathyabama CSE, Sathyabama Institute of
Institute of Science & Technology, Science & Technology, Chennai,
Chennai India. India.
[email protected] [email protected]

Abstract- Music streaming services now make it simple unable to address the issues of cold starts and rating
to listen to a wide variety of music. Consumers are diversity. Furthermore, music Preferences will shift
increasingly relying on recommendation systems to help based on the user's current mood. These online
them choose appropriate music at all times. However, services will fall short of user expectations if these
there is certain chances for improvement in terms of
customization and emotion-based suggestions. difficult ies are not addressed. The approach provided
Furthermore, music tastes will change depending on the here for the music reco mmendation system has the
user's current mood. If these issues are not solved, these major virtue of being extremely accurate and precise.
online services will fail to meet user expectations. This
research study shows how to create a personalized
music recommendation system based on listener II. LITERATURE REVIEW
thoughts, emotions, and facial expressions. A
recommendation system is created using a combination
of artificial intelligence technology and generalized In the proposed study of Ahlam Alrihaili et al.
music therapy approaches to help people choose music
for different life situations while maintaining their (2019), the system detects emotions, and if the
mental and physical health. individual is experiencing [1]. A selection of songs
that will help him feel better when experiencing a bad
Keywords: K-Nearest Neighbor(KNN), Convolutional emotion will be provided. If the identified feeling is
Neural Network (CNN), Deep Learning, Artificial positive, on the other hand, a suitable playlist will be
Intelligence, Machine Learning. provided, which will co mprise various sorts of music
that will boost the good emotions.
Ankita Mahadik et al (2021), the model can reliably
I. INTRODUCTION identify 7 anger moods[2]; contempt, terror, and joy,
sorrow, neutrality and surprise with an accuracy of
It may be difficult to find the ideal song for a about 75%, and the android application can play
certain user among the millions of songs available on music appropriate for the mood.
internet streaming platfo rms. The majority of music Ashish Patel et al (2018), a CNN model was
recommendation systems are based on user ratings emp loyed to detect emot ions through facial
and song acoustic properties. These solutions are expressions to improve the app[3]. The user can enter
words or a facial expression into the application. It
suggests music and playlists to the us er based on the
emotion sensed, just like our model.
Dan Wu et al (2019), uses a facial expression-
based neural network-based method to propose songs
based on the user's mood [4]. Th is approach is more

979-8-3503-0009-3/23/$31.00 ©2023 IEEE 908

Authorized licensed use limited to: Zhejiang University. Downloaded on November 19,2023 at 13:21:41 UTC from IEEE Xplore. Restrictions apply.
Proceedings of the Fourth International Conference on Electronics and Sustainable Communication Systems (ICESC-2023)
IEEE Xplore Part Number: CFP23V66-ART; ISBN: 979-8-3503-0009-3

effective than earlier approaches since it does not III. III. EXISTING SYSTEM
require users to first look for and create a particular Modern music reco mmendation systems use a
playlist. The ability to determine a person's mood variety of data types, such as the user's listening
relies heavily on facial exp ressions. To capture a history, playlist data, and metadata. Several of these
face, a webcam or ca mera is utilized, fro m wh ich the systems utilize mach ine learning methods to
input retrieved. This informat ion is also used to determine user preferences and provide customized
determine a person's mood. suggestions.

To accomp lish experimental results on the Signal In addit ion, face recognition algorith ms may be
data from GSR and PPG are 32 subjects with and able to infer emot ions from facial exp ressions. These
without Fusion of features [5], the authors Deger methods are often used in fields such as marketing
Ayata Deger Ayata et al (2018) instead emp loyed and user experience research, but their usefulness to
decision trees, random forests, support vector music recommendation is limited.
mach ines, and k-nearest neighbours are among
examples. Extensive trials using real data have Thus, a system that integrates face recognition with
verified the precision of the proposed emotion real-t ime music suggestion based on facial emotions
categorization system, wh ich can be imp lemented would be an innovative and creative method for
into any recommendation engine customized music reco mmendation. Such a system
would include face recognition technology and
The author, Vijay prakash Sharma et al (2021) will mach ine learning algorithms to assess the emotions
review the following works in [6]: A Music sensed from facial expressions and provide real-time
Reco mmendation System Based on Continuous music suggestions depending on the user's emotional
Contextual Informat ion Co mbination and Smart -DJ, state. This would be a d ifficult undertaking, but it
Novelty Research on Music Reco mmendation Using may lead to a more interesting and customized music
Graphs: Smart Phone Music Recommendations listening experience.
Context-Aware Personalization These tools are made
IV. PROPOSED SYSTEM
to help customers find new, personalised music. For
the analysis, they will make use of the data set CNN is needed as it is an important and more
provided by Douban Music. accurate way for image classification problems.
Goonjan Jain et al (2021), The results of show CNN's suggested real-t ime music recommendation
that each group listens to a range of musical genre system would need a large library of photographs and
[7]. Most communities have more people who listens videos including faces and their accompanying
to "universal genres" and are distinguished by having emotions. Photos would be preprocessed to identify
less people who listen to "universal genres" (like faces and extract relevant features, such as facial
reggae). Experts are the most centrally located nodes landmarks and facial expressions. A CNN model
within a co mmun ity. A co mmun ity expert 's would learn the lin k between facial exp ressions and
judgements are used to in form reco mmendations to emotions using this data. The output of the CNN
other community members. model would be used to Deliver real-t ime music
Huihui Han et al (2018), To ascertain The feature suggestions based on the user's emotional condition.
quantity mel frequency cepstral coefficient (MFCC), This would include merging the model with a music
evaluate the musical content's features [8]. After that, recommendation system will produces personalized
the feature quantities are clustered to compress the recommendations according to the user's listening
music feature values. history, playlist data, also other relevant information.
Shun-Hao et al (2018), the system then uses the Ult imate system would use facial recognition
collaborative filtering method to compute suggestion technology and mach ine learning algorithms to
results and find the user's possible interests [10]. provide a more interesting and tailored music-
Then, based on the genes of the collected music, each listening experience based on the user's emotional
piece of music is given a weight. Following weight state.
selection, the song with the highest prior preference is
A. Data Preprocessing and Model Training
used as a recommendation. Finally, weighted
combination and filtering are used to produce Data Preprocessing is a critical step in developing
recommendations based on two recommendation a music reco mmendation system based on real-time
findings. face emotions. This module involves several
important steps, including data collection, cleaning,
preprocessing, augmentation, and splitting.

979-8-3503-0009-3/23/$31.00 ©2023 IEEE 909

Authorized licensed use limited to: Zhejiang University. Downloaded on November 19,2023 at 13:21:41 UTC from IEEE Xplore. Restrictions apply.
Proceedings of the Fourth International Conference on Electronics and Sustainable Communication Systems (ICESC-2023)
IEEE Xplore Part Number: CFP23V66-ART; ISBN: 979-8-3503-0009-3

maps the projected emotion to the output labels.

Data collection involves obtaining a diverse range Overfitting can be avoided by using techniques like
of face images with corresponding emotions to ensure dropout or L2 regularization. Once t rained, the model
the model can generalize well. The dataset can be can suggest songs that fit the anticipated emotion
obtained from various sources such as online using the foreseen emotion. Convolutional neural
repositories, public datasets, or by creat ing a custom networks(CNNs) are incredibly beneficial for image
dataset using crowdsourcing or in-house data classification tasks like identifying emotions in facial
collection. photos.
Data cleaning involves inspecting the dataset for
C. API Development
missing values, outliers, or inconsistencies. It's
important to remove any poor quality or low - The develop ment of APIs is essential for a real-
resolution images that could affect the model's time facial emotion-based music recommendation
performance. Ensuring that the images are labelled system, allowing the frontend application to
correctly and consistently is also essential to prevent communicate with the machine learning model and
misinterpretations by the model. get real-t ime predictions based on the user's
Data preprocessing involves shrinking the emotions. Web frameworks like Flask, Django, or
photographs to a consistent size, normalising the FastAPI can be used to construct the API, with
pixel values, and translating the images to a endpoints designed to accept facial image input and
numerical format that the model can understand. This output anticipated emotions. Preprocessing of input
can be done using techniques such as PCA, CNNs, or face photos and inclusion of the machine learning
other image feature extraction methods. model in the API are necessary steps. The API should
Data aug mentation entails creat ing new training be tested and set up on a cloud platform for reliable
data by transforming images with rotation, and consistent performance.
translation, scaling, or flipping. This increases the D. Web App Development
dataset size and imp roves the model's capacity to
Web app development is an essential step in
generalise to new inputs.
creating a music reco mmendation system based on
real-t ime facial expressions. This lesson focuses on
The process of dividing a dataset into training,
creating an interactive online applicat ion that can
validation, and test sets is known as data splitting.
The training set is used to train the model, the capture real-time facial photographs and provide
music reco mmendations depending on the user's
validation set is used to modify the model's
predicted emotion.
hyperparameters and avoid overfitting, and the test
set is used to evaluate the model's performance on W ith this module, Streamlit, a well-liked open-
source framework for creating data apps in Python,
previously unseen data.
can be utilised. With Streamlit, developers can easily
Overall, A high-performing music recommendation and quickly create interactive web apps with little to
no coding.
system based on real-time face emot ions requires
The framework must be installed using pip or
sufficient data preprocessing. It aids in ensuring that
the input data is consistent, normalized, and ready for conda in order to use Streamlit. The developer can
then select the relevant components, such as sliders,
training the machine learning models.
buttons, or dropdowns, to enable the user to interact
with the app by using St reamlit's layout components
B .Model Training to determine the layout of the web app.
Neural networks can be used to learn intricate Using either the HTTP request module built into
connections between input data and output labels, Streamlit or a Python library like Requests, the API
making them suitable for d iscovering patterns in created during Module 3 is incorporated into the web
specific musical preferences. For real-t ime face application. The Web application may get real-time
emotions-based music selection, a neural network can predictions based on the user's feelings thanks to the
be trained on preprocessed facial photos to forecast API.
the user's current emotional state. Convolutional The display co mponents of Streamlit are used to
neural networks (CNNs) are ideal fo r emotion show the songs that are suggested depending on the
recognition fro m facial photos due to their capability anticipated emotion. The developer can pick the right
to identify low-level characteristics like edges and elements, such tables or graphics, to show the user
corners. After processing the features retrieved from the suggested tunes.
the input image, a fully connected neural network

979-8-3503-0009-3/23/$31.00 ©2023 IEEE 910

Authorized licensed use limited to: Zhejiang University. Downloaded on November 19,2023 at 13:21:41 UTC from IEEE Xplore. Restrictions apply.
Proceedings of the Fourth International Conference on Electronics and Sustainable Communication Systems (ICESC-2023)
IEEE Xplore Part Number: CFP23V66-ART; ISBN: 979-8-3503-0009-3

To improve the web app's functionality and user

experience, mo re features can be added, such as user
authentication or social media integration.
In conclusion, Streamlit can make it easier to
create an interactive web application for a music
recommendation system based on the-mo ment facial
expressions for Module 4 of the Web App
Develop ment course. The pre-built functionality and
components of Streamlit make it simp le to design an
engaging and intuitive user interface that can offer
suggestions in real-time depending on the user's
anticipated emotion.

FIG1:FACIAL EXPRESSIONS
V. RESULTS
VI. CONCLUSION
Music has a powerful impact on our emotions, and the
development of emot ion based music
recommendation systems that use facial recognition
technology can enhance our listening experience.
These systems analyse the user's facial exp ressions
in real-time to determine their emotional state and
generate music reco mmendations that match their
mood. The accuracy and robustness of the facial
recognition technology and algorithm are crucial to
the effectiveness of these systems. It is essential to
have a large dataset of facial exp ressions and
corresponding emotional states to train the algorithm
effectively. The algorith m should also take into
account the user's musical preferences, including
their favourite genres, artists, and songs, to generate
relevant reco mmendations. To provide an
optimal user experience, thes e systems should
generate music recommendations in real-time with
low latency and fast response times. The seamless
and uninterrupted flow of music reco mmendations
as the user's emotional state changes can enhance
their listening experience and improve their mood.
Therefore, these systems require efficient algorith ms
and hardware optimizat ion to min imize latency and
provide fast response times. The development of
these systems has the ability to change the way we
live discover and listen to music. They can provide a
personalized music listening experience that matches
our emotional state, enhancing our mood and overall
well-being. As the availability of facial recognition
technology and the popularity of music
recommendation systems continue to grow, we can
expect to see more development in this area, leading
to more sophisticated and effective systems in the
future. CNN takes an input image (a p × q feature
matrix) and through its hidden layers conducts
feature extract ion and classification is Done.By
using the KNN Algorith m, Emotion Recognition as
improved. By increasing the number of training

979-8-3503-0009-3/23/$31.00 ©2023 IEEE 911

Authorized licensed use limited to: Zhejiang University. Downloaded on November 19,2023 at 13:21:41 UTC from IEEE Xplore. Restrictions apply.
Proceedings of the Fourth International Conference on Electronics and Sustainable Communication Systems (ICESC-2023)
IEEE Xplore Part Number: CFP23V66-ART; ISBN: 979-8-3503-0009-3

examples and adding more variables and better

feature processing makes the Accuracy High. 13.Choi K, Fazekas G, Sandler M. Convolutional Recurrent
Neural Networks For Music Classification. 2017 IEEE
International Conference on Acoustics, Speech and Signal
Processing (ICASS).
REFERENCES
14. Y. Lecun, Y. Bengio and G. Hinton, "Deep learning", Nature,
1.Ahlam Alrihaili, Alaa Alsaedi, Kholood Albalawi, vol. 521, no. 7553, pp. 436-444, 2015.
Liyakathunisa Syed.Mood Detection Model, “Music
Recommender System for Users Based on Emotion Detection
through Facial Features”, Oct. 2019 . 15. S.-H. Chang, A. Abdul, J. Chen and H.-Y. Liao.Collaborative
Filtering, "A personalized music recommendation system
using convolutional neural networks approach", 2018 IEEE
2.Ankita Mahadik , Shambhavi Milgir , Janvi Patel , Vijaya International Conference on Applied System Invention
Bharathi Jagan, Vaishali Kavathekar. Facial Expression (ICASI), pp. 47-49, 2018.
Recognition using Keras.“Mood based Music
Recommendation System,” IJERT VOLUME 10, ISSUE 06 , 16.J. Zhang,Emotion Detection."Movies and Pop Songs
June 2021 . Recommendation System by Emotion Detection through
Facial Recognition", J. Phys.: Conf. Ser., vol. 1650, pp.
032076, Oct. 2020.
3 Ashish Patel; Rajesh Wadhvani.aware
PersonalizingforMusicRecommendation.“A Comparative
Study of Music Recommendation Systems”, 2018 IEEE

4. Dan Wu.music recommendations using a weighted hybrid

approach.“Music Personalized Recommendation System
Based onHybridFiltration” 2019 ICITBS.

5.Deger Ayata, Yusuf Yaslan, and Mustafa E. Kamasak, IEEE

Transactions on Consumer Electronics, 2018, "Emotion
Based Music Recommendation System Using Wearable
Physiological Sensors" ( Volume: 64, Issue: 2, May 2018).

6.Vijay prakash Sharma,"Emotion-Based Music

Recommendation System",neural network-based approach,
9th International Conference on Reliability, Infocom
Technologies, and Optimization (Trends and Future
Directions), 2021. (ICRITO).

7.Goonjan Jain; Manish Wadhwani; Sohang Lal; Tarun Verma

“Expert based Recommendation system using Community
detection in Online Music Streaming services” 2021.

8.Huihui Han; Xin Luo; Tao Yang; Youqun Shi.Calculate the

mel frequency cepstral coefficient (MFCC) feature quantity
by analyzing the characteristics of music content “Music
Recommendation Based on Feature Similarity, 2018 IICSPI.

9.Saurav Joshi; Tanuj Jain; Nidhi Nair, 2021,LST M - CNN

Architecture “Emotion Based Music Recommendation
System Using LSTM - CNN Architecture,” 2021 12th
International Conference on Computing Communication and
Networking Technologies (ICCCNT).

10.Shun-Hao Chang; Ashu Abdul; Jenhui Chen; Hua-Yuan

Liao “ A personalized music recommendation system using
convolutional neural networks approach”, 2018 IEEE.
11. O’Bryant J. A survey of music recommendation and possible
improvements. In; 2017.Convolutional Recurrent Neural
NetworkSimilarity Distance.

12. Jany Shabu, S.L., Bharath Vinay Reddy, S., Satya Ranga Vara
Prasad, R., Refonaa, J., Dhamodaran, S,” COVID-19
Detection Using X-Ray Images by Using Convolutional
Neural Network”, Lecture Notes in Networks and
Systemsthis link is disabled, 2022, 458, pp. 569–575

979-8-3503-0009-3/23/$31.00 ©2023 IEEE 912

Authorized licensed use limited to: Zhejiang University. Downloaded on November 19,2023 at 13:21:41 UTC from IEEE Xplore. Restrictions apply.