Master’s Student Develops AI Emotion Recognition Software With Potential Applications in Psychotherapy, Publishes Findings

What if you could go to therapy for a much lower cost and without self-consciousness or fear of stigma — because your therapist is a psychology-trained artificial intelligence (AI) model instead of a human? In many countries around the world, barriers such as cost and societal and cultural factors have prevented millions of people from accessing the help they need. Now, new technology may provide a solution. 

Recent Computer Science master’s graduate Vismay Rathod ‘24 hopes to one day create this solution with machine-learning psychologist models that can provide accessible, affordable and judgment-free therapy.

Making smarter machine learning models

Working under Ph.D. student Necdet Gurkan, who studies cultural consensus theory —  how groups of people think collectively and how this can be researched in tandem with training AI — Rathod has been studying human faces in order to create smarter models of machine learning that can accurately recognize human traits and feelings. Gurkan and Rathod’s article, "Unlocking the Face Code: How Facial Characteristics Drive Social Biases," recently published in the Proceedings of the Annual Meeting of the Cognitive Science Society, investigates how facial features and emotional expressions influence first impressions and social biases. The researchers used computer vision and machine learning to analyze facial morphology and emotional attributes, modeling 34 different first impressions. They found that these facial characteristics significantly shape social biases, providing a comprehensive understanding of the impact of facial features on perception. 

The researchers first showed a group of human participants a set of faces and asked them to identify characteristics such as race, sex and so forth; as well as emotional states and subjective traits like attractiveness. Then they showed the same data set to AI and asked it to guess the emotions and traits. “The AI model was able to detect 998 faces out of 1,000. We got a million data points. We achieved an accuracy of about 40%, and we did emotional analysis on the data set,” Rahod said.  

Forty-percent accuracy is remarkable considering the nuances of how human emotional affect is expressed. Even with cultural differences in expressions, the machines performed much higher than Rathod and Gurkan predicted.  

“If there is some way that we can know the reasoning behind a person’s thoughts or a person’s perspective, that is something that will help a lot in my project to improve the accuracy from 40% to 65%,” he noted.

Vismay Rathod

Removing bias in AI therapy

Rathod and Gurkan’s findings indicate that certain facial features, such as face shape and emotional expressions like smiling, heavily influence perceptions of traits such as trustworthiness, attractiveness and competence. These biases have real-world implications in hiring, legal decisions and social interactions. 

Rathod explained how India and most of Asia and South Asia still carry significant cultural stigma around seeking mental health treatment. In India, just having someone to talk to confidentially is a huge help when there is otherwise no one to confide in: “My parents don’t have any understanding of therapy and judged me when I asked to see a therapist. I only got into it very recently, because even in my undergrad, I still didn’t know what therapy was,” Rathod recounted. He went on to describe how his first few years adjusting to life in the U.S. were challenging, and through meeting psychologists and learning about talk therapy, he began doing his own research to discover the power and potential of this field, and how he could apply it to science. Rathod wishes he had had the pre-knowledge to seek therapy and mental health resources sooner. 

He also described the extreme pressure many young people from Eastern cultures face as they take difficult exams for limited placement in universities and the need, above all, to succeed. Depression and anxiety are soaring in these regions and Rajastan, a city where many students travel to take these exams, is the suicide capital of India. A non-judgemental and compassionate AI therapist for this demographic could literally save lives. 

“The use cases for AI therapy would be really helpful. Of course, it’s not something to completely replace the human touch, but I hope to improve it so it is as close to human as it can be, at a very affordable rate, without bias or judgment," said Rathod. "This is in the very nascent stage; it has to be proven out. Even a million data points is not much. There needs to be billions of rows of data before it would be ready for public use.”

An AI that can “read” people

An AI trained to accurately read and interpret human emotions as they shift would be better able to respond in real time like a human therapist in a video call with a patient. The AI could be modeled to remove the subconscious biases that humans have toward each other regarding race, gender, attractiveness, etc., and also eliminate the risk of transference, the phenomenon in which patients project their emotions and unresolved issues onto the therapist, and vice versa.

“Our research focused on how facial features and expressions shape the way we make snap judgments about things like trustworthiness or competence," said Rathod. "Even though we started by looking at first impressions, the findings have a much bigger potential. If we can train AI to pick up on these subtle cues accurately, it could create interactions that feel more natural—almost like speaking to a real person, which could be useful in therapy. Plus, AI has the advantage of being less influenced by human biases, like those based on race or gender, making it possible to provide a more objective and supportive experience for someone seeking help.”

Of course, there’s much more to teach AI in terms of recognizing the wide variation of expressions in human emotion and behavior. Everything from tone of voice, to eye movements, to irony and sarcasm, to hand gestures, and cultural differences is difficult if not impossible for current computer models to interpret — but Rathod is hopeful for a future where the science is advanced enough to create AI with this level of discernment. 

All of this said, Rathod’s end goal is not to eliminate or replace the need for human therapists, but rather to take the entire field to new frontiers. Just as the internet, social media and AI have made huge leaps in democratizing and expanding access to education, information and technology for new regions and groups of people, the same would be true of psychotherapy. 

“My big dream for this is to be a platform to help people. The way that I see the world right now is that people are struggling a lot to make good connections with each other due to many personal reasons and problems. If there is a platform to help them overcome their personal difficulties, social and academic issues, mental illness, and get back into the world, it would be great. If people are able to get the help they need, become motivated for life again, willingly try to improve the world and not be stuck inside with depression and anxiety, doom scrolling on Instagram and YouTube shorts, the whole world would change. This platform provides a very efficient way to tackle that problem, and could make people’s daily lives better in some way.”