Using Artificial Intelligence to Monitor Social Media Discourse

There is no denying that social media platforms like Facebook, X and Reddit have created an online discourse with little to no barrier to entry. But is the ability to express any opinion without consequence a good thing?

Along with assistant professor Jingyi Sun, who teaches courses in web mining and social network analysis in the Stevens School of Business business intelligence & analytics master’s program, second-year information systems Ph.D. student Xiaoxuan “Cleo” Xu is harnessing the power of natural language processing and large-scale text mining to help develop an answer. Their project explores how banning toxic online communities affects explicit and implicit biases. The study of Reddit has produced “paradoxical” results, finding that while banning reduces blatant hate speech, it may unexpectedly increase subtle gender and racial prejudices.

“We are studying how users react to what we call ‘community level censorship,’ such as banning subreddits or a Facebook group,” Cleo explained. “We study how a user reacts to those policies in terms of their speech toxicity versus their implicit bias against females, people of color, etc. After the policy, users basically become more biased. After the users are banned from a toxic subreddit, they migrate to other subreddits and influence their discussions, so overall subreddit discourses were more biased.”

Cleo’s use of AI technology to advance how platforms can monitor their content is another example of the extraordinary artificial intelligence work women do every day at Stevens. Giving this work higher visibility is the purpose of the first National Women in AI Month, a joint effort by National Day Calendar and the Cadence Giving Foundation to "showcase women as role models, promote their success in the AI and tech sectors, and encourage women to pursue careers in artificial intelligence technology."

How were you first introduced to using AI for your research?

I'm from a non-STEM background and what I'm doing today is highly technical and data-driven. I couldn't imagine how I would have started a big project like this without the help of AI. I earned my master’s degree in media communication and my undergrad is in advertising, both creativity-based, highly qualitative and rhetorical. I do love my previous expertise. It's shaped my personality and who I’ve become, but I want to be more convinced by data. You cannot tell a story without solid data.

How important have your female role models at Stevens been in your work?

Jingyin Sun, my advisor, and [assistant professor] Bei Yan have communication backgrounds like I do, and Jingyi used to work in journalism. It’s super inspiring to me to see how they use social media analytics as a complement to their previous expertise. I also think they are great instructors. This specific project was inspired by my advisor.

What are some of the practical implications of your research into online communities?

The key thing is the effect is paradoxical. The platform was hoping it could create a safer online space, but in fact, it’s not achieving what it wanted. Empirically, we’re hoping to show how to form better content moderation policies. Each platform will be able to build on this research and create more efficient policies on how to manage the implicit bias overall. Social media data is sensitive, so we won’t share it, but we do have the process of how we measure the implicit bias. How we measure the multiple dimensions of implicit bias is super cool. We will be able to share how we tested the results so others can replicate this.

Why is celebrating Women in AI Month important?

I believe this is definitely important for younger women to get an impression of how to use this technology. As a teaching assistant, I’m surprised that not every student, especially female students, are comfortable with AI assistants or know how to better communicate with AI to leverage the power of it. But I understand that feeling because when I was in media communications, I thought running a regression [analysis] was challenging and that some entry-level software was a fancy tool. Now, with the help I’ve received, I have the confidence to learn anything new. I hope all of us have the courage to learn something new with the help of AI. When I was studying femininity versus masculinity, we discussed the stereotypes that men are better at technology, and I think there are gaps that need to be bridged. I think we will be able to fill in those gaps with AI’s help. It reminds me of my six-year-old niece. She doesn't know how to type or read, but using voice recognition, she creates images using AI. She's a so-called AI native, and I see a promising future for her to realize her creativity.