Research & Innovation

Exposing — and Eliminating — the Hidden Biases in AI

Computer scientist Jia Xu works to pinpoint, analyze and remove potential sources of bias from the AI systems that approve or deny our mortgages, recommend jail sentences for criminals and more

Are certain groups of people “red-lined” when applying for credit, not only by people but by automated systems utilized by lending institutions? What about employment or health-insurance applicants who are pre-screened by computerized systems? 

Or criminals sentenced to prison time by judges: Do certain groups’ personal demographics work against them, in the automated recommendations supplied to courts? 

Stevens computer science professor Jia Xu wants to find out. 

Xu, who joined Stevens in 2019, studies algorithms and elements of artificial intelligence that shed light on decisions made by AI systems. She believes fairness and bias mitigation — as well explainability (the ability of AI to explain its decisions) — will be critical challenges for technologists going forward. 

"Your race and your age may not be the important features when determining whether you are a credit risk, for example," points out Xu. "Yet this is how most current AI decides whether you receive a line of credit or a home loan. It incorporates certain societal biases within its data.” 

"We need to build something better, more accurate, more bias-free." 

Toward fairer credit decisions

To examine fairness in personal-loan decisions, Xu and her students examined a set of anonymized credit data on people — some of whom had defaulted — created and made public by the University of California, Irvine. 

Her team then removed certain data points (or "features," in AI jargon), such as a subject's age, marital status, gender and education level, then ran the predictions again. 

Scrubbing those features out, they found, did not affect the AI's power to predict a person’s future credit default at all. Its predictions were equally accurate.

"We may need to rethink our systems for giving or denying people a loan or line of credit," Xu suggests. "A loan officer or online application system could be required to not ask for nor enter your age, gender and marital status at all, for example.” 

"The decision it makes will still be just as accurate, intelligent and profitable for the bank, but it will be made without the biases built into that system." 

In a collaboration with Stevens finance professor and CRAFT (Center for Research toward Advancing Financial Technologies) Director Steve Yang, as well as doctoral candidate Xuting Tang and postdoctoral researcher Abdul Rafae Khan, Xu is currently expanding upon her credit-fairness research

“The conventional approach, which is silencing some protected attributes in a model, cannot entirely eliminate bias from features like zip codes that act as proxies for other data points,” she explains about her methods to reduce bias in AI systems. “So it is essential to understand the sources of bias in a model.” 

Next her team will create a novel learning model that deploys special types of algorithms within the AI that will help observers understand some of the processes and features that drive its decisions.

Xu hopes her investigations can inform the building of improved AI systems that will generate more equitable decisions across ethnic backgrounds, genders and geographic areas. The research may also be useful to consumers, she adds, who wish to improve their own credit worthiness. 

Can courtroom sentencing be made fairer?

In another project focused on potential sources of bias in AI, Xu examines the automated systems that make sentencing recommendations for law enforcement and judicial officers. 

Again, her goal is to develop AI systems that do not express bias, for example on the basis of race, in their decision-making. 

"Since courts are already using computing systems to assist them in determining bail and criminal sentences,” she points out, “we need to better understand how those sentencing-software models are actually making their decisions.” 

To investigate the issue, her team sliced and diced crime data using a crime-prediction model that searches for the most useful factors and patterns in predicting a person’s criminal history — then learns to make good predictions about future recidivism with the best of those factors.

With Stevens postdoctoral researcher Khan and additional collaborators at Rutgers University, Xu analyzed a dataset of more than 17,000 anonymized crime records supplied by the analytics firm KOVID, creating a new learning algorithm in the process that achieves very high accuracy at predicting future crimes when it is fed data about real convicted criminals.

Certain data about those individuals' criminal histories, the group found — such as the time between committing successive crimes, or the recency of a crime — were very powerful in aiding successful crime-prediction. 

Personal characteristics such as age or race, however, were not useful predictors. 

"This model provided some interpretations for its predictions," says Xu, “and those interpretations suggest a person's historical criminal records — rather than personal data — are the key features of a strong model. Information such as a person's race or age contributed very little toward accurate crime prediction.” 

“Judges and sentencing tools could take this into account by removing information about age, race, marital status, and so on.” 

Another interesting side finding of the study analysis of this dataset was that criminals are double-more likely to commit the same type of crime, or one of a similar level of seriousness, in the future rather than different types of crimes over the course of a lifetime.  

If this finding holds up, it could contradict popular wisdom about minor “gateway” crimes normally leading to progressively different and more serious ones. 

Large-language models, explainability and bias

Xu is also interested in large-language-model generative AI, a natural language processing (NLP) area that breakthrough commercial technologies such as ChatGPT, GPT-4 and Bard are built upon.

Language models depend heavily upon NLP techniques that rapidly analyze huge quantities of our writing and speech, learning both factual information and deft conversational skills from the patterns detected in its analyses.

“Contemporary language models demonstrate amazing and revolutionizing capability, sometimes exceeding human imagination,” she says.

Image of professor Jia Xu with fanciful hatXu received her Ph.D. from Germany's RWTH Aachen University in 2010However, she adds, given the power these models have already demonstrated, it is urgent to accelerate our understanding of these models to control risks and increase security and controllability. Increasing NLP models’ interpretability has thus become essential to achieving those aims, but it remains a bottleneck despite considerable prior study.

“One downside of many explainable NLP methodologies is that their scope is limited to pre-defined concepts that reinforce traditional linguistic knowledge,” explains Xu. “They don’t always reflect new concepts learned by a model. This restricted view may limit the interpretation of any new knowledge captured by their pre-trained language models.”

In other words, the results you get may be pre-biased: they will largely mirror what data has been fed into the model, and some of that data can be "noisy" (misleading or erroneous).

Xu is working on this challenge.

With postdoctoral researcher Khan and colleagues from Dalhousie University and Hamid Bin Khalifa University, she has designed a framework called ConceptX that she hopes will help NLP-driven AIs generate more useful and also much more bias-free responses and discoveries.

ConceptX works by discovering and interpreting latent (hidden) concepts learned by its neural network models through a process known as unsupervised learning.

During that process, the neural network aggregates clusters of hidden neural representations and forms concepts, labeling them to the best of its ability. (Xu’s team also uses human experts to assess those discovered aggregations for relevance, confirming the usefulness of what the model has learned.)

In early experiments, ConceptX uncovered a number of hidden concepts that indicate potential bias in what it has learned — which will help AI scientists as they work to “de-bias” future iterations of large-language models.

Enabling student success

As she conducts these various lines of research and others, Xu also mentors Stevens undergraduate and graduate student teams, preparing them to compete in global natural-language competitions. 

One team’s machine-translation work recently took a top prize for Hinglish-to-English translation at the Conference on Machine Translation (WMT22) in Abu Dhabi in December 2022. 

Another team is currently competing in the elite Amazon Alexa Prize competition against teams from Stanford, Carnegie Mellon, Virginia Tech and the University of California, Santa Barbara, among other heavyweights. This year the competition requires university student teams to design conversational tools that can chat realistically with humans for at least 20 minutes on a variety of topics — certainly timely, given the recent white-hot rise of tools like ChatGPT and Bard.

“These events immerse students in real-world challenges, and they also get the excitement of participation with peers and colleagues and, in some cases, of triumph,” concludes Xu.