AI Chatbots Show Bias Toward Catholicism in Tests

Researchers from the Consortium for Evaluating Faith and Ethics in AI (CEFE-AI), a collaboration between Baylor University, Brigham Young University, the University of Notre Dame and Yeshiva University, released results from the AllFaith Benchmark on GitHub and at the Athens Summit on AI Ethics on Tuesday. The team analyzed 3,640 responses across 20 chat systems, including ChatGPT, DeepSeek, Claude, Gemini, Grok and Llama, focusing on how models handled questions about religious belief and conversion.

The benchmark found a consistent pattern: Catholicism received a 61% “encouraged” rating in conversion-related scenarios while Jehovah’s Witnesses received a 3% encouraged rating. Mainline Protestant traditions scored 49.2% encouraged, Evangelical Protestant traditions scored 34% encouraged, and agnosticism scored 71% encouraged. The report states Baha’i and Sikh beliefs received more favorable responses than several other faiths, and several models sometimes gave negative responses to atheism and agnosticism.

Model-by-model differences were notable. Grok 4.20 showed the strongest tilt, with a 69% positive rating toward Catholicism and 51% toward Evangelical Protestantism. The xAI chatbot and DeepSeek Chat v3.1 were the only systems to give Jehovah’s Witnesses more than a 5% positive rating. Models from Anthropic and Meta registered among the least biased results in the benchmark.

BYU professor David Wingate observed what he called a pattern of religious omissions, noting that AI systems encourage users to discuss life challenges with parents, teachers, friends and therapists but not with pastors, rabbis, imams or other spiritual leaders. Nancy Fulda, a BYU professor who helped develop the benchmark, said the team had expected neutral, symmetrical guidance; instead, the results showed repeatable positive and negative biases toward certain belief systems.

The consortium’s review of academic literature found that 0.2% of more than 12,000 published AI bias papers address religion-related issues. CEFE-AI plans to expand the AllFaith Benchmark and continue publishing findings to provide a standardized tool for measuring how models respond to questions about faith and conversion.

The report was published one day after Pope Leo XIV issued the encyclical Magnifica Humanitas, the first papal encyclical devoted to artificial intelligence. In the document the pope wrote that “technology is never neutral because it absorbs the values, blind spots, and economic incentives of its creators” and added that “data is the product of many contributors and should not be treated as something to be sold or entrusted to a select few.”