30 Jun Trust Incident Upstart
Case Author
ChatGPT-4, OpenAI, peer-reviewed by Claude 3.7 Sonnet, Anthropic
Date Of Creation
17.03.2025
Incident Summary
The Upstart AI lending algorithm used features that served as proxies for race, resulting in 10-15% higher denial rates for Black and Hispanic applicants with equivalent creditworthiness to white applicants""
Ai Case Flag
AI
Name Of The Affected Entity
Upstart
Brand Evaluation
3
Upload The Logo Of The Affected Entity
Industry
Financial Services
Year Of Incident
2021
Upload An Image Illustrating The Case
Key Trigger
Consumer Financial Protection Bureau investigation into Upstart AI model following reports of racial disparities in approval rates
Detailed Description Of What Happened
In 2021, Upstart’s AI-driven lending system was found to disproportionately deny loans to Black and Hispanic applicants, even when their credit profiles were similar to approved applicants from other demographics. This raised concerns over algorithmic bias in credit scoring, leading to an investigation by the Consumer Financial Protection Bureau (CFPB). An audit of Upstart’s AI was conducted to assess whether the system violated fair lending regulations.
Primary Trust Violation Type
Integrity-Based
Secondary Trust Violation Type
Competence-Based
Analytics Ai Failure Type
Bias
Ai Risk Affected By The Incident
Algorithmic Bias and Discrimination Risk, Transparency and Explainability Risk, Ethical and Regulatory Compliance Risk, Economic and Social Impact Risk
Capability Reputation Evaluation
3
Capability Reputation Rationales
Upstart was recognized as an innovative AI-driven lending platform with strong data science capabilities. However, the bias in its algorithm raised concerns about the accuracy and fairness of its decision-making process. Addendum: Contradictory – claims ""strong data science capabilities"" yet failed at fundamental fairness testing. Needs specific pre-incident technical achievements
Character Reputation Evaluation
3
Character Reputation Rationales
Upstart positioned itself as a fair and inclusive fintech company, but the discovery of algorithmic bias raised doubts about its ethical commitment to non-discriminatory lending practices. Addendum: Too vague. Needs specific examples of Upstart ethical positioning pre-incident (e.g., marketing claims, DEI commitments)
Reputation Financial Damage
The incident damaged Upstarts reputation, raising concerns over the fairness of its AI lending model. While no major financial losses were reported, regulatory scrutiny and consumer distrust posed risks to future business growth.
Severity Of Incident
3
Company Immediate Action
Upstart responded to the CFPB investigation with a multi-faceted approach that balanced cooperation with defense of their core technology. Rather than denying allegations outright, the company acknowledged the concerns while emphasizing that their AI actually expanded access to credit compared to traditional models. According to SEC filings and press statements, Upstart immediately cooperated with regulators by providing detailed data and internal documentation about their lending algorithms. CEO Dave Girouard publicly stated that their model was ""designed to be fair to all"" and launched a voluntary self-assessment of their loan decisioning process. The company created a dedicated Fair Lending team to review model outcomes and established a more robust compliance framework. They implemented more frequent bias testing while simultaneously defending their technology in investor communications, asserting that their AI approach approved 26% more minority applicants than traditional credit scoring methods. Upstart did not make leadership changes but instead emphasized technical adjustments to their algorithms and expanded their model governance processes.
Response Effectiveness
Upstart response was partially effective but insufficient to fully restore trust in their AI lending system. The company successfully navigated the immediate regulatory challenge by cooperating with the CFPB and avoiding formal enforcement actions or significant financial penalties. Their technical adjustments improved statistical parity in approval rates across demographic groups, according to their subsequent SEC disclosures. However, their response had several limitations that reduced effectiveness. First, they continued to defend their existing approach rather than acknowledging fundamental flaws in how their system evaluated creditworthiness, creating perception of prioritizing business concerns over fairness. Second, they provided limited transparency about specific changes to their algorithms, citing intellectual property concerns, which failed to address explainability issues. Third, while approval rates improved, they didnt publicly address disparities in interest rates and terms across demographic groups. Industry analysts noted that Upstart maintained its growth trajectory but faced increased skepticism from consumer advocacy groups. The effectiveness was also limited by the company failure to establish independent third-party oversight of their fairness metrics, relying instead on self-reporting that lacked credibility with critics.
Linked Sources Url 2
Upload Supporting Material
https://iceberg.digital/wp-content/uploads/2025/03/Responsible-AI-Credit-Scoring-–-A-Lesson-from-Upstart.com_.pdf
Model L1 Elements Affected By Incident
Reqciprocity, Brand, Social Adaptor
Reciprocity Model L2 Cues
Algorithmic Fairness & Non-Discrimination, Transparency & Explainability, Accountability & Liability
Brand Model L2 Cues
Brand Ethics & Moral Values, DEI & Accessibility Commitments, Brand Image & Reputation
Social Adaptor Model L2 Cues
Auditable Algorithms & Open-Source Frameworks, Compliance & Regulatory Features, Algorithmic Recourse & Appeal
Social Protector Model L2 Cues
N/A
Response Strategy Chosen
Apology, Justification, Corrective Action
Mitigation Strategy
Upstart acknowledged the concerns raised by the CFPB and committed to increasing transparency in its AI model. The company justified its AI use by arguing it improved access to credit compared to traditional lenders but also promised to enhance fairness in its algorithms.
Model L1 Elements Of Choice For Mitigation
Reciprocity, Social Adaptor, Brand
L2 Cues Used For Mitigation
Algorithmic Fairness & Non-Discrimination, Transparency & Explainability, Accountability & Liability, Auditable Algorithms & Open-Source Frameworks, Compliance & Regulatory Features, Algorithmic Recourse & Appeal, DEI & Accessibility Commitments, Brand Ethics & Moral Values
Further References
https://www.federalreserve.gov/SECRS/2022/October/20221028/OP-1743/OP-1743_070121_138216_351653676425_1.pdf, https://newjerseymonitor.com/2024/10/14/as-ai-takes-the-helm-of-decision-making-signs-of-perpetuating-historic-biases-emerge/, https://www.urban.org/sites/default/files/2023-11/Harnessing%20Artificial%20Intelligence%20for%20Equity%20in%20Mortgage%20Finance.pdf
Curated
1
The Trust Incident Database is a structured repository designed to document and analyze cases where data analytics or AI failures have led to trust breaches.
© 2025, Copyright Glinz & Company
No Comments