Log in
/
RISK #15
Technical & Emergent Risks

Black-Box Dependence

Systems too complex to interpret. Can't debug, can't audit, can't predict failure.

Severity

HIGH

Likelihood

HIGH

Reversibility

MODERATE

Mitigation

Partial measures

Failure Mode

Flying blind at Mach 10.

Key Insight

Dependence on systems you can't understand is dependence on luck.

Linked Discussions

No discussions linked yet. Be the first to start a conversation about this risk.

Linked Incidents
ChatGPT Allegedly Defamed Norwegian User by Inventing Child Homicide and Imprisonment
Aug 2024 · OpenAI
Google Bard Allegedly Generated Fake Legal Citations in Michael Cohen Case
Dec 2023 · Michael Cohen, David M. Schwartz
Skating Rink’s Facial Recognition Cameras Misidentified Black Teenager as Banned Troublemaker
Jul 2021 · Riverside Arena Skating Rink
Collection of Robotic Surgery Malfunctions
Jul 2015 · Hospitals, Doctors
Springer Nature Book 'Mastering Machine Learning: From Basics to Advanced' Reportedly Published With Numerous Purportedly Nonexistent or Incorrect Citations
Apr 2025 · Springer Nature, Govindakumar Madhavan
ChatGPT Provided Non-Existent Citations and Links when Prompted by Users
Nov 2022 · OpenAI
DoNotPay Allegedly Misrepresented Its AI "Robot Lawyer" Product
Mar 2023 · DoNotPay
Live facial recognition is tracking kids suspected of being criminals
Oct 2020 · Buenos Aires city government
Erroneous Declined Transaction Notification by PayPal AI Assistant
Jun 2024 · PayPal
Year-long AI Surveillance Pilot in Two South Australian Aged Care Facilities Reportedly Overwhelmed Staff with False Positives
Mar 2021 · South Australia Health
Facebook’s Automated Content Moderation Tool Flagged a Post Containing Parts of the Declaration of Independence as Hate Speech by Mistake
Jul 2018 · Facebook
MyPillow Defense Lawyers in Coomer v. Lindell Reportedly Sanctioned for Filing Court Document Allegedly Containing AI-Generated Legal Citations
Feb 2025 · McSweeny Synkar and Kachouroff PLLC, Jennifer T. DeMaster, Christopher I. Kachouroff
Alleged AI-Generated Photo Alteration Leads to Inappropriate Modifications in Speaker's Conference Picture
Oct 2024 · Unknown conference employee
Facewatch Reported to Have Wrongfully Flagged Home Bargains Customer as Shoplifter
May 2024 · Home Bargains
AI-Powered Taco Bell Drive-Thru Reportedly Disrupted by Viral Prank Ordering 18,000 Water Cups
Aug 2025 · Taco Bell
State Farm Allegedly Discriminated against Black Customers in Claim Payout
Jun 2021 · State Farm
ChatGPT-Powered Bing Reportedly Had Problems with Factual Accuracy on Some Controversial Topics
Feb 2023 · Microsoft
Australian Automated Debt Assessment System Issued False Notices to Thousands
Jul 2015 · Australian Department of Human Services
Facial Recognition in Remote Learning Software Reportedly Failed to Recognize a Black Student’s Face
Feb 2021 · unknown
NarxCare’s Risk Score Model Allegedly Lacked Validation and Trained on Data with High Risk of Bias
Jul 2020 · NarxCare, AvertD, Appriss