We are AI

The concept of Artificial Intelligence (AI) is in an important sense a reflection of the choices humans make: systems are trained on material people produced, and they are optimised for goals people set. That inheritance shows up in measurable ways — in who a model gets wrong, and in what it asserts with complete confidence.

Bias is inherited, and it can be measured¶

Three well-documented cases show what "the model reflects its training data" means in practice.

Recruiting. Amazon built an experimental system from 2014 to screen résumés and scrapped it after it taught itself that male candidates were preferable: it penalised résumés containing the word "women's" and downgraded graduates of two all-women's colleges, as Reuters reported in 2018.
Face analysis. The peer-reviewed Gender Shades study (Buolamwini & Gebru, 2018) tested three commercial gender-classification systems and found that darker-skinned women were misclassified in up to 34.7 percent of cases, against a maximum error rate of 0.8 percent for lighter-skinned men. MIT News reported the same disparity: error rates above 20 percent for darker-skinned women, never worse than 0.8 percent for light-skinned men.
Risk scores. ProPublica's 2016 analysis of the COMPAS recidivism tool found that it wrongly flagged Black defendants as future criminals at almost twice the rate of white defendants, while the overall error rate for both groups was roughly equal.

That the model amplifies what is already overrepresented in its training data is the argument of On the Dangers of Stochastic Parrots (Bender, Gebru, McMillan-Major & Shmitchell, 2021): for US and UK English, the authors write, "white supremacist and misogynistic, ageist, etc. views are overrepresented in the training data", which sets models up to amplify those biases further.

Fluent output is not understanding¶

The same paper defines a language model as a system that stitches together sequences of linguistic forms "according to probabilistic information about how they combine, but without any reference to meaning". That is a narrower claim than saying a model lacks empathy — the sources reviewed for this post do not measure empathy, and the two should not be treated as the same thing. What is documented is that fluency and correctness are separate properties: a peer-reviewed survey of hallucination in natural language generation (Ji et al., ACM Computing Surveys, 2023) records that deep-learning text generation is "prone to hallucinate unintended text" that fails user expectations in real-world use. In a decision-support context — a risk score, a screening tool, a summary — that gap is where the risk sits.

What is being built around it¶

Because these systems make consequential choices, the design requirements have moved from opinion to published rules:

Transparency. NIST's AI Risk Management Framework (2023) exists to incorporate "trustworthiness considerations into the design, development, use, and evaluation of AI products, services, and systems".
Disclosure. Article 50 of the EU AI Act (Regulation (EU) 2024/1689) obliges providers to ensure that people are informed when they are interacting with an AI system, unless that is obvious from the context.
Interpretability. Article 13 of the same regulation requires high-risk systems to be designed so that their operation is "sufficiently transparent to enable deployers to interpret a system's output and use it appropriately".

The scale at which this matters¶

These are no longer pilot projects. Stanford HAI's AI Index Report 2025 records that 78 percent of organisations reported using AI in 2024, up from 55 percent a year earlier — and adds that AI-related incidents are rising sharply while standardised responsible-AI evaluations "remain rare among major industrial model developers".

Key takeaways¶

AI systems inherit human choices: the training data and the objective function are the design decision, and their effects are measurable (Gender Shades, COMPAS, Amazon's recruiting tool).
Fluency is not understanding; the documented failure mode is confident, plausible and wrong output (hallucination surveys), not "unfeeling" prose.
Transparency, disclosure and interpretability are now standards and legal requirements rather than preferences (NIST AI RMF; EU AI Act Articles 13 and 50).
Deployment has outrun evaluation: most organisations now use AI, while standardised testing of its risks lags behind.