Why AI Fairness Matters More Than Ever
AI systems that discriminate — against job applicants based on gender, against loan seekers based on race, against patients based on age — are not just an ethical failure. They are a legal risk. The EU AI Act (Regulation 2024/1689), fully applicable from August 2026, imposes mandatory fairness and bias obligations on high-risk AI systems used in employment, credit, healthcare, law enforcement, and other high-stakes domains. Violations carry fines of up to €30 million or 6% of global annual turnover — whichever is higher.
Beyond regulatory pressure, AI bias causes real harm: people denied loans they qualify for, candidates screened out of jobs based on proxy variables for protected characteristics, healthcare diagnoses less accurate for underrepresented demographic groups. Building fair AI is not optional.
EU AI Act Art. 10: Training Data Governance
Article 10 of the EU AI Act is the primary data governance requirement for high-risk AI systems. It requires that training, validation, and testing data sets:
- Are subject to appropriate data governance and management practices
- Are relevant, representative, free of errors, and complete to the extent possible
- Have appropriate statistical properties with regard to the persons or groups against whom the AI system is intended to be used
- Take into account characteristics or elements particular to the specific geographical, behavioural, or functional setting of intended deployment
Article 10(5) creates a narrow exception allowing processing of special categories of personal data (race, ethnicity, health, sexual orientation) to the extent strictly necessary to detect and correct biases — with strict safeguards including DPIA, access controls, and deletion after bias correction.
In practice, Art. 10 requires a formal training data audit before deployment of any high-risk AI system:
- Document data sources, collection methodology, and preprocessing steps
- Analyse demographic representation in training data relative to the target population
- Identify proxy variables that may encode protected attributes (postcode for race, name for gender)
- Assess labeller bias and inter-annotator agreement in supervised learning datasets
- Publish data cards documenting limitations
Fairness Metrics: The Core Vocabulary
There is no single universal definition of algorithmic fairness. Different metrics capture different concepts of fair treatment, and — critically — they are mathematically incompatible (you cannot simultaneously satisfy all of them). Understanding the key metrics is essential for making informed choices:
Demographic Parity (Statistical Parity)
The positive prediction rate is equal across all demographic groups. If a loan approval model approves 70% of applications from Group A, it should approve approximately 70% from Group B. This metric does not account for whether the base rates of the target variable differ across groups.
When to use: Situations where historical disparities in outcomes are themselves a consequence of past discrimination (e.g. credit access). Limitation: May produce less accurate predictions for individual groups if true base rates differ.
Equalised Odds
Both true positive rate (TPR) and false positive rate (FPR) are equal across groups. This means the model is equally accurate for positive cases and equally inaccurate for negative cases across demographics. Proposed by Hardt, Price, and Srebro (2016).
When to use: High-stakes decisions where both false positives and false negatives have serious consequences (recidivism risk scores, medical diagnosis). Limitation: Cannot simultaneously achieve equalised odds and demographic parity unless base rates are equal across groups.
Equal Opportunity
A relaxation of equalised odds — only the true positive rate is equalised across groups. False positive rates may differ. This ensures that qualified individuals from all groups are equally likely to receive a positive prediction.
When to use: Situations where false negatives (missing a qualified candidate) are the primary concern — employment screening, university admissions.
Individual Fairness
Similar individuals should receive similar predictions. Formally: the model output should be a Lipschitz function of a task-specific similarity metric. Requires defining what "similar" means — a non-trivial challenge in practice.
When to use: When group-level metrics are insufficient and case-by-case consistency matters. Challenge: Defining the similarity metric in a legally defensible way.
Calibration
Predicted probabilities match actual outcomes across groups. If a model predicts 70% recidivism risk for individuals from Group A, approximately 70% of those individuals should actually reoffend — and the same should hold for Group B. Calibration is compatible with equalised odds only under equal base rates.
The Impossibility Theorem
Chouldechova (2017) and Kleinberg et al. (2016) proved that calibration, equalised odds, and demographic parity cannot all hold simultaneously when base rates differ between groups. This is not a software bug — it is a mathematical constraint. Organisations must make an explicit, documented choice about which fairness metric to prioritise and why, then get that choice approved by legal, ethics, and business stakeholders.
Common Types of Algorithmic Bias
- Selection bias: Training data not representative of the deployment population. E.g. a hiring model trained on historical successful hires that were predominantly male.
- Measurement bias: Proxy variables encoding protected attributes. Postcode correlates with race and socioeconomic status. Job title correlates with gender. University name correlates with socioeconomic background.
- Label bias: Annotator discrimination in training data labelling. Resume screeners rating the same resume lower when a stereotypically female name is attached (demonstrated in multiple studies).
- Aggregation bias: A single model for heterogeneous groups ignores sub-group differences. A one-size-fits-all medical diagnostic model may perform worse for demographic groups underrepresented in clinical training data.
- Evaluation bias: Benchmark datasets that don't represent all groups. Facial recognition systems evaluated on predominantly light-skinned faces will have inflated accuracy figures.
- Historical bias: Training on historical outcomes encodes past discrimination. A loan approval model trained on decisions made under discriminatory policies will perpetuate those patterns.
Bias Detection: Practical Methods
Pre-Processing Techniques
Applied before model training: reweighting under-represented groups; resampling (oversampling minority groups or undersampling majority groups); suppressing protected attributes and their proxies from the feature set; applying disparate impact remover (Feldman et al. 2015) to reduce feature correlation with protected attributes.
In-Processing Techniques
Applied during model training: fairness constraints added to the optimisation objective (e.g. constraining demographic parity difference to ≤ 0.1); adversarial debiasing (training a second model to predict sensitive attributes and penalising the main model for enabling that prediction); reductions approach (Agarwal et al. 2018) converting fairness into a constrained optimisation problem.
Post-Processing Techniques
Applied to model outputs: threshold adjustment — using different classification thresholds for different groups to achieve equalised odds; calibrated equalised odds (Pleiss et al. 2017); reject option classification — abstaining from predictions in the uncertainty boundary near the threshold.
Intersectional Fairness
Testing fairness along a single demographic dimension (e.g. gender alone) can mask discrimination against people who belong to multiple marginalised groups. A model may be fair across gender and race separately but discriminate significantly against Black women — a phenomenon described by Kimberlé Crenshaw's intersectionality framework.
Practical intersectional testing: create subgroup partitions for all meaningful combinations of protected attributes in scope (gender × race × age, etc.). Test fairness metrics for each subgroup. This requires sufficient data volume in each intersection — small intersectional groups with fewer than ~100 observations may produce unreliable metrics, which itself warrants a note in the audit.
EU AI Act Art. 14: Human Oversight
For high-risk AI systems, Art. 14 requires effective human oversight measures, including: the ability for human reviewers to understand the system's capabilities and limitations; the ability to detect and address malfunctions or unexpected outputs; the ability to decide not to use the system or override its output. This directly implies:
- Explainability mechanisms (SHAP, LIME, attention maps) so reviewers understand why a decision was made
- Counterfactual explanations for adverse decisions ("Changing X would have changed the outcome")
- Training for human reviewers on automation bias — the documented tendency for human reviewers to over-rely on algorithmic recommendations without independent judgment
- Accessible appeal mechanisms for individuals affected by adverse AI decisions (aligned with GDPR Art. 22)
Building a Bias Audit Programme
A practical programme for continuous fairness governance:
- Define scope: Which AI systems are in scope? Prioritise by impact on individuals and EU AI Act risk classification.
- Select fairness metrics: Choose with legal, ethics, and product teams. Document the trade-offs and get sign-off.
- Conduct training data audit: Per Art. 10 — representativeness, proxy variables, label quality.
- Run pre-deployment bias tests: Compute fairness metrics on held-out test set segmented by demographic. Document results.
- Implement monitoring: Track fairness metrics in production at minimum quarterly. Set alert thresholds.
- Complete FRIA: Art. 27 requires high-risk deployers to conduct a Fundamental Rights Impact Assessment before deploying high-risk AI.
- Establish incident response: What happens when a bias spike is detected? Who is notified? How is the model rolled back or adjusted?
- Third-party audit: Annual independent bias audit by an external assessor, especially before major model releases.
Recommended Tools
- Fairlearn (Microsoft): Open-source fairness assessment and mitigation library for Python. Dashboard visualisation, multiple fairness metrics, reductions approach.
- AI Fairness 360 (IBM): 70+ fairness metrics and 11 bias mitigation algorithms. Comprehensive but complex.
- What-If Tool (Google): Visual interface for fairness exploration, counterfactual analysis, and slice-based performance analysis.
- Aequitas: Bias and fairness audit toolkit from the University of Chicago Center for Data Science and Public Policy.
- SHAP / LIME: Model explainability libraries essential for human oversight requirements.
Audit Your AI System's Fairness
ComplyKit's free AI Fairness & Bias Audit Checklist covers 42 controls across training data governance, fairness metrics, model transparency, human oversight, ongoing monitoring, and accountability governance — and generates a professional audit report aligned with EU AI Act Art. 10, 13, 14, and IEEE 7003. No account required.