Submitted:
21 March 2025
Posted:
21 March 2025
You are already at the latest version
Abstract
Keywords:
1. Introduction
1.1. Background Information
1.2. Objectives of the Review Paper
- To examine the root causes of AI bias in machine learning models.
- To analyze the impact of AI bias on decision-making in different domains.
- To evaluate current bias detection and mitigation strategies.
- To propose future research directions for achieving fairness in AI systems.
1.3. Research Importance
2. Methodology
2.1. Research Design
2.1.1. Search Strategy
- (“AI bias” OR “algorithmic discrimination” OR “machine learning fairness”) AND (“decision-making” OR “ethics” OR “governance”)
- (“bias in AI” OR “fair AI” OR “algorithmic bias”) AND (“social impact” OR “mitigation techniques”)
2.1.2. Inclusion and Exclusion Criteria
- Peer-reviewed journal articles, conference papers, and government reports.
- Studies focused on AI bias detection, mitigation, and ethical considerations.
- Papers published in English.
- Opinion articles, blog posts, and non-peer-reviewed sources.
- Studies without empirical evidence or theoretical contributions.
- Papers focusing on AI bias outside decision-making contexts (e.g., AI in gaming).
2.1.3. Data Extraction & Synthesis
2.1.4. Limitations
- Possible selection bias in database indexing.
- Variability in AI bias measurement across studies.
- Ethical considerations are often theoretical, lacking empirical validation.
3. Literature Review
3.1. Origins of AI Bias
3.2. Impact of AI Bias on Decision-Making
3.3. Bias Detection Methods
- Demographic Parity: This metric ensures that the probability of a positive outcome is independent of membership in a protected group. For instance, in a hiring algorithm, demographic parity is achieved if candidates from different demographic groups have equal chances of being selected (Pagano et al., 2022).
- Equalized Odds: This criterion requires that the true positive rate and false positive rate are equal across all demographic groups. It ensures that an AI model’s accuracy is consistent, regardless of group membership (Ghai & Mueller, 2022).
- Disparate Impact Analysis: This analysis evaluates whether a decision-making process disproportionately affects a particular group. A commonly used threshold is the "80% rule," where a selection rate for any group less than 80% of the rate for the most favored group may indicate potential bias (Chakraborty et al., 2020).
- AI Ethics Toolkits: Tools such as AI Fairness 360 offer resources for detecting, understanding, and mitigating unwanted algorithmic biases. These toolkits provide a suite of algorithms and metrics to assess fairness throughout the AI development lifecycle (Pagano et al., 2022).
- Fairness Benchmarks: Standardized benchmarks have been introduced to evaluate the fairness of AI systems. For example, the Fairlearn toolkit provides measurement models for assessing fairness, such as demographic parity, enabling a better understanding of how AI models impact different demographic groups (Ghai & Mueller, 2022).
3.4. Mitigation Strategies
3.5. Review of Relevant Theories
3.6. Theoretical Implications
4. Future Directions
4.1. Longitudinal Studies
- Comprehensive Monitoring: Implement continuous monitoring frameworks to track AI system performance and fairness metrics over extended periods, facilitating the early detection of emerging biases.
- Adaptive Mitigation Techniques: Develop and refine bias mitigation strategies that can adapt to changing data patterns and operational contexts, maintaining their effectiveness as AI systems evolve.
- Cross-Domain Studies: Conduct longitudinal research across diverse application domains to understand how biases manifest differently and to identify domain-specific challenges and solutions.
4.2. Intervention Studies
- Domain-Specific Evaluations: Conduct empirical studies across diverse AI domains, such as finance, healthcare, and criminal justice, to assess the generalizability of bias mitigation strategies.
- Longitudinal Studies: Implement longitudinal research designs to monitor the sustained effectiveness of bias mitigation techniques over time and in dynamic environments.
- Scalability Assessments: Examine the scalability of these strategies in large-scale deployments to ensure their practicality in real-world settings.
4.3. Ethical Frameworks
- Interdisciplinary Collaboration: Encourage collaboration between policymakers, technologists, ethicists, and legal experts to develop comprehensive ethical frameworks that address the multifaceted challenges of AI bias and accountability.
- Continuous Monitoring and Evaluation: Implement mechanisms for the ongoing assessment of AI systems to ensure compliance with established ethical standards and regulatory guidelines, adapting to emerging challenges and technological advancements.
- Public Engagement and Transparency: Promote transparency in AI development processes and engage with diverse stakeholders, including the public, to build trust and ensure that AI systems align with societal values and expectations.
5. Conclusions
- Origins of AI Bias: Bias in AI systems often stems from imbalances in training data, algorithmic design flaws, and subjective human decisions during development and deployment (Mehrabi et al., 2021). Biases are not always evident in early-stage model training but can emerge over time as models interact with real-world data. Addressing these biases requires a proactive approach, incorporating continuous monitoring mechanisms.
- Impact on Various Sectors: AI bias manifests differently across domains. In healthcare, biased AI can lead to disparities in diagnostic accuracy and treatment recommendations. In law enforcement, it may result in racial profiling through predictive policing models. In finance, discriminatory lending practices can arise from biased credit-scoring algorithms. In employment, biases in hiring algorithms can disadvantage certain demographic groups (Mehrabi et al., 2021). While these issues are well-documented, sector-specific interventions remain underdeveloped, requiring tailored bias mitigation techniques for each industry.
- Bias Detection Techniques: Fairness metrics such as demographic parity, equalized odds, and disparate impact analysis are widely utilized to detect and quantify bias in AI systems (Verma & Rubin, 2018). However, current fairness metrics often struggle with the trade-off between interpretability and effectiveness. Many practitioners rely on these metrics without considering their limitations, such as the difficulty in balancing fairness constraints with predictive accuracy.
- Mitigation Strategies: Approaches like data preprocessing to rebalance training datasets and the development of fairness-aware machine learning models are employed to mitigate bias. Continuous evaluation of these strategies is necessary to maintain their effectiveness in real-world applications (Friedler et al., 2019). Many bias mitigation strategies remain experimental and lack empirical validation across diverse datasets and application areas. Future research should focus on refining these techniques to ensure their practical deployment and effectiveness in addressing bias at scale.
Call to Action
- Integration of Bias Detection and Mitigation: AI developers should embed bias detection and mitigation strategies throughout the model development lifecycle. This includes implementing practices such as data balancing—through undersampling, oversampling, or synthetic sampling—and augmentation to create representative datasets. Additionally, algorithmic adjustments may be necessary to address inherent biases in the models (Hardt et al., 2016; Feldman & Peake, 2021).
- Enforcement of AI Governance Frameworks: Policymakers are urged to establish and enforce comprehensive AI governance frameworks that emphasize transparency, accountability, fairness, and ethics. Such frameworks should set standards for data handling, model explainability, and decision-making processes to foster responsible AI innovation while mitigating risks related to bias and privacy breaches (Ferrara, 2023; IEEE, 2022).
- Focus on Long-Term Monitoring and Intervention: Future research should prioritize the continuous monitoring of AI systems to detect and address biases that may emerge over time. This involves systematic real-world testing and evaluation to gauge models' practical impact and to develop intervention strategies that ensure fairness and equity in dynamic environments (Koene, 2017; Sutton et al., 2018).
References
- Bartlett, R., Morse, A., Stanton, R., & Wallace, N. (2019). Consumer-lending discrimination in the FinTech era. National Bureau of Economic Research. [CrossRef]
- Buolamwini, J., & Gebru, T. (2018). Gender shades: Intersectional accuracy disparities in commercial gender classification. Proceedings of Machine Learning Research, 81, 1-15. http://proceedings.mlr.press/v81/buolamwini18a.html.
- Dastin, J. (2018, October 10). Amazon scraps secret AI recruiting tool that showed bias against women. Reuters. https://www.reuters.com/article/us-amazon-com-jobs-automation-insight-idUSKCN1MK08G.
- Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019). Dissecting racial bias in an algorithm used to manage the health of populations. Science, 366(6464), 447-453. [CrossRef]
- Buolamwini, J., & Gebru, T. (2018). Gender shades: Intersectional accuracy disparities in commercial gender classification. Proceedings of Machine Learning Research, 81, 1-15.
- Ferrara, E. (2023). Fairness and bias in artificial intelligence: A brief survey of sources, impacts, and mitigation strategies. arXiv preprint arXiv:2304.07683.
- National Institute of Standards and Technology. (2022). There's more to AI bias than biased data, NIST report highlights. Retrieved from https://www.nist.gov/news-events/news/2022/03/theres-more-ai-bias-biased-data-nist-report-highlights.
- Adamson, A. S., & Smith, A. (2022). Machine learning and health care disparities: A critical review. Journal of the American Medical Association, 327(7), 627–635. [CrossRef]
- Banerjee, A., Chen, S., Reddy, S., & Cooper, J. (2021). The limitations of AI-driven diagnostic models for diverse populations: A dermatology case study. Nature Medicine, 27(5), 747–752. [CrossRef]
- Buolamwini, J., & Gebru, T. (2018). Gender shades: Intersectional accuracy disparities in commercial gender classification. Proceedings of Machine Learning Research, 81, 1–15.
- Chen, L., Liu, Z., & He, Q. (2021). Algorithmic fairness in hiring: Examining AI bias in resume screening models. Journal of Artificial Intelligence Research, 71, 345–362. [CrossRef]
- Fuster, A., Goldsmith-Pinkham, P., Ramadorai, T., & Walther, A. (2022). Predictably unequal? The effects of machine learning on credit markets. Journal of Finance, 77(4), 1813–1850. [CrossRef]
- Raghavan, M., Barocas, S., Kleinberg, J., & Levy, K. (2020). Mitigating bias in algorithmic hiring: Evaluating fairness-aware strategies in recruitment AI systems. Proceedings of the ACM Conference on Fairness, Accountability, and Transparency, 3, 265–279. [CrossRef]
- Richardson, R., Schultz, J. M., & Crawford, K. (2019). Dirty data, bad predictions: How civil rights violations impact predictive policing. New York University Law Review, 94(1), 192–229.
- Chakraborty, J., Majumder, S., Yu, Z., & Menzies, T. (2020). Fairway: A way to build fair ML software. arXiv preprint arXiv:2003.10354.
- Ferrara, E. (2023). Fairness and bias in artificial intelligence: A brief survey of sources, impacts, and mitigation strategies. arXiv preprint arXiv:2304.07683.
- Ghai, B. (2023). Towards fair and explainable AI using a human-centered AI approach. arXiv preprint arXiv:2306.07427.
- Balch, A. (2024). Why algorithms remain unjust: Power structures surrounding algorithmic activity. arXiv preprint arXiv:2405.18461.
- Pagano, T. P., Loureiro, R. B., Lisboa, F. V. N., Peixoto, R. M., Guimarães, G. A. S., Cruz, G. O. R., Araujo, M. M., Santos, L. L., Cruz, M. A. S., Oliveira, E. L. S., & others. (2023). Bias and unfairness in machine learning models: A systematic review on datasets, tools, fairness metrics, and identification and mitigation methods. Big Data and Cognitive Computing, 7(1), 15.
- Fairlearn Development Team. (n.d.). Fairness in machine learning—Fairlearn 0.13.0.dev0 documentation. Retrieved from https://fairlearn.org/main/user_guide/fairness_in_machine_learning.html.
- Chakraborty, J., Majumder, S., Yu, Z., & Menzies, T. (2020). Fairway: A way to build fair ML software. arXiv preprint arXiv:2003.10354.
- Ghai, B., & Mueller, K. (2022). D-BIAS: A causality-based human-in-the-loop system for tackling algorithmic bias. arXiv preprint arXiv:2208.05126.
- Pagano, T. P., Loureiro, R. B., Lisboa, F. V. N., Peixoto, R. M., Guimarães, G. A. S., Cruz, G. O. R., Araujo, M. M., Santos, L. L., Cruz, M. A. S., Oliveira, E. L. S., Winkler, I., & Nascimento, E. G. S. (2022). Bias and unfairness in machine learning models: A systematic literature review. arXiv preprint arXiv:2202.08176.
- Adadi, A., & Berrada, M. (2018). Peeking inside the black-box: A survey on Explainable Artificial Intelligence (XAI). IEEE Access, 6, 52138–52160. [CrossRef]
- Agarwal, R., Dudik, M., Wu, Z. S., & Hanna, J. (2020). Fairlearn: A toolkit for assessing and improving fairness in AI. ACM Conference on Fairness, Accountability, and Transparency. [CrossRef]
- Buijsman, S. (2023). Navigating fairness measures and trade-offs in AI systems. arXiv preprint arXiv:2307.08484.
- Lipton, Z. C. (2018). The mythos of model interpretability. Communications of the ACM, 61(10), 36–43. [CrossRef]
- Mittelstadt, B. (2019). Principles alone cannot guarantee ethical AI. Nature Machine Intelligence, 1(11), 501–507. [CrossRef]
- National Institute of Standards and Technology (NIST). (2022). AI risk management framework: A guide for trustworthy AI development. U.S. Department of Commerce.
- Pistilli, G., Munoz Ferrandis, C., Jernite, Y., & Mitchell, M. (2023). Stronger together: On the articulation of ethical charters, legal tools, and technical documentation in machine learning. arXiv preprint arXiv:2305.18615.
- Strickland, M. J., Farquhar, S., Stoyanovich, J., & Rosner, D. (2021). Fairness versus accuracy trade-offs in AI-driven healthcare systems: A review of bias mitigation strategies. Journal of Biomedical Informatics, 118, 103799. [CrossRef]
- Ferrara, E. (2023). Fairness and bias in artificial intelligence: A brief survey of sources, impacts, and mitigation strategies. arXiv preprint arXiv:2304.07683.
- van der Wal, O., Jumelet, J., Schulz, K., & Zuidema, W. (2022). The birth of bias: A case study on the evolution of gender bias in an English language model. arXiv preprint arXiv:2207.10245.
- Ezzeldin, Y. H., Yan, S., He, C., Ferrara, E., & Avestimehr, S. (2021). FairFed: Enabling group fairness in federated learning. arXiv preprint arXiv:2110.00857.
- Li, J., Li, Z., Wang, Y., Li, Y., & Wang, L. (2023). DBFed: Debiasing federated learning framework based on domain-independent adversarial training. arXiv preprint arXiv:2307.05582.
- Poulain, R., Tarek, M. F. B., & Beheshti, R. (2023). Improving fairness in AI models on electronic health records: The case for federated learning methods. arXiv preprint arXiv:2305.11386.
- Zhang, B. H., Lemoine, B., & Mitchell, M. (2018). Mitigating unwanted biases with adversarial learning. Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, 335–340. [CrossRef]
- Access Now, & Amnesty International. (2018). The Toronto Declaration: Protecting the rights to equality and non-discrimination in machine learning systems. https://www.torontodeclaration.org/.
- Deck, L., Müller, J.-L., Braun, C., Zipperling, D., & Kühl, N. (2024). Implications of the AI Act for non-discrimination law and algorithmic fairness. arXiv preprint arXiv:2403.20089.
- National Telecommunications and Information Administration (NTIA). (2023). Artificial Intelligence Accountability Policy Report. https://www.ntia.gov/report/2023/artificial-intelligence-accountability-policy-report.
- Whittaker, M., Alper, M., Bennett, C. L., Hendren, S., Kaziunas, E., Mills, M., & West, S. M. (2021). Disability, bias, and AI. AI Now Institute. https://ainowinstitute.org/disabilitybiasai-2021.pdf.
- Friedler, S. A., Scheidegger, C., Venkatasubramanian, S., Choudhary, S., Hamilton, E. P., & Roth, D. (2019). A comparative study of fairness-enhancing interventions in machine learning. Proceedings of the Conference on Fairness, Accountability, and Transparency, 329–338. [CrossRef]
- Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2021). A survey on bias and fairness in machine learning. ACM Computing Surveys, 54(6), 1–35. [CrossRef]
- Verma, S., & Rubin, J. (2018). Fairness definitions explained. 2018 IEEE/ACM International Workshop on Software Fairness (FairWare), 1–7. [CrossRef]
- Feldman, T., & Peake, A. (2021). End-to-end bias mitigation: Removing gender bias in deep learning. arXiv preprint arXiv:2104.02532.
- Ferrara, E. (2023). Fairness and bias in artificial intelligence: A brief survey of sources, impacts, and mitigation strategies. arXiv preprint arXiv:2304.07683.
- Hardt, M., Price, E., & Srebro, N. (2016). Equality of opportunity in supervised learning. Advances in Neural Information Processing Systems, 29, 3315–3323.
- IEEE. (2022). IEEE CertifAIEd™ - Ontological specification for ethical algorithmic bias. IEEE.
- Koene, A. (2017). Algorithmic bias: Addressing growing concerns [Leading Edge]. IEEE Technology and Society Magazine, 36(2), 31–32. [CrossRef]
- Sutton, A., Welfare, T., & Cristianini, N. (2018). Biased embeddings from wild data: Measuring, understanding and removing. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society (pp. 328–334). [CrossRef]
- Zou, J., & Schiebinger, L. (2018). AI can be sexist and racist—it's time to make it fair. Nature, 559(7714), 324–326. [CrossRef]
- Ferrara, E. (2023). Fairness and bias in artificial intelligence: A brief survey of sources, impacts, and mitigation strategies. arXiv preprint arXiv:2304.07683.
- Friedler, S. A., Scheidegger, C., Venkatasubramanian, S., Choudhary, S., Hamilton, E. P., & Roth, D. (2019). A comparative study of fairness-enhancing interventions in machine learning. Proceedings of the Conference on Fairness, Accountability, and Transparency, 329–338. [CrossRef]
- Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2021). A survey on bias and fairness in machine learning. ACM Computing Surveys, 54(6), 1–35. [CrossRef]
- Verma, S., & Rubin, J. (2018). Fairness definitions explained. 2018 IEEE/ACM International Workshop on Software Fairness (FairWare), 1–7. [CrossRef]
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2025 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
