Paper: arXiv 2405.09360

Authors: Tolulope Fadina, Thorsten Schmidt

Abstract

Fairness in decision-making processes is often quantified using probabilistic metrics. However, these metrics may not fully capture the real-world consequences of unfairness. In this article, we adopt a utility-based approach to more accurately measure the real-world impacts of decision-making process. In particular, we show that if the concept of $\varepsilon$-fairness is employed, it can possibly lead to outcomes that are maximally unfair in the real-world context. Additionally, we address the common issue of unavailable data on false negatives by proposing a reduced setting that still captures essential fairness considerations. We illustrate our findings with two real-world examples: college admissions and credit risk assessment. Our analysis reveals that while traditional probability-based evaluations might suggest fairness, a utility-based approach uncovers the necessary actions to truly achieve equality. For instance, in the college admission case, we find that enhancing completion rates is crucial for ensuring fairness. Summarizing, this paper highlights the importance of considering the real-world context when evaluating fairness.

Complexity vs Empirical Score

  • Math Complexity: 5.5/10
  • Empirical Rigor: 3.0/10
  • Quadrant: Lab Rats — theoretically deep, empirically untested

Why this score: The paper provides a strong, mathematically sound critique of standard epsilon-fairness metrics by introducing a utility-based framework. However, it relies primarily on theoretical examples and conceptual formulations rather than rigorous empirical backtesting or large dataset evaluations.

Research Flowchart

  flowchart TD
    A[Research Goal: Evaluate real-world impacts of ε-fairness using a utility-based framework] --> B[Inputs: Case Studies & Datasets]
    B -->|College Admissions & Credit Risk| C[Methodology: Utility-Based Metric Framework]
    C --> D[Computational Analysis: Evaluate Probabilistic vs. Utility Outcomes]
    D --> E[Reduced Setting Analysis: Address Missing False Negative Data]
    E --> F[Key Finding: ε-fairness can cause maximal real-world unfairness]
    E --> G[Key Finding: Achieving true fairness requires contextual action]