Paper: arXiv 2508.20097
Authors: Andrew Blair-Stanek, Nils Holzenberger, Benjamin Van Durme
Abstract
We investigate whether large language models can discover and analyze U.S. tax-minimization strategies. This real-world domain challenges even seasoned human experts, and progress can reduce tax revenue lost from well-advised, wealthy taxpayers. We evaluate the most advanced LLMs on their ability to (1) interpret and verify tax strategies, (2) fill in gaps in partially specified strategies, and (3) generate complete, end-to-end strategies from scratch. This domain should be of particular interest to the LLM reasoning community: unlike synthetic challenge problems or scientific reasoning tasks, U.S. tax law involves navigating hundreds of thousands of pages of statutes, case law, and administrative guidance, all updated regularly. Notably, LLM-based reasoning identified an entirely novel tax strategy, highlighting these models’ potential to revolutionize tax agencies’ fight against tax abuse.
Complexity vs Empirical Score
- Math Complexity: 2.0/10
- Empirical Rigor: 7.5/10
- Quadrant: Street Traders — empirical and applied, lower math complexity
Why this score: The paper’s core math is low-complexity linguistic prompting and legal analysis, but it demonstrates high empirical rigor through a handcrafted dataset (Shelter Check), GitHub code, and expert-graded strategy generation from real-world LLMs.
Research Flowchart
flowchart TD A["Research Goal<br>Can LLMs Identify Tax Abuse?"] --> B["Methodology: Three Evaluation Tasks<br>(1) Interpret & Verify<br>(2) Fill Gaps<br>(3) Generate Strategies"] B --> C["Computational Process<br>LLM Reasoning on Tax Law<br>Simulated Expert Interactions"] C --> D["Key Findings & Outcomes<br>Novel Strategy Discovery<br>Verification Capability<br>Potential for Tax Agency Use"]
Evidence Card
What the authors claim, on what data, and who has checked it. Read from the paper's own text by a language model: every quote is verbatim, nothing was re-run. How to read this card
Proposes
The paper introduces the Shelter Check dataset of 36 U.S. tax‑minimization strategies and benchmarks several leading LLMs on interpreting, verifying, and generating such strategies.
Source sentence
We introduce a new domain-expert crafted dataset called Shelter Check, derived from public sources that describe schemes used in the past by taxpayers to minimize taxes.
Assumes 1
- The tax‑minimization strategies in the study depend on textual tax‑law authorities rather than on asset‑valuation uncertainty. data
Source sentence
3 The Shelter Check Dataset The Shelter Check dataset currently contains 36 known U.S. tax-minimization strategies that depend upon applying tex-tual authorities. Why only 36?
Data
- Shelter Check text-news public
Source sentence
We introduce a new domain-expert crafted dataset called Shelter Check, derived from public sources that describe schemes used in the past by taxpayers to minimize taxes. The dataset includes the text of the tax-law authorities, like tax code sections or court decisions, that enabled the scheme.
Costs
Not applicable
Finds 3
- LLM‑based reasoning identified an entirely novel tax strategy, demonstrating potential for tax agencies. other
Source sentence
Notably, LLM-based reasoning identified an entirely novel tax strategy, highlighting these models' potential to revolutionize tax agencies' fight against tax abuse.
- All three reasoning models (o3, claude‑4, gemini‑2.5) outperformed the baseline llama‑3.3‑70B on the step‑cloze and generation tasks. performance figure from elsewhere in the paper
Source sentence
The three reasoning models outperformed the baseline llama-3.3-70B. so that replacing a specified correct step with the adversarial step would keep the strategy from working (i.e. achieving all listed goals).
- When adversarial steps were added, the reasoning models initially showed agreement bias, but after iterative refinement they returned 0/36 correct strategies, indicating resistance to finding non‑viable strategies. negative
Source sentence
But the LLMs proved remarkably resistant to finding that a strategy does not work. We improved the adversarial steps and expressly eliminated unwarranted factual assumptions to minimize the number of strategies the models found cor-rect, until all three reasoning models returned 0/36 as being correct.
Checked by others
No other paper in the archive has been found to support or contradict these findings (compared with 8 related papers). Coverage is partial, so this is neither replication nor refutation.
Extracted 2026-11-14 · same card as JSON · A language model chose these items: check the source sentence before relying on one.
Cited References (64)
- Rethinking the Development of Large Language Models from the Causal Perspective: A Legal Text Prediction Case Study (2024) — link
- RR-PU: A Synergistic Two-Stage Positive and Unlabeled Learning Framework for Robust Tax Evasion Detection (2024) — link
- GxVAEs: two joint VAEs generate hit molecules from gene expression profiles (2024) — link
- Black-Box Optimization with Implicit Constraints for Public Policy (2025) — link
- J & H: Evaluating the Robustness of Large Language Models Under Knowledge-Injection Attacks in Legal Domain (2025) — link
- Blackboard Systems (1986)
- Communication, Simulation, and Intelligent Agents: Implications of Personal Intelligent Machines for Medical Education (1983)
- Classification Problem Solving (1984)
and 56 more
- New Ways to Make Microcircuits Smaller (1980) — link
- New Ways to Make Microcircuits Smaller—Duplicate Entry (1980)
- Strategic explanations for a diagnostic consultation system (1984) — link
- Strategic Explanations in Consultation—Duplicate (1983)
- Poligon: A System for Parallel Problem Solving (1986)
- Transfer of Rule-Based Expertise through a Tutorial Dialogue (1979)
- The Engineering of Qualitative Models (2021)
- Attention Is All You Need (2017) — link
- Pluto: The ‘Other’ Red Planet (2015)
- Deep Blue (2002) — link
- A Survey on Large Language Models for Automated Planning (2025) — link
- Overview and Discussion of the Competition on Legal Information, Extraction/Entailment (COLIEE) 2023 (2024) — link
- Swiss-Judgment-Prediction: A Multilingual Legal Judgment Prediction Benchmark (2021) — link
- JEC-QA: A Legal-Domain Question Answering Dataset (2020) — link
- ILDC for CJPE: Indian Legal Documents Corpus for Court Judgment Prediction and Explanation (2021) — link
- LexGLUE: A Benchmark Dataset for Legal Language Understanding in English (2022) — link
- Computational Discovery of Transaction-Based Financial Crime via Grammatical Evolution: The Case of Ponzi Schemes (2022) — link
- Can AI expose tax loopholes? Towards a new generation of legal policy assistants (2025) — link
- Attention is All you Need (2017) — link
- Accurate structure prediction of biomolecular interactions with AlphaFold 3 (2024)
- Machine science (2010)
- GPT Gets Its First Law School B-pluses (2024)
- Can GPT-3 Perform Statutory Reasoning? (2023) — link
- A Dataset for Statutory Reasoning in Tax Law Entailment and Question Answering (2020) — link
- LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models (2023) — link
- Detecting tax evasion: a co-evolutionary approach (2016) — link
- Using LLMs to Discover Legal Factors (2024) — link
- Predicting Outcomes of Case-Based Legal Arguments (2003) — link
- The Design of a Legal Analysis Program (1983) — link
- Catala: a programming language for the law (2021) — link
- The British Nationality Act as a logic program (1986) — link
- Reflections on ‘‘Taxman": An Experiment in Artificial Intelligence and Legal Reasoning (1977) — link
- Crafting Papers on Machine Learning (2000)
- The Need for Biases in Learning Generalizations (1980)
- Computational Complexity of Machine Learning (1989)
- Machine Learning: An Artificial Intelligence Approach, Vol. I (1983)
- Pattern Classification (2000)
- Suppressed for Anonymity (2021)
- Mechanisms of Skill Acquisition and the Law of Practice (1981)
- Some Studies in Machine Learning Using the Game of Checkers (1959)
- Wiz Research Uncovers Exposed DeepSeek Database Leaking Sensitive Information, Including Chat History (2025) — link
- Tax non-compliance detection using co-evolution of tax evasion risk and audit likelihood (2015)
- The general theory of tax avoidance (1985)
- Federal Taxation of Income, Estates and Gifts \textparagraph 4.3 (2025)
- Esmark, I nc. v. C ommissioner (1988)
- Notice 2003-54, COMMON TRUST FUND STRADDLE TAX SHELTER (2003)
- Common Trust Funds: The Living Fossil of Passthroughs (2010)
- Recognized abusive and listed transactions (2025) — link
- Large language models as tax attorneys: a case study in legal capabilities emergence (2024)
- Tax Gap Projections Methodology: Tax Year 2022 (2024)
- Base erosion and profit shifting (2025) — link
- Rulearena: A benchmark for rule-guided reasoning with llms in real-world scenarios (2024) — link
- The Ideation-Execution Gap: Execution Outcomes of LLM-Generated versus Human Research Ideas (2025) — link
- Shelter Check: Proactively Finding Tax Minimization Strategies via AI (2025) — link
- Intellectual Property Law Solutions to Tax Avoidance (2015)
- AI, Taxation, and Valuation (2023) — link
0 of 64 cited works are in the corpus; resolved via arXiv source bundles, Semantic Scholar and OpenAlex. Corpus-wide: 2596 linked papers.