Publications on Flaky Tests

Beyond Test Flakiness: A Manifesto for a Holistic Approach to Test Suite Health
P McMinn, MF Roslan and GM Kapfhammer
Flaky Tests Workshop (FTW), 2025
[]
@inproceedings{McMinn2025,
  author    = "McMinn, Phil and Roslan, Muhammad Firhard and Kapfhammer, Gregory M.",
  title     = "Beyond Test Flakiness: A Manifesto for a Holistic Approach to Test Suite Health",
  booktitle = "Flaky Tests Workshop (FTW)",
  year      = "2025"
}

Empirically Evaluating Flaky Tests for Autonomous Driving Systems in Simulated Environments
AM Osikowicz, P McMinn and D Shin
Flaky Tests Workshop (FTW), 2025
[]
@inproceedings{Osikowicz2025,
  author    = "Osikowicz, Aleksander M. and McMinn, Phil and Shin, Donghwan",
  title     = "Empirically Evaluating Flaky Tests for Autonomous Driving Systems in Simulated
               Environments",
  booktitle = "Flaky Tests Workshop (FTW)",
  year      = "2025"
}

Do Automatic Test Generation Tools Generate Flaky Tests?
M Gruber, MF Roslan, O Parry, F Scharnböck, P McMinn and G Fraser
International Conference on Software Engineering (ICSE), 2024
[PDF]
[DOI]
[]
@inproceedings{Gruber2024,
  author    = "Gruber, Martin and Roslan, Muhammad Firhard and Parry, Owain and Scharnb{\"o}ck,
               Fabian and McMinn, Phil and Fraser, Gordon",
  title     = "Do Automatic Test Generation Tools Generate Flaky Tests?",
  booktitle = "International Conference on Software Engineering (ICSE)",
  year      = "2024"
}

Empirically Evaluating Flaky Test Detection Techniques Combining Test Case Rerunning and Machine Learning Models
O Parry, M Hilton, GM Kapfhammer and P McMinn
Empirical Software Engineering, 2023
[PDF]
[DOI]
[]
@article{Parry2023,
  author  = "Parry, Owain and Hilton, Michael and Kapfhammer, Gregory M. and McMinn, Phil",
  title   = "Empirically Evaluating Flaky Test Detection Techniques Combining Test Case Rerunning
             and Machine Learning Models",
  journal = "Empirical Software Engineering",
  year    = "2023",
  issue   = "3",
  volume  = "28"
}

What Do Developer-Repaired Flaky Tests Tell Us About the Effectiveness of Automated Flaky Test Detection?
O Parry, M Hilton, GM Kapfhammer and P McMinn
International Conference on Automation of Software Test (AST), 2022
[PDF]
[DOI]
[]
@inproceedings{Parry2022,
  author    = "Parry, Owain and Hilton, Michael and Kapfhammer, Gregory M. and McMinn, Phil",
  title     = "What Do Developer-Repaired Flaky Tests Tell Us About the Effectiveness of Automated
               Flaky Test Detection?",
  booktitle = "International Conference on Automation of Software Test (AST)",
  year      = "2022"
}

Surveying the Developer Experience of Flaky Tests
O Parry, M Hilton, GM Kapfhammer and P McMinn
International Conference on Software Engineering — Software Engineering In Practice track (ICSE-SEIP), 2022
[PDF]
[DOI]
[]
@inproceedings{Parry2022a,
  author    = "Parry, Owain and Hilton, Michael and Kapfhammer, Gregory M. and McMinn, Phil",
  title     = "Surveying the Developer Experience of Flaky Tests",
  booktitle = "International Conference on Software Engineering --- Software Engineering In
               Practice track (ICSE-SEIP)",
  year      = "2022"
}

Evaluating Features for Machine Learning Detection of Order- And Non-Order-Dependent Flaky Tests
O Parry, M Hilton, GM Kapfhammer and P McMinn
International Conference on Software Testing, Verification and Validation (ICST), 2022
[PDF]
[DOI]
[]
@inproceedings{Parry2022b,
  author    = "Parry, Owain and Hilton, Michael and Kapfhammer, Gregory M. and McMinn, Phil",
  title     = "Evaluating Features for Machine Learning Detection of Order- And Non-Order-Dependent
               Flaky Tests",
  booktitle = "International Conference on Software Testing, Verification and Validation (ICST)",
  year      = "2022"
}

A Survey of Flaky Tests
O Parry, M Hilton, GM Kapfhammer and P McMinn
ACM Transactions on Software Engineering and Methodology, 2021
[PDF]
[DOI]
[]
@article{Parry2021,
  author  = "Parry, Owain and Hilton, Michael and Kapfhammer, Gregory M. and McMinn, Phil",
  title   = "A Survey of Flaky Tests",
  journal = "ACM Transactions on Software Engineering and Methodology",
  year    = "2021"
}

Flake It 'Till You Make It: Using Automated Repair to Induce and Fix Latent Test Flakiness
O Parry, M Hilton, GM Kapfhammer and P McMinn
International Workshop on Automated Program Repair (APR), 2020
[PDF]
[DOI]
[]
@inproceedings{Parry2020,
  author    = "Parry, Owain and Hilton, Michael and Kapfhammer, Gregory M. and McMinn, Phil",
  title     = "Flake It 'Till You Make It: Using Automated Repair to Induce and Fix Latent Test
               Flakiness",
  booktitle = "International Workshop on Automated Program Repair (APR)",
  year      = "2020"
}