How Java Programmer Test Exceptional Behavior

Accepted for publication at the 2021 Mining Software Repositories Conference (MSR 2021)

Preprint

Replication package including the tool, the dataset and all scripts used for the analysis

4 minutes summary video

Slides

Abstract

Exceptions often signal faulty or undesired behavior; hence, high-quality test suites should also target exceptional behavior. This paper is a large-scale study of exceptional tests—which exercise exceptional behavior—in 1 157 open-source Java projects hosted on GitHub.

We analyzed JUnit exceptional tests to understand what kinds of exceptions are more frequently tested, what coding patterns are used, and how features of a project, such as its size and number of contributors, correlate to the characteristics of its exceptional tests.

We found that:

The paper also zeroes in on several detailed examples involving some of the largest analyzed projects, which refine the empirical results with qualitative evidence. The study’s findings, and the capabilities ofthe tool we developed to analyze exceptional tests, suggest several implications for the practice of software development and for follow-up empirical studies.