Judging a commit by its cover; or can a commit message predict build failure?
- Published
- Accepted
- Subject Areas
- Data Mining and Machine Learning, Natural Language and Speech, Software Engineering
- Keywords
- commit messages, build status, travis-ci, github, language model, mining software repositories
- Copyright
- © 2016 Santos et al.
- Licence
- This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, reproduction and adaptation in any medium and for any purpose provided that it is properly attributed. For attribution, the original author(s), title, publication source (PeerJ PrePrints) and either DOI or URL of the article must be cited.
- Cite this article
- 2016. Judging a commit by its cover; or can a commit message predict build failure? PeerJ PrePrints 4:e1771v1 https://doi.org/10.7287/peerj.preprints.1771v1
Abstract
Developers summarize their changes to code in commit messages. When a message seems “unusual,” however, this puts doubt into the quality of the code contained in the commit. We trained \(n\)-gram language models and used cross-entropy as an indicator of commit message “unusualness” of over 120 000 commits from open source projects. Build statuses collected from Travis-CI were used as a proxy for code quality. We then compared the distributions of failed and successful commits with regards to the “unusualness” of their commit message. Our analysis yielded significant results when correlating cross-entropy with build status.
Author Comment
This is our work on the MSR 2016 challenge using Boa. The preprint has a little bit more content than the version submitted to MSR; we found the four-page limit a bit limiting, and may want to extend this work into a full paper.
Supplemental Information
Figure 1a: Histograms of commit message cross-entropies, initial model
Note the tall bins which contain a large number of auto-generated commit messages that were not foreseen when training this model.
Figure 1b: Histograms of commit message cross-entropies, refined model
We recalculated the histogram, removing auto-generated commits, as well as many non-English commit messages.
Figure 2: Empirical cumulative distribution functions of commits
Empirical cumulative distribution functions of number of passed (in green), failed (in purple), and errored (in orange) commits as cross-entropy (“unusualness”) increases. Note that failed, initially grows slower than passed and errored; by 10 bits, however, failed is indistinguishable from passed and errored.