John Penix
Works on deploying automated tools into developer workflows. He is on the Steering Committee of the IEEE/ACM International Conference on Automated Software Engineering, and has worked in the Automated Software Engineering R&D group at NASA's Ames Research Center before joining Google. He holds a Ph.D. in Computer Engineering from the University of Cincinnati.
Research Areas
Authored Publications
Sort By
Techniques for improving regression testing in continuous integration development environments
Sebastian Elbaum
Gregg Rothermel
Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, ACM (2014), pp. 235-245
Preview abstract
In continuous integration development environments, software engineers frequently integrate new or changed code with the mainline codebase. This can reduce the amount of code rework that is needed as systems evolve and speed up development time. While continuous integration processes traditionally require that extensive testing be performed following the actual submission of code to the codebase, it is also important to ensure that enough testing is performed prior to code submission to avoid breaking builds and delaying the fast feedback that makes continuous integration desirable. In this work, we present algorithms that make continuous integration processes more cost-effective. In an initial pre-submit phase of testing, developers specify modules to be tested, and we use regression test selection techniques to select a subset of the test suites for those modules that render that phase more cost-effective. In a subsequent post-submit phase of testing, where dependent modules as well as changed modules are tested, we use test case prioritization techniques to ensure that failures are reported more quickly. In both cases, the techniques we utilize are novel, involving algorithms that are relatively inexpensive and do not rely on code coverage information -- two requirements for conducting testing cost-effectively in this context. To evaluate our approach, we conducted an empirical study on a large data set from Google that we make publicly available. The results of our study show that our selection and prioritization techniques can each lead to cost-effectiveness improvements in the continuous integration process.
View details
Introducing Continuous Systematic Testing of Evolving Software
Mary Jean Harrold
Darko Marinov
Stephen Oney
Mauro Pezzè
Adam Porter
Per Runeson
Shin Yoo
Dagstuhl Seminar Proceedings, Schloss Dagstuhl-Leibniz-Zentrum für Informatik (2010)
Preview abstract
In today's evolutionary development of software, continuous testing is needed to ensure that the software is still functioning after changes. Test automation helps partly managing the large number of executions needed, but there is also a limit for how much automated tests may be executed. Then systematic approaches for test selection are needed also for automated tests. This manuscript defines this situation and outlines a general method and tool framework for its solution. Experiences from different companies are collected to illustrate how it may be set into practice.
View details
Experiences Using Static Analysis to Find Bugs
Nathaniel Ayewah
David Hovemeyer
J. David Morgenthaler
William Pugh
IEEE Software, 25 (2008), pp. 22-29
Preview abstract
Static analysis examines code in the absence of input data and without running the code, and can detect potential security violations (e.g., SQL injection), runtime errors (e.g., dereferencing a null pointer) and logical
inconsistencies (e.g., a conditional test that cannot possibly be true). While there is a rich body of literature on algorithms and analytical frameworks used by such tools, reports describing experiences with such tools in industry are much harder to come by.
In this paper, we describe FindBugs, an open source static analysis tool for Java, and experience using it in production settings. FindBugs does not push the envelope in terms of the sophistication of its analysis techniques. Rather, it is designed to evaluate what kinds of defects can be effectively detected with relatively simple techniques and to help us understand how such tools can be incorporated into the software development process. FindBugs has become very popular, downloaded more than 500,000 times and used by many major companies and software projects. We report on experience running FindBugs against Sun’s JDK implementation, using Findbugs at Google where it has been used for more than a year and a half and incorporated into their standard development process, and preliminary results from a survey of FindBugs users.
View details
Predicting Accurate and Actionable Static Analysis Warnings: An Experimental Approach
Joseph Ruthruff
J. David Morgenthaler
Sebastian Elbaum
Gregg Rothermel
Proceedings of the International Conference on Software Engineering, ACM (2008), pp. 341-350
Preview abstract
Static analysis tools report software defects that may or may not be detected by other verification methods. Two challenges complicating the adoption of these tools are spurious false positive warnings and legitimate warnings that are not acted on. This paper reports automated support to help address these challenges using logistic regression models that predict the foregoing types of warnings from signals in the warnings and implicated code. Because examining many potential signaling factors in large software development settings can be expensive, we use a screening methodology to quickly discard factors with low predictive power and cost-effectively build predictive models. Our empirical evaluation indicates that these models can achieve high accuracy in predicting accurate and actionable static analysis warnings, and suggests that the models are competitive with alternative models built without screening.
View details
Evaluating Static Analysis Defect Warnings on Production Software
Nathaniel Ayewah
William Pugh
J. David Morgenthaler
YuQian Zhou
Proceedings of the 7th ACM SIGPLAN-SIGSOFT workshop on Program analysis for software tools and engineering, ACM Press, New York, NY, USA (2007), pp. 1-8
Preview abstract
Classification of static analysis warnings into false positive, trivial or serious bugs: Experience on Java JDK and Google codebase
View details
Using FindBugs on Production Software
Preview
Nathaniel Ayewah
J. David Morgenthaler
William Pugh
YuQian Zhou
Proc. OOPSLA'07, ACM, Montréal (2007)
