Releng 2015 program now available
>> Tuesday, April 28, 2015
Releng 2015 will take place in concert with ICSE in Florence, Italy on May 19, 2015. The program is now available. Register here!
![]() |
pinomoscato, Creative Commons by-nc-sa 2.0 | by ©
Open Source Release Engineering
Releng 2015 will take place in concert with ICSE in Florence, Italy on May 19, 2015. The program is now available. Register here!
![]() |
via romana in firenze by ©pinomoscato, Creative Commons by-nc-sa 2.0 |
Running a large continuous integration farm forces you to deal with many dynamic inputs coupled with capacity constraints. The number of pushes increase. People add more tests. We build and test on a new platform. If the number of machines available remains static, the computing time associated with a single push will increase. You can scale this for platforms that you build and test in the cloud (for us - Linux and Android on emulators), but this costs more money. Adding hardware for other platforms such as Mac and Windows in data centres is also costly and time consuming.
Do we really need to run every test on every commit? If not, which tests should be run? How often do they need to be run in order to catch regressions in a timely manner (i.e. able to bisect where the regression occurred)
Several months ago, jmaher and vaibhav1994, wrote code to analyze the test data and determine the minimum number of tests required to run to identify regressions. They named their software SETA (search for extraneous test automation). They used historical data to determine the minimum set of tests that needed to be run to catch historical regressions. Previously, we coalesced tests on a number of platforms to mitigate too many jobs being queued for too few machines. However, this was not the best way to proceed because it reduced the number of times we ran all tests, not just less useful ones. SETA allows us to run a subset of tests on every commit that historically have caught regressions. We still run all the test suites, but at a specified interval.
![]() |
SETI – The Search for Extraterrestrial Intelligence by ©encouragement, Creative Commons by-nc-sa 2.0 |
![]() |
http://hg.mozilla.org/build/buildbot-configs/file/2d9e77a87dfa/mozilla-tests/config_seta.py#l692 |
![]() |
http://hg.mozilla.org/build/buildbotcustom/file/728dc76b5ad0/misc.py#l2727 |
Here's March 2015's monthly analysis of the pushes to our Mozilla development trees. You can load the data as an HTML page or as a json file.
Trends
The number of pushes increased from those recorded in the previous month with a total of 10943.
Highlights
© Blogger template Simple n' Sweet by Ourblogtemplates.com 2009
Back to TOP