RelEng & RelOps Weekly highlights - March 4, 2016

It was a busy week with many releases in flight, as well as preparation for running beta 1 with release promotion next week.  We also are in the process of adding more capacity to certain test platform pools to lower wait times given all the new e10s tests that have been enabled.

Improve Release Pipeline:

• Nick ran a staging release for 46.0b1 to check for issues before the merge, preventing some bustage for Fennec and ensuring we can fall back to the old system if any unexpected issues show up with release promotion
• Varun improved Balrog’s detection of certain types of bad data.
• Ben finished most of the work involved with preparing Balrog to move to CloudOps infrastructure, including automatically building Docker images.
• We’re hoping to do our first beta with the new release build promotion pipeline next week for 47.0b1. Stay tuned!

Everyone gets a release promotion!  Source: http://i.imgur.com/WMmqSDI.jpg

Improve CI Pipeline:

• Dustin deployed a new version of the TaskCluster tools/login system with much improved UI for handling signing in and out and editing clients and roles.  He also simplified the existing roles, with the result that the set of roles now fits on one screen, and is entirely composed of human-readable names.  All of this works toward two important goals: building a sign-in system that is useful and usable by all mozillians; and configuring the access-control system to give everyone their appropriate permissions and no more.

Release:

The releases calendar is getting busier as we get closer to the end of the cycle. Many releases were shipped or are still in-flight:

• Firefox 45.0b10
• Fennec 45.0b11
• Fennec 45.0 (in-progress)
• Firefox 45.0 (in-progress) - we shipped the RC to the beta channel
• Firefox 45.0esr (in-progress)
• Firefox 38.7.0esr (in-progress)

As always, you can find more specific release details in our post-mortem minutes:
https://wiki.mozilla.org/Releases:Release_Post_Mortem:2016-03-02 
https://wiki.mozilla.org/Releases:Release_Post_Mortem:2016-03-09

Operational:

• Vlad, Alin and Amy reallocated 28 Windows test machines to the w7 pool to help with backlog and e10s testing.
• Jake deployed new OpenSSL packages to protect our infrastructure from Drown Attack and various other recent OpenSSL vulnerabilities

Until next time!

Read more...

RelEng & RelOps Weekly highlights - February 26, 2016

It was a busy week for release engineering as several team members travelled to the Vancouver office to sprint on the release promotion project. The goal of the release promotion project is to promote continuous integration builds to release channels, allowing us to ship releases much more quickly.

Improve Release Pipeline:

• Chris, Jordan, Callek (remotely), Kim, Mihai and Rail had a sprint on Release Promotion. We made so much progress on this project that we decided to use the new process for Firefox 46.0b1. https://bugzil.la/1118794 So many green jobs!

• A new contributor, Kumar, fixed a longstanding annoyance with how Balrog shows changes to Releases. 
• Ben and Nick came up with a new way to model GMP and System Addons in Balrog which will allow for more fine grained permissions and simpler rules. 
• Mihai added a functionality in tctalker to walk the graph and cancel all pending/running tasks. He also added release sanity check logic for en-US binaries within promotion. 
• Kim added support for virus scanning of release promotion artifacts, a new docker image for this task, a task to move source bundles and checksum artifacts, and sha256 checksums for the docker images we use in release promotion. 

Improve CI Pipeline:

• Rok added a button “Purge workers cache” in Taskcluster 
• A new contributor, Varun, is helping to improve our l10n repack process. 
• Aki added python3 compatibility to taskcluster-client.py https://github.com/taskcluster/taskcluster-client.py/pull/44 

Release:

• Ben, Mihai, Nick, Rail and Callek Shipped Firefox 45.0b6, Firefox 45.0b7, Firefox 45.0b9, Fennec 45.0b6, Thunderbird 45.0b2 

Operational:

• Alin landed changes to run mochitest-push-e10s tests on Windows 7 https://bugzil.la/1248729. This is another step toward completing the enabling of e10s tests. 

Read more...

This week in Mozilla Releng - July 4, 2014

This is a special double issue of this week in releng. I was so busy in the last week that I didn't get a chance to post this last week.  Despite the fireworks for Canada Day and Independence Day,  Mozilla release engineering managed to close some bugs. 

Major highlights:

• Armen, although he works on Ateam now, made blobber uploads discoverable and blogged about it.  Blobber is a server and client side set of tools that allow Releng's test infrastructure to upload files without requiring to deploy ssh keys on them. 
• Callek and Coop, who served on buildduty during the past two weeks worked to address capacity issues with our test and build infrastructure.  We hit a record of 88,000 jobs yesterday which led to high pending counts.
• Kim is trying to address the backlog of Android 2.3 test jobs  by moving more test jobs to AWS from our inhouse hardware now that Geoff on the Ateam has found a suitable image.
• Rail switched jacuzzi EBS from magnetic to SSD.  Jacuzzis are similar pools of build machines  and switching their EBS storage from magnetic to SSD in AWS will improve build times.

 Completed work (resolution is 'FIXED'):

• Buildduty
• Stale slaverebooter lockfile
• please upload a new tp5n.zip to get a network access fix
• Please upload blessings to internal PyPI
• Please deploy newer version of blobuploader
• upload a new mobile_tp4.zip pageset to the 3 headed remote talos server
• Problem with cannot create debug link section
• B2G hamachi_eng_nightly failing with "Giving up on gecko git revision for 31de1a84b27f."
• increase disk size for AWS buildbot masters
• Please deploy newer version of blobuploader (1.2.1)
• b-2008-sm-0009 is unreachable
• Something wrong with b-2008-ix-002x slaves
• upload a talos pages to the 3 headed remote talos server to revision f9136c4bc616
• B2G bumper bot hasn't made any commits for 6+ hours
• Windows 8 test jobs pending on Try for coming up to 3 hours
• General Automation
• Need automatic hsts preload list updates on any branches based on Gecko 18 and later
• Make blobber uploads discoverable
• Provision enough in-house master capacity
• Disable mochitest-metro on Cedar
• Allow in-tree json for Gaia-Try to specify which Gecko build to use.
• Please do not use http://puppetagain.pub.build.mozilla.org for high-reliability Python installs
• [tracker] run Android 2.3 test jobs on EC2
• Perma-fail on Linux64 slaves - test_hosted.xul,test_packaged.xul,test_packaged_launch.xul | Error during test: [Exception... "Component returned failure code: 0x80520012 (NS_ERROR_FILE_NOT_FOUND) [nsIFileOutputStream.init]" nsresult: "0x80520012
• generate "flatfish" builds for B2G
• Add the build step or else process name to buildbot's generic command timed out failure strings
• [Dolphin] Create Dolphin builds for 1.4
• [Dolphin] Need a way to build Dolphin builds for 1.4
• Require more free space for android builds
• switch tst* EBS from magnetic to SSD
• Tooltool doesn't work on (at least) windows c-c
• Increase Android 2.3 mochitest chunks, for aws
• Monitor aws_stop_idle.py hungs
• Hazard failures with rsync: connection unexpectedly closed (0 bytes received so far) [sender]
• Add more tst-linux64 slaves
• Update Gu job to build Gaia with DESKTOP_SHIMS=0
• Please clobber bld-linux64-spot-015
• Configure elm for linux builds
• Use tooltool for l10n builds
• Please deploy newer version of blobber
• Race condition between builders that push updates to in-tree files
• Unschedule Goop everywhere but cedar
• Add hazards builds to ash branch
• Start doing mulet builds on Fig
• Terminate buildbot-master65 instance
• Decommission the ionmonkey tree
• switch jacuzzi EBS from magnetic to SSD
• create in-tree CA pinning preload list
• Loan Requests
• loan linux64 ec2 slave to jmaher
• Requesting a loaner machine b2g_ubuntu64_vm to diagnose bug 1017490
• loan gbrown a c3.xlarge to test reftests etc again in AWS
• Request Loan Machine for tst-linux64-spot
• Other
• Allow developers to access Android emulator AVDs
• Provide a Mozilla API key for desktop builds
• sign Thunderbird hot-fix testing addon
• No Firefox win32 l10n builds for mozilla-central since June 25th
• Platform Support
• Upload new version of xulrunner to tooltool
• Redirect runner logging output to /var/log/runner.log
• rc3.d/ symlinks should refresh on initscript updates for runner and buildbot
• Segmentation fault on gaia try server with pull request from bug 1017490
• Update version of pip installed on automation machines from 0.8.2 to 1.5.5
• address high pending count in in-house Linux64 test pool
• Release Automation
• Please disable the servo bors cronjob
• Need an update watershed in 29.0bN for beta channel
• Tree closure hooks for esr31
• Releases
• Sign the Firefox update hotfix (v20140527.01)
• Repos and Hooks
• Allow pushing Instantbird-only changes on a closed c-c- tree
• Create a script to import one Mercurial repository into a subdirectory of another, with linear history
• Add a repository hook to enforce new review requirements for changing .webidl files
• Tools
• update relman scripts based on notes from first usage
• Verify that trychooser's tryload is looking at the right names for pending jobs and adding them up into the correct buckets
• TryChooser conflicts with HTTPS Everywhere
• port b2g branching script to mozharness, with revision locking
• Update SlaveAPI sphynx docs to reflect recent changes
• Update tools/update-packaging/make_incremental_update.sh to allow passing Product Version and Channel ID
• Using slaveapi to reboot a machine on the slave health page results in "list index out of range" under 'output'
• loan jgriffin an AWS linux64 test box, and then bump up the instance cpu/ram til tests pass

In progress work (unresolved and not assigned to nobody):

• Balrog: Backend
• support comparison operators for matching version & buildID in rules
• Buildduty
• loan glandium and mshal a Linux ec2 instance
• Growing pending queues for tegras and time between jobs per tegra > 6 hours
• [tracking] Eliminate buildduty
• Investigate Windows 8 machines that are still out of action
• Reconfigs should be automatic, and scheduled via a cron job
• mountain-lion talos systems using the wrong basedir
• General Automation
• [Tracking] Cleanup And organize mobile (sut) code.
• end_to_end reconfig script should run its python unbuffered
• Provide B2G Emulator builds for Darwin x86
• Figure out the correct path setup and mozconfigs for automation-driven MSVC2013 builds
• Move emulator gaia-ui-tests on cedar from AWS to IX slaves
• [Meta] Some "Android 4.0 debug" tests fail
• Create mozilla-esr31 branch
• Normalize builder names
• [Meta] Fix + unhide broken testsuites or else turn them off to save capacity
• gaia-try: do a merge locally instead of using the pull request reference
• manage_instance_storage.py should consider root volume size
• fx desktop builds in mozharness
• Move Firefox Desktop repacks to use mozharness
• Add signing for firefox and org.mozilla.updater binaries on OSX
• ensure the timezone and time are set properly on tegras (and other devices)
• Partial update generation service
• Run desktop mochitests-browser-chrome on Ubuntu
• Get Mulet builds running on OSX on fig
• port buildapi to relengapi
• Do nightly builds with profiling disabled
• Tracking bug for 21-jul-2014 migration work
• dolphin full image size is incorrect in pvt build
• spidermonkey_build.py looks for gcc in two different places
• Attach a cost figure(in USD) to try reports
• Please add non-unified builds to mozilla-central
• Allow to install additional packages in mock
• Stop rebooting after every job
• Schedule Android 4.0 Debug reftests, crashtests, and robocop tests on all trunk trees and let them ride the trains
• revamp b2g upload configs
• Hazard builds uploading to wrong FTP dir
• Schedule Mnw on cedar on emulator-jb and emulator-kk
• We should stop using setup.sh in tooltool
• Touch CLOBBER on branches being uplifted
• reduce EBS writes by tweaking writeout
• Improve slaveapi logging
• Loan Requests
• loan windows 7 slave to jmaher
• loan linux32 (hardware) slave to jmaher
• Loan t-w732-ix-003, t-w732-ix-004 to Q
• Loan :sfink a linux64 b-linux64-hp-0024
• Need a bld-lion-r5 to test build times with SSD
• Request to borrow an r4 mini
• Other
• Identify next batch of locales which no longer need nightlies on m-c
• Decide and document how many B2G filesystem diffs we keep for updates
• Create a long term archive of opt+debug builds on M-I for Regression Hunting
• mozharness device mixin has various problems
• Locate and identify tegra-122, tegra-123, and tegra-161
• mozharness device_talosrunner has some minor issues
• Set up cron jobs to copy tinderbox-builds content to S3
• [Tracking bug] support git in production on parity with hg
• get a consistent story on graph server data per branch
• [tracking] infrastructure improvements
• Platform Support
• Stop update_shared_repos runner task from pulling Try
• Deploy NSIS 3.0a2 or MozillaBuild 1.9 (includes NSIS 3.0a2) to buildslaves
• [tracker] Stop testing on tegras
• evaluate mac cloud options
• b-2008-sm machines being incorrectly flagged as unreachable
• signing win64 builds is busted
• implement c3.xlarge slave class for Linux64 test spot instances
• Investigate alternative SSH daemon for Windows
• verify script called by mozharness for android devices doesn't reboot via mozpool
• Deploy hg.m.o/build/buildbot production-0.8 to non-windows buildslaves to pick up bug 961075
• disable tegras on 32 to ride the trains
• slave pre-flight tasks
• Tegra Cleanup should delete some additional files on the sdcard
• Cleanup org.mozilla.f3nn3c.PasswordsProvider in verify.py
• Release Automation
• release automation should update ship it at certain points
• Update channels for single locale Beta and Release builds of Firefox for Android 30 (and beyond)
• use a property other than speed to differentiate between slaves in slavealloc
• Add new table(s) to shipit database
• send e-mail to release-drivers when a build is requested through ship it
• Add a standalone process that listens to pulse for release related buildbot messages
• Add REST API entry point to shipit that allows shipit-agent to enter release data into shipit database
• Stop doing a recursive chmod on directories every time we upload to a candidates dir
• Releases
• Sign and dev deploy the Firefox update hotfix (v20140527.01) (v2)
• Releases: Custom Builds
• Remove Rambler config to inactive and remove builds from FTP
• Repos and Hooks
• Error when pushing to user repo: Sending messages to pulse.mozilla.org, Hook returned an exception: 'NoneType' object has no attribute 'filerev'
• Change the webidl hook to allow commits authored by a DOM reviewer
• research if we can remove duplicate single head hg.m.o hook
• Tools
• Fix tools/update-packaging/common.sh for consistent behaviour across platforms
• Intermittent time travel in legacy vcs
• AWS Sanity Check lies about how long an instance was shut down for...
• Figure out tools versioning for partial generation
• tooltool upload should set file permissions to world readable
• tegra/panda health checks (verify.py) should not swallow exceptions
• Set up auto publish of doc index per Boston workweek agreement
• aws_create_instance should not be allowed to run forever
• mozharness mozilla-release merge day script
• Create a new landing page for releng
• mozharness mozilla-beta + mozilla-aurora merge day script
• slaveapi's shutdown_buildslave action doesn't cope well with a machine that isn't connected to buildbot
• self-serve agents' error handling needs more care
• Make mozharness use structured logging for marionette tests
• implement 'aws_create_instance' action in slaveapi
• [tracking] Implement a comprehensive slave health tool
• slaveapi should make use of dev staging inventory and mozdns/api/v1_dns
• Transplant tool (Hg to Hg) for sheriffs

Read more...

This week in Mozilla Releng - June 20, 2014

Ben is away for the next few Fridays, so I'll be covering this blog post for the next couple of weeks.

Major highlights:

• Aki fixed an issue which was causing the sheriffs grief because the specific version of an external repository was not specified in b2g manifests.
• Ben completed work on to support update balrog blob schema to support multiple partials.  This is important because sometimes we need to have multiple partials in a beta or release.
• John is back for his second internship.  He completed his work providing an easy setup of some of our releng applications in a local test environment using Docker and blogged about it here.  This should reduce the barriers for new people to contribute.
• Jordan, wearing the buildduty shield this week,  fixed backlogs on our AWS spot instances that were causing high pending counts. 

Completed work (resolution is 'FIXED'):

• Balrog: Backend
• Add tests for architecture detection in Balrog
• update balrog blob schema to support multiple partials
• Buildduty
• Clean up old release dirs on slaves
• Backlog of linux compile jobs (Amazon AWS instances not being launched)
• Intermittent "Unable to remove C:\slave\test\build!"
• Please upload a new tp5n.zip to get a manifest change (to disable the alipay.com test)
• Something wrong with b-2008-ix-002x slaves
• Windows inbound builds pending for 10 hours
• B2g-inbound tree closed due to pending B2G ICS Emulator Opt builds
• update cache_flusher in http://pypi.pub.build.mozilla.org/pub
• General Automation
• mozilla-central should contain a pointer to the revision of all external B2G repos, not just gaia
• use gzip compression for hg bundles
• pending l10n jobs should spin up aws slaves
• Self-serve should be able to request arbitrary builds on a push (not just retriggers or complete sets of dep/PGO/Nightly builds)
• nagios checks for stub installer bouncer entries
• move jacuzzi repo out of bhearsum's account
• kill b2g26_v1_2 on june 9 merge day
• keep buildbot master twistd logs longer
• Add sendchange functionality to the mozharness library
• Missing hsb and dsb single-locale Aurora Fennec builds
• Proactively check test machines are ok
• create bouncer entries for nightly and aurora stub installers
• Perma-fail on Linux64 slaves - test_hosted.xul,test_packaged.xul,test_packaged_launch.xul | Error during test: [Exception... "Component returned failure code: 0x80520012 (NS_ERROR_FILE_NOT_FOUND) [nsIFileOutputStream.init]" nsresult: "0x80520012
• Remove useless --enable-privilege option from remotereftest.py
• turn off aurora localizer b2g desktop builds
• Generalize the sendchanges logic on mozharness
• Remove config for ubuntu64_hw-b2g-dt platform
• Disable nightly builds and updates for Fig project branch
• Don't force *.chk in MARs
• [b2g] Request for hamachi/buri releng builds from larch twig
• Incorporate AWS slaves into nextSlave decisions
• TP5 Responsiveness is not being run on mozilla-inbound
• manage buildbot master buildbot.tac files
• Version bump from Gecko 32 to 33 has broken Gaia-try
• spot instances should make sure to have up-to-date git repositories
• try-bld-spot in us-west-2 are failing to init mock environment
• switch routing to hg.m.o to use public internet
• Make EC2 instances less susceptible to "abort: No space left on device"
• Intermittent command timed out: 3600 seconds without output, attempting to kill fetching b2g bits from gitmo
• Need automated B2G device builds on Mac (trunk and b2g18)
• consider using StartupItemContext to run mac signing servers
• Gaia related tests should run |git log -n 1| in it's log output
• [tracking] automation support for B2G v1.1.0hd
• Remove spot instances from inventory
• aws masters are very slow to respond to simple http queries
• Build some jacuzzis
• make sure build slaves can continue to talk to aus3-staging and aus4-dev
• Start doing mulet builds on Fig
• Set MINIDUMP_SAVE_PATH environment var to a valid path for Talos runs
• Don't require puppet or DNS to launch new instances
• Use different logic to adjust amount of instances to be started for jacuzzied slaves
• Device Builds Tests/Builds for mozilla-inbound
• Monitoring for AWS slaves
• [tracking bug] continuous integration for release engineering code
• Loan Requests
• loan linux64 hardware slave to jmaher
• loan linux64 hardware slave to :jib
• Other
• Provide a Mozilla API key for desktop builds
• Linux builds on cedar failing to find build dependencies
• slavealloc: use SSL
• Intermittent pseudo-first-line.svg | image comparison (==), max difference: 32, number of differing pixels: 1
• test-masters should test that universal masters still work
• Run rev4 10.6 slaves concurrently with rev3 10.6 and have graph server results for both
• PGO talos is submitting to the Firefox-Non-PGO tree
• [Tracking bug] decommission tinderbox server
• Deprecate tinderbox-builds/old directories for desktop & mobile
• [tracking] move services from cruncher to production
• Platform Support
• Move p3 panda rack
• Add cloud-init to our base AMIs
• Add HD Panda chassis to android production pandas
• Release Automation
• release l10n repacks failed due to failed "rm"
• Move hosted partner bundles into place automatically
• Create a Vagrantfile for Ship It
• update and final verify are incompatible with balrog
• intermittent make_incremental_update.sh bustage
• Releases
• Show Australis whatsnew tour page upon update for all locales on Firefox 29 release builds
• tracking bug for build and release of Firefox and Fennec 30.0
• Link to latest version (30.0) of mobile Firefox on FTP is outdated
• Releases: Custom Builds
• Yandex partner builds for Turkey
• Partner repack changes for web.de, gmx, mail.com and 1&1 (Firefox 30)
• Repos and Hooks
• New git repositories to mirror for Flame
• Tools
• Buildduty Report sorting should not error out (JS) if a section has no bugs
• vcs-sync needs to be able to publish git-hg mappings to mapper
• End-to-end reconfig should also update tools version on foopies
• build-tools: Can't push /opt/vcs2vcs/vcs_sync/build/conversion/build-tools to build-tools-github!
• Setup docker apps for buildbot, buildapi and redis
• add more slave names to non jacuzzi builders
• end_to_end_reconfig.sh should store logs from manage_foopies.py
• Blobber upload files not served with correct content type
• Move buildduty report script into slave_health repo
• Strikethrough or don't display closed dependencies in buildduty report
• Deploy relengapi 0.2.1 and mapper 0.2.1 into production https://api.pub.build.mozilla.org/
• db-based mapper on web cluster
• Deploy updated Buildapi to self-serve agents
• "'NoneType' object has no attribute 'id_'" when slaveapi fails at rebooting unreachable slaves
• slave_health stopped updating after 13:40 Friday
• update_maintenance_wiki.sh is truncating text content
• buildfarm/maintenance/manage_foopies.py not executable

In progress work (unresolved and not assigned to nobody):

• Balrog: Backend
• Match on a list of locales, rather than exact only
• Clean up data handling in AUS3
• Buildduty
• Reconfigs should be automatic, and scheduled via a cron job
• [tracking] Eliminate buildduty
• General Automation
• Make blobber uploads discoverable
• Figure out the correct path setup and mozconfigs for automation-driven MSVC2013 builds
• switch tst* EBS from magnetic to SSD
• Add signing for firefox and org.mozilla.updater binaries on OSX
• Schedule all Android 2.3 armv6 tests, except mochitest-gl, on all trunk trees and make them ride the trains
• manage_instance_storage.py should consider root volume size
• Move Firefox Desktop repacks to use mozharness
• Move emulator gaia-ui-tests on cedar from AWS to IX slaves
• Partial update generation service
• Clean up / redesign of configuration in buildbot-configs/mozilla
• Make b2g_emulator_unittest.py easier to run outside of automation
• Race condition between builders that push updates to in-tree files
• port buildapi to relengapi
• Investigate using distcc for builds (on as many platforms as possible) to reduce build wall clock time
• Increase Android 2.3 mochitest chunks, for aws
• Switch gaia-try to my mozharness again
• Monitor aws_stop_idle.py hungs
• all android single locale repacks broken
• Update Gu job to build Gaia with DESKTOP_SHIMS=0
• Please add non-unified builds to mozilla-central
• Use tooltool for l10n builds
• Don't generate nightly builds on a tree if no new changes have landed since the previous nightly
• revamp b2g upload configs
• FlatFish: Integrate boot.img and recovery.img into the build system
• Schedule Mnw on cedar on emulator-jb and emulator-kk
• We should stop using setup.sh in tooltool
• reduce EBS writes by tweaking writeout
• Loan Requests
• Slave loan request for a talos-r4-snow machine
• Requesting a loaner machine b2g_ubuntu64_vm to diagnose bug 1017490
• Slave loan request for a bld-linux64-ec2 vm
• Other
• [tracking] infrastructure improvements
• Locate and identify tegra-122, tegra-123, and tegra-161
• Platform Support
• Android nightly builds should download the previous.apk *before* uploading a new gecko-unsigned-unaligned.apk to the same URL
• Upload new version of xulrunner to tooltool
• slave pre-flight tasks
• Tegra Cleanup should delete some additional files on the sdcard
• Update version of pip installed on automation machines from 0.8.2 to 1.5.5
• scl1 Move Train D releng config Work
• address high pending count in in-house Linux64 test pool
• Cleanup org.mozilla.f3nn3c.PasswordsProvider in verify.py
• Release Automation
• cache MAR + installer downloads in update verify
• 'ready to release' emails not being sent for release
• Repos and Hooks
• Allow pushing Instantbird-only changes on a closed c-c- tree
• Tools
• cut over gecko.git to the new vcs-sync system
• AWS Sanity Check lies about how long an instance was shut down for...
• port b2g branching script to mozharness, with revision locking
• more vcs-sync email tweaks
• tegra/panda health checks (verify.py) should not swallow exceptions
• tracker to retire legacy vcs2vcs
• slaveapi disable - comment in bug while disabling is in progress and add a reason dep bug option
• cut over l10n repos to the new vcs-sync system
• loan jgriffin an AWS linux64 test box, and then bump up the instance cpu/ram til tests pass
• Possible bug in end_to_end_reconfig.sh when using -p option?
• Make mozharness use structured logging for marionette tests
• permanent location for vcs-sync mapfiles, status json, logs
• Fix tools/update-packaging/common.sh for consistent behaviour across platforms

Read more...