tag:blogger.com,1999:blog-250232802024-03-05T08:11:08.127-08:00Releng of the NerdsOpen Source Release EngineeringKim Moirhttp://www.blogger.com/profile/14700841495895160750noreply@blogger.comBlogger164125tag:blogger.com,1999:blog-25023280.post-55776532582825885642017-05-17T14:12:00.000-07:002017-05-17T14:12:22.831-07:00New blog locationI moved my blog to WordPress.<br />
<br />
New location is here <a href="https://kimmoir.blog/">https://kimmoir.blog/</a>Kim Moirhttp://www.blogger.com/profile/14700841495895160750noreply@blogger.com0tag:blogger.com,1999:blog-25023280.post-80347245337679117042016-09-30T06:10:00.000-07:002016-09-30T06:10:42.149-07:00Beyond the Code 2016 recap<div class="separator" style="clear: both; text-align: left;">
I've had the opportunity to attend the Beyond the Code conference for the past two years. This year, the venue moved to a location in Toronto, the last two events had been held in Ottawa. The conference is organized by Shopify who again managed to have a really <a href="http://beyondthecode.io/pages/speakers">great speaker line</a> up this year on a <a href="http://beyondthecode.io/pages/program">variety of interesting topics</a>. It was a two track conference so I'll summarize some of the talks I attended. <a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgC_Mb93Xerw-UuJ6SQP9rqKbB5WzdXk1F4N5BVvVnCPVT1q995_lv-jh912F1T0PMCd57VgDj6wNcx9gLu38YJy_3SkgUmwD6N7pY5-1AmEOsQ2YTmVgV9gBcADyRB3K8SYezP/s1600/IMG_1535.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgC_Mb93Xerw-UuJ6SQP9rqKbB5WzdXk1F4N5BVvVnCPVT1q995_lv-jh912F1T0PMCd57VgDj6wNcx9gLu38YJy_3SkgUmwD6N7pY5-1AmEOsQ2YTmVgV9gBcADyRB3K8SYezP/s640/IMG_1535.jpg" width="640" /></a></div>
<br />
The conference started off with Anna Lambert of Shopify welcoming everyone to the conference.<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgYCpZG5Y-9JpXK6aZk-3DAftVkRFn9npy64xq1-kTIztAMYJWP9Ggze-pOPMOJRxermUMrFMjri_99Yl_zL-JrDJH-PwhvAaO95JGHl_1pwD3Eb27UtgGIZ30qWCqRF1IsynnH/s1600/Screen+Shot+2016-09-28+at+9.10.15+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgYCpZG5Y-9JpXK6aZk-3DAftVkRFn9npy64xq1-kTIztAMYJWP9Ggze-pOPMOJRxermUMrFMjri_99Yl_zL-JrDJH-PwhvAaO95JGHl_1pwD3Eb27UtgGIZ30qWCqRF1IsynnH/s1600/Screen+Shot+2016-09-28+at+9.10.15+PM.png" /></a></div>
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhqQxcqRDVvpxynkjGw0i0FamG1f7imvcoq6SmfgRcqlUmdXaZMm7koQvPMbYXbF6azjQ_ieOmBlrA3EGNOlfHwoRwls9VwOl3MkhYU9fmOvai1xeAMvGlX8zTauS2gamVEwHTc/s1600/Screen+Shot+2016-09-28+at+9.15.48+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="336" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhqQxcqRDVvpxynkjGw0i0FamG1f7imvcoq6SmfgRcqlUmdXaZMm7koQvPMbYXbF6azjQ_ieOmBlrA3EGNOlfHwoRwls9VwOl3MkhYU9fmOvai1xeAMvGlX8zTauS2gamVEwHTc/s640/Screen+Shot+2016-09-28+at+9.15.48+PM.png" width="640" /></a></div>
<br />
The first speaker was Atlee Clark, Director of App and Developer relations at Shopify who discussed the wheel of diversity. <br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhfRTPkN9p0GG3rMFPJFfC505E7wI8L_zCCovx2jFwUjSAcvdGBB14fwZbMQAniRlAQbq_eiUzmdBtcCN4h1Y7opmpTkNRLp1vaepJLR4Ha6e1CaDBdS7bkqyYsU_RHtrkaxe8f/s1600/IMG_1484.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhfRTPkN9p0GG3rMFPJFfC505E7wI8L_zCCovx2jFwUjSAcvdGBB14fwZbMQAniRlAQbq_eiUzmdBtcCN4h1Y7opmpTkNRLp1vaepJLR4Ha6e1CaDBdS7bkqyYsU_RHtrkaxe8f/s640/IMG_1484.jpg" width="640" /></a></div>
<br />
The wheel of diversity is a way of mapping the characteristics that you're born with (age, gender, gender expression, race or ethnicity, national origin, mental/physical ability), along with those that you acquire through life (appearance, education, political belief, religion, income, language and communication skills, work experience, family, organizational role). When you look at your team, you can map how diverse it is by colour. (Of course, some of these characteristics are personal and might not be shared with others). You can see how diverse the team is by mapping different characteristics with different colours. If you map your team and it's mostly the same colour, then you probably will not bring different perspectives together when you work because you all have similar backgrounds and life experiences. This is especially important when developing products. <br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjhxvwZ1E-ZT-5LsgT_Zu_r8QqYrYxJfm7xF365_zdC1RHP2OyKlGZBhGv_1GPRFq8WkuiM2zpS9C65EUIArFsnlgn1eSlmNzMg2WDDu_1QwdtnKrLF7FXqQnpyLXf89up97hf7/s1600/IMG_1486.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjhxvwZ1E-ZT-5LsgT_Zu_r8QqYrYxJfm7xF365_zdC1RHP2OyKlGZBhGv_1GPRFq8WkuiM2zpS9C65EUIArFsnlgn1eSlmNzMg2WDDu_1QwdtnKrLF7FXqQnpyLXf89up97hf7/s640/IMG_1486.jpg" width="640" /></a></div>
<br />
<br />
This wheel also applies to hiring too. You want to have different perspectives when you're interviewing someone. Atlee mentioned when she was hiring for a new role, she mapped out the characteristics of the people who would be conducting the hiring interviews and found there was a lot of yellow.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgnc2LrB7NoCoYgwvkIxfOjarwmwON-mzwIOI1sxXc0TSy4YyrOzFiz1u73Jr7oTvgFDpCQ2zL9n2kww4HbjWWRQh7OnF1vnUOnvy4o7_jsDX4OUSmPK9fCfxRMubnVK1w7IlEY/s1600/IMG_1487.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgnc2LrB7NoCoYgwvkIxfOjarwmwON-mzwIOI1sxXc0TSy4YyrOzFiz1u73Jr7oTvgFDpCQ2zL9n2kww4HbjWWRQh7OnF1vnUOnvy4o7_jsDX4OUSmPK9fCfxRMubnVK1w7IlEY/s640/IMG_1487.jpg" width="640" /></a></div>
<br />
So she switched up the team that would be conducting the interviews to include people with more diverse perspectives.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhI1P8iA8m-qI2dhLPgj8BG9WzAoln6ohs6p_TCwic7APyPao0XA4QE3UjeE8kKQ1oL9w_ok9RzXb1Lh10SHJORDtez0IoCk6MXC2Swr9a-7Y8r1_I5zbLm6jMfSpfeztwdCUvk/s1600/IMG_1489.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhI1P8iA8m-qI2dhLPgj8BG9WzAoln6ohs6p_TCwic7APyPao0XA4QE3UjeE8kKQ1oL9w_ok9RzXb1Lh10SHJORDtez0IoCk6MXC2Swr9a-7Y8r1_I5zbLm6jMfSpfeztwdCUvk/s640/IMG_1489.jpg" width="640" /></a></div>
She finished by stating that this is just a tool, keep it simple, and practice makes it better. <br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
The next talk was by Erica Joy, who is a build and release engineer at Slack, as well as a diversity advocate. I have to admit, when I saw she was going to speak at Beyond the Code, I immediately pulled out my credit card and purchased a conference ticket. She is one of my tech heroes. Not only did she build the build and release pipeline at Slack from the ground up, she is an amazing writer and advocate for change in the tech industry. I highly recommend reading everything she has written on <a href="https://medium.com/@ericajoy">Medium</a>, her chapter in <a href="http://www.orbooks.com/catalog/lean-out/">Lean Out</a> and all her discussions on twitter. So fantastic.<br />
<br />
Her talk at the conference was "Building a Diverse Corporate Culture: Diversity and Inclusion in Tech". She talked about how literally thousands of companies say they value inclusion and diversity. However, few talk about what they are willing to give up to order to achieve it. Are you willing to give up your window seat with a great view? Something else so that others can be paid fairly? She mentioned that change is never free. People need both mentorship and sponsorship in in order to progress in their career.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhZVQ5QOcXh0XMwcPjBwYhmDX2n_5WZc-s1-X_dGMoZFpFSNt3ArmtkyuFjsf4G3xZ5OfPzN39IuLzWmJUsjtu2_hylqim-ZaRPVS0X1IQ7sMGFZWrmuZBdbVVLzUur_6jPsdVs/s1600/Screen+Shot+2016-09-29+at+7.47.34+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhZVQ5QOcXh0XMwcPjBwYhmDX2n_5WZc-s1-X_dGMoZFpFSNt3ArmtkyuFjsf4G3xZ5OfPzN39IuLzWmJUsjtu2_hylqim-ZaRPVS0X1IQ7sMGFZWrmuZBdbVVLzUur_6jPsdVs/s1600/Screen+Shot+2016-09-29+at+7.47.34+PM.png" /></a></div>
<br />
<br />
<br />
I really liked her discussion around hiring and referrals. She stated that when you're hire people you already know you're probably excluding equally or better qualified that you don't know. By default, women of colour are underpaid.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiO67JQw_QPkoqQROYpuzGT9xcSMnCNRHtSkd2SXraka33JuCGEkbLLXKfAa3O0LZ_o3nZTJKZ_gA4wNjO5ljY4zFH7nB-7YVTVarTvk7G5uCLkUv-EnYb2r7bWOIz37I6N326O/s1600/IMG_1507.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiO67JQw_QPkoqQROYpuzGT9xcSMnCNRHtSkd2SXraka33JuCGEkbLLXKfAa3O0LZ_o3nZTJKZ_gA4wNjO5ljY4zFH7nB-7YVTVarTvk7G5uCLkUv-EnYb2r7bWOIz37I6N326O/s640/IMG_1507.jpg" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Pay gap for white woman, African American women and Hispanic women compared to a white man in the United States.</td></tr>
</tbody></table>
<br />
Some companies have referral system to give larger referral bonuses to people who are underrepresented in tech, she gave the example of Intel which has this in place. This is a way to incentivize your referral system so you don't just hire all your white friends. <br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi0inbGmUbYFbSusGFvJRY4py8dlT83bn1wBDjSM38MDLzbiIR4X0n01osuEEM_jsTBeEn4dcYkdvpnxkhWFTQMiYLbd_v-daP6t2rwPAwz9eg_ylLT0yN15w2Bwjb54UMtPO5J/s1600/IMG_1513.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi0inbGmUbYFbSusGFvJRY4py8dlT83bn1wBDjSM38MDLzbiIR4X0n01osuEEM_jsTBeEn4dcYkdvpnxkhWFTQMiYLbd_v-daP6t2rwPAwz9eg_ylLT0yN15w2Bwjb54UMtPO5J/s640/IMG_1513.jpg" width="640" /></a></td></tr>
<tr align="left"><td class="tr-caption">The average white American has 91 white friends and one black friend so it's not very likely that they will refer non-white people. Not sure what the numbers are like in Canada but I'd guess that they are quite similar.</td></tr>
</tbody></table>
<br />
In addition, don't ask people to work for free, to speak at conferences
or do diversity and inclusion work. Her words were "We can't pay rent
with exposure". <br />
<br />
Spend time talking to diversity and inclusion experts. There are people that have spent their entire lives conducting research in this area and you can learn from their expertise. Meritocracy is a myth, we are just lucky to be in the right place in the right time. She mentioned that her colleague <a href="http://www.durettihirpa.com/">Duretti Hirpa</a> at Slack points out the need for accomplices, not allies. People that will actually speak up for others. So people feeling pain or facing a difficult work environment don't have to do all the work of fighting for change. <br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh5Q4_qOaF10NVOJf-D5JnLbn-QwzRFGSPJ4F6yPfnI8VxIQ6WofTZAmekzeTYcokVokjVs_aQn7bi09xAsrSJTHRoPkt58LtwaiDgyUuf2sWab9NmyEfNNAIyAiT8WzF-zGU-6/s1600/IMG_1515.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh5Q4_qOaF10NVOJf-D5JnLbn-QwzRFGSPJ4F6yPfnI8VxIQ6WofTZAmekzeTYcokVokjVs_aQn7bi09xAsrSJTHRoPkt58LtwaiDgyUuf2sWab9NmyEfNNAIyAiT8WzF-zGU-6/s640/IMG_1515.jpg" width="640" /></a></div>
<br />
<br />
<br />
In most companies, there aren't escalation paths for human issues either. If a person is making sexist or racist remarks, shouldn't that be a firing offense? <br />
<br />
If people were really working hard on diversity and inclusion, we would see more women and people of colour on boards and in leadership positions. But we don't.<br />
<br />
She closed with a quote from Beyonce:<br />
<br />
<div style="text-align: center;">
<i>"If everything was perfect, you would never learn and you would never grow"</i></div>
<div style="text-align: center;">
<br /></div>
<div style="text-align: center;">
💜💜💜
</div>
<div style="background-color: white; border: medium none; color: black; overflow: hidden; text-align: left; text-decoration: none;">
<br /></div>
The next talk I attended was by <a href="http://where.coraline.codes/">Coraline Ada Ehmke</a>, who is an application engineer at Github. Her talk was about the "Broken Promise of Open Source". Open source has the core principals of the free exchange of ideas, success through collaboration, shared ownership and meritocracy.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhjk5SFmfrIQdfwdha9osV91OIJ28SW-LoupyZMaM9MuWoNxjCT79fM151Y-UUvl6fUEk2T_qtfsOyFPrDY93SctivNnlE2HnWHKvJFDhcdQ0fEf0qlGJxR1TBzHQwcq6RRUgke/s1600/IMG_1525.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhjk5SFmfrIQdfwdha9osV91OIJ28SW-LoupyZMaM9MuWoNxjCT79fM151Y-UUvl6fUEk2T_qtfsOyFPrDY93SctivNnlE2HnWHKvJFDhcdQ0fEf0qlGJxR1TBzHQwcq6RRUgke/s640/IMG_1525.jpg" width="640" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: left;">
However, meritocracy is a myth. Currently, only <a href="https://www.toptal.com/open-source/is-open-source-open-to-women" style="outline-offset: 0px; outline: 1px dotted;">6% of Github users are women.</a> The environment can be toxic, which drives a lot of people away. She mentioned that we don't have numbers for diversity in open source other than women, but Github plans to do a survey soon to try to acquire more data.</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgIdOiv0mHiJOgK4KVSMzNtVIiZkMARxRDG1mFVd4HYJmzOjM1O2S5qA10xR3aQQ2kmxjS2b8RdS7lZWThFdQ9iGbWWJ2MSVPt-AMJ-WHlJupwOf6j2PWgJrFbWhV4WAU__dzEL/s1600/IMG_1526.JPG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgIdOiv0mHiJOgK4KVSMzNtVIiZkMARxRDG1mFVd4HYJmzOjM1O2S5qA10xR3aQQ2kmxjS2b8RdS7lZWThFdQ9iGbWWJ2MSVPt-AMJ-WHlJupwOf6j2PWgJrFbWhV4WAU__dzEL/s640/IMG_1526.JPG" width="640" /></a></div>
<br />
Gabriel Fayant from <a href="https://a7g.dotrust.org/">Assembly of Seven Generation</a>'s talk was entitled "Walking in Both Worlds, traditional ways of being and the world of technology". I found this quite interesting, she talked about traditional ceremonies and how they promote the idea of living in the moment, and thus looking at your phone during a drum ceremony isn't living the full experience. A question from the audience from someone who worked in the engineering faculty at the University of Toronto was how we can work with indigenous communities to share our knowledge of the technology and make youth both producers of tech, not just consumers. <br />
<br />
<div style="left: -99999px; position: absolute;">
If everything was perfect, you would never learn and you would never grow.<br />
Read more at: http://www.brainyquote.com/quotes/quotes/b/beyoncekno596349.html</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEizta-JoH-EhDJVTT1oJFCpUbZQmqNkJWxVM2Di1WGVqfi-KGVFLhDgzR8RytkmQfNGqoPSmA3X2bc2LNjdihOgv3gQ2Aj_61JnDbgsWERFKJef6H4oqHcYRvt12RnCjSFm1bu6/s1600/IMG_1527.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEizta-JoH-EhDJVTT1oJFCpUbZQmqNkJWxVM2Di1WGVqfi-KGVFLhDgzR8RytkmQfNGqoPSmA3X2bc2LNjdihOgv3gQ2Aj_61JnDbgsWERFKJef6H4oqHcYRvt12RnCjSFm1bu6/s640/IMG_1527.jpg" width="640" /></a></div>
<br />
<div style="left: -99999px; position: absolute;">
f everything was perfect, you would never learn and you would never grow.<br />
Read more at: http://www.brainyquote.com/quotes/quotes/b/beyoncekno596349.html</div>
The next talk was by <a href="http://www.sandimetz.com/">Sandi Metz</a>, entitled "Madame Santi tells your future". This was a totally fascinating look at the history of printing text from scrolls all the way to computers.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgElcTgAZ8TxGLzFslldqMDLDlKhDKZvvuUtGyQRDEsXvkAdOsf44QlxjAZzpp9VgBLXoOHKSovEZTcrfE_TYSGN9QWpwANO4REWRXKc_QvfFA7rVve3vBCvNi14JFsHum9e9mX/s1600/IMG_1529.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgElcTgAZ8TxGLzFslldqMDLDlKhDKZvvuUtGyQRDEsXvkAdOsf44QlxjAZzpp9VgBLXoOHKSovEZTcrfE_TYSGN9QWpwANO4REWRXKc_QvfFA7rVve3vBCvNi14JFsHum9e9mX/s640/IMG_1529.jpg" width="640" /></a></div>
She gave the same talk at another<a href="https://www.youtube.com/watch?v=JOM5_V5jLAs"> conference earlier so you watch it here</a>. It described the progression of printing technology from 7000 years ago until today. Each new technology disrupted the previous one, and it was difficult for those who worked on the previous technology to make the jump to work on the new one. <br />
<br />
So according to Sandi, what is your future?<br />
<ul>
<li>What you are working on now probably won't be relevant in 10 years </li>
<li>You will all die</li>
<li>All the people you love will die</li>
<li>Your body will start to fail you</li>
<li>Life is short </li>
<li>Tell people that you love them</li>
<li>Guard your health</li>
<li>Spend time with your kids </li>
<li>Get some exercise (she loves to bike)</li>
<li>We are bigger than tech</li>
<li>Community and schools need help</li>
<li>She gave the example of Habitat for Humanity where she volunteers</li>
<li>These organizations also need help to write code, they might not have the knowledge or time to do it right</li>
</ul>
<br />
The last talk I attended was by <a href="https://twitter.com/sabrinageremia">Sabrina Geremia</a> of Google Canada. She talked about the factors that encourage a girl to consider computer science (encouragement, career perception, self-perception and academic exposure.) <br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhOtWnYCB6IKOwQnYxKcLZLUBCU8lCyBYO7nUga15_2sw1Ov1SsRZXuzAffXx7d2xuK2tW08yXeg1pEenmSoj_0PUzAnLVTYmEl83Eo095yCAFefozXLONXc1fN-5RbHVvJeOD0/s1600/IMG_1531.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhOtWnYCB6IKOwQnYxKcLZLUBCU8lCyBYO7nUga15_2sw1Ov1SsRZXuzAffXx7d2xuK2tW08yXeg1pEenmSoj_0PUzAnLVTYmEl83Eo095yCAFefozXLONXc1fN-5RbHVvJeOD0/s640/IMG_1531.jpg" width="640" /></a></div>
<br />
I found that this talk was interesting but it focused a bit too much on the pipeline argument - that the major problem is that girls are not enrolling in CS courses. If you look at all the problems with environment, culture, lack of pay equity and opportunities for promotion due to bias, maybe choosing a career where there is more diversity is a better choice. For instance, law, accounting and medicine have much better numbers for these issues, despite there still being an imbalance.<br />
<br />
At the end of the day, there was a panel to discuss diversity issues:<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh5XIlUNTRNxx7wgf47Tr7u1nJRubEno9yqsLYskMP6g6lLJWggne6wI9tLf4Kf1zlTdHGvJZIkxKHez-_9k0A83wCbJs4yRL8z-N4lY-jlIgT4J2XIfJZCjqVF1QTjdwJ2A-_J/s1600/IMG_1534.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh5XIlUNTRNxx7wgf47Tr7u1nJRubEno9yqsLYskMP6g6lLJWggne6wI9tLf4Kf1zlTdHGvJZIkxKHez-_9k0A83wCbJs4yRL8z-N4lY-jlIgT4J2XIfJZCjqVF1QTjdwJ2A-_J/s640/IMG_1534.jpg" width="640" /></a></td></tr>
<tr align="left"><td class="tr-caption">Moderator: Ariti Sharma, Shopify, Panelists: Mohammed Asaduallah, Format, Katie Krepps, Capital One Canada, Lateesha Thomas, Dev Bootcamp, Ramya Raghavan, Google, Kara Melton, TWG, Gladstone Grant, Microsoft Canada</td></tr>
</tbody></table>
Some of my notes from the panel<br />
<ul>
<li>Be intentional about seeking out talent</li>
<li>Fix culture to be more diverse</li>
<li>Recruit from bootcamps. Better diversity today. Don't wait for universities to change the ratios.</li>
<li>Environment impacts retention</li>
<li>Conduct and engagement survey to see if underrepresented groups feel that their voices are being heard.</li>
<li>There is a need for sponsorship, not just mentoring. Define a role that doesn't exist at the company. A sponsor can make that role happen by advocating for it at higher levels</li>
<li>Mentors do better if matched with demographics. They will realize the challenges that you will face in the industry better than a white man who has never directly experienced sexism or racism.</li>
<li>Sponsors tend to be men due to the demographics of our industry</li>
<li>At Microsoft, when you reach a certain level your are expected to mentor an unrepresented person</li>
<li>Look at compensation and representation across diverse groups</li>
<li>Attrition is normal, it varies by region, especially acute in San Francisco.</li>
<li>Women leave companies at 2x the rate of men due to culture</li>
<li>You shouldn't stay at a place if you are burnt out, take care of yourself.</li>
</ul>
<br />
Compared to the previous two iterations of this conference, it seemed that this time it focused a lot more on solutions to have more diversity and inclusion in your company. The previous two conferences I attended seemed to focus more on technical talks by diverse speakers.<br />
<br />
<br />
As a side note, there were a lot of Shopify folks in attendance because they ran the conference. They sent a bus of people from their head office in Ottawa to attend it. I was really struck at how diverse some of the teams were. I met group of women who described themselves as a team of "five badass women developers" 💯 As someone who has been the only woman on her team for most of her career, this was beautiful to see and gave me hope for the future of our industry. I've visited the Ottawa Shopify office several times (Mr. Releng works there) and I know that the representation of of their office doesn't match the demographics of the Beyond the Code attendees which tended to be more women and people of colour. But still, it is refreshing to see a company making a real effort to make their culture inclusive. I've read that it is easier to make your culture inclusive from the start, rather than trying to make difficult culture changes years later when your teams are all homogeneous. So kudos to them for setting an example for other companies.<br />
<br />
Thank you Shopify for organizing this conference, I learned a lot and I look forward to the next one!<br />
<br />Kim Moirhttp://www.blogger.com/profile/14700841495895160750noreply@blogger.com0tag:blogger.com,1999:blog-25023280.post-59256391176546231882016-07-29T12:06:00.004-07:002016-07-29T12:31:34.717-07:00Ottawa Python Authors Meetup: Artificial Intelligence with PythonLast night, I attended my first Ottawa Python Authors Meetup. It was the first time that I had attended despite wanting to attend for a long time. (Mr. Releng also works with Python and thus every time there's a meetup, we discuss who gets to go and who gets to stay home and take care of little Releng. It depends on if the talk to more relevant to our work interests.)<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEguAHKDbtRLBBFSczA7HTbAlKPnXZ6Xwamoq4BEmV6C0NMHikzPu1XeJ-k8o3Yx1kRsEaIdmgHgaVw7Bd4bGqXMOyxmQcsQtsftQkRPF5hze2XSf1D3_1_Of5G53XgJOuQ7jCAy/s1600/IMG_0483.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="433" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEguAHKDbtRLBBFSczA7HTbAlKPnXZ6Xwamoq4BEmV6C0NMHikzPu1XeJ-k8o3Yx1kRsEaIdmgHgaVw7Bd4bGqXMOyxmQcsQtsftQkRPF5hze2XSf1D3_1_Of5G53XgJOuQ7jCAy/s640/IMG_0483.jpg" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">The venue was across the street from Confederation Park aka land of Pokemon.</td></tr>
</tbody></table>
<br />
<br />
I really enjoyed it. The people I chatted with were very friendly and welcoming. Of course, I ran into some people I used to work with, as is with any tech event in Ottawa it seems. Nice to catch up!<br />
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj-zrA36k1UGNt7wnr00FYLXrq8r1sYr6fMHoMk_-Xjh-7pD4oPuObpVR3yqddovMZ4WRrwUVkKn9suGIoEMEXQk-lsb5L86Hu3_iWbu_o418FNX-Az0FYQAUc1OhkSr1bmZNCq/s1600/IMG_0484.jpg" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj-zrA36k1UGNt7wnr00FYLXrq8r1sYr6fMHoMk_-Xjh-7pD4oPuObpVR3yqddovMZ4WRrwUVkKn9suGIoEMEXQk-lsb5L86Hu3_iWbu_o418FNX-Az0FYQAUc1OhkSr1bmZNCq/s640/IMG_0484.jpg" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">The venue had the Canada Council for the Arts as a tenant, thus the quintessentially Canadian art.</td></tr>
</tbody></table>
<br />
<br />
<div class="separator" style="clear: both; text-align: left;">
The speaker that night was <a href="https://twitter.com/emdaniels">Emily Daniels,</a> developer from <a href="http://www.halogensoftware.com/">Halogen Software</a> who spoke on Artificial Intelligence with Python. (<a href="https://www.dropbox.com/s/so91aw5njfg73fj/Artificial%20Intelligence%20with%20Python.pdf?dl=0">Slides here</a>, <a href="https://github.com/emdaniels/poetic-inner-join">github repo here</a>). She mentioned that she writes Java during the day but works on fun projects in Python at night. She started the talk by going through some examples of artificial intelligence on the web. Perhaps the most interesting one I found was a recurrent neural network called <a href="https://medium.com/artists-and-machine-intelligence/adventures-in-narrated-reality-6516ff395ba3#.r778jniy1">Benjamin</a> which <a href="http://benjamin-ai.tumblr.com/">generates movie script ideas</a> and was trained on existing sci-fi movies and movie scripts. Also, a short film called <a href="http://video.arstechnica.com/watch/sunspring-sci-fi-short-film">Sunspring</a> was made of one of the generated scripts. The dialogue is kind of stilted but it is interesting concept.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhVRw-Xwpb5OzSs6TOZPGSuHVcoNXO07_9NHG_6TJCfz_Bt6GuUfdrepGdO6SGV21q57lzLh0tKNGHtrwmhfiiALPmJKkXeft52VqO5fDq9UTIGocAi4LTX2wsRrxTxWahngzlg/s1600/IMG_0485.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="268" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhVRw-Xwpb5OzSs6TOZPGSuHVcoNXO07_9NHG_6TJCfz_Bt6GuUfdrepGdO6SGV21q57lzLh0tKNGHtrwmhfiiALPmJKkXeft52VqO5fDq9UTIGocAi4LTX2wsRrxTxWahngzlg/s640/IMG_0485.jpg" width="640" /></a></div>
After the examples, Emily then moved on to how it all works. <br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEidy4st1fGGuWIrGJpGrLtFDdu-7btEjwlKr5PP2KH8Z49DYXBfFjc0LqxCS5iL0SW6ST5lMj3NaDBJZinJeS2J-9pyd50xMGEvyYALvinOt5SvquNmmKBfQORPTNGPsG52P7aW/s1600/IMG_0487.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="304" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEidy4st1fGGuWIrGJpGrLtFDdu-7btEjwlKr5PP2KH8Z49DYXBfFjc0LqxCS5iL0SW6ST5lMj3NaDBJZinJeS2J-9pyd50xMGEvyYALvinOt5SvquNmmKBfQORPTNGPsG52P7aW/s640/IMG_0487.jpg" width="640" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: left;">
Deep learning is a type of machine learning that drives meaning out of data using a hierarchy of multiple layers that mimics the neural networks of our brain.</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEga9C5-6sraWphaYIitcIKQSfzpqwph9_Ptp9QjIRTFVnpjcOOgdLDBJgptJU4VQ3XPquphBc5vhne-zwGFvbBalHbA9MneYQsprqJUYCWZffnBzWHrTP0-HhoVFmgqp8BPSsWp/s1600/IMG_0491.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="246" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEga9C5-6sraWphaYIitcIKQSfzpqwph9_Ptp9QjIRTFVnpjcOOgdLDBJgptJU4VQ3XPquphBc5vhne-zwGFvbBalHbA9MneYQsprqJUYCWZffnBzWHrTP0-HhoVFmgqp8BPSsWp/s640/IMG_0491.jpg" width="640" /></a></div>
<br />
She then spoke about a <a href="https://www.emilydaniels.com/2016/05/poetic-inner-join/">project she wrote to create</a> generative poetry from a RNN (recurrent neural network). It was based on a <a href="https://github.com/dennybritz/rnn-tutorial-rnnlm">RNN tutorial</a> that <a href="https://github.com/emdaniels/poetic-inner-join/">she heavily refactored</a> to meet her needs. She went through the code that she developed to generate artificial prose from the works of H.G. Wells and Jane Austen. She talked about how she cleaned up the text to remove EOL delimiters, page breaks, chapters numbers and so on. And then it took a week to train it with the data.<br />
<br />
She then talked about another example which used data from Jack Kerouac and Virginia Woolf novels, which she posts some of the results to <a href="https://twitter.com/ker00lf">twitter</a>. <br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiccX0eQBt2pWLxV0VhboCLhoquL9HzIKbb0b5YITW_PE-TzgTDiuiZw9cFqAWf-0CVpTZjUO5atgTULcTp1M4P_wj2kWVBpBQlbJ28YW9wt9KEri9Dgc0-j5zsIjctNQsMFUNw/s1600/IMG_0498.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="306" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiccX0eQBt2pWLxV0VhboCLhoquL9HzIKbb0b5YITW_PE-TzgTDiuiZw9cFqAWf-0CVpTZjUO5atgTULcTp1M4P_wj2kWVBpBQlbJ28YW9wt9KEri9Dgc0-j5zsIjctNQsMFUNw/s640/IMG_0498.jpg" width="640" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: left;">
She also created a twitter account which posts generated text from her RNN that consumes the content of <a href="https://twitter.com/whitinson">Walt Whitman and Emily Dickinson</a>. (I should mention at this point that she chose these authors for her projects because copyrights have expired on these works and they are available on the <a href="https://www.gutenberg.org/">Gutenberg project</a>)</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
After the talk, she field a number of audience questions which were really insightful. There were discussions on the inherent bias in the data because it was written by humans that are sexist and racist. She mentioned that she doesn't post the results of the model automatically to twitter because some of them are really inappropriate since these novels since they learned from text that humans wrote who are inherently biased. </div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
One thing I found really interesting is that Emily mentioned that she felt a need to ensure that the algorithms and data continue to exist, and that they were faithfully backed up. I began to think about all the Amazon instances that Mozilla releng had automatically killed that day as our capacity had peaked and declined. And of the great joy I feel ripping out code when we deprecate a platform. I personally feel no emotional attachment to bring down machines or deleting used code.</div>
<div class="separator" style="clear: both; text-align: left;">
</div>
<div class="separator" style="clear: both; text-align: left;">
Perhaps the sense of a need for a caretaker for these recurrent neural networks and the data they create is related to the fact that the algorithms that output text that is a simulacrum for the work of an author that we enjoy reading. And perhaps that is why we maybe we aren't as attached to a ephemeral pool of build machines as we are are to our phones. Because the phone provides a sense human of connection to the larger world when we may be sitting alone. </div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
Thank you Emily for the very interesting talk, to the <a href="http://www.meetup.com/ottawapython/">Ottawa Python Authors Group</a> for organizing the meetup, and Shopify for sponsoring the venue. Looking forward to the next one!</div>
<br />
<div class="separator" style="clear: both; text-align: left;">
<b>Further reading</b></div>
<div class="separator" style="clear: both; text-align: left;">
<a href="https://www.emilydaniels.com/2016/05/poetic-inner-join/">Poetic Inner Join - Emily Daniels</a></div>
<div class="separator" style="clear: both; text-align: left;">
<a href="https://www.dropbox.com/s/so91aw5njfg73fj/Artificial%20Intelligence%20with%20Python.pdf?dl=0">Artificial Intelligence with Python</a></div>
<div class="separator" style="clear: both; text-align: left;">
<a href="https://github.com/emdaniels/poetic-inner-join">Poetic Inner Join Github repo</a></div>
Kim Moirhttp://www.blogger.com/profile/14700841495895160750noreply@blogger.com0tag:blogger.com,1999:blog-25023280.post-51550975200297096802016-07-29T12:06:00.003-07:002016-07-29T12:06:32.657-07:00Eclipse Committer EmeritusI received this very kind email in my inbox this morning. <br />
<br />
"David Williams has expired your commit rights to the<br />
eclipse.platform.releng project. The reason for this change is:<br />
<br />
We have all known this day would come, but it does not make it any easier.<br />
It has taken me four years to accept that Kim is no longer helping us with<br />
Eclipse. That is how large her impact was, both on myself and Eclipse as a<br />
whole. And that is just the beginning of why I am designating her as<br />
"Committer Emeritus". Without her, I humbly suggest that Eclipse would not<br />
have gone very far. Git shows her active from 2003 to 2012 -- longer than<br />
most! She is (still!) user number one on the build machine. (In Unix terms,<br />
that is UID 500). The original admin, when "Eclipse" was just the Eclipse<br />
Project.<br />
<br />
She was not only dedicated to her job as a release engineer she was<br />
passionate about doing all she could to make other committer's jobs easier<br />
so they could focus on their code and specialties. She did (and still does)<br />
know that release engineering is a field of its own; a specialized<br />
profession (not something to "tack on" at the end) that just anyone can do)<br />
and good, committed release engineers are critical to the success of any<br />
project.<br />
<br />
For anyone reading this that did not know Kim, it is not too late: you can<br />
follow her blog at<br />
<br />
<a data-saferedirecturl="https://www.google.com/url?hl=en&q=http://relengofthenerds.blogspot.com/&source=gmail&ust=1469888511163000&usg=AFQjCNEiujtD-vAXy2u53j0pCiaGnMkqxA" href="http://relengofthenerds.blogspot.com/" rel="noreferrer" target="_blank">http://relengofthenerds.<wbr></wbr>blogspot.com/</a><br />
<br />
You will see that she is still passionate about release engineering and<br />
influential in her field.<br />
<br />
And, besides all that, she was (I assume still is :) a well-rounded, nice<br />
person, that was easy to work with! (Well, except she likes running for<br />
exercise. :)<br />
<br />
Thanks, Kim, for all that you gave to Eclipse and my personal thanks for<br />
all that you taught me over the years (and I mean before I even tried to<br />
fill your shoes in the Platform).<br />
<br />
We all appreciate your enormous contribution to the success of Eclipse and<br />
happy to see your successes continuing.<br />
<br />
To honor your contributions to the project, David Williams has nominated<br />
you for Committer Emeritus status."<br />
<br />
<br />
Thank you David! I really appreciate your kind words. I learned so much working with everyone in the Eclipse community. I had the intention to contribute to Eclipse when I left IBM but really felt that I have given all I had to give. Few people have the chance to contribute to two fantastic open source communities during their career. I'm lucky to have that opportunity. <br />
<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi9AXSRB3z0ZALzlli1IHgr8OWpAmhY4o3vigWYxUgR_88rhTIapQAFHesCUG8WKWDkHzwgH84HVVLQU-b8E2_IiTdEZUQmgeVzTTbhgtVvPnuKwwFzAyljSyv8KQZ1NBbtUj7x/s1600/IMG_0501.JPG" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi9AXSRB3z0ZALzlli1IHgr8OWpAmhY4o3vigWYxUgR_88rhTIapQAFHesCUG8WKWDkHzwgH84HVVLQU-b8E2_IiTdEZUQmgeVzTTbhgtVvPnuKwwFzAyljSyv8KQZ1NBbtUj7x/s640/IMG_0501.JPG" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">My IBM friends made this neat Eclipse poster when I left. The Mozilla dino displays my IRC handle.</td></tr>
</tbody></table>
<br />Kim Moirhttp://www.blogger.com/profile/14700841495895160750noreply@blogger.com2tag:blogger.com,1999:blog-25023280.post-46475559966457427432016-06-03T11:11:00.004-07:002016-06-06T17:59:20.143-07:00Submissions for Releng 2016: due by July 1, 2016The <a href="http://releng.polymtl.ca/RELENG2016/html/index.html">CFP for Releng 2016 is open!</a> The workshop will be held November 18, 2016 in Seattle. It will be held in conjunction with <a href="http://www.cs.ucdavis.edu/fse2016/">FSE 2016</a>. (Foundations of Software Engineering ACM conference)<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjVLC8nIaA9nogdeRmN69UdmYns2_gAthvojSIqyO9qsb5Nt7uYjfVOBo0RFhew3eaIA3D_giAKP7hDZBw7_6sSeMJjaWePGGSPGyygw7u-8UykHwYY-QpkisXp74DisW7LW2aC/s1600/14482954049_8390600c75_b.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="238" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjVLC8nIaA9nogdeRmN69UdmYns2_gAthvojSIqyO9qsb5Nt7uYjfVOBo0RFhew3eaIA3D_giAKP7hDZBw7_6sSeMJjaWePGGSPGyygw7u-8UykHwYY-QpkisXp74DisW7LW2aC/s640/14482954049_8390600c75_b.jpg" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Picture by howardignatius- Creative Commons Attribution-NonCommercial-NoDerivs 2.0 Generic (CC BY-NC-ND 2.0)<br />
https://www.flickr.com/photos/howardignatius/14482954049/sizes/l</td></tr>
</tbody></table>
If you've done something like<br />
<ul>
<li>Migrated to a new build or continuous integration system</li>
<li>Implemented a new release or deployment pipeline</li>
<li>Implemented tooling to simplify managing your apps in a mobile store</li>
<li>Significantly reduced build time with parallelization or some other interesting optimization!</li>
<li>Moved your build and test system to containers</li>
<li>Refactored your infrastructure code for a live production environment</li>
<li>... we'd love to see your submission to the workshop</li>
</ul>
<br />
We'd like to encourage people new to speaking to apply, as
well as those from underrepresented groups in tech. We'd love to hear from some new voices and new companies ! <br />
<br />
<br />
Submissions are due July 1, 2016. If you have questions on of the submission process, topics to submit, or anything else, I'm happy to help! I'm kmoir and I work at mozilla.com or contact me on <a href="https://twitter.com/kmoir">twitter</a>. Submit early and often!Kim Moirhttp://www.blogger.com/profile/14700841495895160750noreply@blogger.com2tag:blogger.com,1999:blog-25023280.post-85152161391946529482016-06-02T13:28:00.002-07:002016-06-03T10:39:31.724-07:00DevOpsDays Toronto recapLast week I attended <a href="http://www.devopsdays.org/events/2016-toronto/welcome/" target="_blank">DevOpsDays Toronto</a>. It was my first time attending a DevOpsDays event and it was quite interesting. It was held at CBC's Glenn Gould studios which is a quick walk from the Toronto Island airport where I landed after an hour flight from Ottawa. This blog post is an overview of some of the talks at the conference.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhqyJ0go-OcaKMZw_N6cWMghNeWK0GKoFDdB3NoBboCdjKukhWnntP8R0NwtY_wrItoOvHz1rYDcZ5kZB1otu7-vjxnlLaGEXVZtCMolg0-viGJIvKyj7BhKnKonCCEreYbYQzt/s1600/IMG_9786.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhqyJ0go-OcaKMZw_N6cWMghNeWK0GKoFDdB3NoBboCdjKukhWnntP8R0NwtY_wrItoOvHz1rYDcZ5kZB1otu7-vjxnlLaGEXVZtCMolg0-viGJIvKyj7BhKnKonCCEreYbYQzt/s400/IMG_9786.jpg" width="300" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Glenn Gould Studios, CBC, Toronto. </td></tr>
</tbody></table>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEggl0hpGU7KTfEq49mIAwckluV_OrYKls_zC9ZtquPVyLRqRLsRS9sp1s5I-VOJJF8YsF76eN1Gi6cyncdbIoGU6y5RPZf1KyNZoRjSgIATQO7k6SMhBag0Ha3ykq_3ERbbhipJ/s1600/IMG_9787.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEggl0hpGU7KTfEq49mIAwckluV_OrYKls_zC9ZtquPVyLRqRLsRS9sp1s5I-VOJJF8YsF76eN1Gi6cyncdbIoGU6y5RPZf1KyNZoRjSgIATQO7k6SMhBag0Ha3ykq_3ERbbhipJ/s400/IMG_9787.jpg" width="300" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Statue of Glenn Gould outside the CBC studios that bear his name.</td></tr>
</tbody></table>
<br />
<h2>
Day 1</h2>
<br />
The day started out with an introduction from the organizers and a brief overview of history of DevOps days. They also made a point about reminding everyone that they had agreed to the code of conduct when they bought their ticket. I found this explicit mention of the code of conduct quite refreshing. <br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhjmIMbSZio9JtDL5GDN4zEiCTGmtZKNU7zZfpW72L1627H1ivwf7dzTq9e5BzJVs0sP7s3YTMgo5AEwo0qKz8eSiIcj3aJ_QCzApPz97ML6Cp5N9ZApVC2xgrbm7U9uE6ZYqUA/s1600/IMG_9788.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhjmIMbSZio9JtDL5GDN4zEiCTGmtZKNU7zZfpW72L1627H1ivwf7dzTq9e5BzJVs0sP7s3YTMgo5AEwo0qKz8eSiIcj3aJ_QCzApPz97ML6Cp5N9ZApVC2xgrbm7U9uE6ZYqUA/s640/IMG_9788.jpg" width="640" /></a><br />
The first talk of the day was <a href="https://twitter.com/botchagalupe" target="_blank">John Willis</a>, evangelist at Docker. He gave an overview of the state of enterprise devops. I found this a fresh perspective because I really don't know what happens in enterprises with respect to DevOps since I have been working in open source communities for so long. John providing an overview of what DevOps encompasses. <br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhM7r-i3JH8VQJil5mZUNoUqNRwD-FYAvpWMs09SGvn-9y2zvxhkpsDXxnfAwOdHRPW3lWuhWuOLffDot3Bra3AYdMQr-jh8dktJxzCsV7AYM5xE36IEAKepyduZjHgdqL3lr1w/s1600/IMG_9790.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhM7r-i3JH8VQJil5mZUNoUqNRwD-FYAvpWMs09SGvn-9y2zvxhkpsDXxnfAwOdHRPW3lWuhWuOLffDot3Bra3AYdMQr-jh8dktJxzCsV7AYM5xE36IEAKepyduZjHgdqL3lr1w/s640/IMG_9790.jpg" width="640" /></a></div>
<br />
DevOps is a continuous feedback loop. <br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi3vts7KGAmsqOC2AHzWuGnGvqlPkv1ovm7Ris5Ie_DoUliPtrqd9X28oGn3lsrO99bc02iktB7sgz5omP_9Rwuuj0CXEONq1-TWcOHHePpt3suhsLImKcqJ5VHbpjmDxqEdBJq/s1600/IMG_9795.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi3vts7KGAmsqOC2AHzWuGnGvqlPkv1ovm7Ris5Ie_DoUliPtrqd9X28oGn3lsrO99bc02iktB7sgz5omP_9Rwuuj0CXEONq1-TWcOHHePpt3suhsLImKcqJ5VHbpjmDxqEdBJq/s640/IMG_9795.jpg" width="640" /></a></div>
<br />
He talked a lot about how empathy is so important in our jobs. He mentions that at Netflix has a slide deck that describes company culture. He doesn't know if this is still the case, but it he had heard that if you hadn't read the company culture deck and show up for an interview at Netflix, you would be automatically disqualified for further interviews. Etsy and Spotify have similar open documents describing their culture.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgr9UROwbbgnN6sJpaSAtyTTOUGfr9cacqThPgbfj2p1d6HZN0JB0ycnta7QZyUVK8juQzwGwmELdDiMgox6HjoESstXl73PwcKEM54rC7A6Tvt4ytXbYFPSBI_s9GQe7B9S76S/s1600/IMG_9794.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgr9UROwbbgnN6sJpaSAtyTTOUGfr9cacqThPgbfj2p1d6HZN0JB0ycnta7QZyUVK8juQzwGwmELdDiMgox6HjoESstXl73PwcKEM54rC7A6Tvt4ytXbYFPSBI_s9GQe7B9S76S/s640/IMG_9794.jpg" width="640" /></a></div>
Here he discusses the research by <a href="http://psychology.berkeley.edu/people/christina-maslach/" target="_blank">Christina Maslach</a> on the six sources of burnout. <br />
<div id="stcpDiv" style="left: -1988px; position: absolute; top: -1999px;">
Christina Maslach </div>
<div id="stcpDiv" style="left: -1988px; position: absolute; top: -1999px;">
Christina Maslach </div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg6a2tWDpQk-TZcx60ZV4Z5Y_ODGjt7jKsyrSX4fn3Mq9W82QnMu9NyBtjxhyphenhyphenKjrOip_hkdnxbEcdo59gEF1cGhAHUOIjvAWtnhZ8N7_59sGq7YZfRyG6srNgH-gAPsC1aJg9Kj/s1600/IMG_9793.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg6a2tWDpQk-TZcx60ZV4Z5Y_ODGjt7jKsyrSX4fn3Mq9W82QnMu9NyBtjxhyphenhyphenKjrOip_hkdnxbEcdo59gEF1cGhAHUOIjvAWtnhZ8N7_59sGq7YZfRyG6srNgH-gAPsC1aJg9Kj/s640/IMG_9793.jpg" width="640" /></a></div>
He gave us some reading to do. I've read the "Release It!" book which is excellent and has some fascinating stories of software failure in it, I've added the other books to my already long reading list.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhZequexLkgM9dell6gb88-ew1y8uB1rAFSWIhaIyNe1E_THas7JKMt3HxzD741uDG5UmgdgHiLOxaantaxtekcxZXUj45h-O1rb2dhVPfqM-WXlExX-eq56D430Nsl5sbeac5Z/s1600/IMG_9792.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhZequexLkgM9dell6gb88-ew1y8uB1rAFSWIhaIyNe1E_THas7JKMt3HxzD741uDG5UmgdgHiLOxaantaxtekcxZXUj45h-O1rb2dhVPfqM-WXlExX-eq56D430Nsl5sbeac5Z/s640/IMG_9792.jpg" width="640" /></a></div>
The rugged manifesto and realizing that the code you write will always be under attack by malicious authors. ICE stands for <a href="http://radar.oreilly.com/2015/01/devops-keeps-it-cool-with-ice.html" target="_blank">Inclusivity, Complexity and Empathy</a>.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEixy33AmpziL0DbuGnmTXNX_kCGDBIQsuEjldDkUjt3oZYsHU7i3A5Vrkei-uc9TwCv6WcemZQWjBPAjCBEreB4U2l1sXhUFqltlOY0v7YbrXw_vlDTuIxfy0qo8NGsKvJhBmZd/s1600/IMG_9797.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEixy33AmpziL0DbuGnmTXNX_kCGDBIQsuEjldDkUjt3oZYsHU7i3A5Vrkei-uc9TwCv6WcemZQWjBPAjCBEreB4U2l1sXhUFqltlOY0v7YbrXw_vlDTuIxfy0qo8NGsKvJhBmZd/s640/IMG_9797.jpg" width="640" /></a></div>
He stated that it's a long standing mantra that you can have two of either fast, cheap or good but recent research shows that today we can many changes quickly, and if there is a failure the mean time to recovery is short.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjvQb0BvDnlKbeXhWwEn_cFEBApywYY_xu6uhfpbKbAIPLa5fNJ44Y_jdSg5rlf1sqWoy-kaBEqr7bX2Rla9BhkPXyHgngzZUO3XXshi622ZF6ckKqXQVIXKn0HPJiCrE-xWLqm/s1600/IMG_9798.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="456" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjvQb0BvDnlKbeXhWwEn_cFEBApywYY_xu6uhfpbKbAIPLa5fNJ44Y_jdSg5rlf1sqWoy-kaBEqr7bX2Rla9BhkPXyHgngzZUO3XXshi622ZF6ckKqXQVIXKn0HPJiCrE-xWLqm/s640/IMG_9798.jpg" width="640" /></a></div>
He left us with some more books to read. <br />
<br />
The second talk was a really interesting talk by Hany Fahim, CEO of <a href="https://vmfarms.com/" target="_blank">VM Farms.</a> It was a short mystery novella describing how VM Farms servers suddenly experienced a huge traffic spike when the Brazilian government banned Whatsapp as a result of a legal order. I love a good war story.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgGm5bGdVDsrkJTPnxESi_WckdAuNGU-J0_C7LJ_TzVFls44zEK1lmbIlp-ljksNB2lHdQVjuFaKhmMJsFZDWnRD1xLlbuW8v4OtZJR1wQlTLSUbLZBZ9Dn2LUuYzhVltuOWY_1/s1600/IMG_9799.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="332" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgGm5bGdVDsrkJTPnxESi_WckdAuNGU-J0_C7LJ_TzVFls44zEK1lmbIlp-ljksNB2lHdQVjuFaKhmMJsFZDWnRD1xLlbuW8v4OtZJR1wQlTLSUbLZBZ9Dn2LUuYzhVltuOWY_1/s640/IMG_9799.jpg" width="640" /></a></div>
Hany discussed one day VMfarms suddenly saw a huge increase in traffic. <br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg8dvhOaO5D5veowoVtZ0EppvJdCUETg9p2pQ0gpKw3EVPR2UhWWxMx7RiHgSNavrW5HXTsMpXTsn83BIpiAPHi3ZcqxHKq_NILoQQRzhioq8PnZNQcXWqscGcuOVcVgCUAtBrl/s1600/IMG_9800.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="394" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg8dvhOaO5D5veowoVtZ0EppvJdCUETg9p2pQ0gpKw3EVPR2UhWWxMx7RiHgSNavrW5HXTsMpXTsn83BIpiAPHi3ZcqxHKq_NILoQQRzhioq8PnZNQcXWqscGcuOVcVgCUAtBrl/s640/IMG_9800.jpg" width="640" /></a></div>
This was a really important point. When your system is failing to scale, it's important to decide if it's a valid increase in traffic or malicious.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjcqArB2AutzuMC6YMTaVPF-Rjtmxg7DHttMgAYWAZJnCBZc3tE234y6LgvT73xl9SkbW-Ku8V95PUCL9gvRDxwTtckddL0BtUkTtHQRQiQxEnGCRICXlwqEhY7zgjfE34_Mo_b/s1600/IMG_9802.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="432" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjcqArB2AutzuMC6YMTaVPF-Rjtmxg7DHttMgAYWAZJnCBZc3tE234y6LgvT73xl9SkbW-Ku8V95PUCL9gvRDxwTtckddL0BtUkTtHQRQiQxEnGCRICXlwqEhY7zgjfE34_Mo_b/s640/IMG_9802.jpg" width="640" /></a></div>
<br />
Looking on twitter, they found that a court case in Brazil had recently ruled that Whatsup would be blocked for 48 hours. Users started circumventing this block via VPN. Looking at their logs, they determined that most of the traffic was resolving to ip addresses from Brazil and that there was a large connection time during SSL handshakes.<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi2XFScElk73URVdPCF_dUTYypbKILXMchjuQ-JO8dZRZobS0-PsSL2Bt6Dr5ocTn1S8OEhkMB4TmQubGGcWbGf06Y3iYMzCrwI4HglsSQwrPdSmcTWjHNKgvUYIq-F0aY0Z9hu/s1600/IMG_9809.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi2XFScElk73URVdPCF_dUTYypbKILXMchjuQ-JO8dZRZobS0-PsSL2Bt6Dr5ocTn1S8OEhkMB4TmQubGGcWbGf06Y3iYMzCrwI4HglsSQwrPdSmcTWjHNKgvUYIq-F0aY0Z9hu/s640/IMG_9809.jpg" width="640" /></a></div>
<br />
The <a href="http://news.bbc.co.uk/2/hi/business/4602325.stm" target="_blank">government of Brazil encouraged the use of open source</a> software versus Windows, and thus the users became more technically literate, and able to circumvent blocks via VPN.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgq1ZxStrM4YgwF0I4K3DqCewyFqhvTGK-nR2oXA6s9E_cja6c2vfmcXypcSUs0BkXH1-UANp54Ep55RreGhCfuoK877moKGrShGV7AyKJ_RrJTCH9L46Ey2UDBkFoWbPbg60Sq/s1600/IMG_9810.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgq1ZxStrM4YgwF0I4K3DqCewyFqhvTGK-nR2oXA6s9E_cja6c2vfmcXypcSUs0BkXH1-UANp54Ep55RreGhCfuoK877moKGrShGV7AyKJ_RrJTCH9L46Ey2UDBkFoWbPbg60Sq/s640/IMG_9810.jpg" width="640" /></a></div>
<br />
In conclusion, making changes to use multi-core <a href="http://www.haproxy.org/" target="_blank">HAProxy</a> fixed a lot of issues. Also, twitter was and continues to be a great source of information on activity that is happening in other countries. Whatsapp was returned to service and then banned a second time, and their servers were able to keep up with the demand.<br />
<br />
After lunch, we were back to to more talks. The organizers came on stage for a while to discuss the afternoon's agenda. They also remarked that one individual had violated the code of conduct and had been removed from the conference. So, the conference had a code of conduct and steps were taken if it was violated. <br />
<br />
Next up, <a href="https://twitter.com/bridgetkromhout" target="_blank">Bridget Kromhout </a>from Pivotal gave a talk entitled <a href="http://bridgetkromhout.com/speaking/2016/devopsdays-toronto/">Containers will not Fix your Broken Culture</a>. <br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgzyWQLJm4_5rzv8R3nWiTylwg0geIckvqsGncGtqMM5zqc4Ut24_fUMePuOYVh6a3Woi9DRgQnOzbahUntAVCjOfkEuLBnBMGwaU12qLzD2KYVRAzDWbaO4TMAk6fqmH648vX7/s1600/IMG_9817.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgzyWQLJm4_5rzv8R3nWiTylwg0geIckvqsGncGtqMM5zqc4Ut24_fUMePuOYVh6a3Woi9DRgQnOzbahUntAVCjOfkEuLBnBMGwaU12qLzD2KYVRAzDWbaO4TMAk6fqmH648vX7/s640/IMG_9817.jpg" width="640" /></a> I first saw Bridget speak at <a href="http://relengofthenerds.blogspot.ca/2014/10/beyond-code-2014-recap.html" target="_blank">Beyond the Code</a> in Ottawa in 2014 about scaling the streaming services for Drama Fever on AWS. At the time, I was moving our mobile test infrastructure to AWS so I was quite enthralled with her talk because 1) it was excellent 2) I had never seen another woman give a talk about scaling services on AWS. Representation matters.<br />
<br />
The summary of the talk last week was that no matter what tools you adopt, you need to communicate with each other about the cultural changes are required to implement new services. A new microservices architecture is great, but if these teams that are implementing these services are not talking to each other, the implementation will not succeed. <br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgEN8fJPfCg3SRGI9aKcx1E3rOV-P5pFAUfSfXsRvYX8Iscai8Hvl4npxdc7owtMlvbSiDkT-oJh3OKZVukVTpbMEVXMHUS8Sjp0w_9_CEonRGsc6QhZHfWWyYFP62c-dDdhb44/s1600/IMG_9820.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgEN8fJPfCg3SRGI9aKcx1E3rOV-P5pFAUfSfXsRvYX8Iscai8Hvl4npxdc7owtMlvbSiDkT-oJh3OKZVukVTpbMEVXMHUS8Sjp0w_9_CEonRGsc6QhZHfWWyYFP62c-dDdhb44/s640/IMG_9820.jpg" width="640" /></a></div>
Bridget pointing out that the technology we choose to implement is often about what is fashionable. <br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgMGHN2ZLuMwVgsnyR1uZcy_V7pjTtLr4KIY3987ZVXtcoP0PUt09bcI_7huuhjbo1Ju2h1DA7zV5iSw48wzUXr86g_gS1xBR_nxWE5fNv0PbtNFXV7JDcp3lnBoIc5X2D6hrlF/s1600/IMG_9822.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgMGHN2ZLuMwVgsnyR1uZcy_V7pjTtLr4KIY3987ZVXtcoP0PUt09bcI_7huuhjbo1Ju2h1DA7zV5iSw48wzUXr86g_gS1xBR_nxWE5fNv0PbtNFXV7JDcp3lnBoIc5X2D6hrlF/s640/IMG_9822.jpg" width="640" /></a></div>
Shoutout to Jennifer Davis' and Katherine Daniel's Effective DevOps book. (note - I've read it on Safari online and it is excellent. The chapter on hiring is especially good)<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjxwmkylhe6Rsu5zWfU6vC-hh1LgXmJW75qPtDNGKV9s1Q0iNKwGQHDU4dltnVmKR3hSQCURfNzZBywiDW9QiiOeT4_MO0Mr5WieZnXLNMwjkMw4WLZ_9263wz_MP0A90h3INrD/s1600/IMG_9823.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjxwmkylhe6Rsu5zWfU6vC-hh1LgXmJW75qPtDNGKV9s1Q0iNKwGQHDU4dltnVmKR3hSQCURfNzZBywiDW9QiiOeT4_MO0Mr5WieZnXLNMwjkMw4WLZ_9263wz_MP0A90h3INrD/s640/IMG_9823.jpg" width="640" /></a></div>
Loved this poster about the wall of confusion between development and operations. <br />
<br />
In the afternoon, there were were lightning talks and then open spaces. Open spaces are free flowing discussions where the topic is voted upon ahead of time. I attended ones on infrastructure automation, CI/CD at scale and my personal favourite, horror stories. I do love hearing how distributed system can go down and how to recover. I found that the conversations were useful but it seemed like some of them were dominated by a few voices. I think it would be better if the person that suggested to topic for the open space also volunteered to moderate the discussion.<br />
<br />
<h2>
Day 2</h2>
The second day started out with a fantastic talk by John Arthorne of Shopify speaking on <a href="https://speakerdeck.com/jarthorn/devops-toronto-2016-scaling-out-continuous-delivery" target="_blank">scaling their deployment pipeline</a>. As a side note, John and I worked together for more than a decade on <a href="http://relengofthenerds.blogspot.ca/2011/10/history-of-lizard-wrangling-and-other.html" target="_blank">Eclipse while we both worked at IBM </a>so it was great to catch up with him after the talk. <br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhoqZQ_APtR6PCNsItCu2KcX_gVfiVlC4tRHwPslY6ANM7yMn4JnB3N0yvLCqCMYfMuOSTuUdYHt-gsdhTIWDVh7tjy275zPfExRtFvMYora7-JQ50Q_awc_B3f2Lr3Oes526eu/s1600/IMG_9834.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhoqZQ_APtR6PCNsItCu2KcX_gVfiVlC4tRHwPslY6ANM7yMn4JnB3N0yvLCqCMYfMuOSTuUdYHt-gsdhTIWDVh7tjy275zPfExRtFvMYora7-JQ50Q_awc_B3f2Lr3Oes526eu/s640/IMG_9834.jpg" width="640" /></a></div>
<br />
He started by giving some key platform characteristics. Stores on
Shopify have flash sales that have traffic spikes so they need to be able to scale
for these bursts of traffic. <br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEivbLPcD_ai7wMx-VNKDLPb8XRUszR6aNuyHpivsgTWmhdFUHl8xOn3v_x2-QksZg5TC6yCQ0x3g1J75k8j1e54uRwpWm7cNvKHCOkOT1sWAQH9iuQGKruWicGx8EQAphWnNWzS/s1600/IMG_9836.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEivbLPcD_ai7wMx-VNKDLPb8XRUszR6aNuyHpivsgTWmhdFUHl8xOn3v_x2-QksZg5TC6yCQ0x3g1J75k8j1e54uRwpWm7cNvKHCOkOT1sWAQH9iuQGKruWicGx8EQAphWnNWzS/s640/IMG_9836.jpg" width="640" /></a></div>
From commit to deploy in 10 minutes. Everyone can deploy. This has two purposes: Make sure the developer stays involved in the deploy process. If it only takes 10 minutes, they can watch to make sure that their deploy succeeds. If it takes longer, they might move on to another task. Another advantage of this quick deploy process is that it can delight customers with the speed of deployment. They also deploy in small batches to ensure that the mean time to recover is small if the change needs to be rolled back.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhHcY3c2EOJcQyk-vu_5IlUvfwJvZXE2aSzQc_GJvTkaRyk3L6U3ZBF-zC2CAK1LEN65b17BAvTfzpRgA4suKF2yfF5Ny6ma4s13CuanrvdTxl3fgzffhyphenhyphenYHaFM9JNzRPrdFvHr/s1600/IMG_9844.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhHcY3c2EOJcQyk-vu_5IlUvfwJvZXE2aSzQc_GJvTkaRyk3L6U3ZBF-zC2CAK1LEN65b17BAvTfzpRgA4suKF2yfF5Ny6ma4s13CuanrvdTxl3fgzffhyphenhyphenYHaFM9JNzRPrdFvHr/s640/IMG_9844.jpg" width="640" /></a></div>
BuildKite is a third party build and test orchestration service. They wrote a tool called Scrooge that monitors the number of EC2 nodes based on current demand to reduce their AWS bills. (Similar to what Mozilla releng does with <a href="https://github.com/mozilla/build-cloud-tools" target="_blank">cloud-tools</a>)<br />
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgsWNoCqyOiZD18bv6kvD93rbXtif1IH4wbkte1AChem-Wr6QUdLTfEFiX7umjtUUtC8t-__UWyma2ZfgNj_xF0WKvrQv3J3dWxPTOaeiRM0N9vl2q3IfmjyDPSGySWQnwnlDWp/s1600/IMG_9846.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgsWNoCqyOiZD18bv6kvD93rbXtif1IH4wbkte1AChem-Wr6QUdLTfEFiX7umjtUUtC8t-__UWyma2ZfgNj_xF0WKvrQv3J3dWxPTOaeiRM0N9vl2q3IfmjyDPSGySWQnwnlDWp/s640/IMG_9846.jpg" width="640" /></a></div>
Shopify uses a open source orchestration tool called ShipIt. I was sitting next to my colleague <a href="http://armenzg.blogspot.ca/" target="_blank">Armen</a> at the conference and he started chuckling at this point because at Mozilla we also wrote an application called ship-it which release management uses to kick off Firefox releases. Shopify also has a overall view of the ship it deployment process which allows developers to see the percentages of nodes where their change has been deployed. One of the questions after the talk was why they use AWS for their deployment pipeline when they have use machines in data centres for their actual customers. Answer: They use AWS where resilency is not an issue. <br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi8ohfb_QUWAvsl2jbnIcjmA6W03FAWahdRO7-h3io3fqz76ifHI0QrfiuMYzR9qb5-8PW9axA-KivBOvdmN0WuRt3HwnlJoGtXGvOcw5dB72-mj2bArelcS9SWYPkvf16OlSEh/s1600/IMG_9848.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi8ohfb_QUWAvsl2jbnIcjmA6W03FAWahdRO7-h3io3fqz76ifHI0QrfiuMYzR9qb5-8PW9axA-KivBOvdmN0WuRt3HwnlJoGtXGvOcw5dB72-mj2bArelcS9SWYPkvf16OlSEh/s640/IMG_9848.jpg" width="640" /></a></div>
Building containers is computationally expensive. He noted that a lot of
engineering resources went into optimizing the layers in the Docker
containers. To isolate changes to the smallest layer. They build
service called Locutus to build the containers on commit, and push to a
registry. It employs caching to make the builds smaller. <br />
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
One key point that John also mentioned is that they had a team dedicated to optimizing their deployment pipeline. It is unreasonable to expect that developers working on the core Shopify platform to also optimize the pipeline.</div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
In the afternoon , there were a series of lightning talks. Roderick Randolph from Capital One gave an amazing talk about <a href="https://speakerdeck.com/roderickrandolph/supporting-developers-through-devops" target="_blank">Supporting Developers through DevOps</a>.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj8X7YvL_oXxaX45muExWJzgkXIcn4mH-QiQP9m8f5-9h-J5rx6k1VBNyQmqMnEb_pueipE91BuAVzSS4mcKvk29KP8NlZlhOQ-4Imak0w57Fv1JeGeYjtfCzxft-iXa6a3RUae/s1600/IMG_9860.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj8X7YvL_oXxaX45muExWJzgkXIcn4mH-QiQP9m8f5-9h-J5rx6k1VBNyQmqMnEb_pueipE91BuAVzSS4mcKvk29KP8NlZlhOQ-4Imak0w57Fv1JeGeYjtfCzxft-iXa6a3RUae/s640/IMG_9860.jpg" width="640" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: left;">
It was an interesting perspective. I've seen quite a few talks about bringing devops culture and practices to the operations side of the house, but the perspective of teaching developers about it is discussed less often.</div>
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj4tcyDyY8AumNXfCi3va1psEpciIlimFsBjJlkAeca10KV3DaEnhzbX3XJ-_dgxz0DKvsRG9IdzJDXsgwA8zqs0LoXQpQPelWnT-pFeLYJG5_o7bd9c6_AjLYAo2SPPTBZCXvq/s1600/IMG_9861.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj4tcyDyY8AumNXfCi3va1psEpciIlimFsBjJlkAeca10KV3DaEnhzbX3XJ-_dgxz0DKvsRG9IdzJDXsgwA8zqs0LoXQpQPelWnT-pFeLYJG5_o7bd9c6_AjLYAo2SPPTBZCXvq/s640/IMG_9861.jpg" width="640" /></a><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg2wK73YORkV6lJ2HZNk6EfWkz-HRSMNdhdFTyWo-dYzFcG1CwyAZ5Hw6w2kfx5cX_qq410qSEiUlpLa0GeMT5mFI_NaTgL_22WIO935Yu0cT_491iIEmzeyRCa6W1HzliCQDZV/s1600/IMG_9864.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;">'</a></div>
<br />
He emphasized the need to empower developers to use DevOp practices by giving them tools, and showing them how to use them. For instance, if they needed to run docker to test something, walk them through it so they will know how to do it next time. <br />
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjVJPK23ZeB4Ta9vi7buVzLooLUhKyYholbBXkJPcS0jwq5LNofuEGJP7HQzTqjF0Z5xqHr0q72EzYiOYTQCtMfcxocIqSa4_tW1pAIXG3lrQ2kCGtFRjGceu_gUPsdsEaA3eMg/s1600/IMG_9863.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjVJPK23ZeB4Ta9vi7buVzLooLUhKyYholbBXkJPcS0jwq5LNofuEGJP7HQzTqjF0Z5xqHr0q72EzYiOYTQCtMfcxocIqSa4_tW1pAIXG3lrQ2kCGtFRjGceu_gUPsdsEaA3eMg/s640/IMG_9863.jpg" width="640" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<br />
<div class="separator" style="clear: both; text-align: left;">
The final talk I'll mention is by <a href="http://buildingbananas.com/" target="_blank">Will Weaver</a>. He talks about how it is hard to show prospective clients how he had CI and tests experience when that experience is not open to the public. So he implemented tests and CI for his dotfiles on github. </div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEguIwSLWiF6jNtqrxLENgD5jHZySKt__dsvdjdaooiQB4PZ4oTRcp1vaxLGnp0pRkSLLa1oRF4TsHv3nn0-iCyhtNEBLEnEh9YqGgJAyAFwGoQ42Wo-_VmwBWfqsBifnFQ0kkDX/s1600/IMG_9866.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEguIwSLWiF6jNtqrxLENgD5jHZySKt__dsvdjdaooiQB4PZ4oTRcp1vaxLGnp0pRkSLLa1oRF4TsHv3nn0-iCyhtNEBLEnEh9YqGgJAyAFwGoQ42Wo-_VmwBWfqsBifnFQ0kkDX/s640/IMG_9866.jpg" width="640" /></a></div>
<br />
<br />
He had excellent advice on how to work on projects outside of work to showcase skills for future employers.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjPLLBmUiARTn52gFxQwETM0yQc-uCKlSqoeeZL9zHWkSkp3YzWs-YmGJdnb3PtvflewCcoFl0uRZOlF8KAmihuH4MViq0SzWLLET23TZtiTQWVIsNeiNxBIoGGdXiocinMuj_8/s1600/IMG_9867.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="398" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjPLLBmUiARTn52gFxQwETM0yQc-uCKlSqoeeZL9zHWkSkp3YzWs-YmGJdnb3PtvflewCcoFl0uRZOlF8KAmihuH4MViq0SzWLLET23TZtiTQWVIsNeiNxBIoGGdXiocinMuj_8/s640/IMG_9867.jpg" width="640" /></a></div>
<br />
<br />
<br />
<h2>
<b>Diversity and Inclusion</b></h2>
<br />
As an aside, whenever I'm at a conference I note the number of people in the "not a white guy" group. This conference had an all men organizing committee but not all white men. (I recognize the fact that not all diversity is visible i.e. mental health, gender identity, sexual orientation, immigration status etc) They was only one woman speaker, but there were a few non-white speakers. There were very few women attendees. I'm not sure what the process was to reach out to potential speakers other than the CFP. <br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjx8m0jLforCHsBZx0PRl8Bpz60JZqjJvs-oYsEnB8ZApyYq4I7nBuASrEsf5rsNVQdoUpgpsaM4V3bk0Eh_JR0gxmlk12BCV2TxuSIt5mNfmkFxEUkIb7_ku6AbHH4WVvGJw-6/s1600/Screen+Shot+2016-06-01+at+4.25.50+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="452" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjx8m0jLforCHsBZx0PRl8Bpz60JZqjJvs-oYsEnB8ZApyYq4I7nBuASrEsf5rsNVQdoUpgpsaM4V3bk0Eh_JR0gxmlk12BCV2TxuSIt5mNfmkFxEUkIb7_ku6AbHH4WVvGJw-6/s640/Screen+Shot+2016-06-01+at+4.25.50+PM.png" width="640" /></a></div>
<br />
<br />
There were slides that showed diverse developers which was refreshing.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgduIVJXOdJezswUiLKFdeiUYG6eKge1pqRCoWtEVm2CXCZpgo8O1UFst2bZ5nS6PKWh8H143-APRdDqqHKhyB18nyW7psYbW-T7sC2RGElz299uCzUCTIOk7dZ7igYs_OKGJ4n/s1600/Screen+Shot+2016-06-01+at+4.27.50+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="508" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgduIVJXOdJezswUiLKFdeiUYG6eKge1pqRCoWtEVm2CXCZpgo8O1UFst2bZ5nS6PKWh8H143-APRdDqqHKhyB18nyW7psYbW-T7sC2RGElz299uCzUCTIOk7dZ7igYs_OKGJ4n/s640/Screen+Shot+2016-06-01+at+4.27.50+PM.png" width="640" /></a></div>
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg1PD54JfKXyy39pauR4NI9Cetx7w7pPL-tRAPw0l26i2E5HS_yPb1v7-BjEvh_w8982ZDw1c35HSUr_MW9nTfKrwFBRI8dxv6K137n9fDNdsRaYEWHr34y-ilvQWNtH8ZioWQE/s1600/IMG_9864.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg1PD54JfKXyy39pauR4NI9Cetx7w7pPL-tRAPw0l26i2E5HS_yPb1v7-BjEvh_w8982ZDw1c35HSUr_MW9nTfKrwFBRI8dxv6K137n9fDNdsRaYEWHr34y-ilvQWNtH8ZioWQE/s640/IMG_9864.jpg" width="640" /></a></div>
Loved Roderick's ops vs dev slide.<br />
<br />
I learned a lot at the conference and am thankful for all the time that the speakers took to prepare their talks. I enjoyed all the conversations I had learning about the challenges people face in the organizations implementing continuous integration and deployment. It also made me appreciate the culture of relentless automation, continuous integration and deployment that we have at Mozilla.<br />
<br />
I don't know who said this during the conference but I really liked it<br />
<br />
<div style="text-align: center;">
<b><i>Shipping is the heartbeat of your company</i></b></div>
<br />
It was interesting to learn how all these people are making their companies heart beat stronger via DevOps practices and tools.Kim Moirhttp://www.blogger.com/profile/14700841495895160750noreply@blogger.com0tag:blogger.com,1999:blog-25023280.post-87953423090976459792016-05-13T08:20:00.001-07:002016-05-13T08:20:10.360-07:00Welcome Mozilla Releng summer internsWe're delighted to have Francis Kang and Connor Sheehan join the Mozilla release engineering team as summer interns. Francis is studying at the University of Toronto while Connor attends McMaster University in Hamilton, Ontario. We'll have another intern (Anthony) join us later on in the summer who will be working from our San Francisco office.<br />
<br />
Francis and Connor will be working on implementing some <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1261447" target="_blank">new features</a> in <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1270620" target="_blank">release promotion</a> as well as <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1271622" target="_blank">migrating</a> <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1251297" target="_blank">some builds</a> to <a href="https://tools.taskcluster.net/" target="_blank">taskcluster</a>. I'll be mentoring Francis, while <a href="https://rail.merail.ca/" target="_blank">Rail</a> will be mentoring Connor. If you are in the Toronto office, please drop by to say hi to them. Or welcome them on irc as fkang or sheehan. <br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjpEkk8IhAzA_bQferSy82PjWn741MwOrpEPHaZK2FPjG65exiDmwSvmN8gf1Gv0dXjPjvPWDn5UM1qOuQ8lXMS1wDr_p6lV6OM2ItH1zpfmJytDnop5sThCGNO8gq6QpZcC5FM/s1600/IMG_9677.JPG" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjpEkk8IhAzA_bQferSy82PjWn741MwOrpEPHaZK2FPjG65exiDmwSvmN8gf1Gv0dXjPjvPWDn5UM1qOuQ8lXMS1wDr_p6lV6OM2ItH1zpfmJytDnop5sThCGNO8gq6QpZcC5FM/s640/IMG_9677.JPG" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Kim, Francis, Connor and Rail</td></tr>
</tbody></table>
They are both already off to a great start and have pull requests merged into production that fixed some release promotion issues. Their code was used in the Firefox 47.0 beta 5 release promotion that we ran last night so their first week was quite productive. <br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhReZcrJPaNHPEtQwAGd07zA9PiqqqbOyzLbG07y1kruukR-iatxDmFYV09NR6c2PVsyw4fscEZPt7n14wGWnRYwkTCgZpyIfZW-3PNO3PZ_G7G7O3OFXCZiTeW71WdoXHDcfVO/s1600/Screen+Shot+2016-05-13+at+10.49.11+AM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="376" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhReZcrJPaNHPEtQwAGd07zA9PiqqqbOyzLbG07y1kruukR-iatxDmFYV09NR6c2PVsyw4fscEZPt7n14wGWnRYwkTCgZpyIfZW-3PNO3PZ_G7G7O3OFXCZiTeW71WdoXHDcfVO/s640/Screen+Shot+2016-05-13+at+10.49.11+AM.png" width="640" /></a></div>
<br />
Mentoring an intern provides an opportunity to see the systems we run from a fresh perspective. They both have lots of great questions which makes us revisit why design decisions were made, could we do things better? Like all teaching roles, I always find that I learn a tremendous amount from the experience, and hope they have fun learning real world software engineering concepts with respect to running large distributed systems.<br />
<br />
Welcome to Mozilla!<br />
<br />Kim Moirhttp://www.blogger.com/profile/14700841495895160750noreply@blogger.com0tag:blogger.com,1999:blog-25023280.post-792662876474912952016-03-07T07:56:00.001-08:002016-03-07T07:56:45.177-08:00RelEng & RelOps Weekly highlights - March 4, 2016It was a busy week with many releases in flight, as well as preparation for running beta 1 with release promotion next week. We also are in the process of adding more capacity to certain test platform pools to lower wait times given all the new e10s tests that have been enabled.<br /><br /><b>Improve Release Pipeline:</b> <br />
<ul>
<li>Nick ran a staging release for 46.0b1 to check for issues before the merge, preventing some bustage for Fennec and ensuring we can fall back to the old system if any unexpected issues show up with release promotion</li>
<li>Varun improved Balrog’s detection of <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1113111" target="_blank">certain types of bad data</a>.</li>
<li>Ben finished most of the work involved with <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1248751" target="_blank">preparing Balrog to move to CloudOps infrastructure</a>, including automatically building Docker images.</li>
<li>We’re hoping to do our first beta with the new release build promotion pipeline next week for 47.0b1. Stay tuned!</li>
</ul>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://i.imgur.com/WMmqSDI.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="http://i.imgur.com/WMmqSDI.jpg" height="400" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Everyone gets a release promotion! Source: <span class="message"><span class="content"><a class="imgur loading" data-imgur_embed="http://i.imgur.com/WMmqSDI.jpg" href="http://i.imgur.com/WMmqSDI.jpg" rel="noreferrer" target="_blank">http://i.imgur.com/WMmqSDI.jpg</a></span></span></td></tr>
</tbody></table>
<br /><b>Improve CI Pipeline:</b><br />
<ul>
<li>Dustin deployed a new version of the TaskCluster tools/login system with much improved UI for handling signing in and out and editing clients and roles. He also simplified the existing roles, with the result that the set of roles now fits on one screen, and is entirely composed of human-readable names. All of this works toward two important goals: building a sign-in system that is useful and usable by all mozillians; and configuring the access-control system to give everyone their appropriate permissions and no more.</li>
</ul>
<br /><b>Release:</b><br /><br />
The releases calendar is getting busier as we get closer to the end of the cycle. Many releases were shipped or are still in-flight:<br />
<ul>
<li>Firefox 45.0b10</li>
<li>Fennec 45.0b11</li>
<li>Fennec 45.0 (in-progress)</li>
<li>Firefox 45.0 (in-progress) - we shipped the RC to the beta channel</li>
<li>Firefox 45.0esr (in-progress)</li>
<li>Firefox 38.7.0esr (in-progress)</li>
</ul>
As always, you can find more specific release details in our post-mortem minutes:<br /><a href="https://wiki.mozilla.org/Releases:Release_Post_Mortem:2016-03-02" target="_blank">https://wiki.mozilla.org/Releases:Release_Post_Mortem:2016-03-02</a> <br />
<a href="https://wiki.mozilla.org/Releases:Release_Post_Mortem:2016-03-09">https://wiki.mozilla.org/Releases:Release_Post_Mortem:2016-03-09</a><br /><br /><b>Operational:</b><br />
<ul>
<li>Vlad, Alin and Amy reallocated 28 Windows test machines to the w7 pool to help with backlog and e10s testing.</li>
<li>Jake deployed new OpenSSL packages to protect our infrastructure from Drown Attack and various other recent OpenSSL vulnerabilities</li>
</ul>
<br />Until next time!Kim Moirhttp://www.blogger.com/profile/14700841495895160750noreply@blogger.com0tag:blogger.com,1999:blog-25023280.post-79288320820302078402016-02-29T12:22:00.001-08:002016-02-29T12:22:47.911-08:00RelEng & RelOps Weekly highlights - February 26, 2016It was a busy week for release engineering as several team members travelled to the Vancouver office to sprint on the release promotion project. The goal of the release promotion project is to promote continuous integration builds to release channels, allowing us to ship releases much more quickly.<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEigslmmLGbOOci3EtvZrepRuDlo3UY-vBXm3r4fdSPDKutd9DGE3jj1JagJKURhVkQALqYcZrUXMBvbL9W3vugCFr7dFc9xjlmZmItEeEmhTKizT-h3YjbAVz1ouTTFwtGZ6Y6k/s1600/IMG_9419.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEigslmmLGbOOci3EtvZrepRuDlo3UY-vBXm3r4fdSPDKutd9DGE3jj1JagJKURhVkQALqYcZrUXMBvbL9W3vugCFr7dFc9xjlmZmItEeEmhTKizT-h3YjbAVz1ouTTFwtGZ6Y6k/s640/IMG_9419.jpg" width="640" /></a></div>
<br />
<b>Improve Release Pipeline:</b><br />
<ul>
<li>Chris, Jordan, Callek (remotely), Kim, Mihai and Rail had a sprint on Release Promotion. We made so much progress on this project that we decided to use the new process for Firefox 46.0b1. <a href="https://bugzil.la/1118794">https://bugzil.la/1118794</a> So many green jobs! </li>
</ul>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgEncaVjp3jrg_HxRM9wa9TbumE2EKB18zQTz9H3k-GPncdHU4bMa5gVdWwYemtdVcIpAISkHYMZhH2teFtTGrwwON98YBDja5TOv1PZm3HaBFxDUKZU4tJ_3wHkXZTihOF-Haz/s1600/Screen+Shot+2016-02-29+at+3.13.38+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="608" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgEncaVjp3jrg_HxRM9wa9TbumE2EKB18zQTz9H3k-GPncdHU4bMa5gVdWwYemtdVcIpAISkHYMZhH2teFtTGrwwON98YBDja5TOv1PZm3HaBFxDUKZU4tJ_3wHkXZTihOF-Haz/s640/Screen+Shot+2016-02-29+at+3.13.38+PM.png" width="640" /></a></div>
<ul>
<li>A new contributor, Kumar, fixed a longstanding annoyance with how <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1140262">Balrog shows changes to Releases</a>. </li>
<li>Ben and Nick came up with <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1245941#c5">a new way to model GMP and System Addons in Balrog</a> which will allow for more fine grained permissions and simpler rules. </li>
<li>Mihai added a <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1247060">functionality</a> in tctalker to walk the graph and cancel all pending/running tasks. He also added <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1239778">release sanity check</a> logic for en-US binaries within promotion. </li>
<li>Kim added support for <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1210538">virus scanning</a> of release promotion artifacts, <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1247428">a new docker image</a> for this task, <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1249676">a task to move source bundles and checksum artifacts</a>, and <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1251299">sha256 checksums</a> for the docker images we use in release promotion. </li>
</ul>
<br />
<b>Improve CI Pipeline:</b><br />
<ul>
<li>Rok added a button “<a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1250414">Purge workers cache</a>” in Taskcluster </li>
<li>A new contributor, Varun, <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1248713">is helping to improve our l10n repack process</a>. </li>
<li>Aki added python3 compatibility to taskcluster-client.py <a href="https://github.com/taskcluster/taskcluster-client.py/pull/44">https://github.com/taskcluster/taskcluster-client.py/pull/44 </a></li>
</ul>
<br />
<b>Release:</b><br />
<ul>
<li>Ben, Mihai, Nick, Rail and Callek Shipped Firefox 45.0b6, Firefox 45.0b7, Firefox 45.0b9, Fennec 45.0b6, Thunderbird 45.0b2 </li>
</ul>
<b><br /></b>
<b>Operational:</b><br />
<ul>
<li>Alin landed changes to run mochitest-push-e10s tests on Windows 7 <a href="https://bugzil.la/1248729">https://bugzil.la/1248729</a>. This is another step toward completing the enabling of e10s tests.
</li>
</ul>
Kim Moirhttp://www.blogger.com/profile/14700841495895160750noreply@blogger.com0tag:blogger.com,1999:blog-25023280.post-63256816431043753362016-01-08T12:37:00.002-08:002016-01-08T12:37:53.439-08:00Tips from a resume nerdBefore I begin this post a few caveats: <br />
<ul>
<li>I don't work in HR</li>
<li>I'm not a manager (but I interview Mozilla releng candidates)</li>
<li>I'm not looking for a new job. </li>
<li>These are just my observations after working in the tech industry for a long time.</li>
</ul>
I'm kind of a resume and interview nerd. I like helping friends fix their resumes and write amazing cover letters. In the past year I've helped a few (non-Mozilla) friends fix up their resumes, write cover letters, prepare for interviews as they search for new jobs. This post will discuss some things I've found to be helpful in this process.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgsEi8ivKoppH0HC-6XfnmRR1pVi1zKLXJSKlAXfWI1zFD2gj7C7OLri81ItIbpI8BPMQH3OFdFrPtCbqu5i_xrLIzhgiklZtG-lgZpDUKwRP5eanWWzx5IRP_2UBqEjU2f0UWA/s1600/16223669794_4223d03698_b.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="404" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgsEi8ivKoppH0HC-6XfnmRR1pVi1zKLXJSKlAXfWI1zFD2gj7C7OLri81ItIbpI8BPMQH3OFdFrPtCbqu5i_xrLIzhgiklZtG-lgZpDUKwRP5eanWWzx5IRP_2UBqEjU2f0UWA/s640/16223669794_4223d03698_b.jpg" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container"><tbody>
<tr><td class="tr-caption" style="text-align: center;">Picture by GotCredit - Creative Commons Attribution-NonCommercial-NoDerivs 2.0 Generic (CC BY-NC-ND 2.0)<br />
https://www.flickr.com/photos/jakerust/16223669794/sizes/l</td></tr>
</tbody></table>
</td></tr>
</tbody></table>
<br />
<b></b>
<b>Preparation</b><br />
Everyone tends to jump into looking at job descriptions and making their resume look pretty. Another scenario is that people have a sudden realization that they need to get out of their current position and find a new job NOW and frantically start applying for anything that matches their qualifications. Before you do that, take a step back and make a list of things that are important to you. For example, when I applied at Mozilla, my list was something like this<br />
<br />
<ul>
<li>learn release engineering at scale + associated tools/languages</li>
<li>open source</li>
<li>no relocation</li>
<li>work on a team of release engineers (not be the only one)</li>
<li>good team dynamics - people happy to share knowledge and like to ship</li>
<li>work in an organization where release engineering is valued for increasing the productivity of the organization as a whole and is funded (hardware/software/services/training) accordingly</li>
<li>support to attend and present at conferences </li>
</ul>
People spend a lot of time at work. Life is too short to be unhappy every day. Writing a list of what is important serves as a checklist to when you are looking at job descriptions and immediately weed out the ones that don't match your list. <br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhrjxWbA1-M-9W2aGGnHczDoJG8zA-4muFIxOi4CfVYAtBG7_89AzknbpV5jBWizlclNOgr_a6Xsw0-YBqyimrmBn4UdaP3Oamw_I0gD7hzQ0hQmeC8NeyBNbwqXFtwYnXWjPAE/s1600/10519774073_febb78a02c_o.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="436" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhrjxWbA1-M-9W2aGGnHczDoJG8zA-4muFIxOi4CfVYAtBG7_89AzknbpV5jBWizlclNOgr_a6Xsw0-YBqyimrmBn4UdaP3Oamw_I0gD7hzQ0hQmeC8NeyBNbwqXFtwYnXWjPAE/s640/10519774073_febb78a02c_o.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Picture by Mufidah Kassalias - Creative Commons Attribution-NonCommercial-NoDerivs 2.0 Generic (CC BY-NC-ND 2.0)<br />
https://www.flickr.com/photos/mufidahkassalias/10519774073/sizes/o/</td></tr>
</tbody></table>
<br />
People tend focus a lot on the technical skills they want to use or new ones you want to learn. You should also think about what kind of culture where you want to work. Do the goals and ethics of the organization align with your own? Who will you be working with? Will you enjoy working with this team? Are you interested in remote work or do you want to work in an office? How will a long commute impact or relocation your quality of life? What is the typical career progression of someone in this role? Are there both management and technical tracks for advancement?<br />
<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEikgJMfshkElhhVNTGU9s4RCi0Dv_zgZ9SE1yJf2KPnemqwFI8BhrqSecEBSSM-1HXexI2k1Pie8G-tD6qOXQ50TiLgJoHWsOVk2-FjLGbcnD-iFbqSMQQKbe-wzb666Y7BJYHr/s1600/4221455156_3b213660bf_o.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="473" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEikgJMfshkElhhVNTGU9s4RCi0Dv_zgZ9SE1yJf2KPnemqwFI8BhrqSecEBSSM-1HXexI2k1Pie8G-tD6qOXQ50TiLgJoHWsOVk2-FjLGbcnD-iFbqSMQQKbe-wzb666Y7BJYHr/s640/4221455156_3b213660bf_o.jpg" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Picture by mugley - Creative Commons Attribution-NonCommercial-NoDerivs 2.0 Generic (CC BY-NC-ND 2.0) https://www.flickr.com/photos/mugley/4221455156/sizes/o/</td></tr>
</tbody></table>
<br />
<br />
To summarize, itemize the skills you'd like to use or learn,
the culture of the company and the team and why you want to work there.<br />
<br />
<b>Cover letter</b><br />
<br />
Your cover letter should succinctly map your existing skills to the role you are applying for and convey enthusiasm and interest. You don't need to have a long story about how you worked on a project at your current job that has no relevance to your potential new employer. Teams that are looking to hire have problems to solve. Your cover letter needs to paint a picture that your have the skills to solve them.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhCgCIyLiVzoOxAtlpwtYWlQ-IJ-jy5XITdtzfGbC3psfhgCZO7HJ2nPwqzkJxXjHjsmaL4z764An9WfG25VLWFO5cOsWiQz0H8Y5LXbFSQwLgjPDnuFFkrkz-aLfjw55QjHUVT/s1600/10320891856_481e4c8d95_b.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhCgCIyLiVzoOxAtlpwtYWlQ-IJ-jy5XITdtzfGbC3psfhgCZO7HJ2nPwqzkJxXjHjsmaL4z764An9WfG25VLWFO5cOsWiQz0H8Y5LXbFSQwLgjPDnuFFkrkz-aLfjw55QjHUVT/s640/10320891856_481e4c8d95_b.jpg" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Picture by Jim Bauer - Creative Commons Attribution-NonCommercial-NoDerivs 2.0 Generic (CC BY-NC-ND 2.0) https://www.flickr.com/photos/lens-cap/10320891856/sizes/l</td></tr>
</tbody></table>
<br />
<br />
<b>Refactoring your resume</b><br />
<br />
Developers have a lot of opportunities these days, but if you intend to move from another industry, into a tech company, it can be more tricky. The important thing is to convey the skills you have in a a way that people can see they can be applied to the problems they want to hire you to fix. <br />
<br />
Many people describe their skills and accomplishments in a way that is too company specific. They may have a list of acronyms and product names on their resume that are unlikely to be known by people outside the company. When describing the work you did in a particular role, describe the work that you did in a that is measurable way that highlights the skills you have. An excellent example of a resume that describes the skills that without going into company specific detail is <a href="http://juliepagano.com/resume/">here</a>. (Julie Pagano also has a terrific post about how she approached <a href="http://juliepagano.com/blog/2015/08/15/job-search-retrospective/">her new job search</a>.)<br />
<br />
Another tip is to leave out general skills that are very common. For instance, if you are a technical writer, omit the fact that you know how to use Windows and Word and focus on highlighting your skills and accomplishments. <br />
<br />
<br />
<b>Non-technical interview preparation</b><br />
<br />
Every job has different technical requirements and there are many books and blog posts on how to prepare for this aspect of the interview process. So I'm going to just cover the non-technical aspects.<br />
<br />
When I interview someone, I like to hear lots of questions. Questions about the work we do and upcoming projects. This indicates that have taken the time to research the team, company and work that we do. It also shows that enthusiasm and interest. <br />
<br />
Here is a list suggestions to prepare for interviews<br />
<br />
<b>1. Research the company make a list of relevant questions</b><br />
Not every company is open about the work that they do, but most will be
have some public information that you can use to formulate questions
during the interviews. Do you know anyone you can have coffee or skype with to who works for the company and can provide insight? What products/services do the company produce? Is the product nearing end of life? If so, what will it be replaced by? What is the companies market share, is it declining, stable or experiencing growth? Who are their main competitors? What are some of the challenges they face going forward? How will this team help address these challenges? <br />
<br />
<b>2. Prepare a list of questions for every person that interviews you ahead of time </b><br />
Many companies will give you the list of names of people who will interview you.<br />
Have they recently given talks? Watch the videos online or read the slides.<br />
Does the team have github or other open repositories? What are recent projects are they working on? Do they have a blog or are active on twitter? If so, read it and formulate some questions to bring to the interview. <br />
Do they use open bug tracking tools? If so, look at the bugs that have recent activity and add them to the list of questions for your interview. <br />
A friend of mine read the book of a person that interviewed him had written and asked questions about the book in the interview. That's serious interview preparation!<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEguGMHycSGGqDajf-I5uySbyX1yxv0SAeLrt9yPkOQtV93MDU_SKyKlQdNI7VSsdDyOqnRkCuhXVqtc5AGCtwr8buXva9BW8IqebBqKIMgex3_XFpuTE3bw0PFVZ5xj70XbtO2M/s1600/22506109386_392e9d5ff2_b.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="426" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEguGMHycSGGqDajf-I5uySbyX1yxv0SAeLrt9yPkOQtV93MDU_SKyKlQdNI7VSsdDyOqnRkCuhXVqtc5AGCtwr8buXva9BW8IqebBqKIMgex3_XFpuTE3bw0PFVZ5xj70XbtO2M/s640/22506109386_392e9d5ff2_b.jpg" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Photo by https://www.flickr.com/photos/wocintechchat/ https://www.flickr.com/photos/wocintechchat/22506109386/sizes/l</td></tr>
</tbody></table>
<br />
<br />
<b>3. Team dynamics and tools</b><br />
Is the team growing or are you hiring to replace somebody who left?<br />
What's the onboarding process like? Will you have a mentor?<br />
How is this group viewed by the rest of the company? You want to be in a role where you can make a valuable contribution. Joining a team where their role is not valued by the company or not funded adequately is a recipe for disappointment.<br />
What does a typical day look like? What hours do people usually work?<br />
What tools do people use? Are there prescribed tools or are you free to use what you'd like?<br />
<br />
<b>4. Diversity and Inclusion</b><br />
If you're a member of an underrepresented group in tech, the numbers are lousy in this industry with some notable exceptions. And I say that while recognizing that I'm personally in the group that is the lowest common denominator for diversity in tech. <br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://twitter.com/radiomorillo/status/589158122108932096" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="350" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj3ZGdMWFOx3bGBADpAyETMkbr4MdK3SiWV31-VaYrDs5v1tsaXbYYG-E3kLBtO3V_y1yLFW00ewfI46u2AqVxqWdUNI5O_Vn-P5o4d2X2oD0F_xYtXc9ols2zJSYDHnjRQE2KO/s640/Screen+Shot+2016-01-08+at+1.55.21+PM.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><a href="https://twitter.com/radiomorillo/status/589158122108932096"><span style="font-size: small;">The entire thread on this tweet is excellent https://twitter.com/radiomorillo/status/589158122108932096</span></a></td></tr>
</tbody></table>
<br />
<br />
I don't really have good advice for this area other than do your research to ensure you're not entering a toxic environment. If you look around the office where you're being interviewed and nobody looks like you, it's time for further investigation. Look at the company's website - is the management team page white guys all the way down? Does the company support diverse conferences, scholarships or internships? Ask on a mailing list like <a href="http://www.devchix.com/">devchix</a> if others have experience working at this company and what it's like for underrepresented groups. If you ask in the interview why there aren't more diverse people in the office and they say something like "well, we only hire on merit" this is a giant red flag. If the answer is along the lines of "yes, we realize this and these are the steps we are taking to rectify this situation", this is a more encouraging response.<br />
<br />
A final piece of advice, ensure that you meet with your manager that you're going to report to as part of your hiring process. You want to ensure that you have rapport with them and can envision a productive working relationship. <br />
<br />
What advice do you have for people preparing to find a new job?<br />
<br />
<b>Further reading</b><br />
<br />
Katherine Daniels gave at really great talk at Beyond the Code 2014 about how to effectively start a new job. <a href="https://vimeo.com/112521974">Press start: Beginning a New Adventure Job </a><br />
She is also the co-author of <a href="http://shop.oreilly.com/product/0636920039846.do">Effective Devops</a> which has fantastic chapter on hiring.<br />
Erica Joy writes amazing articles about the <a href="https://medium.com/@ericajoy">tech industry and diversity</a>.<br />
Cate Huston has some beautiful posts on how to <a href="http://www.catehuston.com/blog/?s=interview">conduct technical interviews</a> and<a href="https://storify.com/catehstn/12-challenging-steps-to-being-a-better-interviewer?utm_content=buffer247bd&utm_medium=social&utm_source=twitter.com&utm_campaign=buffer"> how to be a better interviewer</a><br />
<a href="http://whilefalse.blogspot.ca/">Camille Fournier's blog</a> is excellent reading on career progression and engineering management.<br />
Mozilla <a href="https://careers.mozilla.org/en-US/listings/">is hiring</a>!<br />
Kim Moirhttp://www.blogger.com/profile/14700841495895160750noreply@blogger.com1tag:blogger.com,1999:blog-25023280.post-15129832280047590762015-11-24T07:57:00.002-08:002015-11-24T07:57:44.206-08:00USENIX Release Engineering Summit 2015 recapNovember 13th, I attended the USENIX Release Engineering Summit in Washington, DC. This summit was along side the larger LISA conference at the same venue. Thanks to Dinah McNutt, Gareth Bowles, Chris Cooper, Dan Tehranian and John O'Duinn for organizing.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjhPLoN3K39GjQCYqN36a4etOD7Kj5hR_rHekusztksLtOBJxkVIwkw6vp5Fqzbq6bR-Pu709u1pybSNLyjfUvPYXW9blxDHN7LB7mr9HF4JPSiiL8Z6Fr5ax5py-Ly1IiR7xxB/s1600/FullSizeRender%25282%2529.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjhPLoN3K39GjQCYqN36a4etOD7Kj5hR_rHekusztksLtOBJxkVIwkw6vp5Fqzbq6bR-Pu709u1pybSNLyjfUvPYXW9blxDHN7LB7mr9HF4JPSiiL8Z6Fr5ax5py-Ly1IiR7xxB/s640/FullSizeRender%25282%2529.jpg" width="488" /></a></div>
<br />
I gave two talks at the summit. One was a long talk on how we have scaled our Android testing infrastructure on AWS, as well as a look back at how it evolved over the years.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjnk2szhFrLPlpZyBWaoo9eTdf8NKUpLMg1Cq-BxdySazPv1S_OhlxxztNd8vvHRpHG0rAcmdo7ZQB-UDZ-LLJ8d8To4vL6X50XmYI4a_ULtt-E3yVuhT6P9p_37RkgLT4mZxDR/s1600/2600844073_025235d192_o.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="456" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjnk2szhFrLPlpZyBWaoo9eTdf8NKUpLMg1Cq-BxdySazPv1S_OhlxxztNd8vvHRpHG0rAcmdo7ZQB-UDZ-LLJ8d8To4vL6X50XmYI4a_ULtt-E3yVuhT6P9p_37RkgLT4mZxDR/s640/2600844073_025235d192_o.jpg" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><div style="text-align: left;">
Picture by Tim Norris - Creative Commons Attribution-NonCommercial-NoDerivs 2.0 Generic (CC BY-NC-ND 2.0)<br />
https://www.flickr.com/photos/tim_norris/2600844073/sizes/o/</div>
</td></tr>
</tbody></table>
<iframe allowfullscreen="" frameborder="0" height="485" marginheight="0" marginwidth="0" scrolling="no" src="//www.slideshare.net/slideshow/embed_code/key/d9mw2dMdE9oePF" style="border-width: 1px; border: 1px solid #CCC; margin-bottom: 5px; max-width: 100%;" width="595"> </iframe> <br />
<div style="margin-bottom: 5px;">
<b> <a href="https://www.slideshare.net/k2moir/scaling-mobile-testing-on-aws-emulators-all-the-way-down" target="_blank" title="Scaling mobile testing on AWS: Emulators all the way down">Scaling mobile testing on AWS: Emulators all the way down</a> </b> from <b><a href="https://www.slideshare.net/k2moir" target="_blank">Kim Moir</a></b><br />
<br />
I gave a second lightning talk in the afternoon on the problems we face with our large distributed continuous integration, build and release pipeline, and how we are working to address the issues. The theme of this talk was that managing a large distributed system is like being the caretaker for the water, or some days, the sewer system for a city. We are constantly looking system leaks and implementing system monitoring. And probably will have to replace it with something new while keeping the existing one running.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg4-SjHsfU0H7Jh_4Aov_6egbhaSBBpqw9_i2-Qh_ZHjdbQ_abDh5hVMWfaGFpl04InXZmcNq_O9iNvwa-2PQqjaaE-OLgoUvoh7uGGgdefkJXEwutHWVL_C2Gz9dK9kLPyJpAS/s1600/14107877324_2a49534976_b.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="310" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg4-SjHsfU0H7Jh_4Aov_6egbhaSBBpqw9_i2-Qh_ZHjdbQ_abDh5hVMWfaGFpl04InXZmcNq_O9iNvwa-2PQqjaaE-OLgoUvoh7uGGgdefkJXEwutHWVL_C2Gz9dK9kLPyJpAS/s640/14107877324_2a49534976_b.jpg" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Picture by Korona Lacasse - Creative Commons 2.0 Attribution 2.0 Generic https://www.flickr.com/photos/korona4reel/14107877324/sizes/l</td></tr>
</tbody></table>
<div class="separator" style="clear: both; text-align: center;">
</div>
<br /></div>
<iframe allowfullscreen="" frameborder="0" height="485" marginheight="0" marginwidth="0" scrolling="no" src="//www.slideshare.net/slideshow/embed_code/key/1m5VpdLt2xcpkn" style="border-width: 1px; border: 1px solid #CCC; margin-bottom: 5px; max-width: 100%;" width="595"> </iframe> <br />
<div style="margin-bottom: 5px;">
<b> <a href="https://www.slideshare.net/k2moir/distributed-systems-at-scale-reducing-the-fail" target="_blank" title="Distributed Systems at Scale: Reducing the Fail">Distributed Systems at Scale: Reducing the Fail</a> </b> from <b><a href="https://www.slideshare.net/k2moir" target="_blank">Kim Moir</a></b> </div>
<br />
In preparation for this talk, I did a lot of reading on complex systems
design and designing for recovery from failure in distributed systems. In particular, I read <a href="https://en.wikipedia.org/wiki/Donella_Meadows">Donatella Meadows</a>' book <a href="https://www.chapters.indigo.ca/en-ca/books/thinking-in-systems-a-primer/9781603580557-item.html?ikwid=thinking+in+systems&ikwsec=Home&ikwidx=0">Thinking in Systems</a>. (Cate Huston reviewed the book <a href="http://www.catehuston.com/blog/2015/04/10/book-thinking-in-systems/">here</a>).
I also watched several talks by people who talked about the challenges
they face managing their distributed systems including the following: <br />
<ul>
<li>All Your Base 2014: Laura Thomson, Director of Engineering, Cloud Services Engineering and Operations, Mozilla – <a href="https://vimeo.com/album/3108317/video/110088288">Many moving parts: monitoring complex systems</a></li>
<li>Velocity Santa Clara 2015: Astrid Atkinson, Director Software Engineering, Google - <a href="https://www.youtube.com/watch?v=p0jGmgIrf_M">Engineering for the long game: Managing complexity in distributed systems</a></li>
<li>Mountain West Ruby Conference, Paul Hinze, Hashicorp - <a href="https://speakerdeck.com/phinze/smoke-and-mirrors-the-primitives-of-high-availability">Primitives of High Availability </a></li>
<li>Strange Loop 2015, Camille Fournier - <a href="https://www.youtube.com/watch?v=TlU1opuCXB0">Hopelessness and Confidence in Distributed Systems Design </a></li>
</ul>
I'd also like to thank all the members of Mozilla releng/ateam who reviewed my slides and provided feedback before I gave the presentations.<br />
<ul>
</ul>
The attendees of the summit attended the same keynote as the LISA attendees. Jez Humble, well known for his <a href="https://www.chapters.indigo.ca/en-ca/books/continuous-delivery-reliable-software-releases/9780321601919-item.html?ikwid=jez+humble&ikwsec=Home&ikwidx=0">Continuous Delivery </a>and <a href="https://www.chapters.indigo.ca/en-ca/books/lean-enterprise-how-high-performance/9781449368425-item.html?ikwid=jez+humble&ikwsec=Home&ikwidx=0">Lean Enterprise</a> books provided a keynote on Lean Configuration Management which I really enjoyed. (Older version of slides from another conference, are available <a href="http://www.slideshare.net/jezhumble/lean-configuration-management">here</a> and <a href="http://www.slideshare.net/jezhumble/architecting-for-continuous-delivery-54192503">here</a>.)<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjZOniqFnOlGk5drqLQakgKFxtb8E7zZhiv1VaQrHY_ficJ90cT-wA49DCBL27R7iENyfgBzYWvXf2S-jw8PbjmjhD5jksEGT8dOjSdaCn_oSeU6BFDrjjW7gJRxhwZlNzjK-ii/s1600/CTsy51XWwAA8FcV.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjZOniqFnOlGk5drqLQakgKFxtb8E7zZhiv1VaQrHY_ficJ90cT-wA49DCBL27R7iENyfgBzYWvXf2S-jw8PbjmjhD5jksEGT8dOjSdaCn_oSeU6BFDrjjW7gJRxhwZlNzjK-ii/s640/CTsy51XWwAA8FcV.jpg" width="640" /></a></div>
<br />
<br />
In particular, I enjoyed his discussion of the cultural aspects of devops. I especially like that he stated that "You should not have to have planned downtime or people working outside business hours to release". He also talked a bit about how many of the leaders that are looked up to as visionaries in the tech industry are known for not treating people very well and this is not a good example to set for others who believe this to be the key to their success. For instance, he said something like "what more could Steve Jobs have accomplished had he treated his employees less harshly".<br />
<br />
Another concept he discussed which I found interesting was that of the strangler application. When moving from a large monolithic application, the goal is to split out the existing functionality into services until the originally application is left with nothing. Exactly what Mozilla releng is doing as we migrate from Buildbot to <a href="http://docs.taskcluster.net/">taskcluster</a>.<br />
<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh7USMbtwyOgXx9unpois9yrLN8k_DxE1LXA_EUacuKdCycmIhWW9OEx6MHbzK-TeXp0Tb1g1aZlQjpR-POgXLLENrl-3qyVm0gsewnnp1dEYnBu2a5Hvxgm7X2d-wr0f-HWyhx/s1600/Screen+Shot+2015-11-18+at+2.34.33+PM.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="433" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh7USMbtwyOgXx9unpois9yrLN8k_DxE1LXA_EUacuKdCycmIhWW9OEx6MHbzK-TeXp0Tb1g1aZlQjpR-POgXLLENrl-3qyVm0gsewnnp1dEYnBu2a5Hvxgm7X2d-wr0f-HWyhx/s640/Screen+Shot+2015-11-18+at+2.34.33+PM.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">http://www.slideshare.net/jezhumble/architecting-for-continuous-delivery-54192503</td><td class="tr-caption" style="text-align: center;"></td><td class="tr-caption" style="text-align: center;"></td><td class="tr-caption" style="text-align: center;"></td></tr>
</tbody></table>
<br />
<br />
At the release engineering summit itself, Lukas Blakk from Pinterest gave a fantastic talk <i><b>Stop Releasing off Your Laptop—Implementing a Mobile App Release Management Process from Scratch in a Startup or Small Company.</b></i> This included grumpy cat picture to depict how Lukas thought the rest of the company felt when that a more structured release process was implemented.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhDUZSQ5M9c-HPeBLs7j4yeaR7WGuhkSDvf82vRBxW2MgAXwOb2GvKrNkqwh_CYyUOnzu7e6qdWcymDk3HoQ6MR-bSDxXMqoDCP5AuHtKI1Cjy8VtZskrBQPS07rtq4mS2cSlpK/s1600/IMG_8237.JPG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhDUZSQ5M9c-HPeBLs7j4yeaR7WGuhkSDvf82vRBxW2MgAXwOb2GvKrNkqwh_CYyUOnzu7e6qdWcymDk3HoQ6MR-bSDxXMqoDCP5AuHtKI1Cjy8VtZskrBQPS07rtq4mS2cSlpK/s640/IMG_8237.JPG" width="640" /></a></div>
<br />
Lukas also included a timeline of the tasks that implemented in her first six months working at Pinterest. Very impressive to see the transition!<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgxsz_kSOo9nL3VFhv09xvojLEYs1M7Rd0i0vOVsxWPc6Pc2JEwo0_yDlLcig7Ok_temy3xL_c4qLdoH22zkW_efUGqlDbEnC0v2vVsMFdtUL1Qz2cZjcjHAkZEQAkVgbtscByJ/s1600/IMG_8238.JPG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgxsz_kSOo9nL3VFhv09xvojLEYs1M7Rd0i0vOVsxWPc6Pc2JEwo0_yDlLcig7Ok_temy3xL_c4qLdoH22zkW_efUGqlDbEnC0v2vVsMFdtUL1Qz2cZjcjHAkZEQAkVgbtscByJ/s640/IMG_8238.JPG" width="640" /></a></div>
<br />
Another talk I enjoyed was <i><b>Chaos Patterns - Architecting for Failure in Distributed Systems
</b></i>by Jos Boumans of Krux. (Similar slides from an earlier conference <a href="http://www.slideshare.net/jiboumans/chaos-patterns-architecting-for-failure-in-distributed-systems?qid=6110514c-d128-4fef-9ccb-45a1dd03c05b&v=qf1&b=&from_search=1">here).</a> He talked about some high profile distributed systems that failed and how chaos engineering can help illuminate these issues before they hit you in production.
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjJoZIV4ODQQ2bLPHezNrX0ttVVODjhER2aJpq6UBWAjg0PUMAsJsGmNdhwnTRW6an7GwI3lL3FYoMIteAh__xpRhCoDCi2oPTyAfjKlJOZTzSo4AzBZONHg4OD-X7XcM9G-uHf/s1600/IMG_8236.JPG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjJoZIV4ODQQ2bLPHezNrX0ttVVODjhER2aJpq6UBWAjg0PUMAsJsGmNdhwnTRW6an7GwI3lL3FYoMIteAh__xpRhCoDCi2oPTyAfjKlJOZTzSo4AzBZONHg4OD-X7XcM9G-uHf/s640/IMG_8236.JPG" width="640" /></a></div>
<br />
For instance, it is impossible for Netflix to model their entire system outside of production given that they consume around one third of nightly downstream bandwidth consumption in the US. <br />
<br />
Evan Willey and Dave Liebreich from Pivotal Cloud Foundry gave a talk entitled "<b>Pivotal Cloud Foundry Release Engineering: Moving Integration Upstream Where It Belongs</b>". I found this talk interesting because they talked about how the built <a href="http://concourse.ci/">Concourse</a>, a CI system that is more scaleable and natively builds pipelines. Travis and Jenkins are good for small projects but they simply don't scale for large numbers of commits, platforms to test or complicated pipelines. We followed a similar path that led us to develop <a href="http://docs.taskcluster.net/">Taskcluster</a>. <br />
<br />
There were many more great talks, hopefully more <a href="https://www.usenix.org/conference/ures15/summit-program">slides</a> will be up soon!<br />
<br />Kim Moirhttp://www.blogger.com/profile/14700841495895160750noreply@blogger.com0tag:blogger.com,1999:blog-25023280.post-9492333208549893722015-09-25T12:47:00.000-07:002015-09-25T17:34:22.553-07:00The mystery of high pending countsIn September, Mozilla release engineering started experiencing high pending counts on our test pools, notably Windows, but also Linux (and consequently Android). High pending counts mean that there are thousands of jobs queued to run on the machines that are busy running other jobs. The time developers have to wait for their test results is longer than ideal.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgEjEkyZmAc5gfKqX6azJUHR27ug3UaUb4fOZmH-nUA69x3H-N-WtZxOECDzeWI4-t6jpW7jIa31z1ExbjNv7ZkjA6dWPLVqpP7SyBppV3U60-P2QUexJX3fvdfhOCFgisergzI/s1600/Screen+Shot+2015-09-25+at+2.48.06+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="251" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgEjEkyZmAc5gfKqX6azJUHR27ug3UaUb4fOZmH-nUA69x3H-N-WtZxOECDzeWI4-t6jpW7jIa31z1ExbjNv7ZkjA6dWPLVqpP7SyBppV3U60-P2QUexJX3fvdfhOCFgisergzI/s640/Screen+Shot+2015-09-25+at+2.48.06+PM.png" width="640" /></a></div>
<br />
Usually, pending counts clear overnight as less code is pushed during the night (in North America) which invokes fewer builds and tests. However, as you can see from the graph above, the Windows test pending counts were flat last night. They did not clear up overnight. You will also note that try, which usually comprises 63% of our load, has very highest pending counts compared to other branches. This is because many people land on try before pushing to other branches, and tests aren't coalesced on try.<br />
<br />
<br />
The work to determine the cause of high pending counts is always an interesting mystery.<br />
<ul>
<li><div class="" id="magicdomid27">
<span class="author-g-qklxbfb5dwnyy0ck">Are end to end times for tests increasing?</span><span class="author-g-qklxbfb5dwnyy0ck"> </span></div>
</li>
<li><span class="author-g-qklxbfb5dwnyy0ck">Have more tests been enabled recently?</span></li>
<li><span class="author-g-qklxbfb5dwnyy0ck">Are retries increasing? (Tests the run multiple times because the initial runs fail due to infrastructure issues)</span></li>
<li><span class="author-g-qklxbfb5dwnyy0ck">Are jobs that are coalesced being backfilled and consuming capacity? </span></li>
<li>Are tests being chunked into smaller jobs that increase end to end time due to the added start up time? <span class="author-g-qklxbfb5dwnyy0ck"><br /></span></li>
</ul>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEie_OG5C5_GLCD_J9ShuIH6AmkpS7R89XOQnCPFzFVkOmZS3Zqh4K6c5uF1eJZ46BBg0x9NJqgZ8lS8wL_qU9IH5z0up7DFhREna0wx-t8wlLQaaxArPYA48AL5twyycXyGs7NE/s1600/6883500405_cbc44561e9_o.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="379" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEie_OG5C5_GLCD_J9ShuIH6AmkpS7R89XOQnCPFzFVkOmZS3Zqh4K6c5uF1eJZ46BBg0x9NJqgZ8lS8wL_qU9IH5z0up7DFhREna0wx-t8wlLQaaxArPYA48AL5twyycXyGs7NE/s640/6883500405_cbc44561e9_o.jpg" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Mystery by <a href="https://flic.kr/p/bugJRt">©Stuart Richards</a>, Creative Commons by-nc-sa 2.0</td></tr>
</tbody></table>
<br />
<a href="https://elvis314.wordpress.com/">Joel Maher </a>and I looked at the data for this last week and discovered what we believe to be the source of the problem. We <span class="author-p-3131">have determined that since the end of August a
number of new test jobs were enabled that increased the compute time per push on
Windows by 13% or 2.5 hours per push. Most of these new test jobs are for
<a href="http://benjamin.smedbergs.us/weekly-updates.fcgi/project/electrolysis">e10s</a>. </span><br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjftUcrFqinPTecKZPCm-mshcSRyDsFpmjanNkOB8ightb62DRUx-fH4xHg-_22JHdiPZYdtmEAzLds8hLdUaD9YJhszT1AV1zmjKiZIKsaUic8GO45FFszPAuyqs6olSyaxffq/s1600/Screen+Shot+2015-09-25+at+3.14.45+PM.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="328" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjftUcrFqinPTecKZPCm-mshcSRyDsFpmjanNkOB8ightb62DRUx-fH4xHg-_22JHdiPZYdtmEAzLds8hLdUaD9YJhszT1AV1zmjKiZIKsaUic8GO45FFszPAuyqs6olSyaxffq/s640/Screen+Shot+2015-09-25+at+3.14.45+PM.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: left;">Increase in seconds that new jobs added to the total compute time per push. (Some existing jobs also reduced their compute time for a total difference about about 2.5 more hours per push on Windows)</td></tr>
</tbody></table>
<span class="author-p-3131">The <a href="https://wiki.mozilla.org/Electrolysis">e10s initiative is an important initiative for Mozilla</a> to make Firefox performance and security even better. However, since new e10s and old tests will continue to run in parallel, we need to get creative on how to have acceptable wait times given the limitations of our current Windows tests pools. (All of our Windows test run on bare metal in our
datacentre, not on Amazon). </span><br />
<br />
Release engineering is working to reduce this pending counts given our current hardware constraints with the following initiatives:<b><span class="author-p-3131"> </span></b><br />
<br />
<b><span class="author-p-3131">To reduce Linux pending counts:</span></b><br />
<div class="" id="magicdomid4">
<ul class="list-bullet1">
<li><span class="author-p-3131">Added 200 new instances to the tst-emulator64 pool (run Android test jobs on Linux emulators) (</span><a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1204756">bug 1204756</a>)</li>
<li><span class="author-p-3131">In process of adding more Linux32 and Linux64 buildbot masters (</span><a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1205409">bug 1205409</a><span class="author-p-3131">) which will allow us to expand our capacity more</span></li>
</ul>
</div>
<br />
<b><span class="author-p-3131">Ongoing work to reduce the Windows pending counts:</span></b><br />
<br />
<ul>
<li><span class="author-p-3131">Disable Linux32 Talos tests and redeploy these machines as Windows test machines (</span><a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1204920" rel="noreferrer" target="_blank">bug 1204920</a><span class="author-p-3131"> and <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1208449">bug 1208449</a>)</span></li>
<li><span class="author-p-3131">Reduce the number of talos jobs by running SETA on talos (<a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1192994">bug 1192994</a>)</span></li>
<li><span class="author-p-3131">Developer productivity team is investigating whether non-operating specific tests that run on multiple windows test platforms can run on fewer platforms.</span></li>
</ul>
<br />
<b><span class="author-p-3131">How can you help? </span></b><br />
<br />
<span class="author-p-3131">Please be considerate when invoking try pushes and
only select the platforms that you explicitly require to test. Each
try push for all platforms and all tests invokes over 800 jobs.</span>Kim Moirhttp://www.blogger.com/profile/14700841495895160750noreply@blogger.com0tag:blogger.com,1999:blog-25023280.post-59711598435897393512015-07-17T08:41:00.000-07:002015-07-17T08:41:29.683-07:00Learning Data Science and evidence based teaching methodsThis spring, I took several online courses on the topic of data science. I became interested in expanding my skills in this area because as release engineers, we <a href="http://relengofthenerds.blogspot.ca/2015/04/less-testing-same-great-firefox-taste.html">deal with a lot of data</a>. I wanted to learn new tools to extract useful information from the <a href="http://www.chesnok.com/daily/2014/05/02/release-engineering-a-draft-of-an-architecture-diagram/">distributed systems behemoth we manage</a>. <br />
<br />
<br />
This xckd reminded me of the challenges of managing our buildfarm somedays :-)<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://imgs.xkcd.com/comics/tamagotchi_hive.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="http://imgs.xkcd.com/comics/tamagotchi_hive.png" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">From http://xkcd.com/1546/</td></tr>
</tbody></table>
<br />
I took three courses from <a href="https://www.coursera.org/specialization/jhudatascience/1?utm_medium=catalog">Coursera's Data Science track from John Hopkins University</a>. As with previous coursera classes I took, all the course material is online (lecture videos and notes). There are quizzes and assignments that are due each week. Each course below was about four weeks long.<br />
<br />
<a href="https://www.coursera.org/course/datascitoolbox">The Data Scientist's Toolbox </a>- This course was pretty easy. Basically a introduction to the questions that data scientists deal as well a primer on installing R, RStudio (IDE for R), and using GitHub.<br />
<a href="https://www.coursera.org/course/rprog">R Programming</a> - Introduction to R. Most of the quizzes and examples used publicly available data for the programming exercises. I found I had to do a lot of reading in the R API docs or on stackoverflow to finish the assignments. The lectures didn't provide a lot of the material needed to complete the assignments. Lots of techniques to learn how to subset data using R which I found quite interesting, reminded me a lot of querying databases with SQL to conduct analysis.<br />
<a href="https://www.coursera.org/course/getdata">Getting and Cleaning Data</a> - More advanced techniques using R. Using publicly available data sources to clean different data sources in different formats, XML, excel spreadsheets, comma or tab delimited. Given this data, we had to answer many questions and conduct specific analysis by writing R programs. The assignments were pretty challenging and took a long time. Again, the course material didn't really cover all the material you needed to do the assignments so a lot of additional reading was required.<br />
<br />
There are six more courses in the Data Science track that I'll start tackling again in the fall that cover subjects such as reproducible research, statistical inference and machine learning. My next coursera class is <a href="https://www.coursera.org/course/introse">Introduction to Systems Engineering</a> which I'll start in a couple of weeks. I've really become interested in learning more about this subject after reading <a href="http://www.catehuston.com/blog/2015/04/10/book-thinking-in-systems/">Thinking in Systems</a>.<br />
<br />
The other course I took this spring was the <a href="https://swcarpentry.github.io/training-course/">Software Carpentry Instructor</a> training course. The Software Carpentry Foundation teachers researchers basic software skills. For instance, if you are a biologist analyzing large data sets it would be useful to learn how to use R, Python, and version control to store the code you wrote to share with others. These are not skills that many scientists acquire in their formal university training, and learning them allows them to work more productively. The instructor course was excellent, thanks Greg Wilson for your work teaching us.<br />
<br />
We read two books for this course:<br />
<a href="http://www.amazon.ca/Building-Better-Teacher-Teaching-Everyone-ebook/dp/B00FPT5MSQ">Building a Better Teacher</a>: An interesting overview of how teacher is taught in different countries and how to make it more effective. Most important: Have more opportunities for other teachers to observe your classroom and provide feedback which I found analogous to how code review makes us better software developers.<br />
<a href="http://www.amazon.ca/How-Learning-Works-Research-Based-Principles/dp/0470484101">How Learning Works: Seven Research-Based Principles for Smart Teaching</a>: A book summarizing the research in disciplines such as education, cognitive science and psychology on the effective techniques for teaching students new material. How assessing student's prior knowledge can help you better design your lessons, how to to ask questions to determine what material students are failing to grasp, how to understand student's motivation for learning and more. Really interesting research.<br />
<br />
For the instructor course, we met every couple of weeks online where Greg would conduct a short discussion on some of the topics on a conference call and we would discuss via etherpad interactively. We would then meet in smaller groups later in the week to conduct practice teaching exercises. We also submitted example lessons to the course repo on GitHub. The final project for the course was to conduct a short lesson to a group of instructors that gave feedback, and submit a pull request to update an existing lesson with a fix. Then we are ready to sign up to teach a Software Carpentry course!<br />
<br />
In conclusion, data science is a great skill to have if you are managing large distributed systems. Also, using evidence based teaching methods to help others learn is the way to go!<br />
<br />
Other fun data science examples include<br />
<a href="http://techblog.netflix.com/2015/07/tracking-down-villains-outlier.html">Tracking down the Villains: Outlier Detection at Netflix</a> - detecting rogue servers with machine learning<br />
<a href="https://vimeo.com/112524564">Finding Shoe Stores in 100k Merchants: Using Data to Group All Things</a> - finding out what Shopify merchants sell shoes using Apache Spark and more<br />
<a href="https://codeascraft.com/speakers/kamelia-aryafar-and-jerry-soung-looking-at-etsy-through-camera-lenses-the-application-of-computer-vision-at-etsy/">Looking Through Camera Lenses: The Application of Computer Vision at Etsy</a><br />
<br />Kim Moirhttp://www.blogger.com/profile/14700841495895160750noreply@blogger.com0tag:blogger.com,1999:blog-25023280.post-45776630817872295022015-06-16T13:52:00.000-07:002015-06-16T13:55:24.201-07:00Test job reduction by the numbersIn an <a href="http://relengofthenerds.blogspot.ca/2015/04/less-testing-same-great-firefox-taste.html">earlier post</a>, I wrote how we had reduced the amount of test jobs that run on two branches to allow us to scale our infrastructure more effectively. We run the tests that historically identify regressions more often. The ones that don't, we skip on every Nth push. We now have data on how this reduced the number of jobs we run since we began implementation in April.<br />
<br />
We run SETA on two branches (<a href="https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound">mozilla-inbound</a> and <a href="https://treeherder.mozilla.org/#/jobs?repo=fx-team">fx-team</a>) and on 18 types of builds. Collectively, these two branches represent about 20% of pushes each month. Implementing SETA allowed us to move from ~400 -> ~240 jobs per push on these two branches<sup>1</sup> We run the tests identified as not reporting regressions on every 10th commit or 90 minutes since the last test was scheduled. We run the critical tests on every commit.<sup>2</sup><br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://people.mozilla.org/~kmoir/reports/SETA/TrackingSETAmozilla-inbound.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="http://people.mozilla.org/~kmoir/reports/SETA/TrackingSETAmozilla-inbound.png" height="396" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Reduction in number of jobs per push on mozilla-inbound as SETA scheduling is rolled out</td></tr>
</tbody></table>
<br />
A graph for <a href="http://people.mozilla.org/~kmoir/reports/SETA/TrackingSETAfx-team.png">the fx-team branch shows a similar trend</a>. It was a staged rollout starting in early April, as I enabled platforms and as the SETA data became available. The dip in early June reflects where I enabled SETA for Android 4.3.<br />
<br />
This data will continue to be updated in our scheduling configuration as it evolves and is updated by the code that <a href="http://alertmanager.allizom.org/seta.html">Joel and Vaibhav wrote to analyze regressions</a>. The analysis identifies that there were<br />
<br />
Jobs to ignore: 440<br />
Jobs to run: 114<br />
Total number of jobs: 554<br />
<br />
which is significant. Our buildbot configurations are updated the latest SETA data with every <a href="https://wiki.mozilla.org/ReleaseEngineering/Buildduty/Reconfigs">reconfig</a>, which occurs usually occurs every couple of days.<br />
<br />
The platforms configured to run fewer tests for both opt and debug are<br />
<br />
MacOSX (10.6, 10.10)<br />
Windows (XP, 7, 8)<br />
Ubuntu 12.04 for linux32, linux64 and ASAN x64<br />
Android 2.3 armv7 API 9<br />
Android 4.3 armv7 API 11+ <br />
<br />
<b>Additional info</b><br />
<sup>1</sup>Tests may have been disabled/added at the same time, this is not taken into account<br />
<sup>2</sup>There still some scheduling issues to be fixed see <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1174870">bug 1174870</a> and <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1174746">bug 1174746</a> for further detailsKim Moirhttp://www.blogger.com/profile/14700841495895160750noreply@blogger.com0tag:blogger.com,1999:blog-25023280.post-1721799093090273242015-06-12T12:28:00.001-07:002015-06-12T12:28:31.685-07:00Mozilla pushes - May 2015Here's May 2015's monthly analysis of the pushes to our Mozilla development trees. You can load the data as an <a href="http://people.mozilla.org/~kmoir/reports/2015-05%20pushes%20report.html">HTML page</a> or as a <a href="http://people.mozilla.org/~kmoir/reports/201505.json">json file</a>. <br />
<br /><br /><b>Trends</b><br /><br />The number of pushes decreased from those recorded in the previous month (8894) with a total of 8363.<b> </b><br />
<br />
<b> </b><br />
<b>Highlights</b><br />
<ul>
<li>8363 pushes</li>
<li>270 pushes/day (average)</li>
<li>Highest number of pushes/day: 445 pushes on May 21, 2015</li>
<li>16.03 pushes/hour (highest average)</li>
</ul>
<br /><b>General Remarks</b><br />
<ul>
<li>Try has around 62% of all the pushes now</li>
<li>The three integration repositories (fx-team, mozilla-inbound and b2g-inbound) account around 27% of all the pushes.</li>
</ul>
<br /><b>Records</b><br />
<ul>
<li>August 2014 was the month with most pushes (13090 pushes)</li>
<li>August 2014 had the highest pushes/day average with 422 pushes/day</li>
<li>July 2014 had the highest average of "pushes-per-hour" with 23.51 pushes/hour</li>
<li>October 8, 2014 had the highest number of pushes in one day with 715 pushes </li>
</ul>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjuwWXg2VHjOUBqhZUC7MoUNSdZhSWbMmmKpvLOogznBXZiG19YRJHdWGYI8LvosHo5MNp2wfdVNrAe4sJH1ZOd_7YqoWoN0nmoqqfLxWhtU3-ac7g6WMUKj_T7_payQfWezidQ/s1600/Screen+Shot+2015-06-12+at+3.22.37+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="260" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjuwWXg2VHjOUBqhZUC7MoUNSdZhSWbMmmKpvLOogznBXZiG19YRJHdWGYI8LvosHo5MNp2wfdVNrAe4sJH1ZOd_7YqoWoN0nmoqqfLxWhtU3-ac7g6WMUKj_T7_payQfWezidQ/s640/Screen+Shot+2015-06-12+at+3.22.37+PM.png" width="640" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiwmp89esnG9J8oN5MB9m8_NTtQLIDaIQEJUATIp1PZi2qeXk-5y4Mo1aqILnliZE0rwZMd5cW0AooIZEvoZcAsv4PRlGf8nhBqDyjQZMwPnEUo3d4jap9MBVDa115eAV1JxjYo/s1600/Screen+Shot+2015-06-12+at+3.22.46+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="246" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiwmp89esnG9J8oN5MB9m8_NTtQLIDaIQEJUATIp1PZi2qeXk-5y4Mo1aqILnliZE0rwZMd5cW0AooIZEvoZcAsv4PRlGf8nhBqDyjQZMwPnEUo3d4jap9MBVDa115eAV1JxjYo/s640/Screen+Shot+2015-06-12+at+3.22.46+PM.png" width="640" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgIe5M6o7T0fAdeJbJNGIz7flzBA3mmm9TJK_XUEPhKa2u86BeQEjSvTE52ztCahRF5O32xqWtQghBgQ-Z5u3hYdH8ppeRVkHxH3_EW05pBYf-0aMST52nHVvRsmqvrvxsrWVxB/s1600/Screen+Shot+2015-06-12+at+3.22.55+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="328" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgIe5M6o7T0fAdeJbJNGIz7flzBA3mmm9TJK_XUEPhKa2u86BeQEjSvTE52ztCahRF5O32xqWtQghBgQ-Z5u3hYdH8ppeRVkHxH3_EW05pBYf-0aMST52nHVvRsmqvrvxsrWVxB/s640/Screen+Shot+2015-06-12+at+3.22.55+PM.png" width="640" /></a></div>
<br />Kim Moirhttp://www.blogger.com/profile/14700841495895160750noreply@blogger.com2tag:blogger.com,1999:blog-25023280.post-86559127053901847742015-05-01T09:44:00.000-07:002015-06-12T12:06:42.660-07:00Mozilla pushes - April 2015Here's April 2015's monthly analysis of the pushes to our Mozilla development trees. You can load the data as an <a href="http://people.mozilla.org/~kmoir/reports/2015-04%20pushes%20report.html">HTML page</a> or as a <a href="http://people.mozilla.org/~kmoir/reports/201504.json">jso<span id="goog_1160394292"></span><span id="goog_1160394293"></span>n file</a>.
<b> </b><br />
<br />
<br />
<b>Trends</b><br />
The number of pushes decreased from those recorded in the previous month with a total of 8894. This is due to the fact that gaia-try is managed by <a href="http://docs.taskcluster.net/">taskcluster</a> and thus these jobs don't appear in the buildbot scheduling databases anymore which this report tracks.<br />
<br />
<br />
<b>Highlights</b><br />
<br />
<ul>
<li>8894 pushes</li>
<li>296 pushes/day (average)</li>
<li>Highest number of pushes/day: 528 pushes on Apr 1, 2015</li>
<li>17.87 pushes/hour (highest average)</li>
</ul>
<br />
<b>General Remarks</b><br />
<br />
<ul>
<li>Try has around 58% of all the pushes now that we no longer track gaia-try</li>
<li>The three integration repositories (fx-team, mozilla-inbound and b2g-inbound) account around 28% of all the pushes.</li>
</ul>
<br />
<b>Records</b><br />
<br />
<ul>
<li>August 2014 was the month with most pushes (13090 pushes)</li>
<li>August 2014 had the highest pushes/day average with 422 pushes/day</li>
<li>July 2014 had the highest average of "pushes-per-hour" with 23.51 pushes/hour</li>
<li>October 8, 2014 had the highest number of pushes in one day with 715 pushes </li>
</ul>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhld548O02ClwrLb5OkZlE0E38F15V2m902Wpbu7rKZ-NKec413Tt7_ZtL1iqeDQDmyzW4HIdhOHJheRF4FJKlpxOAg32Iii_bW-MAHhn6sGF0sf1e_Awds1QnKLDC38k9MMY45/s1600/Screen+Shot+2015-05-01+at+11.59.06+AM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="230" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhld548O02ClwrLb5OkZlE0E38F15V2m902Wpbu7rKZ-NKec413Tt7_ZtL1iqeDQDmyzW4HIdhOHJheRF4FJKlpxOAg32Iii_bW-MAHhn6sGF0sf1e_Awds1QnKLDC38k9MMY45/s1600/Screen+Shot+2015-05-01+at+11.59.06+AM.png" width="640" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh-Dpx8HkA3N8mEv4gQMyWjGUbf6f71z-9h7B-dkba4emYDde_B_iMgMqEh9Glt4mEhuNEk5ukxhHGvZysTiAnVpEPLyxJ1on3PtiRH0EzFCLXo3A-Jgwf1WWkTj4yptTP2Rf0m/s1600/Screen+Shot+2015-05-01+at+11.59.17+AM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="238" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh-Dpx8HkA3N8mEv4gQMyWjGUbf6f71z-9h7B-dkba4emYDde_B_iMgMqEh9Glt4mEhuNEk5ukxhHGvZysTiAnVpEPLyxJ1on3PtiRH0EzFCLXo3A-Jgwf1WWkTj4yptTP2Rf0m/s1600/Screen+Shot+2015-05-01+at+11.59.17+AM.png" width="640" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiCfQz8nxAb9HLdUEcmWy80aKyrS8saqrivijWaFdnTFoSWlis_LoSJEQCt3sL2rPlycxOOMFV0h5Mrrp5FHmwwwBzfztIoIYJ5xBXthPgOCJCyi2A_ikfpl5lCPtVxuUT-uWC8/s1600/Screen+Shot+2015-05-01+at+12.00.19+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="308" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiCfQz8nxAb9HLdUEcmWy80aKyrS8saqrivijWaFdnTFoSWlis_LoSJEQCt3sL2rPlycxOOMFV0h5Mrrp5FHmwwwBzfztIoIYJ5xBXthPgOCJCyi2A_ikfpl5lCPtVxuUT-uWC8/s1600/Screen+Shot+2015-05-01+at+12.00.19+PM.png" width="640" /></a></div>
<b>Note </b><br />
I've changed the graphs to only track 2015 data.
Last month they were tracking 2014 data as well but it looked
crowded so I updated them. Here's a graph showing the number of pushes over the last few years for comparison.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjOfGuIeKf7PP-x-FnIqpSUVPo_bGAXuJDGAehfwS1c09AsjBKS9RRKT5UItBjVG9LGElMGnwYhJI6wiEBw-iWdRvw2pByJdj9i0uakfPwWqa5BvjzJ2pehkDvMELNYIGUxxJaJ/s1600/Screen+Shot+2015-05-01+at+12.32.20+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="256" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjOfGuIeKf7PP-x-FnIqpSUVPo_bGAXuJDGAehfwS1c09AsjBKS9RRKT5UItBjVG9LGElMGnwYhJI6wiEBw-iWdRvw2pByJdj9i0uakfPwWqa5BvjzJ2pehkDvMELNYIGUxxJaJ/s1600/Screen+Shot+2015-05-01+at+12.32.20+PM.png" width="640" /></a></div>
<br />
<br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline;"></span>Kim Moirhttp://www.blogger.com/profile/14700841495895160750noreply@blogger.com0tag:blogger.com,1999:blog-25023280.post-46251423854882015822015-04-28T12:54:00.000-07:002015-04-28T12:54:00.659-07:00Releng 2015 program now availableReleng 2015 will take place in concert with ICSE in Florence, Italy on May 19, 2015. The program is now <a href="http://releng.polymtl.ca/RELENG2015/html/program.html">available</a>. Register <a href="http://releng.polymtl.ca/RELENG2015/html/registration.html">here!</a><br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg-pCM2uAgP6GPaFD3BW3c94ZRF5qeuWKjZwLujGHFC3KGMgEiQftAnD0Qfi-MQo0ZCn6eoKq_RWUB62LT1O0ZSHUnP2pXBI4avE3ABZeHrlIk8j-tJ1kg-bgtyJUonh3bW1547/s1600/5478666411_c157a64598_o.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg-pCM2uAgP6GPaFD3BW3c94ZRF5qeuWKjZwLujGHFC3KGMgEiQftAnD0Qfi-MQo0ZCn6eoKq_RWUB62LT1O0ZSHUnP2pXBI4avE3ABZeHrlIk8j-tJ1kg-bgtyJUonh3bW1547/s1600/5478666411_c157a64598_o.jpg" height="425" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span class=" meta-field photo-title " id="yui_3_16_0_1_1430250442685_20708">via romana in firenze</span> by ©<a href="https://flic.kr/p/9m8ADH">pinomoscato</a>, Creative Commons by-nc-sa 2.0</td></tr>
</tbody></table>
<br />
<br />
<br />Kim Moirhttp://www.blogger.com/profile/14700841495895160750noreply@blogger.com0tag:blogger.com,1999:blog-25023280.post-38113701562065124632015-04-28T11:47:00.000-07:002015-06-12T14:02:50.273-07:00Less testing, same great Firefox taste!<br />
Running a large continuous integration farm forces you to deal with many dynamic inputs coupled with capacity constraints. The <a href="http://relengofthenerds.blogspot.ca/2015/04/mozilla-pushes-march-2015.html">number of pushes increase</a>. People add more tests. We build and test on a new platform. If the number of machines available remains static, the computing time associated with a single push will increase. You can scale this for platforms that you build and test in the cloud (for us - Linux and Android on emulators), but this costs more money. Adding hardware for other platforms such as Mac and Windows in data centres is also costly and time consuming.<br />
<br />
Do we really need to run every test on every commit? If not, which tests should be run? How often do they need to be run in order to catch regressions in a timely manner (i.e. able to bisect where the regression occurred)<br />
<br />
<br />
Several months ago, <a href="https://elvis314.wordpress.com/">jmaher</a> and <a href="https://vaibhavag.wordpress.com/">vaibhav1994</a>, wrote <a href="https://github.com/dminor/ouija">code to analyze the test data</a> and determine the minimum number of tests required to run to identify regressions. They named their software <a href="https://elvis314.wordpress.com/2015/02/06/seta-search-for-extraneous-test-automation/">SETA (search for extraneous test automation)</a>. They used historical data to determine the minimum set of tests that needed to be run to catch historical regressions. Previously, we coalesced tests on a number of platforms to mitigate too many jobs being queued for too few machines. However, this was not the best way to proceed because it reduced the number of times we ran all tests, not just less useful ones. SETA allows us to run a subset of tests on every commit that historically have caught regressions. We still run all the test suites, but at a specified interval. <br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgiSne-w_d6ezklldyNOambG2t2Kr9n6AqbZGYM_xhL_8EX_Yj1nnL-kPcjIooEPnJ-HRJuNLVrwkpfKyFXss_ejzwH9fg9PckOeW-C69oUxreqFkw5s2vECvRIJSlSLVC6kPjx/s1600/14759554777_15061b6b04_o.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="426" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgiSne-w_d6ezklldyNOambG2t2Kr9n6AqbZGYM_xhL_8EX_Yj1nnL-kPcjIooEPnJ-HRJuNLVrwkpfKyFXss_ejzwH9fg9PckOeW-C69oUxreqFkw5s2vECvRIJSlSLVC6kPjx/s1600/14759554777_15061b6b04_o.jpg" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><span class=" meta-field photo-title " id="yui_3_16_0_1_1430143879393_5120">SETI – The Search for Extraterrestrial Intelligence</span> by ©<a href="https://flic.kr/p/oufzqn">encouragement</a>, Creative Commons by-nc-sa 2.0</td></tr>
</tbody></table>
In the last few weeks, I've implemented SETA scheduling in our our buildbot configs to use the data that the analysis that Vaibhav and Joel implemented. Currently, it's implemented on mozilla-inbound and fx-team branches which in aggregate represent around 19.6% (March 2015 data) of total pushes to the trees. The platforms configured to run fewer tests for both opt and debug are<br />
<ul>
<li>MacOSX (10.6, 10.10)</li>
<li>Windows (XP, 7, 8)</li>
<li>Ubuntu 12.04 for linux32, linux64 and ASAN x64</li>
<li>Android 2.3 armv7 API 9</li>
</ul>
<br />
As we gather more SETA data for newer platforms, such as <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1133833">Android 4.3</a>, we can implement SETA scheduling for it as well and reduce our test load. We continue to run the full suite of tests on all platforms other branches other than m-i and fx-team, such as mozilla-central, try, and the beta and release branches. If we did miss a regression by reducing the tests, it would appear on other branches mozilla-central. We will continue to update our configs to incorporate SETA data as it changes.<br />
<br />
<b>How does SETA scheduling work? </b><br />
We specify the tests that we would like to run on a reduced schedule in our buildbot configs. For instance, this specifies that we would like to run these debug tests on every 10th commit or if we reach a timeout of 5400 seconds between tests.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhKk6RcwKAQEbYlsT6oW7qFvlzCkVoRgSOhuHT5pGeaYzro2KtqKRHijPA416ZLiEXBCiLcNP_ihh2Ug4PWGngYTgsQuJt2e2cRaQvTvvR3phYlibI-PxvvAU-ECvmCfFK2RDQq/s1600/Screen+Shot+2015-04-28+at+1.09.10+PM.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="54" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhKk6RcwKAQEbYlsT6oW7qFvlzCkVoRgSOhuHT5pGeaYzro2KtqKRHijPA416ZLiEXBCiLcNP_ihh2Ug4PWGngYTgsQuJt2e2cRaQvTvvR3phYlibI-PxvvAU-ECvmCfFK2RDQq/s1600/Screen+Shot+2015-04-28+at+1.09.10+PM.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">http://hg.mozilla.org/build/buildbot-configs/file/2d9e77a87dfa/mozilla-tests/config_seta.py#l692</td></tr>
</tbody></table>
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
Previously, <a href="http://atlee.ca/blog/index.html">catlee</a> had implemented a scheduling in buildbot that allowed us to coallesce jobs on a certain branch and platform using <a href="http://hg.mozilla.org/build/buildbotcustom/file/134f2e1f9618/scheduler.py#l562">EveryNthScheduler</a>. However, as it was originally implemented, it didn't allow us to specify tests to skip, such as mochitest-3 debug on MacOSX 10.10 on mozilla-inbound. It would only allow us to skip all the debug or opt tests for a certain platform and branch.<br />
<br />
I modified misc.py to parse the configs and create a dictionary for each test specifying the interval at which the test should be skipped and the timeout interval. If the tests has these parameters specified, it should be scheduled using the <a href="http://hg.mozilla.org/build/buildbotcustom/file/134f2e1f9618/scheduler.py#l562">EveryNthScheduler</a> instead of the default scheduler.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjC7cOevu3RMU4A0uXk7jmEYw_eeVFt87qTi8ps4K9s2IbxW-OmnphHuKC2duM7Q009Krd1lpHWtA47IsnSxa6gYz4pEM_k_ka5SiBziwmiJmZfZR1J4a5W_DhPr1CwwFL_ua_g/s1600/Screen+Shot+2015-04-28+at+1.52.41+PM.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="234" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjC7cOevu3RMU4A0uXk7jmEYw_eeVFt87qTi8ps4K9s2IbxW-OmnphHuKC2duM7Q009Krd1lpHWtA47IsnSxa6gYz4pEM_k_ka5SiBziwmiJmZfZR1J4a5W_DhPr1CwwFL_ua_g/s1600/Screen+Shot+2015-04-28+at+1.52.41+PM.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">http://hg.mozilla.org/build/buildbotcustom/file/728dc76b5ad0/misc.py#l2727</td></tr>
</tbody></table>
There are still some quirks to work out but I think it is working out well so far. I'll have some graphs in a future post on how this reduced our test load. <br />
<br />
<b>Further reading</b><br />
<a href="https://elvis314.wordpress.com/2015/02/06/seta-search-for-extraneous-test-automation/">Joel Maher: SETA – Search for Extraneous Test Automation</a><br />
<span class="byline"><span class="author vcard"><a class="url fn n" href="https://vaibhavag.wordpress.com/author/vaibhavmagarwal/" rel="author" title="View all posts by Vaibhav Agrawal">Vaibhav Agrawal</a> <a href="https://vaibhavag.wordpress.com/2015/04/27/a-team-contributions-in-2015/">A-Team Contributions in 2015</a> </span></span><br />
<span class="byline"><span class="author vcard"><span class="byline"><span class="author vcard"><span id="summary_alias_container"><span id="short_desc_nonedit_display"><a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1131269"><span id="summary_alias_container"><span id="short_desc_nonedit_display">Use SETA data to disable unneeded tests</span></span></a> (To do - update configs dynamically)</span></span></span></span> </span></span><br />
<span class="byline"><span class="author vcard"><span id="summary_alias_container">
<span id="short_desc_nonedit_display"><a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1155362">Enable SETA for Android in buildbot scheduling, configs</a></span></span></span></span><br />
<span class="byline"><span class="author vcard"><span id="summary_alias_container"><span id="short_desc_nonedit_display"><br /></span></span></span></span>Kim Moirhttp://www.blogger.com/profile/14700841495895160750noreply@blogger.com1tag:blogger.com,1999:blog-25023280.post-79777806837434954422015-04-15T07:18:00.002-07:002015-04-15T07:18:57.471-07:00Mozilla pushes - March 2015Here's March 2015's monthly analysis of the pushes to our Mozilla development trees. You can load the data as an <a href="http://people.mozilla.org/~kmoir/reports/2015-03%20pushes%20report.html">HTML page</a> or as a <a href="http://people.mozilla.org/~kmoir/reports/201405.json">json file</a>.<br />
<br /><b>Trends</b><br />The number of pushes increased from those recorded in the previous month with a total of 10943. <br /><br />
<b>Highlights</b><br />
<ul>
<li>10943 pushes</li>
<li>353 pushes/day (average)</li>
<li>Highest number of pushes/day: 579 pushes on Mar 11, 2015</li>
<li>23.18 pushes/hour (highest average)</li>
</ul>
<br /><b>General Remarks</b><br />
<ul>
<li>Try keeps on having around 49% of all the pushes</li>
<li>The three integration repositories (fx-team, mozilla-inbound and b2g-inbound) account around 26% of all the pushes.</li>
</ul>
<br /><b>Records</b><br />
<ul>
<li>August 2014 was the month with most pushes (13090 pushes)</li>
<li>August 2014 had the highest pushes/day average with 422 pushes/day</li>
<li>July 2014 had the highest average of "pushes-per-hour" with 23.51 pushes/hour</li>
<li>October 8, 2014 had the highest number of pushes in one day with 715 pushes </li>
</ul>
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEik3Hgi-yqSZC25_rm0aohyphenhyphenqJc-wLNRhhrB1VGfBvMnJyOtDg2FfQjm8iJoEwNlHq7JMn4DXf9i1CHQ4lVZIbXSOfKsWFNEjMeKsM-SjKLiCXthDJ8avKqERrPugOy4C_hn40Gn/s1600/Screen+Shot+2015-04-15+at+10.10.10+AM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEik3Hgi-yqSZC25_rm0aohyphenhyphenqJc-wLNRhhrB1VGfBvMnJyOtDg2FfQjm8iJoEwNlHq7JMn4DXf9i1CHQ4lVZIbXSOfKsWFNEjMeKsM-SjKLiCXthDJ8avKqERrPugOy4C_hn40Gn/s1600/Screen+Shot+2015-04-15+at+10.10.10+AM.png" height="264" width="640" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj2EPSUbIQMk6kz-E09DIocGjqrTjc-lTZ9jrniOOJumdx5saiD9lbWyZEN6lsGXwL-A41ESEed6bzWOf6UJTRB-y1ax0SHxu4ceFbMxguKe8PJbG0gCsOHj1sT_2dAjfk0Ibvw/s1600/Screen+Shot+2015-04-15+at+10.10.26+AM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj2EPSUbIQMk6kz-E09DIocGjqrTjc-lTZ9jrniOOJumdx5saiD9lbWyZEN6lsGXwL-A41ESEed6bzWOf6UJTRB-y1ax0SHxu4ceFbMxguKe8PJbG0gCsOHj1sT_2dAjfk0Ibvw/s1600/Screen+Shot+2015-04-15+at+10.10.26+AM.png" height="254" width="640" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj5P3hxU4iCS2_0Z-gbzdbLvHNUzK4xYv_ewQ5Ogo9PM2bYwn9q8pjmbpOC8tJ13CYeDyBo8QZycVyDiD_WDz-OdTp20_hbT-PmVRyZxzwOapYCzf_XZI-g9t9Mm4MD4Baue2Rr/s1600/Screen+Shot+2015-04-15+at+10.10.34+AM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj5P3hxU4iCS2_0Z-gbzdbLvHNUzK4xYv_ewQ5Ogo9PM2bYwn9q8pjmbpOC8tJ13CYeDyBo8QZycVyDiD_WDz-OdTp20_hbT-PmVRyZxzwOapYCzf_XZI-g9t9Mm4MD4Baue2Rr/s1600/Screen+Shot+2015-04-15+at+10.10.34+AM.png" height="340" width="640" /></a></div>
<br />Kim Moirhttp://www.blogger.com/profile/14700841495895160750noreply@blogger.com0tag:blogger.com,1999:blog-25023280.post-51503655931420155082015-03-20T11:50:00.000-07:002015-03-20T11:52:53.502-07:00Scaling YosemiteWe migrated most of our Mac OS X 10.8 (Mountain Lion) test machines to 10.10.2 (Yosemite) this quarter.<br />
<br />
This project had two major constraints:<br />
1) Use the existing hardware pool (~100 r5 mac minis) <br />
2) Keep wait times sane<sup>1</sup>. (The machines are constantly running tests most of the day due to the distributed nature of the Mozilla community and this had to continue during the migration.)<br />
<br />
So basically upgrade all the machines without letting people notice what you're doing!<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj-VVNQLL0SOHgYUCnI5s3Z_EkomnqtDbuNMfQU_rmCxQueQ9UusYBMcXxOq52sDfacAefwUXLiu5t99nGuB4WR-U4TZPZe1guxWuhEPXZmtfCriBZI2EIeTBPRvFJJFNAFaOSj/s1600/8491263568_e27b3642e1_b.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj-VVNQLL0SOHgYUCnI5s3Z_EkomnqtDbuNMfQU_rmCxQueQ9UusYBMcXxOq52sDfacAefwUXLiu5t99nGuB4WR-U4TZPZe1guxWuhEPXZmtfCriBZI2EIeTBPRvFJJFNAFaOSj/s1600/8491263568_e27b3642e1_b.jpg" height="426" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Yosemite Valley - Tunnel View Sunrise by <a href="https://flic.kr/p/dWkWQ1">©jeffkrause</a>, Creative Commons by-nc-sa 2.0 </td></tr>
</tbody></table>
<br />
Why didn't we just buy more minis and add them to the existing pool of test machines?<br />
<ol>
<li>We run performance tests and thus need to have all the machines running the same hardware within a pool so performance comparisons are valid. If we buy new hardware, we need to replace the entire pool at once. Machines with different hardware specifications = useless performance test comparisons. </li>
<li>We tried to purchase some used machines with the same hardware specs as our existing machines. However, we couldn't find a source for them. As Apple stops production of old mini hardware each time they announce a new one, they are difficult and expensive to source.</li>
</ol>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhWM_-9JrUzCCKDIcB6XJdkxj08o-fBfmoEVfTZ8muBd4x50yLabPYzP_TC5jz3p3wEY6zmfUDXC52vsaCKjJ6acSjwOd7V6CgCM8skM2Xw0WoL66pUVKEujzfcSvhu0sfCY8zz/s1600/16626177600_9816900af1_b.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhWM_-9JrUzCCKDIcB6XJdkxj08o-fBfmoEVfTZ8muBd4x50yLabPYzP_TC5jz3p3wEY6zmfUDXC52vsaCKjJ6acSjwOd7V6CgCM8skM2Xw0WoL66pUVKEujzfcSvhu0sfCY8zz/s1600/16626177600_9816900af1_b.jpg" height="424" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Apple Pi by ©<a href="https://flic.kr/p/rkcvmu">apionid</a>, Creative Commons by-nc-sa 2.0</td></tr>
</tbody></table>
<br />
Given that Yosemite was released last October, why we are only upgrading our test pool now? We wait until the population of users running a new platform<sup>2</sup> surpass those the old one before switching.<br />
<br />
Mountain Lion -> Yosemite is an easy upgrade on your laptop. It's not as simple when you're updating production machines that run tests at scale.<br />
<br />
The first step was to pull a few machines out of production and verify the Puppet configuration was working. In Puppet, you can specify commands to only run certain operating system versions. So we implemented several commands to accommodate changes for Yosemite. For instance, changing the <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1033650">default scrollbar behaviour</a>, new services that interfere with test runs needed to be disabled, debug tests required <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1122875">new Apple security permissions</a> configured etc.<br />
<br />
Once the Puppet configuration was stable, I updated our configs so the people could run tests on <a href="https://wiki.mozilla.org/ReleaseEngineering/TryServer">Try</a> and allocated a few machines to this pool. We <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1121199">opened bugs for tests</a> that failed on Yosemite but passed on other platforms. This was a very iterative process. Run tests on try. Look at failures, file bugs, fix test manifests. Once we had to the opt (functional) tests in a green state on try, we could start the migration.<br />
<br />
<b>Migration strategy</b><br />
<ul>
<li>Disable selected Mountain Lion machines from the production pool</li>
<li>Reimage as Yosemite, update DNS and let them puppetize</li>
<li>Land patches to disable Mountain Lion tests and <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1126493">enable corresponding Yosemite tests</a> on selected branches</li>
<li>Enable Yosemite machines to take production jobs</li>
<li><a href="https://wiki.mozilla.org/ReleaseEngineering:Buildduty:Reconfigs">Reconfig</a> so the buildbot master enable new Yosemite builders and schedule jobs appropriately</li>
<li>Repeat this process in batches</li>
<ul>
<li>Enable Yosemite opt and performance tests on trunk (gecko >= 39) (50 machines)</li>
<li>Enable Yosemite debug (25 more machines)</li>
<li>Enable Yosemite on mozilla-aurora (15 more machines)</li>
</ul>
</ul>
We currently have 14 machines left on Mountain Lion for mozilla-beta and mozilla-release branches.<br />
<br />
As a I mentioned earlier, the two constraints with this project were to use the existing hardware pool that constantly runs tests in production and keep the existing wait times sane. We encountered two major problems that impeded that goal:<br />
<ul>
<li>Persistent and increasing numbers of DNS failures as we migrated more machines to Yosemite. The default Yosemite configuration broadcasts multicast messages via <a href="https://support.apple.com/kb/TA20999?locale=en_CA">Bonjour</a>. This is fine for a few Apple devices talking to each other in your house. This doesn't scale for 100 machines in a colo. I saw many DNS timeout messages <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1144206#c1">in the system log</a>. This manifested itself large numbers of performance tests failing because they couldn't resolve the name of the graphing server to upload their results. We <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1144206">disabled this multicast broadcast via Puppet</a> and our tests turned green again. </li>
<li><a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1125998">Debug tests initially took 1.5-2x longer to complete on Yosemite</a> than they did on Mountain Lion. If we deployed these tests in production, our wait times would increase because of the longer time required to complete each test run. This was fixed by <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1138616">bug 1138616 Don't track addref and release counts in the bloat log.</a></li>
</ul>
<br />
It's a compliment when people say things like "I didn't realize that you updated a platform" because it means the upgrade did not cause large scale fires for all to see. So it was a nice to hear that from one of my colleagues this week.<br />
<br />
Thanks to philor, RyanVM and jmaher for opening bugs with respect to failing tests and greening them up. Thanks to coop for many code reviews. Thanks dividehex for reimaging all the machines in batches and to arr for her valiant attempts to source new-to-us minis! <br />
<br />
<b>References</b><br />
<sup>1</sup>Wait times represent the time from when a job is added to the scheduler database until it actually starts running. We usually try to keep this to under 15 minutes but this really varies on how many machines we have in the pool.<br />
<sup>2</sup>We run tests for our products on a matrix of operating systems and operating system versions. The terminology for operating system x version in many release engineering shops is a platform. To add to this, the list of platform we support varies across branches. For instance, if we're going to deprecate a platform, we'll let this change <a href="http://relengofthenerds.blogspot.ca/2014/04/enabling-tests-that-ride-trains.html">ride the trains to release</a>. <br />
<br />
<b>Further reading</b><br />
<a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1121175">Bug 1121175: [Tracking] Fix failing tests on Mac OSX 10.10 </a><br />
<a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1121199">Bug 1121199: Green up 10.10 tests currently failing on try</a> <br />
<a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1126493">Bug 1126493: rollout 10.10 tests in a way that doesn't impact wait times</a><br />
<a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1144206">Bug 1144206: investigate what is causing frequent talos failures on 10.10</a> <br />
<a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1125998">Bug 1125998: Debug tests initially took 1.5-2x longer to complete on Yosemite</a> <br />
<br />
<br />
<b>Why don't you just run these tests in the cloud?</b><br />
<ol>
<li>The Apple EULA severely restricts virtualization on Mac hardware. </li>
<li>I don't know of any major cloud vendors that offer the Mac as a platform. Those that claim they do are actually renting racks of Macs on a dedicated per host basis. This does not have the inherent scaling and associated cost saving of cloud computing. In addition, the APIs to manage the machines at scale aren't there.</li>
<li>We manage ~350 Mac minis. We have more experience scaling Apple hardware than many vendors. Not many places run CI at Mozilla scale :-) Hopefully this will change and we'll be able to scale testing on Mac products like we do for Android and Linux in a cloud. </li>
</ol>
Kim Moirhttp://www.blogger.com/profile/14700841495895160750noreply@blogger.com0tag:blogger.com,1999:blog-25023280.post-62252221620134155242015-03-17T08:54:00.000-07:002015-03-17T08:54:13.810-07:00Mozilla pushes - February 2015Here's February's 2015 monthly analysis of the pushes to our Mozilla development trees. You can load the data as <a href="http://people.mozilla.org/~kmoir/reports/2015-02%20pushes%20report.html">an HTML page</a> or as a <a href="http://people.mozilla.org/~kmoir/reports/201502.json">json file</a>.<br /><br /><b>Trends</b><br />Although February is a shorter month, the number of pushes were close to those recorded in the previous month. We had a higher average number of daily pushes (358) than in January (348).<br />
<br /><b>Highlights</b><br />10015 pushes<br />358 pushes/day (average)<br />Highest number of pushes/day: 574 pushes on Feb 25, 2015<br />23.18 pushes/hour (highest)<br /><br /><b>General Remarks</b><br />Try had around 46% of all the pushes<br />The three integration repositories (fx-team, mozilla-inbound and b2g-inbound) account around 22% of all the pushes<br /><br /><b>Records</b><br />August 2014 was the month with most pushes (13090 pushes)<br />August 2014 has the highest pushes/day average with 422 pushes/day<br />July 2014 has the highest average of "pushes-per-hour" with 23.51 pushes/hour<br />October 8, 2014 had the highest number of pushes in one day with 715 pushes <br /><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEisNL4lM5CwwGUoz1z6uk61H-cXW-QkgIhhE5A0KdGGWzylbdkhe2MB60GmOfcWts1PRALF8_CSbzO6ROvgW-qyjpURmzcmCNsGCQc_xSFkS4zPkEOQjsS1CcYeQmYl27W5zuGG/s1600/Screen+Shot+2015-03-17+at+11.29.24+AM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEisNL4lM5CwwGUoz1z6uk61H-cXW-QkgIhhE5A0KdGGWzylbdkhe2MB60GmOfcWts1PRALF8_CSbzO6ROvgW-qyjpURmzcmCNsGCQc_xSFkS4zPkEOQjsS1CcYeQmYl27W5zuGG/s1600/Screen+Shot+2015-03-17+at+11.29.24+AM.png" height="284" width="640" /></a></div>
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgRHOPtB2DLuOZX3g4H48b4qLf-bn-kG5OmmCwVcY6-xkcBp_DwPPe8-oUc1XVCC_PuajtGgkXDPUkWbVJ8nkodZJiZtFMKf80MO5domFHpdfkKEFIa2FUyLf3MqFEWFlsIu-dk/s1600/Screen+Shot+2015-03-17+at+11.29.40+AM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgRHOPtB2DLuOZX3g4H48b4qLf-bn-kG5OmmCwVcY6-xkcBp_DwPPe8-oUc1XVCC_PuajtGgkXDPUkWbVJ8nkodZJiZtFMKf80MO5domFHpdfkKEFIa2FUyLf3MqFEWFlsIu-dk/s1600/Screen+Shot+2015-03-17+at+11.29.40+AM.png" height="278" width="640" /></a><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEijvCpO49v1Kk4YONlfOvAMbMTzALlzY4eXPkpjylG5O2B-UrpaXiIFBCWQ_0UlHeguxBKmaRN7xHubKxjpvuaTw_xEC6qiF9Goxss9O4y5IDWJO3UfQj_abxT2v8w0Ag90hH0p/s1600/Screen+Shot+2015-03-17+at+11.29.32+AM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEijvCpO49v1Kk4YONlfOvAMbMTzALlzY4eXPkpjylG5O2B-UrpaXiIFBCWQ_0UlHeguxBKmaRN7xHubKxjpvuaTw_xEC6qiF9Goxss9O4y5IDWJO3UfQj_abxT2v8w0Ag90hH0p/s1600/Screen+Shot+2015-03-17+at+11.29.32+AM.png" height="374" width="640" /></a></div>
<br />
<br />Kim Moirhttp://www.blogger.com/profile/14700841495895160750noreply@blogger.com0tag:blogger.com,1999:blog-25023280.post-46887830846383601162015-02-25T07:26:00.000-08:002015-03-02T06:48:08.927-08:00Release Engineering special issue now availableThe <a href="http://online.qmags.com/ISW0315">release engineering special issue of IEEE software</a> was published yesterday. (Download <a href="http://delivery.qmags.com/d/?pub=ISW&upid=19277&s=OL&fl=others%2fISW%2fISW_20150301_Mar_2015.pdf">pdf here</a>). This issue focuses on the current state of release engineering, from both an industry and research perspective. Lots of exciting work happening in this field!<br />
<br />
I'm interviewed in the <a href="http://online.qmags.com/ISW0315?pg=45&mode=2#pg45&mode2">roundtable article on the future of release engineering</a>, along with <a href="https://twitter.com/chuckr">Chuck Rossi</a> of Facebook and <a href="https://plus.google.com/+BorisDebic/posts">Boris Debic</a> of Google. Interesting discussions on the current state of release engineering at organizations that scale large number of builds and tests, and release frequently. As well, the challenges with mobile releases versus web deployments are discussed. And finally, a discussion of how to find good release engineers, and what the future may hold.<br />
<br />
Thanks to the other guest editors on this issue - Stephany Bellomo, Tamara Marshall-Klein, Bram Adams, Foutse Khomh and Christian Bird - for all their hard work that make this happen!<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh1kKTGckHm0CaAZPh-KoK7wjTmyqKnS9rTdOS3d-HvsnfBNOOTsVjUsvjyV2jQmlBuLRAeAV564QoIiMMNSn4v91r6a5_w8MXjrP5PV6GZYiO3DIJqF7q2U1mgdjAuBOQ3JMVL/s1600/Screen+Shot+2015-02-25+at+10.12.32+AM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh1kKTGckHm0CaAZPh-KoK7wjTmyqKnS9rTdOS3d-HvsnfBNOOTsVjUsvjyV2jQmlBuLRAeAV564QoIiMMNSn4v91r6a5_w8MXjrP5PV6GZYiO3DIJqF7q2U1mgdjAuBOQ3JMVL/s1600/Screen+Shot+2015-02-25+at+10.12.32+AM.png" height="400" width="293" /></a></div>
<br />
As an aside, when I opened the issue, the image on the front cover made me laugh. It's reminiscent of the cover on a mid-century science fiction anthology. I showed Mr. Releng and he said "Robot birds? That is EXACTLY how I pictured working in releng." Maybe it's meant to represent that we let software fly free. In any case, I must go back to tending the flock of robotic avian overlords.Kim Moirhttp://www.blogger.com/profile/14700841495895160750noreply@blogger.com2tag:blogger.com,1999:blog-25023280.post-31475488297722824432015-02-13T08:13:00.000-08:002015-02-13T08:13:16.938-08:00Mozilla pushes - January 2015Here's January 2015's monthly analysis of the pushes to our Mozilla development trees. You can load the data as an <a href="http://people.mozilla.org/~kmoir/reports/2015-01%20pushes%20report.html">HTML page</a> or as a <a href="http://people.mozilla.org/~kmoir/reports/201501.json">json file</a>.<br />
<br />
<b>Trends</b><br />
We're back to regular volume after the holidays. Also, it's really cold outside in some parts of the of the Mozilla world. Maybe committing code > going outside.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjtmfqef_pWLalCFGuVjhV0bvhdhTvLTfDFDX8VAx4B1FPrPxzxNEOQ7Dq2lTNKREkiHkyB-WJ9-usi6DNZilyMKTqsuhfglCoQxoTkXvz7WRiMgyDX9pPNHNS7P_fdbe7iOqKr/s1600/Screen+Shot+2015-02-13+at+11.06.02+AM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjtmfqef_pWLalCFGuVjhV0bvhdhTvLTfDFDX8VAx4B1FPrPxzxNEOQ7Dq2lTNKREkiHkyB-WJ9-usi6DNZilyMKTqsuhfglCoQxoTkXvz7WRiMgyDX9pPNHNS7P_fdbe7iOqKr/s1600/Screen+Shot+2015-02-13+at+11.06.02+AM.png" height="371" width="640" /></a></div>
<br />
<b>Highlights</b><br />
10798 pushes<br />
348 pushes/day (average)<br />
Highest number of pushes/day: 562 pushes on Jan 28, 2015<br />
18.65 pushes/hour (highest)<br />
<br />
<b>General Remarks</b><br />
Try had around around 42% of all the pushes<br />
The three integration repositories (fx-team, mozilla-inbound and b2g-inbound) account around 24% of all of the pushes<br />
<br />
<b>Records</b><br />
August 2014 was the month with most pushes (13,090 pushes)<br />
August 2014 has the highest pushes/day average with 422 pushes/day<br />
July 2014 has the highest average of "pushes-per-hour" with 23.51 pushes/hour<br />
October 8, 2014 had the highest number of pushes in one day with 715 pushes <br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjePI3bdK2ntzQpihv2XfyjzJLlz8p4ihLoZTTt_5ZYznYatdb9QoKUcau51_fqk4__Av8hPPjSCbzxa1PJgpM5eeAo4O0W02mnuRhOXHQqGBahMXyjqrh6FOiCpuEZ81csNxAz/s1600/Screen+Shot+2015-02-13+at+10.47.42+AM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjePI3bdK2ntzQpihv2XfyjzJLlz8p4ihLoZTTt_5ZYznYatdb9QoKUcau51_fqk4__Av8hPPjSCbzxa1PJgpM5eeAo4O0W02mnuRhOXHQqGBahMXyjqrh6FOiCpuEZ81csNxAz/s1600/Screen+Shot+2015-02-13+at+10.47.42+AM.png" height="250" width="640" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgSQITJ9JIEflKMqRewgOBmFUyeVSRmduQBhR3qV-KVK7KE1wAYpG_GdBrGG1m9fGqBLYRp84U74fYxTzYfVT4V4om1aso3XrxqNLRWRYLnZAtHthpwjIzgI5uuj0_xMbyVUZRc/s1600/Screen+Shot+2015-02-13+at+10.48.04+AM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgSQITJ9JIEflKMqRewgOBmFUyeVSRmduQBhR3qV-KVK7KE1wAYpG_GdBrGG1m9fGqBLYRp84U74fYxTzYfVT4V4om1aso3XrxqNLRWRYLnZAtHthpwjIzgI5uuj0_xMbyVUZRc/s1600/Screen+Shot+2015-02-13+at+10.48.04+AM.png" height="248" width="640" /></a></div>
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjX96_AlJJWIlQ3ul7IQezLyupX-VSJOrilrh0PX06zk7Xa897IhVaML1uqqquRWANjK3u9YMeDVlFUVlnmFGT327meUBBffA-wyYS1to4koDNNs0OkVB8g51jbnpU1Jh6wx4Ho/s1600/Screen+Shot+2015-02-13+at+10.47.56+AM.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjX96_AlJJWIlQ3ul7IQezLyupX-VSJOrilrh0PX06zk7Xa897IhVaML1uqqquRWANjK3u9YMeDVlFUVlnmFGT327meUBBffA-wyYS1to4koDNNs0OkVB8g51jbnpU1Jh6wx4Ho/s1600/Screen+Shot+2015-02-13+at+10.47.56+AM.png" height="354" width="640" /></a><br />
<br />Kim Moirhttp://www.blogger.com/profile/14700841495895160750noreply@blogger.com0tag:blogger.com,1999:blog-25023280.post-38571723214914115892015-01-21T12:36:00.003-08:002015-01-21T12:36:48.878-08:00Reminder: Releng 2015 submissions due Friday, January 23Just a reminder that submissions for the Releng 2015 conference are due this Friday, January 23. <br />
<br />
It will be held on <a href="http://releng.polymtl.ca/RELENG2015/html/index.html">May 19, 2015 in Florence Italy</a>. <br /><br />If you've done recent work like<br />
<ul>
<li>migrating your build or test pipeline to the cloud</li>
<li>switching to a new build system</li>
<li>migrating to a new version control system</li>
<li>optimized your configuration management system or switched to a new one</li>
<li>implemented continuous integration for mobile devices</li>
<li>reduced end to end build times</li>
<li>or anything else build, release, configuration and test related</li>
</ul>
we'd love to hear from you. Please consider submitting a talk!<br /><br />In addition, if you have colleagues that work in this space that might have interesting topics to discuss at this workshop, please forward this information. I'm happy to talk to people about the submission process or possible topics if there are questions.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhTydLlR-Vf1pzNxLrDstj-Atn90Em6lA8ZLI1cJD2E8jIRzPtLSNbpU81-6jgXFCr47mMtr3JV3LcY7UhHbdtIw9dJywKxYKeqtFMLMhWtj9xiIu3vrrXo5wHNJE07RByzJkx1/s1600/4677592277_9a0d3f2e5e_o.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhTydLlR-Vf1pzNxLrDstj-Atn90Em6lA8ZLI1cJD2E8jIRzPtLSNbpU81-6jgXFCr47mMtr3JV3LcY7UhHbdtIw9dJywKxYKeqtFMLMhWtj9xiIu3vrrXo5wHNJE07RByzJkx1/s1600/4677592277_9a0d3f2e5e_o.jpg" height="300" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Il Duomo di Firenze by <a href="https://flic.kr/p/88kTdH">©eddi_07</a>, Creative Commons by-nc-sa 2.0 </td></tr>
</tbody></table>
<br />
<br />
Sono nel comitato che organizza la conferenza Releng 2015 che si terrà il 19 Maggio 2015 a Firenze. La scadenza per l’invio dei paper è il 23 Gennaio 2015.<br /><br />http://releng.polymtl.ca/RELENG2015/html/index.html<br /><br />se avete competenze in:<br />
<ul>
<li>migrazione del sistema di build o dei test nel cloud</li>
<li>aggiornamento del processo di build</li>
<li>migrazione ad un nuovo sistema di version control</li>
<li>ottimizzazione o aggiornamento del configuration management system</li>
<li>implementazione di un sistema di continuos integration per dispositivi mobili</li>
<li>riduzione dei tempi di build</li>
<li>qualsiasi cambiamento che abbia migliorato il sistema di build/test/release</li>
</ul>
e volete discutere della vostra esperienza, inviateci una proposta di talk!<br /><br />Per favore inoltrate questa richiesta ai vostri colleghi e alle persone interessate a questi argomenti. Nel caso ci fossero domande sul processo di invio o sui temi di discussione, non esitate a contattarmi.<br />
<br />
(Thanks <a href="https://twitter.com/gerva">Massimo</a> for helping with the Italian translation).<br />
<br />
More information<br />
<a href="http://releng.polymtl.ca/RELENG2015/html/index.html">Releng 2015 web page </a><br />
<a href="http://relengofthenerds.blogspot.ca/2014/12/releng-2015-cfp-now-open.html">Releng 2015 CFP now open </a><br />
<br />Kim Moirhttp://www.blogger.com/profile/14700841495895160750noreply@blogger.com0tag:blogger.com,1999:blog-25023280.post-14542511356668710042015-01-08T09:14:00.000-08:002015-01-13T07:17:11.727-08:00Mozilla pushes - December 2014<br />
Here's December 2014's monthly analysis of the pushes to our Mozilla development trees. You can load the data as an <a href="http://people.mozilla.org/~kmoir/reports/2014-12%20pushes%20report.html">HTML page </a>or as a <a href="http://people.mozilla.org/~kmoir/reports/201412.json">json file</a>.<br />
<br />
<b>Trends</b><br />
There was a low number of pushes this month. I expect this is due to the Mozilla all-hands in Portland in early December where we were encouraged to meet up with other teams instead of coding :-) and the holidays at the end of the month for many countries.<br />
As as side node, in 2014 we had a total number of 124423 pushes, compared to 79233 in 2013 which represents a growth rate of 57% this year.<br />
<br />
<b>Highlights</b><br />
7836 pushes<br />
253 pushes/day (average)<br />
Highest number of pushes/day: 706 pushes on Dec 17, 2014<br />
15.25 pushes/hour (highest)<br />
<br />
<b>General Remarks</b><br />
Try had around around 46% of all the pushes<br />
The three integration repositories (fx-team, mozilla-inbound and b2g-inbound) account around 23% of all of the pushes<br />
<br />
<b>Records</b><br />
August 2014 was the month with most pushes (13,090 pushes)<br />
August 2014 has the highest pushes/day average with 422 pushes/day<br />
July 2014 has the highest average of "pushes-per-hour" with 23.51 pushes/hour<br />
October 8, 2014 had the highest number of pushes in one day with 715 pushes <br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiY5PfhSFjcUDOu2XQjhFHC9W2VpWPRBDJn7rsSiJRhIXovhKqHllTD0nEL29TH17AeAYXAjZktDUT3DZMjn7FzgXsXyTqLR4KFJMbibNBinG8rNIp67cQDHlioU0bvapvKn7WQ/s1600/Screen+Shot+2015-01-08+at+12.00.20+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiY5PfhSFjcUDOu2XQjhFHC9W2VpWPRBDJn7rsSiJRhIXovhKqHllTD0nEL29TH17AeAYXAjZktDUT3DZMjn7FzgXsXyTqLR4KFJMbibNBinG8rNIp67cQDHlioU0bvapvKn7WQ/s1600/Screen+Shot+2015-01-08+at+12.00.20+PM.png" height="318" width="640" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgEF_uYPaoNvI17OXFXUnOwItaYOEqH3BzSApRPLnmy2yxgcDntsEGhh99jb0M6ZbxrLhCPs-LI_xzWs_cOvktv_Qzkb9tpDSgG7b0PgWiwujmtBUbkbKsnq069P3uhq6GZQixb/s1600/Screen+Shot+2015-01-08+at+12.00.28+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgEF_uYPaoNvI17OXFXUnOwItaYOEqH3BzSApRPLnmy2yxgcDntsEGhh99jb0M6ZbxrLhCPs-LI_xzWs_cOvktv_Qzkb9tpDSgG7b0PgWiwujmtBUbkbKsnq069P3uhq6GZQixb/s1600/Screen+Shot+2015-01-08+at+12.00.28+PM.png" height="288" width="640" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjpXx7jpvmkQE0KxZyqBhutl8IXCLSUxvXzo9A6UITYjbzyAM9uo5ANlr2Wk7pxGBG4MZOSimCxN6P_1bVtQV2ISCSZPXK8mkbMHTnmcUd5ss_5jfwkanFaJc8lCvMgl1QtCIJ6/s1600/Screen+Shot+2015-01-08+at+12.00.37+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjpXx7jpvmkQE0KxZyqBhutl8IXCLSUxvXzo9A6UITYjbzyAM9uo5ANlr2Wk7pxGBG4MZOSimCxN6P_1bVtQV2ISCSZPXK8mkbMHTnmcUd5ss_5jfwkanFaJc8lCvMgl1QtCIJ6/s1600/Screen+Shot+2015-01-08+at+12.00.37+PM.png" height="250" width="400" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />Kim Moirhttp://www.blogger.com/profile/14700841495895160750noreply@blogger.com2