A tale of CVS, NFS and Swordplay

Earlier this year, the time that it took our builds to check out code from CVS had slowed to a crawl. Last year, our integration builds that started by 8am were available by noon. However, during the 3.6 cycle, most builds weren't ready until 2pm or even 4pm. Very painful, especially for our European committer friends. Not only were the builds slow, but it took a long time for us to commit code, and timeouts were frequent.

Denis and his merry band of webmasters determined that the source of the problem was twofold.

1) NFS configuration issues causing high CPU load on the servers. (Bug 288293)
2) Anonymous pserver users were holding the cvs lock files for a long time (Bug 293355)

Earlier this week, Denis fixed the NFS configuration issues. This had reduced the time it takes to check out our code significantly. Also, I haven't seen any CVS timeouts lately. Today, our 8am integration build was ready at noon. A big thank you to the webmasters! We are much more productive thanks to your work. Less swordplay while waiting for builds, and more time to fix bugs and implement new features. Again from xkcd.



Really, who doesn't love xkcd? It's so good.

Denis will change the anonymous pserver access to a mirrored copy of the CVS repository on December 11. He sent notes to the committers list with the details. The means that committers won't have to contend with anonymous users for cvs lock files if they check out code via ssh. Psever will be available on a separate file system, but sychronized with the live copy.

In preparation for this change, I modified our build scripts to check out our code using ssh, instead of anonymous pserver. However, our map files remain untouched so anyone can check out our code via pserver. The build scripts modify the map files after they have been checked out. We won't have to deal with file contention for CVS lock files. And ssh connections to eclipse.org have a higher QoS than pserver. Good stuff.

Most teams won't have to make any changes because they run their builds on build.eclipse.org. These users will still pull from the live copy of the data via pserver. And the mirrored copy will be up to date. I'm just trying to do everything I can to avoid infrastructure issues that cause build failures.

See bug 294900 for the platform releng changes.