Prior to this summer, it would take two hours for Microsoft's GitHub to release code to production – if everything went smoothly.
Not only was integration testing a lengthy process, but it also heavily consumed computing power. Committing code to the cloud can take 600 cores of computing resources.
The firm decided that it could do better:
"We're constantly reviewing our internal systems," " Keerthana Kumar, staff software engineering manager at GitHub, told Business Insider. "There's a constant leap to change."
In order to improve the process, GitHub first broke down the steps it takes to release code, including unit testing to test individual units of code, linting to check for programming errors, integration testing to make sure that individual sections of code all work properly together, and deployments to release the code. While most of the jobs required less than 13 minutes to run, integration testing was the biggest bottleneck, taking as much as 45 minutes.
The analysis and engineering efforts to make the process more efficient took three weeks, in which GitHub managed to cut down the integration testing time to 15 minutes, making it three times faster to make sure that new code updates work with the rest of an application. GitHub tested the new process for a week before rolling it out widely mid-summer.
These efforts fall under the niche of continuous integration and continuous delivery (CI/CD) processes, which helps developers automatically test and release code faster and more often and is on the rise as developers move to the cloud. GitHub itself also offers these kinds of tools to customers, and decided to build an internal CI tool, too.
"It's important because if you think about it, it optimizes the time for the developer," Kumar said.
Though, it's not just about the tools: Making GitHub's new time-saving tools successful required better collaboration among developers, too, as they communicate remotely from around the world to ensure that new code works for other teams, Kumar says.
"One thing that helps us build is the trust that we can collaborate across the globe," Kumar said. " Now to go back and fix an issue we hadn't fixed, that required communication. As we are building the tools, we collaborate."
GitHub made other efficiency changes, too
GitHub also identified another issue that could get in the way of making its development process faster. While GitHub releases new updates several times a day to the cloud, its offering for on-premise data centers — GitHub Enterprise Server — releases updates every two weeks and releases a major update every quarter.
Since Enterprise Server releases code on a different schedule, GitHub had to make sure that releasing new updates to the cloud did not break the code for its server customers.
To handle that, GitHub built a deferred compliance tool, which speeds up the time required for testing while maintaining quality for GitHub Enterprise Server by monitoring for test failures.
If one part of the process fails, the system automatically creates a label and tags the people who wrote the code. It also sends a warning message on Slack and triggers a 72-hour timer in which the developer has to fix the issue or make a change. If the issue isn't fixed within 72 hours, GitHub will halt new code releases until tests for Enterprise Server are fixed. This helps developers stay accountable for the code they release.
This timer also helps because it helps prevent a developer in one part of the world from unintentionally blocking releases for a developer in another part of the world.
"This really saved a lot of our developer hours," Kumar said. "We can ship faster and work on more feature development. It saved a lot of machine hours as well."
Now, GitHub saves time and computing power when it comes to releasing code, but it's constantly looking for ways to improve. Making the development process more efficient is a "continual process," Kumar says.
"We are constantly taking a pulse of our developers," Kumar said. "We are finding the bottlenecks and friction points. We identify them all the time. There might be larger friction issues, but we're able to identify a single problem. ... Even now, we're doing multiple changes to our own internal tooling to make our process complete and better."
Got a tip? Contact this reporter via email at rmchan@jkmperu.com, Signal at 646.376.6106, Telegram at @rosaliechan, or Twitter DM at @rosaliechan17. (PR pitches by email only, please.) Other types of secure messaging available upon request.
