Continuous Software Development: Scaling Continuous Integration, Delivery, and Deployment Practices

Continuous development involves testing, updating, releasing, and measuring applications in a continual fashion as they are under development. In the book, WTF?: What's the Future and Why It's Up to Us, author Tim O’Reilly describes how the principles and technical practices of the open source community inspired software development today. As one of the more modern methods, continuous development runs counter to traditional software development practices. In the 1990s, software was distributed annually (at best). This schedule worked for the marketplace demand at the time: According to O’Reilly, less than half of all U.S. households had a personal computer (for comparison, more than 80 percent of those occupants now own a smartphone). When Google was founded in September of 1998, there were 147 million internet users (today there are more than three billion).

O’Reilly asserts that delivering software as a service (SaaS) using an “iterative, user-centered, data-driven development” approach is one of the core contributions to the modern approach. Users play a critical role in the popularity of continuous software development, acting as beta-testers who contribute to the evolution of a product by giving feedback through interaction with a web-based application. For example, product development teams at Amazon monitor an application under development in the user environment (in real time), measure what is working and what could be improved, and inform future development with continuous feedback from the system. To perform this work and deploy software thousands of times a day, Amazon uses iterative programming models, like Agile software development and Kanban, DevOps organizational methods, and technical practices like continuous integration, continuous delivery, and continuous deployment.

Companies that develop software as an asset must continuously find ways to create value for their customers and competitive advantages for their business processes. Continuous iteration of your product and features helps you win in the marketplace, as does the actionable feedback that results from implementing continuous software development practices like CD. With continuous delivery, your software is in a state of readiness, so you manually control the timing of the final deployment to a production environment. Proceeding in this manner is as much a business decision as it is a technical strategy.

Former Netflix engineer and cloud architect Adrian Cockcroft is an advocate of the continuous delivery model. In 2009, Netflix adopted CD and the use of microservices architecture to develop and deploy applications. Microservices refers to developing small, reusable building blocks of code to ensure that the application under development is not affected by the increase in the velocity of deployments. The goal is to shorten the time it takes to deploy changes in code or configuration.

Cockcroft helped develop this cloud-based architecture at Netflix, leveraging tools like Amazon Web Services (AWS) and the use of CD principles so Netflix could react quickly to customer needs and use the momentum to create new experiences that attract more customers. He believes that speed (at scale) wins the marketplace. Optimizing the software development process around the principles of CI and CD places a premium on speed, and the efficiency of operations creates a byproduct of releasing popular products and features faster. Netflix is often cited as an example of a successful DevOps organization that learned to move so quickly through feature development that even today, the competition has trouble keeping up.

CD and the 2017 State of DevOps Report

Jez Humble believes the whole point of practices like continuous delivery is to change organizational behavior. This relates to the business benefits of applying CD practices to software development: If you are developing web-based features or products, firmware, or mobile applications, there is a legitimate business case for CD. This mantra applies to businesses and teams of all sizes and structures.

Humble works with a team at DORA (DevOps Research and Assessment) and another at Puppet on DevOps research, and reports the findings in the annual State of DevOps Report. The report highlights the importance of culture as a vital part of a company’s bottom line performance, as well as the impact on job satisfaction in a performance-oriented team environment. The idea is that even if you build the best systems with the best tools, a corrupt team culture will damage the effectiveness of anything they attempt to accomplish. The emphasis on culture change and adapting to the ethos of highly cooperative, shared risk, learn-from-failure mentality makes CD an excellent fit for teams in a DevOps environment.

The 2017 State of DevOps Report suggests that DevOps “enables mission achievement for any type of organization, independent of industry or sector.” In fact, the document states that high-performing DevOps teams that leverage technical practices like CD report increased frequency of deployment and more stability. High-performing DevOps teams that leverage automation, version control, and the principles of CI, CD, and CDP are more than twice as likely to achieve or exceed goals relating to the number of products and services created, operating efficiency, customer satisfaction, and more. Humble’s research demonstrates several benefits garnered from high-performance CD practices, including the following:

• Using CD technical practices like version control, CI, and test automation is a predictor of high performance IT operations.

• Lower risks in software deployments that are undetectable to users.

• Higher quality of enhanced testing practices and automation.

• Lower costs by eliminating fixed costs associated with releasing software products and features.

• Faster time to market for more efficient testing.

• Better products from faster feedback loops that produce optimized feature development.

• Teams experience less of the burnout associated with deployment or low-value activities.

Continuous integration refers to the practice of frequently merging smaller code changes into a central repository (where the “master” or “main” branch of code lives). In some cases, the frequency means multiple daily integrations; in other cases, several thousand times a day. The integration of individual code to the system is verified by automated builds and automated tests that occur every time there is a change to the version control system. This routine empowers the rapid software release schedules that iterative programming models like Agile and DevOps methods require for modern SaaS development.

Daily continuous integration evolved and originated with Kent Beck’s book, Extreme Programming Explained. The Extreme Programming (XP) development process recommends using CI in the original twelve practices Beck created. Developers share their workflow (code and tests) as quickly as possible after completing tasks by merging code changes into a shared repository (version control). These isolated changes trigger an automated system to build, test, and validate the main branch (sometimes referred to as the trunk, as in trunk-based development).

Martin Fowler is an author, speaker, and Chief Scientist at the software company Thoughtworks (creators of tools like CruiseControl and Go). He is often cited for his influence on CI and Agile approaches to software development. Fowler describes the rationale behind CI as a practice that leads to significantly reduced integration problems and allows a team to develop cohesive software more rapidly (daily). Automated tests are fundamental to this practice. He writes on his blog: “Continuous integration is a software development practice where members of a team integrate their work frequently, (sic) usually each person integrates at least daily — leading to multiple integrations per day.”

According to Fowler, adopting CI to build new features means you never have an integration failure setback for a long period of time. Failed builds will occur, but CI addresses errors quickly to reduce daily problems and help build cohesion with development, QA, and operations teams. Additional benefits of CI include the following:

• Reduced risk of integration errors and bad builds

• Better communication and collaboration for all stakeholders

• Faster iterations and compatibility with Agile principles

• Speedy feedback loops that include business decisions

• Improved productivity

• Frequent on-time updates

In the book, Continuous Integration: Improving Software Quality and Reducing Risk, author Paul Duvall recommends seven core CI practices for running continuous integration on project:

• Commit code frequently

• Address broken builds immediately

• Automate testing (unit, functional, regression, integration, etc.)

• Integration tests must pass to continue

• Run private code builds

• Avoid broken code

Fowler, who wrote the forward to Duvall’s book in 2007, extends these core practices to include the following practices for teams who wish to use CI:

• Maintain one source repository for version control.

• Automate the build and make it self-testing.

• Every developer commits daily workflows to the main branch.

• Use a separate CI server (machine) for these builds.

• Clone the production environment and test there.

• Keep communication open and make it easy for everyone to see the ongoing work.

• Automate deployment.

DevOps refers to the combination of software development, software testing, and software operations methodology. The DevOps approach to organizing workflows replaces siloed development and IT operations with multidisciplinary teams capable of implementing practice such as continuous integration, delivery, and deployment. For more on the DevOps methodology, including a glossary of common terms, see the The Way of DevOps: A Primer on DevOps Principles and Practices.

The relationship between continuous integration (CI), continuous delivery (CD), and continuous deployment (CDP) with DevOps is conceptual. The concepts share an emphasis on technical practices such as automation (build, testing, and deployment) and the culture of cooperation between the traditionally siloed development and operations teams. However, DevOps emphasizes a broad cultural philosophy, whereas a continuous development concept like CI is a technical practice and component of a more extensive DevOps development environment. Teams that implement CI and CD can do so without adopting a vast DevOps culture by utilizing the principles such as automated software testing (unit testing, functional testing, regression testing, integration integration, API testing, etc.).

The principles of CI are critical to a healthy DevOps environment, and are also valuable to teams that do not practice DevOps methodology as a whole. The factors (for example, using CI, trunk-based development, and version control) that contribute positively to CD do so for DevOps. They both make a strong business case for implementation, as the practices promote improved speed and frequency to enable a competitive advantage in the market.

Continuous delivery is a critical component of successful DevOps software development practices because of the emphasis on automating builds, testing, and the ability to implement continuous deployment to production. Also, by architecting the development environment for CI or CD, a team fulfills the requirements for successful DevOps environments such as readily deployable, previously tested applications that are modified and deployed with little to no human interaction. The benefits of CD are similar to the expected benefits of successful DevOps teams:

• Better reaction time to customer demands

• Advanced reliability of systems

• Reduced project risks with emphasis on shared risk

• Visible cost accountability practices

• Improved flexibility for developers

• Accessibility to teams of all sizes and resource levels

• Enhanced culture of collaboration

• Efficient staffing and workforce management practices

Continuous delivery is the practice of keeping code in a deployable state, ready for the production environment (or in the hands of beta users). CD is an extension of CI practices. It requires continuous integration to keep code deployment routine, predictable, and on demand. The emphasis on automated testing (and automated builds) for quality assurance is essential to the practice. CD helps remove obstacles that prevent the frequent deployment of features, which is the fundamental goal of Agile development.

Jez Humble and David Farley introduced the deployment pipeline in their book Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation. The primary focus of CI practices is the development team. However, releasing reliable software, rapidly and securely, involves software testing (QA), and operations (systems administration). If the operations team's work is delayed by development and testing, or if a bug is discovered after development moves forward, software is not deployable. Delayed feedback between the different teams and individuals and bugs in the code prevent CI development from producing benefits.

Humble and Farley created the deployment pipeline model to provide a more holistic and complete approach to developing, testing, and managing the complexity and change involved in delivering software that’s ready for release with CI practices. With continuous delivery, the deployment pipeline process consists of building the code and following the status of the builds through the various stages of testing and deployment.

The deployment pipeline models the process workflows involved with getting software from check-in to release: "[of] getting a feature from the mind of a customer or user into their hands." Also referred to as a continuous integration pipeline, a build pipeline, or a deployment production line, the deployment pipeline is fundamentally an automated software delivery process, according to Humble and Farley.  

According to the authors, the deployment pipeline approach allows teams to "create, test, and deploy complex systems of higher quality and at significantly lower cost and risk than we could otherwise have done." The purpose of implementing continuous delivery and modeling the process with a deployment pipeline is to get fast feedback by giving everyone visibility into the progress.

Implementing continuous delivery and the deployment pipeline is a complicated process and is not as simple as using "one-size-fits-all" technology. However, the authors recommend using tools such as container management, continuous integration servers, and (automated) release management tools to realize the benefits of CI. Here is a list of the list of tools available in this market:

• AnthillPro

• Apache Continuum

• Apache Gump

• AppVeyor

• AWS

• Bamboo

• Bitbucket Server

• Buddy

• Buildbot

• BuildHive

• BuildMaster

• CABIE

• CircleCI

• Cloud 66 Skycap

• Codefresh

• CruiseControl

• Docker

• Fabasoft app.test

• GitHub

• GitLab

• Harrow

• Hudson

• Jenkins

• Microsoft TFS

• Percy

• Probo

• Semaphore

• TeamCity

• Travis CI

• Vexor

The deployment pipeline depends on more than leveraging tools and technology. CI is a practice, not a tool, so using CI software is not the same as implementing a deployment pipeline. Humble and Farley recommend disciplined CI practices to realize the benefits of continuous delivery:

• Release changes that pass build, test, and deployment automation.

• Use configuration management.

• Automate the application’s build and deployment scripts.

• Use test automation to prove that the application delivers user value.

In addition to the baseline practices included here, Humble and Farley outline steps that serve as a checklist for submitting new code and changes with the continuous deployment pipeline in their book. Also, online communities continue to advance the knowledge and practice of continuous integration and delivery. If you already have the software tools and technical practices in place, refer to resources like the Continuous Delivery maturity matrix and a checklist for measuring the maturity of your deployment pipeline.

Realizing the benefits of continuous delivery and a mature deployment pipeline is challenging enough. But if you plan to implement new software to achieve continuous development practices, there is an added element of risk. Here are two software migration planning templates to help with the process:

Download Software Migration Plan Template

Excel | Smartsheet

Agile software development strives to deliver feature requests and updates to the user quickly and efficiently. Using an iterative programming model, software is developed in increments over a shorter period of time. The role that CD plays during these short increments of time (called sprints) happens during the final stages of the Agile sprint, sometimes referred to as “the last mile.”

The last mile of Agile development is the deployment stage. It emphasizes automation, which helps prevent human error and optimize software. Adding automation to the deployment of software code is what sets continuous delivery apart from continuous deployment. Deployment automation is crucial to keeping Agile project release schedules on time.

Continuous delivery requires collaboration and constant feedback loops, and shares Agile development principles that advocate a centralized view of activity and regular updates on the status of individual tasks and priorities. Applying CD to Agile projects promotes collaboration of those involved in the development of software and those responsible for business requirements.

Adopting technical practices like continuous delivery (and the tools that empower its effectiveness) requires sound change management principles and organizational support of leadership and all stakeholders involved in developing software as an asset. A strong strategy is essential to overcome the challenges associated with implementing practices like CI or CD or using their mechanisms to promote a DevOps environment. Using software tools for implementing these technical practices enables the approach, but they do not indicate successful adoption of the practices. Jez Humble recommends asking the following questions to determine if you have genuinely adopted CD:

1. Does your team check in code to the main branch daily?

2. Does your test suite validate (automatically) these code commits?

3. If something is broken is it the top priority of the entire team to fix it?

In the book Effective DevOps, authors Jennifer Davis and Ryn Daniels discuss common misconceptions about adopting and scaling DevOps technical practices like CI or CD and strategies to troubleshoot the everyday challenges. Here are a few of these strategies:

• Create Shared Understanding Around Goals: Communicate the “why” when changing the development environment and the benefits the team can expect from the changes. Make sure all stakeholders understand the pain points you are looking to solve with CD and focus on the buy-in you earn for solving these problems. Look for opportunities to implement CD, and deliver visible ROI early and often to keep support for adoption high. Then, gradually improve using performance data. Select development workflows for applications with more natural migrations (less risk or adversity) that demonstrate the benefits of speed and frequency that successfully implementing CD practices delivers.

• Encourage a Learning Environment: Introducing new tools is inevitable when implementing effective technical practices like CI and CDP in a DevOps environment. Encourage growth and account for a learning curve when adopting new tools for building code, testing, and deploying to production. Consider organizing multi-disciplinary teams that work together. Or, at a minimum, perform “site visits” to different stakeholders involved in developing a product or feature, including the business stakeholders. Encourage acquisition of technical skills, as well as the soft skills that make adopting new technologies and processes more efficient.

• Design Hiring Practices that Attract the Right Talent: Identify if the current hiring process attracts the right talent to help implement CD. Communicate the technical and soft skills or work history that demonstrate previous DevOps experience or the ability to adapt to the new tools and processes you are implementing with your team. Hiring DevOps professionals with expertise in the methodology and structuring teams with experienced professionals will boost adoption of the core practices.

The preferred frequency of deploying code to production (or to users) is the difference between continuous delivery (CD) and continuous deployment (referred to here as CDP). CDP is achieved when code passes the CI stages and enters production on a large system, production environment, or an application automatically. Automation is critical to continuous deployment. Removing human interaction with code deployment empowers high-velocity deployments (again, ideally daily). Only failed software tests prevent code deployment.

Instagram (acquired by Facebook) deploys code for its web servers around 40 times a day. Each change to the master code branch takes an average of 10 minutes — and, thanks to automated practices, requires no human interaction. Michael Gorven, a Production Engineer at Facebook, identified the following expected benefits of implementing continuous deployment of this scale in a case study presentation for O’Reilly Media, Inc. (Continuous Deployment: How Facebook Deploys to Millions, O’Reilly Velocity Conference, 2016):

• Faster Engineering: Facebook engineers don’t have to wait around to push code to the master branch, which gives them more time. CDP allows engineers to iterate multiple times a day, and this momentum empowers engineers to release code changes whenever they want. There is no limit or fixed schedule for deployments.

• Smaller Haystacks: Committing code changes with CDP makes it easier to identify problems caused by adverse code changes (“bad commits”). Therefore, finding the “needle in the haystack” becomes less of an issue. Since Instagram deploys one commit at a time, it shrinks the haystack and makes it easier to find the problematic changes.

• Limited Delays: Instead of having a big mess of deployments and changes to sort through, CDP prevents broken changes from holding back work. The process of finding and fixing problems is quick, so necessary changes deploy quickly, and Instagram’s code base stays optimized in a state of readiness regardless of issues that arise.

According to Gorven, CDP principles are within reach for teams of all sizes and resources, not just large companies like Facebook. He recommends that you start by focusing on your current mechanisms for deployment so you can evolve with the core principles of CDP and avoid common pitfalls that lead to the high costs with low rewards. He recommends starting with the following fundamental principles to achieve the benefits of CDP:

• Faster Testing: Use a fast, automated test suite, with decent (not perfect) coverage. Run this often during code review, and before and after committing code changes to the master branch.

• Canary Tests: Test production changes on a small scale before deploying to the main code branch. This approach involves using an automated test, known as a canary, deployed to a smaller number of users (or a single machine for testing before going into the main branch of code). This approach addresses small, problematic code changes before they cause bugs in the more extensive system. The canary approach empowers developers to evaluate if the small code commits to provide the desired outcome quickly.

• Comfort Culture: Make the team comfortable with the changes presented with CDP by addressing complexity. Focus on the visibility of everyone’s work, promote natural behavior changes, and use clear mechanisms for deployment.

• Detect and React: Expect that problems will arise and emphasize a culture of reaction to failure, so the focus is on monitoring, detecting, and rolling back issues quickly and efficiently.

The Scaled Agile Framework (SAFe®) provides large-scale enterprise operations a prescriptive Agile framework that accounts for the business goals associated with adapting to Agile software development. Adapting and scaling Agile by applying continuous development practices like CDP to large enterprise software development requires flexibility and commitment to culture change. The SAFe framework recommends some essential CDP practices to deliver value to customers more frequently, including the following:

• Match Development, Testing, and Production Environments: Using version control and automated scripts, make sure to validate development and testing activities in an environment that replicates the actual production environment for deployment. The development and test environment needs to closely emulate the production environment whenever possible, although this is a resource-sensitive practice. SAFe recommends investing in the functional equivalent of the production environment. For example, use whatever machines you have access to as a staging environment, regardless of if the actual number of servers used in production isn’t achievable.   

• Deploy Every Iteration: The goal for continuous deployment is frequency and stability. Every iteration is deployed into the production-like staging environment. The purpose of this practice is to empower the state of readiness, which is critical to shortening the lead time and increasing throughput of deployments to production.   

• Automate Testing: The key to this practice is to ensure the various levels of the product or feature under development work as intended at all levels. Advocates of Agile, DevOps, and the continuous development technical practices believe you should automate anything that you can successfully. For the software testing mechanisms of CDP, automation tools are used to automate unit tests, functional tests, API tests, and more. Find out more about software test automation in this article that covers the basics.

• Automate Deployment: Use Application Release Automation (ARA) tools to automate the process of testing and deploying an application to different environments before the release of the final deployment to production. The benefits of deployment automation with ARA software include reduced error, increased throughput frequency, enhanced system stability, faster troubleshooting, enhanced collaboration, and more visibility for all deployment activities across the release pipeline.

• Decouple Deployment from Release: The “release pipeline” refers to the entire process involved in the technical practices of CI and CDP. Deployment is a practice that occurs at different stages of this process. The critical components of an ARA tool include packaging, creating infrastructure dependencies, promoting to varying stages of deployment (for example, from development to test), deployment automation of specific configurations (data, code, artifacts, etc.), and maintenance documentation.

Several practices exist to achieve continuous software development. Many of the core practices involved emphasize automation (automated testing, code builds, and deployment). Three of these practices, continuous integration (CI), continuous delivery (CD), and continuous deployment (CDP) are closely related development and test activities with distinct differences in how the code is deployed to the production environment.

The “continuous” in both CD and CDP refers to automation, which keeps workflow in the pipeline moving as quickly as possible with the aid of automation. The concepts differ from CI in their shared emphasis on automated process. All three share the same expected benefits, however: increased throughput frequency and stable, ready systems.

The popularity of DevOps and the reported benefits are making a strong business case for implementing continuous development, and this stokes the demand for hiring professionals with the knowledge and skills to apply DevOps technical practices. The opportunities for DevOps engineers with continuous integration experience or system administrators with experience managing and operating applications built with cloud-based, microservices architecture continue to rise in popularity. Learn more about continuous development principles and master the technical practices required to thrive in DevOps environments with the following online resources:

• Lynda.com®:  A subscription-based online learning platform from LinkedIn, Lynda.com offers online video-based courses for software development professionals who want to learn continuous development practices including Continuous Everything, CI/CD Best Practices, Continuous testing, Introduction to Continuous Delivery, and Basics of Continuous Integration. The platform offers courses for DevOps professionals including AWS for DevOps, DevOps Foundations: Continuous Delivery/Continuous Integration, and more.

• Udemy: Udemy offers paid video-based courses in development and IT & software for DevOps professionals, including DevOps Fundamentals, Learn DevOps: CI/CD with Jenkins Using Pipelines and Docker, Continuous Integration Concepts and Tools, and Master Jenkins CI For DevOps and Developers, as well as a career preparation course titled DevOps Interview Questions.  

• DevOps Bookmarks: Peruse a curated list of bookmarks for DevOps-related topics by function (for example CI/CD, virtualization and containers, etc.), platform, software license model, and programming language.

Empower your people to go above and beyond with a flexible platform designed to match the needs of your team — and adapt as those needs change. 

The Smartsheet platform makes it easy to plan, capture, manage, and report on work from anywhere, helping your team be more effective and get more done. Report on key metrics and get real-time visibility into work as it happens with roll-up reports, dashboards, and automated workflows built to keep your team connected and informed. 

When teams have clarity into the work getting done, there’s no telling how much more they can accomplish in the same amount of time. Try Smartsheet for free, today.