Chris Murman

26 April 2021

A Definitive Guide to Understanding the CI/CD Pipeline

The CI/CD pipeline is a foundational part of any DevOps practice. DevOps principles can't become operational until you have a CI/CD pipeline in place.

Understanding and implementing a pipeline can be a challenge for several reasons. One is uncertainty about which processes and practices comprise the pipeline. Another is which tools to select when building a pipeline. CI/CD tools vary depending on where your pipeline deploys. Even figuring out what the acronym CI/CD stands for can be confusing. CD can read as Continuous Delivery, Continuous Deployment, or Continuous Development.

If you're unsure what, exactly, a CI/CD pipeline is, why it's important and how to put it in place, keep reading. This article explains everything you need to know to get started with CI/CD pipelines as part of your DevOps journey.

What Is CI/CD?

Definition: CI/CD is a set of processes used to create, deploy, manage and update applications.

The acronym CI/CD is shorthand for Continuous Integration/Continuous Delivery. Some will say the CD bit refers to Continuous Deployment or Continuous Development. The latter is a little different from but relates to Continuous Delivery.

It's best to think of CI/CD as the set of processes by which DevOps teams deliver. The end result is the rapid delivery of applications and updates into production.

What Is a CI/CD Pipeline?

While CI/CD is the set of processes used to deliver software, the pipeline is what you get when you integrate those processes. It makes it easy to transition between one process and another.

You don't need a CI/CD pipeline to deliver software. You could manage each process of the pipeline separately, and manually hand off code between one stage and the next. But that would be inefficient and leave you at risk of errors.

By building a CI/CD pipeline that uses automation tools and streamlined processes to integrate discrete processes together, organizations achieve faster rates of innovation and fewer instances of friction.

Which Processes Comprise CI/CD Pipelines?

The term CI/CD can be a bit misleading in that it implies that there are only two main processes at play:

1. Continuous Integration means the integration of new code into an existing codebase.
2. Continuous Delivery means the rapid delivery of software into production environments.

Although CI and CD are the two core processes in a CI/CD pipeline, they're not the only ones. A complete pipeline consists of more than a half-dozen different processes:

• Planning: Plan the application features or changes that stakeholders want to implement.
  
• Coding: Code the desired changes from development team members.
  
• Integration: Integrate new code by individual developers into the main, shared codebase.
  
• Building: After code updates are complete, compile the modified codebase into binaries.
  
• Testing: Test application binaries to determine whether it meets performance and reliability standards.
  
• Deployment: Deploy the application into production once it passes testing to allow end-user access.
  
• Monitoring: Monitor and manage the new release to address any issues in production.
  
• Feedback: Share feedback with developers to help inform the next round of application updates.

The feedback stage of the pipeline connects back to the planning stage. That allows it to function more like a loop vs a mono-directional pipeline. Each process feeds into the next one until the cycle starts all over again.

A never-ending chain of application updates and innovation.

Continuous, Concurrent CI/CD Processes

The processes in a well-managed pipeline are continuous. They are also concurrent–in the sense that each process happens at the same time as other processes.

A CI/CD pipeline differs from predecessors in that processes can happen without delays. Waterfall, for example, involved waiting for one stage to complete before the next could start.

In CI/CD, developers can work on new code today even while what they wrote yesterday flows down the pipeline. There are no major hold-ups or bottlenecks within the software delivery process.

Canary Deployments and CI/CD

The deployment process can in some cases split into many processes. This happens if teams use a technique known as canary deployments.

A canary model deploys a new release to a certain subset of users (the so-called canary group). Others keep running the previous release until the new one is stable enough to push out to the entire group.

Teams may choose to use a canary model because it limits the impact of a bad application update. It reduces some of the risks associated with pushing out application updates.

However, it does add complexity to the deployment part of the CI/CD pipeline. Teams have to deploy the same release many times: First to the canary group, then to the rest of the user base.

The lesson here is that the deployment process can be more complex than it appears at first glance.

Why Is the CI/CD Pipeline Important?

We've hinted above at the reasons why CI/CD pipelines matter, but let's spell them out here.

There are several factors that make CI/CD pipelines important:

• Velocity: CI/CD pipelines increase the rate at which teams can deliver software. Leading DevOps organizations are able to update applications hundreds of times per day. Traditional software delivery methodologies achieve only a few releases per year.
  
• Gradual change becomes more practical. Changes are pushed out in small batches rather than overhauling your entire application. This makes for a smoother user experience. Your customers experience gradual change instead of wading through a major update.
  
• Consistency comes from breaking delivery into a set of integrated and automated processes. CI/CD pipelines make the work of development and deployment consistent and smooth. There is less room for engineers to act in different ways, which in turn reduces the likelihood of errors.
  
• Rollbacks are helpful in situations where stability or security problems arise in production. A well-designed pipeline enables teams to revert to an earlier version.
  
• Security is not a core focus of CI/CD. That said, pipelines tend to have the effect of making applications more secure. They do this by systematizing the way you write code, test, and deploy it. Which, in turn, reduces the risk of oversights that could invite security problems. It is also possible to integrate security into the CI/CD pipeline. This is a practice associated with DevSecOps.

Tools for Building a CI/CD Pipeline

There are no hard and fast rules on which tools a team needs to use to install a CI/CD pipeline. In general, there are several essential categories of tools that most pipelines include:

• Source code management solutions, which help developers organize source code. Git is a popular tool in this category.
  
• Continuous Integration (CI) servers–as explained above, these tools integrate new code into an existing codebase. There are a variety of CI servers available, such as Jenkins and CircleCI.
  
• Build automation tools, which compile source code into binaries.
  
• Test automation software like Selenium lets developers write and run behavior validation.
  
• Deployment automation software for pushing new application releases into production environments.
  
• Monitoring or observability software for tracking the performance of applications in production.

In some cases, the same tool could fill more than one of the roles described above. For example, many Continuous Integration servers can perform build automation, deployment automation, and code integration.

Note, too, that a team may choose to use many tools at once in the same category. Test automation frameworks could excel at evaluating different aspects of your application builds.

The tools you use to build a pipeline can vary depending on your needs, preferences, and tools.

Tools That Complement the Pipeline

Teams may opt to take advantage of solutions beyond the core tools that comprise a pipeline. These tools don't perform core CI/CD processes. Instead, they complement or enhance them.

These tools can scan source code or containers for vulnerabilities. Code is then automatically vetted as it flows through the pipeline. This is one way to achieve closer integration between CI/CD and security.

Collaboration tools represent another key category that is not part of the pipeline. It is vital for any team that wants to perform CI/CD. Developers and IT engineers use software like Slack or Microsoft Teams to communicate. They can also use automation to integrate them into the pipeline. It helps to create notifications or alerts based on the status of the pipeline.

On-Prem Vs. Cloud-Based Pipelines

CI/CD pipelines can deliver any type of application. They can support both cloud and on-premises deployment environments.

If you host your pipeline in the cloud, take advantage of ready-made solutions. That allows you to install a pipeline using managed services. The Microsoft Azure cloud offers Azure DevOps. It provides most of the functionality teams need to do CI/CD using a SaaS architecture. AWS CodePipeline is a similar solution to the Amazon cloud.

It's possible to build your own CI/CD pipeline within a public cloud by setting up the individual CI/CD tools of your choosing. But the turnkey pipeline services offered by cloud providers are an attractive solution for teams that want to get a pipeline up and running quickly.

If you use an on-prem pipeline, you will have to set up the tooling by installing a CI server. It utilizes build and test automation tools within your on-prem environment. Certain all-in-one CI/CD platforms can help on-premises. GitLab offers a self-hosted option. Also, you can use Azure DevOps Server (an on-prem version of Azure DevOps).

Conclusion: The Pipeline Is What You Make it

While the CI/CD pipeline is a core component of any DevOps practice, it is also a flexible one. Every pipeline includes a key set of basic CI/CD processes. Some teams choose to use more complex ones by using techniques such as canary deployment. There is also a great deal of flexibility in which specific tools you use to build your CI/CD pipeline.

The essential takeaway is that your organization's needs should drive the pipeline.