Tag Archives: ci/cd

Four Best Practices of CI Builds

Continuous integration builds (also called CI builds) are an absolute process game-changer, turning a local ‘works on my box’ coding and deployment strategy into an automatic and reliable process. Once a team gets a taste of an automated and reliable build process, it is easy to fall into a trap of adding all of the things to your CI build. Integration tests, security scans, performance tests, etc, until eventually, the CI build has lost the features that make a CI build implicitly valuable. The best development teams keep continuous integration builds simple.

Best Practice: With no changes to code or steps, the result of CI builds should be the same.

Responsibilities of the CI Build

A CI build’s job is to collect all developer changes, and create a releasable artifact, ensuring changes from all developers on a project are included in the product, and don’t cause conflicts with each other. A CI build must be fast, so that changes that break a build can be immediately corrected; a quick feedback loop is essential to the CI process. A CI build should perform the following steps:

  1. Collect all source code and changes.
  2. Restore dependencies.
  3. Compile / construct the build.
  4. Run unit tests and build verification tests.
  5. Create the artifact.

Assuming the same code input from step one, and no changes to any of the steps, a CI build’s result should be the same, every time.

A new CI build should be executed if the state of any of the five items above are modified. If there’s a code change anywhere, step one has changed. If any dependencies are updated, in the framework or libraries the product uses, step two has changed. If a build setting has changed, like going from 32 bit to 64 bit, or from ‘debug’ mode to ‘release’ mode, that is step three. Unit test changes typically are manifested as code changes, but any test settings changed would be an item for step four. If the artifact construction process changes at all, that’s step five.

Best Practice: Execute a new build on any individual change to those steps.

Tempting Steps You Should Avoid in CI Builds

  • Code Security Checks
  • Integration Tests

Code security checks are tempting items to add to a CI build, but fail the ‘no changes to code or steps’ rule. Most code security checks are rule-based in nature, getting updates when there is a new known vulnerability. The process is similar to a malware database updates. With no code change, and no change to steps, but an invisible-to-the-developer modification to the security changes, builds can succeed or fail seemingly randomly. Also, the scope of secure code changes can be larger than common unit / build verification failures. A security change can be very small, but more commonly is larger and design-related.

Best practice: Include code security checks on a regular schedule, not directly connected to the CI build, and have those check failures create backlog items the developers can research and correct.

Integration tests are tempting to add to a CI build, but fail the rule of failing fast. Any system large enough to have integration tests in the first place will be a fairly complex set of tests to run, which may take quite a while to execute fully. They also tend to rely on a lot of scripting and setup work.

Best practice: Include integration tests as a post-CI step in an automated pipeline separate from the CI build.

To recap: Continuous Integration builds should be quick, automated, and provide a fast feedback loop. They should contain the following steps.

  1. Collect all source code and changes from the repo.
  2. Restore any dependencies.
  3. Compile / construct the build.
  4. Execute Unit Tests and Build Verification tests.
  5. Create the artifact.

Remember the best practices of good CI builds:

  1. With no changes to code or steps, the result of CI builds should be the same.
  2. Execute a new CI Build on any changes to the five steps.
  3. Include code security checks on a regular schedule, not directly connected to the CI build, and have those check failures create backlog items the developers can research and correct.
  4. Include integration tests as a post-CI step in an automated pipeline separate from the CI build.

Happy building!

Coaching Session – Panic at the Deadline

I had a conversation with a new dev lead today. This lead was recently installed into a project with an upcoming deadline that he was genuinely worried about. As we started the conversation, the worry sort of came out of him in piles. When he was finally able to take a breath, I collected from him that 1) he believes that the code-base is largely un-salvageable (he only recently joined the team) and 2) the deadline is completely rushing at him (about a week and a half before the first delivery.) I had spent a few minutes looking through his project and coached him toward doing the following.

1. Get a CI / CD Pipeline Setup

The first thing to do here is to get a Continuous Integration and Delivery pipeline setup, so you can quickly and reliably deploy in an automated fashion. The fact is, no software comes out as a 100% bug-free product the first time and this software will be no exception. By building up the CI pipeline you’ll be able to fix things quickly and consistently. Every commit should create an artifact, and assuming success that artifact is something can release. If the software isn’t good right now, the product can get good over time.

This was the bulk of the conversation, as it felt like to him that I was recommending that he throw sh*t at a wall and hope that it sticks. The big value comes from the ability to iterate.

2. Code Doesn’t Have to be Perfect.

The fact is, when you have a deadline you have already agreed to, you have to get over the fact that the code is crap, and likely will need a rewrite. Code is a tool used to create a product, nothing more, so worrying that it’s “not good enough” is contra-indicated.

The dirty truth about software is that no code is good enough. Ever. There is always something to do better and cleaner. That is not to disparage the craftsmen out there, but a craftsman does not back out of a commitment. When you make a commitment, you keep it.

3. Keep Your Campground Clean

This one was fairly specific to the project, but in development projects un-merged branches and directories all over the place that are largely empty do not help anyone, and make the project look bigger than it is. If you have a REST API and a daemon, you don’t need a ‘Tests\Python’ folder that has nothing in it. Things that look small, will feel small.

I will check in with him again after a few days to see what progress he has made on those items. If he can get the CI/CD pipeline down, and know HOW to get the code into production boxes quickly, we can move into things like refactoring into patterns that might smooth some sailing here.