mirror of
https://github.com/inspec/inspec
synced 2024-11-26 22:50:36 +00:00
fdb789febf
Signed-off-by: Takuya Noguchi <takninnovationresearch@gmail.com> Signed-off-by: Takuya Noguchi <takninnovationresearch@gmail.com>
133 lines
9.4 KiB
Markdown
133 lines
9.4 KiB
Markdown
# About InSpec's Continuous Integration and Continuous Delivery Setup
|
|
|
|
## Major Components
|
|
|
|
### Expeditor
|
|
|
|
[Expeditor](https://expeditor.chef.io/) is the main coordinator of all CI-CD activity at Chef. It is configured through a [YAML file](https://github.com/inspec/inspec/blob/main/.expeditor/config.yml).
|
|
|
|
### BuildKite
|
|
|
|
[Buildkite](https://buildkite.com/chef) is the engine that actually executes all of the tests and builds for InSpec. While Buildkite coordinates all of the work, the actual infrastructure is managed by Chef in a variety of locations, including various clouds and datacenters.
|
|
|
|
Buildkite is divided into two orgs, [Chef](https://buildkite.com/chef) and [Chef-OSS](https://buildkite.com/chef-oss) (Open Source Software). Chef is private and Chef-OSS is public.
|
|
|
|
You will need to have an account on BuildKite and be a member of both orgs to fully utilize the system.
|
|
|
|
### Rakefile
|
|
|
|
The [Rakefile](https://github.com/inspec/inspec/blob/main/Rakefile) defines the tests harness to be run. Most of the test scripts come down to executing "rake test" or similar.
|
|
|
|
### Omnibus
|
|
|
|
Omnibus is a system for building OS-specific packages of software, including all dependencies including Ruby runtimes. We use Omnibus to make RPMs, DEBs, MSIs, DMGs, and several other OS-specific formats that deploy inspec and its dependencies natively to the OS.
|
|
|
|
The omnibus configuration for InSpec is stored at https://github.com/inspec/inspec/tree/master/omnibus and the main configuration file is [inspec.rb](https://github.com/inspec/inspec/blob/main/omnibus/config/projects/inspec.rb).
|
|
|
|
### Rubygems
|
|
|
|
InSpec is published as a set of 4 gems - inspec, inspec-core, inspec-bin, and inspec-core-bin. When we release a new version to the public (a process we call "promotion" and typically happens on Wednesdays), we publish to rubygems.org.
|
|
|
|
Rubygems are configured by declaring them in the expeditor configuration file in the `rubygems` section.
|
|
|
|
### Artifactory
|
|
|
|
Artifactory stores build artifacts (such as RPMs, MSIs, and gems) on a temporary basis. Artifactory is protected by the Progress VPN. [Artifactory inspec package search](http://artifactory.chef.co/ui/packages?name=inspec&type=packages).
|
|
|
|
## What Happens when...
|
|
|
|
### A PR is opened
|
|
|
|
When a PR is opened, the subscription [workload: pull_request_opened:{{agent_id}}:*](https://github.com/inspec/inspec/blob/cb2abf2e10906bba4df24b2ed18ec51b0931eff2/.expeditor/config.yml#L173) is activated. In addition, several defaults also apply.
|
|
|
|
One default pipeline that gets activated is the Buildkite [master verify pipeline](https://buildkite.com/chef-oss/inspec-inspec-master-verify). This default is documented [here](https://expeditor.chef.io/docs/pipelines/verify/).
|
|
|
|
#### verify pipeline
|
|
|
|
The verify pipeline runs the linter, the unit tests, and the functional tests. It verifies that the code being submitted is sound.
|
|
|
|
The verify pipeline is defined first in the [verify.pipeline.yml](https://github.com/inspec/inspec/blob/master/.expeditor/verify.pipeline.yml) file, which defines the separate Ruby versions, platforms, and environment variables to be passed to each one. Each runner calls a shell script, either [verify.sh](https://github.com/inspec/inspec/blob/master/.expeditor/buildkite/verify.sh) or [verify.ps1](https://github.com/inspec/inspec/blob/main/.expeditor/buildkite/verify.ps1). These scripts are thin wrappers that install Bundler dependencies from a cache, then call into the Rakefile.
|
|
|
|
#### habitat artifact pipeline
|
|
|
|
The habitat artifact pipeline runs a smoke test to verify that the habitat build of inspec is valid.
|
|
|
|
The habitat artifact pipeline is defined first in the [artifact.habitat.yml](https://github.com/inspec/inspec/blob/master/.expeditor/artifact.habitat.yml) file. It simply defines a linux runner and a windows runner, each with a dedicated script, [artifact.habitat.test.sh](https://github.com/inspec/inspec/blob/master/.expeditor/buildkite/artifact.habitat.test.sh) or [artifact.habitat.test.ps1](https://github.com/inspec/inspec/blob/master/.expeditor/buildkite/artifact.habitat.test.ps1). The scripts install habitat, setup an origin key, build the package, and then run a [Rakefile](https://github.com/inspec/inspec/blob/main/test/artifact/Rakefile)
|
|
|
|
|
|
### A PR is merged
|
|
|
|
When a PR is merged, the Expeditor actions under `merge_actions` are executed.
|
|
|
|
Watch the Slack channel #inspec-notify for messages about the success or failure of various steps.
|
|
|
|
Connect to the Progress VPN to fetch Expeditor logs in the event of a failure.
|
|
|
|
#### Version Bumping
|
|
|
|
This is controlled by the `built_in:bump_version` and `bash:.expeditor/update_version.sh` subscriptions.
|
|
|
|
If there are no GitHub labels on the PR, the patchlevel of the version will be bumped by executing the [.expeditor/update_version.sh](https://github.com/inspec/inspec/blob/main/.expeditor/update_version.sh) script. First the VERSION file is updated, then the script runs to update the versions in the Ruby files.
|
|
|
|
`built_in:bump_version` is in charge of bumping versions in VERSION, and is controlled by GitHub labels on the PR. Most, though not all, PRs should not have any Expeditor control labels.
|
|
|
|
Here are the Expeditor control labels, and the circumstances under which they should be used:
|
|
|
|
* Expeditor: Bump Minor Version - Use when a significant new feature is being released.
|
|
* Expeditor: Bump Major Version - Use when a major release is made - rarely used.
|
|
* Expeditor: Skip Version Bump - Use for non-code-change PRs, such as website or CI changes.
|
|
|
|
#### Build Omnibus Packages
|
|
|
|
This is controlled by the `trigger_pipeline:omnibus/release` subscription.
|
|
|
|
The Omnibus build creates operating-system-specific packages for each platform on which we release Chef InSpec. Its [expeditor configuration](https://github.com/inspec/inspec/blob/44fe144732e1e0abb2594957a880c5f1821e7774/.expeditor/config.yml#L133) drives a [Buildkite configuration](https://github.com/inspec/inspec/blob/main/.expeditor/release.omnibus.yml), which lists exactly which platforms to build.
|
|
|
|
The Omnibus build is generally reliable, if somewhat slow.
|
|
|
|
When the omnibus build succeeds, omnitruck delivers the packages to various package repos in `unstable` channels for public consumption. The packages are also delivered to [Artifactory](http://artifactory.chef.co/ui/repos/tree/General/omnibus-unstable-local%2Fcom%2Fgetchef%2Finspec) (VPN required)
|
|
|
|
#### Chef Habitat Build
|
|
|
|
The Chef Habitat build creates Habitat .hart packages for Linux and Windows. The [Expeditor configuration](https://github.com/inspec/inspec/blob/44fe144732e1e0abb2594957a880c5f1821e7774/.expeditor/config.yml#L138) drives a [Buildkite configuration](https://github.com/inspec/inspec/blob/main/.expeditor/build.habitat.yml).
|
|
|
|
When the hab build succeeds, the packages will be placed on the Hab builder in the `unstable` channel for public consumption.
|
|
|
|
#### Docker Image Built and Released
|
|
|
|
We also release a Docker image (see [expeditor config](https://github.com/inspec/inspec/blob/44fe144732e1e0abb2594957a880c5f1821e7774/.expeditor/config.yml#L150)), which contains a Linux system and Chef InSpec installed from a gem, with the ENTRYPOINT of the Docker image being `inspec` (see [Dockerfile](https://github.com/inspec/inspec/blob/main/Dockerfile)). It's a simple way to ship the dependencies of `inspec`.
|
|
|
|
When it succeeds, the Docker build is labeled as `current`.
|
|
|
|
#### Gems Built and Placed on Artifactory
|
|
|
|
The `inspec`, `inspec-bin`, `inspec-core`, and `inspec-core-bin` gems are all built and placed on the internal Chef [Artifactory](http://artifactory.chef.co/ui/packages?name=inspec&type=packages) (VPN required) server. During promotion later, they publish to rubygems.org.
|
|
|
|
The difference between the gems is as follows:
|
|
|
|
* `inspec` is a library gem, with full heavyweight dependencies, not encumbered by commercial licensing
|
|
* `inspec-bin` contains an `inspec` executable and is encumbered by commercial licensing
|
|
* `inspec-core` is a library gem, with lightweight dependencies and no compilation required at install time, and is not encumbered by commercial licensing
|
|
* `inspec-core-bin` contains an `inspec` executable and is encumbered by commercial licensing
|
|
|
|
### A release is promoted
|
|
|
|
When expeditor is told to promote a release, using the slack command `/expeditor promote inspec/inspec:master 4.36.4` (for example), Expeditor automatically promotes the Omnibus packages from the unstable channel to the stable channel, publishing them to the various downloads sites. It also creates the `artifact_published:stable` event, which has numerous [actions subscribed](https://github.com/inspec/inspec/blob/8a93f08a13d6bde8f87e447ff4246801bef80f8c/.expeditor/config.yml#L158).
|
|
|
|
Some of the more important ones:
|
|
|
|
#### Update and publish the docker image
|
|
|
|
The Dockerfile is updated - mainly to update version numbers - and then the published Docker image is tagged with the labels "stable" and "latest".
|
|
|
|
#### Rubygems are published to rubygems,org
|
|
|
|
The gems are taken from Artifactory and published to Rubygems.org. This is done using an Expeditor built-in action. The gems must be owned by the user `chef`.
|
|
|
|
#### Release notes are published
|
|
|
|
The [pending release notes](https://github.com/inspec/inspec/wiki/Pending-Release-Notes) are copied to AWS S3 by a [script](https://github.com/inspec/inspec/blob/master/.expeditor/publish-release-notes.sh), and then reset back to an empty state. Another [script](https://github.com/inspec/inspec/blob/main/.expeditor/announce-release.sh) takes the release notes from S3 and creates a post on Discourse.
|
|
|
|
|
|
|
|
|