Mirrors/grype
2
0
Fork 0
mirror of https://github.com/anchore/grype synced 2024-11-10 06:34:13 +00:00
Code Issues Projects Releases Packages Wiki Activity

docs: update quality gate docs (#1032)

Browse source
This commit is contained in:
Keith Zantow 2022-12-12 15:59:25 -05:00 committed by GitHub
parent a869480f89
commit 2ace4c0b11
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
1 changed files with 182 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

182
test/quality/README.md
View file

@ -14,6 +14,8 @@ static set of reference container images. The kinds of comparisons made are:
## Getting started
For information about required setup see: [Required setup](#required-setup).
To capture raw tool output and store into the local `.yardstick` directory for
further analysis:
```
@ -138,3 +140,183 @@ to keep in mind:
Pushing this ceiling will likely raise the number of unlabled matches
significantly for all images. Only bump this ceiling if all possible matches
are labeled.
## Workflow
One way of working is to simply run `yardstick` and `gate.py` in the `test/quality` directory.
You will need to make sure the `vulnerabilty-match-labels` submodule has been initialized. This happens automatically
for some `make` commands, but you can ensure this by `git submodule update --init`. After the submodule has been
initialized, the match data from `vulnerabilty-match-labels` will be available locally.
**TIP**: when dealing with submodules, it may be convenient to set the git config option `submodule.recurse` to `true`
so `git checkout` will automatically update submodules to the correct commit:
```shell
git config submodule.recurse true
```
To do this we need some results to begin with. As noted above, start with (this does ensure the submodule is initialized):
```shell
make capture
```
This will download prebuilt SBOMs for the configured images and generate match results for configured tools (here:
the previous Grype version as well as the local version).
After `make capture` has finished, we should have results and can now start inspecting and
modifying the comparison labels.
To get started, let's assume we see some quality gate failure in like this (something found in CI
or after running `./gate.py`):
```
Running comparison against labels...
Results used:
├── f4fb4e6e-c911-41b6-9a10-f90b3954a41a : grype@v0.53.1-19-g8900767 against docker.io/anchore/test_images@sha256:808f6cf3cf4473eb39ff9bb47ead639d2ed71255b75b9b140162b58c6102bcc9
└── fcebdd0b-d80a-4fe2-b81a-802c7b98d83b : grype@v0.53.1 against docker.io/anchore/test_images@sha256:808f6cf3cf4473eb39ff9bb47ead639d2ed71255b75b9b140162b58c6102bcc9
Match differences between tooling (with labels):
TOOL PARTITION PACKAGE VULNERABILITY LABEL COMMENTARY
grype@v0.53.1 ONLY node@14.18.2 CVE-2021-44531 TruePositive (this is a new FN 😱)
grype@v0.53.1 ONLY node@14.18.2 CVE-2021-44532 TruePositive (this is a new FN 😱)
grype@v0.53.1 ONLY node@14.18.2 CVE-2021-44533 TruePositive (this is a new FN 😱)
Failed quality gate
- current F1 score is lower than the latest release F1 score: current=0.80 latest=0.80 image=docker.io/anchore/test_images@sha256:808f6cf3cf4473eb39ff9bb47ead639d2ed71255b75b9b140162b58c6102bcc9
- current indeterminate matches % is greater than 10%: current=13.60% image=docker.io/anchore/test_images@sha256:808f6cf3cf4473eb39ff9bb47ead639d2ed71255b75b9b140162b58c6102bcc9
- current false negatives is greater than the latest release false negatives: current=6 latest=3 image=docker.io/anchore/test_images@sha256:808f6cf3cf4473eb39ff9bb47ead639d2ed71255b75b9b140162b58c6102bcc9
```
This tells us some important information: which package, version, and vulnerability had a difference;
how it was previously labeled, and most importantly: the image we need to focus on (`docker.io/anchore/test_images@sha256:808f6cf3cf4473eb39ff9bb47ead639d2ed71255b75b9b140162b58c6102bcc9`).
Using the SHA above, we can run `yardstick` to see which results are available:
```shell
$ yardstick result list --result-set pr_vs_latest_via_sbom | grep 808f6cf3cf4473eb39ff9bb47ead639d2ed71255b75b9b140162b58c6102bcc9
5bf0611b-183f-4525-a1ab-f268f62f48b6 docker.io/anchore/test_images:appstreams-centos-stream-8-1a287dd@sha256:808f6cf3cf4473eb39ff9bb47ead639d2ed71255b75b9b140162b58c6102bcc9 grype@v0.53.1 2022-12-09 20:49:56+00:00
43a9650a-d5de-4687-b3ba-459105e32cb8 docker.io/anchore/test_images:appstreams-centos-stream-8-1a287dd@sha256:808f6cf3cf4473eb39ff9bb47ead639d2ed71255b75b9b140162b58c6102bcc9 grype@v0.53.1-15-gf29a32b 2022-12-09 20:49:53+00:00
67913f57-690f-4f35-a2d9-ffccd2a0b2a1 docker.io/anchore/test_images:appstreams-centos-stream-8-1a287dd@sha256:808f6cf3cf4473eb39ff9bb47ead639d2ed71255b75b9b140162b58c6102bcc9 syft@v0.60.1 2022-11-01 20:30:52+00:00
```
We'll need to use the UUIDs to explore the labels, so copy the first UUID, which we can see was run against the last Grype release (`grype@v0.53.1`). Use the UUID to explore and edit the results with
`yardstick label explore`:
```shell
yardstick label explore 5bf0611b-183f-4525-a1ab-f268f62f48b6
```
At this point we can use the TUI to explore and modify the match data, by deleting things or labeling as
true positives, false positives, etc.. **After making changes make sure to save the results** (`Ctrl-S`)!
At this point you can run the quality gate using updated label data. The quality gate can run against
just one image, for example the image we first found in the failure, so run the quality gate and see
how changes to the label data have affected the result:
```shell
./gate.py --image docker.io/anchore/test_images@sha256:808f6cf3cf4473eb39ff9bb47ead639d2ed71255b75b9b140162b58c6102bcc9
```
After iterating on all the changes we need using `yardstick label explore`, we're now ready to commit changes. Since
we're using `git submodules`, we need to do two steps:
1. get the changes merged to the `vulnerability-match-labels` repository `main` branch
2. update the submodule in this repository
To create a pull request for the `vulnerability-match-labels`, make sure you are in the `vulnerability-match-labels`
subdirectory and create a branch -- something like:
```shell
git checkout --no-track -b my-branch-name
```
Commit the changes to this branch, push, create a pull request like normal. NOTE: you may need to add a fork
(`git remote add ...`) and push to the fork if you don't have push permissions against the main
`vulnerability-match-labels` repo. After the PR is approved and merged to `vulnerability-match-labels` repo's `main`
branch, update the submodule locally using:
```shell
git submodule update --remote
```
Next, _commit the submodule change_ as part of any other changes
to the Grype pull request and push as part of the in-progress PR
against Grype. The PR will now use the updated match labels when running
the quality check.
## Required setup
In order to manage Python versions, [pyenv](https://github.com/pyenv/pyenv) can be used. (e.g. `brew install pyenv`)
Both this project and `yardstick` require Python 3.10.
Using `pyenv`, see which python versions are available, for example:
```shell
$ pyenv install --list|grep 3.10
3.10.0
...
3.10.7
...
```
In this case, we see `3.10.7` is the latest version, so we'll use that for the rest of the setup:
Install this version using `pyenv`:
```shell
pyenv install 3.10.7
```
NOTE: to view the specific Python versions installed use `pyenv versions`:
```shell
$ pyenv versions
system
* 3.8.13 (set by /Users/usr/.pyenv/version)
3.10.7
```
To select the `3.10` version use the exact version number:
```shell
pyenv shell 3.10.7
```
(or maybe just: `pyenv shell $(pyenv versions | grep 3.10 | tail -1)`)
Verify this has worked properly by running:
```shell
python --version
```
**Important:** it is also required to have `oras` installed (e.g. `brew install oras`)
**After** setting the working Python version to 3.10, in the `test/quality` directory,
you need to set up a virtual environment using:
```shell
make venv
```
**After** creating the virtual environment, you can now activate it to set up a
working shell using:
```shell
. venv/bin/activate
```
You should now have a shell running in the correct virtual environment, it might look something
like this:
```shell
(venv) user@HOST quality %
```
Now you should be able to run both `yardstick` and `./gate.py`.
## Troubleshooting
As noted above, yardstick requires Python 3.10. If you try to run with an older version, such as
the default macOS 3.8 version, you will likely see an error similar to:
```
Traceback (most recent call last):
File "./vulnerability-match-labels/sboms.py", line 12, in <module>
import yardstick
File "/grype/test/quality/vulnerability-match-labels/venv/lib/python3.8/site-packages/yardstick/__init__.py", line 4, in <module>
from . import arrange, artifact, capture, cli, comparison, label, store, tool, utils
File "/grype/test/quality/vulnerability-match-labels/venv/lib/python3.8/site-packages/yardstick/arrange.py", line 4, in <module>
from yardstick import artifact
File "/grype/test/quality/vulnerability-match-labels/venv/lib/python3.8/site-packages/yardstick/artifact.py", line 482, in <module>
class ResultSet:
File "/grype/test/quality/vulnerability-match-labels/venv/lib/python3.8/site-packages/yardstick/artifact.py", line 484, in ResultSet
state: list[ResultState] = field(default_factory=list)
TypeError: 'type' object is not subscriptable
```