cataloger configuration is respected regardless of source (#1142)

README.md

@@ -95,7 +95,7 @@ nix-shell -p syft
 
 ## Getting started
 
-#### SBOM
+### SBOM
 
 To generate an SBOM for a container image:
 
@@ -109,67 +109,9 @@ The above output includes only software that is visible in the container (i.e.,
 syft <image> --scope all-layers
 ```
 
-#### Format conversion (experimental)
 
-The ability to convert existing SBOMs means you can create SBOMs in different formats quickly, without the need to regenerate the SBOM from scratch, which may take significantly more time.
 
-```
+## Supported sources
-syft convert <ORIGINAL-SBOM-FILE> -o <NEW-SBOM-FORMAT>[=<NEW-SBOM-FILE>]
-```
-
-This feature is experimental and data might be lost when converting formats. Packages are the main SBOM component easily transferable across formats, whereas files and relationships, as well as other information Syft doesn't support, are more likely to be lost.
-
-We support formats with wide community usage AND good encode/decode support by Syft. The supported formats are:
-- Syft JSON
-- SPDX 2.2 JSON
-- SPDX 2.2 tag-value
-- CycloneDX 1.4 JSON
-- CycloneDX 1.4 XML
-
-Conversion example:
-```sh
-syft alpine:latest -o syft-json=sbom.syft.json # generate a syft SBOM
-syft convert sbom.syft.json -o cyclonedx-json=sbom.cdx.json  # convert it to CycloneDX
-```
-
-#### SBOM attestation
-
-### Keyless support
-Syft supports generating attestations using cosign's [keyless](https://github.com/sigstore/cosign/blob/main/KEYLESS.md) signatures.
-
-To use this feature with a format like CycloneDX json simply run:
-```
-syft attest --output cyclonedx-json <IMAGE WITH OCI WRITE ACCESS>
-```
-This command will open a web browser and allow the user to authenticate their OIDC identity as the root of trust for the attestation (Github, Google, Microsoft).
-
-After authenticating, Syft will upload the attestation to the OCI registry specified by the image that the user has write access to.
-
-You will need to make sure your credentials are configured for the OCI registry you are uploading to so that the attestation can write successfully.
-
-Users can then verify the attestation(or any image with attestations) by running:
-```
-COSIGN_EXPERIMENTAL=1 cosign verify-attestation <IMAGE_WITH_ATTESTATIONS>
-```
-
-Users should see that the uploaded attestation claims are validated, the claims exist within the transparency log, and certificates on the attestations were verified against [fulcio](https://github.com/SigStore/fulcio).
-There will also be a printout of the certificates subject `<user identity>` and the certificate issuer URL: `<provider of user identity (Github, Google, Microsoft)>`:
-```
-Certificate subject:  test.email@testdomain.com
-Certificate issuer URL:  https://accounts.google.com
-```
-
-### Local private key support
-
-To generate an SBOM attestation for a container image using a local private key:
-```
-syft attest --output [FORMAT] --key [KEY] [SOURCE] [flags]
-```
-
-The above output is in the form of the [DSSE envelope](https://github.com/secure-systems-lab/dsse/blob/master/envelope.md#dsse-envelope).
-The payload is a base64 encoded `in-toto` statement with the generated SBOM as the predicate. For details on workflows using this command see [here](#adding-an-sbom-to-an-image-as-an-attestation-using-syft).
-
-### Supported sources
 
 Syft can generate a SBOM from a variety of sources:
 
@@ -195,6 +137,47 @@ file:path/to/yourproject/file            read directly from a path on disk (any
 registry:yourrepo/yourimage:tag          pull image directly from a registry (no container runtime required)
 ```
 
+#### Default Cataloger Configuration by scan type
+
+##### Image Scanning:
+- alpmdb
+- rpmdb
+- dpkgdb
+- apkdb
+- portage
+- ruby-gemspec
+- python-package
+- php-composer-installed Cataloger
+- javascript-package
+- java
+- go-module-binary
+- dotnet-deps
+
+##### Directory Scanning:
+- alpmdb
+- apkdb
+- dpkgdb
+- portage
+- rpmdb
+- ruby-gemfile
+- python-index
+- python-package
+- php-composer-lock
+- javascript-lock
+- java
+- java-pom
+- go-module-binary
+- go-mod-file
+- rust-cargo-lock
+- dartlang-lock
+- dotnet-deps
+- cocoapods
+- conan
+- hackage
+
+#### Non Default:
+- cargo-auditable-binary
+
 ### Excluding file paths
 
 Syft can exclude files and paths from being scanned within a source by using glob expressions
@@ -232,7 +215,7 @@ Where the `formats` available are:
 - `table`: A columnar summary (default).
 - `template`: Lets the user specify the output format. See ["Using templates"](#using-templates) below.
 
-#### Using templates
+## Using templates
 
 Syft lets you define custom output formats, using [Go templates](https://pkg.go.dev/text/template). Here's how it works:
 
@@ -265,7 +248,7 @@ Which would produce output like:
 
 Syft also includes a vast array of utility templating functions from [sprig](http://masterminds.github.io/sprig/) apart from the default Golang [text/template](https://pkg.go.dev/text/template#hdr-Functions) to allow users to customize the output format.
 
-#### Multiple outputs
+## Multiple outputs
 
 Syft can also output _multiple_ files in differing formats by appending
 `=<file>` to the option, for example to output Syft JSON and SPDX JSON:
@@ -315,7 +298,7 @@ Here's a simple workflow to mount this config file as a secret into a container
       config.json: <base64 encoded config.json>
     ```
 
-    `kubectl apply -f secret.yaml`
+   `kubectl apply -f secret.yaml`
 
 
 2. Create your pod running syft. The env `DOCKER_CONFIG` is important because it advertises where to look for the credential file. In the below example, setting `DOCKER_CONFIG=/config` informs syft that credentials can be found at `/config/config.json`. This is why we used `config.json` as the key for our secret. When mounted into containers the secrets' key is used as the filename. The `volumeMounts` section mounts our secret to `/config`. The `volumes` section names our volume and leverages the secret we created in step one.
@@ -346,13 +329,74 @@ Here's a simple workflow to mount this config file as a secret into a container
           secretName: registry-config
     ```
 
-    `kubectl apply -f pod.yaml`
+   `kubectl apply -f pod.yaml`
 
 
 3. The user can now run `kubectl logs syft-private-registry-demo`. The logs should show the Syft analysis for the `<private_image>` provided in the pod configuration.
 
 Using the above information, users should be able to configure private registry access without having to do so in the `grype` or `syft` configuration files.  They will also not be dependent on a Docker daemon, (or some other runtime software) for registry configuration and access.
 
+## Format conversion (experimental)
+
+The ability to convert existing SBOMs means you can create SBOMs in different formats quickly, without the need to regenerate the SBOM from scratch, which may take significantly more time.
+
+```
+syft convert <ORIGINAL-SBOM-FILE> -o <NEW-SBOM-FORMAT>[=<NEW-SBOM-FILE>]
+```
+
+This feature is experimental and data might be lost when converting formats. Packages are the main SBOM component easily transferable across formats, whereas files and relationships, as well as other information Syft doesn't support, are more likely to be lost.
+
+We support formats with wide community usage AND good encode/decode support by Syft. The supported formats are:
+- Syft JSON
+- SPDX 2.2 JSON
+- SPDX 2.2 tag-value
+- CycloneDX 1.4 JSON
+- CycloneDX 1.4 XML
+
+Conversion example:
+```sh
+syft alpine:latest -o syft-json=sbom.syft.json # generate a syft SBOM
+syft convert sbom.syft.json -o cyclonedx-json=sbom.cdx.json  # convert it to CycloneDX
+```
+
+## Attestation (experimental)
+### Keyless support
+Syft supports generating attestations using cosign's [keyless](https://github.com/sigstore/cosign/blob/main/KEYLESS.md) signatures.
+
+To use this feature with a format like CycloneDX json simply run:
+```
+syft attest --output cyclonedx-json <IMAGE WITH OCI WRITE ACCESS>
+```
+This command will open a web browser and allow the user to authenticate their OIDC identity as the root of trust for the attestation (Github, Google, Microsoft).
+
+After authenticating, Syft will upload the attestation to the OCI registry specified by the image that the user has write access to.
+
+You will need to make sure your credentials are configured for the OCI registry you are uploading to so that the attestation can write successfully.
+
+Users can then verify the attestation(or any image with attestations) by running:
+```
+COSIGN_EXPERIMENTAL=1 cosign verify-attestation <IMAGE_WITH_ATTESTATIONS>
+```
+
+Users should see that the uploaded attestation claims are validated, the claims exist within the transparency log, and certificates on the attestations were verified against [fulcio](https://github.com/SigStore/fulcio).
+There will also be a printout of the certificates subject `<user identity>` and the certificate issuer URL: `<provider of user identity (Github, Google, Microsoft)>`:
+```
+Certificate subject:  test.email@testdomain.com
+Certificate issuer URL:  https://accounts.google.com
+```
+
+#### Local private key support
+
+To generate an SBOM attestation for a container image using a local private key:
+```
+syft attest --output [FORMAT] --key [KEY] [SOURCE] [flags]
+```
+
+The above output is in the form of the [DSSE envelope](https://github.com/secure-systems-lab/dsse/blob/master/envelope.md#dsse-envelope).
+The payload is a base64 encoded `in-toto` statement with the generated SBOM as the predicate. For details on workflows using this command see [here](#adding-an-sbom-to-an-image-as-an-attestation-using-syft).
+
+
+
 ## Configuration
 
 Configuration search paths:

syft/lib.go

@@ -48,24 +48,25 @@ func CatalogPackages(src *source.Source, cfg cataloger.Config) (*pkg.Catalog, []
 		log.Info("could not identify distro")
 	}
 
-	// conditionally use the correct set of loggers based on the input type (container image or directory)
+	// if the catalogers have been configured, use them regardless of input type
 	var catalogers []cataloger.Cataloger
-	switch src.Metadata.Scheme {
+	if len(cfg.Catalogers) > 0 {
-	case source.ImageScheme:
-		log.Info("cataloging image")
-		catalogers = cataloger.ImageCatalogers(cfg)
-	case source.FileScheme:
-		log.Info("cataloging file")
-		catalogers = cataloger.AllCatalogers(cfg)
-	case source.DirectoryScheme:
-		log.Info("cataloging directory")
-		catalogers = cataloger.DirectoryCatalogers(cfg)
-	default:
-		return nil, nil, nil, fmt.Errorf("unable to determine cataloger set from scheme=%+v", src.Metadata.Scheme)
-	}
-
-	if cataloger.RequestedAllCatalogers(cfg) {
 		catalogers = cataloger.AllCatalogers(cfg)
+	} else {
+		// otherwise conditionally use the correct set of loggers based on the input type (container image or directory)
+		switch src.Metadata.Scheme {
+		case source.ImageScheme:
+			log.Info("cataloging image")
+			catalogers = cataloger.ImageCatalogers(cfg)
+		case source.FileScheme:
+			log.Info("cataloging file")
+			catalogers = cataloger.AllCatalogers(cfg)
+		case source.DirectoryScheme:
+			log.Info("cataloging directory")
+			catalogers = cataloger.DirectoryCatalogers(cfg)
+		default:
+			return nil, nil, nil, fmt.Errorf("unable to determine cataloger set from scheme=%+v", src.Metadata.Scheme)
+		}
 	}
 
 	catalog, relationships, err := cataloger.Catalog(resolver, release, catalogers...)

syft/pkg/cataloger/rust/audit_binary_cataloger.go

@@ -12,7 +12,7 @@ import (
 	rustaudit "github.com/microsoft/go-rustaudit"
 )
 
-const catalogerName = "rust-audit-binary-cataloger"
+const catalogerName = "cargo-auditable-binary-cataloger"
 
 type Cataloger struct{}
 

syft/pkg/cataloger/rust/cataloger.go

@@ -13,5 +13,5 @@ func NewCargoLockCataloger() *common.GenericCataloger {
 		"**/Cargo.lock": parseCargoLock,
 	}
 
-	return common.NewGenericCataloger(nil, globParsers, "rust-cataloger")
+	return common.NewGenericCataloger(nil, globParsers, "rust-cargo-lock-cataloger")
 }

test/cli/packages_cmd_test.go

@@ -229,9 +229,10 @@ func TestPackagesCmdFlags(t *testing.T) {
 		},
 		{
 			name: "catalogers-option",
+			// This will detect enable python-index-cataloger, python-package-cataloger and ruby-gemspec cataloger
 			args: []string{"packages", "-o", "json", "--catalogers", "python,ruby-gemspec", coverageImage},
 			assertions: []traitAssertion{
-				assertPackageCount(6),
+				assertPackageCount(13),
 				assertSuccessfulReturnCode,
 			},
 		},

test/integration/catalog_packages_test.go

@@ -3,6 +3,7 @@ package integration
 import (
 	"testing"
 
+	"github.com/stretchr/testify/assert"
 	"github.com/stretchr/testify/require"
 
 	"github.com/anchore/syft/syft/linux"
@@ -54,7 +55,7 @@ func BenchmarkImagePackageCatalogers(b *testing.B) {
 }
 
 func TestPkgCoverageImage(t *testing.T) {
-	sbom, _ := catalogFixtureImage(t, "image-pkg-coverage", source.SquashedScope, false)
+	sbom, _ := catalogFixtureImage(t, "image-pkg-coverage", source.SquashedScope, nil)
 
 	observedLanguages := internal.NewStringSet()
 	definedLanguages := internal.NewStringSet()
@@ -221,3 +222,24 @@ func TestPkgCoverageDirectory(t *testing.T) {
 		t.Errorf("package coverage incomplete (packages=%d, coverage=%d)", len(definedPkgs), len(observedPkgs))
 	}
 }
+
+func TestPkgCoverageCatalogerConfiguration(t *testing.T) {
+	// Check that cataloger configuration can be used to run a cataloger on a source
+	// for which that cataloger isn't enabled by defauly
+	sbom, _ := catalogFixtureImage(t, "image-pkg-coverage", source.SquashedScope, []string{"rust"})
+
+	observedLanguages := internal.NewStringSet()
+	definedLanguages := internal.NewStringSet()
+	definedLanguages.Add("rust")
+
+	for actualPkg := range sbom.Artifacts.PackageCatalog.Enumerate() {
+		observedLanguages.Add(actualPkg.Language.String())
+	}
+
+	assert.Equal(t, definedLanguages, observedLanguages)
+
+	// Verify that rust isn't actually an image cataloger
+	c := cataloger.DefaultConfig()
+	c.Catalogers = []string{"rust"}
+	assert.Len(t, cataloger.ImageCatalogers(c), 0)
+}

test/integration/convert_test.go

@@ -36,7 +36,7 @@ var convertibleFormats = []sbom.Format{
 func TestConvertCmd(t *testing.T) {
 	for _, format := range convertibleFormats {
 		t.Run(format.ID().String(), func(t *testing.T) {
-			sbom, _ := catalogFixtureImage(t, "image-pkg-coverage", source.SquashedScope, false)
+			sbom, _ := catalogFixtureImage(t, "image-pkg-coverage", source.SquashedScope, nil)
 			format := syft.FormatByID(syftjson.ID)
 
 			f, err := ioutil.TempFile("", "test-convert-sbom-")

test/integration/distro_test.go

@@ -1,16 +1,17 @@
 package integration
 
 import (
-	"github.com/anchore/syft/syft/source"
 	"testing"
 
+	"github.com/anchore/syft/syft/source"
+
 	"github.com/stretchr/testify/assert"
 
 	"github.com/anchore/syft/syft/linux"
 )
 
 func TestDistroImage(t *testing.T) {
-	sbom, _ := catalogFixtureImage(t, "image-distro-id", source.SquashedScope, false)
+	sbom, _ := catalogFixtureImage(t, "image-distro-id", source.SquashedScope, nil)
 
 	expected := &linux.Release{
 		PrettyName: "BusyBox v1.31.1",

test/integration/encode_decode_cycle_test.go

@@ -3,13 +3,14 @@ package integration
 import (
 	"bytes"
 	"fmt"
+	"regexp"
+	"testing"
+
 	"github.com/anchore/syft/internal/formats/cyclonedxjson"
 	"github.com/anchore/syft/internal/formats/cyclonedxxml"
 	"github.com/anchore/syft/internal/formats/syftjson"
 	"github.com/anchore/syft/syft/source"
 	"github.com/google/go-cmp/cmp"
-	"regexp"
-	"testing"
 
 	"github.com/anchore/syft/syft/sbom"
 	"github.com/stretchr/testify/require"
@@ -64,7 +65,7 @@ func TestEncodeDecodeEncodeCycleComparison(t *testing.T) {
 	for _, test := range tests {
 		t.Run(fmt.Sprintf("%s", test.formatOption), func(t *testing.T) {
 			for _, image := range images {
-				originalSBOM, _ := catalogFixtureImage(t, image, source.SquashedScope, false)
+				originalSBOM, _ := catalogFixtureImage(t, image, source.SquashedScope, nil)
 
 				format := syft.FormatByID(test.formatOption)
 				require.NotNil(t, format)

test/integration/mariner_distroless_test.go

@@ -8,7 +8,7 @@ import (
 )
 
 func TestMarinerDistroless(t *testing.T) {
-	sbom, _ := catalogFixtureImage(t, "image-mariner-distroless", source.SquashedScope, false)
+	sbom, _ := catalogFixtureImage(t, "image-mariner-distroless", source.SquashedScope, nil)
 
 	expectedPkgs := 12
 	actualPkgs := 0

test/integration/package_deduplication_test.go

@@ -2,10 +2,11 @@ package integration
 
 import (
 	"fmt"
+	"testing"
+
 	"github.com/anchore/syft/syft/source"
 	"github.com/stretchr/testify/assert"
 	"github.com/stretchr/testify/require"
-	"testing"
 )
 
 func TestPackageDeduplication(t *testing.T) {
@@ -56,7 +57,7 @@ func TestPackageDeduplication(t *testing.T) {
 
 	for _, tt := range tests {
 		t.Run(string(tt.scope), func(t *testing.T) {
-			sbom, _ := catalogFixtureImage(t, "image-vertical-package-dups", tt.scope, false)
+			sbom, _ := catalogFixtureImage(t, "image-vertical-package-dups", tt.scope, nil)
 
 			assert.Equal(t, tt.packageCount, sbom.Artifacts.PackageCatalog.PackageCount())
 			for name, expectedInstanceCount := range tt.instanceCount {

test/integration/package_ownership_relationship_test.go

@@ -3,9 +3,10 @@ package integration
 import (
 	"bytes"
 	"encoding/json"
-	"github.com/anchore/syft/syft/source"
 	"testing"
 
+	"github.com/anchore/syft/syft/source"
+
 	"github.com/anchore/syft/internal/formats/syftjson"
 	syftjsonModel "github.com/anchore/syft/internal/formats/syftjson/model"
 )
@@ -23,7 +24,7 @@ func TestPackageOwnershipRelationships(t *testing.T) {
 
 	for _, test := range tests {
 		t.Run(test.fixture, func(t *testing.T) {
-			sbom, _ := catalogFixtureImage(t, test.fixture, source.SquashedScope, false)
+			sbom, _ := catalogFixtureImage(t, test.fixture, source.SquashedScope, nil)
 
 			output := bytes.NewBufferString("")
 			err := syftjson.Format().Encode(output, sbom)

test/integration/regression_apk_scanner_buffer_size_test.go

@@ -1,16 +1,17 @@
 package integration
 
 import (
-	"github.com/anchore/syft/syft/source"
 	"testing"
 
+	"github.com/anchore/syft/syft/source"
+
 	"github.com/anchore/syft/syft/pkg"
 )
 
 func TestRegression212ApkBufferSize(t *testing.T) {
 	// This is a regression test for issue #212 (https://github.com/anchore/syft/issues/212) in which the apk db could
 	// not be processed due to a scanner buffer that was too small
-	sbom, _ := catalogFixtureImage(t, "image-large-apk-data", source.SquashedScope, false)
+	sbom, _ := catalogFixtureImage(t, "image-large-apk-data", source.SquashedScope, nil)
 
 	expectedPkgs := 58
 	actualPkgs := 0

test/integration/regression_go_bin_scanner_arch_test.go

@@ -1,10 +1,11 @@
 package integration
 
 import (
-	"github.com/anchore/syft/syft/source"
 	"strings"
 	"testing"
 
+	"github.com/anchore/syft/syft/source"
+
 	"github.com/anchore/syft/syft/pkg"
 )
 
@@ -16,7 +17,7 @@ func TestRegressionGoArchDiscovery(t *testing.T) {
 	)
 	// This is a regression test to make sure the way we detect go binary packages
 	// stays consistent and reproducible as the tool chain evolves
-	sbom, _ := catalogFixtureImage(t, "image-go-bin-arch-coverage", source.SquashedScope, false)
+	sbom, _ := catalogFixtureImage(t, "image-go-bin-arch-coverage", source.SquashedScope, nil)
 
 	var actualELF, actualWIN, actualMACOS int
 

test/integration/regression_java_no_main_package_test.go

@@ -1,10 +1,11 @@
 package integration
 
 import (
-	"github.com/anchore/syft/syft/source"
 	"testing"
+
+	"github.com/anchore/syft/syft/source"
 )
 
 func TestRegressionJavaNoMainPackage(t *testing.T) { // Regression: https://github.com/anchore/syft/issues/252
-	catalogFixtureImage(t, "image-java-no-main-package", source.SquashedScope, false)
+	catalogFixtureImage(t, "image-java-no-main-package", source.SquashedScope, nil)
 }

test/integration/rust_audit_binary_test.go

@@ -8,7 +8,7 @@ import (
 )
 
 func TestRustAudit(t *testing.T) {
-	sbom, _ := catalogFixtureImage(t, "image-rust-auditable", source.SquashedScope, true)
+	sbom, _ := catalogFixtureImage(t, "image-rust-auditable", source.SquashedScope, []string{"all"})
 
 	expectedPkgs := 2
 	actualPkgs := 0

test/integration/sqlite_rpmdb_test.go

@@ -11,7 +11,7 @@ import (
 func TestSqliteRpm(t *testing.T) {
 	// This is a regression test for issue #469 (https://github.com/anchore/syft/issues/469). Recent RPM
 	// based distribution store package data in an sqlite database
-	sbom, _ := catalogFixtureImage(t, "image-sqlite-rpmdb", source.SquashedScope, false)
+	sbom, _ := catalogFixtureImage(t, "image-sqlite-rpmdb", source.SquashedScope, nil)
 
 	expectedPkgs := 139
 	actualPkgs := 0

test/integration/utils_test.go

@@ -1,9 +1,10 @@
 package integration
 
 import (
-	"github.com/stretchr/testify/require"
 	"testing"
 
+	"github.com/stretchr/testify/require"
+
 	"github.com/anchore/syft/syft/pkg/cataloger"
 
 	"github.com/anchore/syft/syft/sbom"
@@ -13,7 +14,7 @@ import (
 	"github.com/anchore/syft/syft/source"
 )
 
-func catalogFixtureImage(t *testing.T, fixtureImageName string, scope source.Scope, allCatalogers bool) (sbom.SBOM, *source.Source) {
+func catalogFixtureImage(t *testing.T, fixtureImageName string, scope source.Scope, catalogerCfg []string) (sbom.SBOM, *source.Source) {
 	imagetest.GetFixtureImage(t, "docker-archive", fixtureImageName)
 	tarPath := imagetest.GetFixtureImageTarPath(t, fixtureImageName)
 	userInput := "docker-archive:" + tarPath
@@ -23,11 +24,9 @@ func catalogFixtureImage(t *testing.T, fixtureImageName string, scope source.Sco
 	t.Cleanup(cleanupSource)
 	require.NoError(t, err)
 
-	// TODO: this would be better with functional options (after/during API refactor)
 	c := cataloger.DefaultConfig()
-	if allCatalogers {
+	c.Catalogers = catalogerCfg
-		c.Catalogers = []string{"all"}
+
-	}
 	c.Search.Scope = scope
 	pkgCatalog, relationships, actualDistro, err := syft.CatalogPackages(theSource, c)
 	if err != nil {