trufflehog/pkg/engine/gcs_test.go

109 lines
2.5 KiB
Go
Raw Normal View History

Add gcs scanning integration (#1153) * Setup for GCS scanning. * Update GCS engine w/ projectID req. * Add concurrency field to gcsManager. * add errgroup to gcsManager. * Update gcs manager. * Use defautl ADC. * use ADC.' * Add TOOD. * add log to iterator completion. * use a BinaryReader instead of concrete object for channel type. * initial test for Chunks. * Add tests for chunking objects. * Add concurrency. * update metadata to include content type and acls. * Add object reading code. * Add integration test. * Add entrypoint. * Add removed wg.Wait(). * remove dead code. * remove build. * Remove period from file extension. * remove used. * Add comment. * Setup for GCS scanning. * Update GCS engine w/ projectID req. * Add concurrency field to gcsManager. * add errgroup to gcsManager. * Update gcs manager. * Use defautl ADC. * use ADC.' * Add TOOD. * add log to iterator completion. * use a BinaryReader instead of concrete object for channel type. * initial test for Chunks. * Add tests for chunking objects. * Add concurrency. * update metadata to include content type and acls. * Add object reading code. * Add integration test. * Add entrypoint. * Add removed wg.Wait(). * remove dead code. * remove build. * remove used. * Add file type for objects. * Add check for file type and size. * Add default file size. * Add additinoal auth options and remaining CLI flags. * Handle errors in go routines. * Handle resuming for buckets. * Remove redundant words in comment. * remove ok check on bool check. * remove extra blank line. * Add return if handler handles chunk. * Add comment. * remove extra blank line. * cleanup comment. * Add comment. * move up fxn. * go mod tidy. * Add exclusion to perf testing buckets. * Handle blocking the channel. * remove unused const. * fix tests. * fix tests. * Handle gcs manger options better. * update fxn name. * Remove arg name. * ignore buckets in gcsManager test. * fix test. * propulate gsManagerOpts. * inline err check. * Add readme. * update readme spelling. * fix test.
2023-03-08 01:32:04 +00:00
package engine
import (
"strings"
Add gcs scanning integration (#1153) * Setup for GCS scanning. * Update GCS engine w/ projectID req. * Add concurrency field to gcsManager. * add errgroup to gcsManager. * Update gcs manager. * Use defautl ADC. * use ADC.' * Add TOOD. * add log to iterator completion. * use a BinaryReader instead of concrete object for channel type. * initial test for Chunks. * Add tests for chunking objects. * Add concurrency. * update metadata to include content type and acls. * Add object reading code. * Add integration test. * Add entrypoint. * Add removed wg.Wait(). * remove dead code. * remove build. * Remove period from file extension. * remove used. * Add comment. * Setup for GCS scanning. * Update GCS engine w/ projectID req. * Add concurrency field to gcsManager. * add errgroup to gcsManager. * Update gcs manager. * Use defautl ADC. * use ADC.' * Add TOOD. * add log to iterator completion. * use a BinaryReader instead of concrete object for channel type. * initial test for Chunks. * Add tests for chunking objects. * Add concurrency. * update metadata to include content type and acls. * Add object reading code. * Add integration test. * Add entrypoint. * Add removed wg.Wait(). * remove dead code. * remove build. * remove used. * Add file type for objects. * Add check for file type and size. * Add default file size. * Add additinoal auth options and remaining CLI flags. * Handle errors in go routines. * Handle resuming for buckets. * Remove redundant words in comment. * remove ok check on bool check. * remove extra blank line. * Add return if handler handles chunk. * Add comment. * remove extra blank line. * cleanup comment. * Add comment. * move up fxn. * go mod tidy. * Add exclusion to perf testing buckets. * Handle blocking the channel. * remove unused const. * fix tests. * fix tests. * Handle gcs manger options better. * update fxn name. * Remove arg name. * ignore buckets in gcsManager test. * fix test. * propulate gsManagerOpts. * inline err check. * Add readme. * update readme spelling. * fix test.
2023-03-08 01:32:04 +00:00
"testing"
"github.com/stretchr/testify/assert"
Add gcs scanning integration (#1153) * Setup for GCS scanning. * Update GCS engine w/ projectID req. * Add concurrency field to gcsManager. * add errgroup to gcsManager. * Update gcs manager. * Use defautl ADC. * use ADC.' * Add TOOD. * add log to iterator completion. * use a BinaryReader instead of concrete object for channel type. * initial test for Chunks. * Add tests for chunking objects. * Add concurrency. * update metadata to include content type and acls. * Add object reading code. * Add integration test. * Add entrypoint. * Add removed wg.Wait(). * remove dead code. * remove build. * Remove period from file extension. * remove used. * Add comment. * Setup for GCS scanning. * Update GCS engine w/ projectID req. * Add concurrency field to gcsManager. * add errgroup to gcsManager. * Update gcs manager. * Use defautl ADC. * use ADC.' * Add TOOD. * add log to iterator completion. * use a BinaryReader instead of concrete object for channel type. * initial test for Chunks. * Add tests for chunking objects. * Add concurrency. * update metadata to include content type and acls. * Add object reading code. * Add integration test. * Add entrypoint. * Add removed wg.Wait(). * remove dead code. * remove build. * remove used. * Add file type for objects. * Add check for file type and size. * Add default file size. * Add additinoal auth options and remaining CLI flags. * Handle errors in go routines. * Handle resuming for buckets. * Remove redundant words in comment. * remove ok check on bool check. * remove extra blank line. * Add return if handler handles chunk. * Add comment. * remove extra blank line. * cleanup comment. * Add comment. * move up fxn. * go mod tidy. * Add exclusion to perf testing buckets. * Handle blocking the channel. * remove unused const. * fix tests. * fix tests. * Handle gcs manger options better. * update fxn name. * Remove arg name. * ignore buckets in gcsManager test. * fix test. * propulate gsManagerOpts. * inline err check. * Add readme. * update readme spelling. * fix test.
2023-03-08 01:32:04 +00:00
"github.com/trufflesecurity/trufflehog/v3/pkg/context"
"github.com/trufflesecurity/trufflehog/v3/pkg/decoders"
Add gcs scanning integration (#1153) * Setup for GCS scanning. * Update GCS engine w/ projectID req. * Add concurrency field to gcsManager. * add errgroup to gcsManager. * Update gcs manager. * Use defautl ADC. * use ADC.' * Add TOOD. * add log to iterator completion. * use a BinaryReader instead of concrete object for channel type. * initial test for Chunks. * Add tests for chunking objects. * Add concurrency. * update metadata to include content type and acls. * Add object reading code. * Add integration test. * Add entrypoint. * Add removed wg.Wait(). * remove dead code. * remove build. * Remove period from file extension. * remove used. * Add comment. * Setup for GCS scanning. * Update GCS engine w/ projectID req. * Add concurrency field to gcsManager. * add errgroup to gcsManager. * Update gcs manager. * Use defautl ADC. * use ADC.' * Add TOOD. * add log to iterator completion. * use a BinaryReader instead of concrete object for channel type. * initial test for Chunks. * Add tests for chunking objects. * Add concurrency. * update metadata to include content type and acls. * Add object reading code. * Add integration test. * Add entrypoint. * Add removed wg.Wait(). * remove dead code. * remove build. * remove used. * Add file type for objects. * Add check for file type and size. * Add default file size. * Add additinoal auth options and remaining CLI flags. * Handle errors in go routines. * Handle resuming for buckets. * Remove redundant words in comment. * remove ok check on bool check. * remove extra blank line. * Add return if handler handles chunk. * Add comment. * remove extra blank line. * cleanup comment. * Add comment. * move up fxn. * go mod tidy. * Add exclusion to perf testing buckets. * Handle blocking the channel. * remove unused const. * fix tests. * fix tests. * Handle gcs manger options better. * update fxn name. * Remove arg name. * ignore buckets in gcsManager test. * fix test. * propulate gsManagerOpts. * inline err check. * Add readme. * update readme spelling. * fix test.
2023-03-08 01:32:04 +00:00
"github.com/trufflesecurity/trufflehog/v3/pkg/sources"
)
func TestScanGCS(t *testing.T) {
tests := []struct {
name string
gcsConfig sources.GCSConfig
wantErr bool
}{
{
name: "scanned GCS",
gcsConfig: sources.GCSConfig{
ApiKey: "abc123",
ProjectID: "test-project",
CloudCred: false,
WithoutAuth: false,
ServiceAccount: "",
},
},
{
name: "missing project ID, with auth",
gcsConfig: sources.GCSConfig{ApiKey: "abc123"},
wantErr: true,
},
{
name: "missing project ID, without auth, public scan",
gcsConfig: sources.GCSConfig{WithoutAuth: true},
Add gcs scanning integration (#1153) * Setup for GCS scanning. * Update GCS engine w/ projectID req. * Add concurrency field to gcsManager. * add errgroup to gcsManager. * Update gcs manager. * Use defautl ADC. * use ADC.' * Add TOOD. * add log to iterator completion. * use a BinaryReader instead of concrete object for channel type. * initial test for Chunks. * Add tests for chunking objects. * Add concurrency. * update metadata to include content type and acls. * Add object reading code. * Add integration test. * Add entrypoint. * Add removed wg.Wait(). * remove dead code. * remove build. * Remove period from file extension. * remove used. * Add comment. * Setup for GCS scanning. * Update GCS engine w/ projectID req. * Add concurrency field to gcsManager. * add errgroup to gcsManager. * Update gcs manager. * Use defautl ADC. * use ADC.' * Add TOOD. * add log to iterator completion. * use a BinaryReader instead of concrete object for channel type. * initial test for Chunks. * Add tests for chunking objects. * Add concurrency. * update metadata to include content type and acls. * Add object reading code. * Add integration test. * Add entrypoint. * Add removed wg.Wait(). * remove dead code. * remove build. * remove used. * Add file type for objects. * Add check for file type and size. * Add default file size. * Add additinoal auth options and remaining CLI flags. * Handle errors in go routines. * Handle resuming for buckets. * Remove redundant words in comment. * remove ok check on bool check. * remove extra blank line. * Add return if handler handles chunk. * Add comment. * remove extra blank line. * cleanup comment. * Add comment. * move up fxn. * go mod tidy. * Add exclusion to perf testing buckets. * Handle blocking the channel. * remove unused const. * fix tests. * fix tests. * Handle gcs manger options better. * update fxn name. * Remove arg name. * ignore buckets in gcsManager test. * fix test. * propulate gsManagerOpts. * inline err check. * Add readme. * update readme spelling. * fix test.
2023-03-08 01:32:04 +00:00
},
{
name: "multiple selected auth methods",
gcsConfig: sources.GCSConfig{
ApiKey: "abc123",
ProjectID: "test-project",
CloudCred: true,
WithoutAuth: false,
ServiceAccount: "",
},
wantErr: true,
},
{
name: "no auth method selected",
gcsConfig: sources.GCSConfig{
ProjectID: "test-project",
MaxObjectSize: 10 * 1024 * 1024,
Add gcs scanning integration (#1153) * Setup for GCS scanning. * Update GCS engine w/ projectID req. * Add concurrency field to gcsManager. * add errgroup to gcsManager. * Update gcs manager. * Use defautl ADC. * use ADC.' * Add TOOD. * add log to iterator completion. * use a BinaryReader instead of concrete object for channel type. * initial test for Chunks. * Add tests for chunking objects. * Add concurrency. * update metadata to include content type and acls. * Add object reading code. * Add integration test. * Add entrypoint. * Add removed wg.Wait(). * remove dead code. * remove build. * Remove period from file extension. * remove used. * Add comment. * Setup for GCS scanning. * Update GCS engine w/ projectID req. * Add concurrency field to gcsManager. * add errgroup to gcsManager. * Update gcs manager. * Use defautl ADC. * use ADC.' * Add TOOD. * add log to iterator completion. * use a BinaryReader instead of concrete object for channel type. * initial test for Chunks. * Add tests for chunking objects. * Add concurrency. * update metadata to include content type and acls. * Add object reading code. * Add integration test. * Add entrypoint. * Add removed wg.Wait(). * remove dead code. * remove build. * remove used. * Add file type for objects. * Add check for file type and size. * Add default file size. * Add additinoal auth options and remaining CLI flags. * Handle errors in go routines. * Handle resuming for buckets. * Remove redundant words in comment. * remove ok check on bool check. * remove extra blank line. * Add return if handler handles chunk. * Add comment. * remove extra blank line. * cleanup comment. * Add comment. * move up fxn. * go mod tidy. * Add exclusion to perf testing buckets. * Handle blocking the channel. * remove unused const. * fix tests. * fix tests. * Handle gcs manger options better. * update fxn name. * Remove arg name. * ignore buckets in gcsManager test. * fix test. * propulate gsManagerOpts. * inline err check. * Add readme. * update readme spelling. * fix test.
2023-03-08 01:32:04 +00:00
},
},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
ctx, cancel := context.WithCancel(context.TODO())
defer cancel()
const defaultOutputBufferSize = 64
opts := []func(*sources.SourceManager){
sources.WithSourceUnits(),
sources.WithBufferedOutput(defaultOutputBufferSize),
}
sourceManager := sources.NewManager(opts...)
conf := Config{
Concurrency: 1,
Decoders: decoders.DefaultDecoders(),
Detectors: DefaultDetectors(),
Verify: false,
SourceManager: sourceManager,
Dispatcher: NewPrinterDispatcher(new(discardPrinter)),
}
e, err := NewEngine(ctx, &conf)
assert.NoError(t, err)
e.Start(ctx)
go func() {
resultCount := 0
for range e.ResultsChan() {
resultCount++
}
}()
err = e.ScanGCS(ctx, test.gcsConfig)
if err != nil && !test.wantErr && !strings.Contains(err.Error(), "googleapi: Error 400: Bad Request") {
Add gcs scanning integration (#1153) * Setup for GCS scanning. * Update GCS engine w/ projectID req. * Add concurrency field to gcsManager. * add errgroup to gcsManager. * Update gcs manager. * Use defautl ADC. * use ADC.' * Add TOOD. * add log to iterator completion. * use a BinaryReader instead of concrete object for channel type. * initial test for Chunks. * Add tests for chunking objects. * Add concurrency. * update metadata to include content type and acls. * Add object reading code. * Add integration test. * Add entrypoint. * Add removed wg.Wait(). * remove dead code. * remove build. * Remove period from file extension. * remove used. * Add comment. * Setup for GCS scanning. * Update GCS engine w/ projectID req. * Add concurrency field to gcsManager. * add errgroup to gcsManager. * Update gcs manager. * Use defautl ADC. * use ADC.' * Add TOOD. * add log to iterator completion. * use a BinaryReader instead of concrete object for channel type. * initial test for Chunks. * Add tests for chunking objects. * Add concurrency. * update metadata to include content type and acls. * Add object reading code. * Add integration test. * Add entrypoint. * Add removed wg.Wait(). * remove dead code. * remove build. * remove used. * Add file type for objects. * Add check for file type and size. * Add default file size. * Add additinoal auth options and remaining CLI flags. * Handle errors in go routines. * Handle resuming for buckets. * Remove redundant words in comment. * remove ok check on bool check. * remove extra blank line. * Add return if handler handles chunk. * Add comment. * remove extra blank line. * cleanup comment. * Add comment. * move up fxn. * go mod tidy. * Add exclusion to perf testing buckets. * Handle blocking the channel. * remove unused const. * fix tests. * fix tests. * Handle gcs manger options better. * update fxn name. * Remove arg name. * ignore buckets in gcsManager test. * fix test. * propulate gsManagerOpts. * inline err check. * Add readme. * update readme spelling. * fix test.
2023-03-08 01:32:04 +00:00
t.Errorf("ScanGCS() got: %v, want: %v", err, nil)
return
}
if err := e.Finish(ctx); err != nil && !test.wantErr && !strings.Contains(err.Error(), "googleapi: Error 400: Bad Request") {
t.Errorf("Finish() got: %v, want: %v", err, nil)
return
}
Add gcs scanning integration (#1153) * Setup for GCS scanning. * Update GCS engine w/ projectID req. * Add concurrency field to gcsManager. * add errgroup to gcsManager. * Update gcs manager. * Use defautl ADC. * use ADC.' * Add TOOD. * add log to iterator completion. * use a BinaryReader instead of concrete object for channel type. * initial test for Chunks. * Add tests for chunking objects. * Add concurrency. * update metadata to include content type and acls. * Add object reading code. * Add integration test. * Add entrypoint. * Add removed wg.Wait(). * remove dead code. * remove build. * Remove period from file extension. * remove used. * Add comment. * Setup for GCS scanning. * Update GCS engine w/ projectID req. * Add concurrency field to gcsManager. * add errgroup to gcsManager. * Update gcs manager. * Use defautl ADC. * use ADC.' * Add TOOD. * add log to iterator completion. * use a BinaryReader instead of concrete object for channel type. * initial test for Chunks. * Add tests for chunking objects. * Add concurrency. * update metadata to include content type and acls. * Add object reading code. * Add integration test. * Add entrypoint. * Add removed wg.Wait(). * remove dead code. * remove build. * remove used. * Add file type for objects. * Add check for file type and size. * Add default file size. * Add additinoal auth options and remaining CLI flags. * Handle errors in go routines. * Handle resuming for buckets. * Remove redundant words in comment. * remove ok check on bool check. * remove extra blank line. * Add return if handler handles chunk. * Add comment. * remove extra blank line. * cleanup comment. * Add comment. * move up fxn. * go mod tidy. * Add exclusion to perf testing buckets. * Handle blocking the channel. * remove unused const. * fix tests. * fix tests. * Handle gcs manger options better. * update fxn name. * Remove arg name. * ignore buckets in gcsManager test. * fix test. * propulate gsManagerOpts. * inline err check. * Add readme. * update readme spelling. * fix test.
2023-03-08 01:32:04 +00:00
if err == nil && test.wantErr {
t.Errorf("ScanGCS() got: %v, want: %v", err, "error")
}
})
}
}