* pull out verification logic from github detectors
* deduplicate verify github logic
* pull out nil check
* return nil instead of empty struct
* skip gh old test bc we can't make new tokens
This is a follow-up to #2107 and #2335. It adds a new (hidden) --results flag that allows a user to show any combination of verified, unverified, and indeterminate secrets.
* verify canaries against SNS; get ARN
* clean comments
* Update tests and logic
* added test for invalid canary secret
* added verify logic for canaries
* go mod tidy
---------
Co-authored-by: Dustin Decker <dustin@trufflesec.com>
When we fail to clone a git repository we log the command output to help with diagnosis. However, this output can include credentials in certain cases (such as certain errors associated with redirects). We don't want to log credentials when this happens.
This PR adds the ability to exclude buckets from S3 scans. The capability is pretty rudimentary right now, and does not support globbing. If both lists are specified the source to fail to initialize.
* JDBC test and parsing improvements
- Uses net/url for more robust URI parsing
- Supports common JDBC formats for MySQL
- Supports URI format for MSSQL
- Uses allowlist for params across all drivers
- Uses testcontainers-go for integration testing - much faster, more robust, no port collisions
- Uses gofakeit for random data (db, user, password) generation in integration tests
- Adds connection timeouts
- Use Microsoft's driver for MSSQL
* go mod tidy
* Add Display method to SourceUnit and Kind member to the CommonSourceUnit
* Make SourceUnitID return the ID and a kind
These two values together uniquely represent a unit.
* Add flag to write job reports to disk
* Fix nil pointer / non-nil interface bug
* Synchronize job report writer goroutine
* Log when the report has been written
* Implement SourceUnitEnumChunker for GitLab
* Add GitLab engine integration test
* Use a SliceReporter instead of checking for nil reporters
* Use more generic VisitorReporter
* Merge logic from getReposFromGitlab into getAllProjectRepos
* Update integration test to have a lower bound
Unfortunately, the GitLab integration test does not appear to be
deterministic. Sometimes 36390 chunks are found, sometimes 36312, or
even lower.
* Refactor UnitHook to block the scan if finished metrics aren't handled
* Log once when back-pressure is detected
* Add hook channel size metric
* Use plural "metrics" for consistency
* Replace LRU cache with map
* use diff chan
* correctly use the buffered file writer
* use value from source
* reorder fields
* add tests and update
* Fix issue with buffer slices growing
* fix test
* correctly use the buffered file writer
* use value from source
* reorder fields
* fix
* add singleton
* use shared pool
* optimize
* rename and cleanup
* add metrics
* add print
* rebase
* remove extra inc
* add metrics for checkout time
* add comment
* use microseconds
* add metrics
* add metrics pkg
* add more metrics
* rever test
* remove fields
* fix
* resize and return
* update metric name
* remove comment
* address comments
* add comment
This is a follow-up to #1912, which used the headers from the response to determine rate-limiting information, instead of using the values from RateLimitError.Rate. Although that logic seemed solid, I discovered that it did not work in some circumstances. This lead to the "unexpected" path more often than intended, and periodic instances where requests would be made before the ratelimit was refreshed.
