Commit graph

2571 commits

Author SHA1 Message Date
Richard Gomez
9053d8f4de
refactor(github): enumerateWithToken flow & tests (#2880) 2024-05-31 15:53:44 -05:00
Abdul Basit
2b3284e650
Redis integration test (#2901)
* implemented redis integration test using docker container

* rename the function and use the redis:7.0 image
2024-05-31 11:59:06 -05:00
Richard Gomez
5575514174
fix(falsepositives): remove 'www' (#2896) 2024-05-31 11:37:27 -04:00
Abdul Basit
ddd861d4c7
consistent image of MSSQL for integration testing. (#2898) 2024-05-31 10:56:02 -04:00
Abdul Basit
d93c428b54
Update metadata for DataDog for API + APPKey (#2879)
* put emails from response in metadata for datadog.

* removed unused type in user structure.

* filter user information based on verified, is service account and disabled boolean
also include organization detail if it is available in response.
2024-05-31 10:50:23 -04:00
Hon
793231370e
Add postman to tui (#2895) 2024-05-29 16:07:23 -07:00
James Telfer
0024b6ce77
feat: support docker image history scanning (#2882)
* feat: support docker image history scanning

* refactor: collapse error handling into return

Style suggestion from review feedback.

* fix: associate layers with history entries

Where possible, add the associated layer to the history entry record. This may help tracing any issues discovered.

This also changes the entry reference format to `image-metadata:history:%d:created-by` which _may_ be more self-explanatory.
2024-05-28 14:07:43 -07:00
Abdul Basit
18b81013b8
Added extra data for LaunchDarkly (#2836)
* added extra data for LaunchDarkly
- Token type like api or sdk
- name and role of First token in response of api-tokens
- total token counts associated with api-token

* renamed total_count to total_token_count

* updated & renamed fields of metadata for launchdarkly, specifically for API based tokens
2024-05-28 14:08:53 -04:00
Richard Gomez
5102e3ae11
test(github): fix some errors (#2774) 2024-05-24 13:03:41 -07:00
Richard Gomez
e53f5bd5c5
Improve handling of Gist URLs (#2653)
* feat(github): handle ghes gists

* fix(github): handle all gist URLs

* refactor(github): helper func to check gist urls
2024-05-24 08:36:30 -07:00
Charlie Gunyon
311494e86e
Elastic adapter (#2727)
* Add stub source and elastic API funcs

* Spawn workers and ship chunks

* Now successfully detects a credential

- Added tests
- Added some documentation comments
- Threaded the passed context through to all the API requests

* Linting fixes

* Add integration tests and resolve some bugs they uncovered

* Logstash -> Elasticsearch

* Add support for --index-pattern

* Add support for --query-json

* Use structs instead of string building to construct a search body

* Support --since-timestamp

* Implement additional authentication methods

* Fix some small bugs

* Refactoring to support --best-effort-scan

* Finish implementation of --best-effort-scan

* Implement scan catch-up

* Finish connecting support for nodes CLI arg

* Add some integration tests around the catchup mechanism

* go mod tidy

* Fix some linting issues

* Remove some debugging Prints

* Move off of _doc

* Remove informational Printf and add informational logging

* Remove debugging logging

* Copy the index from the outer loop as well

* Don't burn up the ES API with rapid requests if there's no work to do in subsequent scans

* No need to export UnitOfWork.AddSearch

* Use a better name for the range query variable when building the timestamp range clause in searches

* Replace some unlocking defers with explicit unlocks to make the synchronized part of the code clearer

* found -> ok

* Remove superfluous buildElasticClient method

---------

Co-authored-by: Charlie Gunyon <charlie@spectral.energy>
2024-05-24 09:38:20 -05:00
Richard Gomez
1441289d41
fix(github): scan user repos (#2814) 2024-05-23 09:40:40 -05:00
Cody Rose
f7214cfee3
Log reasons for GitLab repo exclusion (#2875)
We have some evidence that some GitLab repos are getting incorrectly ignored, but it's not clear why this is happening, so this PR adds some more logging to the relevant code.
2024-05-23 09:08:36 -04:00
āh̳̕mͭͭͨͩ̐e̘ͬ́͋ͬ̊̓͂d
b9ea2f5d4e
adding Groq detector (#2873)
* adding Groq detector

* using prefix as keyword
2024-05-22 15:46:14 -04:00
ahrav
fccf7c9a41
[chore] - Use http.NewRequestWithContext (#2870) 2024-05-22 06:58:12 -07:00
Abdul Basit
4b10b8a009
made changes in organization regex for azure devops. (#2866) 2024-05-20 15:02:12 -05:00
Abdul Basit
5dff334ffa
Update azure storage extra data (#2808)
Blob service of Azure storage returns containers name in response. From that, containers name is added in extra data.
2024-05-20 13:42:54 -04:00
Richard Gomez
d71a8056c0
chore(engine): remove verbose log line (#2860) 2024-05-16 22:25:16 -07:00
ahrav
da25ac2e7f
remove redundant chunking (#2855) 2024-05-16 14:38:50 -07:00
ahrav
5257069748
[chore] - move buffers pkg out of writers pkg (#2826)
* Remove specialized handler and archive struct and restructure handlers pkg.

* Refactor RPM archive handlers to use a library instead of shelling out

* make rpm handling context aware

* update test

* Refactor AR/deb archive handler to use an existing library instead of shelling out

* Update tests

* Handle non-archive data within the DefaultHandler

* make structs and methods private

* Remove non-archive data handling within sources

* add max size check

* add filename and size to context kvp

* move skip file check and is binary check before opening file

* fix test

* preserve existing funcitonality of not handling non-archive files in HandleFile

* Handle non-archive data within the DefaultHandler

* rebase

* Remove non-archive data handling within sources

* Adjust check for rpm/deb archive type

* add additional deb mime type

* add gzip

* move diskbuffered rereader setup into handler pkg

* remove DiskBuffereReader creation logic within sources

* update comment

* move rewind closer

* reduce log verbosity

* add metrics for file handling

* add metrics for errors

* make defaultBufferSize a const

* add metrics for file handling

* add metrics for errors

* fix tests

* add metrics for max archive depth and skipped files

* update error

* skip symlinks and dirs

* update err

* Address incompatible reader to openArchive

* remove nil check

* fix err assignment

* Allow git cat-file blob to complete before trying to handle the file

* wrap compReader with DiskbufferReader

* Allow git cat-file blob to complete before trying to handle the file

* updates

* use buffer writer

* update

* refactor

* update context pkg

* revert stuff

* update test

* fix test

* remove

* use correct reader

* add metrics for file handling

* add metrics for errors

* fix tests

* rebase

* add metrics for errors

* add metrics for max archive depth and skipped files

* update error

* skip symlinks and dirs

* update err

* fix err assignment

* rebase

* remove

* Update write method in contentWriter interface

* Add bufferReadSeekCloser

* update name

* update comment

* fix lint

* Remove specialized handler and archive struct and restructure handlers pkg.

* Refactor RPM archive handlers to use a library instead of shelling out

* make rpm handling context aware

* update test

* Refactor AR/deb archive handler to use an existing library instead of shelling out

* Update tests

* add max size check

* add filename and size to context kvp

* move skip file check and is binary check before opening file

* fix test

* preserve existing funcitonality of not handling non-archive files in HandleFile

* Handle non-archive data within the DefaultHandler

* rebase

* Remove non-archive data handling within sources

* Handle non-archive data within the DefaultHandler

* add gzip

* move diskbuffered rereader setup into handler pkg

* remove DiskBuffereReader creation logic within sources

* update comment

* move rewind closer

* reduce log verbosity

* make defaultBufferSize a const

* add metrics for file handling

* add metrics for errors

* fix tests

* add metrics for max archive depth and skipped files

* update error

* skip symlinks and dirs

* update err

* Address incompatible reader to openArchive

* remove nil check

* fix err assignment

* wrap compReader with DiskbufferReader

* Allow git cat-file blob to complete before trying to handle the file

* updates

* use buffer writer

* update

* refactor

* update context pkg

* revert stuff

* update test

* remove

* rebase

* go mod tidy

* lint check

* update metric to ms

* update metric

* update comments

* dont use ptr

* update

* fix

* Remove specialized handler and archive struct and restructure handlers pkg.

* Refactor RPM archive handlers to use a library instead of shelling out

* make rpm handling context aware

* update test

* Refactor AR/deb archive handler to use an existing library instead of shelling out

* Update tests

* add max size check

* add filename and size to context kvp

* move skip file check and is binary check before opening file

* fix test

* preserve existing funcitonality of not handling non-archive files in HandleFile

* Adjust check for rpm/deb archive type

* add additional deb mime type

* update comment

* go mod tidy

* update go mod

* Add a buffered file reader

* update comments

* use Buffered File Readder

* return buffer

* update

* fix

* return

* go mod tidy

* merge

* use a shared pool

* use sync.Once

* reorganzie

* remove unused code

* fix double init

* fix stuff

* nil check

* reduce allocations

* updates

* update metrics

* updates

* reset buffer instead of putting it back

* skip binaries

* skip

* concurrently process diffs

* close chan

* concurrently enumerate orgs

* increase workers

* ignore pbix and vsdx files

* add metrics for gitparse's Diffchan

* fix metric

* update metrics

* update

* fix checks

* fix

* inc

* update

* reduce

* Create workers to handle binary files

* modify workers

* updates

* add check

* delete code

* use custom reader

* rename struct

* add nonarchive handler

* fix break

* add comments

* add tests

* refactor

* remove log

* do not scan rpm links

* simplify

* rename var

* rename

* fix benchmark

* add buffer

* buffer

* buffer

* handle panic

* merge main

* merge main

* add recover

* revert stuff

* revert

* revert to using reader

* fixes

* remove

* update

* fixes

* linter

* fix test

* move buffers pkg out of writers pkg

* rename

* [refactor] - move buffer pool logic into own pkg (#2828)

* move buffer pool logic into own pkg

* fix test

* fix test

* whoops

* [feat] - additional buffer pool (#2829)

* move buffer pool logic into own pkg

* move

* fix test

* fix test

* fix test

* remove

* fix test

* whoops

* revert

* fix
2024-05-16 14:38:36 -07:00
ahrav
896e6e7c66
upgrade github dep (#2858) 2024-05-16 14:35:08 -07:00
Zachary Rice
e0351c215a
add tolower to all keywords, and remove return on error for global vars (#2852) 2024-05-16 14:03:03 -05:00
Abdul Basit
15c6333987
deprecated Integromat detector becuase they are gone. (#2856)
remove the package as well.
2024-05-16 08:29:36 -07:00
ahrav
2db06f0576
[bug] - Handle empty reader case in newFileReader (#2854)
* Correclty handle empty files

* fix

* fix test
2024-05-15 18:25:36 -07:00
ahrav
ead9dd5748
[refactor] - Create separate handler for non-archive data (#2825)
* Remove specialized handler and archive struct and restructure handlers pkg.

* Refactor RPM archive handlers to use a library instead of shelling out

* make rpm handling context aware

* update test

* Refactor AR/deb archive handler to use an existing library instead of shelling out

* Update tests

* Handle non-archive data within the DefaultHandler

* make structs and methods private

* Remove non-archive data handling within sources

* add max size check

* add filename and size to context kvp

* move skip file check and is binary check before opening file

* fix test

* preserve existing funcitonality of not handling non-archive files in HandleFile

* Handle non-archive data within the DefaultHandler

* rebase

* Remove non-archive data handling within sources

* Adjust check for rpm/deb archive type

* add additional deb mime type

* add gzip

* move diskbuffered rereader setup into handler pkg

* remove DiskBuffereReader creation logic within sources

* update comment

* move rewind closer

* reduce log verbosity

* add metrics for file handling

* add metrics for errors

* make defaultBufferSize a const

* add metrics for file handling

* add metrics for errors

* fix tests

* add metrics for max archive depth and skipped files

* update error

* skip symlinks and dirs

* update err

* Address incompatible reader to openArchive

* remove nil check

* fix err assignment

* Allow git cat-file blob to complete before trying to handle the file

* wrap compReader with DiskbufferReader

* Allow git cat-file blob to complete before trying to handle the file

* updates

* use buffer writer

* update

* refactor

* update context pkg

* revert stuff

* update test

* fix test

* remove

* use correct reader

* add metrics for file handling

* add metrics for errors

* fix tests

* rebase

* add metrics for errors

* add metrics for max archive depth and skipped files

* update error

* skip symlinks and dirs

* update err

* fix err assignment

* rebase

* remove

* Update write method in contentWriter interface

* Add bufferReadSeekCloser

* update name

* update comment

* fix lint

* Remove specialized handler and archive struct and restructure handlers pkg.

* Refactor RPM archive handlers to use a library instead of shelling out

* make rpm handling context aware

* update test

* Refactor AR/deb archive handler to use an existing library instead of shelling out

* Update tests

* add max size check

* add filename and size to context kvp

* move skip file check and is binary check before opening file

* fix test

* preserve existing funcitonality of not handling non-archive files in HandleFile

* Handle non-archive data within the DefaultHandler

* rebase

* Remove non-archive data handling within sources

* Handle non-archive data within the DefaultHandler

* add gzip

* move diskbuffered rereader setup into handler pkg

* remove DiskBuffereReader creation logic within sources

* update comment

* move rewind closer

* reduce log verbosity

* make defaultBufferSize a const

* add metrics for file handling

* add metrics for errors

* fix tests

* add metrics for max archive depth and skipped files

* update error

* skip symlinks and dirs

* update err

* Address incompatible reader to openArchive

* remove nil check

* fix err assignment

* wrap compReader with DiskbufferReader

* Allow git cat-file blob to complete before trying to handle the file

* updates

* use buffer writer

* update

* refactor

* update context pkg

* revert stuff

* update test

* remove

* rebase

* go mod tidy

* lint check

* update metric to ms

* update metric

* update comments

* dont use ptr

* update

* fix

* Remove specialized handler and archive struct and restructure handlers pkg.

* Refactor RPM archive handlers to use a library instead of shelling out

* make rpm handling context aware

* update test

* Refactor AR/deb archive handler to use an existing library instead of shelling out

* Update tests

* add max size check

* add filename and size to context kvp

* move skip file check and is binary check before opening file

* fix test

* preserve existing funcitonality of not handling non-archive files in HandleFile

* Adjust check for rpm/deb archive type

* add additional deb mime type

* update comment

* go mod tidy

* update go mod

* Add a buffered file reader

* update comments

* use Buffered File Readder

* return buffer

* update

* fix

* return

* go mod tidy

* merge

* use a shared pool

* use sync.Once

* reorganzie

* remove unused code

* fix double init

* fix stuff

* nil check

* reduce allocations

* updates

* update metrics

* updates

* reset buffer instead of putting it back

* skip binaries

* skip

* concurrently process diffs

* close chan

* concurrently enumerate orgs

* increase workers

* ignore pbix and vsdx files

* add metrics for gitparse's Diffchan

* fix metric

* update metrics

* update

* fix checks

* fix

* inc

* update

* reduce

* Create workers to handle binary files

* modify workers

* updates

* add check

* delete code

* use custom reader

* rename struct

* add nonarchive handler

* fix break

* add comments

* add tests

* refactor

* remove log

* do not scan rpm links

* simplify

* rename var

* rename

* fix benchmark

* add buffer

* buffer

* buffer

* handle panic

* merge main

* merge main

* add recover

* revert stuff

* revert

* revert to using reader

* fixes

* remove

* update

* fixes

* linter

* fix test

* fix comment

* update field name

* fix
2024-05-15 13:40:16 -07:00
Abdul Basit
7025b0aa35
added email and location in metadata. (#2850) 2024-05-15 12:36:22 -05:00
cuiyourong
ead4e8fa2d
chore: fix some typos in comments (#2851)
Signed-off-by: cuiyourong <cuiyourong@gmail.com>
2024-05-15 07:36:21 -07:00
Alexandre GUIOT--VALENTIN
0d8c3335ed
Add "Intra42" detector (#2835)
* Add basic intra42 detector (lacks verification)

* Improve keywords/prefixes for intra42 detector

* Un-lint pkg/pb/detectorspb/detectors.pb.go to avoid bloating PR

* Add client_id match and secret verification

* Improve PrefixRegex

* Add missing entry in DetectorType_name in detectors.pb.go

* Add Intra42 to proto/detectors.proto

* Remove PrefixRegex

* Keep only identifiers as keywords

* Factorize regex (a-f0-9)
2024-05-14 11:33:54 -07:00
ahrav
6df147de58
[feat] - Support bearer auth for docker scans (#2848)
* Support bearer auth for docker scans

* updates

* use no auth by default if no other auth method is provided
2024-05-14 11:30:11 -07:00
Cody Rose
4882d230e0
Use fake detectors in versioned detectors test (#2847)
This automated test used to run with the real GitLab detectors because they were versioned. However, the test doesn't need real detectors to actually validate the functionality in question, and relying on real detectors means that we're susceptible to token expiration, which we recently discovered when it happened. The test has been updated to use fake detectors (which means it can run correctly in the community suite as well now.)
2024-05-14 13:15:06 -04:00
ahrav
f82cf8d76d
[bug] - Fix case-sensitivity issue in PrefixRegex function (#2811)
* correctly remove case insensitivity for the capture group

* update
2024-05-14 08:55:36 -05:00
Richard Gomez
a00587673a
feat(sendgrid): update detector (#2833) 2024-05-13 18:44:37 -07:00
ahrav
9873c144ee
[chore] - Update GitlabV2 detector (#2840)
* replace keyword and replace prefix

* address comment
2024-05-13 14:13:23 -07:00
Abdul Basit
f527da9ecc
Update results's extra data for Twilio (#2807)
* Response structure added for service api of Twilio.
added two response fields in extra data:
1) friendly_name
2) account_sid

* mark credentials verified for non-fatal errors.
also check for atleast one service in response before extracting metadata.
2024-05-13 10:09:35 -04:00
ahrav
570cec7565
[refactor] - Refactor Archive Handling Logic (#2703)
* Remove specialized handler and archive struct and restructure handlers pkg.

* Refactor RPM archive handlers to use a library instead of shelling out

* make rpm handling context aware

* update test

* Refactor AR/deb archive handler to use an existing library instead of shelling out

* Update tests

* add max size check

* add filename and size to context kvp

* move skip file check and is binary check before opening file

* fix test

* preserve existing funcitonality of not handling non-archive files in HandleFile

* Adjust check for rpm/deb archive type

* add additional deb mime type

* update comment

* Remove specialized handler and archive struct and restructure handlers pkg.

* Refactor RPM archive handlers to use a library instead of shelling out

* make rpm handling context aware

* update test

* Refactor AR/deb archive handler to use an existing library instead of shelling out

* Update tests

* add max size check

* add filename and size to context kvp

* move skip file check and is binary check before opening file

* fix test

* preserve existing funcitonality of not handling non-archive files in HandleFile

* Adjust check for rpm/deb archive type

* add additional deb mime type

* update comment

* go mod tidy

* update go mod

* go mod tidy

* add comment

* update max depth check to >

* go mod tidy

* rename

* [refactor] - Refactor Archive Handling Logic - Part 4: Non-Archive Data Handling and Cleanup (#2704)

* Handle non-archive data within the DefaultHandler

* make structs and methods private

* Remove non-archive data handling within sources

* Handle non-archive data within the DefaultHandler

* rebase

* Remove non-archive data handling within sources

* add gzip

* move diskbuffered rereader setup into handler pkg

* remove DiskBuffereReader creation logic within sources

* move rewind closer

* reduce log verbosity

* make defaultBufferSize a const

* use correct reader

* address comments

* update test

* [feat] - Add Prometheus Metrics for File Handlers (#2705)

* add metrics for file handling

* add metrics for errors

* add metrics for file handling

* add metrics for errors

* fix tests

* add metrics for max archive depth and skipped files

* update error

* skip symlinks and dirs

* update err

* fix err assignment

* add metrics for file handling

* add metrics for errors

* fix tests

* rebase

* add metrics for errors

* add metrics for max archive depth and skipped files

* update error

* skip symlinks and dirs

* update err

* fix err assignment

* rebase

* remove

* update metric to ms

* update comments

* address comments

* reduce indentations

* add metrics for archive depth

* [bug] - Enhanced Archive Handling to Address Interface Constraints (#2710)

* add metrics for file handling

* add metrics for errors

* add metrics for file handling

* add metrics for errors

* fix tests

* add metrics for max archive depth and skipped files

* update error

* skip symlinks and dirs

* update err

* Address incompatible reader to openArchive

* remove nil check

* fix err assignment

* wrap compReader with DiskbufferReader

* add metrics for file handling

* add metrics for errors

* fix tests

* rebase

* add metrics for errors

* add metrics for max archive depth and skipped files

* update error

* skip symlinks and dirs

* update err

* fix err assignment

* rebase

* remove

* update metric to ms

* update comments

* address comments

* reduce indentations

* replace diskbuffereader with bufferedfilereader

* updtes

* add metric back

* [bug] -  Fix bug and simplify git cat-file command execution and output handling (#2719)

* add metrics for file handling

* add metrics for errors

* add metrics for file handling

* add metrics for errors

* fix tests

* add metrics for max archive depth and skipped files

* update error

* skip symlinks and dirs

* update err

* Address incompatible reader to openArchive

* remove nil check

* fix err assignment

* Allow git cat-file blob to complete before trying to handle the file

* wrap compReader with DiskbufferReader

* Allow git cat-file blob to complete before trying to handle the file

* updates

* revert stuff

* update test

* remove

* add metrics for file handling

* add metrics for errors

* fix tests

* rebase

* add metrics for errors

* add metrics for max archive depth and skipped files

* update error

* skip symlinks and dirs

* update err

* fix err assignment

* rebase

* remove

* update metric to ms

* update comments

* address comments

* reduce indentations

* inline
2024-05-10 11:36:06 -07:00
Richard Gomez
98912a98f7
test(common/http): fix panic (#2817) 2024-05-09 18:17:14 -07:00
ahrav
c7b72b9867
address linter (#2783) 2024-05-08 13:58:50 -07:00
Cody Rose
a317897d66
increase test chan size (#2797)
This test has a race condition. This change makes it less likely to cause a test failure, and is a stopgap measure to de-flake the test while we investigate the underlying issue.
2024-05-07 11:11:11 -04:00
Dustin Decker
288003519a
Add webhook source protos (#2789)
* add webhook source protos

* update protos

* update proto

* update protos
2024-05-06 19:43:10 -04:00
ahrav
27eae925de
Use custom fp logic for private keys (#2793) 2024-05-06 14:41:00 -07:00
ahrav
3c659a2144
set default buffer size to 64 (#2778) 2024-05-03 08:42:18 -07:00
Abdul Basit
bf25b74224
Update result's extra data for Slack (#2779)
* add name of team and user in extra data of results, received from slack'api

* adding token type in extra data for slack
2024-05-02 15:16:30 -05:00
Zachary Rice
4ea3a1376b
fix for infinite recursion in Postman var sub (#2780)
* fix for infinite recursion

* oneliner
2024-05-02 13:03:03 -05:00
NIKHIL PANWAR
94a165390b
Update rabbitmq.go regex detect amqps protocol (#2609)
* Update rabbitmq.go regex detect amqps protocol

Old one couldn't detect amqps:// connection string, and only the amqp://

* [Revised] Update rabbitmq.go regex detect amqps protocol

Co-authored-by: Richard Gomez <32133502+rgmz@users.noreply.github.com>

---------

Co-authored-by: Richard Gomez <32133502+rgmz@users.noreply.github.com>
2024-05-01 13:20:54 -05:00
ahrav
7bd1fb1dcc
update imports (#2772) 2024-05-01 11:41:43 -05:00
Ankush Goel
79687683ff
Detector-Competition-Fix - fixed the alchemy detector regex (#1821)
* fixed the alchemy detector

* added the chunk filtering for alcht_
2024-04-30 17:01:13 -05:00
Ankush Goel
770459eb57
Detector-Fix: Reintroduce Cloudflareglobalapikey (#2101)
* fixed cloudflare code

* readd email check

---------

Co-authored-by: Zachary Rice <zachary.rice@trufflesec.com>
2024-04-30 15:11:04 -05:00
Cody Rose
2f7029bc4d
Expose detector-specific false positive logic (#2743)
This PR:

Creates an optional interface that detectors can use to customize their false positive detection
Implements this interface on detectors that have custom logic
In most cases this "custom logic" is simply a no-op because the detector does not participate in false positive detection
Eliminates inline (old-style) false positive exclusion in a few detectors that #2643 missed
2024-04-30 16:10:26 -04:00
Ankush Goel
dc930f9594
fixed calendly api key (#2368) 2024-04-30 14:47:36 -05:00
ahrav
88967192e8
[bug] - Improve BufferedFileReader Close Behavior (#2768)
* Dont allow read after close

* update comment

* remove defer
2024-04-30 08:50:16 -07:00
ahrav
7e47b96631
[feat] - Add ReadFrom method to BufferedFileWriter (#2759)
* Update write method in contentWriter interface

* fix lint

* Add a buffered file reader

* update comments

* update comment

* add compile type checks

* fix

* fix test

* inline if

* Add ReadFrom method to the BufferedFileWriter

* update test

* fix test

* update benchmark
2024-04-30 07:41:05 -07:00
ahrav
46d4ae1334
[feat] - buffered file reader (#2731)
* Update write method in contentWriter interface

* fix lint

* Add a buffered file reader

* update comments

* update comment

* add compile type checks

* fix

* fix test

* inline if

* magic

* update comment
2024-04-30 07:31:37 -07:00
Richard Gomez
13bd783d2d
test(git): change length of chunks (#2767)
This fixes one missed test in #2754 (comment).

The number of chunks doubled because each commit now has metadata + data.
2024-04-30 08:34:12 -04:00
Miccah
6cf3a25a04
[chore] Add some happy path logs to GitLab (#2765) 2024-04-29 16:42:35 -07:00
ahrav
99ae8f8035
Update ignore extensions (#2764) 2024-04-29 15:48:42 -07:00
ahrav
591871977c
Correclty set metrics for enumerated orgs (#2757) 2024-04-29 14:26:46 -07:00
Richard Gomez
11e5febeee
feat(git): scan commit metadata (#2754)
This is a follow-up to #2713 that fixes the strange test error.

As suspected, the failure was caused by additional diffs not being included in the test's expected data.
2024-04-29 16:58:45 -04:00
ahrav
0f122edc59
ignore pbix and vsdx files (#2762) 2024-04-29 12:59:21 -07:00
mountcount
1d92655d97
pkg: fix function names in comment (#2761)
Signed-off-by: mountcount <cuimoman@outlook.com>
2024-04-29 11:21:26 -05:00
ahrav
0df300c0ca
[chore] - add additional binary extension (#2760)
* add rust binary extension

* add additinal binary file extension for PyTorch serialized models
2024-04-29 08:08:00 -07:00
ahrav
5d3b90799e
[bug] - Fix the metric for buffered file writer writes (#2750)
* avoid double counting

* add disk write count back
2024-04-25 13:04:07 -07:00
ahrav
d89b0cdace
[bug] - fix buffer size metric (#2749)
* fix metric

* another fix
2024-04-25 11:43:38 -07:00
Miccah
fadf9c6286
[chore] Remove broken test (#2748)
This wasn't actually testing the fix, which is more difficult to
orchestrate than is worth.

See: https://github.com/trufflesecurity/trufflehog/pull/2742
2024-04-25 11:27:17 -07:00
ahrav
b430dae83e
[refactor] - lazy buffer retrieval (#2745)
* only create the contentWriter once

* update test

* Lazily fetch buffer from the pool

* fix tests

* fix test

* remove ctx
2024-04-25 08:27:15 -07:00
ahrav
8d3404804e
[chore] - update buffered file writer metric (#2740)
* missed one

* add comment
2024-04-25 08:17:32 -07:00
ahrav
8ceeb5d5a1
[bug] - Refactor newDiff constructor to avoid double initialization of contentWriter (#2742)
* only create the contentWriter once

* update test

* correclty use mock

* remove deprecated pkg
2024-04-25 08:01:38 -07:00
Cody Rose
11452e8a57
Revert "feat(git): scan commit metadata (#2713)" (#2747)
This reverts commit 81a9c813a1.
2024-04-25 10:56:48 -04:00
Cody Rose
ba5ad5d8a9
Fix SQL Server detector tests (#2716)
These tests were broken so I fixed them and updated them to use testcontainers, which is more robust and used in the JDBC detector tests.
2024-04-25 10:40:46 -04:00
Richard Gomez
81a9c813a1
feat(git): scan commit metadata (#2713)
This fixes #2683. It scans the commit author, committer (which is typically GitHub <noreply@github.com> for GitHub, but can be different), and message.

It also scans Git notes.
2024-04-25 10:13:09 -04:00
ahrav
97599b19e7
update buffer metrics (#2737) 2024-04-24 07:22:06 -07:00
ahrav
ea4d9d2d32
[bug] - Correctly return the checked out buffer to the pool (#2732)
* Make sure to return the buffer to the pool

* update comment

* defer the return

* remove anonymous function
2024-04-23 14:38:28 -07:00
Dustin Decker
0ce02fc827
Make connection issues less jarring (#2730)
* Make connection issues less jarring

* lint

* fix lint issue

* print just the connection issue in yellow

* update terminology
2024-04-23 14:29:38 -07:00
ahrav
f03aa38726
[bug] - Fix disk write metric and update BufferedFileWriter file field (#2733)
* Record the disk write after the file has been written to

* remove comment

* remove print logs
2024-04-23 14:13:13 -07:00
Cody Rose
af095c294c
Add false positive info to proto (#2729)
This PR adds false positive information to the Result protobuf message in anticipation of us tracking it as first-class secret metadata. We're not doing that yet (it's blocked behind #2643) but setting up the messages now means we'll be able to do it later with less of a code delta.
2024-04-23 16:18:45 -04:00
ahrav
4a5fbf8417
[refactor] - Update Write method signature in contentWriter interface (#2721)
* Update write method in contentWriter interface

* fix lint
2024-04-23 08:47:53 -07:00
Dustin Decker
14e44db2be
Move detectors.IsKnownFalsePositive from the detectors and into the engine (#2643)
* Remove detectors.IsKnownFalsePositive from detectors

* Centralize false positive removal in engine

* Don't apply fp filtering on custom regex to preserve previous behavior.

* fix empty branch

* update excludes

* update filtering

* Add result flag option and exclude some detectors
2024-04-22 15:18:04 -07:00
Ankush Goel
3fa86a1008
added onfleet api key detector (#2375)
* added onfleet detector

* use organization get endpoint

---------

Co-authored-by: Ahrav Dutta <ahrav.dutta@trufflesec.com>
2024-04-21 10:06:51 -07:00
Julien Doutre
32652a7498
Detect Slack workflows webhook (#2569) 2024-04-19 07:21:40 -07:00
Luska
e5575cd6f2
Adding Pagarme API key detection (#2665)
* Adding support to Pagarme API key detection

* adding scanner

---------

Co-authored-by: Ahrav Dutta <ahrav.dutta@trufflesec.com>
2024-04-10 17:57:25 -07:00
ahrav
867434331b
[refactor] - template detector (#2692)
* refactor template detector to only check for 200 status code

* Replace ldap.DialTLS w/ ldap.DialURL since the former is deprecated

* sort imports
2024-04-10 17:46:07 -07:00
fml09
c6b454e736
Remove unnecessary space in Vultr regex pattern (#2689)
* Fix incorrect regular expression with missing closing bracket

* Remove unnecessary space in Vultr regex pattern
2024-04-10 17:12:55 -07:00
Shreyas Sriram
08b6f90c81
Add Wiz detector (#2691)
* Implement wiz detector

* Fix tests

* Update false positive logic
2024-04-10 08:19:36 -07:00
Cody Rose
b745cfd495
Enrich Gitlab enumeration logging (#2678)
This PR modifies the GitLab source:

* emits a new "groups enumerated" metric
* logs more information about group enumeration
* emits the repo enumeration metric inside getAllProjectRepos, which means it will work when units are flipped on
* emits the repo enumeration metric more granularly
2024-04-08 10:47:05 -04:00
Cody Rose
14b1a6e236
Handle inactive Slack account tokens (#2668)
This PR updates the Slack detector to accommodate a previously unhandled error type. It also fixes the exiting Slack tests.
2024-04-05 10:26:20 -04:00
ahrav
3cb7aedf4a
[bug] - Add ASCII validation check for base64 decoding (#2671)
* Correclt handle invalid base64 with ascii check

* remove parallel
2024-04-04 16:59:13 -07:00
redismongo
f1957fec59
chore: fix some typos (#2666) 2024-04-04 05:50:47 -07:00
ahrav
a8132839f8
[chore] - update go-github dep manually (#2664)
* update go-github dep

* remove commented out line
2024-04-03 19:19:14 -07:00
ahrav
0fe39db56f
upgrade launchdarkly dep (#2650) 2024-04-03 07:24:20 -07:00
Richard Gomez
d0d59c2e77
chore: remove duplicate jiratoken.v2 detector (#2657) 2024-04-02 18:12:06 -07:00
kenzht
0d3023fe74
add GCP application default credentials detector (#2530)
* add GCP application default credentials detector

* add a comment

* update Keywords to better match the key

---------

Co-authored-by: Ahrav Dutta <ahrav.dutta@trufflesec.com>
2024-04-01 11:27:40 -07:00
Shreyas Sriram
31ad1eed30
Add JupiterOne detector (#2446)
* Add JupiterOne bootstrap

* Implement verification logic

* Cleanup

* Fix verificationError

* Undo unnecessary changes

---------

Co-authored-by: Ahrav Dutta <ahrav.dutta@trufflesec.com>
2024-03-29 19:14:04 -07:00
Richard Gomez
3b58a15a84
Fix GitHub enumeration & rate-limiting logic (#2625)
This is a follow-up to #2379.

It fixes the following issues:

GitHub API calls missing rate-limit handling
The fix for Refactor GitHub source #2379 (comment) inadvertently resulting in duplicate API calls
2024-03-29 10:29:46 -04:00
dylanTruffle
7fccb52b16
Fixing nitro check (#2631)
* Fixing nitro check

* remove dupe detector and disable Alconst

---------

Co-authored-by: Dylan Ayrey <dylan@Dylans-MacBook-Pro.local>
Co-authored-by: Ahrav Dutta <ahrav.dutta@trufflesec.com>
Co-authored-by: ahrav <ahravdutta02@gmail.com>
2024-03-27 11:39:58 -07:00
Dustin Decker
612ff1a0f1
Use Lstat to identify non-regular files in filesystem source (#2628)
* Use Lstat to identify non-regular files in filesystem source

* fix test
2024-03-26 15:22:42 -07:00
ahrav
369d79080e
[bugfix] - Update the Anthropic detector (#2629)
* Update the Anthropic detector

* update comment
2024-03-26 15:01:32 -07:00
fml09
9da396e237
Fix incorrect regular expression with missing closing bracket (#2616) 2024-03-26 13:32:14 -05:00
ahrav
11afc3215b
[chore] - upgrade dep (#2618)
* upgrade dep

* remove dupe deps
2024-03-26 11:21:07 -07:00
Richard Gomez
95dc8d6e16
Fix additional GitHub test errors #2614 2024-03-26 09:34:12 -04:00
Richard Gomez
9d4cf87c02
fix(github): resolve panic & test failures (#2608) 2024-03-22 09:49:01 -07:00
Ankush Goel
6dbe80806b
Dockerhub v2 detector (#2361)
* Dockerhub v2 detector

* update package structure

---------

Co-authored-by: Dustin Decker <dustin@trufflesec.com>
2024-03-22 09:30:35 -07:00
Richard Gomez
441d9ff005
Update Snyk detector (#2559)
* feat(snyk): add extradata from api

* update test

---------

Co-authored-by: Dustin Decker <dustin@trufflesec.com>
2024-03-22 09:14:48 -07:00
faktas2
72fb2b9deb
MaxMind detector uses the right endpoint (#2577)
* MaxMind detector uses the right endpoint

The endpoint that the current detector uses fails in validating the license key as some license keys do not have permissions to the geoip API. This commit is to make the detector use the right endpoint https://dev.maxmind.com/license-key-validation-api

* Remove RawV2

* Remove trimspace and extra if branch

* Add the proper tests

* Use SetVerificationError

* Add SetVerificationError

* update tests

---------

Co-authored-by: Dustin Decker <dustin@trufflesec.com>
2024-03-22 09:10:44 -07:00
Richard Gomez
baf7ea1458
feat(gitparse): avoid uneeded calls to strconv.Unquote (#2605) 2024-03-22 08:35:10 -07:00
Richard Gomez
80e8a67c2d
Refactor GitHub source (#2379)
* refactor(github): cleanup logic

* fix(github): lookup wikis per-repo

* refactor(github): change scanErrs.String output

---------

Co-authored-by: Bill Rich <bill.rich@gmail.com>
2024-03-21 14:07:39 -07:00
Miccah
3a7266e540
[chore] Fix potential resource leak in postman source (#2606)
This moves workspace unpacking to a helper function to leverage a defer,
which ensures the file is always closed.
2024-03-21 10:21:13 -05:00
Zachary Rice
1216fa23c9
strings contain keyword check, add collection name to keywords (#2602) 2024-03-21 09:35:38 -05:00
Zachary Rice
b11ce72338
Postman Source (#2579)
postman source

Co-authored-by: Miccah <m.castorina93@gmail.com>

---------

Co-authored-by: Joe Leon <joe.leon@trufflesec.com>
Co-authored-by: Miccah Castorina <m.castorina93@gmail.com>
2024-03-20 11:36:20 -05:00
Richard Gomez
aa862e46bb
fix(git): decode unicode paths (#2585) 2024-03-19 08:50:27 -07:00
Miccah
931a28a537
[chore] Replace "Trufflehog" with "TruffleHog" (#2584) 2024-03-18 11:01:25 -07:00
Richard Gomez
fd7e7e6e29
fix(github): response can be nil (#2583) 2024-03-16 01:12:44 -07:00
āh̳̕mͭͭͨͩ̐e̘ͬ́͋ͬ̊̓͂d
2ef7a4a49f
pull out verification logic from github detectors (#2554)
* pull out verification logic from github detectors

* deduplicate verify github logic

* pull out nil check

* return nil instead of empty struct

* skip gh old test bc we can't make new tokens
2024-03-15 15:00:45 -04:00
Richard Gomez
f5025fd382
Add --results flag (#2372)
This is a follow-up to #2107 and #2335. It adds a new (hidden) --results flag that allows a user to show any combination of verified, unverified, and indeterminate secrets.
2024-03-15 10:19:31 -04:00
ahrav
800cc6d90b
[chore] - Record metrics before reset (#2556)
* Record buffer metrics before resetting.

* Address comment.
2024-03-12 09:35:18 -07:00
ahrav
b53054b605
Fix flaky test. (#2564) 2024-03-12 08:31:47 -07:00
ahrav
2e65773b27
use custom grow method (#2555) 2024-03-08 12:16:52 -08:00
joeleonjr
0bbb68931c
Canary verification (#2531)
* verify canaries against SNS; get ARN

* clean comments

* Update tests and logic

* added test for invalid canary secret

* added verify logic for canaries

* go mod tidy

---------

Co-authored-by: Dustin Decker <dustin@trufflesec.com>
2024-03-07 18:18:18 -08:00
āh̳̕mͭͭͨͩ̐e̘ͬ́͋ͬ̊̓͂d
d7a33055ad
add version to extra data + moving existing versioned detectors into subdirectory format (#2471)
* moving existing versioned detectors into subdirectory format

* update docs for adding version number to extra data

* nits
2024-03-07 15:48:27 -05:00
Dustin Decker
dbf6965152
DB is not needed for ping command (#2540) 2024-03-06 17:08:37 -08:00
Cody Rose
b7f08db1ef
Redact secret in git command output (#2539)
When we fail to clone a git repository we log the command output to help with diagnosis. However, this output can include credentials in certain cases (such as certain errors associated with redirects). We don't want to log credentials when this happens.
2024-03-06 11:51:35 -05:00
Cody Rose
28ed81f0a2
Add naive S3 ignorelist (#2536)
This PR adds the ability to exclude buckets from S3 scans. The capability is pretty rudimentary right now, and does not support globbing. If both lists are specified the source to fail to initialize.
2024-03-05 08:01:20 -05:00
Richard Gomez
cbc0f0f48e
Create basic escaped unicode decoder (#2456)
* feat(decoders): basic escaped unicode

* wip: handle unicode notation
Experimenting with this.. might remove
2024-03-02 11:27:44 -08:00
ahrav
3da0c5e125
[feat] - Make the client configurable (#2528)
* Make the client configurable

* add comment

* add backoff option
2024-03-01 13:29:25 -08:00
Dylan Ayrey
7620906b07
Ignore canary IDs in notifications (#2526)
* Update aws.go

* Update aws.go

* Update tests

---------

Co-authored-by: Dustin Decker <dustin@trufflesec.com>
2024-02-28 16:52:50 -08:00
Dustin Decker
8ed0c0ad5d
Remove one filter word (#2525)
Removes a problematic word from the unverified filtering while we work on a more holistic fix.
2024-02-28 15:46:39 -08:00
trufflesteeeve
12ff21f245
Improve Gitlab default URL handling (#2491)
Co-authored-by: Miccah <m.castorina93@gmail.com>
2024-02-28 14:15:11 -05:00
Simon Whitaker
431586ce78
Implement detectors.EndpointCustomizer on datadogtoken (#2510)
Closes #2265
2024-02-28 10:52:01 -06:00
Dustin Decker
ad9d4e53e1
JDBC test and parsing improvements (#2516)
* JDBC test and parsing improvements

- Uses net/url for more robust URI parsing
- Supports common JDBC formats for MySQL
- Supports URI format for MSSQL
- Uses allowlist for params across all drivers
- Uses testcontainers-go for integration testing - much faster, more robust, no port collisions
- Uses gofakeit for random data (db, user, password) generation in integration tests
- Adds connection timeouts
- Use Microsoft's driver for MSSQL

* go mod tidy
2024-02-28 08:51:37 -08:00
Dustin Decker
5805f11ac6
Improve monogo and snowflake detectors (#2518)
* Set timeouts on mongo connection string

* use lightened snowflake driver

* update param
2024-02-28 08:26:27 -08:00
Dustin Decker
2d2ca4d3d6
fix prefix check when returning early (#2503) 2024-02-24 09:15:54 -08:00
Dustin Decker
8a825fde52
Clean up some detectors (#2501) 2024-02-23 15:04:02 -08:00
ahrav
9ef5151200
Gitlab scan targets (#2470)
* add method to scan targets

* Add logic to handle targetted scan

* address comments

* remove pagination opts

* add kvp with scan type
2024-02-23 07:40:52 -08:00
Dustin Decker
d53b83b58e
Identify some canary tokens without detonation (#2500)
* Identify canary tokens

* Update README.md

* Update README.md

---------

Co-authored-by: dylanTruffle <52866392+dylanTruffle@users.noreply.github.com>
2024-02-21 09:42:21 -08:00
Miccah
c60443891b
Add Display method to SourceUnit and Kind member to the CommonSourceUnit (#2450)
* Add Display method to SourceUnit and Kind member to the CommonSourceUnit

* Make SourceUnitID return the ID and a kind

These two values together uniquely represent a unit.
2024-02-20 11:24:13 -08:00
Zachary Rice
bccba20d3e
concurrency uint8 to int (#2488)
* concurrency uint8 to uint16

* jk, use int

* git test fix
2024-02-20 09:35:40 -06:00
ahrav
5290023c2d
use read full (#2474) 2024-02-20 07:21:16 -08:00
ahrav
afccf2cf5f
[chore] - upgrade lru cache version (#2487) 2024-02-19 18:07:31 -08:00
ahrav
41301bec8a
move clenaup outside the engine (#2475) 2024-02-17 08:06:24 -08:00
ahrav
5c313c14db
tighten keyword match (#2473) 2024-02-16 13:38:07 -08:00
Miccah
88c1bb3289
[chore] Increase TestMaxDiffSize timeout (#2472) 2024-02-16 11:09:25 -08:00
Zachary Rice
834163acf5
add lazy quantifier to prefixregex (#2466) 2024-02-15 17:08:27 -06:00
ahrav
40bbab8add
[cleanup] - Extract buffer logic (#2409)
* extract the buffer logic into it's own package

* address comments
2024-02-15 11:40:34 -08:00
ahrav
5568b2e0a6
update gitlab proto (#2469)
* update gitlab proto

* update protos
2024-02-15 10:23:41 -08:00
Zachary Rice
bd729ce48e
add missing prefixregex (#2468) 2024-02-15 07:13:57 -06:00
Dustin Decker
a9817a3292
Remove some noisy / less useful detectors (#2467) 2024-02-14 15:27:03 -08:00
Miccah
216a29d7cf
[chore] Add some doc comments to source manager (#2434) 2024-02-13 07:54:48 -08:00
ahrav
e8006f1bee
2396 since commit stopped working (#2402)
* Ensure we handle commits with no diffs correctly.

* cleanup

* add nil check

* address comments

* move comment

* revert

* add comment
2024-02-13 07:21:22 -08:00
Richard Gomez
9572628dc6
chore(gcp): ignore known test creds (#2413) 2024-02-12 10:29:00 -06:00
Miccah
74f1553e06
[fix] Add unit information to error returned by ChunkUnit (#2410) 2024-02-12 08:24:31 -08:00
renovate[bot]
af6099665f
fix(deps): update module github.com/charmbracelet/bubbletea to v0.25.0 (#2326)
* fix(deps): update module github.com/charmbracelet/bubbletea to v0.25.0

* Remove deprecated and unused mouse events

---------

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: Miccah Castorina <m.castorina93@gmail.com>
2024-02-11 12:11:46 -08:00
Miccah
4acf3ccb80
[chore] Ensure Postgres detector respects context deadline (#2408) 2024-02-10 23:32:05 -08:00
Miccah
8f01326468
[chore] Rename file to legacy_reporters.go (#2406) 2024-02-09 18:17:46 -08:00
Miccah
9642d4c8fd
Add flag to write job reports to disk (#2298)
* Add flag to write job reports to disk

* Fix nil pointer / non-nil interface bug

* Synchronize job report writer goroutine

* Log when the report has been written
2024-02-09 12:30:28 -08:00
Miccah
aace92b64d
Implement SourceUnitEnumChunker for GitLab (#2367)
* Implement SourceUnitEnumChunker for GitLab

* Add GitLab engine integration test

* Use a SliceReporter instead of checking for nil reporters

* Use more generic VisitorReporter

* Merge logic from getReposFromGitlab into getAllProjectRepos

* Update integration test to have a lower bound

Unfortunately, the GitLab integration test does not appear to be
deterministic. Sometimes 36390 chunks are found, sometimes 36312, or
even lower.
2024-02-09 11:06:31 -08:00
Miccah
dd4d4a8a96
Refactor UnitHook to block the scan if finished metrics aren't handled (#2309)
* Refactor UnitHook to block the scan if finished metrics aren't handled

* Log once when back-pressure is detected

* Add hook channel size metric

* Use plural "metrics" for consistency

* Replace LRU cache with map
2024-02-08 14:50:58 -08:00
ahrav
6557b3b321
[feat] - buffered file writer metrics (#2395)
* use diff chan

* correctly use the buffered file writer

* use value from source

* reorder fields

* add tests and update

* Fix issue with buffer slices growing

* fix test

* correctly use the buffered file writer

* use value from source

* reorder fields

* fix

* add singleton

* use shared pool

* optimize

* rename and cleanup

* add metrics

* add print

* rebase

* remove extra inc

* add metrics for checkout time

* add comment

* use microseconds

* add metrics

* add metrics pkg

* add more metrics

* rever test

* remove fields

* fix

* resize and return

* update metric name

* remove comment

* address comments

* add comment
2024-02-08 07:38:40 -08:00
Richard Gomez
3b40c4fa63
Update GitParse to handle quoted binary filenames (#2391)
* fix(gitparse): quoted binary files

* fix(gitparse): use bytes.Cut instead of regexp

* fix lint warning

---------

Co-authored-by: Zachary Rice <zachary.rice@trufflesec.com>
2024-02-08 09:25:04 -06:00
Dustin Decker
a00ffe9522
Allow multiple domains for Forager (#2400) 2024-02-08 07:08:30 -08:00
ahrav
bbf1decb39
prevent concurrent map writes (#2399) 2024-02-07 17:45:06 -08:00
Ryan Jacobchick
7296bcdc5d
Allow CLI version pinning in GHA (#2397) (#2398)
* Allow CLI version pinning in GHA (#2397)

* prevent segfault in test-community
2024-02-07 16:58:04 -06:00
Richard Gomez
b3ff12d1e9
Fix handling of GitHub ratelimit information (#2041)
This is a follow-up to #1912, which used the headers from the response to determine rate-limiting information, instead of using the values from RateLimitError.Rate. Although that logic seemed solid, I discovered that it did not work in some circumstances. This lead to the "unexpected" path more often than intended, and periodic instances where requests would be made before the ratelimit was refreshed.
2024-02-07 09:11:12 -05:00
ahrav
7b492a690a
[feat] - use diff chan (#2387)
* use diff chan

* address comments

* add comment

* address comments

* use old ordering

* add correct author line

* Add required *Commit arg to newDiff

* address comments
2024-02-06 10:06:10 -08:00
ahrav
843334222c
[not-fixup] - Reduce memory consumption for Buffered File Writer (#2377)
* correctly use the buffered file writer

* use value from source

* reorder fields

* use only the DetectorKey as a map field

* correctly use the buffered file writer

* use value from source

* reorder fields

* add tests and update

* Fix issue with buffer slices growing

* fix test

* fix

* add singleton

* use shared pool

* optimize

* rename and cleanup

* use correct calculation to grow buffer

* only grow if needed

* address comments

* remove unused

* remove

* rip out Grow

* address coment

* use 2k default buffer

* update comment allow large buffers to be garbage collected
2024-02-06 09:22:25 -08:00
Richard Gomez
8104611d6e
fix: case-insensitive ext check (#2383) 2024-02-06 10:13:53 -05:00
dylanTruffle
901c851698
tightening opsgenie detection and verification (#2389)
Co-authored-by: Dylan Ayrey <dylan@Dylans-MacBook-Pro.local>
2024-02-05 17:31:09 -08:00
Miccah
01c9ac7b59
Fix binary file hanging bug in git sources (#2388)
Waiting for the sub-command will block until all of `stdout` has been
read. In some cases, we return early due to failed chunking without
reading all of the data, and thus, get stuck waiting for the command to
finish. Closing the pipe will ensure `Wait` does not block on that I/O.
2024-02-05 15:28:49 -08:00
ahrav
135cc3eb69
[fixup] - correctly use the buffered file writer (#2373)
* correctly use the buffered file writer

* use value from source

* reorder fields

* use only the DetectorKey as a map field

* address comments and use factory function

* fix optional params

* remove commented out code
2024-02-05 10:43:55 -08:00
ahrav
28d079bdad
use only the DetectorKey as a map field (#2374) 2024-02-05 06:53:08 -08:00
ahrav
a22874f9f0
[feat] - concurently scan the filesystem source (#2364)
* concurently scan the filesystem source

Co-authored-by: Miccah Castorina <m.castorina93@gmail.com>

* fix test

* update test

* remove return

* use error not info

* address comment

---------

Co-authored-by: Miccah Castorina <m.castorina93@gmail.com>
2024-02-03 10:49:14 -08:00
Miccah
27b30e65ed
[chore] Cleanup GitLab source errors (#2345)
* [chore] Cleanup GitLab source errors

* Ungroup compile time interface checks and revert error message
2024-02-02 20:00:34 -08:00
ahrav
382990a6bd
[bug] - use DetectorKey as the key in the detectorKeysWithResults map (#2366)
* use DetectorKey as the key in the map

* nil check

* update comment
2024-02-02 13:43:56 -08:00
Mike Vanbuskirk
f6546ffaf5
Add s3 credential validation (#2362)
* add string non-empty validation to AWS creds

* clean up import spacing

* syntax fixup

* change to non-empty validation only

* convert to lower snake_case

- https://protobuf.dev/programming-guides/style/#message-field-names
2024-02-02 12:49:46 -05:00
ahrav
b2074ad05d
Polite Verification (#2356)
* draft reverify chunks

* remove

* remove

* reduce dupe map cap

* do not verify chunk

* cli arg and use val for dupe lut

* remove counter

* skipp empty results]

* working on test and normalizing val for comparison

* forgot to save file

* optimize normalize

* reuse map

* remove print

* use levenshtein distance to check dupes

* forgot to leave in emptying map

* use slice

* small tweak

* comment

* use bytes

* praise

* use ctx logger

* add len check

* add comments

* use 8x concurrency for reverifier workers

* revert worker count

* use more workers

* process result directly for any collisions

* continue after decoder match for reverifying

* use map

* use map

* otimization and fix the bug.

* revert worker count

* better option naming

* handle identical secrets in chunks

* update comment

* update comment

* fix test

* use DetecotrKey

* rm out of scope tests and testdata

* rename all reverification elements

* don't re-write map entry

* use correct key

* rename worker, remove log val

* test likelydupe, add eq detector check in loop

* add test

* add comment

* add test

* Set verification error

* Update tests

---------

Co-authored-by: Zachary Rice <zachary.rice@trufflesec.com>
Co-authored-by: Dustin Decker <dustin@trufflesec.com>
2024-02-02 09:29:18 -08:00
Dustin Decker
c2ae31d060
Make AzureDevopsPersonalAccessToken verification more robust (#2359)
* Make AzureDevopsPersonalAccessToken verification more robust

* fix snifftest
2024-02-01 08:40:44 -08:00
roxanne-tampus
143e275272
update azure test files to check rawV2 (#2353) 2024-01-31 08:36:52 -08:00
Miccah
24d0680f5c
[chore] Add filesystem integration test (#2358) 2024-01-31 08:27:57 -08:00
Richard Gomez
8e90c4e669
Scan GitHub wikis #2233 2024-01-31 10:52:24 -05:00
Marlon
91d6496a76
added flyio protos (#2357)
* added flyio protos

* added builtwith proto

---------

Co-authored-by: root <root@ubuntutruffle.myguest.virtualbox.org>
2024-01-31 07:02:06 -08:00
ahrav
9867ce8eb8
Allow for configuring the buffered file writer (#2319)
* Write large diffs to tmp files

* address comments

* Move bufferedfilewriter to own pkg

* update test

* swallow write err

* use buffer pool

* use size vs len

* use interface

* fix test

* update comments

* fix test

* Allow for configuring the buffered file writer

* remove unused

* add missing method

* remove

* remove unused

* move parser and commit struct closer to where they are used

* linter change

* fix snifftest

* address comments

* add more kvp pairs to error

* fix test

* update

* add back missing metadata fields

* address comments

* remove bufferedfile writer

* fix

* address comments

* use unint8

* update interface

* adjust interface

* fix tests

* make linter happy

* fix finalize

* address comments

* update test

* address comments

* lint

* remove guard

* fix test

* fix

* add TODO

* fix tests
2024-01-30 12:51:58 -08:00
ahrav
7c59ff95d5
[feat] - tmp file diffs (#2306)
* Write large diffs to tmp files

* address comments

* Move bufferedfilewriter to own pkg

* update test

* swallow write err

* use buffer pool

* use size vs len

* use interface

* fix test

* update comments

* fix test

* remove unused

* remove

* remove unused

* move parser and commit struct closer to where they are used

* linter change

* add more kvp pairs to error

* fix test

* update

* address comments

* remove bufferedfile writer

* address comments

* adjust interface

* fix finalize

* address comments

* lint

* remove guard

* fix

* add TODO
2024-01-30 12:30:51 -08:00
Miccah
6824eb41ea
Fix filesystem enumeration ignore paths bug (#2355) 2024-01-30 12:21:37 -08:00
āh̳̕mͭͭͨͩ̐e̘ͬ́͋ͬ̊̓͂d
7ece4c3e66
Detectors Updates 1 for Tristate Verification (#2187)
* updating alibaba

* updating agora

* updating aeroworkflow

* updating aha

* updating artifactory

* updating abbysale

* updating abstract

* updating abuseipdb

* updating accuweather

* updating adafruitio

* updating adzuna

* cleanup on abuseipdb

* cleanup on aha

* cleanup on abuseipdb

* cleanup on aeroworkflow

* cleanup on adzuna

* cleanup on accuweather

* cleanup/refactor

* update token pattern to be explicitly 73char (old) or 64char (new)

* comment to clarify 403 on Aha

* mocking out verified case for aha + adding inactive account test

* using contact response instead of gock

* update 403 to be determinate
2024-01-30 12:20:56 -05:00
Richard Gomez
232032410c
feat(detectors): update template (#2342) 2024-01-29 21:21:23 -08:00
Dustin Decker
453792de76
Azure function key is throwing FPs (#2352)
* Merge branch 'main' of github.com:trufflesecurity/trufflehog

* AZF throwing FPs

* update snifftest script
2024-01-29 15:32:12 -05:00
Dustin Decker
7befefd369
Improve fp ignore logic (#2351)
* forgot field change

* use aho corasick for filter

* reduce wordlist sensitivity
2024-01-29 11:28:46 -08:00
roxanne-tampus
303e191f38
added azuresearchquerykey detector (#2349)
Co-authored-by: Dustin Decker <dustin@trufflesec.com>
2024-01-29 10:19:51 -08:00
roxanne-tampus
06b7ebf824
added azuresearchadminkey detector (#2348)
* added azuresearchadminkey detector

* additional update

* update import

* fix raw fields for new detectors and entropy check

---------

Co-authored-by: Dustin Decker <dustin@trufflesec.com>
2024-01-29 09:55:46 -08:00
roxanne-tampus
83dc986b07
added azurefunctionkey detector (#2337)
* added azurefunctionkey detector

* update raw field to include url

* clean up and added prefix on key pattern

* update bench script

* update imports, snifftest, and gen proto

---------

Co-authored-by: Dustin Decker <dustin@trufflesec.com>
2024-01-29 09:20:33 -08:00
āh̳̕mͭͭͨͩ̐e̘ͬ́͋ͬ̊̓͂d
fa1c5fa583
updates to plain and json printing to include verification error (#2335) 2024-01-29 11:57:28 -05:00
faktas2
76fcdae3a0
Add the new MaxMind license key format (#2181)
* Add the new MaxMind license key format

* feedback

* reorg rules

---------

Co-authored-by: Dustin Decker <dustin@trufflesec.com>
2024-01-26 11:49:47 -08:00
Dustin Decker
3a6cfd9d97
Prevent print or logging in detectors (#2341)
* Prevent print or logging in detectors

* mount repo

* update job name
2024-01-26 11:39:41 -08:00
ahrav
41cfec4e1c
make sure to close connections after testing (#2343) 2024-01-26 09:24:06 -08:00
Dustin Decker
f078df43eb
Fix test (#2339) 2024-01-25 23:15:37 -08:00
Zubair Khan
76057618ba
add tri-state verification to yelp (#1736)
* init yelp tri state

* fix detector and test

---------

Co-authored-by: Dustin Decker <dustin@trufflesec.com>
2024-01-25 22:28:16 -08:00
Richard Gomez
38eb5d08e7
Improve GitHub scan logging (#2220)
* feat(github): improve scan logging

* Move metric

---------

Co-authored-by: Dustin Decker <dustin@trufflesec.com>
2024-01-25 22:11:01 -08:00
Richard Gomez
c6f73a7d10
Update DockerHub detector logic (#2266)
* feat(dockerhub): update logic

* updates

---------

Co-authored-by: Dustin Decker <dustin@trufflesec.com>
2024-01-25 22:02:36 -08:00
Richard Gomez
d4a0645c29
Add Google oauth2 token detector (#2274)
* feat(google): add oauth2 token detector

* update pr

---------

Co-authored-by: Dustin Decker <dustin@trufflesec.com>
2024-01-25 21:32:03 -08:00
ahrav
f209b04d5d
add priority semaphore (#2336) 2024-01-24 16:43:56 -08:00
joeleonjr
792266afa9
updating doppler logic (#2329)
* updating doppler logic

* added json response struct
2024-01-24 12:20:33 -05:00
roxanne-tampus
47c6539a41
added azuredevopspersonalaccesstoken detector (#2315)
* added azuredevopspersonalaccesstoken detector

* fix comment

* update raw field to include all parts of the credential

---------

Co-authored-by: Dustin Decker <dustin@trufflesec.com>
2024-01-23 20:46:51 -08:00
Miccah
4c698fc1e8
Walk directories in filesystem source enumeration (#2313)
* Walk directories in filesystem source enumeration

* Ignore all directories instead of just the root

* Fix bug with multiple directories

* Skip filesystem TestEnumerate

* Update filesystem enumeration test to create files and folders
2024-01-23 14:57:38 -08:00