Commit graph

2389 commits

Author SHA1 Message Date
ahrav
9873c144ee
[chore] - Update GitlabV2 detector (#2840)
* replace keyword and replace prefix

* address comment
2024-05-13 14:13:23 -07:00
Abdul Basit
f527da9ecc
Update results's extra data for Twilio (#2807)
* Response structure added for service api of Twilio.
added two response fields in extra data:
1) friendly_name
2) account_sid

* mark credentials verified for non-fatal errors.
also check for atleast one service in response before extracting metadata.
2024-05-13 10:09:35 -04:00
ahrav
570cec7565
[refactor] - Refactor Archive Handling Logic (#2703)
* Remove specialized handler and archive struct and restructure handlers pkg.

* Refactor RPM archive handlers to use a library instead of shelling out

* make rpm handling context aware

* update test

* Refactor AR/deb archive handler to use an existing library instead of shelling out

* Update tests

* add max size check

* add filename and size to context kvp

* move skip file check and is binary check before opening file

* fix test

* preserve existing funcitonality of not handling non-archive files in HandleFile

* Adjust check for rpm/deb archive type

* add additional deb mime type

* update comment

* Remove specialized handler and archive struct and restructure handlers pkg.

* Refactor RPM archive handlers to use a library instead of shelling out

* make rpm handling context aware

* update test

* Refactor AR/deb archive handler to use an existing library instead of shelling out

* Update tests

* add max size check

* add filename and size to context kvp

* move skip file check and is binary check before opening file

* fix test

* preserve existing funcitonality of not handling non-archive files in HandleFile

* Adjust check for rpm/deb archive type

* add additional deb mime type

* update comment

* go mod tidy

* update go mod

* go mod tidy

* add comment

* update max depth check to >

* go mod tidy

* rename

* [refactor] - Refactor Archive Handling Logic - Part 4: Non-Archive Data Handling and Cleanup (#2704)

* Handle non-archive data within the DefaultHandler

* make structs and methods private

* Remove non-archive data handling within sources

* Handle non-archive data within the DefaultHandler

* rebase

* Remove non-archive data handling within sources

* add gzip

* move diskbuffered rereader setup into handler pkg

* remove DiskBuffereReader creation logic within sources

* move rewind closer

* reduce log verbosity

* make defaultBufferSize a const

* use correct reader

* address comments

* update test

* [feat] - Add Prometheus Metrics for File Handlers (#2705)

* add metrics for file handling

* add metrics for errors

* add metrics for file handling

* add metrics for errors

* fix tests

* add metrics for max archive depth and skipped files

* update error

* skip symlinks and dirs

* update err

* fix err assignment

* add metrics for file handling

* add metrics for errors

* fix tests

* rebase

* add metrics for errors

* add metrics for max archive depth and skipped files

* update error

* skip symlinks and dirs

* update err

* fix err assignment

* rebase

* remove

* update metric to ms

* update comments

* address comments

* reduce indentations

* add metrics for archive depth

* [bug] - Enhanced Archive Handling to Address Interface Constraints (#2710)

* add metrics for file handling

* add metrics for errors

* add metrics for file handling

* add metrics for errors

* fix tests

* add metrics for max archive depth and skipped files

* update error

* skip symlinks and dirs

* update err

* Address incompatible reader to openArchive

* remove nil check

* fix err assignment

* wrap compReader with DiskbufferReader

* add metrics for file handling

* add metrics for errors

* fix tests

* rebase

* add metrics for errors

* add metrics for max archive depth and skipped files

* update error

* skip symlinks and dirs

* update err

* fix err assignment

* rebase

* remove

* update metric to ms

* update comments

* address comments

* reduce indentations

* replace diskbuffereader with bufferedfilereader

* updtes

* add metric back

* [bug] -  Fix bug and simplify git cat-file command execution and output handling (#2719)

* add metrics for file handling

* add metrics for errors

* add metrics for file handling

* add metrics for errors

* fix tests

* add metrics for max archive depth and skipped files

* update error

* skip symlinks and dirs

* update err

* Address incompatible reader to openArchive

* remove nil check

* fix err assignment

* Allow git cat-file blob to complete before trying to handle the file

* wrap compReader with DiskbufferReader

* Allow git cat-file blob to complete before trying to handle the file

* updates

* revert stuff

* update test

* remove

* add metrics for file handling

* add metrics for errors

* fix tests

* rebase

* add metrics for errors

* add metrics for max archive depth and skipped files

* update error

* skip symlinks and dirs

* update err

* fix err assignment

* rebase

* remove

* update metric to ms

* update comments

* address comments

* reduce indentations

* inline
2024-05-10 11:36:06 -07:00
Richard Gomez
98912a98f7
test(common/http): fix panic (#2817) 2024-05-09 18:17:14 -07:00
ahrav
c7b72b9867
address linter (#2783) 2024-05-08 13:58:50 -07:00
Cody Rose
a317897d66
increase test chan size (#2797)
This test has a race condition. This change makes it less likely to cause a test failure, and is a stopgap measure to de-flake the test while we investigate the underlying issue.
2024-05-07 11:11:11 -04:00
Dustin Decker
288003519a
Add webhook source protos (#2789)
* add webhook source protos

* update protos

* update proto

* update protos
2024-05-06 19:43:10 -04:00
ahrav
27eae925de
Use custom fp logic for private keys (#2793) 2024-05-06 14:41:00 -07:00
ahrav
3c659a2144
set default buffer size to 64 (#2778) 2024-05-03 08:42:18 -07:00
Abdul Basit
bf25b74224
Update result's extra data for Slack (#2779)
* add name of team and user in extra data of results, received from slack'api

* adding token type in extra data for slack
2024-05-02 15:16:30 -05:00
Zachary Rice
4ea3a1376b
fix for infinite recursion in Postman var sub (#2780)
* fix for infinite recursion

* oneliner
2024-05-02 13:03:03 -05:00
NIKHIL PANWAR
94a165390b
Update rabbitmq.go regex detect amqps protocol (#2609)
* Update rabbitmq.go regex detect amqps protocol

Old one couldn't detect amqps:// connection string, and only the amqp://

* [Revised] Update rabbitmq.go regex detect amqps protocol

Co-authored-by: Richard Gomez <32133502+rgmz@users.noreply.github.com>

---------

Co-authored-by: Richard Gomez <32133502+rgmz@users.noreply.github.com>
2024-05-01 13:20:54 -05:00
ahrav
7bd1fb1dcc
update imports (#2772) 2024-05-01 11:41:43 -05:00
Ankush Goel
79687683ff
Detector-Competition-Fix - fixed the alchemy detector regex (#1821)
* fixed the alchemy detector

* added the chunk filtering for alcht_
2024-04-30 17:01:13 -05:00
Ankush Goel
770459eb57
Detector-Fix: Reintroduce Cloudflareglobalapikey (#2101)
* fixed cloudflare code

* readd email check

---------

Co-authored-by: Zachary Rice <zachary.rice@trufflesec.com>
2024-04-30 15:11:04 -05:00
Cody Rose
2f7029bc4d
Expose detector-specific false positive logic (#2743)
This PR:

Creates an optional interface that detectors can use to customize their false positive detection
Implements this interface on detectors that have custom logic
In most cases this "custom logic" is simply a no-op because the detector does not participate in false positive detection
Eliminates inline (old-style) false positive exclusion in a few detectors that #2643 missed
2024-04-30 16:10:26 -04:00
Ankush Goel
dc930f9594
fixed calendly api key (#2368) 2024-04-30 14:47:36 -05:00
ahrav
88967192e8
[bug] - Improve BufferedFileReader Close Behavior (#2768)
* Dont allow read after close

* update comment

* remove defer
2024-04-30 08:50:16 -07:00
ahrav
7e47b96631
[feat] - Add ReadFrom method to BufferedFileWriter (#2759)
* Update write method in contentWriter interface

* fix lint

* Add a buffered file reader

* update comments

* update comment

* add compile type checks

* fix

* fix test

* inline if

* Add ReadFrom method to the BufferedFileWriter

* update test

* fix test

* update benchmark
2024-04-30 07:41:05 -07:00
ahrav
46d4ae1334
[feat] - buffered file reader (#2731)
* Update write method in contentWriter interface

* fix lint

* Add a buffered file reader

* update comments

* update comment

* add compile type checks

* fix

* fix test

* inline if

* magic

* update comment
2024-04-30 07:31:37 -07:00
Richard Gomez
13bd783d2d
test(git): change length of chunks (#2767)
This fixes one missed test in #2754 (comment).

The number of chunks doubled because each commit now has metadata + data.
2024-04-30 08:34:12 -04:00
Miccah
6cf3a25a04
[chore] Add some happy path logs to GitLab (#2765) 2024-04-29 16:42:35 -07:00
ahrav
99ae8f8035
Update ignore extensions (#2764) 2024-04-29 15:48:42 -07:00
ahrav
591871977c
Correclty set metrics for enumerated orgs (#2757) 2024-04-29 14:26:46 -07:00
Richard Gomez
11e5febeee
feat(git): scan commit metadata (#2754)
This is a follow-up to #2713 that fixes the strange test error.

As suspected, the failure was caused by additional diffs not being included in the test's expected data.
2024-04-29 16:58:45 -04:00
ahrav
0f122edc59
ignore pbix and vsdx files (#2762) 2024-04-29 12:59:21 -07:00
mountcount
1d92655d97
pkg: fix function names in comment (#2761)
Signed-off-by: mountcount <cuimoman@outlook.com>
2024-04-29 11:21:26 -05:00
ahrav
0df300c0ca
[chore] - add additional binary extension (#2760)
* add rust binary extension

* add additinal binary file extension for PyTorch serialized models
2024-04-29 08:08:00 -07:00
ahrav
5d3b90799e
[bug] - Fix the metric for buffered file writer writes (#2750)
* avoid double counting

* add disk write count back
2024-04-25 13:04:07 -07:00
ahrav
d89b0cdace
[bug] - fix buffer size metric (#2749)
* fix metric

* another fix
2024-04-25 11:43:38 -07:00
Miccah
fadf9c6286
[chore] Remove broken test (#2748)
This wasn't actually testing the fix, which is more difficult to
orchestrate than is worth.

See: https://github.com/trufflesecurity/trufflehog/pull/2742
2024-04-25 11:27:17 -07:00
ahrav
b430dae83e
[refactor] - lazy buffer retrieval (#2745)
* only create the contentWriter once

* update test

* Lazily fetch buffer from the pool

* fix tests

* fix test

* remove ctx
2024-04-25 08:27:15 -07:00
ahrav
8d3404804e
[chore] - update buffered file writer metric (#2740)
* missed one

* add comment
2024-04-25 08:17:32 -07:00
ahrav
8ceeb5d5a1
[bug] - Refactor newDiff constructor to avoid double initialization of contentWriter (#2742)
* only create the contentWriter once

* update test

* correclty use mock

* remove deprecated pkg
2024-04-25 08:01:38 -07:00
Cody Rose
11452e8a57
Revert "feat(git): scan commit metadata (#2713)" (#2747)
This reverts commit 81a9c813a1.
2024-04-25 10:56:48 -04:00
Cody Rose
ba5ad5d8a9
Fix SQL Server detector tests (#2716)
These tests were broken so I fixed them and updated them to use testcontainers, which is more robust and used in the JDBC detector tests.
2024-04-25 10:40:46 -04:00
Richard Gomez
81a9c813a1
feat(git): scan commit metadata (#2713)
This fixes #2683. It scans the commit author, committer (which is typically GitHub <noreply@github.com> for GitHub, but can be different), and message.

It also scans Git notes.
2024-04-25 10:13:09 -04:00
ahrav
97599b19e7
update buffer metrics (#2737) 2024-04-24 07:22:06 -07:00
ahrav
ea4d9d2d32
[bug] - Correctly return the checked out buffer to the pool (#2732)
* Make sure to return the buffer to the pool

* update comment

* defer the return

* remove anonymous function
2024-04-23 14:38:28 -07:00
Dustin Decker
0ce02fc827
Make connection issues less jarring (#2730)
* Make connection issues less jarring

* lint

* fix lint issue

* print just the connection issue in yellow

* update terminology
2024-04-23 14:29:38 -07:00
ahrav
f03aa38726
[bug] - Fix disk write metric and update BufferedFileWriter file field (#2733)
* Record the disk write after the file has been written to

* remove comment

* remove print logs
2024-04-23 14:13:13 -07:00
Cody Rose
af095c294c
Add false positive info to proto (#2729)
This PR adds false positive information to the Result protobuf message in anticipation of us tracking it as first-class secret metadata. We're not doing that yet (it's blocked behind #2643) but setting up the messages now means we'll be able to do it later with less of a code delta.
2024-04-23 16:18:45 -04:00
ahrav
4a5fbf8417
[refactor] - Update Write method signature in contentWriter interface (#2721)
* Update write method in contentWriter interface

* fix lint
2024-04-23 08:47:53 -07:00
Dustin Decker
14e44db2be
Move detectors.IsKnownFalsePositive from the detectors and into the engine (#2643)
* Remove detectors.IsKnownFalsePositive from detectors

* Centralize false positive removal in engine

* Don't apply fp filtering on custom regex to preserve previous behavior.

* fix empty branch

* update excludes

* update filtering

* Add result flag option and exclude some detectors
2024-04-22 15:18:04 -07:00
Ankush Goel
3fa86a1008
added onfleet api key detector (#2375)
* added onfleet detector

* use organization get endpoint

---------

Co-authored-by: Ahrav Dutta <ahrav.dutta@trufflesec.com>
2024-04-21 10:06:51 -07:00
Julien Doutre
32652a7498
Detect Slack workflows webhook (#2569) 2024-04-19 07:21:40 -07:00
Luska
e5575cd6f2
Adding Pagarme API key detection (#2665)
* Adding support to Pagarme API key detection

* adding scanner

---------

Co-authored-by: Ahrav Dutta <ahrav.dutta@trufflesec.com>
2024-04-10 17:57:25 -07:00
ahrav
867434331b
[refactor] - template detector (#2692)
* refactor template detector to only check for 200 status code

* Replace ldap.DialTLS w/ ldap.DialURL since the former is deprecated

* sort imports
2024-04-10 17:46:07 -07:00
fml09
c6b454e736
Remove unnecessary space in Vultr regex pattern (#2689)
* Fix incorrect regular expression with missing closing bracket

* Remove unnecessary space in Vultr regex pattern
2024-04-10 17:12:55 -07:00
Shreyas Sriram
08b6f90c81
Add Wiz detector (#2691)
* Implement wiz detector

* Fix tests

* Update false positive logic
2024-04-10 08:19:36 -07:00