Commit graph

341 commits

Author SHA1 Message Date
Sahil Silare
d78239c9f5
feat: validation & verification fix for apiscience to apimetrics (#3475) 2024-10-22 11:31:32 -05:00
Cody Rose
3ab60865c8
Extract FP logic correctly at other call site #3476 2024-10-21 09:51:53 -04:00
JonZeolla
4ea311dea9
feat: add github comments timeframe filtering (fixes #3388) (#3390)
* feat: add github comments timeframe filtering

* fixup and generate protos

* Cleanup

---------

Co-authored-by: Zachary Rice <zachary.rice@trufflesec.com>
2024-10-15 15:13:36 -04:00
Abdul Basit
aa17b8eba4
[detector] Implemented Box Detector (#3242)
* Implemented a box detector with test cases.

* corrected comments

* remove generic keyword for box detector
remove PII details of user.

* Added Box Oauth detector
Implemented description for Box detectors.
Separated out test for Box detectors.

* removed user information from ExtraData.

---------

Co-authored-by: 0x1 <13666360+0x1@users.noreply.github.com>
2024-10-15 08:42:37 -05:00
Kyle Dodson
58485f3395
Add detector for SaladCloud API Keys (#3273) 2024-10-10 21:23:25 -07:00
Richard Gomez
23afcd77ee
Log skipped files on debug level (#3383) 2024-10-07 20:39:06 -07:00
Kashif Khan
ce5da505a7
Added Cisco Meraki API Key detector (#3367)
* Added cisco meraki apikey detector

* addressed the comments

* handled api response and saving orgs data in extra data

* fixed linter

---------

Co-authored-by: Zachary Rice <zachary.rice@trufflesec.com>
2024-10-07 12:00:45 -05:00
Miccah
0328a19a9d
[fix] Move detector initialization to DefaultDetectors function (#3341) 2024-09-26 14:03:24 -07:00
Miccah
1a0cf04c3d
[chore] Ensure testing Endpoints() doesn't silently pass on change (#3334)
Since Endpoints() isn't a defined interface, we are testing an
implementation detail of EndpointSetter. If that function changes in
anyway, the test will now fail instead of skipping every detector and
passing.
2024-09-25 15:27:13 -07:00
Miccah
4484bf443b
[fix] Correctly initialize detectors with cloud endpoint customization (#3333)
* [fix] Correctly initialize detectors with cloud endpoint customization

We were only initializing if the detector was configured with a custom
endpoint, but not in the default case.

* Add test

* Fix gitlab.v2 detector
2024-09-25 13:06:00 -07:00
Kashif Khan
eb40243984
RailwayApp Detector (#3331)
* Added RailwayApp detector

* Updated Keywords
2024-09-25 10:17:08 -05:00
dylanTruffle
0f427b3c6a
Adding Descriptions (#3258)
* adding AI generated descriptions of the key types and their capabilities

* removing empty file

* Update abbysale.go

* update to interface

* fixes

* fix

* small cleanup

---------

Co-authored-by: Dylan Ayrey <dxa4481@rit.edu>
Co-authored-by: Dustin Decker <dustin@trufflesec.com>
2024-09-24 16:10:16 -07:00
Kashif Khan
4b6957df66
Endpoint customizer refresh (#3308)
* Refresh EndpointCustomizer for more explicit configuration

Also add CloudProvider interface.

* WIP: Update EndpointSetter

* Updated detectors with new endpoint customizer

* Fixed linter

* Added check for appending cloud endpoints

---------

Co-authored-by: Miccah Castorina <m.castorina93@gmail.com>
2024-09-24 11:41:05 -05:00
dylanTruffle
d201e54305
adding pypi detector (#3287)
* adding pypi detector

* update test and use helper

---------

Co-authored-by: Dylan Ayrey <dxa4481@rit.edu>
Co-authored-by: Dustin Decker <dustin@trufflesec.com>
2024-09-11 19:17:17 -07:00
Richard Gomez
b7411d2922
Clarify "no decoder found for chunk" log message (#3001)
* chore(engine): clarify trace log message

* chore(engine): fix merge conflicts
2024-09-10 13:58:40 -05:00
Daniel Teixeira
f24f62832b
Add detector for Nvidia NGC Personal Keys (#3280)
* Add detector for Nvidia NGC Personal Keys

* Update nvapi.go to use `nvapi-` as the keyword
2024-09-10 08:36:33 -07:00
Dustin Decker
db0108f731
Make worker multipliers configurable (#3267) 2024-09-04 11:36:26 -07:00
Shreyas Sriram
15faaba61c
Add Robinhood Crypto detector (#3254)
* Add Robinhood Crypto detector

* Address comment - use single keyword
2024-08-29 14:05:52 -07:00
ahrav
06bbd6fd49
Update buffer (#3255) 2024-08-29 13:40:26 -07:00
Nash
69f5d9b76d
Th 899 postman panic issue (#3245)
* Fixed the checks for local exported data

* Fixed the check for local export files

* Fixed the check for local export files

* Fixed the check for local export files

* Merge branch 'main' into th-899-postman-panic-issue

* minor changes in the tests

* test update

* test
2024-08-26 14:46:05 -04:00
Cody Rose
8f299ff8cd
Skip filtration for targeted scans #3243
There is a scenario in which results filtration is known to cause problems, and this PR disables it in that scenario. (It should cause problems more generally, but lacking any concrete cases of that, I want to tread lightly.)
2024-08-23 10:59:07 -04:00
Cody Rose
f39a5254ff
Customize results cleaning (using smuggled interface) (#3235)
We have identified some cases in which the results "cleaning" logic (the logic that eliminates superfluous results) should not run. In order to allow this, we need to expose the cleaning logic to the engine. This PR does so by doing these things:

- Create a CustomResultsCleaner interface that can be implemented by detectors that want to use custom cleaning logic
- Implement this interface for the aws and awssessionkey detectors (and remove their previous invocation of their custom cleaning logic)
- Modify the engine to invoke this logic (conditionally)

This PR also removes the "custom" cleaning logic for the opsgenie, razorpay, and twilio detectors, because it was added erroneously.

This is an alternative implementation of #3233.
2024-08-21 09:42:20 -04:00
Cody Rose
9718ec6a51
Capture decoding time metric (#3209)
We're trying to track down some slowness.
2024-08-09 15:19:16 -04:00
Cody Rose
239f35921d
Log when a detector ignores the timeout (#3201)
If a detector ignores the configured timeout it is probably because of I/O blocking, which degrades the efficiency of the detector worker pool when it happens a lot. In the worst case, a detector that fully hangs will zombify its worker, causing really bad performance problems. When this happens, we don't really have a good way to notice other than seeing scan throughput drop suspiciously. This PR adds explicit logging when detection takes longer than it should so we have a better chance of catching this.

(This problem theoretically can spring up anywhere, in any worker, but the detector fleet is vast, uses network I/O, and is implemented by a much larger group of people, so this sort of problem is much more likely to slip into detector implementations than anywhere else in the codebase. We could generalize this mechanism, but I don't want to make that investment before seeing if this smaller change captures the information we need.)
2024-08-08 14:58:23 -04:00
Dustin Decker
c2e5506b95
Change log verbosity for detection errors (#3171) 2024-08-04 20:47:41 -07:00
ahrav
0a3451a1ba
[bug] - Create a new context with timeout per request (#3163)
* Create a new context with timeout per request

* match timeout

* use context timeout

* reduce timeout
2024-08-02 14:46:37 -07:00
ahrav
29613220b0
[chore] - log detector type on error (#3159)
* log detector type on error

* update error message

* update log

* update message
2024-08-02 10:54:59 -07:00
joeleonjr
7d606e2480
CFOR Commit Scanner (#3145)
* alpha feature for scanning hidden commits on github

* improvements re: git operations

* lint updates

* updating with exec block due to no gh token

* reworked logic into new source

* fixed collisions threshold flag input

* fixed IOutil issues

* removed additions from GH config

---------

Co-authored-by: Joe Leon <joe.leon@trufflesec.com>
2024-08-01 23:04:20 -04:00
Miccah
38e844f968
[chore] Only set default detectors if none are provided (#3147) 2024-08-01 17:15:06 -07:00
Harmon Herring
f664472da1
Include default detectors when using a config that contains detectors (#3115)
* include default detectors when config file is used

* fix test
2024-07-29 14:36:40 -07:00
trufflesteeeve
c01428d107
Remove onwater detector (#3088) 2024-07-22 17:00:32 -04:00
Abdul Basit
5b64e1e5a1
implemented a netsuite detector (#3068)
* implemented a netsuite detector

* implemented the netsuite detector with modified test.

* clean up go.sum by running `go mod tidy`

* implemented a netsuite detector

* implemented the netsuite detector with modified test.

* clean up go.sum by running `go mod tidy`

* Incorporated suggestion by Ahrav
	- optimized nonce generation logic.
	- use string builder as compare to concatenation.

* fix go.sum

* fix import

* fix

---------

Co-authored-by: Ahrav Dutta <ahrav.dutta@trufflesec.com>
2024-07-22 12:37:18 -07:00
ahrav
8f172b23ac
[chore] - Reduce VerificationOverlapWorkers (#3082)
* reduce worker count

* reduce detector worker count
2024-07-19 18:09:57 -07:00
Abdul Basit
c20ca0d6a1
Added Twitter v2 Detector (#3016)
* implemented a newer version of twitter included a test.
moved old twitter detector to as V1.

* added version information in twitter test
2024-07-18 12:15:22 -04:00
Ankush Goel
e2e3b870b6
Atlassian Token Detector (#3065)
* atlassian token

* Update atlassian.go

Fixed version

* fixed regex
2024-07-17 12:44:39 -07:00
dylanTruffle
47535830c4
Elevenlabs detector (#3023)
* adding v1 eleven labs

* updating elevenlabs to support old and new version

* fixing status codes

* lint fixes

* adding test for v2

* adding test for v1

* return err

---------

Co-authored-by: Dylan Ayrey <dylan@Dylans-MacBook-Pro.local>
Co-authored-by: āh̳̕mͭͭͨͩ̐e̘ͬ́͋ͬ̊̓͂d <13666360+0x1@users.noreply.github.com>
2024-07-03 10:53:20 -04:00
joeleonjr
01a1499600
New Source: HuggingFace (#3000)
* initial spike on hf

* added in user and org enum

* adding huggingface source

* updated with lint suggestions

* updated readme

* addressing resources that require org approval to access

* removing unneeded code

* updating with new error msg for 403

* deleted unused code + added resource check in main
2024-06-27 13:22:06 -04:00
Shreyas Sriram
e9206c66bb
Add endorlabs detector (#3015)
* Add endorlabs detector

* Remove unrelated changes

* Addrss comments

* remove prefix regex

---------

Co-authored-by: Ahrav Dutta <ahrav.dutta@trufflesec.com>
2024-06-26 21:28:19 -07:00
Abdul Basit
dddeca5224
Adding Larksuite Detectors + Tests (#3008)
* implemented larksuite detectores for tokens and api keys.
test implemented for larksuite token based detectors.

* implemented test for larksuiteapikey detector

* load credentials from GCP secret manager for larksuite api keys
2024-06-24 11:05:56 -05:00
ahrav
347e8a6683
[feat] - Add Option to Retain False Positives During Detection (#2967)
* provide a mechanism to retain false positive findings

* update

* reorganzie

* revert comment

* update test

* typo

* fix test

* fix test

* update

* update
2024-06-18 09:40:21 -07:00
Cody Rose
15719c2c4f
Fix integration tests (#2970)
The recent engine changes (#2887) broke some integration tests. This PR fixes them. I also fixed an unrelated linter issue.
2024-06-14 14:30:18 -04:00
Richard Gomez
2964b3b2d2
feat(detectors): log falsepositive reason (#2969) 2024-06-14 08:26:05 -07:00
ahrav
523a915143
add metrics to the pipeline (#2968) 2024-06-14 07:57:52 -07:00
āh̳̕mͭͭͨͩ̐e̘ͬ́͋ͬ̊̓͂d
7bf3a9b5e2
adding eraser ai detector (#2961)
* adding eraser ai detector

* add eraser to defaults
2024-06-14 10:10:37 -04:00
ahrav
cb072603dc
Modularize scanning engine (#2887)
* POC: Modularize scanning engine.

* fix typo

* update interface name

* fix tests

* update test

* fix moar tests

* fix bug

* fixes.

* fix merge

* add detector verification overrides

* handle --no-verification flag

* support fp

* add test

* update name

* filter

* update test

* explicit use of detector

* updates
2024-06-13 13:47:09 -07:00
Abdul Basit
cb4d332cbf
adding twitter + Consumer key detector (#2963)
* updated the twitter regex.

* updated regex for bearer token.

* clean up the code for existing twitter detector
added and Implemented new detector for twitter consumer key & secrets with test.
proto generated.

* string updated.

* written test for twitter consumer key detector

* reverted the file to avoid conflicts

* corrected the regex library in twitter detector
2024-06-13 09:32:24 -04:00
ahrav
bf77251543
[feat] - Update span calculation logic to use offset magnitude (#2957)
* Add a default start offset

* update

* use keywordIdx
2024-06-11 09:12:31 -07:00
ahrav
68bea576db
[fix] - Refactor Filtering Logic to Fix Known False Positive Handling in Overlapping Cases (#2946)
* Filter results for verification overlap results

* add test
2024-06-11 07:25:12 -07:00
ahrav
bef4a46b65
[fix] - Correctly calculate EntireSpanChunkCalculator span (#2924)
* fix bug when calculating the span for the entire span chunk calculator

* fix rename
2024-06-05 19:21:57 -07:00
ahrav
babe48fdd1
remove stutter in naming (#2926) 2024-06-05 18:36:08 -07:00