Commit graph

95 commits

Author SHA1 Message Date
Brendan Shaklovitz
10902f802a
Add max object size flag for s3 bucket scanning (#1294)
Co-authored-by: Dustin Decker <dustin@trufflesec.com>
2023-04-26 15:39:43 -07:00
ahrav
622700b6ec
update proto to allow for ignoring projects. (#1289) 2023-04-26 07:30:43 -07:00
Shabbir B
6f801f64c7
Added a new detector for percy.io (#1284)
* Feature: Added a new detector for percy.io

* Updated variable name

---------

Co-authored-by: ahrav <ahravdutta02@gmail.com>
2023-04-25 13:18:34 -07:00
ahrav
cec1543894
Add utf16 decoder proto. (#1276) 2023-04-20 15:25:36 -07:00
Zubair Khan
6dd24d17d4
Switch Endpoint Field to Client ID (#1270)
* no longer using endpoint, but do need client id

* use oauth2 credential type
2023-04-19 16:54:37 -04:00
ahrav
5b2b434a8a
Allow multipel team IDs. (#1259) 2023-04-19 11:47:25 -07:00
ah̳̕mͭͭͨͩ̐e̘ͬ́͋ͬ̊̓͂d
1c1f034468
Adding Google drive to MetaData proto (#1264) 2023-04-19 10:49:40 -04:00
ahrav
531e3ef6c3
Add team name to proto. (#1258) 2023-04-14 09:07:23 -07:00
Zubair Khan
2c1d8fbba4
THOG-920/add oss proto (#1240)
* add sharepoint source proto

* create sharepoint oss protos

* add email field, remove oauth2 type, update token to refresh_token

* rename path to link

* restore clientcredentials

* restore s3 and confluence proto changes from make command
2023-04-10 10:30:47 -07:00
iamjpotts
b3d917f9c7
Resolve #1167 by adding support for the AWS_SESSION_TOKEN (#1170)
* Resolve #1167 by adding support for the AWS_SESSION_TOKEN environment variable and adding a --session-token cli arg

* fix error message

---------

Co-authored-by: Dustin Decker <dustin@trufflesec.com>
2023-04-03 14:56:43 -07:00
ahrav
dfc38a135a
Add oauth2 cred as auth type for Teams. (#1221) 2023-03-29 19:37:41 -07:00
ahrav
b8467ee17e
Add Oauth creds to GCS. (#1212) 2023-03-27 10:29:21 -07:00
garg472
3e4496156c
added new detectors and fixed mesibo detector (#1166)
* added new detectors and fixed mesibo detector

* added bscscan.com API detector

* added coinmarketcap detector

* update alchemy

* update blocknative

* update bscscan test

* update cmc test

* update tests

---------

Co-authored-by: Dustin Decker <dustin@trufflesec.com>
2023-03-16 18:57:08 -07:00
ahrav
c617bd7a4e
Add resuming capability to GCS source (#1161)
* Add resuming capability to GCS source.

* Handle no auth scans.

* complete resume logic

* Use custom function type.

* remove functions.

* linter.

* fix test.

* fix test.

* Handle concurrent map writes.

* use string as CLI flag for include/exclude.

* handle emtpy buckets.

* Handle enumeration on initial job run.

* Rename stats to attributes.

* remove redundant return.

* If test fails due to 400, that is fine, it's expected.

* Add unauth GCS source type.

* comments.

* update proto.

* Use short flag.

* address comments.
2023-03-16 17:53:42 -07:00
ahrav
6193509098
add support for json service account and service account file. (#1185) 2023-03-16 13:04:36 -07:00
ahrav
17e8d7030e
Add unauth GCS source type. (#1178) 2023-03-13 16:54:45 -07:00
ahrav
cbf299aa77
Add gcs scanning integration (#1153)
* Setup for GCS scanning.

* Update GCS engine w/ projectID req.

* Add concurrency field to gcsManager.

* add errgroup to gcsManager.

* Update gcs manager.

* Use defautl ADC.

* use ADC.'

* Add TOOD.

* add log to iterator completion.

* use a BinaryReader instead of concrete object for channel type.

* initial test for Chunks.

* Add tests for chunking objects.

* Add concurrency.

* update metadata to include content type and acls.

* Add object reading code.

* Add integration test.

* Add entrypoint.

* Add removed wg.Wait().

* remove dead code.

* remove build.

* Remove period from file extension.

* remove used.

* Add comment.

* Setup for GCS scanning.

* Update GCS engine w/ projectID req.

* Add concurrency field to gcsManager.

* add errgroup to gcsManager.

* Update gcs manager.

* Use defautl ADC.

* use ADC.'

* Add TOOD.

* add log to iterator completion.

* use a BinaryReader instead of concrete object for channel type.

* initial test for Chunks.

* Add tests for chunking objects.

* Add concurrency.

* update metadata to include content type and acls.

* Add object reading code.

* Add integration test.

* Add entrypoint.

* Add removed wg.Wait().

* remove dead code.

* remove build.

* remove used.

* Add file type for objects.

* Add check for file type and size.

* Add default file size.

* Add additinoal auth options and remaining CLI flags.

* Handle errors in go routines.

* Handle resuming for buckets.

* Remove redundant words in comment.

* remove ok check on bool check.

* remove extra blank line.

* Add return if handler handles chunk.

* Add comment.

* remove extra blank line.

* cleanup comment.

* Add comment.

* move up fxn.

* go mod tidy.

* Add exclusion to perf testing buckets.

* Handle blocking the channel.

* remove unused const.

* fix tests.

* fix tests.

* Handle gcs manger options better.

* update fxn name.

* Remove arg name.

* ignore buckets in gcsManager test.

* fix test.

* propulate gsManagerOpts.

* inline err check.

* Add readme.

* update readme spelling.

* fix test.
2023-03-07 17:32:04 -08:00
Miccah
c5b4d6f28b
Support file scanning in filesystem source (#1030)
* Rename directories to paths

* Generate protos

* Add file scanning support to filesystem source

* Add directories back to filesystem proto

* Generate protos

* Combine paths and directories from in source

* Add filesystem filter

* Address comments
2023-02-27 12:15:05 -06:00
ah̳̕mͭͭͨͩ̐e̘ͬ́͋ͬ̊̓͂d
ee5b028c67
Adding initial protos for Google Drive scanner (#1121) 2023-02-22 10:04:46 -05:00
Dustin Decker
26c455d302
add more confluence options (#1105) 2023-02-13 13:58:02 -08:00
ahrav
3f30216356
Add incclude exclude spaces for confluence source. (#1073) 2023-02-06 08:49:45 -08:00
ahrav
e85411b59f
Add file to confluence proto. (#1049) 2023-01-26 17:34:01 -08:00
ahrav
2088f030f9
Add location to Teams metadata. (#1034) 2023-01-23 08:12:05 -08:00
ahrav
009756dce6
add proto that was missing. (#986) 2022-12-23 13:27:07 -08:00
Miccah
861ad057c7
Implement CustomRegex detector (#950)
* Remove verifying successRanges because it is unused in webhook

* Move custom_detectors validation code into its own file

* Initial implementation of custom regex detector

Secret verification is done via webhook.

* Add CustomRegex detector type

* Add upper bound to permutation

* Return early if the context is canceled

* Add headers from configuration

* Add detector name as a key in the JSON body

* Implement faster algorithm for productIndices
2022-12-14 10:26:53 -06:00
Miccah
4409210b87
Add custom detectors configuration parsing (#927)
* Add custom_detectors proto

* Generate proto code

* Create custom_detectors package

Also create protoyaml package to test YAML unmarshalling the
configuration.

* Simplify custom_detectors proto by removing connection

* Generate proto code

* Update custom_detectors parsing tests
2022-11-21 15:10:38 -06:00
Jessica
6e25664a52
add rambbitmq detector (#936)
* add rambbitmq detector

* use fixed length redaction

Co-authored-by: Dustin Decker <dustin@trufflesec.com>
2022-11-21 10:47:41 -08:00
Dustin Decker
ae4b387448
add LDAP detector (#896) 2022-11-18 19:45:11 -08:00
kstilwell
ecd25784f5
Adding Shopify detector (#875)
* Fixes/work based on testing

* Remove some commented code

* Change how verification happens and grab additional information

* Address linter warnings.

* add shopify detector to default detectors.

Co-authored-by: Dustin Decker <dustin@trufflesec.com>
Co-authored-by: Ahrav Dutta <ahravdutta02@gmail.com>
2022-11-08 16:21:57 -08:00
trufflesteeeve
e8cd2e7fae
Add Unknown visibility level (#902) 2022-11-04 14:28:20 -04:00
Dustin Decker
3a143f095b
add option to skip tls verification for confluence (#900) 2022-11-03 13:10:01 -07:00
Dustin Decker
a7fc12240f
Do local URI verification, while attempting to defuse SSRF (#879)
* simplify monogo pattern

* do URI verification locally, while attempting to defuse SSRF

* test SSRF defuse

* simplify err check logic per linter recommendation

* split up detectors

* address comments

* remove unused var
2022-11-01 17:27:24 -07:00
Alexandr Marchenko
60464da3ce
proposal: SqlServer connection string detector (#867)
* sqlserver added to detectors.proto

* make protos

* boilerplate detector generated

* wireup

* initial
2022-10-26 07:46:13 -07:00
ahrav
c203eef86f
[THOG-767] - Ignore Bitbucket and Gitlab repos (#852)
* Add messages to BB and Gitlab source protos to allow ignoring repos.

* remove unsued field in struct.j

* Fix casing.
2022-10-18 14:14:04 -07:00
ahrav
cea2a23c56
[THOG-768] - Add ignore repo list to Github proto (#843)
* Add ignore repo list to Github proto.

* Add proto.

* Add missing proto.
2022-10-11 15:41:33 -07:00
Miccah
2bc4985061
Add SSH config option for the git source (#830)
* Add SSH config option for the git source

The auth message is empty since we use the git binary underneath to
handle the SSH authentication.

* Import digitaloceanv2
2022-09-28 20:40:01 +02:00
Mildred Bernardo
80dcfbe9db
Added DigitalOceanV2 detector (#828) 2022-09-27 17:51:10 -07:00
trufflesteeeve
02310a64f3
Add token auth to JIRA proto (#824) 2022-09-27 15:39:51 -04:00
Bill Rich
1c00014051
Include public/private in github metadata (#812)
* Include public/private in github metadata

* CR feedback

* Fix typos and naming
2022-09-26 14:55:46 -07:00
Joseph Lucas
b02cf7e032
Adding detector for Nvidia NGC (#797)
* template

* minimum viable regex

* valid api 401

* passing tests

* snake to camelcase
2022-09-20 08:20:18 -07:00
Miccah
59d6d29c02
Add location to Slack and Confluence metadata (#802)
Location indicates where in the source a secret was found. For example,
Slack could be in a message or in an attachment.
2022-09-16 22:58:14 +02:00
trufflesteeeve
57e46f9b76
Add SlackRealtime proto message (#803) 2022-09-16 16:49:51 -04:00
ahrav
c4492b1fdc
Add support for MongoDB detector. (#793)
* Add support for MongoDB detector.

* Remove extra line.

* Remove unused arg.

* Add context around found secret test.

* Remove unused arg.
2022-09-15 05:47:09 -07:00
ahrav
e9599db240
[THOG-690] - Add visibility to slack metadata proto message (#796) 2022-09-15 05:01:45 -07:00
roxanne-tampus
90da460fa1
added new detector (#765) 2022-08-31 17:54:23 -07:00
Mildred Bernardo
4c3c103b62
added new detectors (#761) 2022-08-31 11:50:33 -07:00
Marlon
a35786dccd
fix and make_protos (#757) 2022-08-30 17:13:04 -07:00
Marlon
79ebd68068
added new detector (#751)
Co-authored-by: marlon pamisa <marlon.pamisa@banyanhills.com>
2022-08-30 11:28:41 -07:00
roxanne-tampus
fa2d6b90cd
added new detectors (#743) 2022-08-29 16:44:11 -07:00
Marlon
098d4a9e7d
added appointed scanner (#425)
* added appointed scanner

* fix comment

* fix comment

* fix comment

* fix issue

Co-authored-by: Dustin Decker <dustin@trufflesec.com>
2022-08-25 10:40:35 -07:00