Commit graph

2601 commits

Author SHA1 Message Date
Miccah
1a0cf04c3d
[chore] Ensure testing Endpoints() doesn't silently pass on change (#3334)
Since Endpoints() isn't a defined interface, we are testing an
implementation detail of EndpointSetter. If that function changes in
anyway, the test will now fail instead of skipping every detector and
passing.
2024-09-25 15:27:13 -07:00
Miccah
4484bf443b
[fix] Correctly initialize detectors with cloud endpoint customization (#3333)
* [fix] Correctly initialize detectors with cloud endpoint customization

We were only initializing if the detector was configured with a custom
endpoint, but not in the default case.

* Add test

* Fix gitlab.v2 detector
2024-09-25 13:06:00 -07:00
Kashif Khan
eb40243984
RailwayApp Detector (#3331)
* Added RailwayApp detector

* Updated Keywords
2024-09-25 10:17:08 -05:00
dylanTruffle
0f427b3c6a
Adding Descriptions (#3258)
* adding AI generated descriptions of the key types and their capabilities

* removing empty file

* Update abbysale.go

* update to interface

* fixes

* fix

* small cleanup

---------

Co-authored-by: Dylan Ayrey <dxa4481@rit.edu>
Co-authored-by: Dustin Decker <dustin@trufflesec.com>
2024-09-24 16:10:16 -07:00
ahrav
50ff17bd5b
update timeout to 60s (#3330) 2024-09-24 15:13:36 -07:00
Abdul Basit
b612e1e4fd
[Fix] (#3306)
For AWS session token, substring is being searched to avoid false positive session tokens.
Reference: https://nitter.poast.org/TalBeerySec/status/1816449053841838223#m
2024-09-24 11:24:46 -07:00
Kashif Khan
4b6957df66
Endpoint customizer refresh (#3308)
* Refresh EndpointCustomizer for more explicit configuration

Also add CloudProvider interface.

* WIP: Update EndpointSetter

* Updated detectors with new endpoint customizer

* Fixed linter

* Added check for appending cloud endpoints

---------

Co-authored-by: Miccah Castorina <m.castorina93@gmail.com>
2024-09-24 11:41:05 -05:00
Richard Gomez
b2311b4ad2
Ignore glTF & JPEG XL files (#3325)
* feat: ignore glTF files

Inspired by https://github.com/gitleaks/gitleaks/issues/1526

* feat: ignore JPEG XL

* feat: ignore .avifs in addition to .avif
2024-09-24 08:00:02 -07:00
ahrav
26ae7cb09d
[bug] - Improve seekability check for stdout pipes in BufferedReadSeeker (#3189)
* fix bug

* update

* clarify comment

* cleanup

* fix test

* update comment

* remove code for large files

* address comments

* update
2024-09-23 16:55:26 -07:00
Richard Gomez
75557f61ed
Improve MongoDB connection string matching (#1550)
* feat(mongodb): improve conn string matching

* fix(mongodb): err -> verificationErr
2024-09-23 15:42:06 -07:00
Abdul Basit
3e46b3f221
[Analyzer] Test and generated permissions for HuggingFace, Square & Stripe (#3294)
* stripe analyzer unit test

* add huggingface analyzer unit test

* add permissions.yaml for huggingface and fix in analyzer

* square permissions generated

* permissions generated for stripe

* change permissions to lowercase

* skip unknown permissions for square and stripe

---------

Co-authored-by: Abdul Basit <abasit@folio3.com>
2024-09-23 12:17:20 -07:00
Miccah
2f3a410e38
Implement SourceUnitEnumChunker for GitHub (#3298)
* Implement SourceUnitEnumChunker for GitHub

This change refactors the internal scan method to introduce a scanRepo
method to perform the actual scan.

* Export unit fields so the values are captured in the report

* Add comment for scanRepo

* Break out ensureRepoInfoCache into a method

* Update comments and check errors

* Ensure that the repoInfoCache contains the repo during ChunkUnit

* Add integration test for ChunkUnit

* Move s.scanOptions initialization to Init()
2024-09-23 10:56:55 -07:00
Zachary Rice
b78be6d935
hit em w/ a min (#3316) 2024-09-23 11:07:39 -05:00
ahrav
97fd2f80ee
[fix] - Add Size Method to BufferedReadSeeker and Refactor Context Timeout Handling in HandleFile (#3307) 2024-09-23 06:21:07 -07:00
Miccah
77dc2720a8
Update GitHub enumeration to report unique filtered values (#3292)
The reported values should match the values populated in s.repos.
2024-09-18 14:30:10 -07:00
Miccah
b2da2a6a5c
[analyze] Add client filter to detect successful unsafe HTTP requests (#3305)
* Move analyzer client to its own file

* Add analyzer client filter to detect successful unsafe HTTP requests

* Close response body in test
2024-09-18 10:31:21 -07:00
Miccah
401bc4687b
Instrument GitHub source with a ChunkReporter (#3296) 2024-09-16 09:57:09 -07:00
Dustin Decker
7e78ca385f
Add user agent suffix feature flag (#3297)
* Add user agent suffix feature flag

* unecessary concat
2024-09-13 15:20:43 -07:00
Miccah
213bf7e4fd
Fix GitHub analyzer panic on empty organization name (#3295) 2024-09-13 11:31:31 -07:00
tiaoxizhan
07dc95dd58
Fix slice initialization error (#3293) 2024-09-13 06:14:51 -07:00
Abdul Basit
dc9c9a30b3
[analyze] Add analyzer interface for Shopify (#3226)
* implement analyzer interface for shopify

* fixed shopify analyzer according to new code and generated permissions

* shopify analyzer test added

* [chore]
- key validations
- linked analyzer with detectors

* [chore]
- moved redundant initialize to global.

* [chore]
moved expected output of test in json file to neat the code.

* [Fixes]
- Fixed permission and category resource issue in shopify analyzer
- corrected test for shopify analyzer

---------

Co-authored-by: Abdul Basit <abasit@folio3.com>
2024-09-12 11:53:42 -07:00
Abdul Basit
57e58123f1
[analyze] Add Analyzer for Mailgun (#3206)
* implement analyzer interface, add unit test and link with detector for mailgun

* [chore] moved expected output of test in json file to neat the code.
corrected variable name for test in detector bucket

* append domain id in fully qualified name of domain resources

* [Fixes]
domains will be added as resource in bindings and permissions.
updated the test.

---------

Co-authored-by: Abdul Basit <abasit@folio3.com>
2024-09-12 11:48:46 -07:00
Abdul Basit
b0318a9edb
[analyze] Add Analyzer for MySQL (#3193)
* implement analyzer interface for mysql

* add integration test for mysql analyzer

* linked detectors with analyzers for jdbc and mysql
validation for connection string in analyzer

* refactored secretInfoToAnalyzerResult func

* generated permissions for mysql analyzer

* [chore]
- optimization in execution flow
- use test-container library for analyze test.

* added host in secret info struct
simplified the mysql test due to huge structure

---------

Co-authored-by: Abdul Basit <abasit@folio3.com>
2024-09-12 11:48:34 -07:00
Miccah
e89190f3ed
Instrument GitHub source with a UnitReporter (#3284)
* Fix GitHub integration test

* Instrument GitHub source with a UnitReporter

The reporter is currently unused, but is the first step to support
scanning while enumerating.

* Update GitHub unit tests
2024-09-12 10:28:37 -07:00
dylanTruffle
029f03e7bd
adding pypi v1 support (#3289)
Co-authored-by: Dylan Ayrey <dxa4481@rit.edu>
2024-09-11 21:39:01 -07:00
dylanTruffle
d201e54305
adding pypi detector (#3287)
* adding pypi detector

* update test and use helper

---------

Co-authored-by: Dylan Ayrey <dxa4481@rit.edu>
Co-authored-by: Dustin Decker <dustin@trufflesec.com>
2024-09-11 19:17:17 -07:00
Dustin Decker
70c6bb5634
feature flag additional refs (#3282) 2024-09-10 15:51:41 -07:00
Richard Gomez
b7411d2922
Clarify "no decoder found for chunk" log message (#3001)
* chore(engine): clarify trace log message

* chore(engine): fix merge conflicts
2024-09-10 13:58:40 -05:00
Zachary Rice
2fb90295ce
update aha keyword (#3281) 2024-09-10 13:07:27 -05:00
ahrav
5dad5a738b
[chore] - remove unused method and function (#3089)
* remove unused method

* delete more unused stuff
2024-09-10 12:35:49 -05:00
Ankush Goel
0feca62469
Jira Email fix (#3061) 2024-09-10 12:27:55 -05:00
Valentin B.
075f8c67a5
fix(git): config normalization for git sources (#3278)
When normalizing the git source config, the base and head refs should be normalized to commit hashes, in case
a branch or tag name was used. The `resolveAndSetCommit` function was returning a boolean value which should
indicate whether the input ref was changed from its original value. While this is in itself not a problem,
the caller (`normalizeConfig`) was using this boolean as an error marker, and returning early in case of `false`.

This meant that if the config was already containing a commit hash for the base or head ref, `resolveAndSetCommit`
would set the flag to `false` and `normalizeConfig` would early return erreneously. This caused the logic to find
the ancestor commit to be skipped which caused the bug in the issue #3220.

Since the `resolveAndSetCommit` function was only used in `normalizeConfig`, the signature has been changed to only
return the commit object and an error. The check for early return in `normalizeConfig` now instead relies on the
commit object being `nil` to indicate a failure to resolve the ref.

Refs: #3220
2024-09-10 12:12:57 -05:00
Daniel Teixeira
f24f62832b
Add detector for Nvidia NGC Personal Keys (#3280)
* Add detector for Nvidia NGC Personal Keys

* Update nvapi.go to use `nvapi-` as the keyword
2024-09-10 08:36:33 -07:00
Nash
17f6c98119
GitHub source logger clean up (#3269)
* GitHub source logger clean up

* applied pr comments

* applied pr comments

* applied pr comments

* applied PR review comments
2024-09-09 15:44:56 -04:00
Abdul Basit
93d09c78b4
[analyze] Add Analyzer for Postgres (#3192)
* implement analyzer interface for postgres

* added unit test for postgres analyzer

* refactored code in postgres analyzer

* generate permissions for postgres analyzer

* renamed variable

* [chore] corrected the variable name.

* appended hostname to distinguish the resources.
updated the test.

---------

Co-authored-by: Abdul Basit <abasit@folio3.com>
2024-09-06 12:42:55 -07:00
Abdul Basit
a43d451c4d
[analyze] Add Analyzer for SourceGraph (#3173)
* implement analyzer interface for sourcegraph

* created permission for sourcegraph
test for sourcegraph.
added email in resource metadata.

* handling of missing keys in map

* linked sourcegraph detector to analyzer

* update the fullyqualidied name of resource to make it unique.
updated the test.

* add current user email in metadata

---------

Co-authored-by: Abdul Basit <abasit@folio3.com>
2024-09-06 12:40:47 -07:00
Abdul Basit
4cab071032
[analyze] Add Analyzer for Asana (#3139)
* impelmented analyzer interface with data models for Asana

* add unit test for asana analyzer

* link asana detector with analyzer

* added permission for asana
linked detector with only positive cases.

* to make test cleaner moved want json in external file.
Moreover without sorting test will not be able to compare.

* use general functions to avoid code duplication.
optimize app permission making logic.

* [fix] assigned bindings to results.

---------

Co-authored-by: Abdul Basit <abasit@folio3.com>
2024-09-06 12:37:45 -07:00
Abdul Basit
f235b8a442
[analyze] Add Analyzer for Slack (#3207)
* implement analyzer interface for slack

* slack analyzer adjusted for new changes in main, unit test added

* link detector with analyzer for slack

* added generated permissions for slack analyzer

* generate permission fix, keep dot in permissions intact

* removed scope from permission and put it metadata.

* [chore]
moved expected output of test in json file to neat the code.
added team id in fully qualified name of user resource.
check permissions before adding it in bindings.

---------

Co-authored-by: Abdul Basit <abasit@folio3.com>
2024-09-06 12:33:06 -07:00
Abdul Basit
c449129d75
[analyze] Improve SquareUp analyzer and Implemented test (#3231)
* square analyzer fix  assign team members to unbounded resources - unit test for square analyzer

* refactoring

---------

Co-authored-by: Abdul Basit <abasit@folio3.com>
2024-09-06 12:32:52 -07:00
Abdul Basit
5d7e6fc2fa
[analyze] Add Analyze interface for Mailchimp (#3225)
* implement analyzer interface for mailchimp

* link detector with analyzer for mailchimp

* fix analyzer type

* add mailchimp analyzer test

* [chore] appended string to make fulllyqualifiedname as unique.

* [chore]
moved expected output of test in json file to neat the code.
removed PII information in metadata.

---------

Co-authored-by: Abdul Basit <abasit@folio3.com>
2024-09-05 16:33:47 -07:00
Abdul Basit
becc2c4010
[analyze] Add analyze interface for Bitbucket (#3224)
* impelmented analyzer interface with data models for Bitbucket

* Added bitbucket permissions in yaml
add ids to keep resources distinguishable.

* [chore]
moved expected output of test in json file to neat the code.
corrected the fully qualified name for repository resources.

---------

Co-authored-by: Abdul Basit <abasit@folio3.com>
2024-09-05 16:04:35 -07:00
Abdul Basit
899f59fbb4
[analyze] Add Analyzer for Sendgrid (#3174)
* implement analyzer interface for sendgrid

* add unit test for sendgrid analyzer

* fixed sendgrid detector.
linked analyzer with sendgrid detector.
handling if key not found in map

* category as resource.
if subcategory is present then subcategory will become resource with parent category

* corrected test and remove hardcoded boolean for printing in sendgrid

* incorporate code refactoring and suggestion for FullyQualifiedName for subcategory by Miccah.

* generate permissions for sendgrid analyzer

* [NIT] rather than updating the global scopes variable, ProcessPermission will return new list of categories with Permission and eliminate those which are not in generated one.

---------

Co-authored-by: Abdul Basit <abasit@folio3.com>
2024-09-04 15:50:32 -07:00
Abdul Basit
d6e1627f16
[analyze] Add Analyzer for Opsgenie (#3181)
* implement analyzer interface for opsgenie and add unit tests

* Add analyzer interface for opsgenie

linked detector  with analyzers
fixed test cases.

* generate permissions for opsgenie and change scope names to lowercase for consistency

* fixed unboundedresources slice issue.
username as fullqualifiedname

---------

Co-authored-by: Abdul Basit <abasit@folio3.com>
2024-09-04 15:50:19 -07:00
Abdul Basit
5ce1578a6f
[analyze] Add analyzer for Postman (#3180)
* implement analyzer interface for postman and add unit test

* analyzer interface inplementation for postman

linked detector with analyzer for postman
add permission for postman

* [fix] linter in postman

* considered Miccah comments about fullyqualifiedName and code refactoring.

* moved want result to expected output file.

---------

Co-authored-by: Abdul Basit <abasit@folio3.com>
2024-09-04 15:40:12 -07:00
Dustin Decker
db0108f731
Make worker multipliers configurable (#3267) 2024-09-04 11:36:26 -07:00
0x1
7eb5b5b12c
add rotation links (#3257) 2024-09-04 10:45:04 -04:00
Dustin Decker
6bbb683ead
Reduce high freq keywords (#3265) 2024-09-03 16:58:46 -07:00
Dustin Decker
8999eab89d
Add central feature flags (#3264)
* Add central feature flags

* use atomic

* tidy
2024-09-03 15:54:41 -07:00
Hon
f52d8e872d
Add huggingface tui config (#3060)
* add huggingface tui config

* update readme

* undo auto format
2024-09-03 12:54:39 -07:00
Shreyas Sriram
15faaba61c
Add Robinhood Crypto detector (#3254)
* Add Robinhood Crypto detector

* Address comment - use single keyword
2024-08-29 14:05:52 -07:00