* implement analyzer interface for shopify
* fixed shopify analyzer according to new code and generated permissions
* shopify analyzer test added
* [chore]
- key validations
- linked analyzer with detectors
* [chore]
- moved redundant initialize to global.
* [chore]
moved expected output of test in json file to neat the code.
* [Fixes]
- Fixed permission and category resource issue in shopify analyzer
- corrected test for shopify analyzer
---------
Co-authored-by: Abdul Basit <abasit@folio3.com>
* implement analyzer interface, add unit test and link with detector for mailgun
* [chore] moved expected output of test in json file to neat the code.
corrected variable name for test in detector bucket
* append domain id in fully qualified name of domain resources
* [Fixes]
domains will be added as resource in bindings and permissions.
updated the test.
---------
Co-authored-by: Abdul Basit <abasit@folio3.com>
* implement analyzer interface for mysql
* add integration test for mysql analyzer
* linked detectors with analyzers for jdbc and mysql
validation for connection string in analyzer
* refactored secretInfoToAnalyzerResult func
* generated permissions for mysql analyzer
* [chore]
- optimization in execution flow
- use test-container library for analyze test.
* added host in secret info struct
simplified the mysql test due to huge structure
---------
Co-authored-by: Abdul Basit <abasit@folio3.com>
* Fix GitHub integration test
* Instrument GitHub source with a UnitReporter
The reporter is currently unused, but is the first step to support
scanning while enumerating.
* Update GitHub unit tests
When normalizing the git source config, the base and head refs should be normalized to commit hashes, in case
a branch or tag name was used. The `resolveAndSetCommit` function was returning a boolean value which should
indicate whether the input ref was changed from its original value. While this is in itself not a problem,
the caller (`normalizeConfig`) was using this boolean as an error marker, and returning early in case of `false`.
This meant that if the config was already containing a commit hash for the base or head ref, `resolveAndSetCommit`
would set the flag to `false` and `normalizeConfig` would early return erreneously. This caused the logic to find
the ancestor commit to be skipped which caused the bug in the issue #3220.
Since the `resolveAndSetCommit` function was only used in `normalizeConfig`, the signature has been changed to only
return the commit object and an error. The check for early return in `normalizeConfig` now instead relies on the
commit object being `nil` to indicate a failure to resolve the ref.
Refs: #3220
* implement analyzer interface for postgres
* added unit test for postgres analyzer
* refactored code in postgres analyzer
* generate permissions for postgres analyzer
* renamed variable
* [chore] corrected the variable name.
* appended hostname to distinguish the resources.
updated the test.
---------
Co-authored-by: Abdul Basit <abasit@folio3.com>
* implement analyzer interface for sourcegraph
* created permission for sourcegraph
test for sourcegraph.
added email in resource metadata.
* handling of missing keys in map
* linked sourcegraph detector to analyzer
* update the fullyqualidied name of resource to make it unique.
updated the test.
* add current user email in metadata
---------
Co-authored-by: Abdul Basit <abasit@folio3.com>
* impelmented analyzer interface with data models for Asana
* add unit test for asana analyzer
* link asana detector with analyzer
* added permission for asana
linked detector with only positive cases.
* to make test cleaner moved want json in external file.
Moreover without sorting test will not be able to compare.
* use general functions to avoid code duplication.
optimize app permission making logic.
* [fix] assigned bindings to results.
---------
Co-authored-by: Abdul Basit <abasit@folio3.com>
* implement analyzer interface for slack
* slack analyzer adjusted for new changes in main, unit test added
* link detector with analyzer for slack
* added generated permissions for slack analyzer
* generate permission fix, keep dot in permissions intact
* removed scope from permission and put it metadata.
* [chore]
moved expected output of test in json file to neat the code.
added team id in fully qualified name of user resource.
check permissions before adding it in bindings.
---------
Co-authored-by: Abdul Basit <abasit@folio3.com>
* square analyzer fix assign team members to unbounded resources - unit test for square analyzer
* refactoring
---------
Co-authored-by: Abdul Basit <abasit@folio3.com>
* implement analyzer interface for mailchimp
* link detector with analyzer for mailchimp
* fix analyzer type
* add mailchimp analyzer test
* [chore] appended string to make fulllyqualifiedname as unique.
* [chore]
moved expected output of test in json file to neat the code.
removed PII information in metadata.
---------
Co-authored-by: Abdul Basit <abasit@folio3.com>
* impelmented analyzer interface with data models for Bitbucket
* Added bitbucket permissions in yaml
add ids to keep resources distinguishable.
* [chore]
moved expected output of test in json file to neat the code.
corrected the fully qualified name for repository resources.
---------
Co-authored-by: Abdul Basit <abasit@folio3.com>
* implement analyzer interface for sendgrid
* add unit test for sendgrid analyzer
* fixed sendgrid detector.
linked analyzer with sendgrid detector.
handling if key not found in map
* category as resource.
if subcategory is present then subcategory will become resource with parent category
* corrected test and remove hardcoded boolean for printing in sendgrid
* incorporate code refactoring and suggestion for FullyQualifiedName for subcategory by Miccah.
* generate permissions for sendgrid analyzer
* [NIT] rather than updating the global scopes variable, ProcessPermission will return new list of categories with Permission and eliminate those which are not in generated one.
---------
Co-authored-by: Abdul Basit <abasit@folio3.com>
* implement analyzer interface for opsgenie and add unit tests
* Add analyzer interface for opsgenie
linked detector with analyzers
fixed test cases.
* generate permissions for opsgenie and change scope names to lowercase for consistency
* fixed unboundedresources slice issue.
username as fullqualifiedname
---------
Co-authored-by: Abdul Basit <abasit@folio3.com>
* implement analyzer interface for postman and add unit test
* analyzer interface inplementation for postman
linked detector with analyzer for postman
add permission for postman
* [fix] linter in postman
* considered Miccah comments about fullyqualifiedName and code refactoring.
* moved want result to expected output file.
---------
Co-authored-by: Abdul Basit <abasit@folio3.com>
The previous implementation of targeted file scanning pulled patches out of commit data, which didn't work for binary files (because GitHub doesn't return patches for them). This PR changes the system to always just download the requested file and scan it, which means we get binary file support.
* Fixed the checks for local exported data
* Fixed the check for local export files
* Fixed the check for local export files
* Fixed the check for local export files
* Merge branch 'main' into th-899-postman-panic-issue
* minor changes in the tests
* test update
* test
The GitHub source generates chunks for targeted scans differently than it does for "normal" scans. One difference was the presence of leading + and - characters, which can interfere with detection in some cases.
There is a scenario in which results filtration is known to cause problems, and this PR disables it in that scenario. (It should cause problems more generally, but lacking any concrete cases of that, I want to tread lightly.)
We have identified some cases in which the results "cleaning" logic (the logic that eliminates superfluous results) should not run. In order to allow this, we need to expose the cleaning logic to the engine. This PR does so by doing these things:
- Create a CustomResultsCleaner interface that can be implemented by detectors that want to use custom cleaning logic
- Implement this interface for the aws and awssessionkey detectors (and remove their previous invocation of their custom cleaning logic)
- Modify the engine to invoke this logic (conditionally)
This PR also removes the "custom" cleaning logic for the opsgenie, razorpay, and twilio detectors, because it was added erroneously.
This is an alternative implementation of #3233.
If a detector ignores the configured timeout it is probably because of I/O blocking, which degrades the efficiency of the detector worker pool when it happens a lot. In the worst case, a detector that fully hangs will zombify its worker, causing really bad performance problems. When this happens, we don't really have a good way to notice other than seeing scan throughput drop suspiciously. This PR adds explicit logging when detection takes longer than it should so we have a better chance of catching this.
(This problem theoretically can spring up anywhere, in any worker, but the detector fleet is vast, uses network I/O, and is implemented by a much larger group of people, so this sort of problem is much more likely to slip into detector implementations than anywhere else in the codebase. We could generalize this mechanism, but I don't want to make that investment before seeing if this smaller change captures the information we need.)
The GitHub source currently applies its authentication configuration as the first step of enumeration. This is incompatible with both targeted scans and scan job reports, and also means that authentication logic has to be duplicated into the validation flow. This PR moves it into Init so that it's available to targeted scans and, eventually, unit-specific scans. This also allows us to remove the copy of the old logic that was in Validate.
As part of the work I've also cleaned up the integration test suite. (Several of them were apparently disabled back when they ran on every push, but now that we're not doing that, we can re-enable them.)
* implemented analyzer interface with data models for HuggingFace
* correct test for huggingface due to new addition of key in detection result.
---------
Co-authored-by: Abdul Basit <abasit@folio3.com>
* implement analyzer interface for square
* linked detector with analyzer for square
fix test for square.
* code refactoring
---------
Co-authored-by: Abdul Basit <abasit@folio3.com>
* Add progress bar to CFOR
* unused vars
* explicitly ignore progress errors
* removed print statements
* use stderr
---------
Co-authored-by: joeleonjr <20135619+joeleonjr@users.noreply.github.com>
Co-authored-by: Joe Leon <joe.leon@trufflesec.com>
* alpha feature for scanning hidden commits on github
* improvements re: git operations
* lint updates
* updating with exec block due to no gh token
* reworked logic into new source
* fixed collisions threshold flag input
* fixed IOutil issues
* removed additions from GH config
---------
Co-authored-by: Joe Leon <joe.leon@trufflesec.com>
* implement analyzer interface for stripe
* consider cateogry as unbound resource if there is no permission with it.
* check for key existence in map.
pass on analysis info from Stripe detector.
test change to remove analysis info.
* remove Valid boolean from metadata of analyzer result
---------
Co-authored-by: Abdul Basit <abasit@folio3.com>
* Add POC analyze sub-command
* Address lint errors
* added http logging to most analyzers
* Use custom RoundTripper with default http.Client
* [chore] Embed scopes at compile time
* [chore] Move subcommand check up to prevent printing metrics
* Create framework of interfaces, structs, and protos
* Implement Analyzer for airbrake
* Add FullAccess permission constant
* Implement Analyzer for asana
* Implement Analyzer for bitbucket
* Implement Analyzer for github
* Implement Analyzer for gitlab
* Implemente Analyzer for huggingface
* Implement Analyzer for mailchimp
* implement analyzer for mailgun
* update cli cmd
* Implement analyzer for openai
* fix timing issue on scopes
* print permissions only if restricted key
* Implement Analyzer for mysql
* enable loggin check
* fixed the formatting issue to wrap sub-errors
* implemented analyzer for opsgenie
* implemented analyzer for postgres
* use format string
* implemented analyzer for sendgrid
* simplify returning the error
* implemented analyzer for postman
* added handling of workspace error
* Update protos to match OSS
* Generate protos
* Update data structures to match OSS
* Update airbrake implementation
* Remove asana implementation
* Remove mailchimp implementation
* Update openai implementation to match OSS
* Remove gitlab implementation
* Remove huggingface implementation
* Remove bitbucket implementation
* Fix permission in airbrake
* Remove github implementation
* Remove mailgun implementation
* Cleanup compiler errors
* Implement Analyzer interface for github
* Add parents to github resources
* Add fine_grained to github metadata
* Update with changes from main
* Remove unused function stubs
---------
Co-authored-by: Joe Leon <joe.leon@trufflesec.com>
Co-authored-by: Hon <8292703+hxnyk@users.noreply.github.com>
Co-authored-by: Abdul Basit <abasit@folio3.com>
Co-authored-by: Abdul Basit <basit.mussani@gmail.com>
The AWS detector verifies credentials in a weird hacky way to work around some non-obvious STS behavior. This workaround does not work for canary tokens, so I updated the test secrets to use non-canary tokens. This PR updates the tests to match the secrets file changes.
* [analyze] Implement Analyzer interface for github
* Make github repo and user enumeration configurable
* Add AnalysisInfo to github detector
* Use AnalyzeAndPrintPermissions from the CLI
* Add POC analyze sub-command
* Address lint errors
* [chore] Embed scopes at compile time
* [chore] Move subcommand check up to prevent printing metrics
* added http logging to most analyzers
* Use custom RoundTripper with default http.Client
* Create framework of interfaces, structs, and protos
* Merge main
* Add AnalysisInfo to detectors.Result
* Hide analyze subcommand
* Update gen_proto.sh
* Update protos
* Make protos
* Update analyzer data types
* Rename argument to credentialInfo
---------
Co-authored-by: Joe Leon <joe.leon@trufflesec.com>
* implemented a netsuite detector
* implemented the netsuite detector with modified test.
* clean up go.sum by running `go mod tidy`
* implemented a netsuite detector
* implemented the netsuite detector with modified test.
* clean up go.sum by running `go mod tidy`
* Incorporated suggestion by Ahrav
- optimized nonce generation logic.
- use string builder as compare to concatenation.
* fix go.sum
* fix import
* fix
---------
Co-authored-by: Ahrav Dutta <ahrav.dutta@trufflesec.com>
* adding v1 eleven labs
* updating elevenlabs to support old and new version
* fixing status codes
* lint fixes
* adding test for v2
* adding test for v1
* return err
---------
Co-authored-by: Dylan Ayrey <dylan@Dylans-MacBook-Pro.local>
Co-authored-by: āh̳̕mͭͭͨͩ̐e̘ͬ́͋ͬ̊̓͂d <13666360+0x1@users.noreply.github.com>
* initial spike on hf
* added in user and org enum
* adding huggingface source
* updated with lint suggestions
* updated readme
* addressing resources that require org approval to access
* removing unneeded code
* updating with new error msg for 403
* deleted unused code + added resource check in main
* implemented larksuite detectores for tokens and api keys.
test implemented for larksuite token based detectors.
* implemented test for larksuiteapikey detector
* load credentials from GCP secret manager for larksuite api keys
Targeted scans should return their errors so that consumers can process them. By creating a type that combines an error with a targeted secret ID, we can return these errors without having to modify the Source interface.
* updated the twitter regex.
* updated regex for bearer token.
* clean up the code for existing twitter detector
added and Implemented new detector for twitter consumer key & secrets with test.
proto generated.
* string updated.
* written test for twitter consumer key detector
* reverted the file to avoid conflicts
* corrected the regex library in twitter detector
