Commit graph

2516 commits

Author SHA1 Message Date
ahrav
29613220b0
[chore] - log detector type on error (#3159)
* log detector type on error

* update error message

* update log

* update message
2024-08-02 10:54:59 -07:00
ahrav
ddb7211ded
[chore] - set custom transport for the Docker client (#3156)
* set custom transport for docker

* fix lint
2024-08-02 08:51:59 -07:00
Abdul Basit
04a13385a8
Add Analyzers interface for HuggingFace (#3140)
* implemented analyzer interface with data models for HuggingFace

* correct test for huggingface due to new addition of key in detection result.

---------

Co-authored-by: Abdul Basit <abasit@folio3.com>
2024-08-02 08:20:11 -07:00
joeleonjr
f927076483
quick patch for cfor enumeration (#3155)
Co-authored-by: Joe Leon <joe.leon@trufflesec.com>
2024-08-02 11:12:43 -04:00
Abdul Basit
c1645e8c27
Add Analyzers interface for Square (#3141)
* implement analyzer interface for square

* linked detector with analyzer for square
fix test for square.

* code refactoring

---------

Co-authored-by: Abdul Basit <abasit@folio3.com>
2024-08-02 08:00:25 -07:00
Dustin Decker
05e4635824
Add progress bar to CFOR (#3151)
* Add progress bar to CFOR

* unused vars

* explicitly ignore progress errors

* removed print statements

* use stderr

---------

Co-authored-by: joeleonjr <20135619+joeleonjr@users.noreply.github.com>
Co-authored-by: Joe Leon <joe.leon@trufflesec.com>
2024-08-02 07:43:59 -07:00
ahrav
fba1a8b410
[perf] - Leverage pgzip for Parallel decompression (#3149) 2024-08-02 04:11:10 -07:00
joeleonjr
7d606e2480
CFOR Commit Scanner (#3145)
* alpha feature for scanning hidden commits on github

* improvements re: git operations

* lint updates

* updating with exec block due to no gh token

* reworked logic into new source

* fixed collisions threshold flag input

* fixed IOutil issues

* removed additions from GH config

---------

Co-authored-by: Joe Leon <joe.leon@trufflesec.com>
2024-08-01 23:04:20 -04:00
Miccah
38e844f968
[chore] Only set default detectors if none are provided (#3147) 2024-08-01 17:15:06 -07:00
Dustin Decker
605d037e45
add twilio analyze relationships (#3148)
* add twilio analyze relationships

* unused struct
2024-08-01 17:04:44 -07:00
Miccah
f776b481d1
[analyze] Combine access level into permission value (#3144) 2024-08-01 11:09:59 -07:00
Abdul Basit
843120427d
Add Analyze interface to Stripe (#3132)
* implement analyzer interface for stripe

* consider cateogry as unbound resource if there is no permission with it.

* check for key existence in map.
pass on analysis info from Stripe detector.
test change to remove analysis info.

* remove Valid boolean from metadata of analyzer result

---------

Co-authored-by: Abdul Basit <abasit@folio3.com>
2024-08-01 08:23:06 -07:00
ahrav
048ec26c92
move concurrency (#3135) 2024-07-31 18:58:18 -07:00
ahrav
fd257350dd
[chore] - address linter (#3133)
* addres linter

* fix
2024-07-31 17:30:51 -07:00
Hon
555e1ceeee
Export maps from permission generation (#3137)
* Adjust permission generation to make maps exportable

* fix bug and add twilio
2024-07-31 16:49:56 -07:00
Dustin Decker
25b01019b3
Add permissions lookup tables (#3125)
* OpenAI LUT

* github LUT

* cleanup

* add test

* update

* update

* update openai

* update

* Add Analyze interface to Twilio (#3128)

* Add Analyze interface to Twilio

* add readme
2024-07-31 13:01:29 -07:00
Abdul Basit
6fccac7f3d
Separate out printing statements with anlayzer logic for SourceGraph (#3119)
* Separated printing and analyzes functionality for sourcegraph

* remove second call to fetch userinfo in sourcegraph.
2024-07-31 10:08:42 -07:00
Abdul Basit
24b7029d4d
Separate out printing statements with anlayzer logic for Stripe (#3120)
* Separated printing and analyzes functionality for stripe

* removed logging enabled check
2024-07-31 07:07:10 -07:00
Abdul Basit
a2c7219d65
Separate out printing statements with anlayzer logic for Slack (#3121)
* Separated printing and analyzes functionality for slack

* removed logging enabled check
2024-07-31 07:06:46 -07:00
Cody Rose
3ab975edb3
Update GitHub integration tests (#3124)
#1816 and #2995 both updated the GitHub source without updating its integration tests. This PR updates those tests, bringing them back into success.
2024-07-31 09:28:10 -04:00
Dustin Decker
a3d3565248
Add new canary ID (#3117) 2024-07-30 20:44:58 -07:00
Abdul Basit
67c01aee6e
Separated printing and analyzes functionality for twilio (#3118) 2024-07-30 20:44:44 -07:00
Abdul Basit
02fb3879eb
Separated printing and analyzes functionality for square (#3122) 2024-07-30 20:44:07 -07:00
Abdul Basit
acd529d9dc
Separated printing and analyzes functionality for shopify (#3123) 2024-07-30 20:43:46 -07:00
Miccah
20de56d441
Analyzer partial implementations (#3114)
* Add POC analyze sub-command

* Address lint errors

* added http logging to most analyzers

* Use custom RoundTripper with default http.Client

* [chore] Embed scopes at compile time

* [chore] Move subcommand check up to prevent printing metrics

* Create framework of interfaces, structs, and protos

* Implement Analyzer for airbrake

* Add FullAccess permission constant

* Implement Analyzer for asana

* Implement Analyzer for bitbucket

* Implement Analyzer for github

* Implement Analyzer for gitlab

* Implemente Analyzer for huggingface

* Implement Analyzer for mailchimp

* implement analyzer for mailgun

* update cli cmd

* Implement analyzer for openai

* fix timing issue on scopes

* print permissions only if restricted key

* Implement Analyzer for mysql

* enable loggin check

* fixed the formatting issue to wrap sub-errors

* implemented analyzer for opsgenie

* implemented analyzer for postgres

* use format string

* implemented analyzer for sendgrid

* simplify returning the error

* implemented analyzer for postman

* added handling of workspace error

* Update protos to match OSS

* Generate protos

* Update data structures to match OSS

* Update airbrake implementation

* Remove asana implementation

* Remove mailchimp implementation

* Update openai implementation to match OSS

* Remove gitlab implementation

* Remove huggingface implementation

* Remove bitbucket implementation

* Fix permission in airbrake

* Remove github implementation

* Remove mailgun implementation

* Cleanup compiler errors

* Implement Analyzer interface for github

* Add parents to github resources

* Add fine_grained to github metadata

* Update with changes from main

* Remove unused function stubs

---------

Co-authored-by: Joe Leon <joe.leon@trufflesec.com>
Co-authored-by: Hon <8292703+hxnyk@users.noreply.github.com>
Co-authored-by: Abdul Basit <abasit@folio3.com>
Co-authored-by: Abdul Basit <basit.mussani@gmail.com>
2024-07-30 09:13:48 -07:00
Harmon Herring
f664472da1
Include default detectors when using a config that contains detectors (#3115)
* include default detectors when config file is used

* fix test
2024-07-29 14:36:40 -07:00
Cody Rose
ed8bc501e5
Use non-canary credentials for AWS tests (#3109)
The AWS detector verifies credentials in a weird hacky way to work around some non-obvious STS behavior. This workaround does not work for canary tokens, so I updated the test secrets to use non-canary tokens. This PR updates the tests to match the secrets file changes.
2024-07-29 10:11:27 -04:00
ahrav
55fe05d0b4
fix dep versions (#3106) 2024-07-26 17:44:23 -07:00
Miccah
14e7a82fbf
[analyze] Add description and user to openai metadata (#3111)
Also rename the type from "org" to "organization" to match github.
2024-07-26 16:24:50 -07:00
Dustin Decker
c048487739
Support openai project and fine grained tokens (#3112) 2024-07-26 15:31:17 -07:00
Miccah
9d089c2188
[analyze] Implement Analyzer interface for github (#3110)
* [analyze] Implement Analyzer interface for github

* Make github repo and user enumeration configurable

* Add AnalysisInfo to github detector

* Use AnalyzeAndPrintPermissions from the CLI
2024-07-26 14:47:03 -07:00
Miccah
63568b0881
[chore] Move openai log message to proper function (#3105) 2024-07-26 10:39:13 -07:00
Miccah
8a8ef85265
Implement Analyzer interface for openai (#3101)
* Implement Analyzer interface for openai

* Use organization ID for the FQDN

* Update CLI to print openai permissions
2024-07-25 14:20:01 -07:00
Miccah
551dc6578c
[chore] Fix Versioner interface for twitter (#3104) 2024-07-25 14:17:02 -07:00
Miccah
2424683923
Analyze (#3099)
* Add POC analyze sub-command

* Address lint errors

* [chore] Embed scopes at compile time

* [chore] Move subcommand check up to prevent printing metrics

* added http logging to most analyzers

* Use custom RoundTripper with default http.Client

* Create framework of interfaces, structs, and protos

* Merge main

* Add AnalysisInfo to detectors.Result

* Hide analyze subcommand

* Update gen_proto.sh

* Update protos

* Make protos

* Update analyzer data types

* Rename argument to credentialInfo

---------

Co-authored-by: Joe Leon <joe.leon@trufflesec.com>
2024-07-25 12:06:05 -07:00
shangchenglumetro
c4aab3fb51
chore: fix some comments (#3098) 2024-07-25 10:37:13 -07:00
ahrav
ebfbd21707
[bug]- Invalid Seek for Non-Seekable Readers (#3095)
* inital work

* fix and add tests

* uncomment

* fix seek end

* use buffer pool

* revert timeout

* make linter happy

* More linting :()
2024-07-24 19:08:56 -07:00
Abdul Basit
7c0c2e9ff3
fixed crash issue if data array is empty (#3091) 2024-07-23 09:36:27 -07:00
trufflesteeeve
c01428d107
Remove onwater detector (#3088) 2024-07-22 17:00:32 -04:00
Abdul Basit
5b64e1e5a1
implemented a netsuite detector (#3068)
* implemented a netsuite detector

* implemented the netsuite detector with modified test.

* clean up go.sum by running `go mod tidy`

* implemented a netsuite detector

* implemented the netsuite detector with modified test.

* clean up go.sum by running `go mod tidy`

* Incorporated suggestion by Ahrav
	- optimized nonce generation logic.
	- use string builder as compare to concatenation.

* fix go.sum

* fix import

* fix

---------

Co-authored-by: Ahrav Dutta <ahrav.dutta@trufflesec.com>
2024-07-22 12:37:18 -07:00
ahrav
8f172b23ac
[chore] - Reduce VerificationOverlapWorkers (#3082)
* reduce worker count

* reduce detector worker count
2024-07-19 18:09:57 -07:00
ahrav
7a36e89c61
add verify check (#3079) 2024-07-19 18:09:26 -07:00
Abdul Basit
c20ca0d6a1
Added Twitter v2 Detector (#3016)
* implemented a newer version of twitter included a test.
moved old twitter detector to as V1.

* added version information in twitter test
2024-07-18 12:15:22 -04:00
ahrav
42b3a9d999
[perf] - Optimize MIME Type Detection to Reduce Allocations (#3048)
* Streaming file handling.

* cleanup

* update tests

* lint

* defer close on input io.ReadCloser's

* remove redundant mime type detection

* Reduce allocations

* fix test

* update comment

* fix seek bug

* address comment

* undo
2024-07-17 14:04:29 -07:00
ahrav
f865482025
[feat] - Streamlined File Handling with BufferedReaderSeeker (#3041)
* Streaming file handling.

* cleanup

* update tests

* lint

* defer close on input io.ReadCloser's

* fix seek bug

* fix hanging

* clarify errors

* update

* address comments

* revert

* update

* address

* add check to prevent seek without buffering

* revet

* revert

* update comment to make buffer usage more clear
2024-07-17 13:52:18 -07:00
Ankush Goel
e2e3b870b6
Atlassian Token Detector (#3065)
* atlassian token

* Update atlassian.go

Fixed version

* fixed regex
2024-07-17 12:44:39 -07:00
āh̳̕mͭͭͨͩ̐e̘ͬ́͋ͬ̊̓͂d
2a626c4dab
update package name (#3020) 2024-07-09 16:51:46 -05:00
Cody Rose
296379d5a0
Log more GitLab stuff (#3040)
Our GitLab happy path logging could use some love.

go.sum also needed a little love, for some reason.
2024-07-09 10:53:40 -04:00
Cody Rose
1a73442088
Order GitLab repos by ID (#3047) 2024-07-09 10:39:47 -04:00
ahrav
7d349ac7f3
remove dead code (#3044) 2024-07-07 08:59:40 -07:00