Commit graph

1832 commits

Author SHA1 Message Date
Cody Rose
ad57de50cd
Do not nest transports for Github installation client (#1564)
#1454 modified one of the Github enumeration code paths in a way that broke an integration test by causing one client's transport to be used for the construction of a different client, causing authentication failures. This saves the original transport for use, fixing the test.
2023-07-31 11:31:16 -04:00
Richard Gomez
e0faac8d1c
Fix runtime error when scanning Gist comments (#1552)
* fix(github): fix runtime error from gist comments

* fix(github): add flag to scan Gist comments
2023-07-31 08:57:42 -05:00
Miccah
070014f380
Initialize the default logger to output to stderr (#1569) 2023-07-31 03:26:19 -05:00
Dustin Decker
10b6e2898d
Increase log level of engine messages (#1576) 2023-07-28 14:30:43 -07:00
Miccah
6bd48583ae
Fix gitparse from panicking on a nil-pointer (#1570) 2023-07-28 11:15:02 -05:00
Cody Rose
431d26f5fa
move false positive check in alchemy detector (#1532)
This PR makes the Alchemy detector run its known false positive check even if verification is disabled. This isn't the most important detector but it's the template for new ones so getting a good pattern nailed down is important.

Moving the check allowed me to rewrite the determinacy logic to hopefully be more clear.
2023-07-28 11:36:02 -04:00
Zubair Khan
9f3809f19e
gdrive proto change (#1566) 2023-07-28 10:38:51 -04:00
Miccah
e391e89f3e
Initial implementation of JobReport with SourceManager usage (#1557)
* Initial implementation of JobReport with SourceManager usage

* Limit concurrent units

* Only save the last JobReport per handle
2023-07-27 10:49:56 -05:00
Zachary Rice
3897454dbb
add merge support (#1561) 2023-07-27 09:24:49 -05:00
ahrav
ade5d91d5c
Add azure repos protos. (#1559) 2023-07-26 19:53:10 -07:00
Richard Gomez
f925da7cea
fix(mongodb): detect CosmoDB access keys (#1511)
https://learn.microsoft.com/en-us/microsoft-365/compliance/sit-defn-azure-cosmos-db-account-access-key?view=o365-worldwide
2023-07-26 16:50:12 -05:00
Richard Gomez
46823f77c9
feat(github): clarify comment log statement (#1553) 2023-07-26 09:40:30 -05:00
Miccah
10f0963bc9
Add SourceManager tests for Run and Wait methods (#1530)
* Miscellaneous SourceManager updates

* Own the chunks channel instead of accepting it as an input
* Add Chunks and Wait methods
* Fix bug in Enroll so it actually returns the handle
* Add context.Context parameter to the SourceInitFunc type

* Add SourceManager tests for Run and Wait methods

* Rename man variables to mgr
2023-07-26 00:48:28 -05:00
Richard Gomez
2290954b02
fix(github): use apiEndpoint for basic or no auth (#1454) 2023-07-25 20:03:08 -07:00
Richard Gomez
f48a635c34
feat: update gitparse logic (#1486) 2023-07-25 17:52:34 -05:00
Miccah
91cbca941a
Fix data race in context wrapper library (#1546) 2023-07-25 17:09:36 -05:00
Zachary Rice
1a1977f7e6
case insensitive (#1547) 2023-07-25 17:01:15 -05:00
Bill Rich
f39303495a
Add commitsScanned metrics (#1533)
* Add commitsScanned metrics

* Just keep commit count
2023-07-25 11:31:01 -07:00
ahrav
b5b01d3eba
[chore] - optimize chunker (#1535)
* Use chunkbytes that includes the size of peek.

* linter.

* continue.

* add TotalChunkSize const.
2023-07-24 19:30:29 -07:00
Zachary Rice
85f363f093
init (#1538) 2023-07-24 19:09:57 -05:00
Miccah
93c561f324
Add match boundary to okta regular expressions (#1531) 2023-07-24 10:52:50 -05:00
Dustin Decker
fab80445d1
continue scanning on detector / decoder panic (#863) 2023-07-24 07:34:43 -07:00
ahrav
9e0a2e9ddd
[chore] - Remove password info from log (#1528)
* Remove password info from log.

* update.

* one more.
2023-07-22 20:25:45 -07:00
Miccah
91c5472876
Implement SourceManager basics (#1515)
* Implement SourceManager basics

* Rename identifiers and add a default headlessAPI implementation

* Rewrite to use SourceInitFunc

* Update variable name to accurately reflect its value
2023-07-21 15:20:25 -05:00
Miccah
a613bbb979
[chore] Remove parent manipulation in context package (#1525)
The ability to set the parent allowed creating context cycles which
shouldn't be allowed, or at the very least have unintuitive behavior.
2023-07-21 13:51:51 -05:00
Cody Rose
ebf1038392
Support indeterminacy in alchemy and update detector docs (#1510) 2023-07-21 14:50:14 -04:00
Cody Rose
06a562688d
capture json error (#1509) 2023-07-21 10:44:47 -05:00
Miccah
e8b5e3cea3
Revert "[chore] Remove parent setting / getting in Context wrapper (#1516)" (#1519)
This reverts commit 8ec5e4916c.
This commit is somehow causing AWS verification (and possibly others) to
not work.
2023-07-20 23:31:28 -05:00
Miccah
8ec5e4916c
[chore] Remove parent setting / getting in Context wrapper (#1516)
* [chore] Remove parent setting / getting in Context wrapper

* Keep the cancellable context from errgroup
2023-07-20 13:33:09 -05:00
Cody Rose
20b7793828
JDBC indeterminacy (#1507)
This PR adds an indeterminacy check to the JDBC verifiers.
2023-07-19 16:57:57 -04:00
Brandon Yan
8fad5fff79
add dockerhub scanner (#1496)
* add dockerhub scanner

* clean

* clean and fix regex logic and tests

* check length of userMatches before access

* Use camelcase.

---------

Co-authored-by: Ahrav Dutta <ahravdutta02@gmail.com>
2023-07-19 09:26:28 -07:00
Cody Rose
cb1a63a4e2
unify JDBC detector ping logic (#1506)
Previously, the various JDBC detectors would independently try to verify credentials by a process of trying various permutations of candidates one-by-one. The upcoming tri-state verification work will need to add sophistication to this process in the same way for each one, so this PR first combines all of the logic so it can be upgraded in a single spot.
2023-07-19 11:45:56 -04:00
Zubair Khan
be549a7287
add thog enterprise detector for web keys (#1448)
* saving progress

* proto changes

* run make protos

* verify response, add test case

* resolve linter warning about unescaped . in regex pattern

* resolve overlapping proto number
2023-07-18 09:53:12 -04:00
Brandon Yan
cab416b533
add launch_darkly keyword to launchdarkly scanner (#1495) 2023-07-17 14:05:58 -05:00
Cody Rose
ee814a67bd
tweak jdbc redaction (#1490)
JDBC redaction could fail in some irritating edge cases involving passwords that contain the @ character. The logic has been tweaked to eliminate these cases and some tests have been added.
2023-07-17 11:04:12 -04:00
Brandon Yan
17b90b32d0
add couchbase scanner to defaults (#1497)
Co-authored-by: Ahrav Dutta <ahravdutta02@gmail.com>
2023-07-16 17:45:56 -07:00
Brandon Yan
9af31f00a9
add envoy api key scanner (#1482)
* add envoy api key scanner

* Use detectors4.

---------

Co-authored-by: Ahrav Dutta <ahravdutta02@gmail.com>
2023-07-16 16:46:28 -07:00
trufflesteeeve
d03a74776e
Fix URI detector false results when the redacted password has been URL encoded (#1489) 2023-07-14 13:35:50 -04:00
Brandon Yan
aab8fddc67
fix twilio verification side effect (#1494)
change POST request to GET request
2023-07-13 17:48:51 -07:00
Miccah
4e774d1f01
Define SourceUnit chunking interface (#1484)
* Define SourceUnit chunking interface

* Refactor to use a ChunkReporter interface

* Rename shadowed err to scanErr
2023-07-13 14:11:43 -05:00
Miccah
4b7f94dea1
Rewrite SourceUnitEnumerator to use UnitReporter instead of a channel (#1485) 2023-07-13 13:48:33 -05:00
ahrav
a9213a1103
[chore] - Update loop to switch. (#1487)
* Update loop to switch.

* remove unused fxn.
2023-07-12 15:47:43 -07:00
Richard Gomez
1594fddf05
feat(git): include line in github & gitlab links (#1466) 2023-07-11 20:02:27 -07:00
Zachary Rice
b48ac24c46
Dedupe results (#1479)
* init 4 dedupin

* use raw rather than rawv2

* rm comment

* comments

* nits

* clean up and use rawv2 too

* add decoder order test
2023-07-11 15:48:00 -05:00
Cody Rose
a123d5c5e1
do not report 403s as indeterminate in AWS detector (#1481) 2023-07-11 16:35:23 -04:00
Cody Rose
b803a0f701
Report indeterminacy in AWS verifier (#1480) 2023-07-11 15:50:31 -04:00
Zubair Khan
4334af4d34
scan GitHub PR and issue comments (#1435)
* issue comment scanning

* save progress

* test

* test for pr comment and issue comment

* add pagination support

* linter stuff

* make linter happy

* remove debug log

* readd logging

* github issue resolved

* var const block and handle rate limit

* remove magic number

* make gitURLParse a public function to use more generally

* fix test bug

* make comment scanning OPT-IN
2023-07-11 15:13:33 -04:00
Miccah
5c0ffda618
Define SourceUnit enumeration interface (#1428)
* Add CancellableWrite helper function

* Create SourceUnitEnumerator interface and EnumerationResult struct

* Implement SourceUnitEnumerator for the filesystem Source

* Omit explicit zero values
2023-07-10 15:05:40 -05:00
Zachary Rice
d4972313ff
remove old detector (#1474) 2023-07-10 13:02:19 -05:00
Cody Rose
87058dd7fa
Add new verification error message field (#1463) 2023-07-10 11:15:40 -04:00