Commit graph

2330 commits

Author SHA1 Message Date
Marwan Sulaiman
3aa5369608
Add Tailscale detector (#1719)
* Add tailscale detector

* PR feedback: match on first element
2023-09-07 19:11:17 -07:00
ah̳̕mͭͭͨͩ̐e̘ͬ́͋ͬ̊̓͂d
8d66fde6de
updating sendgrid detector to use tri-state verification (#1735)
* updating sendgrid detector to use tri-state verification
2023-09-07 14:21:03 -04:00
ahrav
2a9f34962d
Add optional param to Chunks (#1747)
* Add interface for targeted chunking.

* use optional args.

* update Chunks method signature.

* update tests.

* fix test.

* update QueryCriteria type.
2023-09-07 09:03:37 -07:00
ahrav
f6512ac4ca
Use common chunker for archive handler (#1717)
* optimize the ReadToMax.

* add comment.

* remove dumb comment.

* update comment.

* fix test.

* lint.

* Expired invite link fix (#1713)

* Use comon chunker for archive handler.

---------

Co-authored-by: Zachary Rice <zachary.rice@trufflesec.com>
2023-09-06 09:26:33 -07:00
Harmon Herring
bf581ae9f9
Fix pagerdutyapikey Detector (#1749)
* Fix pagerdutyapikey detector by broadening regex

* Add 'pd' to list of pagerdutyapikey keywords
2023-09-06 09:15:39 -07:00
ah̳̕mͭͭͨͩ̐e̘ͬ́͋ͬ̊̓͂d
d3e7c5acc2
updating jiratoken and jiratokenV2 to use tri-state verification + updating tests (#1744) 2023-09-05 16:32:05 -04:00
ahrav
abb131e502
[chore] - update Docker source (#1708)
* Add concurrency and common chunker.

* lint.

* address comments.
2023-09-05 07:40:38 -07:00
ah̳̕mͭͭͨͩ̐e̘ͬ́͋ͬ̊̓͂d
b28a2b9177
updating sendbird detector to use tri-state verification (#1737)
* updating sendbird detector to use tri-state verification
2023-09-05 10:39:05 -04:00
Cody Rose
afe708519b
Validate S3 source (#1715)
This PR adds S3 source validation. This is accomplished by factoring out common "bucket visiting" logic to be used by both scanning and validation.
2023-09-05 10:18:58 -04:00
ahrav
c9e6086644
Correctly create azure git links. (#1743) 2023-09-01 10:38:51 -07:00
ahrav
000065b225
[chore] - fix sentry detector (#1738)
* fix sentry detector to check response.

* use err.

* address comments.
2023-09-01 10:33:21 -07:00
s.shivasurya
0a949d7131
iterating on suggestions (#1742) 2023-09-01 10:20:18 -07:00
Mike Vanbuskirk
2e4b17d3f4
update jira detector to match new variable tokens (#1720)
* update jira detector to match new variable tokens

* add versioned interface

* use _v2 format for naming packages w. versioner

- also added documentation for internal/external contrib.

* migrate jira and jira_v2 secrets tests to newer version

* add v2 specific domain and email

* add support for tri-state verification

---------

Co-authored-by: Zubair Khan <zkhan124@umd.edu>
2023-09-01 12:14:36 -04:00
Zubair Khan
78bbb89a30
add tri-state verification for twilio detector (#1729)
* add tri state for twilio

* save progress

* fix twilio tristate test

* resolve lint issue
2023-08-31 12:06:18 -04:00
s.shivasurya
6695cf1dce
added sourcegraph token verification detection (#1730) 2023-08-31 08:47:13 -07:00
Miccah
4aa91337e5
Update to Go 1.21 (#1733)
* Update to Go 1.21

* go mod tidy
2023-08-31 08:46:43 -07:00
Zubair Khan
07702ea06d
update slack webhook with tri-state verification (#1724)
* add tristate basics

* update test

* cleanup

* fix leading space

* fix accidental comment

* save changes

* update secret in gsm, fix bug

* fix linter issue

* cleanup

* use defaultClient
2023-08-30 18:18:17 -04:00
Cody Rose
a2c0abbfd6
Unify S3 client creation logic (#1657)
This PR unifies some code paths within the S3 source. This is being done to better support a future implementation of S3 source validation; less code that runs means less code to validate. The logical change is to move the handling of "role-less" operation down the call tree, which allows for a single code path for more of the S3 code.

This PR also fixes a bug that would occur in the (rare) case that the source couldn't create a regional S3 client. Before, an error would be logged, but it would be followed by a panic. Now the bucket in question is skipped.
2023-08-30 17:49:37 -04:00
Miccah
522b2fab29
Add a cancel cause to job cancellation (#1728) 2023-08-30 12:00:44 -07:00
Miccah
c77c1172c8
Add the 'Cause' family of functions to the context wrapper library (#1725)
Go 1.20 introduced `WithCancelCause`, `WithTimeoutCause`, and
`WithDeadlineCause` to allow adding a reason to context cancellations.
Adding it to our wrapper will allow us to use these features.
2023-08-30 07:57:45 -07:00
ahrav
d6afca682b
remove fmt.Print (#1727) 2023-08-30 07:17:38 -07:00
ahrav
4dc5eb7912
Optimize read to max (#1714)
* optimize the ReadToMax.

* add comment.

* remove dumb comment.

* update comment.

* fix test.

* lint.

* address comments.

* use limit reader.

* update equality check.

* update test.'

* use custom limit reader.

* address comments.

* revert fun.
2023-08-29 17:31:40 -07:00
Miccah
7ba880f47a
Add AvailableCapacity method to SourceManager (#1665) 2023-08-29 12:36:44 -07:00
ahrav
2b1b1b5ad0
Add jobID to chunk. (#1721) 2023-08-29 12:02:30 -07:00
ahrav
c51e8f8af5
buffer channel. (#1718) 2023-08-28 18:08:31 -07:00
Zubair Khan
66271aed7b
add detectors that were missed (#1716) 2023-08-28 18:02:11 -07:00
Zachary Rice
d1b1ac408e
Expired invite link fix (#1713) 2023-08-28 14:21:13 -05:00
ahrav
0932ea224b
[chore] - Prevent nil deref panic (#1709) 2023-08-26 20:39:50 -07:00
Miccah
5eb776cd61
Support cancelling a run from a JobProgressRef (#1663) 2023-08-25 10:43:33 -07:00
Cody Rose
33eed42e17
Test S3 role assumption (#1655)
This PR adds a test of the S3 role assumption functionality. It currently only tests role assumption within a single account.
2023-08-25 11:30:08 -04:00
Miccah
61977412df
Add SourceName to JobProgressRef (#1664) 2023-08-25 07:48:25 -07:00
ahrav
4f4a79f62b
Support azure git links (#1662)
* Support azure git links.

* update comment.

* update test names.
2023-08-24 14:36:52 -07:00
Miccah
f2bfcc7ac6
Capture source-reported progress in JobProgress snapshot (#1661) 2023-08-24 11:28:50 -07:00
Miccah
a4401939a8
Add ElapsedTime method to JobProgressMetrics (#1660) 2023-08-24 11:28:33 -07:00
Zubair Khan
519646342e
add snowflake detector (#1653)
Detect Snowflake secrets (compound URI of account, username, password) and enrich Secret Result with account and databases that the secret has access to.
2023-08-24 13:29:58 -04:00
trufflesteeeve
4b7f0c860a
Update launchdarkly regex, support sdk keys, add tri-state verification (#1645) 2023-08-24 12:17:34 -04:00
ahrav
a5fbc54312
[chore] - update benchmarks. (#1641)
* update benchmarks.

* remove dupe timer reset.
2023-08-23 14:34:10 -07:00
ahrav
a2a7a2087e
[chore] - update comments and logs. (#1654)
* update comments and logs.

* Update github.go
2023-08-23 13:18:07 -07:00
ahrav
9ae72308be
Include the job ID in a chunk (#1652)
* Include the job ID in a source's chunk.

* address comments.

* address comments.
2023-08-22 14:00:27 -07:00
Zubair Khan
fd00d2b30b
add rate limit and consumption metrics for GitHub (#1651)
* add rate limit and consumption metrics

* incrment after each repo scanned

* update repo scanned label name
2023-08-22 15:01:59 -04:00
Cody Rose
059ea23a72
update s3 test bucket (#1649)
We're switching our S3 source test account over to a different one, which means we have to change the bucket name.
2023-08-22 12:43:38 -04:00
Miccah
5cfbde783f
Fix reversed ordering of arguments (#1648)
The source manager initialization function was defined as `sourceID`
followed by `jobID`, while the source initialization function is the
reverse. This is confusing and easy to mix up since the parameters are
the same type.

This commit adds a test to make sure the source manager initializes in
the correct order, but it doesn't prevent the library user to make the
same mistake. We may want to consider using different types.
2023-08-22 07:55:56 -07:00
Zubair Khan
9a13c74a35
add thog CLI support for GitHub config validate (#1626)
* add exportable validate function for github

* update validator

* use the context

* gate to prevent panic

* wrap error with context

* wrap error with context for basic auth and unauth
2023-08-22 10:22:39 -04:00
Cody Rose
dbb2c2e319
wait before finishing s3 test (#1647)
The S3 source test verifies that chunking has completed, but it didn't actually wait for completion first, leading to non-deterministic test failures.
2023-08-21 12:36:36 -04:00
Cody Rose
ed06217862
Add tri-state verification to sqlserver detector (#1624)
This is a different detector than the general JDBC detector.
2023-08-21 10:05:45 -04:00
ahrav
d51e3b6d83
Only scan gist comments or repo comments. (#1646) 2023-08-20 11:38:28 -07:00
Mike Vanbuskirk
64dd49f9ce
add role assumption for s3 source (#1477)
* add role assumption for s3 source

* refactor role assumption to repeatable string

user can pass array of roles to assume

* refactor s3 chunks to handle passed roleARNs

* add role-session name

use timestamp to make dynamic

* add docstring for rolearn strings()

* make sure role ars are passed into source

* refactor role assumption functionality

break s3 bucket scanning into sep. function

* add log check on assume role

* fix role iteration

- Make sure s3 struct is populated with roles
- add separate new client instantiation for role-based access
- iterates through each role

* add comment

* protobuf revert for merge

* re-run make proto

* lint cleanup

* cleanup TODOs

* drop redundant switch case in assumerole client

* use less verbose 'ctx' designator

* breakout functionality from Chunks

- separate functions for:
- enumerating buckets to scan
- scanning objects within the buckets

* remake protobuf defs

* allow scan to continue on single bucket err

* add readme docs

* minor fixups
2023-08-17 20:30:20 -04:00
ahrav
0ae8cf5d35
[bug] - handle IOOR panic (#1639)
* handle IOOR panic.

* use a better fxn name.

* increae timeout for test to compete.

* simplify code and add test.

* do it for miccah.
2023-08-17 15:47:11 -07:00
ahrav
55b9d48e0d
updat test file. (#1637) 2023-08-17 10:16:25 -07:00
ahrav
f3c2d5e6c7
[bug] - Correctly reset reader before handling archive chunk data (#1636)
* Correctly reset reader before handling archive chunk.

* stop the re-reader.
2023-08-17 10:04:43 -07:00