Brendan Shaklovitz
da5301ea1e
Exit with non-zero exit code on chunk source error ( #1286 )
...
* Exit with non-zero exit code on chunk source error
* Exit with a non-zero exit code whenever we hit an error getting
chunks. Previously the error would be logged but trufflehog would exit
with a 0 (success) status code.
* fix gcs test
---------
Co-authored-by: Dustin Decker <dustin@trufflesec.com>
Co-authored-by: ahrav <ahravdutta02@gmail.com>
2023-06-26 11:39:57 -05:00
Chris Atkin
6e6895b48e
Update Slack webhook error text for verification ( #1427 )
...
This updates the matched error text to determine the verified status of a Slack webhook, as this has been updated on Slack's API.
2023-06-26 08:44:17 -07:00
Zubair Khan
cd67f6bf16
prevent www from being a key to prevent fp ( #1418 )
2023-06-25 11:55:11 -04:00
Dustin Decker
eeefde1ec9
Ensure results are collected correctly when verification is off, and dedupe twilio ( #1420 )
2023-06-23 14:14:08 -07:00
Miccah
f3152b6885
Implement SourceUnitUnmarshaller for all sources ( #1416 )
...
* Implement CommonSourceUnitUnmarshaller
* Add SourceUnitUnmarshaller to all sources using
All sources, with the exception of git, will use the CommonSourceUnit as
they only contain a single type of unit to scan.
* Fix method comments to adhere to Go's style guide
2023-06-23 11:15:51 -05:00
Dustin Decker
e856a6890d
🎉 Add Docker image scanning 🎉 ( #1412 )
...
* Add Docker source
* Add metrics
* Add test
* Add debugging, address PR comments, fix path output
* review suggestions
2023-06-22 08:02:25 -07:00
dillonstreator
648ef3b52c
fix spelling errors ( #1413 )
2023-06-21 07:15:28 -07:00
Zubair Khan
0c3410c5cd
add new key pat for mailgun detector ( #1375 )
...
* add new detector key pat for mailgun
* resolve mailgun issue
* remove unused tokenPat and commented strings import
* fix closing bracket issue
2023-06-20 19:14:56 -04:00
Zachary Rice
4938d67e66
Custom detector name ( #1400 )
...
* hacky way to add detector name to output
* set name in custom detectors
2023-06-20 13:55:31 -05:00
Miccah
e12f0f84a1
Setup SourceUnit interface ( #1393 )
...
* Test: Asymmetrical unmarshal API
* Test: Symmetric marshal API
* Revert "Test: Symmetric marshal API"
This reverts commit f51c64a797
2023-06-16 10:38:28 -05:00
Bill Rich
401688d0a8
Add Validator interface and example ( #1397 )
...
* Add Validator interface and example
* Close sockets and improve error messages
* Remove duplicate error
* Use var declaration so err slice can be nil
2023-06-15 08:24:32 -07:00
Bill Rich
6d9ae7acbb
Make trace error message so newlines aren't escaped ( #1396 )
2023-06-14 17:24:31 -07:00
Miccah
fb76eaf17b
Use heuristic to choose the most likely UTF-16 decoded string ( #1381 )
...
* Use heuristic to choose the most likely UTF-16 decoded string
* Assume ASCII and include valid BE and LE bytes
* Remove unused code
* Assume ASCII and return nil when not utf16
---------
Co-authored-by: bill-rich <bill.rich@gmail.com>
2023-06-13 17:00:40 -07:00
Zachary Rice
74ffbd2878
add a custom detector check for logging duplicate detector ( #1394 )
...
* add a custom detector check for logging duplicate detector
* use pb type
2023-06-13 14:49:21 -05:00
Bill Rich
c2e3e7d53a
Split files instead of using ReadAll ( #1387 )
...
* Split files instead of using ReadAll
* Remove dup chunk
* Actually break out of loop
2023-06-12 14:09:05 -07:00
Dustin Decker
ca1947291b
Update sqlserver redaction, deduplication, and URI redaction ( #1369 )
...
* Update sqlserver redaction, deduplication, and URI redaction
* don't use pointer
2023-06-09 11:06:54 -07:00
Zachary Rice
c28c70b399
fix new git file plus plus plus bug ( #1386 )
2023-06-08 18:29:11 -05:00
Zubair Khan
dfb1a0cd38
Add DocuSign detector ( #1382 )
...
* init
* look for client id and client secret, encode them for basis auth
* add tests
* test without checking the contents of response
* confirm access_token exists
* cleanup test
* explain in code that an undocumented grant_type is used
* remove use of deprecated ioutil, remove dead code, return errors instead of just logging
* directly pull access token
* update error text, remove redundant body close()
* import new detector into defaults
2023-06-08 13:34:50 -04:00
ahrav
ce4a1fd7e6
[chore] - fix test ( #1383 )
...
* fix test.
* fix import order.
* fix twilio test.
2023-06-06 18:58:00 -07:00
Dustin Decker
572cb0e5dc
Loosen up version check for git
2023-06-01 12:17:48 -07:00
Dustin Decker
183037ab34
Check that git meets version requirements ( #1373 )
2023-06-01 09:41:06 -07:00
ahrav
8b7c50825e
update detector regex. ( #1368 )
2023-06-01 08:16:18 -07:00
Dustin Decker
5358ed776b
fix mockaroo fps ( #1370 )
...
* fix mockaroo fps
* fix test
2023-05-30 20:58:41 -07:00
Dustin Decker
c8944825de
Surface missing git as an error during initialization ( #1362 )
2023-05-26 15:23:08 -07:00
Tim Strazzere
cbfbf5335e
Add Data member to ResultsMetadata struct. ( #1358 )
...
When a Result is emitted, it should include
the `chunk.Data []byte` so that we can utilize
the blob of data which caused the result.
This makes it so something catching the results
does not have to maintain a collection of chunks
to correlate the two together.
2023-05-24 09:21:41 -07:00
Brendan Shaklovitz
3ab864aca9
Make OpenAI regex more specific ( #1345 )
2023-05-22 07:39:18 -07:00
ahrav
1da7720912
Replace context.TODO. ( #1349 )
2023-05-19 11:09:51 -07:00
ahrav
0c386220dd
[chore] - Use correct detector proto ( #1347 )
...
* Use correct detector proto.
* sort imports.
2023-05-18 15:12:38 -07:00
trufflesteeeve
1a81709726
Check to see if StructuredData exists before attempting to print it ( #1346 )
2023-05-18 17:42:19 -04:00
Brendan Shaklovitz
195f9f0798
Add Base64URLSafe decoder ( #1292 )
...
* Add Base64URLSafe decoder
* Add decoder that can decode base64 strings with '_' and '-' instead of
of '+' and '/'.
* Combine url-safe b64 decoder into b64 decoder
2023-05-18 08:30:47 -07:00
RuchitaKshirsagarTR
f831b62a3f
Update generic.go ( #1343 )
...
Generic API keys like shown in the example below is getting excluded:
api_key=9e107d9d372bb6826bd81d3542a419d6 because of following regex patterns:
\b[A-Fa-f0-9]{32}\b
\b[A-Fa-f0-9x]{6,99}\b
The base64 decoding logic is getting hit and NOT returning an error, and thus it continues thinking it is base64 decoded.
2023-05-17 13:30:40 -07:00
ahrav
31844b12e3
[oc-313] - Add GitHub metrics ( #1324 )
...
* Normalize repos during enumeration.
* fix test.
* Add benchmark.
* Add benchmark.
* Add more realistic benchmark values.
* add gist mocks.
* Remove old normalize fxn.
* abstract away the repo cache.
* update test.
* increase repo count.
* increase page limnit to 100.
* move callee fxns below caller for Chunks.
* Add context to normalize.
* remove extra logic in normalize repo.
* Delete new.txt
* Delete old.txt
* Handle errors in a thread safe manner.
* fix test.'
* fix test.
* handle repos that are included by users.
* Abstract include ignore logic within repoCache.
* Add better comment around repoCache.
* Rename params.
* remove commented out code.
* use repos instead of items.
* remove commented out code.
* Use ++ instead of atomic increment.
* update to use logger var.
* use cache pkg.
* Use separate file for repo logic.
* Address comments.
* fix test.
* make less sucky test.
* Update test.
* Add logs for duration and repo size.
* fix integration test.
* address comment.
2023-05-16 08:45:28 -07:00
Brendan Shaklovitz
88b4a283c4
Add extra data and structured data to plain output ( #1316 )
...
* Add extra data and structured data to plain output
* Remove duplicate ExtraData output
2023-05-16 08:14:42 -07:00
Dustin Decker
4250773e92
GitHub basic auth ( #1337 )
2023-05-15 22:04:42 -07:00
ahrav
e81b908e07
Add buildkitev2 detector for newer tokens. ( #1341 )
2023-05-15 12:58:36 -07:00
ahrav
6db770fbe5
use md5 hash for checking if key exists. ( #1257 )
2023-05-15 10:04:14 -07:00
ahrav
948828ba8c
[chore] - move objectManager interface ( #1332 )
...
* Relocate the objectManager interface to the consumer package as per Go
best practices.
* address comment.
2023-05-15 09:30:26 -07:00
vickygoel
4c04bbbe85
added pulumi cloud Access token detector ( #1295 )
...
* added pulumi cloud Access token detector
* removed accidentally committed tokens
* added the databricks token detection
* made recommended changes
* added supabase management api token
* nuget api key detector
* added aiven.io token detector
* added prefect.io api key detector
* update protos.
---------
Co-authored-by: Developer <garg47294+1@gmail.com>
Co-authored-by: Ahrav Dutta <ahravdutta02@gmail.com>
2023-05-11 09:08:48 -07:00
Brendan Shaklovitz
584db86031
Support line numbers in filesystem source ( #1297 )
2023-05-09 08:02:34 -07:00
Brendan Shaklovitz
fad34d4dc6
git worktree scanning fix for #827 ( #1315 )
...
* Fix worktree scan by setting EnableDotGitCommonDir
* Change `PlainOpenOptions` to set `EnableDotGitCommonDir` to true.
In every current usage of this function, it is on an already-cloned
repository, so it should always be valid to have this set. By doing
so, it should fix some issues with worktrees.
* Remove unused go.mod replace directives
* Remove replace directives for libraries that are not in use.
2023-05-09 08:00:47 -07:00
Brendan Shaklovitz
e3213fbdeb
Do extraction after decompression ( #1320 )
...
* Fix error where some files do not get properly scanned due to order of
extraction / decompression steps. Doing decompression first ensures
that a compressed archive (e.g., gzipped zip file), is handled
correctly.
2023-05-09 07:56:08 -07:00
Bill Rich
f2924f3061
Make sure context lines are properly handled ( #1331 )
...
* Make sure context lines are properly handled
* Fix git test to account for context change
2023-05-05 12:51:27 -07:00
Miccah
6699ccd2b5
Generate protos ( #1329 )
2023-05-04 12:26:41 -05:00
Brendan Shaklovitz
87f3f27dab
Fix SquareApp detector type return value ( #1322 )
...
* Change SquareApp detector type to report as SquareApp instead of
Square.
2023-05-04 10:25:20 -07:00
ahrav
deb0f63d25
Update regex. ( #1328 )
2023-05-04 10:23:13 -07:00
ahrav
030c093392
Fix how we scan orgs ( #1327 )
...
* Fix how we scan orgs.
* fix integration test.
2023-05-04 08:07:11 -07:00
Brendan Shaklovitz
be4147a24e
Output git timestamps as UTC times ( #1323 )
2023-05-03 11:47:00 -05:00
ahrav
323c093818
Normalize GitHub repos during enumeration ( #1269 )
...
* Normalize repos during enumeration.
* fix test.
* Add benchmark.
* Add benchmark.
* Add more realistic benchmark values.
* add gist mocks.
* Remove old normalize fxn.
* abstract away the repo cache.
* update test.
* increase repo count.
* increase page limnit to 100.
* move callee fxns below caller for Chunks.
* Add context to normalize.
* remove extra logic in normalize repo.
* Delete new.txt
* Delete old.txt
* Handle errors in a thread safe manner.
* fix test.'
* fix test.
* handle repos that are included by users.
* Abstract include ignore logic within repoCache.
* Add better comment around repoCache.
* Rename params.
* remove commented out code.
* use repos instead of items.
* remove commented out code.
* Use ++ instead of atomic increment.
* update to use logger var.
* use cache pkg.
* Address comments.
* fix test.
* make less sucky test.
* Update test.
2023-05-03 08:35:53 -07:00
ahrav
9cb91a6e4f
Extend cache interface ( #1318 )
...
* Extend cache interface.
* update test.
2023-05-03 08:21:00 -07:00
ahrav
714c480931
Add log to track git log size ( #1325 )
...
* Add log to track git log size.
* Add calc for large commits and last commit.
2023-05-02 16:36:39 -07:00
