Mirrors/trufflehog
2
0
Fork 0
mirror of https://github.com/trufflesecurity/trufflehog.git synced 2024-11-10 07:04:24 +00:00
Code Issues Projects Releases Packages Wiki Activity
3021 commits 139 branches 257 tags 147 MiB
Commit graph

2356 commits

Author SHA1 Message Date
ahrav
61c7d52a43
[bug] - close file after reading (#2203) 
* close file after reading

* inline return
 2023-12-11 15:04:30 -08:00
Richard Gomez
d1a2d9e832
chore: propagate log context to handlers (#2191) 2023-12-10 10:30:11 -08:00
Richard Gomez
6c5fc2f212
feat(privatekey): run checks concurrently (#2139) 2023-12-10 10:11:17 -08:00
ahrav
331336dc0a
[fixup] - skip files in the archive handler (#2195) 2023-12-08 20:23:32 -08:00
ahrav
2728e514d2
move logic to main Chunks method (#2194) 2023-12-08 14:51:24 -08:00
ahrav
2a7813929b
add metrics for gitlab (#2190) 2023-12-08 09:50:09 -08:00
ahrav
4b31b39d6b
[chore] - Refactor common code into a separate function (#2179) 
* Refactor common code into a separate function

* rename vars

* make sure to set the scanOptions fields

* address comments
 2023-12-08 08:44:35 -08:00
Cody Rose
ee6923a241
Remove java archives from ignored extensions (#2188) 
A previous commit (5d0196957f) added .jar/.war/.ear files to the ignored extensions list, but these are archive files that we can scan, so we shouldn't exclude them.
 2023-12-07 15:19:56 -05:00
ahrav
b75991850a
[chore] - Compile regex once (#2176) 
* move regex compilation out of the fxn

* missed a spot

* merge main
 2023-12-07 07:26:27 -08:00
ahrav
f772fd8b44
update regex (#2184) 2023-12-06 17:04:38 -08:00
Dustin Decker
3167dde8a1
Deprecate some detectors (#2186) 2023-12-06 16:57:55 -08:00
ahrav
0595a3baac
allow targets for the source manager (#2182) 
* allow targets to the source manager

* use targets
 2023-12-06 16:38:35 -08:00
ahrav
c6e9b8ff64
use https for verification endpoints (#2185) 2023-12-06 16:06:04 -08:00
ahrav
e6bc7f4451
remove unnecessary Git cmd check (#2175) 2023-12-06 13:38:34 -08:00
ahrav
cb81f7d11a
[feat] - Remove go-git dependency (#2174) 
* remove use of go-git for binary files

* fix it

* use limit reader

* fix comment

* fix test

* address comments

* address comments

* address comments
 2023-12-06 13:38:01 -08:00
ahrav
990274b596
Skip trying to determine MIME type for directories (#2178) 2023-12-06 12:00:18 -08:00
dylanTruffle
96aa50d119
fixing how to rotate URL (#2183) 2023-12-06 11:59:21 -08:00
āh̳̕mͭͭͨͩ̐e̘ͬ́͋ͬ̊̓͂d
11394ea318
[thog-1548] add auto redaction for verification errors (#2106) 
* Updating VerificationError to have auto redaction logic
* find/replace error
 2023-12-05 08:57:52 -05:00
ahrav
13da76d357
skip files we can't scan (#2170) 2023-12-04 13:37:11 -08:00
ahrav
996a11dcc0
[chore] - remove deprecated types (#2168) 
* remove deprecated types

* missed one
 2023-12-04 13:23:58 -08:00
Cody Rose
5d0196957f
Ignore images and binaries (#2162) 
This PR expands the list of excluded file extensions to contain images and other binary files. These files can technically contain secrets, but need decoding to properly be handled, and we don't have any such decoding yet. Down the road if we want to add it we can.
 2023-12-04 13:25:29 -05:00
ahrav
37d9e5eedf
[chore] - Increase pagination limit (#2154) 
* increae pagination limit

* rename
 2023-12-04 10:14:46 -08:00
Dustin Decker
07dc123840
update forager types (#2159) 2023-12-03 13:16:16 -08:00
ahrav
c34efc3cf9
make empty slice delcration consistent (#2144) 2023-12-01 11:03:44 -08:00
ahrav
279f915799
[chore] - fix error comparisons (#2142) 
* fix error comparisons

* fix imports
 2023-12-01 08:32:41 -08:00
ahrav
52ffab1034
[chore] - fix import name clashes (#2143) 
* fix import name clashes

* fix missing var
 2023-12-01 06:53:15 -08:00
Dustin Decker
a367f9ce34
Fix azure panic when invalid URL is constructed (#2137) 2023-11-30 11:33:04 -08:00
ahrav
8880c2e005
fixup cleantemp (#2136) 2023-11-30 09:39:30 -08:00
Miccah
e498c80b3d
Fix nil pointer dereference when checking if a unit IsFinished (#2135) 2023-11-29 14:19:31 -08:00
Miccah
7ecd43ab1e
[chore] Minor cleanup of source_manager.go (#2134) 2023-11-29 11:08:25 -08:00
Dustin Decker
363ccab316
Simplify temp dir cleaning (#2133) 
* Simplify temp dir cleaning

* rename vars

* add test

* update test
 2023-11-28 16:42:17 -08:00
Dustin Decker
ede0c39589
Add new auth method to source (#2132) 2023-11-28 10:58:11 -08:00
Zachary Rice
d552222385
add extradata nil check and use make (#2129) 
* add extradata nil check and use make

* remove some lines
 2023-11-28 09:45:37 -06:00
Miccah
78219a27b3
Call Finish in SourceManager after the semaphore is released (#2121) 2023-11-24 13:22:08 -08:00
Richard Gomez
024aa056b9
chore(github): add a newline between titles and bodies (#2124) 2023-11-23 16:14:28 -08:00
Richard Gomez
1f502fd42c
feat(github): scan issue & pr titles (#1899) 2023-11-22 19:15:27 -08:00
ahrav
0e6e1dce3f
use camelcase var names (#2123) 2023-11-22 09:09:04 -08:00
Oleksandr Redko
7d10e2540e
Remove unused functions (#2122) 2023-11-22 06:58:16 -08:00
Dustin Decker
a7d330a2a5
import missing detectors (#2119) 2023-11-21 10:30:11 -08:00
Dustin Decker
75e869faff
Fix forks and repos counter, add metric for orgs enumerated (#2118) 2023-11-21 08:52:33 -08:00
Richard Gomez
62c628fb52
feat(telegram): add username to extradata (#2100) 2023-11-20 14:00:09 -08:00
Dustin Decker
9e88cdf625
add extra data to github detector (#1909) 
* add extra data to github detector

* Add verification error
 2023-11-20 13:55:16 -08:00
joeleonjr
cd9c1ae186
fixed gist direct link generation (#2115) 
* fixed gist direct link generation

* added two test cases for gist link generation
 2023-11-20 13:41:19 -05:00
Zachary Rice
d69de658b2
fix nil map assignment (#2117) 2023-11-20 11:13:09 -06:00
Miccah
39a603d2dc
[chore] Add JSON tags to job metrics (#2114) 2023-11-16 17:08:33 -08:00
ahrav
d334b3075e
move all Git setup into Init method (#2105) 
* add proto fields for git

* add uri to proto

* move all git setup into Init method

* fix logic for when to use repoPath
 2023-11-16 13:59:53 -08:00
ahrav
fd33198ad8
add proto fields for Git (#2104) 
* add proto fields for git

* add uri to proto

* add comment
 2023-11-16 13:52:38 -08:00
joeleonjr
b2042e4e03
extract AWS account number from ID without verification (#2091) 
* added GetAccountNumFromAWSID function

* refacted aws func, moved to common
 2023-11-16 11:45:47 -05:00
āh̳̕mͭͭͨͩ̐e̘ͬ́͋ͬ̊̓͂d
737d6b764d
Adding Sumo Logic how to rotate (#2103) 2023-11-09 12:48:08 -05:00
ahrav
76a0468580
update protos so we can use the git source for CI (#2102) 2023-11-08 09:07:29 -08:00
Damanpreet Singh
d066a3fa78
Detector-Competition-Feat: Added Replicate API token detector (#2021) 
* Detector-Competition-Feat: Added Replicate API token detector

* fix fullstory

---------

Co-authored-by: Zachary Rice <zachary.rice@trufflesec.com>
 2023-11-07 12:16:39 -06:00
Damanpreet Singh
bcde7856c3
Detector-Competition-Feat: Added Ngrok API token detector (#2024) 
Co-authored-by: Zachary Rice <zachary.rice@trufflesec.com>
 2023-11-07 09:28:05 -06:00
Ankush Goel
1b93c0545c
Competition-Detector-New:added v2 version for fullstory (#2067) 
* added v2 version for fullstory

* added versioner to the v1 fullstory detector
 2023-11-07 08:55:06 -06:00
Miccah
8e3f6e98dc
Add support for user:pass@host to postgres JDBC detector (#2089) 
* Add support for user:pass@host to postgres JDBC detector

* Remove ineffectual assignment
 2023-11-06 17:17:37 -08:00
Corben Leo
1094190ff5
Detector-Competition-Feat: Add Overloop detector (#2080) 
* Detector-Competition-Feat: Add Overloop detector

* add protos and to defaults.go

---------

Co-authored-by: Zachary Rice <zachary.rice@trufflesec.com>
 2023-11-06 16:43:31 -06:00
Damanpreet Singh
da59b72735
Detector-Competition-Feat: Added Request.Finance API token detector (#2020) 
Co-authored-by: Zachary Rice <zachary.rice@trufflesec.com>
 2023-11-06 16:13:33 -06:00
Ankush Goel
703e158648
Detector-Competition-New : created grafana service account detector (#1960) 
* created grafana service account detector

* add import

---------

Co-authored-by: Zachary Rice <zachary.rice@trufflesec.com>
 2023-11-06 15:41:37 -06:00
Ankush Goel
b2d541e0ea
Detector-Competition-Fix: fixed zulipchat detector (#1990) 
* fixed zulipchat detector

* fixed testing scenarios

* fixed test detector

* fixed test

* made chunking keyword from zulipchat to zulip

* fixed email regex

* fixed domain regex
 2023-11-06 12:22:47 -06:00
Ankush Goel
6259b179b9
Grafana (#2096) 
* Created Grafana Cloud API Key detector

* made the regex more bounded

* added boundary to regex
 2023-11-06 11:13:06 -06:00
Ankush Goel
aabfec4cdf
Competition-Detector-New: added eventbrite detector (#2072) 
* added eventbrite detector

* added packagename to defaults.go

---------

Co-authored-by: Zachary Rice <zachary.rice@trufflesec.com>
 2023-11-03 16:42:13 -05:00
Ankush Goel
1371512ff3
logz.io detector (#2076) 
Co-authored-by: Zachary Rice <zachary.rice@trufflesec.com>
 2023-11-03 16:32:35 -05:00
Ankush Goel
06b5fc25ef
Coda Detector (#2075) 
Co-authored-by: Zachary Rice <zachary.rice@trufflesec.com>
 2023-11-03 15:50:05 -05:00
Zachary Rice
50a3a82cbb
fix (#2094) 2023-11-03 12:56:12 -05:00
Corben Leo
de8889b406
Detector-Competition-Fix: Fix LiveAgent Detector & Verifier (#2001) 
* Detector-Competition-Fix: Fix LiveAgent Detector & Verifier

* update regex
 2023-11-03 12:28:20 -05:00
dylanTruffle
0b90265802
pulling short lived AWS keys into their own thing, fixes #1224 (#2088) 
* pulling short lived AWS keys into their own thing, fixes #1224

* Update awssessionkey.go

* fmt

---------

Co-authored-by: Chair <chair@Chairs-MacBook-Pro.local>
Co-authored-by: Zachary Rice <zachary.rice@trufflesec.com>
 2023-11-03 11:58:49 -05:00
Cody Rose
7a156330b5
Support multiple detectors per match (#2065) 
#1711 inadvertently removed the ability to match multiple custom detectors, or multiple detectors of the same type but different version, to a given keyword. (#2060 re-added support for multiple versions of detectors globally, and #2064 re-added support for multiple custom detectors globally, but neither fixed trufflehog's inability to support multiple such detectors for a given keyword match.) This PR re-adds the removed functionality (and narrows the AhoCorasickCore interface in the process.)
 2023-11-03 12:26:18 -04:00
Miccah
600903f391
[chore] Speedup IsKnownFalsePositive using sets (#2090) 
Also check that the match is a valid UTF-8 string.
 2023-11-03 08:45:00 -07:00
Corben Leo
3b9ecaa704
Detector-Competition-Fix: Fix ScraperSite (deprecated) (#2074) 
Co-authored-by: āh̳̕mͭͭͨͩ̐e̘ͬ́͋ͬ̊̓͂d <13666360+0x1@users.noreply.github.com>
 2023-11-03 11:15:53 -04:00
Corben Leo
41e9cc59e2
Detector-Competition-Fix: Fix PassBase (acquired, deprecated) (#2079) 2023-11-03 08:59:32 -05:00
Ankush Goel
b95ed3b41a
Detector-Competition-New - Created Grafana Cloud API Key detector (#1959) 
* Created Grafana Cloud API Key detector

* made the regex more bounded

* added boundary to regex
 2023-11-03 09:25:54 -04:00
Corben Leo
9e52e3e86f
Detector-Competition-Fix: Fix/Deprecate Prospect.io (#2081) 
* Detector-Competition-Feat: Fix/Deprecate Prospect.io

* Detector-Competition-Fix: fix defaults.go
 2023-11-03 07:04:42 -05:00
joeleonjr
a1d74cd887
added resource type mapping to extraData in AWS (#2087) 
* added resource type mapping to extraData in AWS

* updating aws regex + logic for resource type
 2023-11-02 17:03:03 -04:00
Corben Leo
b5cc6c196c
Detector-Competition-Fix: Fix FakeJSON (deprecated) (#2073) 2023-11-02 15:43:49 -05:00
Ankush Goel
ab896890b4
fixed helpscout detector regex and verifier (#2056) 2023-11-02 14:20:26 -05:00
Ankush Goel
965a274de9
Detector-Competition-Fix: fixed regex for databricks domain and fixed tests (#1965) 
* fixed regex for domain and fixed tests

* fixed regex

* fixed an issue with regex subgrouping

* made recommended changes

* made recommended changed

* fixed RawV2
 2023-11-02 11:26:31 -05:00
Ankush Goel
b6469f23ac
modified regex (#2033) 2023-11-02 11:24:37 -05:00
dylanTruffle
4106ce7bf0
Detector-Competition-Feat: Adding Azure Container Registry Password Detector (#1958) 
* implementing azure container registry password detector

* Fixing boundry feedback

* whoops

* update verification code

* fix regex

---------

Co-authored-by: Chair <chair@Chairs-MacBook-Pro.local>
Co-authored-by: Zachary Rice <zachary.rice@trufflesec.com>
 2023-11-02 11:17:01 -05:00
Corben Leo
07f6c84aa4
Detector-Competition-Fix: Fix SentimentInvestor (deprecated) (#2078) 2023-11-01 11:54:40 -05:00
Miccah
9d6bc8c504
Refactor git source to support scanning units (#2083) 2023-11-01 09:52:58 -07:00
Miccah
52600a897a
[chore] Replace chunks channel with ChunkReporter in git based sources (#2082) 
ChunkReporter is more flexible and will allow code reuse for unit
chunking. ChanReporter was added as a way to maintain the original
channel functionality, so this PR should not alter existing behavior.
 2023-11-01 09:22:44 -07:00
ahrav
d55cb56db4
update comment (#2084) 
update Cache.Contents() comment
 2023-11-01 07:36:22 -07:00
Cody Rose
7197e4b3f1
use rawv2 for pubnubpublish (#2062) 
We're seeing secrets of this type flap between verified and unverified, which is expected behavior for multipart secrets without RawV2 defined. This PR adds RawV2 for secrets of this type.
 2023-11-01 10:14:28 -04:00
ahrav
95e0090bc2
[chore] - correctly handle input shorter than 512 bytes (#2077) 
* correctly handle input shorter than 512 bytes

* add tests

* reorder tests

* add another test case

* update test

* address comment
 2023-10-31 16:42:42 -07:00
āh̳̕mͭͭͨͩ̐e̘ͬ́͋ͬ̊̓͂d
74a56de831
update braintreepayments detector to tri-state verification (#1834) 
* update braintreepayments detector to tri-state verification

* Update pkg/detectors/braintreepayments/braintreepayments.go

Co-authored-by: Zachary Rice <zachary.rice@trufflesec.com>

* small nits

* small nits

---------

Co-authored-by: Zachary Rice <zachary.rice@trufflesec.com>
 2023-10-31 16:13:48 -04:00
dylanTruffle
8bac2b15ba
Detector-Competition-Feat: Adding Azure Batch keys (#1956) 
* adding azure batch

* fmt

* fix lint

---------

Co-authored-by: Chair <chair@Chairs-MacBook-Pro.local>
Co-authored-by: Zachary Rice <zachary.rice@trufflesec.com>
 2023-10-31 10:49:04 -05:00
dylanTruffle
499cb64546
Detector-Competition-Fix: Fix redis to now support SSL, and look for azure redis connection strings (#1957) 
* adding azure redis, and fixing the old detector to support ssl too

* fix?

* other way

---------

Co-authored-by: Chair <chair@Chairs-MacBook-Pro.local>
Co-authored-by: Zachary Rice <zachary.rice@trufflesec.com>
 2023-10-31 10:17:55 -05:00
Corben Leo
a4fd17c9d1
Detector-Competition-Fix: Fix AppFollow Detection & Verification (#1933) 
* Detector-Competition-Fix: Fix AppFollow Detection & Verification

* fix(regex): update jwt regex for appfollow
 2023-10-31 09:43:20 -05:00
ahrav
a9b056de0a
Centralize logic for checking archive extraction tools (#2063) 
* Centralize logic for checking archive extraction tools

* simplify
 2023-10-30 20:14:51 -07:00
Miccah
57203a56cd
[chore] Fix SourceManager flaky test (#2059) 
* [chore] Fix SourceManager flaky test

Sorting by EndTime is not deterministic, however sorting by StartTime
should be. StartTime is set in a goroutine that's limited by
WithConcurrentUnits, so it should happen in order that the units are
received.

* Sort by unit ID
 2023-10-30 19:16:55 -07:00
Cody Rose
e58a2913ea
Support multiple custom detectors (#2064) 
#1711 accidentally removed the ability to support multiple custom detectors. This PR partially adds back this capability: Multiple custom detectors are now supported overall, but only one custom detector can be returned for a given keyword match.
 2023-10-30 18:17:17 -04:00
Corben Leo
de4a14b3f9
Detector-Competition-Fix: Fix SalesBlink Detection & Verification (#1950) 2023-10-30 16:10:24 -05:00
Damanpreet Singh
244ba3a214
Detector-Competition-Fix: Update formio regex to match Jwt token (#1935) 2023-10-30 16:08:19 -05:00
Corben Leo
6a15cd8f30
Detector-Competition-Fix: Fix Bitcoin Average detector (#1929) 2023-10-30 16:02:30 -05:00
Corben Leo
509fc6c0eb
Detector-Competition-Fix: Fix currencycloud.com API key (#1917) 
* Detector-Competition-Fix: Fix currencycloud.com API environment

* Detector-Competition-Fix: Fix currencycloud.com API environment

* fix(env): update environment
 2023-10-30 15:56:30 -05:00
Cody Rose
45059864f8
Re-add detector version (#2060) 
#2010 mistakenly removed detector version tracking from the Aho Corasick wrapper. This PR re-adds it.
 2023-10-30 15:34:33 -04:00
Dustin Decker
05fae156e1
Add TravisCI source (#1877) 
* Add TravisCI source

* update test to use sourcestest

* Remove jobPage loop

ListByBuild does not support pagination, so this was infinitely
repeating. https://developer.travis-ci.com/resource/jobs#find

* Continue chunking on error

* review updates

* update readme

---------

Co-authored-by: Miccah Castorina <m.castorina93@gmail.com>
 2023-10-30 07:28:25 -07:00
Cody Rose
876a55821b
Remove verify flag from Aho-Corasick core (#2010) 
The Aho-Corasick wrapper we have tracks information about whether verification should be enabled on an individual detector basis, but that functionality isn't related to the matching functionality of Aho-Corasick, and including it complicates the implementation. This PR removes it to simplify some things.

This PR removes some code that supported a potential future implementation of detector-specific verification settings, but that feature has not actually been implemented yet, so there's no loss of functionality. If we want that feature we can add it back on top of this in a more separated way.
 2023-10-30 09:52:51 -04:00
Ankush Goel
2a66d4117a
adding 'token' keyword to regex for github_old (#2037) 2023-10-29 20:45:35 -07:00
Damanpreet Singh
7a9332152a
Detector-Competition-Feat: Added Reply.io API token detector (#2019) 
Co-authored-by: Ahrav Dutta <ahrav.dutta@trufflesec.com>
 2023-10-29 17:57:36 -07:00
Damanpreet Singh
0068ec54f2
Detector-Competition-Feat: Added Stripo API token detector (#2018) 
* Detector-Competition-Feat: Added Stripo API token detector

* adjust regex

---------

Co-authored-by: Ahrav Dutta <ahrav.dutta@trufflesec.com>
 2023-10-29 17:26:14 -07:00
Richard Gomez
0427985ebe
feat: deno deploy detector (#2040) 
Co-authored-by: Ahrav Dutta <ahrav.dutta@trufflesec.com>
 2023-10-29 16:58:00 -07:00
Damanpreet Singh
3ffc0dfd22
Detector-Competition-Feat: Added Budibase API token detector (#2016) 2023-10-29 10:12:45 -07:00
Damanpreet Singh
52b3c99868
Detector-Competition-Feat: Added LemonSqueezy API token detector (#2017) 
* Detector-Competition-Feat: Added LemonSqueezy API token detector

* fix regex

---------

Co-authored-by: Ahrav Dutta <ahrav.dutta@trufflesec.com>
 2023-10-28 15:03:14 -07:00
Richard Gomez
96b25150d0
Add Coinbase Wallet-as-a-Service detector (#1895) 
* feat(coinbase): basic Wallet-as-a-Service detector

* update test

---------

Co-authored-by: Dustin Decker <dustin@trufflesec.com>
 2023-10-27 10:32:36 -07:00
Damanpreet Singh
eb0c0fa99f
Detector-Competition-Feat: Add Metabase Session Secret Detector (#1902) 
Co-authored-by: Ahrav Dutta <ahrav.dutta@trufflesec.com>
 2023-10-26 20:17:41 -07:00
Damanpreet Singh
bf6ece39ca
Detector-Competition-Feat: Added AppOptics API token detector (#1989) 
Co-authored-by: Ahrav Dutta <ahrav.dutta@trufflesec.com>
 2023-10-26 20:06:30 -07:00
Damanpreet Singh
4d0a40d2f3
Detector-Competition-Feat: Added ZeroTier API token detector (#1988) 
Co-authored-by: Ahrav Dutta <ahrav.dutta@trufflesec.com>
 2023-10-26 19:55:58 -07:00
Damanpreet Singh
f1a75395e8
Detector-Competition-Feat: Added BetterStack API token detector (#1987) 
Co-authored-by: Ahrav Dutta <ahrav.dutta@trufflesec.com>
 2023-10-26 19:46:56 -07:00
Corben Leo
98d2922bee
Detector-Competition-Fix: Fix SurveyBot Verification (#1948) 2023-10-26 12:10:00 -05:00
Bill Rich
00a00ef651
Fix binary handling (#1999) 2023-10-26 10:07:02 -07:00
Mike Vanbuskirk
4636dc08f6
Add temp directory management (#1878) 
* adds func to get scannerPIDs

* add cleanup and call to get pids

* move pid handling to git module

* remove PID logic from main

* refactor testing code to handle different exec name

* cleanup linting errors

* add better logging, fix dir if clause

* some PR fixups

* mod fixup

* add interfaces for helper funcs

* refactor cleanup into main, getPID into git

* lint and test fixups, remove fail on n<2 pids

* simplify pid sorting

* use filepath.Join

* use Args[0] for exec name, fix logger

* formatting fixup

* move functionality into cleantemp pkg

* go mod fixup

* remove redundant testing comment

* fix go.sum issues

* add 15m ticker loop for cleanup

* enclose ticker in function for goroutine defer

fix cleantemp interface

* make time more readable

* add check for non-local Trufflehog PIDs

* allow deletion even if no non-local pids found

* bundle intial cleanup into runCleanup func

* add explicit regex check for tempdir format
 2023-10-26 12:28:56 -04:00
Corben Leo
8505d24d7d
Detector-Competition-Fix: Fix/Remove Flowdock detector (#2004) 
* Detector-Competition-Fix: Fix/Remove Flowdock detector

---------

Co-authored-by: āh̳̕mͭͭͨͩ̐e̘ͬ́͋ͬ̊̓͂d <13666360+0x1@users.noreply.github.com>
Co-authored-by: ahmed <ahmed.zahran@trufflesec.com>
 2023-10-26 09:35:13 -04:00
Corben Leo
b776f9c122
Detector-Competition-Fix: Fix/Remove Happi Detection & Verification (#2003) 
* Detector-Competition-Fix: Fix/Remove Happi Detection & Verification

---------

Co-authored-by: āh̳̕mͭͭͨͩ̐e̘ͬ́͋ͬ̊̓͂d <13666360+0x1@users.noreply.github.com>
Co-authored-by: ahmed <ahmed.zahran@trufflesec.com>
 2023-10-26 09:20:53 -04:00
Corben Leo
6914dacde3
Detector-Competition-Fix: Fix/Remove DataFire, API retired (#1995) 
* Detector-Competition-Fix: Fix/Remove DataFire, API retired

* Detector-Competition-Fix: Depreciate Datafire Proto

* make protos for deprecating datafire

---------

Co-authored-by: āh̳̕mͭͭͨͩ̐e̘ͬ́͋ͬ̊̓͂d <13666360+0x1@users.noreply.github.com>
Co-authored-by: ahmed <ahmed.zahran@trufflesec.com>
 2023-10-25 21:51:54 -04:00
Corben Leo
f7960265ea
Detector-Competition-Fix: Fix/Remove QuickMetrics (shutdown) (#1997) 
* Detector-Competition-Fix: Fix/Remove QuickMetrics (shutdown)

* Detector-Competition-Fix: Fix/Remove QuickMetrics protos

* make protos for deprecating Blablabus (#2002)

* make protos for deprecating quickmetrics

---------

Co-authored-by: āh̳̕mͭͭͨͩ̐e̘ͬ́͋ͬ̊̓͂d <13666360+0x1@users.noreply.github.com>
Co-authored-by: ahmed <ahmed.zahran@trufflesec.com>
 2023-10-25 20:05:26 -04:00
āh̳̕mͭͭͨͩ̐e̘ͬ́͋ͬ̊̓͂d
7af4591356
make protos for deprecating Blablabus (#2002) 2023-10-25 19:25:00 -04:00
Corben Leo
51b7fcc5d6
Detector-Competition-Fix: Fix/Remove BlaBlaBus, API retired (#1996) 
* Detector-Competition-Fix: Fix/Remove BlaBlaBus, API retired

* Detector-Competition-Fix: Depreciate Blabus proto
 2023-10-25 18:45:40 -04:00
Corben Leo
cebd92d79e
Detector-Competition-Fix: Depreciate Glitterly (#2000) 2023-10-25 18:08:50 -04:00
Corben Leo
386c54ecbe
Detector-Competition-Fix: Fix MeaningCloud Verification (#1946) 2023-10-25 14:52:36 -05:00
Corben Leo
cef05b8d5a
Detector-Competition-Fix: Fix ScreenshotAPI Verification (#1949) 
* Detector-Competition-Fix: Fix ScreenshotAPI

* Detector-Competition-Fix: Fix ScreenshotAPI
 2023-10-25 14:50:20 -05:00
Ankush Goel
6c35dcffa5
Detector-Competition-Fix : fixed monday.com regex (#1961) 
* fixed monday.com regex

* updating test secrets to use detectors5

---------

Co-authored-by: ahmed <ahmed.zahran@trufflesec.com>
 2023-10-25 12:50:07 -04:00
Damanpreet Singh
b2811bcf78
Detector-Competition-Feat: Added Vagrant Cloud Access Token Detector (#1941) 
Co-authored-by: Ahrav Dutta <ahrav.dutta@trufflesec.com>
 2023-10-25 11:03:45 -05:00
ahrav
4a9d93fe18
remove detector (#1993) 2023-10-25 07:44:58 -07:00
Corben Leo
c674f1fc34
Detector-Competition-Fix: Fix/Remove baseapi detector (no longer exists) (#1992) 2023-10-25 07:17:08 -07:00
Damanpreet Singh
2189dc9b0f
Detector-Competition-Feat: Added Portainer Detector (#1936) 
Co-authored-by: Ahrav Dutta <ahrav.dutta@trufflesec.com>
 2023-10-25 06:32:57 -07:00
ahrav
5901a92acb
deprecate scan_interval field (#1984) 
* deprecate scan_interval field

* rename

* add link to docs
 2023-10-25 06:18:27 -07:00
Damanpreet Singh
b2702b7839
Detector-Competition-Feat: Added OpenVPN API Detector (#1940) 2023-10-25 04:57:07 -07:00
Ankush Goel
84cb33ce3d
loggly detector (#1782) 
* loggly detector

* fixed the loggly_test.go

* fixed the test file to pass the test

---------

Co-authored-by: dsingdev-rocketx <bughunter00@protonmail.com>
 2023-10-24 20:06:47 -07:00
Damanpreet Singh
f467cf923c
Detector-Competition-Feat: Added PortainerToken Detector (#1938) 
Co-authored-by: Ahrav Dutta <ahrav.dutta@trufflesec.com>
 2023-10-24 13:48:40 -07:00
Damanpreet Singh
664c4884a6
fix #1751: update facebookOauth Detector (#1921) 2023-10-24 11:07:52 -05:00
Damanpreet Singh
8184a62e24
fix: NewRelic Detector: fallback to EU Api for verification (#1932) 2023-10-24 11:02:39 -05:00
Corben Leo
7bc0b77374
Detector-Competition-Fix: Fix CloudSmith detection (#1944) 2023-10-24 11:01:27 -05:00
Cody Rose
e556bdd7b2
Revert "Fix off by one (#1891)" (#1963) 
This reverts commit 7f534d0bb7.
 2023-10-24 08:40:44 -07:00
Bill Rich
c5efa870ff
Use latest dbr (#1955) 2023-10-24 07:52:49 -07:00
ahrav
0f845c8eee
export ShouldVerify (#1962) 2023-10-24 07:27:01 -07:00
ahrav
9ae114f92f
export struct (#1954) 2023-10-24 06:29:26 -07:00
Corben Leo
f3479194d2
Detector-Competition-Fix: Fix CodeClimate verification (#1945) 2023-10-23 20:19:02 -05:00
Damanpreet Singh
855aba2407
Detector-Competition-Feat: Add InstaMojo Payment Detector (#1905) 2023-10-23 16:58:25 -05:00
Corben Leo
893bb3548d
Detector-Competition-Fix: Fix SuperNotes API verification (#1947) 2023-10-23 16:29:55 -05:00
Miccah
0b16142d4f
Add UnitHook and NoopHook implementations (#1930) 
* Add UnitHook and NoopHook implementations

The UnitHook tracks metrics per unit of a job, and emits them on a
channel once finished. It should work even if the Source does not
support source units.

* Refactor channel to use an LRU cache instead

An LRU cache has a more favorable failure mode than the channel. With
the channel, if the consumer stopped consuming metrics, scanning would
block. With the LRU cache, metrics will be dropped when space runs out
and a log message emitted.
 2023-10-23 14:27:01 -07:00
Damanpreet Singh
b4753a60be
Detector-Competition-New: add IP2Location api key detector (#1915) 2023-10-23 13:51:14 -05:00
Miccah
136d8b9428
[chore] Fix glob package name (#1931) 2023-10-23 08:50:16 -07:00
ahrav
68f28a0e34
Filter unique detectors by keywords in chunk (#1711) 
* pre filter detectors that include the keywords in the chunk.

* Optimize the engine to prevent iterating overing all detectors.

* use sync.Map for concurrent access.

* lint.

* use correct verify.

* allow versioned detectors.

* Break apart Start.

* cleanup.

* Update benchmark.

* add comment.

* remove Engine prefix.

* update comments.

* use regular map.

* delete the pool.

* remove old code.

* refactor ahocorasickcore into own file.

* update comments

* move structs to ahocorasickcore

* update comments

* fix

* address comments

* exported some methods and constructor since it will need to be be used by the enterprise pipeline as well

* remove extra log
 2023-10-23 08:02:01 -07:00
Corben Leo
6c75e45958
Detector-Competition-Feat: Add ipinfo.io API key detector (#1889) 
* Detector-Competition-Feat: Add ipinfo.io API key detector

* fix prefix
 2023-10-23 09:00:35 -05:00
Miccah
b8724e87e6
Use the configured include repositories in the GitHub filter (#1926) 2023-10-20 19:03:28 -07:00
Richard Gomez
3acc65b2fb
chore(github): reduce comment log verbosity (#1922) 2023-10-20 16:16:38 -07:00
Corben Leo
4cb67a571d
Detector-Competition-Feat: Add Privacy.com API key detector (#1888) 
* Detector-Competition-Feat: Add Privacy.com API key detector

* Detector-Competition-Feat: Add Privacy.com API key detector

* cleanup: fix prefix
 2023-10-20 08:45:16 -05:00
Cody Rose
7ac7fa8728
Move Github comments check to fix a test #1927 2023-10-19 19:23:55 -04:00
Richard Gomez
4b821e9732
Handle secondary GitHub ratelimits (#1912) 
* fix(github): reduce visibility-related api calls

* fix(github): handle secondary ratelimits
 2023-10-19 14:54:45 -04:00
Miccah
758344711a
Export ChunkError fields and add ErrorsFor convenience method (#1920) 2023-10-19 08:46:49 -07:00
Corben Leo
8058006a92
Detector-Competition-Fix: Fix plaid.com API key detection (#1916) 
* Detector-Competition-Fix: Fix plaid.com API key detection

* Detector-Competition-Fix: Fix plaid.com API key detection

* Update plaidkey_test.go

hardcode dev

---------

Co-authored-by: Zachary Rice <zachary.rice@trufflesec.com>
 2023-10-19 10:46:04 -05:00
ahrav
3d7207ddd5
update regex (#1919) 2023-10-19 07:20:35 -07:00
Richard Gomez
b57b1c1aa7
feat(voiceflow): basic detector (#1900) 2023-10-18 16:17:11 -05:00
Damanpreet Singh
a354cbd796
Fix for #1526: Update Posthog detector (#1910) 2023-10-18 15:21:59 -05:00
Miccah
23ae970bb0
Add generic glob filter (#1858) 
* Add generic glob filter

* Make nil filters safe

* Include glob in error

* Use better example for exclude and include test

* Allow user to configure the ambiguous case

* Rename Pass to ShouldInclude and invert logic

* Test default *Filter and Filter have the same behavior of allow

* Add property based tests

* Remove configuration for the not found ambiguous case
 2023-10-18 11:48:31 -07:00
Dustin Decker
93cf523760
Tighten up regex for twist detector (#1908) 2023-10-18 09:17:31 -07:00
Richard Gomez
b46fb75c73
feat(git): only generate line numbers > 0 (#1898) 2023-10-18 06:53:58 -07:00
Richard Gomez
6ea3a7da4a
fix(github): normalize repo cache (#1897) 2023-10-17 15:07:47 -07:00
Shreyas Sriram
7f534d0bb7
Fix off by one (#1891) 2023-10-17 07:02:27 -07:00
Miccah
d4d4d0ec9a
Add ShannonEntropy test for an empty string (#1893) 2023-10-16 13:50:28 -07:00
Miccah
03dc7cb68d
[chore] Add SourceUnitEnumChunker filesystem tests (#1873) 
* [chore] Add SourceUnitEnumChunker filesystem tests

* Ensure reported units are exactly what is expected
 2023-10-16 10:42:18 -07:00
Corben Leo
072e1f9dcf
Detector-Competition-Fix: Add Personal Access Tokens (API Tokens Depr… (#1871) 
* Detector-Competition-Fix: Add Personal Access Tokens (API Tokens Depreciation)

* fix(test): fix test debug msg

* remove print
 2023-10-16 08:17:12 -05:00
ahrav
5c721d1a73
[bug] - Don't modify global client var (#1890) 
* Create a new client within the verify block

* remove unused var
 2023-10-13 12:32:21 -07:00
s.shivasurya
040167178c
added cody gateway token detection code (#1883) 
* added cody gateway token detection code

* resolved conflict
 2023-10-13 09:09:04 -06:00
Corben Leo
ae3a5d1202
Detector-Competition-Feat: Add Klaviyo API Secret Detector (#1870) 
* Detector-Competition-Feat: Add Klaviyo API Secret Detector

* fix(error): add s1.VerificationError and remove specific code check.

* fix(error): add s1.VerificationError and remove specific code check.
 2023-10-11 08:35:04 -06:00
Dustin Decker
52ed87edb7
Add an option to filter unverified results using shannon entropy (#1875) 
* Add an option to filter unverified results using shannon entropy

* lint

* add test, update test, and optimize
 2023-10-08 19:52:28 -07:00
Miccah
f09bce3f75
[chore] Fix flaky TestJobProgressElapsedTime (#1872) 2023-10-06 17:05:05 -07:00
Dustin Decker
22ee2c5b07
Tighten up keywords (#1874) 2023-10-06 16:28:51 -07:00
Corben Leo
77a82847af
Detector-Competition-Fix: fix notion.so false negative verification (#1866) 
* Detector-Competition-Fix: fix notion.so false negative verification

* Detector-Competition-Fix: fix notion.so verification
 2023-10-05 12:27:06 -05:00
Corben Leo
179a7e4cbc
Detector-Competition-New: add anthropic api key detector (#1861) 
* feat(anthropic): add anthropic api key detector

* Detector-Competition-Fix: fix remove debug println
 2023-10-05 11:34:40 -05:00
Corben Leo
bf1cce43e5
Detector-Competition-New: add ramp.com client id & secret detector (#1862) 2023-10-05 09:40:30 -05:00
ahrav
3d2490ca80
use Repositories field from conn. (#1860) 2023-10-04 13:56:02 -07:00
Miccah
273f1077af
Add include and ignore list to Artifactory (#1857) 
* Add include and ignore list to Artifactory proto

* Generate protos
 2023-10-03 16:48:30 -07:00
ahrav
cee456f484
support insecure TLS for Jira and Jenkins (#1856) 
* support insecure TLS for Jira and Jenkins

* lint
 2023-10-03 09:55:38 -07:00
āh̳̕mͭͭͨͩ̐e̘ͬ́͋ͬ̊̓͂d
8d2d8c8395
add tristate verification to postman (#1837) 2023-10-03 12:38:43 -04:00
Hon Kwok
4598244167
Use placeholder as default if field left empty and is required (#1642) 
* Use placeholder as default if field left empty and is required

Co-authored-by: mcastorina <m.castorina93@gmail.com>

* Drop unused func

* uncomment azure source

* update wording

---------

Co-authored-by: mcastorina <m.castorina93@gmail.com>
 2023-10-02 17:21:40 -07:00
Dylan Ayrey
de535071e1
implemented planet scale creds (passwords and API keys) (#1841) 
* implemented planet scale creds (passwords and API keys)

* Add timeout, fix tests, fix indeterminate

---------

Co-authored-by: counter <counter@counters-MacBook-Air.local>
Co-authored-by: Dustin Decker <dustin@trufflesec.com>
 2023-10-02 14:00:36 -07:00
Dylan Ayrey
f13fe36ae2
adding azure storage detector (#1840) 
* adding azure storage detector

* Fix variable name

* Escape regex

* fix test fields and update expected status code

---------

Co-authored-by: counter <counter@counters-MacBook-Air.local>
Co-authored-by: Dustin Decker <dustin@trufflesec.com>
 2023-10-02 13:45:45 -07:00
Dylan Ayrey
b3555f5419
Adding Howtorotate Guides to TruffleHog (#1839) 
* adding how to rotate guides

* Adding project ID to metadata

* update key name, remove comments, and ensure always present

---------

Co-authored-by: counter <counter@counters-MacBook-Air.local>
Co-authored-by: Dustin Decker <dustin@trufflesec.com>
 2023-10-02 13:45:17 -07:00
āh̳̕mͭͭͨͩ̐e̘ͬ́͋ͬ̊̓͂d
f8f0c984fb
update pagerdutyapikey detector to tri-state verification (#1836) 2023-10-02 16:33:18 -04:00
Miccah
0d451aa806
Fix bug in chunker that surfaces with a flaky passed in io.Reader (#1838) 
* Fix bug in chunker that surfaces with a flaky passed in io.Reader

The chunker was previously expecting the passed in io.Reader to always
successfully read a full buffer of data, however it's valid for a Reader
to return less data than requested. When this happens, the chunker would
peek the same data that it then reads in the next iteration of the loop,
causing the same data to be scanned twice.

Co-authored-by: ahrav <ahravdutta02@gmail.com>

* Fix EOF error check

* Use io.ReadFull in Chunker

---------

Co-authored-by: ahrav <ahravdutta02@gmail.com>
 2023-10-02 09:38:23 -07:00
Dylan Ayrey
b232ec8b4e
fixing razorpay (#1852) 
Co-authored-by: counter <counter@counters-MacBook-Air.local>
 2023-10-02 08:49:40 -05:00
āh̳̕mͭͭͨͩ̐e̘ͬ́͋ͬ̊̓͂d
24748b3de6
add tristate verification to twitch (#1830) 
* add tristate verification to twitch

* return early

* small nits
 2023-09-29 16:17:30 -04:00
āh̳̕mͭͭͨͩ̐e̘ͬ́͋ͬ̊̓͂d
5df6afdbf4
Separate gitlab detectors (#1819) 
* update gitlabv2 to tri-state

* updating secret to s1 to match convention

* consolidating both versions of the gitlab detector

* remove gitlabV2 references

* Delete temp.txt

delete test file (note: these are not real secrets)

* updating gitlabV1 detector to only work w/ v1 secrets, and v2 detector only w/ v2 secrets

* update package name and add to defaults

* cleanup nesting

* lowercase package names

* update v1 detector to explicitly ignore results with glpat

* nit

* update package name
 2023-09-28 12:36:46 -05:00
āh̳̕mͭͭͨͩ̐e̘ͬ́͋ͬ̊̓͂d
e645827fcb
[chore] add figmav2 to defaults (#1820) 
* add figma to defaults

* update figma detector package to use versioning
 2023-09-28 13:35:51 -04:00
āh̳̕mͭͭͨͩ̐e̘ͬ́͋ͬ̊̓͂d
afd0b4cc12
Cleanup jiratoken detector (#1832) 
* cleanup nesting on jiratoken v1

* cleanup nesting on jiratoken v2
 2023-09-28 13:35:30 -04:00
āh̳̕mͭͭͨͩ̐e̘ͬ́͋ͬ̊̓͂d
1f2e9d342f
cleanup nesting (#1831) 2023-09-28 13:34:07 -04:00
āh̳̕mͭͭͨͩ̐e̘ͬ́͋ͬ̊̓͂d
f72c77fb69
Cleanup pubnub detector (#1826) 
* pull out verification logic for pubnub to reduce nesting

* remove comment

* return early
 2023-09-28 13:31:07 -04:00
Zachary Rice
28dbd2f704
Update alchemy_test.go to use detectors5 (#1829) 2023-09-28 11:24:45 -05:00
Zachary Rice
3b99517780
Update web3storage_test.go (#1828) 2023-09-28 11:24:29 -05:00
ahrav
c4bc8fc7fa
[bug] - correctly check err (#1824) 
* correctly check err.

* address comments.

* update.

* add comment.

* update comment.
 2023-09-27 15:52:07 -07:00
Cody Rose
e9efed85c2
Use S3 credentials waterfall (#1823) 
This PR updates the S3 source to use explicitly configured credentials if they're available and follow the normal AWS credentials waterfall if they're not. This is irrespective of whether role assumption is configured. This changes the previous behavior, which was to use waterfall credentials only if role assumption was configured and explicitly configured credentials only when it was not.
 2023-09-27 16:57:47 -04:00
joeleonjr
699547b7d3
consolidated pr and issue descr/comment flags (#1827) 2023-09-27 15:54:02 -04:00
Ankush Goel
faf46175e4
added Web3 Storage detector (#1789) 
* added Web3 Storage detector

* fixed the regex

* removed test and disabled token
 2023-09-27 12:09:39 -05:00
ahrav
bf47fd69bb
Github partial scan (#1804) 
* Add ability for targetted partial scans of Github.

* update comment.

* add more tests.

* add additiional test.

* address comments.
 2023-09-26 12:38:33 -07:00
joeleonjr
1e42dae734
added PR and Issue body scanning (#1816) 
* added PR and Issue body scanning; adjusted CLI args to fit

* removed print statement from debugging

* removed exclude-commits; adjusted CLI flags

* minor changes to match main branch

* fixing logic

* updating README for --issues and --prs
 2023-09-26 12:25:48 -04:00
āh̳̕mͭͭͨͩ̐e̘ͬ́͋ͬ̊̓͂d
1715e7ad23
updating browserstack detector to use tri-state verification (#1785) 
* updating browserstack detector to use tri-state verification

* cleaning up nested conditions
 2023-09-25 15:34:13 -04:00
āh̳̕mͭͭͨͩ̐e̘ͬ́͋ͬ̊̓͂d
af1434e05a
updating microsoft teams webhook detector to use tri-state verification (#1792) 
* updating microsoft teams webhook detector to use tri-state verification

* cleaning up nested conditions
 2023-09-25 15:30:41 -04:00
āh̳̕mͭͭͨͩ̐e̘ͬ́͋ͬ̊̓͂d
ac18096da0
updating myfreshworks detector to use tri-state verification (#1779) 
* updating secret regex to include underscores and updating tests to have secret and domain within 20char range of keyword

* updating myfreshworks detector to use tri-state verification
 2023-09-25 13:27:23 -04:00
āh̳̕mͭͭͨͩ̐e̘ͬ́͋ͬ̊̓͂d
cc9f5739dd
update figma to use tri-state verification (#1814) 2023-09-25 13:26:18 -04:00
āh̳̕mͭͭͨͩ̐e̘ͬ́͋ͬ̊̓͂d
f4ddc8b39e
adding support for new version of figma token (#1813) 2023-09-22 18:13:49 -04:00
ahrav
6affc903e1
add line to link for azure repos. (#1801) 2023-09-21 16:07:11 -07:00
āh̳̕mͭͭͨͩ̐e̘ͬ́͋ͬ̊̓͂d
62b2195502
Adding new function SetProgressOngoing to be used when the source does not yet know how many items it is scanning and does not want to display a percentage complete. (#1802) 
Co-Authored-By: @mcastorina
 2023-09-21 13:26:10 -04:00
āh̳̕mͭͭͨͩ̐e̘ͬ́͋ͬ̊̓͂d
1a1b2ca51a
updating uri detector to use tri-state verification (#1791) 2023-09-21 11:20:40 -04:00
Miccah
efa404942a
Add ability to dynamically scale concurrently running sources (#1790) 
* Add ability to dynamically scale concurrently running sources

Refactor SourceManager to use a counting semaphore to allow for
dymanically changing limits. This complicated `Wait() error` which needs
to return the first error encountered. We previously got that for free
using `errgroup.Group`, however now we need to handle that ourselves.
`Wait()` needs to return an error for use in the engine to set the
correct exit code.

* Group third party imports together
 2023-09-20 16:49:56 -07:00
ahrav
a8c89c59b9
[bug] - fix link line (#1793) 
* fix link line.

* rename.
 2023-09-20 14:46:00 -07:00
ahrav
47d5ddebf2
Ability to update line number in link (#1788) 
* Add functionality to update a source's link in the metadata with the updated line number.

* update comment.

* add logic to engine.

* only update link for non empty links.

* add tests for bb.
 2023-09-19 15:39:13 -07:00
Ankush Goel
63eaccd208
fixed rubygems detector (#1781) 
Co-authored-by: dsingdev-rocketx <bughunter00@protonmail.com>
 2023-09-19 06:59:30 -07:00
Ankush Goel
f9ea22f72b
Update sonarcloud.go (#1784) 2023-09-18 08:24:51 -07:00
ahrav
d2676618c0
[bug] - correclty handle nested archived directories (#1778) 2023-09-15 04:37:15 -07:00
ahrav
22876f8381
replace interface{} with any. (#1771) 2023-09-15 04:35:15 -07:00
Miccah
dbcb888063
Update Source interface to use SourceID and JobID types (#1774) 
The previous implementation used int64 for both, which can be mixed up
easily. Using distinct types adds a layer of type safety checked by the
compiler.
 2023-09-14 11:28:24 -07:00
Cody Rose
1155ee2736
Implement Gitlab source validation (#1765) 
This PR implements validation of Gitlab source configuration.

I was hoping to be able to unify more of the implementation of Validate and Chunks, but there was more divergence than I expected. Specifically, Chunks handles a fair number of Gitlab errors that aren't configuration errors (e.g. "Gitlab returned a repo with an unparseable URL"). Accommodating these in the Validate code path felt wrong, and I wasn't able to create a common code path that could accommodate both Validate and Chunks without looking awful.
 2023-09-13 11:51:12 -04:00
martinohmann
31d17c4f93
fix: add missing error check in archive handler (#1770) 
Fixes #1769

The existing error check `errors.Is(err, archiver.ErrNoMatch) && depth >
0` only conditionally handled a specific error.

Any other error case was not short circuited and ended up causing a
nil-pointer dereference further down the method when `format.Name()` was
invoked.
 2023-09-13 07:07:40 -07:00
Miccah
72b6a9ec6b
Add a SourceType constant to all source packages (#1768) 2023-09-12 17:23:25 -07:00
Miccah
be4d0bcb41
Refactor SourceManager to remove Enrollment (#1740) 
* Refactor SourceManager to remove Enrollment

Initializing the Source will be the responsibility of the caller. The
SourceManager exposes a GetIDs method for getting a source and job ID.

* Update tests

* Update engine usage

* Update apiClient interface to have one GetIDs method

* Update SourceManager usage in engine
 2023-09-12 16:58:38 -07:00
âh̳̕mͭͭͨͩ̐e̘ͬ́͋ͬ̊̓͂d
91340c1e00
updating sendbirdorganizationapi detector to use tri-state verification (#1763) 2023-09-12 10:48:11 -04:00
Mike Vanbuskirk
de540652cb
verbosity updates to s3 source (#1750) 2023-09-11 14:53:43 -05:00
ahrav
90a07f0352
[chore] - fix slackwebhook detector (#1761) 
* fix slackwebhook detector.

* sort imports.

* add test for 400.
 2023-09-11 12:48:40 -07:00
Dustin Decker
72b3fa31a3
Improve private key detector (#1760) 
* Surface extra data and check private keys directly against gitlab and github

* fix encrpypted private key test

* implement feedback

* mod tidy

* fix change

* Set timeout for SSH connections
 2023-09-11 12:05:27 -07:00
Zubair Khan
3f84a6700e
add tri state verification to slack (not slack webhook) (#1731) 
* add client, add known false positive check

* fix idiosyncracies

* cleanup

* cleanup comment

* add unexpected error test case

---------

Co-authored-by: Dustin Decker <dustin@trufflesec.com>
 2023-09-11 08:38:43 -07:00
ahrav
fdeccf06a0
cache dupes w/ different decoders (#1754) 
* only cache dupes that have different decoders.

* add test.

* remove file.

* update comment.
 2023-09-11 08:18:48 -07:00
ahrav
70cdff915b
add test for custom providers. (#1759) 2023-09-11 08:18:34 -07:00
Cody Rose
2c32b14df1
always close aws response body (#1758) 2023-09-11 10:42:14 -04:00
ahrav
e53a72abd2
[chore] - Sentry detector update (#1746) 
* add test case for account deactivated.

* Handle empty case.
 2023-09-11 07:26:09 -07:00
Cody Rose
62ce9bac8b
Retry AWS verification 403s (#1757) 
This PR introduces retries on 403s in the AWS detector in attempt to work around erroneous SignatureDoesNotMatch errors. As part of the work, the detector has been refactored somewhat, resulting in two minor semantic changes:

Errors crafting the verification HTTP request no longer result in the candidate secret being discarded.
The known-words-based false positive check now runs (and potentially discards candidate secrets) even if verification is disabled. This unifies its behavior with the hash-based false positive check.
 2023-09-08 15:32:53 -04:00
trufflesteeeve
e68ace5338
Always attempt to return a git link (#1756) 2023-09-08 15:17:29 -04:00
Marwan Sulaiman
3aa5369608
Add Tailscale detector (#1719) 
* Add tailscale detector

* PR feedback: match on first element
 2023-09-07 19:11:17 -07:00
ah̳̕mͭͭͨͩ̐e̘ͬ́͋ͬ̊̓͂d
8d66fde6de
updating sendgrid detector to use tri-state verification (#1735) 
* updating sendgrid detector to use tri-state verification
 2023-09-07 14:21:03 -04:00
ahrav
2a9f34962d
Add optional param to Chunks (#1747) 
* Add interface for targeted chunking.

* use optional args.

* update Chunks method signature.

* update tests.

* fix test.

* update QueryCriteria type.
 2023-09-07 09:03:37 -07:00
ahrav
f6512ac4ca
Use common chunker for archive handler (#1717) 
* optimize the ReadToMax.

* add comment.

* remove dumb comment.

* update comment.

* fix test.

* lint.

* Expired invite link fix (#1713)

* Use comon chunker for archive handler.

---------

Co-authored-by: Zachary Rice <zachary.rice@trufflesec.com>
 2023-09-06 09:26:33 -07:00
Harmon Herring
bf581ae9f9
Fix pagerdutyapikey Detector (#1749) 
* Fix pagerdutyapikey detector by broadening regex

* Add 'pd' to list of pagerdutyapikey keywords
 2023-09-06 09:15:39 -07:00
ah̳̕mͭͭͨͩ̐e̘ͬ́͋ͬ̊̓͂d
d3e7c5acc2
updating jiratoken and jiratokenV2 to use tri-state verification + updating tests (#1744) 2023-09-05 16:32:05 -04:00
ahrav
abb131e502
[chore] - update Docker source (#1708) 
* Add concurrency and common chunker.

* lint.

* address comments.
 2023-09-05 07:40:38 -07:00
ah̳̕mͭͭͨͩ̐e̘ͬ́͋ͬ̊̓͂d
b28a2b9177
updating sendbird detector to use tri-state verification (#1737) 
* updating sendbird detector to use tri-state verification
 2023-09-05 10:39:05 -04:00
Cody Rose
afe708519b
Validate S3 source (#1715) 
This PR adds S3 source validation. This is accomplished by factoring out common "bucket visiting" logic to be used by both scanning and validation.
 2023-09-05 10:18:58 -04:00
ahrav
c9e6086644
Correctly create azure git links. (#1743) 2023-09-01 10:38:51 -07:00
ahrav
000065b225
[chore] - fix sentry detector (#1738) 
* fix sentry detector to check response.

* use err.

* address comments.
 2023-09-01 10:33:21 -07:00
s.shivasurya
0a949d7131
iterating on suggestions (#1742) 2023-09-01 10:20:18 -07:00
Mike Vanbuskirk
2e4b17d3f4
update jira detector to match new variable tokens (#1720) 
* update jira detector to match new variable tokens

* add versioned interface

* use _v2 format for naming packages w. versioner

- also added documentation for internal/external contrib.

* migrate jira and jira_v2 secrets tests to newer version

* add v2 specific domain and email

* add support for tri-state verification

---------

Co-authored-by: Zubair Khan <zkhan124@umd.edu>
 2023-09-01 12:14:36 -04:00
Zubair Khan
78bbb89a30
add tri-state verification for twilio detector (#1729) 
* add tri state for twilio

* save progress

* fix twilio tristate test

* resolve lint issue
 2023-08-31 12:06:18 -04:00
s.shivasurya
6695cf1dce
added sourcegraph token verification detection (#1730) 2023-08-31 08:47:13 -07:00
Zubair Khan
07702ea06d
update slack webhook with tri-state verification (#1724) 
* add tristate basics

* update test

* cleanup

* fix leading space

* fix accidental comment

* save changes

* update secret in gsm, fix bug

* fix linter issue

* cleanup

* use defaultClient
 2023-08-30 18:18:17 -04:00
Cody Rose
a2c0abbfd6
Unify S3 client creation logic (#1657) 
This PR unifies some code paths within the S3 source. This is being done to better support a future implementation of S3 source validation; less code that runs means less code to validate. The logical change is to move the handling of "role-less" operation down the call tree, which allows for a single code path for more of the S3 code.

This PR also fixes a bug that would occur in the (rare) case that the source couldn't create a regional S3 client. Before, an error would be logged, but it would be followed by a panic. Now the bucket in question is skipped.
 2023-08-30 17:49:37 -04:00
Miccah
522b2fab29
Add a cancel cause to job cancellation (#1728) 2023-08-30 12:00:44 -07:00
Miccah
c77c1172c8
Add the 'Cause' family of functions to the context wrapper library (#1725) 
Go 1.20 introduced `WithCancelCause`, `WithTimeoutCause`, and
`WithDeadlineCause` to allow adding a reason to context cancellations.
Adding it to our wrapper will allow us to use these features.
 2023-08-30 07:57:45 -07:00
ahrav
d6afca682b
remove fmt.Print (#1727) 2023-08-30 07:17:38 -07:00
ahrav
4dc5eb7912
Optimize read to max (#1714) 
* optimize the ReadToMax.

* add comment.

* remove dumb comment.

* update comment.

* fix test.

* lint.

* address comments.

* use limit reader.

* update equality check.

* update test.'

* use custom limit reader.

* address comments.

* revert fun.
 2023-08-29 17:31:40 -07:00
Miccah
7ba880f47a
Add AvailableCapacity method to SourceManager (#1665) 2023-08-29 12:36:44 -07:00
ahrav
2b1b1b5ad0
Add jobID to chunk. (#1721) 2023-08-29 12:02:30 -07:00