ahrav
383f8a1f67
[chore] - reduce test time ( #2321 )
...
* reduce test time
* remove commented out code
2024-01-22 09:40:32 -08:00
ahrav
d3d551d24e
[chore] - Update Chunk struct comment ( #2317 )
...
* update comment to include information on the importance of struct ordering
* more cute tricks
* remove cute tricks
2024-01-20 13:31:27 -08:00
ahrav
8380e1713e
save 8 bytes per chunk ( #2310 )
2024-01-18 13:20:06 -08:00
ahrav
b0fd951652
[chore] - Add regex and keyword for api_org tokens ( #2240 )
...
* Add regex and keyword for api_org tokens.
* handle org token auth struct
* update keywords
2024-01-16 15:25:26 -08:00
Miccah
c5af979aee
Assume unauthenticated github scans have public visibility ( #2308 )
2024-01-16 14:57:06 -08:00
ahrav
a1dc660f41
[fixup ] - Allow ssh cloning with AWS Code Commit ( #2307 )
2024-01-16 11:55:17 -08:00
roxanne-tampus
d6419a8ab2
added azure protos ( #2304 )
2024-01-15 06:59:47 -08:00
Dustin Decker
2596331bca
Disable recently added postgres detector because it it too sensitive ( #2303 )
2024-01-13 19:52:57 -08:00
Cody Rose
b03cc30263
Individuate archive tests #2293
2024-01-12 09:39:48 -05:00
ahrav
651beff492
[feat] - Allow for the use of include/exclude path files for filesystem scans ( #2297 )
...
* Allow for the use of include/exclude path files for filesystem scans
* remove oopsie
2024-01-11 15:41:50 -08:00
ahrav
9408425cc6
[chore] - small updates ( #2288 )
...
* small updates
* fix logic
* simplify fxn
* remove errors
* use strings.EqualFold
2024-01-11 14:27:10 -08:00
ahrav
677238c96c
Extend memory cache ( #2275 )
...
* Extend memory cache to allow for configuring custom expiration and purge interval
* use any for value type
* fix test
* fix test
* address comments
* address
* make new construct more clear
* reduce duplication
* fix test
2024-01-11 08:20:37 -08:00
dylanTruffle
3b4518cbab
adding postgres detector ( #2108 )
...
* adding postgres detector
---------
Co-authored-by: Chair <chair@Chairs-MacBook-Pro.local>
Co-authored-by: ahmed <ahmed.zahran@trufflesec.com>
2024-01-10 16:19:45 -05:00
ahrav
fb927e011b
update test ( #2283 )
2024-01-10 09:56:21 -08:00
Richard Gomez
04bf244f38
fix(signable): ignore common false positives ( #2230 )
2024-01-09 17:53:25 -08:00
Richard Gomez
38f36475de
fix(parseur): ignore false positives ( #2229 )
2024-01-09 17:50:30 -08:00
ahrav
b2c0d50ba8
[chore] - update docs for pre-commit ( #2280 )
...
* update docs for pre-commit
* fix spacing
2024-01-09 15:41:27 -08:00
David
24a09bc37d
1833 Fix syslog udp ( #1835 )
...
* # 1183 - Update syslog UDP listener deadline
* #1833 - Update syslog UDP listener deadline v2
* #1833 - Update syslog UDP listener deadline v3
2024-01-08 09:59:48 -08:00
Cody Rose
fd8e5e9df6
Wrap temp deletion err #2277
2024-01-08 12:48:58 -05:00
Shivasurya
e5289ac8e4
Updated trufflehog sourcegraph secret format ( #2254 )
2024-01-06 08:48:41 -08:00
NIKHIL PANWAR
7365dba9c9
Update stripe detector regex ( #2261 )
2024-01-06 08:43:39 -08:00
Miccah
5e78ad402e
[chore] Add test to check all versioned detectors are non-zero ( #2272 )
2024-01-05 12:06:59 -08:00
Richard Gomez
241e153dfb
fix(gitparse): handle fromFileLine edge case ( #2206 )
2024-01-04 14:53:08 -08:00
Dustin Decker
1cc41e2c75
Use directory iterator instead of walkdir ( #2260 )
...
* Use directory iterator instead of walkdir
* pr comments
2023-12-22 22:45:27 -08:00
Bill Rich
78d8dd3abf
Add handlerOpts back ( #2258 )
2023-12-22 12:11:59 -08:00
Bill Rich
ceff786db4
Skip all binaries ( #2256 )
...
* Skip all binaries
* Remove noop
* Drop handlerOpts
2023-12-22 12:01:07 -08:00
Dustin Decker
7d93adc1d0
Add skip archive support ( #2257 )
2023-12-22 11:55:23 -08:00
Dustin Decker
f699f60e89
use walk dir for tmp cleanup ( #2255 )
2023-12-22 07:52:58 -08:00
ahrav
39f0310f1f
[fixup] - Refactor to Pass Reader for Binary Diffs and Archived Data; Optimize /tmp Directory Cleanup ( #2253 )
2023-12-22 07:41:54 -08:00
Cody Rose
9c8674777c
Dedupe some source log keys ( #2250 )
...
The source manager attaches some context keys, but in certain circumstances, they're already present, resulting in duplicate keys. This PR changes the attachment to be conditional. It also adds some new log messages to track source startup progress.
2023-12-21 10:11:52 -08:00
ahrav
07ae9ec870
Fix goroutine leak ( #2251 )
2023-12-20 21:09:05 -08:00
ahrav
28212c9a82
[chore] - lower logging level ( #2249 )
...
* lower logging level
* move pid len check outside loop
2023-12-20 09:50:32 -08:00
ahrav
4a66dddd81
[chore] - add additional binary extensions to skip ( #2235 )
...
* add additional binary extensions to skip
* remove whl
2023-12-20 06:57:23 -08:00
ahrav
f5d0f3f366
use snake_case for naming ( #2238 )
2023-12-20 06:57:00 -08:00
ahrav
5848f5b8d6
[bug] - Bug archive handler memory leak ( #2247 )
2023-12-20 06:16:58 -08:00
Dustin Decker
6b90a96ca0
Add missing import ( #2246 )
2023-12-19 16:40:31 -08:00
Richard Gomez
69d5e0c993
fix(snowflake): avoid extraneous attempts ( #2057 )
...
Co-authored-by: Dustin Decker <dustin@trufflesec.com>
2023-12-19 16:04:08 -08:00
Richard Gomez
97afd570ae
feat(github): update extradata ( #2219 )
2023-12-19 16:01:07 -08:00
Mike Vanbuskirk
adba91da50
Adds basic if/else check if pid slice is empty ( #2244 )
...
* adds basic if/else check for pid slice
* use continue instead of if
2023-12-18 22:24:49 -05:00
ahrav
64c7365364
add secretID to chunk ( #2242 )
2023-12-18 15:27:49 -08:00
Richard Gomez
eeb4dbd304
feat(shortcut): use tri-state verification ( #2211 )
2023-12-17 15:47:14 -08:00
Richard Gomez
ded8e459bd
feat(huggingface): enhance extradata ( #2222 )
2023-12-17 14:29:45 -08:00
Richard Gomez
69a70a3374
fix(myfreshworks): check for valid JSON ( #2212 )
2023-12-17 10:26:38 -08:00
Miccah
88281bc354
[chore] Add skip_binaries field to AzureRepos proto message ( #2232 )
...
* [chore] Add skip_binaries field to AzureRepos proto message
* Make protos
2023-12-15 12:23:46 -08:00
ahrav
5c6ce693c1
[feat] - Make skipping binaries configurable ( #2226 )
...
* Make skipping binaries configurable
* remove ioutil
* fix
* address comments
* address comments
* use multi-reader
* remove print
* use const
* fix test
* fix my stupidness
2023-12-15 11:46:27 -08:00
Miccah
78b5a95342
[chore] Prevent panic when ChunkError has a nil Unit ( #2227 )
2023-12-15 11:11:28 -08:00
Richard Gomez
b3040b1227
fix(github): remove unused 'members' var ( #2202 )
2023-12-14 11:53:24 -08:00
Miccah
f6bbc59bf6
Check for SourceUnit support dynamically in the SourceManager ( #2205 )
...
* Check for SourceUnit support dynamically in the SourceManager
* Only call the function if we can use source units
2023-12-14 11:48:15 -08:00
Richard Gomez
06b137fd93
fix(gitlab): check for valid JSON ( #2218 )
2023-12-14 11:22:06 -08:00
ahrav
d8cb65833c
Avoid reading decompressed data into memory ( #2196 )
2023-12-14 11:00:11 -08:00
Richard Gomez
e72fdb62e4
fix(gitparse): don't trim filename ( #2201 )
2023-12-14 08:29:46 -08:00
Richard Gomez
22ae6a7a8f
fix(giturl): encode '%' in path ( #2214 )
2023-12-14 08:06:09 -08:00
Miccah
4db20e29f8
Update metabase verification to check for a valid JSON response ( #2210 )
...
* Update metabase verification to check for a valid JSON response
* added test tokens + cleanup
---------
Co-authored-by: ahmed <ahmed.zahran@trufflesec.com>
2023-12-13 12:18:56 -08:00
Miccah
84b7461796
[chore] Remove unnecessary string conversion in tefter detector ( #2209 )
2023-12-13 11:39:16 -08:00
āh̳̕mͭͭͨͩ̐e̘ͬ́͋ͬ̊̓͂d
6987507534
fix and refactor browserstack detector ( #2208 )
...
* fix and refactor browserstack detector
2023-12-12 16:14:31 -05:00
āh̳̕mͭͭͨͩ̐e̘ͬ́͋ͬ̊̓͂d
5e3ea1a8f2
Fix azurestorage detector ( #2207 )
...
* bugfix + cleanup - update azurestorage detector raw string to use key instead of id
2023-12-12 16:07:09 -05:00
Miccah
9f6a47da3f
[chore] Remove omitempty tags on JobProgressMetrics and UnitMetrics ( #2204 )
2023-12-12 10:02:56 -08:00
Mike Vanbuskirk
53f060a08e
Add disk buffer tempfile cleanup ( #2130 )
...
* add tempfile creation
- break PID retrieval into sep. function
* add tmpfile cleanup func
* add file cleanup to main cleanup func
* refactor file logic to only return name string
* add temp buffer naming to gcs
* add temp buffer naming to s3
* add temp buffer naming to filesystem
* add temp buffer naming to git
* consolidate cleanup functions
- have single function handle both files and dirs
- remove interface(not needed with a single func implementation)
- change calls to `New(...)` to reflect config implementation
- simplify automation in main.go
- update disk-buffer-reader dependency
* integrate changes from pr #2133
* merge main
* checkout from main to revert conflict issues
* re-add buffer logic to git
* interface no longer needed
* move string format to global const
---------
Co-authored-by: Ahrav Dutta <ahrav.dutta@trufflesec.com>
2023-12-11 18:31:50 -05:00
Cody Rose
405f356071
Use bad json in slackwebhooks ( #2193 )
...
* add rotation guides to SlackWebhook tests
* begin cleaning up tests
* have slack webhook detector use malformed json
* update test secrets
---------
Co-authored-by: Ahrav Dutta <ahrav.dutta@trufflesec.com>
2023-12-11 15:04:55 -08:00
ahrav
61c7d52a43
[bug] - close file after reading ( #2203 )
...
* close file after reading
* inline return
2023-12-11 15:04:30 -08:00
Richard Gomez
d1a2d9e832
chore: propagate log context to handlers ( #2191 )
2023-12-10 10:30:11 -08:00
Richard Gomez
6c5fc2f212
feat(privatekey): run checks concurrently ( #2139 )
2023-12-10 10:11:17 -08:00
ahrav
331336dc0a
[fixup] - skip files in the archive handler ( #2195 )
2023-12-08 20:23:32 -08:00
ahrav
2728e514d2
move logic to main Chunks method ( #2194 )
2023-12-08 14:51:24 -08:00
ahrav
2a7813929b
add metrics for gitlab ( #2190 )
2023-12-08 09:50:09 -08:00
ahrav
4b31b39d6b
[chore] - Refactor common code into a separate function ( #2179 )
...
* Refactor common code into a separate function
* rename vars
* make sure to set the scanOptions fields
* address comments
2023-12-08 08:44:35 -08:00
Cody Rose
ee6923a241
Remove java archives from ignored extensions ( #2188 )
...
A previous commit (5d0196957f
2023-12-07 15:19:56 -05:00
ahrav
b75991850a
[chore] - Compile regex once ( #2176 )
...
* move regex compilation out of the fxn
* missed a spot
* merge main
2023-12-07 07:26:27 -08:00
ahrav
f772fd8b44
update regex ( #2184 )
2023-12-06 17:04:38 -08:00
Dustin Decker
3167dde8a1
Deprecate some detectors ( #2186 )
2023-12-06 16:57:55 -08:00
ahrav
0595a3baac
allow targets for the source manager ( #2182 )
...
* allow targets to the source manager
* use targets
2023-12-06 16:38:35 -08:00
ahrav
c6e9b8ff64
use https for verification endpoints ( #2185 )
2023-12-06 16:06:04 -08:00
ahrav
e6bc7f4451
remove unnecessary Git cmd check ( #2175 )
2023-12-06 13:38:34 -08:00
ahrav
cb81f7d11a
[feat] - Remove go-git dependency ( #2174 )
...
* remove use of go-git for binary files
* fix it
* use limit reader
* fix comment
* fix test
* address comments
* address comments
* address comments
2023-12-06 13:38:01 -08:00
ahrav
990274b596
Skip trying to determine MIME type for directories ( #2178 )
2023-12-06 12:00:18 -08:00
dylanTruffle
96aa50d119
fixing how to rotate URL ( #2183 )
2023-12-06 11:59:21 -08:00
āh̳̕mͭͭͨͩ̐e̘ͬ́͋ͬ̊̓͂d
11394ea318
[thog-1548] add auto redaction for verification errors ( #2106 )
...
* Updating VerificationError to have auto redaction logic
* find/replace error
2023-12-05 08:57:52 -05:00
ahrav
13da76d357
skip files we can't scan ( #2170 )
2023-12-04 13:37:11 -08:00
ahrav
996a11dcc0
[chore] - remove deprecated types ( #2168 )
...
* remove deprecated types
* missed one
2023-12-04 13:23:58 -08:00
Cody Rose
5d0196957f
Ignore images and binaries ( #2162 )
...
This PR expands the list of excluded file extensions to contain images and other binary files. These files can technically contain secrets, but need decoding to properly be handled, and we don't have any such decoding yet. Down the road if we want to add it we can.
2023-12-04 13:25:29 -05:00
ahrav
37d9e5eedf
[chore] - Increase pagination limit ( #2154 )
...
* increae pagination limit
* rename
2023-12-04 10:14:46 -08:00
Dustin Decker
07dc123840
update forager types ( #2159 )
2023-12-03 13:16:16 -08:00
ahrav
c34efc3cf9
make empty slice delcration consistent ( #2144 )
2023-12-01 11:03:44 -08:00
ahrav
279f915799
[chore] - fix error comparisons ( #2142 )
...
* fix error comparisons
* fix imports
2023-12-01 08:32:41 -08:00
ahrav
52ffab1034
[chore] - fix import name clashes ( #2143 )
...
* fix import name clashes
* fix missing var
2023-12-01 06:53:15 -08:00
Dustin Decker
a367f9ce34
Fix azure panic when invalid URL is constructed ( #2137 )
2023-11-30 11:33:04 -08:00
ahrav
8880c2e005
fixup cleantemp ( #2136 )
2023-11-30 09:39:30 -08:00
Miccah
e498c80b3d
Fix nil pointer dereference when checking if a unit IsFinished ( #2135 )
2023-11-29 14:19:31 -08:00
Miccah
7ecd43ab1e
[chore] Minor cleanup of source_manager.go ( #2134 )
2023-11-29 11:08:25 -08:00
Dustin Decker
363ccab316
Simplify temp dir cleaning ( #2133 )
...
* Simplify temp dir cleaning
* rename vars
* add test
* update test
2023-11-28 16:42:17 -08:00
Dustin Decker
ede0c39589
Add new auth method to source ( #2132 )
2023-11-28 10:58:11 -08:00
Zachary Rice
d552222385
add extradata nil check and use make ( #2129 )
...
* add extradata nil check and use make
* remove some lines
2023-11-28 09:45:37 -06:00
Miccah
78219a27b3
Call Finish in SourceManager after the semaphore is released ( #2121 )
2023-11-24 13:22:08 -08:00
Richard Gomez
024aa056b9
chore(github): add a newline between titles and bodies ( #2124 )
2023-11-23 16:14:28 -08:00
Richard Gomez
1f502fd42c
feat(github): scan issue & pr titles ( #1899 )
2023-11-22 19:15:27 -08:00
ahrav
0e6e1dce3f
use camelcase var names ( #2123 )
2023-11-22 09:09:04 -08:00
Oleksandr Redko
7d10e2540e
Remove unused functions ( #2122 )
2023-11-22 06:58:16 -08:00
Dustin Decker
a7d330a2a5
import missing detectors ( #2119 )
2023-11-21 10:30:11 -08:00
Dustin Decker
75e869faff
Fix forks and repos counter, add metric for orgs enumerated ( #2118 )
2023-11-21 08:52:33 -08:00
Richard Gomez
62c628fb52
feat(telegram): add username to extradata ( #2100 )
2023-11-20 14:00:09 -08:00
Dustin Decker
9e88cdf625
add extra data to github detector ( #1909 )
...
* add extra data to github detector
* Add verification error
2023-11-20 13:55:16 -08:00
joeleonjr
cd9c1ae186
fixed gist direct link generation ( #2115 )
...
* fixed gist direct link generation
* added two test cases for gist link generation
2023-11-20 13:41:19 -05:00
Zachary Rice
d69de658b2
fix nil map assignment ( #2117 )
2023-11-20 11:13:09 -06:00
Miccah
39a603d2dc
[chore] Add JSON tags to job metrics ( #2114 )
2023-11-16 17:08:33 -08:00
ahrav
d334b3075e
move all Git setup into Init method ( #2105 )
...
* add proto fields for git
* add uri to proto
* move all git setup into Init method
* fix logic for when to use repoPath
2023-11-16 13:59:53 -08:00
ahrav
fd33198ad8
add proto fields for Git ( #2104 )
...
* add proto fields for git
* add uri to proto
* add comment
2023-11-16 13:52:38 -08:00
joeleonjr
b2042e4e03
extract AWS account number from ID without verification ( #2091 )
...
* added GetAccountNumFromAWSID function
* refacted aws func, moved to common
2023-11-16 11:45:47 -05:00
āh̳̕mͭͭͨͩ̐e̘ͬ́͋ͬ̊̓͂d
737d6b764d
Adding Sumo Logic how to rotate ( #2103 )
2023-11-09 12:48:08 -05:00
ahrav
76a0468580
update protos so we can use the git source for CI ( #2102 )
2023-11-08 09:07:29 -08:00
Damanpreet Singh
d066a3fa78
Detector-Competition-Feat: Added Replicate API token detector ( #2021 )
...
* Detector-Competition-Feat: Added Replicate API token detector
* fix fullstory
---------
Co-authored-by: Zachary Rice <zachary.rice@trufflesec.com>
2023-11-07 12:16:39 -06:00
Damanpreet Singh
bcde7856c3
Detector-Competition-Feat: Added Ngrok API token detector ( #2024 )
...
Co-authored-by: Zachary Rice <zachary.rice@trufflesec.com>
2023-11-07 09:28:05 -06:00
Ankush Goel
1b93c0545c
Competition-Detector-New:added v2 version for fullstory ( #2067 )
...
* added v2 version for fullstory
* added versioner to the v1 fullstory detector
2023-11-07 08:55:06 -06:00
Miccah
8e3f6e98dc
Add support for user:pass@host to postgres JDBC detector ( #2089 )
...
* Add support for user:pass@host to postgres JDBC detector
* Remove ineffectual assignment
2023-11-06 17:17:37 -08:00
Corben Leo
1094190ff5
Detector-Competition-Feat: Add Overloop detector ( #2080 )
...
* Detector-Competition-Feat: Add Overloop detector
* add protos and to defaults.go
---------
Co-authored-by: Zachary Rice <zachary.rice@trufflesec.com>
2023-11-06 16:43:31 -06:00
Damanpreet Singh
da59b72735
Detector-Competition-Feat: Added Request.Finance API token detector ( #2020 )
...
Co-authored-by: Zachary Rice <zachary.rice@trufflesec.com>
2023-11-06 16:13:33 -06:00
Ankush Goel
703e158648
Detector-Competition-New : created grafana service account detector ( #1960 )
...
* created grafana service account detector
* add import
---------
Co-authored-by: Zachary Rice <zachary.rice@trufflesec.com>
2023-11-06 15:41:37 -06:00
Ankush Goel
b2d541e0ea
Detector-Competition-Fix: fixed zulipchat detector ( #1990 )
...
* fixed zulipchat detector
* fixed testing scenarios
* fixed test detector
* fixed test
* made chunking keyword from zulipchat to zulip
* fixed email regex
* fixed domain regex
2023-11-06 12:22:47 -06:00
Ankush Goel
6259b179b9
Grafana ( #2096 )
...
* Created Grafana Cloud API Key detector
* made the regex more bounded
* added boundary to regex
2023-11-06 11:13:06 -06:00
Ankush Goel
aabfec4cdf
Competition-Detector-New: added eventbrite detector ( #2072 )
...
* added eventbrite detector
* added packagename to defaults.go
---------
Co-authored-by: Zachary Rice <zachary.rice@trufflesec.com>
2023-11-03 16:42:13 -05:00
Ankush Goel
1371512ff3
logz.io detector ( #2076 )
...
Co-authored-by: Zachary Rice <zachary.rice@trufflesec.com>
2023-11-03 16:32:35 -05:00
Ankush Goel
06b5fc25ef
Coda Detector ( #2075 )
...
Co-authored-by: Zachary Rice <zachary.rice@trufflesec.com>
2023-11-03 15:50:05 -05:00
Zachary Rice
50a3a82cbb
fix ( #2094 )
2023-11-03 12:56:12 -05:00
Corben Leo
de8889b406
Detector-Competition-Fix: Fix LiveAgent Detector & Verifier ( #2001 )
...
* Detector-Competition-Fix: Fix LiveAgent Detector & Verifier
* update regex
2023-11-03 12:28:20 -05:00
dylanTruffle
0b90265802
pulling short lived AWS keys into their own thing, fixes #1224 ( #2088 )
...
* pulling short lived AWS keys into their own thing, fixes #1224
* Update awssessionkey.go
* fmt
---------
Co-authored-by: Chair <chair@Chairs-MacBook-Pro.local>
Co-authored-by: Zachary Rice <zachary.rice@trufflesec.com>
2023-11-03 11:58:49 -05:00
Cody Rose
7a156330b5
Support multiple detectors per match ( #2065 )
...
#1711 inadvertently removed the ability to match multiple custom detectors, or multiple detectors of the same type but different version, to a given keyword. (#2060 re-added support for multiple versions of detectors globally, and #2064 re-added support for multiple custom detectors globally, but neither fixed trufflehog's inability to support multiple such detectors for a given keyword match.) This PR re-adds the removed functionality (and narrows the AhoCorasickCore interface in the process.)
2023-11-03 12:26:18 -04:00
Miccah
600903f391
[chore] Speedup IsKnownFalsePositive using sets ( #2090 )
...
Also check that the match is a valid UTF-8 string.
2023-11-03 08:45:00 -07:00
Corben Leo
3b9ecaa704
Detector-Competition-Fix: Fix ScraperSite (deprecated) ( #2074 )
...
Co-authored-by: āh̳̕mͭͭͨͩ̐e̘ͬ́͋ͬ̊̓͂d <13666360+0x1@users.noreply.github.com>
2023-11-03 11:15:53 -04:00
Corben Leo
41e9cc59e2
Detector-Competition-Fix: Fix PassBase (acquired, deprecated) ( #2079 )
2023-11-03 08:59:32 -05:00
Ankush Goel
b95ed3b41a
Detector-Competition-New - Created Grafana Cloud API Key detector ( #1959 )
...
* Created Grafana Cloud API Key detector
* made the regex more bounded
* added boundary to regex
2023-11-03 09:25:54 -04:00
Corben Leo
9e52e3e86f
Detector-Competition-Fix: Fix/Deprecate Prospect.io ( #2081 )
...
* Detector-Competition-Feat: Fix/Deprecate Prospect.io
* Detector-Competition-Fix: fix defaults.go
2023-11-03 07:04:42 -05:00
joeleonjr
a1d74cd887
added resource type mapping to extraData in AWS ( #2087 )
...
* added resource type mapping to extraData in AWS
* updating aws regex + logic for resource type
2023-11-02 17:03:03 -04:00
Corben Leo
b5cc6c196c
Detector-Competition-Fix: Fix FakeJSON (deprecated) ( #2073 )
2023-11-02 15:43:49 -05:00
Ankush Goel
ab896890b4
fixed helpscout detector regex and verifier ( #2056 )
2023-11-02 14:20:26 -05:00
Ankush Goel
965a274de9
Detector-Competition-Fix: fixed regex for databricks domain and fixed tests ( #1965 )
...
* fixed regex for domain and fixed tests
* fixed regex
* fixed an issue with regex subgrouping
* made recommended changes
* made recommended changed
* fixed RawV2
2023-11-02 11:26:31 -05:00
Ankush Goel
b6469f23ac
modified regex ( #2033 )
2023-11-02 11:24:37 -05:00
dylanTruffle
4106ce7bf0
Detector-Competition-Feat: Adding Azure Container Registry Password Detector ( #1958 )
...
* implementing azure container registry password detector
* Fixing boundry feedback
* whoops
* update verification code
* fix regex
---------
Co-authored-by: Chair <chair@Chairs-MacBook-Pro.local>
Co-authored-by: Zachary Rice <zachary.rice@trufflesec.com>
2023-11-02 11:17:01 -05:00
Corben Leo
07f6c84aa4
Detector-Competition-Fix: Fix SentimentInvestor (deprecated) ( #2078 )
2023-11-01 11:54:40 -05:00
Miccah
9d6bc8c504
Refactor git source to support scanning units ( #2083 )
2023-11-01 09:52:58 -07:00
Miccah
52600a897a
[chore] Replace chunks channel with ChunkReporter in git based sources ( #2082 )
...
ChunkReporter is more flexible and will allow code reuse for unit
chunking. ChanReporter was added as a way to maintain the original
channel functionality, so this PR should not alter existing behavior.
2023-11-01 09:22:44 -07:00
ahrav
d55cb56db4
update comment ( #2084 )
...
update Cache.Contents() comment
2023-11-01 07:36:22 -07:00
Cody Rose
7197e4b3f1
use rawv2 for pubnubpublish ( #2062 )
...
We're seeing secrets of this type flap between verified and unverified, which is expected behavior for multipart secrets without RawV2 defined. This PR adds RawV2 for secrets of this type.
2023-11-01 10:14:28 -04:00
ahrav
95e0090bc2
[chore] - correctly handle input shorter than 512 bytes ( #2077 )
...
* correctly handle input shorter than 512 bytes
* add tests
* reorder tests
* add another test case
* update test
* address comment
2023-10-31 16:42:42 -07:00
āh̳̕mͭͭͨͩ̐e̘ͬ́͋ͬ̊̓͂d
74a56de831
update braintreepayments detector to tri-state verification ( #1834 )
...
* update braintreepayments detector to tri-state verification
* Update pkg/detectors/braintreepayments/braintreepayments.go
Co-authored-by: Zachary Rice <zachary.rice@trufflesec.com>
* small nits
* small nits
---------
Co-authored-by: Zachary Rice <zachary.rice@trufflesec.com>
2023-10-31 16:13:48 -04:00
dylanTruffle
8bac2b15ba
Detector-Competition-Feat: Adding Azure Batch keys ( #1956 )
...
* adding azure batch
* fmt
* fix lint
---------
Co-authored-by: Chair <chair@Chairs-MacBook-Pro.local>
Co-authored-by: Zachary Rice <zachary.rice@trufflesec.com>
2023-10-31 10:49:04 -05:00
dylanTruffle
499cb64546
Detector-Competition-Fix: Fix redis to now support SSL, and look for azure redis connection strings ( #1957 )
...
* adding azure redis, and fixing the old detector to support ssl too
* fix?
* other way
---------
Co-authored-by: Chair <chair@Chairs-MacBook-Pro.local>
Co-authored-by: Zachary Rice <zachary.rice@trufflesec.com>
2023-10-31 10:17:55 -05:00
Corben Leo
a4fd17c9d1
Detector-Competition-Fix: Fix AppFollow Detection & Verification ( #1933 )
...
* Detector-Competition-Fix: Fix AppFollow Detection & Verification
* fix(regex): update jwt regex for appfollow
2023-10-31 09:43:20 -05:00
ahrav
a9b056de0a
Centralize logic for checking archive extraction tools ( #2063 )
...
* Centralize logic for checking archive extraction tools
* simplify
2023-10-30 20:14:51 -07:00
Miccah
57203a56cd
[chore] Fix SourceManager flaky test ( #2059 )
...
* [chore] Fix SourceManager flaky test
Sorting by EndTime is not deterministic, however sorting by StartTime
should be. StartTime is set in a goroutine that's limited by
WithConcurrentUnits, so it should happen in order that the units are
received.
* Sort by unit ID
2023-10-30 19:16:55 -07:00
Cody Rose
e58a2913ea
Support multiple custom detectors ( #2064 )
...
#1711 accidentally removed the ability to support multiple custom detectors. This PR partially adds back this capability: Multiple custom detectors are now supported overall, but only one custom detector can be returned for a given keyword match.
2023-10-30 18:17:17 -04:00
Corben Leo
de4a14b3f9
Detector-Competition-Fix: Fix SalesBlink Detection & Verification ( #1950 )
2023-10-30 16:10:24 -05:00
Damanpreet Singh
244ba3a214
Detector-Competition-Fix: Update formio regex to match Jwt token ( #1935 )
2023-10-30 16:08:19 -05:00
Corben Leo
6a15cd8f30
Detector-Competition-Fix: Fix Bitcoin Average detector ( #1929 )
2023-10-30 16:02:30 -05:00
Corben Leo
509fc6c0eb
Detector-Competition-Fix: Fix currencycloud.com API key ( #1917 )
...
* Detector-Competition-Fix: Fix currencycloud.com API environment
* Detector-Competition-Fix: Fix currencycloud.com API environment
* fix(env): update environment
2023-10-30 15:56:30 -05:00
Cody Rose
45059864f8
Re-add detector version ( #2060 )
...
#2010 mistakenly removed detector version tracking from the Aho Corasick wrapper. This PR re-adds it.
2023-10-30 15:34:33 -04:00
Dustin Decker
05fae156e1
Add TravisCI source ( #1877 )
...
* Add TravisCI source
* update test to use sourcestest
* Remove jobPage loop
ListByBuild does not support pagination, so this was infinitely
repeating. https://developer.travis-ci.com/resource/jobs#find
* Continue chunking on error
* review updates
* update readme
---------
Co-authored-by: Miccah Castorina <m.castorina93@gmail.com>
2023-10-30 07:28:25 -07:00
Cody Rose
876a55821b
Remove verify flag from Aho-Corasick core ( #2010 )
...
The Aho-Corasick wrapper we have tracks information about whether verification should be enabled on an individual detector basis, but that functionality isn't related to the matching functionality of Aho-Corasick, and including it complicates the implementation. This PR removes it to simplify some things.
This PR removes some code that supported a potential future implementation of detector-specific verification settings, but that feature has not actually been implemented yet, so there's no loss of functionality. If we want that feature we can add it back on top of this in a more separated way.
2023-10-30 09:52:51 -04:00
Ankush Goel
2a66d4117a
adding 'token' keyword to regex for github_old ( #2037 )
2023-10-29 20:45:35 -07:00
Damanpreet Singh
7a9332152a
Detector-Competition-Feat: Added Reply.io API token detector ( #2019 )
...
Co-authored-by: Ahrav Dutta <ahrav.dutta@trufflesec.com>
2023-10-29 17:57:36 -07:00
Damanpreet Singh
0068ec54f2
Detector-Competition-Feat: Added Stripo API token detector ( #2018 )
...
* Detector-Competition-Feat: Added Stripo API token detector
* adjust regex
---------
Co-authored-by: Ahrav Dutta <ahrav.dutta@trufflesec.com>
2023-10-29 17:26:14 -07:00
Richard Gomez
0427985ebe
feat: deno deploy detector ( #2040 )
...
Co-authored-by: Ahrav Dutta <ahrav.dutta@trufflesec.com>
2023-10-29 16:58:00 -07:00
Damanpreet Singh
3ffc0dfd22
Detector-Competition-Feat: Added Budibase API token detector ( #2016 )
2023-10-29 10:12:45 -07:00
Damanpreet Singh
52b3c99868
Detector-Competition-Feat: Added LemonSqueezy API token detector ( #2017 )
...
* Detector-Competition-Feat: Added LemonSqueezy API token detector
* fix regex
---------
Co-authored-by: Ahrav Dutta <ahrav.dutta@trufflesec.com>
2023-10-28 15:03:14 -07:00
Richard Gomez
96b25150d0
Add Coinbase Wallet-as-a-Service detector ( #1895 )
...
* feat(coinbase): basic Wallet-as-a-Service detector
* update test
---------
Co-authored-by: Dustin Decker <dustin@trufflesec.com>
2023-10-27 10:32:36 -07:00
Damanpreet Singh
eb0c0fa99f
Detector-Competition-Feat: Add Metabase Session Secret Detector ( #1902 )
...
Co-authored-by: Ahrav Dutta <ahrav.dutta@trufflesec.com>
2023-10-26 20:17:41 -07:00
Damanpreet Singh
bf6ece39ca
Detector-Competition-Feat: Added AppOptics API token detector ( #1989 )
...
Co-authored-by: Ahrav Dutta <ahrav.dutta@trufflesec.com>
2023-10-26 20:06:30 -07:00
Damanpreet Singh
4d0a40d2f3
Detector-Competition-Feat: Added ZeroTier API token detector ( #1988 )
...
Co-authored-by: Ahrav Dutta <ahrav.dutta@trufflesec.com>
2023-10-26 19:55:58 -07:00
Damanpreet Singh
f1a75395e8
Detector-Competition-Feat: Added BetterStack API token detector ( #1987 )
...
Co-authored-by: Ahrav Dutta <ahrav.dutta@trufflesec.com>
2023-10-26 19:46:56 -07:00
Corben Leo
98d2922bee
Detector-Competition-Fix: Fix SurveyBot Verification ( #1948 )
2023-10-26 12:10:00 -05:00
Bill Rich
00a00ef651
Fix binary handling ( #1999 )
2023-10-26 10:07:02 -07:00
Mike Vanbuskirk
4636dc08f6
Add temp directory management ( #1878 )
...
* adds func to get scannerPIDs
* add cleanup and call to get pids
* move pid handling to git module
* remove PID logic from main
* refactor testing code to handle different exec name
* cleanup linting errors
* add better logging, fix dir if clause
* some PR fixups
* mod fixup
* add interfaces for helper funcs
* refactor cleanup into main, getPID into git
* lint and test fixups, remove fail on n<2 pids
* simplify pid sorting
* use filepath.Join
* use Args[0] for exec name, fix logger
* formatting fixup
* move functionality into cleantemp pkg
* go mod fixup
* remove redundant testing comment
* fix go.sum issues
* add 15m ticker loop for cleanup
* enclose ticker in function for goroutine defer
fix cleantemp interface
* make time more readable
* add check for non-local Trufflehog PIDs
* allow deletion even if no non-local pids found
* bundle intial cleanup into runCleanup func
* add explicit regex check for tempdir format
2023-10-26 12:28:56 -04:00
Corben Leo
8505d24d7d
Detector-Competition-Fix: Fix/Remove Flowdock detector ( #2004 )
...
* Detector-Competition-Fix: Fix/Remove Flowdock detector
---------
Co-authored-by: āh̳̕mͭͭͨͩ̐e̘ͬ́͋ͬ̊̓͂d <13666360+0x1@users.noreply.github.com>
Co-authored-by: ahmed <ahmed.zahran@trufflesec.com>
2023-10-26 09:35:13 -04:00
Corben Leo
b776f9c122
Detector-Competition-Fix: Fix/Remove Happi Detection & Verification ( #2003 )
...
* Detector-Competition-Fix: Fix/Remove Happi Detection & Verification
---------
Co-authored-by: āh̳̕mͭͭͨͩ̐e̘ͬ́͋ͬ̊̓͂d <13666360+0x1@users.noreply.github.com>
Co-authored-by: ahmed <ahmed.zahran@trufflesec.com>
2023-10-26 09:20:53 -04:00
Corben Leo
6914dacde3
Detector-Competition-Fix: Fix/Remove DataFire, API retired ( #1995 )
...
* Detector-Competition-Fix: Fix/Remove DataFire, API retired
* Detector-Competition-Fix: Depreciate Datafire Proto
* make protos for deprecating datafire
---------
Co-authored-by: āh̳̕mͭͭͨͩ̐e̘ͬ́͋ͬ̊̓͂d <13666360+0x1@users.noreply.github.com>
Co-authored-by: ahmed <ahmed.zahran@trufflesec.com>
2023-10-25 21:51:54 -04:00
Corben Leo
f7960265ea
Detector-Competition-Fix: Fix/Remove QuickMetrics (shutdown) ( #1997 )
...
* Detector-Competition-Fix: Fix/Remove QuickMetrics (shutdown)
* Detector-Competition-Fix: Fix/Remove QuickMetrics protos
* make protos for deprecating Blablabus (#2002 )
* make protos for deprecating quickmetrics
---------
Co-authored-by: āh̳̕mͭͭͨͩ̐e̘ͬ́͋ͬ̊̓͂d <13666360+0x1@users.noreply.github.com>
Co-authored-by: ahmed <ahmed.zahran@trufflesec.com>
2023-10-25 20:05:26 -04:00
āh̳̕mͭͭͨͩ̐e̘ͬ́͋ͬ̊̓͂d
7af4591356
make protos for deprecating Blablabus ( #2002 )
2023-10-25 19:25:00 -04:00
Corben Leo
51b7fcc5d6
Detector-Competition-Fix: Fix/Remove BlaBlaBus, API retired ( #1996 )
...
* Detector-Competition-Fix: Fix/Remove BlaBlaBus, API retired
* Detector-Competition-Fix: Depreciate Blabus proto
2023-10-25 18:45:40 -04:00
Corben Leo
cebd92d79e
Detector-Competition-Fix: Depreciate Glitterly ( #2000 )
2023-10-25 18:08:50 -04:00
Corben Leo
386c54ecbe
Detector-Competition-Fix: Fix MeaningCloud Verification ( #1946 )
2023-10-25 14:52:36 -05:00
Corben Leo
cef05b8d5a
Detector-Competition-Fix: Fix ScreenshotAPI Verification ( #1949 )
...
* Detector-Competition-Fix: Fix ScreenshotAPI
* Detector-Competition-Fix: Fix ScreenshotAPI
2023-10-25 14:50:20 -05:00
Ankush Goel
6c35dcffa5
Detector-Competition-Fix : fixed monday.com regex ( #1961 )
...
* fixed monday.com regex
* updating test secrets to use detectors5
---------
Co-authored-by: ahmed <ahmed.zahran@trufflesec.com>
2023-10-25 12:50:07 -04:00
Damanpreet Singh
b2811bcf78
Detector-Competition-Feat: Added Vagrant Cloud Access Token Detector ( #1941 )
...
Co-authored-by: Ahrav Dutta <ahrav.dutta@trufflesec.com>
2023-10-25 11:03:45 -05:00
ahrav
4a9d93fe18
remove detector ( #1993 )
2023-10-25 07:44:58 -07:00
Corben Leo
c674f1fc34
Detector-Competition-Fix: Fix/Remove baseapi detector (no longer exists) ( #1992 )
2023-10-25 07:17:08 -07:00
Damanpreet Singh
2189dc9b0f
Detector-Competition-Feat: Added Portainer Detector ( #1936 )
...
Co-authored-by: Ahrav Dutta <ahrav.dutta@trufflesec.com>
2023-10-25 06:32:57 -07:00
ahrav
5901a92acb
deprecate scan_interval field ( #1984 )
...
* deprecate scan_interval field
* rename
* add link to docs
2023-10-25 06:18:27 -07:00
Damanpreet Singh
b2702b7839
Detector-Competition-Feat: Added OpenVPN API Detector ( #1940 )
2023-10-25 04:57:07 -07:00
Ankush Goel
84cb33ce3d
loggly detector ( #1782 )
...
* loggly detector
* fixed the loggly_test.go
* fixed the test file to pass the test
---------
Co-authored-by: dsingdev-rocketx <bughunter00@protonmail.com>
2023-10-24 20:06:47 -07:00
Damanpreet Singh
f467cf923c
Detector-Competition-Feat: Added PortainerToken Detector ( #1938 )
...
Co-authored-by: Ahrav Dutta <ahrav.dutta@trufflesec.com>
2023-10-24 13:48:40 -07:00
Damanpreet Singh
664c4884a6
fix #1751 : update facebookOauth Detector ( #1921 )
2023-10-24 11:07:52 -05:00
Damanpreet Singh
8184a62e24
fix: NewRelic Detector: fallback to EU Api for verification ( #1932 )
2023-10-24 11:02:39 -05:00
Corben Leo
7bc0b77374
Detector-Competition-Fix: Fix CloudSmith detection ( #1944 )
2023-10-24 11:01:27 -05:00
Cody Rose
e556bdd7b2
Revert "Fix off by one ( #1891 )" ( #1963 )
...
This reverts commit 7f534d0bb7
2023-10-24 08:40:44 -07:00
Bill Rich
c5efa870ff
Use latest dbr ( #1955 )
2023-10-24 07:52:49 -07:00
ahrav
0f845c8eee
export ShouldVerify ( #1962 )
2023-10-24 07:27:01 -07:00
ahrav
9ae114f92f
export struct ( #1954 )
2023-10-24 06:29:26 -07:00
Corben Leo
f3479194d2
Detector-Competition-Fix: Fix CodeClimate verification ( #1945 )
2023-10-23 20:19:02 -05:00
Damanpreet Singh
855aba2407
Detector-Competition-Feat: Add InstaMojo Payment Detector ( #1905 )
2023-10-23 16:58:25 -05:00
Corben Leo
893bb3548d
Detector-Competition-Fix: Fix SuperNotes API verification ( #1947 )
2023-10-23 16:29:55 -05:00
Miccah
0b16142d4f
Add UnitHook and NoopHook implementations ( #1930 )
...
* Add UnitHook and NoopHook implementations
The UnitHook tracks metrics per unit of a job, and emits them on a
channel once finished. It should work even if the Source does not
support source units.
* Refactor channel to use an LRU cache instead
An LRU cache has a more favorable failure mode than the channel. With
the channel, if the consumer stopped consuming metrics, scanning would
block. With the LRU cache, metrics will be dropped when space runs out
and a log message emitted.
2023-10-23 14:27:01 -07:00
Damanpreet Singh
b4753a60be
Detector-Competition-New: add IP2Location api key detector ( #1915 )
2023-10-23 13:51:14 -05:00
