Mirrors/trufflehog
2
0
Fork 0
mirror of https://github.com/trufflesecurity/trufflehog.git synced 2024-09-20 22:51:55 +00:00
Code Issues Projects Releases Packages Wiki Activity
2609 commits 123 branches 240 tags 83 MiB
Commit graph

127 commits

Author SHA1 Message Date
Miccah
6209a80ce1
[chore] Address more linter errors (#1134) 
* Address lint errors in detectors

* Update deprecated ioutil call
 2023-02-28 10:00:41 -06:00
Miccah
4efe5313f4
[chore] Address lint errors (#1133) 
* Update strings.Title to cases.Title

* Migrate go-genproto to google-cloud-go

See: https://github.com/googleapis/google-cloud-go/blob/main/migration.md

* Check error in test

* Check error from sem.Acquire

* Remove unused code
 2023-02-27 21:03:47 -06:00
Miccah
0ce72ccda3
[chore] Remove logrus from github source (#1086) 
* [chore] Remove logrus from github source

* Fix handleRateLimit test

* Fix tests
 2023-02-09 18:02:04 -06:00
Miccah
1f0fd91205
Skip repo and continue scanning when encountering an error (#1080) 2023-02-08 11:33:01 -06:00
Bill Rich
af6e3f8fdf
Pull gitparse config options out of pkg consts (#1072) 
* Pull gitparse config options out of pkg consts.

* Adjust naming
 2023-02-04 13:19:23 -08:00
Bill Rich
00ebb2ed64
Full git log when targeting base merge commit (#1044) 
* Full git log when targeting merge commits

* Full log is needed whenever base is specified.
 2023-01-26 09:17:54 -08:00
Yassine Ilmi
d720c0c0f3
Switch to retryableHttpClient for GitHub AuthN API Client + More Logs (#995) 
* Adding missing flags to Readme

* Use retryableHttpClient by default for GitHub

* Adding repoUrl for scanning time log

* Use WithField instead of WithFields

* Updating README with lasted --help output
 2023-01-09 09:21:56 -08:00
ahrav
009756dce6
add proto that was missing. (#986) 2022-12-23 13:27:07 -08:00
Miccah
7ac7fdae44
Add more logging for git sources (#974) 2022-12-13 17:51:57 -06:00
Bill Rich
33d32d2de4
Don't scan the --since-commit target (#960) 2022-12-06 13:24:27 -08:00
Bill Rich
f1ec9e74eb
Close files to clean up tmp files (#940) 2022-11-22 13:13:34 -08:00
Bill Rich
79cae3b82b
Add newlines when file is split (#937) 2022-11-22 09:01:39 -08:00
Miccah
86f9e1288f
Initialize scan options if given a nil pointer (#924) 2022-11-15 17:01:59 -06:00
ahrav
dd141fb55f
[oc-147] - Add context to all git methods (#901) 
* Add context to all git methods.

* remove logrus.

* Add ctx.

* Address comments.

* Add error to clone failing.

* Return error.
 2022-11-03 16:36:52 -07:00
Bill Rich
965279421c
Support common ssh repo format (#878) 
* Try ssh repo format

* Add tests
 2022-10-28 11:56:03 -07:00
Bill Rich
ab71b93f7d
Add context to handler (#877) 
* Add context to handler

* Return rather than break out of select
 2022-10-28 08:57:55 -07:00
Bill Rich
d7d614cc5f
Copy buffer bytes (#864) 2022-10-25 09:09:47 -07:00
Bill Rich
958266ea84
Run chunker in pipeline (#859) 
* Run chunker in pipeline

* Move ChunkSize and PeekSize to source package.

* Use new Chunk and Peek size location
 2022-10-24 13:57:27 -07:00
Bill Rich
3d5f697f9a
Use line aware chunking for git. (#858) 2022-10-24 13:00:03 -07:00
Dustin Decker
64ace363af Change commit to trace level logging 2022-10-24 08:59:52 -07:00
Miccah
2bc4985061
Add SSH config option for the git source (#830) 
* Add SSH config option for the git source

The auth message is empty since we use the git binary underneath to
handle the SSH authentication.

* Import digitaloceanv2
 2022-09-28 20:40:01 +02:00
Dustin Decker
752c848640
Show clone path for git repos (#823) 2022-09-26 14:36:55 -07:00
Bill Rich
e3107ad6bb
Move head and base normalization to source (#818) 2022-09-23 08:58:45 -07:00
Bill Rich
509cf8b6fa
Use headref and check empty commits for base (#815) 2022-09-21 16:04:01 -07:00
Bill Rich
912d8e461d
Add context so to avoid splitting creds. (#791) 
* Add context so to avoid splitting creds.

* Add context newlines to expected results
 2022-09-09 15:00:33 -07:00
ahrav
7ba583ca40
[THOG-681] - Handle errors sources (#783) 
* Handle errors w/ github source.

* Fix loop var captured by func literal.

* Fix loop var captured by func literal.

* Set completed progress if the scan completes with no errors.

* Set progress to 100% if the scope and iteration are both 0.

* Fix commentary.

* Fix test.

* Return after the defer to os.RemoveAll.

* Fix unauth scan.

* Inline range loop.

* update tests for partial scan completion with errors. Ensure correct progress is set.

* Update progress for all sources.

* Update github test.

* Address comments.
 2022-09-07 19:40:37 -07:00
Bill Rich
41936169c7
Use gitparse for unstaged changes. (#775) 2022-09-03 18:01:36 -07:00
Dustin Decker
fa9479100e
Add common sentry recover library and add into goroutines (#738) 
* Add common sentry recover library and add into goroutines

* fix nits
 2022-08-29 11:45:37 -07:00
Haz
4cc3529bc5
Added support for SSH URIs (#725) 2022-08-23 16:34:34 -07:00
Bill Rich
a0d44a39f1
Use trufflesec git parser (#729) 
* Use trufflesec git parser.

* wip

* Fix line numbers and linter feedback
 2022-08-23 13:29:20 -07:00
Bill Rich
4a93e49eea
Support scanning binary files in git sources (#684) 
* Scan binary files for git sources

* Create data chunks in for loop

* Linter feedback and newline commit result

* Use disk buffered reader and chunker function
 2022-08-10 16:10:45 -07:00
ahrav
30ebe84e3e
[THOG-608] - Fix linter errors. (#701) 
* Fix linter errors.

* Fix gist adding test.

* Update test string for mock JSON reply.

* Remove if.
 2022-08-09 19:20:02 -07:00
ahrav
21e1ff4a8a
Fix the order to correctly match the params in NewGit. (#676) 2022-07-28 13:23:45 -07:00
trufflesteeeve
176552b07a
Fix commit attribution, git tests, and run make protos (#667) 
* Update dependency to fix commit attribution, fix git tests

* Run make protos to match code with current proto definitions
 2022-07-25 11:44:15 -04:00
trufflesteeeve
96106563a9
Remove git fragment trace (#656) 
The fragment trace was a bit too verbose even at the trace level. We may
want to trace the file being chunked or something like that, but not the
entire diff.
 2022-07-14 13:13:23 -04:00
trufflesteeeve
e123e9f177
Cleanup individual repositories after scanning (#614) 2022-06-10 14:00:50 -04:00
Miccah
fc18a5ae0c
Bug fix and add authentication in shallow clone (#595) 2022-05-31 20:45:28 -05:00
Miccah
67ad2f2247
Shallow clone if --since-commit is provided (#564) 
* Shallow clone if --since-commit is provided

* Set the user before constructing args

* Fix vbout detector

* Address PR comments

* Use a better name for timestamp
* Use net.URL.String method for the remote path
 2022-05-24 10:49:03 -05:00
ahrav
d2605354fe
[THOG-332 ]Remove TokenSource interface from the init method of Source. (#539) 
* Remove TokenSource interface from the init method of Source.

* Remove proto message.

* Remove proto message.

* Fix tests.

* Fix filesystem test.
 2022-05-13 14:35:06 -07:00
ahrav
b0d79180f6
[THOG-314] Add new parameter to the Init method for the source interface. (#529) 
* Add new parameter to the Init method for the source interface.

* Add Oauth Token service.

* remove .test file.

* remove .test file.

* Fix param spelling.

* fix tests with new param in init

* Add missing gock lib.
 2022-05-10 11:11:43 -07:00
ahrav
e12432cef8
[THOG-315] Replace bytes.buffer with strings.builder. (#533) 
* Replace bytes.buffer with string.builder.

* Remove profiling.

* Remove detector changes.

* ignore .test files.

* fix detectors removed.
 2022-05-09 17:02:46 -07:00
ahrav
1bd5e9d9d2
fix merge conflicts. (#497) 2022-05-02 15:04:05 -07:00
ahrav
eb9d3e333c
[THOG-269] Increase test coverage for git. (#496) 2022-05-02 12:47:38 -07:00
ahrav
865bc5baf6
[THOG-248] fix broken gitlab tests (#437) 
* Fix broken gitlab test.

* Close chunks chan from blocking indefinitely.

* Range over chunksCh in the event chunksCh is nil we don't run into an invalid memory address error.
Update warnings and move clone output information back.

* Remove commented out code.

* Remove .Run() because .CombinedOutput() should call .Run()

* Update test to include count check.

* Address PR comments.

* Fix merge issue.
 2022-04-21 18:18:07 -07:00
Bill Rich
d4c33918f6
Ignore safedir when scanning git repo (#438) 2022-04-21 10:02:20 -07:00
Dustin Decker
879e65b5c5
Handle nil file channel (#417) 2022-04-16 15:41:51 -07:00
Bill Rich
09de492ec9
Check that required info is present (#415) 2022-04-15 16:32:16 -07:00
Bill Rich
1f55171437
Scan commit in since-commit (#416) 
* Scan commit in since-commit.

Fixes #413

* address lint issue

Co-authored-by: Dustin Decker <dustin@trufflesec.com>
 2022-04-15 15:58:54 -07:00
Bill Rich
af979d4620
Use .Line to reduce diff strangeness (#380) 2022-04-11 16:38:08 -07:00
Bill Rich
6be5ee18ff
Remove overseer warn, and don't os.Exit on git error. (#348) 2022-04-08 18:20:19 -07:00
Miccah
c620a62c8c
Make remote URL metadata optional for git scanning (#313) 
* Make remote URL metadata optional for git scanning

* Use helper function in ScanUnstaged
 2022-04-07 16:32:55 -07:00
ahrav
cedb3393d1
[THOG-128] Code cleanup/ OSS onboarding (#117) 
* Small amount of code clean up.

* Rename sem to concurrency for better readability and to remove an extra comment.

* fix stashing issue.

Co-authored-by: Ahrav Dutta <ahrav.dutta@trufflesec.com>
 2022-04-01 16:47:27 -07:00
Bill Rich
62ed0878a5
Improve git clone error logging (#114) 2022-03-30 14:27:13 -07:00
steeeve
a770f643df Add placeholder for encoded resume info in SetProgressComplete 2022-03-24 12:43:36 -04:00
Bill Rich
c2b4f0bc39
Include line numbers in git metadata (#97) 
* Include line numbers in git metadata

* Update tests for fragments and line numbers
 2022-03-22 09:27:15 -07:00
Bill Rich
0ea3b93238
Use git cmd for clone (#96) 
* Use git command to clone

* Check for git command when cloning
 2022-03-22 09:26:58 -07:00
Bill Rich
faf30fe8fe
Use author email (#85) 2022-03-15 17:54:03 -07:00
Bill Rich
6354b16810
Use patched go-gitdiff (#84) 2022-03-15 17:02:45 -07:00
Bill Rich
e8234c3514
Remove unused code and add git binary check (#80) 2022-03-14 17:47:18 -07:00
Dustin Decker
72c13c4b8f
Update detector process and readme (#79) 2022-03-14 17:27:14 -07:00
Bill Rich
0ee34a5be6
Use gitleaks git cmd stream (#75) 
* Switch to git cmd stream

* Fix rebase issues
 2022-03-14 17:12:58 -07:00
Dustin Decker
b0e0c26137 update detectors 2022-03-11 10:24:36 -08:00
Bill Rich
55b839fc5a
Only scan files that changed from base to head (#68) 
* Only scan files that changed from base to head

* Acknowledge ignored errors
 2022-03-08 08:48:00 -08:00
Bill Rich
665b0bf928 Add timestamp (#61) 
* Add timestamp to sources

* Include timestamp in sources
 2022-03-04 08:39:23 -08:00
Bill Rich
c144630c54 Add parent results to ignore list (#47) 
* Add parent results to ignore list

* Force concurrency to 1 when base commit is set
 2022-03-04 08:39:22 -08:00
Bill Rich
56dc1b109c Check commit order, dedupe results, and support using a head commit. (#44) 
* Check commit order and support using a head commit.

* Only apply dedupe to git bases source
 2022-03-04 08:39:22 -08:00
Dustin Decker
77418fb3f8 module v3 2022-02-15 18:54:47 -08:00
Bill Rich
2d8756938d Fast git scanning (#40) 
* Fast git scanning

* Use original tests

* Use committer time

Co-authored-by: Bill Rich <bill.rich@trufflesec.com>
 2022-02-15 18:54:47 -08:00
Bill Rich
6b183424f5 Match expected chunks to actual 2022-02-15 18:54:47 -08:00
Dustin Decker
44d113c408 Add concurrency to gitlab source integration 2022-02-15 18:54:47 -08:00
Bill Rich
206b99704b Change log order and path filtering. 2022-02-15 18:54:47 -08:00
Dustin Decker
26184dc2cd Fix incorrect commit skipped error 2022-02-15 18:54:47 -08:00
Bill Rich
28ed0c3b7c Complete support for existing git scan flags (#13) 
* Add `since_commit` to git scan

* Support `max_depth` option for git scan

* Use new options in github and gitlab sources

* Address review feedback

Co-authored-by: Bill Rich <bill.rich@trufflesec.com>
 2022-02-15 18:54:47 -08:00
Dustin Decker
d6ffadb1ee
Initial docs and release automation (#5) 2022-01-18 16:59:18 -08:00
Bill Rich
a70937bfe6
Support remote git repos using https (#9) 
Co-authored-by: Bill Rich <bill.rich@trufflesec.com>
 2022-01-14 16:07:45 -08:00
Bill Rich
1422cc303a
Include and exclude paths args for gitscan (#6) 
* include and exclude paths gitscan args

Add support for include_paths and exclude_paths arguments when scanning
git sources.

* Improve variable name

Co-authored-by: Bill Rich <hrich@Bills-MacBook-Pro.local>
 2022-01-14 12:40:50 -08:00
Dustin Decker
4218c39d99
Initial CLI w/ partially implemented Git source and demo detector (#1) 2022-01-13 12:02:24 -08:00