Mirrors/trufflehog
2
0
Fork 0
mirror of https://github.com/trufflesecurity/trufflehog.git synced 2024-09-20 14:42:03 +00:00
Code Issues Projects Releases Packages Wiki Activity
2794 commits 123 branches 240 tags 83 MiB
Commit graph

366 commits

Author SHA1 Message Date
Miccah
161e499142
[chore] Remove logrus from trufflehog (#1095) 
* [chore] Remove logrus from trufflehog

* Minor fixes

* Fix logFatal call

* Fix logrus call
 2023-02-14 17:00:07 -06:00
Miccah
c6826c4574
Fix nil scan options (#1107) 2023-02-14 12:09:45 -06:00
SAYGIN Metin
f2139a7615
Github filter support for exclude and include (#1087) 
* test

* Add missing head and base hash back.

---------

Co-authored-by: Ahrav Dutta <ahravdutta02@gmail.com>
 2023-02-14 08:40:53 -08:00
ahrav
c5c8d10d28
[chore] - Remove monolithic config struct (#1091) 
* REmove monolithic config struct.

* fix broken test.
 2023-02-10 12:43:00 -08:00
Miccah
d317ddb51a
[chore] Remove logrus from circleci, filesystem, gitlab, and s3 sources (#1089) 
* [chore] Remove logrus from circleci, filesystem, gitlab, and s3 sources

* Address comments
 2023-02-10 11:02:55 -06:00
Miccah
0ce72ccda3
[chore] Remove logrus from github source (#1086) 
* [chore] Remove logrus from github source

* Fix handleRateLimit test

* Fix tests
 2023-02-09 18:02:04 -06:00
ahrav
e47cc2451f
Dont pre-allocate errors slice. (#1083) 2023-02-08 17:33:30 -08:00
Miccah
1f0fd91205
Skip repo and continue scanning when encountering an error (#1080) 2023-02-08 11:33:01 -06:00
ahrav
0d73dbe638
[chore] - Add tests for errors (#1071) 2023-02-08 04:15:44 -08:00
Bill Rich
af6e3f8fdf
Pull gitparse config options out of pkg consts (#1072) 
* Pull gitparse config options out of pkg consts.

* Adjust naming
 2023-02-04 13:19:23 -08:00
ahrav
8be89a593b
Handle errors in a thread safe manner (#1052) 
* Handle errors in a thread safe manner.

* fix test.

* fix linter.

* address comments.
 2023-02-02 11:05:33 -08:00
Alexandr Marchenko
b29b78c10d
filesystem support for exclude and include filters (2nd attemp) (#1033) 
* fix filter issue - empty lines should be ignored

* filesystem support for filter exclude

Co-authored-by: Dustin Decker <dustin@trufflesec.com>
 2023-01-26 09:33:45 -08:00
Bill Rich
00ebb2ed64
Full git log when targeting base merge commit (#1044) 
* Full git log when targeting merge commits

* Full log is needed whenever base is specified.
 2023-01-26 09:17:54 -08:00
Dustin Decker
4ef546a06b
fix github integration tests (#1042) 2023-01-25 08:57:39 -08:00
ahrav
1621403e11
Add concurrency to CircleCi source (#1029) 
* Small cleanup of CircleCi source.

* Add concurrency to circleci.

* merge w/ cleanup branch.

* Rdefine loop var.

* Delete github.go

* reverge file delete.

* Add debug log for scan errors.

* make collecting scanned errors thread safe.

* pre-allocate errors slice.
 2023-01-17 12:24:49 -08:00
ahrav
319ae64a02
[chore] - Small cleanup of CircleCi source (#1028) 
* Small cleanup of CircleCi source.

* address comments.

* Add context to methods as first param.
 2023-01-17 09:36:18 -08:00
Yassine Ilmi
d720c0c0f3
Switch to retryableHttpClient for GitHub AuthN API Client + More Logs (#995) 
* Adding missing flags to Readme

* Use retryableHttpClient by default for GitHub

* Adding repoUrl for scanning time log

* Use WithField instead of WithFields

* Updating README with lasted --help output
 2023-01-09 09:21:56 -08:00
Dustin Decker
5f6143f09a
Add Circle CI source (#997) 
* Add Circle CI source

* remove SHA1 line

* remove trim
 2023-01-05 21:44:37 -08:00
ahrav
009756dce6
add proto that was missing. (#986) 2022-12-23 13:27:07 -08:00
ahrav
936a139596
Allow using a glob for include list. (#977) 
* Allow using a glob for include list.

* Update command flag.

* Make comment more clear.

* update comment.

* Allow scanning repo and org at the same time.
 2022-12-16 13:28:16 -08:00
Bill Rich
36ca2601e0
Add s3 object count to trace logs (#975) 
* Add s3 object count to trace logs

* fix debug level
 2022-12-13 16:46:09 -08:00
Miccah
7ac7fdae44
Add more logging for git sources (#974) 2022-12-13 17:51:57 -06:00
ahrav
26befdd1ec
[bug] - Handle error when scanning s3 bucket. (#969) 
* Handle error when scanning s# bucket.

* move wait outside loop.

* Add logging.

* revert changes.

* remove.

* revert.
 2022-12-12 10:10:06 -08:00
Dustin Decker
7de9bdd12d
Support globbing with ignore repos (#967) 2022-12-09 12:10:42 -08:00
ahrav
a72b9feb35
Only scan org with --org flag. (#931) 2022-12-06 16:18:48 -08:00
Bill Rich
33d32d2de4
Don't scan the --since-commit target (#960) 2022-12-06 13:24:27 -08:00
Bill Rich
1a1c2e275e
Change chunker test source (#959) 
* Change chunker test source

* Emit chunk if the size isn't 0
 2022-12-06 12:45:08 -08:00
Bill Rich
9f99ee470d
Integration test fixes (#956) 
* Adjust repo count for new app

* Fix chunk test count
 2022-12-06 08:42:24 -08:00
Bill Rich
f1ec9e74eb
Close files to clean up tmp files (#940) 2022-11-22 13:13:34 -08:00
Bill Rich
79cae3b82b
Add newlines when file is split (#937) 2022-11-22 09:01:39 -08:00
Dustin Decker
28dd25beeb
S3 scanner improvements (#938) 2022-11-21 19:15:26 -08:00
Miccah
86f9e1288f
Initialize scan options if given a nil pointer (#924) 2022-11-15 17:01:59 -06:00
Jessica
3d501975e4
Add filter as scan option to gitlab module's git scan (#919) 2022-11-15 13:02:37 -08:00
ahrav
dd141fb55f
[oc-147] - Add context to all git methods (#901) 
* Add context to all git methods.

* remove logrus.

* Add ctx.

* Address comments.

* Add error to clone failing.

* Return error.
 2022-11-03 16:36:52 -07:00
ahrav
fe1e475a04
Prevent concurrent read and writes to visibility map. (#892) 2022-11-01 16:20:59 -07:00
Bill Rich
965279421c
Support common ssh repo format (#878) 
* Try ssh repo format

* Add tests
 2022-10-28 11:56:03 -07:00
Bill Rich
ab71b93f7d
Add context to handler (#877) 
* Add context to handler

* Return rather than break out of select
 2022-10-28 08:57:55 -07:00
Bill Rich
d7d614cc5f
Copy buffer bytes (#864) 2022-10-25 09:09:47 -07:00
Bill Rich
958266ea84
Run chunker in pipeline (#859) 
* Run chunker in pipeline

* Move ChunkSize and PeekSize to source package.

* Use new Chunk and Peek size location
 2022-10-24 13:57:27 -07:00
Bill Rich
3d5f697f9a
Use line aware chunking for git. (#858) 2022-10-24 13:00:03 -07:00
Dustin Decker
64ace363af Change commit to trace level logging 2022-10-24 08:59:52 -07:00
ahrav
46bc010165
Add tests for including github repos. (#854) 2022-10-21 07:56:36 -07:00
trufflesteeeve
fb56b9f713
Check rate limit when getting github user (#855) 
Also, don't fetch a github user or their token when both are known. This
currently only affects the Github Token auth type. Github App
installations will continually fetch tokens every time we clone a repo.
In the future we should check the `ExpiresAt` field of the Github App
token and determine if we need to fetch a new one at that point.
 2022-10-20 18:14:28 -04:00
ahrav
029519eb01
[THOG-767] ignore gitlab repos (#853) 
* Add ability to ignore repos.

* use std library slices.Contains.

* Add tests.

* Remove zero values from test.
 2022-10-19 13:55:44 -07:00
ahrav
2d6aadcb46
[THOG-774] - GitHub ignore repo full name (#848) 
* Use github repo full name.

* fix tests.
 2022-10-14 09:20:49 -07:00
ahrav
04c9bb535e
[THOG-768] - Add ability to skip scanning Github repos (#846) 
* Add ability to skip scanning Github repos.

* remove old change.

* rename method.
 2022-10-12 16:28:24 -07:00
Dustin Decker
785cead43e
Ignore URIs where the password is redacted (#842) 
Only `*`s in the password is a redacted basic auth URI.
 2022-10-11 14:18:52 -07:00
Miccah
2bc4985061
Add SSH config option for the git source (#830) 
* Add SSH config option for the git source

The auth message is empty since we use the git binary underneath to
handle the SSH authentication.

* Import digitaloceanv2
 2022-09-28 20:40:01 +02:00
Miccah
891996f546
Do not fail scanning if we cannot enumerate gists (#826) 2022-09-27 20:59:10 +02:00
Bill Rich
1c00014051
Include public/private in github metadata (#812) 
* Include public/private in github metadata

* CR feedback

* Fix typos and naming
 2022-09-26 14:55:46 -07:00
Dustin Decker
97a73710de
403 on listing user gist should not fail org scan (#822) 2022-09-26 14:37:25 -07:00
Dustin Decker
752c848640
Show clone path for git repos (#823) 2022-09-26 14:36:55 -07:00
Bill Rich
e3107ad6bb
Move head and base normalization to source (#818) 2022-09-23 08:58:45 -07:00
ahrav
92f40c2031
[THOG-709] - Recover from detector panics (#810) 2022-09-22 07:01:10 -07:00
trufflesteeeve
63fcf33ce6
Fix improper github org member pagination (#814) 
I'm not sure I fully understand why this issue exists. But I think the
short version is this: When we attempted to paginate users, we would set
a variable's Page value. But that variable appears to not actually be a
pointer, despite being added as one. It probably has to do with how
struct embedding works. Either way, if we make the overall options
variable the whole thing, and update its embedded struct with our page
variable, everything works out.
 2022-09-21 16:22:42 -07:00
Bill Rich
509cf8b6fa
Use headref and check empty commits for base (#815) 2022-09-21 16:04:01 -07:00
Dustin Decker
335e676caa
Provide user when during private clones with token and fix integration tests (#811) 2022-09-19 15:53:21 -07:00
Bill Rich
593f1e6754
Include apiClient in Github source (#804) 2022-09-19 14:31:48 -07:00
trufflesteeeve
945de06858
Fix include-members not working on github (#773) 2022-09-12 13:26:38 -04:00
Bill Rich
912d8e461d
Add context so to avoid splitting creds. (#791) 
* Add context so to avoid splitting creds.

* Add context newlines to expected results
 2022-09-09 15:00:33 -07:00
Dustin Decker
ecfdb0105b
Provide correct username for app cloning and add integration test (#786) 2022-09-08 17:41:53 -07:00
Dustin Decker
80b247286b
Improve GitHub debug logging (#784) 
* close bodies early

* add more debug logging to github

* fix nil check

* Add nil checks for response
 2022-09-08 12:23:40 -07:00
ahrav
7ba583ca40
[THOG-681] - Handle errors sources (#783) 
* Handle errors w/ github source.

* Fix loop var captured by func literal.

* Fix loop var captured by func literal.

* Set completed progress if the scan completes with no errors.

* Set progress to 100% if the scope and iteration are both 0.

* Fix commentary.

* Fix test.

* Return after the defer to os.RemoveAll.

* Fix unauth scan.

* Inline range loop.

* update tests for partial scan completion with errors. Ensure correct progress is set.

* Update progress for all sources.

* Update github test.

* Address comments.
 2022-09-07 19:40:37 -07:00
Bill Rich
41936169c7
Use gitparse for unstaged changes. (#775) 2022-09-03 18:01:36 -07:00
Bill Rich
d11ce27f33
Use correct reader in filesystem source (#756) 2022-08-30 10:24:52 -07:00
Dustin Decker
fa9479100e
Add common sentry recover library and add into goroutines (#738) 
* Add common sentry recover library and add into goroutines

* fix nits
 2022-08-29 11:45:37 -07:00
Bill Rich
0ddd49a1b8
Use file handler and common chunker (#707) 2022-08-23 16:35:52 -07:00
Haz
4cc3529bc5
Added support for SSH URIs (#725) 2022-08-23 16:34:34 -07:00
Bill Rich
a0d44a39f1
Use trufflesec git parser (#729) 
* Use trufflesec git parser.

* wip

* Fix line numbers and linter feedback
 2022-08-23 13:29:20 -07:00
Bill Rich
5ad3bbde37
Use pointer to config (#715) 2022-08-16 09:15:25 -07:00
ahrav
73f9d3f0a0
[chore] - Use config struct instead of pointer for engine scans. (#709) 
* Use a config struct instead of pointer when scanning engine sources.

* use config.
 2022-08-12 09:56:24 -07:00
Bill Rich
4a93e49eea
Support scanning binary files in git sources (#684) 
* Scan binary files for git sources

* Create data chunks in for loop

* Linter feedback and newline commit result

* Use disk buffered reader and chunker function
 2022-08-10 16:10:45 -07:00
Bill Rich
a473b9aa99
Use re-readable reader and common chunker (#703) 
* Use re-readable reader and common chunker

* Linter feedback

* Break on error
 2022-08-10 15:32:49 -07:00
ahrav
dcc102a81c
[Thog-371] Utilize config struct for engine scans (#700) 
* Use a config struct when scanning and engine source.

* fix tests.

* Move test_helpers to the sources pkg.

* Handle ScanGit error in tests.

* adderss comments.

* Use functional options.

* Remove temp var.

* Add better var names for the setup functions for each config.

* Remove unused var.

* fix error logs.

* fix error logs.

* single line.

* remove blank lines.
 2022-08-10 10:11:13 -07:00
ahrav
30ebe84e3e
[THOG-608] - Fix linter errors. (#701) 
* Fix linter errors.

* Fix gist adding test.

* Update test string for mock JSON reply.

* Remove if.
 2022-08-09 19:20:02 -07:00
Bill Rich
7273dc9058
Archive decoder (#683) 
* Archive decoder

* Fix reader handling

* Seek error handling

* Add tests

* Fix extra empty chunk

* Sync chunk size
 2022-08-02 20:36:21 -07:00
ahrav
21e1ff4a8a
Fix the order to correctly match the params in NewGit. (#676) 2022-07-28 13:23:45 -07:00
trufflesteeeve
176552b07a
Fix commit attribution, git tests, and run make protos (#667) 
* Update dependency to fix commit attribution, fix git tests

* Run make protos to match code with current proto definitions
 2022-07-25 11:44:15 -04:00
trufflesteeeve
96106563a9
Remove git fragment trace (#656) 
The fragment trace was a bit too verbose even at the trace level. We may
want to trace the file being chunked or something like that, but not the
entire diff.
 2022-07-14 13:13:23 -04:00
trufflesteeeve
e793f4a5e6
Properly count the number of repos after a github scan resume (#625) 2022-06-17 16:21:22 -04:00
trufflesteeeve
10f4d02c31
Allow gitlab to resume from encoded resume info (#611) 2022-06-17 11:45:17 -04:00
Dustin Decker
2178f1f42e reword and fix error logging 2022-06-13 16:14:22 -07:00
trufflesteeeve
e123e9f177
Cleanup individual repositories after scanning (#614) 2022-06-10 14:00:50 -04:00
Dustin Decker
9bcddbc45a
Change GHE org enum to use since ID instead of pages (#618) 
* Change GHE org enum to use since ID instead of pages

* fix logging
 2022-06-09 15:09:13 -07:00
Dustin Decker
8051b03bbf
improve debug logging for GHE enum (#615) 2022-06-08 13:56:07 -07:00
Dustin Decker
1a12a25f4d
Enumerate all visible orgs in GHE (#612) 2022-06-07 09:24:31 -07:00
Dustin Decker
e3bbf293e2
Fix NPD on mutex (#609) 
* Fix NPD on mutex

* fix test
 2022-06-06 17:20:27 -07:00
Miccah
9074006695
Fix bug in GitHub unit test mocking (#608) 2022-06-06 16:58:34 -07:00
trufflesteeeve
fd79a367f1
Allow github to resume from encoded resume info (#601) 2022-06-06 12:08:57 -04:00
Miccah
fc18a5ae0c
Bug fix and add authentication in shallow clone (#595) 2022-05-31 20:45:28 -05:00
Miccah
67ad2f2247
Shallow clone if --since-commit is provided (#564) 
* Shallow clone if --since-commit is provided

* Set the user before constructing args

* Fix vbout detector

* Address PR comments

* Use a better name for timestamp
* Use net.URL.String method for the remote path
 2022-05-24 10:49:03 -05:00
ahrav
2051fe14ff
remove profililing. (#567) 2022-05-23 11:05:39 -07:00
ahrav
d2605354fe
[THOG-332 ]Remove TokenSource interface from the init method of Source. (#539) 
* Remove TokenSource interface from the init method of Source.

* Remove proto message.

* Remove proto message.

* Fix tests.

* Fix filesystem test.
 2022-05-13 14:35:06 -07:00
ahrav
b0d79180f6
[THOG-314] Add new parameter to the Init method for the source interface. (#529) 
* Add new parameter to the Init method for the source interface.

* Add Oauth Token service.

* remove .test file.

* remove .test file.

* Fix param spelling.

* fix tests with new param in init

* Add missing gock lib.
 2022-05-10 11:11:43 -07:00
ahrav
e12432cef8
[THOG-315] Replace bytes.buffer with strings.builder. (#533) 
* Replace bytes.buffer with string.builder.

* Remove profiling.

* Remove detector changes.

* ignore .test files.

* fix detectors removed.
 2022-05-09 17:02:46 -07:00
Miccah
edaf1e1fd3
Move GitHub integration tests behind a build flag and add unit tests (#527) 
* Add unit tests and refactor some logic

* Move integration tests to a separate file behind a build flag

* Fix bugs in normalizeRepos

* Address lint errors

* Sort slices before comparing because order doesn't matter
 2022-05-09 08:31:00 -07:00
Miccah
85208606bb
Reorganize GitHub source (#517) 
* Reorganize GitHub source

This breaks up the Chunks method into smaller sub-method calls to help
organize and better understand the logic flow. No logic has been
modified (except one obvious bug), just shuffling code around.

* Check errors and revert bug fix
 2022-05-06 05:00:46 -07:00
Bill Rich
212aa9ba1e
Disable tests that take too long (#524) 2022-05-04 16:37:37 -07:00
Bill Rich
c78120e56f
Syslog source (#500) 
* Add syslog source

* only load cert/key with tls

* Cleanup

* Linting

Co-authored-by: Bill Rich <bill.rich@trufflesec.com>
 2022-05-04 15:08:11 -07:00
Miccah
71442320ec
Chunk orgs the same when authenticated as unauthenticated (#501) 
Also debug log the amount of forks we find in addReposByOrg.
 2022-05-02 17:26:01 -07:00
ahrav
1bd5e9d9d2
fix merge conflicts. (#497) 2022-05-02 15:04:05 -07:00
ahrav
eb9d3e333c
[THOG-269] Increase test coverage for git. (#496) 2022-05-02 12:47:38 -07:00
dcRUSTy
a74bea0e8e
fix: regex for hostnames (#494) 
* fix: regex for ends with github.com

* fix: regex for hooks.zapier.com

* fix: regex for hooks.slack.com

* fix: regex for signalwire.com

* fix: regex for kanbantool.com

* fix: regex for invoiceoceam.com

* fix: regex for invoiceocean.com

* fix: regex for freshdesk.com

* fix: regex for discord.com

* fix: regex for deputy.com
 2022-05-02 09:44:37 -07:00
Miccah
8cd0831f77
Test Gitlab basic authentication with password and personal access token (#474) 2022-04-23 17:55:13 -07:00
ahrav
753f116c89
Fix broken test for gitlab. (#473) 2022-04-22 10:28:17 -07:00
ahrav
865bc5baf6
[THOG-248] fix broken gitlab tests (#437) 
* Fix broken gitlab test.

* Close chunks chan from blocking indefinitely.

* Range over chunksCh in the event chunksCh is nil we don't run into an invalid memory address error.
Update warnings and move clone output information back.

* Remove commented out code.

* Remove .Run() because .CombinedOutput() should call .Run()

* Update test to include count check.

* Address PR comments.

* Fix merge issue.
 2022-04-21 18:18:07 -07:00
ahrav
3b35404238
[THOG-204] Use oauth2 as username when cloning (#441) 
* Use oauth2 as the username for cloning private repos with a token.

* Update username string.

* Only set user to "placeholder" if no username is present.
 2022-04-21 17:49:08 -07:00
Bill Rich
d4c33918f6
Ignore safedir when scanning git repo (#438) 2022-04-21 10:02:20 -07:00
dependabot[bot]
5b6c9f7b31
Bump github.com/xanzy/go-gitlab from 0.60.0 to 0.63.0 (#382) 
* Bump github.com/xanzy/go-gitlab from 0.60.0 to 0.63.0

Bumps [github.com/xanzy/go-gitlab](https://github.com/xanzy/go-gitlab) from 0.60.0 to 0.63.0.
- [Release notes](https://github.com/xanzy/go-gitlab/releases)
- [Changelog](https://github.com/xanzy/go-gitlab/blob/master/releases_test.go)
- [Commits](https://github.com/xanzy/go-gitlab/compare/v0.60.0...v0.63.0)

---
updated-dependencies:
- dependency-name: github.com/xanzy/go-gitlab
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Fix name for updated dependency

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: bill-rich <bill.rich@gmail.com>
 2022-04-20 23:15:20 -07:00
Dustin Decker
879e65b5c5
Handle nil file channel (#417) 2022-04-16 15:41:51 -07:00
Bill Rich
09de492ec9
Check that required info is present (#415) 2022-04-15 16:32:16 -07:00
Bill Rich
1f55171437
Scan commit in since-commit (#416) 
* Scan commit in since-commit.

Fixes #413

* address lint issue

Co-authored-by: Dustin Decker <dustin@trufflesec.com>
 2022-04-15 15:58:54 -07:00
Bill Rich
af979d4620
Use .Line to reduce diff strangeness (#380) 2022-04-11 16:38:08 -07:00
Bill Rich
6be5ee18ff
Remove overseer warn, and don't os.Exit on git error. (#348) 2022-04-08 18:20:19 -07:00
Bill Rich
bf5c757cd3
Add head and base support to github (#346) 2022-04-08 14:32:54 -07:00
Dustin Decker
ba6ea9d42f
Improve error logging and continue when there are insufficient permissions (#315) 2022-04-07 18:33:44 -07:00
Miccah
c620a62c8c
Make remote URL metadata optional for git scanning (#313) 
* Make remote URL metadata optional for git scanning

* Use helper function in ScanUnstaged
 2022-04-07 16:32:55 -07:00
Bill Rich
d78c929385
Actually skip file (#299) 2022-04-06 09:48:40 -07:00
Bill Rich
33aa6f9cab
Log error and skip file when stat fails (#296) 2022-04-05 18:58:05 -07:00
Dustin Decker
d41d18af3a fix cli parsing and improve github user scanning error handling 2022-04-03 13:42:23 -07:00
ahrav
cedb3393d1
[THOG-128] Code cleanup/ OSS onboarding (#117) 
* Small amount of code clean up.

* Rename sem to concurrency for better readability and to remove an extra comment.

* fix stashing issue.

Co-authored-by: Ahrav Dutta <ahrav.dutta@trufflesec.com>
 2022-04-01 16:47:27 -07:00
Bill Rich
6ed01500f8
Only add trailing slash when endpoint populated (#120) 2022-04-01 11:47:59 -07:00
Bill Rich
62ed0878a5
Improve git clone error logging (#114) 2022-03-30 14:27:13 -07:00
Bill Rich
155566bbd5
make sure url ends with slash (#110) 2022-03-28 09:56:44 -07:00
steeeve
a770f643df Add placeholder for encoded resume info in SetProgressComplete 2022-03-24 12:43:36 -04:00
Bill Rich
c2b4f0bc39
Include line numbers in git metadata (#97) 
* Include line numbers in git metadata

* Update tests for fragments and line numbers
 2022-03-22 09:27:15 -07:00
Bill Rich
0ea3b93238
Use git cmd for clone (#96) 
* Use git command to clone

* Check for git command when cloning
 2022-03-22 09:26:58 -07:00
Bill Rich
5ab5c6f9d9
Only scan regular files (#87) 
* Only scan regular files

* Remove IsDirectory func
 2022-03-16 16:04:10 -07:00
Bill Rich
faf30fe8fe
Use author email (#85) 2022-03-15 17:54:03 -07:00
Bill Rich
6354b16810
Use patched go-gitdiff (#84) 2022-03-15 17:02:45 -07:00
Bill Rich
e8234c3514
Remove unused code and add git binary check (#80) 2022-03-14 17:47:18 -07:00
Dustin Decker
72c13c4b8f
Update detector process and readme (#79) 2022-03-14 17:27:14 -07:00
Bill Rich
0ee34a5be6
Use gitleaks git cmd stream (#75) 
* Switch to git cmd stream

* Fix rebase issues
 2022-03-14 17:12:58 -07:00
Bill Rich
6486c18565
Add s3 support to CLI (#76) 
* Add s3 support to CLI

* Clean up comments

Co-authored-by: Dustin Decker <dustin@trufflesec.com>
 2022-03-14 17:07:07 -07:00
Dustin Decker
794a082b6c Fix Dockerfiles, readme example, and github rate limit handling 2022-03-12 07:36:42 -08:00
Dustin Decker
b0e0c26137 update detectors 2022-03-11 10:24:36 -08:00
Dustin Decker
8d754a88d4 update dependency 2022-03-09 15:49:57 -08:00
Bill Rich
55b839fc5a
Only scan files that changed from base to head (#68) 
* Only scan files that changed from base to head

* Acknowledge ignored errors
 2022-03-08 08:48:00 -08:00
Bill Rich
2a0ece9eef
Wait until rate limit reset (#69) 2022-03-08 08:47:37 -08:00
Bill Rich
665b0bf928 Add timestamp (#61) 
* Add timestamp to sources

* Include timestamp in sources
 2022-03-04 08:39:23 -08:00
Dustin Decker
c80bd5e905 Fix linting and dogfood in CI 2022-03-04 08:39:23 -08:00
trufflesteeeve
9cb99e5aa4 578 Fix gitlab basic auth with access tokens (#54) 2022-03-04 08:39:22 -08:00
Bill Rich
c144630c54 Add parent results to ignore list (#47) 
* Add parent results to ignore list

* Force concurrency to 1 when base commit is set
 2022-03-04 08:39:22 -08:00
Bill Rich
b25295580a Actually save the normalized repos (#50) 2022-03-04 08:39:22 -08:00
Bill Rich
c742f6a816 Do not continue if semaphore can't be acquired (#49) 2022-03-04 08:39:22 -08:00
Bill Rich
56dc1b109c Check commit order, dedupe results, and support using a head commit. (#44) 
* Check commit order and support using a head commit.

* Only apply dedupe to git bases source
 2022-03-04 08:39:22 -08:00
Dustin Decker
c20e9f4732 improvements 2022-03-04 08:39:17 -08:00
Dustin Decker
77418fb3f8 module v3 2022-02-15 18:54:47 -08:00
Bill Rich
2d8756938d Fast git scanning (#40) 
* Fast git scanning

* Use original tests

* Use committer time

Co-authored-by: Bill Rich <bill.rich@trufflesec.com>
 2022-02-15 18:54:47 -08:00
Dustin Decker
152ef6d4e1 add include forks option (#37) 2022-02-15 18:54:47 -08:00
Dustin Decker
c131a6e4ae add debug pprof server and metrics server 2022-02-15 18:54:47 -08:00
Bill Rich
1fb767247f Add missing pagination on github calls (#30) 
* Add missing pagination on github calls

Includes some refactoring to improve readability and code reuse.

* Close response body and handle rate limit

* Re-include support for including users as repos to github scans

* Fix gist test to match new func signature

* Add current test name to logging

* Support username as org use case

* Also include no-auth user as org

Co-authored-by: Bill Rich <bill.rich@trufflesec.com>
 2022-02-15 18:54:47 -08:00
Bill Rich
6b183424f5 Match expected chunks to actual 2022-02-15 18:54:47 -08:00
Bill Rich
2cc34f4633 Make tests more resilliant and more coverage 2022-02-15 18:54:47 -08:00
Dustin Decker
e15fa3a5be helpful logging 2022-02-15 18:54:47 -08:00
Dustin Decker
44d113c408 Add concurrency to gitlab source integration 2022-02-15 18:54:47 -08:00
Dustin Decker
7e38e699f6 GitHub concurrency (#25) 
* GitHub scan concurrency

* Add raw result to plain output

* Fix flakey test (still flakey)

* Fix race
 2022-02-15 18:54:47 -08:00
Bill Rich
206b99704b Change log order and path filtering. 2022-02-15 18:54:47 -08:00
Dustin Decker
26184dc2cd Fix incorrect commit skipped error 2022-02-15 18:54:47 -08:00
Dustin Decker
3da3f1ec94 Add gitlab pagination support (#26) 2022-02-15 18:54:47 -08:00
Bill Rich
28ed0c3b7c Complete support for existing git scan flags (#13) 
* Add `since_commit` to git scan

* Support `max_depth` option for git scan

* Use new options in github and gitlab sources

* Address review feedback

Co-authored-by: Bill Rich <bill.rich@trufflesec.com>
 2022-02-15 18:54:47 -08:00
Dustin Decker
03ead2f7ed Integrate GitHub source 2022-02-15 18:54:47 -08:00
Dustin Decker
d6ffadb1ee
Initial docs and release automation (#5) 2022-01-18 16:59:18 -08:00
Bill Rich
a70937bfe6
Support remote git repos using https (#9) 
Co-authored-by: Bill Rich <bill.rich@trufflesec.com>
 2022-01-14 16:07:45 -08:00
Bill Rich
1422cc303a
Include and exclude paths args for gitscan (#6) 
* include and exclude paths gitscan args

Add support for include_paths and exclude_paths arguments when scanning
git sources.

* Improve variable name

Co-authored-by: Bill Rich <hrich@Bills-MacBook-Pro.local>
 2022-01-14 12:40:50 -08:00
Dustin Decker
4218c39d99
Initial CLI w/ partially implemented Git source and demo detector (#1) 2022-01-13 12:02:24 -08:00