Mirrors/trufflehog
2
0
Fork 0
mirror of https://github.com/trufflesecurity/trufflehog.git synced 2024-11-10 15:14:38 +00:00
Code Issues Projects Releases Packages Wiki Activity
1687 commits 139 branches 257 tags 147 MiB
Commit graph

124 commits

Author SHA1 Message Date
ahrav
d2605354fe
[THOG-332 ]Remove TokenSource interface from the init method of Source. (#539) 
* Remove TokenSource interface from the init method of Source.

* Remove proto message.

* Remove proto message.

* Fix tests.

* Fix filesystem test.
 2022-05-13 14:35:06 -07:00
ahrav
b0d79180f6
[THOG-314] Add new parameter to the Init method for the source interface. (#529) 
* Add new parameter to the Init method for the source interface.

* Add Oauth Token service.

* remove .test file.

* remove .test file.

* Fix param spelling.

* fix tests with new param in init

* Add missing gock lib.
 2022-05-10 11:11:43 -07:00
ahrav
e12432cef8
[THOG-315] Replace bytes.buffer with strings.builder. (#533) 
* Replace bytes.buffer with string.builder.

* Remove profiling.

* Remove detector changes.

* ignore .test files.

* fix detectors removed.
 2022-05-09 17:02:46 -07:00
Miccah
edaf1e1fd3
Move GitHub integration tests behind a build flag and add unit tests (#527) 
* Add unit tests and refactor some logic

* Move integration tests to a separate file behind a build flag

* Fix bugs in normalizeRepos

* Address lint errors

* Sort slices before comparing because order doesn't matter
 2022-05-09 08:31:00 -07:00
Miccah
85208606bb
Reorganize GitHub source (#517) 
* Reorganize GitHub source

This breaks up the Chunks method into smaller sub-method calls to help
organize and better understand the logic flow. No logic has been
modified (except one obvious bug), just shuffling code around.

* Check errors and revert bug fix
 2022-05-06 05:00:46 -07:00
Bill Rich
212aa9ba1e
Disable tests that take too long (#524) 2022-05-04 16:37:37 -07:00
Bill Rich
c78120e56f
Syslog source (#500) 
* Add syslog source

* only load cert/key with tls

* Cleanup

* Linting

Co-authored-by: Bill Rich <bill.rich@trufflesec.com>
 2022-05-04 15:08:11 -07:00
Miccah
71442320ec
Chunk orgs the same when authenticated as unauthenticated (#501) 
Also debug log the amount of forks we find in addReposByOrg.
 2022-05-02 17:26:01 -07:00
ahrav
1bd5e9d9d2
fix merge conflicts. (#497) 2022-05-02 15:04:05 -07:00
ahrav
eb9d3e333c
[THOG-269] Increase test coverage for git. (#496) 2022-05-02 12:47:38 -07:00
dcRUSTy
a74bea0e8e
fix: regex for hostnames (#494) 
* fix: regex for ends with github.com

* fix: regex for hooks.zapier.com

* fix: regex for hooks.slack.com

* fix: regex for signalwire.com

* fix: regex for kanbantool.com

* fix: regex for invoiceoceam.com

* fix: regex for invoiceocean.com

* fix: regex for freshdesk.com

* fix: regex for discord.com

* fix: regex for deputy.com
 2022-05-02 09:44:37 -07:00
Miccah
8cd0831f77
Test Gitlab basic authentication with password and personal access token (#474) 2022-04-23 17:55:13 -07:00
ahrav
753f116c89
Fix broken test for gitlab. (#473) 2022-04-22 10:28:17 -07:00
ahrav
865bc5baf6
[THOG-248] fix broken gitlab tests (#437) 
* Fix broken gitlab test.

* Close chunks chan from blocking indefinitely.

* Range over chunksCh in the event chunksCh is nil we don't run into an invalid memory address error.
Update warnings and move clone output information back.

* Remove commented out code.

* Remove .Run() because .CombinedOutput() should call .Run()

* Update test to include count check.

* Address PR comments.

* Fix merge issue.
 2022-04-21 18:18:07 -07:00
ahrav
3b35404238
[THOG-204] Use oauth2 as username when cloning (#441) 
* Use oauth2 as the username for cloning private repos with a token.

* Update username string.

* Only set user to "placeholder" if no username is present.
 2022-04-21 17:49:08 -07:00
Bill Rich
d4c33918f6
Ignore safedir when scanning git repo (#438) 2022-04-21 10:02:20 -07:00
dependabot[bot]
5b6c9f7b31
Bump github.com/xanzy/go-gitlab from 0.60.0 to 0.63.0 (#382) 
* Bump github.com/xanzy/go-gitlab from 0.60.0 to 0.63.0

Bumps [github.com/xanzy/go-gitlab](https://github.com/xanzy/go-gitlab) from 0.60.0 to 0.63.0.
- [Release notes](https://github.com/xanzy/go-gitlab/releases)
- [Changelog](https://github.com/xanzy/go-gitlab/blob/master/releases_test.go)
- [Commits](https://github.com/xanzy/go-gitlab/compare/v0.60.0...v0.63.0)

---
updated-dependencies:
- dependency-name: github.com/xanzy/go-gitlab
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Fix name for updated dependency

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: bill-rich <bill.rich@gmail.com>
 2022-04-20 23:15:20 -07:00
Dustin Decker
879e65b5c5
Handle nil file channel (#417) 2022-04-16 15:41:51 -07:00
Bill Rich
09de492ec9
Check that required info is present (#415) 2022-04-15 16:32:16 -07:00
Bill Rich
1f55171437
Scan commit in since-commit (#416) 
* Scan commit in since-commit.

Fixes #413

* address lint issue

Co-authored-by: Dustin Decker <dustin@trufflesec.com>
 2022-04-15 15:58:54 -07:00
Bill Rich
af979d4620
Use .Line to reduce diff strangeness (#380) 2022-04-11 16:38:08 -07:00
Bill Rich
6be5ee18ff
Remove overseer warn, and don't os.Exit on git error. (#348) 2022-04-08 18:20:19 -07:00
Bill Rich
bf5c757cd3
Add head and base support to github (#346) 2022-04-08 14:32:54 -07:00
Dustin Decker
ba6ea9d42f
Improve error logging and continue when there are insufficient permissions (#315) 2022-04-07 18:33:44 -07:00
Miccah
c620a62c8c
Make remote URL metadata optional for git scanning (#313) 
* Make remote URL metadata optional for git scanning

* Use helper function in ScanUnstaged
 2022-04-07 16:32:55 -07:00
Bill Rich
d78c929385
Actually skip file (#299) 2022-04-06 09:48:40 -07:00
Bill Rich
33aa6f9cab
Log error and skip file when stat fails (#296) 2022-04-05 18:58:05 -07:00
Dustin Decker
d41d18af3a fix cli parsing and improve github user scanning error handling 2022-04-03 13:42:23 -07:00
ahrav
cedb3393d1
[THOG-128] Code cleanup/ OSS onboarding (#117) 
* Small amount of code clean up.

* Rename sem to concurrency for better readability and to remove an extra comment.

* fix stashing issue.

Co-authored-by: Ahrav Dutta <ahrav.dutta@trufflesec.com>
 2022-04-01 16:47:27 -07:00
Bill Rich
6ed01500f8
Only add trailing slash when endpoint populated (#120) 2022-04-01 11:47:59 -07:00
Bill Rich
62ed0878a5
Improve git clone error logging (#114) 2022-03-30 14:27:13 -07:00
Bill Rich
155566bbd5
make sure url ends with slash (#110) 2022-03-28 09:56:44 -07:00
steeeve
a770f643df Add placeholder for encoded resume info in SetProgressComplete 2022-03-24 12:43:36 -04:00
Bill Rich
c2b4f0bc39
Include line numbers in git metadata (#97) 
* Include line numbers in git metadata

* Update tests for fragments and line numbers
 2022-03-22 09:27:15 -07:00
Bill Rich
0ea3b93238
Use git cmd for clone (#96) 
* Use git command to clone

* Check for git command when cloning
 2022-03-22 09:26:58 -07:00
Bill Rich
5ab5c6f9d9
Only scan regular files (#87) 
* Only scan regular files

* Remove IsDirectory func
 2022-03-16 16:04:10 -07:00
Bill Rich
faf30fe8fe
Use author email (#85) 2022-03-15 17:54:03 -07:00
Bill Rich
6354b16810
Use patched go-gitdiff (#84) 2022-03-15 17:02:45 -07:00
Bill Rich
e8234c3514
Remove unused code and add git binary check (#80) 2022-03-14 17:47:18 -07:00
Dustin Decker
72c13c4b8f
Update detector process and readme (#79) 2022-03-14 17:27:14 -07:00
Bill Rich
0ee34a5be6
Use gitleaks git cmd stream (#75) 
* Switch to git cmd stream

* Fix rebase issues
 2022-03-14 17:12:58 -07:00
Bill Rich
6486c18565
Add s3 support to CLI (#76) 
* Add s3 support to CLI

* Clean up comments

Co-authored-by: Dustin Decker <dustin@trufflesec.com>
 2022-03-14 17:07:07 -07:00
Dustin Decker
794a082b6c Fix Dockerfiles, readme example, and github rate limit handling 2022-03-12 07:36:42 -08:00
Dustin Decker
b0e0c26137 update detectors 2022-03-11 10:24:36 -08:00
Dustin Decker
8d754a88d4 update dependency 2022-03-09 15:49:57 -08:00
Bill Rich
55b839fc5a
Only scan files that changed from base to head (#68) 
* Only scan files that changed from base to head

* Acknowledge ignored errors
 2022-03-08 08:48:00 -08:00
Bill Rich
2a0ece9eef
Wait until rate limit reset (#69) 2022-03-08 08:47:37 -08:00
Bill Rich
665b0bf928 Add timestamp (#61) 
* Add timestamp to sources

* Include timestamp in sources
 2022-03-04 08:39:23 -08:00
Dustin Decker
c80bd5e905 Fix linting and dogfood in CI 2022-03-04 08:39:23 -08:00
trufflesteeeve
9cb99e5aa4 578 Fix gitlab basic auth with access tokens (#54) 2022-03-04 08:39:22 -08:00
Bill Rich
c144630c54 Add parent results to ignore list (#47) 
* Add parent results to ignore list

* Force concurrency to 1 when base commit is set
 2022-03-04 08:39:22 -08:00
Bill Rich
b25295580a Actually save the normalized repos (#50) 2022-03-04 08:39:22 -08:00
Bill Rich
c742f6a816 Do not continue if semaphore can't be acquired (#49) 2022-03-04 08:39:22 -08:00
Bill Rich
56dc1b109c Check commit order, dedupe results, and support using a head commit. (#44) 
* Check commit order and support using a head commit.

* Only apply dedupe to git bases source
 2022-03-04 08:39:22 -08:00
Dustin Decker
c20e9f4732 improvements 2022-03-04 08:39:17 -08:00
Dustin Decker
77418fb3f8 module v3 2022-02-15 18:54:47 -08:00
Bill Rich
2d8756938d Fast git scanning (#40) 
* Fast git scanning

* Use original tests

* Use committer time

Co-authored-by: Bill Rich <bill.rich@trufflesec.com>
 2022-02-15 18:54:47 -08:00
Dustin Decker
152ef6d4e1 add include forks option (#37) 2022-02-15 18:54:47 -08:00
Dustin Decker
c131a6e4ae add debug pprof server and metrics server 2022-02-15 18:54:47 -08:00
Bill Rich
1fb767247f Add missing pagination on github calls (#30) 
* Add missing pagination on github calls

Includes some refactoring to improve readability and code reuse.

* Close response body and handle rate limit

* Re-include support for including users as repos to github scans

* Fix gist test to match new func signature

* Add current test name to logging

* Support username as org use case

* Also include no-auth user as org

Co-authored-by: Bill Rich <bill.rich@trufflesec.com>
 2022-02-15 18:54:47 -08:00
Bill Rich
6b183424f5 Match expected chunks to actual 2022-02-15 18:54:47 -08:00
Bill Rich
2cc34f4633 Make tests more resilliant and more coverage 2022-02-15 18:54:47 -08:00
Dustin Decker
e15fa3a5be helpful logging 2022-02-15 18:54:47 -08:00
Dustin Decker
44d113c408 Add concurrency to gitlab source integration 2022-02-15 18:54:47 -08:00
Dustin Decker
7e38e699f6 GitHub concurrency (#25) 
* GitHub scan concurrency

* Add raw result to plain output

* Fix flakey test (still flakey)

* Fix race
 2022-02-15 18:54:47 -08:00
Bill Rich
206b99704b Change log order and path filtering. 2022-02-15 18:54:47 -08:00
Dustin Decker
26184dc2cd Fix incorrect commit skipped error 2022-02-15 18:54:47 -08:00
Dustin Decker
3da3f1ec94 Add gitlab pagination support (#26) 2022-02-15 18:54:47 -08:00
Bill Rich
28ed0c3b7c Complete support for existing git scan flags (#13) 
* Add `since_commit` to git scan

* Support `max_depth` option for git scan

* Use new options in github and gitlab sources

* Address review feedback

Co-authored-by: Bill Rich <bill.rich@trufflesec.com>
 2022-02-15 18:54:47 -08:00
Dustin Decker
03ead2f7ed Integrate GitHub source 2022-02-15 18:54:47 -08:00
Dustin Decker
d6ffadb1ee
Initial docs and release automation (#5) 2022-01-18 16:59:18 -08:00
Bill Rich
a70937bfe6
Support remote git repos using https (#9) 
Co-authored-by: Bill Rich <bill.rich@trufflesec.com>
 2022-01-14 16:07:45 -08:00
Bill Rich
1422cc303a
Include and exclude paths args for gitscan (#6) 
* include and exclude paths gitscan args

Add support for include_paths and exclude_paths arguments when scanning
git sources.

* Improve variable name

Co-authored-by: Bill Rich <hrich@Bills-MacBook-Pro.local>
 2022-01-14 12:40:50 -08:00
Dustin Decker
4218c39d99
Initial CLI w/ partially implemented Git source and demo detector (#1) 2022-01-13 12:02:24 -08:00