Commit graph

115 commits

Author SHA1 Message Date
Dustin Decker
752c848640
Show clone path for git repos (#823) 2022-09-26 14:36:55 -07:00
Bill Rich
e3107ad6bb
Move head and base normalization to source (#818) 2022-09-23 08:58:45 -07:00
ahrav
92f40c2031
[THOG-709] - Recover from detector panics (#810) 2022-09-22 07:01:10 -07:00
trufflesteeeve
63fcf33ce6
Fix improper github org member pagination (#814)
I'm not sure I fully understand why this issue exists. But I think the
short version is this: When we attempted to paginate users, we would set
a variable's Page value. But that variable appears to not actually be a
pointer, despite being added as one. It probably has to do with how
struct embedding works. Either way, if we make the overall options
variable the whole thing, and update its embedded struct with our page
variable, everything works out.
2022-09-21 16:22:42 -07:00
Bill Rich
509cf8b6fa
Use headref and check empty commits for base (#815) 2022-09-21 16:04:01 -07:00
Dustin Decker
335e676caa
Provide user when during private clones with token and fix integration tests (#811) 2022-09-19 15:53:21 -07:00
Bill Rich
593f1e6754
Include apiClient in Github source (#804) 2022-09-19 14:31:48 -07:00
trufflesteeeve
945de06858
Fix include-members not working on github (#773) 2022-09-12 13:26:38 -04:00
Bill Rich
912d8e461d
Add context so to avoid splitting creds. (#791)
* Add context so to avoid splitting creds.

* Add context newlines to expected results
2022-09-09 15:00:33 -07:00
Dustin Decker
ecfdb0105b
Provide correct username for app cloning and add integration test (#786) 2022-09-08 17:41:53 -07:00
Dustin Decker
80b247286b
Improve GitHub debug logging (#784)
* close bodies early

* add more debug logging to github

* fix nil check

* Add nil checks for response
2022-09-08 12:23:40 -07:00
ahrav
7ba583ca40
[THOG-681] - Handle errors sources (#783)
* Handle errors w/ github source.

* Fix loop var captured by func literal.

* Fix loop var captured by func literal.

* Set completed progress if the scan completes with no errors.

* Set progress to 100% if the scope and iteration are both 0.

* Fix commentary.

* Fix test.

* Return after the defer to os.RemoveAll.

* Fix unauth scan.

* Inline range loop.

* update tests for partial scan completion with errors. Ensure correct progress is set.

* Update progress for all sources.

* Update github test.

* Address comments.
2022-09-07 19:40:37 -07:00
Bill Rich
41936169c7
Use gitparse for unstaged changes. (#775) 2022-09-03 18:01:36 -07:00
Bill Rich
d11ce27f33
Use correct reader in filesystem source (#756) 2022-08-30 10:24:52 -07:00
Dustin Decker
fa9479100e
Add common sentry recover library and add into goroutines (#738)
* Add common sentry recover library and add into goroutines

* fix nits
2022-08-29 11:45:37 -07:00
Bill Rich
0ddd49a1b8
Use file handler and common chunker (#707) 2022-08-23 16:35:52 -07:00
Haz
4cc3529bc5
Added support for SSH URIs (#725) 2022-08-23 16:34:34 -07:00
Bill Rich
a0d44a39f1
Use trufflesec git parser (#729)
* Use trufflesec git parser.

* wip

* Fix line numbers and linter feedback
2022-08-23 13:29:20 -07:00
Bill Rich
5ad3bbde37
Use pointer to config (#715) 2022-08-16 09:15:25 -07:00
ahrav
73f9d3f0a0
[chore] - Use config struct instead of pointer for engine scans. (#709)
* Use a config struct instead of pointer when scanning engine sources.

* use config.
2022-08-12 09:56:24 -07:00
Bill Rich
4a93e49eea
Support scanning binary files in git sources (#684)
* Scan binary files for git sources

* Create data chunks in for loop

* Linter feedback and newline commit result

* Use disk buffered reader and chunker function
2022-08-10 16:10:45 -07:00
Bill Rich
a473b9aa99
Use re-readable reader and common chunker (#703)
* Use re-readable reader and common chunker

* Linter feedback

* Break on error
2022-08-10 15:32:49 -07:00
ahrav
dcc102a81c
[Thog-371] Utilize config struct for engine scans (#700)
* Use a config struct when scanning and engine source.

* fix tests.

* Move test_helpers to the sources pkg.

* Handle ScanGit error in tests.

* adderss comments.

* Use functional options.

* Remove temp var.

* Add better var names for the setup functions for each config.

* Remove unused var.

* fix error logs.

* fix error logs.

* single line.

* remove blank lines.
2022-08-10 10:11:13 -07:00
ahrav
30ebe84e3e
[THOG-608] - Fix linter errors. (#701)
* Fix linter errors.

* Fix gist adding test.

* Update test string for mock JSON reply.

* Remove if.
2022-08-09 19:20:02 -07:00
Bill Rich
7273dc9058
Archive decoder (#683)
* Archive decoder

* Fix reader handling

* Seek error handling

* Add tests

* Fix extra empty chunk

* Sync chunk size
2022-08-02 20:36:21 -07:00
ahrav
21e1ff4a8a
Fix the order to correctly match the params in NewGit. (#676) 2022-07-28 13:23:45 -07:00
trufflesteeeve
176552b07a
Fix commit attribution, git tests, and run make protos (#667)
* Update dependency to fix commit attribution, fix git tests

* Run make protos to match code with current proto definitions
2022-07-25 11:44:15 -04:00
trufflesteeeve
96106563a9
Remove git fragment trace (#656)
The fragment trace was a bit too verbose even at the trace level. We may
want to trace the file being chunked or something like that, but not the
entire diff.
2022-07-14 13:13:23 -04:00
trufflesteeeve
e793f4a5e6
Properly count the number of repos after a github scan resume (#625) 2022-06-17 16:21:22 -04:00
trufflesteeeve
10f4d02c31
Allow gitlab to resume from encoded resume info (#611) 2022-06-17 11:45:17 -04:00
Dustin Decker
2178f1f42e reword and fix error logging 2022-06-13 16:14:22 -07:00
trufflesteeeve
e123e9f177
Cleanup individual repositories after scanning (#614) 2022-06-10 14:00:50 -04:00
Dustin Decker
9bcddbc45a
Change GHE org enum to use since ID instead of pages (#618)
* Change GHE org enum to use since ID instead of pages

* fix logging
2022-06-09 15:09:13 -07:00
Dustin Decker
8051b03bbf
improve debug logging for GHE enum (#615) 2022-06-08 13:56:07 -07:00
Dustin Decker
1a12a25f4d
Enumerate all visible orgs in GHE (#612) 2022-06-07 09:24:31 -07:00
Dustin Decker
e3bbf293e2
Fix NPD on mutex (#609)
* Fix NPD on mutex

* fix test
2022-06-06 17:20:27 -07:00
Miccah
9074006695
Fix bug in GitHub unit test mocking (#608) 2022-06-06 16:58:34 -07:00
trufflesteeeve
fd79a367f1
Allow github to resume from encoded resume info (#601) 2022-06-06 12:08:57 -04:00
Miccah
fc18a5ae0c
Bug fix and add authentication in shallow clone (#595) 2022-05-31 20:45:28 -05:00
Miccah
67ad2f2247
Shallow clone if --since-commit is provided (#564)
* Shallow clone if --since-commit is provided

* Set the user before constructing args

* Fix vbout detector

* Address PR comments

* Use a better name for timestamp
* Use net.URL.String method for the remote path
2022-05-24 10:49:03 -05:00
ahrav
2051fe14ff
remove profililing. (#567) 2022-05-23 11:05:39 -07:00
ahrav
d2605354fe
[THOG-332 ]Remove TokenSource interface from the init method of Source. (#539)
* Remove TokenSource interface from the init method of Source.

* Remove proto message.

* Remove proto message.

* Fix tests.

* Fix filesystem test.
2022-05-13 14:35:06 -07:00
ahrav
b0d79180f6
[THOG-314] Add new parameter to the Init method for the source interface. (#529)
* Add new parameter to the Init method for the source interface.

* Add Oauth Token service.

* remove .test file.

* remove .test file.

* Fix param spelling.

* fix tests with new param in init

* Add missing gock lib.
2022-05-10 11:11:43 -07:00
ahrav
e12432cef8
[THOG-315] Replace bytes.buffer with strings.builder. (#533)
* Replace bytes.buffer with string.builder.

* Remove profiling.

* Remove detector changes.

* ignore .test files.

* fix detectors removed.
2022-05-09 17:02:46 -07:00
Miccah
edaf1e1fd3
Move GitHub integration tests behind a build flag and add unit tests (#527)
* Add unit tests and refactor some logic

* Move integration tests to a separate file behind a build flag

* Fix bugs in normalizeRepos

* Address lint errors

* Sort slices before comparing because order doesn't matter
2022-05-09 08:31:00 -07:00
Miccah
85208606bb
Reorganize GitHub source (#517)
* Reorganize GitHub source

This breaks up the Chunks method into smaller sub-method calls to help
organize and better understand the logic flow. No logic has been
modified (except one obvious bug), just shuffling code around.

* Check errors and revert bug fix
2022-05-06 05:00:46 -07:00
Bill Rich
212aa9ba1e
Disable tests that take too long (#524) 2022-05-04 16:37:37 -07:00
Bill Rich
c78120e56f
Syslog source (#500)
* Add syslog source

* only load cert/key with tls

* Cleanup

* Linting

Co-authored-by: Bill Rich <bill.rich@trufflesec.com>
2022-05-04 15:08:11 -07:00
Miccah
71442320ec
Chunk orgs the same when authenticated as unauthenticated (#501)
Also debug log the amount of forks we find in addReposByOrg.
2022-05-02 17:26:01 -07:00
ahrav
1bd5e9d9d2
fix merge conflicts. (#497) 2022-05-02 15:04:05 -07:00