Richard Gomez
e0faac8d1c
Fix runtime error when scanning Gist comments ( #1552 )
...
* fix(github): fix runtime error from gist comments
* fix(github): add flag to scan Gist comments
2023-07-31 08:57:42 -05:00
Richard Gomez
46823f77c9
feat(github): clarify comment log statement ( #1553 )
2023-07-26 09:40:30 -05:00
Richard Gomez
2290954b02
fix(github): use apiEndpoint for basic or no auth ( #1454 )
2023-07-25 20:03:08 -07:00
Richard Gomez
1594fddf05
feat(git): include line in github & gitlab links ( #1466 )
2023-07-11 20:02:27 -07:00
Zubair Khan
4334af4d34
scan GitHub PR and issue comments ( #1435 )
...
* issue comment scanning
* save progress
* test
* test for pr comment and issue comment
* add pagination support
* linter stuff
* make linter happy
* remove debug log
* readd logging
* github issue resolved
* var const block and handle rate limit
* remove magic number
* make gitURLParse a public function to use more generally
* fix test bug
* make comment scanning OPT-IN
2023-07-11 15:13:33 -04:00
Miccah
f3152b6885
Implement SourceUnitUnmarshaller for all sources ( #1416 )
...
* Implement CommonSourceUnitUnmarshaller
* Add SourceUnitUnmarshaller to all sources using
All sources, with the exception of git, will use the CommonSourceUnit as
they only contain a single type of unit to scan.
* Fix method comments to adhere to Go's style guide
2023-06-23 11:15:51 -05:00
Dustin Decker
c8944825de
Surface missing git as an error during initialization ( #1362 )
2023-05-26 15:23:08 -07:00
ahrav
1da7720912
Replace context.TODO. ( #1349 )
2023-05-19 11:09:51 -07:00
ahrav
31844b12e3
[oc-313] - Add GitHub metrics ( #1324 )
...
* Normalize repos during enumeration.
* fix test.
* Add benchmark.
* Add benchmark.
* Add more realistic benchmark values.
* add gist mocks.
* Remove old normalize fxn.
* abstract away the repo cache.
* update test.
* increase repo count.
* increase page limnit to 100.
* move callee fxns below caller for Chunks.
* Add context to normalize.
* remove extra logic in normalize repo.
* Delete new.txt
* Delete old.txt
* Handle errors in a thread safe manner.
* fix test.'
* fix test.
* handle repos that are included by users.
* Abstract include ignore logic within repoCache.
* Add better comment around repoCache.
* Rename params.
* remove commented out code.
* use repos instead of items.
* remove commented out code.
* Use ++ instead of atomic increment.
* update to use logger var.
* use cache pkg.
* Use separate file for repo logic.
* Address comments.
* fix test.
* make less sucky test.
* Update test.
* Add logs for duration and repo size.
* fix integration test.
* address comment.
2023-05-16 08:45:28 -07:00
Dustin Decker
4250773e92
GitHub basic auth ( #1337 )
2023-05-15 22:04:42 -07:00
ahrav
030c093392
Fix how we scan orgs ( #1327 )
...
* Fix how we scan orgs.
* fix integration test.
2023-05-04 08:07:11 -07:00
ahrav
323c093818
Normalize GitHub repos during enumeration ( #1269 )
...
* Normalize repos during enumeration.
* fix test.
* Add benchmark.
* Add benchmark.
* Add more realistic benchmark values.
* add gist mocks.
* Remove old normalize fxn.
* abstract away the repo cache.
* update test.
* increase repo count.
* increase page limnit to 100.
* move callee fxns below caller for Chunks.
* Add context to normalize.
* remove extra logic in normalize repo.
* Delete new.txt
* Delete old.txt
* Handle errors in a thread safe manner.
* fix test.'
* fix test.
* handle repos that are included by users.
* Abstract include ignore logic within repoCache.
* Add better comment around repoCache.
* Rename params.
* remove commented out code.
* use repos instead of items.
* remove commented out code.
* Use ++ instead of atomic increment.
* update to use logger var.
* use cache pkg.
* Address comments.
* fix test.
* make less sucky test.
* Update test.
2023-05-03 08:35:53 -07:00
ahrav
a2266b4e28
add additional logging ( #1298 )
...
* add additional logging.
* update test.
* remove continue.
* address comments.
2023-04-27 16:48:04 -07:00
Miccah
c6826c4574
Fix nil scan options ( #1107 )
2023-02-14 12:09:45 -06:00
SAYGIN Metin
f2139a7615
Github filter support for exclude and include ( #1087 )
...
* test
* Add missing head and base hash back.
---------
Co-authored-by: Ahrav Dutta <ahravdutta02@gmail.com>
2023-02-14 08:40:53 -08:00
Miccah
0ce72ccda3
[chore] Remove logrus from github source ( #1086 )
...
* [chore] Remove logrus from github source
* Fix handleRateLimit test
* Fix tests
2023-02-09 18:02:04 -06:00
Dustin Decker
4ef546a06b
fix github integration tests ( #1042 )
2023-01-25 08:57:39 -08:00
Yassine Ilmi
d720c0c0f3
Switch to retryableHttpClient for GitHub AuthN API Client + More Logs ( #995 )
...
* Adding missing flags to Readme
* Use retryableHttpClient by default for GitHub
* Adding repoUrl for scanning time log
* Use WithField instead of WithFields
* Updating README with lasted --help output
2023-01-09 09:21:56 -08:00
ahrav
009756dce6
add proto that was missing. ( #986 )
2022-12-23 13:27:07 -08:00
ahrav
936a139596
Allow using a glob for include list. ( #977 )
...
* Allow using a glob for include list.
* Update command flag.
* Make comment more clear.
* update comment.
* Allow scanning repo and org at the same time.
2022-12-16 13:28:16 -08:00
Dustin Decker
7de9bdd12d
Support globbing with ignore repos ( #967 )
2022-12-09 12:10:42 -08:00
ahrav
a72b9feb35
Only scan org with --org flag. ( #931 )
2022-12-06 16:18:48 -08:00
Bill Rich
9f99ee470d
Integration test fixes ( #956 )
...
* Adjust repo count for new app
* Fix chunk test count
2022-12-06 08:42:24 -08:00
ahrav
dd141fb55f
[oc-147] - Add context to all git methods ( #901 )
...
* Add context to all git methods.
* remove logrus.
* Add ctx.
* Address comments.
* Add error to clone failing.
* Return error.
2022-11-03 16:36:52 -07:00
ahrav
fe1e475a04
Prevent concurrent read and writes to visibility map. ( #892 )
2022-11-01 16:20:59 -07:00
ahrav
46bc010165
Add tests for including github repos. ( #854 )
2022-10-21 07:56:36 -07:00
trufflesteeeve
fb56b9f713
Check rate limit when getting github user ( #855 )
...
Also, don't fetch a github user or their token when both are known. This
currently only affects the Github Token auth type. Github App
installations will continually fetch tokens every time we clone a repo.
In the future we should check the `ExpiresAt` field of the Github App
token and determine if we need to fetch a new one at that point.
2022-10-20 18:14:28 -04:00
ahrav
029519eb01
[THOG-767] ignore gitlab repos ( #853 )
...
* Add ability to ignore repos.
* use std library slices.Contains.
* Add tests.
* Remove zero values from test.
2022-10-19 13:55:44 -07:00
ahrav
2d6aadcb46
[THOG-774] - GitHub ignore repo full name ( #848 )
...
* Use github repo full name.
* fix tests.
2022-10-14 09:20:49 -07:00
ahrav
04c9bb535e
[THOG-768] - Add ability to skip scanning Github repos ( #846 )
...
* Add ability to skip scanning Github repos.
* remove old change.
* rename method.
2022-10-12 16:28:24 -07:00
Miccah
891996f546
Do not fail scanning if we cannot enumerate gists ( #826 )
2022-09-27 20:59:10 +02:00
Bill Rich
1c00014051
Include public/private in github metadata ( #812 )
...
* Include public/private in github metadata
* CR feedback
* Fix typos and naming
2022-09-26 14:55:46 -07:00
Dustin Decker
97a73710de
403 on listing user gist should not fail org scan ( #822 )
2022-09-26 14:37:25 -07:00
trufflesteeeve
63fcf33ce6
Fix improper github org member pagination ( #814 )
...
I'm not sure I fully understand why this issue exists. But I think the
short version is this: When we attempted to paginate users, we would set
a variable's Page value. But that variable appears to not actually be a
pointer, despite being added as one. It probably has to do with how
struct embedding works. Either way, if we make the overall options
variable the whole thing, and update its embedded struct with our page
variable, everything works out.
2022-09-21 16:22:42 -07:00
Dustin Decker
335e676caa
Provide user when during private clones with token and fix integration tests ( #811 )
2022-09-19 15:53:21 -07:00
Bill Rich
593f1e6754
Include apiClient in Github source ( #804 )
2022-09-19 14:31:48 -07:00
trufflesteeeve
945de06858
Fix include-members not working on github ( #773 )
2022-09-12 13:26:38 -04:00
Dustin Decker
ecfdb0105b
Provide correct username for app cloning and add integration test ( #786 )
2022-09-08 17:41:53 -07:00
Dustin Decker
80b247286b
Improve GitHub debug logging ( #784 )
...
* close bodies early
* add more debug logging to github
* fix nil check
* Add nil checks for response
2022-09-08 12:23:40 -07:00
ahrav
7ba583ca40
[THOG-681] - Handle errors sources ( #783 )
...
* Handle errors w/ github source.
* Fix loop var captured by func literal.
* Fix loop var captured by func literal.
* Set completed progress if the scan completes with no errors.
* Set progress to 100% if the scope and iteration are both 0.
* Fix commentary.
* Fix test.
* Return after the defer to os.RemoveAll.
* Fix unauth scan.
* Inline range loop.
* update tests for partial scan completion with errors. Ensure correct progress is set.
* Update progress for all sources.
* Update github test.
* Address comments.
2022-09-07 19:40:37 -07:00
Dustin Decker
fa9479100e
Add common sentry recover library and add into goroutines ( #738 )
...
* Add common sentry recover library and add into goroutines
* fix nits
2022-08-29 11:45:37 -07:00
ahrav
dcc102a81c
[Thog-371] Utilize config struct for engine scans ( #700 )
...
* Use a config struct when scanning and engine source.
* fix tests.
* Move test_helpers to the sources pkg.
* Handle ScanGit error in tests.
* adderss comments.
* Use functional options.
* Remove temp var.
* Add better var names for the setup functions for each config.
* Remove unused var.
* fix error logs.
* fix error logs.
* single line.
* remove blank lines.
2022-08-10 10:11:13 -07:00
ahrav
30ebe84e3e
[THOG-608] - Fix linter errors. ( #701 )
...
* Fix linter errors.
* Fix gist adding test.
* Update test string for mock JSON reply.
* Remove if.
2022-08-09 19:20:02 -07:00
trufflesteeeve
e793f4a5e6
Properly count the number of repos after a github scan resume ( #625 )
2022-06-17 16:21:22 -04:00
trufflesteeeve
10f4d02c31
Allow gitlab to resume from encoded resume info ( #611 )
2022-06-17 11:45:17 -04:00
Dustin Decker
9bcddbc45a
Change GHE org enum to use since ID instead of pages ( #618 )
...
* Change GHE org enum to use since ID instead of pages
* fix logging
2022-06-09 15:09:13 -07:00
Dustin Decker
8051b03bbf
improve debug logging for GHE enum ( #615 )
2022-06-08 13:56:07 -07:00
Dustin Decker
1a12a25f4d
Enumerate all visible orgs in GHE ( #612 )
2022-06-07 09:24:31 -07:00
Dustin Decker
e3bbf293e2
Fix NPD on mutex ( #609 )
...
* Fix NPD on mutex
* fix test
2022-06-06 17:20:27 -07:00
Miccah
9074006695
Fix bug in GitHub unit test mocking ( #608 )
2022-06-06 16:58:34 -07:00
