Cody Rose
de19a39f2c
Return targeted scan errors ( #2995 )
...
Targeted scans should return their errors so that consumers can process them. By creating a type that combines an error with a targeted secret ID, we can return these errors without having to modify the Source interface.
2024-06-21 13:50:56 -04:00
Richard Gomez
4addd81e29
test: fix compile errors ( #2964 )
2024-06-13 08:22:25 -07:00
Richard Gomez
5216142960
refactor(cache): use generics ( #2930 )
2024-06-06 13:08:00 -04:00
Richard Gomez
4d2c8c6e11
refactor(github): improve wiki err handling ( #2917 )
2024-06-05 08:06:01 -04:00
Richard Gomez
9053d8f4de
refactor(github): enumerateWithToken flow & tests ( #2880 )
2024-05-31 15:53:44 -05:00
Richard Gomez
5102e3ae11
test(github): fix some errors ( #2774 )
2024-05-24 13:03:41 -07:00
Richard Gomez
e53f5bd5c5
Improve handling of Gist URLs ( #2653 )
...
* feat(github): handle ghes gists
* fix(github): handle all gist URLs
* refactor(github): helper func to check gist urls
2024-05-24 08:36:30 -07:00
Richard Gomez
1441289d41
fix(github): scan user repos ( #2814 )
2024-05-23 09:40:40 -05:00
ahrav
896e6e7c66
upgrade github dep ( #2858 )
2024-05-16 14:35:08 -07:00
ahrav
591871977c
Correclty set metrics for enumerated orgs ( #2757 )
2024-04-29 14:26:46 -07:00
ahrav
a8132839f8
[chore] - update go-github dep manually ( #2664 )
...
* update go-github dep
* remove commented out line
2024-04-03 19:19:14 -07:00
Richard Gomez
3b58a15a84
Fix GitHub enumeration & rate-limiting logic ( #2625 )
...
This is a follow-up to #2379 .
It fixes the following issues:
GitHub API calls missing rate-limit handling
The fix for Refactor GitHub source #2379 (comment) inadvertently resulting in duplicate API calls
2024-03-29 10:29:46 -04:00
Richard Gomez
95dc8d6e16
Fix additional GitHub test errors #2614
2024-03-26 09:34:12 -04:00
Richard Gomez
9d4cf87c02
fix(github): resolve panic & test failures ( #2608 )
2024-03-22 09:49:01 -07:00
Richard Gomez
80e8a67c2d
Refactor GitHub source ( #2379 )
...
* refactor(github): cleanup logic
* fix(github): lookup wikis per-repo
* refactor(github): change scanErrs.String output
---------
Co-authored-by: Bill Rich <bill.rich@gmail.com>
2024-03-21 14:07:39 -07:00
ahrav
3da0c5e125
[feat] - Make the client configurable ( #2528 )
...
* Make the client configurable
* add comment
* add backoff option
2024-03-01 13:29:25 -08:00
Richard Gomez
b3ff12d1e9
Fix handling of GitHub ratelimit information ( #2041 )
...
This is a follow-up to #1912 , which used the headers from the response to determine rate-limiting information, instead of using the values from RateLimitError.Rate. Although that logic seemed solid, I discovered that it did not work in some circumstances. This lead to the "unexpected" path more often than intended, and periodic instances where requests would be made before the ratelimit was refreshed.
2024-02-07 09:11:12 -05:00
Richard Gomez
8e90c4e669
Scan GitHub wikis #2233
2024-01-31 10:52:24 -05:00
ahrav
9867ce8eb8
Allow for configuring the buffered file writer ( #2319 )
...
* Write large diffs to tmp files
* address comments
* Move bufferedfilewriter to own pkg
* update test
* swallow write err
* use buffer pool
* use size vs len
* use interface
* fix test
* update comments
* fix test
* Allow for configuring the buffered file writer
* remove unused
* add missing method
* remove
* remove unused
* move parser and commit struct closer to where they are used
* linter change
* fix snifftest
* address comments
* add more kvp pairs to error
* fix test
* update
* add back missing metadata fields
* address comments
* remove bufferedfile writer
* fix
* address comments
* use unint8
* update interface
* adjust interface
* fix tests
* make linter happy
* fix finalize
* address comments
* update test
* address comments
* lint
* remove guard
* fix test
* fix
* add TODO
* fix tests
2024-01-30 12:51:58 -08:00
Richard Gomez
38eb5d08e7
Improve GitHub scan logging ( #2220 )
...
* feat(github): improve scan logging
* Move metric
---------
Co-authored-by: Dustin Decker <dustin@trufflesec.com>
2024-01-25 22:11:01 -08:00
Miccah
c5af979aee
Assume unauthenticated github scans have public visibility ( #2308 )
2024-01-16 14:57:06 -08:00
ahrav
9408425cc6
[chore] - small updates ( #2288 )
...
* small updates
* fix logic
* simplify fxn
* remove errors
* use strings.EqualFold
2024-01-11 14:27:10 -08:00
ahrav
677238c96c
Extend memory cache ( #2275 )
...
* Extend memory cache to allow for configuring custom expiration and purge interval
* use any for value type
* fix test
* fix test
* address comments
* address
* make new construct more clear
* reduce duplication
* fix test
2024-01-11 08:20:37 -08:00
Dustin Decker
7d93adc1d0
Add skip archive support ( #2257 )
2023-12-22 11:55:23 -08:00
ahrav
64c7365364
add secretID to chunk ( #2242 )
2023-12-18 15:27:49 -08:00
ahrav
5c6ce693c1
[feat] - Make skipping binaries configurable ( #2226 )
...
* Make skipping binaries configurable
* remove ioutil
* fix
* address comments
* address comments
* use multi-reader
* remove print
* use const
* fix test
* fix my stupidness
2023-12-15 11:46:27 -08:00
Richard Gomez
b3040b1227
fix(github): remove unused 'members' var ( #2202 )
2023-12-14 11:53:24 -08:00
ahrav
b75991850a
[chore] - Compile regex once ( #2176 )
...
* move regex compilation out of the fxn
* missed a spot
* merge main
2023-12-07 07:26:27 -08:00
Richard Gomez
024aa056b9
chore(github): add a newline between titles and bodies ( #2124 )
2023-11-23 16:14:28 -08:00
Richard Gomez
1f502fd42c
feat(github): scan issue & pr titles ( #1899 )
2023-11-22 19:15:27 -08:00
Dustin Decker
75e869faff
Fix forks and repos counter, add metric for orgs enumerated ( #2118 )
2023-11-21 08:52:33 -08:00
Miccah
52600a897a
[chore] Replace chunks channel with ChunkReporter in git based sources ( #2082 )
...
ChunkReporter is more flexible and will allow code reuse for unit
chunking. ChanReporter was added as a way to maintain the original
channel functionality, so this PR should not alter existing behavior.
2023-11-01 09:22:44 -07:00
Miccah
b8724e87e6
Use the configured include repositories in the GitHub filter ( #1926 )
2023-10-20 19:03:28 -07:00
Richard Gomez
3acc65b2fb
chore(github): reduce comment log verbosity ( #1922 )
2023-10-20 16:16:38 -07:00
Cody Rose
7ac7fa8728
Move Github comments check to fix a test #1927
2023-10-19 19:23:55 -04:00
Richard Gomez
4b821e9732
Handle secondary GitHub ratelimits ( #1912 )
...
* fix(github): reduce visibility-related api calls
* fix(github): handle secondary ratelimits
2023-10-19 14:54:45 -04:00
Richard Gomez
6ea3a7da4a
fix(github): normalize repo cache ( #1897 )
2023-10-17 15:07:47 -07:00
ahrav
3d2490ca80
use Repositories field from conn. ( #1860 )
2023-10-04 13:56:02 -07:00
joeleonjr
699547b7d3
consolidated pr and issue descr/comment flags ( #1827 )
2023-09-27 15:54:02 -04:00
ahrav
bf47fd69bb
Github partial scan ( #1804 )
...
* Add ability for targetted partial scans of Github.
* update comment.
* add more tests.
* add additiional test.
* address comments.
2023-09-26 12:38:33 -07:00
joeleonjr
1e42dae734
added PR and Issue body scanning ( #1816 )
...
* added PR and Issue body scanning; adjusted CLI args to fit
* removed print statement from debugging
* removed exclude-commits; adjusted CLI flags
* minor changes to match main branch
* fixing logic
* updating README for --issues and --prs
2023-09-26 12:25:48 -04:00
ahrav
22876f8381
replace interface{} with any. ( #1771 )
2023-09-15 04:35:15 -07:00
Miccah
dbcb888063
Update Source interface to use SourceID and JobID types ( #1774 )
...
The previous implementation used int64 for both, which can be mixed up
easily. Using distinct types adds a layer of type safety checked by the
compiler.
2023-09-14 11:28:24 -07:00
Miccah
72b6a9ec6b
Add a SourceType constant to all source packages ( #1768 )
2023-09-12 17:23:25 -07:00
ahrav
2a9f34962d
Add optional param to Chunks ( #1747 )
...
* Add interface for targeted chunking.
* use optional args.
* update Chunks method signature.
* update tests.
* fix test.
* update QueryCriteria type.
2023-09-07 09:03:37 -07:00
ahrav
2b1b1b5ad0
Add jobID to chunk. ( #1721 )
2023-08-29 12:02:30 -07:00
ahrav
0932ea224b
[chore] - Prevent nil deref panic ( #1709 )
2023-08-26 20:39:50 -07:00
ahrav
4f4a79f62b
Support azure git links ( #1662 )
...
* Support azure git links.
* update comment.
* update test names.
2023-08-24 14:36:52 -07:00
ahrav
a2a7a2087e
[chore] - update comments and logs. ( #1654 )
...
* update comments and logs.
* Update github.go
2023-08-23 13:18:07 -07:00
Zubair Khan
fd00d2b30b
add rate limit and consumption metrics for GitHub ( #1651 )
...
* add rate limit and consumption metrics
* incrment after each repo scanned
* update repo scanned label name
2023-08-22 15:01:59 -04:00
