Commit graph

14 commits

Author SHA1 Message Date
ahrav
3da0c5e125
[feat] - Make the client configurable (#2528)
* Make the client configurable

* add comment

* add backoff option
2024-03-01 13:29:25 -08:00
Miccah
dbcb888063
Update Source interface to use SourceID and JobID types (#1774)
The previous implementation used int64 for both, which can be mixed up
easily. Using distinct types adds a layer of type safety checked by the
compiler.
2023-09-14 11:28:24 -07:00
Miccah
72b6a9ec6b
Add a SourceType constant to all source packages (#1768) 2023-09-12 17:23:25 -07:00
ahrav
2a9f34962d
Add optional param to Chunks (#1747)
* Add interface for targeted chunking.

* use optional args.

* update Chunks method signature.

* update tests.

* fix test.

* update QueryCriteria type.
2023-09-07 09:03:37 -07:00
ahrav
2b1b1b5ad0
Add jobID to chunk. (#1721) 2023-08-29 12:02:30 -07:00
ahrav
13999227b9
Use common chunk reader (#1596)
* Add common chunker.

* add comment.

* use better config name.

* Add common chunk reader to s3.

* Add common chunk reader to git, gcs, circleci.

* revert gcs.

* revert gcs.

* fix chunker.

* revert gcs.

* update cancellablewrite.

* revert impl.

* update to remove totalsize.

* Fix my goof.

* Use unified struct in chunkreader.

* return err instead of logging and returning.

* rename error to err.

* only send single ChunkResult even if there is an error and chunkBytes.

* fix logic.
2023-08-07 12:55:28 -07:00
Miccah
f3152b6885
Implement SourceUnitUnmarshaller for all sources (#1416)
* Implement CommonSourceUnitUnmarshaller

* Add SourceUnitUnmarshaller to all sources using

All sources, with the exception of git, will use the CommonSourceUnit as
they only contain a single type of unit to scan.

* Fix method comments to adhere to Go's style guide
2023-06-23 11:15:51 -05:00
Dustin Decker
1db22599af
update circle test because workflows expire and need re-running (#1251) 2023-04-10 16:21:19 -07:00
Miccah
d317ddb51a
[chore] Remove logrus from circleci, filesystem, gitlab, and s3 sources (#1089)
* [chore] Remove logrus from circleci, filesystem, gitlab, and s3 sources

* Address comments
2023-02-10 11:02:55 -06:00
ahrav
e47cc2451f
Dont pre-allocate errors slice. (#1083) 2023-02-08 17:33:30 -08:00
ahrav
8be89a593b
Handle errors in a thread safe manner (#1052)
* Handle errors in a thread safe manner.

* fix test.

* fix linter.

* address comments.
2023-02-02 11:05:33 -08:00
ahrav
1621403e11
Add concurrency to CircleCi source (#1029)
* Small cleanup of CircleCi source.

* Add concurrency to circleci.

* merge w/ cleanup branch.

* Rdefine loop var.

* Delete github.go

* reverge file delete.

* Add debug log for scan errors.

* make collecting scanned errors thread safe.

* pre-allocate errors slice.
2023-01-17 12:24:49 -08:00
ahrav
319ae64a02
[chore] - Small cleanup of CircleCi source (#1028)
* Small cleanup of CircleCi source.

* address comments.

* Add context to methods as first param.
2023-01-17 09:36:18 -08:00
Dustin Decker
5f6143f09a
Add Circle CI source (#997)
* Add Circle CI source

* remove SHA1 line

* remove trim
2023-01-05 21:44:37 -08:00