ahrav cbf299aa77 Add gcs scanning integration (#1153) ... * Setup for GCS scanning. * Update GCS engine w/ projectID req. * Add concurrency field to gcsManager. * add errgroup to gcsManager. * Update gcs manager. * Use defautl ADC. * use ADC.' * Add TOOD. * add log to iterator completion. * use a BinaryReader instead of concrete object for channel type. * initial test for Chunks. * Add tests for chunking objects. * Add concurrency. * update metadata to include content type and acls. * Add object reading code. * Add integration test. * Add entrypoint. * Add removed wg.Wait(). * remove dead code. * remove build. * Remove period from file extension. * remove used. * Add comment. * Setup for GCS scanning. * Update GCS engine w/ projectID req. * Add concurrency field to gcsManager. * add errgroup to gcsManager. * Update gcs manager. * Use defautl ADC. * use ADC.' * Add TOOD. * add log to iterator completion. * use a BinaryReader instead of concrete object for channel type. * initial test for Chunks. * Add tests for chunking objects. * Add concurrency. * update metadata to include content type and acls. * Add object reading code. * Add integration test. * Add entrypoint. * Add removed wg.Wait(). * remove dead code. * remove build. * remove used. * Add file type for objects. * Add check for file type and size. * Add default file size. * Add additinoal auth options and remaining CLI flags. * Handle errors in go routines. * Handle resuming for buckets. * Remove redundant words in comment. * remove ok check on bool check. * remove extra blank line. * Add return if handler handles chunk. * Add comment. * remove extra blank line. * cleanup comment. * Add comment. * move up fxn. * go mod tidy. * Add exclusion to perf testing buckets. * Handle blocking the channel. * remove unused const. * fix tests. * fix tests. * Handle gcs manger options better. * update fxn name. * Remove arg name. * ignore buckets in gcsManager test. * fix test. * propulate gsManagerOpts. * inline err check. * Add readme. * update readme spelling. * fix test.		2023-03-07 17:32:04 -08:00
..
credentials.proto	Add SSH config option for the git source (#830)	2022-09-28 20:40:01 +02:00
custom_detectors.proto	Add custom detectors configuration parsing (#927)	2022-11-21 15:10:38 -06:00
detectors.proto	Implement CustomRegex detector (#950)	2022-12-14 10:26:53 -06:00
source_metadata.proto	Add gcs scanning integration (#1153)	2023-03-07 17:32:04 -08:00
sources.proto	Add gcs scanning integration (#1153)	2023-03-07 17:32:04 -08:00