trufflehog/pkg/sources
ahrav 570cec7565
[refactor] - Refactor Archive Handling Logic (#2703)
* Remove specialized handler and archive struct and restructure handlers pkg.

* Refactor RPM archive handlers to use a library instead of shelling out

* make rpm handling context aware

* update test

* Refactor AR/deb archive handler to use an existing library instead of shelling out

* Update tests

* add max size check

* add filename and size to context kvp

* move skip file check and is binary check before opening file

* fix test

* preserve existing funcitonality of not handling non-archive files in HandleFile

* Adjust check for rpm/deb archive type

* add additional deb mime type

* update comment

* Remove specialized handler and archive struct and restructure handlers pkg.

* Refactor RPM archive handlers to use a library instead of shelling out

* make rpm handling context aware

* update test

* Refactor AR/deb archive handler to use an existing library instead of shelling out

* Update tests

* add max size check

* add filename and size to context kvp

* move skip file check and is binary check before opening file

* fix test

* preserve existing funcitonality of not handling non-archive files in HandleFile

* Adjust check for rpm/deb archive type

* add additional deb mime type

* update comment

* go mod tidy

* update go mod

* go mod tidy

* add comment

* update max depth check to >

* go mod tidy

* rename

* [refactor] - Refactor Archive Handling Logic - Part 4: Non-Archive Data Handling and Cleanup (#2704)

* Handle non-archive data within the DefaultHandler

* make structs and methods private

* Remove non-archive data handling within sources

* Handle non-archive data within the DefaultHandler

* rebase

* Remove non-archive data handling within sources

* add gzip

* move diskbuffered rereader setup into handler pkg

* remove DiskBuffereReader creation logic within sources

* move rewind closer

* reduce log verbosity

* make defaultBufferSize a const

* use correct reader

* address comments

* update test

* [feat] - Add Prometheus Metrics for File Handlers (#2705)

* add metrics for file handling

* add metrics for errors

* add metrics for file handling

* add metrics for errors

* fix tests

* add metrics for max archive depth and skipped files

* update error

* skip symlinks and dirs

* update err

* fix err assignment

* add metrics for file handling

* add metrics for errors

* fix tests

* rebase

* add metrics for errors

* add metrics for max archive depth and skipped files

* update error

* skip symlinks and dirs

* update err

* fix err assignment

* rebase

* remove

* update metric to ms

* update comments

* address comments

* reduce indentations

* add metrics for archive depth

* [bug] - Enhanced Archive Handling to Address Interface Constraints (#2710)

* add metrics for file handling

* add metrics for errors

* add metrics for file handling

* add metrics for errors

* fix tests

* add metrics for max archive depth and skipped files

* update error

* skip symlinks and dirs

* update err

* Address incompatible reader to openArchive

* remove nil check

* fix err assignment

* wrap compReader with DiskbufferReader

* add metrics for file handling

* add metrics for errors

* fix tests

* rebase

* add metrics for errors

* add metrics for max archive depth and skipped files

* update error

* skip symlinks and dirs

* update err

* fix err assignment

* rebase

* remove

* update metric to ms

* update comments

* address comments

* reduce indentations

* replace diskbuffereader with bufferedfilereader

* updtes

* add metric back

* [bug] -  Fix bug and simplify git cat-file command execution and output handling (#2719)

* add metrics for file handling

* add metrics for errors

* add metrics for file handling

* add metrics for errors

* fix tests

* add metrics for max archive depth and skipped files

* update error

* skip symlinks and dirs

* update err

* Address incompatible reader to openArchive

* remove nil check

* fix err assignment

* Allow git cat-file blob to complete before trying to handle the file

* wrap compReader with DiskbufferReader

* Allow git cat-file blob to complete before trying to handle the file

* updates

* revert stuff

* update test

* remove

* add metrics for file handling

* add metrics for errors

* fix tests

* rebase

* add metrics for errors

* add metrics for max archive depth and skipped files

* update error

* skip symlinks and dirs

* update err

* fix err assignment

* rebase

* remove

* update metric to ms

* update comments

* address comments

* reduce indentations

* inline
2024-05-10 11:36:06 -07:00
..
circleci [feat] - Make the client configurable (#2528) 2024-03-01 13:29:25 -08:00
docker Update Source interface to use SourceID and JobID types (#1774) 2023-09-14 11:28:24 -07:00
filesystem [refactor] - Refactor Archive Handling Logic (#2703) 2024-05-10 11:36:06 -07:00
gcs [refactor] - Refactor Archive Handling Logic (#2703) 2024-05-10 11:36:06 -07:00
git [refactor] - Refactor Archive Handling Logic (#2703) 2024-05-10 11:36:06 -07:00
github Correclty set metrics for enumerated orgs (#2757) 2024-04-29 14:26:46 -07:00
gitlab [chore] Add some happy path logs to GitLab (#2765) 2024-04-29 16:42:35 -07:00
postman fix for infinite recursion in Postman var sub (#2780) 2024-05-02 13:03:03 -05:00
s3 [refactor] - Refactor Archive Handling Logic (#2703) 2024-05-10 11:36:06 -07:00
syslog 1833 Fix syslog udp (#1835) 2024-01-08 09:59:48 -08:00
travisci [feat] - Make the client configurable (#2528) 2024-03-01 13:29:25 -08:00
chunker.go Fix goroutine leak (#2251) 2023-12-20 21:09:05 -08:00
chunker_test.go Fix goroutine leak (#2251) 2023-12-20 21:09:05 -08:00
errors.go fix(github): resolve panic & test failures (#2608) 2024-03-22 09:49:01 -07:00
errors_test.go Fix additional GitHub test errors #2614 2024-03-26 09:34:12 -04:00
job_progress.go pkg: fix function names in comment (#2761) 2024-04-29 11:21:26 -05:00
job_progress_hook.go Add Display method to SourceUnit and Kind member to the CommonSourceUnit (#2450) 2024-02-20 11:24:13 -08:00
job_progress_test.go Add Display method to SourceUnit and Kind member to the CommonSourceUnit (#2450) 2024-02-20 11:24:13 -08:00
legacy_reporters.go [chore] Rename file to legacy_reporters.go (#2406) 2024-02-09 18:17:46 -08:00
metrics.go Refactor UnitHook to block the scan if finished metrics aren't handled (#2309) 2024-02-08 14:50:58 -08:00
mock_job_progress_test.go replace interface{} with any. (#1771) 2023-09-15 04:35:15 -07:00
resume.go [chore] Remove logrus from circleci, filesystem, gitlab, and s3 sources (#1089) 2023-02-10 11:02:55 -06:00
resume_test.go [chore] Remove logrus from circleci, filesystem, gitlab, and s3 sources (#1089) 2023-02-10 11:02:55 -06:00
source_manager.go set default buffer size to 64 (#2778) 2024-05-03 08:42:18 -07:00
source_manager_test.go increase test chan size (#2797) 2024-05-07 11:11:11 -04:00
source_unit.go Add Display method to SourceUnit and Kind member to the CommonSourceUnit (#2450) 2024-02-20 11:24:13 -08:00
sources.go Postman Source (#2579) 2024-03-20 11:36:20 -05:00
sources_test.go [chore] - Update Chunk struct comment (#2317) 2024-01-20 13:31:27 -08:00
test_helpers.go fix github integration tests (#1042) 2023-01-25 08:57:39 -08:00