#1711 inadvertently removed the ability to match multiple custom detectors, or multiple detectors of the same type but different version, to a given keyword. (#2060 re-added support for multiple versions of detectors globally, and #2064 re-added support for multiple custom detectors globally, but neither fixed trufflehog's inability to support multiple such detectors for a given keyword match.) This PR re-adds the removed functionality (and narrows the AhoCorasickCore interface in the process.)
* fixed regex for domain and fixed tests
* fixed regex
* fixed an issue with regex subgrouping
* made recommended changes
* made recommended changed
* fixed RawV2
ChunkReporter is more flexible and will allow code reuse for unit
chunking. ChanReporter was added as a way to maintain the original
channel functionality, so this PR should not alter existing behavior.
We're seeing secrets of this type flap between verified and unverified, which is expected behavior for multipart secrets without RawV2 defined. This PR adds RawV2 for secrets of this type.
* adding azure redis, and fixing the old detector to support ssl too
* fix?
* other way
---------
Co-authored-by: Chair <chair@Chairs-MacBook-Pro.local>
Co-authored-by: Zachary Rice <zachary.rice@trufflesec.com>
* [chore] Fix SourceManager flaky test
Sorting by EndTime is not deterministic, however sorting by StartTime
should be. StartTime is set in a goroutine that's limited by
WithConcurrentUnits, so it should happen in order that the units are
received.
* Sort by unit ID
#1711 accidentally removed the ability to support multiple custom detectors. This PR partially adds back this capability: Multiple custom detectors are now supported overall, but only one custom detector can be returned for a given keyword match.
* Add TravisCI source
* update test to use sourcestest
* Remove jobPage loop
ListByBuild does not support pagination, so this was infinitely
repeating. https://developer.travis-ci.com/resource/jobs#find
* Continue chunking on error
* review updates
* update readme
---------
Co-authored-by: Miccah Castorina <m.castorina93@gmail.com>
The Aho-Corasick wrapper we have tracks information about whether verification should be enabled on an individual detector basis, but that functionality isn't related to the matching functionality of Aho-Corasick, and including it complicates the implementation. This PR removes it to simplify some things.
This PR removes some code that supported a potential future implementation of detector-specific verification settings, but that feature has not actually been implemented yet, so there's no loss of functionality. If we want that feature we can add it back on top of this in a more separated way.
* adds func to get scannerPIDs
* add cleanup and call to get pids
* move pid handling to git module
* remove PID logic from main
* refactor testing code to handle different exec name
* cleanup linting errors
* add better logging, fix dir if clause
* some PR fixups
* mod fixup
* add interfaces for helper funcs
* refactor cleanup into main, getPID into git
* lint and test fixups, remove fail on n<2 pids
* simplify pid sorting
* use filepath.Join
* use Args[0] for exec name, fix logger
* formatting fixup
* move functionality into cleantemp pkg
* go mod fixup
* remove redundant testing comment
* fix go.sum issues
* add 15m ticker loop for cleanup
* enclose ticker in function for goroutine defer
fix cleantemp interface
* make time more readable
* add check for non-local Trufflehog PIDs
* allow deletion even if no non-local pids found
* bundle intial cleanup into runCleanup func
* add explicit regex check for tempdir format
* Detector-Competition-Fix: Fix/Remove DataFire, API retired
* Detector-Competition-Fix: Depreciate Datafire Proto
* make protos for deprecating datafire
---------
Co-authored-by: āh̳̕mͭͭͨͩ̐e̘ͬ́͋ͬ̊̓͂d <13666360+0x1@users.noreply.github.com>
Co-authored-by: ahmed <ahmed.zahran@trufflesec.com>
* loggly detector
* fixed the loggly_test.go
* fixed the test file to pass the test
---------
Co-authored-by: dsingdev-rocketx <bughunter00@protonmail.com>
* Add UnitHook and NoopHook implementations
The UnitHook tracks metrics per unit of a job, and emits them on a
channel once finished. It should work even if the Source does not
support source units.
* Refactor channel to use an LRU cache instead
An LRU cache has a more favorable failure mode than the channel. With
the channel, if the consumer stopped consuming metrics, scanning would
block. With the LRU cache, metrics will be dropped when space runs out
and a log message emitted.
* pre filter detectors that include the keywords in the chunk.
* Optimize the engine to prevent iterating overing all detectors.
* use sync.Map for concurrent access.
* lint.
* use correct verify.
* allow versioned detectors.
* Break apart Start.
* cleanup.
* Update benchmark.
* add comment.
* remove Engine prefix.
* update comments.
* use regular map.
* delete the pool.
* remove old code.
* refactor ahocorasickcore into own file.
* update comments
* move structs to ahocorasickcore
* update comments
* fix
* address comments
* exported some methods and constructor since it will need to be be used by the enterprise pipeline as well
* remove extra log
* Add generic glob filter
* Make nil filters safe
* Include glob in error
* Use better example for exclude and include test
* Allow user to configure the ambiguous case
* Rename Pass to ShouldInclude and invert logic
* Test default *Filter and Filter have the same behavior of allow
* Add property based tests
* Remove configuration for the not found ambiguous case
