* initial spike on hf
* added in user and org enum
* adding huggingface source
* updated with lint suggestions
* updated readme
* addressing resources that require org approval to access
* removing unneeded code
* updating with new error msg for 403
* deleted unused code + added resource check in main
* Add flag to get information if trufflehog being ran from TUI
Co-authored-by: mcastorina <m.castorina93@gmail.com>
* Always use version.BuildVersion
---------
Co-authored-by: mcastorina <m.castorina93@gmail.com>
* Add stub source and elastic API funcs
* Spawn workers and ship chunks
* Now successfully detects a credential
- Added tests
- Added some documentation comments
- Threaded the passed context through to all the API requests
* Linting fixes
* Add integration tests and resolve some bugs they uncovered
* Logstash -> Elasticsearch
* Add support for --index-pattern
* Add support for --query-json
* Use structs instead of string building to construct a search body
* Support --since-timestamp
* Implement additional authentication methods
* Fix some small bugs
* Refactoring to support --best-effort-scan
* Finish implementation of --best-effort-scan
* Implement scan catch-up
* Finish connecting support for nodes CLI arg
* Add some integration tests around the catchup mechanism
* go mod tidy
* Fix some linting issues
* Remove some debugging Prints
* Move off of _doc
* Remove informational Printf and add informational logging
* Remove debugging logging
* Copy the index from the outer loop as well
* Don't burn up the ES API with rapid requests if there's no work to do in subsequent scans
* No need to export UnitOfWork.AddSearch
* Use a better name for the range query variable when building the timestamp range clause in searches
* Replace some unlocking defers with explicit unlocks to make the synchronized part of the code clearer
* found -> ok
* Remove superfluous buildElasticClient method
---------
Co-authored-by: Charlie Gunyon <charlie@spectral.energy>
This is a follow-up to #2107 and #2335. It adds a new (hidden) --results flag that allows a user to show any combination of verified, unverified, and indeterminate secrets.
This PR adds the ability to exclude buckets from S3 scans. The capability is pretty rudimentary right now, and does not support globbing. If both lists are specified the source to fail to initialize.
* Add flag to write job reports to disk
* Fix nil pointer / non-nil interface bug
* Synchronize job report writer goroutine
* Log when the report has been written
* draft reverify chunks
* remove
* remove
* reduce dupe map cap
* do not verify chunk
* cli arg and use val for dupe lut
* remove counter
* skipp empty results]
* working on test and normalizing val for comparison
* forgot to save file
* optimize normalize
* reuse map
* remove print
* use levenshtein distance to check dupes
* forgot to leave in emptying map
* use slice
* small tweak
* comment
* use bytes
* praise
* use ctx logger
* add len check
* add comments
* use 8x concurrency for reverifier workers
* revert worker count
* use more workers
* process result directly for any collisions
* continue after decoder match for reverifying
* use map
* use map
* otimization and fix the bug.
* revert worker count
* better option naming
* handle identical secrets in chunks
* update comment
* update comment
* fix test
* use DetecotrKey
* rm out of scope tests and testdata
* rename all reverification elements
* don't re-write map entry
* use correct key
* rename worker, remove log val
* test likelydupe, add eq detector check in loop
* add test
* add comment
* add test
* Set verification error
* Update tests
---------
Co-authored-by: Zachary Rice <zachary.rice@trufflesec.com>
Co-authored-by: Dustin Decker <dustin@trufflesec.com>
* add tempfile creation
- break PID retrieval into sep. function
* add tmpfile cleanup func
* add file cleanup to main cleanup func
* refactor file logic to only return name string
* add temp buffer naming to gcs
* add temp buffer naming to s3
* add temp buffer naming to filesystem
* add temp buffer naming to git
* consolidate cleanup functions
- have single function handle both files and dirs
- remove interface(not needed with a single func implementation)
- change calls to `New(...)` to reflect config implementation
- simplify automation in main.go
- update disk-buffer-reader dependency
* integrate changes from pr #2133
* merge main
* checkout from main to revert conflict issues
* re-add buffer logic to git
* interface no longer needed
* move string format to global const
---------
Co-authored-by: Ahrav Dutta <ahrav.dutta@trufflesec.com>
* Add TravisCI source
* update test to use sourcestest
* Remove jobPage loop
ListByBuild does not support pagination, so this was infinitely
repeating. https://developer.travis-ci.com/resource/jobs#find
* Continue chunking on error
* review updates
* update readme
---------
Co-authored-by: Miccah Castorina <m.castorina93@gmail.com>
The Aho-Corasick wrapper we have tracks information about whether verification should be enabled on an individual detector basis, but that functionality isn't related to the matching functionality of Aho-Corasick, and including it complicates the implementation. This PR removes it to simplify some things.
This PR removes some code that supported a potential future implementation of detector-specific verification settings, but that feature has not actually been implemented yet, so there's no loss of functionality. If we want that feature we can add it back on top of this in a more separated way.
* adds func to get scannerPIDs
* add cleanup and call to get pids
* move pid handling to git module
* remove PID logic from main
* refactor testing code to handle different exec name
* cleanup linting errors
* add better logging, fix dir if clause
* some PR fixups
* mod fixup
* add interfaces for helper funcs
* refactor cleanup into main, getPID into git
* lint and test fixups, remove fail on n<2 pids
* simplify pid sorting
* use filepath.Join
* use Args[0] for exec name, fix logger
* formatting fixup
* move functionality into cleantemp pkg
* go mod fixup
* remove redundant testing comment
* fix go.sum issues
* add 15m ticker loop for cleanup
* enclose ticker in function for goroutine defer
fix cleantemp interface
* make time more readable
* add check for non-local Trufflehog PIDs
* allow deletion even if no non-local pids found
* bundle intial cleanup into runCleanup func
* add explicit regex check for tempdir format
* added PR and Issue body scanning; adjusted CLI args to fit
* removed print statement from debugging
* removed exclude-commits; adjusted CLI flags
* minor changes to match main branch
* fixing logic
* updating README for --issues and --prs
* add role assumption for s3 source
* refactor role assumption to repeatable string
user can pass array of roles to assume
* refactor s3 chunks to handle passed roleARNs
* add role-session name
use timestamp to make dynamic
* add docstring for rolearn strings()
* make sure role ars are passed into source
* refactor role assumption functionality
break s3 bucket scanning into sep. function
* add log check on assume role
* fix role iteration
- Make sure s3 struct is populated with roles
- add separate new client instantiation for role-based access
- iterates through each role
* add comment
* protobuf revert for merge
* re-run make proto
* lint cleanup
* cleanup TODOs
* drop redundant switch case in assumerole client
* use less verbose 'ctx' designator
* breakout functionality from Chunks
- separate functions for:
- enumerating buckets to scan
- scanning objects within the buckets
* remake protobuf defs
* allow scan to continue on single bucket err
* add readme docs
* minor fixups
* Init attempt at tui with bubble tea.
Co-authored-by: mcastorina <m.castorina93@gmail.com>
* Add starting and source selection options
Co-authored-by: mcastorina <m.castorina93@gmail.com>
* Rewrite models into a state machine
* Update source descriptions
* Make subpages implement tea.Model
* Rename page0 and page1 to be more descriptive
* Adjust styling and adding color consts
Co-authored-by: mcastorina <m.castorina93@gmail.com>
* Add helper generic function to call Update and type cast
* Setup plumbing for source configuration page
* Use CLI introspection for source configuration (WIP)
* Experiment with table view
* Replace table with form fields
Co-authored-by: mcastorina <m.castorina93@gmail.com>
* Change 🔒 to 💸
* Copy components from soft-serve
Co-authored-by: hxnyk <8292703+hxnyk@users.noreply.github.com>
* Copy styles from soft-serve
Co-authored-by: hxnyk <8292703+hxnyk@users.noreply.github.com>
* Copy common from soft-serve
Co-authored-by: hxnyk <8292703+hxnyk@users.noreply.github.com>
* Refactor into pages
This is still a WIP, but the main structure is there.
Co-authored-by: hxnyk <8292703+hxnyk@users.noreply.github.com>
* Trying out selector for wizard intro
Co-authored-by: mcastorina <m.castorina93@gmail.com>
* Use selector with custom View
Co-authored-by: hxnyk <8292703+hxnyk@users.noreply.github.com>
* Change Item to be an enum
Co-authored-by: hxnyk <8292703+hxnyk@users.noreply.github.com>
* Add link pages
Co-authored-by: mcastorina <m.castorina93@gmail.com>
* Update source select to use selector
Co-authored-by: mcastorina <m.castorina93@gmail.com>
* Delete source configure page and add blank tabs
Co-authored-by: hxnyk <8292703+hxnyk@users.noreply.github.com>
* Add tab placeholder pages for configurationi
Co-authored-by: mcastorina <m.castorina93@gmail.com>
* Added headers and style to each tab
Co-authored-by: hxnyk <8292703+hxnyk@users.noreply.github.com>
* Update with new sources
* Remove kingpin attribute from SourceItem
* Add basic form field and source structuring
* Hookup git form fields with an underlying textinput component
Co-authored-by: hxnyk <8292703+hxnyk@users.noreply.github.com>
* Update forms for git and github
Co-authored-by: mcastorina <m.castorina93@gmail.com>
* Add labels per text input
* Add sources and adjust styling
* add basic trufflehog configuration page
* Add skip button to textinputs component
* Emit and handle textinputs skip/submit button commands
* Don't quit when q is pressed on the sourceConfigurePage
* Build trufflehog command based on source config vals
Co-authored-by: mcastorina <m.castorina93@gmail.com>
* Build flags based on truffle config inputs
* Update summary section
* Add generated truffle fields
Co-authored-by: mcastorina <m.castorina93@gmail.com>
* update summary to correctly print info
* Go back a page when escape key is pressed
* WIP run page list
Co-authored-by: hxnyk <8292703+hxnyk@users.noreply.github.com>
* Allow running trufflehog from the run page
Co-authored-by: hxnyk <8292703+hxnyk@users.noreply.github.com>
* Add option to view help docs
Co-authored-by: mcastorina <m.castorina93@gmail.com>
* comment out unused styles and remove unused types
* Capitalize H in TruffleHog
* remove unneeded fmt.Sprintf
---------
Co-authored-by: mcastorina <m.castorina93@gmail.com>
* feat: initial support for bare repositories
* feat: use concatenation instead of formatting and os.Getenv instead of os.Environ
Signed-off-by: Savely Krasovsky <savely@krasovs.ky>
* fix: go-git update with pre-receive hooks fix
Signed-off-by: Savely Krasovsky <savely@krasovs.ky>
* fix: remove info about pre-receive hook from README.md for now
Signed-off-by: Savely Krasovsky <savely@krasovs.ky>
* fix: don't scan staged while using --bare option, fixes to make it work with the latest master
Signed-off-by: Savely Krasovsky <savely@krasovs.ky>
* fix: small refactor according to #1518
Signed-off-by: Savely Krasovsky <savely@krasovs.ky>
---------
Signed-off-by: Savely Krasovsky <savely@krasovs.ky>
* Exit with non-zero exit code on chunk source error
* Exit with a non-zero exit code whenever we hit an error getting
chunks. Previously the error would be logged but trufflehog would exit
with a 0 (success) status code.
* fix gcs test
---------
Co-authored-by: Dustin Decker <dustin@trufflesec.com>
Co-authored-by: ahrav <ahravdutta02@gmail.com>
