* Init attempt at tui with bubble tea.
Co-authored-by: mcastorina <m.castorina93@gmail.com>
* Add starting and source selection options
Co-authored-by: mcastorina <m.castorina93@gmail.com>
* Rewrite models into a state machine
* Update source descriptions
* Make subpages implement tea.Model
* Rename page0 and page1 to be more descriptive
* Adjust styling and adding color consts
Co-authored-by: mcastorina <m.castorina93@gmail.com>
* Add helper generic function to call Update and type cast
* Setup plumbing for source configuration page
* Use CLI introspection for source configuration (WIP)
* Experiment with table view
* Replace table with form fields
Co-authored-by: mcastorina <m.castorina93@gmail.com>
* Change 🔒 to 💸
* Copy components from soft-serve
Co-authored-by: hxnyk <8292703+hxnyk@users.noreply.github.com>
* Copy styles from soft-serve
Co-authored-by: hxnyk <8292703+hxnyk@users.noreply.github.com>
* Copy common from soft-serve
Co-authored-by: hxnyk <8292703+hxnyk@users.noreply.github.com>
* Refactor into pages
This is still a WIP, but the main structure is there.
Co-authored-by: hxnyk <8292703+hxnyk@users.noreply.github.com>
* Trying out selector for wizard intro
Co-authored-by: mcastorina <m.castorina93@gmail.com>
* Use selector with custom View
Co-authored-by: hxnyk <8292703+hxnyk@users.noreply.github.com>
* Change Item to be an enum
Co-authored-by: hxnyk <8292703+hxnyk@users.noreply.github.com>
* Add link pages
Co-authored-by: mcastorina <m.castorina93@gmail.com>
* Update source select to use selector
Co-authored-by: mcastorina <m.castorina93@gmail.com>
* Delete source configure page and add blank tabs
Co-authored-by: hxnyk <8292703+hxnyk@users.noreply.github.com>
* Add tab placeholder pages for configurationi
Co-authored-by: mcastorina <m.castorina93@gmail.com>
* Added headers and style to each tab
Co-authored-by: hxnyk <8292703+hxnyk@users.noreply.github.com>
* Update with new sources
* Remove kingpin attribute from SourceItem
* Add basic form field and source structuring
* Hookup git form fields with an underlying textinput component
Co-authored-by: hxnyk <8292703+hxnyk@users.noreply.github.com>
* Update forms for git and github
Co-authored-by: mcastorina <m.castorina93@gmail.com>
* Add labels per text input
* Add sources and adjust styling
* add basic trufflehog configuration page
* Add skip button to textinputs component
* Emit and handle textinputs skip/submit button commands
* Don't quit when q is pressed on the sourceConfigurePage
* Build trufflehog command based on source config vals
Co-authored-by: mcastorina <m.castorina93@gmail.com>
* Build flags based on truffle config inputs
* Update summary section
* Add generated truffle fields
Co-authored-by: mcastorina <m.castorina93@gmail.com>
* update summary to correctly print info
* Go back a page when escape key is pressed
* WIP run page list
Co-authored-by: hxnyk <8292703+hxnyk@users.noreply.github.com>
* Allow running trufflehog from the run page
Co-authored-by: hxnyk <8292703+hxnyk@users.noreply.github.com>
* Add option to view help docs
Co-authored-by: mcastorina <m.castorina93@gmail.com>
* comment out unused styles and remove unused types
* Capitalize H in TruffleHog
* remove unneeded fmt.Sprintf
---------
Co-authored-by: mcastorina <m.castorina93@gmail.com>
This PR implements tri-state verification in the FTP detector. The verification timeout was made injectable to support a new test case. Some test cases that had already been broken have been fixed as well.
* Add common chunker.
* add comment.
* use better config name.
* Add common chunk reader to s3.
* Add common chunk reader to git, gcs, circleci.
* revert gcs.
* revert gcs.
* fix chunker.
* revert gcs.
* update cancellablewrite.
* revert impl.
* update to remove totalsize.
* Fix my goof.
* Use unified struct in chunkreader.
* return err instead of logging and returning.
* rename error to err.
* only send single ChunkResult even if there is an error and chunkBytes.
* fix logic.
* Add SourceManager to Engine struct
* Update Engine methods to use the SourceManager
* Fix GCS test
The original was testing that `Init()` errors weren't surfaced in
`Finish()`, but the `SourceManager` changed that behavior.
* JobProgress race fixes
* Add contextual values
* Remove unused code
* Add debug logs
* Rename WithConcurrency to WithConcurrentSources
* Always forward chunks to the output chunks channel
This PR implements tri-state verification for the LDAP detector. This implementation looks for network errors to explicitly flag as indeterminate, rather than authentication errors to explicitly flag as determinate; this is because the error that occurs from authentication failures doesn't appear to have its own type and I didn't want to have to match on the error message text.
* feat: initial support for bare repositories
* feat: use concatenation instead of formatting and os.Getenv instead of os.Environ
Signed-off-by: Savely Krasovsky <savely@krasovs.ky>
* fix: go-git update with pre-receive hooks fix
Signed-off-by: Savely Krasovsky <savely@krasovs.ky>
* fix: remove info about pre-receive hook from README.md for now
Signed-off-by: Savely Krasovsky <savely@krasovs.ky>
* fix: don't scan staged while using --bare option, fixes to make it work with the latest master
Signed-off-by: Savely Krasovsky <savely@krasovs.ky>
* fix: small refactor according to #1518
Signed-off-by: Savely Krasovsky <savely@krasovs.ky>
---------
Signed-off-by: Savely Krasovsky <savely@krasovs.ky>
* Github Oauth2 verification
* Use prefix and include RawV2
* Make gh_oauth2 a new detector
* Remove unused struct
* Remove versioner
* Remove unused code
* Refactor git source to allow ScanOptions and use source in engine
Refactor the Chunks method of the git Source to call out to two helper
methods: scanRepos and scanDirs which scans s.conn.Repositories and
s.conn.Directories respectively. The only notable change in behavior is
that a credential is no longer necessary if there are no
s.conn.Repositories to scan.
* Preserve ScanGit functionality of not cleaning up temporary files
* gitparse: Use an object for currentDiff instead of a pointer
* gitparse: Use an object for currentCommit instead of a pointer
* Revert "gitparse: Use an object for currentCommit instead of a pointer"
This reverts commit c5f0708b4a.
* Support fatal errors in job reports
* WIP: JobReporter and JobInspector
* WIP: JobReportHook and JobReportRef
* Add ChunkError type and asyncRun helper method
* Rename JobReport to JobProgress
* Return a closed channel from Done when the JobProgress is nil
* Comment catchFirstFatal function
It turns out that GetCallerIdentity returns a surprising quantity of transient, false-negative 403 responses that carry the SignatureDoesNotMatch error reason. I don't know why this is happening, but their transient nature makes them indeterminate verification failures and they should be flagged as such. The AWS detector has therefore been modified to specifically look for the InvalidClientTokenId error reason in 403 responses and mark all other responses as indeterminate.
In addition to the functional changes this PR contains some updates to the test code that allow us to test them.
#1454 modified one of the Github enumeration code paths in a way that broke an integration test by causing one client's transport to be used for the construction of a different client, causing authentication failures. This saves the original transport for use, fixing the test.
This PR makes the Alchemy detector run its known false positive check even if verification is disabled. This isn't the most important detector but it's the template for new ones so getting a good pattern nailed down is important.
Moving the check allowed me to rewrite the determinacy logic to hopefully be more clear.
* Miscellaneous SourceManager updates
* Own the chunks channel instead of accepting it as an input
* Add Chunks and Wait methods
* Fix bug in Enroll so it actually returns the handle
* Add context.Context parameter to the SourceInitFunc type
* Add SourceManager tests for Run and Wait methods
* Rename man variables to mgr
* Implement SourceManager basics
* Rename identifiers and add a default headlessAPI implementation
* Rewrite to use SourceInitFunc
* Update variable name to accurately reflect its value
Previously, the various JDBC detectors would independently try to verify credentials by a process of trying various permutations of candidates one-by-one. The upcoming tri-state verification work will need to add sophistication to this process in the same way for each one, so this PR first combines all of the logic so it can be upgraded in a single spot.
* saving progress
* proto changes
* run make protos
* verify response, add test case
* resolve linter warning about unescaped . in regex pattern
* resolve overlapping proto number
JDBC redaction could fail in some irritating edge cases involving passwords that contain the @ character. The logic has been tweaked to eliminate these cases and some tests have been added.
* issue comment scanning
* save progress
* test
* test for pr comment and issue comment
* add pagination support
* linter stuff
* make linter happy
* remove debug log
* readd logging
* github issue resolved
* var const block and handle rate limit
* remove magic number
* make gitURLParse a public function to use more generally
* fix test bug
* make comment scanning OPT-IN
* Add CancellableWrite helper function
* Create SourceUnitEnumerator interface and EnumerationResult struct
* Implement SourceUnitEnumerator for the filesystem Source
* Omit explicit zero values
* verify response body with expected keywords
* remove debug log
* add extra test case
* migrate from ioutil to io
* close body and only check for one keyword
* cleanup
* init
* add detector type
* rotate leaked credentials
* tighten up username pattern
* isolated prefixregex as overrriding new line stuff
* passwordPat working now
* add username test
* fix edge case
* cleanup
* make linter happy
* make linter happy rd 2
* skip error logging
* fix test
* add password regex helper func
* make test more robust
* cleanup PR
* remove comments
* clarify prepend rationale
* Exit with non-zero exit code on chunk source error
* Exit with a non-zero exit code whenever we hit an error getting
chunks. Previously the error would be logged but trufflehog would exit
with a 0 (success) status code.
* fix gcs test
---------
Co-authored-by: Dustin Decker <dustin@trufflesec.com>
Co-authored-by: ahrav <ahravdutta02@gmail.com>
* Implement CommonSourceUnitUnmarshaller
* Add SourceUnitUnmarshaller to all sources using
All sources, with the exception of git, will use the CommonSourceUnit as
they only contain a single type of unit to scan.
* Fix method comments to adhere to Go's style guide
* Add Validator interface and example
* Close sockets and improve error messages
* Remove duplicate error
* Use var declaration so err slice can be nil
* Use heuristic to choose the most likely UTF-16 decoded string
* Assume ASCII and include valid BE and LE bytes
* Remove unused code
* Assume ASCII and return nil when not utf16
---------
Co-authored-by: bill-rich <bill.rich@gmail.com>
* init
* look for client id and client secret, encode them for basis auth
* add tests
* test without checking the contents of response
* confirm access_token exists
* cleanup test
* explain in code that an undocumented grant_type is used
* remove use of deprecated ioutil, remove dead code, return errors instead of just logging
* directly pull access token
* update error text, remove redundant body close()
* import new detector into defaults
When a Result is emitted, it should include
the `chunk.Data []byte` so that we can utilize
the blob of data which caused the result.
This makes it so something catching the results
does not have to maintain a collection of chunks
to correlate the two together.
* Add Base64URLSafe decoder
* Add decoder that can decode base64 strings with '_' and '-' instead of
of '+' and '/'.
* Combine url-safe b64 decoder into b64 decoder
Generic API keys like shown in the example below is getting excluded:
api_key=9e107d9d372bb6826bd81d3542a419d6 because of following regex patterns:
\b[A-Fa-f0-9]{32}\b
\b[A-Fa-f0-9x]{6,99}\b
The base64 decoding logic is getting hit and NOT returning an error, and thus it continues thinking it is base64 decoded.
* Fix worktree scan by setting EnableDotGitCommonDir
* Change `PlainOpenOptions` to set `EnableDotGitCommonDir` to true.
In every current usage of this function, it is on an already-cloned
repository, so it should always be valid to have this set. By doing
so, it should fix some issues with worktrees.
* Remove unused go.mod replace directives
* Remove replace directives for libraries that are not in use.
* Fix error where some files do not get properly scanned due to order of
extraction / decompression steps. Doing decompression first ensures
that a compressed archive (e.g., gzipped zip file), is handled
correctly.
* Implement EndpointCustomizer
Add the EndpointCustomizer interface and EndpointSetter convenience struct,
implement EndpointCustomizer for github and gitlab detectors, and add
parsing, verification, and applying user-supplied configuration.
* Check error from SetEndpoints
* Rename variable for clarity
* add sharepoint source proto
* create sharepoint oss protos
* add email field, remove oauth2 type, update token to refresh_token
* rename path to link
* restore clientcredentials
* restore s3 and confluence proto changes from make command
* Add in-memory caching lib, used by the GCS source.
* Use cache for tracking progress for the GCS source.
* fix merge issue.
* fix merge issue.
* fix test.
* Fix static check.
* Add test for NewWithData.
* Use cache for tracking progress for the GCS source.
* fix merge issue.
* fix merge issue.
* fix test.
* update comment.
* update comments.
* Use cache for tracking progress for the GCS source.
* fix merge issue.
* fix merge issue.
* fix test.
* remove unused dep.
* address comments.
* Add exists method.
* Use cache for tracking progress for the GCS source.
* fix merge issue.
* fix merge issue.
* fix test.
* rebase.
* fix test.
* Use cache for tracking progress for the GCS source.
* fix merge issue.
* fix merge issue.
* fix test.
* rebase.
* rebase.
* split encode resume by comma.
* Use a persistable cache.
* fix merge.
* fix merge.
* Add progress as part of the cache given it will be the persistence layer.
* Add test for making sure the cache doesn't persist when the increment value is not met.
* fix tests.
* Resolve#1167 by adding support for the AWS_SESSION_TOKEN environment variable and adding a --session-token cli arg
* fix error message
---------
Co-authored-by: Dustin Decker <dustin@trufflesec.com>
* Adjust types to use DetectorID struct
* Parse versions with detector include and exclude input
* Update detectors filter to use version
Co-authored-by: steeeve <steve@trufflesec.com>
* Implement Versioner for github, gitlab, and npm detectors
Co-authored-by: steeeve <steve@trufflesec.com>
---------
Co-authored-by: steeeve <steve@trufflesec.com>
* Refactor generating CustomRegex results into a helper function
* Added errGroup for createResults, and ensure goroutines are non-blocking
* clean return
---------
Co-authored-by: Miccah Castorina <m.castorina93@gmail.com>
This will concatenate all errors together into a single string. When
possible, it would be better to log the actual errors slice to take
advantage of structured logging.
* Add ability to include and exclude detectors
* Trim space before checking for empty items
* Explicitly check for integer overflow
* Use strconv.ParseInt instead of strconv.Atoi
* Address comments
* Rename directories to paths
* Generate protos
* Add file scanning support to filesystem source
* Add directories back to filesystem proto
* Generate protos
* Combine paths and directories from in source
* Add filesystem filter
* Address comments
In this case just Heroku and LinearAPI. But this includes the Moonclerck
detector, which appears to be a typo that got turned into a separate
detector type.
Co-authored-by: zubairk14 <zubair.khan@trufflesec.com>
* Add Type() to detector interface
The goal here is to allow the detector type information to be used
without the need for reflection. This could possibly allow us to more
easily inject information into detectors or filter them out if
necessary.
Co-authored-by: ahmed <ahmed.zahran@trufflesec.com>
* remove test detector
---------
Co-authored-by: ahmed <ahmed.zahran@trufflesec.com>
