Commit graph

104 commits

Author SHA1 Message Date
Mike Vanbuskirk
4636dc08f6
Add temp directory management (#1878)
* adds func to get scannerPIDs

* add cleanup and call to get pids

* move pid handling to git module

* remove PID logic from main

* refactor testing code to handle different exec name

* cleanup linting errors

* add better logging, fix dir if clause

* some PR fixups

* mod fixup

* add interfaces for helper funcs

* refactor cleanup into main, getPID into git

* lint and test fixups, remove fail on n<2 pids

* simplify pid sorting

* use filepath.Join

* use Args[0] for exec name, fix logger

* formatting fixup

* move functionality into cleantemp pkg

* go mod fixup

* remove redundant testing comment

* fix go.sum issues

* add 15m ticker loop for cleanup

* enclose ticker in function for goroutine defer

fix cleantemp interface

* make time more readable

* add check for non-local Trufflehog PIDs

* allow deletion even if no non-local pids found

* bundle intial cleanup into runCleanup func

* add explicit regex check for tempdir format
2023-10-26 12:28:56 -04:00
Damanpreet Singh
b9f49933b8
Added Support for '-h' Option for Help Documentation (#1901) 2023-10-18 06:57:05 -07:00
Dustin Decker
52ed87edb7
Add an option to filter unverified results using shannon entropy (#1875)
* Add an option to filter unverified results using shannon entropy

* lint

* add test, update test, and optimize
2023-10-08 19:52:28 -07:00
joeleonjr
699547b7d3
consolidated pr and issue descr/comment flags (#1827) 2023-09-27 15:54:02 -04:00
joeleonjr
1e42dae734
added PR and Issue body scanning (#1816)
* added PR and Issue body scanning; adjusted CLI args to fit

* removed print statement from debugging

* removed exclude-commits; adjusted CLI flags

* minor changes to match main branch

* fixing logic

* updating README for --issues and --prs
2023-09-26 12:25:48 -04:00
ahrav
fdeccf06a0
cache dupes w/ different decoders (#1754)
* only cache dupes that have different decoders.

* add test.

* remove file.

* update comment.
2023-09-11 08:18:48 -07:00
Mike Vanbuskirk
64dd49f9ce
add role assumption for s3 source (#1477)
* add role assumption for s3 source

* refactor role assumption to repeatable string

user can pass array of roles to assume

* refactor s3 chunks to handle passed roleARNs

* add role-session name

use timestamp to make dynamic

* add docstring for rolearn strings()

* make sure role ars are passed into source

* refactor role assumption functionality

break s3 bucket scanning into sep. function

* add log check on assume role

* fix role iteration

- Make sure s3 struct is populated with roles
- add separate new client instantiation for role-based access
- iterates through each role

* add comment

* protobuf revert for merge

* re-run make proto

* lint cleanup

* cleanup TODOs

* drop redundant switch case in assumerole client

* use less verbose 'ctx' designator

* breakout functionality from Chunks

- separate functions for:
- enumerating buckets to scan
- scanning objects within the buckets

* remake protobuf defs

* allow scan to continue on single bucket err

* add readme docs

* minor fixups
2023-08-17 20:30:20 -04:00
Zubair Khan
db89e345d7
correct logging output for github comments and add oss flags (#1632)
* correct logging output

* add flags

* respect oss cli flags for github comment scanning

* improve copy
2023-08-16 18:23:59 -04:00
Hon
47c2b6bed9
Add terminal UI (#1593)
* Init attempt at tui with bubble tea.

Co-authored-by: mcastorina <m.castorina93@gmail.com>

* Add starting and source selection options

Co-authored-by: mcastorina <m.castorina93@gmail.com>

* Rewrite models into a state machine

* Update source descriptions

* Make subpages implement tea.Model

* Rename page0 and page1 to be more descriptive

* Adjust styling and adding color consts

Co-authored-by: mcastorina <m.castorina93@gmail.com>

* Add helper generic function to call Update and type cast

* Setup plumbing for source configuration page

* Use CLI introspection for source configuration (WIP)

* Experiment with table view

* Replace table with form fields

Co-authored-by: mcastorina <m.castorina93@gmail.com>

* Change 🔒 to 💸

* Copy components from soft-serve

Co-authored-by: hxnyk <8292703+hxnyk@users.noreply.github.com>

* Copy styles from soft-serve

Co-authored-by: hxnyk <8292703+hxnyk@users.noreply.github.com>

* Copy common from soft-serve

Co-authored-by: hxnyk <8292703+hxnyk@users.noreply.github.com>

* Refactor into pages

This is still a WIP, but the main structure is there.

Co-authored-by: hxnyk <8292703+hxnyk@users.noreply.github.com>

* Trying out selector for wizard intro

Co-authored-by: mcastorina <m.castorina93@gmail.com>

* Use selector with custom View

Co-authored-by: hxnyk <8292703+hxnyk@users.noreply.github.com>

* Change Item to be an enum

Co-authored-by: hxnyk <8292703+hxnyk@users.noreply.github.com>

* Add link pages

Co-authored-by: mcastorina <m.castorina93@gmail.com>

* Update source select to use selector

Co-authored-by: mcastorina <m.castorina93@gmail.com>

* Delete source configure page and add blank tabs

Co-authored-by: hxnyk <8292703+hxnyk@users.noreply.github.com>

* Add tab placeholder pages for configurationi

Co-authored-by: mcastorina <m.castorina93@gmail.com>

* Added headers and style to each tab

Co-authored-by: hxnyk <8292703+hxnyk@users.noreply.github.com>

* Update with new sources

* Remove kingpin attribute from SourceItem

* Add basic form field and source structuring

* Hookup git form fields with an underlying textinput component

Co-authored-by: hxnyk <8292703+hxnyk@users.noreply.github.com>

* Update forms for git and github

Co-authored-by: mcastorina <m.castorina93@gmail.com>

* Add labels per text input

* Add sources and adjust styling

* add basic trufflehog configuration page

* Add skip button to textinputs component

* Emit and handle textinputs skip/submit button commands

* Don't quit when q is pressed on the sourceConfigurePage

* Build trufflehog command based on source config vals

Co-authored-by: mcastorina <m.castorina93@gmail.com>

* Build flags based on truffle config inputs

* Update summary section

* Add generated truffle fields

Co-authored-by: mcastorina <m.castorina93@gmail.com>

* update summary to correctly print info

* Go back a page when escape key is pressed

* WIP run page list

Co-authored-by: hxnyk <8292703+hxnyk@users.noreply.github.com>

* Allow running trufflehog from the run page

Co-authored-by: hxnyk <8292703+hxnyk@users.noreply.github.com>

* Add option to view help docs

Co-authored-by: mcastorina <m.castorina93@gmail.com>

* comment out unused styles and remove unused types

* Capitalize H in TruffleHog

* remove unneeded fmt.Sprintf

---------

Co-authored-by: mcastorina <m.castorina93@gmail.com>
2023-08-09 13:13:55 -07:00
Savely Krasovsky
d062834997
initial support for bare repositories (#1499)
* feat: initial support for bare repositories

* feat: use concatenation instead of formatting and os.Getenv instead of os.Environ

Signed-off-by: Savely Krasovsky <savely@krasovs.ky>

* fix: go-git update with pre-receive hooks fix

Signed-off-by: Savely Krasovsky <savely@krasovs.ky>

* fix: remove info about pre-receive hook from README.md for now

Signed-off-by: Savely Krasovsky <savely@krasovs.ky>

* fix: don't scan staged while using --bare option, fixes to make it work with the latest master

Signed-off-by: Savely Krasovsky <savely@krasovs.ky>

* fix: small refactor according to #1518

Signed-off-by: Savely Krasovsky <savely@krasovs.ky>

---------

Signed-off-by: Savely Krasovsky <savely@krasovs.ky>
2023-08-03 11:23:41 -05:00
ahrav
06d2eab204
include scan duration in output log (#1598)
* add scan duration to output log.

* fix linter.
2023-08-02 11:48:29 -07:00
Zubair Khan
a4b1fb7752
create hidden debug flag to disable overseer (#1582)
* add in new debug flag

* keep localdev local
2023-07-31 22:03:59 -04:00
ahrav
5e7a6ca11c
Concurrent detection (#1580)
* Run detection on each chunk concurrently.

* Add printer functionality.

* Add logic for dedupe.

* cleanup.

* Moddify number of notifier workers.

* Add comment.

* move consts into fxn.

* buffer resutls chan.

* fix test.

* address comments.

* return an error from Finish.

* fix test.

* fix test.

* linter.

* check err.

* address comments.
2023-07-31 11:12:08 -07:00
Miccah
69515bb7ac
Correctly route pprof endpoint (#1527) 2023-07-21 22:45:27 -05:00
Brendan Shaklovitz
da5301ea1e
Exit with non-zero exit code on chunk source error (#1286)
* Exit with non-zero exit code on chunk source error

* Exit with a non-zero exit code whenever we hit an error getting
  chunks. Previously the error would be logged but trufflehog would exit
  with a 0 (success) status code.

* fix gcs test

---------

Co-authored-by: Dustin Decker <dustin@trufflesec.com>
Co-authored-by: ahrav <ahravdutta02@gmail.com>
2023-06-26 11:39:57 -05:00
Dustin Decker
e856a6890d
🎉 Add Docker image scanning 🎉 (#1412)
* Add Docker source

* Add metrics

* Add test

* Add debugging, address PR comments, fix path output

* review suggestions
2023-06-22 08:02:25 -07:00
dillonstreator
648ef3b52c
fix spelling errors (#1413) 2023-06-21 07:15:28 -07:00
dillonstreator
fd4b5d1d14
remove gorilla mux (#1411) 2023-06-20 17:07:03 -07:00
Miccah
b1675194ca
Implement EndpointCustomizer (#1291)
* Implement EndpointCustomizer

Add the EndpointCustomizer interface and EndpointSetter convenience struct,
implement EndpointCustomizer for github and gitlab detectors, and add
parsing, verification, and applying user-supplied configuration.

* Check error from SetEndpoints

* Rename variable for clarity
2023-04-27 12:23:50 -05:00
Brendan Shaklovitz
10902f802a
Add max object size flag for s3 bucket scanning (#1294)
Co-authored-by: Dustin Decker <dustin@trufflesec.com>
2023-04-26 15:39:43 -07:00
Miccah
5a86c18302
Fix include and exclude detector logic (#1267)
* Fix include and exclude detector logic

* Fix test

* Add more clarifying comments
2023-04-26 10:49:54 -05:00
Bill Rich
0507f0eb87
Only add detectors once (#1265) 2023-04-17 14:10:13 -07:00
iamjpotts
b3d917f9c7
Resolve #1167 by adding support for the AWS_SESSION_TOKEN (#1170)
* Resolve #1167 by adding support for the AWS_SESSION_TOKEN environment variable and adding a --session-token cli arg

* fix error message

---------

Co-authored-by: Dustin Decker <dustin@trufflesec.com>
2023-04-03 14:56:43 -07:00
ahrav
0052f60090
Allow for custom verifier (#1070)
* allow for custom verifier.

* Update engine.

* use custom detectors.

* set cap.

* Update verifiers.

* Remove nil check.

* resolved nit

* handle uppercase values

* updating missing url logs

* adding more descriptive variable names

* updating logs to use correct variables

* Removing toLower for urls

* if else nits

* Adding versioning for github and gitlab

---------

Co-authored-by: ahmed <ahmed.zahran@trufflesec.com>
Co-authored-by: ah̳̕mͭͭͨͩ̐e̘ͬ́͋ͬ̊̓͂d <13666360+0x1@users.noreply.github.com>
2023-03-29 12:26:39 -07:00
Dustin Decker
cb454bfc05
Add GitHub Actions output (#1201)
* Add GitHub Actions output

Co-authored-by: Mike Vanbuskirk <mike.vanbuskirk@trufflesec.com>
2023-03-28 09:07:26 -07:00
Zachary Rice
fb9ae75661
Support for exclude globs at the git log level (#1202)
* init

* seems to be working

* better comment

* rm conditional

* Add more context to exclude-globs description
2023-03-28 10:46:03 -05:00
ahrav
c617bd7a4e
Add resuming capability to GCS source (#1161)
* Add resuming capability to GCS source.

* Handle no auth scans.

* complete resume logic

* Use custom function type.

* remove functions.

* linter.

* fix test.

* fix test.

* Handle concurrent map writes.

* use string as CLI flag for include/exclude.

* handle emtpy buckets.

* Handle enumeration on initial job run.

* Rename stats to attributes.

* remove redundant return.

* If test fails due to 400, that is fine, it's expected.

* Add unauth GCS source type.

* comments.

* update proto.

* Use short flag.

* address comments.
2023-03-16 17:53:42 -07:00
ahrav
cbf299aa77
Add gcs scanning integration (#1153)
* Setup for GCS scanning.

* Update GCS engine w/ projectID req.

* Add concurrency field to gcsManager.

* add errgroup to gcsManager.

* Update gcs manager.

* Use defautl ADC.

* use ADC.'

* Add TOOD.

* add log to iterator completion.

* use a BinaryReader instead of concrete object for channel type.

* initial test for Chunks.

* Add tests for chunking objects.

* Add concurrency.

* update metadata to include content type and acls.

* Add object reading code.

* Add integration test.

* Add entrypoint.

* Add removed wg.Wait().

* remove dead code.

* remove build.

* Remove period from file extension.

* remove used.

* Add comment.

* Setup for GCS scanning.

* Update GCS engine w/ projectID req.

* Add concurrency field to gcsManager.

* add errgroup to gcsManager.

* Update gcs manager.

* Use defautl ADC.

* use ADC.'

* Add TOOD.

* add log to iterator completion.

* use a BinaryReader instead of concrete object for channel type.

* initial test for Chunks.

* Add tests for chunking objects.

* Add concurrency.

* update metadata to include content type and acls.

* Add object reading code.

* Add integration test.

* Add entrypoint.

* Add removed wg.Wait().

* remove dead code.

* remove build.

* remove used.

* Add file type for objects.

* Add check for file type and size.

* Add default file size.

* Add additinoal auth options and remaining CLI flags.

* Handle errors in go routines.

* Handle resuming for buckets.

* Remove redundant words in comment.

* remove ok check on bool check.

* remove extra blank line.

* Add return if handler handles chunk.

* Add comment.

* remove extra blank line.

* cleanup comment.

* Add comment.

* move up fxn.

* go mod tidy.

* Add exclusion to perf testing buckets.

* Handle blocking the channel.

* remove unused const.

* fix tests.

* fix tests.

* Handle gcs manger options better.

* update fxn name.

* Remove arg name.

* ignore buckets in gcsManager test.

* fix test.

* propulate gsManagerOpts.

* inline err check.

* Add readme.

* update readme spelling.

* fix test.
2023-03-07 17:32:04 -08:00
Miccah
e6846ede54
Support filtering detectors by version (#1150)
* Adjust types to use DetectorID struct

* Parse versions with detector include and exclude input

* Update detectors filter to use version

Co-authored-by: steeeve <steve@trufflesec.com>

* Implement Versioner for github, gitlab, and npm detectors

Co-authored-by: steeeve <steve@trufflesec.com>

---------

Co-authored-by: steeeve <steve@trufflesec.com>
2023-03-02 16:33:56 -06:00
Yassine Ilmi
0cf9139df6
Disable profiler in debug mode and add profile switch (#1136) 2023-02-28 12:49:54 -08:00
Miccah
dd39848709
Add ability to include and exclude detectors (#1106)
* Add ability to include and exclude detectors

* Trim space before checking for empty items

* Explicitly check for integer overflow

* Use strconv.ParseInt instead of strconv.Atoi

* Address comments
2023-02-27 16:46:45 -06:00
Miccah
c5b4d6f28b
Support file scanning in filesystem source (#1030)
* Rename directories to paths

* Generate protos

* Add file scanning support to filesystem source

* Add directories back to filesystem proto

* Generate protos

* Combine paths and directories from in source

* Add filesystem filter

* Address comments
2023-02-27 12:15:05 -06:00
ahrav
012fdfe3a2
Update helper text for max-archive-size. (#1114) 2023-02-16 13:56:55 -08:00
ahrav
ea71756e20
[chore] - archive size helper text (#1110) 2023-02-15 10:08:26 -08:00
ahrav
ea40c0f306
Add the unit for max archive size. (#1108) 2023-02-15 09:45:27 -08:00
Miccah
161e499142
[chore] Remove logrus from trufflehog (#1095)
* [chore] Remove logrus from trufflehog

* Minor fixes

* Fix logFatal call

* Fix logrus call
2023-02-14 17:00:07 -06:00
SAYGIN Metin
f2139a7615
Github filter support for exclude and include (#1087)
* test

* Add missing head and base hash back.

---------

Co-authored-by: Ahrav Dutta <ahravdutta02@gmail.com>
2023-02-14 08:40:53 -08:00
Dustin Decker
0c66d30c1f
Revert "Make detectors configurable (#1084)" (#1097)
This reverts commit 67784f6928.
2023-02-11 08:12:13 -08:00
ahrav
67784f6928
Make detectors configurable (#1084)
* Make detectors configurable.

* remove redundant check.

* add number of detectors.

* update comment.

* remove reflect.

* inline key.

* replace name w/ type.

* remove temp var.

* fix test name.

* fix engine start.

* add filter unverified to engine.

* reorder engine args.

* Address comments.

* Add include and exclude.

* update comments.

* add comment.

* add comment.
2023-02-10 16:30:38 -08:00
ahrav
c5c8d10d28
[chore] - Remove monolithic config struct (#1091)
* REmove monolithic config struct.

* fix broken test.
2023-02-10 12:43:00 -08:00
Miccah
58e8c1e4ac
[chore] Remove logrus from engine package (#1085) 2023-02-09 16:55:19 -06:00
Bill Rich
7dd2b74f1f
Make archive handler configurable (#1077)
* Make archive handler configurable.

* Use common.IsDone()
2023-02-07 15:25:14 -08:00
Alexandr Marchenko
b29b78c10d
filesystem support for exclude and include filters (2nd attemp) (#1033)
* fix filter issue - empty lines should be ignored

* filesystem support for filter exclude

Co-authored-by: Dustin Decker <dustin@trufflesec.com>
2023-01-26 09:33:45 -08:00
Dustin Decker
5f6143f09a
Add Circle CI source (#997)
* Add Circle CI source

* remove SHA1 line

* remove trim
2023-01-05 21:44:37 -08:00
Yassine Ilmi
330a6f7cdc
Removing Debug version Println to logrus debug (#993) 2023-01-03 10:36:27 -06:00
Miccah
8859771a2a
Remove custom log leveler (#985)
Instead of manually using a log leveler, rely on the global one defined
in the `log` package.
2022-12-20 19:03:53 -06:00
Miccah
f5b83ee2a5
Add configuration parsing and custom detectors to engine (#968)
* Add configuration parsing for custom detectors

* Error on empty filename
2022-12-20 10:14:49 -06:00
ahrav
936a139596
Allow using a glob for include list. (#977)
* Allow using a glob for include list.

* Update command flag.

* Make comment more clear.

* update comment.

* Allow scanning repo and org at the same time.
2022-12-16 13:28:16 -08:00
Bill Rich
36ca2601e0
Add s3 object count to trace logs (#975)
* Add s3 object count to trace logs

* fix debug level
2022-12-13 16:46:09 -08:00
Bill Rich
3b055ce3f9
Add logger to context (#947)
* Add logger to context

* Fatal on no org
2022-11-30 11:10:05 -08:00