mirror of
https://github.com/trufflesecurity/trufflehog.git
synced 2024-09-20 06:31:57 +00:00
Update detector process and readme (#79)
This commit is contained in:
Dustin Decker
GitHub
parent
0ee34a5be6
commit
72c13c4b8f
5 changed files with 85 additions and 46 deletions
81
README.md
81
README.md
|
|
@ -1,15 +1,29 @@
|
|||
# TruffleHog
|
||||
<p align="center">
|
||||
<img alt="GoReleaser Logo" src="https://storage.googleapis.com/trufflehog-static-sources/pixel_pig.png" height="140" />
|
||||
<h2 align="center">TruffleHog</h2>
|
||||
<p align="center">Find leaked credentials.</p>
|
||||
</p>
|
||||
|
||||
---
|
||||
|
||||
|
||||
[](https://github.com/trufflesecurity/trufflehog2/actions)
|
||||
[](https://goreportcard.com/report/github.com/trufflesecurity/trufflehog2)
|
||||
[](https://hub.docker.com/r/trufflesecurity/trufflehog2/)
|
||||
|
||||
|
||||
---
|
||||
|
||||
## Join The Slack
|
||||
Have questions? Feedback? Jump in slack and hang out with us
|
||||
|
||||
https://join.slack.com/t/trufflehog-community/shared_invite/zt-pw2qbi43-Aa86hkiimstfdKH9UCpPzQ
|
||||
|
||||
|
||||
## Demo
|
||||
|
||||
|
||||
|
||||
## Installation
|
||||
|
||||
Several options:
|
||||
|
|
@ -21,35 +35,35 @@ Several options:
|
|||
|
||||
### 3. Docker
|
||||
|
||||
|
||||
> Note: Apple M1 hardware users should run with `docker run --platform linux/arm64` for better performance.
|
||||
|
||||
#### **Most users**
|
||||
|
||||
```bash
|
||||
$ docker run -it -v "$PWD:/pwd" trufflesecurity/trufflehog2:latest github --repo https://github.com/dustin-decker/secretsandstuff --debug
|
||||
🐷🔑🐷 TruffleHog. Unearth your secrets. 🐷🔑🐷
|
||||
|
||||
Found verified result 🐷🔑
|
||||
Detector Type: AWS
|
||||
File: aws
|
||||
Link: https://github.com/dustin-decker/secretsandstuff/blob/90c75f884c65dc3638ca1610bd9844e668f213c2/aws
|
||||
Repository: https://github.com/dustin-decker/secretsandstuff.git
|
||||
Commit: 90c75f884c65dc3638ca1610bd9844e668f213c2
|
||||
Email: dustindecker@protonmail.com
|
||||
|
||||
Found unverified result 🐷🔑❓
|
||||
Detector Type: Github
|
||||
File: slack
|
||||
Link: https://github.com/dustin-decker/secretsandstuff/blob/8afb0ecd4998b1179e428db5ebbcdc8221214432/slack
|
||||
Repository: https://github.com/dustin-decker/secretsandstuff.git
|
||||
Commit: 8afb0ecd4998b1179e428db5ebbcdc8221214432
|
||||
Email: dustindecker@protonmail.com
|
||||
...
|
||||
docker run -it -v "$PWD:/pwd" ghcr.io/trufflesecurity/trufflehog2:latest github --repo https://github.com/trufflesecurity/test_keys --debug
|
||||
```
|
||||
|
||||
### 4. Pip (TODO)
|
||||
pip install trufflehog
|
||||
#### **Apple M1 users**
|
||||
|
||||
### 5. Brew (TODO)
|
||||
brew install trufflehog
|
||||
The `linux/arm64` image is better to run on the M1 than the amd64 image.
|
||||
Even better is running the native darwin binary avilable, but there is not container image for that.
|
||||
|
||||
```bash
|
||||
docker run --platform linux/arm64 -it -v "$PWD:/pwd" ghcr.io/trufflesecurity/trufflehog2:latest github --repo https://github.com/trufflesecurity/test_keys
|
||||
```
|
||||
|
||||
### 4. Pip (help wanted)
|
||||
|
||||
It's possible to distribute binaries in pip wheels.
|
||||
|
||||
Here is an example of a [project that does it](https://github.com/Yelp/dumb-init).
|
||||
|
||||
Help with setting up this packaging would be appreciated!
|
||||
|
||||
### 5. Brew (help wanted)
|
||||
|
||||
We'd love to distribute via brew and could use your help.
|
||||
|
||||
## Usage
|
||||
|
||||
|
|
@ -58,8 +72,9 @@ TruffleHog has a sub-command for each source of data that you may want to scan:
|
|||
- git
|
||||
- github
|
||||
- gitlab
|
||||
- slack
|
||||
- S3
|
||||
- filesystem
|
||||
- file and stdin
|
||||
|
||||
Each subcommand can have options that you can see with the `-h` flag provided to the sub command:
|
||||
|
||||
|
|
@ -94,6 +109,20 @@ For example, to scan a `git` repository, start with
|
|||
$ trufflehog git https://github.com/trufflesecurity/trufflehog2.git
|
||||
```
|
||||
|
||||
# License Change
|
||||
## Contributors
|
||||
|
||||
This project exists thanks to all the people who contribute. [[Contribute](CONTRIBUTING.md)].
|
||||
|
||||
|
||||
<a href="https://github.com/trufflesecurity/trufflehog/graphs/contributors">
|
||||
<img src="https://contrib.rocks/image?repo=trufflesecurity/trufflehog" />
|
||||
</a>
|
||||
|
||||
|
||||
## Stargazers over time
|
||||
|
||||
[](https://starchart.cc/trufflesecurity/trufflehog)
|
||||
|
||||
## License Change
|
||||
|
||||
Since v3.0, TruffleHog is released under a AGPL 3 license, included in [`LICENSE`](LICENSE). TruffleHog v3.0 uses none of the previous codebase, but care was taken to preserve backwards compatibility on the command line interface. The work previous to this release is still available licensed under GPL 2.0 in the history of this repository and the previous package releases and tags. A completed CLA is required for us to accept contributions going forward.
|
||||
|
|
|
|||
|
|
@ -46,9 +46,9 @@ If you think that something should be included outside of these guidelines, plea
|
|||
|
||||
### Creating a new Secret Scanner
|
||||
|
||||
1. Identify the Secret Scanner name from the [proto/secrets.proto](proto/secrets.proto) `DetectorType` enum.
|
||||
1. Identify the Secret Detector name from the [/proto/detectors.proto](/proto/detectors.proto) `DetectorType` enum.
|
||||
|
||||
2. Generate the SecretScanner
|
||||
2. Generate the Secret Detector
|
||||
|
||||
```bash
|
||||
go run hack/generate/generate.go detector <DetectorType enum name>
|
||||
|
|
@ -56,7 +56,7 @@ If you think that something should be included outside of these guidelines, plea
|
|||
|
||||
3. Complete the secret detector.
|
||||
|
||||
The previous step templated a boilerplate + some example code as a package in the `pkg/secrets` folder for you to work on.
|
||||
The previous step templated a boilerplate + some example code as a package in the `pkg/detectors` folder for you to work on.
|
||||
The secret detector can be completed with these general steps:
|
||||
|
||||
1. Add the test secret to GCP Secrets. See [managing test secrets](#managing-test-secrets)
|
||||
|
|
@ -80,7 +80,7 @@ Do not embed test credentials in the test code. Instead, use GCP Secrets Manager
|
|||
Note: `/tmp/s` is a valid path on Linux. You will need to change that for Windows or OSX, otherwise you will see an error. On Windows you will also need to install [WSL](https://docs.microsoft.com/en-us/windows/wsl/install).
|
||||
|
||||
```bash
|
||||
gcloud secrets versions access --project thog-contractor-tests --secret detectors3 latest > /tmp/s
|
||||
gcloud secrets versions access --project trufflehog-testing --secret detectors3 latest > /tmp/s
|
||||
```
|
||||
|
||||
2. Add the secret that you need for testing.
|
||||
|
|
@ -97,7 +97,7 @@ Do not embed test credentials in the test code. Instead, use GCP Secrets Manager
|
|||
3. Update the secret version with your modification.
|
||||
|
||||
```bash
|
||||
gcloud secrets versions add --project thog-contractor-tests detectors3 --data-file /tmp/s
|
||||
gcloud secrets versions add --project trufflehog-testing detectors3 --data-file /tmp/s
|
||||
```
|
||||
|
||||
4. Access the secret value as shown in the [example code](pkg/secrets/heroku/heroku_test.go).
|
||||
|
|
@ -119,7 +119,7 @@ Do not embed test credentials in the test code. Instead, use GCP Secrets Manager
|
|||
```bash
|
||||
dos2unix ./scripts/gen_proto.sh
|
||||
```
|
||||
5. Open [proto/secrets.proto](proto/secrets.proto) file and add new detectors then save it. Make sure Docker is running and run this in Ubuntu command line.
|
||||
5. Open [/proto/detectors.proto](/proto/detectors.proto) file and add new detectors then save it. Make sure Docker is running and run this in Ubuntu command line.
|
||||
```bash
|
||||
make protos
|
||||
```
|
||||
|
|
|
|||
22
main.go
22
main.go
|
|
@ -10,6 +10,7 @@ import (
|
|||
"os"
|
||||
"runtime"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/felixge/fgprof"
|
||||
|
|
@ -38,11 +39,11 @@ func main() {
|
|||
|
||||
gitScan := cli.Command("git", "Find credentials in git repositories.")
|
||||
gitScanURI := gitScan.Arg("uri", "Git repository URL. https:// or file:// schema expected.").Required().String()
|
||||
gitScanIncludePaths := gitScan.Flag("include_paths", "Path to file with newline separated regexes for files to include in scan.").Short('i').String()
|
||||
gitScanExcludePaths := gitScan.Flag("exclude_paths", "Path to file with newline separated regexes for files to exclude in scan.").Short('x').String()
|
||||
gitScanSinceCommit := gitScan.Flag("since_commit", "Commit to start scan from.").String()
|
||||
gitScanIncludePaths := gitScan.Flag("include-paths", "Path to file with newline separated regexes for files to include in scan.").Short('i').String()
|
||||
gitScanExcludePaths := gitScan.Flag("exclude-paths", "Path to file with newline separated regexes for files to exclude in scan.").Short('x').String()
|
||||
gitScanSinceCommit := gitScan.Flag("since-commit", "Commit to start scan from.").String()
|
||||
gitScanBranch := gitScan.Flag("branch", "Branch to scan.").String()
|
||||
gitScanMaxDepth := gitScan.Flag("max_depth", "Maximum depth of commits to scan.").Int()
|
||||
gitScanMaxDepth := gitScan.Flag("max-depth", "Maximum depth of commits to scan.").Int()
|
||||
gitScan.Flag("allow", "No-op flag for backwards compat.").Bool()
|
||||
gitScan.Flag("entropy", "No-op flag for backwards compat.").Bool()
|
||||
gitScan.Flag("regex", "No-op flag for backwards compat.").Bool()
|
||||
|
|
@ -52,7 +53,7 @@ func main() {
|
|||
githubScanRepos := githubScan.Flag("repo", `GitHub repository to scan. You can repeat this flag. Example: "https://github.com/dustin-decker/secretsandstuff"`).Strings()
|
||||
githubScanOrgs := githubScan.Flag("org", `GitHub organization to scan. You can repeat this flag. Example: "trufflesecurity"`).Strings()
|
||||
githubScanToken := githubScan.Flag("token", "GitHub token.").String()
|
||||
githubIncludeForks := githubScan.Flag("include_forks", "Include forks in scan.").Bool()
|
||||
githubIncludeForks := githubScan.Flag("include-forks", "Include forks in scan.").Bool()
|
||||
|
||||
gitlabScan := cli.Command("gitlab", "Find credentials in GitLab repositories.")
|
||||
// TODO: Add more GitLab options
|
||||
|
|
@ -64,14 +65,21 @@ func main() {
|
|||
filesystemDirectories := filesystemScan.Flag("directory", "Path to directory to scan. You can repeat this flag.").Required().Strings()
|
||||
// TODO: Add more filesystem scan options. Currently only supports scanning a list of directories.
|
||||
// filesystemScanRecursive := filesystemScan.Flag("recursive", "Scan recursively.").Short('r').Bool()
|
||||
// filesystemScanIncludePaths := filesystemScan.Flag("include_paths", "Path to file with newline separated regexes for files to include in scan.").Short('i').String()
|
||||
// filesystemScanExcludePaths := filesystemScan.Flag("exclude_paths", "Path to file with newline separated regexes for files to exclude in scan.").Short('x').String()
|
||||
// filesystemScanIncludePaths := filesystemScan.Flag("include-paths", "Path to file with newline separated regexes for files to include in scan.").Short('i').String()
|
||||
// filesystemScanExcludePaths := filesystemScan.Flag("exclude-paths", "Path to file with newline separated regexes for files to exclude in scan.").Short('x').String()
|
||||
|
||||
s3Scan := cli.Command("s3", "Find credentials in S3 buckets.")
|
||||
s3ScanKey := s3Scan.Flag("key", "S3 key used to authenticate.").String()
|
||||
s3ScanSecret := s3Scan.Flag("secret", "S3 secret used to authenticate.").String()
|
||||
s3ScanCloudEnv := s3Scan.Flag("cloud-environment", "Use IAM credentials in cloud environment.").Bool()
|
||||
s3ScanBuckets := s3Scan.Flag("bucket", "Name of S3 bucket to scan. You can repeat this flag.").Strings()
|
||||
|
||||
for i, arg := range os.Args {
|
||||
if strings.HasPrefix(arg, "--") {
|
||||
os.Args[i] = strings.ReplaceAll(arg, "_", "-")
|
||||
}
|
||||
}
|
||||
|
||||
cmd := kingpin.MustParse(cli.Parse(os.Args[1:]))
|
||||
|
||||
// When setting a base commit, chunks must be scanned in order.
|
||||
|
|
|
|||
|
|
@ -220,7 +220,8 @@ func (e *Engine) detectorWorker(ctx context.Context) {
|
|||
logrus.WithField("raw", string(result.Raw)).WithField("repo", repo).WithField("file", file).Debugf("result: %s", result.Raw)
|
||||
_, exists := e.detectedSecret.secret[sid]
|
||||
if exists {
|
||||
logrus.Debugf("skipping duplicate result for %s in commit %s", result.Raw, commit)
|
||||
commit = commit
|
||||
// logrus.Debugf("skipping duplicate result for %s in commit %s", result.Raw, commit)
|
||||
continue
|
||||
}
|
||||
e.detectedSecret.sync.Lock()
|
||||
|
|
|
|||
|
|
@ -4,8 +4,6 @@ import (
|
|||
"bytes"
|
||||
"context"
|
||||
"fmt"
|
||||
"github.com/gitleaks/go-gitdiff/gitdiff"
|
||||
"github.com/rs/zerolog"
|
||||
"io"
|
||||
"io/ioutil"
|
||||
"net/url"
|
||||
|
|
@ -15,19 +13,22 @@ import (
|
|||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/gitleaks/go-gitdiff/gitdiff"
|
||||
"github.com/go-errors/errors"
|
||||
"github.com/go-git/go-git/v5"
|
||||
"github.com/go-git/go-git/v5/plumbing"
|
||||
"github.com/go-git/go-git/v5/plumbing/transport/http"
|
||||
"github.com/rs/zerolog"
|
||||
log "github.com/sirupsen/logrus"
|
||||
"github.com/trufflesecurity/trufflehog/v3/pkg/pb/source_metadatapb"
|
||||
"github.com/trufflesecurity/trufflehog/v3/pkg/pb/sourcespb"
|
||||
"github.com/trufflesecurity/trufflehog/v3/pkg/sanitizer"
|
||||
"github.com/trufflesecurity/trufflehog/v3/pkg/sources"
|
||||
glgo "github.com/zricethezav/gitleaks/v8/git"
|
||||
"golang.org/x/sync/semaphore"
|
||||
"google.golang.org/protobuf/proto"
|
||||
"google.golang.org/protobuf/types/known/anypb"
|
||||
|
||||
"github.com/trufflesecurity/trufflehog/v3/pkg/pb/source_metadatapb"
|
||||
"github.com/trufflesecurity/trufflehog/v3/pkg/pb/sourcespb"
|
||||
"github.com/trufflesecurity/trufflehog/v3/pkg/sanitizer"
|
||||
"github.com/trufflesecurity/trufflehog/v3/pkg/sources"
|
||||
)
|
||||
|
||||
type Source struct {
|
||||
|
|
|
|||
Loading…
Reference in a new issue