trufflehog/hack/docs/Adding_Detectors_Internal.md

Secret Detectors

Secret Detectors have these two major functions:

1. Given some bytes, extract possible secrets, typically using a regex.
2. Validate the secrets against the target API, typically using a HTTP client.

The purpose of Secret Detectors is to discover secrets with exceptionally high signal. High rates of false positives are not accepted.

Table of Contents

• Secret Detectors
  • Table of Contents
  • Getting Started
    • Sourcing Guidelines
    • Development Guidelines
    • Development Dependencies
    • Creating a new Secret Scanner
  • Addendum
    • Managing Test Secrets
    • Setting up Google Cloud SDK

Getting Started

Sourcing Guidelines

We are interested in detectors for services that meet at least one of these criteria

• host data (they store any sort of data provided)
• have paid services (having a free or trial tier is okay though)

If you think that something should be included outside of these guidelines, please let us know.

Development Guidelines

• When reasonable, favor using the net/http library to make requests instead of bringing in another library.
• Use the common.SaneHttpClient for the http.Client whenever possible.
• We recommend an editor with gopls integration (such as Vscode with Go plugin) for benefits like easily running tests, autocompletion, linting, type checking, etc.

Development Dependencies

• A GitLab account
• A Google account
• Google Cloud SDK installed
• Go 1.17+
• Make

Creating a new Secret Scanner

1. Identify the Secret Detector name from the /proto/detectors.proto DetectorType enum.

2. Generate the Secret Detector

   go run hack/generate/generate.go detector <DetectorType enum name>
   

3. Complete the secret detector.

   The previous step templated a boilerplate + some example code as a package in the pkg/detectors folder for you to work on. The secret detector can be completed with these general steps:

   1. Add the test secret to GCP Secrets. See managing test secrets
   2. Update the pattern regex and keywords. Try iterating with regex101.com.
   3. Update the verifier code to use a non-destructive API call that can determine whether the secret is valid or not.
   4. Update the tests with these test cases at minimum:
      1. Found and verified (using a credential loaded from GCP Secrets)
      2. Found and unverified
      3. Not found
      4. Any false positive cases that you come across
   5. Create a merge request for review. CI tests must be passing.

Addendum

Managing Test Secrets

Do not embed test credentials in the test code. Instead, use GCP Secrets Manager.

1. Access the latest secret version for modification.

   Note: /tmp/s is a valid path on Linux. You will need to change that for Windows or OSX, otherwise you will see an error. On Windows you will also need to install WSL.

   gcloud secrets versions access --project trufflehog-testing --secret detectors3 latest > /tmp/s
   

2. Add the secret that you need for testing.

   The command above saved it to /tmp/s.

   The format is standard env file format,

   SECRET_TYPE_ONE=value
   SECRET_TYPE_ONE_INACTIVE=v@lue
   

3. Update the secret version with your modification.

   gcloud secrets versions add --project trufflehog-testing detectors3 --data-file /tmp/s
   

4. Access the secret value as shown in the example code.

Setting up Google Cloud SDK

1. Install the Google Cloud SDK: https://cloud.google.com/sdk/docs/install
2. Authenticate with gcloud auth login --update-adc using your Google account

Adding Protos in Windows

1. Install Ubuntu App in Microsoft Store https://www.microsoft.com/en-us/p/ubuntu/9nblggh4msv6.
2. Install Docker Desktop https://www.docker.com/products/docker-desktop. Enable WSL integration to Ubuntu. In Docker app, go to Settings->Resources->WSL INTEGRATION->enable Ubuntu.
3. Open Ubuntu cli and install dos2unix.

   sudo apt install dos2unix
   

4. Identify the trufflehog local directory and convert scripts/gen_proto.sh file in Unix format.

   dos2unix ./scripts/gen_proto.sh
   

5. Open /proto/detectors.proto file and add new detectors then save it. Make sure Docker is running and run this in Ubuntu command line.

   make protos
   

Testing a detector

   go test ./pkg/detectors/<detector> -tags=detectors