Add more contributing docs

Author: dustin-decker

.gitignore

@@ -1,2 +1,3 @@
 .idea
 dist
+.env

CONTRIBUTING.md

@@ -1,7 +1,13 @@
 # Contribution guidelines
 
-Please create an issue to collect feedback prior to feature additions. If possible try to keep PR's scoped to one feature, and add tests for new features.
+Please create an issue to collect feedback prior to feature additions. If possible try to keep PRs scoped to one feature, and add tests for new features.
 
-When showing intereste, in a bug, enhancement, PR, or issue, please use the thumbs up/thumbs down emoji on the original message rather than adding comments expressing the same.
+When showing interest in a bug, enhancement, PR, or issue, please use the thumbs up/thumbs down emoji on the original message rather than adding comments expressing the same.
 
-Contributors need to [sign our CLA](https://cla-assistant.io/trufflesecurity/trufflehog) before we are able to accept contributions.
+Contributors need to [sign our CLA](https://cla-assistant.io/trufflesecurity/trufflehog) before we are able to accept contributions.
+
+# Resources
+
+## Adding new secret detectors
+
+We have published some [documentation and tooling to get started on adding new secret detectors](hack/docs/Adding_Detectors_external.md). Let's improve detection together!

README.md

@@ -137,6 +137,10 @@ Contributions are very welcome! Please see our [contribution guidelines first](C
 
 We no longer accept contributions to TruffleHog v2, but that code is available in the `v2` branch.
 
+### Adding new secret detectors
+
+We have published some [documentation and tooling to get started on adding new secret detectors](hack/docs/Adding_Detectors_external.md). Let's improve detection together!
+
 ## License Change
 
 Since v3.0, TruffleHog is released under a AGPL 3 license, included in [`LICENSE`](LICENSE). TruffleHog v3.0 uses none of the previous codebase, but care was taken to preserve backwards compatibility on the command line interface. The work previous to this release is still available licensed under GPL 2.0 in the history of this repository and the previous package releases and tags. A completed CLA is required for us to accept contributions going forward.

hack/docs/Adding_Detectors.md hack/docs/Adding_Detectors_Internal.mdhack/docs/Adding_Detectors.md renamed to hack/docs/Adding_Detectors_Internal.md

@@ -41,7 +41,7 @@ If you think that something should be included outside of these guidelines, plea
 - A GitLab account
 - A Google account
 - [Google Cloud SDK installed](#setting-up-google-cloud-sdk)
-- Go 1.16
+- Go 1.17+
 - Make
 
 ### Creating a new Secret Scanner
@@ -100,7 +100,7 @@ Do not embed test credentials in the test code. Instead, use GCP Secrets Manager
    gcloud secrets versions add --project trufflehog-testing detectors3 --data-file /tmp/s
    ```
 
-4. Access the secret value as shown in the [example code](pkg/secrets/heroku/heroku_test.go).
+4. Access the secret value as shown in the [example code](pkg/detectors/heroku/heroku_test.go).
 
 ### Setting up Google Cloud SDK
 

hack/docs/Adding_Detectors_external.md

@@ -0,0 +1,102 @@
+# Secret Detectors
+
+Secret Detectors have these two major functions:
+
+1. Given some bytes, extract possible secrets, typically using a regex.
+2. Validate the secrets against the target API, typically using a HTTP client.
+
+The purpose of Secret Detectors is to discover secrets with exceptionally high signal. High rates of false positives are not accepted.
+
+## Table of Contents
+
+- [Secret Detectors](#secret-detectors)
+  * [Table of Contents](#table-of-contents)
+  * [Getting Started](#getting-started)
+    + [Sourcing Guidelines](#sourcing-guidelines)
+    + [Development Guidelines](#development-guidelines)
+    + [Development Dependencies](#development-dependencies)
+    + [Creating a new Secret Scanner](#creating-a-new-secret-scanner)
+  * [Addendum](#addendum)
+    + [Using a test secret file](#using-a-test-secret-file)
+    + [Adding Protos in Windows](#adding-protos-in-windows)
+
+## Getting Started
+
+### Sourcing Guidelines
+
+We are interested in detectors for services that meet at least one of these criteria
+- host data (they store any sort of data provided)
+- have paid services (having a free or trial tier is okay though)
+
+If you think that something should be included outside of these guidelines, please let us know.
+
+### Development Guidelines
+
+- When reasonable, favor using the `net/http` library to make requests instead of bringing in another library.
+- Use the [`common.SaneHttpClient`](pkg/common/http.go) for the `http.Client` whenever possible.
+
+### Development Dependencies
+
+- Go 1.17+
+- Make
+
+### Creating a new Secret Scanner
+
+1. Identify the Secret Detector name from the [/proto/detectors.proto](/proto/detectors.proto) `DetectorType` enum. If necessary, run `make protos` when adding new ones.
+
+2. Generate the Secret Detector
+
+   ```bash
+   go run hack/generate/generate.go detector <DetectorType enum name>
+   ```
+
+3. Complete the secret detector.
+
+   The previous step templated a boilerplate + some example code as a package in the `pkg/detectors` folder for you to work on.
+   The secret detector can be completed with these general steps:
+
+   1. Create a [test secrets file, and export the variable](#using-a-test-secret-file)
+   2. Update the pattern regex and keywords. Try iterating with [regex101.com](http://regex101.com/).
+   3. Update the verifier code to use a non-destructive API call that can determine whether the secret is valid or not.
+   4. Update the tests with these test cases at minimum:
+      1. Found and verified (using a credential loaded from GCP Secrets)
+      2. Found and unverified
+      3. Not found
+      4. Any false positive cases that you come across
+   5. Create a pull request for review.
+
+## Addendum
+
+### Using a test secret file
+
+1. Create a file called `.env` with this env file format:
+
+   ```bash
+   SECRET_TYPE_ONE=value
+   SECRET_TYPE_ONE_INACTIVE=v@lue
+   ```
+
+2. Export the `TEST_SECRET_FILE` variable, pointing to the env file:
+
+   ```bash
+   export TEST_SECRET_FILE=".env"
+   ```
+
+Now, the detector test should attempt to load the given env key from that file.
+
+### Adding Protos in Windows
+
+1. Install Ubuntu App in Microsoft Store https://www.microsoft.com/en-us/p/ubuntu/9nblggh4msv6.
+2. Install Docker Desktop https://www.docker.com/products/docker-desktop. Enable WSL integration to Ubuntu. In Docker app, go to Settings->Resources->WSL INTEGRATION->enable Ubuntu.
+3. Open Ubuntu cli and install `dos2unix`.
+   ```bash
+   sudo apt install dos2unix
+   ```
+4. Identify the `trufflehog` local directory and convert `scripts/gen_proto.sh` file in Unix format.
+   ```bash
+   dos2unix ./scripts/gen_proto.sh
+   ```
+5. Open [/proto/detectors.proto](/proto/detectors.proto) file and add new detectors then save it. Make sure Docker is running and run this in Ubuntu command line.
+   ```bash
+   make protos
+   ```

pkg/common/secrets.go

@@ -3,6 +3,7 @@ package common
 import (
 	"context"
 	"fmt"
+	"os"
 	"time"
 
 	secretmanager "cloud.google.com/go/secretmanager/apiv1"
@@ -21,14 +22,32 @@ func (s *Secret) MustGetField(name string) string {
 	return val
 }
 
+func GetSecretFromEnv(filename string) (secret *Secret, err error) {
+	data, err := godotenv.Read(filename)
+	if err != nil {
+		return nil, err
+	}
+	return &Secret{kv: data}, nil
+}
+
 func GetTestSecret(ctx context.Context) (secret *Secret, err error) {
+	filename := os.Getenv("TEST_SECRET_FILE")
+	if len(filename) > 0 {
+		return GetSecretFromEnv(filename)
+	}
+
 	return GetSecret(ctx, "trufflehog-testing", "test")
 }
 
 func GetSecret(ctx context.Context, gcpProject, name string) (secret *Secret, err error) {
 	ctx, cancel := context.WithTimeout(ctx, time.Second*10)
 	defer cancel()
 
+	filename := os.Getenv("TEST_SECRET_FILE")
+	if len(filename) > 0 {
+		return GetSecretFromEnv(filename)
+	}
+
 	parent := fmt.Sprintf("projects/%s/secrets/%s/versions/latest", gcpProject, name)
 
 	client, err := secretmanager.NewClient(ctx)

pkg/updater/updater.go

@@ -76,7 +76,7 @@ func (g *OSS) Fetch() (io.Reader, error) {
 	defer resp.Body.Close()
 
 	if resp.StatusCode == http.StatusNoContent {
-		return nil, nil
+		return nil, errors.New("already up to date")
 	}
 
 	log.Debug("fetching trufflehog update")