Commit graph

2227 commits

Author SHA1 Message Date
ahrav
e0db575d4a
[chore] - Use custom context for archive handler of specialized archives (#1629)
* Use custom context for archive handler of specialized archives.

* fix arg.

* fix test.

* use re-reader.

* use re-reader.

* Update error and comments.

* Add better error handling.

* update.
2023-08-16 13:52:55 -07:00
Zubair Khan
62d359eba4
add salesforce detector (#1608)
* setup

* update time out case to return detector result

* fix

* remove unneeded comment

* remove debug print

* cleanup

* more robust error handling

* reflect new detector template changes

* fixes

* mark response body check err as indeterminate
2023-08-16 10:42:04 -04:00
ahrav
6ad5659334
Integration of SpecializedHandler for Enhanced Archive Processing (#1625)
* Add handler for .deb file formats.

* Add handler for .rpm file formats.

* update.

* move logic to general archive handler.

* update const.

* Add compile time guard.

* Remove redundant parens.

* Add checks to make sure we have the tools installed to extract arhives.

* Limit size of temp file for arhive reading.

* handle nested archives.

* add comment.

* use consistent name for tempEnv -> env

* fix handler fxn signature.
2023-08-15 16:08:55 -07:00
Hon
ea8c21051d
fix github org placeholder (#1627) 2023-08-15 15:27:52 -07:00
Zachary Rice
eee01e0361
bump go to 1.21 (#1623) 2023-08-14 15:36:25 -05:00
Brandon Yan
2de5c0b217
change verification endpoint (#1611) 2023-08-14 11:50:11 -07:00
Zubair Khan
ea6e8b6bb5
add huggingface detector (#1621)
* init huggingface detector

* completed test
2023-08-14 14:22:04 -04:00
Miccah
eae66ccf7e
Refactor FragmentLineOffset to match multiline secrets (#1612)
* Refactor FragmentLineOffset to match multiline secrets

* Add tests and benchmarks

* Use bytes.Count and fix an ignore tag edge case
2023-08-14 10:51:41 -07:00
Cody Rose
09795c3591
fix alchemy test error message (#1622) 2023-08-14 13:12:46 -04:00
joeleonjr
fa9469cfc7
Docker scanning by digest (#1615)
* added functionality to scan docker images with digests instead of tags

* cleaned import statement

* added unit test for baseAndTag parsing + remote digest scan
2023-08-11 16:53:12 -05:00
ahrav
e894540632
Use the common chunker for scanning the filesystem source (#1619)
* Use the common chunker for scanning the filesystem source.

* remove unused conts.

* add test.
2023-08-11 13:40:10 -07:00
Cody Rose
c9f58b3780
Support indeterminate verification in Gitlab detector (#1613) 2023-08-11 13:36:01 -04:00
Cody Rose
62cbef5292
stop saving alchemy url (#1614)
Verification of an Alchemy secret requires putting the candidate secret directly into a URL. This makes the URL potentially sensitive, and if the request fails, we don't want to save it anywhere that might inadvertently get logged elsewhere - like the resulting error message. (Despite verification failing, this error message is only saved if the failure is indeterminate, which means that the secret might actually be live.)
2023-08-11 10:09:05 -04:00
Cody Rose
18f854d142
Add tri-state verification to pubnub publish key detector (#1616) 2023-08-11 10:08:26 -04:00
Cody Rose
2570fdc244
fix error msg in alchemy test (#1617) 2023-08-11 09:54:37 -04:00
Hon
47c2b6bed9
Add terminal UI (#1593)
* Init attempt at tui with bubble tea.

Co-authored-by: mcastorina <m.castorina93@gmail.com>

* Add starting and source selection options

Co-authored-by: mcastorina <m.castorina93@gmail.com>

* Rewrite models into a state machine

* Update source descriptions

* Make subpages implement tea.Model

* Rename page0 and page1 to be more descriptive

* Adjust styling and adding color consts

Co-authored-by: mcastorina <m.castorina93@gmail.com>

* Add helper generic function to call Update and type cast

* Setup plumbing for source configuration page

* Use CLI introspection for source configuration (WIP)

* Experiment with table view

* Replace table with form fields

Co-authored-by: mcastorina <m.castorina93@gmail.com>

* Change 🔒 to 💸

* Copy components from soft-serve

Co-authored-by: hxnyk <8292703+hxnyk@users.noreply.github.com>

* Copy styles from soft-serve

Co-authored-by: hxnyk <8292703+hxnyk@users.noreply.github.com>

* Copy common from soft-serve

Co-authored-by: hxnyk <8292703+hxnyk@users.noreply.github.com>

* Refactor into pages

This is still a WIP, but the main structure is there.

Co-authored-by: hxnyk <8292703+hxnyk@users.noreply.github.com>

* Trying out selector for wizard intro

Co-authored-by: mcastorina <m.castorina93@gmail.com>

* Use selector with custom View

Co-authored-by: hxnyk <8292703+hxnyk@users.noreply.github.com>

* Change Item to be an enum

Co-authored-by: hxnyk <8292703+hxnyk@users.noreply.github.com>

* Add link pages

Co-authored-by: mcastorina <m.castorina93@gmail.com>

* Update source select to use selector

Co-authored-by: mcastorina <m.castorina93@gmail.com>

* Delete source configure page and add blank tabs

Co-authored-by: hxnyk <8292703+hxnyk@users.noreply.github.com>

* Add tab placeholder pages for configurationi

Co-authored-by: mcastorina <m.castorina93@gmail.com>

* Added headers and style to each tab

Co-authored-by: hxnyk <8292703+hxnyk@users.noreply.github.com>

* Update with new sources

* Remove kingpin attribute from SourceItem

* Add basic form field and source structuring

* Hookup git form fields with an underlying textinput component

Co-authored-by: hxnyk <8292703+hxnyk@users.noreply.github.com>

* Update forms for git and github

Co-authored-by: mcastorina <m.castorina93@gmail.com>

* Add labels per text input

* Add sources and adjust styling

* add basic trufflehog configuration page

* Add skip button to textinputs component

* Emit and handle textinputs skip/submit button commands

* Don't quit when q is pressed on the sourceConfigurePage

* Build trufflehog command based on source config vals

Co-authored-by: mcastorina <m.castorina93@gmail.com>

* Build flags based on truffle config inputs

* Update summary section

* Add generated truffle fields

Co-authored-by: mcastorina <m.castorina93@gmail.com>

* update summary to correctly print info

* Go back a page when escape key is pressed

* WIP run page list

Co-authored-by: hxnyk <8292703+hxnyk@users.noreply.github.com>

* Allow running trufflehog from the run page

Co-authored-by: hxnyk <8292703+hxnyk@users.noreply.github.com>

* Add option to view help docs

Co-authored-by: mcastorina <m.castorina93@gmail.com>

* comment out unused styles and remove unused types

* Capitalize H in TruffleHog

* remove unneeded fmt.Sprintf

---------

Co-authored-by: mcastorina <m.castorina93@gmail.com>
2023-08-09 13:13:55 -07:00
Cody Rose
e5aeb219de
implement tri-state verification in FTP detector (#1604)
This PR implements tri-state verification in the FTP detector. The verification timeout was made injectable to support a new test case. Some test cases that had already been broken have been fixed as well.
2023-08-09 09:52:23 -04:00
Bill Rich
2d2595a2e3
Move commits_scanned to ScanRepo (#1610) 2023-08-07 14:28:57 -07:00
ahrav
13999227b9
Use common chunk reader (#1596)
* Add common chunker.

* add comment.

* use better config name.

* Add common chunk reader to s3.

* Add common chunk reader to git, gcs, circleci.

* revert gcs.

* revert gcs.

* fix chunker.

* revert gcs.

* update cancellablewrite.

* revert impl.

* update to remove totalsize.

* Fix my goof.

* Use unified struct in chunkreader.

* return err instead of logging and returning.

* rename error to err.

* only send single ChunkResult even if there is an error and chunkBytes.

* fix logic.
2023-08-07 12:55:28 -07:00
Cody Rose
18b3d3dd01
Tweak template detector test code (#1609)
The test code for the Alchemy detector, which is used as a template for new detectors, had a few little warts that I cleaned up.
2023-08-07 11:04:59 -04:00
Bill Rich
c2b49b060b
Detect API keys without app keys (#1605) 2023-08-03 15:11:54 -07:00
Bill Rich
cda88ebdf0
Adjust regex and add tests (#1602) 2023-08-03 15:11:45 -07:00
Miccah
1cd600f70f
Use SourceManager in engine (#1586)
* Add SourceManager to Engine struct

* Update Engine methods to use the SourceManager

* Fix GCS test

The original was testing that `Init()` errors weren't surfaced in
`Finish()`, but the `SourceManager` changed that behavior.

* JobProgress race fixes

* Add contextual values

* Remove unused code

* Add debug logs

* Rename WithConcurrency to WithConcurrentSources

* Always forward chunks to the output chunks channel
2023-08-03 13:36:30 -05:00
Cody Rose
d763097fdf
implement indeterminate LDAP verification (#1574)
This PR implements tri-state verification for the LDAP detector. This implementation looks for network errors to explicitly flag as indeterminate, rather than authentication errors to explicitly flag as determinate; this is because the error that occurs from authentication failures doesn't appear to have its own type and I didn't want to have to match on the error message text.
2023-08-03 14:02:31 -04:00
Miccah
e322c4b29d
Fix nil pointer dereference to git ScanOptions (#1603) 2023-08-03 12:07:24 -05:00
Savely Krasovsky
d062834997
initial support for bare repositories (#1499)
* feat: initial support for bare repositories

* feat: use concatenation instead of formatting and os.Getenv instead of os.Environ

Signed-off-by: Savely Krasovsky <savely@krasovs.ky>

* fix: go-git update with pre-receive hooks fix

Signed-off-by: Savely Krasovsky <savely@krasovs.ky>

* fix: remove info about pre-receive hook from README.md for now

Signed-off-by: Savely Krasovsky <savely@krasovs.ky>

* fix: don't scan staged while using --bare option, fixes to make it work with the latest master

Signed-off-by: Savely Krasovsky <savely@krasovs.ky>

* fix: small refactor according to #1518

Signed-off-by: Savely Krasovsky <savely@krasovs.ky>

---------

Signed-off-by: Savely Krasovsky <savely@krasovs.ky>
2023-08-03 11:23:41 -05:00
ahrav
5a5e8a607e
Common chunk reader (#1594)
* Add common chunker.

* add comment.

* use better config name.

* Add common chunk reader to s3.

* Add common chunk reader to git, gcs, circleci.

* fix chunker.

* revert gcs.

* update cancellablewrite.

* revert impl.

* update to remove totalsize.
2023-08-03 06:27:33 -07:00
Bill Rich
c995e93dcc
Add commits scanned to log (#1600)
* Add commits scanned to log

* Use atomic
2023-08-02 14:10:54 -07:00
ahrav
06d2eab204
include scan duration in output log (#1598)
* add scan duration to output log.

* fix linter.
2023-08-02 11:48:29 -07:00
ahrav
b1947246d9
Make prints to stdout serial. (#1597) 2023-08-02 11:38:10 -07:00
Bill Rich
0c7ed19270
Github Oauth2 verification (#1584)
* Github Oauth2 verification

* Use prefix and include RawV2

* Make gh_oauth2 a new detector

* Remove unused struct

* Remove versioner

* Remove unused code
2023-08-02 11:16:40 -07:00
Bill Rich
1cf419e478
Expand paypal regex (#1599) 2023-08-02 10:58:32 -07:00
Miccah
160fd830dd
Remove capturing the cancel callstack in the context package (#1595)
* Fix race condition in context package

* Remove capturing the cancel callstack
2023-08-01 21:34:00 -05:00
Zubair Khan
0ad46381d9
tighten up regex pattern for timezoneapi (#1591)
* tighten up regex pattern

* add response body check
2023-08-01 17:30:29 -04:00
Zachary Rice
91aa75679b
broken link (#1592) 2023-08-01 16:13:22 -05:00
Zachary Rice
b906a51d5c
updating github templates (#1587)
* updating github templates

* address miccahs comments

* remove community note comment
2023-08-01 15:25:33 -05:00
Zachary Rice
3c79b6b133
Remove CI Badge (#1590) 2023-08-01 13:45:46 -05:00
ahrav
78d06658ca
Dont return in loop. (#1589) 2023-08-01 10:29:01 -07:00
Miccah
69021f59c5
Refactor git source to allow ScanOptions and use source in engine (#1518)
* Refactor git source to allow ScanOptions and use source in engine

Refactor the Chunks method of the git Source to call out to two helper
methods: scanRepos and scanDirs which scans s.conn.Repositories and
s.conn.Directories respectively. The only notable change in behavior is
that a credential is no longer necessary if there are no
s.conn.Repositories to scan.

* Preserve ScanGit functionality of not cleaning up temporary files
2023-08-01 09:52:02 -05:00
ahrav
b8c43ea58f
Fix VirusTotal deetector (#1585) 2023-08-01 05:41:18 -07:00
Zubair Khan
a4b1fb7752
create hidden debug flag to disable overseer (#1582)
* add in new debug flag

* keep localdev local
2023-07-31 22:03:59 -04:00
Cody Rose
7d2f126411
add tri-state verification to mongodb detector (#1575) 2023-07-31 18:23:35 -04:00
ahrav
5043fc8756
[bug] - Fix unlocking an unlocked mutex (#1583)
* use correct mutext.

* remove unused fxn.
2023-07-31 14:06:41 -07:00
ahrav
661c6b47b7
[bug] - fix shodan detector (#1579)
* fix shodan detector.

* fix import order.
2023-07-31 11:12:52 -07:00
ahrav
eb00d0d4e1
[bug] - fix data races (#1577)
* fix data race.

* Add test and fix additional data race.

* address comments.
2023-07-31 11:12:38 -07:00
ahrav
406ce7bc55
Replace magic strings with const (#1568)
* Add normalize repo for azure.

* remove zero values from test cases.

* use const.

* remove azure logic.
2023-07-31 11:12:26 -07:00
ahrav
5e7a6ca11c
Concurrent detection (#1580)
* Run detection on each chunk concurrently.

* Add printer functionality.

* Add logic for dedupe.

* cleanup.

* Moddify number of notifier workers.

* Add comment.

* move consts into fxn.

* buffer resutls chan.

* fix test.

* address comments.

* return an error from Finish.

* fix test.

* fix test.

* linter.

* check err.

* address comments.
2023-07-31 11:12:08 -07:00
Miccah
b54683acb9
gitparse: Use an object for currentDiff (#1573)
* gitparse: Use an object for currentDiff instead of a pointer

* gitparse: Use an object for currentCommit instead of a pointer

* Revert "gitparse: Use an object for currentCommit instead of a pointer"

This reverts commit c5f0708b4a.
2023-07-31 11:39:14 -05:00
Miccah
32e3f1f015
Fix pubnub regular expression (#1565)
One of the sub-groups of the UUIDv4 was missing the characters 0-9.
2023-07-31 11:37:25 -05:00
Miccah
a07b6664f8
Support fatal errors in job reports (#1562)
* Support fatal errors in job reports

* WIP: JobReporter and JobInspector

* WIP: JobReportHook and JobReportRef

* Add ChunkError type and asyncRun helper method

* Rename JobReport to JobProgress

* Return a closed channel from Done when the JobProgress is nil

* Comment catchFirstFatal function
2023-07-31 11:28:30 -05:00