Commit graph

14 commits

Author SHA1 Message Date
Miccah
dd4d4a8a96
Refactor UnitHook to block the scan if finished metrics aren't handled (#2309)
* Refactor UnitHook to block the scan if finished metrics aren't handled

* Log once when back-pressure is detected

* Add hook channel size metric

* Use plural "metrics" for consistency

* Replace LRU cache with map
2024-02-08 14:50:58 -08:00
Miccah
2d96b89554
Add prometheus metrics to measure hook execution time (#2312)
* Add prometheus metrics to measure hook execution time

* Move metrics to separate file and reduce buckets
2024-01-22 11:47:45 -08:00
Miccah
78b5a95342
[chore] Prevent panic when ChunkError has a nil Unit (#2227) 2023-12-15 11:11:28 -08:00
Miccah
9f6a47da3f
[chore] Remove omitempty tags on JobProgressMetrics and UnitMetrics (#2204) 2023-12-12 10:02:56 -08:00
Miccah
39a603d2dc
[chore] Add JSON tags to job metrics (#2114) 2023-11-16 17:08:33 -08:00
Miccah
758344711a
Export ChunkError fields and add ErrorsFor convenience method (#1920) 2023-10-19 08:46:49 -07:00
Miccah
dbcb888063
Update Source interface to use SourceID and JobID types (#1774)
The previous implementation used int64 for both, which can be mixed up
easily. Using distinct types adds a layer of type safety checked by the
compiler.
2023-09-14 11:28:24 -07:00
Miccah
522b2fab29
Add a cancel cause to job cancellation (#1728) 2023-08-30 12:00:44 -07:00
Miccah
5eb776cd61
Support cancelling a run from a JobProgressRef (#1663) 2023-08-25 10:43:33 -07:00
Miccah
61977412df
Add SourceName to JobProgressRef (#1664) 2023-08-25 07:48:25 -07:00
Miccah
f2bfcc7ac6
Capture source-reported progress in JobProgress snapshot (#1661) 2023-08-24 11:28:50 -07:00
Miccah
a4401939a8
Add ElapsedTime method to JobProgressMetrics (#1660) 2023-08-24 11:28:33 -07:00
Miccah
1cd600f70f
Use SourceManager in engine (#1586)
* Add SourceManager to Engine struct

* Update Engine methods to use the SourceManager

* Fix GCS test

The original was testing that `Init()` errors weren't surfaced in
`Finish()`, but the `SourceManager` changed that behavior.

* JobProgress race fixes

* Add contextual values

* Remove unused code

* Add debug logs

* Rename WithConcurrency to WithConcurrentSources

* Always forward chunks to the output chunks channel
2023-08-03 13:36:30 -05:00
Miccah
a07b6664f8
Support fatal errors in job reports (#1562)
* Support fatal errors in job reports

* WIP: JobReporter and JobInspector

* WIP: JobReportHook and JobReportRef

* Add ChunkError type and asyncRun helper method

* Rename JobReport to JobProgress

* Return a closed channel from Done when the JobProgress is nil

* Comment catchFirstFatal function
2023-07-31 11:28:30 -05:00