From b611c9c2c8e1885ed71802eff586c7ca61e4acc3 Mon Sep 17 00:00:00 2001 From: Clinton Wolfe Date: Sun, 19 Jun 2022 22:02:54 -0400 Subject: [PATCH 1/3] First pass at enhanced outcomes design doc Signed-off-by: Clinton Wolfe --- dev-docs/enhanced-outcomes.md | 58 +++++++++++++++++++++++++++++++++++ 1 file changed, 58 insertions(+) create mode 100644 dev-docs/enhanced-outcomes.md diff --git a/dev-docs/enhanced-outcomes.md b/dev-docs/enhanced-outcomes.md new file mode 100644 index 000000000..ba03314b2 --- /dev/null +++ b/dev-docs/enhanced-outcomes.md @@ -0,0 +1,58 @@ +# Enhanced Outcomes + +Enhanced Outcomes refers to the addition of several new control outcomes to the InSpec vocabulary. + +## Test Outcomes vs Control Outcomes + +It is important to be clear that Enhanced Outcomes refers to new **control outcomes** - the results of running a control. The results of running a test - a `describe block` are determined by RSpec, and are limited to Pass, Fail, and Skip. Test outcomes are much more difficult to extend than control outcomes. + +## New Control Outcomes + +Enhanced Outcomes adds three new control outcomes to the existing Pass, Fail and Skip outcomes. + +### Error + +The Error outcome represents the situation when the system has encountered an error when attempting to evaluate the control code. But to a technical limitation, an accurate outcome is not possible to be obtained. + +In the first iteration of Enhanced Outcomes, Error outcome will be detected: + + * if the status of a test is Failed AND + * if the message of the test includes the text "Control source error" + +then the entire control should be marked Error. + +Additional means of detecting Error may be developed in the future. + +Error's abbreviation is ERR. + +Error's UI color assignment is Indigo. + +### Not Applicable + +If the control is not in Error and the impact of a control is 0.0, then the control's outcome is Not Applicable. + +Not Applicable's abbreviation is N/A. + +Not Applicable's UI color assignment is Sky Blue. + +### Not Reviewed + +If the control is not in Error or Not Applicable and all test results are Skipped, then the control outcome is Not Reviewed. + +Not Reviewed replaces Skipped as a control outcome. + +Not Reviewed's abbreviation is N/R. + +Not Reviewed's UI color assignment is Amber. + +## The --enhanced-outcomes option + +A new CLI option will be introduced for `inspec exec`, `inspec shell`, and `inspec schema` that controls the Enhanced Outcomes functionality. + +### InSpec 5 + +In InSpec 5.x, a user must request the enhanced outcomes functionality explicitly by adding the --enhanced-outcomes option. + +### InSpec 6 + +In InSpec 6.x, --enhanced-outcomes will default to true. A user may request disabling the enhanced outcomes functionality by adding the --no-enhanced-outcomes option. From 1dc2faf2059217d7f0174c8306c56b3f497788ac Mon Sep 17 00:00:00 2001 From: Deepa Kumaraswamy Date: Thu, 23 Jun 2022 18:33:23 +0530 Subject: [PATCH 2/3] Proof-read Signed-off-by: Deepa Kumaraswamy --- dev-docs/enhanced-outcomes.md | 50 +++++++++++++++++------------------ 1 file changed, 24 insertions(+), 26 deletions(-) diff --git a/dev-docs/enhanced-outcomes.md b/dev-docs/enhanced-outcomes.md index ba03314b2..514e7b9c1 100644 --- a/dev-docs/enhanced-outcomes.md +++ b/dev-docs/enhanced-outcomes.md @@ -1,58 +1,56 @@ # Enhanced Outcomes -Enhanced Outcomes refers to the addition of several new control outcomes to the InSpec vocabulary. +Enhanced Outcomes refers to the addition of new control outcomes to the InSpec vocabulary. -## Test Outcomes vs Control Outcomes +## Test Outcomes vs. Control Outcomes -It is important to be clear that Enhanced Outcomes refers to new **control outcomes** - the results of running a control. The results of running a test - a `describe block` are determined by RSpec, and are limited to Pass, Fail, and Skip. Test outcomes are much more difficult to extend than control outcomes. +It is essential to understand that Enhanced Outcomes refers to new **control outcomes**, the results of running a control. -## New Control Outcomes +The results of running a test, a `describe block` are determined by RSpec and are limited to `Pass`, `Fail`, and `Skip`. -Enhanced Outcomes adds three new control outcomes to the existing Pass, Fail and Skip outcomes. +Test outcomes are much more difficult to extend than control outcomes. + + + +Enhanced Outcomes adds three new control outcomes to the existing `Pass`, `Fail`, and `Skip` outcomes. ### Error -The Error outcome represents the situation when the system has encountered an error when attempting to evaluate the control code. But to a technical limitation, an accurate outcome is not possible to be obtained. +The **Error outcome** represents the situation when the system has encountered an error when attempting to evaluate the control code. However, due to a technical limitation, an accurate outcome cannot be obtained. -In the first iteration of Enhanced Outcomes, Error outcome will be detected: +In the first iteration of Enhanced Outcomes, the Error outcome is detected: - * if the status of a test is Failed AND - * if the message of the test includes the text "Control source error" +* if the status of a test is `Failed`, and +* if the message of the test includes the text, *Control source error* -then the entire control should be marked Error. +Then, the entire control should be marked `Error`. -Additional means of detecting Error may be developed in the future. +Additional means of detecting error may be developed in the future. -Error's abbreviation is ERR. - -Error's UI color assignment is Indigo. +Error's abbreviation is `ERR`. Error's UI color assignment is `Indigo`. ### Not Applicable -If the control is not in Error and the impact of a control is 0.0, then the control's outcome is Not Applicable. +If the control is not in `Error` and the impact of control is `0.0`, then the control's outcome is `Not Applicable`. -Not Applicable's abbreviation is N/A. - -Not Applicable's UI color assignment is Sky Blue. +Not Applicable's abbreviation is `N/A`. Not Applicable's UI color assignment is `Sky Blue`. ### Not Reviewed -If the control is not in Error or Not Applicable and all test results are Skipped, then the control outcome is Not Reviewed. +If the control is not in `Error` or `Not Applicable`, and all test results are `Skipped`, then the control outcome is `Not Reviewed`. -Not Reviewed replaces Skipped as a control outcome. +Not Reviewed replaces `Skipped` as a control outcome. -Not Reviewed's abbreviation is N/R. +Not Reviewed's abbreviation is `N/R`. Not Reviewed's UI color assignment is `Amber`. -Not Reviewed's UI color assignment is Amber. - -## The --enhanced-outcomes option +## The `--enhanced-outcomes` option A new CLI option will be introduced for `inspec exec`, `inspec shell`, and `inspec schema` that controls the Enhanced Outcomes functionality. ### InSpec 5 -In InSpec 5.x, a user must request the enhanced outcomes functionality explicitly by adding the --enhanced-outcomes option. +In InSpec 5.x, a user must request the enhanced outcomes functionality explicitly by adding the `--enhanced-outcomes` option. ### InSpec 6 -In InSpec 6.x, --enhanced-outcomes will default to true. A user may request disabling the enhanced outcomes functionality by adding the --no-enhanced-outcomes option. +In InSpec 6.x, --enhanced-outcomes will default to `true`. A user may request disabling the enhanced outcomes functionality by adding the `--no-enhanced-outcomes` option. From fe8248bfb2535f2e1c7684ee17a6438b96656ff0 Mon Sep 17 00:00:00 2001 From: Clinton Wolfe Date: Thu, 23 Jun 2022 10:28:25 -0400 Subject: [PATCH 3/3] Adjust criteria for Error Signed-off-by: Clinton Wolfe --- dev-docs/enhanced-outcomes.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/dev-docs/enhanced-outcomes.md b/dev-docs/enhanced-outcomes.md index ba03314b2..fe94532f1 100644 --- a/dev-docs/enhanced-outcomes.md +++ b/dev-docs/enhanced-outcomes.md @@ -16,8 +16,8 @@ The Error outcome represents the situation when the system has encountered an er In the first iteration of Enhanced Outcomes, Error outcome will be detected: - * if the status of a test is Failed AND - * if the message of the test includes the text "Control source error" + * if the message of any test includes the text "Control source error" OR + * the result of any test includes a backtrace then the entire control should be marked Error.