Commit graph

27 commits

Author SHA1 Message Date
Sonu Saha
b5fcc141d2
CHEF-6437: Implement different version of inspec export (#6816)
* Failing test for export - should not evaluate

Signed-off-by: Clinton Wolfe <clintoncwolfe@gmail.com>

* Sketch out a info_from_parse method

Signed-off-by: Clinton Wolfe <clintoncwolfe@gmail.com>

* Temporary commit to checkpoint experimental work

Signed-off-by: Clinton Wolfe <clintoncwolfe@gmail.com>

* Basic control ids extraction

Signed-off-by: Clinton Wolfe <clintoncwolfe@gmail.com>

* Modify to capture entire block

Signed-off-by: Clinton Wolfe <clintoncwolfe@gmail.com>

* Ability to parse desc, impact and title of a control (#6662)

Signed-off-by: Sonu Saha <sonu.saha@progress.com>

* Rework per-control metadata collectors to be class-based

Signed-off-by: Clinton Wolfe <clintoncwolfe@gmail.com>

* REFACTOR: make a common base class for collectors

Signed-off-by: Clinton Wolfe <clintoncwolfe@gmail.com>

* memoise `info_from_parse`

Signed-off-by: Sathish <sbabu@progress.com>

* Add --legacy-export option to inspec export (#6661)

* support legacy export option

Signed-off-by: Sathish <sbabu@progress.com>

* ability to run legacy export option

Signed-off-by: Sathish <sbabu@progress.com>

---------

Signed-off-by: Sathish <sbabu@progress.com>

* Improve ControlIDCollector and other fields of export data (#6686)

* Parse tags & refs from the ast nodes

Signed-off-by: Sonu Saha <sonu.saha@progress.com>

* ENHANCE: Improve Desc collector to collect description

Signed-off-by: Sonu Saha <sonu.saha@progress.com>

* ENHANCE: Only loop through the child node of begin block

Signed-off-by: Sonu Saha <sonu.saha@progress.com>

* FIX: Fix bug/todo to handle duplicacy of control ids

Signed-off-by: Sonu Saha <sonu.saha@progress.com>

* TEST - a profile which fails to properly be exported but is likely to be used by MITRE

Signed-off-by: Clinton Wolfe <clintoncwolfe@gmail.com>

* Revert "FIX: Fix bug/todo to handle duplicacy of control ids"

This reverts commit 46d66e0026.

* Revert "ENHANCE: Only loop through the child node of begin block"

This reverts commit 47c92d8746.

* ADD: Add code key in control data

Signed-off-by: Sonu Saha <sonu.saha@progress.com>

* ADD: Add source_location key in controls data

Signed-off-by: Sonu Saha <sonu.saha@progress.com>

* HACK: Update the location ref for the controls

Signed-off-by: Sonu Saha <sonu.saha@progress.com>

* FIX: Update variable name as latest changes

Signed-off-by: Sonu Saha <sonu.saha@progress.com>

* FIX: Fix source location ref for all controls in a file

Signed-off-by: Sonu Saha <sonu.saha@progress.com>

* FIX: Improve tagcollector to handle other data types

Signed-off-by: Sonu Saha <sonu.saha@progress.com>

* FIX: Improve tagcollector to handle different types of tags

Signed-off-by: Sonu Saha <sonu.saha@progress.com>

* ENHANCE & TEST: Improve tag collector to collector different tag styles and add test for it

Signed-off-by: Sonu Saha <sonu.saha@progress.com>

* update groups

Signed-off-by: Sathish <sbabu@progress.com>

* Add yml data to export info_from_parse

Signed-off-by: Sonu Saha <sonu.saha@progress.com>

* Add inputs to export data info_from_parse

Signed-off-by: Sonu Saha <sonu.saha@progress.com>

* Add status and status_messages

Signed-off-by: Sonu Saha <sonu.saha@progress.com>

* Initialize all control fields

Signed-off-by: Sonu Saha <sonu.saha@progress.com>

* WIP: Filter controls using --controls

Signed-off-by: Sonu Saha <sonu.saha@progress.com>

* Add inputs collector class - rules remaining

Signed-off-by: Sonu Saha <sonu.saha@progress.com>

* Parse inputs from dsl - 1

Signed-off-by: Sonu Saha <sonu.saha@progress.com>

* TEST: Uncomment tests to verify export

Signed-off-by: Sonu Saha <sonu.saha@progress.com>

* TEST: Include test for different desc

Signed-off-by: Sonu Saha <sonu.saha@progress.com>

* TEST: Include test for different title

Signed-off-by: Sonu Saha <sonu.saha@progress.com>

* TEST: Include test for different ref

Signed-off-by: Sonu Saha <sonu.saha@progress.com>

* Default impact to 0.5 and add test

Signed-off-by: Sonu Saha <sonu.saha@progress.com>

* FIX: Avoid duplicate inputs

Signed-off-by: Sonu Saha <sonu.saha@progress.com>

* Add test for inputs

Signed-off-by: Sonu Saha <sonu.saha@progress.com>

* REFACTOR: Minor refactoring of tests

Signed-off-by: Sonu Saha <sonu.saha@progress.com>

* Uncomment test for refs

Signed-off-by: Sonu Saha <sonu.saha@progress.com>

---------

Signed-off-by: Sonu Saha <sonu.saha@progress.com>
Signed-off-by: Clinton Wolfe <clintoncwolfe@gmail.com>
Signed-off-by: Sathish <sbabu@progress.com>
Co-authored-by: Clinton Wolfe <clintoncwolfe@gmail.com>
Co-authored-by: Sathish <sbabu@progress.com>

* Update option to match inspec's coding standard

Signed-off-by: Sonu Saha <sonu.saha@progress.com>

* Handle inputs within control block

Signed-off-by: Sonu Saha <sonu.saha@progress.com>

* TEST & ENHANCE: Enhance parser and add more tests

Signed-off-by: Sonu Saha <sonu.saha@progress.com>

* FIX: Fix broken test for profile_test

Signed-off-by: Sonu Saha <sonu.saha@progress.com>

* Update groups after filtering control

Signed-off-by: Sonu Saha <sonu.saha@progress.com>

* Add --legacy-export support to inspec json

Signed-off-by: Sonu Saha <sonu.saha@progress.com>

* TEST: Fix broken test & fix group filters

Signed-off-by: Sonu Saha <sonu.saha@progress.com>

* DOCS: Manually update cli.md to include export cmd

Signed-off-by: Sonu Saha <sonu.saha@progress.com>

* Add tag filtering support to export

Signed-off-by: Sonu Saha <sonu.saha@progress.com>

* TEST: Add test for tag and control based filtering

Signed-off-by: Sonu Saha <sonu.saha@progress.com>

* LINT: Fix lint offense

Signed-off-by: Sonu Saha <sonu.saha@progress.com>

* CHORE: Remove addressed todo and update comments

Signed-off-by: Sonu Saha <sonu.saha@progress.com>

* CHEF-6493: Support `--legacy-export` option in `inspec archive` (#6829)

* Introduce --legacy-export flag to archive command

Signed-off-by: Sonu Saha <sonu.saha@progress.com>

* Add more test to verify --legacy-export with archive

Signed-off-by: Sonu Saha <sonu.saha@progress.com>

* Update logic to fetch info based on --legacy-export flag

Signed-off-by: Sonu Saha <sonu.saha@progress.com>

---------

Signed-off-by: Sonu Saha <sonu.saha@progress.com>

* Enhance InputCollector to match pattern instead of to indexing children type to avoid nil errors

Signed-off-by: Sonu Saha <sonu.saha@progress.com>

* Improve RefCollector to handle ref   ({:ref=>'Some ref', :url=>'https://'\}\) syntax

Signed-off-by: Sonu Saha <sonu.saha@progress.com>

* Improve RefCollector and TagCollector to handle variables values from inputs/attributes

Signed-off-by: Sonu Saha <sonu.saha@progress.com>

* Run inspec check using output info_from_parse (#6673)

* Add test fixture profile that emits evaluation markers on stderr

Signed-off-by: Clinton Wolfe <clintoncwolfe@gmail.com>

* Failing test for export - should not evaluate

Signed-off-by: Clinton Wolfe <clintoncwolfe@gmail.com>

* Sketch out a info_from_parse method

Signed-off-by: Clinton Wolfe <clintoncwolfe@gmail.com>

* Temporary commit to checkpoint experimental work

Signed-off-by: Clinton Wolfe <clintoncwolfe@gmail.com>

* Basic control ids extraction

Signed-off-by: Clinton Wolfe <clintoncwolfe@gmail.com>

* Modify to capture entire block

Signed-off-by: Clinton Wolfe <clintoncwolfe@gmail.com>

* Ability to parse desc, impact and title of a control (#6662)

Signed-off-by: Sonu Saha <sonu.saha@progress.com>

* Rework per-control metadata collectors to be class-based

Signed-off-by: Clinton Wolfe <clintoncwolfe@gmail.com>

* REFACTOR: make a common base class for collectors

Signed-off-by: Clinton Wolfe <clintoncwolfe@gmail.com>

* memoise `info_from_parse`

Signed-off-by: Sathish <sbabu@progress.com>

* Add --legacy-export option to inspec export (#6661)

* support legacy export option

Signed-off-by: Sathish <sbabu@progress.com>

* ability to run legacy export option

Signed-off-by: Sathish <sbabu@progress.com>

---------

Signed-off-by: Sathish <sbabu@progress.com>

* Parse tags & refs from the ast nodes

Signed-off-by: Sonu Saha <sonu.saha@progress.com>

* ENHANCE: Improve Desc collector to collect description

Signed-off-by: Sonu Saha <sonu.saha@progress.com>

* ENHANCE: Only loop through the child node of begin block

Signed-off-by: Sonu Saha <sonu.saha@progress.com>

* FIX: Fix bug/todo to handle duplicacy of control ids

Signed-off-by: Sonu Saha <sonu.saha@progress.com>

* TEST - a profile which fails to properly be exported but is likely to be used by MITRE

Signed-off-by: Clinton Wolfe <clintoncwolfe@gmail.com>

* Revert "FIX: Fix bug/todo to handle duplicacy of control ids"

This reverts commit 46d66e0026.

* Revert "ENHANCE: Only loop through the child node of begin block"

This reverts commit 47c92d8746.

* ADD: Add code key in control data

Signed-off-by: Sonu Saha <sonu.saha@progress.com>

* ADD: Add source_location key in controls data

Signed-off-by: Sonu Saha <sonu.saha@progress.com>

* HACK: Update the location ref for the controls

Signed-off-by: Sonu Saha <sonu.saha@progress.com>

* FIX: Update variable name as latest changes

Signed-off-by: Sonu Saha <sonu.saha@progress.com>

* FIX: Fix source location ref for all controls in a file

Signed-off-by: Sonu Saha <sonu.saha@progress.com>

* FIX: Improve tagcollector to handle other data types

Signed-off-by: Sonu Saha <sonu.saha@progress.com>

* FIX: Improve tagcollector to handle different types of tags

Signed-off-by: Sonu Saha <sonu.saha@progress.com>

* ENHANCE & TEST: Improve tag collector to collector different tag styles and add test for it

Signed-off-by: Sonu Saha <sonu.saha@progress.com>

* update groups

Signed-off-by: Sathish <sbabu@progress.com>

* Add yml data to export info_from_parse

Signed-off-by: Sonu Saha <sonu.saha@progress.com>

* Add inputs to export data info_from_parse

Signed-off-by: Sonu Saha <sonu.saha@progress.com>

* Add status and status_messages

Signed-off-by: Sonu Saha <sonu.saha@progress.com>

* Initialize all control fields

Signed-off-by: Sonu Saha <sonu.saha@progress.com>

* make description `default` as a symbol

Signed-off-by: Sathish Babu <sbabu@progress.com>

* define `checks` as Set

Signed-off-by: Sathish Babu <sbabu@progress.com>

* Collect tests as part of collector
and store it in `checks`

Signed-off-by: Sathish Babu <sbabu@progress.com>

* refactor to read `ID` from controls which is an Array now unlike an Hash in `params.controls`

Signed-off-by: Sathish Babu <sbabu@progress.com>

* read yaml params from metadata

Signed-off-by: Sathish Babu <sbabu@progress.com>

* use to Array to simply DS as the o/p ie being converted to JSON

Signed-off-by: Sathish Babu <sbabu@progress.com>

* move old check as legacy check

Signed-off-by: Sathish Babu <sbabu@progress.com>

* support `legacy_check` as an option to run checks in legacy mode

Signed-off-by: Sathish Babu <sbabu@progress.com>

* fix tests to support `legacy_checks`

Signed-off-by: Sathish Babu <sbabu@progress.com>

* update document for check

Signed-off-by: Sathish Babu <sbabu@progress.com>

* Update usage doc for --legaccy-check

Signed-off-by: Clinton Wolfe <clintoncwolfe@gmail.com>

---------

Signed-off-by: Clinton Wolfe <clintoncwolfe@gmail.com>
Signed-off-by: Sonu Saha <sonu.saha@progress.com>
Signed-off-by: Sathish <sbabu@progress.com>
Signed-off-by: Sathish Babu <sbabu@progress.com>
Co-authored-by: Clinton Wolfe <clintoncwolfe@gmail.com>
Co-authored-by: Sonu Saha <98935583+ahasunos@users.noreply.github.com>
Co-authored-by: Sonu Saha <sonu.saha@progress.com>

* LINT: Fix lint offense

Signed-off-by: Sonu Saha <sonu.saha@progress.com>

* do not include tests to controls by default

Signed-off-by: Sathish Babu <sbabu@progress.com>

* generate info with tests for check

Signed-off-by: Sathish Babu <sbabu@progress.com>

---------

Signed-off-by: Clinton Wolfe <clintoncwolfe@gmail.com>
Signed-off-by: Sonu Saha <sonu.saha@progress.com>
Signed-off-by: Sathish <sbabu@progress.com>
Signed-off-by: Sathish Babu <sbabu@progress.com>
Co-authored-by: Clinton Wolfe <clintoncwolfe@gmail.com>
Co-authored-by: Sathish <sbabu@progress.com>
Co-authored-by: Sathish Babu <80091550+sathish-progress@users.noreply.github.com>
2023-11-07 15:15:45 +05:30
Nikita Mathur
b360d64a1c
Changed params usage in inspec archive to skip evaluation (#6741)
* Using metadata to fetch basic info as name and version required in archive

Signed-off-by: Nik08 <nikita.mathur@progress.com>

* Typo fixes

Signed-off-by: Nik08 <nikita.mathur@progress.com>

---------

Signed-off-by: Nik08 <nikita.mathur@progress.com>
2023-09-30 19:33:36 +05:30
Sathish Babu
e1d39e581a
CHEF-6422 Modify inspec archive to not check or export by default (#6659)
* make profile check optional

Signed-off-by: Sathish <sbabu@progress.com>

* Clarify error message when making inspec check optional

Signed-off-by: Clinton Wolfe <clintoncwolfe@gmail.com>

* Add test fixture profile that emits evaluation markers on stderr

Signed-off-by: Clinton Wolfe <clintoncwolfe@gmail.com>

* Make export to inspec.json optional, not enabled by default

Signed-off-by: Clinton Wolfe <clintoncwolfe@gmail.com>

* Add test confirming non-evaluative archive

Signed-off-by: Clinton Wolfe <clintoncwolfe@gmail.com>

* Fix failing plugin search test, failing since we published a new test gem

Signed-off-by: Clinton Wolfe <clintoncwolfe@gmail.com>

---------

Signed-off-by: Sathish <sbabu@progress.com>
Signed-off-by: Clinton Wolfe <clintoncwolfe@gmail.com>
Co-authored-by: Clinton Wolfe <clintoncwolfe@gmail.com>
2023-09-29 18:12:37 -04:00
Nikita Mathur
1de38f3b13 Build fix for archive and vendor tests due to cookstyle offenses
Signed-off-by: Nikita Mathur <nikita.mathur@chef.io>
2021-09-29 13:23:35 +05:30
Nick Schwaderer
989d86c2ce Stores output of inspec json to archive and artifact
Signed-off-by: Nick Schwaderer <nschwaderer@chef.io>
2020-05-20 12:34:17 +01:00
Clinton Wolfe
28f8d6cc5f Removing skip in archive test just seems to work, too
Signed-off-by: Clinton Wolfe <clintoncwolfe@gmail.com>
2020-01-28 12:24:29 -05:00
Ryan Davis
07dc5e3192 First pass at cleaning deprecations for old minitest/spec-style tests.
3 files left to go, and they're behaving oddly so I'm leaving them out
in this pass. Looks like 21 deprecations left.

Signed-off-by: Ryan Davis <zenspider@chef.io>
2019-10-03 13:45:19 -07:00
Ryan Davis
fc1ad67ffc More parallelization for functionals.
Signed-off-by: Ryan Davis <zenspider@chef.io>
2019-09-17 11:50:31 -07:00
Ryan Davis
917a531a92 Skipped all failing tests on windows AT the failures site.
This activates a BUNCH of tests but keeps us green.

Whether these tests are valid or not is not being addressed here.

Signed-off-by: Ryan Davis <zenspider@chef.io>
2019-07-24 18:09:08 -07:00
Ryan Davis
18ce4850f0 Initial pass to clean up functional testing wrt exit codes and output.
This should get windows tests running again? Hopefully? They're
running SO slowly on my vagrant box that I'm just pushing for now.
They're clean on the mac side.

Signed-off-by: Ryan Davis <zenspider@chef.io>
2019-07-24 18:06:25 -07:00
Ryan Davis
fae4230a41 Removing inspec/profile_vendor from inspec/base_cli.
This was the next most expensive require in the analysis.

Also rearranged the way that ui handled tables to be lazy.

```
% SLOW=1 time rake test:functional

before: Finished in  681.514579s, 0.5136 runs/s, 2.9919 assertions/s.
after : Finished in  642.655918s, 0.5446 runs/s, 3.1728 assertions/s.
```

Signed-off-by: Ryan Davis <zenspider@chef.io>
2019-06-11 19:16:58 -07:00
Ryan Davis
a5309ea392 blindly applied chefstyle -a
Signed-off-by: Ryan Davis <zenspider@chef.io>
2019-06-11 18:52:03 -07:00
Ryan Davis
58fa148773 Stabilize windows functional testing.
Skip most everything.

After some digging, in those tests that didn't have 100% failures, of
the ~10+% passing, those tests weren't checking enough. So I skip them
too in the hopes that we improve testing across the board.

At this point, we need appveyor to be green more than we need these
tests to be fixed. If that means we skip them, so be it.

These tests will time-bomb at the end of July.

Signed-off-by: Ryan Davis <zenspider@chef.io>
2019-06-03 23:27:26 -07:00
Miah Johnson
2a1b1d8e88 Remove all leading newlines.
Signed-off-by: Miah Johnson <miah@chia-pet.org>
2019-05-31 11:43:44 -07:00
Miah Johnson
6c19e81a06 Remove authors tags from source files.
Signed-off-by: Miah Johnson <miah@chia-pet.org>
2019-05-28 12:47:39 -07:00
Miah Johnson
659b4b373a Remove # encoding: utf8 magic comments
Signed-off-by: Miah Johnson <miah@chia-pet.org>
2019-05-07 16:06:23 -07:00
Clinton Wolfe
f7d58ad92c Replace most mentions of attribute with input in test/
Signed-off-by: Clinton Wolfe <clintoncwolfe@gmail.com>
2019-02-20 20:41:10 -05:00
Clinton Wolfe
785888a209 Rename functional tests, unit tests, and test fixtures
Signed-off-by: Clinton Wolfe <clintoncwolfe@gmail.com>
2019-02-08 18:23:01 -05:00
Clinton Wolfe
102505a937 Use new config file system to read config
Signed-off-by: Clinton Wolfe <clintoncwolfe@gmail.com>
2019-02-01 14:24:26 -05:00
Jared Quick
d33d189d93
Create appvayor workers for Windows functional tests. (#3397)
* Add more windows functional tests and create some appvayor workers.
* Fix uuid test for automate

Signed-off-by: Jared Quick <jquick@chef.io>
2018-11-08 12:00:14 -05:00
Jared Quick
0fc83871a1
Fix functional tests. (#3572)
Signed-off-by: Jared Quick <jquick@chef.io>
2018-11-02 12:52:54 -04:00
Jared Quick
cbb3b600d9
Fix archive with required attributes (#3468)
* Skip require check unless we are in a exec call.
* Add testing
* Update comment.

Signed-off-by: Jared Quick <jquick@chef.io>
2018-10-05 16:24:26 -04:00
Jerry Aldrich
9d031053ea Various improvements to vendor command (#3286)
Signed-off-by: Jerry Aldrich <jerryaldrichiii@gmail.com>
2018-09-06 23:28:08 -04:00
Lee Briggs
02530310cd Updating tests for profile versions
Signed-off-by: Lee Briggs <lbriggs@apptio.com>
2016-12-14 16:20:27 +00:00
Steven Danna
3777f06927
Remove some warnings during the test run 2016-09-09 14:14:36 +01:00
Steven Danna
3a6e610de9 Allow functional tests to pass on OSX
A few minor issues were causing 3 functional test failures on OS X.
These were not program errors but where rather the result of the
profiles under test assuming a linux environment.

Since many of the developers who will work on this project in the future
will be running OS X, let's ensure they can run the functional tests
easily.

Signed-off-by: Steven Danna <steve@chef.io>
2016-08-26 15:25:59 +02:00
Dominik Richter
17840e0299 split up functional tests 2016-03-25 01:32:03 +01:00