From 29242deb7ceb3b1e292c7ea58501953c89b31c81 Mon Sep 17 00:00:00 2001 From: Clinton Wolfe Date: Mon, 22 Jul 2024 09:56:24 -0400 Subject: [PATCH] Usage Telemetry v3 (#6012) * Remove unused telemetry v1 code Signed-off-by: Clinton Wolfe * Sketch out basics of telemetry, with start/stop of invocation telemetry Signed-off-by: Clinton Wolfe * Data structure for run telemetry - job capture Signed-off-by: Clinton Wolfe * Add per-control and per-run feature detection Signed-off-by: Clinton Wolfe * CHEF-4017 Telemetry job api updations (#6965) * Added initial changes to jobs api Signed-off-by: Nik08 * Feature flag changes for telemetry Signed-off-by: Nik08 * move base, debug and null to its own file structure Signed-off-by: Sathish * make HTTP client post requests Signed-off-by: Sathish * remove old logic Signed-off-by: Sathish * make backend class as `HTTP` Signed-off-by: Sathish * CHEF-7258 Fetch and use licensing information for telemetry (#6964) * Added method to fetch license ids for inspec Signed-off-by: Nik08 * Added free license check for performing telemetry api call Signed-off-by: Nik08 * move base, debug and null to its own file structure Signed-off-by: Sathish * make HTTP client post requests Signed-off-by: Sathish * remove old logic Signed-off-by: Sathish * make backend class as `HTTP` Signed-off-by: Sathish --------- Signed-off-by: Nik08 Signed-off-by: Sathish Co-authored-by: Sathish * Updated control tags and desc value to be used in jobs api Signed-off-by: Nik08 * Added checks for automate run context and free license check Signed-off-by: Nik08 * capture target mode and id Signed-off-by: Sathish * profile doesn't need ID Signed-off-by: Sathish * use run context to set environment data Signed-off-by: Sathish * refactor `create_wrapper` to be localized Signed-off-by: Sathish * change all timestamps to be UTC Signed-off-by: Sathish * Null checks for response and corrected job api endpoint Signed-off-by: Nik08 * Fixed tag values to be sent as string in api call Signed-off-by: Nik08 * make version as float Signed-off-by: Sathish * add platform name Signed-off-by: Sathish * Added control result data in jobs api payload Signed-off-by: Nik08 * Debug logs added for telemetry call Signed-off-by: Nik08 * Removed unwanted telemetry debug class Signed-off-by: Nik08 * Payload fix to pass features data only on per control basis Signed-off-by: Nik08 * Added class function to list all invoked features by feature sub system Signed-off-by: Nik08 * Using feature system to get all invoked features list to be used in jobs api Signed-off-by: Nik08 * Unit tests cases updated and fixed Signed-off-by: Nik08 * License type check downcased Signed-off-by: Nik08 * Lint fix Signed-off-by: Nik08 * CHEF-7265 Telemetry opt-in for CINC users (#6966) * Enabled telemtry opt-in Signed-off-by: Nik08 * Removed old comments Signed-off-by: Nik08 * Unit test case added to validate the disabling telemetry behaviour for inspec user Signed-off-by: Nik08 --------- Signed-off-by: Nik08 --------- Signed-off-by: Nik08 Signed-off-by: Sathish Co-authored-by: Sathish * Product team review changes - only disable telemetry for commercial license users Signed-off-by: Nik08 * Connection failure handling for telemetry http call Signed-off-by: Nik08 * Testing fix - Remove usage of deleted library Signed-off-by: Nik08 * Telemetry test case fix - Issue caused because unit test are run without feature flag env set Signed-off-by: Nik08 * Fixed and replaced tightly coupled semver versioning regex matching test for telemetry data Signed-off-by: Nik08 * Telemery test fix to use license key from env or a dummy value if not set in env Signed-off-by: Nik08 * Added error logs in case the http call is not successful for telemetry Signed-off-by: Nik08 * Error handling for telemetry start and run calls Signed-off-by: Nik08 * Telemetry opt-in changes (#7055) * Removed usage of feature system to enable telemetry - making it opt-in by default Signed-off-by: Nik08 * Telemetry disable check fix when no option is passed in args Signed-off-by: Nik08 * Fix in test to use license specs defined for testing Signed-off-by: Nik08 --------- Signed-off-by: Nik08 * (Restoring) CHEF-10392 load default telemetry url conditionally (#7059) * load default telemetry url conditionally Signed-off-by: Sathish * remove version base path version base path is defined in jobs path already Signed-off-by: Sathish * use `CHEF_` prefix for the ENV Signed-off-by: Sathish --------- Signed-off-by: Sathish Co-authored-by: Sathish * Typo fix in features list Signed-off-by: Nik08 * Stub added for CI license key Signed-off-by: Nik08 * License usage telemetry correction - not track control results (#7060) Signed-off-by: Nik08 * Changes to disable telemetry for other InSpec distros (#7065) Signed-off-by: Nik08 * Lint issue fix Signed-off-by: Nik08 * Removing disable telemetry test - breaks on CI because of commercial license usage Signed-off-by: Nik08 * CHEF-13228 Chef licensing telemetry documentation (#7056) * WIP chef telemetry env variable usage updated Signed-off-by: Nik08 * WIP intro added for chef telemetry - requires edit Signed-off-by: Nik08 * Correction in opt in behaviour of telemetry Signed-off-by: Nik08 * Doc update after default opt in changes Signed-off-by: Nik08 * Doc edit from product Signed-off-by: Nik08 * Doc edit Signed-off-by: Nik08 * Edits Signed-off-by: Ian Maddaus --------- Signed-off-by: Nik08 Signed-off-by: Ian Maddaus Co-authored-by: Ian Maddaus * Updated version pinning of chef licensing to version 1 for chef telemetry Signed-off-by: Nik08 --------- Signed-off-by: Clinton Wolfe Signed-off-by: Nik08 Signed-off-by: Sathish Signed-off-by: Ian Maddaus Co-authored-by: Nikita Mathur Co-authored-by: Sathish Co-authored-by: Nik08 Co-authored-by: Ian Maddaus --- docs-chef-io/content/inspec/license.md | 8 + etc/features.sig | 12 +- etc/features.yaml | 3 + inspec-core.gemspec | 3 +- lib/inspec.rb | 1 - lib/inspec/cli.rb | 2 +- lib/inspec/runner.rb | 3 + lib/inspec/utils/telemetry.rb | 78 +++++++++- lib/inspec/utils/telemetry/base.rb | 147 ++++++++++++++++++ lib/inspec/utils/telemetry/collector.rb | 81 ---------- lib/inspec/utils/telemetry/data_series.rb | 44 ------ lib/inspec/utils/telemetry/global_methods.rb | 22 --- lib/inspec/utils/telemetry/http.rb | 40 +++++ lib/inspec/utils/telemetry/null.rb | 11 ++ .../utils/telemetry/run_context_probe.rb | 14 +- .../reporters/run_data_test_profile_a.json | 1 + test/fixtures/valid_client_api_data.json | 57 +++++++ test/unit/utils/telemetry/collector_test.rb | 60 ------- test/unit/utils/telemetry/data_series_test.rb | 60 ------- .../utils/telemetry/global_methods_test.rb | 33 ---- test/unit/utils/telemetry_test.rb | 124 +++++++++++++++ 21 files changed, 491 insertions(+), 313 deletions(-) create mode 100644 lib/inspec/utils/telemetry/base.rb delete mode 100644 lib/inspec/utils/telemetry/collector.rb delete mode 100644 lib/inspec/utils/telemetry/data_series.rb delete mode 100644 lib/inspec/utils/telemetry/global_methods.rb create mode 100644 lib/inspec/utils/telemetry/http.rb create mode 100644 lib/inspec/utils/telemetry/null.rb create mode 100644 test/fixtures/reporters/run_data_test_profile_a.json create mode 100644 test/fixtures/valid_client_api_data.json delete mode 100644 test/unit/utils/telemetry/collector_test.rb delete mode 100644 test/unit/utils/telemetry/data_series_test.rb delete mode 100644 test/unit/utils/telemetry/global_methods_test.rb create mode 100644 test/unit/utils/telemetry_test.rb diff --git a/docs-chef-io/content/inspec/license.md b/docs-chef-io/content/inspec/license.md index aa9418daf..ba64ebadc 100644 --- a/docs-chef-io/content/inspec/license.md +++ b/docs-chef-io/content/inspec/license.md @@ -211,3 +211,11 @@ inspec exec ``` This capability is basic and you must synchronize the license servers, otherwise you may get inconsistent results. + +## Licensing Telemetry service + +The Chef Licensing Telemetry service gathers product activation, product usage trends and statistics, environment information, bugs, and other data related to the use of Chef InSpec. + +This feature is enabled for free and trial tiers only and isn't enabled for commercial users. + +For more information on the data gathered by the Licensing Telemetry service, see the [Progress Privacy Policy](https://www.progress.com/legal/privacy-policy). diff --git a/etc/features.sig b/etc/features.sig index bb9282400..aa97863b5 100644 --- a/etc/features.sig +++ b/etc/features.sig @@ -1,6 +1,6 @@ -LoJePRrMIqFz6d1uu5n3QBqQAPD8wLuLM8PfvdDerFjuX/TFJDFdwdcNZ8b8 -KBxFjR5qUTMZizjIUp5Jd6FFI4gSm0RIMKa4UeJCQQAWKJGo/tIbSKLPLWlV -m1X1Z869AkvQSJxyaXvS2oKPck/znCbRKEDhuk2kqSyDJlC2BILTVa0sx3nd -4W2J2CwFBlqmYWI1FARkZCMGlfzkjcUqrVrCb3RcZ7bcEYOT5ebIm9zZlbuV -n2Di29KFZhl8paEoGq3EYJvxEC7rVtLccei8UteNQcSOWihG61dtPGhHnpS+ -/7RNGjrS8s4i/dQHjZlZgV6guki6EqB+DIirVek9PQ== +nr7EKXZMiAwYI0Kon1ctCMkDulEkovRbT/FRezvP04yx8wVhJaSi7dMhL/mP +NvTzMOuT9G4R/QsP6VV7QKs4eBmAOPGrvgZgyfXDvfe1TPYcvpsVncSXm5rx +TO+g7i0XGz9s/FtvdzOpl2urhgOsQ35wk7IsNu9Ktij2HqZw7UmxMvtT954s +aQuW6eVvvM9n+bobEBVSErkhgvOvJ7jZyz5r0cv/uuhrayIC6V1qegod9QHa +uCdasmmEqglyNQYXIM7V7iNrnfuYB80or44Ewi640edHarSw8YU/Tul2Y2l/ +DWeXRHsXxmuEL1wXA9ZIV6wqK0RsxaufwY6M7bqWSQ== diff --git a/etc/features.yaml b/etc/features.yaml index bc0afc4fe..1f42324aa 100644 --- a/etc/features.yaml +++ b/etc/features.yaml @@ -92,3 +92,6 @@ inspec-audit-logging: description: Use audit logging. env_preview: true + inspec-telemetry-client: + description: Perform license usage telemetry. + env_preview: true diff --git a/inspec-core.gemspec b/inspec-core.gemspec index afa5e17aa..fc607a2a9 100644 --- a/inspec-core.gemspec +++ b/inspec-core.gemspec @@ -62,5 +62,6 @@ Source code obtained from the Chef GitHub repository is made available under Apa spec.add_dependency "cookstyle" spec.add_dependency "train-core", ">= 3.11.0" - spec.add_dependency "chef-licensing", ">= 0.7.5" + # Minimum major version 1 is required for Chef licensing telemetry + spec.add_dependency "chef-licensing", ">= 1.0.0" end diff --git a/lib/inspec.rb b/lib/inspec.rb index bce81b4c5..a4c8a5a62 100644 --- a/lib/inspec.rb +++ b/lib/inspec.rb @@ -19,7 +19,6 @@ require "inspec/rspec_extensions" require "inspec/globals" require "inspec/impact" require "inspec/utils/telemetry" -require "inspec/utils/telemetry/global_methods" require "inspec/plugin/v2" require "inspec/plugin/v1" diff --git a/lib/inspec/cli.rb b/lib/inspec/cli.rb index 7dd5eb8c4..00fa4e7fe 100644 --- a/lib/inspec/cli.rb +++ b/lib/inspec/cli.rb @@ -57,7 +57,7 @@ class Inspec::InspecCLI < Inspec::BaseCLI desc: "Disable loading all plugins that the user installed." class_option :enable_telemetry, type: :boolean, - desc: "Allow or disable telemetry", default: false + desc: "Allow or disable telemetry", default: true require "license_acceptance/cli_flags/thor" include LicenseAcceptance::CLIFlags::Thor diff --git a/lib/inspec/runner.rb b/lib/inspec/runner.rb index edacbc919..63274e16e 100644 --- a/lib/inspec/runner.rb +++ b/lib/inspec/runner.rb @@ -12,6 +12,7 @@ require "inspec/dist" require "inspec/reporters" require "inspec/runner_rspec" require "chef-licensing" +require "inspec/utils/telemetry" # spec requirements module Inspec @@ -179,6 +180,7 @@ module Inspec } Inspec::Log.debug "Starting run with targets: #{@target_profiles.map(&:to_s)}" + Inspec::Telemetry.run_starting(runner: self, conf: @conf) load run_tests(with) rescue ChefLicensing::SoftwareNotEntitled @@ -227,6 +229,7 @@ module Inspec @run_data = @test_collector.run(with) # dont output anything if we want a report render_output(@run_data) unless @conf["report"] + Inspec::Telemetry.run_ending(runner: self, run_data: @run_data, conf: @conf) @test_collector.exit_code end diff --git a/lib/inspec/utils/telemetry.rb b/lib/inspec/utils/telemetry.rb index 43e8c7129..c22d927c6 100644 --- a/lib/inspec/utils/telemetry.rb +++ b/lib/inspec/utils/telemetry.rb @@ -1,3 +1,75 @@ -require "inspec/utils/telemetry/collector" -require "inspec/utils/telemetry/data_series" -require "inspec/utils/telemetry/global_methods" +require "time" unless defined?(Time.zone_offset) +require "chef-licensing" +require_relative "telemetry/null" +require_relative "telemetry/http" +require_relative "telemetry/run_context_probe" + +module Inspec + class Telemetry + + @@instance = nil + @@config = nil + + def self.instance + @@instance ||= determine_backend_class.new + end + + def self.determine_backend_class + # Don't perform telemetry action for other InSpec distros + # Don't perform telemetry action if running under Automate - Automate does LDC tracking for us + # Don't perform telemetry action if license is a commercial license + + if Inspec::Dist::EXEC_NAME != "inspec" || + Inspec::Telemetry::RunContextProbe.under_automate? || + license&.license_type&.downcase == "commercial" + + return Inspec::Telemetry::Null + end + + if Inspec::Dist::EXEC_NAME == "inspec" && telemetry_disabled? + # Issue a warning if an InSpec user is explicitly trying to opt out of telemetry using cli option + Inspec::Log.warn "Telemetry opt-out is not permissible." + end + + Inspec::Log.debug "Determined HTTP instance for telemetry" + + Inspec::Telemetry::HTTP + end + + def self.license + Inspec::Log.debug "Fetching license context for telemetry" + @license = ChefLicensing.license_context + end + + ###### + # These class methods make it convenient to call from anywhere within the InSpec codebase. + ###### + def self.run_starting(opts) + Inspec::Log.debug "Initiating telemetry for InSpec" + @@config ||= opts[:conf] + instance.run_starting(opts) + rescue StandardError => e + Inspec::Log.debug "Encountered error in Telemetry start run call -> #{e.message}" + end + + def self.run_ending(opts) + @@config ||= opts[:conf] + instance.run_ending(opts) + Inspec::Log.debug "Finishing telemetry for InSpec" + rescue StandardError => e + Inspec::Log.debug "Encountered error in Telemetry end run call -> #{e.message}" + end + + def self.note_feature_usage(feature_name) + instance.note_feature_usage(feature_name) + end + + def self.config + @@config + end + + def self.telemetry_disabled? + config.telemetry_options["enable_telemetry"].nil? ? false : !config.telemetry_options["enable_telemetry"] + end + end +end diff --git a/lib/inspec/utils/telemetry/base.rb b/lib/inspec/utils/telemetry/base.rb new file mode 100644 index 000000000..ded362fe4 --- /dev/null +++ b/lib/inspec/utils/telemetry/base.rb @@ -0,0 +1,147 @@ +# frozen_string_literal: true +require "chef-licensing" +require "securerandom" unless defined?(SecureRandom) +require "digest" unless defined?(Digest) +require_relative "../../dist" +module Inspec + class Telemetry + class Base + VERSION = 2.0 + TYPE = "job" + JOB_TYPE = "InSpec" + + attr_accessor :scratch + + def fetch_license_ids + Inspec::Log.debug "Fetching license IDs for telemetry" + @license_keys ||= ChefLicensing.license_keys + end + + def create_wrapper + Inspec::Log.debug "Initialising wrapper for telemetry" + { + version: VERSION, + createdTimeUTC: Time.now.getutc.iso8601, + environment: Inspec::Telemetry::RunContextProbe.guess_run_context, + licenseIds: fetch_license_ids, + source: "#{Inspec::Dist::EXEC_NAME}:#{Inspec::VERSION}", + type: TYPE, + } + end + + def note_feature_usage(feature_name) + @scratch ||= {} + @scratch[:features] ||= [] + @scratch[:features] << feature_name + end + + def run_starting(_opts = {}) + @scratch ||= {} + @scratch[:features] ||= [] + @scratch[:run_start_time] = Time.now.getutc.iso8601 + end + + def run_ending(opts) + note_per_run_features(opts) + + payload = create_wrapper + + train_platform = opts[:runner].backend.backend.platform + payload[:platform] = train_platform.name + + payload[:jobs] = [{ + type: JOB_TYPE, + + # Target platform info + environment: { + host: obscure(URI(opts[:runner].backend.backend.uri).host) || "unknown", + os: train_platform.name, + version: train_platform.release, + architecture: train_platform.arch || "", + id: train_platform.uuid, + }, + + runtime: Inspec::VERSION, + content: [], # one content == one profile + steps: [], # one step == one control + }] + + opts[:run_data][:profiles].each do |profile| + payload[:jobs][0][:content] << { + name: obscure(profile[:name]), + version: profile[:version], + sha256: profile[:sha256], + maintainer: profile[:maintainer] || "", + type: "profile", + } + + profile[:controls].each do |control| + payload[:jobs][0][:steps] << { + id: obscure(control[:id]), + name: "inspec-control", + description: control[:desc] || "", + target: { + mode: opts[:runner].backend.backend.backend_type, + id: opts[:runner].backend.backend.platform.uuid, + }, + resources: [], + features: [], + tags: format_control_tags(control[:tags]), + } + + control[:results]&.each do |resource_block| + payload[:jobs][0][:steps].last[:resources] << { + type: "inspec-resource", + name: resource_block[:resource_class], + id: obscure(resource_block[:resource_title].respond_to?(:resource_id) ? resource_block[:resource_title].resource_id : nil) || "unknown", + } + end + + # Per-control features. + payload[:jobs][0][:steps].last[:features] = scratch[:features].dup + end + end + + Inspec::Log.debug "Final data for telemetry upload -> #{payload}" + # Return payload object for testing + payload + end + + def format_control_tags(tags) + tags_list = [] + tags.each do |key, value| + tags_list << { name: key.to_s, value: (value || "").to_s } + end + tags_list + end + + # Hash text if non-nil + def obscure(cleartext) + return nil if cleartext.nil? + return nil if cleartext.empty? + + Digest::SHA2.new(256).hexdigest(cleartext) + end + + def note_per_run_features(opts) + note_all_invoked_features + note_gem_dependency_usage(opts) + end + + def note_all_invoked_features + Inspec::Feature.list_all_invoked_features.each do |feature| + Inspec::Telemetry.note_feature_usage(feature.to_s) + end + end + + def note_gem_dependency_usage(opts) + unless opts[:runner].target_profiles.map do |tp| + tp.metadata.gem_dependencies + \ + tp.locked_dependencies.list.map { |_k, v| v.profile.metadata.gem_dependencies }.flatten + end.flatten.empty? + Inspec::Telemetry.note_feature_usage("inspec-gem-deps-in-profiles") + end + end + end + end +end diff --git a/lib/inspec/utils/telemetry/collector.rb b/lib/inspec/utils/telemetry/collector.rb deleted file mode 100644 index 5e6b42036..000000000 --- a/lib/inspec/utils/telemetry/collector.rb +++ /dev/null @@ -1,81 +0,0 @@ -require "inspec/config" -require "inspec/utils/telemetry/data_series" -require "singleton" unless defined?(Singleton) - -module Inspec::Telemetry - # A Singleton collection of data series objects. - class Collector - include Singleton - - attr_reader :config - - def initialize - @data_series = [] - @telemetry_toggled_off = false - load_config - end - - # Allow loading a configuration, useful when testing. - def load_config(config = Inspec::Config.cached) - @config = config - end - - # Add a data series to the collection. - # @return [True] - def add_data_series(data_series) - @data_series << data_series - end - - # The loaded configuration should have a option to configure - # telemetry, if not default to false. - # @return [True, False] - def telemetry_enabled? - if @telemetry_toggled_off - false - else - config_telemetry_options.fetch("enable_telemetry", false) - end - end - - # A way to disable the telemetry system. - def disable_telemetry - @telemetry_toggled_off = true - end - - # The entire data series collection. - # @return [Array] - def list_data_series - @data_series - end - - # Finds the data series object with the specified name and returns it. - # If it does not exist then creates a new data series with that name - # and returns it. - # @return [Inspec::Telemetry::DataSeries] - def find_or_create_data_series(name) - ds = @data_series.select { |data_series| data_series.name.eql?(name) } - if ds.empty? - new_data_series = Inspec::Telemetry::DataSeries.new(name) - @data_series << new_data_series - new_data_series - else - ds.first - end - end - - # Blanks the contents of the data series collection. - # Reset telemetry toggle - # @return [True] - def reset! - @data_series = [] - @telemetry_toggled_off = false - end - - private - - # Minimize exposure of Inspec::Config interface - def config_telemetry_options - config.telemetry_options - end - end -end diff --git a/lib/inspec/utils/telemetry/data_series.rb b/lib/inspec/utils/telemetry/data_series.rb deleted file mode 100644 index 291e262a5..000000000 --- a/lib/inspec/utils/telemetry/data_series.rb +++ /dev/null @@ -1,44 +0,0 @@ -require "json" unless defined?(JSON) - -module Inspec; end - -# A minimal Dataseries Object -# Stores the name of the data series and an array of data. -# Stored data should be a object that supports #to_s -module Inspec::Telemetry - class DataSeries - def initialize(name) - @name = name - @enabled = true - @data ||= [] - end - - attr_reader :data, :name - - # This needs to also be set by configuration. - def enabled? - @enabled - end - - def disable - @enabled = false - end - - def <<(appending_data) - data << appending_data - end - - alias push << - - def to_h - { - name: @name, - data: @data, - } - end - - def to_json - to_h.to_json - end - end -end diff --git a/lib/inspec/utils/telemetry/global_methods.rb b/lib/inspec/utils/telemetry/global_methods.rb deleted file mode 100644 index 731279fd3..000000000 --- a/lib/inspec/utils/telemetry/global_methods.rb +++ /dev/null @@ -1,22 +0,0 @@ -require "inspec/utils/telemetry/collector" - -module Inspec - # A Global method to add a data series object to the Telemetry Collection. - # `data_series_name`s are unique, so `:dependency_group` will always return - # the same object. - # `data_point` is optional, you may also supply a block with several data points. - # All data points should allow #to_s - def self.record_telemetry_data(data_series_name, data_point = nil) - coll = Inspec::Telemetry::Collector.instance - return unless coll.telemetry_enabled? - - ds = coll.find_or_create_data_series(data_series_name) - return unless ds.enabled? - - if block_given? - ds << yield - else - ds << data_point - end - end -end diff --git a/lib/inspec/utils/telemetry/http.rb b/lib/inspec/utils/telemetry/http.rb new file mode 100644 index 000000000..403c46565 --- /dev/null +++ b/lib/inspec/utils/telemetry/http.rb @@ -0,0 +1,40 @@ +# frozen_string_literal: true +require_relative "base" +require "faraday" unless defined?(Faraday) +require "inspec/utils/licensing_config" +module Inspec + class Telemetry + class HTTP < Base + TELEMETRY_JOBS_PATH = "v1/job" + TELEMETRY_URL = if ChefLicensing::Config.license_server_url&.match?("acceptance") + ENV["CHEF_TELEMETRY_URL"] + else + "https://services.chef.io/telemetry/" + end + def run_ending(opts) + payload = super + response = connection.post(TELEMETRY_JOBS_PATH) do |req| + req.body = payload.to_json + end + if response.success? + Inspec::Log.debug "HTTP connection with Telemetry Client successful." + Inspec::Log.debug "HTTP response from Telemetry Client -> #{response.to_hash}" + true + else + Inspec::Log.debug "HTTP connection with Telemetry Client faced an error." + Inspec::Log.debug "HTTP error -> #{response.to_hash[:body]["error"]}" if response.to_hash[:body] && response.to_hash[:body]["error"] + false + end + rescue Faraday::ConnectionFailed + Inspec::Log.debug "HTTP connection failure with telemetry url -> #{TELEMETRY_URL}" + end + + def connection + Faraday.new(url: TELEMETRY_URL) do |config| + config.request :json + config.response :json + end + end + end + end +end diff --git a/lib/inspec/utils/telemetry/null.rb b/lib/inspec/utils/telemetry/null.rb new file mode 100644 index 000000000..6360ccd04 --- /dev/null +++ b/lib/inspec/utils/telemetry/null.rb @@ -0,0 +1,11 @@ +# frozen_string_literal: true +require_relative "base" +module Inspec + class Telemetry + class Null < Base + def run_starting(_opts); end + def run_ending(_opts); end + end + end +end + diff --git a/lib/inspec/utils/telemetry/run_context_probe.rb b/lib/inspec/utils/telemetry/run_context_probe.rb index 4a2eb886b..d3eb40e30 100644 --- a/lib/inspec/utils/telemetry/run_context_probe.rb +++ b/lib/inspec/utils/telemetry/run_context_probe.rb @@ -1,9 +1,21 @@ module Inspec - module Telemetry + class Telemetry # Guesses the run context of InSpec - how were we invoked? # All stack values here are determined experimentally class RunContextProbe + # Guess if we are running under Automate + def self.under_automate? + # Currently assume we are under automate if we have an automate-based reporter + Inspec::Config.cached[:reporter] + .keys + .map(&:to_s) + .any? { |n| n =~ /automate/ } + end + + # Guess, using stack introspection, if we were called under + # test-kitchen, cli, audit-cookbook, or otherwise. + # TODO add compliance-phase of chef-infra def self.guess_run_context(stack = nil) stack ||= caller_locations return "test-kitchen" if kitchen?(stack) diff --git a/test/fixtures/reporters/run_data_test_profile_a.json b/test/fixtures/reporters/run_data_test_profile_a.json new file mode 100644 index 000000000..c7fb330b1 --- /dev/null +++ b/test/fixtures/reporters/run_data_test_profile_a.json @@ -0,0 +1 @@ +{"controls":[{"status":"passed", "code_desc":"File / is expected to be directory", "run_time":0.009208, "start_time":"2024-04-18T15:54:25+05:30", "resource_title":"File /", "expectation_message":"is expected to be directory", "waiver_data":null, "resource_class":"file", "resource_params":["/"]}, {"status":"passed", "code_desc":"example_config version is expected to eq \"1.0\"", "run_time":0.001179, "start_time":"2024-04-18T15:54:25+05:30", "resource_title":"example_config", "expectation_message":"version is expected to eq \"1.0\"", "waiver_data":null, "resource_class":"example_config", "resource_params":[]}, {"status":"passed", "code_desc":"File / is expected to be directory", "run_time":0.004369, "start_time":"2024-04-18T15:54:25+05:30", "resource_title":"File /", "expectation_message":"is expected to be directory", "waiver_data":null, "resource_class":"file", "resource_params":["/"]}], "other_checks":[], "profiles":[{"name":"profile_a", "title":"InSpec Profile", "maintainer":"The Authors", "copyright":"The Authors", "copyright_email":"you@example.com", "license":"Apache-2.0", "summary":"An InSpec Compliance Profile", "version":"0.1.0", "depends":[{"name":"profile_c", "path":"../profile_c", "status":"loaded"}], "supports":[], "controls":[{"title":"Create /tmp directory", "desc":"An optional description...", "descriptions":{"default":"An optional description..."}, "impact":0.7, "refs":[], "tags":{"tag-profilec1":null}, "code":"control 'profilec-1' do # A unique ID for this control\n impact 0.7 # The criticality, if this control fails.\n title 'Create /tmp directory' # A human-readable title\n desc 'An optional description...'\n tag 'tag-profilec1'\n describe file('/') do # The actual test\n it { should be_directory }\n end\nend\n", "source_location":{"ref":"/Users/nmathur/chef/inspec/test/fixtures/profiles/dependencies/profile_c/controls/example.rb", "line":2}, "id":"profilec-1"}, {"title":"Create / directory", "desc":"An optional description...", "descriptions":{"default":"An optional description..."}, "impact":0.7, "refs":[], "tags":{"tag-profilea1":null}, "code":"control 'profilea-1' do # A unique ID for this control\n impact 0.7 # The criticality, if this control fails.\n title 'Create / directory' # A human-readable title\n desc 'An optional description...'\n tag \"tag-profilea1\"\n describe file('/') do # The actual test\n it { should be_directory }\n end\nend\n", "source_location":{"ref":"test/fixtures/profiles/dependencies/profile_a/controls/example.rb", "line":13}, "id":"profilea-1", "results":[{"status":"passed", "code_desc":"File / is expected to be directory", "run_time":0.009208, "start_time":"2024-04-18T15:54:25+05:30", "resource_title":"File /", "expectation_message":"is expected to be directory", "waiver_data":null, "resource_class":"file", "resource_params":["/"]}], "waiver_data":{}}, {"title":null, "desc":null, "descriptions":{}, "impact":0.5, "refs":[], "tags":{"tag-profilea2":null}, "code":"control 'profilea-2' do\n tag \"tag-profilea2\"\n describe example_config do\n its('version') { should eq('1.0') }\n end\nend\n", "source_location":{"ref":"test/fixtures/profiles/dependencies/profile_a/controls/example.rb", "line":23}, "id":"profilea-2", "results":[{"status":"passed", "code_desc":"example_config version is expected to eq \"1.0\"", "run_time":0.001179, "start_time":"2024-04-18T15:54:25+05:30", "resource_title":"example_config", "expectation_message":"version is expected to eq \"1.0\"", "waiver_data":null, "resource_class":"example_config", "resource_params":[]}], "waiver_data":{}}], "groups":[{"title":"sample section", "controls":["profilea-1", "profilea-2"], "id":"controls/example.rb"}, {"title":null, "controls":["profilec-1"], "id":"/Users/nmathur/chef/inspec/test/fixtures/profiles/dependencies/profile_c/controls/example.rb"}], "inputs":[], "sha256":"db35c749d86f9bee60fc0d2cb33584b144bb7422814fbc6373ee379239898e00", "status_message":"", "status":"loaded"}, {"name":"profile_c", "title":"InSpec Profile", "maintainer":"The Authors", "copyright":"The Authors", "copyright_email":"you@example.com", "license":"Apache-2.0", "summary":"An InSpec Compliance Profile", "version":"0.1.0", "supports":[], "controls":[{"title":"Create /tmp directory", "desc":"An optional description...", "descriptions":{"default":"An optional description..."}, "impact":0.7, "refs":[], "tags":{"tag-profilec1":null}, "code":"control 'profilec-1' do # A unique ID for this control\n impact 0.7 # The criticality, if this control fails.\n title 'Create /tmp directory' # A human-readable title\n desc 'An optional description...'\n tag 'tag-profilec1'\n describe file('/') do # The actual test\n it { should be_directory }\n end\nend\n", "source_location":{"ref":"/Users/nmathur/chef/inspec/test/fixtures/profiles/dependencies/profile_c/controls/example.rb", "line":2}, "id":"profilec-1", "results":[{"status":"passed", "code_desc":"File / is expected to be directory", "run_time":0.004369, "start_time":"2024-04-18T15:54:25+05:30", "resource_title":"File /", "expectation_message":"is expected to be directory", "waiver_data":null, "resource_class":"file", "resource_params":["/"]}], "waiver_data":{}}], "groups":[{"title":null, "controls":["profilec-1"], "id":"controls/example.rb"}], "inputs":[], "sha256":"fcfaa8fb85ad5ecebb504940ee2e2e57fcf7c84643b78568e4c7acc50c957e6e", "parent_profile":"profile_a", "status_message":"", "status":"loaded"}], "platform":{"name":"mac_os_x", "release":"23.3.0", "target":"local://", "target_id":"b2abc19a-f4b4-5d40-941c-c52a0240a076"}, "version":"6.6.15", "statistics":{"duration":0.017777, "controls":{"total":3, "passed":{"total":3}, "skipped":{"total":0}, "failed":{"total":0}}}} \ No newline at end of file diff --git a/test/fixtures/valid_client_api_data.json b/test/fixtures/valid_client_api_data.json new file mode 100644 index 000000000..2c027405a --- /dev/null +++ b/test/fixtures/valid_client_api_data.json @@ -0,0 +1,57 @@ +{ + "data":{ + "cache": { + "lastModified": "2023-01-16T12:05:40Z", + "evaluatedOn": "2023-01-16T12:07:20.114370692Z", + "expires": "2023-01-17T12:07:20.114370783Z", + "cacheControl": "private,max-age:42460" + }, + "client": { + "license": "Free", + "status": "Active", + "changesTo": "Grace", + "changesOn": "2024-11-01", + "changesIn": "2 days", + "usage": "Active", + "used": 2, + "limit": 2, + "measure": 2 + }, + "assets": [ + { + "id": "assetguid1", + "name": "Test Asset 1" + }, + { + "id": "assetguid2", + "name": "Test Asset 2" + } + ], + "features": [ + { + "id": "featureguid1", + "name": "Test Feature 1" + }, + { + "id": "featureguid2", + "name": "Test Feature 2" + } + ], + "entitlement": { + "id": "3ff52c37-e41f-4f6c-ad4d-365192205968", + "name": "Inspec", + "start": "2022-11-01", + "end": "2024-11-01", + "licenses": 2, + "limits": [ + { + "measure": "nodes", + "amount": 2 + } + ], + "entitled": false + } + }, + "message": "", + "status_code": 200 + } \ No newline at end of file diff --git a/test/unit/utils/telemetry/collector_test.rb b/test/unit/utils/telemetry/collector_test.rb deleted file mode 100644 index 031cf6f99..000000000 --- a/test/unit/utils/telemetry/collector_test.rb +++ /dev/null @@ -1,60 +0,0 @@ -require "inspec/utils/telemetry" -require "helper" - -class TestTelemetryCollector < Minitest::Test - def setup - @collector = Inspec::Telemetry::Collector.instance - @collector.reset! - end - - def test_collector_singleton - assert_equal Inspec::Telemetry::Collector.instance, @collector - end - - def test_add_data_series - assert_empty @collector.list_data_series - assert @collector.add_data_series(Inspec::Telemetry::DataSeries.new("/resource/File")) - refute_empty @collector.list_data_series - end - - def test_list_data_series - assert_empty @collector.list_data_series - @collector.add_data_series(Inspec::Telemetry::DataSeries.new("/resource/File")) - @collector.add_data_series(Inspec::Telemetry::DataSeries.new(:deprecation_group)) - assert_equal 2, @collector.list_data_series.count - assert_equal 1, @collector.list_data_series.select { |d| d.name.eql?(:deprecation_group) }.count - assert_kind_of Array, @collector.list_data_series - assert_kind_of Inspec::Telemetry::DataSeries, @collector.list_data_series.first - end - - def test_find_or_create_data_series - dg = @collector.find_or_create_data_series(:deprecation_group) - assert_kind_of Inspec::Telemetry::DataSeries, dg - assert_equal :deprecation_group, dg.name - assert_equal @collector.find_or_create_data_series(:deprecation_group), dg - end - - def test_reset_singleton - data_series = Inspec::Telemetry::DataSeries.new("/resource/File") - @collector.add_data_series(data_series) - @collector.reset! - assert_equal 0, @collector.list_data_series.count - end - - def test_telemetry_enabled - @collector.load_config(Inspec::Config.mock("enable_telemetry" => true)) - assert @collector.telemetry_enabled? - end - - def test_telemetry_disabled - @collector.load_config(Inspec::Config.mock("enable_telemetry" => false)) - refute @collector.telemetry_enabled? - end - - def test_disable_telemetry - @collector.load_config(Inspec::Config.mock("enable_telemetry" => true)) - assert @collector.telemetry_enabled? - @collector.disable_telemetry - refute @collector.telemetry_enabled? - end -end diff --git a/test/unit/utils/telemetry/data_series_test.rb b/test/unit/utils/telemetry/data_series_test.rb deleted file mode 100644 index 13476f455..000000000 --- a/test/unit/utils/telemetry/data_series_test.rb +++ /dev/null @@ -1,60 +0,0 @@ -require "inspec/utils/telemetry" -require "json" -require "helper" - -class TestTelemetryDataSeries < Minitest::Test - def test_name - ds = Inspec::Telemetry::DataSeries.new("fizz") - refute_nil ds - assert_equal "fizz", ds.name - end - - def test_data - ds = Inspec::Telemetry::DataSeries.new("fizz") - refute_nil ds.data - assert_kind_of Array, ds.data - assert_empty ds.data - end - - def test_data_append - ds = Inspec::Telemetry::DataSeries.new("fizz") - assert_empty ds.data - assert ds << "foo" - assert_equal ["foo"], ds.data - end - - def test_data_push_alias - ds = Inspec::Telemetry::DataSeries.new("fizz") - assert_empty ds.data - assert ds.push "bar" - assert_equal ["bar"], ds.data - end - - def test_to_h - ds = Inspec::Telemetry::DataSeries.new("fizz") - ds << "foo" - assert_kind_of Hash, ds.to_h - assert_equal "fizz", ds.to_h[:name] - assert_equal ["foo"], ds.to_h[:data] - end - - def test_to_json - ds = Inspec::Telemetry::DataSeries.new("fizz") - ds << "foo" - assert_kind_of String, ds.to_json - assert_equal '{"name":"fizz","data":["foo"]}', ds.to_json - assert JSON.parse(ds.to_json) - end - - def test_enabled - ds = Inspec::Telemetry::DataSeries.new("fizz") - assert ds.enabled? - end - - def test_disable - ds = Inspec::Telemetry::DataSeries.new("fizz") - assert ds.enabled? - ds.disable - refute ds.enabled? - end -end diff --git a/test/unit/utils/telemetry/global_methods_test.rb b/test/unit/utils/telemetry/global_methods_test.rb deleted file mode 100644 index a0655e4db..000000000 --- a/test/unit/utils/telemetry/global_methods_test.rb +++ /dev/null @@ -1,33 +0,0 @@ -require "inspec/utils/telemetry" -require "helper" - -class TestTelemetryGlobalMethods < Minitest::Test - def setup - @collector = Inspec::Telemetry::Collector.instance - @collector.load_config(Inspec::Config.mock("enable_telemetry" => true)) - @collector.reset! - end - - def test_record_telemetry_data - assert Inspec.record_telemetry_data(:deprecation_group, "serverspec_compat") - - depgrp = @collector.find_or_create_data_series(:deprecation_group) - assert_equal ["serverspec_compat"], depgrp.data - assert_equal :deprecation_group, depgrp.name - end - - def test_record_telemetry_data_with_block - Inspec.record_telemetry_data(:deprecation_group) do - "serverspec_compat" - end - - depgrp = @collector.find_or_create_data_series(:deprecation_group) - assert_equal ["serverspec_compat"], depgrp.data - assert_equal :deprecation_group, depgrp.name - end - - def test_telemetry_disabled - @collector.load_config(Inspec::Config.mock(telemetry: false)) - refute Inspec.record_telemetry_data(:deprecation_group, "serverspec_compat") - end -end diff --git a/test/unit/utils/telemetry_test.rb b/test/unit/utils/telemetry_test.rb new file mode 100644 index 000000000..2b315c7dd --- /dev/null +++ b/test/unit/utils/telemetry_test.rb @@ -0,0 +1,124 @@ +require_relative "../../helper" +require_relative "../../../lib/inspec/utils/telemetry" +require_relative "../../../lib/inspec/runner" + +module Inspec + class Telemetry::Mock < Telemetry::Base + attr_reader :run_ending_payload + def run_ending(opts) + @run_ending_payload = super(opts) + end + end +end + +REGEX = { + version: /^(\d+|\d+\.\d+|\d+\.\d+\.\d+)(?:-((?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*)(?:\.(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*))*))?(?:\+([0-9a-zA-Z-]+(?:\.[0-9a-zA-Z-]+)*))?$/, + datetime: /^(\d{4})-(\d{2})-(\d{2})T(\d{2}):(\d{2}):(\d{2}(?:\.\d*)?)((-(\d{2}):(\d{2})|Z)?)$/, + uuid: /^[0-9a-fA-F]{8}\b-[0-9a-fA-F]{4}\b-[0-9a-fA-F]{4}\b-[0-9a-fA-F]{4}\b-[0-9a-fA-F]{12}$/, + transport: /^[a-z0-9\-\_]+$/, + sha256: /^[0-9a-fA-F]{64}|unknown$/, +}.freeze + +describe "Telemetry" do + let(:conf) { Inspec::Config.new({ "enable_telemetry" => false }) } + let(:runner) { Inspec::Runner.new({ command_runner: :generic, reporter: [], conf: conf }) } + let(:run_data) { JSON.parse(File.read("test/fixtures/reporters/run_data_test_profile_a.json"), symbolize_names: true) } + let(:repo_path) { File.expand_path("../../..", __dir__) } + let(:mock_path) { File.join(repo_path, "test", "fixtures") } + let(:valid_client_api_data) { File.read("#{repo_path}/test/fixtures/valid_client_api_data.json") } + let(:profile_path) { File.join(mock_path, "profiles") } + let(:profile) { File.join(profile_path, "dependencies", "profile_a") } + let(:tm) { Inspec::Telemetry::Mock.new } + let(:chef_license_key) { "free-42727540-ddc8-4d4b-0000-80662e03cd73-0000" } + + before do + stub_request(:get, "#{ChefLicensing::Config.license_server_url}/v1/listLicenses") + .to_return( + body: { + "data": [chef_license_key], + "message": "", + "status_code": 200, + }.to_json, + headers: { content_type: "application/json" } + ) + + stub_request(:get, "#{ChefLicensing::Config.license_server_url}/v1/client") + .with(query: { licenseId: chef_license_key, entitlementId: ChefLicensing::Config.chef_entitlement_id }) + .to_return( + body: valid_client_api_data , + headers: { content_type: "application/json" } + ) + + stub_request(:get, "#{ChefLicensing::Config.license_server_url}/v1/client") + .with(query: { licenseId: [chef_license_key, ENV["CHEF_LICENSE_KEY"]].join(","), entitlementId: ChefLicensing::Config.chef_entitlement_id }) + .to_return( + body: valid_client_api_data , + headers: { content_type: "application/json" } + ) + + stub_request(:get, "#{ChefLicensing::Config.license_server_url}/v1/client") + .with(query: { licenseId: [ENV["CHEF_LICENSE_KEY"], chef_license_key].join(","), entitlementId: ChefLicensing::Config.chef_entitlement_id }) + .to_return( + body: valid_client_api_data , + headers: { content_type: "application/json" } + ) + end + + describe "when it runs with a nested profile" do + it "sets the wrapper fields" do + ChefLicensing::Context.license = ChefLicensing.client(license_keys: [chef_license_key]) + Inspec::Telemetry.expects(:instance).returns(tm).at_least_once + Inspec::Telemetry.run_ending(runner: runner, run_data: run_data, conf: conf) + runner.add_target(profile) + runner.run + _(tm.run_ending_payload).wont_be_empty + _(tm.run_ending_payload).must_be_kind_of Hash + _(tm.run_ending_payload[:source]).must_match(/^inspec:\d+\.\d+\.\d+$/) + _(tm.run_ending_payload[:licenseIds]).wont_be_empty + _(tm.run_ending_payload[:createdTimeUTC]).must_match(REGEX[:datetime]) + _(tm.run_ending_payload[:type]).must_match(/^job$/) + end + + it "sets the job fields" do + ChefLicensing::Context.license = ChefLicensing.client(license_keys: [chef_license_key]) + Inspec::Telemetry.expects(:instance).returns(tm).at_least_once + Inspec::Telemetry.run_ending(runner: runner, run_data: run_data, conf: conf) + runner.add_target(profile) + runner.run + j = tm.run_ending_payload[:jobs][0] + _(j).wont_be_empty + _(j).must_be_kind_of Hash + _(j[:type]).must_equal("InSpec") + + _(j[:environment][:host]).must_match(/^\S+$/) + _(j[:environment][:os]).must_match(/^\S+$/) + _(j[:environment][:version]).must_match(REGEX[:version]) # looser version matching + _(j[:environment][:architecture]).wont_be_empty + _(j[:environment][:id]).must_match(REGEX[:uuid]) + + _(j[:content]).must_be_kind_of Array + _(j[:content].count).must_equal 2 + j[:content].each do |c| + _(c[:name]).wont_be_empty + _(c[:version]).must_match(REGEX[:version]) + _(c[:sha256]).must_match(REGEX[:sha256]) + _(c[:maintainer]).wont_be_empty + end + + _(j[:steps]).must_be_kind_of Array + _(j[:steps].count).must_equal 4 + j[:steps].each do |s| + _(s[:name]).must_equal "inspec-control" + _(s[:id]).must_match(REGEX[:sha256]) + _(s[:resources]).must_be_kind_of Array + _(s[:features]).wont_be_empty + _(s[:tags]).wont_be_empty + s[:resources].each do |r| + _(r[:type]).must_equal "inspec-resource" + _(r[:name]).wont_be_empty + _(r[:id]).must_match(REGEX[:sha256]) + end + end + end + end +end