DevOps Zone is brought to you in partnership with:

Patrick Debois has been working on closing the gap between development and operations for many years. In 2009 he organized the first devopsdays.org conference and since then the world is stuck with the term 'devops'. Always seeking for opportunities to optimize the global IT instead of local optimizations. Patrick is a DZone MVB and is not an employee of DZone and has posted 39 posts at DZone. You can read more from them at their website. View Full User Profile

Puppet unit testing like a pro

12.19.2011
| 8510 views |
  • submit to reddit

A big thanks to Atlassian for allowing me to post this series!!

In our previous blogpost on Puppet Versioning, we described the most basic check to see if a puppet manifest was valid. We used the parseonly function to see if it would compile.

Until know this means we have only have if the compiler is happy, not that it performs the function it needs to do. In 2009 after the first devopsdays I wrote a collection of Test Driven Infrastructure Links . This was obviously inspired by Lindsay Holmwood's talk on cucumber-nagios.

On the Opscode chef front, Stephen Nelson-Smith wrote a great book Test-driven Infrastructure with Chef on how to do this. Also see the cuken project where re-usable cucumber steps are grouped.

Because we are using Puppet here at Atlassian, I was out to understand the current state of puppet testing. A lot can already be found at http://puppetlabs.com/blog/testing-modules-in-the-puppet-forge/

Note that I've purposely named this blog 'Puppet unit testing', as the tests I'm describing now, don't run against an actual system. Therefore it's hard to test the actual behavior.

Tip 1: cucumber-puppet

Inspired by Lindsay Holmwood's talk on cucumber-nagios and Ohad Levy's manitest Nikolay Sturm created cucumber-puppet

In his post on Thoughts on testing puppet manifests he explains that the idea of writing tests is NOT about duplicating the code, and he identified the most common problems he was facing are:

  • catalog does not compile: syntax errors, missing template files, ..
  • catalog does compile, but cannot be applied: unreachable or non-existent resources, missing file resources in repo
  • catalog does applies, but is faulty: faulty files, due to empty manifests variables or wrong values, missing dependencies (wrong order ...), files are installed without ensuring a directory ...

An important advice is:

Resource specifications can be useful for documentation purposes or refactorings. However, there is a risk of reimplementing your Puppet manifest, so be wary.


$ cd puppet-mymodule
$ gem install cucumber-puppet

Write features per module, this is the structure we are aiming at:

module
  +-- manifests
  +-- lib
  +-- features
       +-- support
       |     +-- hooks.rb
       |     +-- world.rb
       +-- catalog
       +-- feature..


Generate a cucumber-puppet world:

$ cucumber-puppet-gen world
Generating with world generator:
     [ADDED]  features/support/hooks.rb
     [ADDED]  features/support/world.rb
     [ADDED]  features/steps

# Adjust the paths to your modules and manifests
$ cat features/support/hooks.rb
Before do
  # adjust local configuration like this
  # @puppetcfg['confdir']  = File.join(File.dirname(__FILE__), '..', '..')
  # @puppetcfg['manifest'] = File.join(@puppetcfg['confdir'], 'manifests', 'site.pp')
  # @puppetcfg['modulepath']  = "/srv/puppet/modules:/srv/puppet/site-modules"

  # adjust facts like this
  @facts['architecture'] = "i386"
end

# Nothing exciting here
$ cat features/support/world.rb

require 'cucumber-puppet/puppet'
require 'cucumber-puppet/steps'

World do
  CucumberPuppet.new
end

Generating a policy feature:

$ cucumber-puppet-gen policy
Generating with policy generator:
     [ADDED]  features/catalog

# Notice the <hostname>.example.com.yaml
# These files contain the facts to test your catalog against
# 
$ cat features/catalog/policy.feature 
Feature: General policy for all catalogs
  In order to ensure applicability of a host's catalog
  As a manifest developer
  I want all catalogs to obey some general rules

  Scenario Outline: Compile and verify catalog
    Given a node specified by "features/yaml/<hostname>.example.com.yaml"
    When I compile its catalog
    Then compilation should succeed
    And all resource dependencies should resolve

    Examples:
      | hostname  |
      | localhost |

To do an actual run:

$ cucumber-puppet features/catalog/policy.feature 
Feature: General policy for all catalogs
  In order to ensure applicability of a host's catalog
  As a manifest developer
  I want all catalogs to obey some general rules

  Scenario Outline: Compile and verify catalog                            # features/catalog/policy.feature:6
    Given a node specified by "features/yaml/<hostname>.example.com.yaml" # cucumber-puppet-0.3.6/lib/cucumber-puppet/steps.rb:1
    When I compile its catalog                                            # cucumber-puppet-0.3.6/lib/cucumber-puppet/steps.rb:14
    Then compilation should succeed                                       # cucumber-puppet-0.3.6/lib/cucumber-puppet/steps.rb:48
    And all resource dependencies should resolve                          # cucumber-puppet-0.3.6/lib/cucumber-puppet/steps.rb:28

    Examples: 
      | hostname  |
      | localhost |
      Cannot find node facts features/yaml/localhost.example.com.yaml. (RuntimeError)
      features/catalog/policy.feature:7:in `Given a node specified by "features/yaml/<hostname>.example.com.yaml"'

Failing Scenarios:
cucumber features/catalog/policy.feature:6 # Scenario: Compile and verify catalog

1 scenario (1 failed)
4 steps (1 failed, 3 skipped)
0m0.006s

List of commands:

Generators for cucumber-puppet

Available generators
    feature                          Generate a cucumber feature
    policy                           Generate a catalog policy
    testcase                         Generate a test case for the test suite
    testsuite                        Generate a test suite for puppet features
    world                            Generate cucumber step and support files

General options:
    -p, --pretend                    Run, but do not make any changes.
    -f, --force                      Overwrite files that already exist.
    -s, --skip                       Skip files that already exist.
    -d, --delete                     Delete files that have previously been generated with this generator.
        --no-color                   Don't colorize the output
    -h, --help                       Show this message
        --debug                      Do not catch errors


He has also added support for testing exported-resources.

And for a more practical explanation, see how Oliver Hookins describes the way Nokia uses cucumber-puppet

Scenario: Proxy host and port have sensible defaults
  Given a node of class "mymodule::myapp"
  And we have loaded "test" settings
  And we have unset the fact "proxy_host"
  And we have unset the fact "proxy_port"
  When I compile the catalog
  Then there should be a file "/etc/myapp/config.properties"
  And the file should contain "proxy.port=-1"
  And the file should contain /proxy\.host=$/

----

Then /^the file should contain "(.*)"$/ do |text|
  fail "File parameter 'content' was not specified" if @resource["content"].nil?
  fail "Text content [#{text}] was not found" unless @resource["content"].include?(text)
end

Then /^the file should contain \/([^\"].*)\/$/ do |regex|
  fail "File parameter 'content' was not specified" if @resource["content"].nil?
  fail "Text regex [/#{regex}/] did not match" unless @resource["content"] =~ /#{regex}/
end


Tip 2: rspec-puppet

While the idea on using specs and puppet is not new (https://github.com/jes5199/puppet_spec), the new tool on the block is rspec-puppet brought to us by Tim Sharpe. The same person who gave us vim-puppet and puppet-lint

Like the cucumber-puppet structure, the idea is to have specs directory close to your module:

module
  +-- manifests
  +-- lib
  +-- spec
       +-- spec_helper.rb
       +-- classes
       |     +-- <class_name>_spec.rb
       +-- defines
       |     +-- <define_name>_spec.rb
       +-- functions
             +-- <function_name>_spec.rb


I found it useful to change the default spec_helper.rb as the default

require 'rspec-puppet'

RSpec.configure do |c|
   c.module_path = File.expand_path(File.join(File.dirname(__FILE__), '..', '..'))
   c.manifest_dir = File.expand_path(File.join(File.dirname(__FILE__), '..', '..','..','manifests'))
end


desc "Run specs check on puppet manifests"
RSpec::Core::RakeTask.new(:spec) do |t|
   t.pattern = './demo-puppet/modules/**/*_spec.rb' # don't need this, it's default
   t.verbose = true
   t.rspec_opts = "--format documentation --color"
    # Put spec opts in a file named .rspec in root
  end

Here is a quick example for checking if the class apache installs a package httpd when on a Debian system

require "#{File.join(File.dirname(__FILE__),'..','spec_helper')}"

describe 'apache', :type => :class do
  let (:title { 'basic' })
  let(:params) { { } }
  let(:facts) { {:operatingsystem => 'Debian', :kernel => 'Linux'} }

  it { should contain_package('httpd').with_ensure('installed') }
end

A more detailed description can be found at

For more generic information on rspec:

Conclusion cucumber-puppet vs rspec-puppet

I think you can write your tests in both to do the same. Currently they both support 2.6 and 2.7

I found the rspec-puppet a bit simpler to juggle with providing params like :name or :facts. The yaml file didn't feel to flexible to me. Also cucumber seems to install more dependent gems, that might inflict with other projects.

But as Nikolay already said:

 

"don't duplicate your manifests in your tests" Focus on the catalog problems he described earlier and test your logic. Don't test if puppet is doing it's job, test that your logic it's doing it's job.


This is why I called them unit-tests, they don't test the real functionality. (That's for the next blogpost)

Tip 3: puppet-lint

To check you files against programming style you can use https://github.com/rodjek/puppet-lint. It will check for Rules on Spacing, Identation & Whitespace , Quoting, Resources, Conditionals, Classes

An easy way to integrate it in your Rakefile is:

require 'puppet-lint'

desc "Run lint check on puppet manifests"
task :lint do
linter =  PuppetLint.new
  Dir.glob('./demo-puppet/modules//**/*.pp').each do |puppet_file|
    puts "Evaluating #{puppet_file}"
    linter.file = puppet_file
    linter.run
  end
  fail if linter.errors?
 end

Now you can simply run:

$ rake lint

Tip 4: go wild and build your own test/catalog logic

After having a look at the rspec-puppet logic, I looked deeper in the way to walk trough the catalog object. This is pretty much work in progress, but the idea is find a way to look at changes in the catalog.

The following is a list of useful examples on understanding on how to work with puppet in ruby code:

The first list of links are some fun tools written by Dean Wilson of www.puppetcookbook.com fame:

R.I. Pienaar of Mcollective Fame shows a way to create diff on a catalog. this can be useful to understand what tests to run in between changes:

This final gist shows how to walk through the catalog and check the classes and resources available:

https://gist.github.com/1430062#file_puppet_demo.rb


Source: http://www.jedi.be/blog/2011/12/05/puppet-unit-testing-like-a-pro/
Published at DZone with permission of Patrick Debois, author and DZone MVB.

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)

Comments

Robert Craft replied on Thu, 2012/01/26 - 6:12am

What about higher-level acceptance tests? For example:

Given a tomcat node
When I install the tomcat module
Then I should be able to access the default tomcat index page

Rather than telling you whether your puppet module executed the expected operations (install package, copy conf file, start service, etc.) it will tells you whether you achieved the desired result

Spring Security

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.