windows git pre-commit hook for puppet development

For a recent client I was asked to provide a puppet development setup on the windows platform. The toolchain included Git for windows / Git bash / Turtoise GIT / Atom and some others. It also includes the windows adaption of my git pre-commit hook.

To use this hook, you need to install the puppet v4 agent and install a couple of ruby gems within the provided ruby environment. Something like :

c:\program files\puppetlabs\puppet\sys\ruby\bin\gem.bat install r10k
c:\program files\puppetlabs\puppet\sys\ruby\bin\gem.bat install puppet-lint

(from an elevated command prompt)

The hook needs to be saved in every git repository that contains puppet code in het directory [repo]/.git/hooks/

Warning : The windows linefeed bit is really recent and might need some additional testing.

 

#!/bin/bash
# Requires bash, as it uses the [[ ]] syntax.
#
# https://puppetlabs.com/blog/using-puppet-lint-to-save-yourself-from-style-faux-pas
# https://docs.puppetlabs.com/guides/templating.html#syntax-checking
#
# If it's puppet code, lint it up.
# 20150915 syncaddict
# - Added support for erb syntax checking
#
# 20151020 syncaddict
# - Added support for YAML syntax checking
# - more verbose operation
#
# 20170615 syncaddict 
# - version that works on windows
#
# 20170809 syncaddict
# - detect / convert windows linefeeds
#
# Variables goes hither

 

PUPPETLINT="/c/Progra~1/Puppet~1/Puppet/sys/ruby/bin/puppet-lint.bat"
PUPPETAGENT="/c/Progra~1/Puppet~1/Puppet/bin/puppet.bat"
ERB="/c/Progra~1/Puppet~1/Puppet/sys/ruby/bin/erb.bat"
RUBY="/c/Progra~1/Puppet~1/Puppet/sys/ruby/bin/ruby.exe"

DOS2UNIX=/usr/bin/dos2unix
GREP=/usr/bin/grep
WC=/usr/bin/wc

declare -a FILES

IFS="
"
FILES=$(git diff --cached --name-only --diff-filter=ACM )
 

for file in ${FILES[@]}
do

   ## replace windows linefeeds on all changed files - WIP
   LFCHECK=`$GREP "\r\n$" $file | $WC -l`
   if [[ $? > 0 ]]; then echo "LF check failed"; fi

   if [[ $LFCHECK -gt 0 ]];
   then
     $DOS2UNIX $file
     echo "Converted linefeeds on $file, please re-add and retry your commit"
     exit 666
   fi

 

 

  case $file in
    *\.pp*)
      echo "Checking puppet file $file"
      $PUPPETLINT --no-puppet_url_without_modules-check --no-arrow_on_right_operand_line-check --no-140chars-check --fail-on-warnings --fix --with-filename "$file"
      RC=$?
      if [ $RC -ne 0 ]; then exit $RC;fi

      $PUPPETAGENT parser validate "$file"
      RC=$?
      if [ $RC -ne 0 ]; then exit $RC;fi
    ;;
    *\.erb*)
      echo "Checking erb template $file"
      $ERB -P -x -T '-' $file | $RUBY -c
      RC=$?
      if [ $RC -ne 0 ]; then exit $RC;fi
    ;;
    *\.yaml*)
      echo "Checking yaml file $file"
      $RUBY -e "require 'yaml'; YAML.load_file('$file')"
      RC=$?
      if [ $RC -ne 0 ]; then exit $RC;fi
    ;;
    *)
      echo "Not checking file $file"
    ;;
  esac
done

exit 0

Validating puppet code changes using octocatalog-diff

During Puppetconf 2016 Github announces it was releasing one of its internal test tools called octocatalog-diff as open source. What the tools basicly does is compile the catalog for a certain machine using your old and new puppetcode to show you the diff output. This allows you to see the impact of your code without actually deploying and running it on puppet clients.

As this could potentially save me a lot of time, I decided to delve into the tool to see what it could bring me. This post describes to the proces of setting the tool up and the surprises I came across.

Environment

For my initial testing I used a vagrant provisioned box running Ubuntu 16.04. Next to that I installed the puppet agent 3.8.7 that is in the standard Ubuntu repositories.

Preparation

The following gems were needed during my trails. Just run ‘sudo gem install [GEMNAME]’ to get them on your system. When using Puppet v4 take note to use the gem binary in /opt/puppetlabs/puppet/bin/gem

  • hiera-eyaml
  • r10k
  • bundler
  • rspec

The tool can query the puppetdb instance of your setup, or you can use the facts in yaml format for the node(s) you are testing. As the puppetdb did not work for my at this point, I copied the contents of /var/lib/puppet/yaml/facts from my puppetmaster to my local test environment.

The tool also needs some develop utilities upon install. Get them using your systems package manager :

  • cmake
  • pkg-config
  • ruby-dev

Installing the tool

After trying the gem, I’ve installed the source version following the instructions on https://github.com/github/octocatalog-diff/blob/master/doc/installation.md . The rake test gave me a a couple of errors that I reported.

Setting up

There were three files I added to my puppet repo :

  • A hiera.yaml configuration valid for my setup
  • A bootstrap script to run r10k on my Puppetfile and to symlink local modules to the common modules dir (see below)
  • A .octocatalog-diff.cfg.rb configured as per the instructions on github Please note the the current example file has a key settings[:hiera_yaml_file]  that should read settings[:hiera_config] . (Github issue)

Running the tool

/vagrant/octocatalog-diff/bin/octocatalog-diff -f production -t [your current branch] -n [NODENAME] --fact-file /vagrant/puppetdb_facts/[NODENAME].yaml --to-fact-override vagrant_puppetrole=nagiosserver --from-fact-override vagrant_puppetrole=nagiosserver --bootstrap-script repo-bootstrap.sh

Installation paths may vary. Note: the fact override thing is my way to assign a role to host without a real ENC / foreman. This value is used by a small piece of code in my site.pp :

node default {
  ## This is a small hook to support local vagrant
  ## development. This special var get set as part
  ## of the vagrant provisioning process.
  if $vagrant_puppetrole != undef {
    class { "roles::${vagrant_puppetrole}": }
  }
}

When removing a single package from my ‘baseline’ it resulted in the expected output:

screen-shot-2016-11-03-at-14-26-40

 

 

 

diff production/NODENAME fea-puppetv4/NODENAME
*******************************************
  Package[tcpdump] =>
   parameters =>
     ensure =>
      - present
      + absent
*******************************************

Contents of  repo-bootstrap.sh

r10k puppetfile install Puppetfile
for i in site/*; do BLA=`echo $i |sed -e 's#site/##'`; ln -s ../site/$BLA modules/$BLA; done

Final notes

When trying to use the tool, I started out using Puppet 4.8. I run into some trouble with puppetlabs firewall module 1.8.1 (( [Puppet Error] Could not autoload puppet/provider/firewall/iptables: undefined method `value’ for nil:NilClass) . As soon as I downgraded to puppet 3.8.7 the firewall module stopped producing errors. Not sure if this is related to the puppet version or the combination with octocatalog-diff.

I will use the tool the coming weeks. The next step could be integrating it in the build pipeline(s).

 

git pre-commit hook for puppet, erb and yaml files

A pre-commit hook is a great way to run custom actions before handing over your work to GIT (and thus your CI tool chain). For puppet related GIT repositories I’ve assembled a pre-commit hook that checks the puppet code, changed ERB templates and any changed YAML files  for formatting errors.

All not rocket science when put together, but a great time saver none the less. I never fly without it these days. Just paste the code block below in your [GITREPO]/.git/hooks/pre-commit and make sure the result is executable for your own account. The code is just bits that I pieced together.

There are a few dependencies that you might have to install depending on your platform.

#!/bin/bash
# Requires bash, as it uses the [[ ]] syntax.
#
# https://puppetlabs.com/blog/using-puppet-lint-to-save-yourself-from-style-faux-pas
# https://docs.puppetlabs.com/guides/templating.html#syntax-checking
#
# If it's puppet code, lint it up.

# If we don't have puppet-lint, so just exit and leave them be.
which puppet-lint >/dev/null 2>&1 || exit

# 20150915 syncaddict
# - Added support for erb syntax checking
#
# 20151020 syncaddict
# - Added support for YAML syntax checking
# - more verbose operation
#
# Variables goes hither

declare -a FILES
IFS="
"
FILES=$(git diff --cached --name-only --diff-filter=ACM )


for file in ${FILES[@]}
do
  case $file in
    *\.pp*)
      echo "Checking puppet file $file"
      puppet-lint --no-puppet_url_without_modules-check --no-80chars-check --fix --with-filename "$file"
      RC=$?
      if [ $RC -ne 0 ]; then exit $RC;fi

      puppet parser validate "$file"
      RC=$?
      if [ $RC -ne 0 ]; then exit $RC;fi
    ;;
    *\.erb*)
      echo "Checking erb template $file"
      erb -P -x -T '-' $file | ruby -c
      RC=$?
      if [ $RC -ne 0 ]; then exit $RC;fi
    ;;
    *\.yaml*)
      echo "Checking yaml file $file"
      ruby -e "require 'yaml'; YAML.load_file('$file')"
      RC=$?
      if [ $RC -ne 0 ]; then exit $RC;fi
    ;;
    *)
      echo "Not checking file $file"
    ;;
  esac
done

exit 0