Speeding up puppet runs by using checksums when running execs

During my integration work for a client, I was running third party puppet code to  integrate automatically deployed application containers with the third party deployment tool (the deployit / XL Deploy module for puppet)

This module was written to run an exec for three different actions per container, all resulting in running Jython code in a local Java JVM. It would result in a couple of API / REST calls on the central deployment server to register this particular container.

Running this code every 30 minutes on 40+ hosts was dead slow and it was pounding the central server receiving the calls. Al while the properties being transferred had not changed and did not need any resending. Ugh 🙁

Kinda hoping that the vendor would come up with something smart, this situation was there for some time. But about a week ago I found the time to create an alternative. Instead of using the third party module, I wrote a shell scripts that parses a puppet managed properties file.

The script starts by reading all the properties and checksumming the concatenated result. If the sha256 sum has not changed since the last run, the script ends there. If there is no old checksum or the checksum does not match, the original Jython running in a JVM doing a rest call thing is run.

Before I started writing this alternative, puppet runs would easily run for minutes and minutes, waiting for the execs to finish. Now, with the central server offloaded, the typical puppet run is reduced to 80 seconds when running the execs and 29 seconds when the checksums matches.

[root@example ~]# puppet agent -t
Info: Retrieving plugin
Info: Loading facts in /var/opt/lib/pe-puppet/lib/facter/pe_build.rb
Info: Loading facts in /var/opt/lib/pe-puppet/lib/facter/pe_version.rb
Info: Loading facts in /var/opt/lib/pe-puppet/lib/facter/xldeploy_facts.rb
[snip]
Info: Caching catalog for example.local
Info: Applying configuration version '1426664207'
Notice: /Stage[main]/app1::Tomcat/my_tomcat::Instance[app1-service]/Exec[xldeploy-register-tomcat-app1-service]/returns: executed successfully
Notice: /Stage[main]/app2::Tomcat/my_tomcat::Instance[app2]/Exec[xldeploy-register-tomcat-app2]/returns: executed successfully
Notice: /Stage[main]/app3::Tomcat/my_tomcat::Instance[app3-service]/Exec[xldeploy-register-tomcat-app3-service]/returns: executed successfully
Notice: /Stage[main]/app4::Tomcat/my_tomcat::Instance[app4]/Exec[xldeploy-register-tomcat-app4]/returns: executed successfully
Notice: Finished catalog run in 28.85 seconds
[root@example ~]#

Xldeploy register script

#!/bin/bash
#
#
# This script can be used as an alternative to the slow puppet resources for
# XL Deploy. Use puppet to fill a property file for your XL Deploy container
# instance and call this script with the property file as argument.
#
# When it needs to register stuff in XL Deploy it is still slow as hell
# (the XL Deploy cli is just slow), but what this script does is test if
# the properties have changed since the last run and skip everything when
# nothing seems to need any updating. Should save as tons of time during
# puppet runs AND should shave us countless REST calls on the XL Deploy
# instance.
#
# 12-mar-2015
#

BASEDIR=/opt/dummy
VARDIR=/tmp

XLD_HOST=xldeploy.example.com
XLD_USER=admin
XLD_PASS=<%= @xldeploy_admin_password %>

export DEPLOYIT_CLI_HOME=/opt/deployit-cli
export DEPLOYIT_CLI_OPTS="-Xmx512m"

if [[ "X$1" == "X" ]];
then
  echo "Usage $0 [propertie file]"
  exit 1;
fi

if [[ ! -r $1 ]];
then
  echo "Unable to read propertie file $1"
  exit 2;
else
  EXT_PROPERTIES=$1
fi

function report()
{
  logger "$1"
  echo "$1"
}

function get_property()
{
  KEYWORD=$1
  VALUE=`/bin/grep $KEYWORD $EXT_PROPERTIES | /bin/cut -f1 -d '=' --complement`
  echo $VALUE
}

## run a command using the CLI. If it fails, remove any existing checksum to force
## a retry during the next run of this script.
function run_xld_command()
{
  COMMAND=$1
  /opt/deployit-cli/bin/cli.sh -q -host $XLD_HOST -port 443 -secure -context /deployit/deployit -username $XLD_USER -password $XLD_PASS $COMMAND
  if [[ $? > 0  && -f $CHECKSUM ]]; then rm $CHECKSUM; fi
}

function create_ci()
{
  CI_ID=$1
  CI_TYPE=$2
  CI_PARAMS=$3
  run_xld_command "-f /opt/deployit-puppet-module/create-ci.py -- $CI_ID $CI_TYPE $CI_PARAMS"
}

function set_tags()
{
  CI_ID=$1
  CI_TAG=$2
  run_xld_command "-f /opt/deployit-puppet-module/set-tags.py -- $CI_ID $CI_TAG"
}

function set_environment()
{
  CI_ID=$1
  CI_ENVIRONMENT=$2
  run_xld_command "-f /opt/deployit-puppet-module/set-envs.py -- $CI_ID $CI_ENVIRONMENT"
}

# The function below checks if anything in the XL Deploy settings for this
# CI has changed since the last run. If anything has changed the checksum on
# disk is updated (and the caller should rerun the command on the XL Deploy
# server to update this CI.
function xlconfig_changed()
{
  CHECK_STRING=$1
  CURRENT_SUM=`echo $CHECK_STRING | /usr/bin/sha256sum | /bin/awk '{ print $1; }'`
  if [[ -r $CHECKSUM ]];
  then
    PREVIOUS_SUM=`cat $CHECKSUM`
  else
    PREVIOUS_SUM="none"
  fi

  if [[ $PREVIOUS_SUM != $CURRENT_SUM ]];
  then
    CHANGED=true
    logger "Checksum $CURRENT_SUM does not match $PREVIOUS_SUM"
    echo $CURRENT_SUM > $CHECKSUM
  else
    CHANGED=false
    logger "Checksum $CURRENT_SUM same as $PREVIOUS_SUM"
  fi

  echo $CHANGED
}


HOST_CI_ID=$(get_property "host.ci.id")
HOST_CI_TYPE=$(get_property "host.ci.type")
HOST_CI_TAG=$(get_property "host.ci.tag")
HOST_CI_PARAMS=$(get_property "host.ci.params")

CONTAINER_CI_ID=$(get_property "container.ci.id")
CONTAINER_CI_TYPE=$(get_property "container.ci.type")
CONTAINER_CI_TAG=$(get_property "container.ci.tag")
CONTAINER_CI_PARAMS=$(get_property "container.ci.params")

CI_ENVIRONMENT=$(get_property "ci.environment")

CHECKSUM=$(get_property "checksum.path")

UPDATE_NEEDED=$(xlconfig_changed "$HOST_CI_ID $HOST_CI_TYPE $HOST_CI_TAG $HOST_CI_PARAMS $CONTAINER_CI_ID $CONTAINER_CI_TYPE $CONTAINER_CI_TAG $CONTAINER_CI_PARAMS $CI_ENVIRONMENT")

if [[ $UPDATE_NEEDED == "true" ]];
then
  report "Running the java/python code to update this CI in the XL Deploy server"
  create_ci "$HOST_CI_ID" "$HOST_CI_TYPE" "$HOST_CI_PARAMS"
  create_ci "$CONTAINER_CI_ID" "$CONTAINER_CI_TYPE" "$CONTAINER_CI_PARAMS"
  set_tags "$HOST_CI_ID" "$HOST_CI_TAG"
  set_tags "$CONTAINER_CI_ID" "$CONTAINER_CI_TAG"
  set_environment "$HOST_CI_ID" "$CI_ENVIRONMENT"
  set_environment "$CONTAINER_CI_ID" "$CI_ENVIRONMENT"
else
  report "Nothing to do here"
fi

exit 0

Example properties file

# managed by puppet
#
host.ci.id=<%= @host_ci_id %>
host.ci.type=<%= @host_ci_type %>
host.ci.tag=<%= @host_ci_tag %>
host.ci.params=<%= @host_ci_params %>

container.ci.id=<%= @container_ci_id %>
container.ci.type=<%= @container_ci_type %>
container.ci.tag=<%= @container_ci_tag %>
container.ci.params=<%= @container_ci_params %>

ci.environment=<%= @ci_environment %>
checksum.path=<%= @checksum_path %>

This code is propably way to specific to be of any use as is, but maybe the checksumming solution for ‘expensive’ exec calls during puppet runs is applicable to other scenarios as well.

It’s 2015 and I’m installing Snow Leopard

The old trusty MacBook white was getting slower and slower. I upgraded it to new OS X releases when they came available over the years. As my kids became the primary user, the complaints started to pile up. I tried moving to Windows 7 and to Ubuntu. But the machine just stayed slow and sluggish.

As a final attempt to fix the issues I wanted to downgrade the machine to Snow Leopard. But I got rid of alle DVD software media years ago.

When searching the Interwebs for a second hand OS X DVD, I actually found a link to the apple store offering new Snow Leopard DVD kits! So it was 2015 and I ordered a fresh DVD of Snow Leopard in the Apple Store for only 20 euro.

IMG_0463IMG_0462A couple of days later I did a fresh install of the old Macbook white and it has been running so smooth ever since. Downgrading  the machine for 20 euros was a pretty good choice it seems. Kids are pretty happy with it!