Standalone Puppet with Capistrano

Mike English and I frequently make use of Puppet to provision servers and manage server configurations. It is very convenient to be able to setup a server (while simultaneously documenting its configuration) by writing Puppet modules and manifests and then simply running the Puppet agent. For larger setups, we have used the Puppet agent/master model (client/server), but for smaller setups (such as for a single server), this doesn’t make much sense — especially if it is a standalone server for a client running privately in the cloud.

However, it feels rather “un-Puppet-like” to SSH into a server directly, make Puppet manifest changes, and then run the Puppet agent in order to make configuration changes to a server; it is not very efficient.

The Solution: Capistrano

We felt it would be more effective to have a single point from which we could modify manifests, and then run commands to make the necessary configuration changes on a server, while not actually needing to run a Puppet master.

To achieve this, we’ve started using Capistrano to bootstrap Puppet, upload manifests, and run the Puppet agent in standalone mode. For a while now, we’ve used Capistrano to deploy applications to servers. We’ve found it incredibly convenient to have both application deployment information and server configuration information (Puppet manifests) in the same place — allowing us to readily provision and configure a server, deploy an application, and document the whole setup in one source control repository.

Overview

At a high level, once we have an application that needs to be deployed, we:

• Add Capistrano to the source code repository, if not already in use.
• Add Puppet manifests and modules to the source code repository.
• Configure Capistrano to bootstrap Puppet, upload Puppet manifests and modules, and run Puppet.
• Run Puppet via Capistrano to provision and configure the servers.
• Configure Capistrano to deploy the application, making use of Capistrano multi-stage if we need to handle multiple environments or servers.
• Run Capistrano to deploy and manage the application.

Implementation

In practice, we write Puppet manifests and modules incrementally — writing some Puppet code, running the Puppet agent to verify it produces expected results, and repeating the process. After we have finished writing the Puppet manifests and modules, we do a full test run on a clean system to verify that our code behaves correctly from bootstrapping to final configuration implementation.

After we have the Puppet code finalized, we use a similar process on the Capistrano recipes to verify that deployment completes correctly. Often we will complete this process on our local machines, making use of virtual machines that we can snapshot and revert as needed to facilitate the code development process.

The remainder of the post gives some specific examples and code snippets which show how we configure Capistrano to bootstrap and run puppet, how we bootstrap RVM and Puppet, and how we tie it all together.

Repository Structure

For a generic Rails application, our source control repository may look something like this (emphasis on directories used by Capistrano and Puppet):

├── Capfile
├── Gemfile
├── Gemfile.lock
├── README
├── Rakefile
├── app
├── bin
├── config
│   ├── deploy
│   │   ├── demo.rb
│   │   ├── production.rb
│   │   └── sandbox.rb
│   ├── deploy.rb
|	└── ...
├── config.ru
├── db
├── doc
├── features
├── lib
├── log
├── public
├── puppet
│   ├── bootstrap.sh
│   ├── manifests
│   │   └── site.pp
│   └── modules
│   │   ├── module1
│   │   ├── module2
│   │   ├── module3
│   │   └── ...
├── script
├── spec
├── tasks
└── vendor

Capistrano Modifications for Puppet

Note: We presume that the given user has sudo privileges on the system already. The “bootstrap” namespace involves bootstrapping Puppet, which includes installing RVM. The “puppet” namespace involves running puppet after bootstrapping.

require "rvm/capistrano"
...
namespace :bootstrap do
  task :default do
    # Specific RVM string for managing Puppet; may or may not match the RVM string for the application
    set :user, "ubuntu"

    # Set the default_shell to "bash" so that we don't use the RVM shell which isn't installed yet...
    set :default_shell, "bash"

    # We tar up the puppet directory from the current directory -- the puppet directory within the source code repository
    system("tar czf 'puppet.tgz' puppet/")
    upload("puppet.tgz","/home/#{user}",:via => :scp)

    # Untar the puppet directory, and place at /etc/puppet -- the default location for manifests/modules
    run("tar xzf puppet.tgz")
    try_sudo("rm -rf /etc/puppet")
    try_sudo("mv /home/#{user}/puppet/ /etc/puppet")

    # Bootstrap RVM/Puppet!
    try_sudo("bash /etc/puppet/bootstrap.sh")
  end 
end
    
namespace :puppet do
  task :default do
    # Specific RVM string for managing Puppet; may or may not match the RVM string for the application
    set :rvm_ruby_string, '1.9.3-p125'
    set :rvm_type, :system
    set :user, "ubuntu"

    # We tar up the puppet directory from the current directory -- the puppet directory within the source code repository
    system("tar czf 'puppet.tgz' puppet/")
    upload("puppet.tgz","/home/#{user}",:via => :scp)

    # Untar the puppet directory, and place at /etc/puppet -- the default location for manifests/modules
    run("tar xzf puppet.tgz")
    try_sudo("rm -rf /etc/puppet")
    try_sudo("mv puppet/ /etc/puppet")

    # Run RVM/Puppet!
    run("rvmsudo -p '#{sudo_prompt}' puppet apply /etc/puppet/manifests/site.pp")
  end 
end

RVM/Puppet Bootstrap

Note: This particular shell script is written for Ubuntu-based systems. It may also work for other Debian-based systems. It shouldn’t be hard to tweak for Redhat/Fedora-based systems. We install Ruby 1.9.3-p125 to manage Puppet. This may or may not be the version of Ruby needed for the application that will be deployed. That can be specified elsewhere in Capistrano, in the individual stage files as :rvm_ruby_string.

#!/bin/bash

# Update our package manager...
sudo apt-get update
# Install dependencies for RVM and Ruby...
sudo apt-get install -y build-essential libxslt1-dev libxml2-dev libreadline-dev zlib1g-dev libssl-dev curl git-core

# Get and install RVM
curl -L https://get.rvm.io | sudo bash -s stable

# Source rvm.sh so we have access to RVM in this shell
source /etc/profile.d/rvm.sh

# Install Ruby 1.9.3-125
rvmsudo rvm install 1.9.3-p125
rvmsudo rvm alias create default 1.9.3-p125

source /etc/profile.d/rvm.sh

# Update rubygems, and pull down facter and then puppet...
rvmsudo rvm 1.9.3-p125 do gem update --system
rvmsudo rvm 1.9.3-p125 do gem install facter --no-ri --no-rdoc
rvmsudo rvm 1.9.3-p125 do gem install puppet --no-ri --no-rdoc
rvmsudo rvm 1.9.3-p125 do gem install libshadow --no-ri --no-rdoc

# Create necessary Puppet directories...
sudo mkdir -p /etc/puppet /var/lib /var/log /var/run

Making it All Work

Now that we have all of our Puppet manifests and modules in one repository with our application code and Capistrano tying everything together, we can setup our server and get our application running:

• cap bootstrap
• cap puppet
• cap deploy:setup
• cap deploy

And if we had stages setup for separate servers with Capistrano multi-stage, we could easily setup our server and application on each stage:

• cap [stage] bootstrap
• cap [stage] puppet
• cap [stage] deploy:setup
• cap [stage] deploy