Extending Puppet

Table of Contents

Extending Puppet

Credits

Foreword

About the Author

Acknowledgments

About the Reviewers

www.PacktPub.com

Support files, eBooks, discount offers, and more

Why subscribe?

Free access for Packt account holders

Preface

What this book covers

What you need for this book

Who this book is for

Conventions

Reader feedback

Customer support

Downloading the example code

Errata

Piracy

Questions

1. Puppet Essentials

The Puppet ecosystem

Why configuration management matters

Puppet components

Installation and configuration

Puppet in action

Resources

The resource abstraction layer

Nodes

Classes and defines

Class inheritance

Resource defaults

Resource references

Variables, facts, and scopes

System's facts

User variables in Puppet DSL

User variables in an ENC

User variables in Hiera

Puppet's built-in variables

A variable's scope

Meta parameters

Managing order and dependencies

Run stages

Reserved names and allowed characters

Conditionals

Comparison operators

The In operator

Expressions combinations

Exported resources

Virtual resources

Modules

The paths of a module and autoloading

ERB templates

Restoring files from a filebucket

Summary

2. Hiera

Installing and configuring Hiera

Global settings

Backend-specific settings

The hiera.yaml examples

Working with the command line on a YAML backend

Using Hiera in Puppet

Dealing with hashes in the Puppet code

Puppet 3 automatic parameter lookup

Evolving usage patterns for class parameters

Additional Hiera backends

The hiera-file backend

The hiera-gpg backend

The hiera-eyaml backend

The hiera-http and hiera-mysql backends

Using Hiera as an ENC

Summary

3. PuppetDB

Installation and configuration

PuppetDB configurations

The init script configuration

Configuration settings

Logging configuration

Configurations on the Puppet Master

Dashboards

PuppetDB performance dashboard

Puppetboard – query PuppetDB from the Web

The PuppetDB API

Querying PuppetDB (Read)

The PuppetDB commands (Write)

Querying PuppetDB for fun and profit

The /facts endpoint

The /resources endpoint

The /nodes endpoint

The /catalogs endpoint

The /facts-names endpoint

The /metrics endpoint

The /reports endpoint

The /events endpoint

The /event-counts endpoint

The /aggregated-event-counts endpoint

The /server-time endpoint

The /version endpoint

The puppetdbquery module

Query format

Query from the command line

Query from Puppet manifests

The PuppetDB Hiera backend

How Puppet code may change in the future

Summary

4. Designing Puppet Architectures

The components of Puppet architecture

Defining the classes to include in each node

Defining the parameters to use for each node

Defining the configuration files provided to the nodes

Defining custom resources and classes

The Foreman

Roles and profiles

The data and the code

Sample architectures

The default approach

Basic ENC, logic in the site module, and Hiera backend

The Foreman and Hiera

The Hiera-based setup

The Hiera-only setup

Foreman smart variables

Fact-driven truths

Nodeless site.pp

Node inheritance done right

Summary

5. Using and Writing Reusable Modules

The evolution of modules' layouts

Class parameters – from zero to data bindings

The params pattern

Data in modules

Files and class names

The anchor pattern

The parameters dilemma

Naming standards

Reusability patterns

Managing files

Managing configuration hash patterns

Managing multiple configuration files

Managing users and dependencies

Managing the installation options

Managing extra resources

Summary

6. Higher Abstraction Modules

Understanding the need for higher abstractions

The OpenStack example

Component (application) modules

Raising abstraction – the official openstack module

Raising abstraction – the Puppet Labs OpenStack module

Raising abstraction – the scenario-based approach

Taking an alternate approach

An approach to reusable stack modules

Summary

7. Deploying and Migrating Puppet

Examining the potential scenarios and approaches

New infrastructures

Existing manually managed infrastructures

Node migration

Node update

Existing automated infrastructures

Upgrading the Puppet Master

Patterns for extending Puppet coverage

Raising the bar, step by step

Knowing the territory

Defining priorities

Automate servers deployment

Automate common configurations

Automate the most important roles

Refine and automate application deployments

Integrate what already works well

Automate monitoring

Evaluating solutions

Coding

Applying changes

Things change

Infrastructure as code

Versioning

Reproducibility and predictability

Testing

Maintenance

Evolution of the system administrator

Designing a Puppet-friendly infrastructure

Summary

8. Code Workflow Management

Writing the Puppet code

Geppetto

Vim

Git workflows

Git's basic principles and commands

Git hooks

Environments and branches

Branch-based automatic environments

Simplified developer workdir environments

Code review

Gerrit

Online resources for peer review

Testing the Puppet code

Using rspec-puppet

rspec-system-puppet and Beaker

Vagrant

Deploying the Puppet code

Using librarian-puppet for deployments

Deploying code with r10k

Propagating Puppet changes

Puppet Continuous Integration tools

Travis

Jenkins

Summary

9. Scaling Puppet Infrastructures

Scaling Puppet

As simple as puppet apply

Default Puppet Master

Puppet Master with Passenger

Installing and configuring Passenger

Multi-Master scaling

Managing certificates

Managing SSL termination

Managing code and data

Load balancing alternatives

Masterless Puppet

Configurations and infrastructure optimizations

Traffic compression

Caching

Distributing Puppet execution times

Checking the interval for changes in files

Scaling stored configs

Measuring performance

Puppet metrics

Optimizing code

Testing different Puppet versions

Summary

10. Writing Puppet Plugins

Anatomy of a Puppet run, under the hood

ENC and Hiera extendibility

Modules' pluginsync

Puppet indirector and its termini

Custom functions

Custom facts

Ruby facts distributed via pluginsync

External facts in the facts.d directory

Custom types and providers

Custom report handlers

Custom faces

Summary

11. Beyond the System

Puppet on a network equipment

A Proxy mode with the puppet device application

Native Puppet on the network equipment

Cisco onePK

Juniper and netdev_stdlib

Puppet for cloud and virtualization

VMware

VM provisioning on vCenter and vSphere

The integration of vCloud Automation Center

The configuration of vCenter

Amazon Web Services

Cloud provisioning on AWS

AWS provisioning and configuration with resource types

Managing CloudFormation

Cloud provisioning on Google Compute Engine

Puppet on storage devices

Summary

12. Future Puppet

Introducing the future parser

Lambdas and Iterations

Manipulating and validating hashes and arrays

Other features

Restrictions and backward incompatibilities

Directory environments

Heredoc support

The new type system

Other new features

EPP Templates

The contain function

Serialization with MessagePack

Improved error system

Data in modules

Facter 2

Trapperkeeper

Summary

Index

Extending Puppet

Extending Puppet

Copyright © 2014 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

First published: June 2014

Production reference: 1170614

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham B3 2PB, UK.

ISBN 978-1-78398-144-1

www.packtpub.com

Cover image by Alessandro Franceschi (<[email protected]>)

Credits

Author

Alessandro Franceschi

Reviewers

Dhruv Ahuja

C. N. A. Corrêa

Brice Figureau

Commissioning Editor

Edward Gordon

Acquisition Editor

Llewellyn Rozario

Content Development Editor

Azharuddin Sheikh

Technical Editors

Rohit Kumar Singh

Pratish Soman

Copy Editors

Sayanee Mukherjee

Karuna Narayanan

Alfida Paiva

Adithi Shetty

Laxmi Subramanian

Project Coordinator

Kartik Vedam

Proofreaders

Simran Bhogal

Maria Gould

Ameesha Green

Paul Hindle

Indexer

Hemangini Bari

Production Coordinator

Adonia Jones

Cover Work

Adonia Jones

Foreword

I first met Alessandro in person at the inaugural Puppet Camp in San Francisco, 2009, but by this time, we'd already chatted on IRC and the Puppet Users mailing list. This was a small event by the standards of Puppet community events today, with about 60 people in attendance, and it's been great to see how many of that original crowd have continued to be active participants in the community, especially Alessandro.

While I was running Puppet at Google, I kept getting a lot of questions from attendees about how we were managing to scale our Puppet infrastructure technically. Alessandro, however, was already prodding me about how I was managing workflow and code layout for reusability and shareability, a topic that he's been very much focused on over the last five years.

When I initially left Google and moved to Puppet Labs in late 2010 to handle products, it became even more apparent how much Alessandro cared about guiding the community towards standards for Puppet content that allowed for reusability and shareability, yet allowed sysadmins to work quickly. We saw this with his promotion of the params pattern to consolidate input variables in a single location, and to allow for a first step towards separating data from code, well before the existence of Hiera as a formal project.

Alessandro saw this need well before most of us, and regularly raised it with the community as well as just about every time we ran into each other at conferences and events. As new projects appeared that added to the capabilities of the Puppet ecosystem, he modified his thinking and raised new proposals.

I'm thrilled to see this new book by Alessandro on Puppet architectures and design patterns, and I can't think of a better person to write it. He's cared about these principles for a long time, and he's promoted them as a responsible community member.

Nigel Kersten

CIO, Puppet Labs

About the Author

Alessandro Franceschi is a freelance Puppet and DevOps consultant. Surviving IT battlegrounds since 1995, he has worked as an entrepreneur, web developer, trainer, and system and network administrator.

He has been using Puppet since 2007, automating a remarkable amount of customers' infrastructures of different sizes, natures, and complexities.

He has attended several PuppetConf and Puppet Camps as a speaker and participant, always enjoying the vibrant and friendly community, learning something new each time.

During the following years, he started to publish his Puppet code, trying to make it reusable in different scenarios.

The result of this work is the Example42 Puppet modules set, a widely used collection of modules based on reusability principles and with some optional, opinionated twists to automate firewalling, monitoring, systems' inventory, and application deployments.

For more information on Example42 modules, visit www.example42.com. His Twitter handle is @alvagante.

Acknowledgments

This is my first book. When Packt Publishing asked me to write a book about Puppet, I was surprised, flattered, and intrigued. The idea of sharing my experience about this wonderful tool was really attractive.

I have been using Puppet for seven years. I have loved it since the beginning.

I have seen a great community of people doing wonderful things with it, and I tried to credit as many people as possible, knowing that many worthy contributors have been forgotten.

I have assisted in its evolution and the tumultuous growth of the company behind it, Puppet Labs.

I have definitely not seen any attack ships on fire off the shoulder of Orion, but I think I have gathered enough experience about Puppet to have valuable things to write about and share.

Please forgive my approximate grasp of the language; if you are a native English speaker, you will surely find some sentences weird or just wrong.

I tried to avoid the temptation to build phrases based on my mother language constructs; I believe I have failed in more than one place.

Various people have helped me with suggestions and corrections; they couldn't solve all my language idiosyncrasies and content limitations, but their input has been very important to make this book better.

I'd like to thank in particular Brice Figureau, Joshua Hoblitt, and Azharuddin Sheikh for the invaluable help and corrections, and Nigel Kersten, Jon Forrest, Calogero Bonasia, Monica Colangelo, and Kartik Vedam for the precious suggestions.

A big hug to Chiara, who is always patient and supportive, and to all my family, even the younger one who is extremely skilled in kicking me out of the bed, encouraging early morning writing sessions.

It's time to enjoy some more weekends together, finally.

About the Reviewers

Dhruv Ahuja is a senior DevOps engineer at a leading financial data vendor. He specializes in orchestration and configuration management in an enterprise, heterogeneous setting. His first brush with Puppet was in 2011 when he developed a solution for dynamically scaling compute nodes for a multipurpose grid platform. He also holds a Master's degree in Advanced Software Engineering from King's College London, and won the Red Hat UK Channel Consultant of the Year award in 2012 for delivering progressive solutions. A long history in conventional software development and traditional systems administration equip him with aptness in both areas. In this era of infrastructure as code, he believes that declarative abstraction is essential for a maintainable systems life-cycle process.

C. N. A. Corrêa (@cnacorrea) is an IT operations manager and consultant, and is also a Puppet enthusiast and an old-school Linux hacker. He has a Master's degree in Systems Virtualization and holds CISSP and RHCE certifications. Backed by a 15-year career in systems administration, Carlos leads the IT operations teams for companies in Brazil, Africa, and the USA. He is also a part-time professor for graduate and undergraduate courses in Brazil. Carlos has co-authored several research papers on network virtualization and OpenFlow, and has presented at peer-reviewed IEEE and ACM conferences worldwide.

I thank God for all the opportunities of hard work and all the lovely people I always find on my way. To the sweetest of them all, my wife Nanda, I thank for all the loving care and support that pushes me forward. Also, to my parents, Nilton and Zélia, for being such a big inspiration for all the things I do.

Brice Figureau works at Days of Wonder, a board game publisher best known for its award-winning train game Ticket to Ride, where he designs, manages, and programs distributed online game servers and the infrastructure they run on. In several previous job roles, he programmed 3D-rendering engines, Photoshop plugins, early mobile Internet services, and voice-recognition-based phone services and learned system administration. He likes to spend time contributing to various open source projects and has started some of his own. He's been using Puppet since Version 0.23.7 and contributed several major features to the Puppet core code that helped make Puppet what it is now. He also maintains www.planetpuppet.org and helps to organize the Paris DevOps Meetups and the DevopsDays Paris conference when time permits. You might find him hanging around in IRC under the masterzen nickname on Twitter with the @_masterzen_ handle, or at different open source, DevOps, or Configuration Management conferences around the world.

www.PacktPub.com

Support files, eBooks, discount offers, and more

You might want to visit www.PacktPub.com for support files and downloads related to your book.

Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.PacktPub.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at for more details.

At www.PacktPub.com, you can also read a collection of free technical articles, sign up for a range of free newsletters and receive exclusive discounts and offers on Packt books and eBooks.

http://PacktLib.PacktPub.com

Do you need instant solutions to your IT questions? PacktLib is Packt's online digital book library. Here, you can access, read and search across Packt's entire library of books.

Why subscribe?

Fully searchable across every book published by Packt

Copy and paste, print and bookmark content

On demand and accessible via web browser

Free access for Packt account holders

If you have an account with Packt at www.PacktPub.com, you can use this to access PacktLib today and view nine entirely free books. Simply use your login credentials for immediate access.

Preface

Puppet has changed the way we manage our systems. When it was released, other configuration management tools were around, but it was clear that it had something special. It came at the right time with the right approach. The challenges of IT infrastructures were beginning to step up to a new level, and the need to automate common activities such as a quick setup and configuration of systems was becoming a requirement. Puppet presented a sane model, based on abstraction of resources and the definition of the expected state of a system, using a clear and sysadmin-friendly language.

There are various books about Puppet around, and most of them are very good. This one tries to contribute with solid and no frills content (few pictures and few large blocks of copied and pasted text) and some new perspectives and topics. It begins with an intense technical overview of Puppet, Hiera, and PuppetDB so that you can use them to design appropriate Puppet architectures that fit your IT infrastructure.

We will explore where our data can be placed, how to design reusable modules, and how they can be used as building blocks for higher abstraction classes. We will try to give a clearer and wider view of what it means to work with Puppet, and what are the challenges we might face when we introduce it on our systems, from code management to deployment rollouts. We will dive into Puppet's internal details and its extension points, showing the multiple ways we can tweak, extend, and hack with it. We will also give a look to less traditional fields, such as Puppet as a configuration-management tool for network devices or cloud services.

The last chapter is about the future: how Puppet is evolving and what we can expect to do with it in the next years.

I'd dare to say that this is the book I'd have liked to read when I was trying to connect the dots and figure out how to do things in the right way, struggling to grasp Puppet's inner concepts and reusability patterns.

Years of pain, experience, evolution, and research are poured in these pages and I really hope they can be useful for your personal adventure exploring Puppet.

What this book covers

Chapter 1, Puppet Essentials, is an intense and condensed summary of the most important Puppet concepts: the baseline needed to understand the chapters that follow and a good occasion to refresh and maybe enrich knowledge about the Puppet language and model.

Chapter 2, Hiera, is dedicated to how to manage our data with Hiera: how to define the lookup hierarchy, organize data, and use different backends.

Chapter 3, PuppetDB, covers the installation, configuration, and usage of PuppetDB, and explores the great possibilities that it may enable in the next generations of modules.

Chapter 4, Designing Puppet Architectures, outlines the components to manage when defining a Puppet architecture: the available tools, how to integrate them, how to cope with data and code, and organize resources to be applied to nodes.

Chapter 5, Using and Writing Reusable Modules, covers the most important Puppet element from the user's perspective, modules, and how to write them in order to be able to reuse them in different infrastructures.

Chapter 6, Higher Abstraction Modules, takes a step further and focuses on modules that use different application modules to compose more complex and wider scenarios.

Chapter 7, Deploying and Migrating Puppet, analyzes the approaches that can be taken when introducing Puppet in a new or existing infrastructure: methodologies, patterns, techniques, and tips for a successful deployment.

Chapter 8, Code Workflow Management, focuses on how to manage Puppet code, from when it is written in an editor to its management with an SCM, its testing and deployment to production.

Chapter 9, Scaling Puppet Infrastructures, covers the challenges you might face in growing infrastructures and how it is possible to make Puppet scale with them.

Chapter 10, Writing Puppet Plugins, covers the many available possibilities to extend the core code with custom plugins and gives a deeper view on how Puppet internals are organized.

Chapter 11, Beyond the System, takes a journey outside the traditional territories, exploring how we can manage with Puppet network and storage equipment and cloud instances.

Chapter 12, Future Puppet, is a step towards Puppet 4 and how its new features may influence the way we work with Puppet.

What you need for this book

You can test the Puppet code present in this book on any Linux system connected to the Internet. You can use the Vagrant environment provided in the example code and have your test machines running a VirtualBox instance on your computer. For this, you need both Vagrant and VirtualBox installed on your system.

Who this book is for

This book is accessible to any Puppet user.

If you are totally new to Puppet, be sure to have given a thorough read of Chapter 1, Puppet Essentials, and to have well understood its principles before continuing your reading.

If you are an intermediate user, enjoy reading the following chapters in order.

If you are an advanced user, you may pick in different pages' useful information and new insights on topic you should already know.

Conventions

In this book you will find a number of styles of text that distinguish between different kinds of information. Here are some examples of these styles, and an explanation of their meaning.

Code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles are shown as follows: For inline documentation about a resource, use the describe subcommand.

A block of code is set as follows:

:backends:

  - http

:http:

  :host: 127.0.0.1

  :port: 5984

  :output: json

  :failure: graceful

  :paths:

    - /configuration/%{fqdn}

    - /configuration/%{env}

    - /configuration/common

Any command-line input or output is written as follows:

puppet agent -t puppet resource package puppet resource service

New terms and important words are shown in bold. Words that you see on the screen, in menus or dialog boxes for example, appear in the text like this: At the end of a Puppet run, we can have metrics that let us understand how much time the Master spent in compiling and delivering the catalog (Config retrieval time).

Note

Warnings or important notes appear in a box like this.

Tip

Tips and tricks appear like this.

Reader feedback

Feedback from our readers is always welcome. Let us know what you think about this book—what you liked or may have disliked. Reader feedback is important for us to develop titles that you really get the most out of.

To send us general feedback, simply send an e-mail to <[email protected]>, and mention the book title through the subject of your message.

If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, see our author guide on www.packtpub.com/authors.

Customer support

Now that you are the proud owner of a Packt book, we have a number of things to help you to get the most from your purchase.

Downloading the example code

You can download the example code files for all Packt books you have purchased from your account at http://www.packtpub.com. If you purchased this book elsewhere, you can visit http://www.packtpub.com/support and register to have the files e-mailed directly to you.

Errata

Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you find a mistake in one of our books—maybe a mistake in the text or the code—we would be grateful if you would report this to us. By doing so, you can save other readers from frustration and help us improve subsequent versions of this book. If you find any errata, please report them by visiting http://www.packtpub.com/submit-errata, selecting your book, clicking on the errata submission form link, and entering the details of your errata. Once your errata are verified, your submission will be accepted and the errata will be uploaded to our website, or added to any list of existing errata, under the Errata section of that title.

Piracy

Piracy of copyright material on the Internet is an ongoing problem across all media. At Packt, we take the protection of our copyright and licenses very seriously. If you come across any illegal copies of our works, in any form, on the Internet, please provide us with the location address or website name immediately so that we can pursue a remedy.

Please contact us at <[email protected]> with a link to the suspected pirated material.

We appreciate your help in protecting our authors, and our ability to bring you valuable content.

Questions

You can contact us at <[email protected]> if you are having a problem with any aspect of the book, and we will do our best to address it.

Chapter 1. Puppet Essentials

There are moments in our professional life when we encounter technologies that trigger an inner wow effect. We realize there's something special in them, and we start to wonder how they can be useful for our current needs and, eventually, wider projects.

Puppet, for me, has been one of these turning point technologies. I have reasons to think that we might share a similar feeling.

If you are new to Puppet, you are probably starting from the wrong place, since there are better fitting titles around to grasp its basic concepts.

This book won't indulge too much in the fundamentals, but don't despair as this chapter might help for a quick start.

It provides the basic Puppet background needed to understand the rest of the contents and may also offer valuable information to more experienced users.

In this chapter, we are going to review the following topics:

The Puppet ecosystem: The components, its history, and the basic concepts behind configuration management

How to install and configure Puppet: Commands and paths to understand where things are placed

The core components and concepts: Terms such as manifests, resources, nodes, and classes will become familiar

The main language elements: Variables, references, resource defaults, ordering, conditionals, comparison operators, and virtual and exported resources

How Puppet stores the changes it makes and how to revert them

The contents of this chapter are quite dense; take your time to review and assimilate them; if they sound new or look too complex, it is because the path towards Puppet awareness is never too easy.

The Puppet ecosystem

Puppet is a configuration management and automation tool. We use it to install, configure, and manage the components of our servers.

Written in Ruby and released with an open source license (Apache 2), it can run on any Linux distribution, many other UNIX variants (Solaris, *BSD, AIX, and Mac OS X), and Windows.

Its development started in 2005 by Luke Kanies as an alternate approach to the existing configuration management tools (most notably, CFEngine and BladeLogic).

The project has