Deploy User Manual

Deploy 3.0

Deploy User Manual

Managing System Deployment with Deploy

Edition 3.0


Legal Notice

Copyright © 2015 Deploy Foundation. This material may only be distributed subject to the terms and conditions set forth in the GNU Free Documentation License (GFDL), V1.2 or later (the latest version is presently available at http://www.gnu.org/licenses/fdl.txt).
May 2015

Chapter 1. Preface

1.1. Audience

This document is intended for software developers and system administrators. It assumes basic experience with software packaging and package repositories, as well as with system installation, system and application configuration, and software updating. Background knowledge in XML file creation and editing, the XML Path Language (XPath), system automation using BASH or other scripting languages, and networking and machine virtualization will also assist the reader.

1.2. What's Not Covered

This manual complements several Deploy documents. The "Installation Instructions" and "Getting Started Guide" provide basic information about obtaining and installing Deploy. The "Deploy Definition File Reference" (DDFR) contains the syntax for definition elements and attributes. Finally, Deploy has a set of man pages explaining the deploy command, its options and the Deploy configuration file, deploy.conf. This manual contains references to all four throughout the text.
These Deploy documents are available at the Deploy website.

1.3. How to Read

The intent of the manual is to help you use Deploy effectively to deploy and maintain systems and applications using software package repositories. This manual takes a top-down approach, explaining the concepts before giving steps, commands and examples that you can use to create and test your own repositories.

1.4. Conventions Used

The document conventions are alphabetically listed below.
Commands
Command names in Deploy or function names in other programs appear throughout the document.
Elements and Attributes
Elements and attributes in XML are referred to by their XPath equivalents. The active node is always considered to be the root, repo; thus, the first element in the XPath expression is the top-level element, while subsequent elements/attributes are its children. Top-level elements in the DDFR and their child elements use the parent/child/child format, while @attribute-names are preceded by an @ symbol. The assumed parent element is repo. For more details on XPath syntax, see XML Path Language.
Example
Code examples appear in a box, displayed below.
<!-- This is a comment in an example code section -->
Filenames
Filenames, folder names, directory trees and file extensions all use a monospace format.
Important
Critical items appear with an "Important" set off tag:

Important

The main top-level element is required.
Notes
Within definition examples, the traditional XML comment tags "<!--" and "-->" contain notes. See Also Example.
A set-off tag highlights things that a user needs to know.

Note

The top-level elements can be in any order in the definition.
Screen output
Computer output to the screen can be inline computer output, usually in Courier monospace font, or included in a box, as in the case below.
/usr/bin/deploy
User Input
Commands entered by the user appear as inline commands or formatted as an Example. If the latter, then special characters indicate different users, as shown below.
# precedes lines entered as a root user

$ precedes lines entered as a normal user
Variables
Deploy variables are presented in monospace font, e.g. %{id}.

1.5. Getting Deploy

Deploy and associated documents are available at the Deploy website at www.deployproject.org.

Chapter 2. Introduction

2.1. Deploy Overview

Deploy is a free, open source platform for managing system and application deployment. The current version works with CentOS and Red Hat Enterprise Linux.
Deploy allows creating two types of repositories: system and package.
System repositories, also known as custom distributions, are useful for system administrators who wish to deploy CentOS or Red Hat Enterprise Linux within their organizations, but with added, or removed, packages and modified default configuration.
Package repositories are useful for independent software vendors (ISVs) and system administrators who wish to deliver software that can be easily deployed and updated using standard operating system package management tools.
Deploy automates common system and application deployment tasks:
  • Builds packages starting from SRPMs or from user-provided files and scripts.
  • Creates package repositories.
  • Creates system installation images (for system repositories).
  • Tests packages, repositories and installation images.
  • Deploys systems and applications.
  • Updates systems and applications.
Deploy Overview

2.2. Repositories

Deploy creates RPM package repositories. RPM is a package management system used by CentOS, Red Hat Enterprise Linux, and others to install, update and remove software and configuration on client systems. Packages are discrete units of software and configuration managed by RPM.
Repositories are collections of packages, typically located on remote servers or CD/DVD media. Repositories are generally organized by the architecture (e.g. i386 or x86_64) and version (e.g. 6) of a base operating system.
RPM package repositories are prevalent in Linux software distribution today. They are used by operating system providers, by software and hardware vendors, and by systems administrators.
Repositories have several key benefits for managing deployment of software and configuration:
  • Uniform. Repositories can be used to deploy software and configuration uniformly, independent of the source (OS provider, OEM, ISV, or internal IT). This consistency eases integration and facilitate automation.
  • Scalable. Package repositories are scalable to meet the needs of a wide range of deployment scenarios including consumer/OEM, small business, enterprise, hosted, and even web farm and massively scaled.
  • Modular. Because packages describe their relationships and dependencies internally, it becomes easy to mix, match, replace, update and remove them to create custom repositories talored to a variety of needs.
Deploy creates two types of repositories: system and package.

2.2.1. System Repositories

System repositories are used to deploy and maintain complete client machines. They contain installation images along with all the software and configuration (in the form of RPM packages) necessary to perform client installation and updates. Clients can be updated against the system repository using either traditional or synchronization methods, the latter ensuring that client packages are kept in strict compliance with the system repository. Deploy provides the YUM Sync Plugin to enable this support.

2.2.2. Package Repositories

Package repositories have two uses. They can serve as input to system repositories. They can also be used for deploying applications to unmanaged client machines, such as consumer desktops. Package repositories do not contain installation images. Neither generally do they contain packages from the base operating system, as these are assumed to be either previously installed on the client, or available through standard operating system update methods, i.e. YUM.
This document will describe creating both system and package repositories.

2.3. Building Repositories

During the build process, Deploy gathers content, processes it, creates a repository, and optionally runs scripts to test, deploy or maintain client machines.
Deploy Build Process

2.3.1. Input

Deploy uses three classes of input: a base operating system repository, additional package repositories, and configuration files and scripts.
  • Base Operating System Repository. Deploy requires a CentOS-compatible operating system repository as input for creating new repositories. Currently supported operating system repositories include CentOS and Red Hat Enterprise Linux, version 6 for i386 and x86_64 architectures, and RedHat Enterprise Linux version 7 for x86_64. See Section 3.3.4, “Specifying Repositories” for more information.
  • Additional Package Repositories. Deploy uses RPM package repositories to locate and download additional software components for the repository. A typical definition will specify multiple additional repositories, including a operating system update repository, and repositories made available from other internal and external providers. See Section 3.3.4, “Specifying Repositories” for more information.
  • Configuration Files and Scripts. The final category of repository content is user-provided configuration files and scripts. Files are copied and scripts are executed during client machine installation and updates. For more information, see Chapter 4, Building RPM Packages.

2.3.2. Process

Deploy uses information from the definition to gather repository content. It uses this information to complete three primary steps.
  • Create repository-specific RPM packages. Deploy creates two types of packages, the release-rpm and optional config-rpms. The release-rpm contains Deploy-generated YUM update configuration. Config-rpm packages contain user-provided files and scripts specified in the definition. Both are discussed under Chapter 4, Building RPM Packages.
    During this step, Deploy also creates a data file, if one does not exist. This file is used to store Deploy generated input data, such as release numbers for generated RPMs, that should be maintained along with the definition.
  • Assemble a package repository. Using the definition, Deploy compiles a list of required packages. It then resolves package dependencies, downloads packages, and creates a complete, consistent package repository. The package repository is used during network installations to client machines. It is also used during client machine updates.
  • Customize system installation images (optional). For system-type repositories, Deploy downloads network installation images from the CentOS-compatible repository and modifies them to include repository-specific information. Deploy can be used to completely automate client installations. The chapters on Chapter 5, Automating System Testing and Chapter 6, Deployment Scenarios describe this in detail.
Deploy stores repository content, both input and generated, in the cache folder, by default /var/cache/deploy/. It reuses information from the cache on subsequent runs, where possible, to improve build performance.

2.3.3. Output

Deploy outputs a complete repository, a data file and a log file.
  • Repository. The primary output of the build process is a complete YUM package repository containing RPM packages. For system-type repositories, the repository also contains system installation images. Together, the YUM repository and system installation images support the complete range of system and application deployment and maintenance scenarios. See the chapters on Chapter 5, Automating System Testing, Chapter 6, Deployment Scenarios and Chapter 7, Package Repositories for more information.
  • Repository Data File. As mentioned above, Deploy creates a data file during the build process. This file is used to track Deploy generated input information.
  • Deploy Log File. Deploy sends output both to the screen and to a log file, located by default at /var/log/deploy.log.

2.3.4. Test, Deploy and Update

The repository generated by Deploy is used to perform testing, deployment and maintenance to client machines. These steps can be performed in one of three ways:
  • Manually by the user or administrator.
  • Automatically by Deploy using deployment scripts.
  • Or, because repositories are in a standard format, automatically by any systems management system that can deploy and maintain native CentOS and Red Hat Enterprise Linux systems.

2.4. Deploy Files and Folders

Deploy has several key files and folders, listed in the table below.
/etc/deploy/deploy.conf Deploy looks for this file to locate deploy configuration. See the deploy and deploy.conf man pages for more information.
/usr/bin/deploy The deploy command.
/usr/lib/python$version/site-packages/deploy/ Python code for Deploy, where $version is the Python version.
/usr/share/doc/deploy-$version/ Documentation for Deploy, where $version is the Deploy version.
/usr/share/deploy/schemas/ Relax NG schemas for definition validation.
/usr/share/deploy/templates/ Templates for use in definition creation. Discussed in Section 3.2, “Working with Templates”.
/var/cache/deploy/ Deploy stores cached content in this folder. See the deploy and deploy.conf man pages for cache-related options.
/var/lib/deploy/ Location where Deploy stores variable data including data for troubleshooting installed config-rpm and deployment scripts.
/var/log/deploy.log Deploy's default log file.
/var/www/html/repos/ Deploy publishes completed repositories to system and package subfolders of this folder by default. An alternative location can be specified on a per-repository basis within a definition.

2.5. Glossary of Terms

Following are terms specific to the Deploy environment. They are arranged in alphabetical order.
build machine
The machine on which Deploy is installed.
client machine
The target machine on which repository content is installed.
config-rpms
RPM packages created by Deploy during the build process. These packages contain user-specified configuration files and scripts.
For more information on config-rpms, see the Chapter 4, Building RPM Packages chapter in this document. See also the section on config-rpms in the Deploy Definition File Reference (DDFR).
data file
A file maintained by Deploy to track Deploy-generated input data. Data files are created in data folders. Both are named using the repository ID with the .dat extension. For example, given the repository ID "web-server-centos-6-x86_64", Deploy will create a data folder and data file as follows:
web-server-centos-6-x86_64.dat/web-server-centos-6-x86_64.dat
By default, Deploy creates the data folder in the same folder as the definition. An alternative directory location can be specified on the deploy command line using the '--data-root' parameter. See the Deploy man page for more information.

Warning

Care must be taken not to delete the data file. Doing so will cause Deploy to regenerate missing information. As a result, installed client machines may fail to download updated RPMs or may report errors when performing RPM GPG key checking.
definition
A file that defines actions Deploy should take during the build process. By convention the file uses the .definition extension. The file is in XML format as defined in the Deploy Definition File Reference (DDFR).
Deploy Definition File Reference (DDFR)
The rules of syntax for the definition file. See the separate document at http://www.deployproject.org/docs/ .
include element
An element that allows reusing content from external documents. See the Deploy Definition File Reference (DDFR) for general information on using include elements within definitions.
release-rpm
An RPM package created by Deploy during the build process. This package contains the YUM Sync Plugin and YUM configuration for the repository.
For more information on the release-rpm and how to customize its content, see the Chapter 4, Building RPM Packages chapter in this document. See also the section on release-rpm in the Deploy Definition File Reference (DDFR).
repository
A deployment repository created during the Deploy build process. Provides RPM packages and, optionally, installation images for use during client machine installations and updates. Can be accessed over the network from the repository URL.
repository ID
A unique id for a repository. Specified in the definition. Typically the definition should contain the name, os, version and architecture of the repository, e.g. 'web-server-centos-6-x86_64'.
repository URL
The web location (URL) for downloading repository content. By default the repository URL is as follows, where $IP_ADDRESS is the address of the build machine, %{type} is the type of the repository (system or package), and %{id} is the repository ID
http://$IP_ADDRESS/deploy/%{type}s/%{id}
For example, if the IP Address for the build machine is 192.168.1.5 and the repository type is system with an id of web-server-centos-6-x86_64 the repository URL is as follows:
http://192.168.1.5/deploy/systems/web-server-deploy-6-x86_64
The repository URL is configurable using the remote-url element described in the Deployment Elements chapter of the Deploy Definition File Reference (DDFR).
templates
Deploy includes a number of templates to assist with common definition creation tasks. These templates are located in the folder at /usr/share/deploy/templates/. See Section 3.2, “Working with Templates” for additional information.
YUM Sync Plugin
A YUM plugin provided by Deploy that allows synchronizing packages on client machines with packages in the repository. The plugin is included by default in the release-rpm, which serves as the mechanism for installing it on client machines.
The plugin establishes the repository as the master repository, disables all other repositories, and provides a new argument, sync, to YUM. Help for the sync argument is available on client machines using the yum help command. Running yum sync allows YUM to calculate package additions and removals based on packages available in the repository. For more information, see Section 4.1, “Building a Release Package”.

Chapter 3. Building a System Repository

3.1. Configuring a Build Machine

Deploy installation requires a machine running CentOS or Red Hat Enterprise Linux (RHEL) version 6 for the x86_64 architecture. Deploy is designed to work in a networked environment.
Recent advances in virtualization, including the introduction of a Kernel-based Virtual Machine system (KVM), have made it both easy and compelling to use virtual machines for testing client installation and updates. For this reason, we recommend installing Deploy on a machine with one of the following processor architectures:
  • an Intel processor with the Intel VT and the Intel 64 extensions, or
  • an AMD processor with the AMD-V and AMD64 extensions.
Refer to the Red Hat Enterprise Linux 6 Virtualization Administration Guide to determine if your processor has the virtualization extensions.
For capacity planning purposes, the machine should have a separate logical or physical drive for the /var directory. Deploy uses the /var/cache/deploy folder for caching repository content and the /var/www/html/repos folder for storing completed repositories. The recommended minimum drive space allocated for each of these folders is 15GB (30GB total). Depending on the number and size of repositories created, you may need more disk space.
Disk access is another consideration. Due to the size of the files handled, fast disk access is preferable. To accommodate the speed requirements, it is best to use hard disk drives with Serial Advanced Technology Attachment (SATA) interfaces.
To install Deploy, follow the instructions on the download page at http://www.deployproject.org.
Now you are ready to begin defining your repository!

3.2. Working with Templates

Before you can use Deploy, you must first create a definition. Definitions describe the content to include in your system, as well as instructions for ongoing testing, deployment, and maintenance. Definitions can feel somewhat daunting at first, but Deploy includes a number of templates to help with the task. In fact, you will almost never create a definition from scratch. It is much easier to copy an existing template and modify.
After you install Deploy, you can find the default templates in the /usr/share/deploy/templates folder. You should not modify the templates in this folder, however, as your changes may be overwritten in future updates.
Instead, you should copy the templates to a new folder, and direct Deploy to look for them in this location. This will allow you to make changes, while still preserving the original files for reference.
Here are the steps:
1. Create a new templates folder and copy files to it. Substitute the name of your folder for the text "/my/templates/folder" below.
mkdir -p /my/templates/folder
cp -a /usr/share/deploy/templates/* /my/templates/folder
2. Add a line to /etc/deploy/deploy.conf as shown below:
<deploy>
<templates-path>/my/templates/folder</templates-path>
</deploy>
As you are creating definitions, you will often want to tell Deploy to include a file, or a portion of a file, from a templates folder. You can do this using the %{templates-dir} macro. When Deploy sees this macro it will look for the specified file in a folder listed in deploy.conf (multiple folders can be listed). If it cannot find the file, it will then search look for the file in the default templates folder.
You will see the %{templates-dir} macro used frequently throughout the remainder of this document.
Information on templates and template folders is also available in the Deploy Definition File Reference (DDFR) and the Deploy Configuration File Manual Page (man deploy.conf).

3.3. Creating a Definition

For the purpose of this user guide, examples, processes and illustrations will focus on defining, building, deploying and updating an Apache web server based on CentOS version 6. Apache is a project of the Apache Software Foundation.

3.3.1. Creating a definition

A definition is a text file in XML format. It can be created using any text editor - vi, emacs, nano, or gedit - or a dedicated XML editor, if you prefer. The tags <definition> and </definition> encapsulate all additional elements and attributes. The example below shows a basic web server definition. It is taken from the file at /usr/share/deploy/templates/el6/httpd/web-server.definition.
<?xml version="1.0" encoding="utf-8"?>
<definition>

<macro id='name'>web-server</macro>
<macro id='os'>centos</macro>
<macro id='edition'>server</macro>
<macro id='version'>6</macro>
<macro id='arch'>x86_64</macro>
<macro id='id'>%{name}-%{os}-%{version}-%{arch}</macro>
<macro id='test-install-enabled'>false</macro>

<main>
<fullname>Example Web Server</fullname>
<name>%{name}</name>
<os>%{os}</os>
<version>%{version}</version>
<arch>%{arch}</arch>
<id>%{id}</id>
</main>

<repos>
<include href='%{templates-dir}/%{norm-os}/common/repos.xml'
         xpath="./repo[@id='%{os}-%{edition}-base' or 
                       @id='%{os}-%{edition}-updates']"/>
</repos>

<packages>
<include href="%{templates-dir}/%{norm-os}/common/packages.xml"
         xpath="./packages[@id='core']/*"/>
</packages>

<config-rpms>
<include href='%{templates-dir}/%{norm-os}/iptables/iptables-update.xml'/>
<include href='%{templates-dir}/%{norm-os}/httpd/httpd-config.xml'/>
</config-rpms>

<test-update>
<include href="%{templates-dir}/%{norm-os}/libvirt/deploy.xml"
         xpath="./*"/>
<include href="%{templates-dir}/%{norm-os}/httpd/test-httpd-config.xml"/>
</test-update>

<test-install enabled="%{test-install-enabled}">
<include href="%{templates-dir}/%{norm-os}/libvirt/deploy.xml"
         xpath="./*"/>
<include href="%{templates-dir}/%{norm-os}/httpd/test-httpd-config.xml"/>
</test-install>

</definition>
The first line, beginning in "<?xml", identifies the file as an XML file. This line is not required, but it is a good practice to include it for compatibility with future XML format changes.

3.3.2. Saving a definition

Definitions are commonly saved with the name of the repository and the file extension ".definition", e.g. web-server.definition.

Warning

Definitions may contain sensitive data including system root passwords and GPG private keys. Take care to save your definitions, and related files including data files and templates, in a secure location.

3.3.3. Naming a system

The first required element of the definition, main, includes the name of the system, the base operating system, the OS version and architecture, and a system id. Deploy uses the system id to create folders for storing the completed system (under /var/www/html/deploy/systems, by default) and for caching system content (under /var/cache/deploy/systems, by default).

Note

The elements can be in any order within a definition.
Typically you will want to complete the main section of your definition using macros. Macros make your definition flexible, so that you can provide content once, and reuse it through the remainder of your definition.
To make use of macros, create your definition as shown below:
<macro id='name'>web-server<macro>
<macro id='os'>centos<macro>
<macro id='edition'>server<macro>
<macro id='version'>6<macro>
<macro id='arch'>x86_64<macro>
<macro id='id'>%{name}-%{os}-%{version}-%{arch}<macro>

<main>
  <name>%{name}</name>
  <os>%{os}</os>
  <version>%{version}</version>
  <arch>%{arch}</arch>
  <id>%{id}</id>
</main>
The Deploy Definition File Reference (DDFR) contains additional details on the main section, as well as on creating and using macros.

3.3.4. Specifying Repositories

During the system build process, Deploy obtains software from existing repositories. In this section, you will define repositories to be used for creating your system.
The easiest way to define repositories is to include them from the repos.xml template, as shown below:
<repos>
<include href="%{templates-dir}/%{norm-os}/common/repos.xml"
         xpath="./repo[@id='%{os}-%{edition}-base' or 
                       @id='%{os}-%{edition}-updates']"/>
</repos>
The example above does the following:
  • It includes content from an external document using the include element .
  • It uses the attribute href='%{templates-dir}/%{norm-os}/common/repos.xml' to specify the location of the external document. Deploy resolves the %{templates-dir} and %{norm-os} macros at runtime to locate the file in an appropriate folder.
  • It uses the attribute xpointer="xpointer(./repo[@id='%{os}-%{edition}-base' or @id='%{os}-%{edition}-updates'])" to specify an XPath query selecting specific content from the document.
Templates are discussed previously in this document under Section 3.2, “Working with Templates”. In addition, the Deploy Definition File Reference (DDFR) contains a chapter on Templates, Macros and Include Elements.

3.3.4.1. Working with CentOS repositories

The repos.xml template is configured to access CentOS base and update repositories from the mirror at http:// mirror.centos.org/centos. For improved performance, you may wish to modify the template to use an alternative mirror, as follows:
  • View the list of CentOS mirrors at http://www.centos.org/download/mirrors and select an alternative.
  • Open the repos.xml file(s) in the templates folder and modify the baseurl and gpgkey paths to use the new location.

3.3.4.2.  Working with Red Hat Enterprise Linux repositories

Before you can use the repos.xml template with Red Hat repositories, you will need to complete several additional steps. These steps require at least one active software subscription. Visit the Red Hat website to purchase a subscription.
3.3.4.2.1. Red Hat Enterprise Linux base repositories
Once you have an active subscription, complete the following steps to create a base operating system repository and modify the repos.xml template to use it.
First, create a base operating system repository.
  1. From the Red Hat Customer Portal, download an ISO image for the desired version of Red Hat Enterprise Linux. A single DVD image is recommended for easier mounting.
  2. Mount the ISO image to a folder location as follows:
    TMPDIR=/mnt/tmp
    mkdir $TMPDIR
    mount -o loop downloaded.iso $TMPDIR
  3. Copy contents of the ISO image to a permanent folder. Be sure to also copy the hidden .treeinfo file from the root of the ISO image to the permanent folder to avoid errors during Deploy operation. The example below shows coping image contents to a web server location on the localhost.
    DESTDIR=mirror/rhel/enterprise/6Server/en/os/x86_64
    mkdir -p /var/www/html/$DESTDIR
    cp -pr $TMPDIR/* /var/www/html/$DESTDIR
    cp -p $TMPDIR/.treeinfo /var/www/html/$DESTDIR
    umount $TMPDIR
    See the Red Hat Enterprise Linux 6 Installation Guide, under the topic Preparing for a Network Installation, for more information.
Then, modify the repos.xml template to use the repository you have created.
3.3.4.2.2. Red Hat Enterprise Linux updates (CDN) repositories
Updates repositories for Red Hat Enterprise Linux are provided by the Red Hat Content Delivery Network (CDN). To access repositories from CDN, you will need to obtain SSL certificates, and copy them to the location expected by the repos.xml template. See the steps below:
  1. Install Red Hat Enterprise Linux on a client machine.
  2. Register the machine with Red Hat Subscription Manager (RHSM). This can be done during installation, or at first boot. Or, post installation using the subscription-manager command as follows:
    subscription-manager register
    subscription-manager attach --auto
    
  3. Once the machine has an attached subscription, the necessary SSL key and certificates can be located as follows, where xxx below is a multiple-digit subscription id:
    Certificate/Key Type Certificate/Key Location
    sslcacert /etc/rhsm/ca/redhat-uep.pem
    sslclientcert /etc/pki/entitlement/xxx.pem
    sslclientkey /etc/pki/entitlement/xxx-key.pem
    To use the the key and certificates with Deploy, copy them to the %{templates-dir}/%{norm-os}/common/rhsm folder. Then rename the xxx.pem and xxx-key.pem files as follows, where %{edition} and %{version} are the edition and version of the installed operating system, e.g. "server" and "6":
    mv xxx.pem %{edition}-%{version}.pem
    mv xxx-key.pem %{edition}-%{version}-key.pem
    

3.3.5. Specifying Packages

Once you have defined external repositories, you can specify packages and groups to include. Use the packages top-level element to accomplish this.
For system repositories, at a minimum you will need the core group and, depending on the OS version, a few extra packages. Because the minimum set varies across OS versions, Deploy provides a template, packages.xml, to assist with package selection.
The example below shows including a minimum set of packages using the packages.xml template.
<packages>
  <include href="%{templates-dir}/%{norm-os}/common/packages.xml"
           xpath="./packages[@id='core']/*"/>
</packages>
In addition, you may wish to include the minimal set of packages required for a Graphical User Interface (GUI). Again, the set of packages varies across OS version, and again the packages.xml template can assist, as shown in the example below:
<packages>
  <include href="%{templates-dir}/%{norm-os}/common/packages.xml"
           xath="./packages[@id='core' or @id='gui']/*"/>
</packages>
Many additional groups exist that may be useful based on your scenario. You can browse groups provided by an external repository by opening the repository's groups file. The groups file can be located in the repository's repodata folder, with a filename typically ending with comps.xml.
In addition to groups, you can also specify individual packages, as shown below.
<package>httpd</package>
Often, however, for greater flexibility and content reuse, individual packages are specified within the context of config-rpms. Config-rpms will be discussed later in this document, in Chapter 4, Building RPM Packages.

3.4. Validating a Definition

Once your definition is complete, you can use Deploy to validate its syntax.
#deploy --validate-only web-server.definition
If your definition is valid, Deploy will display the complete definition and exit. Otherwise, it will display error information, allowing you to locate and correct issues.

3.5. Building a System Repository

Now that you have defined your repository, you use the deploy command to build it. Running Deploy validates the definition syntax, creates the repository, and logs process results. The default level of screen output includes errors, build status and file download and creation status. Deploy also sends this output to the log file, by default /var/log/deploy.log. If a critical error occurs, Deploy includes detailed error information in the log file.
The initial build of a repository can be time consuming, on the order of 2-10 minutes or more, depending on the location of package repositories (local or remote), network connection speed, the number of packages included in the repository, and the amount of content that can be reused from the Deploy shared cache. Subsequent builds complete in much less time, often less than a minute, depending on the level and type of change since the previous build.
To build the web-server repository we have been defining, enter the command below:
#deploy web-server.definition
The output for an initial run details all the steps for downloading packages, checking package dependencies and creating images. If there are any errors, Deploy notes those and, in many cases, halts the process.
When the process is complete, you can locate your new repository at /var/www/html/deploy/systems/web-server-6-i386.
You will also notice that Deploy has created a data file at web-server-6-i386.dat/web-server-6-i386.dat, in the same location as you definition. Take care not to delete the data file as it contains important information for your system.

3.5.1. Using command-line macros

Because we were careful to make use of macros while creating the definition, the same definition can be used for multiple operating system versions and architectures. To provide the desired version and arch when executing Deploy, use the --macro command line option. The example below shows using command line macros to specify both os and arch.
#deploy --macro "os:rhel" --macro "arch:x86_64" web-server.definition

3.6. Updating a System Repository

To keep your system content current, you will need to run Deploy on a regular basis. Each time Deploy runs, it checks for changes in the definition or in input data - such as new packages available from an input repository - and refreshes the repository as necessary.
If Deploy finds updated packages, it refreshes the repository to contain just the current set of packages. Specifically, when Deploy determines that packages have changed, it calculates the list of packages needed to satisfy repository requirements, and then adds and removes packages as necessary to bring the repository into conformance with the list. This behavior allows the YUM Sync Plugin to rely on packages in the repository as a master list of packages to be installed on client machines.
To update your repository, you use the same command as you used to initially build it, for example:
# deploy web-server.definition
You can automate repository updates using cron. The following example shows a shell script that updates the web-server repository.
#!/bin/sh
DEF_DIR=/root/definitions
PARAMS="\
  $DEF_DIR/web-server.definition \
  "

for params in $PARAMS; do
  deploy $params 
done
You can extend this script to update additional repositories by adding lines to the PARAMS variable. To run the script on a nightly basis, save it to a file (e.g. nightly_build.sh), copy the file to the /etc/cron.daily folder, and make it executable (e.g. by using the command chmod +x /etc/cron.daily/nightly_build.sh).

3.7. Cleanup and Maintenance

Each time you change the id element in a definition, either directly or indirectly using a command-line macro, Deploy creates new folders in the publish and cache folders (/var/www/html/deploy and /var/cache/deploy, by default). To preserve hard drive space, you may wish to review the publish and cache folders on occasion. Folders for obsolete system and package repositories can be deleted using standard file system methods, as shown below:
rm -rf obsolete_folder

Chapter 4. Building RPM Packages

Now that you have completed a basic definition you can add to it by creating RPM packages that are specific to your repository. Deploy builds three types of RPMs: the release-rpm, config-rpms, and RPMs that start from source RPMs (SRPMs).
These topics will be discussed in the following sections.

4.1. Building a Release Package

A release package is an essential component of a YUM repository. It contains information that allows client machines to connect to the repository using YUM. YUM then manages download, install and update of RPM packages on the individual client machines.
Deploy automatically creates a release package for the repository. You can control aspects of the release package using the release-rpm element.
The release package, referred to as the release-rpm is created by default, even if no release-rpm element is specified in the definition. In the default case, the release-rpm performs two actions:
  • Installs a YUM repository definition for the repository in the /etc/yum.repos.d folder. This configures client machines to check for updates in the repository each time YUM is run.
  • For system-type repositories, installs a plugin module for YUM that supports addition and removal of packages on the client machine in synchronization with changes in the repository. See the section below for more information on the YUM Sync Plugin.
The release-rpm element allows you to disable installation of the Yum Sync Plugin, as well as to control GPG key checking for downloaded RPM packages. See the Deploy Definition File Reference (DDFR) for complete information on the release-rpm element.
You can disable creation of the release-rpm, if desired, as shown below. This may be useful if you desire to create a custom release package, for example as shown in Section 4.3, “Building Custom Packages”.
<release-rpm enabled='false'/>

4.1.1. YUM Sync Plugin

Deploy provides an extension to YUM, called the YUM sync plugin, that allows keeping installed clients up to date with changes in a "master" repository. The YUM sync plugin is installed to client machines by default (for system-type repositories) via the release-rpm. For package-type repositories, the plugin is automatically disabled.
Updates
In addition to installing the sync plugin, the release-rpm also configures the sync plugin to use the Deploy-generated repository as the master repository. This disables all other package repositories on the client machine, allowing the plugin to get an accurate list of what packages should be installed on the client using packages available in the repository.
The sync plugin provides a new command line argument sync for use with YUM:
# yum sync
The sync argument behaves similarly to the YUM update argument, with two additions: 1) it adds packages from the repository that are not installed on the client, and 2) it removes packages from the client that do not exist in the repository.
The sync plugin can be excluded from the release-rpm using the release-rpm/updates element as shown below:
<release-rpm>
<updates sync="false"/>
</release-rpm>
See the Deploy Definition File Reference (DDFR) for more information on the release-rpm element.
In addition, the sync plugin can be disabled on individual client machines using the /etc/yum/yumplugin.d/sync.conf file. It can also be disabled at YUM runtime using the --disableplugins command line parameter, for example:
# yum --disableplugins sync ...
See Chapter 6, Deployment Scenarios for information on using the YUM sync plugin in various deployment scenarios.

4.2. Building Configuration Packages

Deploy allows you to deploy and update client configuration in the same manner as you deploy and update individual software components, using RPM packages. This simplifies your tasks as a system administrator by allowing you to manage software and configuration using uniform tools and processes. Creating RPM packages manually can be cumbersome and time consuming. Deploy reduces overhead by creating RPMs automatically during the build process, based on information you provide in the definition, and by providing a framework for you to test RPMs.
In Deploy, configuration packages are defined using config-rpm elements. This section covers config-rpm parent elements, defining config-rpms and creating config-rpm templates. Config-rpm testing is covered in Chapter 5, Automating System Testing.

4.2.1. Config-rpm parent elements

Config-rpm elements can be used within four top-level, or parent, elements of the definition. Those elements include the following:
  • The config-rpms (plural) element, a simple container for config-rpm (singular) elements. This is the most common parent element, and will be shown in the examples below.
  • Deployment elements including the test-install, test-update and publish elements. Specifying config-rpms within the context of deployment elements is useful for deployment template authors. It allows them to ensure that deployment-environment specific configuration is included within the repository. This topic is discussed further in Chapter 5, Automating System Testing, later in this document.
See also the Deploy Definition File Reference (DDFR) for more information on the above-mentioned parent elements.

4.2.2. Defining config-rpms

Using Deploy, you can define config-rpms to complete the following:
  • Require external packages or capabilities to be installed prior to installing the config-rpm. This prevents configuration files from being overwritten, and ensures that scripts operate as expected.
  • Install files and folder of files, either as new files or to replace files installed by other packages.
  • Run scripts that perform actions when the config-rpm is installed, updated or uninstalled.
  • Run scripts that perform actions when other RPM packages are installed or uninstalled.
The sections below discuss each of these items.

4.2.2.1. Creating a config-rpm element

The first step for defining a config-rpm is to create the config-rpm element. The element requires an @id attribute which is used in the filename of the created RPM. For the web server example, you will give the config-rpm an id of httpd-config as shown below. Note that the config-rpm element must be included within a containing element, the most common of which is the config-rpms element shown below.
<config-rpms>
<config-rpm id="httpd-config">

...

</config-rpm>
</config-rpms>
You can provide additional information about the config-rpm using summary, description and license elements. See the section on config-rpms in the Deploy Definition File Reference (DDFR) for complete information.

4.2.2.2. Requiring packages and capabilities

Using requires and repo elements you can specify packages or capabilities, such as files or virtual capabilities, that must be installed prior to installing the config-rpm.
See the config-rpm section in the Deploy Definition File Reference (DDFR) for more information on requires and repo elements and on capabilities.
For the web-server example, several packages and several executable files must be installed as prerequisites for files and scripts contained within the httpd-config package.
<config-rpm id="httpd-config">

<requires>httpd</requires>
<requires>iptables</requires>
<requires>/bin/ps</requires>
<requires>/sbin/pidof</requires>

</config-rpm>
In some cases a config-rpm may do nothing more than specify required capabilities. This is essentially an alternative to specifying packages using packages element discussed in the previous chapter.
Using config-rpm/requires elements to specify capabilities has several advantages over using package elements.
  • It supports content reuse. For example, many repositories may require web server configuration. By listing web server requirements in a config-rpm element, the config-rpm element can be saved as a template in a separate file, and included across multiple repositories. This approach is covered in more detail under Section 4.2.3, “Using config-rpm templates”.
  • It allows specifying files and virtual capabilities in addition to packages. This may also allow for greater definition reuse as filenames and virtual capabilities are sometimes more stable than package names.
The config-rpm/repo element, discussed in the DDFR, allows you to identify additional input repositories. These elements have the same syntax as repos/repo elements, and operationally Deploy treats them the same. Specifying input repositories within the context of a config-rpm is useful when creating config-rpm templates, as it allows template authors to ensure that repositories are available to satisfy config-rpm requirements.

4.2.2.3. Installing files

Config-rpm elements allow you to specify files to be installed at certain locations on client machines. This is accomplished using files elements.
The example below shows configuring the client machine firewall for web server access. The files element directs Deploy to create a file named iptables-httpd using text provided in the body of the element, and install it to the /etc/rc.d/init.d folder with the file mode "644". The file itself is written to be OS version independent by making use of the %{version} macro to include conditional logic.
<files content='text' destname='iptables-httpd' destdir='/etc/rc.d/init.d'
       mode='750'>
#!/bin/sh
#
#chkconfig: 2345 07 91
#description: Iptables configuration for HTTP
#
VERSION=%{version}
FILE=/etc/sysconfig/iptables

case "$1" in
  "start")
# Enable http traffic through the firewall
if [[ ! `fgrep -e "--dport 80 " $FILE` ]]; then
  if [[ $VERSION == 5 ]]
    then
      sed -i '1,/REJECT/ {/REJECT/i\
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 80 -j ACCEPT
}' $FILE
    else # version 6 and future
      sed -i '1,/REJECT/ {/REJECT/i\
-A INPUT -m state --state NEW -m tcp -p tcp --dport 80 -j ACCEPT
}' $FILE
  fi
fi
     ;;
  "stop")
     exit 0
     ;;
esac
</files>

Note

The attribute value content="text" provides a convenient method to include files in the repository without needing to create and manage files on disk. See the Deploy Definition File Reference (DDFR) for more information on the config-rpm/files element.

4.2.2.4. Using scripts

Config-rpm elements also allow you to run scripts when the config-rpm is installed, uninstalled or updated.
For the web-server example, let's include a script that accomplishes two functions:
  • Starts the Apache web service (httpd) and configures it to start at system boot
  • Determines if the new service (iptables-httpd - added using the files element) has changed and, if so, configures it to start at system boot. Also starts the iptables-httpd service and restarts the iptables service, unless the system is in the middle of a clean install (i.e. anaconda is in the list of active processes).
We will specify the type for this script as post. Post scripts are executed each time the config-rpm is installed or updated.
<script type='post'>
# Start web server and configure to start at system boot
SERVICE=httpd
if [[ -z `pidof $SERVICE` ]] ; then
  service $SERVICE start
fi
chkconfig --level 345 $SERVICE on

# Enable http traffic through the firewall
if [[ $changed =  *\ /etc/rc.d/init.d/iptables-httpd\ * ]]; then
  chkconfig --add iptables-httpd
  # /etc/sysconfig/iptables doesn't exist yet if we're running under anaconda
  if ! ps aux | grep "[/]usr/bin/anaconda"; then
    service iptables-httpd start
    service iptables restart
  fi
fi
</script>
4.2.2.4.1. Writing idempotent scripts
Depending on the type of your script, it can run each time the config-rpm is installed, uninstalled or updated. For this reason, scripts should be written to be idempotent, meaning that they can be run multiple times with the same result.
Following are some tips for writing idempotent scripts:
  • Test if a file exists before deleting it, or use an underlying utility that does this for you, for example-
    if [[ -e file ]]; then rm file; fi
    or
    rm -f file
  • Test if a service is running before starting it, or if service interruption can be tolerated, restart the service each time instead, for example-
    if [[ -z `pidof service` ]]; then service service start; fi
    or
    service service restart
  • Test if text exists in a file before adding it, for example-
    if ! grep "some text" file; then echo "some text" >> file; fi
These are a few of many possible examples. By keeping in mind that your scripts can run more than once, you can make the necessary adjustments.
4.2.2.4.2. Running scripts only when installed files change
You may want to perform certain actions, such as restarting system services, only when specific files that you are installing change. This eliminates unnecessary disruption to previously installed client machines. To do this, you can make use of the $changed variable from within your scripts. This variable contains a string-separated list of files, currently or previously installed by the config-rpm, that meet one or more of the following criteria:
  • added since the last config-rpm update
  • modified since the last config-rpm update
  • differs from the file on disk
  • removed since the last config-rpm update
The section above shows an example script that uses the $changed variable to execute only when the iptables-httpd file changes.
4.2.2.4.3. Additional resources on writing RPM scripts
Writing scripts for use in RPMs is a broad topic. Fortunately, a number of resources exist to provide additional information and examples. Here is a list to get you started.

4.2.2.5. Using triggers

A trigger is a script, much like the scripts described above, except that it runs when an external package or capability is installed or uninstalled. Triggers allow config-rpms that depend on other packages to properly configure themselves when those other packages are installed or removed. See the Fedora Project's documentation website for general information on triggers. See also the Deploy Definition File Reference (DDFR) chapter on config-rpms for complete information on the trigger element, including an example.

4.2.3. Using config-rpm templates

Often you will want to reuse a config-rpm definition across multiple definitions. This can be accomplished by saving the config-rpm definition to a separate file, referred to as a template, and inserting the template within a definition using an include element .
The following example shows a file named httpd-config.xml, which is a compilation of the individual examples of this section. For reference, the httpd-config.xml file is also included in the Deploy templates folder.
<?xml version="1.0" encoding="UTF-8"?>
<config-rpm id='httpd-config'>
<!-- http -->
<requires>httpd</requires>
<requires>mod_ssl</requires>
<requires>/sbin/pidof</requires>

<!-- for use with iptables-update -->
<include href='%{templates-dir}/%{norm-os}/iptables/iptables-update-client.xml'
         xpath='./*'/>
<files destname='%{rpm-id}' destdir='/etc/sysconfig/iptables-update.d'
       content='text'>
-A INPUT -m state --state NEW -m tcp -p tcp --dport 80 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 443 -j ACCEPT
</files>

<script type = 'post'>
# Start web server and configure to start at system boot
if [[ -z `pidof httpd` ]] ; then
  service httpd start > /dev/null
fi
chkconfig --level 345 httpd on
</script>
</config-rpm>
This file can be inserted within a definition using an include element as shown below:
<config-rpms>
<include href="%{templates-dir}/%{norm-os}/httpd/httpd-config.xml"/>
</config-rpms>
See the Deploy Definition File Reference (DDFR) for more information and examples regarding include elements.

4.3. Building Custom Packages

Deploy allows building custom RPMs starting from user-specified Source RPMs (SRPMs). SRPMs are a standard mechanism for distributing source code in a way that is easy to modify and compile into installation packages (RPMs) for different operating system architectures. See the Fedora Project's RPM Guide for information on creating and modifying Source RPMs.
The process Deploy uses to build RPMs from SRPMs includes these steps:
  • Obtaining a user-provided SRPM.
  • Building RPMs using a default or user-provided template. The template specifies the rpmbuild environment as well as scripts to complete the actual rpmbuild and verification process.
  • Signing the built RPMs.
  • Including the built RPMs in the repository.
Using the srpmbuild element, you specify individual SRPMs to build. For each SRPM, you can specify the following:
  • The path to an SRPM file, or
  • The name of an SRPM and the path to an SRPM repository where it can be located, or
  • A script that completes desired actions and finally copies an SRPM to a location specified by Deploy. This is useful, for example, for creating an SRPM from scratch starting from a Source Code Management system, or for patching an existing SRPM.
  • Optionally, you can specify a custom template for Deploy to use for creating the rpmbuild environment and for building and verifying the RPMs.
See the Deploy Definition File Reference (DDFR) for complete information and examples for using the srpmbuild element.

4.4. Providing GPG Signing Keys

RPM packages are typically signed with GPG keys as a measure of ensuring package authenticity. GPG keys come in pairs, secret and public. Packages are signed using the secret key. The private key is distributed to client systems, which use it to verify the package at install time.
Deploy automatically signs all packages it creates using release-rpm, config-rpm and srpmbuild elements. To sign the packages, it uses either automatically generated or user-provided keys.
In the default cause, Deploy automatically generates a key pair, one per repository, and stores key data in the data file. If the YUM Sync Plugin is enabled for the repository, this approach is typically sufficient as public keys are automatically distributed and installed on client systems by the plugin, and end users do not need to recognize and explicitly trust the Deploy-generated key.
When the YUM sync plugin is disabled, however, either manually via the release-rpm element or automatically for package-type repositories, the opposite is true. In this case, users are prompted to install GPG keys after viewing summary information and, thus, it is often desirable to provide GPG signing keys containing specific, user-recognizable information about the key and provider.
See the release-rpm chapter for more information on the YUM sync plugin.
Using the gpgsign element, you can provide signing keys for use with your repository. The process involves three steps: creating keys, exporting keys, and creating the gpgsign element.

4.4.1. Creating keys

For our example, we will create keys using the batch mode for GPG key creation. While the key is being created, the process may stall and you may need to do some additional work (move the mouse, switch windows, download some web pages, etc.) to create sufficient random data.
Create a file with the following content. Modify the lines starting with Name-Real and Passphrase. The passphrase line is optional.
Key-Type: DSA
Key-Length: 1024
Subkey-Type: ELG-E
Subkey-Length: 1024
Expire-Date: 0
Name-Real: My Deploy Signing Key
Passphrase: The quick brown fox jumped over the lazy dog.
Next, type the following at the command line, replacing myfile with the name of the file created above:
# gpg --batch --gen-key myfile
Once the process is complete, check that the key was generated successfully using the following command:
# gpg --list-keys
You should see an item with the uid of "My Deploy Signing Key"

4.4.2. Exporting keys

To export the newly created keys from the GPG keyring, complete the following:
# gpg --export -a "My Deploy Signing Key" > RPM-GPG-KEY-mykey
# gpg --export-secret-key -a "My Deploy Signing Key" > RPM-GPG-KEY-mykey-secret
You can verify the exported files using the commands:
# cat RPM-GPG-KEY-mykey
# cat RPM-GPG-KEY-mykey-secret
As each file is listed, it should begin with the following text, respectively:
-----BEGIN PGP PUBLIC KEY BLOCK-----
-----BEGIN PGP PRIVATE KEY BLOCK-----
After verifying the exported keys, it is no longer necessary to keep them in the GPG keyring. You can delete them from the keyring using the following commands:
# gpg --delete-secret-keys "My Deploy Signing Key"  
# gpg --delete-keys "My Deploy Signing Key"

4.4.3. Creating the gpgsign element

The final step for providing signing keys manually is to add a gpgsign element. You can do this by copying and pasting the content of the public and secret key files directly into the definition (see the gpgsign section in the Deploy Definition File Reference for an example), or by using include elements as in the example below.
To use include elements, first copy the public and secret key files to a location relative to the definition, e.g. to the same folder. Then add a gpgsign element as follows:
<gpgsign>
  <public><include href="RPM-GPG-KEY-mykey" parse="text"><public>
  <secret><include href="RPM-GPG-KEY-mykey-secret" parse="text"></secret>
  <passphrase>The quick brown fox jumped over the lazy dog.</passphrase>
<gpgsign>
This concludes our chapters on creating system repositories The next chapters will discuss system testing, deployment and maintenance.

Chapter 5. Automating System Testing

Deploy makes it easy to automate installation and update testing using customizable deployment templates.
This chapter covers the following:
  • Testing system installation.
  • Testing system updates.
  • Creating custom deployment templates.
  • Troubleshooting installation and update issues.
In this chapter, we will discuss testing installation and updates using virtual machines running on the Deploy build machine. In doing so, we assume a basic level of knowledge about using and managing libvirt-based virtual machines on CentOS and Red Hat Enterprise Linux systems. To learn more about virtualization, see the Virtualization Getting Started Guide, Virtualization Host Configuration and Guest Installation Guide and Virtualization Administration Guide included with the Red Hat Enterprise Linux documentation.
Before using the instructions in this section, you will need to be running Deploy on a KVM-compatible machine. You will also need to install the deploy-virt-config RPM package. See Section 3.1, “Configuring a Build Machine” for information on determining whether your machine is KVM-compatible. See the Deploy download page at http://www.deployproject.org/download for information on installing the deploy-virt-config package.

5.1. Testing System Installation

Deploy enables installation testing using the test-install element. As usual, the easiest way to complete this element is using a template. The example below shows testing installation using the libvirt/deploy.xml template.
<test-install enabled='true'>
  <macro id='test-install-post-delete>false</macro>
  <include href="%{templates-dir}/%{norm-os}/libvirt/deploy.xml" xpath="./*"/>
</test-install>
When the test-install element shown above is present, Deploy performs the following tasks:
  • Creates a test-install repository in the test-install folder at /var/www/html/deploy/systems/test-install, by default.
  • Uses scripts in the deploy.xml template to install a new virtual machine running on the local system. In our web-server example, the hostname will be web-server-centos-6-x86-64-test-install. You can connect to the virtual machine using SSH as follows:
    ssh web-server-centos-6-x86-64-test-install
  • Verifies the client machine installation, halting process if an error occurs. This allows errors to be resolved prior to publishing changes to the system repository. See the Section 5.4, “Troubleshooting” section for help resolving errors.
During subsequent build cycles, Deploy completes the following:
  • Synchronizes any repository changes to the test-install location.
  • Determines whether changes have occurred that require retesting installation.
  • If so, it deletes the existing machine, and completes the installation and verification steps listed above.
Once development is complete and the system is stable, you may wish to change the value of the test-install-post-delete macro to true. This will cause the virtual machine to be deleted once testing is complete, freeing up system resources.

5.2. Testing System Updates

Deploy enables update testing using the test-update element. As with test-install, testing is most easily accomplished using a template.
<test-update>
  <macro id='test-update-poweroff'>false</macro>
  <include href="%{templates-dir}/%{norm-os}/libvirt/deploy.xml" xpath="./*"/>
</test-update>
When the test-update element shown above is present, Deploy performs the following tasks:
  • Creates a test-update repository in the test-update folder at /var/www/html/deploy/systems/test-update, by default.
  • During the first build cycle after the test-update element has been added, uses scripts in the deploy.xml template to create a new machine to use as a baseline for future update testing. In our web-server example, the hostname will be web-server-centos-6-x86-64-test-update. You can connect to the virtual machine using SSH as follows:
    ssh web-server-centos-6-x86-64-test-update
  • During subsequent runs, updates packages on the test machine to conform to packages in the system repository.
  • Verifies client machine updates, halting process if an error occurs. This allows errors to be resolved prior to publishing changes to the system repository. See the Section 5.4, “Troubleshooting” section for help resolving errors.
To maintain a consistent testing environment, Deploy does not automatically delete and reinstall test-update machines. Instead, if an error occurs, Deploy will indicate that a machine already exists. Users will have to correct the error, or delete the test-update system manually before Deploy can continue.
Once development is complete and the system is stable, you may wish to change the value of the test-update-poweroff macro to true. This will cause the virtual machine to be powered down once testing is complete, freeing up system resources.

5.3. Customizing Deployment Templates

The Deploy model for automated deployment is simple, yet flexible and extensible. Deployment can occur in a variety of operating environments, for example using virtual machines running on a local host, virtual machines running on remote or cloud-based hosts, physical machines managed by enterprise systems management systems, or unmanaged physical machines such as consumer desktops.
The model includes three deployment cases, test-install, test-update and publish. The first two cases were introduced previously in Section 5.1, “Testing System Installation” and Section 5.2, “Testing System Updates”. The publish case will be introduced later under Chapter 6, Deployment Scenarios. Each case is represented by an element of the corresponding name in the definition.
Each deployment element supports a number of options, or sub-elements. These sub-elements are consistent across deployment elements, although the default behavior varies slightly between them in some instances. The sub-elements are described in detail in the chapter on Deployment Elements in the Deploy Definition File Reference (DDFR).
At a high level, deployment sub-elements serve to specify -
  • Deployment repository location, both file path and a web URL.
  • Client machine information including hostname, domain and password.
  • Installer options including boot options and a kickstart file.
  • Deployment specific config-rpms.
  • Scripts to delete, install, verify, and update client machines.
Many of these sub-elements can be reused across definitions, making them ideal candidates to be created as templates. In fact, because many of the sub-elements can be used in a mix-and-match fashion, a common practice is to save single sub-elements as individual templates. These individual templates can then be included directly in definitions, or nested within composite templates.
Deploy includes a number of deployment templates in the templates folder. In the previous section, we discussed one of these, the libvirt deploy.xml template.
This section will cover deployment templates in greater detail. It will also describe how you can customize deployment templates to meet the needs of individual deployment scenarios.

5.3.1. Exploring deployment templates

The libvirt deployment template deploy.xml provides an example for how Deploy can be used to deploy and maintain client systems. The deploy.xml template providing settings and scripts to accomplish the following:
  • Specify a domain, local, for client machines (domain).
  • Automate operating system installation using a kickstart file (kickstart).
  • Configure client machines to allow SSH access from the Deploy build machine (ssh-config.xml).
  • Determine whether the client machine should be reinstalled (test-triggers.xml).
  • Activate, delete, and install a virtual machine running on the Deploy build machine (virt-activate.xml, virt-delete.xml and virt-install.xml).
  • Verify the client machine installation (verify-install.xml).
  • Save information about client installation for future use (save-triggers.xml).
  • Update the client machine (update.xml).
  • Determine whether a new kernel has been installed and restart the system as necessary (check-kernel.xml).
The deploy.xml template, shown below, has very little content of its own and is largely composed by including sub-templates:
<?xml version="1.0" encoding="utf-8"?>
<xml>
<!-- install and update a libvirt vm on the local machine -->
<domain>local</domain>
<kickstart>
<include href='%{templates-dir}/%{norm-os}/common/ks.xml'
         xpath='./node()'/>
<include href='%{templates-dir}/%{norm-os}/libvirt/ks-post-ssh.xml'
         xpath='./node()'/>
</kickstart>
<include href='%{templates-dir}/%{norm-os}/common/acpid-config.xml'/>
<include href='%{templates-dir}/%{norm-os}/common/ssh-config.xml'/>
<include href='%{templates-dir}/%{norm-os}/libvirt/libvirt-guest-config.xml'
         xpath='./*'/>
<include href='%{templates-dir}/%{norm-os}/libvirt/verify-kvm.xml'/>
<include href='%{templates-dir}/%{norm-os}/libvirt/create-guestname.xml'/>
<include href='%{templates-dir}/%{norm-os}/libvirt/test-exists.xml'/>
<include href='%{templates-dir}/%{norm-os}/libvirt/activate.xml'/>
<include href='%{templates-dir}/%{norm-os}/common/connect.xml'/>
<include href='%{templates-dir}/%{norm-os}/common/ssh-write-host-key.xml'/>
<include href='%{templates-dir}/%{norm-os}/common/test-triggers.xml'/>
<include href='%{templates-dir}/%{norm-os}/libvirt/delete.xml'/>
<include href='%{templates-dir}/%{norm-os}/common/ssh-delete-host-key.xml'/>
<include href='%{templates-dir}/%{norm-os}/libvirt/install.xml'/>
<include href='%{templates-dir}/%{norm-os}/common/verify-install.xml'/>
<include href='%{templates-dir}/%{norm-os}/common/save-triggers.xml'/>
<include href='%{templates-dir}/%{norm-os}/common/update-yum-conf.xml'/>
<include href='%{templates-dir}/%{norm-os}/common/update.xml'/>
<include href="%{templates-dir}/%{norm-os}/common/check-kernel.xml"/>
<include href="%{templates-dir}/%{norm-os}/common/conditional-test-remove-custom-pkgs.xml"/>
<include href="%{templates-dir}/%{norm-os}/libvirt/conditional-poweroff.xml"/>
<include href="%{templates-dir}/%{norm-os}/libvirt/conditional-post-delete.xml"/>
</xml>
The sections below discuss how you can reuse content from the deploy.xml template and sub-templates, or provide alternate templates, to accommodate additional deployment scenarios.

5.3.2. Providing a custom kickstart

Kickstart files let you automate operating system installation by providing answers to language, network configuration, keyboard, file system configuration, mouse, timezone, root password and other selections.
The libvirt deployment template uses ks.xml and ks-post-ssh.xml templates. The first provides basic kickstart options. The second configures the client for ssh access. Contents of these templates are shown below.
ks.xml
<?xml version="1.0" encoding="utf-8"?>
<kickstart>
<macro id='timezone'>America/Vancouver</macro>
<macro id='kickstart-text'>text</macro>
install
%{kickstart-text}
network --device=link --bootproto=dhcp --hostname %{hostname} --activate
rootpw --iscrypted "%{crypt-password}"
lang en_US
keyboard us
zerombr
bootloader --location=mbr
clearpart --all --initlabel
part / --fstype ext4 --size 1 --grow --asprimary
part swap --recommended
part /boot --fstype ext3 --size 512 --asprimary
timezone --utc %{timezone}
auth --enablemd5 --enableshadow --enablecache
selinux --enforcing
reboot

%packages --nobase
@core
%end
</kickstart>
ks-post-ssh.xml
<?xml version="1.0" encoding="utf-8"?>
<xml>
%post
# allow build machine to access client via ssh
# set some convenience variables
sshdir=/root/.ssh
authkeys=$sshdir/authorized_keys
pubkey="%{build-host-pubkey}"

# make sshdir if it doesn't exist
[[ -d $sshdir ]] || mkdir $sshdir 

# create authorized_keys file if it doesn't exist
[[ -f $authkeys ]] || touch $authkeys

# copy pubkey into authorized keys if it's not there already
[[ `cat $authkeys` == $pubkey ]] || echo $pubkey >> $authkeys
  
# set permissions
chmod 700 $sshdir
chmod 600 $authkeys
[[ -f /sbin/restorecon ]] &amp;&amp; /sbin/restorecon -r $sshdir
%end
</xml>

The two files can be combined within a single kickstart element as follows:
<kickstart>
  <include href="%{templates-dir}/%{norm-os/common/ks.xml" xpath="./node()"/>
  <include href="%{templates-dir}/%{norm-os}/libvirt/ks-post-ssh.xml" xpath="./node()"/>
<kickstart>
The default ks.xml file uses macros (see the DDFR for information on deployment macros) to allow the hostname and root password to be filled in at Deploy runtime. Often, however, you will want to customize additional kickstart options such as timezone, or drive partitioning configuration. To do this, you will need to provide a custom kickstart element. This can be accomplished using several different methods. Perhaps the easiest method, however, is shown in the example test-install element below.
In this example, you start with a test-install element, add a password element and include the libvirt deployment template. The include element varies from ones you have used previously, however, in that it contains an xpointer attribute. Xpointer is used to include all elements from the file, except elements named kickstart. Next, you add a kickstart element. This element contains kickstart content to suit your needs; in the example below only the timezone has been changed. Finally, to the kickstart element you also include content from the ks-post-ssh.xml template.
<test-install>
<include href="%{templates-dir}/%{norm-os}/libvirt/deploy.xml" 
      xpath="./*[name()!='kickstart']"/>
<kickstart>
network --hostname %{fqdn}
rootpw --iscrypted "%{crypt-password}"
lang en_US
zerombr
bootloader --location=mbr
clearpart --all --initlabel
part / --fstype ext4 --size 1 --grow --asprimary
part swap --recommended
part /boot --fstype ext3 --size 512 --asprimary
timezone America/Denver  #change timezone to Denver
auth --enablemd5 --enableshadow --enablecache
selinux --enforcing
reboot

%packages --nobase
@core
%end

<include href="%{templates-dir}/%{norm-os}/libvirt/ks-post-ssh.xml" xpath="./node()"/>
</kickstart>
</test-install>
See the DDFR for more information on the kickstart element, including notes on several kickstart options to avoid when using kickstart in combination with Deploy.

5.3.3. Providing custom scripts

Deploy allows users to provide scripts for completing various deployment and maintenance actions. These scripts can be written in any scripting language.
The following example shows -
  • Providing a custom install script. In this case we are setting the virtual machine RAM at 2GB and the disk image size at 40GB.
  • Including all elements from the libvirt deployment template, except the install script.
<test-install>
<script id='install' type='install'>
#!/bin/bash
virt-install --name %{fqdn} --ram 2000 \
             --file /var/lib/libvirt/images/%{fqdn}.img \
             --file-size 40 \
             --location %{webroot} \
             --extra-args "%{boot-options} ks=%{webroot}/ks.cfg"\
             --noreboot
</script>
<include href="%{templates-dir}/%{norm-os}/libvirt/deploy.xml" 
            xpath="./*[@id!='virt-install']"/>
</test-install>
See the DDFR for complete information on deployment scripts, including the comes-before and comes-after attributes which allow ordering of script execution.

5.3.4. Installing a virtual machine to a remote host

The libvirt deployment templates makes it easy to create, deploy and maintain virtual machines on the local machine running a libvirt compatible hypervisor (KVM). Your particular deployment scenario, however, may require you to work with virtual machines running on remote hosts with different hypervisors (e.g. VMware, Hyper-V and Amazon Web Services). To support these cases, you will need to create new templates. These templates will be similar to the libvirt deployment templates.
See the windows-azure and openstack-deploy folders in the templates folder for examples.

5.4. Troubleshooting

As you work with Deploy to create deployment templates and config-rpms, you will occasionally run into errors. When these occur, Deploy halts operation and displays error information to the console. It also records error messages in the Deploy log file, by default /var/log/deploy.log.
In addition to displaying and logging error messages, Deploy provides assistance for troubleshooting errors that occur during client-side script processing. Client-side scripts include some deployment scripts and config-rpm scripts and triggers.

5.4.1. Troubleshooting errors in client-side deployment scripts

When Deploy executes a deployment script using SSH, it first copies the script to the client machine and then executes it from there. To assist with troubleshooting, Deploy leaves copied scripts on the client machine. They can be located in the folder at /var/lib/deploy/deploy.
Each script exists in a separate file, with a filename corresponding to the script's @id attribute. By executing the script from command line on the client machine, you will be able to reproduce and troubleshoot errors.

Note

You can view virtual clients created by the virt-deploy template by typing virt-manager at the command line of the Deploy build machine.

5.4.2. Troubleshooting RPM scriptlet errors

By default, Deploy halts processing when it detects scriptlet failed errors during RPM package installation on client machines. This error detection occurs for all RPM packages, both Deploy-created packages (config-rpms) and packages created outside of Deploy. Also, error detection occurs during both initial system install and ongoing system updates.
Error detection is accomplished using two deployment scripts, verify-install.xml and update.xml. These scripts are located within the common subfolder of the templates folder. They, or custom variations based on them, are generally recommended for inclusion within all deployment templates.
When scriptlet failed errors do occur, as will be somewhat common while you are developing new definitions, the following steps will allow you to save time as you troubleshoot and resolve errors:
  1. If you have not previously created a test-update machine, add a test-update element to your definition. See Section 5.2, “Testing System Updates” for more information.
  2. Ensure the sync plugin is enabled, as it is by default. The sync plugin is helpful during troubleshooting. See Section 4.1, “Building a Release Package” for more information.
  3. Comment out test-install and publish elements, if they exist in your definition. These take time and slow down troubleshooting. You can add them back after errors are resolved.
  4. Comment out the package or packages with failed scriptlet. Generally you will do this by commenting out config-rpm elements, but you may also need to comment out package or requires elements. The goal is to complete an error-free test-update machine installation, from which you can add back troublesome packages as you resolve issues.
  5. Run Deploy to create the test-update machine. Iterate through this step and the previous until you complete an error-free system installation or update.
  6. Uncomment a failed package (see step 4 above).
  7. Run Deploy again.
  8. SSH to the test-update system, for example:
    ssh web-server-centos-6-x86-64-test-update
  9. Run "yum sync"
  10. If the scriptlet failed error is related to a Deploy-created config-rpm, you can locate the problem script in the /var/lib/deploy/config folder, in a subfolder named after the config-rpm, e.g. httpd-config. Within the config-rpm subfolder, files are named by script type, e.g. post-script and postun-script. Trigger scripts are named by trigger type, followed by the name of the trigger, e.g. triggerin-drupal-script.
    Errors in trigger scripts can be more difficult to troubleshoot. You will need to search folders in the /var/lib/deploy/config folder to locate which config-rpm, if any, provided the errant trigger script.
    Once you have located the error-producing script, you can execute it from the command line by typing
    ./script_name
    This will allow you to reproduce and troubleshoot the error, ultimately updating the original config-rpm script element.
    If the script that produces the scriptlet failed error is provided by a non config-rpm script, the process is more complex. From the test-update machine, you can list scripts for an rpm by typing the following command:
    rpm -q --scripts package_name
    This lists all scripts for the package. From there you will need to separate out the relevant script and save as an individual file to perform testing. Updates will need to be applied to the original rpm specfile.
  11. Rerun deploy and yum sync until package updates complete successfully. Repeat for additional failed packages, starting at step 6.
  12. Reverse changes from steps 1-3, as desired.

Chapter 6. Deployment Scenarios

Because Deploy creates system repositories in a standard format, they can be used in a wide range of deployment scenarios. This chapter builds on techniques described in previous chapters to discuss client deployment in consumer/OEM, small business/IT Consultant, enterprise, hosted, and even web farm and massively scaled environments.
Examples in this chapter show a variety of uses for the publish element. Complete information on the publish element is available from the chapter on Deployment Elements in the Deploy Definition File Reference (DDFR).

6.1. Consumer/OEM

In consumer scenarios, system deployment is typically managed by Original Equipment Manufacturers (OEMs). In these scenarios, OEMs preinstall client systems with a default set of software and settings, and end users have complete flexibility to add and remove software as desired. Client maintenance is performed by the end user, or by an automated process running on behalf of the end user. The end result is that YUM checks, downloads and installs updates from a number of package repositories, many of which are maintained by 3rd parties.
Deploy can be used in consumer deployment scenarios in two ways. First, by automating the creation and maintenance of package repositories for distributing hardware drivers and other OEM provided software and configuration. This will be discussed further in Chapter 7, Package Repositories. Second, by automating client system deployment. The latter is discussed below.

6.1.1. Consumer repository definitions

For consumer scenarios, repository definitions are used primarily to assemble packages needed for the system (the packages themselves will exist in previously created repositories), and to automate deployment to the client.
In general, consumer repository definitions should not create RPM packages. A consumer device such as a desktop or laptop has dozens of configuration options - from operating system, to applications, to hardware drivers and software - and maintaining repositories to represent the combination of all options will likely be undesirable.
Instead, consumer system repositories should be thought of as temporal mechanisms for automating client installation, potentially one client at a time.
Key aspects of consumer system repository definitions include the following:
  • Disabled release-rpm element. Disabling the release-rpm prevents installed client systems from attempting to connect to the system repository (which may no longer exist) during YUM updates.
  • No config-rpm or srpmbuild elements. Again, since the system repository is temporal, all RPMs should be provided by stand-alone package repositories.
  • Optional test-install and test-update elements. In general, these elements are not recommended. It is preferable to perform testing at the package repository level to avoid overhead at the time of client installation.
  • Publish element with a custom kickstart and one or more scripts to automate client installation.
The following example shows a skeleton definition for a consumer system repository. The firstboot package along with the firstboot --reconfig kickstart option allow consumers to reconfigure the system during first boot.
Consumer system repository definition
<?xml version="1.0" encoding="utf-8"?>
<definition>

<main>...</main>

<repos>...</repos>

<packages>
...
<package>firstboot</package>
</packages>

<release-rpm enabled='false'/>

<publish>
<kickstart>
...
firstboot --reconfig
</kickstart>

<script id='install' type='install'>
# custom script to install the repository to a client machine
</script>
</publish>

</repo>

6.1.2. Automating consumer repository creation

Manually creating a repository for each consumer system is time consuming and potentially error prone. This may be sufficient for a low volume of systems. For a higher volume of systems, OEMs can integrate the creation of repository definitions with the system selection process. Because definitions are in standard XML format, they can be generated using a number of programming languages. Once created, they can be verified, and the repositories created, by executing the Deploy application programmatically.

6.1.3. Replicated deployment

For high-volume OEM scenarios where the same software and configuration will be installed across multiple client systems, an image copy-based approach will be preferable to the per-machine installation approach described above. See Section 6.5, “Replicated Clients/Scale-Out” later in this chapter for more information.

6.2. Small Business/IT Consultant

Small business system deployment is often managed by IT consultants who service multiple clients. These consultants work remotely when possible, and visit the customer location only as needed. Using Deploy, IT Consultants can perform the tasks of system integration and testing remotely. Physical installation, which involves burning a CD/DVD and using it to boot the physical machine, can be performed by a tech-savvy individual within the small business, or by the IT consultant during an on-site visit.
To enable this scenario, the IT Consultant must first establish a web server that customers will use to access system repositories. This can be a web server at the customer location, a hosted web server, or an internet-accessible web server maintained by the IT Consultant. The web server can be running on any operating system. The only requirements for the web server are that:
  • customer machines can access it over HTTP or HTTPS,
  • the IT Consultant has file system access to the server, for example via SSH, and
  • permissions are set so that only the customer and the consultant have access to customer repositories.
For this scenario, we assume that the final production machines (e.g. laptops, desktops or servers) will be installed manually via CD/DVD. In small environments, manual deployment is often more practical than setting up the infrastructure required to support automated deployment. Installation can be performed by the IT Consultant or a tech-savvy individual within the small business. Maintenance will occur automatically using YUM.
We assume for this scenario that production machines are strictly managed, specifically that all applications and configuration on the machines are controlled by the IT Consultant rather than by end users. Some users may require a greater degree of control over their machines, in which case IT consultants can accommodate these users following a strategy similar to that described under Section 6.1, “Consumer/OEM”.
Key aspects of small business system repository definitions include the following:
  • Use of the config-rpm element to configure clients to periodically check the web server for updates. In the example below, this is done by creating a config-rpm with the ID yum-sync-cron. This config-rpm configures the client to run the yum sync command on a daily basis using the cron utility. It does this by installing a file named yum-sync to the /etc/cron.daily folder, and setting the file mode to executable.
  • Use of the test-install and test-update elements to perform ongoing testing prior to making the repository available for use in the customer environment.
  • Use of the remote-url element within the publish element to specify the web server location of the production repository. This is the location where customers/consultants can access system installation images for creating CD/DVD discs. It is also the location where client machines will access updates. In the example below, the remote-url is specified as http://www.company.com/deploy/systems/.
  • Use of the password element within the publish element to specify the root user password for the production client. If the same repository will be used to deploy multiple clients, the password element can be omitted and a unique, per-machine password can be provided manually at install time.
  • Use of the kickstart element within the publish element to automate CD/DVD-based system installation. If the same repository will be used to deploy multiple client machines, the network option should be omitted from the kickstart to avoid machine hostname collision. In this case the hostname can be provided manually during the CD/DVD installation process. Also, the password option may be omitted and the password provided at install time as described above.
  • Use of the script element within the publish element to automate synchronization of the repository from the Deploy build machine to the web server location specified by the remote-url. This synchronization will only occur if the test-install and test-update processes have completed without error.
    In the example below, this is done by providing a script with the id 'sync'. The type attribute for the script is set to 'update', which will cause Deploy to run the script each time changes occur in repository content. The ssh attribute is set to false, directing Deploy to execute the script on the build machine, rather than on a remote client. The script synchronizes the published repository on the build machine with the customer-accessible repository specified by remote-url.
    The script assumes that SSH public key authentication has been previously configured between the build machine and the remote server. See the Deployment Guide for Red Hat Enterprise Linux at the Red Hat Documentation website http://docs.redhat.com for information on configuring key-based authentication for OpenSSH.
The following example shows a skeleton definition for a small business system repository.
Small business system repository definition
<?xml version="1.0" encoding="utf-8"?>
<definition>

<main>...</main>

<repos>...</repos>

<packages>...</packages>

<config-rpms>
<config-rpm id='yum-sync-cron'>
<files content='text' destdir='/etc/cron.daily' destname='yum-sync' mode='755'/>
#!/bin/sh
/usr/bin/yum -q -y sync
</files>
</config-rpm>
...
</config-rpms>

<test-install>
<password>test-password<password>
<include href="%{templates-dir}/%{norm-os}/libvirt/deploy.xml" xpath="./*"/>
</test-install>

<test-update>
<password>test-password<password>
<include href="%{templates-dir}/%{norm-os}/libvirt/deploy.xml" xpath="./*"/>
</test-update>

<publish>
<web-url>http://remote.company.com/deploy/systems/<web-url>

<password>production-password<password>

<kickstart>
<?xml version="1.0" encoding="utf-8"?>
<kickstart>
<macro id='timezone'>America/Vancouver</macro>
<macro id='kickstart-text'>text</macro>
install
%{kickstart-text}
network --device=link --bootproto=dhcp --hostname %{hostname} --activate
rootpw --iscrypted "%{crypt-password}"
lang en_US
keyboard us
zerombr
bootloader --location=mbr
clearpart --all --initlabel
part / --fstype ext4 --size 1 --grow --asprimary
part swap --recommended
part /boot --fstype ext3 --size 512 --asprimary
timezone --utc %{timezone}
auth --enablemd5 --enableshadow --enablecache
selinux --enforcing
reboot

%packages --nobase
@core
%end
</kickstart>

	
</kickstart>

<script id='sync-to-production-server' type='update' ssh='false'>
#!/bin/sh
rsync -a --delete -e ssh \
"/var/www/html/deploy/systems/%{id}/" \
"user@www.company.com:/var/www/html/deploy/systems/%{id}"
</script>
</publish>

</definition>

6.2.1. Deploying Clients Manually

Because Deploy creates system repositories in a standard format, clients can be deployed manually using traditional tools and methods. For the small business scenario, we will discuss installation using CD/DVD boot media. See the Red Hat Enterprise Linux Installation Guide and Red Hat Enterprise Linux Virtualization Guide for information on additional installation methods.

6.2.1.1. CD/DVD Network Installation

In this method, the consultant/customer will use the network install image (boot.iso) to create an installation CD or DVD. The image can be found in the images folder for the repository, e.g. /var/www/html/deploy/systems/web-server-centos-6-x86_64/images/.
The first step is to download the boot.iso file to a local machine, and then use CD writing software to burn it to CD/DVD media. Depending on the operating system of the local machine, this may be as simple as right clicking the file and choosing "Burn disc image" or "Write to Disk". On older Windows operating systems, a separate application may be needed to provide ISO image burning capability.
Once CD or DVD disk is created, the next step is to place it into the CD/DVD drive of the client to be installed, and restart the machine, directing the machine to boot from the CD/DVD device if necessary. The boot process will load the installer which will complete the installation, retrieving RPM packages over the network from the repository URL. If the hostname and root password are missing from the kickstart file, the installer will prompt for them at the beginning of installation.

6.2.1.2. CD/DVD Complete Installation

Deploy can also create complete CD/CVD install media. In this case, the installer program will retrieve RPM packages directly from the CD/DVD media, rather than downloading them over the network. This method is convenient for installing clients with limited network connectivity.
To direct Deploy to create complete CD/DVD installation images, add an iso element to the definition, as follows:
<iso>
<set>DVD</set>
</iso>
See the Deploy Definition File Reference (DDFR) for the complete syntax of the iso element.
After running Deploy again, the iso image (or images) will be available from the iso folder of the repository, e.g. /var/www/html/deploy/systems/web-server-centos-6-x86_64/iso/. Burn the resulting image(s) to media, and use them to boot client machines as described under Section 6.2.1.1, “CD/DVD Network Installation” above.

6.3. Enterprise

Enterprise system deployment is typically managed by in-house system administrators, often with the assistance of deployment automation infrastructure such as network installation servers or more highly integrated systems/operations management solutions. Enterprise system administrators can use Deploy to perform system integration, testing and maintenance, while using existing infrastructure to automate system deployment.
Two primary strategies can be followed for integrating Deploy repositories with enterprise deployment management systems. One of those is to use Deploy repositories for performing Preboot Execution Environment (PXE)-based system installations. PXE is an efficient deployment infrastructure, supported by most enterprise management solutions. The other strategy is to use Deploy to generate system images in a format supported by the deployment system. PXE-based deployment will be discussed in this chapter. System image-based deployment will be covered in Section 6.4, “Virtual/Hosted”.
PXE deployment is recommended over system image deployment, if supported by your management system, as it eliminates the added step of creating and maintaining system images. See the documentation for your particular management solution to learn if it provides support for PXE.
Deploy enables PXE deployment by creating system repositories in a standard operating system distribution format known as an installation tree. An installation tree is a folder in the file system that contains all the needed files and subfolders to perform a complete system installation.

6.3.1. PXE-based System Deployment

Because Deploy system repositories are created in a standard installation tree format, they can be used with any deployment system that supports the Preboot Execution Environment (PXE) protocol.
If you organization does not have an existing systems management solution, you can implement a simple PXE-based deployment system using components available in the default CentOS and Red Hat Enterprise Linux server operating systems. See the advanced topics section of the Red Hat Enterprise Linux Installation Guide for more information.
To integrate your system repositories with a PXE-based deployment system you will first create repositories, and then configure the PXE service to expose those repositories.

6.3.1.1. Create repositories

To enable this scenario, system administrators must first establish a web server for deploying system repositories. Similar to the small business scenario described in the previous section, the web server can be running on any operating system - the only requirements being that 1) client systems can access it over HTTP/HTTPS and 2) the Deploy build machine can copy/synchronize files to its underlying file system.
For this scenario, we assume that the final production machines (e.g. laptops, desktops or servers) will be installed using PXE-capable deployment systems. As with the small business scenario, maintenance will occur automatically using YUM.
Again as in the small business scenario, we assume that production machines are strictly managed, specifically that all applications and configuration on the machines are controlled by the system administrator rather than by end users. Some users may require a greater degree of control over their machines, in which case system administrators can accommodate these users following a strategy similar to that described under Section 6.1, “Consumer/OEM”.
In fact, the major difference between this scenario and the small business scenario is the additional use of PXE deployment infrastructure. All other aspects of the small business scenario, including the ability to manually install systems via CD/DVD, still apply.
See Section 6.2, “Small Business/IT Consultant” for an overview and example skeleton system repository definition that applies to both small business and enterprise.

Note

For enterprise scenarios, you will typically omit the network --hostname option from the kickstart file for the repository, as the hostname will be provided by DHCP.

6.3.1.2. Configure the PXE service to expose repositories

Instructions for PXE boot configuration vary across deployment management solutions. In this section, we will discuss PXE boot configuration generally. Consult the documentation for your deployment management solution for details.
In general, configuring your PXE service will involve specifying the location for three files, the kernel and initrd images, and the kickstart file.
You may need to copy the kernel and initrd image files from the production web server to the PXE boot server. This can be accomplished using additional elements in the definition as shown below. These additions will cause the kernel and initrd images to be copied to the PXE server each time Deploy updates production images.
<publish>
...
<triggers>treeinfo<triggers>
<script id='sync-to-pxe-server' type='install'/>
#!/bin/sh
rsync -a --delete -e ssh \
"/var/www/html/deploy/systems/%{id}/images/pxeboot" \
"user@pxe.company.com:/pxe/file/location/%{id}"
</script>
<publish>
Once the kernel and initrd images are available to the PXE server, you configure the PXE service to install client machines using these files. Again, configuration varies from system to system, but in general you should specify the boot kernel as the path to the vmlinuz file copied in the script above. In addition, you should specify at least two kernel boot arguments. These arguments will be the paths to the initrd image and the kickstart file. For example -
kernel /path/to/vmlinuz/on/pxeboot/server
append initrd=/path/to/initrd.img/on/pxeboot/server ks=http:/www.company.com/deploy/systems/web-server-centos-6-x86_64/ks.cfg ...
In addition to the initrd and ks arguments, a number of additional boot arguments, also known as boot options, can be specified. These boot options are described in the Advanced Installation options section of the Red Hat Enterprise Linux Installation Guide.

6.4. Virtual/Hosted

Virtual machine and hosted system deployment is another scenario supported by Deploy. This scenario covers deployment to a range of hypervisor and hosted solutions including VMWare, Hyper-V, KVM, Xen and Amazon Web Services. Some virtual machine management solutions support PXE-based deployment. If so, PXE deployment is recommended. See the section on Section 6.3, “Enterprise” for information on PXE-based deployment.
If your virtual machine management solution does not support PXE-based deployment, another option is to use Deploy to create virtual machine images in the desired format. These images can then be deployed to the production environment.
To use Deploy for creating virtual machine images, you start with a definition as shown in the Section 6.2, “Small Business/IT Consultant” scenario. To this definition you provide an additional update-type script to create the virtual machine image. Deploy will execute the script each time the repository is updated, ensuring that the installation image is always current. See the script element below for an example.
Example
The example below shows providing a script to create a virtual machine image. In this example, we reuse content from the virt-install.xml template to create a machine that will run under a KVM hypervisor. Other hypervisor systems will provide similar utilities for creating virtual machine images. See the documentation for your hypervisor for additional information. We then copy the created image to a production location where it can be accessed by the virtual machine management solution.
<publish>
...
<script id='create-virtual-machine-image' type='update' ssh='false'>
# reuse content from the virt-install.xml template to create the vm image
<include href='%{templates-dir}/%{norm-os}/deploy/deploy.xml'
         xpath='/script/text()'/>

# copy the created image to a production location
rsync -a --delete -e ssh \
"/var/lib/libvirt/images/%{fqdn}.img" \
"user@www.company.com:/vm_images"
</script>
</publish>

</repo>

6.5. Replicated Clients/Scale-Out

Deploy also supports replicated client and scale-out deployments. In this scenario, many client machines share duplicate software and configuration. For these environments, traditional client installation, which can take 5-10 minutes per client, is inefficient. Instead, it is preferable to make a single machine image which can be applied to client hard drives using an efficient low-level disk copy process.
Deploy supports this scenario in the same manner as the Section 6.4, “Virtual/Hosted” scenario. In particular, you will start with a definition similar to that shown in the Section 6.2, “Small Business/IT Consultant” scenario. To that definition you will add a publish/script element with a @type attribute of 'install' and a @ssh attribute of 'false'. The script will first create the machine image. Then it either copy the image to a location where it can be used in the production deployment process, or more likely, send an email notification that a new image is available. The latter case allows greater control over changes in the production environment. The details of the script will vary based on requirements of the production deployment system.

6.6. Configuration Management Systems

A final scenario supported by Deploy is integration with configuration management systems such as Puppet. In this scenario, system administrators use Deploy for initial client deployment, and the configuration management system for ongoing client maintenance.
When creating repositories for integration with configuration management systems, you will want to do the following:
  • Disable the YUM Sync Plugin using the release-rpm element. Disabling the YUM Sync Plugin prevents clients from attempting to remove RPM packages installed by the configuration management system, but not included in the repository.
  • Add a config-rpm for configuring the configuration management system. This RPM will install any client side software and settings needed by the configuration management system.
  • Add test-install and test-update elements to perform ongoing pre-production testing.
  • Add a publish element with a custom kickstart and one or more scripts to integrate the repository with a production deployment system. See the previous sections for information on integrating Deploy repositories within various deployment environments.
Example
The following example shows a skeleton definition for integrating with a configuration management system.
<?xml version="1.0" encoding="utf-8"?>
<definition>

<main>...</main>

<repos>...</repos>

<packages>... </packages>

<release-rpm>
<updates sync='false'/>
</release-rpm>

<config-rpms>
<config-rpm id='configuration-management-system-config>
<requires>configuration-management-client</requires>
<script type='post'/>
# script to perform client configuration for the configuration management system
</script>
</config-rpm>
</config-rpms>

<publish>
<kickstart>...</kickstart>

<script ...>
# script to integrate with production deployment management system
</script>
</publish>

</definition>

Chapter 7. Package Repositories

Package repositories are typically created and maintained by Independent Software Vendors (ISVs) who distribute consumer and enterprise applications, and by systems administrators, who manage applications and configuration across multiple clients within an organization. Package repositories are useful for deploying software and configuration that is applicable across multiple clients, and that does not change from client to client.
In this chapter we will discuss defining, creating and using package repositories.

7.1. Defining Package Repositories

Defining package repositories is much like defining system repositories, as discussed in Chapter 3, Building a System Repository and Chapter 4, Building RPM Packages earlier in this document. Technically, package repositories are subsets of system repositories, containing just the packages needed to deploy application-specific or organization-specific software and configuration.
Four aspects of particular relevance when creating package repository definitions are as follows:
  • Setting the repository type
  • Identifying external packages to include
  • Providing GPG signing keys
  • Using configuration templates

7.1.1. Setting the repository type

The first step for creating a package repository is to add a type element to the main top-level element, as shown in the example below.
<main>
  ...
  <type>package</type>
</main>
After using Deploy to build the repository, you can locate it in the package repositories folder, /var/www/html/deploy/packages by default.

7.1.2. Including external packages

Packages in your repository will often have dependencies on packages in external repositories. As a convenience, Deploy allows you to download packages from external repositories and include them in your repository. As a result, users of your repository will have fewer repositories to track and manage. You can include packages from external repositories using the repo/download element. By default, the value of this element is 'false'.
In some cases, you will not want to include packages from external repositories:
  • Distribution rights. You may not have the rights to distribute some external packages.
  • Operating system independence. You may want to create a single package repository that is applicable to users of CentOS or Red Hat Enterprise Linux, rather than maintaining separate repositories for each.
For more information on the download element, and the repos element in general, see the Repos section in the Deploy Definition File Reference (DDFR)

7.1.3. Providing GPG signing keys

Deploy automatically signs packages it creates using GPG signing keys. When users of your repository first install created packages, they will be asked to confirm installation of the repository's public key given summary information. To provide your users with specific, recognizable information on which to base their decision, you will want to manually create GPG signing keys for use with your package repositories.
See Section 4.4, “Providing GPG Signing Keys” for complete information on creating keys and using them within your definitions.

7.1.4. Using configuration templates

In some cases you will use package repositories to distribute software for external use. This is a common scenario for ISV applications, and applies to community developed and internal applications as well.
Often such software will support multiple possible configurations. You can use configuration templates to define and test common configurations. You can distribute these templates along with your repository, making it easy for external users to customize and deploy your software in their own environments.
In the example below you will create and use configuration templates for a small repository containing four packages. The first package contains application software. We will refer to this package as application. The remaining three packages contain configuration for three alternative application scenarios. The package names are scenario1, scenario2 and scenario3 respectively.
Your definition will consist of four files in the file system as shown below:
application.definition
templates/
  scenario1-config.xml
  scenario2-config.xml
  scenario3-config.xml
The application.definition file defines the repository, creates the application package, includes the config templates, and copies the templates folder to the repository tree.
application.definition
<?xml version="1.0" encoding="utf-8"?>
<definition>
<main>
<name>application</name>
<arch>x86_64</arch>
<version>6</version>
<type>package</type>
</main>

<srpmbuild id='application'>
<script>
# script to create the application package
</script>
</srpmbuild>

<config-rpms>
<include href="templates/scenario1-config.xml"/>
<include href="templates/scenario2-config.xml"/>
<include href="templates/scenario3-config.xml"/>
</config-rpms>

<files>
<files>templates</files>
</files>
</definition>
The template files define each of the three config-rpms for the repository, for example:
scenario1-config.xml
<?xml version="1.0" encoding="utf-8"?>
<config-rpm id='scenario1-config'>
...
</config-rpm>
After using Deploy to create your repository, you can locate it at /var/www/html/deploy/packages/application-6-x86_64. The tree structure of the repository will look as follows:
gpgkeys/
Packages/
repodata/
templates/
The templates folder contains copies of the configuration templates. Customers can use these as starting points for further customizing and deploying your application within their organizations.

7.2. Package Repository Types

Deploy supports several common package repository types:
  • Configuration Repositories - used for distributing organization-specific operating system or application configuration.
  • Patch Repositories - used for distributing patched software or configuration, typically within an organization.
  • Aggregation Repositories - used for aggregating packages from multiple repositories into a single convenience repository.
  • Application Repositories - used for distributing application software and default configuration.
These types are not mutually exclusive, in fact they often overlap. In this section, we provide skeleton definitions for each package repository type.

7.2.1. Configuration Repositories

Configuration repositories are used by enterprise systems administrators for distributing enterprise-specific configuration, such as client authentication (Kerberos/LDAP, etc.) and network (DHCP/DNS) configuration.
The skeleton template below shows elements common to configuration repositories.
Skeleton configuration repository definition
<?xml version="1.0" encoding="utf-8"?>
<definition>

<main>
<name>config-repo</name>
<os>centos</os>
<version>6</version>
<arch>x86_64</arch>
<type>package</type>
</main>

<gpgsign>
<!-- gpgsign elements -->
</gpgsign>

<config-rpms>
<!-- config-rpm definitions -->
</config-rpms>

</definition>

7.2.2. Patch Repositories

Patch repositories are used by ISVs or enterprise IT departments for distributing modified software applications and default configuration.
The skeleton template below shows elements common to patch repositories.
Skeleton patch repository definition
<?xml version="1.0" encoding="utf-8"?>
<definition>

<main>
<name>patch-repo</name>
<os>centos</os>
<version>6</version>
<arch>x86_64</arch>
<type>package</type>
</main>

<gpgsign>
<!-- gpgsign elements -->
</gpgsign>

<srpmbuild>
<!-- srpm definitions -->
</srpmbuild>

</definition>

7.2.3. Aggregation Repositories

Aggregation repositories are used by ISVs or enterprise IT departments for aggregating packages from multiple repositories into a single convenience repository.
The skeleton template below shows elements common to aggregation repositories.
Skeleton aggregation repository definition
<?xml version="1.0" encoding="utf-8"?>
<definition>

<main>
<name>aggregation-repo</name>
<os>centos</os>
<version>6</version>
<arch>x86_64</arch>
<type>package</type>
</main>

<repos>
<!-- repository definitions -->
</repos>

<packages>
<!-- package and group listings -->
</packages>

</definition>

7.2.4. Application Repositories

Application repositories are used by ISVs or enterprise IT departments for distributing software applications and default configuration.
Application repositories are the most complex package repository type. Often they combine elements of all the other types - configuration, patch and aggregation.
The skeleton template below shows elements common to application repositories.
Skeleton application repository definition
<?xml version="1.0" encoding="utf-8"?>
<definition>

<main>
<name>application-repo</name>
<os>centos</os>
<version>6</version>
<arch>x86_64</arch>
<type>package</type>
</main>

<gpgsign>
<!-- gpgsign elements -->
</gpgsign>

<repos>
<!-- repository definitions -->
</repos>

<packages>
<!-- package and group listings -->
</packages>

<config-rpms>
<!-- config-rpm definitions -->
</config-rpms>

<srpmbuild>
<!-- srpm build definitions -->
</srpmbuild>

<files>
<!-- file inclusion definitions -->
</files>
</definition>

7.3. Using Package Repositories

Once you have completed your package repository, customers can use it with Deploy for creating their own systems. Or they can install packages from your repository manually on existing systems. We will discuss each of these scenarios below.

7.3.1. With Deploy

Your customers can use Deploy for including content from your repository within their own system and package repositories. They can do this by including the following elements in their definitions:
  • A repo element pointing to your published repository.
  • Package elements naming packages in your repository.
  • Config-rpm elements reusing and customizing configuration templates you have made available in your repository. See Section 7.1.4, “Using configuration templates” for more information on providing configuration templates for your customers to modify.

7.3.2. On Existing Systems

Your customers can also install packages from your repository manually on existing systems using YUM. To do this, they will need to complete the following steps:
  • Download the release-rpm from your package repository. You can make this easy for your customers by providing them with a full path to the release-rpm in your final published repository, e.g. https://repos.mycompany.com://application/application-release-6-1.el6.noarch.rpm.
  • Install the release rpm using rpm. You can make this easy for your customers by directing them to execute the following command as the root user:
    rpm -Uvh path/to/downloaded/rpm
  • Install packages as desired from your repository using YUM, again as the root user, for example:
    yum install package1 package2