Quattor
Encyclopedia
Quattor is a generic open-source tool-kit used to install, configure, and manage computers. Quattor was originally developed in the framework of European Data Grid project (2001-2004). Since its first release in 2003, Quattor has been maintained and extended by a volunteer community of users and developers, primarily from the community of grid system administrators.
The Quattor tool-kit, like other configuration management systems, reduces the manpower required to maintain a cluster and facilitates reliable change management. However, three unique features make it particularly attractive for managing grid
resources:
These features are also attractive beyond the grid context. This has been confirmed by the growing adoption of Quattor, by both commercial companies and academic institutions, most of them using the tool-kit to manage consistently their grid and non-grid systems.
the years programming language designers have attacked this problem from many angles. While
trends change, the basic principles are well understood. Features such as encapsulation, abstraction,
modularity, and typing produce clear benefits. We believe that similar principles apply when sharing
configuration information across administrative domains.
The Quattor configuration tool-kit derives its architecture from LCFG
, improving it in several
aspects. At the core of Quattor is Pan, a high-level, typed language with flexible include mechanisms,
a range of data structures, and validation features familiar to modern programmers. Pan
allows collaborative administrators to build up a complex set of configuration templates describing
service types, hardware components, configuration parameters, users etc. The use of a high-level
language facilitates code reuse in a way that goes beyond cut-and-paste of configuration snippets.
The principles embodied in Quattor are in line with those established within the system administration
community. In particular, all managed nodes retrieve their configurations from
a configuration server backed by a source-control system (or systems in the case of devolved management).
This allows individual nodes to be recreated in the case of hardware failure. Quattor
handles both distributed and traditional (single-site) infrastructures.
Devolved management includes the following features: consistency over a multi-site
infrastructure, multiple management points, and the ability to accommodate the specific needs
of constituent sites. There is no single “correct” model for a devolved infrastructure, thus great
flexibility is needed in the architecture of the configuration system itself. Sometimes a set of highly autonomous
sites wish to collaborate loosely. In this case each site will host a fairly comprehensive
set of configuration servers, with common configuration information being retrieved from a shared
database and integrated with the local configuration.
Distributing the management task can potentially introduce new costs. For example, transmitting
configuration information over the WAN introduces latency and security concerns. Quattor
allows servers to be placed at appropriate locations in the infrastructure to reduce latency, and
the use of standard tools and protocols means that existing security systems (such as a public key
infrastructure) can be harnessed to encrypt and authenticate communications.
high-level configuration templates, the Pan compiler that validates templates and translates them
to XML profiles, and a machine profile repository that serves the profiles to client nodes. Only the
Pan compiler is strictly necessary in a Quattor system; the other two subsystems
can be replaced by any service providing similar functionality.
Devolved management in a cross-domain environment requires users to be authenticated and
their operations to be authorized. For the configuration database, we chose to adopt X.509 certificates
because of the support offered by many standard tools, and access control lists (ACLs)
because they allow a fine-grained control (an ACL can be attached to each template). When many
users interact with the system, conflicts and misconfiguration may arise which require a roll back
mechanism; to this purpose, a simple concurrent transaction mechanism, based on standard version
control systems, was implemented.
Quattor’s modular architecture allows the three configuration management subsystems to be
deployed in either a distributed or centralized fashion. In the distributed approach, profile compilation
(at development stage) is carried out on client systems, templates are then checked in to a
suitable database, and finally the deployment is initiated by invoking a separate operation on the
server. The centralized approach provides strict control of configuration data. The compilation
burden is placed onto the central server, and users can only access and modify templates via a
dedicated interface.
Since the two paradigms provide essentially the same functionality, the choice between them
depends on which fits the management model of an organization better. For instance, the centralized
approach fits large computer centres well because of its strictly controlled work-flow, whereas multi-site
organizations such as GRIF prefer the distributed approach because it allows different parts of
the whole configuration set to be handled autonomously.
written in the Pan configuration language
by system administrators and produces XML
files (profiles) that are easily consumed by Quattor clients. The Pan language itself has a simple,
declarative syntax that allows simultaneous definition of configuration information and an associated
schema. In this section, we focus only on the Pan features that are relevant to devolved
management of distributed sites: validation, configuration reuse, and modularization.
Validation. The extensive validation features in the Pan language maximize the probability of
finding configuration problems at compile time, minimizing costly clean-ups of deployed misconfiguration.
Pan enables system administrators to define atomic or compound types with associated
validation functions; when a part of the configuration schema is bound to a type, the declared
constraints are automatically enforced.
Configuration reuse. Pan allows identification and reuse of configuration information through
“structure templates.” These identify small, reusable chunks of Pan-level configuration information
which can be used whenever an administrator identifies an invariant (or nearly invariant) configuration
sub-tree.
Modularization. With respect to the original design, two new features have been developed
to promote modularization and large-scale reuse of configurations: the name-spacing and load-path
mechanisms.
A full site configuration typically consists of a large number of templates organized into directories
and subdirectories. The Pan template name-spacing mimics (and enforces) this organization
much as is done in the Java language. The name-space hierarchy is independent of the configuration
schema. The configuration schema is often organized by low-level services such as firewall settings
for ports, account generation, log rotation entries, cron
entries, and the like. In contrast, the Pan
templates are usually organized based on other criteria like high-level services (web server, mail
server, etc.) or by responsible person/group.
The name-spacing allows various parts of the configuration to be separated and identified. To
effectively modularize part of the configuration for reuse, administrators must be able to import
the modules easily into a site’s configuration and to customize them. Users of the Pan compiler
combine a load-path with the name-spacing to achieve this. The compiler uses the load-path to search
multiple root directories for particular, named templates; the first version found on the load-path
is the one that is used by the compiler. This allows modules to be kept in a pristine state while
allowing sites to override any particular template.
Further, module developers can also expose global variables to parameterize the module, permitting
a system administrator to use a module without having to understand the inner workings
of the module’s templates.
Quattor Working Group (QWG) templates are used to configure grid middleware
services.
The QWG templates use all of the features of Pan to allow distributed sites to share grid middleware
expertise.
install machines, possibly from a remote location. To this purpose, Quattor provides a modular
framework called the Automated Installation Infrastructure (AII). This framework is responsible
for translating the configuration parameters embodied in node profiles into installation instructions
suitable for use by standard installation tools. Current AII modules use node profiles to configure
DHCP servers, PXE boot and Kickstart-guided installations.
Normally AII is set up with an install server at each site. However, the above mentioned
technologies allow the transparent implementation of multi-site installations, by setting up a central
server and appropriate relays using standard protocols.
Each node has a set of configuration agents (components) that are each registered with a particular part of the configuration schema.
For example, the component that manages user accounts is registered with the path /software/components/accounts.
A dispatcher program running on the node performs an analysis of the freshly retrieved configuration for changes in the relevant sections, and triggers the appropriate components.
Run-time dependencies may be expressed in the node’s profile, so that a partial order can be enforced on component execution.
For example, it is important that the user accounts component runs before the file creation component, to ensure that file ownership can be correctly specified.
By design, no control loop is provided for ensuring the correct execution of configuration components.
Site administrators typically use standard monitoring systems to detect and respond to
configuration failures. Nagios
and Lemon are both being used at Quattor sites for this
purpose. In fact, Lemon has been developed in tandem with Quattor, and provides sensors to
detect failures in Quattor component execution.
While nodes normally update themselves automatically, administrators can configure the system
to disable automatic change deployment. This is crucial in a devolved system where the responsibilities
for, respectively, modifying and deploying the configuration may be separated. A typical
scenario is that top-level administrators manage the shared configuration of multiple remote sites
and local managers apply it according to their policies. For instance, software updates might be
scheduled at different times.
The Quattor tool-kit, like other configuration management systems, reduces the manpower required to maintain a cluster and facilitates reliable change management. However, three unique features make it particularly attractive for managing grid
Grid computing
Grid computing is a term referring to the combination of computer resources from multiple administrative domains to reach a common goal. The grid can be thought of as a distributed system with non-interactive workloads that involve a large number of files...
resources:
- Federated Management: The open, modular nature of the tool-kit permits system administrators at different institutes to share the management of their distributed resources.
- Shared Configuration and Management Efficiency: Quattor encourages the re-use of configuration information in such a way that it can be distributed and used with little or no modification at different sites, facilitating the distribution of best practices without the need for each site to implement configuration changes.
- Coherent Site Model: Quattor allows an administrator to develop a site model that, once constructed, can be used to manage a range of different resources, such as real machines, virtual machines and cloud resources.
These features are also attractive beyond the grid context. This has been confirmed by the growing adoption of Quattor, by both commercial companies and academic institutions, most of them using the tool-kit to manage consistently their grid and non-grid systems.
Principles
The challenge of structuring and sharing components in a collaborative system is not new; overthe years programming language designers have attacked this problem from many angles. While
trends change, the basic principles are well understood. Features such as encapsulation, abstraction,
modularity, and typing produce clear benefits. We believe that similar principles apply when sharing
configuration information across administrative domains.
The Quattor configuration tool-kit derives its architecture from LCFG
LCFG
LCFG stands for "Local ConFiGuration system". Developed at the University of Edinburgh beginning around 1993, it is "a system for automatically installing and managing the configuration of large numbers of computer systems...
, improving it in several
aspects. At the core of Quattor is Pan, a high-level, typed language with flexible include mechanisms,
a range of data structures, and validation features familiar to modern programmers. Pan
allows collaborative administrators to build up a complex set of configuration templates describing
service types, hardware components, configuration parameters, users etc. The use of a high-level
language facilitates code reuse in a way that goes beyond cut-and-paste of configuration snippets.
The principles embodied in Quattor are in line with those established within the system administration
community. In particular, all managed nodes retrieve their configurations from
a configuration server backed by a source-control system (or systems in the case of devolved management).
This allows individual nodes to be recreated in the case of hardware failure. Quattor
handles both distributed and traditional (single-site) infrastructures.
Devolved management includes the following features: consistency over a multi-site
infrastructure, multiple management points, and the ability to accommodate the specific needs
of constituent sites. There is no single “correct” model for a devolved infrastructure, thus great
flexibility is needed in the architecture of the configuration system itself. Sometimes a set of highly autonomous
sites wish to collaborate loosely. In this case each site will host a fairly comprehensive
set of configuration servers, with common configuration information being retrieved from a shared
database and integrated with the local configuration.
Distributing the management task can potentially introduce new costs. For example, transmitting
configuration information over the WAN introduces latency and security concerns. Quattor
allows servers to be placed at appropriate locations in the infrastructure to reduce latency, and
the use of standard tools and protocols means that existing security systems (such as a public key
infrastructure) can be harnessed to encrypt and authenticate communications.
Configuration management system
Quattor’s configuration management system is composed of a configuration database that storeshigh-level configuration templates, the Pan compiler that validates templates and translates them
to XML profiles, and a machine profile repository that serves the profiles to client nodes. Only the
Pan compiler is strictly necessary in a Quattor system; the other two subsystems
can be replaced by any service providing similar functionality.
Devolved management in a cross-domain environment requires users to be authenticated and
their operations to be authorized. For the configuration database, we chose to adopt X.509 certificates
because of the support offered by many standard tools, and access control lists (ACLs)
because they allow a fine-grained control (an ACL can be attached to each template). When many
users interact with the system, conflicts and misconfiguration may arise which require a roll back
mechanism; to this purpose, a simple concurrent transaction mechanism, based on standard version
control systems, was implemented.
Quattor’s modular architecture allows the three configuration management subsystems to be
deployed in either a distributed or centralized fashion. In the distributed approach, profile compilation
(at development stage) is carried out on client systems, templates are then checked in to a
suitable database, and finally the deployment is initiated by invoking a separate operation on the
server. The centralized approach provides strict control of configuration data. The compilation
burden is placed onto the central server, and users can only access and modify templates via a
dedicated interface.
Since the two paradigms provide essentially the same functionality, the choice between them
depends on which fits the management model of an organization better. For instance, the centralized
approach fits large computer centres well because of its strictly controlled work-flow, whereas multi-site
organizations such as GRIF prefer the distributed approach because it allows different parts of
the whole configuration set to be handled autonomously.
Pan language
The Pan language compiler https://trac.lal.in2p3.fr/Quattor/wiki/Doc/panc sits at the core of the Quattor tool-kit. It compiles machine configurationswritten in the Pan configuration language
Pan (programming language)
The pan configuration language allows the definition of machine configuration information and an associatedschema with a simple, human-accessible syntax...
by system administrators and produces XML
files (profiles) that are easily consumed by Quattor clients. The Pan language itself has a simple,
declarative syntax that allows simultaneous definition of configuration information and an associated
schema. In this section, we focus only on the Pan features that are relevant to devolved
management of distributed sites: validation, configuration reuse, and modularization.
Validation. The extensive validation features in the Pan language maximize the probability of
finding configuration problems at compile time, minimizing costly clean-ups of deployed misconfiguration.
Pan enables system administrators to define atomic or compound types with associated
validation functions; when a part of the configuration schema is bound to a type, the declared
constraints are automatically enforced.
Configuration reuse. Pan allows identification and reuse of configuration information through
“structure templates.” These identify small, reusable chunks of Pan-level configuration information
which can be used whenever an administrator identifies an invariant (or nearly invariant) configuration
sub-tree.
Modularization. With respect to the original design, two new features have been developed
to promote modularization and large-scale reuse of configurations: the name-spacing and load-path
mechanisms.
A full site configuration typically consists of a large number of templates organized into directories
and subdirectories. The Pan template name-spacing mimics (and enforces) this organization
much as is done in the Java language. The name-space hierarchy is independent of the configuration
schema. The configuration schema is often organized by low-level services such as firewall settings
for ports, account generation, log rotation entries, cron
Cron
Cron is a time-based job scheduler in Unix-like computer operating systems. Cron enables users to schedule jobs to run periodically at certain times or dates...
entries, and the like. In contrast, the Pan
templates are usually organized based on other criteria like high-level services (web server, mail
server, etc.) or by responsible person/group.
The name-spacing allows various parts of the configuration to be separated and identified. To
effectively modularize part of the configuration for reuse, administrators must be able to import
the modules easily into a site’s configuration and to customize them. Users of the Pan compiler
combine a load-path with the name-spacing to achieve this. The compiler uses the load-path to search
multiple root directories for particular, named templates; the first version found on the load-path
is the one that is used by the compiler. This allows modules to be kept in a pristine state while
allowing sites to override any particular template.
Further, module developers can also expose global variables to parameterize the module, permitting
a system administrator to use a module without having to understand the inner workings
of the module’s templates.
Quattor Working Group (QWG) templates are used to configure grid middleware
Middleware
Middleware is computer software that connects software components or people and their applications. The software consists of a set of services that allows multiple processes running on one or more machines to interact...
services.
The QWG templates use all of the features of Pan to allow distributed sites to share grid middleware
Middleware
Middleware is computer software that connects software components or people and their applications. The software consists of a set of services that allows multiple processes running on one or more machines to interact...
expertise.
Automated installation management
A key feature for administering large distributed infrastructures is the ability to automaticallyinstall machines, possibly from a remote location. To this purpose, Quattor provides a modular
framework called the Automated Installation Infrastructure (AII). This framework is responsible
for translating the configuration parameters embodied in node profiles into installation instructions
suitable for use by standard installation tools. Current AII modules use node profiles to configure
DHCP servers, PXE boot and Kickstart-guided installations.
Normally AII is set up with an install server at each site. However, the above mentioned
technologies allow the transparent implementation of multi-site installations, by setting up a central
server and appropriate relays using standard protocols.
Node configuration management
In Quattor, managed nodes handle their configuration process autonomously; all actions are initiated locally, once the configuration profile has been retrieved from the repository.Each node has a set of configuration agents (components) that are each registered with a particular part of the configuration schema.
For example, the component that manages user accounts is registered with the path /software/components/accounts.
A dispatcher program running on the node performs an analysis of the freshly retrieved configuration for changes in the relevant sections, and triggers the appropriate components.
Run-time dependencies may be expressed in the node’s profile, so that a partial order can be enforced on component execution.
For example, it is important that the user accounts component runs before the file creation component, to ensure that file ownership can be correctly specified.
By design, no control loop is provided for ensuring the correct execution of configuration components.
Site administrators typically use standard monitoring systems to detect and respond to
configuration failures. Nagios
Nagios
Nagios is a popular open source computer system and network monitoring software application. It watches hosts and services, alerting users when things go wrong and again when they get better....
and Lemon are both being used at Quattor sites for this
purpose. In fact, Lemon has been developed in tandem with Quattor, and provides sensors to
detect failures in Quattor component execution.
While nodes normally update themselves automatically, administrators can configure the system
to disable automatic change deployment. This is crucial in a devolved system where the responsibilities
for, respectively, modifying and deploying the configuration may be separated. A typical
scenario is that top-level administrators manage the shared configuration of multiple remote sites
and local managers apply it according to their policies. For instance, software updates might be
scheduled at different times.