As a cluster administrator, you need to create cluster resources for every resource or application you run on servers in your cluster. Cluster resources can include Web sites, e-mail servers, databases, file systems, virtual machines, and any other server-based applications or services you want to make available to users at all times.
For an overview of resource types you can create, refer to Section 4.2.3, Masters.
There are three types of RAs (Resource Agents) available with the cluster (for background information, see Section 4.2.2, Linux Standards Base (LSB) Scripts). To create a cluster resource use the crm tool. To add a new resource to the cluster, proceed as follows:
Open a shell and become root.
Enter crm configure to open the internal shell
Configure a primitive IP address:
crm(live)configure# primitive myIP ocf:heartbeat:IPaddr \
params ip=127.0.0.99 op monitor interval=60s
The previous command configures a primitive
with the
name myIP. You need to choose a class (here
ocf), provider (heartbeat), and
type (IPaddr). Furthermore, this primitive expects
other parameters like the IP address. Change the address to your
setup.
Display and review the changes you have made:
crm(live)configure# showCommit your changes to take effect:
crm(live)configure# commitTo set up the NFS server, you need to complete the following:
Configure DRBD.
Set up a file system Resource.
Set up the NFS server and configure the IP address.
Learn how to achieve this in the following subsection.
Before starting with the DRBD High Availability configuration, set up a DRBD device manually. Basically this is configuring DRBD and letting it synchronize. The exact procedure is described in Section 13.0, Distributed Replicated Block Device (DRBD). For now, assume that you configured a resource r0 that may be accessed at the device /dev/drbd_r0 on both of your cluster nodes.
The DRBD resource is an OCF master/slave resource. This can be found in the description of the metadata of the DRBD resource agent. However, it is important that the actions promote and demote exist in the actions section of the metadata. These are mandatory for master/slave resources and commonly not available to other resources.
For High Availability, master/slave resources may have multiple masters on different nodes. It is even possible to have a master and slave on the same node. Therefore, configure this resource in a way that there is exactly one master and one slave, each running on different nodes. Do this with the meta attributes of the master resource. Master/slave resources are special types of clone resources in High Availability. Every master and every slave counts as a clone.
Proceed as follows to configure a DRBD resource:
Open a shell and become root.
Enter crm configure to open the internal shell.
If you have a two-node cluster, set the following properties per ms resource:
crm(live)configure# primitive my-stonith stonith:external/ipmi ... crm(live)configure# ms ms_drbd_r0 res_drbd_r0 meta \ globally-unique=false ... crm(live)configure# property no-quorum-policy=ignore crm(live)configure# property stonith-enabled=true
Create a primitive DRBD resource:
crm(live)configure# primitive drbd_r0 ocf:linbit:drbd params \
drbd drbd_resource=r0 op monitor interval="30s"Create a master/slave resource:
crm(live)configure# ms ms_drbd_r0 res_drbd_r0 meta master-max=1 \
master-node-max=1 clone-max=2 clone-node-max=1 notify=trueSpecify an colocation and order constraint:
crm(live)configure# colocation fs_on_drbd_r0 inf: res_fs_r0 ms_drbd_r0:Master crm(live)configure# order fs_after_drbd_r0 inf: ms_drbd_r0:promote res_fs_r0:start
Display your changes with the show command.
Commit your changes with the commit command.
The filesystem resource is configured as an OCF primitive resource with DRBD. It has the task of mounting and unmounting a device to a directory on start and stop requests. In this case, the device is /dev/drbd_r0 and the directory to use as mount point is /srv/failover. The file system used is xfs.
Use the following commands in the crm shell to configure a file system resource:
crm(live)# configure crm(live)configure# primitive filesystem_resource \ ocf:linbit:drbd \ params device=/dev/drbd_r0 directory=/srv/failover fstype=xfs
To make the NFS server always available at the same IP address, use an additional IP address as well as the ones the machines use for their normal operations. This IP address is then assigned to the active NFS server in addition to the system's IP address.
The NFS server and the IP address of the NFS server should always be active on the same machine. In this case, the start sequence is not very important. They may even be started at the same time. These are the typical requirements for a group resource.
Before starting the High Availability RA configuration, configure the NFS server with YaST. Do not let the system start the NFS server. Just set up the configuration file. If you want to do that manually, see the manual page exports(5) (man 5 exports). The configuration file is /etc/exports. The NFS server is configured as an LSB resource.
Configure the IP address completely with the High Availability RA configuration. No additional modification is necessary in the system. The IP address RA is an OCF RA.
crm(live)# configure crm(live)configure# primitive nfs_resource ocf:nfsserver \ params nfs_ip=10.10.0.1 nfs_shared_infodir=/shared crm(live)configure# primitive ip_resource ocf:heartbeat:IPaddr \ params ip=10.10.0.1 crm(live)configure# group nfs_group nfs_resource ip_resource crm(live)configure# show primitive ip_res ocf:heartbeat:IPaddr \ params ip="192.168.1.10" primitive nfs_res ocf:heartbeat:nfsserver \ params nfs_ip="192.168.1.10" nfs_shared_infodir="/shared" group nfs_group nfs_res ip_res crm(live)configure# commit crm(live)configure# end crm(live)# quit
From the crm perspective, a STONITH device is just another resource. To create a STONITH resource, proceed as follows:
Open a shell and become root.
Enter crm to open the internal shell.
Get a list of all STONITH types with the following command:
crm(live)# ra list stonith
apcmaster apcsmart baytech
cyclades drac3 external/drac5
external/hmchttp external/ibmrsa external/ibmrsa-telnet
external/ipmi external/kdumpcheck external/rackpdu
external/riloe external/sbd external/ssh
external/vmware external/xen0 external/xen0-ha
ibmhmc ipmilan meatware
null nw_rpc100s rcd_serial
rps10 ssh suicideChoose a STONITH type from the above list and view the list of possible options. Use the following command:
crm(live)# ra info stonith:external/ipmi
IPMI STONITH external device (stonith:external/ipmi)
ipmitool based power management. Apparently, the power off
method of ipmitool is intercepted by ACPI which then makes
a regular shutdown. If case of a split brain on a two-node
it may happen that no node survives. For two-node clusters
use only the reset method.
Parameters (* denotes required, [] the default):
hostname (string): Hostname
The name of the host to be managed by this STONITH device.
...Create the STONITH resource with the stonith class, the type you have chosen in , and the respective parameters if needed, for example:
crm(live)# configure crm(live)configure# primitive my-stonith stonith:external/ipmi \ params hostname="node1" ipaddr="192.168.1.221" \ userid="admin" passwd="secret" \ op monitor interval=60m timeout=120s
Having all the resources configured is only one part of the job. Even if the cluster knows all needed resources, it might still not be able to handle them correctly. For example, try not to mount the file system on the slave node of drbd (in fact, this would fail with drbd). Define constraints to make these kind of information available to the cluster.
For more information about constraints, see Section 4.4, Resource Constraints.
This type of constraint may be added multiple times for each resource. All location constraints are evaluated for a given resource. A simple example that expresses a preference to run a resource with the ID fs1-loc on the node with the name earth to 100 would be the following:
crm(live)configure# location fs1-loc fs1 100: earth
Another example is a location with pingd:
crm(live)configure# primitive pingd pingd \ params name=pingd dampen=5s multiplier=100 host_list="r1 r2" crm(live)configure# location node_pref internal_www \ rule 50: #uname eq node1 \ rule pingd: defined pingd
The collocation command is used to define what resources should run on the same or on different hosts.
It is only possible to set a score of either +inf or -inf, defining resources that must always or must never run on the same node. It is also possible to use non-infinite scores. In that case the collocation is called advisory and the cluster may decide not to follow them in favour of not stopping other resources if there is a conflict.
For example, to the resources with the IDs filesystem_resource and nfs_group always on the same host, use the following constraint:
crm(live)configure# colocation nfs_on_filesystem inf: nfs_group filesystem_resource
For a master slave configuration, it is necessary to know if the current node is a master in addition to running the resource locally.
Sometimes it is necessary to provide an order of resource actions or operations. For example, you cannot mount a file system before the device is available to a system. Ordering constraints can be used to start or stop a service right before or after a different resource meets a special condition, such as being started, stopped, or promoted to master. Use the following commands in the crm shell to configure an ordering constraint:
crm(live)configure# order nfs_after_filesystem mandatory: filesystem_resource nfs_group
The example used for this chapter would not work without additional constraints. It is essential that all resources run on the same machine as the master of the drbd resource. The drbd resource must be master before any other resource starts. Trying to mount the DRBD device when it is not the master simply fails. The following constraints must be fulfilled:
The file system must always be on the same node as the master of the DRBD resource.
crm(live)configure# colocation filesystem_on_master inf: \
filesystem_resource drbd_resource:MasterThe NFS server as well as the IP address must be on the same node as the file system.
crm(live)configure# colocation nfs_with_fs inf: \
nfs_group filesystem_resourceThe NFS server as well as the IP address start after the file system is mounted:
crm(live)configure# order nfs_second mandatory: \
filesystem_resource:start nfs_groupThe file system must be mounted on a node after the DRBD resource is promoted to master on this node.
crm(live)configure# order drbd_first inf: \
drbd_resource:promote filesystem_resourceTo determine a resource failover, use the meta attribute migration-threshold. For example:
crm(live)configure# location r1-node1 r1 100: node1
Normally, r1 prefers to run on node1. If it fails there, migration-threshold is checked and compared to the failcount. If failcount >= migration-threshold then it is migrated to the node with the next best preference.
Start failures set the failcount to inf depend on the start-failure-is-fatal option. Stop failures cause fencing. If there is no STONITH defined, the resource will not migrate at all.
For an overview, refer to Section 4.4.3, Failover Nodes.
A resource might fail back to its original node when that node is back online and in the cluster. If you want to prevent a resource from failing back to the node it was running on prior to failover, or if you want to specify a different node for the resource to fail back to, you must change its resource stickiness value. You can either specify resource stickiness when you are creating a resource, or afterwards.
For an overview, refer to Section 4.4.4, Value is -INFINITY:.
Configuring Placement of Resources Based on Load Impact
Not all resources are equal. Some, such as Xen guests, require that the node hosting them meets their capacity requirements. If resources are placed such that their combined need exceed the provided capacity, the resources diminish in performance (or even fail).
To take this into account, the High Availability Extension allows you to specify the following parameters:
The capacity a certain node provides.
The capacity a certain resource requires.
An overall strategy for placement of resources.
For detailed background information about the parameters and a configuration example, refer to Section 4.4.5, balanced.
To configure the resource's requirements and the capacity a node provides, use utilization attributes as described in Adding Or Modifying Utilization Attributes. You can name the utilization attributes according to your preferences and define as many name/value pairs as your configuration needs.
In the following example, we assume that you already have a basic configuration of cluster nodes and resources and now additionally want to configure the capacities a certain node provides and the capacity a certain resource requires.
Start the crm shell with the following command:
crm configureTo specify the capacity a node provides, use the following command and replace the placeholder NODE_1 with the name of your node:
crm(live)configure# node NODE_1 utilization memory=16384 cpu=8
With these values, NODE_1 would be assumed to provide 16GB of memory and 8 CPU cores to resources.
To specify the capacity a resource requires, use:
crm(live)configure# primitive xen1 ocf:heartbeat:Xen ... \
utilization memory=4096 cpu=4
This would make the resource consume 4096 of those memory units from nodeA, and 4 of the cpu units.
Configure the placement strategy with the property command:
crm(live)configure# property ...
Four values are available for the placement strategy:
Utilization values are not taken into account at all, per default. Resources are allocated according to location scoring. If scores are equal, resources are evenly distributed across nodes.
Utilization values are taken into account when deciding whether a node is considered eligible if it has sufficient free capacity to satisfy the resource's requirements. However, load-balancing is still done based on the number of resources allocated to a node.
Utilization values are taken into account when deciding whether a node is eligible to serve a resource; an attempt is made to concentrate the resources on as few nodes as possible, thereby enabling possible power savings on the remaining nodes.
Utilization values are taken into account when deciding whether a node is eligible to serve a resource; an attempt is made to spread the resources evenly, optimizing resource performance.
The placing strategies are best-effort, and do not yet utilize complex heuristic solvers to always reach an optimum allocation result. Ensure that resource priorities are properly set so that your most important resources are scheduled first.
Commit your changes before leaving the crm shell:
crm(live)configure# commitThe following example demonstrates a three node cluster of equal nodes, with 4 virtual machines:
crm(live)configure# node node1 utilization memory="4000" crm(live)configure# node node2 utilization memory="4000" crm(live)configure# node node3 utilization memory="4000" crm(live)configure# primitive xenA ocf:heartbeat:Xen \ utilization memory="3500" meta priority="10" crm(live)configure# primitive xenB ocf:heartbeat:Xen \ utilization memory="2000" meta priority="1" crm(live)configure# primitive xenC ocf:heartbeat:Xen \ utilization memory="2000" meta priority="1" crm(live)configure# primitive xenD ocf:heartbeat:Xen \ utilization memory="1000" meta priority="5" crm(live)configure# property placement-strategy="minimal"
With all three nodes up, xenA will be placed onto a node first, followed by xenD. xenB and xenC would either be allocated together or one of them with xenD.
If one node failed, too little total memory would be available to host them all. xenA would be ensured to be allocated, as would xenD; however, only one of xenB or xenC could still be placed, and since their priority is equal, the result is not defined yet. To resolve this ambiguity as well, you would need to set a higher priority for either one.
To monitor a resource, there are two possibilities: either define a monitor operation with the op keyword or use the monitor command. The following example configures an Apache resource and monitors it every 30 minutes with the op keyword:
crm(live)configure# primitive apache apache \ params ... \ op monitor interval=60s timeout=30s
The same can be done with:
crm(live)configure# primitive apache apache \ params ... crm(live)configure# monitor apache 60s:30s
For an overview, refer to Section 4.3, Resource Monitoring.
One of the most common elements of a cluster is a set of resources that needs to be located together. Start sequentially and stop in the reverse order. To simplify this configuration we support the concept of groups. The following example creates two primitives (an IP address and an e-mail resource):
Run the crm command as system administrator. The prompt changes to crm(live).
Configure the primitives:
crm(live)# configure crm(live)configure# primitive Public-IP ocf:IPaddr:heartbeat \ params ip=1.2.3.4 crm(live)configure# primitive Email lsb:exim
Group the primitives with their relevant identifiers in the correct order:
crm(live)configure# group shortcut Public-IP EmailFor an overview, refer to Groups.
Clones were initially conceived as a convenient way to start N instances of an IP resource and have them distributed throughout the cluster for load balancing. They have turned out to quite useful for a number of other purposes, including integrating with DLM, the fencing subsystem and OCFS2. You can clone any resource, provided the resource agent supports it.
Learn more about cloned resources in Stateful Clones.
To create an anonymous clone resource, first create a primitive resource and then refer to it with the clone command. Do the following:
Open a shell and become root.
Enter crm configure to open the internal shell.
Configure the primitive, for example:
crm(live)configure# primitive Apache lsb:apacheClone the primitive:
crm(live)configure# clone apache-clone Apache
To create an stateful clone resource, first create a primitive resource and then the master/slave resource.
Open a shell and become root.
Enter crm configure to open the internal shell.
Configure the primitive. Change the intervals if needed:
crm(live)configure# primitive myRSC ocf:myCorp:myAppl \
op monitor interval=60 \
op monitor interval=61 role=MasterCreate the master slave resource:
crm(live)configure# clone apache-clone Apache