Novell SUSE Linux Enterprise Server 11
Novell SUSE Linux Enterprise Desktop 11
Novell SUSE Linux Enterprise Server 10
Novell SUSE Linux Enterprise Desktop 10

By default, a XEN installation will allow dom0 access to all physical memory installed in the system. XEN will then attempt to free (and reclaim) memory as needed for domU's...a process called "ballooning".

Testing has shown that this process may cause dom0 to become unresponsive for periods of several minutes during the shutdown/creation/migration of a domU if the domU has been assigned a memory amount > 512M or dom0 has been restricted to a single VCPU.

To avoid this condition:

• "Lock down" the amount of memory assigned to dom0.
• Disable ballooning
  
• Do not restrict dom0 to a single VCPU
• Only use Logical Volume Manager (LVM) on Dom0 or DomU, but not both
  

Restricting the amount of memory assigned to dom0 at boot time:

First determine a reasonable amount of memory for dom0. This amount will vary depending on your hardware and configuration. Then use an editor to change the hypervisor line in /boot/grub/menu.lst to have "dom0_mem=X", where X is the memory that will be assigned to dom0. In the example below 2048m (2G) was determined to be the amount:

title Xen -- SUSE Linux Enterprise Server 10 SP1 - 2.6.16.54-0.2.5
root (hd0,1)
kernel /xen.gz dom0_mem=2048m
module /vmlinuz-2.6.16.54-0.2.5-xen root=/dev/disk/by-id/scsi-3600508e00000000015153e9beb87870e-part1 vga=0x314 resume=/dev/sda3 splash=0 showopts
module /initrd-2.6.16.54-0.2.5-xen

Disable ballooning:

For SLES10: Change dom0-min-mem in /etc/xen/xend-config.sxp to match the value specified in the dom0_mem= boot option
For SLES11: Change (enable-dom0-ballooning yes) in /etc/xen/xend-config.sxp to (enable-dom0-ballooning no).

Ensure that dom0 has more than 1 processor.

This is the default (as long as the server has more than 1 cpu). However, you can verify this by using `xm list` and checking the VCPUs column for Domain-0 (example below):

Name ID Mem VCPUs State Time(s)
Domain-0 0 4096 16 r----- 478.2
sles-10-sp1 55000 1 27.4

If for some reason dom0 has been restricted to a single vcpu change it back by editing /etc/xen/xend-config.sxp. In that file there is a line that normally reads "(dom0-cpus 0)". If dom0 is coming up with only 1 vcpu then the line will read (dom0-cpus 1). Changing it back to the default will restore the desired behavior.

Only use LVM on Dom0 or DomU, but not both

Testing has shown that using LVM on both the Dom0 and a DomU, I/O performance suffers. It is recommended that you only use LVM on Dom0 or DomU, but not both.

After making these changes, reboot the server.

Frequently asked questions:

1- What is "ballooning" and how/why does it work against us here?

"Ballooning" is the process by which the amount of memory assigned to a domain is changed (up or down). In the case of this issue, domain-0 automatically "balloons" to free up memory domU needs to reclaim for shutdown.

2- Is there a formula for computing memory allocation. If I understand correctly, the total assignment to dom0 + hypervisor + vms must be < or = total physical memory in the machine. Is this correct?

Correct.

The amount of memory needed by the hypervisor, domain0, and each domU varies based on machine specifications, type of VM, etc. Therefore, there are no fixed values that can be assigned to any of those variables. However, 'xm info' can provide a view into the current state of the system. There are 5 fields in the list that provide information about the memory status:

total_memory : 8190 # Total RAM available to the system
free_memory : 177 # Free RAM (not in use by hypervisor, dom0, or any domU)
max_free_memory : 5610 # Maximum value possible for"free_memory" after ballooning dom0
max_para_memory : 5606 # Maximum amount of memory that could be assigned to a new pv guest
max_hvm_memory : 5584 # Maximum amount of memory that could be assigned to a new fv guest

3- Is there a limit regarding how much memory can be assigned to dom0? i.e. In a machine with 16G of RAM will "dom0_mem=12G" work as long as no more than 4G is allocated to vms and the hypervisor?

The only maximum for dom0 is the amount of memory in the physical system. The bigger concern should be the minimum that can be allocated to dom0. Ideally, find a value that is small enough to maximize the amount of memory that can be assigned to the guests, yet large enough that dom0 can handle all of the required I/O, management tasks, etc. Again, this value varies based on numerous factors, so experimentation is required in every environment. Once the appropriate value has been determined, the best thing to do is to set the initial/maximum memory assigned to dom0 using the "dom0_mem=" boot option and then prevent dom0 from ballooning below that value by changing the value of dom0-min-mem in /etc/xen/xend-config.sxp. This, in effect, disables the ballooning process.

4- Is there a formula for computing how much memory is required by the hypervisor (how much to leave "unassigned" for the hypervisor's needs)? Is this a static value?

This is not a static value. At very least there is going to be variation based on the hardware. For example, the hypervisor needs more memory to track the RAM on a 128 GB machine than it does on a machine with only 1 GB of memory. However, it is possible to get a good idea of how much is required by booting XEN in the default configuration (all available memory assigned to dom0). In this configuration, the total system memory minus the amount of memory assigned to dom0 equals the amount required by the hypervisor.