Environment
SUSE Linux Enterprise Server 11
Novell Open Enterprise Server (Linux based)Novell eDirectory 8.7.3 for Linux
SUSE Linux Enterprise Server 9
SUSE Linux Enterprise Server 10
VMware ESX server
NTP Time Synchronization
Situation
Time drifting when running a Linux guest under VMware ESX server.
When running eDirectory on the guest, ndsrepair -T reports that time is not in sync.
ERROR: -659 adding a server to the tree
ERROR: -659 adding a server to the tree
BACKGOUND:
An operating system needs to have some way to schedule work in the future at a precise time. A processor supporting such an operating system must have a programmable interrupt timer that can periodically interrupt the processor in order for it to run scheduled work. This periodic interrupt is known as a system clock tick. All activities are scheduled using this constant "heartbeat". Within the Linux kernel time is seen as the amount of clock ticks since the system booted. This measurement is known as jiffies.
The Linux 2.6 kernel in SLES9 changes the base interrupt rate as compared to the 2.4 kernel in SLES8 from 100 per second to 1000 per second. (SLES 10 SP1's kernel is set to 250 HZ.) A dual-processor Linux 2.6 kernel can fire up to 3000 per second. This is usually not an issue when running on a bare metal server under moderate load. However, issues can easily arise in a virtualized environment where multiple instances of the Linux kernel are in competition for the host's actual hardware based processor's attention. When the system gets busy for extended periods lost ticks can occur (in a virtualized environment little actual guest OS activity is required). These lost ticks are usually due to I/O and system services that must wake up and mask or disable interrupts for a time while they run. When the interrupts are disabled for an extended amount of time ticks will pile up and the kernel can lose track of their count - they are consideredlost.
Due to the descrepency of how many interrupts the Linux kernel guest occurances can generate and the amount the ESX host kernel can service, lost ticks are a frequent concern. Code was added to the 2.6 kernel to detect lost ticks and correct for them. However, it is difficult at times to accurately detect when a tick was lost. Due to the way x86 queues the interrupt handlers an extra tick may be added by this code when, in fact, none were actually lost. These extra ticks can lead to time gaining or moving forward beyond the server's time source. Lost ticks in themselves lead to time being lost (time slowing from the source).
Resolution
To minimize time advancing beyond the time source due to the kernel's compensation code adding too many ticks use the following boot parameter:
clock=pit
(For a SLES 10 SP1 kernel running under ESX 3.x this is likely the only setting that will have to be changed.)
The following settings and recommendations for Linux guests running under ESX should avoid timesync issues in this environment.
NOTE: SLES 11 requires no kernel parameters.
SETUP NTP ON THE VIRTUALIZED GUESTS
a. Ensure that an external reliable NTP time source is used both for the ESX host as well as its virtualized guests. The timesource should not be the ESX host itself nor a virtual server running under it. Doing otherwise will only add to a virtual machine's interrupt load and exacerbate the issue.
b. Set up the NTP Source.
- Modify the /etc/ntp.conf file. Comment out the following entries so the local clock will not be used by placing # signs before them:
#server 127.127.1.0 # local clock
#fudge 127.127.1.0 stratum 10 - Add an entry for this time source at the end of the file replacing x.x.x.x with the server's IP address:
server x.x.x.x prefer
- Stop the NTP daemon. On SLES10:
/etc/init.d/ntp stopOn SLES9:
/etc/init.d/xntpd stop
- Slam the current time by typing:
ntpdate x.x.x.x(where x.x.x.x is the NTP source's IP address)
(Force this command several times until the offset displayed is less than one second.) -
Start the NTP daemon again. On SLES10:
/etc/init.d/ntp startOn SLES9:
/etc/init.d/xntpd -
Wait for this server to show a reach of 377 by typing:
ntpq -p(This may take 15-20 minutes.)
Make changes to the VMware Environment
ESX 2.x only -Misc.TimerHardPeriod
This helps with time lagging. If time is lagging on a ESX 3.x server with SLES 10 SP1 kernels it is likely NTP is misconfigured, the source is unreliable or there is too much load on the ESX server. Moving virtual machines off the server to another and\or tuning the priority of the guest may help as this gives the guests more processor cycles. VMware recommends not changing the default values in ESX 3.0 for the max and min times.
Using the ESX 2.0 Status Monitor select Options - Advanced Settings - Scroll to the setting"Misc.TimerHardPeriod". Change the value from the default of 1000 to 333 (or 250) and OK it. This change is dynamic and takes place immediately.
Change the Kernel Boot Parameters
ESX 2.x and 3.x - clock=pit
This change helps with overcorrection of time - time moving forward due to the the extra ticks generated by the kernel's compensation code.
Within the guest, open the /boot/grub/menu.lst file within vi and add "clock=pit" to the end of the kernel line in the title Linux section.
NOTE: There was a report of clock=pit causing a reboot when clustering services start when running in a clustering environment. The recommendation was to change this to clock=pmtmr.
Below is an example:
kernel (hd0,2)/boot/vmlinuz root=/dev/hda3 vga=0x317 selinux=0 splash=silent
resume=/dev/hda1 elevator=cfq showopts clock=pit
Should you find NTP time is not being reliably synchronized the following steps may be useful:
- Verify that the choosen time souce has given us time and is trusted:
ntpq -p
that there is an asterisk by the chosen time provider entry.The virtual machine may need to its APIC and/or ACPI disabled at boot time before loading the OS in addition to these steps.
If an asterisk is not seen using the ntpq -p command APIC could be interferring with time synchronization. This can be disabled by passing the "noapic" parameter at bootup by adding this parameter to the boot switch in /boot/grub/menu.lst.
Type
ntpd
At the command window grab a list of association indexes by typing "as". Then grab a detailed debug report on this server by typing "rv assid" where assid is the number seen in the previous step. Many of the values displayed are useful in debugging NTP. One of the most useful is the flash value. This displays the results of the sanity checks placed on the incoming NTP packets. This value should be "flash=00 ok".
Additional Information
Perl Script for testing the time drift:
By using the following perl script it is possible to test the success of the suggestions mentioned above:
#!/usr/bin/perl
# Get drift
my $lines = `ntpdate -q pool.ntp.org`;
my @lines = split(/\s/,$lines);
my $dummy = pop(@lines);
my $drift = pop(@lines);
my $nowstring = `date +"%Y%m%d-%H%M%S"`;
chomp $nowstring;
print "$nowstring -> $drift\n";
Save this script e.g. into the file /usr/local/bin/timedrift-check.pl and make it executable (by using chmod +x).
The script prints a line in the following format:
->
When starting the script frequently you can see how the time drift evolve.
VMware Knowledgebase Articles
TID history
Formerly known as TID# 10100597
By using the following perl script it is possible to test the success of the suggestions mentioned above:
#!/usr/bin/perl
# Get drift
my $lines = `ntpdate -q pool.ntp.org`;
my @lines = split(/\s/,$lines);
my $dummy = pop(@lines);
my $drift = pop(@lines);
my $nowstring = `date +"%Y%m%d-%H%M%S"`;
chomp $nowstring;
print "$nowstring -> $drift\n";
Save this script e.g. into the file /usr/local/bin/timedrift-check.pl and make it executable (by using chmod +x).
The script prints a line in the following format:
When starting the script frequently you can see how the time drift evolve.
VMware Knowledgebase Articles
- "Linux Guest Clock Runs Slowly on ESX Server 3", http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&externalId=2219
- "Clock in a Linux Guest Runs More Slowly or Quickly Than Real Time", http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1420
TID history
Formerly known as TID# 10100597