VMware ESX 3.5 ntpdate strangeness

We just noticed that the time was very far off on our sparkly new VMware EXS 3.5 server. When I went to run ntpdate to bring it up to sync, I was suprised to find that it could not make a connection to the time server because outbound UDP 123 traffic was blocked by the internal firewall. Here is what I got:

# /usr/sbin/ntpdate -u time.nist.gov
9 Apr 03:47:53 ntpdate[20245]: sendto(192.43.244.18): Operation not permitted
9 Apr 03:47:54 ntpdate[20245]: sendto(192.43.244.18): Operation not permitted
9 Apr 03:47:55 ntpdate[20245]: sendto(192.43.244.18): Operation not permitted
9 Apr 03:47:56 ntpdate[20245]: sendto(192.43.244.18): Operation not permitted
9 Apr 03:47:57 ntpdate[20245]: no server suitable for synchronization found

Normally I would just add a rule to the “/etc/sysconfig/iptables” file to allow traffic out on this port, but Vmware ESX server does not use iptables… It uses its own firewall, so I had to figure out how to change it. Happily, it turns out that there is a handy “esxcfg-firewall” command built just for such things.

Running this:
/usr/sbin/esxcfg-firewall -q | grep 123

12300 1803K valid-tcp-flags  tcp  --  *   *     0.0.0.0/0        0.0.0.0/0

Confirmed that UDP port 123 outbound was disallowed.

Running this opened it up:
/usr/sbin/esxcfg-firewall -e ntpClient

Grep out “123” again just to be sure:
/usr/sbin/esxcfg-firewall -q | grep 123

1  76 ACCEPT  udp  --  *    *    0.0.0.0/0      0.0.0.0/0     udp dpt:123

And you can now run ntpdate to sync up the time:
/usr/sbin/ntpdate -u time.nist.gov

9 Apr 09:52:54 ntpdate[20319]: step time server 192.43.244.18 offset 21689.039217 sec

Getting ntpd to work correctly on RHEL

When many new servers are delivered from the factory, the system clock is way off. Most UNIX systems run “ntpd” to keep the time in sync with internet time servers, which are, in turn synchronized against an atomic clock. This results in a system time that is very very close to the “actual” time of day. The downside, however, is that even a properly configured “ntpd” will not synchronize the system clock if it is too far out of sync with the time server. To remedy this, we first have to run “ntpdate” to get the system clock close to the correct time, and then enable “ntpd” to keep it there.

The first thing we have to do is “ntpd” to free up the port for “ntpdate”:

[root@server /]# /sbin/service ntpd stop
Shutting down ntpd:                                        [  OK  ]

This frees up the port for ntpdate. Next we run:

[root@server /]# /usr/sbin/ntpdate time.apple.com

Now the time should be set correctly. We then change the default time servers to something like the following in /etc/ntp.conf:

# --- OUR TIMESERVERS -----
time-a.timefreq.bldrdoc.gov
time-b.timefreq.bldrdoc.gov
time-c.timefreq.bldrdoc.gov

We can use any time server we want, but I like these and find them to be reliable.

Finally, start backup up your “ntpd” service, and your all set to go.

[root@server /]# /sbin/service ntpd start
Starting ntpd:                                        [  OK  ]

Remember to use “chkconfig” to make sure “ntpd” is enabled to come up when the system starts.