Friday, December 14, 2012

Damn it feels good to be a stratum 1 gangster

Alternate title: YET ANOTHER GARMIN GPS NTP POST

I've been bored recently and needed a project to keep me occupied.  I fell into a hole looking at folks pages describing how they built stratum 1 NTP servers (usually fed with GPS, but not always).  I looked at other methods such as:
  • Radio sourced (WWV / WWVB in the US)
    • I'm not enough of a radio geek to do WWV right (antenna / rcvr).  It's very tempting though because the audio decoder is built into ntpd.  There's also an automatic tuner that speaks ICOM control to tune up/down as the signal changes throughout the day !
    • I'm a fair distance from Colorado.
    • WWVB is what your Brookstone / Radioshack "atomic clock" uses.  They aren't atomic clocks, they are "radio clocks".  There are some folks who have cannibalized these and hacked together an interface that ntpd can read.  Looks interesting, maybe a new project for a new day.
  • CDMA
    • $$$
    • I don't see a cheap solution for this.  You can get a CDMA mini-pci card - but there is no inherent support in ntpd to pull the timing.  The 'appliances' for this are nice but expensive.
    • Cheapest would probably be this: http://www.endruntechnologies.com/time-frequency-reference-cdma.htm  Need to dig in and see if I can find a price for those.
  • Atomic
    • $$$$$$$$$
    • Not trying to compete with NIST.
So GPS seemed to be the way to go.  Digging into this I soon realized that the time code from GPS itself doesn't buy you much without PPS (pulse per second).  So the intersection of all that is the Garmin GPS 18x LVC model.  The -PC model (attached DE9 connector) and -USB models do not provide PPS.  There is also a 5hz model which provides 5PPS.  I don't think ntpd supports this, and also it's twice the price.  My understanding is that the difference between the -X and older non-X version are sensitivity.  I was able to get fix in the basement indoors with my 18x.

GPS 18x LVC

The LVC model appeared to come with bare wires exposed, but it seems they are putting a small header on them now.

Box contents

Original connector

Some of the projects I've found online have used a DE9 hood directly on the end, with another feeder cable coming back out for USB (power).  The LVC needs to get it's power through two of the leads, and with a DE9 alone (typically) you won't get this.

I decided to build an interface box that would let me combine the power from USB, and put this on two unused DE9 pins.

Inside project box

USB power cable

I borrowed ideas from several folks pages online (listed at the bottom), and decided to incorporate a fuse, and an LED wired to the PPS line (DCD).  This is nice because after getting power and acquiring enough satellites, the GPS starts PPS and the led flashes with each pulse.  Also, blinkenlights :)

Unit - GPS side

Unit - PC side

After the interface box was complete I was able to at least get the GPS connected and powered.  I used a Windows7 machine in the house to do this.  Even though configuration can be done through minicom in Linux - the Windows utility also lets you upgrade the firmware on the GPS itself.

I installed SNSRXCFG and also downloaded the 3.80 firmware from: http://www8.garmin.com/support/collection.jsp?product=010-00321-31

To upgrade the firmware, it seems I needed to fiddle with UAC in Windows (ughh).  So I disabled it temporarily to do this.  SNSRXCFG calls another program to do the firmware and I never seemed to get prompted by UAC to allow it.  Once I got through all that - it told me I was already running the same (3.80) firwmare anyway :/

Also in the utility I made the following config changes:
  • Set all GPS sentences to disabled, except $GPRMC.  This significantly lessens the amount of data sent in a cycle (to a single line)
  • Check that PPS itself is on
  • Set PPS length to 200ms
  • Set "PPS Auto off" on
    • This seems a bit counterintuitive.  From what I read - with this setting on, if the GPS loses satellite acquisition, it will kill PPS.  This should allow ntpd to switch to another source.
I decided to mount the GPS outside.  The server room is in the basement and has windows at ground level.  I built a mount out of PVC pipe and routed the cable inside through the window.  I used a Dremel to make a very shallow notch at the point where the windows slide together as to not crush the cable.

GPS mounting complete

I bought a PCI serial card to add to the server.  I wanted to make sure that I had real 16550 UARTs, and not introduce any buffering or latency from a USB/Serial converter.

My server is Linux.  Specifically Ubuntu server 12.04 LTS.  Originally this was 10.04 - but 10.04 does not include all needed kernel modules for PPS.  Who doesn't like an unforeseen late night re-install :)

I also decided to take the advice of several sites and use setserial to set low_latency on the GPS port.  My understanding is that this disables or tells the kernel to NOT use the buffers on the UART - rather send and recieve data immediately.  For something as time sensitive as stratum 1 time - every bit of delay you cut out the better.

/var/lib/setserial/autoserial.conf:

* Note the comments, you must follow those instructions regarding dpkg-reconfigure so that the file is not overwritten:

# If you want to configure this file by hand, use 
# dpkg-reconfigure setserial
# and change the configuration mode of the file to MANUAL. 
# If you do not do this, this file may be overwritten automatically the next time
# you upgrade the package.
#
/dev/ttyS0 uart 16550A port 0x03f8 irq 4 baud_base 115200 spd_normal skip_test
/dev/ttyS4 uart 16550A port 0xe800 irq 21 baud_base 115200 spd_normal skip_test low_latency
/dev/ttyS5 uart 16550A port 0xe400 irq 21 baud_base 115200 spd_normal skip_test

/dev/ttyS4 is the first port on the PCI card and where the GPS lives.

Next up is getting PPS working.  For this, ldattach is used.  This is a userspace process, which attaches a device to a 'line discipline'.  For this, our line discipline (or ldisc) is PPS.  Being a userspace process, ldattach will continue to run for the duration.

To set all this up, as well as create the symlinks that GPS expects I wrote a simple init.d script.  To this extent I'm committing Ubuntu blasphemy two-fold: not using udev to create the symlinks as well as continuing to use init.d as opposed to upstart.  :)  The relevant lines of the script are:

	# attach PPS line discipline
	ldattach PPS /dev/ttyS4

	# create symlinks for devices that ntpd expects

	# gps / pps
	ln -sf /dev/ttyS4 /dev/gps0
	chmod 666 /dev/gps0
	ln -sf /dev/pps0 /dev/gpspps0
	chmod 666 /dev/gpspps0
	# acts modem
	ln -sf /dev/ttyS0 /dev/acts0
	chmod 666 /dev/acts0

At this point I wanted to test to make sure PPS is really coming through.  The best way to do this is to use the pps-tools package, specifically the ppswatch utility.  Unfortunately my aforementioned double Ubuntu blasphemy has caught up to me - and pps-tools is NOT in 12.04.  However it is in 12.10, so I went for a third offense and manually downloaded and installed it from 12.10 by looking it up on http://packages.ubuntu.com

Once you have this, you should run ppswatch against /dev/pps0 and see something like:

ppswatch /dev/pps0
trying PPS source "/dev/pps0"
found PPS source "/dev/pps0"
timestamp: 1355443840, sequence: 126, offset:  201326869
timestamp: 1355539785, sequence: 127, offset:  200000916
timestamp: 1355539785, sequence: 127, offset:  200000916
timestamp: 1355539786, sequence: 128, offset:  199999743
timestamp: 1355539786, sequence: 128, offset:  199999743
timestamp: 1355539787, sequence: 129, offset:  199998954
^C

Total number of PPS signals: 6
Maximum divergence: 201326869

I'm going to be using the PPS support that is built into the GPS NMEA refclock driver .  In many (older) HOWTOs and posts the PPS driver (ATOM) is defined and used separately.  I decided to use the 'integrated' version, hence the links above pointing gps0 and gpspp0 to essentially the same device (since ldattach is going to create pps0 <--> ttyS4).

My relevant ntp.conf section for the GPS:

# gps / pps
server 127.127.20.0 mode 1 minpoll 4 maxpoll 4
fudge 127.127.20.0 flag1 1 flag2 0 flag3 1 time2 0.600

Here is what all of that means:

  • server 127.127.20.0
    • Definition for GPS NMEA driver
  • mode 1
    • mode 0 is default, use the first recieved GPS sentence in a cycle.  mode 1 means only pay attention to $GPMRC
  • minpoll 4 maxpoll 4
    • Lock polling to 16 seconds, no more no less
  • flag1 1
    • Enable PPS signal processing
  • flag2 0
    • Capture PPS pulse on rising edge.  This is the default - I really don't need this.
  • flag3 1
    • Use kernel PPS interface
  • time2 0.600
If it isn't clear already - I'm using the standard ntp.org ntpd.  The package that comes with Ubuntu 12.04 is 4.2.6p3.  There were changes/fixes to the GPS NMEA driver > 4.2.6p3.  I don't know if they are related - but my experience is that I get better stability with a compiled from source ntpd compared to the Ubuntu packaged version.  I downloaded and compiled 4.2.6p5 from ntp.org.

The fruits of all this labor:

vom@ice:~$ ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
oGPS_NMEA(0)     .GPS.            0 l    8   16  377    0.000   -0.001   0.001
+name1.glorb.com 209.51.161.238   2 u   36   64  377   30.927    2.583   1.697
-clock01.chil01. 131.107.13.100   2 u   21   64  377   44.486    4.650   1.665
+clock02.chil01. 131.107.13.100   2 u   14   64  377   44.070    4.257   0.575
-clock01.sctn01. 131.107.13.100   2 u   33   64  377   55.393   -3.165   0.547
-clock02.sctn01. 131.107.13.100   2 u   10   64  277   46.121   -0.836   2.133
*clock.nyc.he.ne .CDMA.           1 u   59   64  377   33.833    2.284   5.762
 ACTS_MODEM(0)   .NIST.           0 l  28h   64    0    0.000   -2.507   0.000

vom@ice:~$ ntpq -c rv
associd=0 status=041d leap_none, sync_uhf_radio, 1 event, kern,
version="ntpd 4.2.6p5@1.2349 Sun Dec  9 10:07:32 UTC 2012 (2)",
processor="x86_64", system="Linux/3.2.0-34-generic", leap=00, stratum=1,
precision=-23, rootdelay=0.000, rootdisp=0.428, refid=GPS,
reftime=d47671a1.4549604e  Fri, Dec 14 2012 22:48:49.270,
clock=d47671ae.7e476db8  Fri, Dec 14 2012 22:49:02.493, peer=24779, tc=4,
mintc=3, offset=-0.001, frequency=-5.498, sys_jitter=0.001,
clk_jitter=0.000, clk_wander=0.000

For monitoring, originally I was using MRTG (I'm old school).  MRTG by itself though, doesn't use RRD (by default), doesn't handle decimal places and doesn't handle negative numbers.  I had an external wrapper script calling ntpq -p and converting to microseconds and then adding an offset (i.e. 5000 us).  This would mean that an offset of -2 ms (-2000us) would get 5000us added to it, and be represented as 3000us.  The 'middle line' is 5000us, which is really zero.  So after all this, you would see deviations and dips above/below the middle line.

Once I had GPS/PPS though - I decided to pursue something a bit more modern than good old MRTG.  I decided on collectd which uses RRD, is very easy to set up and even has a plugin for ntpd.  Now I can get graphs like:



Here's some of the sites/links that I relied on heavily to put all this together:

http://www.satsignal.eu/ntp/FreeBSD-GPS-PPS.htm
http://time.qnan.org
http://blog.doylenet.net/?p=145
http://www.rjsystems.nl/en/2100-ntpd-garmin-gps-18-lvc-gpsd.php
http://www.eecis.udel.edu/~mills/ntp/html/drivers/driver20.html
http://support.ntp.org/bin/view/Support/ConfiguringNMEARefclocks#Section_6.1.12.2.