MODIS Level 1 failures caused by leapsec and utcpole files

[gumley]

Two IMAPP VA users recently notified me that they were seeing failures in the MODIS Level 1 processing with messages like these in the processing log files:

---start of log file message---
PGS_TD_UTCjdtoTAIjd():PGSTD_E_NO_LEAP_SECS:28171
input Julian day out of range for tabulated corrections - approximate value used

PGS_TD_LeapSec():PGSTD_W_JD_OUT_OF_RANGE:27652
input time (at UTC Julian Date 2455509.591885) is past end of table --
a linear approximation was used to determine the leap seconds value

PGS_TD_TAIjdtoUTCjd():PGSTD_E_NO_LEAP_SECS:28171
input julian day out of range for tabulated corrections - approximate value used

PGS_TD_TAItoUTC():PGSTD_E_NO_LEAP_SECS:28171
input julian day out of range for tabulated corrections - approximate value used

PGS_TD_LeapSec():PGSTD_W_JD_OUT_OF_RANGE:27652
input time (at UTC Julian Date 2455509.591394) is past end of table --
a linear approximation was used to determine the leap seconds value
---end of log file message---

This problem is caused by old versions of leapsec.dat and/or utcpole.dat. These files must be kept up to date, because even if the information inside them has not changed, they contain a header line which records the time of the last update. The MODIS L1 software (and several other packages in the IMAPP VA) check the header time, and if it is more than 90 days old, it exits with an error.

There is an IMAPP VA script running under cron which is responsible for updating these files once a week. The script is

dbvm/scripts/update_leapsec_utcpole.bash

This script downloads new versions of these files from a site maintained by the Ocean Biology Processing Group (OBPG) at NASA GSFC. In the past, these files were hosted on an FTP server at

ftp://oceans.gsfc.nasa.gov/COMMON/leapsec.dat
ftp://oceans.gsfc.nasa.gov/COMMON/utcpole.dat

However OBPG now wants to encourage users to switch to a HTTP server at

http://oceandata.sci.gsfc.nasa.gov/Anci ... eapsec.dat
http://oceandata.sci.gsfc.nasa.gov/Anci ... tcpole.dat

and they have configured the old FTP server to respond with a timeout if a site requests two files in rapid succession. This is why the IMAPP VA update script is failing, and you should be able to see evidence of this problem in dbvm/logs/update_leapsec_utcpole.log.

You can edit your local copy of dbvm/scripts/update_leapsec_utcpole.bash to use the new file locations as shown above by typing

cd $DBVM_HOME/scripts
gedit update_leapsec_utcpole.bash

and changing the line containing the "lftp -c" command. In version 1.1 of the IMAPP VA (now in testing), the URLs for these files are configurable in the dbvm_env.bash file, and the update script has logic built in to handle cases where the Internet connection is down.

Cheers,
Liam.

[gumley]

It turns out that using lftp in "continue mode" can cause problems when updating these files. The update_leapsec_utcpole.bash script may report that it updated the files successfully, when in fact the contents of the file were unchanged.

One solution is to use curl instead of lftp to download the files. Change the following lines in dbvm/scripts/update_leapsec_utcpole.bash:

lftp -c "get -c http://oceandata.sci.gsfc.nasa.gov/Anci ... eapsec.dat"
lftp -c "get -c http://oceandata.sci.gsfc.nasa.gov/Anci ... tcpole.dat"

to

curl -O http://oceandata.sci.gsfc.nasa.gov/Anci ... eapsec.dat
curl -O http://oceandata.sci.gsfc.nasa.gov/Anci ... tcpole.dat

You can run the script manually by typing

update_leapsec_utcpole.bash

at the command line in the IMAPP VA. This should update your leapsec.dat and utcpole.dat files successfully.

Cheers,
Liam.