Polar wonder ancillary file conversion in ADL 4.0

[jhuangadl]

I was using the new conversion tools in ADL4.0 successfully until I want to start some massive processing:
For example, I have a bunch of Polar wonder files in the loading zone for the converter. However, the converter only processed two of the files. See below.
What could be the cause of such? Any solutions that we can work around to meet massive processing needs?
Thanks!
Jingfeng

Code: Select all

[scientist@localhost anc_msd]$ ls /home/scientist/CSPP/ADL/data/MSD_LZ/ANC
off_USNO-PolarWander-UT1-ANC_Ser7_USNO_000f_20120427_201204271200Z_20120427000003Z_ee20120504120000Z_np.msd
off_USNO-PolarWander-UT1-ANC_Ser7_USNO_000f_20120427_201204271200Z_20120427000003Z_ee20120504120000Z_np.msd.CMP_FILENAME
off_USNO-PolarWander-UT1-ANC_Ser7_USNO_000f_20120504_201205041200Z_20120504000003Z_ee20120511120000Z_np.msd
off_USNO-PolarWander-UT1-ANC_Ser7_USNO_000f_20120504_201205041200Z_20120504000003Z_ee20120511120000Z_np.msd.CMP_FILENAME
off_USNO-PolarWander-UT1-ANC_Ser7_USNO_000f_20120511_201205111200Z_20120511000004Z_ee20120518120000Z_np.msd
off_USNO-PolarWander-UT1-ANC_Ser7_USNO_000f_20120511_201205111200Z_20120511000004Z_ee20120518120000Z_np.msd.CMP_FILENAME
off_USNO-PolarWander-UT1-ANC_Ser7_USNO_000f_20120518_201205181200Z_20120518000003Z_ee20120525120000Z_np.msd
off_USNO-PolarWander-UT1-ANC_Ser7_USNO_000f_20120518_201205181200Z_20120518000003Z_ee20120525120000Z_np.msd.CMP_FILENAME
off_USNO-PolarWander-UT1-ANC_Ser7_USNO_000f_20120525_201205251200Z_20120525000003Z_ee20120601120000Z_np.msd
off_USNO-PolarWander-UT1-ANC_Ser7_USNO_000f_20120525_201205251200Z_20120525000003Z_ee20120601120000Z_np.msd.CMP_FILENAME
off_USNO-PolarWander-UT1-ANC_Ser7_USNO_000f_20120601_201206011200Z_20120601000003Z_ee20120608120000Z_np.msd
off_USNO-PolarWander-UT1-ANC_Ser7_USNO_000f_20120601_201206011200Z_20120601000003Z_ee20120608120000Z_np.msd.CMP_FILENAME
[scientist@localhost anc_msd]$ /home/scientist/CSPP/ADL/script/runMsd.pl -a
Running ING MSD...
ING MSD completed successfully
All file from Landing Zone converted.
Number of newly converted output files: 4
Checking NCEP-ANC-Int products...
Checking NAAPS-ANC-Int products...
Checking NOGAPS-ANC-Int products...
[scientist@localhost anc_msd]$ ls $ADL_HOME/data/output/withMetadata/IngMsdOutputs/
50572fc7-a8a8a-007f0100-00cb9751.asc
50572fc7-a8a8a-007f0100-00cb9751.off_USNO-PolarWander-UT1-ANC_Ser7_USNO_000f_20120525_201205251200Z_20120525000003Z_ee20120601120000Z_np.msd
50572fc7-b6467-007f0100-00cc711e.asc
50572fc7-b6467-007f0100-00cc711e.off_USNO-PolarWander-UT1-ANC_Ser7_USNO_000f_20120525_201205251200Z_20120525000003Z_ee20120601120000Z_np.msd_Internal
50572fc7-c4fd1-007f0100-00cd5c98.asc
50572fc7-c4fd1-007f0100-00cd5c98.off_USNO-PolarWander-UT1-ANC_Ser7_USNO_000f_20120601_201206011200Z_20120601000003Z_ee20120608120000Z_np.msd
50572fc7-d14f6-007f0100-00ce21ad.asc
50572fc7-d14f6-007f0100-00ce21ad.off_USNO-PolarWander-UT1-ANC_Ser7_USNO_000f_20120601_201206011200Z_20120601000003Z_ee20120608120000Z_np.msd_Internal

[jhuangadl]

It only processes two USNO files, maximum. Any chance we can change any configuration files to add more allowance?

[kbisanz]

This is expected behavior in the operational system. It's something that kind of snuck in to ADL and we didn't notice it until you brought it up.

In the config file $ADL_HOME/cfg/ADL_CFG_MSD.xml there are entries for "WILL_PURGE_DMS" and "NUMBER_OF_DMS_COPIES". Consider the following scenario:
1. You put multiple data files of a single type in the landing zone.
2. If WILL_PURGE_DMS is true, the ingest functionality will allow a maximum of NUMBER_OF_DMS_COPIES files in DMS for a given type.
3. If DMS already contains 2 items and a newer version comes along, the older of the existing 2 will be deleted and the newest one will be inserted.

Judging by your output, the 2 newest files remain in your DMS. But if you want to run data before they become effective (before May 25), you may be out of luck until you change that setting.

To fix your problem I believe you want to change every setting of WILL_PURGE_DMS from true to false in $ADL_HOME/cfg/ADL_CFG_MSD.xml and then rerun runMsd.pl.

[jhuangadl]

Thanks a lot, Kevin, it worked out beautifully!
The NCEP and NAAPS did not have such problem, and I have seen that in the file you mentioned, the settings are 'false' for NCEP and NAAPS.
For the USNO setting (called 'Earth Orientation' in the file), I changed it from 'true' to 'false', and that is the only change I made in that file.
Then everything worked out for our purposes of generating ancillary files for running aerosol algorithm.