Determining exact input file set from Output file metadata

Data formats, HDF5, XML profiles, etc.
Post Reply
houchin
Posts: 128
Joined: Mon Jan 10, 2011 6:20 am

Determining exact input file set from Output file metadata

Post by houchin »

Hi all,

I am in the process of making the Aerospace ADL configuration more DMS like; instead of a input template containing a specific set of input LUTs, I am going to better leverage multiple input directories specified in the execution XML file. For example, I will have a directory for all valid versions of the VIIRS-SDR-F-LUT installed in the system. By default, ADL will use the DMS rules to select the correct version of that file. If the user desires to override the DMS selection and use a specific file, that directory will not be included as an input directory, and a link to the specific file will be placed into the "primary" input directory along with other this-specific-run files, like the unpacked RDR.

From a using ADL perspective, this is an important change as it allows us to mimic the input files that would have been used on the IDSD by default, whereas with the previous configuration, this was not possible.

Unfortunately, this is going to make it more difficult to gather all of the inputs for a given ADL run into a single tarball that we can submit to DPE for the LUT update process much more difficult. For the submission to be valid, we need to tar up every single input file used (minus the tile data), and thus it is essential that I be able to figure out which of each of those inputs ADL selected.

I can get part of this information by looking at the metadata file for one of the outputs. However, the information for the different files appears to be inconsistently specified.

Consider the following two files placed in an input directory for the ADL run:

Code: Select all

parmentier:inputs$ ls -l 4ed4e84c-7098d-dd828c19-8bdc5952* 4f06fd13-d1690-dd828c19-0f65f9bc*
lrwxrwxrwx 1 sh27576 viirs  83 Feb  8 11:15 4ed4e84c-7098d-dd828c19-8bdc5952.asc -> /tcs/ago/group/viirs/adl/lut/I1.5.06J/unpacked/4ed4e84c-7098d-dd828c19-8bdc5952.asc*
lrwxrwxrwx 1 sh27576 viirs 212 Feb  8 11:15 4ed4e84c-7098d-dd828c19-8bdc5952.VIIRS-SDR-DELTA-C-LUT_npp_20110822010100Z_20111101010000Z_ee00000000000000Z_PS-1-N-CCR-11-117-JPSS-DPA-001-PE-_noaa_all_all-_all.bin -> /tcs/ago/group/viirs/adl/lut/I1.5.06J/unpacked/4ed4e84c-7098d-dd828c19-8bdc5952.VIIRS-SDR-DELTA-C-LUT_npp_20110822010100Z_20111101010000Z_ee00000000000000Z_PS-1-N-CCR-11-117-JPSS-DPA-001-PE-_noaa_all_all-_all.bin
lrwxrwxrwx 1 sh27576 viirs  83 Feb  8 11:15 4f06fd13-d1690-dd828c19-0f65f9bc.asc -> /tcs/ago/group/viirs/adl/lut/I1.5.06J/unpacked/4f06fd13-d1690-dd828c19-0f65f9bc.asc*
lrwxrwxrwx 1 sh27576 viirs 219 Feb  8 11:15 4f06fd13-d1690-dd828c19-0f65f9bc.VIIRS-SDR-BB-TEMP-COEFFS-LUT_npp_20111109010000Z_20111101010000Z_ee00000000000000Z_PS-1-N-CCR-11-221-JPSS-DPA-001-PE-_noaa_all_all-_all.bin -> /tcs/ago/group/viirs/adl/lut/I1.5.06J/unpacked/4f06fd13-d1690-dd828c19-0f65f9bc.VIIRS-SDR-BB-TEMP-COEFFS-LUT_npp_20111109010000Z_20111101010000Z_ee00000000000000Z_PS-1-N-CCR-11-221-JPSS-DPA-001-PE-_noaa_all_all-_all.bin

parmentier:inputs$ ls -l 4f32c9be-28378-dd828c19-3102698f.*
-rwxrwxrwx 1 sh27576 viirs 1108 Feb  8 11:15 4f32c9be-28378-dd828c19-3102698f.asc*
lrwxrwxrwx 1 sh27576 viirs   47 Feb  8 11:15 4f32c9be-28378-dd828c19-3102698f.VIIRS-SDR-F-LUT -> /tcs/ago/home/sh27576/VIIRS-SDR-F-LUT-SN003.bin

parmentier:inputs$ ls -l /tcs/ago/group/viirs/adl/lut/I1.5.06J/unpacked/4ed4e84c-7098d-dd828c19-8bdc5952.asc*
-rwxrwxrwx 1 sh27576 viirs 1376 Nov 29 06:12 /tcs/ago/group/viirs/adl/lut/I1.5.06J/unpacked/4ed4e84c-7098d-dd828c19-8bdc5952.asc*
parmentier:inputs$ ls -l /tcs/ago/group/viirs/adl/lut/I1.5.06J/unpacked/4ed4e84c-7098d-dd828c19-8bdc5952.VIIRS-SDR-DELTA-C-LUT_npp_20110822010100Z_20111101010000Z_ee00000000000000Z_PS-1-N-CCR-11-117-JPSS-DPA-001-PE-_noaa_all_all-_all.bin
-rw-rw---- 1 sh27576 viirs 4505920 Feb  8 10:37 /tcs/ago/group/viirs/adl/lut/I1.5.06J/unpacked/4ed4e84c-7098d-dd828c19-8bdc5952.VIIRS-SDR-DELTA-C-LUT_npp_20110822010100Z_20111101010000Z_ee00000000000000Z_PS-1-N-CCR-11-117-JPSS-DPA-001-PE-_noaa_all_all-_all.bin
parmentier:inputs$ ls -l /tcs/ago/group/viirs/adl/lut/I1.5.06J/unpacked/4f06fd13-d1690-dd828c19-0f65f9bc.asc*
-rwxrwxrwx 1 sh27576 viirs 1467 Jan  6 05:54 /tcs/ago/group/viirs/adl/lut/I1.5.06J/unpacked/4f06fd13-d1690-dd828c19-0f65f9bc.asc*
parmentier:inputs$ ls -l /tcs/ago/group/viirs/adl/lut/I1.5.06J/unpacked/4f06fd13-d1690-dd828c19-0f65f9bc.VIIRS-SDR-BB-TEMP-COEFFS-LUT_npp_20111109010000Z_20111101010000Z_ee00000000000000Z_PS-1-N-CCR-11-221-JPSS-DPA-001-PE-_noaa_all_all-_all.bin
-rw-rw---- 1 sh27576 viirs 240 Feb  8 10:37 /tcs/ago/group/viirs/adl/lut/I1.5.06J/unpacked/4f06fd13-d1690-dd828c19-0f65f9bc.VIIRS-SDR-BB-TEMP-COEFFS-LUT_npp_20111109010000Z_20111101010000Z_ee00000000000000Z_PS-1-N-CCR-11-221-JPSS-DPA-001-PE-_noaa_all_all-_all.bin
I realize that listing is a mess, but the files are as follows:
  • The Delta C and BB-TEMPS binary files and metadata files are symbolic links using absolute paths to the target
  • The targets of the Delta C and BB-TEMPS binary and metadata files are real files
  • The F-LUT binary file is a symbolic link to an absolute path, and the target of that link is again a real file
  • The F-LUT metadata file in the input directory is a real file (the ADL front end "unpacks" the binary LUT, creating the metadata file as part of staging that file)
However, in the metadata for the M7 SDR output for that run, I see the following

Code: Select all

    ("N_Aux_Filename" STRING EQ "../unpacked/4ed4e84c-7098d-dd828c19-8bdc5952.VIIRS-SDR-DELTA-C-LUT_npp_20110822010100Z_20111101010000Z_ee00000000000000Z_PS-1-N-CCR-11-117-JPSS-DPA-001-PE-_noaa_all_all-_all.bin")
... snip ...
    ("N_Aux_Filename" STRING EQ "/tcs/ago/group/viirs/adl/lut/I1.5.06J/unpacked/4f06fd13-d1690-dd828c19-0f65f9bc.VIIRS-SDR-BB-TEMP-COEFFS-LUT_npp_20111109010000Z_20111101010000Z_ee00000000000000Z_PS-1-N-CCR-11-221-JPSS-DPA-001-PE-_noaa_all_all-_all.bin")
    ("N_Aux_Filename" STRING EQ "VIIRS-SDR-F-LUT-SN003.bin")


This makes it very difficult to find the specific files. Is there a reason they are different, and if not (or the reason isn't important), can we get an ADL update that writes absolute paths for all ancillary and auxiliary files listed in the output metadata.

We are heading toward a near term weekly update of the F and H LUTs, and a slightly longer-term daily update cycle, so it is important to be able to communicate clearly for a given output exactly the input files that were used.
Scott Houchin, Senior Engineering Specialist, The Aerospace Corporation
15049 Conference Center Dr CH3/310, Chantilly, VA 20151; 571-307-3914; scott.houchin@aero.org
houchin
Posts: 128
Joined: Mon Jan 10, 2011 6:20 am

Re: Determining exact input file set from Output file metada

Post by houchin »

So as I've continued to make the updates to my DMS-like configuration, I discovered that the values written into the output products are the literal values pulled out of the input metadata files. This explains the inconsistency. So certainly I can update all of my metadata files to contain the full path of the file in its current location, and I probably will end up doing that sometime in the next couple weeks.

However, this doesn't solve the problem if the files are moved.

It still would be great if when ADL read in the file, it adjusted the value in memory to be the absolute path of the current location of the file, which then would be written out correctly when products are written.
Scott Houchin, Senior Engineering Specialist, The Aerospace Corporation
15049 Conference Center Dr CH3/310, Chantilly, VA 20151; 571-307-3914; scott.houchin@aero.org
kbisanz
Posts: 280
Joined: Wed Jan 05, 2011 7:02 pm
Location: Omaha NE

Re: Determining exact input file set from Output file metada

Post by kbisanz »

You are correct about the names being passed through from the input files.

In ADL Phase 4, we can investigate adding an absolute path when an item is being output. This would only affect outputs. We would not adjust any input .asc files already existing on disk.
Kevin Bisanz
Raytheon Company
houchin
Posts: 128
Joined: Mon Jan 10, 2011 6:20 am

Re: Determining exact input file set from Output file metada

Post by houchin »

kbisanz wrote:In ADL Phase 4, we can investigate adding an absolute path when an item is being output. This would only affect outputs. We would not adjust any input .asc files already existing on disk.
Agreed.

Thanks!
Scott Houchin, Senior Engineering Specialist, The Aerospace Corporation
15049 Conference Center Dr CH3/310, Chantilly, VA 20151; 571-307-3914; scott.houchin@aero.org
johnyjohn
Posts: 2
Joined: Fri Jul 13, 2012 7:48 am

Re: Determining exact input file set from Output file metada

Post by johnyjohn »

As a general rule, options are applied to the next specified file.
Therefore, order is important, and you can have the same option on the
command line multiple times. Each occurrence is then applied to the
next input or output file.
Post Reply