Test Input Included In 3.0

[hcronk]

This question is specifically about VIIRS Aerosol EDR input but hopefully is general enough to help others as well. The input path specified by default in the VIIRS Aerosol EDR execution config file is a directory containing multiple files of most every available input (as specified by the Offical Short Names in the algorithm config file). I am wondering what/where specifies which of these files to use as input? Somewhat related, I am also wondering if later I can put all input files in a single location for use (similar to what I see currently) or if I must keep each granule's worth of inputs in an individual location and change the input path in the execution config file each time?

Thanks,
Heather

[houchin]

In ADL, all of the inputs, both data and setup (LUT, PC, ...) must be placed in the input directory. However, what you can also do is to place a symbolic link to the desired file in that directory, rather than the actual file itself (see the sample inputs with ADL 3).

In metadata mode, there are a couple fields in the config file that specify which data inputs to use. In noMetadata mode, the inputs need to be the specific shortnames with no other identifier (e.g., VIIRS-SCIENCE-RDR). In those cases, it is probably best to create a new input directory with symbolic links to all of the LUT/PC files and to the specific inputs you want to use, and then put that input directory into the config file.

The convention I am following is to create the output directory for my ADL run, then create an input directory inside that directory. Into the input directory go all of the symbolic links to the inputs, wherever they are found. The config file then also goes into the input directory. The benefit of this is you now have pseudo-built-in metadata for a given set of outputs, which inputs were used.

[kbisanz]

It is also possible to list multiple input paths in the <inputPath> tag. They should be separated by colons. You can even list the output directory in the input path if you want outputs to feed back into an algorithm the next time it is executed. See $ADL_HOME/cfg/dynamic/withMetadata/ProSdrViirsSolarDiffuserLwFile.xml for an example.

A few things should be noted if you're using multiple input paths:
--The exact same item (with the same URID) cannot exist in multiple places. The code will fail during initialization.
--The order of directories in <inputPath> does not matter and does not give any files in those directories preference over other files.
--If the combined list of files in all input directories gets large (above ~1,000 files) you may start to notice a slow down in the algorithm. The algorithm will still complete, it may just take longer. The reason is that the ADL data store does not utilize a database for fast searches.

As for which of the inputs to choose, the code builds up a collection of "metadata" and then uses that to whittle down the list of available inputs. For example, here is a small piece of a debug log file from running an algorithm in "with metadata" mode:

Code: Select all

2011/06/30 19:11:35.153 (974904.1): DBG_LOW DmCoreURMetadataFilter.cpp|137| tid-1 DMS query start: Searching for items matching the following metadata:
Metadata Collection METADATALIST (size 3)
item 0, 1201ded30 id=0 urId=0 parent=0 parentType=0 ItemIndex_=0 type=STRING name=N_Collection_Short_Name, comparison=EQ, value=NCEP-ANC-Int
item 1, 11f63c648 id=0 urId=0 parent=0 parentType=0 ItemIndex_=0 type=INTEGER name=DatasetLock, comparison=EQ,value=0
item 2, 11e774ef0 id=0 urId=0 parent=0 parentType=0 ItemIndex_=0 type=DATETIMERANGE name=Effectivity, comparison=EQ2003-01-25 07:18:13.368600,2003-01-25 07:18:13.368600

In this case, the code is looking for an item with a name of "NCEP-ANC-Int" and the Effectivity range specified. If you run the algorithm in "no metadata" mode, a metadata collection is still used internally, but contains fewer elements. It may contain only the "N_Collection_Short_Name", which in no metadata mode is the file name on disk.

If you're running in "with metadata" mode, you should be able to put all inputs for multiple granules into the same directory, provided they have the appropriate metadata to differentiate themselves from each other. For example, most things would be differentiated by their granule ID and collection short name.

If you're trying to determine why a certain input was not able to be found, I would recommend reading section "5.8.7.1.2 Input Query Errors" in the User Manual Part 2, located in $ADL_HOME/doc.

[hcronk]

[quote="kbisanz"] As for which of the inputs to choose, the code builds up a collection of "metadata" and then uses that to whittle down the list of available inputs.

I'm not sure I understand this. Does it mean I need to change metadata parameters in the code rather than a config file to select a certain input from the input directory?

[JimBiard]

It depends. I'm going to assume that you are working with metadata. The *LwFile.xml file for the application specifies the granule ID for each data input to be processed. If you want to select a different granule (or granules), you specify the new granule ID in the *LwFile.xml file. Ancillary and auxiliary data that are required are specified by collection short name in the various *CFG.xml files that are listed in the *GuideList.cfg file named in the *LwFile.xml file. The ancillary and auxiliary data all have associated metadata that specifies effectivity date ranges, and the appropriate files to use are selected based on the acquisition times of the granules you are processing.

All of the input files (or links to them) must be in one of the folders specified in the *LwFile.xml file as containing input. If you want to modify the contents of an ancillary or auxiliary file, you can add a new one with new metadata (and a new reference ID) and edit the effectivity dates in the old and new metadata files to cause the new file to be used instead of the old one, or you can leave the metadata alone and modify the contents of the binary blob file.

Of course, if you are running without metadata, then there are only binary blob files named according to their collection short names, and there can only be one of each (with the exception of leading and trailing data granule files and spacecraft diary files, but those are disambiguated with appended numbers on the file names).

If you are talking about adding a completely new input data item to your algorithm that wasn't used before, then yes, there will be some programming involved. In truth, a lot of the base-level work will be done by the ADL source generator once you have defined the data product with the XML editor and created the appropriate *CFG.xml file and added it to the *GuideList.cfg file.

[hcronk]

Thanks Jim, that helped a bunch! I had seen what I thought was the granule ID in the *LwFile.xml but couldn't find anything that corresponded to that ID in the input names. I didn't realize that the ID from the *LwFile.xml corresponded to one IN the metadata text file and not in the file name.

[hcronk]

Is the granule ID the only thing I should need to change in the *LwFile.xml to run a new granule and get new output?

[kbisanz]

Is the granule ID the only thing I should need to change in the *LwFile.xml to run a new granule and get new output?

Yes, it should be. Of course you'll need to make sure you have the inputs for that new granule in the input directory since it'll be looking for inputs with a different granule ID.