how to generate binary ancillary files for ADL3.0

[ljiang]

The conversion tool seems to work. But I still have a couple of questions: there is no metadata output for the blob ancillary file. Without metadata we cannot output final results in HDF. It seems not difficult to generate the meta file by ourselves since it's just dozens of lines, but I'd like to know which fields are important for running in ADL.

Also, from the ancillary blob metadata file that comes with ADL3.0 under ProSdrViirsControllerInputs, it shows:

("METADATALIST" COLLECTION
("DatasetLock" INTEGER EQ 0)
("Effectivity" DATETIMERANGE EQ "2003-01-25 06:30:00.000000" "2003-01-25 07:29:59.999999")
("N_Collection_Short_Name" STRING EQ "NCEP-ANC-Int")
("RangeDateTime" DATETIMERANGE EQ "2010-12-21 00:19:45.989236" "2010-12-21 00:19:45.989238")
("UpdateDateTime" DATETIME EQ "2010-12-21 00:19:46.142684")
("CreationDateTime" DATETIME EQ "2010-12-21 00:19:45.436529")
("N_Dataset_Type_Tag" STRING EQ "ANC-Int")
("AncFilename" GROUP EQ
("METADATALIST" COLLECTION
("N_Anc_Filename" STRING EQ "off_NCEP-GFS-06HR-ANC_GFS_NCEP_006f_20030125_200301250000Z_20030125000000Z_ee20070125000000Z_np")
("N_Anc_Filename" STRING EQ "off_NCEP-GFS-09HR-ANC_GFS_NCEP_009f_20030125_200301250000Z_20030125000000Z_ee20070125000000Z_np")
)
)


There are two files, 6hr forecast and 9 hr forecast, and the "Effectivity" is from 6:30~7:30. So IDPS seems to ingest NCEP data in 1-hr intervals, by interpolating temporally between two NCEP GFS files. The conversion tool seems to miss this point. Do I understand this correctly?

[kbisanz]

You are mostly correct. That particular file, which has an effectivity of 1 hour, was created by averaging 2 other files (off_NCEP-GFS-06HR-ANC_GFS_NCEP_006f_20030125_200301250000Z_20030125000000Z_ee20070125000000Z_np and off_NCEP-GFS-09HR-ANC_GFS_NCEP_009f_20030125_200301250000Z_20030125000000Z_ee20070125000000Z_np) which are the 6 and 9 hour forecasts.

In this particular case, IDPS ingested off_NCEP-GFS-06HR-ANC_GFS_NCEP_006f_20030125_200301250000Z_20030125000000Z_ee20070125000000Z_np and off_NCEP-GFS-09HR-ANC_GFS_NCEP_009f_20030125_200301250000Z_20030125000000Z_ee20070125000000Z_np and created 2 files with a collection short name of NCEP-ANC-Int. Each NCEP-ANC-Int is effective for 1 hour (6 and 9hr) at that point. At this point, there are "effectivity holes" in the group of NCEP-ANC-Int files because nothing is effective for the 7 and 8 hours. Then ProAncTemporalInterpolation.exe runs and creates 2 more NCEP-ANC-Int files (by averaging the 6 and 9 hr forecasts), each having 1 hour effectivities for 7 and 8 hours.

My guess is that after you run the native ancillary convertor tool and create .asc files, you need to run ProAncTemporalInterpolation.exe

Off the top of my head, I know the following fields are critically important to ADL:
DatasetLock
Effectivity
N_Collection_Short_Name
N_Dataset_Type_Tag

I don't believe the other pieces of metadata are critical, but I think you'll have some non-fatal errors if you look closely at the debug log. Additionally, metadata in algorithm outputs will not have some metadata correctly populated. It'd be best if all the metadata you listed could be populated. I have not tested my theories, so you'll have to give it a try using the ones I listed above as important.

I've never used the conversion tool, but it seems like it'd be easy for the tool to create a .asc file. Perhaps that's something UW could add. You would have to ask them.

[kbisanz]

Yes, ProAncTemporallInterpolation.exe should work. Its use is apparently not described very well from what I can tell.

After talking with another person, here's what I found out:
--It only works in "with metadata" mode.
--You use it by changing the "task details" in the LW file. So you'll need to change $ADL_HOME/cfg/dynamic/withMetadata/ProAncTemporalInterpolationLwFile.xml

Currently the LW contains this:

Code: Select all

  <task>
    <taskType>EDR</taskType>
    <taskDetails1>NCEP-ANC-Int</taskDetails1>
    <taskDetails2>1433008832000000</taskDetails2>
    <taskDetails3>6</taskDetails3>
    <taskDetails4></taskDetails4>
  </task>

You'll have to change items 1, 2, and 3. taskDetails1 is the collection short name. If you're using NCEP data, it's already set correctly. If you're using NOGAPS, you'll need to change it. taskDetails2 is the synoptic time in IET format. taskDetails3 is the leading forecast time.

You can use $ADL_HOME/tools/bin/string2iet.exe to convert to/from IET time and a human readable format. It reads times from STDIN. For example:

Code: Select all

$ADL_HOME/tools/bin > ./string2iet.exe 
Enter the time: 1433008832000000
2003-05-30 18:00:00.000000Z = 1433008832000000 = 1433008832.00000000
Enter the time: 2003-05-30 18:00:00.000000Z
2003-05-30 18:00:00.000000Z = 1433008832000000 = 1433008832.00000000
Enter the time: 2011-10-27 00:00:00.000000Z
2011-10-27 00:00:00.000000Z = 1698364834000000 = 1698364834.00000000
Enter the time: $ADL_HOME/tools/bin > 

ADL provided some test data after it's already been converted to the internal format in $ADL_HOME/data/input/withMetadata/ProAncTemporalInterpolationInputs. Say you wanted to temporally interpolate the 6 and 9 hr forecasts for NCEP (that's what is already configured).
1. Set taskDetails1 to NCEP-ANC-Int
2. View 481b520e-d624a-9b9deab9-e3c64ef9.asc and 481b5231-626ad-9b9deab9-e3bf137f.asc and note that the SynopticTime is "2003-05-30 18:00:00.000000".
3. Use $ADL_HOME/tools/bin/string2iet.exe to convert "2003-05-30 18:00:00.000000" to IET time. Note that you have to add a trailing "Z" (for Zulu time) when you provide the time to string2iet.exe.

Code: Select all

$ADL_HOME/tools/bin > echo 2003-05-30 18:00:00.000000Z | ./string2iet.exe 
Enter the time: 2003-05-30 18:00:00.000000Z = 1433008832000000 = 1433008832.00000000
Enter the time: $ADL_HOME/tools/bin >

4. Add 1433008832000000 to taskDetails2
5. Of the 2 forecast times you're interpolating (6 and 9 in this case), add the first one to taskDetails3. In this case it's "6"

Following the above steps is how

Code: Select all

  <task>
    <taskType>EDR</taskType>
    <taskDetails1>NCEP-ANC-Int</taskDetails1>
    <taskDetails2>1433008832000000</taskDetails2>
    <taskDetails3>6</taskDetails3>
    <taskDetails4></taskDetails4>
  </task>

was constructed.

[ljiang]

Kevin, thanks very much for the useful instructions. It seems the ProAncTemporalInterpolation.exe generate binary files in the byte order of local machine (which is LE for us according to the metadata file generated). So I guess I need to change the byte order for it to be usable in ADL since ADL always expect Big-Endian?

Also the first file generated has an effectivity of 21:45:00 ~ 22:29:59.999999 ( consistent with truth output ), instead of 21:30:00 ~ 22:29:59.9999 , I wonder if that will cause any problem for data between 21:30:00~21:45:00

Yes, ProAncTemporallInterpolation.exe should work. Its use is apparently not described very well from what I can tell.

After talking with another person, here's what I found out:
--It only works in "with metadata" mode.
--You use it by changing the "task details" in the LW file. So you'll need to change $ADL_HOME/cfg/dynamic/withMetadata/ProAncTemporalInterpolationLwFile.xml

Currently the LW contains this:

Code: Select all

  <task>
    <taskType>EDR</taskType>
    <taskDetails1>NCEP-ANC-Int</taskDetails1>
    <taskDetails2>1433008832000000</taskDetails2>
    <taskDetails3>6</taskDetails3>
    <taskDetails4></taskDetails4>
  </task>

You'll have to change items 1, 2, and 3. taskDetails1 is the collection short name. If you're using NCEP data, it's already set correctly. If you're using NOGAPS, you'll need to change it. taskDetails2 is the synoptic time in IET format. taskDetails3 is the leading forecast time.

You can use $ADL_HOME/tools/bin/string2iet.exe to convert to/from IET time and a human readable format. It reads times from STDIN. For example:

Code: Select all

$ADL_HOME/tools/bin > ./string2iet.exe 
Enter the time: 1433008832000000
2003-05-30 18:00:00.000000Z = 1433008832000000 = 1433008832.00000000
Enter the time: 2003-05-30 18:00:00.000000Z
2003-05-30 18:00:00.000000Z = 1433008832000000 = 1433008832.00000000
Enter the time: 2011-10-27 00:00:00.000000Z
2011-10-27 00:00:00.000000Z = 1698364834000000 = 1698364834.00000000
Enter the time: $ADL_HOME/tools/bin > 

ADL provided some test data after it's already been converted to the internal format in $ADL_HOME/data/input/withMetadata/ProAncTemporalInterpolationInputs. Say you wanted to temporally interpolate the 6 and 9 hr forecasts for NCEP (that's what is already configured).
1. Set taskDetails1 to NCEP-ANC-Int
2. View 481b520e-d624a-9b9deab9-e3c64ef9.asc and 481b5231-626ad-9b9deab9-e3bf137f.asc and note that the SynopticTime is "2003-05-30 18:00:00.000000".
3. Use $ADL_HOME/tools/bin/string2iet.exe to convert "2003-05-30 18:00:00.000000" to IET time. Note that you have to add a trailing "Z" (for Zulu time) when you provide the time to string2iet.exe.

Code: Select all

$ADL_HOME/tools/bin > echo 2003-05-30 18:00:00.000000Z | ./string2iet.exe 
Enter the time: 2003-05-30 18:00:00.000000Z = 1433008832000000 = 1433008832.00000000
Enter the time: $ADL_HOME/tools/bin >

4. Add 1433008832000000 to taskDetails2
5. Of the 2 forecast times you're interpolating (6 and 9 in this case), add the first one to taskDetails3. In this case it's "6"

Following the above steps is how

Code: Select all

  <task>
    <taskType>EDR</taskType>
    <taskDetails1>NCEP-ANC-Int</taskDetails1>
    <taskDetails2>1433008832000000</taskDetails2>
    <taskDetails3>6</taskDetails3>
    <taskDetails4></taskDetails4>
  </task>

was constructed.

[kbisanz]

Regarding the byte order, I would recommend adding

Code: Select all

("ByteOrder" STRING EQ "LE")

to the .asc files you're creating. Then the code will know it's little endian and it will not do any byte swapping on your LE platform. If the code can find the "ByteOrder" metadata, it will take the appropriate action (or no action). If it can not find "ByteOrder" it will assume it is BE and do what it needs to do to convert to native endianess. In "with metadata" mode, the assumption is big endian, in "no metadata" mode, the assumption is native endianess if "DataEndianType" not found in the *_CFG.xml file. Note that in "with metadata" mode the ByteOrder metadata is what matters and the DataEndianType in the *_CFG.xml is ignored.

A lot of the provided data is from an IDPS DMS where "ByteOrder" is not present. So that's why it's not present on most of the provided data. However on outputs of ADL algorithms, it should be present. Sorry I forgot to mention you should add ByteOrder in the previous post.

Regarding the 21:45 time, I'm guessing you're trying to use a 3 and 6 hour forecast? The 3 hour forecast has a rule in the either the EDR-IR or EDR-PR where you can't use it until it's 3:45 old. So that's the reason for that effectivity. So data for 21:30:00~21:45:00 will not use the 3 hr forecast. It'll use the 9 hr forecast from the previous synoptic time.

Additionally, I think the ending effectivity on the 24 hour product extends for a week.

[ljiang]

thanks again. It seems I just have too many questions. There is another one for you. In ADL3.1 the OCC algorithm is now contained as algorithm 2 in the OCC controller, which also include ancillary file controller for granulating ancillary files as algorithm 1. So now we should be able to have ungranulated ancillary file as the input such as NCEP blob and terrain tiles instead of the granulated ANC files? However, the test input did not include the NCEP blob (only granulated files) but the OCC controller still ran and produced outputs.

Regarding the byte order, I would recommend adding

Code: Select all

("ByteOrder" STRING EQ "LE")

to the .asc files you're creating. Then the code will know it's little endian and it will not do any byte swapping on your LE platform. If the code can find the "ByteOrder" metadata, it will take the appropriate action (or no action). If it can not find "ByteOrder" it will assume it is BE and do what it needs to do to convert to native endianess. In "with metadata" mode, the assumption is big endian, in "no metadata" mode, the assumption is native endianess if "DataEndianType" not found in the *_CFG.xml file. Note that in "with metadata" mode the ByteOrder metadata is what matters and the DataEndianType in the *_CFG.xml is ignored.

A lot of the provided data is from an IDPS DMS where "ByteOrder" is not present. So that's why it's not present on most of the provided data. However on outputs of ADL algorithms, it should be present. Sorry I forgot to mention you should add ByteOrder in the previous post.

Regarding the 21:45 time, I'm guessing you're trying to use a 3 and 6 hour forecast? The 3 hour forecast has a rule in the either the EDR-IR or EDR-PR where you can't use it until it's 3:45 old. So that's the reason for that effectivity. So data for 21:30:00~21:45:00 will not use the 3 hr forecast. It'll use the 9 hr forecast from the previous synoptic time.

Additionally, I think the ending effectivity on the 24 hour product extends for a week.

[kbisanz]

Config guides (*_CFG.xml files) have something called the "Environment Run Name". There's an entry for both official and substitute mode. Note that substitute mode is not supported in ADL. The entry you want to look for is "EnvironmentRunNameOfficial". That entry tells the name of the executable that the algorithm will run in.

So, if you look at ProEdrViirsOCC_CFG.xml, the EnvironmentRunNameOfficial is "ProEdrViirsOCCController", which is why it runs in the OCC controller. If you look at ProAncViirsController_CFG.xml, the EnvironmentRunNameOfficial is "AlwaysCreate", meaning it (but not its sub algorithms) will always be created. If you look at one of the sub algorithms of the ANC Controller, you'll see an EnvironmentRunNameOfficial that is not ProEdrViirsOCCController. For example, ProAncViirsGranulateSurfPres_CFG.xml lists ProSdrViirsController.

So, the Gran Anc algorithms are really not running in the OCC controller. That's why nothing failed when NCEP data was missing.

There are a couple ways to tell from a debug log if an algorithm is truly being executed:

• --Search for "doProcessing". If an algorithm is running, it will output a DBG_MED message when its doProcessing method executes. For example: "2011/10/28 15:00:30.484 (1106658.1): DBG_MED ProCmnMethodAudit.cpp|123|ProAncViirsGranulateTerrainGeopotentialHeight[ProAncViirsGranulateTerrainGeopotentialHeight]::doProcessing(NPP001212026331) [111f7b350] entered"
  --To know if an algorithm will be executed, search for a DBG_LOW message from ProCmnControllerAlgorithm.cpp containing "will create". For example: "2011/10/28 15:00:21.885 (1106658.1): DBG_LOW ProCmnControllerAlgorithm.cpp|606| will create ProAncViirsGranulateTerrainGeopotentialHeight"
  --To know if an algorithm will not be executed, search for a DBG_LOW message from ProCmnControllerAlgorithm.cpp containing "will not create". For example: "2011/10/28 15:00:22.228 (1106658.1): DBG_LOW ProCmnControllerAlgorithm.cpp|618| will not create ProAncViirsGranulateNhfSurfTemp". For algorithms not created (for example, due to a different EnvironmentRunNameOfficial value), there is also a DBG_HIGH message.

If you want to run Gran Anc in the OCC controller, you should be able to edit ProAncViirs*_CFG.xml to have ProEdrViirsOCCController for the EnvironmentRunNameOfficial. Of course you will need the inputs required by Gran Anc.

In the operational system, VIIRS Gran Anc runs in the VIIRS SDR controller.

[ljiang]

Thanks.

So it means an algorithm can only be run in the controller (or itself) determined by it's environmental run name, unless it's 'AlwaysCreate' in which case the algorithm will run in any controller as long as it is one of the algorithms in the controller's configuration file.

So if I change the environmental run name of all the granulating algorithms (such as ProAncViirsGranulateSurfPres_CFG.xml) to "AlwaysCreate" then it should run in OCC controller, correct?

[kbisanz]

Yes, your understanding of the environment run name is correct.

I *believe* you can use AlwaysCreate in any algorithm, not just a controller.

So yes, you should be able to change them all to AlwaysCreate and they should work. If not, you'll have to change them to a specific name. I know that will work.

In the operational system, we only want an algorithm to run at a single given point, so that's why you don't see the AlwaysCreate much.

[ljiang]

Regarding the 21:45 time, I'm guessing you're trying to use a 3 and 6 hour forecast? The 3 hour forecast has a rule in the either the EDR-IR or EDR-PR where you can't use it until it's 3:45 old. So that's the reason for that effectivity. So data for 21:30:00~21:45:00 will not use the 3 hr forecast. It'll use the 9 hr forecast from the previous synoptic time.

Additionally, I think the ending effectivity on the 24 hour product extends for a week.

Hi Kevin, do you know where I can find detailed rules about which NCEP forecast files to use to generate a NCEP ADL blob? or can you confirm if my following understanding is correct or not?


23:30-00:30: 6hr forecast from 18:00Z
00:30-01:30: file1 generated from interpolate between 6hr and 9hr forecast from 18:00Z
01:30-02:30: file2 generated from interpolate between 6hr and 9hr forecast from 18:00Z
02:30-02:45: 9hr forecast from 18:00Z
02:45-03:30: 3hr forecast from 00:00Z
03:30-04:30: file1 generated from interpolate between 3hr and 6hr forecast from 00:00Z
04:30-05:30: file2 generated from interpolate between 3hr and 6hr forecast from 00:00Z
05:30-06:30: 6hr forecast from 00:00Z
06:30-07:30: file1 generated from interpolate between 6hr and 9hr forecast from 00:00Z
07:30-08:30: file2 generated from interpolate between 6hr and 9hr forecast from 00:00Z
09:30-09:45: 9hr forecast from 00:00Z
09:45-10:30: 3hr forecast from 06:00Z
....

The similar thing repeat every 6 hrs and only 3hr, 6hr, and 9hr forecast are used.