SSEC forums

Hi,

I am trying to run a granule on 01/24 around t12:53 between t12:52 and t12:55. I gtp the following NCEP ancillary data from GRAVITE.
But I feel very lost with so many files. I believe I should pick up something like between 12HR and 15HR, but at each 12HR, there are four files at 0,6,12 and 18z.

How do I decide which one to pick for the metacreater, data coverter and data interpolation?

And even after a conversion and interpolation to hourly internal binary format, what do we suppose to granulize the data to generate the inputs for ADL to ingest? I believe ADL3.1 only take granulized Ancillary data inputs, am I right?

Moreover, the granulized anc inputs are: each file for each variable, for example, ozone or precipitable water etc., will that variable separation be handled in the granulization process?

Thanks!!!

jhuang% gtp list between ts "2012-01-24 00:00:00" "2012-01-24 24:00:00" type=off_NPP | grep NCEP

72482470 off_NCEP-GFS-15HR-ANC_GFS_NCEP_015f_20120124_201201241800Z_20120124214246Z_ee20120125120000Z_np.h5 f51608ff836adad68d3bc87ef9007cd1 15060063
72481749 off_NCEP-GFS-21HR-ANC_GFS_NCEP_021f_20120124_201201241800Z_20120124214533Z_ee20120125180000Z_np.h5 d049384c3bba682f857e4f741c529cf1 15172108
72481743 off_NCEP-GFS-18HR-ANC_GFS_NCEP_018f_20120124_201201241800Z_20120124214410Z_ee20120125150000Z_np.h5 03272fc1a6c4811cb0a1b88e6dfc369d 15126836
72475323 off_NCEP-GFS-12HR-ANC_GFS_NCEP_012f_20120124_201201241800Z_20120124214123Z_ee20120125090000Z_np.h5 7afcafeb705be33b03b9a1b81a0d5d04 15065045
72475319 off_NCEP-GFS-06HR-ANC_GFS_NCEP_006f_20120124_201201241800Z_20120124213622Z_ee20120125030000Z_np.h5 6a9ee158f5ee950da2f28d374d326778 14834678
72475310 off_NCEP-GFS-09HR-ANC_GFS_NCEP_009f_20120124_201201241800Z_20120124213745Z_ee20120125060000Z_np.h5 9a7e64ecc7ce4af5230fd39fb16f0ab5 14995619
72471854 off_NCEP-GFS-24HR-ANC_GFS_NCEP_024f_20120124_201201241800Z_20120124214658Z_ee20120131180000Z_np.h5 6c91631a29405fb827762d63ad727fc3 15201957
72471847 off_NCEP-GFS-03HR-ANC_GFS_NCEP_003f_20120124_201201241800Z_20120124213132Z_ee20120125000000Z_np.h5 f0bd889a5def23862333d456657cc0f4 15152255
72415157 off_NCEP-GFS-15HR-ANC_GFS_NCEP_015f_20120124_201201241200Z_20120124154156Z_ee20120125060000Z_np.h5 f4633acd71c657f5eed480e38a44233b 15048998
72412627 off_NCEP-GFS-06HR-ANC_GFS_NCEP_006f_20120124_201201241200Z_20120124153747Z_ee20120124210000Z_np.h5 0c2accf26f38e8c04b5d1cd8484af5af 15031000
72412625 off_NCEP-GFS-12HR-ANC_GFS_NCEP_012f_20120124_201201241200Z_20120124154033Z_ee20120125030000Z_np.h5 ef6a41519d8fe2a96c88fb40551ab843 15005239
72412624 off_NCEP-GFS-09HR-ANC_GFS_NCEP_009f_20120124_201201241200Z_20120124153910Z_ee20120125000000Z_np.h5 cd2f67de23d1b8bbccb2f57ed01653b9 15043662
72412621 off_NCEP-GFS-18HR-ANC_GFS_NCEP_018f_20120124_201201241200Z_20120124154320Z_ee20120125090000Z_np.h5 d845e4cca20cc138c8d25f431ad55cb3 15092164
72412620 off_NCEP-GFS-21HR-ANC_GFS_NCEP_021f_20120124_201201241200Z_20120124154443Z_ee20120125120000Z_np.h5 868dcdf01e0293e7c086b12203da3dd0 15143874
72412617 off_NCEP-GFS-03HR-ANC_GFS_NCEP_003f_20120124_201201241200Z_20120124153624Z_ee20120124180000Z_np.h5 f0f4f9409f03026bdc86b923309c3d1e 15244482
72412615 off_NCEP-GFS-24HR-ANC_GFS_NCEP_024f_20120124_201201241200Z_20120124154607Z_ee20120131120000Z_np.h5 0a47ceb2b21275a16d66eca9d2008958 15186396
72357466 off_NCEP-GFS-24HR-ANC_GFS_NCEP_024f_20120124_201201240600Z_20120124094624Z_ee20120131060000Z_np.h5 404a7bd079c65dc8653c0bcdbb6e4379 15142622
72357445 off_NCEP-GFS-15HR-ANC_GFS_NCEP_015f_20120124_201201240600Z_20120124094124Z_ee20120125000000Z_np.h5 aa68f3902e927b1e480333b13998c47c 15158062
72356511 off_NCEP-GFS-21HR-ANC_GFS_NCEP_021f_20120124_201201240600Z_20120124094410Z_ee20120125060000Z_np.h5 8852ba994046c3f8634a3e7056a1f439 15123024
72356504 off_NCEP-GFS-12HR-ANC_GFS_NCEP_012f_20120124_201201240600Z_20120124093910Z_ee20120124210000Z_np.h5 d7208dd74fef24dfa94c57b0b76e5754 15210127
72356502 off_NCEP-GFS-18HR-ANC_GFS_NCEP_018f_20120124_201201240600Z_20120124094247Z_ee20120125030000Z_np.h5 e7dc06fdc1e19f935eca08e4c8155d41 15099581
72356497 off_NCEP-GFS-09HR-ANC_GFS_NCEP_009f_20120124_201201240600Z_20120124093746Z_ee20120124180000Z_np.h5 5ada0c123a5d338578e88a65f90a7f70 15129654
72356495 off_NCEP-GFS-06HR-ANC_GFS_NCEP_006f_20120124_201201240600Z_20120124093623Z_ee20120124150000Z_np.h5 cd6170c356ef3e687950f3fad9faf30d 15016017
72356483 off_NCEP-GFS-03HR-ANC_GFS_NCEP_003f_20120124_201201240600Z_20120124093231Z_ee20120124120000Z_np.h5 1bdf249935824c870bbf599d4dd3b637 15212868
72295697 off_NCEP-GFS-24HR-ANC_GFS_NCEP_024f_20120124_201201240000Z_20120124034701Z_ee20120131000000Z_np.h5 07feefda34cd3b10314da18726f40128 15240429
72292386 off_NCEP-GFS-21HR-ANC_GFS_NCEP_021f_20120124_201201240000Z_20120124034537Z_ee20120125000000Z_np.h5 574a9a063acb61520a2e797e60dbd425 15334394
72291247 off_NCEP-GFS-18HR-ANC_GFS_NCEP_018f_20120124_201201240000Z_20120124034412Z_ee20120124210000Z_np.h5 950e395eb00e6b19a4cd5a7ad6930f3c 15309349
72291241 off_NCEP-GFS-12HR-ANC_GFS_NCEP_012f_20120124_201201240000Z_20120124034124Z_ee20120124150000Z_np.h5 c8c1e2b234623be77fb74fd3eb310f08 15193773
72291237 off_NCEP-GFS-09HR-ANC_GFS_NCEP_009f_20120124_201201240000Z_20120124033747Z_ee20120124120000Z_np.h5 9729e880b2d3b7923c851fcf34574905 15188928
72288812 off_NCEP-GFS-06HR-ANC_GFS_NCEP_006f_20120124_201201240000Z_20120124033623Z_ee20120124090000Z_np.h5 96adbc9808670a8834a15c87f1ba34d8 15068990
72288807 off_NCEP-GFS-03HR-ANC_GFS_NCEP_003f_20120124_201201240000Z_20120124033124Z_ee20120124060000Z_np.h5 89f1f5857f7857dc7333be24e9a4ed2d 15280181

jhuang% gtp list between ts "2012-01-24 00:00:00" "2012-01-24 24:00:00" type=off_NPP | grep NCEP-GFS-12HR
72475323 off_NCEP-GFS-12HR-ANC_GFS_NCEP_012f_20120124_201201241800Z_20120124214123Z_ee20120125090000Z_np.h5 7afcafeb705be33b03b9a1b81a0d5d04 15065045
72412625 off_NCEP-GFS-12HR-ANC_GFS_NCEP_012f_20120124_201201241200Z_20120124154033Z_ee20120125030000Z_np.h5 ef6a41519d8fe2a96c88fb40551ab843 15005239
72356504 off_NCEP-GFS-12HR-ANC_GFS_NCEP_012f_20120124_201201240600Z_20120124093910Z_ee20120124210000Z_np.h5 d7208dd74fef24dfa94c57b0b76e5754 15210127
72291241 off_NCEP-GFS-12HR-ANC_GFS_NCEP_012f_20120124_201201240000Z_20120124034124Z_ee20120124150000Z_np.h5 c8c1e2b234623be77fb74fd3eb310f08 15193773

Hi,

If you can download an EDR which you know used ancillary data, you can look at its N_Anc_Filename metadata and it will tell which file(s) were used. For example:
("N_Anc_Filename" STRING EQ "off_NCEP-GFS-06HR-ANC_GFS_NCEP_006f_20030125_200301250000Z_20030125000000Z_ee20070125000000Z_np")
("N_Anc_Filename" STRING EQ "off_NCEP-GFS-09HR-ANC_GFS_NCEP_009f_20030125_200301250000Z_20030125000000Z_ee20070125000000Z_np")

In the above example, the 3 and 6 hour ones are shown because the algorithm used either a 4 or 5 hour temporal interpolated forecast (which was created from the deliverable 3 and 6 hour forecasts). If a 6 hour forecast was used, only the off_NCEP-GFS-06HR-ANC_GFS_NCEP_* file have been shown.

EDR algorithms in ADL 3.1 and the upcoming 4.0 require granulated ancillary data. This is produced in the VIIRS SDR Controller. The VIIRS SDR Controller requires "internally" format NCEP products, which are different from off_NCEP-GFS-*HR-ANC_*_np.h5. In ADL 3.1 you must use external tools (like the DynAncMetaCreator, DynAncConverter, and DynAncInterpolater) to create such internal files. In ADL 4.0 that process will get easier as ADL will have code to ingest and convert the native files.

The data is granulated by the ProSdrViirsController.exe process. Inside of $ADL_HOME/cfg/ProSdrViirsController_CFG.xml, you can see it runs several controllers, one of which is ProAncViirsController. Inside of ProAncViirsController_CFG.xml, you can see it runs numerous algorithms which chunk up the data into granule sized pieces for each field.

The easiest way to run the granulation process without the pain of running the entire SDR process is to tweak ProSdrViirsController_CFG.xml as described in User Manual Part 2 section 5.9.7 (found in $ADL_HOME/doc). However, you will need a geolocation file for that granule which will not be produced from ProSdrViirsController_CFG.xml if you configured only ProAncViirsController to run. You might be able to find the geo on CLASS.

Does the above info help?

Thanks a lot, Kevin, Your reply definitely helped a lot!

In your case, I have seen that both 06HR and 09HR have 200301250000Z in the filename.

("N_Anc_Filename" STRING EQ "off_NCEP-GFS-06HR-ANC_GFS_NCEP_006f_20030125_200301250000Z_20030125000000Z_ee20070125000000Z_np")
("N_Anc_Filename" STRING EQ "off_NCEP-GFS-09HR-ANC_GFS_NCEP_009f_20030125_200301250000Z_20030125000000Z_ee20070125000000Z_np")

In my case, my question was, I am having 200301250000Z, 200301250600Z, 200301251200Z, 200301251800Z four files, does it matter at all? or should we just stay with 200301250000Z and forget the other three? What is their (these four files) relationship to the 06HR or 09HR? I got very confused with the naming convention here obviously.

I think somehow I figured out the conversion and interpolation tools already, which means I have those 'internally' binary files already that can serve as inputs to the ProSdrViirsController.exe process. My quick question is, if I only have these 'internal' binary files and geo file, but not any RDR data in the input folder, then I tweak the ProSdrViirsController_CFG.xml to make it only run the granulation process, will it work? Does a RDR data have to be there or not in such case? Or a geo file will be sufficient?

Thanks a lot again!

According to the CDFCB-X Volume 1, table 3.4.5-1 "Ancillary File-Naming Convention Description", the fields (separated by _) have the following meanings. I'm using off_NCEP-GFS-15HR-ANC_GFS_NCEP_015f_20120124_201201241800Z_20120124214246Z_ee20120125120000Z_np.h5 as an example

off: File type; "off" is official ancillary data file
NCEP-GFS-15HR-ANC: Collection short name
GFS: Model used to generate this file
NCEP: Producer
015f: Forecast time
20120124: Production date; "This field identifies the date that the data was released. Provides year, month, and day – this information is defined by the specific delivered file."
201201241800Z: Synoptic Time (UTC); "This field identifies the model run’s synoptic (collection) time for the data within the file. The field is in years, months, days, hours, and minutes and is appended with a ‘Z’. The field implies the data effectivity start time for products other than model output."
20120124214246Z: Version; "This field provides a timestamp representing the time of update/receive. The field is in years, months, days, hours, minutes, and seconds and is appended with a ‘Z’."
ee20120125120000Z: Effectivity stop; "The field indicates the effectivity stop time by giving the year, month, day, hour, minutes, and seconds and is appended with a ‘Z’. The timestamp is prepended with ‘ee’ (effectivity end). If the stop time does not apply to a particular file, then ee00000000000000Z should be used."
np: Program ID; np is NPOESS
h5: File extension

It would be best to use the file specified in the metadata because that's what the operational code used.

If you have the internal binary file and the geo (VIIRS-MOD-RGEO-TC or VIIRS-MOD-GEO-TC) you should be ok for most of them. ProAncViirsGranulateNitrateDepletion requires Nitr-Depl-Temp-ANC. ProAncViirsGranulateTerrainGeopotentialHeight_CFG.xml and ProAncViirsGranulateBathymetry_CFG.xml both require Terrain-Eco-ANC-Tile. If you don't care about those, you can edit ProAncViirsController_CFG.xml to not execute those. If you do care about them, you'll need to have access to the Terrain-Eco-ANC-Tile data for the tile IDs specified in the N_Anc_Filename of an EDR for that granule. RDR data should not be needed.

Thank you very very much, Kevin,
I am testing it out.
It seems I can use the example inputs and selected only Anc Granulation subalgorithm and do the granulation for the example inputs (well, all the files are there).
I wonder however:
I only supplied the code with one NCEP file in binary format, not all the cross-granule 3 files, not all the NCEP, NAAPS, PW files etc, and I supplied it with the binary GEO file as well, it did not work out.
The NCEP file and the GEO file do match in terms of the information from the EDR meta data.
The question is, do I have to provide the module with all the cross-granule files, and all the necessary Ancillary files (sufficient for SDR algorithm), so that the sub-algorithm can run?
Or I can run the sub-algorithm with whatever ancillary file in the input folder as long as it matches the GEO file?
******
More notes after some more testing:
1) I only turned on the Anc granulation subalgorithm;
2) I tested with the example inputs provided;
3) Besides GEO file, the following files seems needed:
a) *Terrain-Eco-ANC-Tile* ($ADL_HOME/data/tiles/Terrain-Eco-ANC-Tile --- Global coverage data is available on disk 3 of the ADL 3.1 delivery in adlTerEcoDataLinks.tgz
https://jpss.ssec.wisc.edu/adl/download ... aLinks.tgz)
b) *LUT* (static inputs?)
c) *NCEP-ANC-Int* (Outputs from DynAncInterpolator)
d) *Static* (static inputs?)
e) *USNO-PolarWonder* (available on GRAVITE, CLASS)
f) *CMNGEO* (where to get those?)
Am I right? If so, where we could get these files? Thanks!
******
Sorry for more questions! I believe we are almost there.

Thanks!

-- Jingfeng

I am having the same question as Jingfeng, although currently I am using the static inputs coming with the ADL3.1 package.

Finally, I think I got it working this Monday morning. What a waste of weekend~

Although it seems obvious, it is worthwhile to mention that changes of the following are the key:
Environment Variable:
NPP_GRANULE_ID_BASETIME
In LW files:
DataStartIET
DataEndIET
N_Granule_ID

And for VIIRS-SDR Granulation Sub-algorithm, make sure you have the following files in your input folder:
*USNO* (Polar Wander)
*Terrain-Eco-Anc-Tile* (I tried to use multiple input folder because all the tile files are actually installed with the package, I separated the two input folders with ':', but it did not work out -- more insightful help are appreciated!)
*CMNGEO*
*Grid-SIN-Tiles*
*NCEP-ANC-Int*
*MOD-GEO-TC*
*Grid-To-Gran*
*Off_Nitr*
*Off-Planet*

I'm glad to hear you got it working.

The NPP_GRANULE_ID_BASETIME is *very* important. For the staged data, you need to have it set to 1300968033000000. For real NPP data, you need to use 1698019234000000.

To run an algorithm, you should need only what's in the corresponding *_CFG.xml file. The only exception is that for algorithms which require the use of the common geoloation library ("Cmn Geo"), you need to also have the inputs listed in PRO_CMN_GEO_CFG.xml. I *believe* that none of the Gran Anc algorithms require Cmn Geo. Of course since you're running a series of Gran Anc algorithms, it requires the inputs listed in each running algorithm's *_CFG.xml file.

First, I'm a little puzzled that you required the following inputs if you're running *only* the algorithms in ProAncViirsController:
*USNO* (Polar Wander)
*Grid-SIN-Tiles*
*Grid-To-Gran*
*Off-Planet*
*CMNGEO*

Second, it looks like you're running in "no metadata" mode if modifying DataStartIET and DataEndIET made a difference? The framework should only check those tags in no metadata mode. Otherwise it'll use the information from metadata.

Third, using a ":" in the input path (no surrounding spaces) should have worked. You should have referenced either the ".../Terrain-Eco-ANC-Tile/withMetadata/" or ".../Terrain-Eco-ANC-Tile/noMetadata/" directory (with "..." being the full path).

Thanks a lot, Kevin,
Very insightful comments.
I will continue my testing process and see whether those files are 'indeed' needed.
The way I was testing is like, by using the example inputs, if I remove certain files, whether the sub-algorithm will run successfully or not.
It turned out it was not running properly if I removed any of the above types of inputs.

And yes, the way of multiple input paths were actually working.

Third, using a ":" in the input path (no surrounding spaces) should have worked. You should have referenced either the ".../Terrain-Eco-ANC-Tile/withMetadata/" or ".../Terrain-Eco-ANC-Tile/noMetadata/" directory (with "..." being the full path).

Thanks!

I was running 'with metadata'. And you are right that my latest test indicated that we don't have to change DataStartIET and DataEndIET, which makes our life a little easier now.

kbisanz wrote: Second, it looks like you're running in "no metadata" mode if modifying DataStartIET and DataEndIET made a difference? The framework should only check those tags in no metadata mode. Otherwise it'll use the information from metadata.