VIIRS AF algrthm - how to run just fires, not entire PRO chn

[Bwind]

Hello,

For my Active Fires Science Team's needs, we just want to run the VIIRS Active Fires mask (EDR) algorithm on operational data for Cal/Val. That's all.

To this end, I have built ADL from source as well as, separately, installed the science appliance in fusion 4 on my Mac. I also downloaded and ran all of the 'dvd EDR test data', then figured out the trick for just Fires of running just

ProEdrViirsMasksController.exe cfg/dynamic/withMetadata/ProEdrViirsMasksControllerLwFile.xml

and, better still, doing the same but removing group "Algorithm_1" (which was 'ProAncViirsController', which I established empirically from runs with the test data that I did not need) from cfg/ProEdrViirsMasksController_CFG.xml
and promoting what had been the second group 'ProEdrViirsActiveFires' to be the new 'Algorithm_1' group in the xml, and (per UGpt2) 'x'-ing out the rest with "Algorithm_Nx" for their respective group name's (all of this so that I could run *only* the AF algorithm as opposed to other spurious algorithms as well)

and finally, best of all, doing the same but using only the set of test data that I actually needed for the job, which I was able to establish -- as well as confirm by moving only those test data files actually required to my fusion installation from the other installation where I had downloaded and ran with all 'dvd EDR test data' -- consisted of the following file unique data file types (2 'cross-granule' scans were required and the actual file names long with 'URID' prefix and std. fname conventions hence I list them here only by unique type per the xml CFG both for brevity and clarity) :

"ActiveFires_Thresholds" : OfficialShortName 'VIIRS-AF-EDR-AC-Int'
"GIP_QSTLWM_GRAN" : 'VIIRS-GridIP-VIIRS-Qst-Lwm-Mod-Gran'
"Moderate_Band16" : 'VIIRS-M16-F/SDR'
"Moderate_Band15" : 'VIIRS-M15-F/SDR'
"Moderate_Band13" : 'VIIRS-M13-SDR'
"Moderate_Band11" : 'VIIRS-M11-F/SDR'
"Moderate_Band7" : 'VIIRS-M7-F/SDR'
"Moderate_Band5" : 'VIIRS-M5-F/SDR'
"GridRowCol_Mod" : 'VIIRS-MOD-GRC'
"Geolocation_Mod_TC" : 'VIIRS-MOD-R/GEO-TC'

So far so good. I was able to faithfully re-generate everywhere the same identical fires mask from the minimal required set of provided test data from the 'dvd EDR test data'.

Now the problem is, it seems that's all that I can do. I can't run any other granules for Cal/Val from operational data. Unless I want to run the entire PRO chain from RDR (which we definitely do not per line 1 of this e-mail). So it would seem that all of this effort was for naught. Because:

a) although I have guesses as to how I might run a new granule, I don't have the complete recipe for doing so (do I get the *.h5 data and convert it to blob and simply update the granule ID in the *Lw*xml; but what about the URID prefix on the static, non-granule-specific VIIRS-AF-EDR-AC-Int AF Thresholds file - does the code magically have the "smarts" just to find whichever file this is under the inputs folder)

b) perhaps worst of all, unless this can be Rectified in CLASS and/or in the code asap, the following two input types from the required list above do not appear to be available to our group anywhere after 1.5 weeks now and several of us scouring the landscape looking for where to locate these data outside of the ADL provided 'dvd EDR test data' (or, I gather, from running the entire PRO chain from RDR which we have neither the resources nor particular interest in doing for just our Active Fires EDR Cal/Val):

"GIP_QSTLWM_GRAN" : 'VIIRS-GridIP-VIIRS-Qst-Lwm-Mod-Gran'
"GridRowCol_Mod" : 'VIIRS-MOD-GRC'


So you see this has seemed like quite a lot of effort for nothing unless there is a simple solution or three to these Issues that we are just missing.

Thanks sincerely for advising as to how best to proceed towards the essential goal of line 1 of this e-mail (!) ,

Brad

[kbisanz]

The short answer is that it's not going to be as easy as you had hoped to run the VIIRS Active Fires EDR.

First, I would like to say I'm glad to hear you were able to configure things to only run the VIIRS Active Fires EDR. I would not say your effort was for naught because you will need to do similar things to get the data for the Active Fires EDR.

Let me see if I can answer some of your questions.

You asked this:

a) although I have guesses as to how I might run a new granule, I don't have the complete recipe for doing so (do I get the *.h5 data and convert it to blob and simply update the granule ID in the *Lw*xml; but what about the URID prefix on the static, non-granule-specific VIIRS-AF-EDR-AC-Int AF Thresholds file - does the code magically have the "smarts" just to find whichever file this is under the inputs folder)

You are correct about how you'd run an algorithm for a given granule:
1. Retrieve the appropriate data from CLASS (see below for a common pitfall)
2. Extract from *.h5 (see below for a common pitfall)
3. Create/update the appropriate *Lw*.xml file
4. Run the algorithm(s)

The URID prefix should not cause any problems. The code doesn't use the URID for much, other than to keep file names unique. Inside of the file URID.asc is all the metadata for the corresponding URID.<collection short name> binary file. The code uses the metadata information in the .asc to determine the file to retrieve.

A common pitfall for item 1 above is that sometimes you need to retrieve more granules than you'd initially expect. This is due to "cross granules", where for a given algorithm to produce granule ID X, it needs inputs from granules X-1, X, and X+1 (the previous, current, and next granules). There are multiple instances of "cross granules" in the VIIRS chain, which you will need to partially run. I would recommend given this set of slides a read: https://jpss.ssec.wisc.edu/adl/download ... _Chain.pdf . If you don't have an account, you can get one here: https://jpss.ssec.wisc.edu/adl/

While the slides talk about running CrIMSS, slides 3-7 still generally apply to your case. Looking at the picture on slide 7, you can see Active Fires (AF) in the picture. The Masks Controller (which is Active Fires and Cloud Mask) requires a previous, current, and next granule of VIIRS SDR. Besides the chart, you can tell this from the algorithm config guide. In ProEdrViirsCM_CFG.xml is an entry for VIIRS-M1-FSDR, with a "Crossgranule_Scans" entry of 5. Note the 5 is the number of scans (not granules) required on each side of the primary granule. Because a granule is 48 scans (and 5 is smaller than 48), the algorithm will use the last 5 scans of the previous granule, all 48 scans of the current granule, and the first 5 scans of the next granule. You can then see that in the chart that the SDR requires a previous, current, and next granule of the RDR. This is verified by looking at Crossgranule_Scans for VIIRS-SCIENCE-RDR in ProSdrViirsCal_CFG.xml. This point of this paragraph is that in order to run a given algorithm in a chain, you man need to run other granules for algorithms prior in the chain.

A common pitfall of item 2 above is that sometimes the data you require is present in the h5 file, but is not in the format the algorithm expects. This can be a tricky problem to solve if you're running ADL 3.1, which I suspect you are. If you're running ADL 3.1, and need to update an algorithm coefficient file (such as VIIRS-AF-EDR-AC-Int), the easiest (though still not very easy) way is to compare the external version from the h5 file (Collection Short Name = VIIRS-AF-EDR-AC) (which will be XML) with the XML version you have. If you don't have an XML version, you can get it from Common CM. Then, once you know the differences, you can use the binary editor at $ADL_HOME/script/bined to edit the binary internal format and apply the same changes. This process is much easier in ADL 4.0: You just drop the XML file from the h5 into a directory, run the Ingest executable, and an internally formatted binary comes out.

After looking closely at your situation, I think you *might* be partly in luck. You need to run the VIIRS SDR controller to create VIIRS-GridIP-VIIRS-Qst-Lwm-Mod-Gran and VIIRS-MOD-GRC. VIIRS-GridIP-VIIRS-Qst-Lwm-Mod-Gran is created by the algorithm using ProGipViirsGridToGranQstLwm_CFG.xml, which requires a mapping IP, which requires geolocation. The VIIRS-MOD-GRC is produced by the geolocation algorithm.

Assuming you're just running Active Fires and not Cloud Mask, I *think* can you generate VIIRS-GridIP-VIIRS-Qst-Lwm-Mod-Gran and VIIRS-MOD-GRC by running ProSdrViirsController.exe for only the granule for which you're trying to run Active Fires. You'll want to edit ProSdrViirsController_CFG.xml and disable execution of certain algorithms, using the "Algorithm_Nx" trick you've already done. You want to disable ProSdrViirsCal (you don't need to run this in your case if you're getting SDRs from CLASS. This would introduce cross granule requirements), ProGipViirsNNImg (the output is not needed in your case), ProGipViirsWCalcGrid (the output is not needed in your case), and ProAncViirsController (the outputs are not needed in your case). So you should be running ProSdrViirsVerifiedRdr, ProSdrViirsGeo, ProGipViirsWCalcGran, and ProGipViirsGridToGranController from the VIIRS SDR Controller. Then in ProEdrViirsMasksController_CFG.xml you also want to disable ProEdrViirsCM from the Mask Controller. So, if you want to produce granule X for Active Fires, you first need to run granule X through the VIIRS SDR Controller. I *believe* the disabled algorithms remove any cross granule requirements. Note, I have not tested the configuration I just described, so there is a possibility that I missed something.

I agree that the process of getting appropriate data for an algorithm can be very painful at times. ADL really isn't in a position to be able to solve parts of that issue. Those decisions go to whoever decided what's delivered to CLASS (and how much disk space they want to purchase). ADL 4.0 does provide Ingest capability, which makes it much easier to use externally formatted ancillary/auxiliary data.

I've thrown a lot of information out here and probably forgot to describe some details, so please post back if you have any questions or issues.

[kbisanz]

I need to correct something I said in my earlier post. The statement I said about "I *believe* the disabled algorithms remove any cross granule requirements." is not correct. Active fires itself is cross granule (look for "Crossgranule_Scans" in ProEdrViirsActiveFires_CFG.xml). So, when running the modified VIIRS SDR controller, you'll need to run a previous, current, and next granule so that Active Fires has surrounding granules of the GRC and QSTLWM.

I apologize for any confusion.

[Bwind]

Hi Kevin,

It sounds like based on your reply that we are looking for a service to provide at least on-demand running of the Sdr process and saving off to the side, at least temporarily, the intermediate data files required for active fires. This to satisfy the input data requirements of the ADL Active Fires algorithm.

I really appreciate the detailed reply. It is fascinating to understand the structure of all of the upstream chain. Truly, I mean that.

However, here is the essential problem:

I can't run the entire chain. We don't have the time or direct contact with someone inside to help us at the level of your e-mail ongoing. My group has already told me no, not from RDR, as fascinating an exercise as that might be for me/us to undertake. And I'm not saying if time were no object, I wouldn't go do it. It's simply outside of my job.

The way this needs to work for it to be interactive and not have someone from Raytheon appointed to lias'ing our science team multiple times daily (lest that is an option??), is the disk space to cache whatever inputs, temporary/intermediate what have you, are necessary for a small albeit important product like Active Fires to run needs to be purchased and installed. We need to be able to order up what we need to run just like we order the EDR's themselves from CLASS. This component of the ADL platform is not an afterthought. It is every bit as signifcant as the ADL software itself. Disk space is cheap compared to man hours, which are also physically limited. It is beyond inefficient to have every scientist and engineer on every science team have to learn every detail of every other algorithm upstream in the system, even at the level of understanding the LUTs and configurations of each component etc.

For true Cal/Val of our product, we need to have access to inputs for Active Fires for ADL in an official sense; not "we hope we're doing this right" and crossing our fingers, as alluring as it is to be able to pull off these technical feats few others can do. No, inputs need to be identical to ops, really, to be useful for our activities. That, or the software itself needs to change somehow to lessen substantially the burden of running so we're processing in as near to ops fashion as humanly possible. These e-mails, while helpful, are as I understand from reading your words, "best guesses." It can be put simply like this:

When we run MODIS science granules for testing, it is never the case we run from Level 0.

We would never get any work done! (which to us means science and algorithm development and hand-off to integration, btw.)

I understand all of the issues that you raise about the Cross-granule-scans. I actually read every page of the ADL user's manual, parts one and two. Basically, this scans issue is in effect but a multiplier of an already heavy data+processing burden. It addresses the need to run multiple granules at a particular step (e.g., the EDR fires algorithm) in the chain. Our algorithm b/c of the way it works, yes, requires additional granules (two, in point of fact, one on each side of "the granule" in the middle). I know this from parsing the xml visually. Unfortunately, the requirement of the cross-granules also multiplies the data and processing burden at *every other step* in the chain as well. This quickly gets out of hand as the next paragraph will document.

Now, coming back to the main take away of this e-mail, that we have to focus on what we're responsible for I also know it can easily take one night of labor (take this past Friday for me) to find available data from CLASS or a like-subscription for *just* running fires. There are a couple dozen files one must wrangle (and this assuming all data is available--a big assumption lately) just in terms of sensor data just to try to run one granule (well, okay 3 if to count the cross-granule scans, but that's the minimum one can run, yes) of our one simple algorithm. To do the same for the upstream Sdr processes and all of their dependencies, as attested to by the length and detail of your e-mail and a few like it from others I have seen recently, is simply beyond our scope and capabilities at the Edr level of analysis. Similarly, you mentioned Common CM. That tool tried to download ~10Gb of stuff to our server's hard drive when I set it off early one afternoon to get the PRO folder and hours later I had to kill the job b/c I had to go home. I find all of the activity fascinating, however in that 10Gb of data what is not only what I need but manageable to me is but a handful of code files <1Mb. So the tool is not only overkill, it's simply unusable to us at least as-provided. I share this b/c it's somewhat symptomatic of the similar issues with ADL. More thought must be given to our actually using these tools for Cal/Val, I'm afraid, or there are simply alternatives that are necessary because we don't have all of the time in the world.

I think perhaps in the case of Active Fires Mask we are just "unlucky" as the case may be. Because others do not have the burden of having to run the Sdr just to run the Edr??

I'll repeat the call for help from the top of this e-mail: We are looking for a service to provide at least on-demand running of the Sdr process and saving off to the side, at least temporarily, the intermediate data files required for active fires. This to satisfy the input data requirements of the ADL Active Fires algorithm.

Very best regards and sincerely thank you for the help and insight into the current version of the software and data system,
I hope this exchange is helpful,

Brad

[Bwind]

Re. the non-granule-specific VIIRS-AF-EDR-AC-Int AF Thresholds file - does the code magically have the "smarts" just to find whichever file this is under the inputs folder)

> You are correct about how you'd run an algorithm for a given granule:
> 1. Retrieve the appropriate data from CLASS (see below for a common pitfall)
> 2. Extract from *.h5 (see below for a common pitfall)
> 3. Create/update the appropriate *Lw*.xml file
> 4. Run the algorithm(s)

> The URID prefix should not cause any problems. The code doesn't use the URID for much, other than to keep file names unique. Inside of the file URID.asc is all the metadata for the corresponding URID.<collection short name> binary file. The code uses the metadata information in the .asc to determine the file to retrieve.

Ok, but *both* *.asc and VIIRS-AF-EDR-AC-Int files are marked with the (time-stamp -based) uniqueness-enforcing URID. So how then does the code "find" the approrpriate static thresholds table in the inputs folder?? What if there are two? ...

Please advise if you can, as we will likely still need to know this one even if we find assistance running the Sdr's up the chain.

It is not clear at all to me (and I would suspect based on experiences so far with the software that it would fail as soon as I try to run a different granule) how to make sure the inputs folder contains the appropriate static '*Int' table in this case. (URID_GEnerator.exe -- or I forget the name -- seems not to accept any args and only keys off of the current system time to generate the id which obviously wouldn't match the rest of the granule data.)

[kbisanz]

Regarding how the code has the smarts to find the correct file in the input folder:

If you look in the AF input directory, you'll see a pair of files like this:

Code: Select all

4f1dbf9d-a9ef4-0a4f180b-59772e4c.VIIRS-AF-EDR-AC_npp_20020101010000Z_20020101010000Z_ee00000000000000Z_1_devl_dev_noaa_ops.xml_Internal
4f1dbf9d-a9ef4-0a4f180b-59772e4c.asc

The URID (the random part on the front) may be different for you. Those files are associated with each other because they have the same URID. The file 4f1dbf9d-a9ef4-0a4f180b-59772e4c.VIIRS-AF-EDR-AC_npp_20020101010000Z_20020101010000Z_ee00000000000000Z_1_devl_dev_noaa_ops.xml_Internal contains the binary data. The file
4f1dbf9d-a9ef4-0a4f180b-59772e4c.asc contains the "metadata" which controls how files are queried for. Here is an example of what one contains:

Code: Select all

("4f1dbf9d-a9ef4-0a4f180b-59772e4c" UR "2012-01-23 20:14:21.000000"
 ("FILELIST" COLLECTION
  ("/runtime_PRO/tjensen/oracle/data//DmSm//shared_1.5.06.P/23_20/4f1dbf9d-a9ef4-0a4f180b-59772e4c.VIIRS-AF-EDR-AC_npp_20020101010000Z_20020101010000Z_ee00000000000000Z_1_devl_dev_noaa_ops.xml_Internal" FILE
   ("METADATALIST" COLLECTION)
  )
 )
 ("METADATALIST" COLLECTION
  ("DatasetLock" INTEGER EQ 0)
  ("Effectivity" DATETIMERANGE EQ "2002-01-01 01:00:00.000000" "2029-12-31 23:59:58.000000")
  ("N_Collection_Short_Name" STRING EQ "VIIRS-AF-EDR-AC-Int")
  ("RangeDateTime" DATETIMERANGE EQ "2002-01-01 01:00:00.000000" "2002-01-01 01:00:00.000000")
  ("UpdateDateTime" DATETIME EQ "2012-01-23 20:14:21.723214")
  ("CreationDateTime" DATETIME EQ "2012-01-23 20:14:21.696554")
  ("Platform_Short_Name" STRING EQ "NPP")
  ("N_Dataset_Type_Tag" STRING EQ "AUX-Int")
  ("Mission_Name" STRING EQ "NPP")
  ("N_Dataset_Source" STRING EQ "devl")
  ("N_Processing_Domain" STRING EQ "ops")
  ("SourceDomain" STRING EQ "dev")
  ("AuxFilename" GROUP EQ
   ("METADATALIST" COLLECTION
    ("N_Aux_Filename" STRING EQ "VIIRS-AF-EDR-AC_npp_20020101010000Z_20020101010000Z_ee00000000000000Z_1_devl_dev_noaa_ops")
   )
  )
  ("N_Software_Version" STRING EQ "I1.5.06.00")
 )
)

At the top of the file, is a long directory path. The directory info doesn't matter (the path is the location on our system), but the file name portion is important. When the code reads the above .asc file, it assumes there is a file named 4f1dbf9d-a9ef4-0a4f180b-59772e4c.VIIRS-AF-EDR-AC_npp_20020101010000Z_20020101010000Z_ee00000000000000Z_1_devl_dev_noaa_ops.xml_Internal in the same directory as the .asc file.

Another piece of metadata in the file is the Effectivity:

Code: Select all

("Effectivity" DATETIMERANGE EQ "2002-01-01 01:00:00.000000" "2029-12-31 23:59:58.000000")

When the code queries for files with a N_Collection_Short_Name of VIIRS-AF-EDR-AC-Int, it sorts the files by Effectivity with the latest starting time first.

For example, say you're running granule ID X, which is for June 05, 2012. Also say you had some files for VIIRS-AF-EDR-AC-Int, created on January 1, 2, and 3 of 2012 and you also had a file you wanted to start using on July 1, 2012. You would have 4 file pairs (the binary and metadata files) for VIIRS-AF-EDR-AC-Int in your input directory. They would have Effectivities similar to this:

Code: Select all

("Effectivity" DATETIMERANGE EQ "2012-01-01 01:00:00.000000" "2029-12-31 23:59:58.000000")
("Effectivity" DATETIMERANGE EQ "2012-01-02 01:00:00.000000" "2029-12-31 23:59:58.000000")
("Effectivity" DATETIMERANGE EQ "2012-01-03 01:00:00.000000" "2029-12-31 23:59:58.000000")
("Effectivity" DATETIMERANGE EQ "2012-07-01 01:00:00.000000" "2029-12-31 23:59:58.000000")

When the query is constructed, it would first look at the granule ID that AF is processing and convert that to a time (June 05, 2012). When the code queried for VIIRS-AF-EDR-AC-Int, it would find files effective on June 05, 2012. That would be the 3 files with start Effectivity in January. It would then look at the Effectivity start time and return the one with the latest starting time, which would be the "2012-01-03 01:00:00.000000" file. So it would be the "2012-01-03 01:00:00.000000" file returned from the query. Once a granule on July 1 or later was processed, the "2012-07-01 01:00:00.000000" file would be used.

[Bwind]

That last post about which thresholds table ADL uses was very informative and helpful, both. Thanks!

I gather from it that we're good to run whichever table (as long as one is present in the designated 'inputs' folder), once we "clear" the two other obstacles in terms of missing input data files (GridRowCol and QstLw).

PS -- I did also have a look at the bined the other week, so I saw how to edit the binary version of this xml in case we need to. Thx again!

[kbisanz]

Regarding your comments about some data not being available from CLASS:
I understand your frustration and actually agree with you on most points. Unfortunately, the ADL team is not in a position to change what data is or is not delivered from the operational system to CLASS. Paul Meade is the person you want to contact if you believe something should be changed. He doesn't have the power to make the changes himself, but he's the appropriate person to get the ball rolling with other NASA people.

Regarding your problem about VIIRS-GridIP-VIIRS-Qst-Lwm-Mod-Gran and VIIRS-MOD-GRC for active fires. I've looked into a work around for you that requires minimal extra work and minimal extra runtime.

The workaround is to configure the algorithm which generates the QST LWM data to run in the AF EDR process if desired. However, we need 3 granules of QST LWM for each AF EDR, so we can't run them at the same time. So, in addition to the "normal" AF EDR process which executes Active Fires, we'll make an alternate configuration which executes only the QST LWM related code (and not AF). Then depending on which execution configuration file ("LW file") is provided to ProEdrViirsMasksController.exe, it will execute either the QST LWM granulation or the AF EDR.

It's not perfect, but I think it's the best there is in the near term.

First, we'll deal with the requirement of VIIRS-MOD-GRC. The reason this is needed is that something called "Common Adjacency" requires that. Common Adjacency can be configured to run with geolocation, which you ready have. To do that, perform the following 2 changes in $ADL_HOME/cfg:
1. In PRO_CMN_ADJ_CFG.xml, near the top of the file, change this
<configValue>GROWCOL</configValue>
to this
<configValue>GEO</configValue>
2. From ProEdrViirsActiveFires_CFG.xml, remove the code that requires VIIRS-MOD-GRC: Starting with the line "<group name="GridRowCol_Mod">" remove the ~21 lines though the corresponding ending </group> tag.

Active fires should now run without the requirement of VIIRS-MOD-GRC. Common Adjacency can use geolocation (VIIRS-MOD-GEO-TC), but prefers the VIIRS-MOD-GRC because it has some precomputed information to make things faster. FYI, in ADL 4.0, VIIRS-MOD-GRC-TC is used so it's in sync with the terrain correct geolocation (VIIRS-MOD-GEO-TC).

After this change, you need to go to $ADL_HOME/EDR/VIIRS/land/ActiveFires/src and execute "make clean library". The reason is that we've removed an input so we need to rebuild the algorithm's auto generated code (AutoGeneratedProEdrViirsActiveFires.cpp).

Unfortunately, we can't just substitute another product for VIIRS-GridIP-VIIRS-Qst-Lwm-Mod-Gran. However, with some configuration we can generate it without running the rest of the SDR. (Though you'll still need to run it 3 times to run a single Active Fires granule.)

First, in $ADL_HOME/cfg, edit ProGipViirsGridToGranQstLwm_CFG.xml and ProGipViirsWCalcGran_CFG.xml so that the EnvironmentRunNameOfficial value near the top of the file is ProEdrViirsMasksController instead of ProSdrViirsController. EnvironmentRunNameOfficial controls where an algorithm is executed. The change will allow the algorithms to execute in ProEdrViirsMasksController. Note if you ever run the VIIRS SDR, you'll need to revert those changes.

Then edit ProEdrViirsMasksController_CFG.xml so that only ProEdrViirsActiveFires is executed. I think you've already done this.

Now, in $ADL_HOME/cfg, 2 new files should be created:
1. Copy ProEdrViirsMasksController_GuideList.cfg to GridToGran_ProEdrViirsMasksController_GuideList.cfg and add the following lines at the bottom of GridToGran_ProEdrViirsMasksController_GuideList.cfg:
# Files for running GridToGran
ProGipViirsWCalcGran_CFG.xml
ProGipViirsGridToGranController_CFG.xml
ProGipViirsGridToGranQstLwm_CFG.xml

Also in GridToGran_ProEdrViirsMasksController_GuideList.cfg, remove "ProEdrViirsMasksController_CFG.xml" and replace it with "GridToGran_ProEdrViirsMasksController_CFG.xml".

2. Copy ProEdrViirsMasksController_CFG.xml to GridToGran_ProEdrViirsMasksController_CFG.xml then do the following to GridToGran_ProEdrViirsMasksController_CFG.xml :
A. Delete Algorithm_3 and Algorithm_4
B. Change Algorithm_1 to be ProGipViirsWCalcGran
C. Change Algorithm_2 to be ProGipViirsGridToGranController
D. Ensure Algorithm_1 and Algorithm_2 do not have any "x" on them.

Now, you will need to create an additional execution configuration files (*LwFile.xml).
1. In $ADL_HOME/cfg/dynamic/withMetadata, copy ProEdrViirsMasksControllerLwFile.xml to GridToGran_ProEdrViirsMasksControllerLwFile.xml
2. In GridToGran_ProEdrViirsMasksControllerLwFile.xml, change the <configGuideName> to be GridToGran_ProEdrViirsMasksController_GuideList.cfg (which we created above)
3. If you don't already have it, download the QST LWM global tile set. You can get it from https://jpss.ssec.wisc.edu/adl/download ... aLinks.tgz Unfortunately all tiles are packaged together, so you will get more than just QSTLWM tiles. After the download you can delete everything that's not a QST LWM file.
4. Add the path to the QST LWM tiles to GridToGran_ProEdrViirsMasksControllerLwFile.xml: To <inputPath>, add the location of the QSTLWM global tile database location. Multiple paths are separated with colon, just like $PATH on Linux.
5. To the <inputPath> of GridToGran_ProEdrViirsMasksControllerLwFile.xml, add the location of the data you've downloaded for the AF EDR. We'll need the geolocation data that's been downloaded.
6. In GridToGran_ProEdrViirsMasksControllerLwFile.xml, configure an <outputPath> entry

In $ADL_HOME/cfg/dynamic/withMetadata/, modify the existing ProEdrViirsMasksControllerLwFile.xml
1. To <inputPath>, add the output directory from GridToGran_ProEdrViirsMasksControllerLwFile.xml. This way when you generate the VIIRS-GridIP-VIIRS-Qst-Lwm-Mod-Gran product AF will be able to find it.

Now we're done with the major configuration.

To run AF for a given granule, you first must generate the VIIRS-GridIP-VIIRS-Qst-Lwm-Mod-Gran for the previous, current, and next granule. For example, I'll use the staged data, which is granule NPP001212016942.

You can determine the previous and next granule in a couple of ways:
1. Run AF and let it fail and see what granule ID it fails for.
2. Use the granule ID utility:
Get the start time for the current granule using the granule ID utility. Ensure the environment variable INFUTIL_CFGDOMAIN is set to $ADL_HOME/cfg/GuideList.txt. There was an error in ADL 3.1 where that was not set. Add it to $ADL_HOME/build/envSetup.*sh if it doesn't exist. Also ensure that NPP_GRANULE_ID_BASETIME is set to 1300968033000000 if you're using the provided test data or to 1698019234000000 if you're using real NPP data.

Here we run the granule ID utility at $ADL_HOME/tools/bin/granID.exe and provide it the granule ID of the AF granule we want to execute. My $ADL_HOME is /testdata/kbisanz/afTest/ADL and "/testdata/kbisanz/afTest/ADL/tools/bin > " is my shell prompt.

Code: Select all

/testdata/kbisanz/afTest/ADL/tools/bin > ./granID.exe 
Sensor = VIIRS
   PSN = VIIRS-SCIENCE
Enter the time or VIIRS Granule ID: NPP001212016942
 Input VIIRS Granule ID = NPP001212016942
Output VIIRS Granule ID = NPP001212016942
Start: 1422169727250000 = 2003-01-25 07:08:15.250000Z
  End: 1422169812600000 = 2003-01-25 07:09:40.600000Z

Enter the time or VIIRS Granule ID: /testdata/kbisanz/afTest/ADL/tools/bin > 
/testdata/kbisanz/afTest/ADL/tools/bin > 

The start time of the granule is 1422169727250000. That is the number of microseconds since January 1, 1958. This is IET time. Now look at INF_CFG.xml and get the value of NPP_VIIRS-SCIENCE_GRANULE_SIZE (85350000). This is the size of a VIIRS granule. Now add 1422169727250000 + 85350000 = 1422169812600000. This is the start time of the next granule. Now provide 1422169812600000 to granID.exe to get the next granule (NPP001212017796). For example:

Code: Select all

/testdata/kbisanz/afTest/ADL/tools/bin > ./granID.exe 
Sensor = VIIRS
   PSN = VIIRS-SCIENCE
Enter the time or VIIRS Granule ID: 1422169812600000
Input Time: 1422169812600000 = 2003-01-25 07:09:40.600000Z
Output VIIRS Granule ID = NPP001212017796
Start: 1422169812600000 = 2003-01-25 07:09:40.600000Z
  End: 1422169897950000 = 2003-01-25 07:11:05.950000Z

Enter the time or VIIRS Granule ID: /testdata/kbisanz/afTest/ADL/tools/bin > 
/testdata/kbisanz/afTest/ADL/tools/bin > 

Now take 1422169727250000 - 85350000 = 1422169641900000 and provide that to granID.exe to get the previous granule (NPP001212016089). For example:

Code: Select all

/testdata/kbisanz/afTest/ADL/tools/bin > ./granID.exe 
Sensor = VIIRS
   PSN = VIIRS-SCIENCE
Enter the time or VIIRS Granule ID: 1422169641900000
Input Time: 1422169641900000 = 2003-01-25 07:06:49.900000Z
Output VIIRS Granule ID = NPP001212016089
Start: 1422169641900000 = 2003-01-25 07:06:49.900000Z
  End: 1422169727250000 = 2003-01-25 07:08:15.250000Z

Enter the time or VIIRS Granule ID: /testdata/kbisanz/afTest/ADL/tools/bin > 
/testdata/kbisanz/afTest/ADL/tools/bin > 

So, to run AF for NPP001212016942, you need to granulate QST LWM data for 3 granules:
Previous granule = NPP001212016089
Current granule = NPP001212016942
Next granule = NPP001212017796

The file GridToGran_ProEdrViirsMasksControllerLwFile.xml is the execution configuration file that will run the granulation of QST LWM, but not run Active Fires. In GridToGran_ProEdrViirsMasksControllerLwFile.xml, the <configGuideName> entry references GridToGran_ProEdrViirsMasksController_GuideList.cfg, which contains a reference to the new/modified GridToGran_ProEdrViirsMasksController_CFG.xml. That file says to run the GridToGran related algorithms we've configured.

To create the 3 granules of QST LWM, you now need to run "$ADL_HOME/bin/ProEdrViirsMasksController.exe $ADL_HOME/cfg/dynamic/withMetadata/GridToGran_ProEdrViirsMasksControllerLwFile.xml" 3 times, changing <taskDetails1> to be each of NPP001212016089, NPP001212016942, and NPP001212017796.

To execute Active Fires (finally!) you can run "$ADL_HOME/bin/ProEdrViirsMasksController.exe $ADL_HOME/cfg/dynamic/withMetadata/ProEdrViirsMasksControllerLwFile.xml" with <taskDetails1> set to NPP001212016942.

When you want to run AF for the next granule (NPP001212017796), you will first need to generate QSTLWM data for NPP001212016942, NPP001212017796, and NPP001212018649. Data for NPP001212016942 and NPP001212017796 should already exist from the previous run.

Following the above steps should allow you to run Active Fires.

If you provide me an email address, I could send you a copy of my files for reference.

[kbisanz]

The ADL team recently held a design review for ADL 4.1, which is the version of ADL currently in development. One of the topics was the problem of EDRs requiring ANC and GIP data that is not currently delivered to CLASS. Because it's very difficult and time consuming to change what is delivered to CLASS, the solution proposed was to make the EDRs smart enough to granulate their own ANC/GIP data.

The user would still need to download the required native ANC files (such as NCEP 3 hour forecasts) and ingest them to an internal format. However, after that, the EDR algorithm (actually something that runs in the same executable prior to the "science" part of the algorithm) would be smart enough to determine the missing ANC/GIP data and granulate it, including cross granule requirements.

I wanted to post back which this information because it should make this issue less of a problem. You will still need to get the external data, but that is delivered to CLASS.

[Bwind]

For the time being we're still using ADL 3.1, although the 4.x improvements to ease running of the AF EDR sound encouraging.

Thank you for the detailed reply one month ago. I got pulled away from monitoring this message board and since had another person or two trying to help me with similar approaches as you described in your detailed reply both to: i) generate the Qstlwm; and ii) obviate the necessity of the GRC input. So far, I am still waiting for further fixes to the processes suggested by my help and so am still stuck on i). ii) sounds like it should work now that you independently verified a similar approach as to what was suggested to me. So the Qstlwm generation remains the proverbial roadblock at the moment.

If you provide me an email address, I could send you a copy of my files for reference.

Perhaps the additional light you have been able to shed for me with your further detail will give me the insight I need to fix my installation for generation of the Qstlwm so that hopefully I can finally use the ADL to run just the Active Fires EDR. It could help to have your configuration files etc. as you suggest.

Generally I don't tend to put my e-mail address on message boards, but here is the link to my department where you can find it (your profile here Kevin nor the Raytheon site seemed to have yours -- I spent some time looking just now -- or I would have just e-mailed directly, sorry).

I'm Brad Wind and my e-mail address is on this web page. I hope to hear from you soon, Kevin. Thanks again,

http://www.geog.umd.edu/staff

Brad