Dear all,
In addition to adding more quality flag, we may also require adding more processing coefficient to the PCT file for the VIIRS aerosol product
Any chance you can guide me though the list of files I need to change in the ADL framework and how we can add more processing coefficients to the PCT file?
Thanks a lot!
Sincerely,
Jingfeng
Adding more processing coefficients in PCT --- VIIRS Aerosol
Re: Adding more processing coefficients in PCT --- VIIRS Aer
JingFeng,
These steps should get you started for modifying the Aerosol PCT input.
1. Update the PRO PCT product file for new fields.
$ADL_HOME/xml/VIIRS/VIIRS_Aeros_EDR_AC_Int.xml (Use EBX unless not on ADL 4.2, otherwise use the old XML Editor tool)
2. Update the DQM XML file to add the new field entries
$ADL_HOME/cfg/dqmtables/NPP/VIIRS/EDR/Aeros/VIIRS-Aeros-EDR-AC_npp_20120826000000Z_20130101000000Z_ee00000000000000Z_PS-1-O-CCR-12-0788-JPSS-DPA-003-PE_noaa_all_all-_all.xml
3. Update the ING coefficient header to add new fields
/vobs/IDPS-CAT3/ADL/CMN/Utilities/ING/include/IngMsdCoefficients_ViirsAerosolStruct.h
4. Make code changes to the ING converter source code similar to other corresponding field changes, for instance look at changes done for moltau field.
$ADL_HOME/CMN/Utilities/ING/MSD/Coefficients/src/IngMsdCoefficients_ViirsAerosolConverter.cpp
5. Update Aerosol algorithm to make use of new configurable parameters, for example look at changes similar to usage of moltau field
$ADL_HOME/EDR/VIIRS/aerosol/src/ProEdrViirsAerosol.cpp
6. Do a full rebuild of ADL from a clean to ensure auto generation software is regenerated with new format definition.
7. Drop new XML file from step 2 into ING Landing Zone and run ING MSD in order to create the new internal binary format for the updated aerosol algorithm to use as the input.
8. Execute the Aerosol algorithm in ADL and ensure results are as expected.
Let me know if you need further assistance.
Thanks,
Bryan Henderson
Raytheon Company
These steps should get you started for modifying the Aerosol PCT input.
1. Update the PRO PCT product file for new fields.
$ADL_HOME/xml/VIIRS/VIIRS_Aeros_EDR_AC_Int.xml (Use EBX unless not on ADL 4.2, otherwise use the old XML Editor tool)
2. Update the DQM XML file to add the new field entries
$ADL_HOME/cfg/dqmtables/NPP/VIIRS/EDR/Aeros/VIIRS-Aeros-EDR-AC_npp_20120826000000Z_20130101000000Z_ee00000000000000Z_PS-1-O-CCR-12-0788-JPSS-DPA-003-PE_noaa_all_all-_all.xml
3. Update the ING coefficient header to add new fields
/vobs/IDPS-CAT3/ADL/CMN/Utilities/ING/include/IngMsdCoefficients_ViirsAerosolStruct.h
4. Make code changes to the ING converter source code similar to other corresponding field changes, for instance look at changes done for moltau field.
$ADL_HOME/CMN/Utilities/ING/MSD/Coefficients/src/IngMsdCoefficients_ViirsAerosolConverter.cpp
5. Update Aerosol algorithm to make use of new configurable parameters, for example look at changes similar to usage of moltau field
$ADL_HOME/EDR/VIIRS/aerosol/src/ProEdrViirsAerosol.cpp
6. Do a full rebuild of ADL from a clean to ensure auto generation software is regenerated with new format definition.
7. Drop new XML file from step 2 into ING Landing Zone and run ING MSD in order to create the new internal binary format for the updated aerosol algorithm to use as the input.
8. Execute the Aerosol algorithm in ADL and ensure results are as expected.
Let me know if you need further assistance.
Thanks,
Bryan Henderson
Raytheon Company
Re: Adding more processing coefficients in PCT --- VIIRS Aer
Thank you very much, Bryan,
I will give it a try.
Have a nice day!
Jingfeng
I will give it a try.
Have a nice day!
Jingfeng
Re: Adding more processing coefficients in PCT --- VIIRS Aer
Hi Bryan,
Any chance you can elaborate a little more about the Step 7?
For the *h5 ancillary data, I understand we need to unpack it to MSD file, then sort MSD data to landing zone by using:
$ADL_HOME/script/sortMsdIntoLz.pl
then run the conversion
$ADL_HOME/script/runMsd.pl -m
However, I never tried before with XML file. Any kind advices in details?
Thanks!
Jingfeng
Any chance you can elaborate a little more about the Step 7?
For the *h5 ancillary data, I understand we need to unpack it to MSD file, then sort MSD data to landing zone by using:
$ADL_HOME/script/sortMsdIntoLz.pl
then run the conversion
$ADL_HOME/script/runMsd.pl -m
However, I never tried before with XML file. Any kind advices in details?
Thanks!
Jingfeng
bhenders wrote: 7. Drop new XML file from step 2 into ING Landing Zone and run ING MSD in order to create the new internal binary format for the updated aerosol algorithm to use as the input.
Re: Adding more processing coefficients in PCT --- VIIRS Aer
The idea is that just like with dynamic ancillary data, Ingest will convert from an external format to an internal format that the algorithm is expecting.
The file edited in step 2, $ADL_HOME/cfg/dqmtables/NPP/VIIRS/EDR/Aeros/VIIRS-Aeros-EDR-AC_npp_20120826000000Z_20130101000000Z_ee00000000000000Z_PS-1-O-CCR-12-0788-JPSS-DPA-003-PE_noaa_all_all-_all.xml is the external format of the processing coefficients. I am assuming you updated values and/or added new values to that file.
In step 4, you need to update the Ingest code to handle any new fields that were added to the external XML file.
In step 7, you use Ingest to create a new internal binary file that the algorithm can use.
You need to do the other steps too, I'm just pointing out the ones involving Ingest.
The file edited in step 2, $ADL_HOME/cfg/dqmtables/NPP/VIIRS/EDR/Aeros/VIIRS-Aeros-EDR-AC_npp_20120826000000Z_20130101000000Z_ee00000000000000Z_PS-1-O-CCR-12-0788-JPSS-DPA-003-PE_noaa_all_all-_all.xml is the external format of the processing coefficients. I am assuming you updated values and/or added new values to that file.
In step 4, you need to update the Ingest code to handle any new fields that were added to the external XML file.
In step 7, you use Ingest to create a new internal binary file that the algorithm can use.
You need to do the other steps too, I'm just pointing out the ones involving Ingest.
Kevin Bisanz
Raytheon Company
Raytheon Company
Re: Adding more processing coefficients in PCT --- VIIRS Aer
Thank you, Kevin,
I think I understand the general idea about the conversion, similar to what we did for the Ancillary data.
I would like to know more details about the commands used for the Step 7:
1. How to drop XML file to the land zone? Is it by $ADL_HOME/script/sortMsdIntoLz.pl? It does not look that way.
2. How to Ingest/convert? Is it $ADL_HOME/script/runMsd.pl -m? Not likely maybe. I checked the folders in the $ADL_MSD_LZ_PATH, but I did not find a particular folder designated for XML files. It has ANC, AUX, Temp and TILED only.
Any more details please?
Thank you, very much!
Jingfeng
I think I understand the general idea about the conversion, similar to what we did for the Ancillary data.
I would like to know more details about the commands used for the Step 7:
1. How to drop XML file to the land zone? Is it by $ADL_HOME/script/sortMsdIntoLz.pl? It does not look that way.
2. How to Ingest/convert? Is it $ADL_HOME/script/runMsd.pl -m? Not likely maybe. I checked the folders in the $ADL_MSD_LZ_PATH, but I did not find a particular folder designated for XML files. It has ANC, AUX, Temp and TILED only.
Any more details please?
Thank you, very much!
Jingfeng
Re: Adding more processing coefficients in PCT --- VIIRS Aer
Jingfeng,
See my responses for your questions below.
1. How to drop XML file to the land zone? Is it by $ADL_HOME/script/sortMsdIntoLz.pl? It does not look that way.
You would use the $ADL_HOME/script/sortMsdIntoLz.pl to sort the updated VIIRS Aerosol PCT XML file into the correct landing zone directory. It will place the file into the AUX Landing Zone folder area.
2. How to Ingest/convert? Is it $ADL_HOME/script/runMsd.pl -m? Not likely maybe. I checked the folders in the $ADL_MSD_LZ_PATH, but I did not find a particular folder designated for XML files. It has ANC, AUX, Temp and TILED only.
The sortMsdIntoLz.pl script will move the XML file to the AUX subdirectory (the parsed XML file becomes an AUX-Int internal data item), then you should run $ADL_HOME/script/runMsd.pl -m which will parse the XML file and create the internal binary file that PRO uses as input. The outputs will then be placed in ${ADL_HOME}/data/output/withMetadata/IngMsdOutputs by default. One would then either need to include this directory in the aerosol input path or move the files placed here to the normal input directory for VIIRS Aerosol.
Thanks,
Bryan Henderson
Raytheon Company
See my responses for your questions below.
1. How to drop XML file to the land zone? Is it by $ADL_HOME/script/sortMsdIntoLz.pl? It does not look that way.
You would use the $ADL_HOME/script/sortMsdIntoLz.pl to sort the updated VIIRS Aerosol PCT XML file into the correct landing zone directory. It will place the file into the AUX Landing Zone folder area.
2. How to Ingest/convert? Is it $ADL_HOME/script/runMsd.pl -m? Not likely maybe. I checked the folders in the $ADL_MSD_LZ_PATH, but I did not find a particular folder designated for XML files. It has ANC, AUX, Temp and TILED only.
The sortMsdIntoLz.pl script will move the XML file to the AUX subdirectory (the parsed XML file becomes an AUX-Int internal data item), then you should run $ADL_HOME/script/runMsd.pl -m which will parse the XML file and create the internal binary file that PRO uses as input. The outputs will then be placed in ${ADL_HOME}/data/output/withMetadata/IngMsdOutputs by default. One would then either need to include this directory in the aerosol input path or move the files placed here to the normal input directory for VIIRS Aerosol.
Thanks,
Bryan Henderson
Raytheon Company
Re: Adding more processing coefficients in PCT --- VIIRS Aer
I see, Bryan, thanks! I did not realize AUX is for XML, but now I know. I will give it a try 