DQTT Questions

Issues related to the ATMS SDR algorithm and data
Post Reply
mark.tolman
Posts: 12
Joined: Thu Nov 08, 2012 7:41 am
Location: Lexington, MA USA

DQTT Questions

Post by mark.tolman »

I'm using ADL 4.1 to test turning on ATMS SDR DQTT. I've edited the xml file located in $ADL_HOME/cfg/dqmtables/NPP/ATMS/SDR/SDR to set the thresholds and make the tests active. I've found RDR granules that should trigger the DQTT, but I'm not certain how to verify that it worked. There doesn't appear the be anything in the resulting SDR hdf5 output.

How do I know if the tests would result in a DQ Notification? Is there something in the logs I should be looking for or some application that needs to be running?

Thanks
Mark Tolman
MIT Lincoln Laboratory


If you give someone a program, you will frustrate them for a day;
if you teach them how to program, you will frustrate them for a lifetime.
- Anonymous
bhenders
Posts: 72
Joined: Wed Jan 05, 2011 9:27 am
Location: Omaha, NE

Re: DQTT Questions

Post by bhenders »

Mark,

Editing the xml file located in $ADL_HOME/cfg/dqmtables/NPP/ATMS/SDR/SDR is the first step on your way to modifying the DQTT used by the ATMS SDR. However, that XML file is not read directly by the ATMS SDR algorithm but rather the algorithm wants an internal binary file that is created by the Ingest Mission Support Data process from reading and converting the XML file to binary.

After editing the file you should run the following script:

Run the sortMsdIntoLz.pl script on the modified xml file to move it to the landing zone.

> ${ADL_HOME}/script/sortMsdIntoLz.pl -c $ADL_HOME/cfg/dqmtables/NPPATMS/SDR/SDR /*.xml
> ls $ADL_MSD_LZ_PATH/*

Then Run the runMsd.pl script to convert the XML file into the Internal binary format.
> ${ADL_HOME}/script/runMsd.pl -m

The internal binary file for the DQTT should then be in:

> ls ${ADL_HOME}/data/output/withMetadata/IngMsdOutputs/

Move the files created here to your ATMS SDR input directory, so the algorithm will have the updated DQTT as an input.

If the test actually triggers a data quality notification, a ATMS-SDR-DQN product should be produced as an output. If you grep the log file, you should also find a debug message "created a DQN product. Test Id's" indicating that a DQN was triggered.

Thanks,

Bryan Henderson
Raytheon Company
mark.tolman
Posts: 12
Joined: Thu Nov 08, 2012 7:41 am
Location: Lexington, MA USA

Re: DQTT Questions

Post by mark.tolman »

The resulting DQN product appears to be a binary file. How does one read it?

Also, is there some documentation on this, so I can RTFM?
bhenders
Posts: 72
Joined: Wed Jan 05, 2011 9:27 am
Location: Omaha, NE

Re: DQTT Questions

Post by bhenders »

The DQN product is a binary output, as you guessed. The actual format can be found in the DPIS ICD Volume 3 section 3.3.5.8, or one can see the format of the structure by opening $ADL_HOME/xml/VIIRS/DQN_INFO.xml with a browser so that the XML file is displayed with the style sheet. I've uploaded a screen shot which shows the summary of the format. It's basically made up of 3 "granule" level fields, then followed by 15 records which are slots for the individual tests that are triggered.

I was going to upload a simple program that we have in development to convert them to ASCII, however it seems I can't upload C++ files to this site. Therefore I'll just paste the code in here which you can paste into a file such as parseDQN.cpp and compile it. I'm guessing this is basically what you are looking for. If not post back.

It should compile with g++ parseDQN.cpp -I $ADL_HOME/include

Bryan Henderson
Raytheon Company
---------------------------------------------------------------------------------------------------------------------------------------------------

#include <cstring>
#include <stdio.h>
#include <cstdlib>
#include <iostream>
#include <fstream>
#include <sstream>
#include <vector>

using namespace std;

#include <ProCmnQuality.h>

int main(int argc, char** argv)
{
ProCmnDQNHeader dqn;
ProCmnDQNRecord dqnRecord;

if (argc != 2)
{
cout << "Usage : " << argv[0] << " <DQNfile>" << endl;
exit (-1);
}

string fileName(argv[1]);

ifstream inputFile(fileName.data(), ios::binary | ios::in);
if (!inputFile.good())
{
if (!inputFile.is_open())
{
cerr << "Unable to open input file : " << fileName.data() << endl;
}
else
{
cerr << "Errors with input file " << fileName.data();
cerr << ", state=" << inputFile.rdstate() << endl;
}
return(-1);
}


// go the beginning of the file and read in the data
inputFile.seekg(0, ios::beg);
inputFile.read((char *) &dqn, sizeof(dqn));

if (dqn.testCount <= 0)
{
cerr << "Invalid number of tests (" << dqn.testCount << ") in dqn: " << fileName.c_str() << endl;
inputFile.close();
return (-1);
}

cout << fileName.data() << " has " << dqn.testCount << " test entries" << endl;

for (int i = 0; i < dqn.testCount; i++)
{

memset((void *) &(dqnRecord), 0, sizeof(dqnRecord));

inputFile.read((char *) &(dqnRecord), sizeof(dqnRecord));

cout << "------------------------------------------" << endl;
cout << "Test ID : " << dqnRecord.testID << endl;
cout << "Minimum Value : " << dqnRecord.valueMin << endl;
cout << "Maximum Value : " << dqnRecord.valueMax << endl;
cout << "Thresholds Count : " << dqnRecord.thresholdTestCount << endl;
cout << "Thresholds Result: " << dqnRecord.thresholdTestResults << endl;
cout << "Severity Level : " << dqnRecord.severityLevel << endl;
cout << "DQN Information : " << dqnRecord.dqnInfo << endl;

cout << "The version of the DQN : " << dqn.dqttVersion << endl;

}

inputFile.close();

return(1);
}
Attachments
DQN.PNG
DQN.PNG (19.27 KiB) Viewed 17860 times
mark.tolman
Posts: 12
Joined: Thu Nov 08, 2012 7:41 am
Location: Lexington, MA USA

Re: DQTT Questions

Post by mark.tolman »

Thank you. The code you provided worked well.

Once I get the hdf5 Packer to stop core dumping I'll be able to test it on multiple granules. I've applied a patch that fixed the Unpacker, so I'm making progress. :D

Mark Tolman
MIT Lincoln Laboratory
Post Reply