Page 1 of 1

DQTT Questions

Posted: Fri Nov 09, 2012 11:25 am
by mark.tolman
I'm using ADL 4.1 to test turning on ATMS SDR DQTT. I've edited the xml file located in $ADL_HOME/cfg/dqmtables/NPP/ATMS/SDR/SDR to set the thresholds and make the tests active. I've found RDR granules that should trigger the DQTT, but I'm not certain how to verify that it worked. There doesn't appear the be anything in the resulting SDR hdf5 output.

How do I know if the tests would result in a DQ Notification? Is there something in the logs I should be looking for or some application that needs to be running?

Thanks
Mark Tolman
MIT Lincoln Laboratory


If you give someone a program, you will frustrate them for a day;
if you teach them how to program, you will frustrate them for a lifetime.
- Anonymous

Re: DQTT Questions

Posted: Sun Nov 11, 2012 9:46 pm
by bhenders
Mark,

Editing the xml file located in $ADL_HOME/cfg/dqmtables/NPP/ATMS/SDR/SDR is the first step on your way to modifying the DQTT used by the ATMS SDR. However, that XML file is not read directly by the ATMS SDR algorithm but rather the algorithm wants an internal binary file that is created by the Ingest Mission Support Data process from reading and converting the XML file to binary.

After editing the file you should run the following script:

Run the sortMsdIntoLz.pl script on the modified xml file to move it to the landing zone.

> ${ADL_HOME}/script/sortMsdIntoLz.pl -c $ADL_HOME/cfg/dqmtables/NPPATMS/SDR/SDR /*.xml
> ls $ADL_MSD_LZ_PATH/*

Then Run the runMsd.pl script to convert the XML file into the Internal binary format.
> ${ADL_HOME}/script/runMsd.pl -m

The internal binary file for the DQTT should then be in:

> ls ${ADL_HOME}/data/output/withMetadata/IngMsdOutputs/

Move the files created here to your ATMS SDR input directory, so the algorithm will have the updated DQTT as an input.

If the test actually triggers a data quality notification, a ATMS-SDR-DQN product should be produced as an output. If you grep the log file, you should also find a debug message "created a DQN product. Test Id's" indicating that a DQN was triggered.

Thanks,

Bryan Henderson
Raytheon Company

Re: DQTT Questions

Posted: Tue Nov 13, 2012 9:27 am
by mark.tolman
The resulting DQN product appears to be a binary file. How does one read it?

Also, is there some documentation on this, so I can RTFM?

Re: DQTT Questions

Posted: Tue Nov 13, 2012 2:36 pm
by bhenders
The DQN product is a binary output, as you guessed. The actual format can be found in the DPIS ICD Volume 3 section 3.3.5.8, or one can see the format of the structure by opening $ADL_HOME/xml/VIIRS/DQN_INFO.xml with a browser so that the XML file is displayed with the style sheet. I've uploaded a screen shot which shows the summary of the format. It's basically made up of 3 "granule" level fields, then followed by 15 records which are slots for the individual tests that are triggered.

I was going to upload a simple program that we have in development to convert them to ASCII, however it seems I can't upload C++ files to this site. Therefore I'll just paste the code in here which you can paste into a file such as parseDQN.cpp and compile it. I'm guessing this is basically what you are looking for. If not post back.

It should compile with g++ parseDQN.cpp -I $ADL_HOME/include

Bryan Henderson
Raytheon Company
---------------------------------------------------------------------------------------------------------------------------------------------------

#include <cstring>
#include <stdio.h>
#include <cstdlib>
#include <iostream>
#include <fstream>
#include <sstream>
#include <vector>

using namespace std;

#include <ProCmnQuality.h>

int main(int argc, char** argv)
{
ProCmnDQNHeader dqn;
ProCmnDQNRecord dqnRecord;

if (argc != 2)
{
cout << "Usage : " << argv[0] << " <DQNfile>" << endl;
exit (-1);
}

string fileName(argv[1]);

ifstream inputFile(fileName.data(), ios::binary | ios::in);
if (!inputFile.good())
{
if (!inputFile.is_open())
{
cerr << "Unable to open input file : " << fileName.data() << endl;
}
else
{
cerr << "Errors with input file " << fileName.data();
cerr << ", state=" << inputFile.rdstate() << endl;
}
return(-1);
}


// go the beginning of the file and read in the data
inputFile.seekg(0, ios::beg);
inputFile.read((char *) &dqn, sizeof(dqn));

if (dqn.testCount <= 0)
{
cerr << "Invalid number of tests (" << dqn.testCount << ") in dqn: " << fileName.c_str() << endl;
inputFile.close();
return (-1);
}

cout << fileName.data() << " has " << dqn.testCount << " test entries" << endl;

for (int i = 0; i < dqn.testCount; i++)
{

memset((void *) &(dqnRecord), 0, sizeof(dqnRecord));

inputFile.read((char *) &(dqnRecord), sizeof(dqnRecord));

cout << "------------------------------------------" << endl;
cout << "Test ID : " << dqnRecord.testID << endl;
cout << "Minimum Value : " << dqnRecord.valueMin << endl;
cout << "Maximum Value : " << dqnRecord.valueMax << endl;
cout << "Thresholds Count : " << dqnRecord.thresholdTestCount << endl;
cout << "Thresholds Result: " << dqnRecord.thresholdTestResults << endl;
cout << "Severity Level : " << dqnRecord.severityLevel << endl;
cout << "DQN Information : " << dqnRecord.dqnInfo << endl;

cout << "The version of the DQN : " << dqn.dqttVersion << endl;

}

inputFile.close();

return(1);
}

Re: DQTT Questions

Posted: Thu Nov 15, 2012 11:48 am
by mark.tolman
Thank you. The code you provided worked well.

Once I get the hdf5 Packer to stop core dumping I'll be able to test it on multiple granules. I've applied a patch that fixed the Unpacker, so I'm making progress. :D

Mark Tolman
MIT Lincoln Laboratory