missing / fill data in VIIRS RDRs

Issues related to the VIIRS SDR algorithm and data
Post Reply
graemem
Posts: 14
Joined: Tue Jan 11, 2011 5:44 pm

missing / fill data in VIIRS RDRs

Post by graemem »

This email was forwarded to me and seems like a good topic for discussion on the forum. I'm hoping someone from Raytheon will provide an answer.

On Mar 7, 2011, at 3:57 PM, Alice Isaacman wrote:

Hi Scott,

Thanks very much.

I'm working an issue that a person named Scott Houchin has brought up -- he's working for Frank deLucca and is busy using Rice decompression on the VIIRS raw RDRs furnished with the ADL package.

Scott is curious about the lengths of the 3 raw RDR files furnished with ADL, all of which are exactly the same -- and there seems to be a lot of fill data at the end of them. Is that perception correct?

He also noticed that there seem to be data which are missing in the middle of the raw RDRs. He'd like to know if that was deliberate.

If you could take these questions to the Raytheon ADL folks, that would be a help.

Thanks again,
Alice
Graeme Martin
UW - Madison, Space Science and Engineering Center
graemem@ssec.wisc.edu
graemem
Posts: 14
Joined: Tue Jan 11, 2011 5:44 pm

Re: missing / fill data in VIIRS RDRs

Post by graemem »

I will add that while my own experience with VIIRS RDRs is nil, I have noticed blocks of missing data in CrIS RDRs. The packet trackers in the header indicate that space has been reserved for packet data but but not filled. So this is not necessarily an indication that there is bad data, but I can see why someone who is working on compression would be wondering about this. I believe there is a common RDR format across instruments.
Graeme Martin
UW - Madison, Space Science and Engineering Center
graemem@ssec.wisc.edu
houchin
Posts: 128
Joined: Mon Jan 10, 2011 6:20 am

Re: missing / fill data in VIIRS RDRs

Post by houchin »

So since I'm the one that started some of this, I'll give more details. I have been making updates to vdext, the Aerospace CCSDS extractor for VIIRS. The previous version worked fine on UAID and application packet stream data, but did not work on streams extracted from RDR files. In making the updates, I ran into two issues:

First, I am generally dealing with packet streams that are "blindly" extracted from HDF files, or otherwise act as if they are extracted. These streams contain huge amounts of zeroed out data. In particular, the sample VIIRS RDR file distributed as part of ADL 2.0 has 161MB of zeros at the end. When vdext sees that data, it interprets it as very long sequence of packets of APID 0 with 0 data length, and thus slows to a glacial crawl as it parses the rest of the file two bytes at a time. If there is a way to recognize when I've reached the end of valid data, can someone point me to where that is documented. For the time being, if my code finds more than 100 "empty" packets in a row, it declares that it has reached the real end of file.

Second, I am seeing extra non-zero data inside aggregation zones for HRD operational packets. As the most simple example, we are seeing M4 detector 1 packets with zone 1 checksum offsets of 16, indicating that there should be 8 bytes of actual zone data. Since this detector is bowtie deleted, that data should not be there. For other packets where there is actual data to be rice decompressed, we are finding additional data inside the zone once the rice decompressor extracts the expected number of samples for that zone. The Rice decompression code we're using is a port of the code from Raytheon's PSD tool. Alice looked at that code and indicate it is similar to the operational code in ADL but not exactly the same. I have not had a chance to look at that nor compare the byte counts I see to what is put in the debug output from ADL yet. However, when vdext is run on UAID data or on the VIIRS application packet streams, I get exactly the same answers as the Raytheon extractor made available earlier in the test program.
Scott Houchin, Senior Engineering Specialist, The Aerospace Corporation
15049 Conference Center Dr CH3/310, Chantilly, VA 20151; 571-307-3914; scott.houchin@aero.org
bhurley
Posts: 4
Joined: Tue Jan 11, 2011 6:21 pm

Re: missing / fill data in VIIRS RDRs

Post by bhurley »

Scott -

I spoke with the IDPS Ingest Subsystem lead about your questions, and I’ll summarize what he told me.

The RDRs delivered with the ADL 2.0 package do have additional space in them.

The reason for the empty space at the end of the VIIRS RDRs is that the VIIRS packets contain compression, so the packet size can be somewhat variable. Due to this, the RDRs are sized for non-compressed detector data (worst case) to ensure the fixed sized RDR always has enough space. Note that in HDF5 RDRs (as will be in CLASS), the empty space at the end of the RDR will be chopped off to minimize waste.

As for the missing data in the middle, you are probably noticing some gaps in the Packet Tracker section of the RDR. The gaps in the Packet Tracker section are due to it being segregated by APID and there are several conditions that cause a smaller number of packets to arrive than are sized for. VIIRS has several all by itself. The first is that when VIIRS is in Night-mode, some APIDs are not created, but the RDR has allocated space for them. These will never arrive. Additionally, as scan times are not perfectly uniform, the RDR has to account for the scenario where (N-1) scans, called a "short granule", is all that fits in the temporal range of an RDR. Some instruments such as ATMS and CrIS do not have a scan time that can be analyzed for this condition and therefore each APID has an extra slot for each granule that in most cases never arrives.

The “Common Data Format Control Book - External (CDFCB-X) Vol. 2 - RDR Formats” contains a description of the common RDR format in sections 3.0 and 3.1. Additionally, it describes how to access the packets.

Probably the simplest approach for parsing packets out of an RDR is to walk the PktTracker array for each packet, this is one of the reasons it is included in the structure. The CDFCB-X shows the format of this section and how to use the APID List section to determine how many entries there are in the array.

Another approach is to use a value in the Static Header, called nextPktPos, that can be used to bound the parse of the storage section. The value in this field represents the offset that another packet can be stored to when received. One less than this offset would be the end of the last packet received. You can simply bound your packet parse to this byte location.

-------------------------------------------------------------------------------------

I’m not familiar with the PSD tool you are using for Rice decompression, what is the source of this tool? There is a special modified version of the Rice compression code (used in sensor firmware and also used to create proxy test data), there is also a special modified version of the Rice decompression code. The standard version of the Rice code will not work correctly for decompressing VIIRS packets. The Rice decompression code packaged with the VIIRS SDR software in ADL 2.0 is the correct version for decompressing VIIRS packets.
houchin
Posts: 128
Joined: Mon Jan 10, 2011 6:20 am

Re: missing / fill data in VIIRS RDRs

Post by houchin »

Thanks for the reply. I'll look into the packet tracker data and see if that is actually available as part of the raw data stream. It may be better to get the code that extracts the raw stream to strip out those missing packets.

The PSD tool is a Raytheon provided tool (which Raytheon, I'm not sure) that was provided for VIIRS data testing as part of the EDU and F1 instrument test programs. It's a Labview tool with a C backend. It is very centric to UAID formatted directories.

The one thing I don't have right now is the decompressed/depredicted output of the ADL parsed VIIRS-SCIENCE-RDR file in a format I can read. This is why we're looking for documentation on the Verified RDR format; we have hopes that we can use that to verify whether our data extractors are working correctly. Note that the output of our extractors is still in DN space, not radiance.
Scott Houchin, Senior Engineering Specialist, The Aerospace Corporation
15049 Conference Center Dr CH3/310, Chantilly, VA 20151; 571-307-3914; scott.houchin@aero.org
Post Reply