Split SDR data

VIIRS Sensor Data Record
Post Reply
lubo
Posts: 23
Joined: Fri Sep 02, 2016 7:04 am

Split SDR data

Post by lubo »

Hello,

we use CSPP_SDR_V2.2.3 including leap second update from Dec 2016 that is embedded in MEOS Polar software (Kongsberg Spacetec).

From time to time, we see that the SDR data from a single pass are split into 2 files with consecutive timestamps, e.g.:

SVI01_npp_d20170214_t1027397_e1031550_b27468_c20170214161229073365_cspp_dev.h5
SVI01_npp_d20170214_t1031562_e1040274_b27468_c20170214161231013416_cspp_dev.h5
.
.
GITCO_npp_d20170214_t1027397_e1031550_b27468_c20170214161041172982_cspp_dev.h5
GITCO_npp_d20170214_t1031562_e1040274_b27468_c20170214161046565861_cspp_dev.h5

though there is just one input RDR file from the pass. Our RDR files are produced by the DRL RT-STPS software v5.6. I tried to use also the RDR files from the latest version v5.9 but result was the same.

Could there be some problem with granules aggregation or something similar? Or is it normal to have the data split like this?

We would like to solve the problem also because the doubled files cause the software crashes due to memory overflow and so we lose some of the data.

If it can be of any help, I attached the MEOS Polar log file including outputs from the CSPP SDR software.

Thanks.

Lubo
Attachments

[The extension log has been deactivated and can no longer be displayed.]

kathys
Posts: 484
Joined: Tue Jun 22, 2010 4:51 pm

Re: Split SDR data

Post by kathys »

Hello,

We believe this problem is due to the nagg aggregation utility. It always tries to align the output SDRs based upon the IDPS archive granule boundary times. This affects the output granule aggregation counts when the initial and end granules do not match up exactly with the IDPS times as is often the case with direct broadcast passes.

We would like to investigate this further. Can you post your RDR file somewhere that we can access it? Feel free to post at our anonymous ftp site:

ftp://ftp.ssec.wisc.edu/pub/incoming/

Kathy
lubo
Posts: 23
Joined: Fri Sep 02, 2016 7:04 am

Re: Split SDR data

Post by lubo »

Hello, Kathy,

You can find the corresponding RDR file at:
ftp://ftp.ssec.wisc.edu/pub/incoming/RN ... ll-_dev.h5

Thanks for your help.

Lubo
lubo
Posts: 23
Joined: Fri Sep 02, 2016 7:04 am

Re: Split SDR data

Post by lubo »

Hello,
I have checked the granules boundary times in the RDR file and compared them with boundary times of the output SDR granules. Here are my findings:

1./ All granules inside the RDR file from the split pass have the same time duration. I calculated the duration from the "Beginning_time" and "Ending_time" values found e.g. under Data_products -> VIIRS-SCIENCE-RDR -> VIIRS-SCIENCE-RDR_Gran_0.

2./ The boundary times of granules inside the RDR file and the SDR granules slightly differ. E.g. the first granule in the RDR begins at 10:27:39.0 and ends at 10:29:04.35 whereas the first SDR granule at 10:27:39.797997 and 10:29:04.230365, respectively (Data_Products -> VIIRS-I1-SDR -> VIIRS-I1-SDR_Gran_0 -> Beginning_Time and Ending_Time). I found these time ranges called as "RangeDatiTime" and "ObservedDateTime" in the output .asc files:

("RangeDateTime" DATETIMERANGE EQ "2017-02-14 10:27:39.000000" "2017-02-14 10:29:04.350000")
("ObservedDateTime" DATETIMERANGE EQ "2017-02-14 10:27:39.797997" "2017-02-14 10:29:04.230365")

The mismatch of granules boundary times in the RDR and SDR files seems to occur in all passes (not only the split ones). What is the meaning of these two time ranges and how is the "ObservedDateTime" derived?

3./ Duration (or "ObservedDateTime" range) of the first SDR granule from the investigated pass is somewhat longer compared with the other granules. Nevertheless, when this longer granule was omitted from aggregation, the SDR data were split at the same time as when all granules were aggregated (i.e. at 10:31:55).

Lubo
kathys
Posts: 484
Joined: Tue Jun 22, 2010 4:51 pm

Re: Split SDR data

Post by kathys »

Hello Lubo,

I have downloaded your data. Thank you for staging it. We will investigate your data and questions and respond back to you.

Kathy
kathys
Posts: 484
Joined: Tue Jun 22, 2010 4:51 pm

Re: Split SDR data

Post by kathys »

Hi Lubo,

We have tracked your problem to the aggregation utility nagg. It supports certain granule boundary times, based upon the NOAA IDPS long granule archive requirements. So even if you want to have all the granules in a pass aggregated, such as the example RDR that you provided to us, it will always split the data set up based upon these calculated times. However, the new version of nagg includes an execution option (--onefile) that can bypass this behavior. To utilize it in your CSPP v2.2 software, follow these instructions:

cd $CSPP_SDR_HOME/common

mv adl_post_process.py adl_post_process.py.old
wget ftp://ftp.ssec.wisc.edu/pub/CSPP/aggreg ... process.py

The new adl_post_process.py replaces the original nagg argument (-n num_gran) with the new one (--onefile):

118 if geo:
119 # args = [NAGG, "-g", productid, "-n", str(num_gran), "-O", DPE_SITE_ID, "-D", DPE_DOMAI N, "-S"]
120 args = [NAGG, "-g", productid, "--onefile", "-O", DPE_SITE_ID, "-D", DPE_DOMAIN, "-S"]
121 else:
122 # args = [NAGG, "-g", "no", "-n", str(num_gran), "-O", DPE_SITE_ID, "-D", DPE_DOMAIN, "- S", "-t", productid]
123 args = [NAGG, "-g", "no", "--onefile", "-O", DPE_SITE_ID, "-D", DPE_DOMAIN, "-S", "-t", productid]

Once you make this change, you should not see the split aggregation occur anymore. I tested this out with the RDR that you provided, and it produced one aggregated set of VIIRS output files.

Your questions about RDR versus SDR granule boundary times have to do with rt-stps utility. We have previously reported the time differences to the NASA direct readout lab.

Kathy
lubo
Posts: 23
Joined: Fri Sep 02, 2016 7:04 am

Re: Split SDR data

Post by lubo »

Hi Kathy,

the --onefile option is exactly what I was looking for. It works great.
Thanks a lot for your work.

Lubo
Post Reply