To process historical data with VIIRS SDR V4.0.

[i3asanuma]

Dear:
I have tried to generate SDR products using CSPP VIIRS SDR v4.0. My system is configured as follows:
CPU : Intel Xeon E-2134, Memory : 128GB.
Operating system : Ubuntu 20.04.4.
CSPP SDR v4.0 is working very well with the recent level0 data, like;
RNSCA-RVIRS_j02_d20230520_t1721339_e1735474_b00001_c20230520173504143000_drlu_ops.h5.
Unfortunately, I have faced into the trouble while processing the historical data, which was
downloaded from NOAA/CLASS system like;
RNSCA-RVIRS_j01_d20210609_t1031208_e1037022_b18431_c20210609143717248308_oeau_ops.h5.
CSPP SDR v4.0 created the geolocation files, but could not create SDR products as the log.
The log mentioned we have to conduct sdr_luts.sh -t yyyymmdd and sdr_ancillary.sh -t yyyymmdd.
I have checked FAQ and followed your suggestion, but it didnot work.
What I did;
$ export CSPP_SDR_HOME=/home/cspp/CSPP/SDR_4_0
$ source $CSPP_SDR_HOME/cspp_sdr_env.sh
As it was impossible to find ancillary file on 9th, June 2021 on cspp_v_6_0, I have used "cspp_v_4_0".
$ export JPSS_REMOTE_ANC_DIR=http://jpssdb.ssec.wisc.edu/cspp_v_4_0/ ... 06_09_160/
or
$ export JPSS_REMOTE_ANC_DIR=http://jpssdb.ssec.wisc.edu/cspp_v_4_0/ancillary/
Also,
$ export JPSS_REMOTE_LUT_DIR=http://jpssdb.ssec.wisc.edu/cspp_luts_v_4_0/
Then I have issued;
$ viirs_sdr.sh -a -p 3 -g -m -d -l $infile
, where $infile is pointing RNSCA-RVIRS_j01_d20210609_t1031208_e1037022_b18431_c20210609143717248308_oeau_ops.h5.

I have tried some trials as you have suggested in the past posts like;
$ rm -r /home/cspp/CSPP/SDR_4_0/anc/cache/*.
I could observe that sdr_luts.sh or sdr_ancillary.sh is creating data base like;
$ ls CSPP/SDR_4_0/anc/cache/2021_06_09_160
TLE-AUX_j01_20210609051025Z_20210609000000Z_ee00000000000000Z_-_oeac_ops_all-_ops.h5
TLE-AUX_npp_20210609051017Z_20210609000000Z_ee00000000000000Z_-_oeac_ops_all-_ops.h5
VIIRS-RSBAUTOCAL-HISTORY-AUX_j01_20210609015210Z_20210609000000Z_ee00000000000000Z_-_oeac_ops_all-_ops.h5
VIIRS-RSBAUTOCAL-HISTORY-AUX_npp_20210609013311Z_20210609000000Z_ee00000000000000Z_-_oeac_ops_all-_ops.h5

The recent data are processed with this condition and SDR v4.0 created geolocation and sdr files without problem.
I am looking forward to hearing from you, especiall the sequence to process the historical data.
Best regards, Ichio.

A part of log is as follows; ------------------------------------------------------------------------------------------------------
2023-06-15 01:37:21(ERROR):adl_sdr.py:250:/home/cspp/CSPP/SDR_4_0/libexec/ADL/bin/ProSdrViirsCalController.exe failed on '/raid/cspp/work/J01003039067692/sdr_viirs_geo_J01003039067692.xml:/raid/cspp/work/J01003039067692/sdr_viirs_cal_J01003039067692.xml': CalledProcessError(1, ['/home/cspp/CSPP/SDR_4_0/libexec/ADL/bin/ProSdrViirsCalController.exe', '/raid/cspp/work/J01003039067692/sdr_viirs_cal_J01003039067692.xml']). Continuing...
2023-06-15 01:37:21(INFO):product_operations.py:65:Check for Granule: J01003039067692 products.
2023-06-15 01:37:21(WARNING):product_operations.py:80:Problem : J01003039067692 VIIRS-DNB-SDR No product produced. SVDNB
2023-06-15 01:37:21(WARNING):product_operations.py:80:Problem : J01003039067692 VIIRS-DualGain-Cal-IP No product produced. IVCDB
2023-06-15 01:37:21(WARNING):product_operations.py:80:Problem : J01003039067692 VIIRS-I1-SDR No product produced. SVI01
2023-06-15 01:37:21(WARNING):product_operations.py:80:Problem : J01003039067692 VIIRS-I2-SDR No product produced. SVI02
2023-06-15 01:37:21(WARNING):product_operations.py:80:Problem : J01003039067692 VIIRS-I3-SDR No product produced. SVI03
2023-06-15 01:37:21(WARNING):product_operations.py:80:Problem : J01003039067692 VIIRS-I4-SDR No product produced. SVI04
2023-06-15 01:37:21(WARNING):product_operations.py:80:Problem : J01003039067692 VIIRS-I5-SDR No product produced. SVI05
2023-06-15 01:37:21(WARNING):product_operations.py:80:Problem : J01003039067692 VIIRS-M1-SDR No product produced. SVM01
2023-06-15 01:37:21(WARNING):product_operations.py:80:Problem : J01003039067692 VIIRS-M10-SDR No product produced. SVM10
2023-06-15 01:37:21(WARNING):product_operations.py:80:Problem : J01003039067692 VIIRS-M11-SDR No product produced. SVM11
2023-06-15 01:37:21(WARNING):product_operations.py:80:Problem : J01003039067692 VIIRS-M12-SDR No product produced. SVM12
2023-06-15 01:37:21(WARNING):product_operations.py:80:Problem : J01003039067692 VIIRS-M13-SDR No product produced. SVM13
2023-06-15 01:37:21(WARNING):product_operations.py:80:Problem : J01003039067692 VIIRS-M14-SDR No product produced. SVM14
2023-06-15 01:37:21(WARNING):product_operations.py:80:Problem : J01003039067692 VIIRS-M15-SDR No product produced. SVM15
2023-06-15 01:37:21(WARNING):product_operations.py:80:Problem : J01003039067692 VIIRS-M16-SDR No product produced. SVM16
2023-06-15 01:37:21(WARNING):product_operations.py:80:Problem : J01003039067692 VIIRS-M2-SDR No product produced. SVM02
2023-06-15 01:37:21(WARNING):product_operations.py:80:Problem : J01003039067692 VIIRS-M3-SDR No product produced. SVM03
2023-06-15 01:37:21(WARNING):product_operations.py:80:Problem : J01003039067692 VIIRS-M4-SDR No product produced. SVM04
2023-06-15 01:37:21(WARNING):product_operations.py:80:Problem : J01003039067692 VIIRS-M5-SDR No product produced. SVM05
2023-06-15 01:37:21(WARNING):product_operations.py:80:Problem : J01003039067692 VIIRS-M6-SDR No product produced. SVM06
2023-06-15 01:37:21(WARNING):product_operations.py:80:Problem : J01003039067692 VIIRS-M7-SDR No product produced. SVM07
2023-06-15 01:37:21(WARNING):product_operations.py:80:Problem : J01003039067692 VIIRS-M8-SDR No product produced. SVM08
2023-06-15 01:37:21(WARNING):product_operations.py:80:Problem : J01003039067692 VIIRS-M9-SDR No product produced. SVM09
2023-06-15 01:37:21(INFO):product_operations.py:101:5 out of 28 products produced for granule J01003039067692
2023-06-15 01:37:21(INFO):adl_log.py:77:Checking Log file /raid/cspp/work/J01003039067692/ProSdrViirsCalController.exe_1234_549794.log for errors.
2023-06-15 01:37:21(INFO):log_common.py:107:
( ADL Error Hint Missing or out of date ancillary input, Check effectivity for listed Shortname. log_file ProSdrViirsCalController.exe_1234_549794.log line 95 )

2023-06-15 01:37:21(ERROR):adl_log.py:113:2023-06-15 01:37:21.048000Z DBG_HIGH ProCmnMessage.cpp(502)[ProSdrViirsCalC (22925763766144)]: INFSTATUS: ProSdrViirsCal SDR_PREREQ_ABSENT VIIRS-SDR-DNB-DN0-LUT ( Required input not available for Shortname: VIIRS-SDR-DNB-DN0-LUT , Granule ID: Failure(s): (VIIRS-SDR-DNB-DN0-LUT, ) ) TaskType=GRANULE, GranuleId=J01003039067692, GranuleVersion=A1, Spacecraft=J01, Sensor=VIIRS
2023-06-15 01:37:21(INFO):log_common.py:107:
( ADL Error Hint Missing LUTS for data, run "sdr_luts.sh -tYYYYMMDD" with time of data from input RDR log_file ProSdrViirsCalController.exe_1234_549794.log line 95 )

2023-06-15 01:37:21(ERROR):adl_log.py:113:2023-06-15 01:37:21.048000Z DBG_HIGH ProCmnMessage.cpp(502)[ProSdrViirsCalC (22925763766144)]: INFSTATUS: ProSdrViirsCal SDR_PREREQ_ABSENT VIIRS-SDR-DNB-DN0-LUT ( Required input not available for Shortname: VIIRS-SDR-DNB-DN0-LUT , Granule ID: Failure(s): (VIIRS-SDR-DNB-DN0-LUT, ) ) TaskType=GRANULE, GranuleId=J01003039067692, GranuleVersion=A1, Spacecraft=J01, Sensor=VIIRS
2023-06-15 01:37:21(INFO):log_common.py:107:
( ADL Error Hint Missing or out of date ancillary input, Check effectivity for listed Shortname. log_file ProSdrViirsCalController.exe_1234_549794.log line 96 )

-------------------------------------------------------------------end of message-------------------------------------------------------------------

[kathys]

Hello,

The CSPP SDR Version 4.0 software documentation includes this section:

2.2.1 Processing Historical Data Sets with CSPP SDR Version 4.0
CSPP SDR v4.0 can be used for creating VIIRS, CrIS, ATMS and OMPS SDRs from historical NOAA-20 and S-NPP RDRs beginning with data collected on 18 July 2022.

You will not be able to process historical data that was observed before this date. Each version of ADL has a different effectivity start date, sometimes with different Look-Up-Tables that may not be compatible with other versions. Since the target use community is direct broadcast, we suggest that you can order the data from the NOAA CLASS archive or the new Amazon Web Services (AWS) Cloud too. AWS does not contain a complete archive of SDRs, but you may find what you need there, and files can be downloaded directly for free:

NOAA CLASS:

• https://www.avl.class.noaa.gov/

JPSS AWS:

• https://registry.opendata.aws/noaa-jpss/

Good luck.

[i3asanuma]

Dear Kathy:
Sorry, I missed the description on the history data.
SDR 4.0 works on data since July 18, 2022.
SDR 3.3 works on data since August 9, 2021.
SDR 3.2 ???
Is it possible to download the older version of SDR software?
Best regards, Ichio.

[kathys]

CSPP SDR Version 3.2.2 can be used to process historical data starting on 26 November 2019.

I have temporarily placed the CSPP SDR tar files and documentation so that you can retrieve them here:

https://bin.ssec.wisc.edu/pub/ssec/kathys/sdr_v3.2.2/

Good luck!

Kathy

[i3asanuma]

Dear Kathy:
The web address provided does not work.
Would you kindly check the address?

https://bin.ssec.wisc.edu/pub/ssec/kathys/sdr_v3.2.2/

I am strongly looking forward to downloading the SDR v3.2.2.

Best regards, Ichio.

[kathys]

Sorry about that. Large files get scrubbed from this site very quickly.

I moved the files to a system that will not be scrubbed. You can access the files through this temporary anonymous ftp site:

ftp://ftp.ssec.wisc.edu/pub/kathys/TEMP/sdr_v3_2_2

Good luck,

Kathy

[i3asanuma]

Dear Kathy:
"viirs_sdr.sh" worked for the historical data around June 2021 under SDR V3.2.
Thank you very much.
Ichio.