To process historical data with VIIRS SDR V4.0.
Posted: Wed Jun 14, 2023 11:26 pm
Dear:
I have tried to generate SDR products using CSPP VIIRS SDR v4.0. My system is configured as follows:
CPU : Intel Xeon E-2134, Memory : 128GB.
Operating system : Ubuntu 20.04.4.
CSPP SDR v4.0 is working very well with the recent level0 data, like;
RNSCA-RVIRS_j02_d20230520_t1721339_e1735474_b00001_c20230520173504143000_drlu_ops.h5.
Unfortunately, I have faced into the trouble while processing the historical data, which was
downloaded from NOAA/CLASS system like;
RNSCA-RVIRS_j01_d20210609_t1031208_e1037022_b18431_c20210609143717248308_oeau_ops.h5.
CSPP SDR v4.0 created the geolocation files, but could not create SDR products as the log.
The log mentioned we have to conduct sdr_luts.sh -t yyyymmdd and sdr_ancillary.sh -t yyyymmdd.
I have checked FAQ and followed your suggestion, but it didnot work.
What I did;
$ export CSPP_SDR_HOME=/home/cspp/CSPP/SDR_4_0
$ source $CSPP_SDR_HOME/cspp_sdr_env.sh
As it was impossible to find ancillary file on 9th, June 2021 on cspp_v_6_0, I have used "cspp_v_4_0".
$ export JPSS_REMOTE_ANC_DIR=http://jpssdb.ssec.wisc.edu/cspp_v_4_0/ ... 06_09_160/
or
$ export JPSS_REMOTE_ANC_DIR=http://jpssdb.ssec.wisc.edu/cspp_v_4_0/ancillary/
Also,
$ export JPSS_REMOTE_LUT_DIR=http://jpssdb.ssec.wisc.edu/cspp_luts_v_4_0/
Then I have issued;
$ viirs_sdr.sh -a -p 3 -g -m -d -l $infile
, where $infile is pointing RNSCA-RVIRS_j01_d20210609_t1031208_e1037022_b18431_c20210609143717248308_oeau_ops.h5.
I have tried some trials as you have suggested in the past posts like;
$ rm -r /home/cspp/CSPP/SDR_4_0/anc/cache/*.
I could observe that sdr_luts.sh or sdr_ancillary.sh is creating data base like;
$ ls CSPP/SDR_4_0/anc/cache/2021_06_09_160
TLE-AUX_j01_20210609051025Z_20210609000000Z_ee00000000000000Z_-_oeac_ops_all-_ops.h5
TLE-AUX_npp_20210609051017Z_20210609000000Z_ee00000000000000Z_-_oeac_ops_all-_ops.h5
VIIRS-RSBAUTOCAL-HISTORY-AUX_j01_20210609015210Z_20210609000000Z_ee00000000000000Z_-_oeac_ops_all-_ops.h5
VIIRS-RSBAUTOCAL-HISTORY-AUX_npp_20210609013311Z_20210609000000Z_ee00000000000000Z_-_oeac_ops_all-_ops.h5
The recent data are processed with this condition and SDR v4.0 created geolocation and sdr files without problem.
I am looking forward to hearing from you, especiall the sequence to process the historical data.
Best regards, Ichio.
A part of log is as follows; ------------------------------------------------------------------------------------------------------
2023-06-15 01:37:21(ERROR):adl_sdr.py:250:/home/cspp/CSPP/SDR_4_0/libexec/ADL/bin/ProSdrViirsCalController.exe failed on '/raid/cspp/work/J01003039067692/sdr_viirs_geo_J01003039067692.xml:/raid/cspp/work/J01003039067692/sdr_viirs_cal_J01003039067692.xml': CalledProcessError(1, ['/home/cspp/CSPP/SDR_4_0/libexec/ADL/bin/ProSdrViirsCalController.exe', '/raid/cspp/work/J01003039067692/sdr_viirs_cal_J01003039067692.xml']). Continuing...
2023-06-15 01:37:21(INFO):product_operations.py:65:Check for Granule: J01003039067692 products.
2023-06-15 01:37:21(WARNING):product_operations.py:80:Problem : J01003039067692 VIIRS-DNB-SDR No product produced. SVDNB
2023-06-15 01:37:21(WARNING):product_operations.py:80:Problem : J01003039067692 VIIRS-DualGain-Cal-IP No product produced. IVCDB
2023-06-15 01:37:21(WARNING):product_operations.py:80:Problem : J01003039067692 VIIRS-I1-SDR No product produced. SVI01
2023-06-15 01:37:21(WARNING):product_operations.py:80:Problem : J01003039067692 VIIRS-I2-SDR No product produced. SVI02
2023-06-15 01:37:21(WARNING):product_operations.py:80:Problem : J01003039067692 VIIRS-I3-SDR No product produced. SVI03
2023-06-15 01:37:21(WARNING):product_operations.py:80:Problem : J01003039067692 VIIRS-I4-SDR No product produced. SVI04
2023-06-15 01:37:21(WARNING):product_operations.py:80:Problem : J01003039067692 VIIRS-I5-SDR No product produced. SVI05
2023-06-15 01:37:21(WARNING):product_operations.py:80:Problem : J01003039067692 VIIRS-M1-SDR No product produced. SVM01
2023-06-15 01:37:21(WARNING):product_operations.py:80:Problem : J01003039067692 VIIRS-M10-SDR No product produced. SVM10
2023-06-15 01:37:21(WARNING):product_operations.py:80:Problem : J01003039067692 VIIRS-M11-SDR No product produced. SVM11
2023-06-15 01:37:21(WARNING):product_operations.py:80:Problem : J01003039067692 VIIRS-M12-SDR No product produced. SVM12
2023-06-15 01:37:21(WARNING):product_operations.py:80:Problem : J01003039067692 VIIRS-M13-SDR No product produced. SVM13
2023-06-15 01:37:21(WARNING):product_operations.py:80:Problem : J01003039067692 VIIRS-M14-SDR No product produced. SVM14
2023-06-15 01:37:21(WARNING):product_operations.py:80:Problem : J01003039067692 VIIRS-M15-SDR No product produced. SVM15
2023-06-15 01:37:21(WARNING):product_operations.py:80:Problem : J01003039067692 VIIRS-M16-SDR No product produced. SVM16
2023-06-15 01:37:21(WARNING):product_operations.py:80:Problem : J01003039067692 VIIRS-M2-SDR No product produced. SVM02
2023-06-15 01:37:21(WARNING):product_operations.py:80:Problem : J01003039067692 VIIRS-M3-SDR No product produced. SVM03
2023-06-15 01:37:21(WARNING):product_operations.py:80:Problem : J01003039067692 VIIRS-M4-SDR No product produced. SVM04
2023-06-15 01:37:21(WARNING):product_operations.py:80:Problem : J01003039067692 VIIRS-M5-SDR No product produced. SVM05
2023-06-15 01:37:21(WARNING):product_operations.py:80:Problem : J01003039067692 VIIRS-M6-SDR No product produced. SVM06
2023-06-15 01:37:21(WARNING):product_operations.py:80:Problem : J01003039067692 VIIRS-M7-SDR No product produced. SVM07
2023-06-15 01:37:21(WARNING):product_operations.py:80:Problem : J01003039067692 VIIRS-M8-SDR No product produced. SVM08
2023-06-15 01:37:21(WARNING):product_operations.py:80:Problem : J01003039067692 VIIRS-M9-SDR No product produced. SVM09
2023-06-15 01:37:21(INFO):product_operations.py:101:5 out of 28 products produced for granule J01003039067692
2023-06-15 01:37:21(INFO):adl_log.py:77:Checking Log file /raid/cspp/work/J01003039067692/ProSdrViirsCalController.exe_1234_549794.log for errors.
2023-06-15 01:37:21(INFO):log_common.py:107:
( ADL Error Hint Missing or out of date ancillary input, Check effectivity for listed Shortname. log_file ProSdrViirsCalController.exe_1234_549794.log line 95 )
2023-06-15 01:37:21(ERROR):adl_log.py:113:2023-06-15 01:37:21.048000Z DBG_HIGH ProCmnMessage.cpp(502)[ProSdrViirsCalC (22925763766144)]: INFSTATUS: ProSdrViirsCal SDR_PREREQ_ABSENT VIIRS-SDR-DNB-DN0-LUT ( Required input not available for Shortname: VIIRS-SDR-DNB-DN0-LUT , Granule ID: Failure(s): (VIIRS-SDR-DNB-DN0-LUT, ) ) TaskType=GRANULE, GranuleId=J01003039067692, GranuleVersion=A1, Spacecraft=J01, Sensor=VIIRS
2023-06-15 01:37:21(INFO):log_common.py:107:
( ADL Error Hint Missing LUTS for data, run "sdr_luts.sh -tYYYYMMDD" with time of data from input RDR log_file ProSdrViirsCalController.exe_1234_549794.log line 95 )
2023-06-15 01:37:21(ERROR):adl_log.py:113:2023-06-15 01:37:21.048000Z DBG_HIGH ProCmnMessage.cpp(502)[ProSdrViirsCalC (22925763766144)]: INFSTATUS: ProSdrViirsCal SDR_PREREQ_ABSENT VIIRS-SDR-DNB-DN0-LUT ( Required input not available for Shortname: VIIRS-SDR-DNB-DN0-LUT , Granule ID: Failure(s): (VIIRS-SDR-DNB-DN0-LUT, ) ) TaskType=GRANULE, GranuleId=J01003039067692, GranuleVersion=A1, Spacecraft=J01, Sensor=VIIRS
2023-06-15 01:37:21(INFO):log_common.py:107:
( ADL Error Hint Missing or out of date ancillary input, Check effectivity for listed Shortname. log_file ProSdrViirsCalController.exe_1234_549794.log line 96 )
-------------------------------------------------------------------end of message-------------------------------------------------------------------
I have tried to generate SDR products using CSPP VIIRS SDR v4.0. My system is configured as follows:
CPU : Intel Xeon E-2134, Memory : 128GB.
Operating system : Ubuntu 20.04.4.
CSPP SDR v4.0 is working very well with the recent level0 data, like;
RNSCA-RVIRS_j02_d20230520_t1721339_e1735474_b00001_c20230520173504143000_drlu_ops.h5.
Unfortunately, I have faced into the trouble while processing the historical data, which was
downloaded from NOAA/CLASS system like;
RNSCA-RVIRS_j01_d20210609_t1031208_e1037022_b18431_c20210609143717248308_oeau_ops.h5.
CSPP SDR v4.0 created the geolocation files, but could not create SDR products as the log.
The log mentioned we have to conduct sdr_luts.sh -t yyyymmdd and sdr_ancillary.sh -t yyyymmdd.
I have checked FAQ and followed your suggestion, but it didnot work.
What I did;
$ export CSPP_SDR_HOME=/home/cspp/CSPP/SDR_4_0
$ source $CSPP_SDR_HOME/cspp_sdr_env.sh
As it was impossible to find ancillary file on 9th, June 2021 on cspp_v_6_0, I have used "cspp_v_4_0".
$ export JPSS_REMOTE_ANC_DIR=http://jpssdb.ssec.wisc.edu/cspp_v_4_0/ ... 06_09_160/
or
$ export JPSS_REMOTE_ANC_DIR=http://jpssdb.ssec.wisc.edu/cspp_v_4_0/ancillary/
Also,
$ export JPSS_REMOTE_LUT_DIR=http://jpssdb.ssec.wisc.edu/cspp_luts_v_4_0/
Then I have issued;
$ viirs_sdr.sh -a -p 3 -g -m -d -l $infile
, where $infile is pointing RNSCA-RVIRS_j01_d20210609_t1031208_e1037022_b18431_c20210609143717248308_oeau_ops.h5.
I have tried some trials as you have suggested in the past posts like;
$ rm -r /home/cspp/CSPP/SDR_4_0/anc/cache/*.
I could observe that sdr_luts.sh or sdr_ancillary.sh is creating data base like;
$ ls CSPP/SDR_4_0/anc/cache/2021_06_09_160
TLE-AUX_j01_20210609051025Z_20210609000000Z_ee00000000000000Z_-_oeac_ops_all-_ops.h5
TLE-AUX_npp_20210609051017Z_20210609000000Z_ee00000000000000Z_-_oeac_ops_all-_ops.h5
VIIRS-RSBAUTOCAL-HISTORY-AUX_j01_20210609015210Z_20210609000000Z_ee00000000000000Z_-_oeac_ops_all-_ops.h5
VIIRS-RSBAUTOCAL-HISTORY-AUX_npp_20210609013311Z_20210609000000Z_ee00000000000000Z_-_oeac_ops_all-_ops.h5
The recent data are processed with this condition and SDR v4.0 created geolocation and sdr files without problem.
I am looking forward to hearing from you, especiall the sequence to process the historical data.
Best regards, Ichio.
A part of log is as follows; ------------------------------------------------------------------------------------------------------
2023-06-15 01:37:21(ERROR):adl_sdr.py:250:/home/cspp/CSPP/SDR_4_0/libexec/ADL/bin/ProSdrViirsCalController.exe failed on '/raid/cspp/work/J01003039067692/sdr_viirs_geo_J01003039067692.xml:/raid/cspp/work/J01003039067692/sdr_viirs_cal_J01003039067692.xml': CalledProcessError(1, ['/home/cspp/CSPP/SDR_4_0/libexec/ADL/bin/ProSdrViirsCalController.exe', '/raid/cspp/work/J01003039067692/sdr_viirs_cal_J01003039067692.xml']). Continuing...
2023-06-15 01:37:21(INFO):product_operations.py:65:Check for Granule: J01003039067692 products.
2023-06-15 01:37:21(WARNING):product_operations.py:80:Problem : J01003039067692 VIIRS-DNB-SDR No product produced. SVDNB
2023-06-15 01:37:21(WARNING):product_operations.py:80:Problem : J01003039067692 VIIRS-DualGain-Cal-IP No product produced. IVCDB
2023-06-15 01:37:21(WARNING):product_operations.py:80:Problem : J01003039067692 VIIRS-I1-SDR No product produced. SVI01
2023-06-15 01:37:21(WARNING):product_operations.py:80:Problem : J01003039067692 VIIRS-I2-SDR No product produced. SVI02
2023-06-15 01:37:21(WARNING):product_operations.py:80:Problem : J01003039067692 VIIRS-I3-SDR No product produced. SVI03
2023-06-15 01:37:21(WARNING):product_operations.py:80:Problem : J01003039067692 VIIRS-I4-SDR No product produced. SVI04
2023-06-15 01:37:21(WARNING):product_operations.py:80:Problem : J01003039067692 VIIRS-I5-SDR No product produced. SVI05
2023-06-15 01:37:21(WARNING):product_operations.py:80:Problem : J01003039067692 VIIRS-M1-SDR No product produced. SVM01
2023-06-15 01:37:21(WARNING):product_operations.py:80:Problem : J01003039067692 VIIRS-M10-SDR No product produced. SVM10
2023-06-15 01:37:21(WARNING):product_operations.py:80:Problem : J01003039067692 VIIRS-M11-SDR No product produced. SVM11
2023-06-15 01:37:21(WARNING):product_operations.py:80:Problem : J01003039067692 VIIRS-M12-SDR No product produced. SVM12
2023-06-15 01:37:21(WARNING):product_operations.py:80:Problem : J01003039067692 VIIRS-M13-SDR No product produced. SVM13
2023-06-15 01:37:21(WARNING):product_operations.py:80:Problem : J01003039067692 VIIRS-M14-SDR No product produced. SVM14
2023-06-15 01:37:21(WARNING):product_operations.py:80:Problem : J01003039067692 VIIRS-M15-SDR No product produced. SVM15
2023-06-15 01:37:21(WARNING):product_operations.py:80:Problem : J01003039067692 VIIRS-M16-SDR No product produced. SVM16
2023-06-15 01:37:21(WARNING):product_operations.py:80:Problem : J01003039067692 VIIRS-M2-SDR No product produced. SVM02
2023-06-15 01:37:21(WARNING):product_operations.py:80:Problem : J01003039067692 VIIRS-M3-SDR No product produced. SVM03
2023-06-15 01:37:21(WARNING):product_operations.py:80:Problem : J01003039067692 VIIRS-M4-SDR No product produced. SVM04
2023-06-15 01:37:21(WARNING):product_operations.py:80:Problem : J01003039067692 VIIRS-M5-SDR No product produced. SVM05
2023-06-15 01:37:21(WARNING):product_operations.py:80:Problem : J01003039067692 VIIRS-M6-SDR No product produced. SVM06
2023-06-15 01:37:21(WARNING):product_operations.py:80:Problem : J01003039067692 VIIRS-M7-SDR No product produced. SVM07
2023-06-15 01:37:21(WARNING):product_operations.py:80:Problem : J01003039067692 VIIRS-M8-SDR No product produced. SVM08
2023-06-15 01:37:21(WARNING):product_operations.py:80:Problem : J01003039067692 VIIRS-M9-SDR No product produced. SVM09
2023-06-15 01:37:21(INFO):product_operations.py:101:5 out of 28 products produced for granule J01003039067692
2023-06-15 01:37:21(INFO):adl_log.py:77:Checking Log file /raid/cspp/work/J01003039067692/ProSdrViirsCalController.exe_1234_549794.log for errors.
2023-06-15 01:37:21(INFO):log_common.py:107:
( ADL Error Hint Missing or out of date ancillary input, Check effectivity for listed Shortname. log_file ProSdrViirsCalController.exe_1234_549794.log line 95 )
2023-06-15 01:37:21(ERROR):adl_log.py:113:2023-06-15 01:37:21.048000Z DBG_HIGH ProCmnMessage.cpp(502)[ProSdrViirsCalC (22925763766144)]: INFSTATUS: ProSdrViirsCal SDR_PREREQ_ABSENT VIIRS-SDR-DNB-DN0-LUT ( Required input not available for Shortname: VIIRS-SDR-DNB-DN0-LUT , Granule ID: Failure(s): (VIIRS-SDR-DNB-DN0-LUT, ) ) TaskType=GRANULE, GranuleId=J01003039067692, GranuleVersion=A1, Spacecraft=J01, Sensor=VIIRS
2023-06-15 01:37:21(INFO):log_common.py:107:
( ADL Error Hint Missing LUTS for data, run "sdr_luts.sh -tYYYYMMDD" with time of data from input RDR log_file ProSdrViirsCalController.exe_1234_549794.log line 95 )
2023-06-15 01:37:21(ERROR):adl_log.py:113:2023-06-15 01:37:21.048000Z DBG_HIGH ProCmnMessage.cpp(502)[ProSdrViirsCalC (22925763766144)]: INFSTATUS: ProSdrViirsCal SDR_PREREQ_ABSENT VIIRS-SDR-DNB-DN0-LUT ( Required input not available for Shortname: VIIRS-SDR-DNB-DN0-LUT , Granule ID: Failure(s): (VIIRS-SDR-DNB-DN0-LUT, ) ) TaskType=GRANULE, GranuleId=J01003039067692, GranuleVersion=A1, Spacecraft=J01, Sensor=VIIRS
2023-06-15 01:37:21(INFO):log_common.py:107:
( ADL Error Hint Missing or out of date ancillary input, Check effectivity for listed Shortname. log_file ProSdrViirsCalController.exe_1234_549794.log line 96 )
-------------------------------------------------------------------end of message-------------------------------------------------------------------