sdr_lut.sh from v 3.3.1 fails on ubuntu jammy

VIIRS Sensor Data Record
Post Reply
trygveas
Posts: 5
Joined: Mon Oct 08, 2018 7:18 am

sdr_lut.sh from v 3.3.1 fails on ubuntu jammy

Post by trygveas »

Hi

I have setup a new instance with ubuntu jammy. I see you recommend centos8 in the install instructions, but I have succeeded with ubuntu in the past.

Thu unpack of the source files works of course fine, but when I want to run sdr_luts.sh the first time I get:

ubuntu@cspp-processing-a:~$ sdr_luts.sh
CSPP_DYNAMIC_ANCIL_DIR=/software/CSPP/SDR_3_3/anc/cache
2022-06-22 09:18:51(INFO):ancillary_control.py:1176:product version: DPE_VER for CSPP SDR CSPP_SDR_3_3_1
2022-06-22 09:18:51(INFO):ancillary_control.py:1177:JPSS_REMOTE_ANC_DIR = https://jpssdb.ssec.wisc.edu/cspp_v_4_0/ancillary
2022-06-22 09:18:51(INFO):ancillary_control.py:1178:JPSS_REMOTE_LUT_DIR = https://jpssdb.ssec.wisc.edu/cspp_luts_v_5_0/
2022-06-22 09:18:51(INFO):sdr_config_paths.py:256:Checking static data
2022-06-22 09:18:51(INFO):sdr_config_paths.py:270:Found *static_luts* signature at /software/CSPP/SDR_3_3/anc/static/shipped_luts
2022-06-22 09:18:51(INFO):sdr_config_paths.py:270:Found *static_tiles* signature at /software/CSPP/SDR_3_3/anc/static/ecotiles
2022-06-22 09:18:51(INFO):sdr_config_paths.py:270:Found *stray_luts* signature at /software/CSPP/SDR_3_3/anc/static/stray_light_luts
2022-06-22 09:18:51(INFO):sdr_config_paths.py:277:Checking dynamic data
2022-06-22 09:18:51(INFO):sdr_config_paths.py:284:Writable cache directory, /software/CSPP/SDR_3_3/anc/cache
2022-06-22 09:18:51(INFO):sdr_config_paths.py:295:end check installation
2022-06-22 09:18:51(INFO):ancillary_control.py:1183:Arguments
Namespace(ancillary='LUTS', debug=False, local=False, package=False, separate=False, time=None, verbosity=2, work_dir='.')
2022-06-22 09:18:51(INFO):ancillary_control.py:1185:CSPP execution work directory is '/home/ubuntu'
CSPP_DYNAMIC_ANCIL_DIR=/software/CSPP/SDR_3_3/anc/cache
2022-06-22 09:18:51(INFO):ancillary_control.py:1188:CSPP execution work directory is '/home/ubuntu'
2022-06-22 09:18:51(INFO):sdr_config_paths.py:256:Checking static data
2022-06-22 09:18:51(INFO):sdr_config_paths.py:277:Checking dynamic data
2022-06-22 09:18:51(INFO):sdr_config_paths.py:295:end check installation
CSPP_DYNAMIC_ANCIL_DIR=/software/CSPP/SDR_3_3/anc/cache
2022-06-22 09:18:51(INFO):ancillary_control.py:1053:check_for_cache_database_initalization /software/CSPP/SDR_3_3/anc/cache/SDR_3_3_DB/db
2022-06-22 09:18:51(INFO):ancillary_control.py:1081:Database initialized or in process [Errno 17] File exists: '/software/CSPP/SDR_3_3/anc/cache/SDR_3_3_DB/db'
2022-06-22 09:18:51(INFO):http.py:105:Connection problem httpsjpssdb.ssec.wisc.edu/cspp_luts_v_5_0/:// >VIIRS-SDR-GEO-MOD-PARAM-V2-LUT_npp_.*?bin<
2022-06-22 09:18:51(INFO):http.py:105:Connection problem httpsjpssdb.ssec.wisc.edu/cspp_luts_v_5_0/:// >VIIRS-SDR-GEO-MOD-PARAM-V2-LUT_npp_.*?bin<
2022-06-22 09:18:51(INFO):http.py:105:Connection problem httpsjpssdb.ssec.wisc.edu/cspp_luts_v_5_0/:// >off_Planet-Eph-ANC_Static.*?bin<
2022-06-22 09:18:51(INFO):http.py:105:Connection problem httpsjpssdb.ssec.wisc.edu/cspp_luts_v_5_0/:// >off_Planet-Eph-ANC_Static.*?bin<
2022-06-22 09:18:52(INFO):http.py:105:Connection problem httpsjpssdb.ssec.wisc.edu/cspp_luts_v_5_0/:// >off_Planet-Eph-ANC_Static.*?bin<
2022-06-22 09:18:52(INFO):http.py:105:Connection problem httpsjpssdb.ssec.wisc.edu/cspp_luts_v_5_0/:// >off_Planet-Eph-ANC_Static.*?bin<
2022-06-22 09:18:52(INFO):http.py:105:Connection problem httpsjpssdb.ssec.wisc.edu/cspp_luts_v_5_0/:// >OMPS-SURFTYPE-LUT_npp_.*?bin<
2022-06-22 09:18:52(INFO):http.py:105:Connection problem httpsjpssdb.ssec.wisc.edu/cspp_luts_v_5_0/:// >OMPS-SURFTYPE-LUT_npp_.*?bin<
2022-06-22 09:18:52(INFO):http.py:105:Connection problem httpsjpssdb.ssec.wisc.edu/cspp_luts_v_5_0/:// >OMPS-TC-TIMING-PATTERN-GND-PI_npp_.*?bin<
2022-06-22 09:18:52(INFO):http.py:105:Connection problem httpsjpssdb.ssec.wisc.edu/cspp_luts_v_5_0/:// >OMPS-TC-TIMING-PATTERN-GND-PI_npp_.*?bin<
2022-06-22 09:18:53(INFO):http.py:105:Connection problem httpsjpssdb.ssec.wisc.edu/cspp_luts_v_5_0/:// >VIIRS-I2-SDR-DQTT_npp_.*?bin<
2022-06-22 09:18:53(INFO):http.py:105:Connection problem httpsjpssdb.ssec.wisc.edu/cspp_luts_v_5_0/:// >VIIRS-I2-SDR-DQTT_npp_.*?bin<
2022-06-22 09:18:53(INFO):http.py:105:Connection problem httpsjpssdb.ssec.wisc.edu/cspp_luts_v_5_0/:// >VIIRS-M10-SDR-DQTT_npp_.*?bin<
2022-06-22 09:18:53(INFO):http.py:105:Connection problem httpsjpssdb.ssec.wisc.edu/cspp_luts_v_5_0/:// >VIIRS-M10-SDR-DQTT_npp_.*?bin<
2022-06-22 09:18:53(INFO):http.py:105:Connection problem httpsjpssdb.ssec.wisc.edu/cspp_luts_v_5_0/:// >VIIRS-RSBAUTOCAL-VOLT-LUT_npp_.*?bin<
2022-06-22 09:18:53(INFO):http.py:105:Connection problem httpsjpssdb.ssec.wisc.edu/cspp_luts_v_5_0/:// >VIIRS-RSBAUTOCAL-VOLT-LUT_npp_.*?bin<


etc

Any idea what is happening here? And if I can do something to fix this?

Trygve Aspenes
kathys
Posts: 484
Joined: Tue Jun 22, 2010 4:51 pm

Re: sdr_lut.sh from v 3.3.1 fails on ubuntu jammy

Post by kathys »

Hello Trygve,

Sorry for the delay in responding. This is the week of the CSPP Users' Group Meeting. We will investigate your issue, and post an answer as soon as possible.

Kathy
kathys
Posts: 484
Joined: Tue Jun 22, 2010 4:51 pm

Re: sdr_lut.sh from v 3.3.1 fails on ubuntu jammy

Post by kathys »

Hello Trygve,

We suspect that the problem is with the ssl and crypto libraries on Ubuntu. Can you please reset your ancillary and LUT environment variables to http?

export JPSS_REMOTE_LUT_DIR=http://jpssdb.ssec.wisc.edu/cspp_luts_v_5_0/
export JPSS_REMOTE_ANC_DIR=http://jpssdb.ssec.wisc.edu/cspp_v_4_0/ancillary

You will need to reset these after executing the "source $CSPP_SDR_HOME/cspp_sdr_env.sh" script.

Please let us know if this resolves your problem.

Kathy
trygveas
Posts: 5
Joined: Mon Oct 08, 2018 7:18 am

Re: sdr_lut.sh from v 3.3.1 fails on ubuntu jammy

Post by trygveas »

Thanks Kathy

I hope you had a nice CSPP Users' Group Meeting! One day I should go.

I thought I tried setting those env variables. Anyway.

I deleted the installation and did a new unpack of the CSPP source files and set the home directory and running the source. Then setting the variables as you described in the post.

Then I got like this:
ubuntu@cspp-processing-jammy:~/cspp/SDR_3_3$ sdr_luts.sh
Warning overridden default: JPSS_REMOTE_ANC_DIR=http://jpssdb.ssec.wisc.edu/cspp_v_4_0/ancillary
Warning overridden default: JPSS_REMOTE_LUT_DIR=http://jpssdb.ssec.wisc.edu/cspp_luts_v_5_0/
CSPP_DYNAMIC_ANCIL_DIR=/home/ubuntu/cspp/SDR_3_3/anc/cache
2022-06-29 10:31:19(INFO):ancillary_control.py:1176:product version: DPE_VER for CSPP SDR CSPP_SDR_3_3_1
2022-06-29 10:31:19(INFO):ancillary_control.py:1177:JPSS_REMOTE_ANC_DIR = http://jpssdb.ssec.wisc.edu/cspp_v_4_0/ancillary
2022-06-29 10:31:19(INFO):ancillary_control.py:1178:JPSS_REMOTE_LUT_DIR = http://jpssdb.ssec.wisc.edu/cspp_luts_v_5_0/
2022-06-29 10:31:19(INFO):sdr_config_paths.py:256:Checking static data
2022-06-29 10:31:19(INFO):sdr_config_paths.py:270:Found *static_luts* signature at /home/ubuntu/cspp/SDR_3_3/anc/static/shipped_luts
2022-06-29 10:31:19(INFO):sdr_config_paths.py:270:Found *static_tiles* signature at /home/ubuntu/cspp/SDR_3_3/anc/static/ecotiles
2022-06-29 10:31:19(INFO):sdr_config_paths.py:270:Found *stray_luts* signature at /home/ubuntu/cspp/SDR_3_3/anc/static/stray_light_luts
2022-06-29 10:31:19(INFO):sdr_config_paths.py:277:Checking dynamic data
2022-06-29 10:31:19(INFO):sdr_config_paths.py:284:Writable cache directory, /home/ubuntu/cspp/SDR_3_3/anc/cache
2022-06-29 10:31:19(INFO):sdr_config_paths.py:295:end check installation
2022-06-29 10:31:19(INFO):ancillary_control.py:1183:Arguments
Namespace(ancillary='LUTS', debug=False, local=False, package=False, separate=False, time=None, verbosity=2, work_dir='.')
2022-06-29 10:31:19(INFO):ancillary_control.py:1185:CSPP execution work directory is '/home/ubuntu/cspp/SDR_3_3'
CSPP_DYNAMIC_ANCIL_DIR=/home/ubuntu/cspp/SDR_3_3/anc/cache
2022-06-29 10:31:19(INFO):ancillary_control.py:1188:CSPP execution work directory is '/home/ubuntu/cspp/SDR_3_3'
2022-06-29 10:31:19(INFO):sdr_config_paths.py:256:Checking static data
2022-06-29 10:31:19(INFO):sdr_config_paths.py:277:Checking dynamic data
2022-06-29 10:31:19(INFO):sdr_config_paths.py:295:end check installation
CSPP_DYNAMIC_ANCIL_DIR=/home/ubuntu/cspp/SDR_3_3/anc/cache
2022-06-29 10:31:19(INFO):ancillary_control.py:1053:check_for_cache_database_initalization /home/ubuntu/cspp/SDR_3_3/anc/cache/SDR_3_3_DB/db
2022-06-29 10:31:19(INFO):adl_query_dms.py:205:create_cache_database /home/ubuntu/cspp/SDR_3_3
2022-06-29 10:31:19(INFO):log_common.py:93:
( One time initialization, takes some time )

2022-06-29 10:31:19(INFO):adl_query_dms.py:248:st_src os.stat_result(st_mode=16888, st_ino=6166321, st_dev=64513, st_nlink=3, st_uid=1000, st_gid=1000, st_size=4096, st_atime=1656497919, st_mtime=1629399494, st_ctime=1656497921)
2022-06-29 10:31:23(INFO):log_common.py:93:
( Execution Time: 3.520650 Sec )

2022-06-29 10:31:23(INFO):ancillary_control.py:1066:Doing post installation initialization of LUTS database
2022-06-29 10:31:23(INFO):ancillary_control.py:1067:This is a long process, Do not interrupt
2022-06-29 10:32:08(INFO):get.py:306:Add downloaded file VIIRS-SDR-DNB-DN0-LUT_npp_20220531000000Z_20220530000000Z_ee00000000000000Z_PS-1-O-CCR-5819-127-LE-PE_all-_all_all-_ops.bin to cache
2022-06-29 10:33:23(INFO):get.py:306:Add downloaded file OMPS-TC-DARKS-GND-PI_npp_20220621191109Z_20220619120158Z_ee00000000000000Z_PS-1-O-CCR-5958-0482-LE-PE-V002-000_all-_all_all-_ops.bin to cache
2022-06-29 10:33:41(INFO):get.py:306:Add downloaded file OMPS-NP-WAVELENGTH-GND-PI_npp_20220621000000Z_20220624000000Z_ee00000000000000Z_PS-1-O-CCR-5988-134-LE-PE-V001-001_all-_all_all-_ops.bin to cache
2022-06-29 10:35:48(INFO):get.py:306:Add downloaded file VIIRS-SDR-DNB-GAIN-RATIOS-LUT_npp_20220531000000Z_20220530000000Z_ee00000000000000Z_PS-1-O-CCR-5819-107-LE-PE_all-_all_all-_ops.bin to cache
2022-06-29 10:36:51(INFO):http.py:105:Connection problem httpjpssdb.ssec.wisc.edu/cspp_luts_v_5_0/:// >OMPS-NP-OSOL-LUT_npp_.*?bin<
2022-06-29 10:37:11(INFO):http.py:105:Connection problem httpjpssdb.ssec.wisc.edu/cspp_luts_v_5_0/:// >OMPS-NP-OSOL-LUT_npp_.*?bin<
2022-06-29 10:37:32(INFO):http.py:105:Connection problem httpjpssdb.ssec.wisc.edu/cspp_luts_v_5_0/:// >VIIRS-SDR-REFLECTIVE-LUT_npp_.*?bin<
2022-06-29 10:37:52(INFO):http.py:105:Connection problem httpjpssdb.ssec.wisc.edu/cspp_luts_v_5_0/:// >VIIRS-SDR-REFLECTIVE-LUT_npp_.*?bin<
2022-06-29 10:38:12(INFO):http.py:105:Connection problem httpjpssdb.ssec.wisc.edu/cspp_luts_v_5_0/:// >VIIRS-M16-SDR-DQTT_npp_.*?bin<
2022-06-29 10:38:32(INFO):http.py:105:Connection problem httpjpssdb.ssec.wisc.edu/cspp_luts_v_5_0/:// >VIIRS-M16-SDR-DQTT_npp_.*?bin<
2022-06-29 10:38:53(INFO):http.py:105:Connection problem httpjpssdb.ssec.wisc.edu/cspp_luts_v_5_0/:// >OMPS-TC-WAVELENGTH-GND-PI_npp_.*?bin<
2022-06-29 10:39:13(INFO):http.py:105:Connection problem httpjpssdb.ssec.wisc.edu/cspp_luts_v_5_0/:// >OMPS-TC-WAVELENGTH-GND-PI_npp_.*?bin<
2022-06-29 10:39:33(INFO):http.py:105:Connection problem httpjpssdb.ssec.wisc.edu/cspp_luts_v_5_0/:// >VIIRS-M6-SDR-DQTT_npp_.*?bin<
2022-06-29 10:39:53(INFO):http.py:105:Connection problem httpjpssdb.ssec.wisc.edu/cspp_luts_v_5_0/:// >VIIRS-M6-SDR-DQTT_npp_.*?bin<
2022-06-29 10:40:14(INFO):http.py:105:Connection problem httpjpssdb.ssec.wisc.edu/cspp_luts_v_5_0/:// >VIIRS-I5-SDR-DQTT_npp_.*?bin<
2022-06-29 10:40:34(INFO):http.py:105:Connection problem httpjpssdb.ssec.wisc.edu/cspp_luts_v_5_0/:// >VIIRS-I5-SDR-DQTT_npp_.*?bin<
2022-06-29 10:40:54(INFO):http.py:105:Connection problem httpjpssdb.ssec.wisc.edu/cspp_luts_v_5_0/:// >OMPS-NP-FAM-LUT_npp_.*?bin<
2022-06-29 10:41:14(INFO):http.py:105:Connection problem httpjpssdb.ssec.wisc.edu/cspp_luts_v_5_0/:// >OMPS-NP-FAM-LUT_npp_.*?bin<
2022-06-29 10:44:06(INFO):get.py:306:Add downloaded file OMPS-NP-DARKS-GND-PI_npp_20220621185913Z_20220619120158Z_ee00000000000000Z_PS-1-O-CCR-5959-0478-LE-PE-V002-000_all-_all_all-_ops.bin to cache
2022-06-29 10:46:51(INFO):get.py:306:Add downloaded file VIIRS-SDR-DNB-DN0-LUT_j01_20220531000000Z_20220530000000Z_ee00000000000000Z_PS-1-O-CCR-5831-059-LE-PE_all-_all_all-_ops.bin to cache
2022-06-29 10:48:09(INFO):get.py:306:Add downloaded file OMPS-TC-DARKS-GND-PI_j01_20220619172311Z_20220619093029Z_ee00000000000000Z_PS-1-O-CCR-5978-0229-LE-PE-V001-000_all-_all_all-_ops.bin to cache
2022-06-29 10:48:27(INFO):get.py:306:Add downloaded file OMPS-NP-WAVELENGTH-GND-PI_j01_20220613000000Z_20220616000000Z_ee00000000000000Z_PS-1-O-CCR-5987-G248-LE-PE-V101-004_all-_all_all-_ops.bin to cache
2022-06-29 10:48:59(INFO):get.py:306:Add downloaded file VIIRS-SDR-F-PREDICTED-LUT_j01_20211021102930Z_20211021000000Z_ee00000000000000Z_PS-1-O-CCR-5724-007-LE-PE_all-_all_all-_ops.bin to cache
2022-06-29 10:50:03(INFO):get.py:306:Add downloaded file VIIRS-SDR-DNB-GAIN-RATIOS-LUT_j01_20220531000000Z_20220530000000Z_ee00000000000000Z_PS-1-O-CCR-5831-058-LE-PE_all-_all_all-_ops.bin to cache
2022-06-29 10:50:46(INFO):get.py:306:Add downloaded file OMPS-NP-OSOL-LUT_j01_20220613000000Z_20220616000000Z_ee00000000000000Z_PS-1-O-CCR-5987-G248-LE-PE-V101-004_all-_all_all-_ops.bin to cache
2022-06-29 10:52:57(INFO):get.py:306:Add downloaded file OMPS-NP-DARKS-GND-PI_j01_20220619172016Z_20220619093029Z_ee00000000000000Z_PS-1-O-CCR-5979-0229-LE-PE-V001-000_all-_all_all-_ops.bin to cache
2022-06-29 10:54:05(INFO):ancillary_control.py:1072:It is recommended that the user still runs sdr_luts.sh to retrieve any LUTS updated since the release of this software.
2022-06-29 10:54:05(INFO):ancillary_control.py:1073:running sdr_luts.sh and sdr_ancillary.sh is recommended after installation
2022-06-29 10:54:05(INFO):ancillary_control.py:856:Add static tiles
2022-06-29 10:54:10(INFO):log_common.py:93:
( 2022-06-29 10:54:10 Execution Time: 4.264309 Sec Cmd "/home/ubuntu/cspp/SDR_3_3/common/ADL/tools/bin/DmHmiCLI.exe" )

2022-06-29 10:54:10(INFO):ancillary_control.py:1285:LUT update, Normal Completion.


So I guess that is a success?

There are some connection problems reported, but when I reran the sdr_luts.sh it looked ok.

But will the http sites stay available in the future?

Trygve Aspenes
kathys
Posts: 484
Joined: Tue Jun 22, 2010 4:51 pm

Re: sdr_lut.sh from v 3.3.1 fails on ubuntu jammy

Post by kathys »

Hello Trygve,

We hope that you will be able to attend CSPP Users' Group Meetings in the future. I think we and alot of other users are very interested in the things you do with direct broadcast data.

It does appear that your initialization of the CSPP SDR database completed successfully. We are not sure why you received those connection error problems. I am not aware of any issues with our ancillary data server at our end.

The http sites will remain available for the near-future since it seems alot of users have problems with https. If we decide to go with https only ancillary distribution, we will try and provide a warning to users well before that change is made.

Kathy
Post Reply