sdr_luts.sh fails to download anciliary data

[asierra]

Hello,

Until one week ago, everything was fine but then we started to get problems to download automatically anciliary data with sdr_luts.sh.

Code: Select all

 sdr_luts.sh
CSPP_DYNAMIC_ANCIL_DIR=/data/opt2/CSPP/SDR_4_0/anc/cache
2024-04-15 18:24:48(INFO):ancillary_and_luts.py:1011:product version: DPE_VER for CSPP SDR  CSPP_SDR_4_0_0
2024-04-15 18:24:48(INFO):ancillary_and_luts.py:1012:JPSS_REMOTE_ANC_DIR =  http://jpssdb.ssec.wisc.edu/cspp_v_4_0/ancillary
2024-04-15 18:24:48(INFO):ancillary_and_luts.py:1013:JPSS_REMOTE_LUT_DIR =  http://jpssdb.ssec.wisc.edu/cspp_luts_v_6_0/
2024-04-15 18:24:48(INFO):sdr_config_paths.py:296:Checking static data
2024-04-15 18:24:48(INFO):sdr_config_paths.py:318:Found *static_luts* signature at /data/opt2/CSPP/SDR_4_0/anc/static/shipped_luts
2024-04-15 18:24:48(INFO):sdr_config_paths.py:318:Found *static_tiles* signature at /data/opt2/CSPP/SDR_4_0/anc/static/ecotiles
2024-04-15 18:24:48(INFO):sdr_config_paths.py:318:Found *stray_luts* signature at /data/opt2/CSPP/SDR_4_0/anc/static/stray_light_luts
2024-04-15 18:24:48(INFO):sdr_config_paths.py:331:Checking dynamic data
2024-04-15 18:24:48(INFO):sdr_config_paths.py:338:Writable cache directory, /data/opt2/CSPP/SDR_4_0/anc/cache
2024-04-15 18:24:48(INFO):sdr_config_paths.py:351:end check installation
2024-04-15 18:24:48(INFO):ancillary_and_luts.py:1016:Arguments 
 Namespace(work_dir='.', debug=False, time=None, verbosity=2, local=False, processor=1, separate=False, package=False, ancillary='LUTS') 
2024-04-15 18:24:48(INFO):ancillary_and_luts.py:1018:CSPP execution work directory is '/home/lanotadm'
CSPP_DYNAMIC_ANCIL_DIR=/data/opt2/CSPP/SDR_4_0/anc/cache
2024-04-15 18:24:48(INFO):ancillary_and_luts.py:1022:CSPP execution work directory is '/home/lanotadm/tmpxetz0qxv'
CSPP_DYNAMIC_ANCIL_DIR=/data/opt2/CSPP/SDR_4_0/anc/cache
2024-04-15 18:24:48(INFO):ancillary_and_luts.py:876:check_for_cache_database_initalization /data/opt2/CSPP/SDR_4_0/anc/cache/SDR_4_0_DB/db
2024-04-15 18:24:48(INFO):ancillary_and_luts.py:908:Database initialized or in process [Errno 17] File exists: '/data/opt2/CSPP/SDR_4_0/anc/cache/SDR_4_0_DB/db'
2024-04-15 18:28:35(WARNING):http_wrap.py:125:Not OK /usr/bin/curl -I  http://jpssdb.ssec.wisc.edu/cspp_luts_v_6_0/OMPS-TC-DARKS-GND-PI_npp_20220821190729Z_20220821103724Z_ee00000000000000Z_PS-1-O-CCR-6057-0491-LE-PE-V002-000_all-_all_all-_ops.bin
2024-04-15 18:28:35(WARNING):http_wrap.py:126:rc 7
2024-04-15 18:28:35(ERROR):http_wrap.py:130:  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:02:06 --:--:--     0curl: (7) Failed connect to jpssdb.ssec.wisc.edu:80; Connection timed out

However, if we try to connect to the server by hand, we can do it.



We don't know what changed from a week to now. We don't think anything changed from our side and our network.

What should we do?

Thank you in advance for any help.

Alejandro A. Sierra,
LANOT, UNAM

[kathys]

Hello Alejandro,

I do not know what is causing the failure of your sdr_luts.sh scripts. To help us investigate the cause, can you please try executing the script again using the "-v" option, and sending us the log results?

For instance:

sdr_luts.sh -v sdrluts_output.txt 2>&1

and send us the sdrluts_output.txt file.

Cheers, Kathy

[asierra]

Hello Kathy:

You can find that file here:

http://132.247.103.154/tmp/sdrluts_output.txt

Cheers,

A.

[scottm]

Try running this from the command line.

/usr/bin/curl -I http://jpssdb.ssec.wisc.edu/cspp_luts_v ... l-_ops.bin

You should see a bunch of output. If this fails it is likely that something changed at your installation. You can also try it using https instead of http

[asierra]

Code: Select all

[lanotadm@tsom03 ~]$ curl -I http://jpssdb.ssec.wisc.edu/cspp_luts_v_6_0/OMPS-TC-DARKS-GND-PI_npp_20220821190729Z_20220821103724Z_ee00000000000000Z_PS-1-O-CCR-6057-0491-LE-PE-V002-000_all-_all_all-_ops.bin
HTTP/1.1 200 OK
date: Tue, 16 Apr 2024 18:53:36 GMT
server: Apache
last-modified: Fri, 26 Aug 2022 20:04:09 GMT
etag: "1071a8-5e72a6abd8040"
accept-ranges: bytes
Content-Length: 1077672
content-type: application/octet-stream
set-cookie: SERVERID=elm80; path=/
cache-control: private

Does it help?

[kathys]

Alejandro, thank you for sending us this information. We are not sure what the problem is.

- The sdr_luts.sh -v log file that you sent shows most of the files were downloaded successfully, but not all of them.
- The curl command that you executed looks fine.

So I think the next thing to do is to get more details on when you are executing the sdr_luts.sh command. The next time that you see a problem with the downloads, let me know right away, and I can get our Technical Computing group to check on our end if they see any problems. They may have some commands that you can run that will help them trace where the problem might be. They have not changed anything in the last couple of weeks on our end with the LUT distribution website.

I am sorry I do not have a quick and easy solution for you.

Kathy