sdr_ancillary.sh fails to download new needed data

[trygveas]

Hi

Some time the last24 hours the sdr_ancillary.sh fails to download new needed data like this

Code: Select all

sdr_ancillary.sh 
CSPP_DYNAMIC_ANCIL_DIR=/software/CSPP/SDR_4_0/anc/cache
2024-02-21 11:24:31(INFO):ancillary_staging.py:340:product version: DPE_VER for CSPP SDR  CSPP_SDR_4_0_0
2024-02-21 11:24:31(INFO):ancillary_staging.py:341:JPSS_REMOTE_ANC_DIR =  http://jpssdb.ssec.wisc.edu/cspp_v_4_0/ancillary
2024-02-21 11:24:31(INFO):ancillary_staging.py:342:JPSS_REMOTE_LUT_DIR =  http://jpssdb.ssec.wisc.edu/cspp_luts_v_6_0/
2024-02-21 11:24:31(INFO):sdr_config_paths.py:331:Checking dynamic data
2024-02-21 11:24:31(INFO):sdr_config_paths.py:338:Writable cache directory, /software/CSPP/SDR_4_0/anc/cache
2024-02-21 11:24:31(INFO):sdr_config_paths.py:351:end check installation
2024-02-21 11:24:31(INFO):ancillary_staging.py:345:Arguments 
 Namespace(work_dir='.', debug=False, time=None, verbosity=2, local=False, separate=True, package=False, ancillary='cache') 
2024-02-21 11:24:31(INFO):ancillary_staging.py:347:CSPP execution work directory is '/home/ubuntu'
CSPP_DYNAMIC_ANCIL_DIR=/software/CSPP/SDR_4_0/anc/cache
2024-02-21 11:24:31(INFO):ancillary_staging.py:351:CSPP execution work directory is '/home/ubuntu'
CSPP_DYNAMIC_ANCIL_DIR=/software/CSPP/SDR_4_0/anc/cache
2024-02-21 11:24:31(INFO):ancillary_staging.py:154:update_ancillary_cache False npp all
2024-02-21 11:24:33(ERROR):get.py:129:Exception 'NoneType' object has no attribute 'groupdict'
2024-02-21 11:24:33(ERROR):ancillary_staging.py:406:Traceback (most recent call last):
  File "/software/CSPP/SDR_4_0/scripts/common/ancillary_staging.py", line 393, in main
    luts = staging.update_ancillary_cache(
  File "/software/CSPP/SDR_4_0/scripts/common/ancillary_staging.py", line 173, in update_ancillary_cache
    luts += self._get_anc_list_from_fetch(work_dir, desired_dt, remote_pattern,
  File "/software/CSPP/SDR_4_0/scripts/common/ancillary_staging.py", line 125, in _get_anc_list_from_fetch
    back = self._ancillary_and_luts.fetch_anc(work_dir, dataset,
  File "/software/CSPP/SDR_4_0/scripts/common/ancillary_and_luts.py", line 388, in fetch_anc
    initial_files_back = sdr_ancillary(
  File "/software/CSPP/SDR_4_0/libexec/cspp_fetch-0.3.1-py3.9.egg/cspp_fetch/cspp_sdr_entry.py", line 139, in sdr_ancillary
    total_files = get.get_first(both,
  File "/software/CSPP/SDR_4_0/libexec/cspp_fetch-0.3.1-py3.9.egg/cspp_fetch/get.py", line 297, in get_first
    get(remote, local)
  File "/software/CSPP/SDR_4_0/libexec/cspp_fetch-0.3.1-py3.9.egg/cspp_fetch/get.py", line 138, in get
    raise RuntimeError("Couldn't successfully retrieve %s" % (remote))
RuntimeError: Couldn't successfully retrieve http://jpssdb.ssec.wisc.edu/cspp_v_4_0/ancillary/2024_02_21_052/VIIRS-RSBAUTOCAL-HISTORY-AUX_npp_20240221013140Z_20240221000000Z_ee00000000000000Z_-_oebc_ops_all-_ops.h5

2024-02-21 11:24:33(INFO):ancillary_staging.py:442:Ancillary update, Normal Completion. Encountered some problems (refer to previous error messages).

I have nail it down to the file size checking cspp_fetch/getters/http_wrap.py the line `cl=re.compile(r'Content-Length:[ ]+(?P<FS>[0-9]+).*')`

Problem is that the curl -I <important file> gives:

HTTP/1.1 200 OK
date: Wed, 21 Feb 2024 11:16:01 GMT
server: Apache
last-modified: Wed, 21 Feb 2024 05:11:03 GMT
etag: "15ae92-611dd5a76cbac"
accept-ranges: bytes
content-length: 1420946
set-cookie: SERVERID=aspen80; path=/
cache-control: private

Where content-length is all lowercase.

I suspect this is your apache somehow changed the output. Fixing mentioned script gives me the expected output.

Please advise how I should proceed.

Trygve Aspenes

[kathys]

Hello,

The CSPP SDR software is failing due to a problem with ancillary data downloads from our server. We are currently working to resolve the problem. As a temporary solution, use the sdr execution scripts (for example viirs_sdr.sh) with the “-l” option which will bypass attempts to download the files. We will let you know when this problem has been resolved.

We are really sorry for this inconvenience,

Kathy

[kathys]

The problem with the CSPP ancillary downloads that resulted in the failure of the SDR software, has been resolved. It turned out to be a problem with an Apache web server update that was done on February 20. If you modified your SDR script execution to use the "-l" option, please remove this option to keep your local dynamic ancillary database updated.

We apologize for any inconvenience this has caused. If you have any other questions or concerns, please let us know.