Problem with anciliary data both with cache and not

[asierra]

Hello:

A few days ago, the processing of L1b data with AITF started to fail, reporting lack of ancillary data. For some days it worked without the --cache_only flag but then failed again. We first tested our network and apparently it is not in our side.

Code: Select all

>>> Running script (Downloading dynamic data): ['/data/tmp/fd-s20241071340/work/run-01-download']...
Downloading dynamic data...   0% (0/1)

Downloading dynamic data (/data/tmp/fd-s20241071340/work/run-01-download) exited with return code 1
    Wall time   : 471.97086103260517 s
    User time   : 0.158642 s
    System time : 0.024147 s
    Maximum RSS : 21640 kiB
    Block input : 8 operations
    Block output: 248 operations
ERROR: Failed: Downloading dynamic data. /data/tmp/fd-s20241071340/work/run-01-download exited with return code 1
ERROR: Last 20 lines of output from run-01-download
ERROR:     debug: /data/aitf_2_1_cache/gfs_grib2_0.5deg/2024/04/16/gfs.t06z.pgrbf06 already exists (download_all
+pre-lock). Skipping.
ERROR:     debug: AtomicCreateIfMissing(/data/aitf_2_1_cache/gfs_grib2_0.5deg/2024/04/16/gfs.t06z.pgrbf09

In case it can be useful, I uploaded one of the generated log files.

http://132.247.103.154/tmp/aitf.log

We will appreciate any help.

Regards,

Alejandro A. Sierra
LANOT, UNAM

[kathys]

Hello Alejandro. I hope that you are well.

I can answer your question about the AIT failure and the log that shows ancillary data downloads failing. We have experienced an ancillary data outage starting late yesterday afternoon (15 April). I have just finished getting the files up to date. Please check your processing to see if it is now working.

This does not answer your question about what started happening a few days ago. Let me see if anyone has any ideas about what is causing your problems. I will respond in another post.

Cheers,

Kathy

[asierra]

I am not sure either. But after your adjustment, it is working again.

Code: Select all

Cleaning up... 100% (1/1) done 0.4 seconds
>>> Results (Cleaning up): [0]...
Finished in 10.9 minutes
Moving /data/tmp/fd-s20241071810/aitf.log to
+/data/tmp/fd-s20241071810/GOES16_ABI_2KM_FD_2024107_1810_20_aitf.log

Thank you very much!

A.

[kathys]

Alejandro, Graeme is looking at this more closely and is working on a response.

Kathy

[jbraun]

Hi Alejandro,

The team talked and we would ask that if you encounter this problem again to please run your aitf-update-cache processing command with increased verbosity by adding

-vvvv

to the aitf-update-cache script.

This should then emit more debugging information to the screen output which will be useful for us for debugging. The ideal way if possible is to run that command manually and copy what is output to the screen.

Thanks,
~Jess

[jbraun]

Alejandro,

Just a note that I updated the previous post to correct the script and additional information about what output we would want to see the next time you encounter this issue.

Let us know if you have any questions.
Thanks,
~Jess