Page 1 of 1

AIT Framework v2 beta4 tips and user feedback

Posted: Thu Jan 20, 2022 3:58 pm
by graemem
A user reported an issue with AITF v2 beta4 that may cause ancillary data downloads to fail. The problem was indicated by "CRITICAL ERROR: Downloading dynamic data failed" and "TimeLimit expired" messages in output from the `aitf` and `aitf-update-cache` scripts.

The problem was determined to be due to a bug introduced in the AITF v2 beta4 software that results in a 30 second connection timeout erroneously being applied to the ancillary data download itself. Since 30 seconds is likely not enough time for most users to download larger ancillary files, the following workaround is recommended for all users running AITF v2 beta4.

Workaround: Set the CSPP_GEO_AITF_ANCIL_TIMEOUT environment variable as shown below before running the `aitf` or `aitf-update-cache` script. This line can be added to your .bashrc or equivalent.

export CSPP_GEO_AITF_ANCIL_TIMEOUT=7200

This sets the timeout on ancillary data downloads to 7200 seconds (2 hours). Any sufficiently high value can be used.

This issue will be addressed in the next AITF v2 release, at which point the environment setting will no longer be needed.

Please contact csppgeo.issues@ssec.wisc.edu with any questions, or post to this thread.

The CSPP Geo Team

Re: AIT Framework v2 beta4 tips and user feedback

Posted: Tue Feb 22, 2022 12:02 pm
by jbraun
A bug was discovered in the AIT Framework logic that determines what temporal inputs to use, triggered by a specific state of the temporal cache. If a user receives a seg-fault error in processing, and has issues with processing additional timesteps, the error state may have occurred. This has been found to happen if a user initially runs without a temporal product (for example, fog) and then attempts to add it in.

This bug will be fixed in a future beta or in the production release, but in the meantime users can try a workaround that the CSPP Geo team successfully tested, which is to switch to a new, empty temporal cache starting with the first job that includes fog. This is done by setting CSPP_GEO_AITF_TEMPORAL_CACHE to indicate a pre-existing empty directory, and continuing to use that setting for all subsequent jobs.

For example:

Code: Select all

export CSPP_GEO_AITF_TEMPORAL_CACHE=$PWD/cache-temporal-2
mkdir -p $CSPP_GEO_AITF_TEMPORAL_CACHE
After you do this there will be a ramp-up period for winds and fog while temporal context is accumulated, but it should avoid any interruption in other products. You will have to set up cleanup of the new cache to avoid filing your disk.

If you have any additional questions, please contact us at csppgeo.issues@ssec.wisc.edu.

Re: AIT Framework v2 beta4 tips and user feedback

Posted: Fri Mar 11, 2022 12:36 pm
by jbraun
An issue has been identified that can result in degraded Low Cloud and Fog products in the terminator region due to a failure to cache Cloud Top Phase products from previous timesteps. This issue only occurs when aitf is called with option --products or --skip-products to explicitly select or exclude products; if the default set of products is generated the problem does not occur.

Workaround: The recommended workaround, for users who are explicitly selecting Low Cloud and Fog via option --products or --skip-products, is to also explicitly select Cloud Top Phase.

For example, a user who is running with:

Code: Select all

--products fog
Should run with:

Code: Select all

--products actp,fog