ACSPO dynamic ancillary data

[hutianxing]

Hello,

I have a question to ASCPO dynamic ancillary data, according to manual, the global sst analysis data like:

"20170205120000-CMC-L4_GHRSST-SSTfnd-CMC0.1deg-GLOB-v02.0-fv03.0.nc"

is required.
But I find that this data update very slow everyday.
It is almost 14:30 GMT now but it is still cannot be downloaded from https://jpssdb.ssec.wisc.edu/cspp_v_2_0 ... 08_22_235/.
Even I run with -q with ACSPO, it still show error, see attachment.
Could help to solve this issue? Thanks!

[kathys]

Hello,

The dynamic ancillary CMC SST file will always be at least one day behind real-time; we acquire and post the file as soon as it is available from the provider. There are times when it can be delayed more than just a day.

The software is not dependent on an exact match of days. It can use files that are not from the data date as long as it falls within a short tolerance. The software will look for files locally first, and then try to download the closest matching file from the ancillary download site.

Are you getting errors when using the software without the "-q" flag, and if so, what does the log file show?

Kathy

[hutianxing]

Hello Kathy,


Without "-q" flag, the command will stuck for a long time, I was wating for more than 1 hour.

In processing dir, it generated a folder named "TMP_******", and only one file exist named "script_file.lftp".

Can help to solve this problem? Thanks.


Regards,
Hu

[kathys]

Hello Hu,

Can you please try executing this command:

wget -v https://jpssdb.ssec.wisc.edu/cspp_v_2_0 ... -fv03.0.nc

If that does not work, can you try this one?

wget -v http://jpssdb.ssec.wisc.edu/cspp_v_2_0/ ... -fv03.0.nc

Let me know if you are successful with either command. Good luck,

Kathy

[hutianxing]

Hello Kathy,

Both commands able to work, maybe it is issue with connection speed, I will test further.

And there's another thing:

I'm testing CSPP MIRS too, and meet some problem:

1. some error in code, in script "cspp_mirs.py" line 398, there's excrescent "ing" and when I run test cases it causes some error.

Code: Select all

nprocs = requested
    if int(nprocs) > number_available:
        LOG.warninging("More processors requested %s than available %d" %
                    (nprocs, number_available))
        nprocs = number_available - 1
        if nprocs <= 0:
            nprocs = 1
    return nprocs

2. When I run MIRS test case, Metop and NOAA-19 is ok, but some error happens when running NOAA-20&21 case:

Code: Select all

Running noaa20 ...
run_mirs.bash -a -d dynanc -W work/noaa20 -p 6 input/noaa20
2024-08-26 20:41:22(INFO):cspp_mirs.py:679:CSPP execution work directory is '/home/test/CSPP_MIRS/CSPP_MIRS_TESTDATA_4_0/work/noaa20'
2024-08-26 20:41:22(INFO):cspp_mirs.py:687:Arguments 
 Namespace(work_dir='work/noaa20', dynanc='dynanc', verbosity=2, local=False, quarantine=False, processor=6, input_directory=['input/noaa20'], resolution='HI', SFR=True, noSFR=False, agg=True, noagg=False, xfiles=True, clean=True, noclean=False) 
2024-08-26 20:41:22(INFO):basics.py:183:environment variable CSPP_DYNAMIC_ANCIL_DIR is set dynanc
2024-08-26 20:41:22(INFO):cspp_mirs.py:698:Cache dir /home/test/CSPP_MIRS/CSPP_MIRS_TESTDATA_4_0/dynanc
2024-08-26 20:41:22(INFO):cspp_mirs.py:714:Static data installation checked.
2024-08-26 20:41:22(INFO):cspp_mirs.py:724:Input directory existence checked
2024-08-26 20:41:22(WARNING):cspp_mirs.py:398:More processors requested 6 than available 4
2024-08-26 20:41:22(INFO):log_common.py:107:
                 ( Create inventory 1 granule groups to process )

2024-08-26 20:41:22(INFO):basics.py:183:environment variable CSPP_DYNAMIC_ANCIL_DIR is set /home/test/CSPP_MIRS/CSPP_MIRS_TESTDATA_4_0/dynanc
2024-08-26 20:41:22(INFO):mirs_ancillary.py:182:[Errno 17] File exists: '/home/test/CSPP_MIRS/CSPP_MIRS_TESTDATA_4_0/dynanc/processed_gfs/work'
2024-08-26 20:41:22(INFO):cspp_mirs.py:154:work /home/test/CSPP_MIRS/CSPP_MIRS_TESTDATA_4_0/work/noaa20 input /home/test/CSPP_MIRS/CSPP_MIRS_TESTDATA_4_0/input/noaa20 
2024-08-26 20:41:23(INFO):cspp_mirs.py:238:20231129 2004426 Produce these ['NPR-MIRS-IMG_v11r9', 'NPR-MIRS-SND_v11r9']
2024-08-26 20:41:23(INFO):basics.py:183:environment variable CSPP_DYNAMIC_ANCIL_DIR is set /home/test/CSPP_MIRS/CSPP_MIRS_TESTDATA_4_0/dynanc
2024-08-26 20:41:23(INFO):log_common.py:107:
                 ( Run /home/test/CSPP_MIRS/CSPP_MIRS_4_0/libexec/app/bin/scripts/n20_scs_nde.bash in /home/test/CSPP_MIRS/CSPP_MIRS_TESTDATA_4_0/work/noaa20/j01_20231129_2004426 )

2024-08-26 20:43:14(INFO):log_common.py:107:
                 ( mirs 2024-08-26 20:43:14 Execution Time: 111.424958 Sec Cmd "/home/test/CSPP_MIRS/CSPP_MIRS_4_0/libexec/app/bin/scripts/n20_scs_nde.bash" )

2024-08-26 20:43:14(INFO):basics.py:269:Mon Aug 26 20:41:23 PDT 2024

2024-08-26 20:43:14(INFO):basics.py:270:
2024-08-26 20:43:14(INFO):basics.py:282:Mon Aug 26 20:41:23 PDT 2024

2024-08-26 20:43:14(INFO):basics.py:283:
2024-08-26 20:43:14(ERROR):basics.py:284:stderr:
2024-08-26 20:43:14(ERROR):basics.py:290:
2024-08-26 20:43:14(INFO):cspp_mirs.py:64:0 products produced in /home/test/CSPP_MIRS/CSPP_MIRS_TESTDATA_4_0/work/noaa20/j01_20231129_2004426
2024-08-26 20:43:14(INFO):cspp_mirs.py:269:Not all products created num_requested 2 != 0 good /home/test/CSPP_MIRS/CSPP_MIRS_TESTDATA_4_0/work/noaa20/j01_20231129_2004426
2024-08-26 20:43:14(INFO):log_common.py:107:
                 ( Product group mirs 1 out of 0 granules processed, 1 products produced successfully )

2024-08-26 20:43:14(INFO):log_common.py:107:
                 ( Tot Execution Time:  "111.985326" Product type "mirs" in /home/test/CSPP_MIRS/CSPP_MIRS_TESTDATA_4_0/work/noaa20/j01_20231129_2004426 )

2024-08-26 20:43:14(INFO):cspp_mirs.py:525:Normal Completion. Encountered some problems (refer to previous error messages).

Seems something error when excuting the "execute_binary_captured_io" function, but I didn't find where it was invoked.

I'm running CSPP MIRS in VMware, with CentOS 7.9 and 4 cores, 16GB memory.


Regards,
Hu

[kathys]

Hello Hu,

ACSPO: After setting the CSPP_ACSPO_HOME environment variable and sourcing the environment script as described in the ACSPO Installation Guide v2.0 on page 6, set the external ancillary website to http instead of https:

export JPSS_REMOTE_ANC_DIR=http://jpssdb.ssec.wisc.edu/cspp_v_2_0/ancillary

Once you have done this, try executing the ACSPO software again without using the “-q” option to see if this resolves your problem.


MiRS: The MiRS target operating system is Rocky Linux 8. It has also been tested on Rocky 9. It will not execute properly on CentOS 7. Sorry.

Kathy

[hutianxing]

Hello Kathy,


Much thanks to your kindly reply, I have test and ACSPO runs well.

Meanwhile I test MIRS on RockyOS 8 and all case test ok.

But I find that the same error I metioned before happens when I run MIRS test case on VMware, seems it always report warning when running with virtual cpu cores.

Regards,
Hu

[kathys]

Hi Hu,

Glad that you are able to successfully execute the ACSPO software now.

MiRS: The MiRS test script uses 6 cores for each execution (-p 6). However, your VM has only 4 cores. That is why you are seeing that error message. You can edit the test script or run the test case manually to use -p 4 or -p 2. That should resolve your problem.

Good luck,

Kathy

[hutianxing]

Hello Kathy,

Thanks to your advise, I modified the test script and it runs well.

And refer to my reply in 26 Aug, there's a little mistake in "$CSPP_MIRS_HOME/libexec/scripts/cspp_mirs.py" line 398, it should be LOG.warning. This will make the user's task exit with error not just return a warning if core is not enough. Maybe you could get colleague to fix it up in next version, thanks.


Regards,
Hu

[kathys]

Hello Hu,

I have provided the information to the software engineer that packaged the CSPP MiRS for release. Thank you for your valuable feedback.

Please let me know if I can help you in some other way.

Cheers,

Kathy