Can not run geocat 1.0b

[luthfi_a]

Hi Geoff, thanks for checking the files.
I have run the script again and uploaded the product files as geocat_B06.tar and geocat_hdf files as geocat_hdf.tar.

Looking at the log, it seems the script still work with B05 data, not B06. If the script work for all bands of FLDK data, shall it provide only one output files for level 2? I do apologize I am a newbie in himawari data processing.

Thank you and best regards,
Luthfi

[luthfi_a]

Hi Geoff,

I run the following script and try to use all bands:

geocat_l2.sh --satellite him8 -W /home/luthfi/Desktop/work /home/luthfi/Desktop/hsd/*FLDK* --num_cpu 1 --viewport_xy 100 1200 100 1200 --line_segments 1 --element_segments 1 -vvv

I compare l2 product file with previous attempt (using only B06 in input paremeter), it seems both have the same histogram. I have uploaded the new product file in ftp as luthfi.work.tar.gz.

Suppose that the script work well, I made more trial by removing --viewport_xy 100 1200 100 1200, but it failed. Is it related with hardware resource problem? My system work with 4 GB RAM.

Thank you for your advice. Best regards,
Luthfi

[geoffc]

I ran your test case (for FLDK), and the outputs were normal, so we know there's not an issue with the input data. To clarify, there's no difference between giving "*B05*.DAT" or "*B06*.DAT", the package just uses what you supply to try and construct the filenames of all of the other required files that match your input file glob. You could just give "HS_H08_20190326_2130_*_FLDK_*.DAT", to try and process the single timestamp "20190326_2130" for the full disk "FLDK", having "B05" or "R20_S0110" in the input glob doesn't affect anything.

Could you try again, this time with the command

Code: Select all

geocat_l2.sh --satellite him8 -W /home/luthfi/Desktop/work /home/luthfi/Desktop/hsd/*FLDK* --num_cpu 1 --viewport_xy 100 1200 100 1200 --line_segments 1 --element_segments 1 --geocat_nscans 200 -vvv

where "--geocat_nscans 200" is the maximum number of lines geocat tries to process at once. This might enable you to get around only having 4Gb of memory. Also, could you run the command "ulimit -a" in your processing environment and report the result back.

Geoff

[luthfi_a]

Hi Geoff,

I have tried the last command you have provided, I uploaded the product files as luthfi_nscans.tar.gz. Then I removed viewport_xy parameter to try processing full coverage of the data, but it was failed, and I uploaded the resulted log files as luthfi_full_xy.tar.gz. I also uploaded luthfi_ulimit.txt where I saved the result of using "ulimit -a" command in that file.

Btw I have more question on using geocat software:
1. I have old data from the year 2015, but I checked in http://geodb.ssec.wisc.edu/ancillary/ that the ancillary needed are not there. Could you please help how to process this data?
2. Is file naming shall follow specific format like we have used in test data? If I am not mistaken it is standard file naming from JMA. CMIIW.
3. Is there a way to crop the data to be processed by geometric coordinate? I read that viewport has coordinate as parameter, but I am not sure I can use it for my purpose.

I do apologize for many questions. Thank you and best regards.
Luthfi

[geoffc]

Hi Lutfi,
it looks like your nscans example worked, so in order to try and process the full disk, in addition to the "nscans" option, try to use the option "--xstride 10", which will only process every 10th pixel in the x-direction.

For your "full_xy" example, the error was unrelated to the data size: the geocat binary generates HDF4 files, and then we convert to NetCDF4 using another program. It is the latter that failed due to unexpected metadata in the HDF4 file. When you upload future examples, could you also include the files in "geocat_hdf", in this case the files "/home/luthfi/Desktop/cspp-geo-geocat-1.0.3b/anc/cache/geocat_hdf/HS_H08_20190326_2130_B05_FLDK_R20_S0110_seg_0/geocatL*.HIMAWARI-8.2019085.213000.hdf".

There is currently no way to crop the processing based on a lat/lon coordinate, but the following will give a subset over Indonesia

Code: Select all

geocat_l2.sh --satellite him8 -W /home/luthfi/Desktop/work /home/luthfi/Desktop/hsd/*FLDK* --preserve_cache --viewport_xy 2000 3500 800 3000 --line_segments 1 --element_segments 1 --num_cpu 1 --geocat_nscans 200 --xstride 1 -vvv

Increase the argument to "--xstride" if you need to thin out the data.

As to your other questions:
1. We only store ancillary data for the last few months. Are you able to get newer HSD data? Is there a reason you need to process data from 2015?
2. The filename format of the files you have been using is what geocat expects.
3. Has been answered above.

Regards,
Geoff

[luthfi_a]

Thank you for your guidance, Geoff.

I still have several questions:
1. About xstride argument, if I use bigger number than 1 (10 for example), will it decrease spatial resolution or accuracy?
2. How to define cloud type from level2 product? Is there any document already shared describing about it?
3. About using old data, I am going to make experiment on historical data to analyze cloud product. Then I suppose I have to download ancillary manually and put the files in specific directory. I will consider to learn about it.

Thank you and best regards,
Luthfi

[luthfi_a]

Hi Geoff,
I forgot to let you know that I have uploaded product files and geocat_hdf files for Indonesia subset area, named as xy_indonesia.tar.gz and xy_indonesia_hdf.tar.gz.

Best regards,
Luthfi

[geoffc]

Thanks Lufthi, the "xy_indonesia" files look fine. However I needed the HDF4 files for your "full_xy" example that failed, are you able to provide those?

Geoff

[luthfi_a]

Hi Geoff,
I rerun the command for full_xy, and uploaded the geocat hdf files to ftp named as full_xy_hdf.tar.gz.

Thank you and best regards,
Luthfi

[geoffc]

Hi Luthfi,
I got your files, and there is no data in them, so something went wrong, but with no supporting information I have no way of knowing what happened, as I shouldn't assume the HDF4 files were from the same run as the "full_xy" files you sent me a few days ago.

Can you try the following:

1. Run the command:

Code: Select all

geocat_l2.sh --satellite him8 -W /home/luthfi/Desktop/work /home/luthfi/Desktop/hsd/*FLDK* --preserve_cache --line_segments 1 --element_segments 1 --num_cpu 1 --geocat_nscans 200 --xstride 1 -vvv

2. Collect the contents of the work directory "/home/luthfi/Desktop/work", including all log files, HDF4 files from "/home/luthfi/Desktop/cspp-geo-geocat-1.0.3b/anc/cache/geocat_hdf", and output NetCDF4 files, into a tarball.

3. Send tarball to ftp.ssec.wisc.edu/pub/incoming


Geoff