Issue in Generating True Color image

Post Reply
rajashekar_g
Posts: 4
Joined: Thu Aug 13, 2020 1:27 am

Issue in Generating True Color image

Post by rajashekar_g »

I am attempting to process Himawari 8 full disk data to generate products over India in the Albers Equal Area projection using geo2grid.

In my first attempt I used Himawari-8 Standard Data (Full disk) from http://www.data.jma.go.jp/mscweb/en/him ... _hisd.html

I was able to generate all the individual bands and the natural color image successfully. I loaded the natural color tif in QGIS and overlaid it with a vector boundary of India in AEA and verified that it was in the desired projection.

The true color product is however not generated.

I have attached below

a) the command line script

Code: Select all

#!/bin/bash
echo "HIMAWARI PROCESSING"
$GEO2GRID_HOME/bin/geo2grid.sh \
 -r ahi_hsd \
 -w geotiff \
 -f /home/nrsc/him/data \
 --progress \
 -v \
 -p true_color  \
 --ll-bbox 68 6 98 38 \
 --grid-configs /home/nrsc/geo2grid_v_1_0_1/grid_configs/alb_ind.conf -g alb_ind 
b) my projection file

Code: Select all

# grid_name,proj4,proj4_str,width,height,pixel_size_x,pixel_size_y,origin_x,origin_y
alb_ind,proj4,+proj=aea +lat_1=28 +lat_2=12 +lat_0=20 +lon_0=78 +x_0=2000000 +y_0=2000000 +ellps=WGS84 +datum=WGS84 +units=m +no_defs ,None,None,1000,1000,None,None
and c) the contents of the log file

Code: Select all

[2020-08-14 00:13:48,825] : PID   7905 : DEBUG    : ahi_hsd_geotiff : main : Starting script with arguments: /home/nrsc/geo2grid_v_1_0_1/lib/python3.7/site-packages/polar2grid/glue.py -r ahi_hsd -w geotiff -f /home/nrsc/him/data --progress -v -p true_color --ll-bbox 68 6 98 38 --grid-configs /home/nrsc/geo2grid_v_1_0_1/grid_configs/alb_ind.conf -g alb_ind -vv
[2020-08-14 00:13:48,828] : PID   7905 : INFO     : ahi_hsd_geotiff : main : Sorting and reading input files...
[2020-08-14 00:13:48,828] : PID   7905 : DEBUG    : satpy.scene : __init__ : Setting 'PPP_CONFIG_DIR' to '/home/nrsc/geo2grid_v_1_0_1/etc/satpy'
[2020-08-14 00:13:48,844] : PID   7905 : DEBUG    : satpy.readers : read_reader_config : Reading ['/home/nrsc/geo2grid_v_1_0_1/lib/python3.7/site-packages/satpy/etc/readers/ahi_hsd.yaml']
[2020-08-14 00:13:49,316] : PID   7905 : DEBUG    : satpy.readers.yaml_reader : create_filehandlers : Assigning to ahi_hsd: ['/home/nrsc/him/data/HS_H08_20170623_0250_B01_FLDK_R10_S0910.DAT.bz2', '/home/nrsc/him/data/HS_H08_20170623_0250_B01_FLDK_R10_S0210.DAT.bz2', '/home/nrsc/him/data/
<snip>
[2020-08-14 00:16:17,077] : PID   7905 : DEBUG    : satpy.composites : load_sensor_composites : Looking for composites config file ahi.yaml
[2020-08-14 00:16:17,177] : PID   7905 : DEBUG    : satpy.composites : load_sensor_composites : Looking for composites config file visir.yaml
[2020-08-14 00:16:17,256] : PID   7905 : DEBUG    : polar2grid.core.script_utils : rename_log_file : Log renamed from '/home/nrsc/him/ahi_hsd_geotiff_fail.log' to 'ahi_hsd_geotiff_20170623_025020.log'
[2020-08-14 00:16:17,256] : PID   7905 : INFO     : ahi_hsd_geotiff : main : Loading product metadata from files...
[2020-08-14 00:16:17,264] : PID   7905 : DEBUG    : satpy.node : _get_compositor_prereqs : Skipping optional solar_zenith_angle: Unknown dataset solar_zenith_angle
[2020-08-14 00:16:17,480] : PID   7905 : DEBUG    : ahi_hsd : _read_header : Time_interval: 2017-06-23 02:50:21.213544 - 2017-06-23 02:50:54.568814
[2020-08-14 00:16:17,555] : PID   7905 : DEBUG    : ahi_hsd : read_band : Reading time 0:00:00.075322
<snip>
[2020-08-14 00:16:20,951] : PID   7905 : DEBUG    : satpy.composites : _apply_correction : Apply the standard sun-zenith correction [1/cos(sunz)]
[2020-08-14 00:16:20,994] : PID   7905 : DEBUG    : satpy.composites : __call__ : Sun-zenith correction applied. Computation time:   0.1 (sec)
[2020-08-14 00:16:21,572] : PID   7905 : INFO     : satpy.composites : __call__ : Removing Rayleigh scattering with atmosphere 'us-standard' and aerosol type 'marine_clean_aerosol' for 'B03'
[2020-08-14 00:16:21,579] : PID   7905 : INFO     : pyspectral.rayleigh : __init__ : Atmosphere chosen: us-standard
[2020-08-14 00:16:21,593] : PID   7905 : DEBUG    : pyspectral.rayleigh : __init__ : LUT filename: /home/nrsc/geo2grid_v_1_0_1/pyspectral_data/marine_clean_aerosol/rayleigh_lut_us-standard.h5
[2020-08-14 00:16:21,604] : PID   7905 : DEBUG    : pyspectral.rsr_reader : _get_filename : Filename: /home/nrsc/geo2grid_v_1_0_1/pyspectral_data/rsr_ahi_Himawari-8.h5
[2020-08-14 00:16:21,604] : PID   7905 : DEBUG    : pyspectral.rsr_reader : __init__ : Filename: /home/nrsc/geo2grid_v_1_0_1/pyspectral_data/rsr_ahi_Himawari-8.h5
[2020-08-14 00:16:21,625] : PID   7905 : DEBUG    : pyspectral.rsr_reader : load : No detectors found - assume only one...
[2020-08-14 00:16:21,674] : PID   7905 : DEBUG    : pyspectral.rayleigh : get_effective_wavelength : Band name: ch3  Effective wavelength: 0.635378
[2020-08-14 00:16:21,782] : PID   7905 : DEBUG    : pyspectral.rayleigh : get_reflectance : Time - Interpolation: 0.056167
[2020-08-14 00:16:21,793] : PID   7905 : DEBUG    : satpy.scene : _generate_composite : Delaying generation of DatasetID(name='B01', wavelength=(0.45, 0.47, 0.49), resolution=1000, polarization=None, calibration='reflectance', level=None, modifiers=('sunz_corrected', 'rayleigh_corrected')) because of incompatible areas
[2020-08-14 00:16:21,796] : PID   7905 : DEBUG    : satpy.composites : __call__ : Applying sun zen correction
[2020-08-14 00:16:21,796] : PID   7905 : DEBUG    : satpy.composites : __call__ : Computing sun zenith angles.
[2020-08-14 00:16:21,857] : PID   7905 : DEBUG    : satpy.composites : _apply_correction : Apply the standard sun-zenith correction [1/cos(sunz)]
[2020-08-14 00:16:21,899] : PID   7905 : DEBUG    : satpy.composites : __call__ : Sun-zenith correction applied. Computation time:   0.1 (sec)
[2020-08-14 00:16:21,899] : PID   7905 : DEBUG    : satpy.scene : _generate_composite : Delaying generation of DatasetID(name='B02', wavelength=(0.49, 0.51, 0.53), resolution=1000, polarization=None, calibration='reflectance', level=None, modifiers=('sunz_corrected', 'rayleigh_corrected')) because of incompatible areas
[2020-08-14 00:16:21,901] : PID   7905 : DEBUG    : satpy.composites : __call__ : Applying sun zen correction
[2020-08-14 00:16:21,901] : PID   7905 : DEBUG    : satpy.composites : __call__ : Computing sun zenith angles.
[2020-08-14 00:16:21,963] : PID   7905 : DEBUG    : satpy.composites : _apply_correction : Apply the standard sun-zenith correction [1/cos(sunz)]
[2020-08-14 00:16:22,004] : PID   7905 : DEBUG    : satpy.composites : __call__ : Sun-zenith correction applied. Computation time:   0.1 (sec)
[2020-08-14 00:16:22,005] : PID   7905 : DEBUG    : satpy.scene : _get_prereq_datasets : Delaying generation of DatasetID(name='green', wavelength=None, resolution=None, polarization=None, calibration=None, level=None, modifiers=None) because of dependency's delayed generation: DatasetID(name='B04', wavelength=(0.83, 0.85, 0.87), resolution=1000, polarization=None, calibration='reflectance', level=None, modifiers=('sunz_corrected',))
[2020-08-14 00:16:22,005] : PID   7905 : DEBUG    : satpy.scene : _get_prereq_datasets : Delayed prerequisite for 'DatasetID(name='green', wavelength=None, resolution=None, polarization=None, calibration=None, level=None, modifiers=None)': 'DatasetID(name='B04', wavelength=(0.83, 0.85, 0.87), resolution=1000, polarization=None, calibration='reflectance', level=None, modifiers=('sunz_corrected',))'
[2020-08-14 00:16:22,006] : PID   7905 : DEBUG    : satpy.scene : _get_prereq_datasets : Delaying generation of DatasetID(name='true_color', wavelength=None, resolution=None, polarization=None, calibration=None, level=None, modifiers=None) because of dependency's delayed generation: DatasetID(name='B01', wavelength=(0.45, 0.47, 0.49), resolution=1000, polarization=None, calibration='reflectance', level=None, modifiers=('sunz_corrected', 'rayleigh_corrected'))
[2020-08-14 00:16:22,006] : PID   7905 : DEBUG    : satpy.scene : _get_prereq_datasets : Delayed prerequisite for 'DatasetID(name='true_color', wavelength=None, resolution=None, polarization=None, calibration=None, level=None, modifiers=None)': 'DatasetID(name='B01', wavelength=(0.45, 0.47, 0.49), resolution=1000, polarization=None, calibration='reflectance', level=None, modifiers=('sunz_corrected', 'rayleigh_corrected'))'
[2020-08-14 00:16:22,007] : PID   7905 : DEBUG    : satpy.scene : _generate_composite : Delaying generation of DatasetID(name='B02', wavelength=(0.49, 0.51, 0.53), resolution=1000, polarization=None, calibration='reflectance', level=None, modifiers=('sunz_corrected', 'rayleigh_corrected')) because of incompatible areas
[2020-08-14 00:16:22,011] : PID   7905 : DEBUG    : satpy.scene : _get_prereq_datasets : Delaying generation of DatasetID(name='green', wavelength=None, resolution=None, polarization=None, calibration=None, level=None, modifiers=None) because of dependency's delayed generation: DatasetID(name='B04', wavelength=(0.83, 0.85, 0.87), resolution=1000, polarization=None, calibration='reflectance', level=None, modifiers=('sunz_corrected',))
[2020-08-14 00:16:22,012] : PID   7905 : DEBUG    : satpy.scene : _get_prereq_datasets : Delayed prerequisite for 'DatasetID(name='green', wavelength=None, resolution=None, polarization=None, calibration=None, level=None, modifiers=None)': 'DatasetID(name='B04', wavelength=(0.83, 0.85, 0.87), resolution=1000, polarization=None, calibration='reflectance', level=None, modifiers=('sunz_corrected',))'
[2020-08-14 00:16:22,012] : PID   7905 : WARNING  : satpy.scene : load : The following datasets were not created and may require resampling to be generated: DatasetID(name='true_color', wavelength=None, resolution=None, polarization=None, calibration=None, level=None, modifiers=None)
[2020-08-14 00:16:22,012] : PID   7905 : DEBUG    : satpy.scene : unload : Unloading dataset: DatasetID(name='B01', wavelength=(0.45, 0.47, 0.49), resolution=1000, polarization=None, calibration='reflectance', level=None, modifiers=())
[2020-08-14 00:16:22,013] : PID   7905 : DEBUG    : satpy.scene : unload : Unloading dataset: DatasetID(name='B04', wavelength=(0.83, 0.85, 0.87), resolution=1000, polarization=None, calibration='reflectance', level=None, modifiers=())
[2020-08-14 00:16:22,013] : PID   7905 : DEBUG    : satpy.scene : unload : Unloading dataset: DatasetID(name='B03', wavelength=(0.62, 0.64, 0.66), resolution=500, polarization=None, calibration='reflectance', level=None, modifiers=())
[2020-08-14 00:16:22,014] : PID   7905 : DEBUG    : satpy.scene : unload : Unloading dataset: DatasetID(name='B02', wavelength=(0.49, 0.51, 0.53), resolution=1000, polarization=None, calibration='reflectance', level=None, modifiers=())
[2020-08-14 00:16:22,037] : PID   7905 : DEBUG    : polar2grid.grids.manager : __init__ : Loading grid configuration '/home/nrsc/geo2grid_v_1_0_1/grid_configs/alb_ind.conf'
[2020-08-14 00:16:22,039] : PID   7905 : DEBUG    : satpy.scene : __init__ : Setting 'PPP_CONFIG_DIR' to '/home/nrsc/geo2grid_v_1_0_1/etc/satpy'
[2020-08-14 00:16:22,265] : PID   7905 : DEBUG    : ahi_hsd_geotiff : main : Products to preserve resolution for: set()
[2020-08-14 00:16:22,265] : PID   7905 : DEBUG    : ahi_hsd_geotiff : main : Products to use new resolution for: {DatasetID(name='true_color', wavelength=None, resolution=None, polarization=None, calibration=None, level=None, modifiers=None)}
[2020-08-14 00:16:39,377] : PID   7905 : DEBUG    : ahi_hsd_geotiff : main : Setting default resampling to 'nearest' for grid 'alb_ind'
[2020-08-14 00:16:39,377] : PID   7905 : INFO     : ahi_hsd_geotiff : main : Resampling data to 'alb_ind'
[2020-08-14 00:16:39,377] : PID   7905 : DEBUG    : satpy.scene : __init__ : Setting 'PPP_CONFIG_DIR' to '/home/nrsc/geo2grid_v_1_0_1/etc/satpy'
[2020-08-14 00:16:39,378] : PID   7905 : DEBUG    : satpy.scene : _resampled_scene : Resampling DatasetID(name='B01', wavelength=(0.45, 0.47, 0.49), resolution=1000, polarization=None, calibration='reflectance', level=None, modifiers=('sunz_corrected',))
<snip>

[2020-08-14 00:17:07,770] : PID   7905 : DEBUG    : numpy : write : Warning: invalid value encountered in subtract

[2020-08-14 00:17:07,827] : PID   7905 : DEBUG    : satpy.resample : _check_numpy_cache : Check if ./resample_lut-83e41c358626e756fa217e0727220ce50d1aa23e.npz exists
[2020-08-14 00:17:07,827] : PID   7905 : DEBUG    : satpy.resample : precompute : Computing kd-tree parameters
[2020-08-14 00:17:07,965] : PID   7905 : DEBUG    : satpy.resample : compute : Resampling None
[2020-08-14 00:17:08,079] : PID   7905 : DEBUG    : satpy.scene : _resampled_scene : Resampling DatasetID(name='B03', wavelength=(0.62, 0.64, 0.66), resolution=500, polarization=None, calibration='reflectance', level=None, modifiers=('sunz_corrected', 'rayleigh_corrected'))
[2020-08-14 00:17:08,080] : PID   7905 : DEBUG    : satpy.resample : _check_numpy_cache : Check if ./resample_lut-83e41c358626e756fa217e0727220ce50d1aa23e.npz exists
[2020-08-14 00:17:08,080] : PID   7905 : DEBUG    : satpy.resample : precompute : Computing kd-tree parameters
[2020-08-14 00:17:08,216] : PID   7905 : DEBUG    : satpy.resample : compute : Resampling None
[2020-08-14 00:17:08,240] : PID   7905 : DEBUG    : satpy.scene : _resampled_scene : Resampling DatasetID(name='B04', wavelength=(0.83, 0.85, 0.87), resolution=1000, polarization=None, calibration='reflectance', level=None, modifiers=('sunz_corrected',))
[2020-08-14 00:17:08,241] : PID   7905 : DEBUG    : satpy.resample : _check_numpy_cache : Check if ./resample_lut-ab6c280f5acc738c9f2914fe0fb4ed4eb72be32b.npz exists
[2020-08-14 00:17:08,241] : PID   7905 : DEBUG    : satpy.resample : precompute : Computing kd-tree parameters
[2020-08-14 00:17:08,364] : PID   7905 : DEBUG    : satpy.resample : compute : Resampling None
[2020-08-14 00:17:08,568] : PID   7905 : INFO     : satpy.composites : __call__ : Removing Rayleigh scattering with atmosphere 'us-standard' and aerosol type 'marine_clean_aerosol' for 'B02'
[2020-08-14 00:17:08,569] : PID   7905 : INFO     : pyspectral.rayleigh : __init__ : Atmosphere chosen: us-standard
[2020-08-14 00:17:08,570] : PID   7905 : DEBUG    : pyspectral.rayleigh : __init__ : LUT filename: /home/nrsc/geo2grid_v_1_0_1/pyspectral_data/marine_clean_aerosol/rayleigh_lut_us-standard.h5
[2020-08-14 00:17:08,571] : PID   7905 : DEBUG    : pyspectral.rsr_reader : _get_filename : Filename: /home/nrsc/geo2grid_v_1_0_1/pyspectral_data/rsr_ahi_Himawari-8.h5
[2020-08-14 00:17:08,571] : PID   7905 : DEBUG    : pyspectral.rsr_reader : __init__ : Filename: /home/nrsc/geo2grid_v_1_0_1/pyspectral_data/rsr_ahi_Himawari-8.h5
[2020-08-14 00:17:08,572] : PID   7905 : DEBUG    : pyspectral.rsr_reader : load : No detectors found - assume only one...
[2020-08-14 00:17:08,644] : PID   7905 : DEBUG    : pyspectral.rayleigh : get_effective_wavelength : Band name: ch2  Effective wavelength: 0.509195
[2020-08-14 00:17:08,736] : PID   7905 : DEBUG    : pyspectral.rayleigh : get_reflectance : Time - Interpolation: 0.050131
[2020-08-14 00:17:08,913] : PID   7905 : INFO     : satpy.composites : __call__ : Removing Rayleigh scattering with atmosphere 'us-standard' and aerosol type 'marine_clean_aerosol' for 'B01'
[2020-08-14 00:17:08,915] : PID   7905 : INFO     : pyspectral.rayleigh : __init__ : Atmosphere chosen: us-standard
[2020-08-14 00:17:08,915] : PID   7905 : DEBUG    : pyspectral.rayleigh : __init__ : LUT filename: /home/nrsc/geo2grid_v_1_0_1/pyspectral_data/marine_clean_aerosol/rayleigh_lut_us-standard.h5
[2020-08-14 00:17:08,916] : PID   7905 : DEBUG    : pyspectral.rsr_reader : _get_filename : Filename: /home/nrsc/geo2grid_v_1_0_1/pyspectral_data/rsr_ahi_Himawari-8.h5
[2020-08-14 00:17:08,916] : PID   7905 : DEBUG    : pyspectral.rsr_reader : __init__ : Filename: /home/nrsc/geo2grid_v_1_0_1/pyspectral_data/rsr_ahi_Himawari-8.h5
[2020-08-14 00:17:08,917] : PID   7905 : DEBUG    : pyspectral.rsr_reader : load : No detectors found - assume only one...
[2020-08-14 00:17:08,976] : PID   7905 : DEBUG    : pyspectral.rayleigh : get_effective_wavelength : Band name: ch1  Effective wavelength: 0.469333
[2020-08-14 00:17:09,069] : PID   7905 : DEBUG    : pyspectral.rayleigh : get_reflectance : Time - Interpolation: 0.053037
[2020-08-14 00:17:09,079] : PID   7905 : INFO     : satpy.composites.ahi : __call__ : Boosting vegetation on green band
[2020-08-14 00:17:09,100] : PID   7905 : DEBUG    : satpy.composites : __call__ : Sharpening image with high resolution red band
[2020-08-14 00:17:09,202] : PID   7905 : DEBUG    : satpy.scene : unload : Unloading dataset: DatasetID(name='B01', wavelength=(0.45, 0.47, 0.49), resolution=1000, polarization=None, calibration='reflectance', level=None, modifiers=('sunz_corrected',))
[2020-08-14 00:17:09,203] : PID   7905 : DEBUG    : satpy.scene : unload : Unloading dataset: DatasetID(name='B02', wavelength=(0.49, 0.51, 0.53), resolution=1000, polarization=None, calibration='reflectance', level=None, modifiers=('sunz_corrected',))
[2020-08-14 00:17:09,203] : PID   7905 : DEBUG    : satpy.scene : unload : Unloading dataset: DatasetID(name='B03', wavelength=(0.62, 0.64, 0.66), resolution=500, polarization=None, calibration='reflectance', level=None, modifiers=('sunz_corrected',))
[2020-08-14 00:17:09,203] : PID   7905 : DEBUG    : satpy.scene : unload : Unloading dataset: DatasetID(name='B03', wavelength=(0.62, 0.64, 0.66), resolution=500, polarization=None, calibration='reflectance', level=None, modifiers=('sunz_corrected', 'rayleigh_corrected'))
[2020-08-14 00:17:09,203] : PID   7905 : DEBUG    : satpy.scene : unload : Unloading dataset: DatasetID(name='B04', wavelength=(0.83, 0.85, 0.87), resolution=1000, polarization=None, calibration='reflectance', level=None, modifiers=('sunz_corrected',))
[2020-08-14 00:17:09,203] : PID   7905 : DEBUG    : satpy.scene : unload : Unloading dataset: DatasetID(name='B02', wavelength=(0.49, 0.51, 0.53), resolution=1000, polarization=None, calibration='reflectance', level=None, modifiers=('sunz_corrected', 'rayleigh_corrected'))
[2020-08-14 00:17:09,203] : PID   7905 : DEBUG    : satpy.scene : unload : Unloading dataset: DatasetID(name='B01', wavelength=(0.45, 0.47, 0.49), resolution=1000, polarization=None, calibration='reflectance', level=None, modifiers=('sunz_corrected', 'rayleigh_corrected'))
[2020-08-14 00:17:09,204] : PID   7905 : DEBUG    : satpy.scene : unload : Unloading dataset: DatasetID(name='green', wavelength=None, resolution=1000, polarization=None, calibration=None, level=None, modifiers=None)
[2020-08-14 00:17:09,204] : PID   7905 : DEBUG    : satpy.writers : read_writer_config : Reading ['/home/nrsc/geo2grid_v_1_0_1/lib/python3.7/site-packages/satpy/etc/writers/geotiff.yaml']
[2020-08-14 00:17:09,484] : PID   7905 : DEBUG    : satpy.writers : apply : Enhancement configuration options: [{'name': 'crefl_scaling', 'method': <function crefl_scaling at 0x7fc3ea8aa710>, 'kwargs': {'idx': [0.0, 25.0, 55.0, 100.0, 255.0], 'sc': [0.0, 90.0, 140.0, 175.0, 255.0]}}]
[2020-08-14 00:17:09,484] : PID   7905 : DEBUG    : satpy.enhancements : crefl_scaling : Applying the crefl_scaling
[2020-08-14 00:17:09,665] : PID   7905 : INFO     : ahi_hsd_geotiff : main : Computing products and saving data to writers...
[2020-08-14 00:18:58,067] : PID   7905 : ERROR    : ahi_hsd_geotiff : exc_handler : Unexpected error. Enable debug messages (-vvv) or see log file for details.
[2020-08-14 00:18:58,392] : PID   7905 : DEBUG    : ahi_hsd_geotiff : exc_handler : Unexpected error exception: 
Traceback (most recent call last):
  File "/home/nrsc/geo2grid_v_1_0_1/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/nrsc/geo2grid_v_1_0_1/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/nrsc/geo2grid_v_1_0_1/lib/python3.7/site-packages/polar2grid/glue.py", line 436, in <module>
    sys.exit(main())
  File "/home/nrsc/geo2grid_v_1_0_1/lib/python3.7/site-packages/polar2grid/glue.py", line 430, in main
    compute_writer_results(to_save)
  File "/home/nrsc/geo2grid_v_1_0_1/lib/python3.7/site-packages/satpy/writers/__init__.py", line 553, in compute_writer_results
    da.compute(delayeds)
  File "/home/nrsc/geo2grid_v_1_0_1/lib/python3.7/site-packages/dask/base.py", line 436, in compute
    results = schedule(dsk, keys, **kwargs)
  File "/home/nrsc/geo2grid_v_1_0_1/lib/python3.7/site-packages/dask/threaded.py", line 81, in get
    **kwargs
  File "/home/nrsc/geo2grid_v_1_0_1/lib/python3.7/site-packages/dask/local.py", line 486, in get_async
    raise_exception(exc, tb)
  File "/home/nrsc/geo2grid_v_1_0_1/lib/python3.7/site-packages/dask/local.py", line 316, in reraise
    raise exc
  File "/home/nrsc/geo2grid_v_1_0_1/lib/python3.7/site-packages/dask/local.py", line 222, in execute_task
    result = _execute_task(task, data)
  File "/home/nrsc/geo2grid_v_1_0_1/lib/python3.7/site-packages/dask/core.py", line 119, in _execute_task
    return func(*args2)
  File "/home/nrsc/geo2grid_v_1_0_1/lib/python3.7/site-packages/dask/optimization.py", line 982, in __call__
    return core.get(self.dsk, self.outkey, dict(zip(self.inkeys, args)))
  File "/home/nrsc/geo2grid_v_1_0_1/lib/python3.7/site-packages/dask/core.py", line 149, in get
    result = _execute_task(task, cache)
  File "/home/nrsc/geo2grid_v_1_0_1/lib/python3.7/site-packages/dask/core.py", line 118, in _execute_task
    args2 = [_execute_task(a, cache) for a in args]
  File "/home/nrsc/geo2grid_v_1_0_1/lib/python3.7/site-packages/dask/core.py", line 118, in <listcomp>
    args2 = [_execute_task(a, cache) for a in args]
  File "/home/nrsc/geo2grid_v_1_0_1/lib/python3.7/site-packages/dask/core.py", line 118, in _execute_task
    args2 = [_execute_task(a, cache) for a in args]
  File "/home/nrsc/geo2grid_v_1_0_1/lib/python3.7/site-packages/dask/core.py", line 118, in <listcomp>
    args2 = [_execute_task(a, cache) for a in args]
  File "/home/nrsc/geo2grid_v_1_0_1/lib/python3.7/site-packages/dask/core.py", line 115, in _execute_task
    return [_execute_task(a, cache) for a in arg]
  File "/home/nrsc/geo2grid_v_1_0_1/lib/python3.7/site-packages/dask/core.py", line 115, in <listcomp>
    return [_execute_task(a, cache) for a in arg]
  File "/home/nrsc/geo2grid_v_1_0_1/lib/python3.7/site-packages/dask/core.py", line 115, in _execute_task
    return [_execute_task(a, cache) for a in arg]
  File "/home/nrsc/geo2grid_v_1_0_1/lib/python3.7/site-packages/dask/core.py", line 115, in <listcomp>
    return [_execute_task(a, cache) for a in arg]
  File "/home/nrsc/geo2grid_v_1_0_1/lib/python3.7/site-packages/dask/core.py", line 120, in _execute_task
    elif not ishashable(arg):
  File "/home/nrsc/geo2grid_v_1_0_1/lib/python3.7/site-packages/dask/core.py", line 20, in ishashable
    hash(x)
  File "/home/nrsc/geo2grid_v_1_0_1/lib/python3.7/site-packages/pyproj/crs/crs.py", line 869, in __hash__
    return hash(self.to_wkt())
  File "pyproj/_crs.pyx", line 451, in pyproj._crs.Base.to_wkt
  File "pyproj/_crs.pyx", line 120, in pyproj._crs._to_wkt
  File "pyproj/_crs.pyx", line 24, in pyproj._crs.cstrdecode
  File "/home/nrsc/geo2grid_v_1_0_1/lib/python3.7/site-packages/pyproj/compat.py", line 21, in pystrdecode
    return cstr.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 2: invalid start byte
[2020-08-14 00:19:02,852] : PID   7905 : ERROR    : traceback : exc_handler : 'utf-8' codec can't decode byte 0x80 in position 2: invalid start byte
Traceback (most recent call last):
  File "/home/nrsc/geo2grid_v_1_0_1/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/nrsc/geo2grid_v_1_0_1/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/nrsc/geo2grid_v_1_0_1/lib/python3.7/site-packages/polar2grid/glue.py", line 436, in <module>
    sys.exit(main())
  File "/home/nrsc/geo2grid_v_1_0_1/lib/python3.7/site-packages/polar2grid/glue.py", line 430, in main
    compute_writer_results(to_save)
  File "/home/nrsc/geo2grid_v_1_0_1/lib/python3.7/site-packages/satpy/writers/__init__.py", line 553, in compute_writer_results
    da.compute(delayeds)
  File "/home/nrsc/geo2grid_v_1_0_1/lib/python3.7/site-packages/dask/base.py", line 436, in compute
    results = schedule(dsk, keys, **kwargs)
  File "/home/nrsc/geo2grid_v_1_0_1/lib/python3.7/site-packages/dask/threaded.py", line 81, in get
    **kwargs
  File "/home/nrsc/geo2grid_v_1_0_1/lib/python3.7/site-packages/dask/local.py", line 486, in get_async
    raise_exception(exc, tb)
  File "/home/nrsc/geo2grid_v_1_0_1/lib/python3.7/site-packages/dask/local.py", line 316, in reraise
    raise exc
  File "/home/nrsc/geo2grid_v_1_0_1/lib/python3.7/site-packages/dask/local.py", line 222, in execute_task
    result = _execute_task(task, data)
  File "/home/nrsc/geo2grid_v_1_0_1/lib/python3.7/site-packages/dask/core.py", line 119, in _execute_task
    return func(*args2)
  File "/home/nrsc/geo2grid_v_1_0_1/lib/python3.7/site-packages/dask/optimization.py", line 982, in __call__
    return core.get(self.dsk, self.outkey, dict(zip(self.inkeys, args)))
  File "/home/nrsc/geo2grid_v_1_0_1/lib/python3.7/site-packages/dask/core.py", line 149, in get
    result = _execute_task(task, cache)
  File "/home/nrsc/geo2grid_v_1_0_1/lib/python3.7/site-packages/dask/core.py", line 118, in _execute_task
    args2 = [_execute_task(a, cache) for a in args]
  File "/home/nrsc/geo2grid_v_1_0_1/lib/python3.7/site-packages/dask/core.py", line 118, in <listcomp>
    args2 = [_execute_task(a, cache) for a in args]
  File "/home/nrsc/geo2grid_v_1_0_1/lib/python3.7/site-packages/dask/core.py", line 118, in _execute_task
    args2 = [_execute_task(a, cache) for a in args]
  File "/home/nrsc/geo2grid_v_1_0_1/lib/python3.7/site-packages/dask/core.py", line 118, in <listcomp>
    args2 = [_execute_task(a, cache) for a in args]
  File "/home/nrsc/geo2grid_v_1_0_1/lib/python3.7/site-packages/dask/core.py", line 115, in _execute_task
    return [_execute_task(a, cache) for a in arg]
  File "/home/nrsc/geo2grid_v_1_0_1/lib/python3.7/site-packages/dask/core.py", line 115, in <listcomp>
    return [_execute_task(a, cache) for a in arg]
  File "/home/nrsc/geo2grid_v_1_0_1/lib/python3.7/site-packages/dask/core.py", line 115, in _execute_task
    return [_execute_task(a, cache) for a in arg]
  File "/home/nrsc/geo2grid_v_1_0_1/lib/python3.7/site-packages/dask/core.py", line 115, in <listcomp>
    return [_execute_task(a, cache) for a in arg]
  File "/home/nrsc/geo2grid_v_1_0_1/lib/python3.7/site-packages/dask/core.py", line 120, in _execute_task
    elif not ishashable(arg):
  File "/home/nrsc/geo2grid_v_1_0_1/lib/python3.7/site-packages/dask/core.py", line 20, in ishashable
    hash(x)
  File "/home/nrsc/geo2grid_v_1_0_1/lib/python3.7/site-packages/pyproj/crs/crs.py", line 869, in __hash__
    return hash(self.to_wkt())
  File "pyproj/_crs.pyx", line 451, in pyproj._crs.Base.to_wkt
  File "pyproj/_crs.pyx", line 120, in pyproj._crs._to_wkt
  File "pyproj/_crs.pyx", line 24, in pyproj._crs.cstrdecode
  File "/home/nrsc/geo2grid_v_1_0_1/lib/python3.7/site-packages/pyproj/compat.py", line 21, in pystrdecode
    return cstr.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 2: invalid start byte
Could you please help solve this issue.

Regards
Rajashekar
davidh
Posts: 116
Joined: Tue Jun 04, 2013 11:19 am

Re: Issue in Generating True Color image

Post by davidh »

Hi,

This is a known bug and we plan on releasing a bug fix for it soon. You can get access to the beta release here:

ftp://ftp.ssec.wisc.edu/pub/davidh/geo2 ... 2b0.tar.gz

Let me know if this works better for you. Good luck.

Dave
rajashekar_g
Posts: 4
Joined: Thu Aug 13, 2020 1:27 am

Re: Issue in Generating True Color image

Post by rajashekar_g »

davidh wrote:Let me know if this works better for you. Good luck.
Dave
Thanks.

That solved the issue and I could generate a nice true color image.

Regards
Rajashekar
rajashekar_g
Posts: 4
Joined: Thu Aug 13, 2020 1:27 am

Re: Issue in Generating True Color image

Post by rajashekar_g »

Tried with imagery in the HSD format from the JAXA p-tree system (for 0900 UTC on the 17th Aug).

I had no issues with generating products over India in AEA (individual bands as well as the true color were successful)

I tried generating a true_color_night composite but the script failed. It would not generate the true color of the full disk either.

The script announced that geo2grid line 45 was killed while writing the final products. I have attached the log file (for the true color fill disk run) below. (the true_color_night terminated in a similar manner)

Code: Select all

[2020-08-19 01:08:07,316] : PID   4732 : DEBUG    : ahi_hsd_geotiff : main : Starting script with arguments: /home/nrsc/geo2grid-swbundle-1.0.2b0/lib/python3.7/site-packages/polar2grid/glue.py -r ahi_hsd -w geotiff -f /home/nrsc/him/0900FLDK --progress -v -p true_color -vv
[2020-08-19 01:08:07,323] : PID   4732 : INFO     : ahi_hsd_geotiff : main : Sorting and reading input files...
[2020-08-19 01:08:07,323] : PID   4732 : DEBUG    : satpy.scene : __init__ : Setting 'PPP_CONFIG_DIR' to '/home/nrsc/geo2grid-swbundle-1.0.2b0/etc/satpy'
[2020-08-19 01:08:07,342] : PID   4732 : DEBUG    : satpy.readers : read_reader_config : Reading ['/home/nrsc/geo2grid-swbundle-1.0.2b0/lib/python3.7/site-packages/satpy/etc/readers/ahi_hsd.yaml']
[2020-08-19 01:08:07,708] : PID   4732 : DEBUG    : satpy.readers.yaml_reader : create_filehandlers : Assigning to ahi_hsd: ['/home/nrsc/him/0900FLDK/HS_H08_20200817_0900_B01_FLDK_R10_S0810.DAT.bz2', '/home/nrsc/him/0900FLDK/HS_H08_20200817_0900_B01_FLDK_R10_S0610.DAT.bz2', 
<SNIP>
[2020-08-19 01:08:07,745] : PID   4732 : INFO     : satpy.readers.utils : unzip_file : Using temp file for BZ2 decompression: /tmp/tmpx4pp8gl8
<SNIP>
[2020-08-19 01:10:15,489] : PID   4732 : DEBUG    : satpy.composites : load_sensor_composites : Looking for composites config file ahi.yaml
[2020-08-19 01:10:15,612] : PID   4732 : DEBUG    : satpy.composites : load_sensor_composites : Looking for composites config file visir.yaml
[2020-08-19 01:10:15,687] : PID   4732 : DEBUG    : polar2grid.core.script_utils : rename_log_file : Log renamed from '/home/nrsc/him/ahi_hsd_geotiff_fail.log' to 'ahi_hsd_geotiff_20200817_090019.log'
[2020-08-19 01:10:15,687] : PID   4732 : INFO     : ahi_hsd_geotiff : main : Loading product metadata from files...
[2020-08-19 01:10:15,693] : PID   4732 : DEBUG    : satpy.node : _get_compositor_prereqs : Skipping optional solar_zenith_angle: Unknown dataset solar_zenith_angle
[2020-08-19 01:10:15,696] : PID   4732 : DEBUG    : satpy.node : _get_compositor_prereqs : Skipping optional satellite_azimuth_angle: Unknown dataset satellite_azimuth_angle
[2020-08-19 01:10:15,698] : PID   4732 : DEBUG    : satpy.node : _get_compositor_prereqs : Skipping optional satellite_zenith_angle: Unknown dataset satellite_zenith_angle
[2020-08-19 01:10:15,700] : PID   4732 : DEBUG    : satpy.node : _get_compositor_prereqs : Skipping optional solar_azimuth_angle: Unknown dataset solar_azimuth_angle
[2020-08-19 01:10:15,702] : PID   4732 : DEBUG    : satpy.node : _get_compositor_prereqs : Skipping optional solar_zenith_angle: Unknown dataset solar_zenith_angle
[2020-08-19 01:10:15,723] : PID   4732 : DEBUG    : satpy.node : _get_compositor_prereqs : Skipping optional solar_zenith_angle: Unknown dataset solar_zenith_angle
[2020-08-19 01:10:15,727] : PID   4732 : DEBUG    : satpy.node : _get_compositor_prereqs : Skipping optional satellite_azimuth_angle: Unknown dataset satellite_azimuth_angle
[2020-08-19 01:10:15,729] : PID   4732 : DEBUG    : satpy.node : _get_compositor_prereqs : Skipping optional satellite_zenith_angle: Unknown dataset satellite_zenith_angle
[2020-08-19 01:10:15,731] : PID   4732 : DEBUG    : satpy.node : _get_compositor_prereqs : Skipping optional solar_azimuth_angle: Unknown dataset solar_azimuth_angle
[2020-08-19 01:10:15,733] : PID   4732 : DEBUG    : satpy.node : _get_compositor_prereqs : Skipping optional solar_zenith_angle: Unknown dataset solar_zenith_angle
[2020-08-19 01:10:15,737] : PID   4732 : DEBUG    : satpy.node : _get_compositor_prereqs : Skipping optional solar_zenith_angle: Unknown dataset solar_zenith_angle
[2020-08-19 01:10:15,741] : PID   4732 : DEBUG    : satpy.node : _get_compositor_prereqs : Skipping optional solar_zenith_angle: Unknown dataset solar_zenith_angle
[2020-08-19 01:10:15,762] : PID   4732 : DEBUG    : satpy.node : _get_compositor_prereqs : Skipping optional satellite_azimuth_angle: Unknown dataset satellite_azimuth_angle
[2020-08-19 01:10:15,764] : PID   4732 : DEBUG    : satpy.node : _get_compositor_prereqs : Skipping optional satellite_zenith_angle: Unknown dataset satellite_zenith_angle
[2020-08-19 01:10:15,766] : PID   4732 : DEBUG    : satpy.node : _get_compositor_prereqs : Skipping optional solar_azimuth_angle: Unknown dataset solar_azimuth_angle
[2020-08-19 01:10:15,768] : PID   4732 : DEBUG    : satpy.node : _get_compositor_prereqs : Skipping optional solar_zenith_angle: Unknown dataset solar_zenith_angle
[2020-08-19 01:10:15,884] : PID   4732 : DEBUG    : ahi_hsd : _read_header : Band number = 4
[2020-08-19 01:10:15,884] : PID   4732 : DEBUG    : ahi_hsd : _read_header : Time_interval: 2020-08-17 09:00:20.204129 - 2020-08-17 09:00:53.561871
[2020-08-19 01:10:15,930] : PID   4732 : DEBUG    : ahi_hsd : read_band : Reading time 0:00:00.030667
[2020-08-19 01:10:15,957] : PID   4732 : DEBUG    : ahi_hsd : calibrate : Calibration time 0:00:00.027546
<SNIP>
[2020-08-19 01:10:16,803] : PID   4732 : DEBUG    : ahi_hsd : _read_header : Band number = 3
<SNIP>
[2020-08-19 01:10:17,512] : PID   4732 : DEBUG    : ahi_hsd : _read_header : Band number = 2
<SNIP>
[2020-08-19 01:10:18,232] : PID   4732 : DEBUG    : ahi_hsd : _read_header : Band number = 1
<SNIP>
<SNIP>
[2020-08-19 01:10:18,856] : PID   4732 : DEBUG    : ahi_hsd : calibrate : Calibration time 0:00:00.022993
[2020-08-19 01:10:18,893] : PID   4732 : DEBUG    : satpy.composites : __call__ : Applying sun zen correction
[2020-08-19 01:10:18,893] : PID   4732 : DEBUG    : satpy.composites : __call__ : Computing sun zenith angles.
[2020-08-19 01:10:18,918] : PID   4732 : DEBUG    : satpy.composites : _apply_correction : Apply the standard sun-zenith correction [1/cos(sunz)]
[2020-08-19 01:10:18,939] : PID   4732 : DEBUG    : satpy.composites : __call__ : Sun-zenith correction applied. Computation time:   0.0 (sec)
[2020-08-19 01:10:18,940] : PID   4732 : DEBUG    : satpy.composites : __call__ : Applying sun zen correction
[2020-08-19 01:10:18,940] : PID   4732 : DEBUG    : satpy.composites : __call__ : Computing sun zenith angles.
[2020-08-19 01:10:18,986] : PID   4732 : DEBUG    : satpy.composites : _apply_correction : Apply the standard sun-zenith correction [1/cos(sunz)]
[2020-08-19 01:10:19,027] : PID   4732 : DEBUG    : satpy.composites : __call__ : Sun-zenith correction applied. Computation time:   0.1 (sec)
[2020-08-19 01:10:19,029] : PID   4732 : DEBUG    : satpy.composites : __call__ : Applying sun zen correction
[2020-08-19 01:10:19,029] : PID   4732 : DEBUG    : satpy.composites : __call__ : Computing sun zenith angles.
[2020-08-19 01:10:19,074] : PID   4732 : DEBUG    : satpy.composites : _apply_correction : Apply the standard sun-zenith correction [1/cos(sunz)]
[2020-08-19 01:10:19,113] : PID   4732 : DEBUG    : satpy.composites : __call__ : Sun-zenith correction applied. Computation time:   0.1 (sec)
[2020-08-19 01:10:19,456] : PID   4732 : DEBUG    : satpy.scene : _generate_composite : Delaying generation of DatasetID(name='B02', wavelength=(0.49, 0.51, 0.53), resolution=1000, polarization=None, calibration='reflectance', level=None, modifiers=('sunz_corrected', 'rayleigh_corrected')) because of incompatible areas
[2020-08-19 01:10:19,458] : PID   4732 : DEBUG    : satpy.composites : __call__ : Applying sun zen correction
[2020-08-19 01:10:19,458] : PID   4732 : DEBUG    : satpy.composites : __call__ : Computing sun zenith angles.
[2020-08-19 01:10:19,483] : PID   4732 : DEBUG    : satpy.composites : _apply_correction : Apply the standard sun-zenith correction [1/cos(sunz)]
[2020-08-19 01:10:19,523] : PID   4732 : DEBUG    : satpy.composites : __call__ : Sun-zenith correction applied. Computation time:   0.1 (sec)
[2020-08-19 01:10:19,523] : PID   4732 : DEBUG    : satpy.scene : _get_prereq_datasets : Delaying generation of DatasetID(name='green', wavelength=None, resolution=None, polarization=None, calibration=None, level=None, modifiers=None) because of dependency's delayed generation: DatasetID(name='B04', wavelength=(0.83, 0.85, 0.87), resolution=1000, polarization=None, calibration='reflectance', level=None, modifiers=('sunz_corrected',))
[2020-08-19 01:10:19,523] : PID   4732 : DEBUG    : satpy.scene : _get_prereq_datasets : Delayed prerequisite for 'DatasetID(name='green', wavelength=None, resolution=None, polarization=None, calibration=None, level=None, modifiers=None)': 'DatasetID(name='B04', wavelength=(0.83, 0.85, 0.87), resolution=1000, polarization=None, calibration='reflectance', level=None, modifiers=('sunz_corrected',))'
[2020-08-19 01:10:19,525] : PID   4732 : DEBUG    : satpy.scene : _generate_composite : Delaying generation of DatasetID(name='B01', wavelength=(0.45, 0.47, 0.49), resolution=1000, polarization=None, calibration='reflectance', level=None, modifiers=('sunz_corrected', 'rayleigh_corrected')) because of incompatible areas
[2020-08-19 01:10:19,526] : PID   4732 : DEBUG    : satpy.scene : _generate_composite : Delaying generation of DatasetID(name='B02', wavelength=(0.49, 0.51, 0.53), resolution=1000, polarization=None, calibration='reflectance', level=None, modifiers=('sunz_corrected', 'rayleigh_corrected')) because of incompatible areas
[2020-08-19 01:10:19,666] : PID   4732 : INFO     : satpy.composites : __call__ : Removing Rayleigh scattering with atmosphere 'us-standard' and aerosol type 'marine_clean_aerosol' for 'B03'
[2020-08-19 01:10:19,686] : PID   4732 : INFO     : pyspectral.rayleigh : __init__ : Atmosphere chosen: us-standard
[2020-08-19 01:10:19,709] : PID   4732 : DEBUG    : pyspectral.rayleigh : __init__ : LUT filename: /home/nrsc/geo2grid-swbundle-1.0.2b0/pyspectral_data/marine_clean_aerosol/rayleigh_lut_us-standard.h5
[2020-08-19 01:10:19,722] : PID   4732 : DEBUG    : pyspectral.rsr_reader : _get_filename : Filename: /home/nrsc/geo2grid-swbundle-1.0.2b0/pyspectral_data/rsr_ahi_Himawari-8.h5
[2020-08-19 01:10:19,722] : PID   4732 : DEBUG    : pyspectral.rsr_reader : __init__ : Filename: /home/nrsc/geo2grid-swbundle-1.0.2b0/pyspectral_data/rsr_ahi_Himawari-8.h5
[2020-08-19 01:10:19,755] : PID   4732 : DEBUG    : pyspectral.rsr_reader : load : No detectors found - assume only one...
[2020-08-19 01:10:19,804] : PID   4732 : DEBUG    : pyspectral.rayleigh : get_effective_wavelength : Band name: ch3  Effective wavelength: 0.635378
[2020-08-19 01:10:19,928] : PID   4732 : DEBUG    : pyspectral.rayleigh : get_reflectance : Time - Interpolation: 0.069165
[2020-08-19 01:10:19,939] : PID   4732 : DEBUG    : satpy.scene : _get_prereq_datasets : Delaying generation of DatasetID(name='true_color', wavelength=None, resolution=None, polarization=None, calibration=None, level=None, modifiers=None) because of dependency's delayed generation: DatasetID(name='B01', wavelength=(0.45, 0.47, 0.49), resolution=1000, polarization=None, calibration='reflectance', level=None, modifiers=('sunz_corrected', 'rayleigh_corrected'))
[2020-08-19 01:10:19,939] : PID   4732 : DEBUG    : satpy.scene : _get_prereq_datasets : Delayed prerequisite for 'DatasetID(name='true_color', wavelength=None, resolution=None, polarization=None, calibration=None, level=None, modifiers=None)': 'DatasetID(name='B01', wavelength=(0.45, 0.47, 0.49), resolution=1000, polarization=None, calibration='reflectance', level=None, modifiers=('sunz_corrected', 'rayleigh_corrected'))'
[2020-08-19 01:10:19,940] : PID   4732 : WARNING  : satpy.scene : load : The following datasets were not created and may require resampling to be generated: DatasetID(name='true_color', wavelength=None, resolution=None, polarization=None, calibration=None, level=None, modifiers=None)
[2020-08-19 01:10:19,940] : PID   4732 : DEBUG    : satpy.scene : unload : Unloading dataset: DatasetID(name='B04', wavelength=(0.83, 0.85, 0.87), resolution=1000, polarization=None, calibration='reflectance', level=None, modifiers=())
[2020-08-19 01:10:19,941] : PID   4732 : DEBUG    : satpy.scene : unload : Unloading dataset: DatasetID(name='B03', wavelength=(0.62, 0.64, 0.66), resolution=500, polarization=None, calibration='reflectance', level=None, modifiers=())
[2020-08-19 01:10:19,941] : PID   4732 : DEBUG    : satpy.scene : unload : Unloading dataset: DatasetID(name='B02', wavelength=(0.49, 0.51, 0.53), resolution=1000, polarization=None, calibration='reflectance', level=None, modifiers=())
[2020-08-19 01:10:19,941] : PID   4732 : DEBUG    : satpy.scene : unload : Unloading dataset: DatasetID(name='B01', wavelength=(0.45, 0.47, 0.49), resolution=1000, polarization=None, calibration='reflectance', level=None, modifiers=())
[2020-08-19 01:10:19,945] : PID   4732 : DEBUG    : polar2grid.grids.manager : __init__ : Using default grid configuration: 'grids.conf' 
[2020-08-19 01:10:19,945] : PID   4732 : DEBUG    : polar2grid.grids.manager : __init__ : Loading grid configuration 'grids.conf'
[2020-08-19 01:10:20,046] : PID   4732 : DEBUG    : polar2grid.grids.manager : parse_proj4_config_line : Converted grid 'dwd_germany' origin from (lon: -2.000000, lat: 56.000000) to (x: -755570.370463, y: -3554679.114710)
[2020-08-19 01:10:20,048] : PID   4732 : DEBUG    : polar2grid.grids.manager : parse_proj4_config_line : Converted grid 'australia' origin from (lon: 105.000000, lat: 5.000000) to (x: -3896182.177765, y: 557305.257275)
[2020-08-19 01:10:20,050] : PID   4732 : DEBUG    : polar2grid.grids.manager : parse_proj4_config_line : Converted grid 'australia2' origin from (lon: 105.000000, lat: 5.000000) to (x: -3896182.177765, y: 557305.257275)
[2020-08-19 01:10:20,052] : PID   4732 : DEBUG    : polar2grid.grids.manager : parse_proj4_config_line : Converted grid '211e' origin from (lon: -123.044000, lat: 59.844000) to (x: -1950510.636800, y: 4368587.226913)
[2020-08-19 01:10:20,054] : PID   4732 : DEBUG    : polar2grid.grids.manager : parse_proj4_config_line : Converted grid '211e_10km' origin from (lon: -123.044000, lat: 59.844000) to (x: -1950510.636800, y: 4368587.226913)
[2020-08-19 01:10:20,056] : PID   4732 : DEBUG    : polar2grid.grids.manager : parse_proj4_config_line : Converted grid '211e_hi' origin from (lon: -123.044000, lat: 59.844000) to (x: -1950510.636800, y: 4368587.226913)
[2020-08-19 01:10:20,058] : PID   4732 : DEBUG    : polar2grid.grids.manager : parse_proj4_config_line : Converted grid '211w' origin from (lon: -152.855000, lat: 54.536000) to (x: -4226045.730436, y: 4368634.765713)
[2020-08-19 01:10:20,059] : PID   4732 : DEBUG    : polar2grid.grids.manager : parse_proj4_config_line : Converted grid '211w_10km' origin from (lon: -152.855000, lat: 54.536000) to (x: -4226045.730436, y: 4368634.765713)
[2020-08-19 01:10:20,069] : PID   4732 : DEBUG    : polar2grid.grids.manager : parse_proj4_config_line : Converted grid '203' origin from (lon: 115.601000, lat: 44.646000) to (x: -4952960.856044, y: 381022.610378)
[2020-08-19 01:10:20,072] : PID   4732 : DEBUG    : polar2grid.grids.manager : parse_proj4_config_line : Converted grid '203_10km' origin from (lon: 115.601000, lat: 44.646000) to (x: -4952960.856044, y: 381022.610378)
[2020-08-19 01:10:20,074] : PID   4732 : DEBUG    : polar2grid.grids.manager : parse_proj4_config_line : Converted grid '204' origin from (lon: 110.000000, lat: 60.644000) to (x: -7314463.252793, y: 8020503.279488)
[2020-08-19 01:10:20,076] : PID   4732 : DEBUG    : polar2grid.grids.manager : parse_proj4_config_line : Converted grid '204_10km' origin from (lon: 110.000000, lat: 60.644000) to (x: -7314463.252793, y: 8020503.279488)
[2020-08-19 01:10:20,078] : PID   4732 : DEBUG    : polar2grid.grids.manager : parse_proj4_config_line : Converted grid '205' origin from (lon: -115.304000, lat: 36.257000) to (x: -4952924.911284, y: -3429056.260492)
[2020-08-19 01:10:20,098] : PID   4732 : DEBUG    : polar2grid.grids.manager : parse_proj4_config_line : Converted grid '210' origin from (lon: -77.000000, lat: 26.422000) to (x: 0.000000, y: 2864298.891608)
[2020-08-19 01:10:20,098] : PID   4732 : DEBUG    : ahi_hsd_geotiff : main : Products to preserve resolution for: set()
[2020-08-19 01:10:20,098] : PID   4732 : DEBUG    : ahi_hsd_geotiff : main : Products to use new resolution for: {DatasetID(name='true_color', wavelength=None, resolution=None, polarization=None, calibration=None, level=None, modifiers=None)}
[2020-08-19 01:10:20,098] : PID   4732 : DEBUG    : ahi_hsd_geotiff : main : Setting default resampling to 'native' for grid 'MAX'
[2020-08-19 01:10:20,098] : PID   4732 : INFO     : ahi_hsd_geotiff : main : Resampling data to 'MAX'
[2020-08-19 01:10:20,099] : PID   4732 : DEBUG    : satpy.scene : __init__ : Setting 'PPP_CONFIG_DIR' to '/home/nrsc/geo2grid-swbundle-1.0.2b0/etc/satpy'
[2020-08-19 01:10:20,099] : PID   4732 : DEBUG    : satpy.scene : _resampled_scene : Resampling DatasetID(name='B01', wavelength=(0.45, 0.47, 0.49), resolution=1000, polarization=None, calibration='reflectance', level=None, modifiers=('sunz_corrected',))
[2020-08-19 01:10:20,099] : PID   4732 : DEBUG    : pyresample.geometry : get_area_slices : Projections for data and slice areas are identical: +proj=geos +lon_0=140.7 +h=35785863 +x_0=0 +y_0=0 +a=6378137 +b=6356752.3 +units=m +no_defs +type=crs
[2020-08-19 01:10:20,189] : PID   4732 : DEBUG    : satpy.scene : _resampled_scene : Resampling DatasetID(name='B03', wavelength=(0.62, 0.64, 0.66), resolution=500, polarization=None, calibration='reflectance', level=None, modifiers=('sunz_corrected',))
[2020-08-19 01:10:20,189] : PID   4732 : DEBUG    : pyresample.geometry : get_area_slices : Projections for data and slice areas are identical: +proj=geos +lon_0=140.7 +h=35785863 +x_0=0 +y_0=0 +a=6378137 +b=6356752.3 +units=m +no_defs +type=crs
[2020-08-19 01:10:20,195] : PID   4732 : DEBUG    : satpy.scene : _resampled_scene : Resampling DatasetID(name='B02', wavelength=(0.49, 0.51, 0.53), resolution=1000, polarization=None, calibration='reflectance', level=None, modifiers=('sunz_corrected',))
[2020-08-19 01:10:20,345] : PID   4732 : DEBUG    : satpy.scene : _resampled_scene : Resampling DatasetID(name='B04', wavelength=(0.83, 0.85, 0.87), resolution=1000, polarization=None, calibration='reflectance', level=None, modifiers=('sunz_corrected',))
[2020-08-19 01:10:20,445] : PID   4732 : DEBUG    : satpy.scene : _resampled_scene : Resampling DatasetID(name='B03', wavelength=(0.62, 0.64, 0.66), resolution=500, polarization=None, calibration='reflectance', level=None, modifiers=('sunz_corrected', 'rayleigh_corrected'))
[2020-08-19 01:10:20,593] : PID   4732 : INFO     : satpy.composites : __call__ : Removing Rayleigh scattering with atmosphere 'us-standard' and aerosol type 'marine_clean_aerosol' for 'B02'
[2020-08-19 01:10:20,594] : PID   4732 : INFO     : pyspectral.rayleigh : __init__ : Atmosphere chosen: us-standard
[2020-08-19 01:10:20,595] : PID   4732 : DEBUG    : pyspectral.rayleigh : __init__ : LUT filename: /home/nrsc/geo2grid-swbundle-1.0.2b0/pyspectral_data/marine_clean_aerosol/rayleigh_lut_us-standard.h5
[2020-08-19 01:10:20,596] : PID   4732 : DEBUG    : pyspectral.rsr_reader : _get_filename : Filename: /home/nrsc/geo2grid-swbundle-1.0.2b0/pyspectral_data/rsr_ahi_Himawari-8.h5
[2020-08-19 01:10:20,596] : PID   4732 : DEBUG    : pyspectral.rsr_reader : __init__ : Filename: /home/nrsc/geo2grid-swbundle-1.0.2b0/pyspectral_data/rsr_ahi_Himawari-8.h5
[2020-08-19 01:10:20,597] : PID   4732 : DEBUG    : pyspectral.rsr_reader : load : No detectors found - assume only one...
[2020-08-19 01:10:20,653] : PID   4732 : DEBUG    : pyspectral.rayleigh : get_effective_wavelength : Band name: ch2  Effective wavelength: 0.509195
[2020-08-19 01:10:20,745] : PID   4732 : DEBUG    : pyspectral.rayleigh : get_reflectance : Time - Interpolation: 0.072506
[2020-08-19 01:10:20,755] : PID   4732 : INFO     : satpy.composites.ahi : __call__ : Boosting vegetation on green band
[2020-08-19 01:10:20,903] : PID   4732 : INFO     : satpy.composites : __call__ : Removing Rayleigh scattering with atmosphere 'us-standard' and aerosol type 'marine_clean_aerosol' for 'B01'
[2020-08-19 01:10:20,904] : PID   4732 : INFO     : pyspectral.rayleigh : __init__ : Atmosphere chosen: us-standard
[2020-08-19 01:10:20,905] : PID   4732 : DEBUG    : pyspectral.rayleigh : __init__ : LUT filename: /home/nrsc/geo2grid-swbundle-1.0.2b0/pyspectral_data/marine_clean_aerosol/rayleigh_lut_us-standard.h5
[2020-08-19 01:10:20,906] : PID   4732 : DEBUG    : pyspectral.rsr_reader : _get_filename : Filename: /home/nrsc/geo2grid-swbundle-1.0.2b0/pyspectral_data/rsr_ahi_Himawari-8.h5
[2020-08-19 01:10:20,906] : PID   4732 : DEBUG    : pyspectral.rsr_reader : __init__ : Filename: /home/nrsc/geo2grid-swbundle-1.0.2b0/pyspectral_data/rsr_ahi_Himawari-8.h5
[2020-08-19 01:10:20,907] : PID   4732 : DEBUG    : pyspectral.rsr_reader : load : No detectors found - assume only one...
[2020-08-19 01:10:20,946] : PID   4732 : DEBUG    : pyspectral.rayleigh : get_effective_wavelength : Band name: ch1  Effective wavelength: 0.469333
[2020-08-19 01:10:21,062] : PID   4732 : DEBUG    : pyspectral.rayleigh : get_reflectance : Time - Interpolation: 0.076445
[2020-08-19 01:10:21,120] : PID   4732 : DEBUG    : satpy.composites : __call__ : Sharpening image with high resolution red band
[2020-08-19 01:10:21,369] : PID   4732 : DEBUG    : satpy.scene : unload : Unloading dataset: DatasetID(name='B01', wavelength=(0.45, 0.47, 0.49), resolution=1000, polarization=None, calibration='reflectance', level=None, modifiers=('sunz_corrected',))
[2020-08-19 01:10:21,369] : PID   4732 : DEBUG    : satpy.scene : unload : Unloading dataset: DatasetID(name='B03', wavelength=(0.62, 0.64, 0.66), resolution=500, polarization=None, calibration='reflectance', level=None, modifiers=('sunz_corrected',))
[2020-08-19 01:10:21,369] : PID   4732 : DEBUG    : satpy.scene : unload : Unloading dataset: DatasetID(name='B02', wavelength=(0.49, 0.51, 0.53), resolution=1000, polarization=None, calibration='reflectance', level=None, modifiers=('sunz_corrected',))
[2020-08-19 01:10:21,369] : PID   4732 : DEBUG    : satpy.scene : unload : Unloading dataset: DatasetID(name='B04', wavelength=(0.83, 0.85, 0.87), resolution=1000, polarization=None, calibration='reflectance', level=None, modifiers=('sunz_corrected',))
[2020-08-19 01:10:21,370] : PID   4732 : DEBUG    : satpy.scene : unload : Unloading dataset: DatasetID(name='B03', wavelength=(0.62, 0.64, 0.66), resolution=500, polarization=None, calibration='reflectance', level=None, modifiers=('sunz_corrected', 'rayleigh_corrected'))
[2020-08-19 01:10:21,370] : PID   4732 : DEBUG    : satpy.scene : unload : Unloading dataset: DatasetID(name='B02', wavelength=(0.49, 0.51, 0.53), resolution=1000, polarization=None, calibration='reflectance', level=None, modifiers=('sunz_corrected', 'rayleigh_corrected'))
[2020-08-19 01:10:21,370] : PID   4732 : DEBUG    : satpy.scene : unload : Unloading dataset: DatasetID(name='green', wavelength=None, resolution=1000, polarization=None, calibration=None, level=None, modifiers=None)
[2020-08-19 01:10:21,370] : PID   4732 : DEBUG    : satpy.scene : unload : Unloading dataset: DatasetID(name='B01', wavelength=(0.45, 0.47, 0.49), resolution=1000, polarization=None, calibration='reflectance', level=None, modifiers=('sunz_corrected', 'rayleigh_corrected'))
[2020-08-19 01:10:21,371] : PID   4732 : DEBUG    : satpy.writers : read_writer_config : Reading ['/home/nrsc/geo2grid-swbundle-1.0.2b0/lib/python3.7/site-packages/satpy/etc/writers/geotiff.yaml']
[2020-08-19 01:10:21,646] : PID   4732 : DEBUG    : satpy.writers : apply : Enhancement configuration options: [{'name': 'crefl_scaling', 'method': <function crefl_scaling at 0x7fe95c200710>, 'kwargs': {'idx': [0.0, 25.0, 55.0, 100.0, 255.0], 'sc': [0.0, 90.0, 140.0, 175.0, 255.0]}}]
[2020-08-19 01:10:21,646] : PID   4732 : DEBUG    : satpy.enhancements : crefl_scaling : Applying the crefl_scaling
[2020-08-19 01:10:21,964] : PID   4732 : INFO     : ahi_hsd_geotiff : main : Computing products and saving data to writers...
davidh
Posts: 116
Joined: Tue Jun 04, 2013 11:19 am

Re: Issue in Generating True Color image

Post by davidh »

Hi,

Could you provide more information on the system you are running this on? How much memory? How much disk space in your current working directory? My best guess is you ran out of memory or disk space. One thing to try would be to pass "--num-workers 1 --tiled" which may improve memory usage by:

1. Only using a single worker at a time instead of the default of 4.
2. Saving the geotiff in "tiled" format which means each chunk of data can be saved independently; reducing the amount of data held in memory at any one time.

Side note: I noticed you are passing .bz2 files to the reader. These were being uncompressed to "/tmp/tmpx4pp8gl8". You may want to make sure this directory was automatically deleted (since the program crashed) so you don't fill up /tmp.

Dave
rajashekar_g
Posts: 4
Joined: Thu Aug 13, 2020 1:27 am

Re: Issue in Generating True Color image

Post by rajashekar_g »

davidh wrote:Hi,

Could you provide more information on the system you are running this on? How much memory? How much disk space in your current working directory? My best guess is you ran out of memory or disk space. One thing to try would be to pass "--num-workers 1 --tiled" which may improve memory usage by:

1. Only using a single worker at a time instead of the default of 4.
2. Saving the geotiff in "tiled" format which means each chunk of data can be saved independently; reducing the amount of data held in memory at any one time.

Side note: I noticed you are passing .bz2 files to the reader. These were being uncompressed to "/tmp/tmpx4pp8gl8". You may want to make sure this directory was automatically deleted (since the program crashed) so you don't fill up /tmp.

Dave
I have been running it in a VM Ware virtual machine on a Xeon workstation, the goal for now being to evaluate the data/ possibilities.

I did suspect memory issues because the VM running Centos has only 3 GB RAM (against the 16 GB RAM (minimum) recommendation), I could produce the country scale products and the log file truncated abruptly. I had not figured out how to handle it.

I will launch it again with the --num-workers --tiled Create tiled geotiffs options set with the official release just announced.

Thanks
Rajashekar
Post Reply