No filenames found for reader ahi_hsd

[torresrodyvito]

So I'm trying to use bash in order to convert HSD to GeoTiff. However we always get an error for this line:

geo2grid.sh -r ahi_hsd -w geotiff --list-products -f /ahi8/data/

we replaced "/ahi8/data" to our directory which is <home>/geo2grid_v_1_0_0/bin/<data>

We always get an error telling that it cannot open the file specified.

Thank you for your help.

[davidh]

What files are in `/ahi/data`? Can you show me the contents? Could you paste the contents of the log file that is generated here too?

The `-f` option does not recursively search for files in a directory, you must have the HSD files in `/ahi/data` and not in a subdirectory like `/ahi/data/20190404/`.

[torresrodyvito]

Here is what we did in bash:

export GEO2GRID_HOME=/torres/geo2grid_v_1_0_0/
export PATH=$PATH:$POLAR2GRID_HOME/bin

geo2grid.sh -r ahi_hsd -w geotiff --list-products -f /torres/geo2dgrid_v_1_0_0/bin/HSD/HS_H08_20190403_0200_B04_FLDK_R10_S0110.DAT/ -vvv

INFO : Sorting and reading input files...
WARNING : No filenames found for reader: ahi_hsd
WARNING : Don't know how to open the following files: {'/geo2grid_v_1_0_0/bin/HSD'}
ERROR : No supported files found | Enable debug message (-vvv) or see log file for details.
DEBUG : Further error information:
Traceback (most recent call last):
File "/home/torres/geo2grid_v_1_0_0/lib/python3.6/site-packages/polar2grid/glue.py", line 253, in main
scn = Scene(**scene_args)
File "/home/torres/geo2grid_v_1_0_0/lib/python3.6/site-packages/satpy/scene.py", line 156, in _init_
reader_kwargs=reader_kwargs)
File "/home/torres/geo2grid_v_1_0_0/lib/python3.6/site-packages/satpy/scene.py", line 201, in create_reader_instances
ppp_config_dir=self.ppp_config_dir)
File "/home/torres/geo2grid_v_1_0_0/lib/python3.6/site-packages/satpy/readers/__init__.py", line 761, in load_readers
raise ValueError("No supported files found")
ValueError: No supported files found
--------------------------------------------------------------------------------------------------------------------------------------------------------------------

The /torres/geo2dgrid_v_1_0_0/bin/HSD/ only contains HSD file. Should we have a dedicated folder exactly named "/ahi/data/" to contain the HSD file. We are not experts of bash and geo2grid so your help is really appreciated. Thank you.

[davidh]

Are you sure this is the output for that exact command? Your command says the filename(s) is:

/torres/geo2dgrid_v_1_0_0/bin/HSD/HS_H08_20190403_0200_B04_FLDK_R10_S0110.DAT/

There are a couple things that are odd about this. First, you end it with a "/" character which is usually meant for directories, is this a directory or a file? Second, is there a reason you are putting data in the `bin` directory of the geo2grid distribution? I would recommend not doing this to make future geo2grid updates easier and lessen the chance of it causing issues with the rest of geo2grid.

Next, the log shows this message:

WARNING : Don't know how to open the following files: {'/geo2grid_v_1_0_0/bin/HSD'}

The only way that I can think of for why geo2grid would list that as the filename is if:

1. The directory you showed in your command is not what was run to produce that output.
2. That directory/file has a space in it which is treated by geo2grid as two (or more) separate arguments.
3. "/torres" has a non-ascii (unicode) character that was parsed incorrectly by geo2grid
4. Some unknown issue.

Can you shed some light on some of these questions? Thanks.

Dave

[torresrodyvito]

For some reason we we're able to continue with the process probably correcting the pathing/directory using:

$ sudo $G/geo2grid.sh -r ahi_hsd -w geotiff -f $G/HSD/

However is it normal to get the following warnings even it said SUCCESS afterwards?

WARNING : Required file type 'hsd_b14' not found or loaded for 'B14'
WARNING : Required file type 'hsd_b16' not found or loaded for 'B16'
WARNING : Required file type 'hsd_b12' not found or loaded for 'B12'
WARNING : Required file type 'hsd_b10' not found or loaded for 'B10'
WARNING : Required file type 'hsd_b06' not found or loaded for 'B06'
WARNING : Required file type 'hsd_b13' not found or loaded for 'B13'
WARNING : Required file type 'hsd_b11' not found or loaded for 'B11'
WARNING : Required file type 'hsd_b08' not found or loaded for 'B08'
WARNING : Required file type 'hsd_b07' not found or loaded for 'B07'
WARNING : Required file type 'hsd_b15' not found or loaded for 'B15'
WARNING : Required file type 'hsd_b09' not found or loaded for 'B09'

The said HSD files for all the bands were in the folder HSD.
Thank you for your recommendation about the directory and will follow it in the future.

[davidh]

However is it normal to get the following warnings even it said SUCCESS afterwards?

This is only normal if you *know* you didn't give it some of the bands' files. If you do "ls $G/HSD` what do you get?

[torresrodyvito]

When using the command list products only these things show:

B01
B02
B03
B04
B05
cloud_phase_distinction
convection
day_microphysics_ahi
day_microphysics_emu
green
ir_cloud_day
mid_vapor
natural_color
natural_color_sun
overview
true_color
true_color_night
true_color_raw
water_vapors1
water_vapors2

------------------------

No other bands were listed as products even though HSD files for all bands were in the HSD folder.

Also to confirm, does the converted geotiff product contains surface reflectance for each band?

[davidh]

Please provide me the list of filenames you are providing to Geo2Grid. If you could also provide me the log file generated by geo2grid, that would also help.

For visible bands, reflectance is saved. For IR bands it is a brightness temperature.

[torresrodyvito]

Here is the contents of the folder HSD:

05-Apr-19 10:46 AM 24,201,523 HS_H08_20170623_0250_B01_FLDK_R10_S0110.DAT
05-Apr-19 10:46 AM 24,201,523 HS_H08_20170623_0250_B01_FLDK_R10_S0210.DAT
05-Apr-19 10:47 AM 24,201,523 HS_H08_20170623_0250_B01_FLDK_R10_S0310.DAT
05-Apr-19 10:47 AM 24,201,533 HS_H08_20170623_0250_B01_FLDK_R10_S0410.DAT
05-Apr-19 10:47 AM 24,201,523 HS_H08_20170623_0250_B01_FLDK_R10_S0510.DAT
05-Apr-19 10:47 AM 24,201,523 HS_H08_20170623_0250_B01_FLDK_R10_S0610.DAT
05-Apr-19 10:47 AM 24,201,533 HS_H08_20170623_0250_B01_FLDK_R10_S0710.DAT
05-Apr-19 10:47 AM 24,201,523 HS_H08_20170623_0250_B01_FLDK_R10_S0810.DAT
05-Apr-19 10:47 AM 24,201,523 HS_H08_20170623_0250_B01_FLDK_R10_S0910.DAT
05-Apr-19 10:47 AM 24,201,523 HS_H08_20170623_0250_B01_FLDK_R10_S1010.DAT
05-Apr-19 10:47 AM 24,201,523 HS_H08_20170623_0250_B02_FLDK_R10_S0110.DAT
05-Apr-19 10:47 AM 24,201,523 HS_H08_20170623_0250_B02_FLDK_R10_S0210.DAT
05-Apr-19 10:47 AM 24,201,523 HS_H08_20170623_0250_B02_FLDK_R10_S0310.DAT
05-Apr-19 10:47 AM 24,201,533 HS_H08_20170623_0250_B02_FLDK_R10_S0410.DAT
05-Apr-19 10:47 AM 24,201,523 HS_H08_20170623_0250_B02_FLDK_R10_S0510.DAT
05-Apr-19 10:47 AM 24,201,523 HS_H08_20170623_0250_B02_FLDK_R10_S0610.DAT
05-Apr-19 10:43 AM 24,201,533 HS_H08_20170623_0250_B02_FLDK_R10_S0710.DAT
05-Apr-19 10:43 AM 24,201,523 HS_H08_20170623_0250_B02_FLDK_R10_S0810.DAT
05-Apr-19 10:43 AM 24,201,523 HS_H08_20170623_0250_B02_FLDK_R10_S0910.DAT
05-Apr-19 10:43 AM 24,201,523 HS_H08_20170623_0250_B02_FLDK_R10_S1010.DAT
05-Apr-19 10:44 AM 96,801,523 HS_H08_20170623_0250_B03_FLDK_R05_S0110.DAT
05-Apr-19 10:44 AM 96,801,523 HS_H08_20170623_0250_B03_FLDK_R05_S0210.DAT
05-Apr-19 10:44 AM 96,801,523 HS_H08_20170623_0250_B03_FLDK_R05_S0310.DAT
05-Apr-19 10:44 AM 96,801,533 HS_H08_20170623_0250_B03_FLDK_R05_S0410.DAT
05-Apr-19 10:44 AM 96,801,523 HS_H08_20170623_0250_B03_FLDK_R05_S0510.DAT
05-Apr-19 10:45 AM 96,801,523 HS_H08_20170623_0250_B03_FLDK_R05_S0610.DAT
05-Apr-19 10:45 AM 96,801,533 HS_H08_20170623_0250_B03_FLDK_R05_S0710.DAT
05-Apr-19 10:45 AM 96,801,523 HS_H08_20170623_0250_B03_FLDK_R05_S0810.DAT
05-Apr-19 10:45 AM 96,801,523 HS_H08_20170623_0250_B03_FLDK_R05_S0910.DAT
05-Apr-19 10:46 AM 96,801,523 HS_H08_20170623_0250_B03_FLDK_R05_S1010.DAT
05-Apr-19 10:46 AM 24,201,523 HS_H08_20170623_0250_B04_FLDK_R10_S0110.DAT
05-Apr-19 10:46 AM 24,201,523 HS_H08_20170623_0250_B04_FLDK_R10_S0210.DAT
05-Apr-19 10:46 AM 24,201,523 HS_H08_20170623_0250_B04_FLDK_R10_S0310.DAT
05-Apr-19 10:46 AM 24,201,533 HS_H08_20170623_0250_B04_FLDK_R10_S0410.DAT
05-Apr-19 10:46 AM 24,201,523 HS_H08_20170623_0250_B04_FLDK_R10_S0510.DAT
05-Apr-19 10:46 AM 24,201,523 HS_H08_20170623_0250_B04_FLDK_R10_S0610.DAT
05-Apr-19 10:46 AM 24,201,533 HS_H08_20170623_0250_B04_FLDK_R10_S0710.DAT
05-Apr-19 10:46 AM 24,201,523 HS_H08_20170623_0250_B04_FLDK_R10_S0810.DAT
05-Apr-19 10:46 AM 24,201,523 HS_H08_20170623_0250_B04_FLDK_R10_S0910.DAT
05-Apr-19 10:46 AM 24,201,523 HS_H08_20170623_0250_B04_FLDK_R10_S1010.DAT
05-Apr-19 10:46 AM 6,051,523 HS_H08_20170623_0250_B05_FLDK_R20_S0110.DAT
05-Apr-19 10:46 AM 6,051,523 HS_H08_20170623_0250_B05_FLDK_R20_S0210.DAT
05-Apr-19 10:46 AM 6,051,523 HS_H08_20170623_0250_B05_FLDK_R20_S0310.DAT
05-Apr-19 10:46 AM 6,051,533 HS_H08_20170623_0250_B05_FLDK_R20_S0410.DAT
05-Apr-19 10:46 AM 6,051,523 HS_H08_20170623_0250_B05_FLDK_R20_S0510.DAT
05-Apr-19 10:46 AM 6,051,523 HS_H08_20170623_0250_B05_FLDK_R20_S0610.DAT
05-Apr-19 10:46 AM 6,051,533 HS_H08_20170623_0250_B05_FLDK_R20_S0710.DAT
05-Apr-19 10:46 AM 6,051,523 HS_H08_20170623_0250_B05_FLDK_R20_S0810.DAT
05-Apr-19 10:46 AM 6,051,523 HS_H08_20170623_0250_B05_FLDK_R20_S0910.DAT
05-Apr-19 10:46 AM 6,051,523 HS_H08_20170623_0250_B05_FLDK_R20_S1010.DAT

--------------------------------------------------------------------------------------------------------------

Also, to clarify, does the geotiff contain "surface reflectance" or "top of the atmosphere" reflectance?

[torresrodyvito]

Moreover, how can we access/extract the surface reflectance values in the converted geotiff?

We tried opening it in ENVI and ArcGIS, however there are only 2 accessible fields: (1) values (which ranges from 0 to 255) , and (2) count.
Is the "values" the reflectance or we need another script/software to access the reflectance?

Thank you.