Some questions about Polar2Grid

Dmitry
Posts: 34
Joined: Mon Feb 29, 2016 3:56 am

Some questions about Polar2Grid

Post by Dmitry »

Dear users and developers!

1.Tell me how you can get a composite RGB image (for example, in combination 2-2-1) for the AVHRR data.
2.Can I get a true color rgb kmz file directly (without forming geotiff) from the source file VIIRS SDR or MODIS.

Regards,
Dmitry
davidh
Posts: 116
Joined: Tue Jun 04, 2013 11:19 am

Re: Some questions about Polar2Grid

Post by davidh »

Hi Dmitry,

1. There are instructions on how to create your own RGB images in the installation guide on pages 31 and 32 I believe. The example is for MODIS, but it should be easy enough to convert things for AVHRR.

2. Currently there is no way to create a KMZ directly from the source SDRs. The "gtiff2kmz.sh" script in Polar2Grid uses the tools provided by GDAL to convert geotiff images to KMZ files and there is no easy way around that. A KMZ file is a zip file of jpeg images created from the geotiff images. If this is something you do often it should be possible to place the commands involved in a simple script.

Hope that helps,
Dave
Dmitry
Posts: 34
Joined: Mon Feb 29, 2016 3:56 am

Re: Some questions about Polar2Grid

Post by Dmitry »

Dear Dave,

Thanks, OK.

Can I get advice the following question.
When you try to process the VIIRS data using crefl2gtiff I get the following log files and error (data description http://navigator.eumetsat.int/discovery ... 8CDAC2E1BB)

IPOPP@DGORSKI:~/Downloads/polar2grid_v_2_0/bin$ ./crefl2gtiff.sh -v -g wgs84_fit_1km --method nearest --no-tc -f /home/IPOPP/AVHRR_raw/
DEBUG : Starting script with arguments: /home/IPOPP/Downloads/polar2grid_v_2_0/ShellB3/lib/python2.7/site-packages/polar2grid-2.0.0-py2.7-linux-x86_64.egg/polar2grid/glue.py crefl gtiff true_color -v -g wgs84_fit_1km --method nearest --no-tc -f /home/IPOPP/AVHRR_raw/ -vv
INFO : Initializing swath extractor...
DEBUG : Searching '/home/IPOPP/AVHRR_raw/' for useful files
DEBUG : Searching '/home/IPOPP/AVHRR_raw/' for useful files
DEBUG : Analyzing /home/IPOPP/AVHRR_raw/SVM12_npp_d20160315_t1038271_e1039513_b22701_c20160315104450530043_cspp_dev.h5
DEBUG : Unnecessary hdf5 file: /home/IPOPP/AVHRR_raw/SVM12_npp_d20160315_t1038271_e1039513_b22701_c20160315104450530043_cspp_dev.h5
DEBUG : Analyzing /home/IPOPP/AVHRR_raw/SVM13_npp_d20160315_t1038271_e1039513_b22701_c20160315104450574003_cspp_dev.h5
DEBUG : Unnecessary hdf5 file: /home/IPOPP/AVHRR_raw/SVM13_npp_d20160315_t1038271_e1039513_b22701_c20160315104450574003_cspp_dev.h5
DEBUG : Analyzing /home/IPOPP/AVHRR_raw/SVM15_npp_d20160315_t1038271_e1039513_b22701_c20160315104450686295_cspp_dev.h5
DEBUG : Unnecessary hdf5 file: /home/IPOPP/AVHRR_raw/SVM15_npp_d20160315_t1038271_e1039513_b22701_c20160315104450686295_cspp_dev.h5
DEBUG : Analyzing /home/IPOPP/AVHRR_raw/SVM04_npp_d20160315_t1038271_e1039513_b22701_c20160315104450149665_cspp_dev.h5
DEBUG : Loading .Platform_Short_Name from SVM04_npp_d20160315_t1038271_e1039513_b22701_c20160315104450149665_cspp_dev.h5
DEBUG : Loading /Data_Products/VIIRS-M4-SDR/VIIRS-M4-SDR_Aggr.AggregateBeginningDate from SVM04_npp_d20160315_t1038271_e1039513_b22701_c20160315104450149665_cspp_dev.h5
DEBUG : Loading /Data_Products/VIIRS-M4-SDR/VIIRS-M4-SDR_Aggr.AggregateBeginningTime from SVM04_npp_d20160315_t1038271_e1039513_b22701_c20160315104450149665_cspp_dev.h5
DEBUG : Loading /Data_Products/VIIRS-M4-SDR/VIIRS-M4-SDR_Aggr.AggregateEndingDate from SVM04_npp_d20160315_t1038271_e1039513_b22701_c20160315104450149665_cspp_dev.h5
DEBUG : Loading /Data_Products/VIIRS-M4-SDR/VIIRS-M4-SDR_Aggr.AggregateEndingTime from SVM04_npp_d20160315_t1038271_e1039513_b22701_c20160315104450149665_cspp_dev.h5
DEBUG : Analyzing /home/IPOPP/AVHRR_raw/SVM11_npp_d20160315_t1038271_e1039513_b22701_c20160315104450486587_cspp_dev.h5
DEBUG : Loading .Platform_Short_Name from SVM11_npp_d20160315_t1038271_e1039513_b22701_c20160315104450486587_cspp_dev.h5
DEBUG : Loading /Data_Products/VIIRS-M11-SDR/VIIRS-M11-SDR_Aggr.AggregateBeginningDate from SVM11_npp_d20160315_t1038271_e1039513_b22701_c20160315104450486587_cspp_dev.h5
DEBUG : Loading /Data_Products/VIIRS-M11-SDR/VIIRS-M11-SDR_Aggr.AggregateBeginningTime from SVM11_npp_d20160315_t1038271_e1039513_b22701_c20160315104450486587_cspp_dev.h5
DEBUG : Loading /Data_Products/VIIRS-M11-SDR/VIIRS-M11-SDR_Aggr.AggregateEndingDate from SVM11_npp_d20160315_t1038271_e1039513_b22701_c20160315104450486587_cspp_dev.h5
DEBUG : Loading /Data_Products/VIIRS-M11-SDR/VIIRS-M11-SDR_Aggr.AggregateEndingTime from SVM11_npp_d20160315_t1038271_e1039513_b22701_c20160315104450486587_cspp_dev.h5
DEBUG : Analyzing /home/IPOPP/AVHRR_raw/SVM16_npp_d20160315_t1038271_e1039513_b22701_c20160315104450729735_cspp_dev.h5
DEBUG : Unnecessary hdf5 file: /home/IPOPP/AVHRR_raw/SVM16_npp_d20160315_t1038271_e1039513_b22701_c20160315104450729735_cspp_dev.h5
DEBUG : Analyzing /home/IPOPP/AVHRR_raw/SVM09_npp_d20160315_t1038271_e1039513_b22701_c20160315104450399541_cspp_dev.h5
DEBUG : Unnecessary hdf5 file: /home/IPOPP/AVHRR_raw/SVM09_npp_d20160315_t1038271_e1039513_b22701_c20160315104450399541_cspp_dev.h5
DEBUG : Analyzing /home/IPOPP/AVHRR_raw/SVM10_npp_d20160315_t1038271_e1039513_b22701_c20160315104450442969_cspp_dev.h5
DEBUG : Loading .Platform_Short_Name from SVM10_npp_d20160315_t1038271_e1039513_b22701_c20160315104450442969_cspp_dev.h5
DEBUG : Loading /Data_Products/VIIRS-M10-SDR/VIIRS-M10-SDR_Aggr.AggregateBeginningDate from SVM10_npp_d20160315_t1038271_e1039513_b22701_c20160315104450442969_cspp_dev.h5
DEBUG : Loading /Data_Products/VIIRS-M10-SDR/VIIRS-M10-SDR_Aggr.AggregateBeginningTime from SVM10_npp_d20160315_t1038271_e1039513_b22701_c20160315104450442969_cspp_dev.h5
DEBUG : Loading /Data_Products/VIIRS-M10-SDR/VIIRS-M10-SDR_Aggr.AggregateEndingDate from SVM10_npp_d20160315_t1038271_e1039513_b22701_c20160315104450442969_cspp_dev.h5
DEBUG : Loading /Data_Products/VIIRS-M10-SDR/VIIRS-M10-SDR_Aggr.AggregateEndingTime from SVM10_npp_d20160315_t1038271_e1039513_b22701_c20160315104450442969_cspp_dev.h5
DEBUG : Analyzing /home/IPOPP/AVHRR_raw/SVM08_npp_d20160315_t1038271_e1039513_b22701_c20160315104450359583_cspp_dev.h5
DEBUG : Loading .Platform_Short_Name from SVM08_npp_d20160315_t1038271_e1039513_b22701_c20160315104450359583_cspp_dev.h5
DEBUG : Loading /Data_Products/VIIRS-M8-SDR/VIIRS-M8-SDR_Aggr.AggregateBeginningDate from SVM08_npp_d20160315_t1038271_e1039513_b22701_c20160315104450359583_cspp_dev.h5
DEBUG : Loading /Data_Products/VIIRS-M8-SDR/VIIRS-M8-SDR_Aggr.AggregateBeginningTime from SVM08_npp_d20160315_t1038271_e1039513_b22701_c20160315104450359583_cspp_dev.h5
DEBUG : Loading /Data_Products/VIIRS-M8-SDR/VIIRS-M8-SDR_Aggr.AggregateEndingDate from SVM08_npp_d20160315_t1038271_e1039513_b22701_c20160315104450359583_cspp_dev.h5
DEBUG : Loading /Data_Products/VIIRS-M8-SDR/VIIRS-M8-SDR_Aggr.AggregateEndingTime from SVM08_npp_d20160315_t1038271_e1039513_b22701_c20160315104450359583_cspp_dev.h5
DEBUG : Analyzing /home/IPOPP/AVHRR_raw/GMODO_npp_d20160315_t1038271_e1039513_b22701_c20160315104448631693_cspp_dev.h5
DEBUG : Loading .Platform_Short_Name from GMODO_npp_d20160315_t1038271_e1039513_b22701_c20160315104448631693_cspp_dev.h5
DEBUG : Loading /Data_Products/VIIRS-MOD-GEO/VIIRS-MOD-GEO_Aggr.AggregateBeginningDate from GMODO_npp_d20160315_t1038271_e1039513_b22701_c20160315104448631693_cspp_dev.h5
DEBUG : Loading /Data_Products/VIIRS-MOD-GEO/VIIRS-MOD-GEO_Aggr.AggregateBeginningTime from GMODO_npp_d20160315_t1038271_e1039513_b22701_c20160315104448631693_cspp_dev.h5
DEBUG : Loading /Data_Products/VIIRS-MOD-GEO/VIIRS-MOD-GEO_Aggr.AggregateEndingDate from GMODO_npp_d20160315_t1038271_e1039513_b22701_c20160315104448631693_cspp_dev.h5
DEBUG : Loading /Data_Products/VIIRS-MOD-GEO/VIIRS-MOD-GEO_Aggr.AggregateEndingTime from GMODO_npp_d20160315_t1038271_e1039513_b22701_c20160315104448631693_cspp_dev.h5
DEBUG : Analyzing /home/IPOPP/AVHRR_raw/SVM03_npp_d20160315_t1038271_e1039513_b22701_c20160315104450094546_cspp_dev.h5
DEBUG : Loading .Platform_Short_Name from SVM03_npp_d20160315_t1038271_e1039513_b22701_c20160315104450094546_cspp_dev.h5
DEBUG : Loading /Data_Products/VIIRS-M3-SDR/VIIRS-M3-SDR_Aggr.AggregateBeginningDate from SVM03_npp_d20160315_t1038271_e1039513_b22701_c20160315104450094546_cspp_dev.h5
DEBUG : Loading /Data_Products/VIIRS-M3-SDR/VIIRS-M3-SDR_Aggr.AggregateBeginningTime from SVM03_npp_d20160315_t1038271_e1039513_b22701_c20160315104450094546_cspp_dev.h5
DEBUG : Loading /Data_Products/VIIRS-M3-SDR/VIIRS-M3-SDR_Aggr.AggregateEndingDate from SVM03_npp_d20160315_t1038271_e1039513_b22701_c20160315104450094546_cspp_dev.h5
DEBUG : Loading /Data_Products/VIIRS-M3-SDR/VIIRS-M3-SDR_Aggr.AggregateEndingTime from SVM03_npp_d20160315_t1038271_e1039513_b22701_c20160315104450094546_cspp_dev.h5
DEBUG : Analyzing /home/IPOPP/AVHRR_raw/SVM07_npp_d20160315_t1038271_e1039513_b22701_c20160315104451498017_cspp_dev.h5
DEBUG : Loading .Platform_Short_Name from SVM07_npp_d20160315_t1038271_e1039513_b22701_c20160315104451498017_cspp_dev.h5
DEBUG : Loading /Data_Products/VIIRS-M7-SDR/VIIRS-M7-SDR_Aggr.AggregateBeginningDate from SVM07_npp_d20160315_t1038271_e1039513_b22701_c20160315104451498017_cspp_dev.h5
DEBUG : Loading /Data_Products/VIIRS-M7-SDR/VIIRS-M7-SDR_Aggr.AggregateBeginningTime from SVM07_npp_d20160315_t1038271_e1039513_b22701_c20160315104451498017_cspp_dev.h5
DEBUG : Loading /Data_Products/VIIRS-M7-SDR/VIIRS-M7-SDR_Aggr.AggregateEndingDate from SVM07_npp_d20160315_t1038271_e1039513_b22701_c20160315104451498017_cspp_dev.h5
DEBUG : Loading /Data_Products/VIIRS-M7-SDR/VIIRS-M7-SDR_Aggr.AggregateEndingTime from SVM07_npp_d20160315_t1038271_e1039513_b22701_c20160315104451498017_cspp_dev.h5
DEBUG : Analyzing /home/IPOPP/AVHRR_raw/SVM01_npp_d20160315_t1038271_e1039513_b22701_c20160315104450010103_cspp_dev.h5
DEBUG : Unnecessary hdf5 file: /home/IPOPP/AVHRR_raw/SVM01_npp_d20160315_t1038271_e1039513_b22701_c20160315104450010103_cspp_dev.h5
DEBUG : Analyzing /home/IPOPP/AVHRR_raw/SVM14_npp_d20160315_t1038271_e1039513_b22701_c20160315104450641850_cspp_dev.h5
DEBUG : Unnecessary hdf5 file: /home/IPOPP/AVHRR_raw/SVM14_npp_d20160315_t1038271_e1039513_b22701_c20160315104450641850_cspp_dev.h5
DEBUG : Analyzing /home/IPOPP/AVHRR_raw/SVM02_npp_d20160315_t1038271_e1039513_b22701_c20160315104450050354_cspp_dev.h5
DEBUG : Unnecessary hdf5 file: /home/IPOPP/AVHRR_raw/SVM02_npp_d20160315_t1038271_e1039513_b22701_c20160315104450050354_cspp_dev.h5
DEBUG : Analyzing /home/IPOPP/AVHRR_raw/SVM06_npp_d20160315_t1038271_e1039513_b22701_c20160315104450258745_cspp_dev.h5
DEBUG : Unnecessary hdf5 file: /home/IPOPP/AVHRR_raw/SVM06_npp_d20160315_t1038271_e1039513_b22701_c20160315104450258745_cspp_dev.h5
DEBUG : Analyzing /home/IPOPP/AVHRR_raw/SVM05_npp_d20160315_t1038271_e1039513_b22701_c20160315104450202990_cspp_dev.h5
DEBUG : Loading .Platform_Short_Name from SVM05_npp_d20160315_t1038271_e1039513_b22701_c20160315104450202990_cspp_dev.h5
DEBUG : Loading /Data_Products/VIIRS-M5-SDR/VIIRS-M5-SDR_Aggr.AggregateBeginningDate from SVM05_npp_d20160315_t1038271_e1039513_b22701_c20160315104450202990_cspp_dev.h5
DEBUG : Loading /Data_Products/VIIRS-M5-SDR/VIIRS-M5-SDR_Aggr.AggregateBeginningTime from SVM05_npp_d20160315_t1038271_e1039513_b22701_c20160315104450202990_cspp_dev.h5
DEBUG : Loading /Data_Products/VIIRS-M5-SDR/VIIRS-M5-SDR_Aggr.AggregateEndingDate from SVM05_npp_d20160315_t1038271_e1039513_b22701_c20160315104450202990_cspp_dev.h5
DEBUG : Loading /Data_Products/VIIRS-M5-SDR/VIIRS-M5-SDR_Aggr.AggregateEndingTime from SVM05_npp_d20160315_t1038271_e1039513_b22701_c20160315104450202990_cspp_dev.h5
INFO : Could not find any existing crefl output will use VIIRS SDRs to create some
DEBUG : Loading the VIIRS frontend to check for daytime data
DEBUG : Day fraction set to 0.100000
DEBUG : Night fraction set to 0.100000
DEBUG : SZA threshold set to 100.000000
INFO : No files or paths provided as input, will search the current directory...
DEBUG : Searching '.' for useful files
ERROR : No useable files loaded
ERROR : Could not create crefl files from SDRs
DEBUG : Frontend exception:
Traceback (most recent call last):
File "/home/IPOPP/Downloads/polar2grid_v_2_0/ShellB3/lib/python2.7/site-packages/polar2grid-2.0.0-py2.7-linux-x86_64.egg/polar2grid/glue.py", line 257, in main
f = fcls(search_paths=args.data_files, **args.subgroup_args["Frontend Initialization"])
File "/home/IPOPP/Downloads/polar2grid_v_2_0/ShellB3/lib/python2.7/site-packages/polar2grid.crefl-2.0.0-py2.7.egg/polar2grid/crefl/crefl2swath.py", line 457, in __init__
self.create_viirs_crefl_files()
File "/home/IPOPP/Downloads/polar2grid_v_2_0/ShellB3/lib/python2.7/site-packages/polar2grid.crefl-2.0.0-py2.7.egg/polar2grid/crefl/crefl2swath.py", line 607, in create_viirs_crefl_files
f = viirs_module.Frontend(search_paths=geo_files)
File "/home/IPOPP/Downloads/polar2grid_v_2_0/ShellB3/lib/python2.7/site-packages/polar2grid.viirs-2.0.0-py2.7.egg/polar2grid/viirs/swath.py", line 491, in __init__
self._load_files(self.find_files_with_extensions())
File "/home/IPOPP/Downloads/polar2grid_v_2_0/ShellB3/lib/python2.7/site-packages/polar2grid.viirs-2.0.0-py2.7.egg/polar2grid/viirs/swath.py", line 548, in _load_files
raise ValueError("No useable files loaded")
ValueError: No useable files loaded
ERROR : crefl frontend failed to load and sort data files (see log for details)
davidh
Posts: 116
Joined: Tue Jun 04, 2013 11:19 am

Re: Some questions about Polar2Grid

Post by davidh »

Hi Dmitry,

The crefl2gtiff.sh script will only work for VIIRS and MODIS. As far as I know there is no corrected reflectance algorithm for AVHRR, at least Polar2Grid does not have it. You'll need to create your own custom RGB "compositor" which I think is mentioned on pages 31-32 of the installation guide, but I'll have to check that later once I find my copy of the document. The functionality that you need is pretty new in Polar2Grid and is considered an advanced concept (and will likely change in the future). I'll get you instructions if they are not in the installation guide. Once you create your custom RGB compositor you would then call the script like this:

Code: Select all

avhrr2gtiff.sh my_comp --compositor-configs /path/to/custom_compositors.ini ...
Where "..." are whatever arguments you normally use and "my_comp" is the name of the compositor you've created in your "custom_compositors.ini" file.

I'll get back to you later today.
davidh
Posts: 116
Joined: Tue Jun 04, 2013 11:19 am

Re: Some questions about Polar2Grid

Post by davidh »

Hi again,

It turns out I documented it already. You can find instructions for creating a custom RGB in section 7.1.3 of this page: http://www.ssec.wisc.edu/software/polar ... itors.html

Note that these instructions are for VIIRS, but you should be able to make the obvious changes to process avhrr.

Dave
Dmitry
Posts: 34
Joined: Mon Feb 29, 2016 3:56 am

Re: Some questions about Polar2Grid

Post by Dmitry »

Dave,

My question was just for VIIRS data . Perhaps you are confused by the way in which located VIIRS data (/ home / IPOPP / AVHRR_raw /).
Can I get advice the following question.
When you try to process the VIIRS data using crefl2gtiff I get the following log files and error (data description http://navigator.eumetsat.int/discovery ... 8CDAC2E1BB)
I understood everything by subject "AVHRR", Thanks!

Additionally: Can I form a multispectral Geotiff file (containing more than 3 channels of the image).

Regards,
Dmitry
davidh
Posts: 116
Joined: Tue Jun 04, 2013 11:19 am

Re: Some questions about Polar2Grid

Post by davidh »

Dmitry,

Yes it seems I misunderstood. Is it possible for you to put your data on an FTP server or somewhere else so that I could get access to it? From what I can tell it should be working.

Dave
Dmitry
Posts: 34
Joined: Mon Feb 29, 2016 3:56 am

Re: Some questions about Polar2Grid

Post by Dmitry »

Please, try this link
https://drive.google.com/open?id=0B-QeK ... Xk3TjlZdUk
Additionally: Can I form a multispectral Geotiff file (containing more than 3 channels of the image).
Dmitry
davidh
Posts: 116
Joined: Tue Jun 04, 2013 11:19 am

Re: Some questions about Polar2Grid

Post by davidh »

Hi Dmitry,

I figured out the problem and it seems to be a bug in Polar2Grid. The bug involves using the spheroid (not terrain corrected) geolocation files. Polar2Grid is looking for the wrong type of file. The simplest fix is to use the terrain corrected geolocation files if that is OK with you and you have access to them. The M band terrain corrected files would have the prefix "GMTCO" instead of "GMODO". If it is important to you to use the not terrain corrected geolocation files let me know and I can tell you what lines in the code to "hack" to fix this bug.

Thanks for finding this bug.

Dave
Dmitry
Posts: 34
Joined: Mon Feb 29, 2016 3:56 am

Re: Some questions about Polar2Grid

Post by Dmitry »

Dear Dave,

Thank you for your analysis.
Our data contain only GMODO files, because of this I need assistance for bugging the code and process this data .

Dmitry
Post Reply