Polar2Grid VIIRS DNB hdf5 to Geotiff Problem

Post Reply
PearceRJ
Posts: 2
Joined: Fri Oct 26, 2018 3:57 am

Polar2Grid VIIRS DNB hdf5 to Geotiff Problem

Post by PearceRJ »

Hi there, I have been trying to convert the DNB hdf5 SDR files downloaded from the NOAA CLASS system into Geotiff with real values (the dtype real). However I think there's a problem with my files as I keep getting an error on 'adaptive_dnb' (see attached). Any advice?
Thanks in advance :)

Code: Select all

(envs) robert@robert-UX303UA:~$ /home/robert/polar2grid_v_2_2_1/bin/polar2grid.sh  viirs_sdr gtiff -p adaptive_dnb --dtype real4 -v -f /home/robert/viirs/data/05112018
DEBUG    : Starting script with arguments: /home/robert/polar2grid_v_2_2_1/common/ShellB3/lib/python2.7/site-packages/polar2grid/glue.py viirs_sdr gtiff -p adaptive_dnb --dtype real4 -v -f /home/robert/viirs/data/05112018 -vv
INFO     : Initializing reader...
DEBUG    : Day fraction set to 0.100000
DEBUG    : Night fraction set to 0.100000
DEBUG    : SZA threshold set to 100.000000
DEBUG    : Searching '/home/robert/viirs/data/05112018' for useful files
DEBUG    : Loading .Platform_Short_Name from GDNBO-SVDNB_npp_d20181104_t0105599_e0111403_b36372_c20181105105739877605_noac_ops.h5
DEBUG    : Loading /Data_Products/VIIRS-DNB-GEO/VIIRS-DNB-GEO_Aggr.AggregateBeginningDate from GDNBO-SVDNB_npp_d20181104_t0105599_e0111403_b36372_c20181105105739877605_noac_ops.h5
DEBUG    : Loading /Data_Products/VIIRS-DNB-GEO/VIIRS-DNB-GEO_Aggr.AggregateBeginningTime from GDNBO-SVDNB_npp_d20181104_t0105599_e0111403_b36372_c20181105105739877605_noac_ops.h5
DEBUG    : Loading /Data_Products/VIIRS-DNB-GEO/VIIRS-DNB-GEO_Aggr.AggregateEndingDate from GDNBO-SVDNB_npp_d20181104_t0105599_e0111403_b36372_c20181105105739877605_noac_ops.h5
DEBUG    : Loading /Data_Products/VIIRS-DNB-GEO/VIIRS-DNB-GEO_Aggr.AggregateEndingTime from GDNBO-SVDNB_npp_d20181104_t0105599_e0111403_b36372_c20181105105739877605_noac_ops.h5
DEBUG    : Loading .Platform_Short_Name from GDNBO-SVDNB_npp_d20181104_t0111415_e0117219_b36372_c20181105105739851428_noac_ops.h5
DEBUG    : Loading /Data_Products/VIIRS-DNB-GEO/VIIRS-DNB-GEO_Aggr.AggregateBeginningDate from GDNBO-SVDNB_npp_d20181104_t0111415_e0117219_b36372_c20181105105739851428_noac_ops.h5
DEBUG    : Loading /Data_Products/VIIRS-DNB-GEO/VIIRS-DNB-GEO_Aggr.AggregateBeginningTime from GDNBO-SVDNB_npp_d20181104_t0111415_e0117219_b36372_c20181105105739851428_noac_ops.h5
DEBUG    : Loading /Data_Products/VIIRS-DNB-GEO/VIIRS-DNB-GEO_Aggr.AggregateEndingDate from GDNBO-SVDNB_npp_d20181104_t0111415_e0117219_b36372_c20181105105739851428_noac_ops.h5
DEBUG    : Loading /Data_Products/VIIRS-DNB-GEO/VIIRS-DNB-GEO_Aggr.AggregateEndingTime from GDNBO-SVDNB_npp_d20181104_t0111415_e0117219_b36372_c20181105105739851428_noac_ops.h5
DEBUG    : Log renamed from '/home/robert/viirs_sdr2gtiff_fail.log' to 'viirs_sdr2gtiff_20181104_010559.log'
INFO     : Initializing remapping...
DEBUG    : Using default grid configuration: 'grids.conf' 
DEBUG    : Loading grid configuration 'grids.conf'
DEBUG    : Converted grid 'dwd_germany' origin from (lon: -2.000000, lat: 56.000000) to (x: -755570.370463, y: -3554679.114710)
DEBUG    : Converted grid 'australia' origin from (lon: 105.000000, lat: 5.000000) to (x: -3896182.177765, y: 557305.257275)
DEBUG    : Converted grid 'australia2' origin from (lon: 105.000000, lat: 5.000000) to (x: -3896182.177765, y: 557305.257275)
DEBUG    : Converted grid '211e' origin from (lon: -123.044000, lat: 59.844000) to (x: -1950510.636800, y: 4368587.226913)
DEBUG    : Converted grid '211e_10km' origin from (lon: -123.044000, lat: 59.844000) to (x: -1950510.636800, y: 4368587.226913)
DEBUG    : Converted grid '211e_hi' origin from (lon: -123.044000, lat: 59.844000) to (x: -1950510.636800, y: 4368587.226913)
DEBUG    : Converted grid '211w' origin from (lon: -152.855000, lat: 54.536000) to (x: -4226045.730436, y: 4368634.765713)
DEBUG    : Converted grid '211w_10km' origin from (lon: -152.855000, lat: 54.536000) to (x: -4226045.730436, y: 4368634.765713)
DEBUG    : Converted grid '203' origin from (lon: 115.601000, lat: 44.646000) to (x: -4952960.856044, y: 381022.610378)
DEBUG    : Converted grid '203_10km' origin from (lon: 115.601000, lat: 44.646000) to (x: -4952960.856044, y: 381022.610378)
DEBUG    : Converted grid '204' origin from (lon: 110.000000, lat: 60.644000) to (x: -7314463.252793, y: 8020503.279488)
DEBUG    : Converted grid '204_10km' origin from (lon: 110.000000, lat: 60.644000) to (x: -7314463.252793, y: 8020503.279488)
DEBUG    : Converted grid '205' origin from (lon: -115.304000, lat: 36.257000) to (x: -4952924.911284, y: -3429056.260492)
DEBUG    : Converted grid '210' origin from (lon: -77.000000, lat: 26.422000) to (x: 0.000000, y: 2864298.891608)
INFO     : Initializing backend...
DEBUG    : Loading rescale configuration files:
	polar2grid.core:rescale_configs/rescale.ini
DEBUG    : Loading package provided configuration file: 'polar2grid.core:rescale_configs/rescale.ini'
INFO     : Initializing compositor objects...
INFO     : Extracting swaths from data files available...
DEBUG    : Loading scene data...
DEBUG    : Searching for dependencies for 'adaptive_dnb'
DEBUG    : Product Dependency: To create 'adaptive_dnb', 'dnb' must be created first
DEBUG    : Product Dependency: To create 'adaptive_dnb', 'dnb_solar_zenith_angle' must be created first
DEBUG    : Product Dependency: To create 'adaptive_dnb', 'dnb_lunar_zenith_angle' must be created first
INFO     : Creating navigation product 'dnb_longitude'
DEBUG    : Using file type 'FT_GDNBO' and getting file key 'tclongitude' for product 'dnb_longitude'
DEBUG    : Writing product 'dnb_longitude' data to binary file
DEBUG    : Writing binary data for 'tclongitude' to file 'dnb_longitude.dat'
DEBUG    : Loading /All_Data/VIIRS-DNB-GEO_All/Longitude_TC from GDNBO-SVDNB_npp_d20181104_t0105599_e0111403_b36372_c20181105105739877605_noac_ops.h5
DEBUG    : No scaling factors for tclongitude
DEBUG    : 3072 rows in output file
DEBUG    : Loading /All_Data/VIIRS-DNB-GEO_All/Longitude_TC from GDNBO-SVDNB_npp_d20181104_t0111415_e0117219_b36372_c20181105105739851428_noac_ops.h5
DEBUG    : No scaling factors for tclongitude
DEBUG    : 6144 rows in output file
DEBUG    : File dnb_longitude.dat has shape (6144, 4064)
DEBUG    : Loading /All_Data/VIIRS-DNB-GEO_All/Longitude_TC from GDNBO-SVDNB_npp_d20181104_t0105599_e0111403_b36372_c20181105105739877605_noac_ops.h5
DEBUG    : Loading /All_Data/VIIRS-DNB-GEO_All/Longitude_TC from GDNBO-SVDNB_npp_d20181104_t0111415_e0117219_b36372_c20181105105739851428_noac_ops.h5
INFO     : Creating navigation product 'dnb_latitude'
DEBUG    : Using file type 'FT_GDNBO' and getting file key 'tclatitude' for product 'dnb_latitude'
DEBUG    : Writing product 'dnb_latitude' data to binary file
DEBUG    : Writing binary data for 'tclatitude' to file 'dnb_latitude.dat'
DEBUG    : Loading /All_Data/VIIRS-DNB-GEO_All/Latitude_TC from GDNBO-SVDNB_npp_d20181104_t0105599_e0111403_b36372_c20181105105739877605_noac_ops.h5
DEBUG    : No scaling factors for tclatitude
DEBUG    : 3072 rows in output file
DEBUG    : Loading /All_Data/VIIRS-DNB-GEO_All/Latitude_TC from GDNBO-SVDNB_npp_d20181104_t0111415_e0117219_b36372_c20181105105739851428_noac_ops.h5
DEBUG    : No scaling factors for tclatitude
DEBUG    : 6144 rows in output file
DEBUG    : File dnb_latitude.dat has shape (6144, 4064)
DEBUG    : Loading /All_Data/VIIRS-DNB-GEO_All/Latitude_TC from GDNBO-SVDNB_npp_d20181104_t0105599_e0111403_b36372_c20181105105739877605_noac_ops.h5
DEBUG    : Loading /All_Data/VIIRS-DNB-GEO_All/Latitude_TC from GDNBO-SVDNB_npp_d20181104_t0111415_e0117219_b36372_c20181105105739851428_noac_ops.h5
DEBUG    : Loading /All_Data/VIIRS-DNB-GEO_All/Latitude_TC from GDNBO-SVDNB_npp_d20181104_t0105599_e0111403_b36372_c20181105105739877605_noac_ops.h5
DEBUG    : Loading /All_Data/VIIRS-DNB-GEO_All/Latitude_TC from GDNBO-SVDNB_npp_d20181104_t0111415_e0117219_b36372_c20181105105739851428_noac_ops.h5
INFO     : Creating data product 'dnb'
ERROR    : Could not create product 'dnb' because some data files are missing
ERROR    : Could not create raw product 'dnb'
INFO     : Creating data product 'dnb_solar_zenith_angle'
DEBUG    : Using file type 'FT_GDNBO' and getting file key 'solar_zenith_angle' for product 'dnb_solar_zenith_angle'
DEBUG    : Writing product 'dnb_solar_zenith_angle' data to binary file
DEBUG    : Writing binary data for 'solar_zenith_angle' to file 'dnb_solar_zenith_angle.dat'
DEBUG    : Loading /All_Data/VIIRS-DNB-GEO_All/SolarZenithAngle from GDNBO-SVDNB_npp_d20181104_t0105599_e0111403_b36372_c20181105105739877605_noac_ops.h5
DEBUG    : No scaling factors for solar_zenith_angle
DEBUG    : 3072 rows in output file
DEBUG    : Loading /All_Data/VIIRS-DNB-GEO_All/SolarZenithAngle from GDNBO-SVDNB_npp_d20181104_t0111415_e0117219_b36372_c20181105105739851428_noac_ops.h5
DEBUG    : No scaling factors for solar_zenith_angle
DEBUG    : 6144 rows in output file
DEBUG    : File dnb_solar_zenith_angle.dat has shape (6144, 4064)
DEBUG    : Setting persist to False for child 'swath_definition'
DEBUG    : Loading /All_Data/VIIRS-DNB-GEO_All/SolarZenithAngle from GDNBO-SVDNB_npp_d20181104_t0105599_e0111403_b36372_c20181105105739877605_noac_ops.h5
DEBUG    : Loading /All_Data/VIIRS-DNB-GEO_All/SolarZenithAngle from GDNBO-SVDNB_npp_d20181104_t0111415_e0117219_b36372_c20181105105739851428_noac_ops.h5
INFO     : Creating data product 'dnb_lunar_zenith_angle'
DEBUG    : Using file type 'FT_GDNBO' and getting file key 'lunar_zenith_angle' for product 'dnb_lunar_zenith_angle'
DEBUG    : Writing product 'dnb_lunar_zenith_angle' data to binary file
DEBUG    : Writing binary data for 'lunar_zenith_angle' to file 'dnb_lunar_zenith_angle.dat'
DEBUG    : Loading /All_Data/VIIRS-DNB-GEO_All/LunarZenithAngle from GDNBO-SVDNB_npp_d20181104_t0105599_e0111403_b36372_c20181105105739877605_noac_ops.h5
DEBUG    : No scaling factors for lunar_zenith_angle
DEBUG    : 3072 rows in output file
DEBUG    : Loading /All_Data/VIIRS-DNB-GEO_All/LunarZenithAngle from GDNBO-SVDNB_npp_d20181104_t0111415_e0117219_b36372_c20181105105739851428_noac_ops.h5
DEBUG    : No scaling factors for lunar_zenith_angle
DEBUG    : 6144 rows in output file
DEBUG    : File dnb_lunar_zenith_angle.dat has shape (6144, 4064)
DEBUG    : Setting persist to False for child 'swath_definition'
DEBUG    : Loading /All_Data/VIIRS-DNB-GEO_All/LunarZenithAngle from GDNBO-SVDNB_npp_d20181104_t0105599_e0111403_b36372_c20181105105739877605_noac_ops.h5
DEBUG    : Loading /All_Data/VIIRS-DNB-GEO_All/LunarZenithAngle from GDNBO-SVDNB_npp_d20181104_t0111415_e0117219_b36372_c20181105105739851428_noac_ops.h5
INFO     : Creating secondary product 'adaptive_dnb'
DEBUG    : Loading /All_Data/VIIRS-DNB-GEO_All/MoonIllumFraction from GDNBO-SVDNB_npp_d20181104_t0105599_e0111403_b36372_c20181105105739877605_noac_ops.h5
DEBUG    : Loading /All_Data/VIIRS-DNB-GEO_All/MoonIllumFraction from GDNBO-SVDNB_npp_d20181104_t0111415_e0117219_b36372_c20181105105739851428_noac_ops.h5
ERROR    : Could not create product (unexpected error): 'adaptive_dnb'
DEBUG    : Could not create product (unexpected error): 'adaptive_dnb'
Traceback (most recent call last):
  File "/home/robert/polar2grid_v_2_2_1/common/ShellB3/lib/python2.7/site-packages/polar2grid/viirs/swath.py", line 812, in create_scene
    one_swath = product_func(product_name, swath_def, products_created)
  File "/home/robert/polar2grid_v_2_2_1/common/ShellB3/lib/python2.7/site-packages/polar2grid/viirs/swath.py", line 882, in create_adaptive_dnb
    dnb_product = products_created[dnb_product_name]
KeyError: 'dnb'
DEBUG    : Removing associated file that is no longer needed: 'dnb_solar_zenith_angle.dat'
DEBUG    : Removing associated file that is no longer needed: 'dnb_lunar_zenith_angle.dat'
DEBUG    : Removing associated file that is no longer needed: 'dnb_longitude.dat'
DEBUG    : Removing associated file that is no longer needed: 'dnb_latitude.dat'
ERROR    : No products were returned by the frontend
DEBUG    : Frontend data extraction exception: 
Traceback (most recent call last):
  File "/home/robert/polar2grid_v_2_2_1/common/ShellB3/lib/python2.7/site-packages/polar2grid/glue.py", line 344, in main
    raise RuntimeError("No products were returned by the frontend")
RuntimeError: No products were returned by the frontend
ERROR    : Frontend data extraction failed (see log for details)
davidh
Posts: 116
Joined: Tue Jun 04, 2013 11:19 am

Re: Polar2Grid VIIRS DNB hdf5 to Geotiff Problem

Post by davidh »

Hi Robert,

I know we're talking on the pytroll slack right now, but my guess is that Polar2Grid is having the same issue satpy is having. That being that aggregated files may not be supported at this time. I'd have to check the code again for the ability to handle these files. Since SatPy did not support them and you had to download the un-aggregated versions, you could retry polar2grid and probably get the results you wanted with the un-aggregated files.

Dave
Post Reply