Remapping AVHRR L1B files using "nearest" method

Post Reply
lubo
Posts: 23
Joined: Fri Sep 02, 2016 7:04 am

Remapping AVHRR L1B files using "nearest" method

Post by lubo »

Hello,

I have problems with remapping AVHRR L1B files produced by AAPP v8.3 using Polar2grid v2.2.1 with "nearest" remapping algorithm. Some files can be successfuly processed but some are producing a remapping error:

Code: Select all

[2019-06-06 09:20:52,233] : PID  16981 : DEBUG    : avhrr2gtiff : main : Starting script with arguments: /home/polar/polar2grid_v_2_2_1/common/ShellB3/lib/python2.7/site-packages/polar2grid/glue.py avhrr gtiff --grid-configs /home/polar/polar2grid_v_2_0/grid_configs/grid.conf -g CE_1KM --overwrite --method nearest -f hrpt_M03_20190605_1812_02994.l1b -vv
[2019-06-06 09:20:52,233] : PID  16981 : INFO     : avhrr2gtiff : main : Initializing reader...
[2019-06-06 09:20:52,233] : PID  16981 : DEBUG    : polar2grid.avhrr.avhrr2swath : __init__ : Day fraction set to 0.100000
[2019-06-06 09:20:52,234] : PID  16981 : DEBUG    : polar2grid.avhrr.avhrr2swath : __init__ : SZA threshold set to 100.000000
[2019-06-06 09:20:52,234] : PID  16981 : DEBUG    : polar2grid.avhrr.avhrr2swath : load_files : Recognize file /home/polar/test_polar2grid/hrpt_M03_20190605_1812_02994.l1b as file type FT_AAPP
[2019-06-06 09:20:52,236] : PID  16981 : DEBUG    : polar2grid.core.script_utils : rename_log_file : Log renamed from '/home/polar/test_polar2grid/avhrr2gtiff_fail.log' to 'avhrr2gtiff_20190605_181227.log'
[2019-06-06 09:20:52,236] : PID  16981 : INFO     : avhrr2gtiff : main : Initializing remapping...
[2019-06-06 09:20:52,237] : PID  16981 : DEBUG    : polar2grid.grids.manager : __init__ : Loading grid configuration '/home/polar/polar2grid_v_2_0/grid_configs/grid.conf'
[2019-06-06 09:20:52,237] : PID  16981 : DEBUG    : polar2grid.grids.manager : parse_proj4_config_line : Converted grid 'p4_211e' origin from (lon: -122.940000, lat: 59.860000) to (x: -1948646.135314, y: 4364965.144534)
[2019-06-06 09:20:52,238] : PID  16981 : DEBUG    : polar2grid.grids.manager : parse_proj4_config_line : Converted grid 'SK_375M' origin from (lon: 14.064000, lat: 51.074000) to (x: -375015.571226, y: 281262.419400)
[2019-06-06 09:20:52,238] : PID  16981 : DEBUG    : polar2grid.grids.manager : parse_proj4_config_line : Converted grid 'CE_750M' origin from (lon: 8.088000, lat: 53.192000) to (x: -749977.875688, y: 562514.487241)
[2019-06-06 09:20:52,238] : PID  16981 : DEBUG    : polar2grid.grids.manager : parse_proj4_config_line : Converted grid 'SK_HKM' origin from (lon: 12.133000, lat: 51.810000) to (x: -499986.282546, y: 375000.711078)
[2019-06-06 09:20:52,238] : PID  16981 : DEBUG    : polar2grid.grids.manager : parse_proj4_config_line : Converted grid 'CE_1KM' origin from (lon: 3.800000, lat: 54.444000) to (x: -1000030.365839, y: 749972.522314)
[2019-06-06 09:20:52,239] : PID  16981 : DEBUG    : polar2grid.grids.manager : parse_proj4_config_line : Converted grid 'EU_1500M' origin from (lon: -44.726000, lat: 59.743000) to (x: -2999978.647599, y: 2250015.950697)
[2019-06-06 09:20:52,239] : PID  16981 : DEBUG    : polar2grid.grids.manager : parse_proj4_config_line : Converted grid 'EU_1500M_merc' origin from (lon: -30.749000, lat: 69.773000) to (x: -3000026.073477, y: 6453392.566320)
[2019-06-06 09:20:52,239] : PID  16981 : DEBUG    : polar2grid.grids.manager : parse_proj4_config_line : Converted grid 'MSG-HRV' origin from (lon: -32.146000, lat: 58.196000) to (x: -2609998.550415, y: 1957551.824550)
[2019-06-06 09:20:52,239] : PID  16981 : DEBUG    : polar2grid.grids.manager : parse_proj4_config_line : Converted grid 'colorado' origin from (lon: -107.338000, lat: 39.251000) to (x: -150037.401209, y: 112536.245998)
[2019-06-06 09:20:52,239] : PID  16981 : DEBUG    : polar2grid.grids.manager : parse_proj4_config_line : Converted grid 'grand_canyon' origin from (lon: -114.936000, lat: 37.756000) to (x: -249985.194471, y: 187447.912307)
[2019-06-06 09:20:52,240] : PID  16981 : DEBUG    : polar2grid.grids.manager : parse_proj4_config_line : Converted grid 'barrow' origin from (lon: -169.311000, lat: 74.301000) to (x: -374993.013984, y: -1676110.707409)
[2019-06-06 09:20:52,240] : PID  16981 : INFO     : avhrr2gtiff : main : Initializing backend...
[2019-06-06 09:20:52,240] : PID  16981 : DEBUG    : polar2grid.core.rescale : __init__ : Loading rescale configuration files:
	polar2grid.core:rescale_configs/rescale.ini
[2019-06-06 09:20:52,240] : PID  16981 : DEBUG    : polar2grid.core.roles : open_config_file : Loading package provided configuration file: 'polar2grid.core:rescale_configs/rescale.ini'
[2019-06-06 09:20:52,283] : PID  16981 : INFO     : avhrr2gtiff : main : Initializing compositor objects...
[2019-06-06 09:20:52,283] : PID  16981 : INFO     : avhrr2gtiff : main : Extracting swaths from data files available...
[2019-06-06 09:20:52,283] : PID  16981 : DEBUG    : polar2grid.avhrr.avhrr2swath : create_scene : Loading scene data...
[2019-06-06 09:20:52,284] : PID  16981 : DEBUG    : polar2grid.avhrr.avhrr2swath : create_scene : No products specified to frontend, will try to load logical defaults
[2019-06-06 09:20:52,284] : PID  16981 : DEBUG    : polar2grid.core.frontend_utils : dependency_ordered_products : Searching for dependencies for 'band2_vis'
[2019-06-06 09:20:52,284] : PID  16981 : DEBUG    : polar2grid.core.frontend_utils : get_product_dependencies : Product Dependency: Additional processing will be required for product 'band2_vis'
[2019-06-06 09:20:52,284] : PID  16981 : DEBUG    : polar2grid.core.frontend_utils : dependency_ordered_products : Searching for dependencies for 'band3a_vis'
[2019-06-06 09:20:52,284] : PID  16981 : DEBUG    : polar2grid.core.frontend_utils : get_product_dependencies : Product Dependency: Additional processing will be required for product 'band3a_vis'
[2019-06-06 09:20:52,284] : PID  16981 : DEBUG    : polar2grid.core.frontend_utils : dependency_ordered_products : Searching for dependencies for 'band1_vis'
[2019-06-06 09:20:52,284] : PID  16981 : DEBUG    : polar2grid.core.frontend_utils : get_product_dependencies : Product Dependency: Additional processing will be required for product 'band1_vis'
[2019-06-06 09:20:52,284] : PID  16981 : DEBUG    : polar2grid.core.frontend_utils : dependency_ordered_products : Searching for dependencies for 'band3b_bt'
[2019-06-06 09:20:52,284] : PID  16981 : DEBUG    : polar2grid.core.frontend_utils : get_product_dependencies : Product Dependency: Additional processing will be required for product 'band3b_bt'
[2019-06-06 09:20:52,284] : PID  16981 : DEBUG    : polar2grid.core.frontend_utils : dependency_ordered_products : Searching for dependencies for 'band4_bt'
[2019-06-06 09:20:52,284] : PID  16981 : DEBUG    : polar2grid.core.frontend_utils : dependency_ordered_products : Searching for dependencies for 'band5_bt'
[2019-06-06 09:20:52,284] : PID  16981 : INFO     : polar2grid.avhrr.avhrr2swath : create_scene : Creating navigation product 'longitude1km'
[2019-06-06 09:20:52,285] : PID  16981 : DEBUG    : polar2grid.avhrr.avhrr2swath : create_raw_swath_object : Using file type 'FT_AAPP' and getting file key 'LongitudeVar' for product 'longitude1km'
[2019-06-06 09:20:52,285] : PID  16981 : DEBUG    : polar2grid.avhrr.avhrr2swath : create_raw_swath_object : Writing product 'longitude1km' data to binary file
[2019-06-06 09:20:52,285] : PID  16981 : DEBUG    : polar2grid.core.frontend_utils : write_var_to_flat_binary : Writing binary data for 'LongitudeVar' to file 'longitude1km.dat'
[2019-06-06 09:20:52,290] : PID  16981 : DEBUG    : polar2grid.avhrr.readers : geolocation_calibration : Interpolating 40km navigation to 1km...
[2019-06-06 09:20:54,010] : PID  16981 : DEBUG    : polar2grid.core.fbf : append : 2218 rows in output file
[2019-06-06 09:20:54,010] : PID  16981 : DEBUG    : polar2grid.core.frontend_utils : write_var_to_flat_binary : File longitude1km.dat has shape (2218, 2048)
[2019-06-06 09:20:54,011] : PID  16981 : INFO     : polar2grid.avhrr.avhrr2swath : create_scene : Creating navigation product 'latitude1km'
[2019-06-06 09:20:54,011] : PID  16981 : DEBUG    : polar2grid.avhrr.avhrr2swath : create_raw_swath_object : Using file type 'FT_AAPP' and getting file key 'LatitudeVar' for product 'latitude1km'
[2019-06-06 09:20:54,011] : PID  16981 : DEBUG    : polar2grid.avhrr.avhrr2swath : create_raw_swath_object : Writing product 'latitude1km' data to binary file
[2019-06-06 09:20:54,011] : PID  16981 : DEBUG    : polar2grid.core.frontend_utils : write_var_to_flat_binary : Writing binary data for 'LatitudeVar' to file 'latitude1km.dat'
[2019-06-06 09:20:54,057] : PID  16981 : DEBUG    : polar2grid.core.fbf : append : 2218 rows in output file
[2019-06-06 09:20:54,057] : PID  16981 : DEBUG    : polar2grid.core.frontend_utils : write_var_to_flat_binary : File latitude1km.dat has shape (2218, 2048)
[2019-06-06 09:20:54,057] : PID  16981 : INFO     : polar2grid.avhrr.avhrr2swath : create_scene : Creating data product 'band2_vis'
[2019-06-06 09:20:54,058] : PID  16981 : DEBUG    : polar2grid.avhrr.avhrr2swath : create_raw_swath_object : Using file type 'FT_AAPP' and getting file key 'Band2Var' for product 'band2_vis'
[2019-06-06 09:20:54,058] : PID  16981 : DEBUG    : polar2grid.avhrr.avhrr2swath : create_raw_swath_object : Writing product 'band2_vis' data to binary file
[2019-06-06 09:20:54,058] : PID  16981 : DEBUG    : polar2grid.core.frontend_utils : write_var_to_flat_binary : Writing binary data for 'Band2Var' to file 'band2_vis.dat'
[2019-06-06 09:20:54,083] : PID  16981 : DEBUG    : polar2grid.avhrr.readers : _vis_calibrate : No valid operational coefficients, fall back to pre-launch
[2019-06-06 09:20:54,256] : PID  16981 : DEBUG    : polar2grid.core.fbf : append : 2218 rows in output file
[2019-06-06 09:20:54,256] : PID  16981 : DEBUG    : polar2grid.core.frontend_utils : write_var_to_flat_binary : File band2_vis.dat has shape (2218, 2048)
[2019-06-06 09:20:54,256] : PID  16981 : DEBUG    : polar2grid.core.containers : set_persist : Setting persist to False for child 'swath_definition'
[2019-06-06 09:20:54,257] : PID  16981 : INFO     : polar2grid.avhrr.avhrr2swath : create_scene : Creating data product 'band3a_vis'
[2019-06-06 09:20:54,257] : PID  16981 : DEBUG    : polar2grid.avhrr.avhrr2swath : create_raw_swath_object : Using file type 'FT_AAPP' and getting file key 'Band3aVar' for product 'band3a_vis'
[2019-06-06 09:20:54,257] : PID  16981 : DEBUG    : polar2grid.avhrr.avhrr2swath : create_raw_swath_object : Writing product 'band3a_vis' data to binary file
[2019-06-06 09:20:54,257] : PID  16981 : DEBUG    : polar2grid.core.frontend_utils : write_var_to_flat_binary : Writing binary data for 'Band3aVar' to file 'band3a_vis.dat'
[2019-06-06 09:20:54,274] : PID  16981 : DEBUG    : polar2grid.avhrr.readers : _vis_calibrate : No valid operational coefficients, fall back to pre-launch
[2019-06-06 09:20:54,444] : PID  16981 : DEBUG    : polar2grid.core.fbf : append : 2218 rows in output file
[2019-06-06 09:20:54,444] : PID  16981 : DEBUG    : polar2grid.core.frontend_utils : write_var_to_flat_binary : File band3a_vis.dat has shape (2218, 2048)
[2019-06-06 09:20:54,444] : PID  16981 : DEBUG    : polar2grid.core.containers : set_persist : Setting persist to False for child 'swath_definition'
[2019-06-06 09:20:54,445] : PID  16981 : INFO     : polar2grid.avhrr.avhrr2swath : create_scene : Creating data product 'band1_vis'
[2019-06-06 09:20:54,445] : PID  16981 : DEBUG    : polar2grid.avhrr.avhrr2swath : create_raw_swath_object : Using file type 'FT_AAPP' and getting file key 'Band1Var' for product 'band1_vis'
[2019-06-06 09:20:54,445] : PID  16981 : DEBUG    : polar2grid.avhrr.avhrr2swath : create_raw_swath_object : Writing product 'band1_vis' data to binary file
[2019-06-06 09:20:54,445] : PID  16981 : DEBUG    : polar2grid.core.frontend_utils : write_var_to_flat_binary : Writing binary data for 'Band1Var' to file 'band1_vis.dat'
[2019-06-06 09:20:54,462] : PID  16981 : DEBUG    : polar2grid.avhrr.readers : _vis_calibrate : No valid operational coefficients, fall back to pre-launch
[2019-06-06 09:20:54,635] : PID  16981 : DEBUG    : polar2grid.core.fbf : append : 2218 rows in output file
[2019-06-06 09:20:54,635] : PID  16981 : DEBUG    : polar2grid.core.frontend_utils : write_var_to_flat_binary : File band1_vis.dat has shape (2218, 2048)
[2019-06-06 09:20:54,635] : PID  16981 : DEBUG    : polar2grid.core.containers : set_persist : Setting persist to False for child 'swath_definition'
[2019-06-06 09:20:54,635] : PID  16981 : INFO     : polar2grid.avhrr.avhrr2swath : create_scene : Creating data product 'band3b_bt'
[2019-06-06 09:20:54,636] : PID  16981 : DEBUG    : polar2grid.avhrr.avhrr2swath : create_raw_swath_object : Using file type 'FT_AAPP' and getting file key 'Band3bVar' for product 'band3b_bt'
[2019-06-06 09:20:54,636] : PID  16981 : DEBUG    : polar2grid.avhrr.avhrr2swath : create_raw_swath_object : Writing product 'band3b_bt' data to binary file
[2019-06-06 09:20:54,636] : PID  16981 : DEBUG    : polar2grid.core.frontend_utils : write_var_to_flat_binary : Writing binary data for 'Band3bVar' to file 'band3b_bt.dat'
[2019-06-06 09:20:54,725] : PID  16981 : DEBUG    : polar2grid.avhrr.readers : _ir_calibrate : Suspicious scan lines: [ 693  694  695 ..., 2215 2216 2217]
[2019-06-06 09:20:54,799] : PID  16981 : DEBUG    : polar2grid.avhrr.readers : _ir_calibrate : Suspect scan lines: [ 693  694  695 ..., 2215 2216 2217]
[2019-06-06 09:20:54,948] : PID  16981 : DEBUG    : polar2grid.core.fbf : append : 2218 rows in output file
[2019-06-06 09:20:54,948] : PID  16981 : DEBUG    : polar2grid.core.frontend_utils : write_var_to_flat_binary : File band3b_bt.dat has shape (2218, 2048)
[2019-06-06 09:20:54,949] : PID  16981 : DEBUG    : polar2grid.core.containers : set_persist : Setting persist to False for child 'swath_definition'
[2019-06-06 09:20:54,949] : PID  16981 : INFO     : polar2grid.avhrr.avhrr2swath : create_scene : Creating data product 'band4_bt'
[2019-06-06 09:20:54,949] : PID  16981 : DEBUG    : polar2grid.avhrr.avhrr2swath : create_raw_swath_object : Using file type 'FT_AAPP' and getting file key 'Band4Var' for product 'band4_bt'
[2019-06-06 09:20:54,949] : PID  16981 : DEBUG    : polar2grid.avhrr.avhrr2swath : create_raw_swath_object : Writing product 'band4_bt' data to binary file
[2019-06-06 09:20:54,949] : PID  16981 : DEBUG    : polar2grid.core.frontend_utils : write_var_to_flat_binary : Writing binary data for 'Band4Var' to file 'band4_bt.dat'
[2019-06-06 09:20:55,280] : PID  16981 : DEBUG    : polar2grid.core.fbf : append : 2218 rows in output file
[2019-06-06 09:20:55,280] : PID  16981 : DEBUG    : polar2grid.core.frontend_utils : write_var_to_flat_binary : File band4_bt.dat has shape (2218, 2048)
[2019-06-06 09:20:55,281] : PID  16981 : DEBUG    : polar2grid.core.containers : set_persist : Setting persist to False for child 'swath_definition'
[2019-06-06 09:20:55,281] : PID  16981 : INFO     : polar2grid.avhrr.avhrr2swath : create_scene : Creating data product 'band5_bt'
[2019-06-06 09:20:55,281] : PID  16981 : DEBUG    : polar2grid.avhrr.avhrr2swath : create_raw_swath_object : Using file type 'FT_AAPP' and getting file key 'Band5Var' for product 'band5_bt'
[2019-06-06 09:20:55,281] : PID  16981 : DEBUG    : polar2grid.avhrr.avhrr2swath : create_raw_swath_object : Writing product 'band5_bt' data to binary file
[2019-06-06 09:20:55,281] : PID  16981 : DEBUG    : polar2grid.core.frontend_utils : write_var_to_flat_binary : Writing binary data for 'Band5Var' to file 'band5_bt.dat'
[2019-06-06 09:20:55,611] : PID  16981 : DEBUG    : polar2grid.core.fbf : append : 2218 rows in output file
[2019-06-06 09:20:55,611] : PID  16981 : DEBUG    : polar2grid.core.frontend_utils : write_var_to_flat_binary : File band5_bt.dat has shape (2218, 2048)
[2019-06-06 09:20:55,611] : PID  16981 : DEBUG    : polar2grid.core.containers : set_persist : Setting persist to False for child 'swath_definition'
[2019-06-06 09:20:55,611] : PID  16981 : INFO     : polar2grid.avhrr.avhrr2swath : create_scene : Creating secondary product 'band3b_bt'
[2019-06-06 09:20:55,618] : PID  16981 : DEBUG    : numpy : write : Warning: invalid value encountered in less

[2019-06-06 09:20:55,687] : PID  16981 : INFO     : polar2grid.avhrr.avhrr2swath : create_scene : Creating secondary product 'band1_vis'
[2019-06-06 09:20:56,008] : PID  16981 : DEBUG    : polar2grid.avhrr.avhrr2swath : day_check_reflectance : Reflectance product's scene has 99.943692% day data
[2019-06-06 09:20:56,008] : PID  16981 : INFO     : polar2grid.avhrr.avhrr2swath : create_scene : Creating secondary product 'band3a_vis'
[2019-06-06 09:20:56,015] : PID  16981 : DEBUG    : polar2grid.avhrr.avhrr2swath : _get_day_percentage : Using cached day percentage
[2019-06-06 09:20:56,015] : PID  16981 : DEBUG    : polar2grid.avhrr.avhrr2swath : day_check_reflectance : Reflectance product's scene has 99.943692% day data
[2019-06-06 09:20:56,015] : PID  16981 : INFO     : polar2grid.avhrr.avhrr2swath : create_scene : Creating secondary product 'band2_vis'
[2019-06-06 09:20:56,016] : PID  16981 : DEBUG    : polar2grid.avhrr.avhrr2swath : _get_day_percentage : Using cached day percentage
[2019-06-06 09:20:56,016] : PID  16981 : DEBUG    : polar2grid.avhrr.avhrr2swath : day_check_reflectance : Reflectance product's scene has 99.943692% day data
[2019-06-06 09:20:56,016] : PID  16981 : DEBUG    : avhrr2gtiff : main : Writer known grids: None
[2019-06-06 09:20:56,016] : PID  16981 : DEBUG    : avhrr2gtiff : main : Forced Grids: ['CE_1KM']
[2019-06-06 09:20:56,016] : PID  16981 : DEBUG    : avhrr2gtiff : main : Grids that will be mapped to: ['CE_1KM']
[2019-06-06 09:20:56,016] : PID  16981 : INFO     : avhrr2gtiff : main : Remapping to grid CE_1KM
[2019-06-06 09:20:56,016] : PID  16981 : DEBUG    : polar2grid.remap.remap : remap_scene : Remap scene being run with method 'nearest'
[2019-06-06 09:20:56,016] : PID  16981 : DEBUG    : polar2grid.remap.remap : remap_scene : Running ll2cr on the highest resolution swath to determine if it fits
[2019-06-06 09:20:56,016] : PID  16981 : DEBUG    : polar2grid.remap.remap : run_ll2cr : Swath '1km_nav' -> Grid 'CE_1KM'
[2019-06-06 09:20:56,074] : PID  16981 : DEBUG    : polar2grid.remap.ll2cr : ll2cr : Copying longitude and latitude arrays because inplace processing could not be done
[2019-06-06 09:20:56,096] : PID  16981 : DEBUG    : polar2grid.remap.ll2cr : ll2cr : Running static version of ll2cr...
[2019-06-06 09:20:56,545] : PID  16981 : DEBUG    : polar2grid.remap.ll2cr : ll2cr : Copying result arrays back to provided inplace array
[2019-06-06 09:20:56,560] : PID  16981 : DEBUG    : polar2grid.remap.remap : run_ll2cr : Grid information:
	grid_name: CE_1KM
	proj4_definition: +proj=aea +lon_0=19.5 +lat_0=48.7 +lat_1=38.7 +lat_2=58.7 +units=km +ellps=WGS84
	cell_width: 1000.0
	cell_height: -1000.0
	width: 2000
	height: 1500
	origin_x: -1000030.36584
	origin_y: 749972.522314
[2019-06-06 09:20:56,561] : PID  16981 : DEBUG    : polar2grid.remap.remap : run_ll2cr : Data fits in grid CE_1KM and uses 11.681391% of the swath
[2019-06-06 09:20:56,589] : PID  16981 : INFO     : polar2grid.remap.remap : remap_scene : Grid information:
	grid_name: CE_1KM
	proj4_definition: +proj=aea +lon_0=19.5 +lat_0=48.7 +lat_1=38.7 +lat_2=58.7 +units=km +ellps=WGS84
	cell_width: 1000.0
	cell_height: -1000.0
	width: 2000
	height: 1500
	origin_x: -1000030.36584
	origin_y: 749972.522314
[2019-06-06 09:20:56,589] : PID  16981 : DEBUG    : polar2grid.remap.remap : _remap_scene_nearest : Running ll2cr on the geolocation data for the following products:
	band2_vis
	band3a_vis
	band1_vis
	band3b_bt
	band4_bt
	band5_bt
[2019-06-06 09:20:56,589] : PID  16981 : DEBUG    : polar2grid.remap.remap : _remap_scene_nearest : Swath name: 1km_nav
[2019-06-06 09:20:56,589] : PID  16981 : DEBUG    : polar2grid.remap.remap : _remap_scene_nearest : Running nearest neighbor for the following products:
	band2_vis
	band3a_vis
	band1_vis
	band3b_bt
	band4_bt
	band5_bt
[2019-06-06 09:20:56,645] : PID  16981 : DEBUG    : polar2grid.remap.remap : _remap_scene_nearest : Combining data masks before building KDTree for nearest neighbor: band2_vis
[2019-06-06 09:20:56,652] : PID  16981 : DEBUG    : polar2grid.remap.remap : _remap_scene_nearest : Combining data masks before building KDTree for nearest neighbor: band3a_vis
[2019-06-06 09:20:56,659] : PID  16981 : DEBUG    : polar2grid.remap.remap : _remap_scene_nearest : Combining data masks before building KDTree for nearest neighbor: band1_vis
[2019-06-06 09:20:56,666] : PID  16981 : DEBUG    : polar2grid.remap.remap : _remap_scene_nearest : Combining data masks before building KDTree for nearest neighbor: band3b_bt
[2019-06-06 09:20:56,673] : PID  16981 : DEBUG    : polar2grid.remap.remap : _remap_scene_nearest : Combining data masks before building KDTree for nearest neighbor: band4_bt
[2019-06-06 09:20:56,681] : PID  16981 : DEBUG    : polar2grid.remap.remap : _remap_scene_nearest : Combining data masks before building KDTree for nearest neighbor: band5_bt
[2019-06-06 09:20:56,710] : PID  16981 : DEBUG    : polar2grid.remap.remap : _remap_scene_nearest : Remapping exception: 
Traceback (most recent call last):
  File "/home/polar/polar2grid_v_2_2_1/common/ShellB3/lib/python2.7/site-packages/polar2grid/remap/remap.py", line 445, in _remap_scene_nearest
    dist, i = cKDTree(x).query(xi, distance_upper_bound=kwargs["distance_upper_bound"])
  File "scipy/spatial/ckdtree.pyx", line 517, in scipy.spatial.ckdtree.cKDTree.__init__ (scipy/spatial/ckdtree.cxx:5636)
  File "/home/polar/polar2grid_v_2_2_1/common/ShellB3/lib/python2.7/site-packages/numpy-1.11.2-py2.7-linux-x86_64.egg/numpy/core/fromnumeric.py", line 2297, in amax
    out=out, **kwargs)
  File "/home/polar/polar2grid_v_2_2_1/common/ShellB3/lib/python2.7/site-packages/numpy-1.11.2-py2.7-linux-x86_64.egg/numpy/core/_methods.py", line 26, in _amax
    return umr_maximum(a, axis, None, out, keepdims)
ValueError: zero-size array to reduction operation maximum which has no identity
[2019-06-06 09:20:56,711] : PID  16981 : ERROR    : polar2grid.remap.remap : _remap_scene_nearest : Remapping error
[2019-06-06 09:20:56,711] : PID  16981 : DEBUG    : polar2grid.remap.remap : _safe_remove : Removing intermediate file 'll2cr_rows_CE_1KM_1km_nav.dat'...
[2019-06-06 09:20:56,712] : PID  16981 : DEBUG    : polar2grid.remap.remap : _safe_remove : Removing intermediate file 'll2cr_cols_CE_1KM_1km_nav.dat'...
[2019-06-06 09:20:56,712] : PID  16981 : DEBUG    : avhrr2gtiff : main : Remapping data exception: 
Traceback (most recent call last):
  File "/home/polar/polar2grid_v_2_2_1/common/ShellB3/lib/python2.7/site-packages/polar2grid/glue.py", line 382, in main
    gridded_scene = remapper.remap_scene(scene, grid_name, **remap_kwargs)
  File "/home/polar/polar2grid_v_2_2_1/common/ShellB3/lib/python2.7/site-packages/polar2grid/remap/remap.py", line 174, in remap_scene
    return func(swath_scene, grid_def, **kwargs)
  File "/home/polar/polar2grid_v_2_2_1/common/ShellB3/lib/python2.7/site-packages/polar2grid/remap/remap.py", line 511, in _remap_scene_nearest
    raise RuntimeError("Nearest neighbor resampling could not remap any of the data to grid '%s'" % (grid_name,))
RuntimeError: Nearest neighbor resampling could not remap any of the data to grid 'CE_1KM'
[2019-06-06 09:20:56,712] : PID  16981 : ERROR    : avhrr2gtiff : main : Remapping data failed
[2019-06-06 09:20:56,730] : PID  16981 : DEBUG    : polar2grid.core.containers : cleanup : Removing associated file that is no longer needed: 'band2_vis.dat'
[2019-06-06 09:20:56,734] : PID  16981 : DEBUG    : polar2grid.core.containers : cleanup : Removing associated file that is no longer needed: 'band3a_vis.dat'
[2019-06-06 09:20:56,738] : PID  16981 : DEBUG    : polar2grid.core.containers : cleanup : Removing associated file that is no longer needed: 'band1_vis.dat'
[2019-06-06 09:20:56,743] : PID  16981 : DEBUG    : polar2grid.core.containers : cleanup : Removing associated file that is no longer needed: 'band3b_bt.dat'
[2019-06-06 09:20:56,746] : PID  16981 : DEBUG    : polar2grid.core.containers : cleanup : Removing associated file that is no longer needed: 'band4_bt.dat'
[2019-06-06 09:20:56,750] : PID  16981 : DEBUG    : polar2grid.core.containers : cleanup : Removing associated file that is no longer needed: 'band5_bt.dat'
[2019-06-06 09:20:56,754] : PID  16981 : DEBUG    : polar2grid.core.containers : cleanup : Removing associated file that is no longer needed: 'longitude1km.dat'
[2019-06-06 09:20:56,758] : PID  16981 : DEBUG    : polar2grid.core.containers : cleanup : Removing associated file that is no longer needed: 'latitude1km.dat'
Processing the same files using "ewa" method works correctly but I would prefer the nearest neighbour method since the images from the "ewa" method look blurred at swath edges. Playing with the "--distance-upper-bound" settings did not help.

Can you, please, explain what is causing this error and advice some solution?

Thanks.

Lubo
davidh
Posts: 116
Joined: Tue Jun 04, 2013 11:19 am

Re: Remapping AVHRR L1B files using "nearest" method

Post by davidh »

Hi Lubo,

It is hard to tell why this is happening, but here are some guesses and questions I have:

1. The images produced with EWA resampling for these specific files, do they look correct? By that I mean do they have actual data in the image? My first guess is that this data doesn't actually overlap with the grid you've selected, but if EWA produces a valid image then this can't be the problem.

2. What values for distance upper bound did you try?

3. What happens if you change your custom grid to be "+units=m" instead of kilometers? It is possible there is a bug in nearest neighbor that assumes things are in meters.

Dave
lubo
Posts: 23
Joined: Fri Sep 02, 2016 7:04 am

Re: Remapping AVHRR L1B files using "nearest" method

Post by lubo »

Hi David,

thanks, here are my answers:

1. The images from problematic passes produced by EWA are covered with data. But it seems that all of them have a gap (missing scan line) as e.g. in attached geotiff files from METOP-B and METOP-C. None of the images produced so far, for which the nearest method was working have such gaps.

2. I have tried different values ranging from 0 to 1000.

3. There was no change with "+units=m" in the grid definition.

It seems the data gaps are causing the remapping error. But what is strange to me, there were no significant signal drops or poor signal quality during these overpasses. Maybe some problem with AAPP software..?

Lubo
Attachments
metopc_avhrr_band5_bt_20190605_181227_CE_1KM.tif
metopc_avhrr_band5_bt_20190605_181227_CE_1KM.tif (658.52 KiB) Viewed 22258 times
metopb_avhrr_band5_bt_20190606_182027_CE_1KM.tif
metopb_avhrr_band5_bt_20190606_182027_CE_1KM.tif (1.24 MiB) Viewed 22258 times
kathys
Posts: 487
Joined: Tue Jun 22, 2010 4:51 pm

Re: Remapping AVHRR L1B files using "nearest" method

Post by kathys »

Hi Lubo,

Can you please try attaching the tif files again? We are not able to download them.

Kathy
lubo
Posts: 23
Joined: Fri Sep 02, 2016 7:04 am

Re: Remapping AVHRR L1B files using "nearest" method

Post by lubo »

Hi Kathy and Dave,

Sorry for that, I have attached the tif files again but looking at the Preview it seems they are still not downloadable. It is strange becasue there was no error message uploading the files.

If it does not work I will upload the images to some data storage site and send you the link.

Lubo
Attachments
MetOp-B image with missing line
MetOp-B image with missing line
metopb_avhrr_band5_bt_20190606_182027_CE_1KM.tif (1.24 MiB) Viewed 22242 times
MetOp-C image with missing line
MetOp-C image with missing line
metopc_avhrr_band5_bt_20190605_181227_CE_1KM.tif (658.52 KiB) Viewed 22242 times
lubo
Posts: 23
Joined: Fri Sep 02, 2016 7:04 am

Re: Remapping AVHRR L1B files using "nearest" method

Post by lubo »

Hi,

you can download the problematic tif images together with corresponding AAPP L1B files from here:
https://drive.google.com/open?id=1vYXj1 ... 6-JmYrbQoP

Thanks.

Lubo
davidh
Posts: 116
Joined: Tue Jun 04, 2013 11:19 am

Re: Remapping AVHRR L1B files using "nearest" method

Post by davidh »

Hi Lubo,

I donwloaded your L1b files and ran Polar2Grid with EWA (default) and nearest neighbor resampling. I was able to reproduce your error, but if I removed the `+units=km` part of the grid definition then it worked for me with nearest neighbor resampling. Here is what my grid configuration script looks like:

Code: Select all

CE_1KM,proj4, +proj=aea +ellps=WGS84 +lon_0=19.5 +lat_0=48.7 +lat_1=38.7 +lat_2=58.7, 2000, 1500, 1000.0, -1000.0, -1000030.36584, 749972.522314
Note, I am running a Polar2Grid 2.3 alpha version that I'm developing locally, but the code for nearest neighbor resampling and AVHRR processing hasn't changed much. The only change made was a small fix for handling bad values of 0: https://github.com/ssec/polar2grid/pull/169

Does your grid definition look like mine? Could you try removing the `+units=km` part again and see if it helps.

Dave
lubo
Posts: 23
Joined: Fri Sep 02, 2016 7:04 am

Re: Remapping AVHRR L1B files using "nearest" method

Post by lubo »

Hi Dave,

my original grid definition looks like this:

Code: Select all

CE_1KM, proj4, +proj=aea +lon_0=19.5 +lat_0=48.7 +lat_1=38.7 +lat_2=58.7 +units=km +ellps=WGS84, 2000, 1500, 1000.000, -1000.000, 3.800deg, 54.444deg

I have removed the units part as you suggested and also tried to use your grid definition but, unfortunatelly, I still get the same remapping error with nearest neighbor method.

I have seen several times a warning about division by zero:
$HOME/polar2grid_v_2_0/ShellB3/lib/python2.7/site-packages/polar2grid.avhrr-1.0.0-py2.7.egg/polar2grid/avhrr/readers.py:371: RuntimeWarning: divide by zero encountered in divide
numpy.log(1 + ir_const_1 * cwnum * cwnum * cwnum / rad)

but the remapping error occurred also in cases when this warning did not appear.

Despite the small changes in software versions, do you think it would make sense to test with the same version as mine, i.e. v2.2.1 (if possible)? I do not know what other things can be different (hardware architecture, OS, libraries, ..?).

Thanks.

Lubo
davidh
Posts: 116
Joined: Tue Jun 04, 2013 11:19 am

Re: Remapping AVHRR L1B files using "nearest" method

Post by davidh »

You are absolutely correct. This doesn't seem to be working in version 2.2 and I'm not exactly sure why. Since it seems to be working in my 2.3 beta version I think the best option is to wait for version 2.3 to be released in the next month or so.
lubo
Posts: 23
Joined: Fri Sep 02, 2016 7:04 am

Re: Remapping AVHRR L1B files using "nearest" method

Post by lubo »

OK, I will wait for the new version, then.
Thanks.

Lubo
Post Reply