Fail to run polar2gridv2.3 nucaps and viirs_edr_active_fires
Posted: Fri Oct 15, 2021 3:18 am
Dear all:
I installed polar2grid2.3 on centos7.8 and passed the test example. I have also tested several other readers, and they are running very well.
But when I use nucaps or viirs_edr_active_fires as the reader, I encountered some problems. I attached the error log below:
1. viirs_edr_active_fires:
The command I used is:
polar2grid.sh viirs_edr_active_fires gtiff -p confidence_pct T13 -f ../../active_fire/AFMOD_j01_d20210908_t0644280_e0645526_b19719_c20210909034222777833_cspp_dev.nc
result is:
INFO : Initializing reader...
INFO : Initializing remapping...
INFO : Initializing backend...
INFO : Initializing compositor objects...
INFO : Extracting swaths from data files available...
INFO : Writing longitude data to disk cache...
ERROR : Frontend data extraction failed (see log for details)
logfile is:
[2021-10-15 14:41:42,109] : PID 103198 : DEBUG : viirs_edr_active_fires2gtiff : main : Starting script with arguments: /home/qsw/program/p2g/polar2grid_v_2_3/lib/python3.7/site-packages/polar2grid/glue_legacy.py viirs_edr_active_fires gtiff -p confidence_pct T13 -f ../../active_fire/AFMOD_j01_d20210908_t0644280_e0645526_b19719_c20210909034222777833_cspp_dev.nc -vv
[2021-10-15 14:41:42,109] : PID 103198 : INFO : viirs_edr_active_fires2gtiff : main : Initializing reader...
[2021-10-15 14:41:42,110] : PID 103198 : DEBUG : satpy.scene : __init__ : Setting 'PPP_CONFIG_DIR' to '/home/qsw/program/p2g/polar2grid_v_2_3/etc/satpy'
[2021-10-15 14:41:42,112] : PID 103198 : DEBUG : satpy.readers : read_reader_config : Reading ['/home/qsw/program/p2g/polar2grid_v_2_3/lib/python3.7/site-packages/satpy/etc/readers/viirs_edr_active_fires.yaml']
[2021-10-15 14:41:42,360] : PID 103198 : DEBUG : satpy.readers.yaml_reader : create_filehandlers : Assigning to viirs_edr_active_fires: ['/mnt/hgfs/p2g_test/active_fire/AFMOD_j01_d20210908_t0644280_e0645526_b19719_c20210909034222777833_cspp_dev.nc']
[2021-10-15 14:41:42,382] : PID 103198 : DEBUG : satpy.composites : load_sensor_composites : Looking for composites config file viirs.yaml
[2021-10-15 14:41:42,515] : PID 103198 : DEBUG : satpy.composites : load_sensor_composites : Looking for composites config file visir.yaml
[2021-10-15 14:41:42,593] : PID 103198 : DEBUG : polar2grid.core.script_utils : rename_log_file : Log renamed from '/mnt/hgfs/p2g_test/output/active_fire/viirs_edr_active_fires2gtiff_fail.log' to 'viirs_edr_active_fires2gtiff_20210908_064428.log'
[2021-10-15 14:41:42,594] : PID 103198 : INFO : viirs_edr_active_fires2gtiff : main : Initializing remapping...
[2021-10-15 14:41:42,594] : PID 103198 : DEBUG : polar2grid.grids.manager : __init__ : Using default grid configuration: 'grids.conf'
[2021-10-15 14:41:42,594] : PID 103198 : DEBUG : polar2grid.grids.manager : __init__ : Loading grid configuration 'grids.conf'
[2021-10-15 14:41:42,685] : PID 103198 : DEBUG : polar2grid.grids.manager : parse_proj4_config_line : Converted grid 'dwd_germany' origin from (lon: -2.000000, lat: 56.000000) to (x: -755570.370463, y: -3554679.114710)
[2021-10-15 14:41:42,713] : PID 103198 : DEBUG : polar2grid.grids.manager : parse_proj4_config_line : Converted grid 'australia' origin from (lon: 105.000000, lat: 5.000000) to (x: -3896182.177765, y: 557305.257275)
[2021-10-15 14:41:42,715] : PID 103198 : DEBUG : polar2grid.grids.manager : parse_proj4_config_line : Converted grid 'australia2' origin from (lon: 105.000000, lat: 5.000000) to (x: -3896182.177765, y: 557305.257275)
[2021-10-15 14:41:42,717] : PID 103198 : DEBUG : polar2grid.grids.manager : parse_proj4_config_line : Converted grid '211e' origin from (lon: -123.044000, lat: 59.844000) to (x: -1950510.636800, y: 4368587.226913)
[2021-10-15 14:41:42,719] : PID 103198 : DEBUG : polar2grid.grids.manager : parse_proj4_config_line : Converted grid '211e_10km' origin from (lon: -123.044000, lat: 59.844000) to (x: -1950510.636800, y: 4368587.226913)
[2021-10-15 14:41:42,721] : PID 103198 : DEBUG : polar2grid.grids.manager : parse_proj4_config_line : Converted grid '211e_hi' origin from (lon: -123.044000, lat: 59.844000) to (x: -1950510.636800, y: 4368587.226913)
[2021-10-15 14:41:42,723] : PID 103198 : DEBUG : polar2grid.grids.manager : parse_proj4_config_line : Converted grid '211w' origin from (lon: -152.855000, lat: 54.536000) to (x: -4226045.730436, y: 4368634.765713)
[2021-10-15 14:41:42,724] : PID 103198 : DEBUG : polar2grid.grids.manager : parse_proj4_config_line : Converted grid '211w_10km' origin from (lon: -152.855000, lat: 54.536000) to (x: -4226045.730436, y: 4368634.765713)
[2021-10-15 14:41:42,726] : PID 103198 : DEBUG : polar2grid.grids.manager : parse_proj4_config_line : Converted grid '203' origin from (lon: 115.601000, lat: 44.646000) to (x: -4952960.856044, y: 381022.610378)
[2021-10-15 14:41:42,728] : PID 103198 : DEBUG : polar2grid.grids.manager : parse_proj4_config_line : Converted grid '203_10km' origin from (lon: 115.601000, lat: 44.646000) to (x: -4952960.856044, y: 381022.610378)
[2021-10-15 14:41:42,730] : PID 103198 : DEBUG : polar2grid.grids.manager : parse_proj4_config_line : Converted grid '204' origin from (lon: 110.000000, lat: 60.644000) to (x: -7314463.252793, y: 8020503.279488)
[2021-10-15 14:41:42,732] : PID 103198 : DEBUG : polar2grid.grids.manager : parse_proj4_config_line : Converted grid '204_10km' origin from (lon: 110.000000, lat: 60.644000) to (x: -7314463.252793, y: 8020503.279488)
[2021-10-15 14:41:42,734] : PID 103198 : DEBUG : polar2grid.grids.manager : parse_proj4_config_line : Converted grid '205' origin from (lon: -115.304000, lat: 36.257000) to (x: -4952924.911284, y: -3429056.260492)
[2021-10-15 14:41:42,735] : PID 103198 : DEBUG : polar2grid.grids.manager : parse_proj4_config_line : Converted grid '210' origin from (lon: -77.000000, lat: 26.422000) to (x: 0.000000, y: 2864298.891608)
[2021-10-15 14:41:42,736] : PID 103198 : INFO : viirs_edr_active_fires2gtiff : main : Initializing backend...
[2021-10-15 14:41:42,737] : PID 103198 : DEBUG : polar2grid.core.rescale : __init__ : Loading rescale configuration files:
polar2grid.core:rescale_configs/rescale.ini
[2021-10-15 14:41:42,739] : PID 103198 : DEBUG : polar2grid.core.roles : open_config_file : Loading package provided configuration file: 'polar2grid.core:rescale_configs/rescale.ini'
[2021-10-15 14:41:42,775] : PID 103198 : INFO : viirs_edr_active_fires2gtiff : main : Initializing compositor objects...
[2021-10-15 14:41:42,777] : PID 103198 : INFO : viirs_edr_active_fires2gtiff : main : Extracting swaths from data files available...
[2021-10-15 14:41:42,780] : PID 103198 : DEBUG : polar2grid.readers : create_scene : Loading scene data...
[2021-10-15 14:41:42,781] : PID 103198 : DEBUG : satpy.readers.yaml_reader : _load_dataset_area : No coordinates found for DatasetID(name='latitude', wavelength=None, resolution=None, polarization=None, calibration=None, level=None, modifiers=())
[2021-10-15 14:41:42,820] : PID 103198 : DEBUG : satpy.readers.yaml_reader : _load_dataset_area : No coordinates found for DatasetID(name='longitude', wavelength=None, resolution=None, polarization=None, calibration=None, level=None, modifiers=())
[2021-10-15 14:41:43,032] : PID 103198 : INFO : polar2grid.readers : area_to_swath_def : Writing longitude data to disk cache...
[2021-10-15 14:41:43,035] : PID 103198 : DEBUG : viirs_edr_active_fires2gtiff : main : Frontend data extraction exception:
Traceback (most recent call last):
File "/home/qsw/program/p2g/polar2grid_v_2_3/lib/python3.7/site-packages/polar2grid/glue_legacy.py", line 338, in main
scene = convert_satpy_to_p2g_swath(f, scene)
File "/home/qsw/program/p2g/polar2grid_v_2_3/lib/python3.7/site-packages/polar2grid/readers/__init__.py", line 317, in convert_satpy_to_p2g_swath
overwrite_existing=overwrite_existing)
File "/home/qsw/program/p2g/polar2grid_v_2_3/lib/python3.7/site-packages/polar2grid/readers/__init__.py", line 113, in area_to_swath_def
lon_arr = np.memmap(filename, mode="w+", dtype=lons.dtype, shape=lons.shape)
File "/home/qsw/program/p2g/polar2grid_v_2_3/lib/python3.7/site-packages/numpy/core/memmap.py", line 264, in __new__
mm = mmap.mmap(fid.fileno(), bytes, access=acc, offset=start)
ValueError: cannot mmap an empty file
[2021-10-15 14:41:43,037] : PID 103198 : ERROR : viirs_edr_active_fires2gtiff : main : Frontend data extraction failed (see log for details)
2. nucaps:
My data name is different from the data name on the nucaps website(https://www.ssec.wisc.edu/software/pola ... ucaps.html). The name format required on the website is: NUCAPS-EDR_v1r0_npp_s201603011158009_e201603011158307_c201603011222270.nc. But the data name format I got is NUCAPS-AGG-EDR_v2r0_m02_s202107270107170_e202107270119160_c2022107nc. And the data cannot be recognized by polar2gird. Then I changed the name of my data to NUCAPS-EDR_v2r0_m02_s202107270107170_e202107270119160_c202107270133220.nc. I tried to list products, but an error was reported. The result shows that there was no “mask surface” in the data.
The command I used is:
polar2grid.sh nucaps gtiff --list-products -f ../../nucaps/m02.202107270929test/NUCAPS-EDR_v2r0_m02_s202107270107170_e202107270119160_c202107270133220.nc
result is:
INFO : Initializing reader...
ERROR : Unexpected error. Enable debug messages (-vvv) or see log file for details.
logfile is:
[2021-10-15 15:31:58,282] : PID 76403 : DEBUG : nucaps2gtiff : main : Starting script with arguments: /home/qsw/program/p2g/polar2grid_v_2_3/lib/python3.7/site-packages/polar2grid/glue_legacy.py nucaps gtiff --list-products -f ../../nucaps/m02.202107270929test/NUCAPS-EDR_v2r0_m02_s202107270107170_e202107270119160_c202107270133220.nc -vv
[2021-10-15 15:31:58,283] : PID 76403 : INFO : nucaps2gtiff : main : Initializing reader...
[2021-10-15 15:31:58,296] : PID 76403 : DEBUG : satpy.scene : __init__ : Setting 'PPP_CONFIG_DIR' to '/home/qsw/program/p2g/polar2grid_v_2_3/etc/satpy'
[2021-10-15 15:31:58,301] : PID 76403 : DEBUG : satpy.readers : read_reader_config : Reading ['/home/qsw/program/p2g/polar2grid_v_2_3/lib/python3.7/site-packages/satpy/etc/readers/nucaps.yaml']
[2021-10-15 15:31:58,445] : PID 76403 : DEBUG : satpy.readers.yaml_reader : create_filehandlers : Assigning to nucaps: ['/mnt/hgfs/p2g_test/nucaps/m02.202107270929test/NUCAPS-EDR_v2r0_m02_s202107270107170_e202107270119160_c202107270133220.nc']
[2021-10-15 15:31:58,452] : PID 76403 : ERROR : nucaps2gtiff : exc_handler : Unexpected error. Enable debug messages (-vvv) or see log file for details.
[2021-10-15 15:31:58,452] : PID 76403 : DEBUG : nucaps2gtiff : exc_handler : Unexpected error exception:
Traceback (most recent call last):
File "/home/qsw/program/p2g/polar2grid_v_2_3/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/home/qsw/program/p2g/polar2grid_v_2_3/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/qsw/program/p2g/polar2grid_v_2_3/lib/python3.7/site-packages/polar2grid/glue_legacy.py", line 506, in <module>
sys.exit(main())
File "/home/qsw/program/p2g/polar2grid_v_2_3/lib/python3.7/site-packages/polar2grid/glue_legacy.py", line 285, in main
f = fcls(search_paths=args.data_files, **args.subgroup_args["Frontend Initialization"])
File "/home/qsw/program/p2g/polar2grid_v_2_3/lib/python3.7/site-packages/polar2grid/readers/nucaps.py", line 290, in __init__
super(Frontend, self).__init__(**kwargs)
File "/home/qsw/program/p2g/polar2grid_v_2_3/lib/python3.7/site-packages/polar2grid/readers/__init__.py", line 362, in __init__
self.scene = Scene(reader=self.reader, filenames=pathnames, reader_kwargs=kwargs)
File "/home/qsw/program/p2g/polar2grid_v_2_3/lib/python3.7/site-packages/satpy/scene.py", line 157, in __init__
reader_kwargs=reader_kwargs)
File "/home/qsw/program/p2g/polar2grid_v_2_3/lib/python3.7/site-packages/satpy/scene.py", line 202, in create_reader_instances
ppp_config_dir=self.ppp_config_dir)
File "/home/qsw/program/p2g/polar2grid_v_2_3/lib/python3.7/site-packages/satpy/readers/__init__.py", line 727, in load_readers
reader_instance.create_filehandlers(loadables, fh_kwargs=reader_kwargs_without_filter)
File "/home/qsw/program/p2g/polar2grid_v_2_3/lib/python3.7/site-packages/satpy/readers/yaml_reader.py", line 531, in create_filehandlers
fh_kwargs=fh_kwargs)
File "/home/qsw/program/p2g/polar2grid_v_2_3/lib/python3.7/site-packages/satpy/readers/yaml_reader.py", line 517, in new_filehandlers_for_filetype
return list(filtered_iter)
File "/home/qsw/program/p2g/polar2grid_v_2_3/lib/python3.7/site-packages/satpy/readers/yaml_reader.py", line 488, in filter_fh_by_metadata
for filehandler in filehandlers:
File "/home/qsw/program/p2g/polar2grid_v_2_3/lib/python3.7/site-packages/satpy/readers/yaml_reader.py", line 429, in new_filehandler_instances
yield filetype_cls(filename, filename_info, filetype_info, *req_fh, **fh_kwargs)
File "/home/qsw/program/p2g/polar2grid_v_2_3/lib/python3.7/site-packages/satpy/readers/nucaps.py", line 57, in __init__
super(NUCAPSFileHandler, self).__init__(*args, **kwargs)
TypeError: __init__() got an unexpected keyword argument 'mask_surface'
[2021-10-15 15:31:58,456] : PID 76403 : ERROR : traceback : exc_handler : __init__() got an unexpected keyword argument 'mask_surface'
Traceback (most recent call last):
File "/home/qsw/program/p2g/polar2grid_v_2_3/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/home/qsw/program/p2g/polar2grid_v_2_3/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/qsw/program/p2g/polar2grid_v_2_3/lib/python3.7/site-packages/polar2grid/glue_legacy.py", line 506, in <module>
sys.exit(main())
File "/home/qsw/program/p2g/polar2grid_v_2_3/lib/python3.7/site-packages/polar2grid/glue_legacy.py", line 285, in main
f = fcls(search_paths=args.data_files, **args.subgroup_args["Frontend Initialization"])
File "/home/qsw/program/p2g/polar2grid_v_2_3/lib/python3.7/site-packages/polar2grid/readers/nucaps.py", line 290, in __init__
super(Frontend, self).__init__(**kwargs)
File "/home/qsw/program/p2g/polar2grid_v_2_3/lib/python3.7/site-packages/polar2grid/readers/__init__.py", line 362, in __init__
self.scene = Scene(reader=self.reader, filenames=pathnames, reader_kwargs=kwargs)
File "/home/qsw/program/p2g/polar2grid_v_2_3/lib/python3.7/site-packages/satpy/scene.py", line 157, in __init__
reader_kwargs=reader_kwargs)
File "/home/qsw/program/p2g/polar2grid_v_2_3/lib/python3.7/site-packages/satpy/scene.py", line 202, in create_reader_instances
ppp_config_dir=self.ppp_config_dir)
File "/home/qsw/program/p2g/polar2grid_v_2_3/lib/python3.7/site-packages/satpy/readers/__init__.py", line 727, in load_readers
reader_instance.create_filehandlers(loadables, fh_kwargs=reader_kwargs_without_filter)
File "/home/qsw/program/p2g/polar2grid_v_2_3/lib/python3.7/site-packages/satpy/readers/yaml_reader.py", line 531, in create_filehandlers
fh_kwargs=fh_kwargs)
File "/home/qsw/program/p2g/polar2grid_v_2_3/lib/python3.7/site-packages/satpy/readers/yaml_reader.py", line 517, in new_filehandlers_for_filetype
return list(filtered_iter)
File "/home/qsw/program/p2g/polar2grid_v_2_3/lib/python3.7/site-packages/satpy/readers/yaml_reader.py", line 488, in filter_fh_by_metadata
for filehandler in filehandlers:
File "/home/qsw/program/p2g/polar2grid_v_2_3/lib/python3.7/site-packages/satpy/readers/yaml_reader.py", line 429, in new_filehandler_instances
yield filetype_cls(filename, filename_info, filetype_info, *req_fh, **fh_kwargs)
File "/home/qsw/program/p2g/polar2grid_v_2_3/lib/python3.7/site-packages/satpy/readers/nucaps.py", line 57, in __init__
super(NUCAPSFileHandler, self).__init__(*args, **kwargs)
TypeError: __init__() got an unexpected keyword argument 'mask_surface'
Maybe it's a problem with the data? If it is a problem of my data, where can I get the test data of nucaps and viirs_edr_active_fires.
Thanks.
Suwei
I installed polar2grid2.3 on centos7.8 and passed the test example. I have also tested several other readers, and they are running very well.
But when I use nucaps or viirs_edr_active_fires as the reader, I encountered some problems. I attached the error log below:
1. viirs_edr_active_fires:
The command I used is:
polar2grid.sh viirs_edr_active_fires gtiff -p confidence_pct T13 -f ../../active_fire/AFMOD_j01_d20210908_t0644280_e0645526_b19719_c20210909034222777833_cspp_dev.nc
result is:
INFO : Initializing reader...
INFO : Initializing remapping...
INFO : Initializing backend...
INFO : Initializing compositor objects...
INFO : Extracting swaths from data files available...
INFO : Writing longitude data to disk cache...
ERROR : Frontend data extraction failed (see log for details)
logfile is:
[2021-10-15 14:41:42,109] : PID 103198 : DEBUG : viirs_edr_active_fires2gtiff : main : Starting script with arguments: /home/qsw/program/p2g/polar2grid_v_2_3/lib/python3.7/site-packages/polar2grid/glue_legacy.py viirs_edr_active_fires gtiff -p confidence_pct T13 -f ../../active_fire/AFMOD_j01_d20210908_t0644280_e0645526_b19719_c20210909034222777833_cspp_dev.nc -vv
[2021-10-15 14:41:42,109] : PID 103198 : INFO : viirs_edr_active_fires2gtiff : main : Initializing reader...
[2021-10-15 14:41:42,110] : PID 103198 : DEBUG : satpy.scene : __init__ : Setting 'PPP_CONFIG_DIR' to '/home/qsw/program/p2g/polar2grid_v_2_3/etc/satpy'
[2021-10-15 14:41:42,112] : PID 103198 : DEBUG : satpy.readers : read_reader_config : Reading ['/home/qsw/program/p2g/polar2grid_v_2_3/lib/python3.7/site-packages/satpy/etc/readers/viirs_edr_active_fires.yaml']
[2021-10-15 14:41:42,360] : PID 103198 : DEBUG : satpy.readers.yaml_reader : create_filehandlers : Assigning to viirs_edr_active_fires: ['/mnt/hgfs/p2g_test/active_fire/AFMOD_j01_d20210908_t0644280_e0645526_b19719_c20210909034222777833_cspp_dev.nc']
[2021-10-15 14:41:42,382] : PID 103198 : DEBUG : satpy.composites : load_sensor_composites : Looking for composites config file viirs.yaml
[2021-10-15 14:41:42,515] : PID 103198 : DEBUG : satpy.composites : load_sensor_composites : Looking for composites config file visir.yaml
[2021-10-15 14:41:42,593] : PID 103198 : DEBUG : polar2grid.core.script_utils : rename_log_file : Log renamed from '/mnt/hgfs/p2g_test/output/active_fire/viirs_edr_active_fires2gtiff_fail.log' to 'viirs_edr_active_fires2gtiff_20210908_064428.log'
[2021-10-15 14:41:42,594] : PID 103198 : INFO : viirs_edr_active_fires2gtiff : main : Initializing remapping...
[2021-10-15 14:41:42,594] : PID 103198 : DEBUG : polar2grid.grids.manager : __init__ : Using default grid configuration: 'grids.conf'
[2021-10-15 14:41:42,594] : PID 103198 : DEBUG : polar2grid.grids.manager : __init__ : Loading grid configuration 'grids.conf'
[2021-10-15 14:41:42,685] : PID 103198 : DEBUG : polar2grid.grids.manager : parse_proj4_config_line : Converted grid 'dwd_germany' origin from (lon: -2.000000, lat: 56.000000) to (x: -755570.370463, y: -3554679.114710)
[2021-10-15 14:41:42,713] : PID 103198 : DEBUG : polar2grid.grids.manager : parse_proj4_config_line : Converted grid 'australia' origin from (lon: 105.000000, lat: 5.000000) to (x: -3896182.177765, y: 557305.257275)
[2021-10-15 14:41:42,715] : PID 103198 : DEBUG : polar2grid.grids.manager : parse_proj4_config_line : Converted grid 'australia2' origin from (lon: 105.000000, lat: 5.000000) to (x: -3896182.177765, y: 557305.257275)
[2021-10-15 14:41:42,717] : PID 103198 : DEBUG : polar2grid.grids.manager : parse_proj4_config_line : Converted grid '211e' origin from (lon: -123.044000, lat: 59.844000) to (x: -1950510.636800, y: 4368587.226913)
[2021-10-15 14:41:42,719] : PID 103198 : DEBUG : polar2grid.grids.manager : parse_proj4_config_line : Converted grid '211e_10km' origin from (lon: -123.044000, lat: 59.844000) to (x: -1950510.636800, y: 4368587.226913)
[2021-10-15 14:41:42,721] : PID 103198 : DEBUG : polar2grid.grids.manager : parse_proj4_config_line : Converted grid '211e_hi' origin from (lon: -123.044000, lat: 59.844000) to (x: -1950510.636800, y: 4368587.226913)
[2021-10-15 14:41:42,723] : PID 103198 : DEBUG : polar2grid.grids.manager : parse_proj4_config_line : Converted grid '211w' origin from (lon: -152.855000, lat: 54.536000) to (x: -4226045.730436, y: 4368634.765713)
[2021-10-15 14:41:42,724] : PID 103198 : DEBUG : polar2grid.grids.manager : parse_proj4_config_line : Converted grid '211w_10km' origin from (lon: -152.855000, lat: 54.536000) to (x: -4226045.730436, y: 4368634.765713)
[2021-10-15 14:41:42,726] : PID 103198 : DEBUG : polar2grid.grids.manager : parse_proj4_config_line : Converted grid '203' origin from (lon: 115.601000, lat: 44.646000) to (x: -4952960.856044, y: 381022.610378)
[2021-10-15 14:41:42,728] : PID 103198 : DEBUG : polar2grid.grids.manager : parse_proj4_config_line : Converted grid '203_10km' origin from (lon: 115.601000, lat: 44.646000) to (x: -4952960.856044, y: 381022.610378)
[2021-10-15 14:41:42,730] : PID 103198 : DEBUG : polar2grid.grids.manager : parse_proj4_config_line : Converted grid '204' origin from (lon: 110.000000, lat: 60.644000) to (x: -7314463.252793, y: 8020503.279488)
[2021-10-15 14:41:42,732] : PID 103198 : DEBUG : polar2grid.grids.manager : parse_proj4_config_line : Converted grid '204_10km' origin from (lon: 110.000000, lat: 60.644000) to (x: -7314463.252793, y: 8020503.279488)
[2021-10-15 14:41:42,734] : PID 103198 : DEBUG : polar2grid.grids.manager : parse_proj4_config_line : Converted grid '205' origin from (lon: -115.304000, lat: 36.257000) to (x: -4952924.911284, y: -3429056.260492)
[2021-10-15 14:41:42,735] : PID 103198 : DEBUG : polar2grid.grids.manager : parse_proj4_config_line : Converted grid '210' origin from (lon: -77.000000, lat: 26.422000) to (x: 0.000000, y: 2864298.891608)
[2021-10-15 14:41:42,736] : PID 103198 : INFO : viirs_edr_active_fires2gtiff : main : Initializing backend...
[2021-10-15 14:41:42,737] : PID 103198 : DEBUG : polar2grid.core.rescale : __init__ : Loading rescale configuration files:
polar2grid.core:rescale_configs/rescale.ini
[2021-10-15 14:41:42,739] : PID 103198 : DEBUG : polar2grid.core.roles : open_config_file : Loading package provided configuration file: 'polar2grid.core:rescale_configs/rescale.ini'
[2021-10-15 14:41:42,775] : PID 103198 : INFO : viirs_edr_active_fires2gtiff : main : Initializing compositor objects...
[2021-10-15 14:41:42,777] : PID 103198 : INFO : viirs_edr_active_fires2gtiff : main : Extracting swaths from data files available...
[2021-10-15 14:41:42,780] : PID 103198 : DEBUG : polar2grid.readers : create_scene : Loading scene data...
[2021-10-15 14:41:42,781] : PID 103198 : DEBUG : satpy.readers.yaml_reader : _load_dataset_area : No coordinates found for DatasetID(name='latitude', wavelength=None, resolution=None, polarization=None, calibration=None, level=None, modifiers=())
[2021-10-15 14:41:42,820] : PID 103198 : DEBUG : satpy.readers.yaml_reader : _load_dataset_area : No coordinates found for DatasetID(name='longitude', wavelength=None, resolution=None, polarization=None, calibration=None, level=None, modifiers=())
[2021-10-15 14:41:43,032] : PID 103198 : INFO : polar2grid.readers : area_to_swath_def : Writing longitude data to disk cache...
[2021-10-15 14:41:43,035] : PID 103198 : DEBUG : viirs_edr_active_fires2gtiff : main : Frontend data extraction exception:
Traceback (most recent call last):
File "/home/qsw/program/p2g/polar2grid_v_2_3/lib/python3.7/site-packages/polar2grid/glue_legacy.py", line 338, in main
scene = convert_satpy_to_p2g_swath(f, scene)
File "/home/qsw/program/p2g/polar2grid_v_2_3/lib/python3.7/site-packages/polar2grid/readers/__init__.py", line 317, in convert_satpy_to_p2g_swath
overwrite_existing=overwrite_existing)
File "/home/qsw/program/p2g/polar2grid_v_2_3/lib/python3.7/site-packages/polar2grid/readers/__init__.py", line 113, in area_to_swath_def
lon_arr = np.memmap(filename, mode="w+", dtype=lons.dtype, shape=lons.shape)
File "/home/qsw/program/p2g/polar2grid_v_2_3/lib/python3.7/site-packages/numpy/core/memmap.py", line 264, in __new__
mm = mmap.mmap(fid.fileno(), bytes, access=acc, offset=start)
ValueError: cannot mmap an empty file
[2021-10-15 14:41:43,037] : PID 103198 : ERROR : viirs_edr_active_fires2gtiff : main : Frontend data extraction failed (see log for details)
2. nucaps:
My data name is different from the data name on the nucaps website(https://www.ssec.wisc.edu/software/pola ... ucaps.html). The name format required on the website is: NUCAPS-EDR_v1r0_npp_s201603011158009_e201603011158307_c201603011222270.nc. But the data name format I got is NUCAPS-AGG-EDR_v2r0_m02_s202107270107170_e202107270119160_c2022107nc. And the data cannot be recognized by polar2gird. Then I changed the name of my data to NUCAPS-EDR_v2r0_m02_s202107270107170_e202107270119160_c202107270133220.nc. I tried to list products, but an error was reported. The result shows that there was no “mask surface” in the data.
The command I used is:
polar2grid.sh nucaps gtiff --list-products -f ../../nucaps/m02.202107270929test/NUCAPS-EDR_v2r0_m02_s202107270107170_e202107270119160_c202107270133220.nc
result is:
INFO : Initializing reader...
ERROR : Unexpected error. Enable debug messages (-vvv) or see log file for details.
logfile is:
[2021-10-15 15:31:58,282] : PID 76403 : DEBUG : nucaps2gtiff : main : Starting script with arguments: /home/qsw/program/p2g/polar2grid_v_2_3/lib/python3.7/site-packages/polar2grid/glue_legacy.py nucaps gtiff --list-products -f ../../nucaps/m02.202107270929test/NUCAPS-EDR_v2r0_m02_s202107270107170_e202107270119160_c202107270133220.nc -vv
[2021-10-15 15:31:58,283] : PID 76403 : INFO : nucaps2gtiff : main : Initializing reader...
[2021-10-15 15:31:58,296] : PID 76403 : DEBUG : satpy.scene : __init__ : Setting 'PPP_CONFIG_DIR' to '/home/qsw/program/p2g/polar2grid_v_2_3/etc/satpy'
[2021-10-15 15:31:58,301] : PID 76403 : DEBUG : satpy.readers : read_reader_config : Reading ['/home/qsw/program/p2g/polar2grid_v_2_3/lib/python3.7/site-packages/satpy/etc/readers/nucaps.yaml']
[2021-10-15 15:31:58,445] : PID 76403 : DEBUG : satpy.readers.yaml_reader : create_filehandlers : Assigning to nucaps: ['/mnt/hgfs/p2g_test/nucaps/m02.202107270929test/NUCAPS-EDR_v2r0_m02_s202107270107170_e202107270119160_c202107270133220.nc']
[2021-10-15 15:31:58,452] : PID 76403 : ERROR : nucaps2gtiff : exc_handler : Unexpected error. Enable debug messages (-vvv) or see log file for details.
[2021-10-15 15:31:58,452] : PID 76403 : DEBUG : nucaps2gtiff : exc_handler : Unexpected error exception:
Traceback (most recent call last):
File "/home/qsw/program/p2g/polar2grid_v_2_3/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/home/qsw/program/p2g/polar2grid_v_2_3/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/qsw/program/p2g/polar2grid_v_2_3/lib/python3.7/site-packages/polar2grid/glue_legacy.py", line 506, in <module>
sys.exit(main())
File "/home/qsw/program/p2g/polar2grid_v_2_3/lib/python3.7/site-packages/polar2grid/glue_legacy.py", line 285, in main
f = fcls(search_paths=args.data_files, **args.subgroup_args["Frontend Initialization"])
File "/home/qsw/program/p2g/polar2grid_v_2_3/lib/python3.7/site-packages/polar2grid/readers/nucaps.py", line 290, in __init__
super(Frontend, self).__init__(**kwargs)
File "/home/qsw/program/p2g/polar2grid_v_2_3/lib/python3.7/site-packages/polar2grid/readers/__init__.py", line 362, in __init__
self.scene = Scene(reader=self.reader, filenames=pathnames, reader_kwargs=kwargs)
File "/home/qsw/program/p2g/polar2grid_v_2_3/lib/python3.7/site-packages/satpy/scene.py", line 157, in __init__
reader_kwargs=reader_kwargs)
File "/home/qsw/program/p2g/polar2grid_v_2_3/lib/python3.7/site-packages/satpy/scene.py", line 202, in create_reader_instances
ppp_config_dir=self.ppp_config_dir)
File "/home/qsw/program/p2g/polar2grid_v_2_3/lib/python3.7/site-packages/satpy/readers/__init__.py", line 727, in load_readers
reader_instance.create_filehandlers(loadables, fh_kwargs=reader_kwargs_without_filter)
File "/home/qsw/program/p2g/polar2grid_v_2_3/lib/python3.7/site-packages/satpy/readers/yaml_reader.py", line 531, in create_filehandlers
fh_kwargs=fh_kwargs)
File "/home/qsw/program/p2g/polar2grid_v_2_3/lib/python3.7/site-packages/satpy/readers/yaml_reader.py", line 517, in new_filehandlers_for_filetype
return list(filtered_iter)
File "/home/qsw/program/p2g/polar2grid_v_2_3/lib/python3.7/site-packages/satpy/readers/yaml_reader.py", line 488, in filter_fh_by_metadata
for filehandler in filehandlers:
File "/home/qsw/program/p2g/polar2grid_v_2_3/lib/python3.7/site-packages/satpy/readers/yaml_reader.py", line 429, in new_filehandler_instances
yield filetype_cls(filename, filename_info, filetype_info, *req_fh, **fh_kwargs)
File "/home/qsw/program/p2g/polar2grid_v_2_3/lib/python3.7/site-packages/satpy/readers/nucaps.py", line 57, in __init__
super(NUCAPSFileHandler, self).__init__(*args, **kwargs)
TypeError: __init__() got an unexpected keyword argument 'mask_surface'
[2021-10-15 15:31:58,456] : PID 76403 : ERROR : traceback : exc_handler : __init__() got an unexpected keyword argument 'mask_surface'
Traceback (most recent call last):
File "/home/qsw/program/p2g/polar2grid_v_2_3/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/home/qsw/program/p2g/polar2grid_v_2_3/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/qsw/program/p2g/polar2grid_v_2_3/lib/python3.7/site-packages/polar2grid/glue_legacy.py", line 506, in <module>
sys.exit(main())
File "/home/qsw/program/p2g/polar2grid_v_2_3/lib/python3.7/site-packages/polar2grid/glue_legacy.py", line 285, in main
f = fcls(search_paths=args.data_files, **args.subgroup_args["Frontend Initialization"])
File "/home/qsw/program/p2g/polar2grid_v_2_3/lib/python3.7/site-packages/polar2grid/readers/nucaps.py", line 290, in __init__
super(Frontend, self).__init__(**kwargs)
File "/home/qsw/program/p2g/polar2grid_v_2_3/lib/python3.7/site-packages/polar2grid/readers/__init__.py", line 362, in __init__
self.scene = Scene(reader=self.reader, filenames=pathnames, reader_kwargs=kwargs)
File "/home/qsw/program/p2g/polar2grid_v_2_3/lib/python3.7/site-packages/satpy/scene.py", line 157, in __init__
reader_kwargs=reader_kwargs)
File "/home/qsw/program/p2g/polar2grid_v_2_3/lib/python3.7/site-packages/satpy/scene.py", line 202, in create_reader_instances
ppp_config_dir=self.ppp_config_dir)
File "/home/qsw/program/p2g/polar2grid_v_2_3/lib/python3.7/site-packages/satpy/readers/__init__.py", line 727, in load_readers
reader_instance.create_filehandlers(loadables, fh_kwargs=reader_kwargs_without_filter)
File "/home/qsw/program/p2g/polar2grid_v_2_3/lib/python3.7/site-packages/satpy/readers/yaml_reader.py", line 531, in create_filehandlers
fh_kwargs=fh_kwargs)
File "/home/qsw/program/p2g/polar2grid_v_2_3/lib/python3.7/site-packages/satpy/readers/yaml_reader.py", line 517, in new_filehandlers_for_filetype
return list(filtered_iter)
File "/home/qsw/program/p2g/polar2grid_v_2_3/lib/python3.7/site-packages/satpy/readers/yaml_reader.py", line 488, in filter_fh_by_metadata
for filehandler in filehandlers:
File "/home/qsw/program/p2g/polar2grid_v_2_3/lib/python3.7/site-packages/satpy/readers/yaml_reader.py", line 429, in new_filehandler_instances
yield filetype_cls(filename, filename_info, filetype_info, *req_fh, **fh_kwargs)
File "/home/qsw/program/p2g/polar2grid_v_2_3/lib/python3.7/site-packages/satpy/readers/nucaps.py", line 57, in __init__
super(NUCAPSFileHandler, self).__init__(*args, **kwargs)
TypeError: __init__() got an unexpected keyword argument 'mask_surface'
Maybe it's a problem with the data? If it is a problem of my data, where can I get the test data of nucaps and viirs_edr_active_fires.
Thanks.
Suwei