Unable to process certain POES AVHRR satellite data

Post Reply
terry123
Posts: 2
Joined: Sat Dec 25, 2021 8:13 pm

Unable to process certain POES AVHRR satellite data

Post by terry123 »

Hello,

My current workflow involves:
- Downloading data from NOAA CLASS
- Processing to AAPP with 'noaa_class_to_aapp' script
- Creating gtiff with p2g

As for AAPP data from NOAA-11 to 14 p2g seems unable to process and print error message. However NOAA-6 to 10 and NOAA-15 to 19 data works just fine.

Here's the log for NOAA-14:

Code: Select all

[2021-12-26 11:57:59,625] : PID  25416 : DEBUG    : avhrr2gtiff : main : Starting script with arguments: /mnt/dt3/geo2grid_v_1_0_2/lib/python3.7/site-packages/polar2grid/glue_legacy.py avhrr gtiff -f ./NSS.HRPT.NJ.D96255.S1816.E1830.B0876666.WI.l1b -vv
[2021-12-26 11:57:59,625] : PID  25416 : INFO     : avhrr2gtiff : main : Initializing reader...
[2021-12-26 11:57:59,625] : PID  25416 : DEBUG    : polar2grid.avhrr.avhrr2swath : __init__ : Day fraction set to 0.100000
[2021-12-26 11:57:59,625] : PID  25416 : DEBUG    : polar2grid.avhrr.avhrr2swath : __init__ : SZA threshold set to 100.000000
[2021-12-26 11:57:59,625] : PID  25416 : DEBUG    : polar2grid.avhrr.avhrr2swath : load_files : Recognize file /mnt/dt3/avhrr/poes/8151553095/001/NSS.HRPT.NJ.D96255.S1816.E1830.B0876666.WI.l1b as file type FT_AAPP
[2021-12-26 11:57:59,626] : PID  25416 : ERROR    : polar2grid.avhrr.readers : __init__ : Could not parse basic information from AVHRR file: /mnt/dt3/avhrr/poes/8151553095/001/NSS.HRPT.NJ.D96255.S1816.E1830.B0876666.WI.l1b
[2021-12-26 11:57:59,626] : PID  25416 : DEBUG    : avhrr2gtiff : main : Frontend exception: 
Traceback (most recent call last):
  File "/mnt/dt3/geo2grid_v_1_0_2/lib/python3.7/site-packages/polar2grid/glue_legacy.py", line 285, in main
    f = fcls(search_paths=args.data_files, **args.subgroup_args["Frontend Initialization"])
  File "/mnt/dt3/geo2grid_v_1_0_2/lib/python3.7/site-packages/polar2grid/avhrr/avhrr2swath.py", line 109, in __init__
    self.load_files(self.find_files_with_extensions())
  File "/mnt/dt3/geo2grid_v_1_0_2/lib/python3.7/site-packages/polar2grid/avhrr/avhrr2swath.py", line 133, in load_files
    self.file_readers[h.file_type].add_file(h)
  File "/mnt/dt3/geo2grid_v_1_0_2/lib/python3.7/site-packages/polar2grid/core/frontend_utils.py", line 501, in add_file
    self.file_readers.append(self.single_class(fn, self.file_type_info))
  File "/mnt/dt3/geo2grid_v_1_0_2/lib/python3.7/site-packages/polar2grid/avhrr/readers.py", line 527, in __init__
    self.satellite = self.SAT_ID_MAP[int(self.file_handle["satid"][0])]
KeyError: 3
[2021-12-26 11:57:59,627] : PID  25416 : ERROR    : avhrr2gtiff : main : avhrr frontend failed to load and sort data files (see log for details)
Thanks for any helps.
davidh
Posts: 116
Joined: Tue Jun 04, 2013 11:19 am

Re: Unable to process certain POES AVHRR satellite data

Post by davidh »

Sorry you're having trouble with this. The reader is based on the AVHRR AAPP reader provided by the Pytroll Satpy Python library. This library uses the "satid" information provided in the file content to determine what platform the data came from. The currently supported platforms are limited to the following IDs which are mapped to the associated platform name:

Code: Select all

AVHRR_PLATFORM_IDS2NAMES = {4: 'NOAA-15',
                            2: 'NOAA-16',
                            6: 'NOAA-17',
                            7: 'NOAA-18',
                            8: 'NOAA-19',
                            11: 'Metop-B',
                            12: 'Metop-A',
                            13: 'Metop-C',
                            14: 'Metop simulator'}
So NOAA-15 to NOAA-19 are supported, but not 6 to 14. I'm confused how your NOAA-6 to NOAA-10 data is working at all. As the error message suggests, your NOAA-14 file contains a satellite ID of 3. I'm not really familiar with the history of these platforms and the AAPP format so I will have to ask around to gather more information. My basic "excuse" answer for now is that pre-NOAA-15 satellites are not supported by this reader.
terry123
Posts: 2
Joined: Sat Dec 25, 2021 8:13 pm

Re: Unable to process certain POES AVHRR satellite data

Post by terry123 »

Hello

I've gone back to run through some datasets and generated these log files, maybe you'll find them useful

https://mega.nz/file/A3gXXK6R#Xw_luBoZw ... VAN1pOhZCw
davidh
Posts: 116
Joined: Tue Jun 04, 2013 11:19 am

Re: Unable to process certain POES AVHRR satellite data

Post by davidh »

Based on the filenames that are being produced it doesn't seem to be understanding the satellite ID properly or the utility you are using is producing the wrong ID. The NOAA-6 log shows NOAA-16 in the filename, while the NOAA-7 log shows NOAA-15. Either there is a different way we should be determining the satellite of the data (other metadata fields in the file) or the utility you are using is "cheating" by making the data seem like it is some other satellite. I will ask around the Pytroll/Satpy community to see if anyone knows more.
Post Reply