Page 1 of 1

Problem with processing l1b Metop data

Posted: Tue Jun 22, 2021 1:57 am
by zion0210
Hi colleagues.

I have a problem with processing AMSU/MHS l1b data.
I used MIRS_v2.4 and l1b NOAA data (https://www.avl.class.noaa.gov/).

For example, I run the following command:

Code: Select all

CSPP_MIRS_2_4//scripts/run_mirs.bash -x --noSFR -d /home/terra/NPP_SOFT//MIRS_v2.4/CSPP_MIRS_2_4//data/dynamic_ancillary/ -w /home/terra/NPP_SOFT//WORK/ -s metopB -p 8 -i /home/terra/ATMS_NPP/metop_test/ -vvv
The inputs files are NSS.AMAX.M1.D21166.S0316.E0405.B4535555.MM and NSS.MHSX.M1.D21166.S0316.E0405.B4535555.MM .

And receive the following log:

Code: Select all

DEBUG %options
==============
'dynanc' => '/NFS/snpp2-4/ATMS_GFS/dynamic_ancillary',
'sat' => 'metopB',
'block' => 2,
'input_files' => '/home/terra/ATMS_NPP/metop_test/',
'xfiles' => 1,
'clean' => 1,
'verbose' => 3,
'processors' => 8,
'nothing' => 0,
'SFR' => 0,
'quarantine' => 0,
'logs' => 1,
'yikes' => 0,
'aggregate' => 1,
'resolution' => 'HI',
'work' => '/HDDS/HDD1/public/NPP_SOFT/WORK'
DEBUG %options _END_
---------------
DEBUG %filemask
===============
'metopA' => '*.{AMAX,MHSX}.M2*',
'metopB' => '*.{AMAX,MHSX}.M1*',
'metopC' => '*.{AMAX,MHSX}.M3*',
'n19' => '*.{AMAX,MHSX}.NP*',
'n21' => '{GATMO,SATMS,TATMS}_j02*.h5',
'npp' => '{GATMO,SATMS,TATMS}_npp*.h5',
'n18' => '*.{AMAX,MHSX}.NN*',
'n20' => '{GATMO,SATMS,TATMS}_j01*.h5'
DEBUG %filemask _END_
-----------------
DEBUG %dynanc_loc
=================
'GFSPROTO' => 'https://',
'GFSLOCAL' => '/NFS/snpp2-4/ATMS_GFS/dynamic_ancillary/%Y_%m_%d_%j',
'GFSSITE' => 'jpssdb.ssec.wisc.edu',
'GFSREMOTE' => 'cspp_v_2_0/ancillary/%Y_%m_%d_%j',
'GFSTEMPLATE' => 'gfs.t%Hz.pgrb2.0p50.f0%f.%Y%m%d'
DEBUG %dynanc_loc _END_
chdir to working directory /HDDS/HDD1/public/NPP_SOFT/WORK/TMP_Bzlt_K
PCF file is:  /HDDS/HDD1/public/NPP_SOFT/WORK/TMP_Bzlt_K/cspp_metopB_pcf.bash
SCS file is:  /HDDS/HDD1/public/NPP_SOFT/WORK/TMP_Bzlt_K/cspp_metopB_scs.bash
run_mirs.pl::make_L1B_bigEndian: /home/terra/ATMS_NPP/metop_test/NSS.AMAX.M1.D21166.S0316.E0405.B4535555.MM endian indeterminate
Abort at /home/terra/NPP_SOFT//MIRS_v2.4/CSPP_MIRS_2_4//scripts/run_mirs.pl line 1014.
I also receive identical error for old version MIRS.

Regards,
zion0210

Re: Problem with processing l1b Metop data

Posted: Tue Jun 22, 2021 11:21 am
by jimd
The run_mirs.pl script tries to determine whether the input file is big- or little-endian by decoding the header record both ways and checking if the decoded year makes sense. I am wondering if your file has no header record - which would be a problem, not just for endian-ness but, as the code is presently written, also for a filename extracted from the header record.
If you could please send me the two files (AMAX and MHSX) I can take a look. Best way might be to upload to ftp://ftp.ssec.wisc.edu/pub/incoming/ and then drop me a line.
Was there an option on CLASS to get files with/without header? I would have tested CSPP_MIRS with CLASS data, but perhaps something has changed since I did so.
Cheers,
-jim.

Re: Problem with processing l1b Metop data

Posted: Tue Jun 22, 2021 11:58 pm
by zion0210
Hi Jim.

Files NSS.AMAX.M1.D21166.S0316.E0405.B4535555.MM and NSS.MHSX.M1.D21166.S0316.E0405.B4535555.MM located at ftp://ftp.ssec.wisc.edu/pub/incoming/

I used the following form getting l1b metop data
https://www.avl.class.noaa.gov/saa/prod ... submit.y=6

Regards,
zion0210

Re: Problem with processing l1b Metop data

Posted: Wed Jun 23, 2021 1:08 pm
by kathys
Hello zion0210,

We have investigated your data and we believe we have a solution.

There is header information that needs to be included in the data that you order from CLASS. To include the header information, you need to follow these instructions:

1). On the CLASS website, along the Left side options, under "User Account", choose "Preferences". This brings you to a "User Preferences" page.

2). Select "Yes" for the Delivery Preference: "Prepend TBM/ARS headers to my data sets"

3). Then scroll to the bottom of the page and select the button "Update".

Once you follow these steps, the data that you order from CLASS will include the header information that you need to execute the MiRS software.

I downloaded the same files that you are working on using this preference, and we were able to process the data using the CSPP MiRS software. I put those new CLASS input files here: https://bin.ssec.wisc.edu/pub/CSPP/test/

Good luck,

Kathy