Hello,
Could anyone please tell us what the following messages from ADL mean?
We have been struggling all afternoon with ADL 3.0, ADL4.0, ADL4.1 in efforts once again to tempt fate and run but a single meager granule using data straight out of GRAVITE obtained using the 'gtp' tool. The file sizes look appropriate and other hdf5-compatible tools work just fine with the input data. Alas, we keep running up against this same error of below. Some sort of anticipated versus real data product file size DMS(?) gibberish.
Unfortunately, we have exhausted everything worth trying we know to do w/o seeking help, & this is pretty much out of our hands whether it works. I wish I had written the converter that generated the blob inputs from the *.h5 files. It cannot seem to produce the data in the correct format to satisfy the ADL ("Granulate" QST LWM step, for subsequent Active Fires -- see adjacent Thread Topic 'VIIRS AF algrthm - how to run just fires, not entire PRO chn ' for full explication if this isn't clear) EDR code. W/o properly formatted inputs of course we cannot run said code, yet keep trying to use the puzzle pieces here that don't quite fit together as designed??
Sample output "ADL_Unpacker.exe" output:
...
ProCmnDataItem[13ProCmnSDRItem]::setDataVectorExisting(VIIRS-MOD-GEO-TC) [0x66f2ef0] ROOT PRO_FAIL wrong file size: mapped file size 81103832 != expected size 81103784 from file ProCmnDataItem.cpp, line 2719
ProCmnAlgorithm[ProGipViirsWCalcGran]::handleInputQuery(NPP000318203577) [0x66d9710] ROOT PRO_FAIL Required input not available for Shortname: VIIRS-MOD-RGEO-TC , Granule ID: NPP000318203577 Failure(s): (VIIRS-MOD-RGEO-TC, NPP000318203577), (VIIRS-MOD-GEO-TC, NPP000318203577) from file ProCmnAlgorithm.cpp, line 7880
ProCmnDataItem[13ProCmnSDRItem]::setDataVectorExisting(VIIRS-MOD-GEO-TC) [0x70b4dc0] ROOT PRO_FAIL wrong file size: mapped file size 81103832 != expected size 81103784 from file ProCmnDataItem.cpp, line 2719
ProCmnAlgorithm[ProGipViirsWCalcGran]::handleInputQuery(NPP000318204430) [0x709b5e0] ROOT PRO_FAIL Required input not available for Shortname: VIIRS-MOD-RGEO-TC , Granule ID: NPP000318204430 Failure(s): (VIIRS-MOD-RGEO-TC, NPP000318204430), (VIIRS-MOD-GEO-TC, NPP000318204430) from file ProCmnAlgorithm.cpp, line 7880
ProCmnDataItem[13ProCmnSDRItem]::setDataVectorExisting(VIIRS-MOD-GEO-TC) [0x7b03dc0] ROOT PRO_FAIL wrong file size: mapped file size 81103832 != expected size 81103784 from file ProCmnDataItem.cpp, line 2719
ProCmnAlgorithm[ProGipViirsWCalcGran]::handleInputQuery(NPP000318205284) [0x7aea5e0] ROOT PRO_FAIL Required input not available for Shortname: VIIRS-MOD-RGEO-TC , Granule ID: NPP000318205284 Failure(s): (VIIRS-MOD-RGEO-TC, NPP000318205284), (VIIRS-MOD-GEO-TC, NPP000318205284) from file ProCmnAlgorithm.cpp, line 7880
ProCmnAlgorithm[ProEdrViirsActiveFires]::handleInputQuery(NPP000318204430) [0x29e9e80] ROOT PRO_FAIL Required input not available for Shortname: VIIRS-GridIP-VIIRS-Qst-Lwm-Mod-Gran , Granule ID: NPP000318204430 Failure(s): (VIIRS-GridIP-VIIRS-Qst-Lwm-Mod-Gran, NPP000318204430) from file ProCmnAlgorithm.cpp, line 7880
Data Staging Issue 'mapped file size != expected size'
Re: Data Staging Issue 'mapped file size != expected size'
The error messages are denoting an inconsistency in the sizes of the geolocation inputs.
wrong file size: mapped file size 81103832 != expected size 81103784
In Mx6.2, the VIIRS gelocation products changed format to add an additional scan quality flag, which in turn changed or grew the size of the geolocation products by 48 bytes. One byte for each scan. The ADL framework does input size consistency checks to ensure that the format that the software has and the input file match, in your case they don't.
I believe that you are trying to run post Mx6.2 produced geolocation data into ADL 4.0 (Mx6.0) or ADL 3.1 in order to get this size mismatch. Where it looks like your file on disk is the larger size (newer format with the additional 48 byte scan field), but your software you are trying to run is an ADL version prior to Mx6.2. In order to get in sync you will need to run ADL 4.1 (sync'ed with Mx6.3) or and ADL 4.0 sync'ed version with Mx6.3 that can be found on the Common CM portal.
ADL 4.1 will also automatically run the granulate ancillary and GridToGran algorithms if the inputs are missing in the EDR/IP algorithms, such that you won't have to go back and run VIIRS SDR.
I hope this is enough information to help you resolve the problem, but if not post back and I'll respond further.
Thanks,
Bryan Henderson
Raytheon Company
wrong file size: mapped file size 81103832 != expected size 81103784
In Mx6.2, the VIIRS gelocation products changed format to add an additional scan quality flag, which in turn changed or grew the size of the geolocation products by 48 bytes. One byte for each scan. The ADL framework does input size consistency checks to ensure that the format that the software has and the input file match, in your case they don't.
I believe that you are trying to run post Mx6.2 produced geolocation data into ADL 4.0 (Mx6.0) or ADL 3.1 in order to get this size mismatch. Where it looks like your file on disk is the larger size (newer format with the additional 48 byte scan field), but your software you are trying to run is an ADL version prior to Mx6.2. In order to get in sync you will need to run ADL 4.1 (sync'ed with Mx6.3) or and ADL 4.0 sync'ed version with Mx6.3 that can be found on the Common CM portal.
ADL 4.1 will also automatically run the granulate ancillary and GridToGran algorithms if the inputs are missing in the EDR/IP algorithms, such that you won't have to go back and run VIIRS SDR.
I hope this is enough information to help you resolve the problem, but if not post back and I'll respond further.
Thanks,
Bryan Henderson
Raytheon Company
Re: Data Staging Issue 'mapped file size != expected size'
Brian, first thanks for your quick reply.
I am using "ADL 4.0 IDPS Mx6.3 (Early Release) Algorithms for ADL 4.0" per a detailed five page pdf file that Paul Meade circulated on October 1, e-mail Subject: "ADL 4.0 with Mx6.2 or M6.3 (pre-release) algorithms." We followed all of the instructions and, additionally, customized for running our active fires EDR the new $ADL_HOME tree per the other thread that I referenced 'VIIRS AF algrthm - how to run just fires, not entire PRO chn'. When I have time, I will have to look at the new GridToGrans time-saver you mentioned. In the meantime, we thought we had a recipe worked out per that thread (what we are really interested in is our own particular science EDR) and right now there appear to be much bigger fish to fry once again just getting anything in the present data stream to work as before.
It seems like geolocation should have been compatible with this early release patch, or the patch itself doesn't make much sense.
In light of this is there an easy solution to this sizes issue besides going to the trouble of downloading another huge system in the complete 4.1 VM kitchen sink?
I am clocking how long all of this activity is requiring, and in these past couple of months 95% of more than a week's worth of effort re. ADL has been spent merely updating the software to try to bring our installation up to being Mx6.3 compliant. This is even using the virtual machines.
We still haven't gotten back to the point where we can simply run the EDR without updating yet again as it sounds like you are suggesting. That will take now yet another several hours when we factor in downloading, re-installing, reconfiguring, and customizing for our particular little EDR yet another virtual machine, the 4.1, which I see has just now appeared***. We now have three (3.1,4.0, 4.0 "patched") what sound to be essentially useless(for forward processing) installations laying around already. So you can see why we are hesitant about going over to yet another.
Will the 4.1 be backwards compatible with the older geolocation? Or should we keep 4.0 at least laying around. It bears asking because particularly the multiple VMs consume a lot of disk and resources.
[UPDATE: This question appears to have been asked in an adjacent thread with that poster's mileage being that backwards compability with the older pre-Mx6.3 geolocation was _not_ preserved in ADL 4.1.]
Patching-in and rebuilding (detailed instructions required if you can, pls.) with the one or two files, whichever they are, necessary to clear this file size check hurdle would be a much, much better way to work if that is at all possible. I wouldn't say this if updating to the new version was turn-key, but it most definitely is not.
Please let us know what would be the best way to proceed. Thanks again so much for your time.
Brad Wind
Active Fires Group
*** Request for the SSEC Site: It would be helpful if out on the downloads page there were timestamps indicating when each of the various downloads appeared so that The Community had some sense of the 'freshness' of each link what with the pace that everything seems to be changing at present.
It may also be that I am just not on a particular mailing list, but is there a notification list to which one can subscribe that will e-mail when the new downloads (especially major releases) become available? It is too difficult to monitor the website manually with any real-time frequency for this kind of information and we keep finding that we are only vaguely aware of the sudden appearance of new download links. Thanks kindly again/ahead.
This is good to know. You are correct that we are looking at an Mx6.3 granule for a problem scene for our active fires EDR. I suspect what you are saying is correct. Unfortunately, ADL (pre-4.1 at least), does not seem to be capable of dealing with this new feature of the geolocation product. Please read on.bhenders wrote:The error messages are denoting an inconsistency in the sizes of the geolocation inputs.
wrong file size: mapped file size 81103832 != expected size 81103784
In Mx6.2, the VIIRS gelocation products changed format to add an additional scan quality flag, which in turn changed or
grew the size of the geolocation products by 48 bytes. One byte for each scan. The ADL framework does input size consistency checks to ensure that the format that the software has and the input file match, in your case they don't.
No. Not exactly.bhenders wrote: I believe that you are trying to run post Mx6.2 produced geolocation data into ADL 4.0 (Mx6.0) or ADL 3.1 in order to get this size mismatch.
I am using "ADL 4.0 IDPS Mx6.3 (Early Release) Algorithms for ADL 4.0" per a detailed five page pdf file that Paul Meade circulated on October 1, e-mail Subject: "ADL 4.0 with Mx6.2 or M6.3 (pre-release) algorithms." We followed all of the instructions and, additionally, customized for running our active fires EDR the new $ADL_HOME tree per the other thread that I referenced 'VIIRS AF algrthm - how to run just fires, not entire PRO chn'. When I have time, I will have to look at the new GridToGrans time-saver you mentioned. In the meantime, we thought we had a recipe worked out per that thread (what we are really interested in is our own particular science EDR) and right now there appear to be much bigger fish to fry once again just getting anything in the present data stream to work as before.
It seems like geolocation should have been compatible with this early release patch, or the patch itself doesn't make much sense.
As I recall from a month ago back when we did the update, I had to get the actual "ADL 4.0 sync'ed version with Mx6.3" source code from Common CM. At the time at least, and I confess I haven't checked since, it was not available on the SSEC site as the pdf had suggested. So we did go to Common CM. Has this code been tested on any EDRs with actual Mx6.3 data? I ask b/c most (all?) of the EDRs use Geolocation. Like I say, ours doesn't seem to work b/c of the size difference that you note, and I have tried in the Mx6.3 patched 4.0 and gotten the same errors as in my original post as recently as this morning once again.bhenders wrote: Where it looks like your file on disk is the larger size (newer format with the additional 48 byte scan field), but your software you are trying to run is an ADL version prior to Mx6.2. In order to get in sync you will need to run ADL 4.1 (sync'ed with Mx6.3) or and ADL 4.0 sync'ed version with Mx6.3 that can be found on the Common CM portal.
In light of this is there an easy solution to this sizes issue besides going to the trouble of downloading another huge system in the complete 4.1 VM kitchen sink?
I am clocking how long all of this activity is requiring, and in these past couple of months 95% of more than a week's worth of effort re. ADL has been spent merely updating the software to try to bring our installation up to being Mx6.3 compliant. This is even using the virtual machines.
We still haven't gotten back to the point where we can simply run the EDR without updating yet again as it sounds like you are suggesting. That will take now yet another several hours when we factor in downloading, re-installing, reconfiguring, and customizing for our particular little EDR yet another virtual machine, the 4.1, which I see has just now appeared***. We now have three (3.1,4.0, 4.0 "patched") what sound to be essentially useless(for forward processing) installations laying around already. So you can see why we are hesitant about going over to yet another.
Will the 4.1 be backwards compatible with the older geolocation? Or should we keep 4.0 at least laying around. It bears asking because particularly the multiple VMs consume a lot of disk and resources.
[UPDATE: This question appears to have been asked in an adjacent thread with that poster's mileage being that backwards compability with the older pre-Mx6.3 geolocation was _not_ preserved in ADL 4.1.]
Patching-in and rebuilding (detailed instructions required if you can, pls.) with the one or two files, whichever they are, necessary to clear this file size check hurdle would be a much, much better way to work if that is at all possible. I wouldn't say this if updating to the new version was turn-key, but it most definitely is not.
Please let us know what would be the best way to proceed. Thanks again so much for your time.
Brad Wind
Active Fires Group
*** Request for the SSEC Site: It would be helpful if out on the downloads page there were timestamps indicating when each of the various downloads appeared so that The Community had some sense of the 'freshness' of each link what with the pace that everything seems to be changing at present.
It may also be that I am just not on a particular mailing list, but is there a notification list to which one can subscribe that will e-mail when the new downloads (especially major releases) become available? It is too difficult to monitor the website manually with any real-time frequency for this kind of information and we keep finding that we are only vaguely aware of the sudden appearance of new download links. Thanks kindly again/ahead.
I hope this is enough information to help you resolve the problem, but if not post back and I'll respond further.
Thanks,
Bryan Henderson
Raytheon Company
Last edited by Bwind on Wed Nov 21, 2012 4:11 pm, edited 1 time in total.
Re: Data Staging Issue 'mapped file size != expected size'
Brad,
Thank you for your detailed post. I've confirmed that the geolocation format change went into Mx6.3. An ADL 4.0 IDPS Mx6.3 (Early Release) Algorithms for ADL 4.0 should be compatible with Mx6.3 operational data. However, based on your mismatch error you seem to have the older version of the geolocation XML files built in the ADL version you are running.
I have a developer that is writing a script that he said would take about an hour to write, that will convert geolocation binary files from one format to the other and vice versa. I can understand the desire for not wanting to roll forward in ADL versions and the desire to potentially run old data through newer versions of ADL. The only solution I have for you is to manipulate the geolocation products you do have to be the correct format or version for the ADL version you are trying to execute. In this case, the additional field of the 48 bytes can be removed from the binary file, but it requires the manipulation of the binary file.
As soon as the tool is functional, I'll post it in a follup posting. I have not seen Paul Meades five page pdf file, but I'll try to confirm that ADL 4.0 and the Mx6.3 source tar file's from Common CM does indeed have the correct geolocation format in it for Mx6.3.
I think converting the geolocation products with this new script that is being written is going to be the easiest solution for you at the current time.
Bryan Henderson
Raytheon Company
Thank you for your detailed post. I've confirmed that the geolocation format change went into Mx6.3. An ADL 4.0 IDPS Mx6.3 (Early Release) Algorithms for ADL 4.0 should be compatible with Mx6.3 operational data. However, based on your mismatch error you seem to have the older version of the geolocation XML files built in the ADL version you are running.
I have a developer that is writing a script that he said would take about an hour to write, that will convert geolocation binary files from one format to the other and vice versa. I can understand the desire for not wanting to roll forward in ADL versions and the desire to potentially run old data through newer versions of ADL. The only solution I have for you is to manipulate the geolocation products you do have to be the correct format or version for the ADL version you are trying to execute. In this case, the additional field of the 48 bytes can be removed from the binary file, but it requires the manipulation of the binary file.
As soon as the tool is functional, I'll post it in a follup posting. I have not seen Paul Meades five page pdf file, but I'll try to confirm that ADL 4.0 and the Mx6.3 source tar file's from Common CM does indeed have the correct geolocation format in it for Mx6.3.
I think converting the geolocation products with this new script that is being written is going to be the easiest solution for you at the current time.
Bryan Henderson
Raytheon Company
Re: Data Staging Issue 'mapped file size != expected size'
Brad,
I've uploaded the script that was written to convert the geolocation products from one format to another.
Since it manipulates the binary files, you may want to make a back up copy of geolocation files before using it.
bhenders-> ./cnvtGEO.csh
Converts GEO products from Mx6.2 format to Mx6.3 and vice versa.
Converting to 6.3 inserts a zero filled ScanQuality2 Field
Converting to 6.2 removes ScanQuality2 Field
Usage: ./cnvtGEO.csh fileName [replaceFile]
fileName: Must be a fully qualified path and filename to be converted
replaceFile: Any non blank value will cause the input file to be replaced
by the converted file.
where you can execute it as follows:
./cnvtGEO.csh <filename> -r
to convert a geolocation file in place.
In order to upload the file to this forum, I had to rename it as cnvtGEO.txt, so download and rename as cnvtGEO.csh
Bryan Henderson
Raytheon Company
I've uploaded the script that was written to convert the geolocation products from one format to another.
Since it manipulates the binary files, you may want to make a back up copy of geolocation files before using it.
bhenders-> ./cnvtGEO.csh
Converts GEO products from Mx6.2 format to Mx6.3 and vice versa.
Converting to 6.3 inserts a zero filled ScanQuality2 Field
Converting to 6.2 removes ScanQuality2 Field
Usage: ./cnvtGEO.csh fileName [replaceFile]
fileName: Must be a fully qualified path and filename to be converted
replaceFile: Any non blank value will cause the input file to be replaced
by the converted file.
where you can execute it as follows:
./cnvtGEO.csh <filename> -r
to convert a geolocation file in place.
In order to upload the file to this forum, I had to rename it as cnvtGEO.txt, so download and rename as cnvtGEO.csh
Bryan Henderson
Raytheon Company
- Attachments
-
[The extension txt has been deactivated and can no longer be displayed.]
Re: Data Staging Issue 'mapped file size != expected size'
Hi we try the conversion script on some Geolocation data like this one,
ftp://dds.gina.alaska.edu/public/NPP/vi ... spp_dev.h5
What I am doing is just run something like this,
./cnvtGEO_UAF.sh GMTCO_npp_d20130606_t1548531_e1550157_b00001_c20130606161732831673_cspp_dev.h5 GMTCO_npp_d20130606_t1548531_e1550157_b00001_c20130606161732831673_cspp_dev.rem.h5
And it turns out
"Skipping unknown conversion"
I checked the code and found that only a few specific in-file-sizes are accepted by the script. Does that imply that I provided the wrong inputs? If so, What should I provide then?
Thanks in advance!
ftp://dds.gina.alaska.edu/public/NPP/vi ... spp_dev.h5
What I am doing is just run something like this,
./cnvtGEO_UAF.sh GMTCO_npp_d20130606_t1548531_e1550157_b00001_c20130606161732831673_cspp_dev.h5 GMTCO_npp_d20130606_t1548531_e1550157_b00001_c20130606161732831673_cspp_dev.rem.h5
And it turns out
"Skipping unknown conversion"
I checked the code and found that only a few specific in-file-sizes are accepted by the script. Does that imply that I provided the wrong inputs? If so, What should I provide then?
Thanks in advance!
Re: Data Staging Issue 'mapped file size != expected size'
I believe the script that was provided only opertes on the binary files and not the HDF5 files. Though, I'm not sure what ./cnvtGEO_UAF.sh has in it. It's possible the script has morphed into something since the original posting.
You can use the ADL_Unpacker.exe to convert the H5 products to the binary files that the originial script operated on. Otherwise I might need to see the contents of the script you have.
Thanks,
Bryan Henderson
Raytheon Company
You can use the ADL_Unpacker.exe to convert the H5 products to the binary files that the originial script operated on. Otherwise I might need to see the contents of the script you have.
Thanks,
Bryan Henderson
Raytheon Company
Re: Data Staging Issue 'mapped file size != expected size'
Hi Bryan,
Thanks for the quick response. I think you are right, we can not apply the scripts on the HDF files.
The script is just the attachment of the post above mine, named cnvtGEO.txt.
Please have a look at that and your feedback is very welcome!
Wei
Thanks for the quick response. I think you are right, we can not apply the scripts on the HDF files.
The script is just the attachment of the post above mine, named cnvtGEO.txt.
Please have a look at that and your feedback is very welcome!
Wei
bhenders wrote:I believe the script that was provided only opertes on the binary files and not the HDF5 files. Though, I'm not sure what ./cnvtGEO_UAF.sh has in it. It's possible the script has morphed into something since the original posting.
You can use the ADL_Unpacker.exe to convert the H5 products to the binary files that the originial script operated on. Otherwise I might need to see the contents of the script you have.
Thanks,
Bryan Henderson
Raytheon Company
Re: Data Staging Issue 'mapped file size != expected size'
Wei,
The above script only works on the extracted geolocation binary files from the HDF5 files. You can use the ADL HDF5 Unpacker tool found in $ADL_HOME/tools/bin/
ADL_Unpacker.exe
Usage: ADL_Unpacker.exe [-h] [<filenames>]
-h Help option.
filenames Names of the HDF5 files to parse.
This tool will extract out the binary geolocation files from the HDF5 files, then you can run the above script to convert the formats to the later version.
Bryan Henderson
Raytheon Company
The above script only works on the extracted geolocation binary files from the HDF5 files. You can use the ADL HDF5 Unpacker tool found in $ADL_HOME/tools/bin/
ADL_Unpacker.exe
Usage: ADL_Unpacker.exe [-h] [<filenames>]
-h Help option.
filenames Names of the HDF5 files to parse.
This tool will extract out the binary geolocation files from the HDF5 files, then you can run the above script to convert the formats to the later version.
Bryan Henderson
Raytheon Company
Re: Data Staging Issue 'mapped file size != expected size'
Hi Bryan,
Thank you for the quick feedback. I just try to run ADL_Unpacker.exe on some sample GMTCO h5 file and it give me a Segmentation fault (core dumped) error. I read the help information carefully and checked the file path and name. Do I need to set any environment variables to run this application or it is just a standalone application?
~$ /home/CSPP/EDR_1_0/common/ADL/tools/bin/ADL_Unpacker.exe /raid/pub/wisc/npp/viirs/sdr/2013_06_06_157_2032/GMTCO_npp_d20130606_t2040286_e2041530_b00001_c20130606205128236673_cspp_dev.h5
Thanks!
Wei
Thank you for the quick feedback. I just try to run ADL_Unpacker.exe on some sample GMTCO h5 file and it give me a Segmentation fault (core dumped) error. I read the help information carefully and checked the file path and name. Do I need to set any environment variables to run this application or it is just a standalone application?
~$ /home/CSPP/EDR_1_0/common/ADL/tools/bin/ADL_Unpacker.exe /raid/pub/wisc/npp/viirs/sdr/2013_06_06_157_2032/GMTCO_npp_d20130606_t2040286_e2041530_b00001_c20130606205128236673_cspp_dev.h5
Thanks!
Wei
bhenders wrote:Wei,
The above script only works on the extracted geolocation binary files from the HDF5 files. You can use the ADL HDF5 Unpacker tool found in $ADL_HOME/tools/bin/
ADL_Unpacker.exe
Usage: ADL_Unpacker.exe [-h] [<filenames>]
-h Help option.
filenames Names of the HDF5 files to parse.
This tool will extract out the binary geolocation files from the HDF5 files, then you can run the above script to convert the formats to the later version.
Bryan Henderson
Raytheon Company