ERROR 1: TIFFAppendToStrip:Maximum TIFF file size exceeded.

KWX
Posts: 19
Joined: Wed May 16, 2018 9:42 am

ERROR 1: TIFFAppendToStrip:Maximum TIFF file size exceeded.

Post by KWX »

Good Evening,

I'm receiving the following error:

ERROR 1: TIFFAppendToStrip:Maximum TIFF file size exceeded. Use BIGTIFF=YES creation option.

Is there a specific file I could modify to add the "BIGTIFF=YES" to allow tiff of over 4GB to be created? This only happens when I'm working with the Full Disk images, at least on the current tests I've performed.

Thanks
kathys
Posts: 487
Joined: Tue Jun 22, 2010 4:51 pm

Re: ERROR 1: TIFFAppendToStrip:Maximum TIFF file size exceed

Post by kathys »

Can you provide us with more information? What command are you executing that is resulting in this error message?

Thank you, Kathy
davidh
Posts: 116
Joined: Tue Jun 04, 2013 11:19 am

Re: ERROR 1: TIFFAppendToStrip:Maximum TIFF file size exceed

Post by davidh »

Right now there is no extra flag that can be given to enable bigtiff support from the command line. From what I can tell you have a couple options (assuming that bigtiff support is built in to our version of libgeotiff):

1. Try setting and increasing `PYTROLL_CHUNK_SIZE` to something like 2048 or 4096. This should provide larger chunks to the geotiff library and possibly (unlikely) reduce the file size a little. Note that this will reduce performance of geo2grid overall, but shouldn't be too bad.
2. Try using the "--tiled", "--blockxsize 256", and "--blockysize 256" command line flags. This will create a tiled geotiff which usually compresses better than a normal "striped" geotiff.
3. This is a hack, but edit `GEO2GRID_HOME/lib/python3.6/site-packages/trollimage/xrimage.py` and on line 315 create a new line with:

bigtiff=yes,

I think this might work. See https://www.gdal.org/frmt_gtiff.html for more information on the bigtiff setting. WARNING: Setting this to "yes" will create a bigtiff *every time* which may not be desired. You'll have to find a way (environment variables?) to set this when you want it if the first two options don't help.

Good luck.
Dave
KWX
Posts: 19
Joined: Wed May 16, 2018 9:42 am

Re: ERROR 1: TIFFAppendToStrip:Maximum TIFF file size exceed

Post by KWX »

kathys wrote:Can you provide us with more information? What command are you executing that is resulting in this error message?

Thank you, Kathy
Hi Kathy,

Here's he command:

geo2grid.sh -r abi_l1b -w geotiff --output-filename FD_Test.tif -p true_color -f grb_common/OR_ABI-L1b-RadF-M3C*_G16_s20190651500307_e20190651511074_*.nc

Thanks
Kelvyn
KWX
Posts: 19
Joined: Wed May 16, 2018 9:42 am

Re: ERROR 1: TIFFAppendToStrip:Maximum TIFF file size exceed

Post by KWX »

davidh wrote:Right now there is no extra flag that can be given to enable bigtiff support from the command line. From what I can tell you have a couple options (assuming that bigtiff support is built in to our version of libgeotiff):

1. Try setting and increasing `PYTROLL_CHUNK_SIZE` to something like 2048 or 4096. This should provide larger chunks to the geotiff library and possibly (unlikely) reduce the file size a little. Note that this will reduce performance of geo2grid overall, but shouldn't be too bad.
2. Try using the "--tiled", "--blockxsize 256", and "--blockysize 256" command line flags. This will create a tiled geotiff which usually compresses better than a normal "striped" geotiff.
3. This is a hack, but edit `GEO2GRID_HOME/lib/python3.6/site-packages/trollimage/xrimage.py` and on line 315 create a new line with:

bigtiff=yes,

I think this might work. See https://www.gdal.org/frmt_gtiff.html for more information on the bigtiff setting. WARNING: Setting this to "yes" will create a bigtiff *every time* which may not be desired. You'll have to find a way (environment variables?) to set this when you want it if the first two options don't help.

Good luck.
Dave
Dave,

Currently testing your suggestions.

Will let you guys know soon.

Thanks!!!
Kelvyn
KWX
Posts: 19
Joined: Wed May 16, 2018 9:42 am

Re: ERROR 1: TIFFAppendToStrip:Maximum TIFF file size exceed

Post by KWX »

Dave,

Going with values at or above 2048 for PYTROLL_CHUNK_SIZE, assisted in getting rid of the error without any other changes.

Passing the --tiled flag, definitely brought down the size down considerably from 4.1GB down to 985MB, while retaining the same image quality (with or without chunk size adjustments).

The error also does not appear if you're using a CHUNK_SIZE of 1024 and using the --tiled flag at the same time. So pretty much, treating the geotiffs as tiles takes care of the errors for sure... at least on my case.

I did however tried the bigtiff=yes flag, but appears that is not implemented yet as I would get the following: "NameError: name 'yes' is not defined".

Appreciate the assistance.

Thanks
Kelvyn
davidh
Posts: 116
Joined: Tue Jun 04, 2013 11:19 am

Re: ERROR 1: TIFFAppendToStrip:Maximum TIFF file size exceed

Post by davidh »

Sorry. "yes" should have been a string, try putting quotes around it.
davidh
Posts: 116
Joined: Tue Jun 04, 2013 11:19 am

Re: ERROR 1: TIFFAppendToStrip:Maximum TIFF file size exceed

Post by davidh »

Just curious, is there anything in your command execution that you forgot to include in your example command? Are you changing compression? Are you using a different data type (not 8-bit integer)? Unless my math is wrong, shouldn't an uncompressed RGBA full disk ABI image (at 500m resolution) be ~1.7GB + metadata?
KWX
Posts: 19
Joined: Wed May 16, 2018 9:42 am

Re: ERROR 1: TIFFAppendToStrip:Maximum TIFF file size exceed

Post by KWX »

davidh wrote:Sorry. "yes" should have been a string, try putting quotes around it.
Thanks David, treating it as a string did the job. Increased the size a bit, but not a lot, for me it was in the rage of KB.
KWX
Posts: 19
Joined: Wed May 16, 2018 9:42 am

Re: ERROR 1: TIFFAppendToStrip:Maximum TIFF file size exceed

Post by KWX »

davidh wrote:Just curious, is there anything in your command execution that you forgot to include in your example command? Are you changing compression? Are you using a different data type (not 8-bit integer)? Unless my math is wrong, shouldn't an uncompressed RGBA full disk ABI image (at 500m resolution) be ~1.7GB + metadata?
In regards question #1, I believe I've not left any additional parameters when I issued the command I posted. Now, if there's information I'm missing, please enlighten me :) , as I'm still testing options.

As far as question #2, I've tested the zlevel (in the geotiff.yaml file) values of 1 and the default of 6 with little to no change in file size and/or image quality, so likely this setting is being overridden by other process(es). I'm currently not aware of another location to change compression options. I'm hopefully using the same data set as you guys, which if I understood correctly, is 8 bit and which I obtain via L-Band from GOES-16.
Post Reply