ERROR 1: TIFFAppendToStrip:Maximum TIFF file size exceeded.

davidh
Posts: 116
Joined: Tue Jun 04, 2013 11:19 am

Re: ERROR 1: TIFFAppendToStrip:Maximum TIFF file size exceed

Post by davidh »

I'm currently not aware of another location to change compression options.
There is a `--compress` flag which defaults to LZW right now. I'm not sure if this respects the zlevel from the yaml config or not.

Do you do any other python development? Is it possible you have some config files or environment variables that are being incorrectly picked up by geo2grid? Are all of your full disk true colors this large? Could you provide us the data in some way (FTP, HTTP download, etc)?
KWX
Posts: 19
Joined: Wed May 16, 2018 9:42 am

Re: ERROR 1: TIFFAppendToStrip:Maximum TIFF file size exceed

Post by KWX »

davidh wrote:
I'm currently not aware of another location to change compression options.
There is a `--compress` flag which defaults to LZW right now. I'm not sure if this respects the zlevel from the yaml config or not.

Do you do any other python development? Is it possible you have some config files or environment variables that are being incorrectly picked up by geo2grid? Are all of your full disk true colors this large? Could you provide us the data in some way (FTP, HTTP download, etc)?
Interesting, totally overlooked the geotiff writer flags. Will test those out for sure.

I don't do python dev on the specific box I have geo2grid running. In matter fact, is a basic CentOS 7 deployment with the bare minimums. Yeah, all the day time true colors are ~985MB in size give or take 20MB or so. Of course, if I test with FD images that are for times during/close to dawn/dusk, then they'll be smaller, but still ranging in the 500MB to 800MB range.

I sure can provide you guys with the netCDF files. Let me know where you like me to upload them to.
davidh
Posts: 116
Joined: Tue Jun 04, 2013 11:19 am

Re: ERROR 1: TIFFAppendToStrip:Maximum TIFF file size exceed

Post by davidh »

You should be able to do an anonymous upload to the "pub/incoming" directory of our FTP server: ftp://ftp.ssec.wisc.edu/pub/incoming
Otherwise, if you have your own FTP or HTTP server you could put them there. Kathy may have other suggestions.
KWX
Posts: 19
Joined: Wed May 16, 2018 9:42 am

Re: ERROR 1: TIFFAppendToStrip:Maximum TIFF file size exceed

Post by KWX »

davidh wrote:You should be able to do an anonymous upload to the "pub/incoming" directory of our FTP server: ftp://ftp.ssec.wisc.edu/pub/incoming
Otherwise, if you have your own FTP or HTTP server you could put them there. Kathy may have other suggestions.
David/Kathy, I've uploaded the following:

OR_ABI-L1b-RadF-M3C01_G16_s20190711645303_e20190711656070_c20190711656116.nc
OR_ABI-L1b-RadF-M3C02_G16_s20190711645303_e20190711656070_c20190711656106.nc
OR_ABI-L1b-RadF-M3C03_G16_s20190711645303_e20190711656070_c20190711656115.nc
davidh
Posts: 116
Joined: Tue Jun 04, 2013 11:19 am

Re: ERROR 1: TIFFAppendToStrip:Maximum TIFF file size exceed

Post by davidh »

Very interesting. I ran the equivalent code for Geo2Grid 1.0 (but with slightly different C libraries) on my Macbook and got ~2.6GB true color geotiff. When I ran Geo2Grid 1.0 on a CentOS 6 and CentOS 7 machine I got the same results of ~1.1GB. Kathy Strabala has mentioned to me that it seems to depend on what machine you are running it on. This may be something out of our control and may depend on the CPU being run on.

What operating system are you running on? Any idea what CPU(s) you are using?
KWX
Posts: 19
Joined: Wed May 16, 2018 9:42 am

Re: ERROR 1: TIFFAppendToStrip:Maximum TIFF file size exceed

Post by KWX »

davidh wrote:Very interesting. I ran the equivalent code for Geo2Grid 1.0 (but with slightly different C libraries) on my Macbook and got ~2.6GB true color geotiff. When I ran Geo2Grid 1.0 on a CentOS 6 and CentOS 7 machine I got the same results of ~1.1GB. Kathy Strabala has mentioned to me that it seems to depend on what machine you are running it on. This may be something out of our control and may depend on the CPU being run on.

What operating system are you running on? Any idea what CPU(s) you are using?
Well, that's definitely very interesting, I haven't seen such differences before. I can't disagree with Kathy's observations based on the tests you have done and what I get.

I'm currently running a CentOS 7 VM. The underlying hardware is an Intel i7-6700K CPU (8 w/ Hyperthread), but I'm working on moving it to a physical system with an i7-3770K CPU (8 w/ Hyperthread) and running CentOS 7 as well. I'm looking forward now, to what I'm going to get on the physical one. I like to test things out in a VM, before I spend too much time with a physical setup. :)

I will say that on my end, I don't mind the large file sizes, since I don't keep them for too long. I prefer image quality over disk.
davidh
Posts: 116
Joined: Tue Jun 04, 2013 11:19 am

Re: ERROR 1: TIFFAppendToStrip:Maximum TIFF file size exceed

Post by davidh »

I completely agree and understand on the VM before physical hardware thing, whenever possible. If you wouldn't mind updating this post when you move to physical hardware and let us know what size differences you get if any (original default operation, tiled operation, etc), that would be great for our own debugging. The two linux servers I ran on had:

Intel(R) Xeon(R) CPU E5-2643 0 @ 3.30GHz

and

Intel(R) Xeon(R) CPU E5-2640 v4 @ 2.40GHz

And produced the same results. My macbook has:

Intel(R) Core(TM) i7-4980HQ CPU @ 2.80GHz

It will be interesting to see how this progresses. As far as I can tell the same server produces the same results every time, so it isn't some kind of race condition (ex. one worker finishing something before another).
KWX
Posts: 19
Joined: Wed May 16, 2018 9:42 am

Re: ERROR 1: TIFFAppendToStrip:Maximum TIFF file size exceed

Post by KWX »

I sure will.
KWX
Posts: 19
Joined: Wed May 16, 2018 9:42 am

Re: ERROR 1: TIFFAppendToStrip:Maximum TIFF file size exceed

Post by KWX »

Here're the CPU specs:

Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz

Now, here're the options and completion times for each test:

NOTE
- All test using the same GOES-16 Full Disk dataset, for time of day where FD was fully illuminated (by the sun).
- The following switches were used on all tests: --progress -p true_color
- "geo2grid.sh -r abi_l1b -w geotiff --output-filename FD_Test.tif --progress -p true_color -f OR_ABI-L1b-RadF-M3C0*_G16_s20190741700371_e20190741711138_*.nc"

Run 1:
Using default settings fails with "ERROR 1: TIFFAppendToStrip:Maximum TIFF file size exceeded. Use BIGTIFF=YES creation option."

Run 2:
--tiled
991M - 6min 14.3s

Run 3:
bigtiff=yes
4.6G - 12min 9.7s

Run 3 with different compression options:
--compress DEFLATE : 3.8G - 18min 15.7s
--compress NONE : 1.8G - 6min 34.1s
Below using CHUNK Size of 2048
--compress DEFLATE : 3.3G - 18min 20.3s
--compress NONE : 1.8G - 7min 8.7s

Run 4:
bigtiff=yes
--tiled
991M - 6min 17.6s

Run 4 with different compression options:
--compress DEFLATE : 799M - 8min 15.0s
--compress NONE : 1.8G - 6min 3.3s
Below using CHUNK Size of 2048
--compress DEFLATE : 799M - 8min 37.4s
--compress NONE : 1.8G - 6min 40.0s
davidh
Posts: 116
Joined: Tue Jun 04, 2013 11:19 am

Re: ERROR 1: TIFFAppendToStrip:Maximum TIFF file size exceed

Post by davidh »

I had a breakthrough on this. How much memory do your systems have? You can try playing with the `GDAL_CACHEMAX` variable which is described here: https://trac.osgeo.org/gdal/wiki/Config ... L_CACHEMAX

From what I understand of the situation, GDAL defaults to using 5% of available memory for holding on to data (caching) before writing it to disk. This means that on lower memory systems (16GB) it won't be holding on to as much data as it would on a more powerful server (128GB) before compressing it and writing it to disk. Since it isn't using large amounts of data in its compression, it isn't doing a great job.

What happens if you set the environment variable to something like:

Code: Select all

export GDAL_CACHEMAX="15%"
before running geo2grid?

Dave
Post Reply