Interesting, well, I currently have 8GB (which should explain the larger file sizes, based on the statement), but I'll add more later and see how it behaves. I shall nonetheless try with various GDAL_CACHEMAX values and let you know later today.
Thanks David
Kelvyn
ERROR 1: TIFFAppendToStrip:Maximum TIFF file size exceeded.
Re: ERROR 1: TIFFAppendToStrip:Maximum TIFF file size exceed
No go, tried up to 50% but no real changes in decreased file sizes with 8GB.
Will try again when I upgrade the memory.
Will try again when I upgrade the memory.
Re: ERROR 1: TIFFAppendToStrip:Maximum TIFF file size exceed
Dave,
Ok, so here're the results.
My system can go as high as 32GB of RAM. I upgraded from 8GB @1333MHz to 32GB @1600MHz.
All tests performed are now finishing at ~5min 45s +/- 4s.
Minimum GDAL_CACHEMAX value where noticeable processing performance was achieved is at 25%. I found that using any values higher than 25%, do not introduce additional reduction in processing time (which is kind of expected since I'm only processing 1 FD image at a time, which is all I can do anyway). I do expect that on higher end systems with 16 cores or more and 128GB of RAM, that increasing the % should provide additional performance. Now, decreasing this value tends to give you a penalty of close to 45s to 50s of increased processing time.
I did also tested cutting the amount of RAM by half (16GB) and processing time increased by also 45s to 50s. So my conclusion is, and one to be expected, that is best to shoot for memory with as high of a clock as possible while keeping the memory size at 32GB or higher. This should keep your processing times in the 5min range or less per FD image (assuming you're processing 1 image at a time).
In regards disk usage, there was little change. Using NONE for compression gave me 1.8GB which is to be expected, with default compression I got 981MB, and with DEFLATE I got 792MB. All which are approximately the same as before.
Now, I've found in regards compression, that using the --compress JPEG flag gave me the best compression and image quality remains the same as rest, but I do have to keep in mind that older viewing software might not be able to read the 12bit JPEG compression (if I'm not mistaking) if this flag is used, but so far so good. With this flag I'm able to get the file size down to 191MB.
Finally, in regards order of best compression to use for fastest write to disk post processing (from fastest to slowest): NONE - LZW (default) - JPEG - DEFLATE.
Thanks
Kelvyn
Ok, so here're the results.
My system can go as high as 32GB of RAM. I upgraded from 8GB @1333MHz to 32GB @1600MHz.
All tests performed are now finishing at ~5min 45s +/- 4s.
Minimum GDAL_CACHEMAX value where noticeable processing performance was achieved is at 25%. I found that using any values higher than 25%, do not introduce additional reduction in processing time (which is kind of expected since I'm only processing 1 FD image at a time, which is all I can do anyway). I do expect that on higher end systems with 16 cores or more and 128GB of RAM, that increasing the % should provide additional performance. Now, decreasing this value tends to give you a penalty of close to 45s to 50s of increased processing time.
I did also tested cutting the amount of RAM by half (16GB) and processing time increased by also 45s to 50s. So my conclusion is, and one to be expected, that is best to shoot for memory with as high of a clock as possible while keeping the memory size at 32GB or higher. This should keep your processing times in the 5min range or less per FD image (assuming you're processing 1 image at a time).
In regards disk usage, there was little change. Using NONE for compression gave me 1.8GB which is to be expected, with default compression I got 981MB, and with DEFLATE I got 792MB. All which are approximately the same as before.
Now, I've found in regards compression, that using the --compress JPEG flag gave me the best compression and image quality remains the same as rest, but I do have to keep in mind that older viewing software might not be able to read the 12bit JPEG compression (if I'm not mistaking) if this flag is used, but so far so good. With this flag I'm able to get the file size down to 191MB.
Finally, in regards order of best compression to use for fastest write to disk post processing (from fastest to slowest): NONE - LZW (default) - JPEG - DEFLATE.
Thanks
Kelvyn
Re: ERROR 1: TIFFAppendToStrip:Maximum TIFF file size exceed
Wow, thanks for testing so many things. It's too bad that the disk usage didn't change. I really thought the file size would go down by changing this. We've got our own testing planned for this so we'll see what we can find on our systems and can then compare.
One last question, do you have any idea how much free memory you had just before starting geo2grid processing on the new system? Were any other "large" operations running?
One last question, do you have any idea how much free memory you had just before starting geo2grid processing on the new system? Were any other "large" operations running?
Re: ERROR 1: TIFFAppendToStrip:Maximum TIFF file size exceed
I roughly had close to 31GB of RAM available just before geo2grid was started.
So far, 7GB has the max amount of memory that it has used for a short duration during processing. Of course, this memory usage will vary depending on the compression you select to use.
In my case, at least for the time being, my system is dedicated to geo2grid, so nothing else, other than the typical OS related usage which is close to 350 to 400MB for any given time.
So far, 7GB has the max amount of memory that it has used for a short duration during processing. Of course, this memory usage will vary depending on the compression you select to use.
In my case, at least for the time being, my system is dedicated to geo2grid, so nothing else, other than the typical OS related usage which is close to 350 to 400MB for any given time.
Re: ERROR 1: TIFFAppendToStrip:Maximum TIFF file size exceed
UPDATE
Performed some additional tests. Nothing new in regards file sizes, but processing time have changed.
Tested 16GB configuration @1600MHz and only a ~4s decrease from using a 32GB @1600MHz.
On my previous test, noticed a higher processing time with 16GB, which was likely attributed to a slower disk access than expected at the time of the test.
In conclusion, if some likes to run at 8GB, they sure can do it, but definitely 16GB as recommended for a minimum configuration is good, but just keep up those memory clocks as high as possible.
Dave, are settings defined in this file being overridden (sure appears to be the case): /lib/python3.6/site-packages/satpy/etc/writers/geotiff.yaml? Was hoping to be able to do some additional tests.
Thanks.
Performed some additional tests. Nothing new in regards file sizes, but processing time have changed.
Tested 16GB configuration @1600MHz and only a ~4s decrease from using a 32GB @1600MHz.
On my previous test, noticed a higher processing time with 16GB, which was likely attributed to a slower disk access than expected at the time of the test.
In conclusion, if some likes to run at 8GB, they sure can do it, but definitely 16GB as recommended for a minimum configuration is good, but just keep up those memory clocks as high as possible.
Dave, are settings defined in this file being overridden (sure appears to be the case): /lib/python3.6/site-packages/satpy/etc/writers/geotiff.yaml? Was hoping to be able to do some additional tests.
Thanks.
Re: ERROR 1: TIFFAppendToStrip:Maximum TIFF file size exceed
Thanks for the extra info.
Yes, the values in that configuration file are defaults, but geo2grid overrides all defaults with its command line arguments regardless of whether they are specified or not (the command line arguments have their own defaults).
Yes, the values in that configuration file are defaults, but geo2grid overrides all defaults with its command line arguments regardless of whether they are specified or not (the command line arguments have their own defaults).