Descartes merge tool on huge area

Hi Mike,

If I understand correctly, you have 21 iTIFF64 uncompressed files created from 21 ECW files and you are trying to merge those 21 iTIFF64 file into one big uncompressed iTIFF64 file, is that correct?

Does the iTIFF64 images to merge are reprojected? If so that might be the culprit, since reprojection involve further complex mathematical operation which can slow the whole thing down.

If so I would suggest that you avoid doing a reprojection during the merge and use instead reprojection during the a Save As operation (not sure it would be faster, but it is worth a try).

Thanks,

Mathieu

Hi Mathieu,

No we are merging without reprojection for the reason you mention.

Unknown said:

use instead reprojection during the a Save As operation

I didn't know save as actually reprojected the data. I had assumed a change in the GCS through 'save as' would correct the projection definition but not reproject the data. I thought I would need to do another 'merge' operation after reprojecting the one 'BIG' itiff64.

Reprojection usually involves a rotation. Wouldn't It need to be resampled ('merged'?) to create the pixels to be square to their intended projection grid?

Mike

Hi Mathieu,

Yes the initial rate of file size increase is rapid then after a day or so it slows to a crawl even though microsoft resource monitor reports a constant rate of ~ 20-25Mb/s.

I should mention I am doing this in Bentley map enterprise. Also I am running another instance of bentley map enterprise on the same machine (we can't afford to have the licence tied up for days just to complete one task).

Mike

Hi Mike,

I just did a test doing a merge of about 100 compressed TIFF files (total size of about 5 GB) to create a huge around 390 uncompressed GB iTIFF64 file.

I didn't let the process terminated since I don't have that free space, but in 14 minutes my iTIFF64 file had already 37 GB of data.

Note that my computer is about 3 years old, no Solid State Drive in it.

The source and destination files were on the same magnetic hard drive.

So I'm suspecting that there is some computer problem involved. By two dedicated local drives, do you mean external USB drives? That could be causing the problem.

You also mention that you are also running another instance of Bentley Map Entreprise. Me during the Merge my computer went so slow that I could barely do another thing. So it seems the transfer of pixel data is not using as much computer resource as mine.

I would thus suggest that you drive doing a smaller merge (10-15 GB) directly with everything on your computer hard drive and check the performance. I expect that you would have the kind of performance I'm seeing.

Thanks,

Mathieu

Hi Mathieu,

Thanks for testing. During 

This machine is a Dual Zeon 2.3Ghz with 32 GB with solid state primary. I have two 1TB western digital blacks connected directly to sata ports The Idea was to read from one and write to the other. Running multiple instances doesn't seem to bother this thing as far as operational speed.

I wrote a script to log the file size every half minute. The script and results are in the attached zip. (The script is in AHK and readable in a text editor.) @395.5 minutes the file size spiked and then slowed to a crawl thereafter.

Note I started the logging script a few hours after it began the merge operation so 0 minutes is actually some time after it had began the merge operation.

I'm going to kill it now as it looks like its going nowhere.

Re: trying a smaller merge.
Yes. we do smaller merges without issue. Just this job is huge! 

When the process takes a long time anyway it sure is difficult to trouble shoot.

Mike

Hi Mike,

I'm wondering if your solid state primary is not stealing all the speed on your bus. Also maybe the motherboard has been built so the solid state can interrupt the SATA disks, which would explain why you are not experiencing any lagging while working during the Merge.

In any case even at the beginning of your script 45 Mb per minute seems awfully slow compared to the 2500 Mb per minute on average up to 37 Gb!

My last advice would be, find a computer without a solid state, and plug the two SATA (or even better, use only one SATA) and retry the merge. My guess is that in 6-7 hours the Merge should be complete.

Thanks,

Mathieu

Actually in that test I used LZW compression on the output. Sorry to confuse things.

Using uncompressed output I get a much faster write initially but then it slows down.

During the fast write speed there is no actual reading of the source images! So what I believe happens is that when writing uncompressed  the file gets created first (like a blank image) and then the data is added to it.

You can confirm this by using windows resource monitor. filter by system and ustation and look at the file reads/writes on the drive. The source images are not read until the file reaches it's maximum size. It is this second phase that seems to take a long time ( the actual transfer of the data from source to destination image).

So in your test you probably stopped it before there was any transfer of image data.

Mike

Actually in that test I used LZW compression on the output. Sorry to confuse things.

Using uncompressed output I get a much faster write initially but then it slows down.

During the fast write speed there is no actual reading of the source images! So what I believe happens is that when writing uncompressed  the file gets created first (like a blank image) and then the data is added to it.

You can confirm this by using windows resource monitor. filter by system and ustation and look at the file reads/writes on the drive. The source images are not read until the file reaches it's maximum size. It is this second phase that seems to take a long time ( the actual transfer of the data from source to destination image).

So in your test you probably stopped it before there was any transfer of image data.

Mike

Hi Mike,

Ah ok, now I see, you seem right, the destination raster is first written with background color (which when using LZW compression of the size of the destination raster is very small pixels with same color can easily be compressed).

So using the size of the raster to evaluate the progress is not possible thus, even more when the destination data is uncompressed.

I let a 23 GB  uncompressed export run till the end and it takes about 7 hours.

I am not sure why the performance bottleneck is so I have entered Defect 114991.

Thanks,

Mathieu