The following article is providing with more technical details on the article within our blog at the link http://www.citilabs.com/citilabs_blog/how-to-significantly-reduce-runtime-in-cube-voyager-by-handling-matrices-in-memory/
AUTOMDARRAY is a sub-keyword for DBI and MATI.
It has been used in the below example loading matrix data (MATI) into multi-dimensional arrays before any script processing, replacing the usage of MATVAL to obtain random access to matrix values. This way, selected tables can be loaded into arrays stored in memory to access cells values within the script.
When dealing with big matrices and reading matrix cells from disk using MATVAL, looping through many complex operations can be slow. Using AUTOMDARRAY can help in reducing run time for the single Matrix programs with major gains in the overall run time.
MATVAL is a MATRIX function providing random access to matrix values. MATVAL reads the desired matrix row from disk as needed, saving a number of rows in memory just in case they are needed again. If there is enough memory available to hold all the matrix tables needed, AUTOMDARRAY is indeed much faster.
Attention should be paid when dealing with very big matrices, to avoid memory limitations issues, with the following advice (there is not an exact rule in terms of memory usage):
As an example of usage of AUTOMDARRAY for an individual Matrix program, we used a triple indexing application developed by the Western Australia DOT, producing the skim matrix for Park&Ride.
This program was originally looping through several matrices cells using MATVAL to calculate the combined Park&Ride Origin-Destination skim matrix.
Original script before improvement:Matrix #1: The original script for the Park&Ride skim matrix used MATVAL (MATI[#], Matrix “Tab” #, I, J [, E]) to access the matrix cells and had a runtime of 00:02:00.
Improvement step #1: AUTOMDARRAY_1Matrix #3 uses AUTOMDARRAY as a sub-keyword in the modified script after the MATI keyword defining the input matrices. This is shown here below for the first input matrix:
The input cells values stored in memory are then called using the name of the multidimensional array given above name[#][i][j] (MATI1, …, in the example) as below (within the I, K, J loop).
Using this step, our example had a runtime of 00:01:22.
Improvement step #2: AUTOMDARRAY_2 Splitting the process into two sub-steps, in our example, total run time was reduced to 00:01:06 (circa):
Therefore, cutting in half the total run time, as shown in the below diagram.
This reduction in run time has been achieved only by storing in memory the input matrices with AUTOMDARRAY instead of looping through them accessing them from disk using MATVAL. If this type of programs are within a major loop, running it several times, a reduction in run time of the 50% can have a big impact. Note that the article is not suggesting to replace the usage of MATI+MW within your scripts instead.
Please refer to the Cube Voyager Help at the section Matrix Program > Control Statements > FILEI for more details.
Finally, further improvements can be achieved by using Cluster Intrastep within your matrix process, with a reduction in run time to a fraction of the initial value (around 20 seconds in the above Park&Ride example).