A couple of days ago I found out that gzread from the zlib library can be used for fast reading of binary files in Maple from the disk to memory - about 100 times faster than readbytes - something like in the following simplified example,
A:=Array(1..2^26,99,datatype=integer[1]):
time(writebytes("A",A));close("A");
9.360
B:=Array(1..2^26,1,datatype=integer[1]):
time(readbytes("A",B));close("A");
8.065
B[1],B[-1];
99, 99
myreadbytes:=proc(f)
local gzopen, gzread, gzclose, n, p, A;
gzopen:=define_external('gzopen',
'path'::string,
'mode'::string,
'RETURN'::integer[4],
'LIB'="zlibwapi.dll");
gzread:=define_external('gzread',
'file'::integer[4],
'buf'::REF(ARRAY(datatype=integer[1])),
'len'::integer[4],
'RETURN'::integer[4],
'LIB'="zlibwapi.dll");
gzclose:=define_external('gzclose',
'file'::integer[4],
'RETURN'::integer[4],
'LIB'="zlibwapi.dll");
n:=FileTools:-Size(f);
A:=Array(1..n,datatype=integer[1]);
try p:=gzopen(f,"rb");
if gzread(p,A,n)=n
then return A end if
finally gzclose(p)
end try
end proc:
time(assign(C=myreadbytes("A")));
0.062
C[1],C[-1];
99, 99
'time(myreadbytes("A"))'$5;
0.078, 0.062, 0.046, 0.046, 0.046
E:=Array(1..2^26,2,datatype=integer[1]):
time(ArrayTools:-Copy(A,E));
0.093
That needs some tweaking, because that works only on uncompressed files. If a file ("A" in this example) was gzipped, then the gzread would ungzip n (uncompressed) bytes in it in this example, instead of copying it into the memory - but it is not a big deal, in general.
Does anybody know about a similar replacement for writebytes? gzwrite doesn't work for copying (it compresses the array.)
I used the zlibwapi.dll library from http://www.winimage.com/zLibDll/index.html, it is a version of zlib 1.2.5 (written by Jean-Loup Gailly and Mark Adler) built by Gilles Vollant. The code is for a 32-bit system (Windows). That should work in 32-bit Linux after replacing that dll with standard libz.so.1, as well as on 64-bit systems after replacing integer[4] with integer[8] in most places.
