Several of those LinearAlgebra calls are not being done "in-place". So unnecessary Matrices are being produced, which is more garbage requiring collection than is necessary.
Note that every time one sees those "bytes used" messages in TTY Maple (or the bottom counters get updated in the GUI) there is garbage collection occuring. When bytesused goes up much faster than bytealloc, it usually means that a great deal of garbage is being produced. Garbage collection is relatively expensive, and very expensive compared to hardware numeric operations doen externally.
Also, the use of that RandomTools routine as the Matrix initializer doesn't seem best. I believe that using it (as posted, originally) involves many procedure calls. Maple procedure calls are very expensive. And those would return software float scalars, which will all become more garbage. But the Statistics package can populate a hardware datatype Vector, using compiled external code to generate the random numbers. And then ArrayTools may be used to copy such random hardware data (again using compiled external code) into the complex[8] Matrix, without any software floats being produced.
It may be possible to use ArrayTools:-Alias, as an alternative to some of my ArrayTools use below.
Does this code do the same thing? (I may have made mistakes.)
Feynman_random_rho := proc(N::posint, d::posint:=2)::'Matrix'(complex[8]);
local rho, temp, X, a, rtemp, temp2;
X := Statistics:-RandomVariable(Normal(0,1)):
temp := Matrix(d^N, d^N,datatype=complex[8]);
a:=Statistics:-Sample(X,d^N * d^N):
rtemp := ArrayTools:-ComplexAsFloat(temp);
ArrayTools:-Copy(d^N * d^N, a, 0 , 1, rtemp, 0, 2);
temp2 := Matrix(d^N, d^N,datatype=complex[8]);
a:=Statistics:-Sample(X,d^N * d^N):
rtemp := ArrayTools:-ComplexAsFloat(temp2);
ArrayTools:-Copy(d^N * d^N, a, 0 , 1, rtemp, 0, 2);
LinearAlgebra[Add](temp, temp2, 1,I,inplace=true);
rho := LinearAlgebra[HermitianTranspose](temp).temp;
LinearAlgebra:-MatrixScalarMultiply(rho,1/LinearAlgebra[Trace](rho),inplace=true);
end proc:
acer

One of the big differences between Maple and regular LaTeX is that the former is (almost) "What You See Is What You Get" while the latter is a scheme for markup that must usually be further processed prior to display.
That's a simplification, of course. There are WYSIWYG applications for LaTeX, some commerical and some free.

LyX is one of the older free ones. And Maple brings huge computer algebra functionality to the table.
In my experience, people really wanting fine-grained control of layout and typesetting often eventually need to resort to something with as much power as LaTeX (or even of its underlying engine, TeX). As soon as one encounters any special issues with a WYSIWYG editor, the question of finer control arises.
You named one such instance: programmatic control of typesetting, for entering symbols, say. A great many people do not consider that aspect, up front. So if it's a concern of major importance for you, then do get the right answers.
As far as I know, there is no mechanism for programmatically entering all the symbols that Maple knows how to typeset. And I don't know of any means for importing LaTeX or TeX into Maple, either. I'd be delighted to learn otherwise.
When Maple's "2D Math" suffices, it can save a great deal of time. It allows lots of nice, easily entered 2D Math typesetting. And it's wonderful to be able to immediately do symbolic calculations on that input. Maple also has abilities to convert expressions and worksheets to LaTeX.
There's very little that can provide the sort of fine, powerful, typesetting control that TeX does. Maybe you won't need to prepare photo-ready quality typesetting such as major journals need. Perhaps, if you got to that point, you could use Maple's exporting facilities to generate LaTeX from what you already had.
acer

There are several really nice texts on LaTeX. One that I've always enjoyed is _The LaTeX Companion_ by Goossens, Mittelbach, and Samarin. Look for the later edition.
There's lots online, to get you started. You might also enjoy this thorough

overview of TeX at wikipedia.
Leslie Lamport wrote LaTeX as a simplified, easier-to-use format of TeX. It is a delight to use: powerful, but straightforward. I doubt that you would regret learning it.
I suggest

CTAN as a good site, for getting started.
acer

To answer the question at the end of your post, no, use of subscripts need not mean use of indexed names.
If you toggle the subscripted object (input by typing, say, m_0) to be a unique name (distinct from m) then the later subs m=k will not affect it. You can convert the subscripted object into a unique name, distinct from m or indexed m, by using the context-sensitive menu on it.
Select just the object such as what you got from typing m_0, with the mouse, and then right click to activate the context-sensitive menus. Choose the context-menu action,
2D Math -> Convert To -> Atomic Identifier
acer

This reminds me of a pet peeve of mine. When I get email with .mw attachments, they always get corrupted by my ISP's webmail interface. I think that it may be related to XML as the format. I've seen three different webmail systems do this. It gets tiresome, having to ask people to resend their .mw attachments zipped, which works.
Now, somewhere I read that maplenet or some other thing can use a special compressed .mwz format. Since worksheets are not usually huge, maybe .mwz could be made the default worksheet format, as decompression on the fly wouldn't be so bad a penalty.
acer

?algcurve
That should pull up a related worksheet, in Maple 11.
There is a references section named "Related Information" at the bottom.
acer

eq:=x^3+x-1+0;
fsolve(eq);
Digits:=14:
fsolve(eq);
acer

with(LinearAlgebra):
A := RandomMatrix(2);
sol1 := MatrixFunction(A,cos(x),x);
# You are currently computing something like this?
# I presume that you meant to take the cosine of
# each of the eigenvalues in L?
# (Did you want normalization in P?)
evals,evecs := Eigenvectors(A):
sol2 := simplify( evecs . DiagonalMatrix(map(cos,evals)) . evecs^(-1) );
acer

Your final do-loop assigns columns, in-place, into what you intend to be Matrix P. But you haven't created the Matrix P, yet. Try adding,
P := Matrix(3,3):
before that final do-loop.
You might also think that P should be purely real. You could try simplify(map(fnormal,P)) at the end. But maybe you would rather compute the eigenvectors of a floating-point symmetric version of stif2, instead of doing floating-point evaluation of the eigenvectors of the exact stif2? Ie,,
Eigenvectors(evalf(Matrix(stif2,shape=symmetric)));
acer

You posted this in the category, "How do I ...? with Maple". So here's how to do it in Maple,
y := ln(sin(x)) - (1/2*sin(x)^2);
diff(y,x);
acer

Enter any of these at the maple prompt, and then press the key.
?scatterplot
?Statistics,scatterplot
?stats,scatterplot
acer

Sorry for not being clear enough

here. What I was trying to get across was that there may be neither a name nor some known, especially nice, closed form for the convergent result.
Just don't

name it after yourself. :)
acer

Maple 11.00, 64 Linux,
> ba,bu,st:=kernelopts(bytesalloc),kernelopts(bytesused),time():
> N:= 10^6:
> R := [ add( rand(0..1), k=1..12 )-6$N ]():
> kernelopts(bytesalloc)-ba,kernelopts(bytesused)-bu,time()-st;
376107760, 664232360, 39.998
New session (not restart),
> ba,bu,st:=kernelopts(bytesalloc),kernelopts(bytesused),time():
> N := 10^6:
> g := rand(0..1):
> R := Vector[row](N,datatype=integer[4]):
> for i from 1 to N do
> R[i] := add(g(),j=1..12) - 6;
> od:
> kernelopts(bytesalloc)-ba,kernelopts(bytesused)-bu,time()-st;
10352792, 217286880, 12.098
So, my posted method seems to use 1/36th the allocated memory, and take 1/3 of the time, for N=10^6.
There may be even leaner ways, perhaps using Statistics.
acer

Could it be that you don't really want to keep each of the one million sequences?
N := 10^6:
g := rand(0..1):
R := Vector[row](N,datatype=integer[4]):
for i from 1 to N do
R[i] := add(g(),j=1..12) - 6;
od:
After the above, the results are all stored in Vector R.
acer

There are various ways to accomplish this, and for your purposes you might be free to pick and choose.
I like the way that Joe suggested. But perhaps it's not to your taste, because you originally considered, map(x->1/x,test) rather than, say, map(`/`,test).
You might also wish to consider map(t->map(x->1/x,t),test) , if it's more to your style or liking.
acer