@assma

If I copy-and-paste your code and run it exactly as you have it, except using a colon after end do, in Maple 2016, I get AVor = -4.633913767. If I use a second method (using add and evalhf), I get -4.63391167924315983, which is close enough to the first value. If I use a third method (compiling the code), I get -4.63391167924588299, which also is close enough. These are very different from the value that you got!

The code for the second method, which I'm nearly sure will run in Maple 18, is

f:= proc(x,y) option inline; sin(x)*cos(y) end proc:
CodeTools:-Usage(
   evalhf(add(add(f(.1*x, .1*y), x= 0..1000), y= 0..1000))
);

The code for the third method, which may not run in Maple 18 due to lack of a compiler, is

F:= proc()::float;
local i::integer, j::integer, AVor::float:= 0, x::float, y::float;
   for i from 0 to 1000 do
      x:= i*.1;     
      for j from 0 to 1000 do
         y:= j*.1;
         AVor:= AVor + sin(x)*cos(y)
      end do
   end do;
   AVor
end proc:

FC:= Compiler:-Compile(F):
FC();

"Extended" typesetting is not meant to be a correction for bugs present in Standard typesetting. It ought to display correctly in both.

Can you give more details about what you mean by "filter"? Do you mean something like removing outliers? For example, you could remove any datum that's more the 3 standard deviations from the mean.

By "code", Kitonum means that you should upload your worksheet file to this site. Use the green uparrow on the toolbar of the MaplePrimes editor.

Function application and index application ought to be listed in the chart at ?operators,precedence. In addition to being generaly useful, this would be a help for the people writing the typesetting code. Since there are no explicit infix symbols for these, people often fail to think of them as binary infix operators; but of course they are, as your example shows. I've mentioned this a few times over the years.

A slightly different way, using a few fewer lines of code, is Eigenvalues(F, V). For a floating-point problem this is more robust, but I doubt that there's much difference in computational effort for a symbolic problem such as this.

Acer, as far as I can tell, the OP's Reply immediately above is correct: The indexed occurrences of lam are being generated internally by a bug in int in the line beginning RRn:=. The OP doesn't appear to be at fault for this.

@tsunamiBTP Look up "catasthrophic cancellation" and especially the Wikipedia article "Loss of significance." The case at hand is an example of catasthrophic cancellation: the substraction of two quantities whose difference is many orders of magnitude smaller than the quantities.

@Math Pi Euler Try copy-and-pasting Tom's code. I think that you may have been entering it manually.

@Math Pi Euler Yeah, thanks, I figured that out by now. Anyway, I think that the Answer by Tom Leslie is sufficient for your purpose. If not, let me know.

What are the limits of summation?

Try typing your expressions here the same way as you would in Maple.

You say "verify the following formula" but I don't see any formula. You'll need to provide more information.

Update: The above comment applied to an earlier version of the Question.

@macmp91 So, we see that the minimal value of what you call w^2 eventually becomes negative. Why not use that as the criterion to stop your while loop? Don't take the sqrt in procedure Fn; take it afterwards.

@macmp91 Let's forget about the singular values for now. I don't know if they're appropriate for your problem. Why do you take the square root of the eigenvalues? Why not just stop your while loop when the minimal eigenvalue is negative?

@dbain1 The numerical integral is very difficult to evaluate for some values of p, such as 0.15. Your integrand looks a bit strange. Did you perhaps intend for it to be the absolute value of the expression that is to be raised to the 1/p power? If I include abs thus, there is no problem doing the numeric integration. Without abs, we end up raising negative values to fractional powers, obtaining imaginary results.