@Aaeru Michi 

As you see in the practice example, Maple does not accept an integro-differential system of this kind.
You must transform it into ODEs, and it's simple; why do you reject this? You have no other choice.

I don't understand your last system.  If you mean:
y'(t)=x(t)^2+z(t)
x''(t)=y(t)+t
z(t)=int ( x(t1)*y(t1), t1=0 .. t )

then you must transform it as above: z' = x*y, z(0)=0.

@_Maxim_ 

Scaling is not a problem when solve/series works. The bottleneck is solve itself.
Try e.g. the similar  F := (x, y) -> ln((1+x)*y)+exp(x^2+y-1)-x-cos(x),  F(0,1)=0. ( D[2](F)(0,1) = 2 <> 0. )

p:=ln((1+x)*y) + exp((x^2)*(y^2)) - x - cos(x):
Order:=10:
'ytaylor'=solve(series(p,x),y);
ytaylor:=convert(solve(series(p,x,5),y), polynom): # useful only near 0
with(plots):
ip:=implicitplot(p, x=0..1, y=0..3, color=green):
display(plot(ytaylor,x=0..1,color=red), ip);

@Math Pi Euler 

p:=ln((1+x)*y) + exp((x^2)*(y^2)) - x - cos(x):
solve(series(p,x),y);

@Markiyan Hirnyk 

OK, thank you; it seems that only Maple on 64 bit is affected.

@Markiyan Hirnyk 

Exporting in pdf works but it's a low resolution bitmap.

Compare with a Maxima plot:
max.pdf

The dropbox .eps  was it from classic or standard interface?
[for 64 bit, the classic interface is absent].

A 2D .pdf can be obtained exporting in .eps format (encapsulated postscript) and then converting into .pdf using Ghostscript.
It results a nice vector graphic.
Note that on my system, exporting directly in .pdf (with the context menu) it results a low resolution bitmap.

The big problem is that the .eps is totally broken for 3d (for many years!). Practically, Maple does not have 3d vector graphics!

p:=plot3d(x^2+y^2, x=-1..1,y=-1..1):
plottools:-exportplot("v3d.eps",p);

@_Maxim_ 

Tracing quickly the original code shows that `DEtools/formal_sol`  also has a remember mechanism
so probably a deeper analysis is needed. I do not envy those who are going to debug this.

@Carl Love 

The problem was about using GF(p,n), without defining it ad-hoc as a quotient ring. Using Domains should be (theoretically) faster. Of course, if p, n and the polynomials are small then this simpler method is preferable.

@Parham2016 

You have Int(...,  r=0..1);

@digerdiga 

It does not redefine the variable.
See  ?MathematicalFunctions,Assume

@Jasagredo 

Ok, you want to start from scratch and not take advantage of the code already existing in Domains.
I admit that the documentation for Domains is less than minimal, and the user is adviced to "study the code of existing domains", but then you are on your own.

@_Maxim_ 

In my answer (after edit) it also works if expressions are used instead of functions.
It would be interesting to investigate the source of the bug.

Added later.
The RootOf obtained by solve is a bit simpler.
For the OP's RootOf, Maple enters somehow in an infinite loop of simplifications/conversions.

@Axel Vogt 

Yes, the picture is the geometric interpretation.

I am a mathematician, not a physicist and I do not use the Physics package (except a few commands, very seldom).
I'd like to ask a few principial questions.
The Physics package looks to me like a "state within a state" with those Setups, redefined basic commands and operators (including *, .), special typesetting etc.
So, the questions.
- Is Maple going to evolve in these directions?
- Why in Maple is missing an AbstractLinearAlgebra or a Rings package (similar to the GroupTheory package)? Many of their commands could be used by Physics and would have a larger audience.
- Are the basic commands in Physics (such as Assume) going to be merged/unified with the existing ones (assume in this case)?