@verdin The original eq[2] was undefined at (0,0,0).  The modified one is well-defined at (0,0,0), however it is not differentiable there, and therefore the equation cannot be linearized.  Here are the details.

Download mw.mw

In your worksheet you have eq[1], eq[3], eq[3].  I assume that the middle one is meant to be eq[2].

Let's evaluate your eq[2] at (x1,x2,x3)=(0,0,0):
eval(eq[2], [x1(t)=0, x2(t)=0, x3(t)=0]);

We see that Maple complains with:
Error, numeric exception: division by zero

This says that your eq[2] is undefined at (x1,x2,x3)=(0,0,0), therefore linearizing about (x1,x2,x3)=(0,0,0) is not meaningful. You need to think that over.

@Carl Love As the parameter y changes, the graph of the function makes a transition from having two extrema (the blue curve below), to having a flat plateau (red curve) to monotone decreasing (green curve).  At the plateau both the first and second derivatives are zero.

@jan123 The labels (x and y in this case) are taken from the names of the variables in the equations in sys, but you may override them by specifying labels explicitly.  For instance,

There is no direct way of specifying label positions; they are placed automatically by Maple. For full control, you may disable the labels altogether, as in:
p1 := DEtools:-DEplot(sys, [x(t),y(t)], t=-2..2, [ic], x=-1..1, y=-1..1,
    linecolor=blue, color=red, thickness=1, tickmarks=[0,0],
    labels=["",""]);

and then design your own labels with the help of plots:-textplot(), as in

p2 := plots:-textplot([[0.9,-0.1,e__1], [0.1,0.9,e__2]]);

and then display p1 and p2 together:

plots:-display([p1,p2]);

@Carl Love I also see the behavior described by the OP.  Perhaps that's specific to Linux.  Here's what I see:

Download mw.mw

@Alger

Download orthogonality.mw

@Kitonum That's a nice animation—the red segments stand out and nicely demonstrate their equal lengths and orthogonality.

It is interesting that when one of the sides of the quadrilateral degenerates to zero length, that is, when the quadrilateral turns into a triangle (even a right triangle!), the theorem's assertion still remains valid and nontrivial.  I don't recall having seen that special case stated anywhere.  One would think that Euclid would have noticed that.

@Markiyan Hirnyk I invite you to go back and read (carefully now!) the reply under the heading "Yes, I have the response" and don't pull things out of your imagination, or somewhere else.

@Markiyan Hirnyk I am not going to stop you from laughing, but consider reading carefully what is written rather than what you imagine is written.  And an apology will not hurt either.

@Markiyan Hirnyk The introduction of biorthogonality in this discussion is a red herring.  For the purpose applying the Fourier transform, what is needed is the orthogonality of the basis functions which in the context this problem consists of the family
{ C(n,x), S(m,x) }_{n = 0...∞, m = 1...∞},
and where the orthogonality is relative to the given weighted L² inner product.  I pointed out that family of functions given in the original post was not orthogonal, that's why an attempt to apply the Fourier transform was leading to an incorrect result.   The family of functions in the amended post is orthogonal, therefore applying the Fourier transform is legitimate in that case.  That's all.

@Alger OK, then.  After that change, the issue raised in your original post should go away.

@vv Nice application of complex algebra!

Digging through my archive of Maple files, I found this animation from some yeara ago:

prob-02-quad.mw

@ibndirac I don't quite understand the setting of this problem. On the one hand you impose boundary conditions at x=0 and x=L, perhaps implying that you seek a solution g(x,t) where 0 < x < L. On the other hand, to determine beta, you need the values of g(x,t) for all x ∈ (−∞,∞). What is the domain of g? Some clarification is needed.

You will get more helpful responses if you post a real worksheet rather than snippets, especially since the snippets that you have posted seem to be inaccurate.  For instance, what is the meaning of the derivative diff(g(x, t), v) since g(x,t) has no v in it?

To post a worksheet, click the big fat green arrow that appears in the toolbar of the window where you edit your message for posting.