Differentiate the first equation of (18) and substitute G' from the second equation into the first equation. This yields a homogeneous linear ODE of second order. Its eigenvalues ​​sqrt(lambda*mu) are obvious and the general solution for F is known. The second equation of (18) is used in an analogous manner to calculate G.

@dharr 

As a beginner, I can learn a lot from this :-) . Computation time about 19 minutes.

The possibility of simplifying the equations by f_i - f_j has already been mentioned. Since only "cos" is left after that, the "cos(k*x)" should be expanded into "cos(x)" and then the addition theorem for "cos(x+x)" should be applied (Gradstein/Ryshik, Volume 1, page 55). However, this leads to a system of polynomials.

Another possibility is to convert the search for a solution for the system into a minimum problem. To do this, a suitable norm must be chosen for the entire equation defect. The problem here is always the starting point to be chosen in R^5. Will all real solutions be found after varying the starting point...?

Another possibility is to systematically search for solutions in a cuboid of R^5 that is to be defined. The cuboid must first be provided with grid points, which are then sampled mathematically. If there is a tendency towards a minimum, the grid is then locally refined and sampled again.

Unfortunately, as a Maple beginner, I am not yet able to do this in Maple and my old Mathcad is obviously overwhelmed by this.

I have often used the latter methods successfully in the past when I had to solve wickedly stubborn equations.

@vv 

...for the clumsy/unfortunate formulation "it's a shame". My English is not particularly good. It is not my native language.
Your solution is impressive. I hope it pleased you. My old purely constructive solution without calculation is also interesting. With this task I only wanted to ensure that important theorems such as Desargue and, for me in particular (not relevant here), the theorems of Hausdorf and Lebesgue (functional analysis) are not forgotten. These have been very effective in developing my way of thinking over the last 60 years of training and work.

It's a shame that no one is interested. This task arose when working on a graphical method for calculating kinematic chains in the training of engineers in the subject of "structural analysis". The background was a reference in Hirschfeld's book, Structural Analysis, page 27. Therefore, here is just a hint for the solution:
"Desargue's theorem"

@Scot Gould 

The references to "YouTube/@MapleProf" and "shortcut keys" are very helpful in learning how to use the program. Now I have to learn and practice to solve this problem.

@Scot Gould 

Thanks, I'll try it. Please answer this question:
How do I delete an empty/unused input line for execution commands that is located anywhere in the worksheet?

@vv 

"The polygon with maximal area is the cyclic one" - that is not obvious. More details in Lit.:
[1] Mathematics Magazine, Vol. 39, 1966, issue 4
[2] Geometric Problems on Minima and Maxima, Titu Andreescu, Oleg Mushkarov, Luchezar Stoyanov, Birkhäuser, Boston Basel Berlin

Additional question:
The circumcircle has a center. Which of the polygon vertices, ordered according to increasing length of the sides, are the vertices of the triangle whose perpendicular bisectors intersect at the circumcircle center?

@nm 

You're right. I should read more carefully :-(.

@Carl Love 

...not the calculation of the second initial condition for y´(0). Rather, the recognition that differentiation of the original equation to "deq" and subtraction of "eq" makes the integral disappear is important as training for future solution ideas - what remains is an ordinary differential equation that is easy to master with the help of Maple. Using Maple is, after this realization, simple, understanding "handwork" :-).

@mmcdara 

It is enough to calculate deq minus eq and the integral is gone. The task doesn't require any more tricks.

@mehdi jafari 

... us have a look ;-)

@Kitonum 

Yes, there is power in these Maple commands. In the past, the formation law had to be laboriously determined using the characteristic equation (literature e.g. Markuschewitsch, Lin. Rekursionen). It is important that pupils and students learn this background-knowledge in order to be able to use it if the computer breaks down ;-).

@salim-barzani 

...wavelets? I have no experience with solitons in solving methods. But as far as I can remember, there are methods that use wavelets, e.g.:

https://ijsts.shirazu.ac.ir/article_2153_836c1c7bda66ef58fcead78d7a6f9f97.pdf

Maple offers something about this in the help.
Good luck!

@salim-barzani 

..., but I can't help you with that. I was only interested in solving your equation out of curiosity in memory of "old times" even without knowing your specific physics background.
Finally, a link:
https://www.colorado.edu/amath/sites/default/files/attached-files/2015_uc-london.pdf