This is a toy example illustrating three possible ways to define a group.

An icosahedron:

with(GroupTheory): with(Student[LinearAlgebra]): with(geom3d):

icosahedron(ii): vv := vertices(ii):

PLOT3D(POLYGONS(op(evalf(faces(ii))), TRANSPARENCY(.75)),
  op(zip(TEXT, 1.1*evalf(vv), [`$`(1..12)])), AXESSTYLE(NONE));

The group of rotations is generated by the rotation around the diagonal (1,4) and the rotation around the line joining the midpoints of the edges (1,2) and (3,4), by the angles 2*Pi/5 and Pi respectively.

Define the group by how the two rotations permute the 6 main diagonals of the icosahedron:

gr := PermutationGroup({[[2, 5, 3, 4, 6]], [[1, 2], [5, 6]]});

IdentifySmallGroup(gr);
                             60, 5

Or define the group by the relations between the two rotations:

gr2 := FPGroup([a, b], [[a$5], [b$2], [a, b, a, b, a, b]]);

IdentifySmallGroup(gr2);
                             60, 5

Finally, define a group with elements that are rotation matrices:

m1 := RotationMatrix(2*Pi/5, Vector(op(4, vv) - op(1, vv))):
m2 := RotationMatrix(Pi, Vector(add(op(3..4, vv) - op(1..2, vv)))):
m1, m2 := op(evala(convert([m1, m2], radical))):

gr3 := CustomGroup([m1, m2], `.` = evala@`.`, `/` = evala@rcurry(`^`, -1), `=` = Equal);

IdentifySmallGroup(gr3);
                             60, 5

AreIsomorphic(SmallGroup(60, 5), AlternatingGroup(5));
                              true

One question is how to find out that the group is actually A₅ without looking up the group (60, 5) elsewhere.

Also, it doesn't matter for this example, but adding `1`=IdentityMatrix(3) to the CustomGroup definition gives an error.