Group Actions

In this chapter we’ll consider some of the consequences of an object acting on a set. It is an important and recurring idea in mathematics to study how one object acts on another, and if we have enough time, we’ll more fully explore this with representation theory.

Orbits and Stabilizers

In the first chapter we introduced group actions as a way for a group to act by symmetries or transformations of a set. We now return to them more systematically.

Recall the definition of a group action: A group action of $G$ on a set $X$ (sometimes denoted $G\curvearrowright X$) is a rule that assigns, for every $g\in G$ and $x\in X$ an element $g\cdot x\in X$ so that the identity acts trivially and the action is compatible with the group operation:

$$e\cdot x=x\text{ for all }x\in X\text{and}g\cdot (h\cdot x)=(gh)\cdot x\text{ for all }g,h\in G\text{ and }x\in X.$$

We learned that equivalently, it is a homomorphism $\rho :G\to \mathrm{Sym}(X)$ from $G$ to the group of permutations of $X$, called the permutation representation of the action. This was because if you fix $g$, the map $x\mapsto g\cdot x$ is a bijection of $X$ to itself, and the compatibility condition ensures that $\rho$ is a homomorphism.

Perhaps the first question you should ask about a group action is: where do points go? That is, if I keep applying elements of $G$ to a point $x\in X$, what are all the possible places I can get to? This leads us to the notion of an orbit.

Definition

Let $G\curvearrowright X$ be a group action. The orbit of an element $x\in X$ is

$${\mathrm{Orb}}_G(x)=\{g\cdot x\mid g\in G\}.$$

We will think of an orbit as an equivalence class: two points $x,y\in X$ are in the same orbit if there is some $g\in G$ such that $g\cdot x=y$. The set of orbits is denoted $X/G$.

This means that the orbit of an action breaks $X$ apart into orbits, and we can for example, count $|X|$ by counting the number of orbits and the size of each orbit.

Definition

Let $G\curvearrowright X$ be a group action. The stabilizer of an element $x\in X$ is the set of group elements that fix $x$ under the action:

$${\mathrm{Stab}}_G(x)=\{g\in G\mid g\cdot x=x\}.$$

Example

Let $G$ act on $S$. An element $s\in S$ is a fixed point of the action if $g\cdot s=s$ for every $g\in G$. Equivalently, $s$ is a fixed point if ${\mathrm{Orb}}_G(s)=\{s\}$, or if ${\mathrm{Stab}}_G(s)=G$.

So the orbit measures how much the point moves; the stabilizer measures how much symmetry remains after we pin down the point. One important difference is that the orbit is a subset of $X$, while the stabilizer is a subgroup of $G$.

Lemma

Let $G$ act on $X$, and let $x\in X$. Then ${\mathrm{Stab}}_G(x)$ is a subgroup of $G$.

Proof

The stabilizer is nonempty because $e_G\cdot x=x$, so $e_G\in {\mathrm{Stab}}_G(x)$. We use the one-step subgroup test. Suppose $a,b\in {\mathrm{Stab}}_G(x)$. Since $b\cdot x=x$, note that the inverse $b^{-1}$ also fixes $x$:

$$b^{-1}\cdot x=b^{-1}\cdot (b\cdot x)=(b^{-1}b)\cdot x=e_G\cdot x=x.$$

Therefore

$$(a{b}^{-1})\cdot x=a\cdot (b^{-1}\cdot x)=a\cdot x=x.$$

Thus $a{b}^{-1}\in {\mathrm{Stab}}_G(x)$, and hence ${\mathrm{Stab}}_G(x)\le G$.

The main counting theorem for actions says that the size of an orbit is controlled by the size of a stabilizer. A useful mnemonic is

LOIS = Length of the Orbit Is the index of the Stabilizer.

LOIS

Let $G$ act on a set $X$. For every $x\in X$,

$$|{\mathrm{Orb}}_G(x)|=[G:{\mathrm{Stab}}_G(x)].$$

Proof

Let $H={\mathrm{Stab}}_G(x)$, and let $\mathcal{L}$ be the set of left cosets of $H$ in $G$. We’ll show that there is a bijection

$$\Phi :\mathcal{L}\to {\mathrm{Orb}}_G(x),\Phi (gH)=g\cdot x.$$

We first check that this is well-defined. If $gH=kH$, then $k^{-1}g\in H$, so

$$(k^{-1}g)\cdot x=x.$$

Applying $k$ to both sides gives $g\cdot x=k\cdot x$.

The same calculation also proves injectivity. If $\Phi (gH)=\Phi (kH)$, then $g\cdot x=k\cdot x$. Applying $k^{-1}$ gives

$$(k^{-1}g)\cdot x=x,$$

so $k^{-1}g\in H$, and therefore $gH=kH$.

Finally, $\Phi$ is surjective by the definition of the orbit: every element of ${\mathrm{Orb}}_G(x)$ has the form $g\cdot x$ for some $g\in G$. Thus $\Phi$ is a bijection, so the number of elements in the orbit is the number of left cosets of the stabilizer.

Exercise

Let $G$ act transitively on a set $X$, and fix $x\in X$. Let $H={\mathrm{Stab}}_G(x)$. Prove that

$$G/H⟶X,gH\mapsto g\cdot x$$

is a well-defined bijection that respects the $G$-actions.

Note that one consequence of LOIS is that for each $x$ in an orbit, the size of the stabilizer is the same. More explicitly, if $y=g\cdot x$, then

$${\mathrm{Stab}}_G(y)={\mathrm{Stab}}_G(g\cdot x)=g{\mathrm{Stab}}_G(x)g^{-1}.$$

Orbit-Stabilizer Theorem

Let $G$ be a finite group acting on a set $X$. For every $x\in X$,

$$|G|=|{\mathrm{Orb}}_G(x)|\cdot |{\mathrm{Stab}}_G(x)|.$$

Proof

By LOIS,

$$|{\mathrm{Orb}}_G(x)|=[G:{\mathrm{Stab}}_G(x)].$$

Since $G$ is finite, Lagrange’s theorem gives

$$[G:{\mathrm{Stab}}_G(x)]=\frac{|G|}{|{\mathrm{Stab}}_G(x)|}.$$

Rearranging gives the result.

Remark

Let $G$ act on a finite set $X$. Choose one representative $x_1,\dots ,x_m$ from each orbit that has more than one element. Then

$$|X|=|\{x\in X\mid x\text{ is fixed}\}|+\sum _{i=1}^m|{\mathrm{Orb}}_G(x_i)|.$$

Using LOIS, we can also write this as

$$|X|=|\{x\in X\mid x\text{ is fixed}\}|+\sum _{i=1}^m[G:{\mathrm{Stab}}_G(x_i)].$$

This is often called the orbit formula. We will use it soon in the special case where a group acts on itself by conjugation.

Exercise

Let $S_n$ act on ${\mathbb{C}}^n$ by permuting coordinates. For $(a_1,\dots ,a_n)\in {\mathbb{C}}^n$, define the elementary symmetric polynomials

$$\begin{array}{rl}{e}_1(a_1,\dots ,a_n)& =a_1+\cdots +a_n,\\ e_2(a_1,\dots ,a_n)& =\sum _{i<j}{a}_i{a}_j,\\ & ⋮\\ e_n(a_1,\dots ,a_n)& =a_1{a}_2\cdots a_n.\end{array}$$

Show that $e_1,\dots ,e_n$ are constant on $S_n$-orbits. Then prove the converse: two points of ${\mathbb{C}}^n$ have the same values of $e_1,\dots ,e_n$ if and only if they lie in the same $S_n$-orbit.

Interpret this as saying that the orbit space ${\mathbb{C}}^n/S_n$ parametrizes monic degree $n$ polynomials by their roots:

$$(a_1,\dots ,a_n)⟼\prod _{i=1}^n(t-a_i).$$

There is also a neat way to count the number of orbits of a group action, called Burnside’s lemma, though it is more accurately due to Cauchy and Frobenius.

Burnside's lemma

Let $G$ be a finite group acting on a finite set $X$. For $g\in G$, let

$$\mathrm{Fix}(g)=\{x\in X\mid g\cdot x=x\}.$$

be all the fixed points of $g$. Then the number of orbits is

$$|X/G|=\frac{1}{|G|}\sum _{g\in G}|\mathrm{Fix}(g)|.$$

Proof

Count the set

$$P=\{(g,x)\in G\times X\mid g\cdot x=x\}$$

enumerating all fixed pairs in two ways. First, if we fix $g\in G$, then the possible $x$ are exactly the elements of $\mathrm{Fix}(g)$. Thus

$$|P|=\sum _{g\in G}|\mathrm{Fix}(g)|.$$

Now count by fixing $x\in X$. The possible $g$ are exactly the elements of ${\mathrm{Stab}}_G(x)$, so

$$|P|=\sum _{x\in X}|{\mathrm{Stab}}_G(x)|.$$

Now group the sum by orbits. If $O$ is an orbit and $x\in O$, then every point of $O$ has a stabilizer of the same size as ${\mathrm{Stab}}_G(x)$. Thus the contribution of this orbit to the sum is

$$|O|\cdot |{\mathrm{Stab}}_G(x)|=|G|$$

by LOIS. Hence each orbit contributes $|G|$, and since there are $|X/G|$ orbits, we get

$$|P|=|G|\cdot |X/G|.$$

Using the first equality for $|P|$ gives the formula.

Example

How many ways are there to color the vertices of a square black or white, up to rotation? Let $C_4=⟨r\mid r^4=1⟩$ act on the set $X$ of all $2^4=16$ colorings by rotating the square. The answer will be the number of orbits — since the orbits identify colorings that are equivalent under rotation.

The identity fixes all $16$ colorings. The rotations $r$ and $r^3$ each fix only the two constant colorings. The rotation $r^2$ fixes the colorings where opposite vertices have the same color, so it fixes $2^2=4$ colorings. By Burnside’s lemma,

$$|X/C_4|=\frac{1}{4}(16+2+4+2)=6.$$

Thus there are $6$ colorings up to rotation.

Exercise

Use Burnside’s lemma to count the number of ways to color the vertices of a regular pentagon with three colors, up to all symmetries of the pentagon.

We can now use these facts to compute with concrete symmetry groups. First, let’s get some definitions:

Definition

An action of $G$ on $X$ is transitive if there is only one orbit, i.e., for every $x,y\in X$, there is some $g\in G$ such that $g\cdot x=y$.

Definition

An action of $G$ on $X$ is faithful if the only group element that fixes every element of $X$ is the identity. Equivalently, the associated homomorphism $\phi :G\to \mathrm{Sym}(X)$ has trivial kernel.

Exercise

Let

$$O(k)=\{Q\in {\mathrm{GL}}_k(\mathbb{R})\mid Q^{T}Q=I\}$$

be the orthogonal group. Let $O(m)\times O(n)$ act on the set $M_{m\times n}(\mathbb{R})$ of real $m\times n$ matrices by

$$(U,V)\cdot A=UA{V}^T.$$

1. Show that this defines a group action. Why does it correspond to rotating the rows and columns of $A$?
2. Show that the eigenvalues of $A^{T}A$, and hence the singular values of $A$, are constant on orbits.
3. Use singular value decomposition to show that two matrices lie in the same $O(m)\times O(n)$-orbit if and only if they have the same singular values.

Lemma

Let $G$ act on a set $X$, and let $\phi :G\to \mathrm{Sym}(X)$ be the associated permutation representation. Then

$$\ker (\phi )=\bigcap _{x\in X}{\mathrm{Stab}}_G(x).$$

In particular, the action is faithful if and only if

$$\bigcap _{x\in X}{\mathrm{Stab}}_G(x)=\{e\}.$$

Proof

An element $g\in G$ lies in $\ker (\phi )$ if and only if $g$ acts trivially on every element of $X$. This means exactly that

$$g\cdot x=x$$

for every $x\in X$, or equivalently, that $g\in {\mathrm{Stab}}_G(x)$ for every $x\in X$.

Example

Let $G$ be the group of rotational symmetries of a cube. We can count $G$ by imagining that we pick up the cube and put it back in the same place. How many ways are there to do this?

$G$ acts on the six faces of the cube. This action is transitive, because I can pick up the cube and rotate it so that any face is on top. The stabilizer of a face consists of the four rotations by $0,\pi /2,\pi ,$ and $3\pi /2$ around the axis passing through the center of $F$ and the center of the opposite face. So

$$|G|=|{\mathrm{Orb}}_G(F)|\cdot |{\mathrm{Stab}}_G(F)|=6\cdot 4=24.$$

The cube has four long diagonals joining opposite vertices. Every rotational symmetry permutes these four diagonals, so we get a homomorphism

$$\rho :G\to S_4.$$

We claim this is actually an isomorphism. This action is faithful: a rotational symmetry that fixes all four long diagonals must fix the whole cube. To see this, consider a line $L=\{A,B\}$ joining two opposite vertices $A$ and $B$. Any $g\in G$ that fixes all four long diagonals must fix $L$ as a set, and thus either fixes both $A$ and $B$ or swaps them. If it fixes $A$ and $B$, rotating around $L$ fixes the other lines. If it swaps $A$ and $B$, this is a rotation around another line $L$ and therefore fixes all the lines. Therefore $\rho$ is injective. Since $|G|=24=|S_4|$, the map $\rho$ is an isomorphism. Thus the rotational symmetry group of the cube is isomorphic to $S_4$.

Example

Let $Y$ be a regular dodecahedron centered at the origin in ${\mathbb{R}}^3$, and let $G$ be the group of orientation-preserving isometries of ${\mathbb{R}}^3$ that send $Y$ to itself:

$$G=\{\alpha :{\mathbb{R}}^3\to {\mathbb{R}}^3\mid \alpha \text{ is an isometry, }\alpha \text{ preserves orientation, and }\alpha (Y)=Y\}.$$

The group $G$ acts on the set of $12$ faces of $Y$. This action is transitive, since any face can be rotated to any other face. If $F$ is a face, then

$$|{\mathrm{Orb}}_G(F)|=12.$$

The stabilizer of $F$ consists of the five rotations around the axis through the center of $F$ and the center of the opposite face:

$$0,\frac{2\pi }{5},\frac{4\pi }{5},\frac{6\pi }{5},\frac{8\pi }{5}.$$

Therefore $|{\mathrm{Stab}}_G(F)|=5$, and the Orbit-Stabilizer Theorem gives

$$|G|=|{\mathrm{Orb}}_G(F)|\cdot |{\mathrm{Stab}}_G(F)|=12\cdot 5=60.$$

The Class Equation

The main goal of this section is to apply the Orbit-Stabilizer Theorem to the action of a group on itself by conjugation. Recall that every group $G$ acts on itself by

$$g\cdot x=gx{g}^{-1}.$$

Definition

Let $G$ be a group. Two elements $g,g^{\prime }\in G$ are conjugate if there exists $h\in G$ such that

$$g^{\prime }=hg{h}^{-1}.$$

The conjugacy class of $g\in G$ is

$$[g{]}_c=\{hg{h}^{-1}\mid h\in G\}.$$

Thus the conjugacy class of $g$ is exactly the orbit of $g$ under conjugation.

Remark

For any group $G$, we have

$$ge{g}^{-1}=e$$

for every $g\in G$. Therefore $[e{]}_c=\{e\}$.

Let us first understand conjugacy classes in symmetric groups. Remember that in symmetric groups, every permutation factors uniquely into a product of disjoint cycles.

Definition

The cycle type of a permutation is the list of lengths of the cycles in its disjoint cycle decomposition, ignoring cycles of length $1$.

Theorem

Two elements of $S_n$ are conjugate if and only if they have the same cycle type.

Exercise

This exercise will prove Theorem symmetric group conjugacy cycle type. Start with the key lemma: conjugation only relabels cycles. That is, if $\sigma \in S_n$, and $i_1,\dots ,i_k$ are distinct elements of $\{1,\dots ,n\}$, then

$$\sigma (i_1{i}_2\cdots i_k){\sigma }^{-1}=(\sigma (i_1)\sigma (i_2)\cdots \sigma (i_k)).$$

Example

The conjugacy classes of $S_4$ are determined by cycle type:

• the identity class $\{e\}$;
• the class of $(12)$, consisting of all transpositions, with $\left(\genfrac{}{}{0ex}{}{4}{2}\right)=6$ elements;
• the class of $(123)$, consisting of all $3$-cycles, with $4\cdot 2=8$ elements;
• the class of $(1234)$, consisting of all $4$-cycles, with $3!=6$ elements;
• the class of $(12)(34)$, consisting of products of two disjoint transpositions, with $3$ elements.

The sizes add to

$$1+6+8+6+3=24=|S_4|.$$

Exercise

Write down all the conjugacy classes of $S_5$, and verify that their sizes add to $|S_5|=120$.

Remark

If $G$ is a nontrivial group, then no conjugacy class is all of $G$. Indeed, $[e{]}_c=\{e\}$, and the conjugacy classes partition $G$.

Definition

Let $G$ be a group and let $S\subseteq G$. The centralizer of $S$ in $G$ is

$$C_G(S)=\{x\in G\mid xs=sx\text{ for all }s\in S\}.$$

If $S=\{a\}$, we write $C_G(a)$ instead of $C_G(\{a\})$.

Definition

Let $G$ be a group and let $S\subseteq G$. The normalizer of $S$ in $G$ is

$$N_G(S)=\{g\in G\mid gS{g}^{-1}=S\}.$$

Exercise

Let $G$ be a group and let $S\subseteq G$. Prove that $C_G(S)$ and $N_G(S)$ are subgroups of $G$.

Lemma

Let $S\subseteq G$. Then $C_G(S)\le N_G(S)$.

Proof

Let $x\in C_G(S)$. For every $s\in S$, we have $xs=sx$, so

$$xs{x}^{-1}=s\text{and}{x}^{-1}sx=s.$$

Therefore $xS{x}^{-1}\subseteq S$ and $x^{-1}Sx\subseteq S$. To prove the reverse containment, let $s\in S$. Since $x^{-1}sx\in S$, we can write

$$s=x(x^{-1}sx)x^{-1}\in xS{x}^{-1}.$$

Hence $S\subseteq xS{x}^{-1}$, so $xS{x}^{-1}=S$. Thus $x\in N_G(S)$.

Remark

If $G$ is abelian, then $C_G(S)=G=N_G(S)$ for every subset $S\subseteq G$.

Exercise

Let $H\le G$, and let $S\subseteq H$. Prove that

$$C_H(S)=C_G(S)\cap H\text{and}{N}_H(S)=N_G(S)\cap H.$$

Exercise

Let $G$ be a group and let $H\le G$.

1. Prove that $N_G(H)/C_G(H)$ is isomorphic to a subgroup of $\mathrm{Aut}(H)$.
2. Deduce that if $H⊴G$, then $C_G(H)⊴G$, and $G/C_G(H)$ is isomorphic to a subgroup of $\mathrm{Aut}(H)$.

Lemma

Let $G$ act on itself by conjugation. For every $g\in G$,

$${\mathrm{Orb}}_G(g)=[g{]}_c,{\mathrm{Stab}}_G(g)=C_G(g),|[g{]}_c|=[G:C_G(g)].$$

Proof

The first equality is the definition of the conjugacy class. For the stabilizer,

$$h\in {\mathrm{Stab}}_G(g)⟺hg{h}^{-1}=g⟺hg=gh⟺h\in C_G(g).$$

The final equality follows from Theorem LOIS.

Corollary

If $G$ is finite, then the size of every conjugacy class divides $|G|$.

Proof

By Lemma conjugacy class orbit stabilizer, a conjugacy class is an orbit. By the Orbit-Stabilizer Theorem, the size of the orbit always divides the size of the group.

Exercise

Let $G$ be a finite group. The commuting probability of $G$ is the probability that two randomly chosen elements of $G$ commute:

$$\mathrm{cp}(G)=\frac{|\{(x,y)\in G\times G\mid xy=yx\}|}{|G{|}^2}.$$

If $k(G)$ is the number of conjugacy classes of $G$, prove that

$$\mathrm{cp}(G)=\frac{k(G)}{|G|}.$$

Compute $\mathrm{cp}(S_3)$ and $\mathrm{cp}(D_4)$.

We now apply the orbit formula to the conjugation action. First we identify the fixed points.

Lemma

Let $G$ act on itself by conjugation. An element $g\in G$ is a fixed point of this action if and only if $g\in Z(G)$.

Proof

Suppose $g\in Z(G)$. Then $g$ commutes with every $h\in G$, so

$$hg{h}^{-1}=(hg)h^{-1}=g(h{h}^{-1})=g.$$

Thus $g$ is fixed by every element of $G$ under conjugation.

Conversely, suppose $g$ is fixed by every element of $G$ under conjugation. Then for every $h\in G$,

$$hg{h}^{-1}=g.$$

Multiplying on the right by $h$ gives $hg=gh$. Hence $g\in Z(G)$.

The Class Equation

Let $G$ be a finite group. For each conjugacy class of size greater than $1$, choose exactly one representative, and call the resulting list $g_1,\dots ,g_r$. Then

$$|G|=|Z(G)|+\sum _{i=1}^r[G:C_G(g_i)].$$

Proof

The conjugacy classes are the orbits of the conjugation action. By Lemma conjugation fixed points center, the fixed points of this action are exactly the elements of $Z(G)$.

Theorem LOIS gives

$$|G|=|Z(G)|+\sum _{i=1}^r|[g_i{]}_c|.$$

For each chosen representative $g_i$, Lemma conjugacy class orbit stabilizer gives

$$|[g_i{]}_c|=[G:C_G(g_i)].$$

Substituting these equalities into the orbit formula proves the class equation.

Remark

If $G$ is abelian, then $Z(G)=G$, so the class equation is not saying much: there are no conjugacy classes of size greater than $1$. The theorem becomes most useful when $G$ is nonabelian, because it forces noncentral conjugacy classes to account for the part of $G$ outside the center.

Exercise

Prove that if $G$ is a nonabelian group of order $21$, then there is only one possible class equation for $G$, up to reordering the noncentral conjugacy-class sizes.

Exercise

Let $p$ be prime, and let $G$ be a finite group of order $p^m$ for some $m>0$. Show $Z(G)$ is not the trivial group.

Lemma

Let $G$ be a group and let $N⊴G$. The conjugation action of $G$ on itself restricts to an action of $G$ on $N$. In particular, $N$ is a disjoint union of conjugacy classes of $G$.

Proof

Define $g\cdot n=gn{g}^{-1}$ for $g\in G$ and $n\in N$. Since $N$ is normal in $G$, the element $gn{g}^{-1}$ lies in $N$, so this rule is well-defined as an action on $N$. The action axioms are inherited from the conjugation action of $G$ on itself.

The orbit of an element $n\in N$ under this restricted action is

$$\{gn{g}^{-1}\mid g\in G\}=[n{]}_c,$$

its conjugacy class in $G$. The orbits of an action partition the set being acted on, so $N$ is a disjoint union of conjugacy classes of $G$.

Remark

Lemma normal subgroup union conjugacy classes says that if $N⊴G$, then every conjugacy class of $G$ that touches $N$ is completely contained in $N$.

This should not be confused with the conjugation action of $N$ on itself. If two elements of $N$ are conjugate by an element of $N$, then they are certainly conjugate by an element of $G$. The converse need not hold: two elements of $N$ might be conjugate in $G$ only by elements outside of $N$.

Exercise

Using the conjugacy classes of $S_4$, prove that the only normal subgroups of $S_4$ are

$$\{e\},V=\{e,(12)(34),(13)(24),(14)(23)\},A_4,S_4.$$

Other Group Actions with Applications

The action on left cosets

Let $G$ be a group and let $H\le G$. Let $\mathcal{L}$ be the set of left cosets of $H$ in $G$:

$$\mathcal{L}=\{xH\mid x\in G\}.$$

When $H$ is normal, this is the quotient group $G/H$. Here we do not assume that $H$ is normal; we are only using $\mathcal{L}$ as a set.

The group $G$ acts on $\mathcal{L}$ by left multiplication. This action is transitive, since every coset is obtained from $H$ by $x\cdot H$. The stabilizer of the coset $H$ is

$${\mathrm{Stab}}_G(H)=\{g\in G\mid gH=H\}=H.$$

This is consistent with LOIS:

$$|{\mathrm{Orb}}_G(H)|=|\mathcal{L}|=[G:H],[G:{\mathrm{Stab}}_G(H)]=[G:H].$$

The associated permutation representation is

$$\phi :G\to \mathrm{Sym}(\mathcal{L}),\phi (g)(xH)=(gx)H.$$

If $[G:H]=n<\infty$, then after choosing a bijection $\mathcal{L}\cong \{1,\dots ,n\}$, we may view this as a homomorphism $\phi :G\to S_n$.

Lemma

Let $G$ be a group and let $H\le G$. Consider the action of $G$ on the set $\mathcal{L}$ of left cosets of $H$, and let $\phi :G\to \mathrm{Sym}(\mathcal{L})$ be the corresponding permutation representation. Then

$$\ker (\phi )=\bigcap _{x\in G}xH{x}^{-1}.$$

In particular, $\ker (\phi )\le H$.

Proof

An element $g\in G$ lies in $\ker (\phi )$ if and only if it fixes every left coset of $H$. Thus

$$\begin{array}{rl}g\in \ker (\phi )& ⟺(gx)H=xH\text{ for every }x\in G\\ & ⟺x^{-1}gx\in H\text{ for every }x\in G\\ & ⟺g\in xH{x}^{-1}\text{ for every }x\in G\\ & ⟺g\in \bigcap _{x\in G}xH{x}^{-1}.\end{array}$$

Taking $x=e$ shows that this intersection is contained in $H$.

The subgroup

$$\bigcap _{x\in G}xH{x}^{-1}$$

is the largest normal subgroup of $G$ contained in $H$. It is often called the core of $H$ in $G$.

Remark

The action of $G$ on the left cosets of $H$ may or may not be faithful. By Lemma coset action kernel, it is faithful if and only if

$$\bigcap _{x\in G}xH{x}^{-1}=\{e\}.$$

If $H⊴G$, then $xH{x}^{-1}=H$ for all $x\in G$, so the kernel is $H$. Thus, for normal $H$, this action is faithful if and only if $H=\{e\}$.

Example

Let $G=S_3$ and let $H=⟨(12)⟩$. The action of $S_3$ on the left cosets of $H$ is faithful. Indeed, if $\omega =(13)$, then

$$\omega H{\omega }^{-1}=\{e,(23)\}.$$

Hence

$$H\cap \omega H{\omega }^{-1}=\{e\}.$$

By Lemma coset action kernel, the kernel of the coset action is contained in this intersection, so the kernel is trivial.

Theorem

Let $G$ be a finite group, and let $H\le G$ have index $p$, where $p$ is the smallest prime divisor of $|G|$. Then $H⊴G$.

Proof

Let $\mathcal{L}$ be the set of left cosets of $H$ in $G$. Since $[G:H]=p$, the action of $G$ on $\mathcal{L}$ gives a homomorphism $\phi :G\to S_p$. Let $N=\ker (\phi )$. By Lemma coset action kernel, we have $N\le H$.

By the First Isomorphism Theorem,

$$[G:N]=|G/N|=|\mathrm{im}(\phi )|.$$

Since $\mathrm{im}(\phi )\le S_p$, Lagrange’s theorem implies that $[G:N]$ divides $p!$. Also $[G:N]$ divides $|G|$. Therefore $[G:N]$ divides $\gcd (|G|,p!)$. Because $p$ is the smallest prime divisor of $|G|$, this greatest common divisor is $p$. Hence

$$[G:N]=1\text{or}[G:N]=p.$$

The first option is impossible because $N\le H<G$. Thus $[G:N]=p$. Since $N\le H$ and $[G:H]=p$, we must have $N=H$. Therefore $H=\ker (\phi )$, so $H$ is normal in $G$.

This generalizes the earlier exercise that every subgroup of index $2$ is normal.

The action on subgroups by conjugation

Another useful action comes from conjugating subgroups. If $H\le G$ and $g\in G$, then

$$gH{g}^{-1}=\{gh{g}^{-1}\mid h\in H\}$$

is again a subgroup of $G$, and the map $H\to gH{g}^{-1}$ given by $h\mapsto gh{g}^{-1}$ is a bijection. Thus conjugate subgroups have the same cardinality. In particular, if $H$ is the unique subgroup of $G$ of order $|H|$, then $gH{g}^{-1}=H$ for every $g\in G$, so $H⊴G$.

Let

$$\mathcal{S}(G)=\{H\mid H\le G\}$$

be the set of all subgroups of $G$. Then $G$ acts on $\mathcal{S}(G)$ by

$$g\cdot H=gH{g}^{-1}.$$

Definition

Two subgroups $A$ and $B$ of a group $G$ are conjugate if there exists $g\in G$ such that

$$A=gB{g}^{-1}.$$

Equivalently, two subgroups are conjugate if they lie in the same orbit for the action of $G$ on $\mathcal{S}(G)$ by conjugation.

Lemma

Let $G$ be a group and let $H\le G$. The number of subgroups of $G$ conjugate to $H$ is $[G:N_G(H)]$.

Proof

The subgroups conjugate to $H$ are exactly the elements in the orbit of $H$ under the conjugation action of $G$ on $\mathcal{S}(G)$. The stabilizer of $H$ under this action is

$$\{g\in G\mid gH{g}^{-1}=H\}=N_G(H).$$

By LOIS, the size of the orbit is $[G:N_G(H)]$.

The normalizer $N_G(H)$ is the largest subgroup of $G$ in which $H$ is normal. Indeed, $H⊴N_G(H)$, and if $H⊴K\le G$, then every element of $K$ normalizes $H$, so $K\le N_G(H)$. In particular,

$$H⊴G\text{if and only if}{N}_G(H)=G.$$

These actions on cosets and subgroups both turn group-theoretic structure into permutation representations. In the next chapter we linearize this idea: an action on a finite set gives a representation by letting $G$ permute basis vectors.