You start with two equations:
- the rate of change in population of the predator with respect to time as a function of the populations
- the rate of change in population of the prey with respect to time as a function of the populations
You then form a matrix representation of the functions and find the eigenvalues and eigenvectors, which are used to plot a phase diagram (a graphical representation of the solution to the differential equations). The magnitude and sign of the eigenvalues determine the stability of the fluctuations between the populations of the two species and the rate of growth of each population. Of the several different possibilities for the phase diagrams, there are two relevant situations: concentric circles, or saddle point.
A saddle point is unstable and suggests the extinction of one of the populations, as shown below.
This is a topic that was covered in Math 4B, but you can also read about it here.