We show that there are two different families of (weakly) orthogonal polynomials associated to the quasi-exactly solvable Razavy potential V(x) = (cos 2x-M)² (>0, M). One of these families encompasses the four sets of orthogonal polynomials recently found by Khare and Mandal, while the other one is new. These results are extended to the related periodic potential U(x) = -(cos 2x-M)², for which we also construct two different families of weakly orthogonal polynomials. We prove that either of these two families yields the ground state (when M is odd) and the lowest-lying gaps in the energy spectrum of the latter periodic potential up to and including the (M - 1)th gap and having the same parity as M - 1. Moreover, we show that the algebraic eigenfunctions obtained in this way are the well known finite solutions of the Whittaker-Hill (or Hill's three-term) periodic differential equation. Thus, the foregoing results provide a Lie-algebraic justification of the fact that the Whittaker-Hill equation (unlike, for instance, Mathieu's equation) admits finite solutions.