For certain types of quantum graphs we show that the random matrix form factor can be recovered to at least third order in the scaled time τ from periodic-orbit theory. We consider the contributions from pairs of periodic orbits represented by diagrams with up to two self-intersections connected by up to four arcs and explain why all other diagrams are expected to give higher-order corrections only. For a large family of graphs with ergodic classical dynamics the diagrams that exist in the absence of time-reversal symmetry sum to zero. The mechanism for this cancellation is rather general which suggests that it also applies at higher orders in the expansion. This expectation is in full agreement with the fact that in this case the linear-τ contribution, the diagonal approximation, already reproduces the random matrix form factor for τ < 1. For systems with time-reversal symmetry there are more diagrams which contribute at third order. We sum these contributions for quantum graphs with uniformly hyperbolic dynamics, obtaining +2τ³, in agreement with random matrix theory. As in the previous calculation of the leading-order correction to the diagonal approximation we find that the third-order contribution can be attributed to exceptional orbits representing the intersection of diagram classes.