We consider a system of q diffusing particle species A₁, A₂, ..., A_q that are all equivalent under a symmetry operation. Pairs of particles may annihilate according to A_i + A_j → 0 with reaction rates k_ij that respect the symmetry, and without self-annihilation (k_ii = 0). For spatial dimension d > 2 mean-field theory predicts that the total particle density decays as ρ(t) ~ t ^-1, provided that the system remains spatially uniform. We determine the conditions on the matrix k under which there exists a critical segregation dimension d_seg below which this uniformity condition is violated; the symmetry between the species is then locally broken. We argue that in those cases the density decay slows down to ρ(t) ~ t ^-d/d_seg for 2<d < d_seg. We show that when d_seg exists, its value can be expressed in terms of the ratio of the smallest to the largest eigenvalue of k. The existence of a conservation law (as in the special two-species annihilation A + B → 0), although sufficient for segregation, is shown not to be a necessary condition for this phenomenon to occur. We work out specific examples and present Monte Carlo simulations compatible with our analytical results.