Cluster abundance measurements are among the most sensitive probes of the amplitude of matter fluctuations in the universe, which in turn can help constrain other cosmological parameters, like the dark energy equation of state or neutrino mass. However, difficulties in calibrating the relation between the cluster observable and the halo mass, and the lack of completeness information, make this technique particularly susceptible to systematic errors. Here we argue that a cluster abundance analysis using statistical weak lensing on the stacked clusters leads to a robust lower limit on the amplitude of fluctuations. The method compares the average weak lensing signal measured around the whole cluster sample to a theoretical prediction, assuming that the clusters occupy the centres of all of the most massive halos above some minimum mass threshold. If the amplitude of fluctuations is below a certain limiting value, there are too few massive clusters in this model and the theoretical prediction falls below the observations. Since any effects that modify the model assumptions can only further decrease the prediction of the model in the context of this method, the limiting amplitude becomes a robust lower limit. Here, we apply it to a volume limited sample of 16 000 group/cluster candidates identified from isolated luminous red galaxies (LRGs) in the Sloan Digital Sky Survey (SDSS). We find σ₈(Ω_m/0.25)^0.5>0.62 at the 95% confidence level after taking into account observational errors in the lensing analysis. While this is a relatively weak constraint, both the scatter in the LRG luminosity–halo mass relation and the lensing errors are large. The constraints could improve considerably in the future with more sophisticated cluster identification algorithms and smaller errors in the lensing analysis. We argue that the existence of a lower limit from cluster abundance is rather general, and demonstrate that Malmquist bias dominates over Eddington bias in this type of analysis.