An important emerging scientific issue is the real time filtering through observations of noisy signals for nonlinear dynamical systems as well as the statistical accuracy of spatio-temporal discretizations for filtering such systems. From the practical standpoint, the demand for operationally practical filtering methods escalates as the model resolution is significantly increased. For example, in numerical weather forecasting the current generation of global circulation models with resolution of 35 km has a total of billions of state variables. Numerous ensemble based Kalman filters (Evensen 2003 Ocean Dyn. 53 343–67; Bishop et al 2001 Mon. Weather Rev. 129 420–36; Anderson 2001 Mon. Weather Rev. 129 2884–903; Szunyogh et al 2005 Tellus A 57 528–45; Hunt et al 2007 Physica D 230 112–26) show promising results in addressing this issue; however, all these methods are very sensitive to model resolution, observation frequency, and the nature of the turbulent signals when a practical limited ensemble size (typically less than 100) is used. In this paper, we implement a radical filtering approach to a relatively low (40) dimensional toy model, the L-96 model (Lorenz 1996 Proc. on Predictability (ECMWF, 4–8 September 1995) pp 1–18) in various chaotic regimes in order to address the 'curse of ensemble size' for complex nonlinear systems. Practically, our approach has several desirable features such as extremely high computational efficiency, filter robustness towards variations of ensemble size (we found that the filter is reasonably stable even with a single realization) which makes it feasible for high dimensional problems, and it is independent of any tunable parameters such as the variance inflation coefficient in an ensemble Kalman filter.

This radical filtering strategy decouples the problem of filtering a spatially extended nonlinear deterministic system to filtering a Fourier diagonal system of parametrized linear stochastic differential equations (Majda and Grote 2007 Proc. Natl Acad. Sci. 104 1124–9; Castronovo et al 2008 J. Comput. Phys. 227 3678–714); for the linear stochastically forced partial differential equations with constant coefficients such as in (Castronovo et al 2008 J. Comput. Phys. 227 3678–714), this Fourier diagonal decoupling is a natural approach provided that the system noise is chosen to be independent in the Fourier space; for a nonlinear problem, however, there is a strong mixing and correlations between different Fourier modes. Our strategy is to radically assume for the purposes of filtering that different Fourier modes are uncorrelated. In particular, we introduce physical model errors by replacing the nonlinearity in the original model with a suitable Ornstein–Uhlenbeck process. We show that even with this 'poor-man's' stochastic model, when the appropriate parametrization strategy is guided by mathematical offline test criteria, it is able to produce reasonably skilful filtered solutions. In the highly turbulent regime with infrequent observation time, this approach is at least as good as trusting the observations while the ensemble Kalman filter implemented in a perfect model scenario diverges. Since these Fourier diagonal linear filters have large model error compared with the nonlinear dynamics, an essential part of the study below is the interplay between this error and the mathematical criteria for a given linear filter in order to produce skilful filtered solutions through the radical strategy.