How Fast is Fast Enough? Industry Perspectives on the Use of Large-eddy Simulation in Wind Energy

The use of graphics processing units (GPUs) has facilitated unprecedented performance gains for computational fluids dynamics in recent years. In many industries this has enabled the integration of large-eddy simulation (LES) in the engineering practice. Flow modelling in the wind industry though still primarily relies on models with significantly lower fidelity. This paper seeks to investigate the reasons why wind energy applications of LES are still an exception in the industrial practice. On that account, we present a survey among industry experts on the matter. The survey shows that the large runtimes and computational costs of LES are still seen as a main obstacle. However, other reasons such as a lack of expertise and user experience, the need for more validation, and lacking trust in the potential benefits of LES reveal that computational efficiency is not the only concern. Lastly, we present an exemplary simulation of a generic offshore wind farm using a GPU-resident Lattice Boltzmann LES framework. The example shows that the runtime requirements stated by a large part of the respondents can already now be fulfilled with reasonable hardware effort.


Introduction
Larger wind farms, bigger wind turbine rotors, wind farm control, and the expansion of wind power to complex or forested terrain are some of the main trends in the wind industry today.All of these developments are crucial for the increasing contribution of wind power to the energy transition.At the same time, they bring about significant modeling challenges [1].These challenges are often model-specific but affect most low-to mid-fidelity engineering models in one way or another.Regardless of the modelling approach, they typically originate from inherent model limitations (e.g., steady-state or linearized formulations), simplifying assumptions that neglect certain physical phenomena (e.g, missing induction, or surface layer scaling) or insufficient tuning of empirical parameters.
Large-eddy simulation (LES) can address many of the current modeling challenges as it provides a transient numerical solution of the (filtered) governing non-linear equations, while involving comparatively little empiricism.Today, LES is the go-to method for fundamental numerical investigations of atmospheric boundary layers [2], wind turbine wakes, and wind farm dynamics [3].Nonetheless, LES has traditionally played only a minor role for wind turbine or farm modeling in the industrial practice.The main reason usually given for this is the immense computational demand of the method.
Over the past two decades, numerous new many-core processor concepts have emerged, with graphics processing units (GPUs) being the most prominent example.GPUs have enabled unprecedented efficiency gains for parallelizable computing tasks and are increasingly adopted for computational fluid dynamics (CFD) in recent years [4,5].This includes classical incompressible Navier-Stokes solvers that are being ported from CPUs to GPUs [6,7].More specialized frameworks employ numerical schemes that are particularly suited for GPU applications.This typically implies a minimization of memory accesses using, for instance, Cartesian grids, weakly compressible formulations, explicit solvers, or a combination of all [8,9].Another scheme that inherently possesses these features is the lattice Boltzmann method (LBM) [10,11].And, the lattice Boltzmann community has been among the early-adopters of GPUs for high-performance computing [12][13][14].Developments in the field of GPU-based CFD also include LES solvers with specific capabilities for atmospheric applications and wind energy.See, for instance, the highresolution weather forecasting frameworks GALES [15,16] or FastEddy [17] as well as Navier-Stokes or LBM solvers with specific wind energy capabilities as presented by Lopez et al. [18], Asmuth et al. [19], or Schottenhamml et al. [20].
GPU-resident CFD frameworks have enabled engineering applications of LES in various industries over the past years.Today, most major vendors of CFD software offer GPUbased solver packages for car or building aerodynamics, aerospace applications, or naval hydrodynamics, to name a few.In that sense, we raise the provoking question: why is LES still an exception and not the rule when it comes to typical wind energy flow modeling applications like wind resource assessment (WRA) or AEP (annual energy production) predictions?Undoubtedly, the answers to this question are manifold.The most obvious assumption, however, is that computational efficiencies are still not sufficient for production runs of such applications.Thus, How fast is fast enough? arises as an obvious follow-up question.Other reasons might be far less technical and relate to issues such as the bankability of model predictions or the integration of new models into established work flows.From an academic research perspective, it is therefore important not only to identify the reasons for this, but also to determine whether the remaining barriers for the adoption of LES can be overcome through further research efforts.
With this paper we intend to bring forward the exchange between industry and fundamental academic research on the applicability of LES.The center of this study are the results of a questionnaire-based survey among industry experts in wind power modeling.By means of this survey we, firstly, aim to get a representative overview of the current modeling practices in the industry.Secondly, we enquire the main reasons hindering the use of high-fidelity models.And, thirdly, we seek to quantify the run-time requirements for LES to become more applicable.Lastly, we present an exemplary wind farm simulation using a GPU-resident LBM framework.This case shall help to illustrate how the capabilities of modern LES approaches compare to the runtime requirements of the industry.The remainder of this paper is organised as follows.In Sec. 2 we describe some general aspects of the design and conduction of the survey.Sec. 3 provides information about the background of the respondents.The main results of the survey are outlined in Sec. 4. In Sec. 5 we discuss the exemplary LES case.We conclude with a discussion and final remarks in Sec. 6.

Survey Design
The survey was conducted online using an openly accessible, anonymous questionnaire.A link to the questionnaire was send out directly via email to industry experts and shared on social media (Twitter and LinkedIn) and in topic-specific online forums.Thus, there was no strict beforehand control if the respondents met the targeted requirements in expertise.The latter was merely controlled retrospectively based on the answers to initial questions about the respondent's experience in wind energy and CFD.The questionnaire comprised a total of 11 questions covering the aforementioned areas of interest as well as the background of the respondents.The majority of the questions were to be answered by single-or multiple-choice as well as Likert-scales ranging from 1 to 5. A final free comment section allowed for general remarks on the topic.A total of 48 respondents participated in the survey between October 10 and December 7, 2022.Unfortunately, the general significance of this sample for the wind energy community is difficult to quantify.To our best knowledge, the most specific available employment statistics of the wind industry include categories such as R&D or Engineering whereof WRA, load assessment, forecasting, etc., are presumably only a small fraction.It can, however, be said that random sub-samples of 50% of the responses show similar trends as the whole sample.Therefore, we expect the presented results to be sufficiently representative for the purpose of this study.In the remainder of this paper direct quotations of either questions or answers from the survey will the given in quotation marks ("...").

Background of the Respondents
An overview of the background of the respondents is shown in Fig. 1.With more than 40%, the majority is active as consultants, followed by 32% working in the field of wind farm development and about 11% for original equipment manufacturers (OEMs).Overall, it can be said that almost all types of industrial stakeholders in the field of turbine and farm modeling are represented in the sample.Apart from the software user side, this also includes a small fraction from companies which develop and sell wind farm modeling software.The only company type not represented in the survey are certifying bodies (not shown in Fig. 1a but an option in the survey).It is possible, though, that employees from certifiers categorized themselves as consultants due to the common overlap of business areas.As for the professional experience, more than 50% look back at 10 or more years in the wind industry, while less than 10% have less than 2 years of experience.When it comes to the expertise in CFD, only about 6% have "little to no understanding of CFD".At the same time, these particular respondents have at least 2 to 5 years of experience in wind energy.We therefore still assume a sufficient overall level of expertise.Another 32% have a basic understanding of CFD, specified in the survey as "e.g., Bachelor or Master's courses".Another 34% have advanced knowledge referring to "several years [of] actively using CFD" and 23% even have expert knowledge having "e.g., actively [been] developing CFD software, [or a] PhD in CFD".In summary, we consider the background and expertise of the respondents suitable to obtain profound assessments for this study.

Survey Results
In the following we outline the results from the main part of the survey.Sec.4.1 discusses the current modelling practices including the limiting factors for the use of CFD.Sec.4.2 is concerned with the current requirements needed for an adoption of LES, and its potential benefits.Lastly, in Sec.4.3 we present the respondent's definitions for a potential use-case for LES and the corresponding run-time requirements.

Modelling Practices
Fig. 2a shows which types of model are being used "on a regular basis" in the companies of the respondents.Note, that this can include any type of wind-energy-related flow modelling task including, e.g., WRA, AEP or load calculations.Not surprisingly, engineering wake models (EWMs) are the most widespread model type with 83%.With 63% and 48%, respectively, also RANS (Reynolds-averaged Navier-Stokes) models and the Weather Research and Forecasting model (WRF) [21] are frequently used by a large fraction.Interestingly, mid-fidelity models that can be used for wake simulations, i.e., dynamic wake meandering (DWM) models, vortex models or linearized CFD codes (see, for instance, [22], [23][24][25] or, [26], respectively), are being used to a significantly lesser extent.Indeed, 23% are regularly using LES, and another 13% WRF-LES.
At this point, it should be clarified that the differentiation between "LES" and "WRF-LES" was originally intended to distinguish between the computationally more expensive wake-resolving LES applications, like typical academic farm flow studies using actuator disk and line models, and micro-meteorological LES applications.However, later discussions with industry experts revealed that some also considered the latter as "LES".A clear differentiation between the two is therefore difficult and would have required a more precise definition in the survey.Large run times (66%) and the large computational cost (79%) are given as the main reasons limiting a wider use of CFD, as shown in Fig. 2b.This is clearly in line with the common notion mentioned earlier.The next important limiting factor, with 42%, is a lack of expertise.Figure 3: (a) Importance of aspects that need to be fulfilled in order to consider the use of LES, ranked on a scale from 1 to 5, with 1 referring to "not important at all/the status quo is sufficient" and 5 referring to "absolutely necessary/requires major improvements".(b) Modelling challenges that would benefit from the use LES, ranked on a scale 1 to 5, with 1 referring to "not relevant at all/low-fidelity models are sufficient" and 5 being "high relevance/lower-fidelity models are inadequate".Results illustrated as diverging stacked bar charts (total length referring to 100%) centered around the middle of category 3. Ticks on the vertical axis mark intervals of 25%.
This aspect is particularly worth highlighting.Ultimately, any model requires trained staff to run and evaluate it.Such personnel might become more scarce and more expensive to employ the higher the model fidelity [27].Another significant limitation are "unclear benefits" with 33%.Strictly speaking this aspect is not a limiting factor per se.It rather implies that lowerfidelity models are either considered to be sufficiently accurate and/or that high-fidelity models are not expected to outperform more simple models in order to justify the higher cost.Lastly, bankability issues are seen as a limiting factor by 23%.An example for this are requirements by investors to compare AEP predictions (e.g., from a developer) against model results from a third party.In such a case it might not be worth running a high-fidelity model if the third party does not do so either.

Requirements for the Use of LES and Its Potential Benefits
As a counter part to the limiting factors discussed above, participants were asked to rate the importance of potential improvements that would lead them to consider using LES.The results are depicted in Fig. 3a.The highest importance is given to "faster run times" and "lower computational cost".This is consistent with the number of mentions of the corresponding limiting factors, albeit with a reversed ranking.The former two are closely followed by "more validation".As to that, it is evident that LES, despite being well-established in fundamental research, has not been validated for farm simulations or WRA to the same extent as common engineering models.After all, established models in the industrial practice have been used for years if not decades.This also implies countless internal validations and valuable user experience.As for LES, such experience only exists in a handful of specialised companies and in academia.offshore, < 10 offshore, 10 -50 offshore, 50 -100 offshore, > 100 onshore (complex), < 10 onshore (complex), 10 -50 onshore (complex), 50 -100 onshore (complex), > 100 onshore (flat), < 10 onshore (flat), 10 -50 onshore (flat), 50 -100 onshore (flat), > 100 (a) Typical use-cases The high importance given to this requirement might therefore also explain the aforementioned notion that the benefits remain unclear.In addition to the quantitative data of the survey, several respondents explicitly mentioned this aspect in the comment section. 1 Less important requirements for the adoption of LES are the "user-friendliness of open-source models" and the "availability of commercial models".
In the second question of this section the respondents were asked to rank if they expect benefits from using LES for some of the current modelling challenges; see Fig. 3b.Here, the relevance of LES for each modelling challenge was inversely linked to the adequacy of lower fidelity models.The highest relevance was given to wind farm wakes, followed by complex terrain.Overall, the majority of the respondents expects some or even large benefits from using LES for all given use cases.For each challenge at least 84% stated a relevance of 3 or more.

Runtime Requirements for a Potential Use-case
Lastly, the respondents were asked to specify a typical case for wind farm modelling, where they expect benefits from using high-fidelity models.Subsequently, they were asked to specify the runtime that would be acceptable for this case.The results are shown in Fig. 4. The case specifications included a differentiation between offshore, onshore (flat) and onshore (complex), as well as the amount of turbines.The primary intention related to these specifications was to estimate the computational demand of such a typical case, i.e. the size of the domain, the grid resolution and the time required to obtain statistically converged results.For instance, in a flat onshore or offshore case, respectively, the inflow can be imposed a few boundary-layer heights upstream of the wind farm and can be pre-computed in a precursor simulation; see, e.g., [28].On the other hand, a complex terrain case will require a significantly larger fetch upstream of the farm in order to capture the footprint of all relevant terrain features [29].A sufficient grid resolution within the farm then again depends on the turbine diameter D as a characteristic length scale of the turbine, and the boundary layer characteristics, first of all the stratification [30].Thus, for smaller turbines the former will be the determining factor for the resolution (a sufficient resolution for the wake is finer than required for the boundary layer) while for large modern offshore turbines it tends to be the latter (the wake is adequately resolved with a resolution that is sufficient for the boundary layer).
The two most mentioned cases are offshore with more than 100 turbines which can also comprise several wind farms, and onshore with complex terrain and 10 to 50 turbines.Generally, the large amount of mentions of complex terrain cases is not surprising given the well-known challenges of modelling wind farms in such conditions.The particular case of 10 to 50 turbines simply reflects the typical wind farm size being build onshore today.Similarly, large modern offshore farms can comprise more than 100 turbines.Furthermore, the interaction of several wind farms is becoming an important modeling aspect (as discussed above) explaining the dominant offshore case specification.
Overall, no clear trend can be observed for the runtime requirements.For about 23%, 6 to 12 h would already be sufficient.The possibility of such overnight runs has often been stated as a critical requirement for the applicability of CFD in general [8].Yet, 27%, for instance, require runtimes between 10 min and 1 h, and another 27% require even less than 10 min.Furthermore, it should be emphasized that there is no clear correlation between the type of case specified earlier and the required runtime.We therefore assume that the runtime requirement rather relates to the use-case envisioned by the respondent.An extreme example could be a layout optimisation that requires hundreds to thousands of realizations of a single farm.In contrast, LES might just be used as a complement to existing low-fidelity models.In that case, only a handful of simulations would be needed and larger runtimes might be acceptable.

Current Capabilities of GPU-resident LES frameworks
To complement the results from the survey, we provide a comparison of the computational performance of a GPU-resident LES framework with the above requirements.For the sake of brevity, we limit the comparison to a single simulation of a generic offshore wind farm with an average size by today's standards.The case size is chosen such that a grid with moderate resolution can still be computed on a single GPU, in order to illustrate the computational possibilities with minimal hardware usage.

Case Description and Numerical Set-up
The farm consists of 64 turbines (DTU 10MW reference wind turbine [31]) with a rated power of 10MW, diameter D = 198 m, and hub height h = 119 m.The turbines are placed in an array of 8 × 8, with a stream-wise spacing of 7 D and a lateral spacing of 5 D. The simulations are performed with the LBM solver VirtualFluids using the parametrized cumulant LBM [32].The sub-grid scales are modelled with the QR eddy-viscosity model by Verstappen and coworkers [33,34] with a model coefficient C = 1/3.The computational domain measures 20 km in the stream-wise direction x, 15 km in the lateral direction y, and 1.6 km in the vertical direction z.The isotropic Cartesian grid has a spacing of ∆x = 10m in the lowest 500 m of the domain.The 500 m above and the uppermost 600 m of the domain are coarsened by a factor of two with respect to the respective grid level below.The grid is similarly coarsened in the stream-wise direction in the last 500 m before the outlet.Overall, the grid comprises 187 • 10 6 grid points.The simulation is run at a Mach number Ma = 0.1 referring to a time-step of ∆t = XXX on the finest grid level.The surface shear-stress is modelled using Monin-Obukhov similarity theory and applied with the inverse momentum exchange method.Further details thereupon can be found in [35].A free-slip boundary condition is applied at the top of the domain.The undisturbed inflow refers to an isothermal pressure-driven boundary layer (with friction  velocity u * = 0.425 m s −1 and roughness length z 0 = 0.01 m) that is pre-generated in a precursor simulation.The mean velocity and turbulence intensity at hub height are u 0 = 11.2 m s −1 and TI = 6.1%, respectively.The turbines are modelled with an actuator line model [19].Statistics are gathered for 6000 s (corresponding to approximately 3.3 domain flow-through-times) after an initial spin-up of 2000 s.A compilation of some of the simulation results is shown in Fig. 5.

Computational Performance
The case is run in single precision floating point format (FP32) on an NVidia A100 GPU with 40 GB device memory.The thermal design power (TDP) of the GPU is 250 W. This is similar to current HPC CPUs.Therefore, it can be assumed that the total energy consumption and operating costs per hour of wall time are of a similar order of magnitude.
The simulation ran with an average performance of 3404 MNUPS (Million Node Updates Per Second).This refers to a total runtime of 5112 s and implies a ratio of simulated time to wall time of 1.56.Hence, running on a single GPU this case already satisfies the requirements set by about 45% of the respondents.Further reductions of the runtime can obviously be achieved by parallelising the case on more GPUs.An additional comparison of the computational performance of this particular case against classical LES frameworks is omitted here for the sake of brevity.Nonetheless, based on previous studies we can typically expect efficiency differences of two orders of magnitude and more when compared to CPU-based Navier-Stokes solvers.See, e.g., [20,36,37].

Discussion and Concluding Remarks
The advent of novel many-core processors and the development of numerical frameworks that efficiently leverage the capabilities of such hardware are expanding the possible applications of LES.With this study we set out to capture the current views of industry wind energy modelling experts on LES and its future perspectives.In the following we shall summarize the main findings from the survey and present the main conclusions concerning future research needs.
Expectedly, LES is found to be an exception in the industrial practice.However, the survey also shows that most industry experts do anticipate large benefits from an adoption of the method for the current modelling challenges.The main obstacles for this adoption seen by the industry experts are the large runtimes and costs.As for classical CPU-based calculations, this consideration is not surprising, even considering today's relatively cheap and easy access to large-scale cloud computing services.However, leveraging modern GPUs and suitable numerical frameworks can increase computational efficiencies by up to two orders of magnitude [17,38].In this way, the runtime requirements, at least for small to moderate case sizes, can already now be met with minimal hardware use, as illustrated by the case shown above.Similar conclusions can be drawn from other recent studies.See, e.g., [18,37,39,40].The awareness of many of the respondents for these computational capabilities still appears rather low, judging from the survey results and some of the responses in the comment section.Still, further improvements in computational efficiency might be required to enable sufficiently fast and cheap simulations for a wider range of applications.For instance, large offshore cases including several wind farms imply grid sizes that are at least one order of magnitude larger than the case shown earlier [41].This survey only quantified what runtimes would be acceptable for such applications, not which cost.However, the latter would be the decisive criterion for determining which computational efficiency is ultimately sufficient.
From a research perspective, the second important aspect highlighted by the survey is the call for more validation.On the one hand, this requirement might originate from the fact that there is little practical experience in the industry with LES.On the other hand, applications of LES to complex real-world scenarios are generally challenging and experience arguably limited, even in academia.Even after 20 years of fundamental academic research on LES in wind energy, only a handful of studies validated LES against full-scale measurements of wind turbines or farms including comparisons of inflow and wake statistics as well as the turbine response.Comprehensive validations involving such detailed comparisons then again highlight various persisting challenges [42][43][44].The need for more validation mentioned by the respondents thus underlines the conclusions of many recent academic investigations.Furthermore, more validation studies will help to demonstrate which applications actually benefit the most from the use LES and help as guidance for industry modellers to choose the right model.This can also help to address the issue that the benefits of high-fidelity models are still unclear to a notable fraction of the respondents.
A final issue to be addressed to increase the use of high-fidelity models is the lacking expertise in the industry (stated by 42% of the respondents).An immediate solution to the issue obviously requires action by the industry which needs to invest in building this expertise by training or hiring staff.Nonetheless, academia can actively support such efforts by promoting education in the field and providing accessible and concise disseminations of state-of-the-art research.In light of the unawareness discussed above, the latter can obviously be improved.

Figure 1 :
Figure 1: Background of the survey respondents.(a) main field of activity of the company, (b) years of experience in the field of wind energy, and (c) expertise in CFD.

Figure 2 :
Figure 2: Survey results concerning the modeling practices in the companies of the respondents.(a)Model types that are being used on a regular basis, and (b) the main limiting factors for a wider use of CFD in the company.Both questions allowed for multiple answers.Answers in parentheses were given by the respondents in the category "other".
f a s t e r r u n t i m e s l o w e r c o m p u t a t i o n a l c o s t m o r e v a l i d a t i o n u s e r -f r i e n d .o p e n -s o u r c e m o d .a v a i l .c o m m e r .m o d .(a) Requirements for the use of LES c o m p l e x t e r r a i n s t r a t i fi c a t i o n b l o c k a g e a n d g r a v i t y w a v e s w i n d f a r

Figure 4 :
Figure 4: Specifications of a potential use case for LES.(a) classification of the case in terms of offshore, onshore (flat) and onshore (complex), combined with the amount of turbines.Note, that more than 100 turbines can even imply several wind farms.(b) run time requirements for the case.

Figure 5 :
Figure 5: Overview of a selection of results from the exemplary LES case.(a) contour plot of the instantaneous stream-wise velocity u at hub height.(b) mean power Pi of the turbines in the fourth column of the array normalised with the mean power of the first turbine P0 = 9.1 MW.(c) mean stream-wise velocity ū at hub height along the fourth column of turbines.