Open code and data are not enough: understandability as design goal for energy system models

Energy system models do not represent natural processes but are assumption-laden representations of complex engineered systems, making validation practically impossible. Post-normal science argues that in such cases, it is important to communicate embedded values and uncertainties, rather than establishing whether a model is ‘true’ or ‘correct’. Here, we examine how open energy modelling can achieve this aim by thinking about what ‘a model’ is and how it can be broken up into manageable parts. Collaboration on such building blocks—whether they are primarily code or primarily data—could become a bigger focus area for the energy modelling community. This collaboration may also include harmonisation and intercomparison of building blocks, rather than full models themselves. The aim is understandability, which will make life easier for modellers themselves (by making it easier to develop and apply problem-specific models) as well as for users far away from the modelling process (by making it easier to understand what is qualitatively happening in a model—without putting undue burden on the modellers to document every detail).


Introduction
What is a model?A dictionary would say something like: a simplified representation of a system, often on a smaller scale than the original it represents.This interpretation works well when thinking of an architectural model of a building, a model of the solar system, or a weather forecasting model.With energy system models, the story is more complicated.Their purpose is less to understand and predict a natural phenomenon or to represent an engineered system like a single building.Instead, it is to aid decision-making when planning and operating a complex engineered system, tightly interwoven not just with natural phenomena (e.g.weather influence on electricity demand and supply) but with the economy and society, and thus with human behaviour and political decision-making.Not surprisingly, there are debates on the use of models that claim to objectively and mathematically summarise these messy facets of reality (Anderson and Jewell 2019).Pfenninger et al (2017) asserted that open energy system modelling is important because it would lead to improved scientific quality and allow sharing the burden of model development across teams and institutions.We can define open modelling as consisting of three aspects: free and open-source computer code, either under a copyleft license like the GPL or a permissive one like the MIT license, open data, for which comparable license choices like the different creative commons licenses exist (Morrison 2022), and open access, which means cost-free access to research reports and papers (Morrison 2018, Pfenninger et al 2018).These are both practical issues, i.e. the ability to obtain and examine these artefacts at no cost, but importantly legal ones: the ability to legally do so, and to reuse or expand on prior work without consent from the original authors.Given its growing prominence, modellers seem to have found value in openness (Ringkjøb et al 2018, Chang et al 2021, Scheller et al 2021).However, energy system models exist at the science-policy boundary and are useful only to the extent to which they can help improve decision-making.The idea of post-normal science (Funtowicz and Ravetz 1993) is useful here: science that sets itself the goal of understanding complex human-natural systems where there are no controlled laboratory conditions, where decision stakes and uncertainty are high, and where a diversity of values and opinions exist.This includes the problem of eliminating greenhouse gas emissions from the world's energy supply.Funtowicz and Ravetz argued that in post-normal science, researchers should focus on making the values underlying their work explicit, and on managing rather than eliminating uncertainty (Funtowicz and Ravetz 1993).Have open energy models allowed this to happen?We discuss this question by first reviewing the role of models as thought experiments for decision support, identifying understandability as a key but often under-addressed issue, then, we discuss how models could become more understandable through developing and maintaining modular 'building blocks' , then, discuss the tailoring of such an effort to different user groups, and finally, pull these threads together to make concrete suggestions for next steps.

Models as thought experiments for decision support
First, we should clarify what kinds of models we are talking about: energy system models are techno-economic models of the energy system.We can define them as models that are used for decision support in planning and implementing the energy transition, for example, to produce pathways for a carbon-neutral energy supply (e.g.Jacobson et al 2017), to inform capacity expansion of the power grid, or to examine trade-offs between different renewable energy deployment strategies (e.g. Brown et al 2018, Tröndle et al 2020).They depict energy generation, conversion and transport processes with costs and efficiencies, and are often formulated as cost-minimising or welfare-maximising optimisation problems, with a range of techno-economic constraints to influence the optimal solution, such as minimum shares of certain technologies, limits on emissions, or a price on carbon.Of key importance is that these models do not model a natural process and are not predictive, but explorative or normative.Thus, while it is possible to validate some of their components-for example, a time series of wind power generation based on weather conditions-there is no way to validate an energy system model as a whole.It is possible to compare modelled scenarios with historic developments (Trutnevyte 2016).However, this does not answer the question of whether a model that was used to produce different alternative visions of the future to guide decision-making can was useful in making decisions on which of these alternatives to implement.
A useful way to think of energy system models is as thought experiments (Ellenbeck and Lilliestam 2019): as an exploration of what-if questions about the future.If we consider a stylised process of modelling (figure 1(a)), it starts with a research problem or research question, leading to the selection of data and making of assumptions, then to the model formulation (including software choice), and the generation of raw result data that are then analysed and interpreted in a final product such as a paper.The first three steps together are what is often considered a 'model' .This model is a set of data and mathematical equations, but it is not an objective representation of reality.It is based on and contains the world view of the modeller and the societal discourses leading to the research question (Ellenbeck and Lilliestam 2019).Often, modelling is commissioned by policymakers; the scope, data and assumptions used may well be influenced by these policymakers (Süsser et al 2021).
When thinking of energy system models as thought experiments, it is less problematic that they cannot be validated.The removal of the fact-value distinction is a feature rather than a bug, if the model is a tool to compare different possible assumptions, whether they are fact-based or value-based.Using the language of post-normal research, for the model itself to be useful, it is the quality of information that matters, not whether it is true or correct.Therefore, the thoughts behind the experiment ought to be well-documented and clearly communicated.This includes not just the fact that a specific number was chosen for a specific parameter, but why that specific number was chosen.In other words, what matters is that the model-this combination of assumptions, data, formulation, and code, applied to a specific question with a specific purpose-is understandable.
This understandability is important also for the modellers themselves because there is a possible danger (figure 1(b)): once a model exists, that is, once an experimental setup has been established, often with considerable investment of time, it makes sense to apply it to additional research questions.However, re-using one experimental setup in a new context can be dangerous.If the assumptions behind the model are not clear, subsequent users will have difficulties untangling the original research question from the model design decisions.They no longer have the means to understand the thoughts behind the thought experiment.When applied to new questions, the original authors' choices on assumptions, data, and model formulation may no longer be applicable.Thus, for models to be useful as decision-support, it is important that their users are able to understand and trace the embedded worldviews and assumptions.This includes the modellers themselves, who ought to be careful when they present model results as a 'rational' or 'unbiased' analysis of alternatives.

Understandable modelling through understandable building blocks
Because these models are only useful if they help make better informed decisions, understandability should be a key consideration in the modelling process.We have argued that a collection of data and code, built with a specific research question in mind, is what constitutes a model, that it is impossible to validate such a model, and that it should be re-used with caution because of the many possible embedded assumptions which may be valid only for the original research question.This does not mean that arguments for open modelling such as increased transparency or the increased potential for re-use and collaboration are invalid.Rather, these arguments may not be sufficient for 'the model' as a whole.
More than 10 years ago, DeCarolis et al (2012) made a call for repeatable analysis with energy-economy optimisation models, providing recommendations such as making code and data openly available.While the volume of academic literature published in this space has grown significantly since then, repeatability remains an elusive goal.However, the final recommendation made by DeCarolis et al-'Work toward interoperability among models'-provides our departure point.Instead of models as a whole, we can consider the building blocks that make up the model, for example, a manually curated dataset of different possible electricity storage technologies and their associated efficiencies and costs, or a dataset of geospatial solar energy potentials along with the code which generates this dataset, or code which simulates the electricity demand of a fleet of electric vehicles based on assumptions about user behaviour.The analysis performed with a model might rely on many such individual building blocks, and importantly, these building blocks themselves can published as a piece of open-source code, or an open dataset.
With sufficient knowledge of what blocks exist, analyses do not have to be built from scratch each time a researcher wants to answer a novel research question: one starts with a box of building blocks, knowing what each of them do, then puts them together to build the question-specific model.In that process, one might make adjustments to some blocks, or add a novel building block or two to the mix.If these building blocks are free, open, well-documented, and well-understood, and therefore, can easily be adapted and re-used, the process of building a model-based thought experiment to answer a new research question is faster, easier, and more transparent.Therefore, understandable modelling is open modelling through and through.It requires ensuring that all three types of openness apply: access to the description of the question that led to the model, the thinking and assumptions underlying it, access to the numerical data, and to the code that operates on these data to deliver answers to the question (figure 2).
The open energy modelling community could put more focus on establishing what kinds of building blocks are useful, and on constructing and maintaining them collaboratively.There are some successful examples of this already: software building blocks to achieve commonly needed tasks, for example resampling of time series (Kotzur et al 2018) or sharing data across models (Gidden and Huppmann 2019), or data building blocks that can be independently validated, for example estimating the generation potential of wind and solar power (Pfenninger and Staffell 2016, Staffell and Pfenninger 2016, Hofmann et al 2021).Well-documented building blocks can be re-used across models and iteratively developed across institutions.
Even if a given model-based analysis is not fully repeatable, striving towards replacing more and more building blocks with repeatable alternatives will eventually lead to a fully repeatable analysis.For example, many large energy system optimisation models which are otherwise fully open still rely on commercial optimisation solvers.Thus, the existing community efforts to build fully open-source alternative solvers like the HiGHS project (Huangfu and Hall 2018) will eventually allow this particular building block to be replaced with an open alternative.Finally, having a broad set of reusable building blocks available makes it easier to avoid the 'hammer-in-search-of-nails' problem of model development, by making it easier to build understandable, question-specific models.There need be no clear and hard definition of what of a building block can consist of, as long as it is some set of code, data, or assumptions, that can be meaningfully packaged up, documented, re-used, and perhaps most importantly, re-mixed, that is, adapted to a different context.For example, a qualitative policy paper might produce a set of narrative scenarios that could be used to represent different possible futures in a model experiment; such scenarios could be considered a building block.
If the models built from such building blocks are not detached, neutral scientific products, but driven by socially determined assumptions, then calls to harmonise the models in order to provide sound scientific decision support (Nikas et al 2021) are probably misguided.Doing so would lead to groupthink and a narrowing of options.However, intercomparison and harmonisation of the underlying building blocks is very valuable-for example to establish how different ways to simplify the representation of the power grid affect model results (Priesmann et al 2019).Comparing and possibly harmonising building blocks such as methods for estimating time series of renewable generation, or different ways in which the future demand for electricity demand from electrified vehicles can be estimated, is scientifically useful and methodologically possible.

Tailoring openness and understandability to different user groups
Openness that fosters understandability is relevant not just for the modellers themselves, but also for two additional groups of users: the 'expert' and 'non-expert' users of models and of model-based results (Kunkel et al 2016).Broadly, we can divide users into three groups: the core group of modellers, including the developers of modelling tools, an intermediate group of experts, consisting of people directly using models or working with model-based results, and finally the non-experts, consuming analysis produced by the first two groups (figure 3).The further away from the modelling process a user is, the more they may have difficulties locating where the most relevant uncertainties in the model lie, and whether assumptions are scientifically grounded or politically determined (Schmidt-Scheele 2020).These users may not care about code or data, but still need to be able to understand and assess the credibility and trustworthiness of the thought experiments represented by a particular model result.They want to make real-world implementation decisions on issues like infrastructure planning or policy design with confidence.These farther-off users may not strictly need access to data or code.But if we think again of a model as made up of smaller building blocks to answer a specific practically relevant question-say, examining trade-offs between new wind farm development and transmission grid expansion-then these users could still conceivably want to understand the building blocks, and how they were combined for that particular application.Furthermore, some of the non-modeller-experts (the middle group in figure 3) may have deep real-world domain expertise that modellers do not, whether that is expertise related to technical systems or other aspects such as policy.The co-creation process by which they contribute this expertise to a model-based analysis is much smoother and easier if they can focus their effort on understanding and contributing to a well-delineated, documented and explained piece of the model-a building block.Models (especially linear optimisation ones) are often vulnerable to small changes in just a few key parameters.For example, this could be the cost or performance of electric vehicles, if transport decarbonisation is one of the modelled decisions.If the data processing and assumptions related to transport are clearly contained in a 'transport' building block, a transportation expert could more easily investigate and critically these model components.This is another reason why open code and data alone are not enough.Openness needs to support all possible users, not just the modellers themselves and technically capable users, in understanding what may have brought about a certain model result.Having a clear understanding of what modular components a model was built up from would help communicate this.For example, it makes it possible to delineate parts that can be more easily validated (e.g.technical data, geospatial data, or simulation models used as inputs) from parts which consist primarily of value-based judgements and assumptions (e.g.geopolitical assumptions or costs).Zooming into the detailed inner workings of one of these building blocks would be necessary only for somebody that wants to apply the building block to a different question.Modular design is a technique used in many engineering domains.By allowing a 'separation of concerns' (Dijkstra 1982), it makes it possible to focus on and improve one part of a model without the mental overhead of dealing with the entire model at once.Thinking about a model this way therefore also makes it easier for modellers to communicate what their models can and cannot do to a broad audience.

Discussion
Scheller et al (2021) interviewed energy modellers, finding that PhD students working on energy system modelling research need to invest large amounts of time to get up and running, and that they report lower grades for model usability and documentation than more senior researchers.This leads the authors to hypothesise that with layer after layer of complexity added to a model, only the senior researchers are still able to fully grasp what the model does.Even if a PhD student ultimately arrives at an understanding of the complete chain of embedded questions, assumptions, data, and code, as outlined in figure 2, the findings of Scheller et al do not sound promising for energy models being understandable outside of an insular caste of modeller-priests.Does it make sense to ask for broader understandability to become one of the main goals for energy system modellers?If we accept the post-normal science view, then yes, it is a critical aspect that warrants more attention.In indeed Silvast et al (2020) confirm in an ethnographic study that modellers themselves see policy relevance as one of the key ways through which their models acquire legitimacy.
In the building-block view of modelling, understanding the model means first understanding the building blocks, then understanding the rationale between the way in which they have been put together to answer a particular question.Focusing community efforts on identifying generally useful building blocks, then sharing efforts to collaboratively design, develop, and compare them, would ensure that open modelling does not become a mere box-ticking exercise-such as dropping code in an online repository with an appropriate license, with little practical hope of it ever being understandable by third parties.When it comes to code in particular, practices from the software industry could be helpful.A range of software architectural patterns exist to tackle the problem of modularisation and re-usability of code.The microservices approach, for example, builds larger applications out of modular and re-usable 'services'-for example a database service-which communicate with each other through standard protocols and are configured and managed through an automated system like Docker (Boettiger 2015).Common data formats for energy model data can serve as the standard protocol layer, for example the IAMC (Gidden and Huppmann 2019) or friendly_data formats (Ali 2022).In terms of complexity, at the very simple end of the scale, an immediately practical approach is to make the building blocks so easy to use that a standard interface is not needed.For example, a tool produces renewable electricity supply time series data in an easily understandable CSV files means that users can adapt it to whatever input format they need-if need be, in a manual processing step.On the other end of the complexity scale are standards like the Functional Mock-up Interface (FMI), which defines a container standard and interface to let smaller simulation models work together to simulate larger systems (Modelica Association Project FMI 2023).
For building blocks to be (re-)useable, and the resulting models understandable, sufficient documentation is of key importance.Here also, approaches from other domains could prove useful.Mitchell et al (2019) propose 'model cards' to accompany trained machine learning models, giving background information on how a model was trained, and in which contexts the authors think a model should be used.In the context of energy models, different kinds of users require different degrees of detail.While each building block should come with documentation detailed enough not just to understand but also re-use and adapt it, documentation for a model that (partially) relies on pre-existing building blocks can simply point to the detailed building block documentation.This could be done much in the same way as food labelling works: in most countries, food products must be accompanied by a nutrition facts label and list all of their ingredients.This works because the labels themselves are standardised to summarise information at an appropriate level for the 'end user' , the consumer of a food.In the European Union, it is sufficient to list additives via their 'E numbers'-the detailed 'documentation' for an additive can then be looked up by a consumer if needed.Similarly, a 'nutrition fact' label for models could give a high-level overview of the building blocks used.
Finally, who should be responsible for taking action?Extra effort on top of what researchers and modellers already do know clearly comes with an extra cost.Is it worth the time spent on organisation, documentation, standardisation, and writing interface code (where standard interfaces are used)?It should be clear that pay-offs to such an investment will accrue particularly in the longer term, for individuals (e.g.making an individual researcher more productive in re-using their own work months or years later), teams (e.g. through better and smoother collaboration), and research and society as a whole (e.g. by making it easier to build on other people's work).Thus arguments in favour of efforts like this are similar to arguments in favour of open science more generally (Allen and Mehler 2019).Besides the effort needed, there is also a danger of building blocks with conflicting assumptions being used together; an issue identified as holding back holistic energy system modelling in general by Silvast et al (2020).However, the conceptual and practical process of designing, maintaining, discussing and combining building blocks could make it easier to understand how energy model evidence is constructed and where disciplines under-represented in the process can better contribute.For example, the process could help counter the exclusion of social sciences and humanities (Royston and Foulds 2021), by opening up avenues for integrating the modelling of social aspects of the energy transition (Krumm et al 2022).Diekmann and Peterson (2013) argue that not only are engineering models not value-free, but that they should be at least partially determined by value judgements.Separating concerns by using building blocks could make it easier to identify and communicate such value judgements, reducing the risk that model results are or can be used to silence dissent or support already-prevailing interests (Royston et al 2023).
Non-open models can also benefit from this process.The UK Energy Research Centre (UKERC) has defined a three-level model transparency categorisation ranging from 'open description' , where at least documentation is accessible, 'open access' , where access to a model is shared across multiple users and documentation and data are publicly available, through 'open source' , which they describe as 'fully transparent and accessible models available for any user to download and use' (Li and Strachan 2021).While the goal in this paper has been to argue for improvements in the state-of-the-art of this last, fully open-source category, building blocks could also let existing models be opened up in a step-by-step fashion or could be re-used in less-than-fully open models.Even if model developers cannot make the leap to the highest level of accessibility, open source, they could potentially make parts of their workflow available as an open building block for others.
Climate change mitigation is an urgent problem.Models and model results are wielded as weapons in battles between competing interests involving not just researchers but actors across society.Open modelling should be transparent and understandable modelling that does not hide but lays bare the embedded assumptions and agendas, helping resolve debates rather than advancing partisan interests.Making this process easier will prevent misunderstandings amongst modellers and users alike.

Figure 1 .
Figure 1.(a) A stylised process of energy system modelling.(b) The danger of 'the model' becoming a black box, re-applied to different questions for which original assumptions may no longer be valid.Adapted from Pfenninger et al (2018) CC BY 4.0.

Figure 2 .
Figure 2. A modified view of openness in relation to the energy modelling process, Adapted from Pfenninger et al (2018) CC BY 4.0.All three kinds of openness-open access to publications, open data, and open-source code-contribute to a fully understandable model, a transparent rather than a black box.The understandable model's code and data are built from modular, re-usable building blocks as much as possible.

Figure 3 .
Figure 3. Three stylised user groups of models and model results, and the extent to which they are involved in modelling or implementation.