Professional Documents
Culture Documents
Journal of Hydrology: Trine Enemark, Luk J.M. Peeters, Dirk Mallants, Okke Batelaan T
Journal of Hydrology: Trine Enemark, Luk J.M. Peeters, Dirk Mallants, Okke Batelaan T
Journal of Hydrology
journal homepage: www.elsevier.com/locate/jhydrol
This manuscript was handled by Peter K. Hydrogeological conceptual models are collections of hypotheses describing the understanding of groundwater
Kitanidis, Editor-in-Chief, with the assistance of systems and they are considered one of the major sources of uncertainty in groundwater flow and transport
Jian Luo, Associate Editor modelling. A common method for characterizing the conceptual uncertainty is the multi-model approach, where
Keywords: alternative plausible conceptual models are developed and evaluated. This review aims to give an overview of
Conceptual models how multiple alternative models have been developed, tested and used for predictions in the multi-model ap-
Model evaluation proach in international literature and to identify the remaining challenges.
Model rejection The review shows that only a few guidelines for developing the multiple conceptual models exist, and these
Multi-model framework
are rarely followed. The challenge of generating a mutually exclusive and collectively exhaustive range of
Conceptual model uncertainty
plausible models is yet to be solved. Regarding conceptual model testing, the reviewed studies show that a
challenge remains in finding data that is both suitable to discriminate between conceptual models and relevant
to the model objective.
We argue that there is a need for a systematic approach to conceptual model building where all aspects of
conceptualization relevant to the study objective are covered. For each conceptual issue identified, alternative
models representing hypotheses that are mutually exclusive should be defined. Using a systematic, hypothesis
based approach increases the transparency in the modelling workflow and therefore the confidence in the final
model predictions, while also anticipating conceptual surprises. While the focus of this review is on hydro-
geological applications, the concepts and challenges concerning model building and testing are applicable to
spatio-temporal dynamical environmental systems models in general.
1. Introduction uncertainty and give false confidence in model results, as vividly illu-
strated in Bredehoeft (2005).
Groundwater model conceptualization is a crucial first step in To develop conceptual models, two major approaches have been
groundwater model development (Anderson et al., 2015a). It provides a traditionally applied: (i) the consensus model approach (Brassington
systematic, internally consistent overview of system boundaries, prop- and Younger, 2010) and (ii) the multi-model approach (Neuman and
erties and processes relevant to the research question, bridging the gap Wierenga, 2003) (Fig. 1). The development of conceptual models is
between hydrogeological characterization and groundwater modelling. based on the available geological and hydrological information, which
As the conceptualization is related to the fundamentals of the pro- are observed data, such as water levels, borehole information and tracer
blem definition, it is considered one of the major sources of uncertainty concentrations, but often also include a component of soft knowledge,
in numerical groundwater modelling (Gupta et al., 2012). Estimating such as geological insights or expert interpretation.
parameters through calibration with an inadequate conceptual model In the single consensus conceptual model approach all available
may lead to biased parameter values (Doherty and Welter, 2010). observations and knowledge is iteratively integrated into a single con-
Biased parameter values are especially problematic when extrapolating ceptual model (Barnett et al., 2012; Izady et al., 2014), providing a
to predictions that are of a different type than the calibration data, staircase of confidence (Gedeon et al., 2013). In this case, the con-
represent a different stress regime, or have a longer timeframe than ceptual model represents the current consensus on system behaviour
the calibration period (White et al., 2014). Not accounting for con- (Brassington and Younger, 2010).
ceptual model uncertainty can potentially greatly underestimate total As illustrated in Schwartz et al. (2017), conceptual model
Corresponding author at: CSIRO Land and Water, Gate 4 Waite Rd, Locked Bag 2, Glen Osmond, SA 5064, Australia.
⁎
E-mail addresses: [email protected] (T. Enemark), [email protected] (L.J.M. Peeters), [email protected] (D. Mallants),
[email protected] (O. Batelaan).
https://doi.org/10.1016/j.jhydrol.2018.12.007
Received 27 July 2018; Received in revised form 6 November 2018; Accepted 4 December 2018
Available online 17 December 2018
0022-1694/ © 2018 Elsevier B.V. All rights reserved.
T. Enemark et al. Journal of Hydrology 569 (2019) 310–329
Fig. 1. Iterative process for the conceptual modelling process via the consensus or multi-model approach. Modified from Environment Agency (2002) and Suzuki
et al. (2008). Each model test step involves introducing new data and thereby identifying new plausible models uncovering conceptual surprises, and rejecting other
models that are inconsistent with the new data.
uncertainty is generally accounted for in the consensus approach by conceptual surprises than the consensus approach, it is not exempt from
increasing the complexity of the model. Increasing complexity effec- it. Using statistical terminology, as explained by Neuman (2003), both
tively turns conceptual model uncertainty into parameter uncertainty the consensus approach and the multi-model approach are prone to
by adding more processes to the model and/or increasing resolution in Type I errors (underestimating model uncertainty by undersampling the
space and time. Increasing the degrees of freedom means that non- model space) and Type II errors (relying on invalid model(s)). However,
uniqueness increases, which is often balanced through optimal model by using the multi-model approach we are less likely to commit either.
complexity favouring the simplest model that can adequately reproduce This paper aims to provide an overview of the current status of the
historical conditions (Young et al., 1996). The main advantage is that it international literature on using multiple conceptual models in
comprehensively captures conceptual issues in the model. The main groundwater modelling. Reviews of the multi-model approach to date,
drawback is that models quickly become intractable and too compu- such as Diks and Vrugt (2010), Schöniger et al. (2014), and Singh et al.
tationally demanding to carry out parameter inference. Another me- (2010) mainly focus on the evaluation of multiple models and sum-
chanism that is often applied to account for conceptual uncertainty, is marising of model results. Much less attention has been devoted to
conservatism, favouring the conceptualization that will result in the approaches that systematically develop and test different conceptual
largest impact (Wingefors et al., 1999). Although inherently biased, the models. This review is therefore organized around the following four
main advantage is that introducing conservative assumptions make the research questions:
problem tractable and provides confidence that the simulated impacts
are not underestimated. The largest drawback however, is that con- 1. What is conceptual model uncertainty?
servative assumptions depend on the type of impact investigated, may 2. How are alternative conceptualizations developed?
not be internally consistent and can lead to missed opportunities 3. How can alternative conceptualizations be tested?
(Freedman et al., 2017). 4. How are different conceptualizations used for predictions?
The alternative to the consensus approach is the multi-model ap-
proach, in which an ensemble of different conceptualizations is con- Each section provides an overview of approaches in published stu-
sidered throughout the model process in parallel rather than sequen- dies, summarized in Tables A.1 and A.2, and remaining challenges.
tially. This approach reflects that the hydrogeological functioning of an While this review will focus on applications in a hydrogeological con-
aquifer system can be interpreted in different ways, especially if the text, the concepts and challenges concerning model building and testing
available data is scarce (Anderson et al., 2015a; Beven, 2002; Neuman are applicable to spatio-temporal dynamical environmental systems
and Wierenga, 2003; Refsgaard et al., 2006). In the multi-model ap- models in general.
proach the aim is not to find the single best model, but to find an en-
semble of alternative conceptual models, each with a different hy-
2. What is conceptual model uncertainty?
pothesis on system behaviour. As depicted in Fig. 1, this is also an
iterative process, in which conceptual models are removed from the
Anderson and Woessner (1992) and Meyer and Gee (1999) define a
ensemble when they are falsified by increased knowledge or data, and
conceptual model as a pictorial, qualitative description of the ground-
where conceptual models are added when new data or insights prompt
water system in terms of its hydrogeological units, system boundaries
the development of a new hypothesis on model behaviour.
(including time-varying inputs and outputs), and hydraulic as well as
In the consensus approach, once committed to a particular con-
transport properties (including their spatial variability). The conceptual
ceptualization, there is considerable inertia to change it as this would
model is often seen as a hypothesis or a combination of hypotheses for
often involve a complete overhaul of the numerical model (Ferré,
the aspects of the groundwater system that are relevant to the model
2017). However, in the multi-model approach, given alternative con-
objective.
ceptual models are developed and evaluated in parallel, it aids in sol-
Table A.1 provides a review of internationally peer reviewed pub-
ving the problem of conceptual “surprises” (Bredehoeft, 2005) as they
lications that explicitly consider hydrogeological conceptual model
are sought out. Even though the multi-model approach is less prone to
uncertainty. These 59 studies have been identified from the Google
311
T. Enemark et al. Journal of Hydrology 569 (2019) 310–329
Scholar database, where the search term “groundwater model” is the first two steps, as well as the Spatial Variability Structure (Fig. 2).
combined with “conceptual model uncertainty”, “structural model un- The latter is included in our discussion of aspects of conceptualization
certainty”, “alternative conceptual models” or “multi-model approach”. as some studies in Table A.1 consider alternative models of the Spatial
Only studies that include alternative conceptual models developed for Variability Structure as conceptual uncertainty.
groundwater modelling, for the purpose of either increasing system The Conceptual Physical Structure captures the hydrostratigraphy
understanding or characterizing conceptual uncertainty, have been in- as well as the horizontal and vertical extent of the system (respectively
cluded. This list is considered to be representative of the treatment of a watershed divide and an impermeable bottom boundary in Fig. 2).
conceptual model uncertainty through the multi-model approach in The Conceptual Physical Structure further defines the hydrostrati-
groundwater research in the last two decades. It is beyond the scope of graphic units and their extent, the barriers and/or conduits to
this review to address the consensus conceptual model building ap- groundwater flow (faults) and the compartmentalisation of the
proach. For each study, Table A.1 provides a short summary of the groundwater system into aquifers and aquitards. The Spatial Variability
alternative conceptualizations, whether or not the objectives are ex- Structure is the description of the time-invariant hydraulic properties of
plicitly defined and which aspects of the conceptualization are con- the system and their spatial variability (magnifying glass in Fig. 2). The
sidered. Conceptual Process Structure contains the boundary conditions that are
In this section we discuss what is included in model con- time variant, such as heads and fluxes in and out of the system. These
ceptualization, how this needs to be linked to the objective of the can be externally controlled and largely independent from the
modelling and the linguistic ambiguity in discussing conceptual model groundwater system dynamics (e.g., rainfall, pumping rates, drainage
uncertainty. levels for mine dewatering, lateral zero-flow boundary) or internally
controlled and largely dependent on the groundwater system dynamics
(e.g., surface water-groundwater interaction, evapotranspiration).
2.1. Conceptual model aspects
Gupta et al. (2012) outlines five formal stages in the model building 2.2. Modelling objective
process: i) Conceptual Physical Structure, ii) Conceptual Process
Structure, iii) Spatial Variability Structure, iv) Equation Structure and Despite being identified as the crucial first step in any modelling
v) Computational Structure. The first two steps are part of the con- study (Anderson et al., 2015a; Barnett et al., 2012; Brassington and
ceptual model, the third and fourth are part of the mathematical model Younger, 2010), only 33 out of 59 studies explicitly define the purpose
and the last step is the computational model. This review will focus on or objective of the model in the introduction of their paper. This is
Fig. 2. Elements of a conceptual model. Items in green illustrate the Conceptual Process Structure, while items in blue illustrate the Spatial Variability Structure
represented in the magnifying glass (Kh = horizontal hydraulic conductivity, Kv = vertical hydraulic conductivity, n = porosity, Ss = Specific storage, Sy = Specific
yield). Items in orange illustrate the Conceptual Physical Structure representing the system geometry and hydrostratigraphy. (For interpretation of the references to
colour in this figure legend, the reader is referred to the web version of this article.)
312
T. Enemark et al. Journal of Hydrology 569 (2019) 310–329
especially relevant as some conceptualization aspects (such as detailed 2.4. Summary of what is considered conceptual model uncertainty
description of spatial variability of hydraulic properties) might be im-
portant to one type of prediction (e.g., travel time distribution), but Groundwater system conceptualization is a collection of hypotheses
might be less relevant to another type of prediction (e.g., hydraulic describing the understanding of the different aspects of the ground-
head distribution) (Refsgaard et al., 2012; Zhou and Herath, 2016). water system that are important to the modelling objective. Conceptual
Alternative conceptualizations are, for instance, directly linked to model uncertainty is the uncertainty due to the limited data and
model objectives when multiple conceptual models are developed to knowledge about a groundwater system. It is the first-order, epistemic
increase system understanding (Passadore et al., 2011) or aid in water uncertainty that is generally considered reducible but cannot be char-
management strategy (Højberg and Refsgaard, 2005). Many of the acterized by continuously varying a variable. Linguistic ambiguity and
studies in which a model objective is not explicitly defined, are focused vague definitions of what constitutes conceptual uncertainty however,
on method development, such as combining model averaging techni- hinders transparent discussions of this major source of uncertainty. We
ques (Rojas et al., 2008), comparing ranking strategies (Foglia et al., will therefore adopt the terminology of Suzuki et al. (2008) and focus
2007) or model selection (Poeter and Anderson, 2005). on first-order uncertainty.
313
T. Enemark et al. Journal of Hydrology 569 (2019) 310–329
Fig. 3. Conceptual model development approaches in the multi-model approach. Illustration of how different conceptualizations of the Conceptual Physical Structure
could take shape if based on the same data (boreholes in this case) through Varying Complexity (a), Alternative Interpretation (b) or Hypothesis Testing (c) strategy.
Based on illustrations of alternative models in Harrar et al. (2003), Schöniger et al. (2015), Seifert et al. (2008) and Troldborg et al. (2007).
conceptualizations may end up being very similar and it is difficult to models developed is collectively exhaustive, but it is more likely for
ensure that independent interpretations are mutually exclusive. Hypothesis Testing to generate an ensemble of mutually exclusive
In the Hypothesis Testing strategy, as advocated by Beven (2018), models.
an ensemble of models is generated by stating different hypotheses The next sections review model building approaches and are
about the system. Rather than multiple teams formulating their best structured around the three key components of the conceptual model
interpretation of the same data in the Alternative Interpretation illustrated in Fig. 2; Conceptual Physical Structure (Section 3.1), Spatial
strategy, the Hypothesis Testing strategy involves the same team aiming Variability Structure (Section 3.2), and Conceptual Process Structure
to maximise the difference between alternative conceptualizations, (Section 3.3). The focus is on different approaches to building multiple
while still adhering to the same dataset. In Fig. 3 this is exemplified conceptual models within these three aspects and how the different
through the presence or absence of a palaeovalley in two alternative strategies to multi-model building have been applied (Fig. 3). Finally,
conceptualizations. Both alternatives are consistent with the borehole Section 3.4 discusses assigning prior probabilities to alternative models.
data, but the interpretation with the palaeovalley present may be
considered less likely. The chances are slim that such a vastly different
3.1. Conceptual Physical Structure
conceptualization would be part of an ensemble generated through the
Alternative Interpretation strategy, where only the most likely model is
Table A.1 lists several examples where the Conceptual Physical
sought. None of the three strategies guarantees that the ensemble of
Structure of conceptual models has been tested through the Alternative
314
T. Enemark et al. Journal of Hydrology 569 (2019) 310–329
Interpretation and the Hypothesis Testing strategy. Using an Alternative lack of attention in literature is mainly due to the process description
Interpretation strategy approach, five alternative hydrostratigraphic typically being assumed to be complete. However, as illustrated by
models were generated by five different (hydro)geologists in the study examples in (Bredehoeft, 2005), conceptual surprises might also occur
by Seifert et al. (2012) resulting in different number of layers, pro- for the Conceptual Process Structure as well as for the other compo-
portions of sand and clay in the quaternary sequence and the location of nents of the conceptual model.
a limestone surface. Using the Hypothesis Testing strategy, Troldborg Among the many boundary conditions imposed on a groundwater
et al. (2007) developed three different models by assuming different model, groundwater recharge is by far the one that has received most
depositional histories and thereby different number of layers in the attention in the literature. A number of methods exist for calculating
models. groundwater recharge that take into account different sources of in-
While it is possible to test a global geometrical hypothesis about the formation (Doble and Crosbie, 2017; Scanlon et al., 2002) which can
Conceptual Physical Structure (e.g. Troldborg et al. (2007)), it is more lead to different estimates of recharge when used in an Alternative
common to test specific geometrical features through local hypotheses. Interpretation strategy approach. Ye et al. (2010) used the Maxey-Eakin
A local hypothesis can, for instance, test the presence of a palaeovalley method, the chloride mass balance method and the net infiltration
(Seifert et al., 2008), the connection between two aquifers (La Vigna method to derive different estimates of recharge to assess the con-
et al., 2014), or the extent of an aquifer (Aphale and Tonjes 2017). If ceptual uncertainty. Each of the different interpretation methods re-
one of the hypotheses is falsified in these studies, the system under- sulted in a different spatial distribution of recharge.
standing will improve in regards to that specific feature. Different levels of model complexity have often been used across
different spatial scales, such as for groundwater recharge estimation
3.2. Spatial Variability Structure (Doble and Crosbie, 2017). Models range from simplified heuristic
models at a global scale (Döll and Fiedler, 2008), simple 1-D bucket
Spatial Variability Structure is the component of the conceptual models for regional scale areas (Flint et al., 2000) to more complex
model that is most often included in a multi-model approach. Because numerical solutions of Richards’ equation at the field scale (Leterme
hydraulic and transport properties are often scale-dependent and the et al., 2012; Neto et al., 2016). Nettasana (2012) tested the complexity
adequate level of complexity depends on the modelling purpose, the of zonation of recharge by defining recharge based only on soil type in
description of properties is often tested by developing models with the one model and in another model both on soil type and land use.
Varying Complexity strategy. The strategy is applied either through The Hypothesis Testing approach for recharge estimation mainly
dividing the study area into different zones of homogeneous hydraulic focuses on a specific feature (Kikuchi et al., 2015; Rojas et al., 2010a).
conductivities, so alternative representations can be generated by Aphale and Tonjes (2017) investigate the effect of a landfill on local
combining the different zones (e.g. Foglia et al., 2007), or by re- recharge with three different hypotheses. Hypothesis Testing for lateral
presenting the geology in different conceptual models as homogenous, boundary conditions has been applied to lateral exchange fluxes with
layered/zoned, or as heterogeneous (e.g. Schöniger et al., 2015). adjacent aquifers (Lukjan et al., 2016; Mechal et al., 2016; Nettasana,
In the INTRAVAL Las Cruces trench experiment five different 2012). Kikuchi et al. (2015) test the existence of underflow through a
modelling teams developed unsaturated zone flow and transport models subsurface zone into an adjacent basin.
using the Alternative Interpretation strategy (Hills and Wierenga,
1994). Despite differences between the models, such as isotropic/ani- 3.4. Assigning a prior probability
sotropic and spatially uniform/heterogeneous soil properties, none of
the models was clearly superior considering several performance cri- A crucial aspect in any Bayesian modelling approach is assigning the
teria. prior probabilities. This prior is based on an initial understanding of the
Geostatistical variogram based approaches facilitate the stochastic probability of a model related to the alternative models and is updated
generation of many pixel-based K realizations based on the same data when additional data is introduced in the model testing step (Section 4).
and assumptions to characterize the lower-order uncertainty. The assigned prior for the reviewed studies are presented in the first
Hypothesis Testing strategy has been applied assuming different var- column of Table A.2.
iogram models to represent the K variation within the system (Samper In order to be objective and unbiased, different conceptual models
and Neuman, 1989; Ye et al., 2004). Rather than defining different are often considered to be equally likely, uninformed by data or
facies variogram, Pham and Tsai (2015; 2016) used three different knowledge. From the 26 studies in Table A.2 that assign a prior prob-
variogram based geostatistical approaches (indicator kriging, indicator ability, 21 use a uniform, and thus uninformed, prior probability. Prior
zonation and general parameterization (Elshall et al., 2013)) to describe probabilities do however have a large influence on the posterior
the variation between clay and sand units as smooth or sharp. probability if the data used for updating the prior has limited in-
In the multipoint geostatistics approach (MPS) (Strebelle, 2002) formation content. Rojas et al. (2009) showed that including proper
different conceptualizations can be represented by adopting different prior knowledge about the conceptualizations increased predictive
training images using the Hypothesis Testing strategy. Studies that have performance when compared to assigning uninformed priors. Ad-
applied the MPS approach using more than one training image in ditionally, uninformed priors are not consistent with the Hypothesis
groundwater modelling are still rare but include studies by He et al. Testing approach, as shown in Fig. 3c. If no other palaeovalleys were
(2014), Hermans et al. (2015) and Linde et al. (2015a). observed in the area, the palaeovalley hypothesis would be possible,
Groundwater flow through fractured rock aquifers complicates the but unlikely. A uniform prior probability would assign each hypothesis
conceptualization as the groundwater flow occurs through both matrix equal likelihood, which would not be appropriate.
and fractures. Selroos et al. (2002) considered e.g. stochastic continuum In the reviewed studies the prior has been based on expert opinion,
models and discrete fracture networks as alternative conceptualizations data consistency and model complexity. For instance, using expert
of fractured rock in Sweden; the models were shown to have different opinion in the study by Ye et al. (2008), the prior probability was based
results in terms of solute transport behaviour. on expert’s belief in alternative recharge models considering the con-
sistency with available data and knowledge. Systematic expert elicita-
3.3. Conceptual Process Structure tion is a well-established technique in environmental risk assessment
and modelling (Krueger et al., 2012) to formalize expert belief into
The Conceptual Process Structure is the component in the con- model priors. There are, however, few published studies on expert
ceptual model that is considered least in the multi-model approaches in elicitation in groundwater conceptualization context. Elshall and Tsai
the analysed studies (Table A.1). According to Gupta et al. (2012) this (2014) used data consistency to inform the prior probability by basing
315
T. Enemark et al. Journal of Hydrology 569 (2019) 310–329
it on calibration of hydrofacies using lithological data. Finally, using priors that accounts for unknown unknowns remains a challenge.
model complexity to inform the prior, in the study by Ye et al. (2005) The Hypothesis Testing strategy seems to be the only model de-
higher probabilities were assigned to favour models with fewer para- velopment strategy that can ensure the models developed are mutually
meters. This was also suggested by Rojas et al. (2010a) as a means of exclusive. However, hypotheses might still overlap. For example,
penalizing increased complexity. Nearing et al. (2016) argues that as- Bresciani et al. (2018) test three hypotheses to explain mountain range
signment of probabilities should not be based on a single component of recharge to a basin aquifer governed either by i) mountain-front re-
the model but rather be based on the whole model. In the reviewed charge, ii) mountain-block recharge or iii) both mountain-front re-
literature the priors have however, only been based on individual charge and mountain-block recharge. Some might argue that the third
components. hypothesis overlaps to some extent with the other two, violating the
mutually exclusive principle. However, only including the two first
3.5. Remaining challenges hypotheses, claiming they are mutually exclusive and collectively ex-
haustive, would set up a false dilemma as parts of both hypotheses can
The review of studies in Table A.1 has shown that alternative be correct at the same time. It is thereby not always possible to state
models have been developed either by i) varying complexity of model mutually exclusive hypotheses in hydrogeology, where the answer will
description, ii) making alternative interpretations or iii) stating dif- be Boolean (true or false), for instance connectivity or no connectivity
ferent hypotheses about the groundwater system. The goal of the multi- between aquifers (Troldborg et al., 2010). Sometimes the mutually
model development process is to define a mutually exclusive, collec- exclusive hypotheses will have to be stated as endmembers (e.g.
tively exhaustive range of models in which the true unknown model mountain-front recharge and mountain-block recharge) and the answer
exists and where the risk of uncovering a conceptual surprise is zero. will be somewhere in between.
This is obviously unattainable and we therefore discuss the remaining Guillaume et al. (2016) discuss two methods to accommodate the
challenges next. conceptual surprises in the model development process: Adopting
First, Table A.1 shows that studies typically focus on exploring adaptive management and applying models that explore the unknown.
different hypotheses for a single aspect of the model (Conceptual In the first approach, management plans are kept open towards change
Physical/Conceptual Process/Spatial Variability Structure). Only 5 out and the iterative modelling process, illustrated in Fig. 1, is a part of the
of 59 papers consider all three aspects simultaneously (Aphale and modelling plan. The second method anticipates surprises by placing
Tonjes, 2017; Foglia et al., 2013; Mechal et al., 2016; Rojas et al., fewer restrictions on what is considered possible. By stating bold hy-
2010a; Ye et al., 2010). For the range of models to be collectively ex- potheses about a system ensures that system understanding can pro-
haustive, all conceptually uncertain aspects have to be considered. gress (Caers, 2018). A bold hypothesis around recharge inflows from
Second, the study objective is not always considered when alter- faults and deep fissures connected to an adjacent aquifer is tested by
native models are developed for the multi-model approach (Table A.1). Rojas et al. (2010a). The available data did not give reason to reject
Models should encapsulate the behaviour that is important to the either of the models to achieve an increase in system understanding, but
modelling objective (Jakeman et al., 2006), and the same should be the alternatives were bold. We argue that by being forced to be bold
true when characterizing conceptual uncertainty. On the other hand, when developing hypotheses, the risk of rejecting plausible models by
“what may seem like inconsequential choices in model construction, omission and adopting an invalid range of models is greatly reduced.
may be important to predictions” (Foglia et al., 2013). To avoid ig- However, defining bold hypotheses does not preclude rejecting plau-
noring the inconsequential model choices, the model objective should sible models by omission. Hunt and Welter (2010) suggest to use ter-
be used to guide the development of alternative models. This does minology that recognize the existence of these unknown unknowns by
imply that ensembles are not necessarily the same for all model ob- presenting results with a specification of which aspects of the model
jectives (Haitjema, 2005). that has been considered, thereby enhancing transparency. An ap-
Third, alternative conceptual models are not always defined as proach that aims at directly identifying unknown unknowns through
mutually exclusive (i.e. if model A is true, models B and C are false). bold hypotheses, taking into account the largest possible range of the
Falsification, which is welcomed in the multi-model approach (Beven, conceptual uncertainty, have not been applied yet and remain a subject
2018), will increase system understanding (Beven and Young, 2013), for further research.
but how much will depend on how the conceptual models are defined.
In the Alternative Interpretation and Varying Complexity strategy, the 4. How are different conceptualizations tested?
models are not necessarily mutually exclusive in the sense that they do
not represent different ideas about the groundwater system. In the After developing a set of conceptual models, the models should be
Varying Complexity approach, alternative models are generated based tested to establish to what degree they are consistent with the available
on the same conceptual model represented in different complexities. A data and knowledge (Neuman and Wierenga, 2003; Refsgaard et al.,
risk in the Alternative Interpretation strategy is that alternative models 2006). Groundwater models used for safety assessment of nuclear waste
are almost identical in terms of understanding of the groundwater repositories, for instance, have been subject of considerable validation
system. efforts (Hassan, 2003; Rogiers et al., 2014; Tsang, 1987, 1991). Model
Fourth, the way the alternative models are developed does not al- testing and validation covers the same model evaluation process in
ways reduce the risk of conceptual surprises. Using the Alternative which models are confronted with new data. However, the term vali-
Interpretation strategy, many groups will come up with what they be- dation is avoided in this review as models can never be proven correct
lieve to be the most likely model, e.g. Seifert et al. (2012). Using the (Konikow and Bredehoeft, 1992). Also, there is no internationally
Varying Complexity strategy, only the complexity and not conceptual agreed definition of validation, which has led several organizations to
ideas will be tested. It is therefore unlikely that a conceptual surprise develop their own operational definitions of validation (Perko et al.,
will be found before one is surprised in both Alternative Interpretation 2009). Finally, validation encourages testing to have a positive result
and Varying Complexity strategy. (Oreskes et al., 1994), that is, models are not expected to be wrong. As
Last, when assigning priors to a range of models that we cannot falsification is important in order to advance our understanding of a
ensure are collectively exhaustive, how do we account for unknown system (Beven, 2018), the term model testing is preferred here.
unknowns? The sum of prior probabilities for the ensemble of models Models are rejected if they are found to be inconsistent with data. In
always add up to one in the reviewed studies, thereby assuming a a Bayesian context, however, a conceptual model can never be com-
collectively exhaustive range of models have been defined. As discussed pletely rejected; its probability can only be greatly reduced. As there is
already, this is extremely difficult to ensure, so an approach to assign a risk of eliminating models that could turn out to be good
316
T. Enemark et al. Journal of Hydrology 569 (2019) 310–329
representations when new data is introduced, Guillaume et al. (2016) 4.1. Conceptual model testing data
suggest to keep rejection decisions temporary to be able to return to
otherwise excluded models. The models that are consistent with ob- Three basic requirements for the nature of the data used for model
servational data are, however, only conditionally validated because they testing are typically discussed: i) it should be different from the data
have not been proven to be inconsistent with data yet (Beven and used for developing the conceptual models (Tarantola, 2006), ii) it
Young, 2013; Oreskes et al., 1994). should be different from the data used for calibrating the model
Testing of conceptual models is not always done as part of the multi- (Neuman and Wierenga, 2003; Refsgaard et al., 2006), and iii) it should
model approach to groundwater modelling (Pfister and Kirchner, depend on the modelling purpose (Beven, 2018).
2017). In Table A.2, only 30 out of 59 studies applied some form of
model testing. However, model testing presents three major ad- 4.1.1. Model testing data and model building data
vantages. Tarantola (2006) distinguishes between a priori information used to
First, systematically developing and testing conceptual models will develop hypotheses and observations used to test models. Post-hoc
allow one to explain why no other conceptual models are plausible theorizing (failing to separate model development and testing data and
(Neuman and Wierenga, 2003), and thereby reducing the risk of accepting the resulting model) might lead to models being conditionally
adopting an invalid range of models. Through systematic documenta- validated due to circular reasoning, e.g. the model should look this way
tion and rejection of conceptual models, the modelling workflow be- to explain the data and the model is true because it explains the data.
comes transparent and traceable, potentially avoiding court cases Another reason for keeping those two groups of data separate is to
challenging the validity of conceptual models. In the impact assessment avoid underestimating conceptual uncertainty. By using geophysical
of the Carmichael Coalmine in Queensland (Australia), available geo- SkyTEM data to both build a training image conceptual model and as
logical and hydrological data allowed for at least one other con- soft constraint as part of a multiple-point geostatistics algorithm, He
ceptualization of ecological and culturally significant springs that could et al. (2014) demonstrated that this over-conditioning lead to an un-
potentially be impacted by the coalmine (Currell et al., 2017). How- derestimation of uncertainty.
ever, a conceptual model leading to an acceptably low modelled impact
on the springs was adopted, which lead to the approval of the mine. A 4.1.2. Model testing data and model calibration data
systematic model development and testing approach for conceptual Testing data should also be different from calibration data to avoid
modelling through the multi-model approach would be able to shed that the conditional confirmation becomes an extension of the cali-
light on this type of confirmation bias. bration (Neuman and Wierenga, 2003). In a review of handling geo-
Second, model testing can lead to uncovering of unknown un- logical uncertainty, Refsgaard et al. (2012) highlighted that it is pos-
knowns (Bredehoeft, 2005). Not many papers exist that actually reject sible to compensate for conceptual errors in groundwater flow models
all of the initial conceptual models or hypotheses about a groundwater by calibrating parameters to fit the solution. The best test for any
system and come up with new plausible explanations, which renders conceptualization involves comparison of model predictions to ob-
this advantage of the model testing procedure somewhat invisible servations outside the calibration base. Cross-validation techniques,
(Beven, 2018). There are, however, a few examples where models are standard practice in statistical inference, are underutilised in ground-
conditionally validated after ad-hoc modifications to the model (e.g. water modelling. Methodologies that minimize error variance provide
Krabbenhoft and Anderson, 1986; Nishikawa, 1997; Woolfenden, some safeguard against calibration-induced acceptance of improper
2008). Ad-hoc modifications are slight changes applied to a current conceptualizations (Kohavi, 1995; Moore and Doherty, 2005; Tonkin
model in order to explain conflicting data, but without falsifying the et al., 2007).
model as a whole. For example, Sanford & Buapeng (1996) developed a
steady-state groundwater flow model for the Bangkok area, which was 4.1.3. Model testing data and the modelling objective
falsified by apparent groundwater ages. An ad-hoc modification that Refsgaard et al. (2012) further concluded that models that perform
assumed groundwater velocities were higher during the last glacial well according to one dataset might not perform well according to
maximum yielded a simulated apparent age closer to the observations, another dataset. This suggests that updating of prior probability should
thereby conditionally validating the model with the ad-hoc modifica- preferably be based upon the data type that the models are to make
tion. Ad-hoc hypotheses are sometimes criticized as they make models predictions about. Davis et al. (1991) argues that testing model per-
unfalsifiable and knowledge does not progress through modifications formance outside areas relevant to the model objective can lead to re-
(Caers, 2018). However, their existence illustrate the difficulty of de- jection of models that might actually be fit-for-purpose. However, in
veloping a collectively exhaustive range of models initially and model many instances, the data type that the models are used to make pre-
testing is imperative if we want to uncover this. dictions about, such as groundwater fluxes or water balances, may not
Third, Bayesian multi-model approaches benefit from allowing their be directly available (Jakeman et al., 2006). On the other hand, Rojas
prior probabilities to be updated because it dilutes the effect of the et al. (2010b) showed that by introducing more and more data in a
choice of priors (Rojas et al., 2009). It is here worth mentioning that multi-model approach, they were able to further and further dis-
most of the studies in Table A.2 that apply a Bayesian approach, update criminate between retained conceptual models, suggesting the more
the prior probability using criteria-based weights (Section 5.1) while diverse and numerous data that are used for testing, the more con-
only eight studies apply a model testing procedure. fidence in the conceptualization.
In the subsequent sections, data relevant to conceptual model
testing (Section 4.1), steps undertaken when testing conceptual models 4.2. Conceptual model testing steps
(Section 4.2), and the remaining challenges within model testing (4.3)
are discussed. Table A.2 presents an overview of the model testing In the previous discussion the type and nature of auxiliary data to
applied in the studies identified using the multi-model approach test conceptual models were introduced. But how should such data be
(Section 2). incorporated to undertake a conceptual model testing exercise?
317
T. Enemark et al. Journal of Hydrology 569 (2019) 310–329
Table 1
Comparison of model testing steps (pros and cons) and examples of applications in literature. The terminology of Step 1–3 is from model testing steps by Neuman and
Wierenga (2003); definition of post-audit is from Anderson and Woessner (1992).
Conceptual model testing step Pros (P) and cons (C) Example
Step 1: “Avoid conflict with data” Narrows down range of plausible models before Hermans et al. (2015) tests training images for MPS against geophysical data.
conversion to mathematical model (P)
Step 2: “Preliminary mathematical Holistic test of the system (P) La Vigna et al. (2014) tests the cause of hydraulic connection between two sand
model testing” Parameters can compensate for conceptual error aquifers against hydraulic head in a simple numerical model and is able to reject two
(C) out of three scenarios.
Narrows down range of plausible models before
complex mathematical model (P)
Step 3: “Confirm model” Holistic test of the system (P) Parameters: Poeter and Anderson (2005) were able to reject 13 out of 61 models
Parameters can compensate for conceptual error where the parameter distribution was wrong.
(C) State variables: Rojas et al. (2008) tested alternative conceptual models against
hydraulic head and rejected two models but were unable to discriminate strongly
between the rest of the models.
Convergence: Poeter and Anderson (2005) rejects two models based on non-
convergence.
Step 4: Post audit Waiting time (C) Nordqvist and Voss (1996) concluded that a supply well was in risk of contamination
Holistic test of the system (P) through a multi-model approach. After the completion of the study, increased levels
Parameters can compensate for conceptual error of contamination were observed in the well which conditionally validated the models.
(C)
Neuman and Wierenga (2003) introduced a three-step workflow for model is set up and compared with testing data (hydraulic head). The
testing and updating prior probability of alternative conceptual models advantage of applying this model testing step is that spending time on
(Table 1). In addition to these three steps, a fourth step, the post-audit setting up complex mathematical models for poor conceptual models
(Anderson and Woessner, 1992) will be reviewed here. is avoided.
318
T. Enemark et al. Journal of Hydrology 569 (2019) 310–329
all but one conceptual model by manual comparison of the direct without looking for other plausible models. However, in many studies
simulation of 4He concentrations with observed data. However, in independent data might not be available and saving some data for the
cases where the lower order uncertainty is characterized within each model testing process is a trade-off between being able to define a
conceptualization, automatic procedures are necessary to efficiently more complete model and being able to test assumptions. Cross-va-
search for models that match field data (Rogiers et al., 2014; Rojas lidation can partly address this issue during inference or calibration,
et al., 2010b,c,a; Schöniger et al., 2015; Zeng et al., 2015). For in- but will remain impractical in the conceptualization phase (model
stance, Rojas et al. (2008) used the importance sampling technique testing step 1) as biases towards existing but unavailable data might
Generalized Likelihood Uncertainty Estimation (GLUE) (Beven and be made.
Binley, 1992) to sample combinations of parameter sets and con- Second, in theory the data used for model testing should depend on
ceptual models and reject models according to an acceptance the model objective, in order to not extrapolate when making predic-
threshold for the misfit between simulated and observed model pre- tions. A challenge arises when having to ensure that the model found
dictions. fit-for-purpose for one dataset (e.g. hydraulic head), will also be fit-for-
Finally, non-convergence of the groundwater model can indicate an purpose to predict another dataset (e.g. concentrations). For example,
error in the conceptual model (Anderson et al., 2015b). The interaction the alternative models developed by Castro and Goblet (2003) all per-
of assumptions that lead to groundwater models not converging has in formed well when calibrated with hydraulic head; however, all but one
many studies been regarded as sufficient evidence of conceptual model model was rejected when tracer data was introduced. Sensitivity and
invalidity (Aphale and Tonjes, 2017; Poeter and Anderson, 2005). In uncertainty analysis can potentially be used to identify which para-
Rojas et al. (2008) the models that did not meet the convergence ac- meters are relevant to the predictions and to what extent they can be
ceptance criteria were assigned a likelihood of zero, eliminating their constrained by the available data.
contribution to the model ensemble predictive distribution. However, Third, the information content in the model testing data is in
conceptual models that do not converge may potentially be valid if no many studies relatively limited (e.g. Rojas et al., 2010c). The in-
effort towards making them converge is made. The effort towards formation content of model testing data relates to the amount and
making a model converge in the consensus approach will probably be type of data available, but also the uncertainty of the data. For ex-
larger than in the multi-model approach as there will still be other ample, as discussed in relation to comparing calibrated hydraulic
models left, in the latter approach. conductivity values to observed hydraulic conductivity values in
Section 4.2, such comparison can be unreliable. The consequence of
4.2.4. Model testing step 4 only limited information content in the model testing data is that
The last model testing step considered in this review is the post- discrimination among alternative models often cannot be made
audit. The post-audit is performed years after the end of the modelling (Seifert et al., 2008). In addition, in a Bayesian context the con-
process, evaluating forecasts of the model on newly collected data. sequence of limited information content in the testing data is that the
Anderson and Woessner (1992) summarize some modelling studies that prior probability will have a large influence on the posterior prob-
have used post-audits while Bredehoeft (2005) focussed on identifying ability (e.g. Rojas et al., 2009).
the conceptual surprises that occurred in these modelling studies as a Another challenge relates to when a model can be considered fal-
result of a post-audit. The advantage of the post-audit is that the model sified. Models are groups of hypotheses rather than a single hypothesis
testing data is by default independent from the model development in itself and many other assumptions are made in groundwater models
data, satisfying one of the basic requirements of model testing data such as model code and the characterization of lower order uncertainty.
(Section 4.1). However, it is inconvenient to rely on this type of model The model prediction thereby depends on many interactions of in-
testing as there may potentially be a long waiting period from the end dependent hypotheses and assumptions. Inconsistencies between model
of the model process until new data is collected. and data should therefore not necessarily be attributed to a single hy-
pothesis and result in the falsification of that hypothesis (Pfister and
4.3. Remaining challenges Kirchner, 2017).
To accommodate these challenges, a more systematic approach to
This review has shown that models can be tested in at least four model development and testing is needed, where parts of the available
different steps in the modelling process: i) as a conceptual model, ii) as data are used only for model testing. Ideally the data selected for model
a simple numerical model, iii) as a complex numerical model and iv) as testing should depend on the model objective and the information
a complex numerical model years after development. In each step the content should be large enough to discriminate between models. There
prior probability can be updated and sometimes models can be rejected is thereby an opportunity for systematic (quantitative or qualitative)
based on lack of support by observation of state variables, parameters assessment prior to study (i) which aspects of the model will be relevant
or because the model did not converge. Identifying suitable data for to the objectives and (ii) what data are needed to distinguish between
model testing remains challenging. hypotheses.
First, in theory the notion that testing data should be independent
is sound, but in practice the separation of data is difficult. Many 5. How are different conceptualizations used for predictions?
studies rely on ranking criteria to update the prior probability (which
we will discuss in Section 5.1), rather than updating prior probability What has emerged from several of the studies so far in this review is
based on data that is independent of the model development. In using that multiple plausible models may coexist for a given study area. So,
all data when developing models, it is no surprise that the models how are predictions made with multiple models? For some studies (e.g.
actually fit data. Post-hoc theorizing can easily result in under- Foglia et al., 2013), one model (the most likely based on the highest
sampling of the model space (Kerr, 1998), as an initial range of support in data) is selected for predictive purposes (Section 4.1), while
plausible models will be accepted (because of circular reasoning) other studies (e.g. Tsai and Li, 2008) focus on ensemble predictions
319
T. Enemark et al. Journal of Hydrology 569 (2019) 310–329
based on all plausible models (Section 4.2). A modelling step that re- information criteria to reduce computational effort constituting the
ceives increasing attention in the literature is the identification of ad- Maximum Likelihood BMA (MLBMA) approach suggested by Neuman
ditional data needs in order to be able to discriminate between the (2003). Given many of the information criteria are developed as model
alternative conceptual models (e.g. Kikuchi et al., 2015) (Section 4.3). selection criteria, they tend to assign a large weight to only a few
The last four columns in Table A.2 present an overview of approaches models (e.g. Nettasana, 2012; Rojas et al., 2010c; Ye et al., 2010),
being adopted when making predictions with multiple models. As which is the main drawback of the MLBMA approach. This leads to the
mentioned in the introduction, several literature reviews (Diks and introduction of a statistical scaling factor to the information criteria
Vrugt, 2010; Schöniger et al., 2014; Singh et al., 2010) have already (Tsai and Li 2008), leading to a flatter weight distribution among the
focussed on the model prediction and evaluation aspect of the multi- alternative models.
model approach. It is therefore not the aim to give a comprehensive One of the disadvantages of the averaging procedures is that the
review here, but to give a general overview of the most often applied system details of how each conceptual model affects the prediction, is
approaches and instead focus on how the model development approach lost (Gupta et al., 2012). To solve this problem, Tsai and Elshall (2013)
(discussed in Section 3) affects the predictions. suggested the hierarchal BMA (H-BMA) approach where the individual
conceptual model components are evaluated through a BMA tree. In the
5.1. Model weighing and selection techniques BMA tree model components are organized at separate levels and the
contribution of uncertainty of each aspect to the total uncertainty is
Model weighing and selection techniques rank models according to quantified. By separating the uncertain model components in a BMA
how well they fit data, where the models with the lowest rank or weight tree, the different aspects can be prioritized and provide an under-
have least support in the data. The purpose of ranking is to select the standing of the uncertainty propagation through each uncertain aspect
“best” model, but for many of the studies in Table A.2 ranking also in the conceptual model.
provides weights for a model averaging technique (Section 5.2). For an
excellent review of model selection techniques the reader is referred to 5.3. Identify additional data needs
Schöniger et al. (2014).
In selecting between models, two principles often receive attention: Refining the prediction made by multiple models may sometimes be
The Principle of Parsimony (favouring the simplest model) and The necessary in order to decrease the range of model predictions.
Principle of Maximum Likelihood (favouring the model that gives the Considering too many conceptual models, one may lose the purpose of
highest chance to facts we have observed). However, the Principle of model development because it indicates high model prediction un-
Consistency (favouring models that do not contradict any effects we certainty (Bredehoeft, 2005; Højberg and Refsgaard, 2005). Therefore,
know) is even more important to consider when choosing between some studies have focussed on identifying additional data needs that
models (Martinez and Gupta, 2011). The most commonly applied could potentially discriminate between alternative conceptual models
ranking techniques in the analysed studies in Table A.2 are the In- to reduce conceptual uncertainty (e.g. Kikuchi et al., 2015; Pham and
formation Criteria, including Akaike’s Information Criterion (AIC) Tsai, 2015, 2016). The goal of collecting new data is not to confirm
(Akaike, 1973), corrected AIC (AICc) (Sugiura 1978; Hurvich and Tsai existing conceptual models, but to be able to discriminate between
1989), Bayesian Information Criterion (BIC) (Schwarz, 1978) and Ka- them.
shyap Information Criterion (KIC) (Neuman, 2003) and GLUE. The Kikuchi et al. (2015) offers a short review of optimal design studies
ranking from the information criteria depends on an error term re- in hydrogeology that attempt to identify the optimal measurement sets
presenting model fit to observations and a penalty term that penalizes for monitoring networks to maximize a data utility function. For a few
model complexity. In GLUE the ranking is only based on an error studies conceptual model discrimination is the design objective
term. (Knopman et al., 1991; Knopman and Voss, 1988, 1989; Usunoff et al.,
1992; Yakirevich et al., 2013), but this approach has yet not received
5.2. Model averaging techniques much attention in hydrogeology according to Kikuchi et al. (2015).
Identifying additional data needs will guide the post audit activity
Model averaging techniques seek to summarize the results from the (Section 4.2) and the use of these data for model testing will ensure the
multiple model approach into an optimal prediction and a single data is independent from the model development data.
measure of the total uncertainty by averaging the posterior distribu-
tions (Raftery et al., 2005). This posterior is obtained through an 5.4. Remaining challenges
averaging approach that weighs the different model predictions ac-
cording to the weight they obtained from the testing or ranking, com- This review shows that current studies often either used criteria-
bined with a prior probability of the individual models. For excellent based weights, either to identify the most plausible models or to pro-
summaries of model averaging techniques the reader is referred to Diks vide weights for a model averaging technique. The current methods are
and Vrugt (2010) and Singh et al. (2010). generally limited by what is attainable through the model development
The most commonly applied approach to averaging predictions of approach. The main limitations and thereby consequences of the model
conceptually different hydrogeological models is Bayesian Model development approach for current methods on making predictions with
Averaging (BMA) (Hoeting et al., 1999). The averaged predictions from multiple conceptualizations are discussed next.
multiple models have been shown to be more robust and less biased First, we can never make sure that we have developed a collectively
than the prediction from a single model (Vrugt and Robinson, 2007). exhaustive range of conceptual models (e.g. Ferré, 2017; Hunt and
Furthermore, they produce a more realistic and reliable description of Welter, 2010; Nearing and Gupta, 2018) (as discussed in Section 3) but
the predictive uncertainty (Rojas et al., 2010a). the prediction methods and the approaches in identifying additional
The Bayesian model evidence is sometimes approximated with the data types rely on this. Undersampling the model space will lead to
320
T. Enemark et al. Journal of Hydrology 569 (2019) 310–329
underestimation of the prediction uncertainty in the model averaging been discussed in Section 3.5. The challenges mentioned in the re-
approaches. Furthermore, by focussing the collection of additional data maining two points occur because of the reliance on methods that
on data that can discriminate between currently known con- assume a single best model can be found. A way forward to accom-
ceptualizations, it is assumed that we already know all plausible con- modate these challenges could be full probabilistic approaches.
ceptualizations. A challenge remains in directing additional data col- Transdimensional inference methods have been applied in geophysics
lection towards uncovering unknown unknown plausible conceptual (e.g. Ray and Key, 2012) and reservoir geology (e.g. Sambridge et al.,
models. 2006) for similar problems. In these approaches, e.g. reversible jump
Second, we can never make sure that the adopted range of models Markov Chain Monte Carlo (Green, 1995), sampling occurs within the
developed is valid (Type II error) (e.g Nearing and Gupta, 2018) same dimension (conceptual model), but also between dimensions
nevertheless both the BMA and the criteria-based model weighing (conceptual models) exploring both the conceptual model space and
techniques rely on the best approximation of reality being in the en- the parameter space.
semble. In the model selection approaches we can therefore never make
sure that the best approximation of reality is selected as it will always 6. Conclusion
be conditional on the developed range of models. In the model aver-
aging approaches, adopting an invalid range of models leads to biased A review of 59 studies applying the multi-model approach for hy-
predictions, which remains a challenge. drogeological conceptual model development, has shown the following:
Third, in BMA it is assumed that models are mutually exclusive, so
that none of the predictions are given a higher weight, following al- 1. A significant linguistic uncertainty still exists of what is considered
most identical models give similar predictions. Not having mutually conceptual uncertainty. There is a need for more consistent termi-
exclusive models gives a false sense of confidence in the modelling nology.
results, as a large number of alternative models considered will give 2. Current studies in conceptual model uncertainty often only focus on
the impression that a large range of the model space has been un- a single or limited set of conceptualization issues. There is a need for
covered. a systematic conceptualization approach to ensure all aspects of
Fourth, the criteria-based model weighing techniques rely only on conceptualization are covered and documented.
the Principle of Parsimony and the Principle of Maximum Likelihood, 3. Current studies rarely consider the objective of the model before
while the Principle of Consistency is disregarded through calibration. developing alternative models for the multi-model approach. The
Through the calibration step the model is trained to compensate for a objective of the model should have an influence on both the model
possible conceptual error through biased parameters (Refsgaard development and the data used for model testing.
et al., 2012; White et al., 2014) and the Principle of Consistency is 4. For each conceptual issue identified, alternative conceptual models
therefore not taken into account. Criteria-based model weighing should be formulated as hypotheses which, at least in theory, can be
techniques use the same data twice in the modelling process, which as refuted. Hypothesis testing, especially bold hypothesis testing, is
discussed in Section 4.1, leads to circular reasoning giving a false essential to increase system understanding and avoiding conceptual
confidence in the result. Also, inconsistent assumptions in the con- surprises.
ceptual model cannot be identified without introducing new data, but 5. In Bayesian inference with multiple models, informed priors are
in the criteria-based model weighing techniques, models are readily recommend, especially if the information content in the hypothesis
rejected through zero-weight as they tend to inflate the weights of a testing data is low.
few best models (e.g. Ye et al., 2010). The models that best com- 6. The current multi-model prediction methods assume that there is a
pensate for conceptual errors through biased parameters are then single outcome of the modelling process and that the developed
combined to make predictions through model averaging, where it is models are mutually exclusive and collectively exhaustive.
claimed that conceptual model uncertainty is taken into account. Presenting results requires a shift in mentality towards presenting
However, given the biased parameters of the models, circular rea- ranges and acknowledging that unknown unknowns exist.
soning and rejection of plausible models, this result may be both
biased and over-conservative. The multi-model approach is superior to the consensus approach as
Last, the model averaging techniques assume that a single result is it is transparent and accounts for conceptual uncertainty. However, to
valid, however if the range of plausible model are mutually exclusive, benefit fully from the multi-model approach, challenges remain in
they might lead to distinctly different predictions. One model might being more systematic in regards to both developing and testing al-
have a distinctly different prediction than the ensemble average or ternative models.
the probability mass may concentrate in multiple areas. This is the
case for the synthetic example in the study by Kikuchi et al. (2015),
where the spring flow depletion prediction is bimodal. In this case the Declaration of interests
average prediction is an outlier to where the probability mass is
concentrated. The average prediction of an ensemble, especially bi- or None.
multi-modally distributed ensembles, may not be a valid model out-
come (Winter and Nychka, 2010). It is therefore preferable to sum- Acknowledgement
marise ensembles through more robust metrics, such as percentiles
(e.g. 5th, 50th and 95th) as these will always be actual results made The authors would like to thank Russell Crosbie, Rebecca Doble,
by a model. Hoshin V. Gupta, Ty Ferré, Rodrigo Rojas and one anonymous reviewer
Suggestions on solving the remaining challenges in relation to for review and constructive comments on the manuscript. This research
populating the model space (first, second, third point) has already was conducted as part of a PhD project funded by CSIRO.
321
Appendix A
Table A1
T. Enemark et al.
Examples of approaches to develop conceptually different models for the Conceptual Physical Structure (Ph), Conceptual Process Structure (Pr) and the Spatial Variability Structure (SVS). Approaches to developing
different models include hypothesis testing (H), complexity testing (C) and interpretation testing (I), i.e. Fig. 3. If the model objective is defined in the introduction of the paper the objective of the model is here considered
well defined. The model objective is relevant to this table as the model objective should have an impact on what to include in the conceptualization.
Study Is the model objective well Conceptual multi-model development approach Ph Pr SVS
defined?
Altman et al. (1996) Yes Two different representations describing unsaturated zone flow through fractured media including equivalent continuum and a dual H
permeability model.
Aphale and Tonjes (2017) No Top of semi-confining unit either as uniform surface or undulating based on interpolation between boreholes (H). H H H
Northern extent of semi-confining unit represented by two different models (H).
Vertical discretization of downward fining sediment in aquifer as either uniform or variable (H).
Landfill effect on recharge either (i) no effect on recharge, (ii) recharge diverted to recharge basins adjacent to the landfill mounds,
(iii) all recharge collected for off-site treatment (H).
Drains segmented or not (H).
Carrera and Neuman (1986) No Ten alternative zonation patterns of hydraulic conductivity for synthetic aquifer. C
Castro and Goblet (2003) Yes Four alternative models where constraints within a formation is imposed (i.e., linear, exponential or with increasing distance decrease H
in hydraulic conductivity or constant hydraulic conductivity values for all formations).
Elshall and Tsai (2014) No Two different geological formation dips propositions (H). H H
Three indicator geostatistical methods for representing geometry: indictor zonation, generalized parameterization and indicator
kriging (H).
Engelhardt et al. (2014) No Seven alternative conceptual models varying the number of parameters (horizontal and vertical hydraulic conductivity and specific C
yield) in 10 homogeneous zones by lumping zones together.
Feyen and Caers (2006) Yes Two different training images representing two different braiding and sinuosity scenarios of a fluvial system (H). H
Three different affinity and angle maps representing local variation in channel width and orientation (H).
322
Three different variogram types: spherical, exponential or Gaussian (H).
Foglia et al. (2007) No Five alternative models that differs in zonation of hydraulic conductivity. Alternatives developed by lumping together different zones C
of homogeneous hydraulic conductivity.
Foglia et al. (2013) Yes Two different bedrock geometries defining the bottom of the groundwater system based on different data (I) I C I/H
Five different zonation of hydraulic conductivity (C).
Recharge either zero, spatially uniform, zonated based on soil types or simulated through rainfall-runoff model (I).
Streams are described with MODFLOW’s SFR and River package in alternative models imposing different assumptions (H).
Gedeon et al. (2013) Yes An initial model including a crude description of e.g. a clay aquitard and an update of the initial model including new information to N/A N/A N/A
update the description of the aquitard. This is an example of a consensus approach allowing for updates and the classification system
presented by Fig. 3 therefore does not apply.
Harrar et al. (2003) Yes Two manually created alternative geological models are based on the same data and contains the same five sediment types but is I
interpreted by two different geologist. They differ in regards to the way the sediment type is assigned to the cells based on borehole
data and the number of layers. Thereby one model reflects a more heterogeneous system while the other reflects a stratified system.
He et al. (2014) No Two training images for an MPS algorithm where one is based on SkyTEM data and the other is based on a Boolean simulation. H
Hermans et al. (2015) Yes In the field example four different training images are produced through a Boolean simulation for an MPS algorithm to describe H
variation between sand, clay and gravel.
Hills and Wierenga (1994) Yes Unsaturated zone and transport models developed by five different teams. The models differed in regards to soil being modelled as I
isotropic or anisotropic and homogeneous or heterogeneous.
Højberg and Refsgaard (2005) Yes Three hydrogeological models manually generated by three different teams for different purposes. I
Johnson et al. (2002) Yes A one-layer, two layer and three layer model is considered to represent a layered basalt and interbedded sediment aquifer. H
Kikuchi et al. (2015) Yes Inclusion of zero, one or two lenses of higher hydraulic conductivity in an otherwise homogeneous unconfined aquifer (H). H H
Mountain front recharge as either a continuous line parallel to mountain front or through discrete stream features (H).
Two models with and without underflow through subsurface zone to adjacent basin (H).
(continued on next page)
Journal of Hydrology 569 (2019) 310–329
Table A1 (continued)
Study Is the model objective well Conceptual multi-model development approach Ph Pr SVS
defined?
T. Enemark et al.
Knopman and Voss (1988), Knopman and Voss Yes Input of solute at upstream boundary of either i) constant, ii) decaying or iii) spatially varying initial condition (H). C H
(1989) Two different models in regards to whether first-order decay is affecting the transport (H).
One or three layers to describe the medium of well-sorted sand and gravel (C)
Knopman et al. (1991) Yes One-dimensional models of solute transport differing in regards to whether first-order decay is affecting the transport (H). C H
One, two or three layer to describe the medium of well-sorted sand and gravel (C)
La Vigna et al. (2014) Yes Three models considered to explain connection between two sand aquifers is i) outside of groundwater model, ii) through silty-sandy H
lense and 3) through old, not backfilled well.
Lee et al. (1992) Yes Homogeneous, layered and randomly heterogeneous geologic description to model tracer migration. C
Li and Tsai (2009) Yes In the Baton Rouge Area case study: Three different influences of a fault in regards to connectivity between aquifers is considered: i) H
impermeable fault model, ii) low permeability model and iii) no fault model.
Linde et al. (2015a) No Two training images for an MPS algorithm where one is based on a local outcrop and the other is based on an aquifer analogue. H
Lukjan et al. (2016) No Two hydrogeological interpretations, homogeneous or zoned (C). C H
Five models by combining different outer boundary conditions as either head or no-flow boundaries (H).
Mechal et al. (2016) No Two different models with two different fault sets and one model not representing faults at all (H). H C C/H
Five models with increasing number of transmissivity zones (C).
Two models with one representing all rivers and one only representing the major river (C).
Two models of lateral boundary conditions where one considers outflow to an adjacent aquifer and one does not (H).
Meyer et al. (2003) No Nine different variogram models to explain log air permeability variation in unsaturated fractured tuff. H
Meyer et al. (2007) Yes Two alternative models of spatial distribution of K: Homogeneous and zoned. C C
A steady-state and a transient boundary condition to a stream.
Nettasana (2012); Nettasana et al. (2012) No/Yes Three/two different independent interpretations of geology that differ in regards to e.g. number of layers (I). I C/H
Two different zonation of recharge based on either soil type, or soil type and land use (C).
Two models where some lateral boundaries are either no-flow or head boundaries to test outflow to adjacent aquifers (H).
Nishikawa (1997) Yes Two models of different geometry where in the first the aquifers are horizontally layered and in the second the layers are folded H
offshore which would create a shorter pathway for seawater to intrude through an outcrop.
323
Nordqvist and Voss (1996) Yes Three models differing in zonation of transmissivity values, i) including description of esker core and outwash material, ii) a C
homogeneous model, iii) including an esker core with a discontinuity and outwash material.
Passadore et al. (2011) Yes Alternative descriptions of how aquitards pinches out in sedimentary basin affecting the connectivity of aquifers. H
Pham and Tsai (2015, 2016) No Geological description by either indicator kriging, indicator zonation or general parameterization (H). H C
Two different fault permeability architectures: i) the same for all lithologies or ii) different for the three different lithologies (C).
Poeter and Anderson (2005) No 61 alternatives models by varying number and distribution of hydraulic conductivity zones generated by Sequential indicator C
simulations.
Refsgaard et al. (2006) Yes In an example five different consultants are asked to assess the vulnerability of aquifers towards pollution. They solve this task with I I I
different models in terms of geometry, processes and casual relationships and end up with vastly different predictions.
Rogiers et al. (2014) Yes A geostatistical representation of an aquifer is tested against a homogeneous representation. Within the geostatistical representation C
50 realization are generated representing the lower order uncertainty.
Rojas et al. (2008) No Seven alternative representations of geometry in a synthetic study differing in regards to number of layers and which layers are spatial I
correlated.
Rojas et al. (2010a) Yes Models either consider a one or a two layer hydrostratigraphic system. H H H
The hydraulic conductivity field is either described by i) constant hydraulic conductivity for each layer, ii) spatial zonation approach
within the layer or iii) using Random Space Functions either conditional or unconditional.
Recharge inflows originating from an eastern sub-basin described as i) diffuse recharge rates distributed over small areas of an alluvial
fan, ii) point recharge fluxes at the apex of an alluvial fan or iii) recharge fluxes distributed over long sections of the eastern boundary.
An additional recharge mechanism spatially distributed over the entire model domain that assumes a connection to adjacent aquifer is
tested.
Rojas et al. (2010c) Yes Three alternative descriptions of geometry differing the number of hydrostratigraphic units included to test the worth of “soft” H
geological knowledge.
(continued on next page)
Journal of Hydrology 569 (2019) 310–329
Table A1 (continued)
Study Is the model objective well Conceptual multi-model development approach Ph Pr SVS
defined?
T. Enemark et al.
Samani et al. (2017) No Three models consisting of different number of zones of hydraulic conductivity (C). C C/H
Recharge divided in four or five zones (C).
Highland recharge represented by either i) a head boundary or ii) a flux boundary (H).
River represented by either i) recharge boundary or ii) flux boundary (H).
Samper and Neuman (1989) No Five different semi variogram models (exponential, quadratic, spherical, pure nugget and exponential with nugget). H
Schöniger et al. (2015) Yes Four alternative representations of a sandbox in a synthetic study going from simple to complex (homogenous through zonation/ C
layered to geostatistical based on pilot points and to fully geostatistical).
Seifert et al. (2008) Yes Two alternative model developed with and without the representation of a palaeovalley. For the study area the presence of the H
palaeovalley is known, but it is investigated what the impact on predicted vulnerability would be if the existence of the palaeovalley
was not known.
Seifert et al. (2012) No Five alternative hydrostratigraphic models were generated by five different (hydro) geologists in a manual approach to geological I
model building.
Selroos et al. (2002) Yes Three different models describing the flow through fractured rock: i) Stochastic continuum, ii) discrete fractures, or iii) channel I
network.
Troldborg et al. (2007) No Four alternative models developed different in regards to a global hypothesis about depositional history, zonation of an aquifer and H/I
which well logs to use for the interpretation.
Troldborg et al. (2010) Yes Two models that differ in regards to contact between two sand aquifers potentially separated by a clay layer (H). H H
Two models with a different description of source zone for contamination (H).
Tsai (2010) Yes Experimental, spherical and Gaussian semivariogram models to describe hydraulic conductivity distribution. H
Tsai and Elshall (2013) No Three alternative variogram to explain spatial variability of the hydrofacies (exponential, pentaspherical and Gaussian) (H). H H/C
One variogram applied globally or local variograms by dividing model domain in zones (C)
Two fault model or one fault model dividing the model domain into three or two zones respectively (H).
Tsai and Li (2008) No Voronoi tessellation, natural neighbour interpolation, inverse, square distance interpolation, ordinary kriging and three Generalized H
Parameterization methods (that are combinations of previous zonation approaches) to parameterize hydraulic conductivity.
Usunoff et al. (1992) No Three different models describing solute transport with the processes: i) Fickian dispersion and diffusion, ii) fickian dispersion and H
324
neglected diffusion and iii) non-fickian dispersion and diffusion.
Yakirevich et al. (2013) Yes Two models where one described a layered media and the other described a layered media with lenses based on boreholes. C
Ye et al. (2004) No Seven alternative variogram models for log permeability variations in unsaturated fractured tuff H
Ye et al. (2010), Reeves et al. (2010) No Five geological interpretations by three different companies. Three models are developed in response to non-unique interpretations of I/H I/H
specific geological features (a thrust fault, a barrier to groundwater flow and a combination of the two).
Five groundwater recharge scenarios informed by different methods (chloride mass balance, net infiltration method, Maxey-Eakin
method) (I). Also included the effect of a surface water runon-runoff component and whether recharge occurs beneath a specific
elevation in some models to test these hypothesis (H).
Zeng et al. (2015) No Seven different representation of geometry by varying number of layers and the hydraulic conductivity distribution within the layers H
in a synthetic study.
Zhou and Herath (2016) Yes Three different models of geometry varying the number and extent of layers in a synthetic study. H
Zyvoloski et al. (2003) Yes To explain large hydraulic gradient a baseline model features a low permeability east-west zone, but there is no evidence for this H
feature, therefore three other models are proposed: i) Lower permeability hydrothermal alteration zone, ii) Alteration zone and NW-
SE trending fault zone, iii) like the aforementioned but with additional fault features.
Journal of Hydrology 569 (2019) 310–329
Table A2
Examples of approaches to test and make predictions with multiple plausible conceptual models. The ‘Prior’ column specifies if the prior probability in a Bayesian context is uninformed or informed by data or expert
opinion. The sub-columns in the ‘Model Testing’ and ‘Model Predictions’ columns refer to modelling steps in the guideline by (Neuman and Wierenga, 2003). The fourth model testing step, the post-audit, is not included in
this table as only one reviewed study (Nordqvist and Voss, 1996) applied this step (see Table 1). In the model testing steps the data type used for testing in the different steps are specified. In ‘Model Prediction’ the method
T. Enemark et al.
used for ranking and making predictions is provided, where ‘X’ refers to methods not specified in the text. Additional data needs refers to the process of identifying additional data that could potentially discriminate
between the conceptual models (as opposed to reducing parameter or prediction uncertainty).
Study Prior Model Testing Model Predictions
Uninformed/ Step 1 Step 2 Step 3 Model Individual Ensemble Predictions Additional data
informed Ranking Predictions needs
325
(2005)
Johnson et al. (2002) – – – Drawdown – – – –
Kikuchi et al. (2015) Uninformed – – – – – X OD4
Knopman and Voss – – – – – X – OD4
(1988)
Knopman and Voss OD4
(1989)
Knopman et al. (1991) OD4
La Vigna et al. (2014) – – Hydraulic head – – – – –
Lee et al. (1992) – – – Tracer plume obs. – – – –
Li and Tsai (2009) Uninformed – – – IC var5 – MLBMA6 –
Linde et al. (2015a) – Geophysical data – – – – – –
Lukjan et al. (2016) Uninformed – – – IC1 X – –
Mechal et al. (2016) – – – Baseflow, transmissivity IC1 X – –
Meyer et al. (2003) Uninformed – – – IC1 – MLBMA6 –
Meyer et al. (2007) Uninformed – Hydraulic head, uranium – IC1 – MLBMA6 –
concentrations
Nettasana (2012) Uninformed, – – Hydraulic head IC1, GLUE7 – GLUE-BMA8, –
informed MLBMA6
Nettasana et al. (2012) – – – – – X – –
Nishikawa (1997) – – – Hydraulic conductivity. – X – –
Nordqvist and Voss – – – – – X – OD4
(1996)
Passadore et al. (2011) – Seismic data and stratigraphic – – – X – –
records
Pham and Tsai (2015) Uninformed – – – IC1 – H-(ML)BMA2 OD4
Pham and Tsai (2016) Uninformed – – – X – BMA9 OD4
Poeter and Anderson – – – Hydraulic conductivity. Model IC1 – X –
(2005) convergence.
Journal of Hydrology 569 (2019) 310–329
Uninformed/ Step 1 Step 2 Step 3 Model Individual Ensemble Predictions Additional data
informed Ranking Predictions needs
326
Ye et al. (2010) Informed – – – IC1, GLUE7 – GLUE-BMA7
Zeng et al. (2015) Uninformed – – Hydraulic head? Model convergence. – – GLUE-BMA7 –
Zhou and Herath (2016) – – – Water balance, travel time IC1 – – –
distribution.
Zyvoloski et al. (2003) – – – Flow paths are inferred from – X – –
hydrogeochemical data
1
Information Criteria including AIC, BIC, KIC etc. (IC).
2
Hierarchal Bayesian Model Averaging (H-BMA).
3
Cross-Validation (CV).
4
Optimal design (OD).
5
Information criterion corrected with variance window (IC var).
6
Maximum Likelihood Bayesian Model Averaging (MLBMA).
7
Generalized Likelihood Uncertainty Estimation Bayesian Model Averaging (GLUE-BMA).
8
Generalized Likelihood Uncertainty Estimation (GLUE).
9
Bayesian Model Averaging (BMA).
Journal of Hydrology 569 (2019) 310–329
T. Enemark et al. Journal of Hydrology 569 (2019) 310–329
327
T. Enemark et al. Journal of Hydrology 569 (2019) 310–329
Water Resour. 15, 75–83. https://doi.org/10.1016/0309-1708(92)90033-X. propositions and conceptual model uncertainty reduction. Adv. Water Resour. 83,
Krabbenhoft, D.P., Anderson, M.P., 1986. Use of a numerical ground-water flow model for 148–159. https://doi.org/10.1016/j.advwatres.2015.05.024.
hypothesis testing. Ground Water 24, 49–55. Poeter, E., Anderson, D., 2005. Multimodel ranking and inference in ground water
Krueger, T., Page, T., Hubacek, K., Smith, L., Hiscock, K., 2012. The role of expert opinion modeling. Ground Water 43, 597–605. https://doi.org/10.1111/j.1745-6584.2005.
in environmental modelling. Environ. Model. Softw. 36, 4–18. https://doi.org/10. 0061.x.
1016/j.envsoft.2012.01.011. Raftery, A.E., Gneiting, T., Balabdaoui, F., Polakowski, M., 2005. Using bayesian model
La Vigna, F., Demiray, Z., Mazza, R., 2014. Exploring the use of alternative groundwater averaging to calibrate forecast ensembles. Mon. Weather Rev. 133, 1155–1174.
models to understand the hydrogeological flow processes in an alluvial context (Tiber https://doi.org/10.1175/MWR2906.1.
River, Rome, Italy). Envrionment Earth Sci. 71, 1115–1121. https://doi.org/10. Ray, A., Key, K., 2012. Bayesian inversion of marine CSEM data with a trans-dimensional
1007/s12665-013-2515-8. self parametrizing algorithm. Geophys. J. Int. 191, 1135–1151. https://doi.org/10.
Lee, R.R., Ketelle, R.H., Bownds, J.M., Rizk, T.A., 1992. Aquifer analysis and modeling in 1111/j.1365-246X.2012.05677.x.
a fractured heterogeneous medium. Ground Water 30, 589–597. Reeves, D.M., Pohlmann, K.F., Pohll, G.M., Ye, M., Chapman, J.B., 2010. Incorporation of
Leterme, B., Mallants, D., Jacques, D., 2012. Sensitivity of groundwater recharge using conceptual and parametric uncertainty into radionuclide flux estimates from a frac-
climatic analogues and HYDRUS-1D. Hydrol. Earth Syst. Sci. 16, 2485–2497. https:// tured granite rock mass. Stoch. Environ. Res. Risk Assess. 24, 899–915. https://doi.
doi.org/10.5194/hess-16-2485-2012. org/10.1007/s00477-010-0385-0.
Li, X., Tsai, F.T.C., 2009. Bayesian model averaging for groundwater head prediction and Refsgaard, J.C., Christensen, S., Sonnenborg, T.O., Seifert, D., Højberg, A.L., Troldborg,
uncertainty analysis using multimodel and multimethod. Water Resour. Res. 45, L., 2012. Review of strategies for handling geological uncertainty in groundwater
1–14. https://doi.org/10.1029/2008WR007488. flow and transport modeling. Adv. Water Resour. 36, 36–50. https://doi.org/10.
Linde, N., Lochbühler, T., Dogan, M., Van Dam, R.L., 2015a. Tomogram-based compar- 1016/j.advwatres.2011.04.006.
ison of geostatistical models: application to the Macrodispersion Experiment (MADE) Refsgaard, J.C., van der Sluijs, J.P., Brown, J., van der Keur, P., 2006. A framework for
site. J. Hydrol. 531, 543–556. https://doi.org/10.1016/j.jhydrol.2015.10.073. dealing with uncertainty due to model structure error. Adv. Water Resour. 29,
Linde, N., Renard, P., Mukerji, T., Caers, J., 2015b. Geological realism in hydrogeological 1586–1597. https://doi.org/10.1016/j.advwatres.2005.11.013.
and geophysical inverse modeling: a review. Adv. Water Resour. 86, 86–101. https:// Rogiers, B., Vienken, T., Gedeon, M., Batelaan, O., Mallants, D., Huysmans, M.,
doi.org/10.1016/j.advwatres.2015.09.019. Dassargues, A., 2014. Multi-scale aquifer characterization and groundwater flow
Lukjan, A., Swasdi, S., Chalermyanont, T., 2016. Importance of alternative conceptual model parameterization using direct push technologies. Environ. Earth Sci. 72,
model for sustainable groundwater management of the Hat Yai basin. Thailand. 1303–1324. https://doi.org/10.1007/s12665-014-3416-1.
Procedia Eng. 154, 308–316. https://doi.org/10.1016/j.proeng.2016.07.480. Rojas, R.M., Batelaan, O., Feyen, L., Dassargues, A., 2010a. Assessment of conceptual
Martinez, G.F., Gupta, H.V., 2011. Hydrologic consistency as a basis for assessing com- model uncertainty for the regional aquifer Pampa del Tamarugal – North Chile.
plexity of monthly water balance models for the continental United States. Water Hydrol. Earth Syst. Sci. Discuss. 6, 5881–5935. https://doi.org/10.5194/hessd-6-
Resour. Res. 47, 1–18. https://doi.org/10.1029/2011WR011229. 5881-2009.
Mechal, A., Birk, S., Winkler, G., Wagner, T., Mogessie, A., 2016. Characterizing regional Rojas, R.M., Feyen, L., Batelaan, O., Dassargues, A., 2010b. On the value of conditioning
groundwater flow in the Ethiopian Rift: A multi- model approach applied to Gidabo data to reduce conceptual model uncertainty in groundwater modeling. Water
River Basin. Austrian. J. Earth Sci. 109. Resour. Res. 46. https://doi.org/10.1029/2009WR008822.
Meyer, P., Gee, G., 1999. Information on hydrologic conceptual models, parameters, Rojas, R.M., Feyen, L., Dassargues, A., 2009. Sensitivity analysis of prior model prob-
uncertainty analysis, and data sources for dose assessments at decommissioning sites, abilities and the value of prior knowledge in the assessment of conceptual model
NUREG/CR-6656. Washington, D.C. uncertainty in groundwater modelling. Hydrol. Process. 23, 1131–1146. https://doi.
Meyer, P.D., Ye, M., Neuman, S.P., Cantrell, K.J., 2003. Combined estimation of hydro- org/10.1002/hyp.
geologic conceptual model and parameter uncertainty. NUREG/CR-6843 Report. Rojas, R.M., Feyen, L., Dassargues, A., 2008. Conceptual model uncertainty in ground-
Washington, D.C. water modeling: combining generalized likelihood uncertainty estimation and
Meyer, P.D., Ye, M., Rockhold, M.L., Neuman, S.P., Cantrell, K.J., 2007. Combined Bayesian model averaging. Water Resour. Res. 44. https://doi.org/10.1029/
Estimation of Hydrogeologic Conceptual Model, Parameter, and Scenario Uncertainty 2008WR006908.
with Application to Uranium Transport at the Hanford Site 300 Area. US Nucl. Regul. Rojas, R.M., Kahunde, S., Peeters, L., Batelaan, O., Feyen, L., Dassargues, A., 2010c.
Commision NUREG/CR-6. Application of a multimodel approach to account for conceptual model and scenario
Moore, C., Doherty, J., 2005. Role of the calibration process in reducing model predictive uncertainties in groundwater modelling. J. Hydrol. 394, 416–435. https://doi.org/
error. Water Resour. Res. 41, 1–14. https://doi.org/10.1029/2004WR003501. 10.1016/j.jhydrol.2010.09.016.
Nearing, G.S., Gupta, H.V., 2018. Ensembles vs. information theory: supporting science Samani, S., Moghaddam, A.A., Ye, M., 2017. Investigating the effect of complexity on
under uncertainty. Front. Earth Sci. 1–8. https://doi.org/10.1007/s11707-018- groundwater flow modeling uncertainty. Stoch. Environ. Res. Risk Assess. 643–659.
0709-9. https://doi.org/10.1007/s00477-017-1436-6.
Nearing, G.S., Tian, Y., Gupta, H.V., Clark, M.P., Harrison, K.W., Weijs, S.V., 2016. A Sambridge, M., Gallagher, K., Jackson, A., Rickwood, P., 2006. Trans-dimensional inverse
philosophical basis for hydrological uncertainty. Hydrol. Sci. J. 61, 1666–1678. problems, model comparison and the evidence. Geophys. J. Int. 167, 528–542.
https://doi.org/10.1080/02626667.2016.1183009. https://doi.org/10.1111/j.1365-246X.2006.03155.x.
Neto, D.C., Chang, H.K., van Genuchten, M.T., 2016. A mathematical view of water table Samper, F.J., Neuman, S.P., 1989. Estimation of spatial covariance structures by adjoint.
fluctuations in a shallow aquifer in Brazil. Groundwater 54, 82–91. https://doi.org/ Water Resour. Res. 25, 373–384.
10.1111/gwat.12329. Sanford, W.E., Buapeng, S., 1996. Assesment of a groundwater flow model of the bangkok
Nettasana, T., 2012. Conceptual Model Uncertainty in the Management of the Chi River basin, Thailand using carbon-14-based ages and paleohydrology. Hydrogeol. J. 4.
Basin, Thailand. University of Waterloo, PhD Thesis. Scanlon, B.R., Healy, R.W., Cook, P.G., 2002. Choosing appropriate techniques for
Nettasana, T., Craig, J., Tolson, B., 2012. Conceptual and numerical models for sustain- quantifying groundwater recharge. Hydrogeol. J. 10, 18–39. https://doi.org/10.
able groundwater management in the Thaphra area, Chi River Basin. Thailand. 1007/s10040-0010176-2.
Hydrogeol. J. 20, 1355–1374. https://doi.org/10.1007/s10040-012-0887-6. Schöniger, A., Illman, W.A., Wöhling, T., Nowak, W., 2015. Finding the right balance
Neuman, S.P., 2003. Maximum likelihood Bayesian averaging of uncertain model pre- between groundwater model complexity and experimental effort via Bayesian model
dictions. Stoch. Environ. Res. Risk Assess. 17, 291–305. https://doi.org/10.1007/ selection. J. Hydrol. 531, 96–110. https://doi.org/10.1016/j.jhydrol.2015.07.047.
s00477-003-0151-7. Schöniger, A., Wöhling, T., Samaniego, L., Nowak, W., 2014. Model selection on solid
Neuman, S.P., Wierenga, P.J., 2003. A comprehensive strategy of hydrogeologic modeling ground: Rigorous comparison of nine ways to evaluate Bayesian model evidence.
and uncertainty analysis for nuclear facilities and sites. NUREG/CR-6805 311. Water Resour. Res. 50, 9484–9513. https://doi.org/10.1002/2014WR016062.
Nishikawa, T., 1997. Testing alternative conceptual models of seawater intrusion in a Schwartz, F.W., Liu, G., Aggarwal, P., Schwartz, C.M., 2017. Naïve simplicity: the over-
coastal aquifer using computer simulation, southern California, USA. Hydrogeol. J. looked piece of the complexity-simplicity paradigm. Groundwater 1–9. https://doi.
https://doi.org/10.1007/s100400050116. org/10.1111/gwat.12570.
Nordqvist, R., Voss, C.I., 1996. A simulation-based approach for designing effective field- Schwarz, G., 1978. Estimating the dimension of a model. Ann. Stat. 6, 461–464. https://
sampling programs to evaluate contamination risk of groundwater supplies. doi.org/10.1214/aos/1176344136.
Hydrogeol. J. 4, 23–39. Seifert, D., Sonnenborg, T.O., Refsgaard, J.C., Højberg, A.L., Troldborg, L., 2012.
Oreskes, N., Shrader-frechette, K., Belitz, K., 1994. Verification, validation and con- Assessment of hydrological model predictive ability given multiple conceptual geo-
firmation of numerical models in the earth sciences. Science (80-.) 263, 641–646. logical models. Water Resour. Res. 48, 1–16. https://doi.org/10.1029/
Passadore, G., Monego, M., Altissimo, L., Sottani, A., Putti, M., 2011. Alternative con- 2011WR011149.
ceptual models and the robustness of groundwater management scenarios in the Seifert, D., Sonnenborg, T.O., Scharling, P., Hinsby, K., 2008. Use of alternative con-
multi-aquifer system of the Central Veneto Basin. Italy. https://doi.org/10.1007/ ceptual models to assess the impact of a buried valley on groundwater vulnerability.
s10040-011-0818-y. Hydrogeol. J. 16, 659–674. https://doi.org/10.1007/s10040-007-0252-3.
Perko, J., Seetharam, S.C., Mallants, D., Vermariën, E., Wilmot, R., 2009. Long-term Selroos, J.-O., Walker, D.D., Ström, A., Gylling, B., Follin, S., 2002. Comparison of al-
evolution of the near surface disposal facility at Dessel. Project near surface disposal ternative modelling approaches for groundwater flow in fractured rock. J. Hydrol.
of category A waste at Dessel. 257, 174–188. https://doi.org/10.1016/S0022-1694(01)00551-0.
Pfister, L., Kirchner, J.W., 2017. Debates – hypothesis testing in hydrology: theory and Singh, A., Mishra, S., Ruskauff, G., 2010. Model averaging techniques for quantifying
practice. Water Resour. Researh 53, 1792–1798. https://doi.org/10.1002/ conceptual model uncertainty. Ground Water 48, 701–715. https://doi.org/10.1111/
2016WR020116.Received. j.1745-6584.2009.00642.x.
Pham, H.V., Tsai, F.T.C., 2016. Optimal observation network design for conceptual model Strebelle, S., 2002. Conditional simulation of complex geological structures using mul-
discrimination and uncertainty reduction. Water Resour. Res. 52, 1245–1264. tiple-point statistics. Math. Geol. 34, 1–21. https://doi.org/10.1023/
https://doi.org/10.1002/2015WR017474. A:1014009426274.
Pham, H.V., Tsai, F.T.C., 2015. Bayesian experimental design for identification of model Sugiura, N., 1978. Further analysts of the data by Akaike’ s information criterion and the
328
T. Enemark et al. Journal of Hydrology 569 (2019) 310–329
finite corrections. Commun. Stat. - Theory Methods 7, 13–26. https://doi.org/10. White, J.T., Doherty, J.E., Hughes, J.D., 2014. Quantifying the predictive consequences of
1080/03610927808827599. model error with linear subspace analysis. Water Resour. Res. 50, 1152–1173.
Sun, N.-Z., Yeh, W.W.G., 1985. Identification of parameter structure in groundwater in- https://doi.org/10.1002/2013WR014767.
verse problem. Water Resour. Res. 21, 869–883. Wingefors, S., Andersson, J., Norrby, S., Eisenberg, N.A., Lee, M.P., Federline, M.V.,
Suzuki, S., Caumon, G., Caers, J., 2008. Dynamic data integration for structural modeling: Sagar, B., Wittmeyer, G.W., 1999. Regulatory perspectives on model validation in
model screening approach using a distance-based model parameterization. Comput. high-level radioactive waste management programs: A Joint NRC/SKI White Paper.
Geosci. 12, 105–119. https://doi.org/10.1007/s10596-007-9063-9. Stockholm, Sweden and Washington DC, USA.
Tarantola, A., 2006. Popper, Bayes and the inverse problem. Nat. Phys. 2, 4–7. Winter, C.L., Nychka, D., 2010. Forecasting skill of model averages. Stoch. Environ. Res.
Tonkin, M., Doherty, J., Moore, C., 2007. Efficient nonlinear predictive error variance for Risk Assess. 24, 633–638. https://doi.org/10.1007/s00477-009-0350-y.
highly parameterized models. Water Resour. Res. 43, 1–15. https://doi.org/10.1029/ Woolfenden, L.R., 2008. Use of a groundwater flow model to assess the location, extent,
2006WR005348. and hydrologic properties of faults in the Rialto-Colton Basin. California, in:
Troldborg, L., Refsgaard, J.C., Jensen, K.H., Engesgaard, P., 2007. The importance of MODFLOW and More 2008, 78–82.
alternative conceptual models for simulation of concentrations in a multi-aquifer Yakirevich, A., Pachepsky, Y.A., Gish, T.J., Guber, A.K., Kuznetsov, M.Y., Cady, R.E.,
system. Hydrogeol. J. 15, 843–860. https://doi.org/10.1007/s10040-007-0192-y. Nicholson, T.J., 2013. Augmentation of groundwater monitoring networks using
Troldborg, M., Nowak, W., Tuxen, N., Bjerg, P.L., Helmig, R., Binning, P.J., 2010. information theory and ensemble modeling with pedotransfer functions. J. Hydrol.
Uncertainty evaluation of mass discharge estimates from a contaminated site using a 501, 13–24. https://doi.org/10.1016/j.jhydrol.2013.07.032.
fully Bayesian framework. Water Resour. Res. 46, 1–19. https://doi.org/10.1029/ Ye, M., Neuman, S.P., Meyer, P.D., 2004. Maximum likelihood Bayesian averaging of
2010WR009227. spatial variability models in unsaturated fractured tuff. Water Resour. Res. 40, 1–17.
Tsai, F.T.C., 2010. Bayesian model averaging assessment on groundwater management https://doi.org/10.1029/2003WR002557.
under model structure uncertainty. Stoch. Environ. Res. Risk Assess. 24, 845–861. Ye, M., Neuman, S.P., Meyer, P.D., Pohlmann, K.F., 2005. Sensitivity analysis and as-
https://doi.org/10.1007/s00477-010-0382-3. sessment of prior model probabilities in MLBMA with application to unsaturated
Tsai, F.T.C., Elshall, A.S., 2013. Hierarchical Bayesian model averaging for hydrostrati- fractured tuff. Water Resour. Res. 41, 1–14. https://doi.org/10.1029/
graphic modeling: uncertainty segregation and comparative evaluation. Water 2005WR004260.
Resour. Res. 49, 5520–5536. https://doi.org/10.1002/wrcr.20428. Ye, M., Pohlmann, K.F., Chapman, J.B., 2008. Expert elicitation of recharge model
Tsai, F.T.C., Li, X., 2008. Multiple parameterization for hydraulic conductivity identifi- probabilities for the Death Valley regional flow system. J. Hydrol. 354, 102–115.
cation. Ground Water 46, 851–864. https://doi.org/10.1111/j.1745-6584.2008. https://doi.org/10.1016/j.jhydrol.2008.03.001.
00478.x. Ye, M., Pohlmann, K.F., Chapman, J.B., Pohll, G.M., Reeves, D.M., 2010. A model-aver-
Tsang, C., 1991. The modelling process and model validation. Ground Water 29, aging method for assessing groundwater conceptual model uncertainty. Groundwater
825–831. 48, 716–728. https://doi.org/10.1111/j.1745-6584.2009.00633.x.
Tsang, C., 1987. Technical note: comments on model validation. Transp. Porous Media 2, Young, P., Parkinson, S., Lees, M., 1996. Simplicity out of complexity in environmental
623–629. modelling: occam’s razor revisited. J. Appl. Stat. https://doi.org/10.1080/
Usunoff, E., Carrera, J., Mousavi, S.F., 1992. An approach to the design of experiments for 02664769624206.
discriminating among alternative conceptual models. Adv. Water Resour. 15, Zeng, X., Wang, D., Wu, J., Zhu, X., Wang, L., Zou, X., 2015. Evaluation of a groundwater
199–214. https://doi.org/10.1016/0309-1708(92)90024-V. conceptual model by using a multimodel averaging method. Hum. Ecol. Risk Assess.
Vrugt, J.A., 2016. Markov chain Monte Carlo simulation using the DREAM software An Int. J. 21, 1246–1258. https://doi.org/10.1080/10807039.2014.957945.
package: theory, concepts, and MATLAB implementation. Environ. Model. Softw. 75, Zhou, Y., Herath, H.M.P.S.D., 2016. Evaluation of alternative conceptual models for
273–316. https://doi.org/10.1016/j.envsoft.2015.08.013. groundwater modelling. Geosci. Front. 8, 437–443. https://doi.org/10.1016/j.gsf.
Vrugt, J.A., Robinson, B.A., 2007. Treatment of uncertainty using ensemble methods: 2016.02.002.
Comparison of sequential data assimilation and Bayesian model averaging. Water Zyvoloski, G., Kwicklis, E., Eddebbarh, A.A., Arnold, B., Faunt, C., Robinson, B.A., 2003.
Resour. Res. 43. https://doi.org/10.1029/2005WR004838. The site-scale saturated zone flow model for Yucca Mountain: calibration of different
Walker, W.E., Harremoes, P., Rotmans, J., van der Sluijs, J.P., van Asselt, M.B.A., Janssen, conceptual models and their impact on flow paths. J. Contam. Hydrol. 62–63,
P., Krayer Von Krauss, M.P., 2003. A Conceptual Basis for Uncertainty Management. 731–750. https://doi.org/10.1016/S0169-7722(02)00190-0.
Integr, Assesment, pp. 4.
329