The need for control and replication in ecological assessments

Why are benchmarks needed in ecological assessments? Simply put, a benchmark should serve the same purpose as a control treatment in a manipulative experiment. Thus, benchmarks are one element of a study design that helps resource managers draw scientifically credible inferences regarding the ecological condition of the resources they manage. Like the controls used in a good manipulative experiment, benchmarks should accomplish 2 main tasks: 1) control for the effects of factors other than the ones being studied (e.g., natural factors vs anthropogenic stressors) and 2) provide sufficient replication to estimate the range of variation in the values of an ecological index that is associated with both natural variability and sampling variability. The extent to which we accomplish these 2 tasks will greatly affect the robustness of inferences regarding the ecological condition of a water body.

Experimental control and replication are fundamental requirements in any ecological assessment, but the specific way these design elements have been applied has been strongly influenced by practical constraints imposed by a heterogeneous landscape and by the statistical sophistication of practitioners. Tightly matched controls can be applied when conducting manipulative field experiments designed to evaluate the potential effects of one of more types of stressor or levels of a stressor on an ecological variable of interest (see review by Cooper and Barmuta 1993), but such highly controlled approaches are difficult if not impossible to use for post hoc assessments of the effects of environmental alterations that have already occurred (potentially over a period of ≥100 y) at individual sites (e.g., Underwood 1994). The need to conduct post hoc assessments at individual sites prompted development of approaches to experimental control that were designed to characterize better the sources and magnitude of natural variability that occur at the scale of entire ecosystems (e.g., Landres et al. 1999).

Two general approaches to matching control sites with an assessed site are frequently used in freshwater ecological assessments. The 1^st approach requires the identification of nonperturbed reference sites that are assumed to be sufficiently similar to one another and the predisturbance condition of the assessed site that they can be used as replicates of fixed controls. The 2^nd approach assumes that natural environmental and ecological variability among reference sites will be so great that their direct use as fixed replicates could mask human-caused effects (see following paragraph). Instead of using reference sites as fixed replicates, this approach uses biota–environment relationships derived from the reference sites to predict the most likely ecological reference condition at any individual assessed site (analogous to regression).

Replication is a critical component in any type of assessment, but its use in ecological assessment and monitoring has differed in important ways from its use in classic controlled experiments (i.e., analysis of variance [ANOVA]-type designs). Replication in controlled experiments assumes treatments are fixed, and replicates are used to estimate sampling error. That estimate of sampling error is then used to test statistically the likelihood that the observed control and treatment means are different. In ecological assessments of individual sites, the null hypothesis that control and treatment means are identical is not appropriate because there is only one treatment observation. It might be tempting to use replicate samples taken within an assessed site to conduct such a test, but the variance associated with within-site replicates typically will be smaller than the variance observed among reference sites (within-site + among-site variation), thereby invalidating the assumption of equal variance and, hence, the statistical test. If used in such tests, the replicate samples taken within a site represent pseudoreplicates (sensu Hurlbert 1984), and their use in a t-test (or similar test) would inflate the likelihood of observing a statistically significant difference in means. The appropriate hypothesis when assessing individual sites is whether the condition observed at that site is outside the range of index values expected among an appropriate set of reference sites (see McBride et al. 1993, Kilgour et al. 1998, Smith et al. 2005, Bowman and Somers 2006 for appropriate statistical approaches).

Precision, bias, and inferences in ecological assessments

Reliable inferences of ecological condition assume that predictions of the reference condition are both acceptably precise and unbiased. Ideally, the only variation associated with estimating the reference condition is that associated with random sampling variability, i.e., the variation in an ecological attribute that would occur among replicate samples taken at an individual site at the same time. In reality, sampling variability is usually confounded with 2 additional types of variation in ecological assessments: 1) natural, systematic variation that occurs over time at individual sites and that exists among the populations of reference sites used as spatial replicates and 2) systematic variation associated with prediction error. Because all natural ecosystems are inherently dynamic, they exhibit a range of conditions associated with temporal variation in both abiotic and biotic forces (Fig. 2). Short-term temporal variation can be associated with either predictable seasonal processes, such as life-history schedules of component species or effects of seasonal variation in climate, or stochastic events, such as individual floods or periods of extreme drought. Because of this seasonal variation in assemblage structure, and thus, metric or index values, assessments based on comparisons of samples taken in one season with reference expectations derived from other seasons will lead to biased predictions and incorrect inferences (e.g., Linke et al. 1999, Reece et al. 2001, Feio et al. 2006). Longer-term year-to-year differences in environmental conditions associated with climate can directly or indirectly affect annual variation in species composition, richness, or relative abundances via effects on the recruitment success and survivorship of individual species (McElravy et al. 1989, Bradley and Ormerod 2001, Scarsbrook 2002, Bêche and Resh 2007).

The effects of natural variation typically have been treated in different ways. Controlling for seasonal variability can be done, in part, by specifying a relatively short period within which samples are collected (e.g., an index period; Barbour et al. 1999; Fig. 1) or by pooling multiple samples taken across seasons to estimate the annual ecological conditions that characterize a site (e.g., Wright 2000). Year-to-year variation typically is considered to be an important component of the range of natural variability that characterizes a site. This type of variation is fundamentally different from random sampling variability in that it is clearly a consequence of reasonably well-understood causal processes. It is also an important component of ecological variation that should be used to assess whether sites are in reference condition (Landres et al. 1999). Spatial variation among reference sites is an outcome of the pervasive and multidimensional environmental heterogeneity that exists naturally across all landscapes and waterways. Thus, such sites can be, at best, coarse replicates of one another. In regionalization or typology approaches (i.e., any type of landscape classification), this variation is explicitly or implicitly assumed to represent part of the range of natural variation associated with a given water body type, and some lower value of that distribution of values is used as a benchmark below which an assessed site would be considered to be in nonreference condition. In site-specific approaches, this type of variation represents a potentially important confounding factor that can reduce estimates of precision and, therefore, should be taken into account by modeling.

Systematic prediction bias occurs as a consequence of nonrandom prediction errors and can occur in any type of prediction scheme. For example, given that we know that the distribution of biota varies strongly with stream size, a classification of sites that treats small and large reference streams as equivalent would probably either over- or underpredict the true reference value for any individual site. Such biased predictions also can occur in models that are developed to control for naturally occurring environmental variation among sites. Predictions at some sites will be inaccurate if the models imperfectly describe environment–biota relationships.

These issues of prediction errors have attracted increasing scrutiny as their existence and magnitude have become more apparent over the last ∼10 y. Most analyses of assessment error have focused on aspects of precision (Clarke 2000, Hawkins et al. 2000b [Fig. 1], Ostermiller and Hawkins 2004, Clarke and Hering 2006 [Fig. 1], Davy-Bowker et al. 2006 [Fig. 1], Nichols et al. 2006, Van Sickle et al. 2007, Stribling et al. 2008, Mazor et al. 2009). Recent research has been directed toward documenting and understanding the magnitude of systematic prediction errors (e.g., Ostermiller and Hawkins 2004, Hawkins 2006, Cao et al. 2007, Hargett et al. 2007, Rehn et al. 2007, Ode et al. 2008). These studies have made it increasingly clear that substantial variation in reference-site biota and the indices derived from them can be associated with natural gradients, even after modeling or adjusting for the effects of natural environmental variation. For example, as much as 30% of the variation in reference-site index values was associated with natural environmental gradients in the USA's recent National Wadeable Stream Assessment (USEPA 2006) (Table 4). The magnitude of this variation implies that assessment errors associated with biased predictions might be frequent and that the issue of bias needs much more attention than it has received.

A brief history of approaches used to estimate ecological benchmarks

Ecological benchmarks can be estimated in at least 4 ways: 1) use of the historical record, 2) extrapolation or interpolation from extant reference-quality sites of appropriate type, 3) hindcasting based on models that describe current stressor–indicator relationships, and 4) prediction derived from mechanistic models that describe how natural processes influence an ecological attribute of interest. Matching of existing reference-quality sites with assessed sites provides a means of estimating aspects of both assemblage composition (what taxa should occur) and structure (what abundances should occur) and is, by far, the main approach used in modern ecological assessments. Matching can be done in several ways, of which landscape regionalizations, typologies, and site-specific predictions are most common. Use of the historical record, hindcasting, and mechanistic modeling are much less well developed (discussed below).

Landscape regionalizations.—In an effort to produce ecological assessments broadly based on the design elements of control and replication, Hughes et al. (1986) advocated use of a regional reference site approach. In the regional reference site approach, some set of ecologically least-disturbed sites is identified within a relatively homogeneous terrestrial landscape (the controls), and the range of biotic attributes observed across this population of sites (the replicates) is used to estimate the range of natural variability that presumably would occur at an individual test site (a site of unknown condition) assuming it was in reference condition. If the value of an ecological index falls outside a threshold value determined from the distribution of reference site values, the site would be inferred to be in nonreference condition. Inferences regarding the condition of a test site initially were based on somewhat arbitrary and simple thresholds (e.g., the 10^th or 25^th percentile of reference-site values), but formal statistical tests (e.g., Kilgour et al. 1998, Bowman and Somers 2006) can be applied to such data to determine more rigorously whether an assessed site is in reference condition.

Hughes et al. (1986) proposed the use of ecoregions (Omernik 1987; Fig. 1) as landscape strata within which assessed sites would be compared with reference sites. The general rationale for using ecoregions as a basis for predicting aquatic biota was based, in part, on Hynes' (1975) view that streams are a product of the terrestrial ecosystems they traverse (see fig. 1 in Hughes et al. 1986). Hughes et al. (1986) suggested that if ecoregions are defined in terms of their climate, topography, geology, soil, and vegetation (Omernik 1987), then their streams should differ in water chemistry, flow regime, habitat structure, and food sources. In turn, these proximate environmental features should strongly influence the biotic character of streams. Thus, ecoregions were viewed as a way to partition (control for) the collective effects of the most important natural factors that control the distribution and abundance of aquatic biota. An important property of the ecoregion approach to regionalization is that the spatial units used for prediction are generally all unique, i.e., ecoregions are a type of geographic-dependent classification in which each class typically has only one member. Thus, no true replicates of ecoregions exist, and assessing the transferability of knowledge gleaned from one ecoregion (or set of ecoregions) to others is problematic.

The regional reference site approach developed nearly in parallel with a major effort by the US Environmental Protection Agency (EPA) to define and describe ecoregions within the USA (Omernik 1987), and subsequently, an ecoregional reference site approach was widely promoted by the EPA as a method for controlling, in whole or part, natural ecological variation when conducting biological assessments in the USA (Gibson et al. 1996 [Fig. 1], Barbour et al. 1999). A regionalization approach to defining reference conditions was subsequently applied elsewhere (e.g., European Commission 2000, Moog et al. 2004, Verdonschot and Nijboer 2004). Studies published in J-NABS did not lay the original foundations for the use of such landscape regionalizations in ecological assessments, but several papers that documented refinement of the approach and case studies of its application appeared in J-NABS (e.g., Lenat 1988, 1993, Barbour et al. 1996, Pan et al. 1996, 2000, Gerritsen et al. 2000, Maxted et al. 2000, Van Sickle and Hughes 2000, Waite et al. 2000).

Typologies.—The eventual recognition that almost any type of regionalization (ecoregion, drainage basin, political boundaries) was associated with about the same, and relatively low, amounts of biotic variation among sites (Hawkins et al. 2000a, Marchant et al. 2000, Sandin and Johnson 2000, Van Sickle and Hughes 2000, Waite et al. 2000, Herlihy et al. 2006, 2008, Herlihy and Sifneos 2008) implied that reassessment was needed of assumptions regarding which specific proximate features most strongly influence the distribution of aquatic biota (e.g., Power et al. 1988, Hawkins et al. 1997, Poff 1997, Mykrä et al. 2007) and what landscape and catchment attributes best predicted those specific features. This recognition anticipated development and testing of more finely resolved, geographic-independent classifications for streams based on spatially repeatable ecotypes or landscape typologies (e.g., Brierley and Fryirs 2000, Waite et al. 2000, Snelder and Biggs 2002 [Fig. 1], Fryirs 2003, Balestrini et al. 2004, Munné and Prat 2004, Snelder et al. 2004a, b, Higgins et al. 2005, Chessman et al. 2006, Seelbach et al. 2006, Schmitt et al. 2007, Orr et al. 2008). Although not initially a main thrust of stream classification research, such typological thinking had an earlier start in the hydrogeomorphic classification of wetlands (e.g., Brinson 1993 [Fig. 1], Brinson and Rheinhardt 1996, Shaffer et al. 1999). The mandate of the European Union Water Framework Directive (WFD) that ecological assessments must be comparable across member nations prompted several studies that assessed the effectiveness of different typologies in bioassessment (e.g., Lorenz et al. 2004, Verdonschot and Nijboer 2004, Dodkins et al. 2005, Ferréol et al. 2005, Aroviita et al. 2008, Turak and Koop 2008), especially regarding how to develop optimal typologies based on only those catchment and waterbody features that most strongly influence biota (e.g., Snelder et al. 2004b, 2007, Dodkins et al. 2005, Snelder and Hughey 2005, Heino and Mykrä 2006, Sánchez-Montoya et al. 2007, Aroviita et al. 2008). However, tests of the effectiveness of the typological approach in partitioning natural ecological variation have not produced encouraging results (Davy-Bowker et al. 2006, Inglis et al. 2008).

Papers published in J-NABS were among the first to explore the utility of a priori classifications other than ecoregions for bioassessment purposes (e.g., review by Hawkins et al. 2000a, Waite et al. 2000), and general ecological syntheses describing the ecological knowledge that underlies the conceptual foundations on which typologies are based were published in J-NABS (e.g., Power et al. 1988, Poff 1997). However, most of the recent advances in development and testing of typologies (see above) have been published elsewhere, although the pioneering analyses of Snelder et al. (2004b) is an important exception.

Site-specific prediction of reference conditions.—During the same time period that the regional reference site approach was being developed, an alternative approach to establishing reference conditions was being developed and refined in Great Britain. This approach was designed to derive nearly site-specific reference expectations by adjusting index values for the effects of natural environmental attributes that can vary both among and within regions. Moss et al. (1987) (see also Furse et al. 1984 [Fig. 1], Wright et al. 1984, Wright 1995 [Fig. 1], Clarke et al. 1996 [Fig. 1], 2003, Moss et al. 2001) developed statistical procedures (RIVPACS) to predict taxon-specific probabilities of detection (P) at different sites from naturally occurring environmental features. The statistical models on which RIVPACS-type systems are based are calibrated with observations made at many reference-quality sites, and the models relate taxon occurrences to multiple environmental gradients. In effect, this approach simultaneously models the niche relationships of many taxa.

In the original RIVPACS and its many refinements and derivatives (e.g., Reynoldson et al. 1995 [Fig. 1], Parsons and Norris 1996, Smith et al. 1999, Hawkins et al. 2000b, Simpson and Norris 2000 [Fig. 1], Joy and Death 2002, Davy-Bowker et al. 2006, Hawkins 2006, Kokeš et al. 2006, Feio et al. 2009, Poquet et al. 2009), no a priori classification of reference sites is used (unlike the regional reference or typology approaches). Instead, RIVPACS-type models predict P by first classifying reference-quality streams based on their taxonomic composition or assemblage structure (not their landscape or channel environmental attributes) and then applying a multivariate predictive model (e.g., linear discriminant functions models [LDMs]) to predict class membership from environmental variables that are generally invariant (unresponsive to anthropogenic activity). Because LDMs predict the probabilities that new observations belong to each of the different classes, the Ps at a site can be estimated by weighting the frequencies of occurrence of each taxon within each group by these probabilities of group membership. This weighting allows prediction of P continuously across the entire predictor variable space encompassed by the reference sites. In essence, the models interpolate the likelihood of collecting different taxa at a site from the reference-site groupings. Excessive extrapolation is prevented by determining if the predictor space occupied by an assessed site is outside that described by the reference sites (e.g., Moss et al. 1987, Clarke et al. 2003).

In the RIVPACS approach, taxon-specific Ps are used to derive ecological quality ratios (EQRs) (Clarke et al. 1996, 2003). For example, O/E_taxa is calculated by first summing the Ps to estimate the number of taxa expected (E) in a sample. The O/E_taxa index is the proportion of those specific taxa predicted to occur in the sample (i.e., E) that are observed (O). When used in conjunction with other ecological information, such as tolerance values, the Ps also can be used to estimate EQRs that assess if an observed tolerance-based biotic index value is different from expected (e.g., the Biological Monitoring Working Party [BMWP] indices; Clarke et al. 2003). In both cases, inferences regarding the ecological condition of a site are based on estimates of modeling error, i.e., the distribution of model residual values. In theory, the distribution of O/E values calculated at reference sites should have a mean of 1, and the variance of this distribution is a direct measure of the precision of a model. If these values are normally distributed, they allow the same type of statistical tests of whether an assessed site is in reference condition or not as previously mentioned (e.g., Kilgour et al. 1998, Bowman and Somers 2006). As discussed above, matching the spatial and temporal extents over which indices are calculated and how reference-site variability (i.e., modeling error + natural variation) is estimated should reduce Type I and II statistical errors of inference. Although EQRs typically have been estimated following the traditional RIVPACS modeling approach, EQRs (including O/E_taxa) also can be calculated from an a priori classification of sites (e.g., Aroviita et al. 2008, 2009). Moreover, in the null model approach developed by Van Sickle et al. (2005) as an aid in evaluating the performance of RIVPACS-type models, O/E is calculated by assuming that all reference sites belong to a single regional class.

As with the regional reference site approach, foundational research in developing site-specific reference conditions was not published first in J-NABS. However, J-NABS has published the 2^nd-most papers that the ISI Web of Knowledge (as of July 2009) identified from the keywords RIVPACS or AusRivAS (Australian River Assessment System) (15 of 106 papers). Moreover, a J-NABS paper (Reynoldson et al. 1997) has been cited more often (201 citations) than any of these other papers. Many of the J-NABS papers on RIVPACS-type topics focus on ways to improve predictions (Van Sickle et al. 2005, 2006 [Fig. 1], Cao et al. 2007, Mykrä et al. 2008b), better estimate and interpret prediction errors (Ostermiller and Hawkins 2004, Van Sickle et al. 2005, Yuan 2006, Ode et al. 2008, Yuan et al. 2008), or derive more sensitive indices based on model outputs (Van Sickle 2008). Other papers describe case studies that illustrate the utility and potential of a site-specific reference condition approach (Sloane and Norris 2003, Hose et al. 2004, Cao et al. 2007).

In contrast, J-NABS has not led in the publication of RIVPACS-type assessments in which methods other than biotic-based site classification coupled with LDMs are used to predict probabilities of capture. Some recent work has focused on predictive schemes that do not require a preclassification step. This work appears to have been stimulated primarily by the argument that the classification step used in RIVPACS might not minimize prediction errors. For example, Joy and Death (2003) and Olden (2003) explored the effectiveness of linking the outcomes of many taxon-specific models to estimate Ps. Others have used artificial neural networks (ANNs) to jointly predict multitaxa Ps directly from environmental data (e.g., Joy and Death 2005, Olden et al. 2006). Still others have developed approaches that identify that subset of reference sites that are best physically matched with a test site of interest—a type of site-specific typology (e.g., Linke et al. 2005, Prins and Smith 2007, Chessman et al. 2008). These modeling techniques generally exhibit desirable performance in precision, accuracy, or sensitivity (i.e., ability to identify a site as being in nonreference condition) but have yet to be widely incorporated into practice, perhaps because their results do not show uniform improvement over traditional RIVPACS models or because the improvements are marginal.

Back to the future: the regional reference site approach revisited.—Estimation of site-specific reference conditions through modeled interpolation circumvented one of the major concerns that Hughes (1995) had with site-specific biological criteria—that site-specific reference expectations derived from individual sites could not be adequately extrapolated to other sites. Even though proponents of the regional reference site approach clearly recognized that “Ecoregional reference sites still require some level of habitat classification at the site scale” (Hughes 1995, p. 35), there was concern that each test site would have to be matched to ≥3 appropriate reference sites (i.e., replication based on matching specific site characteristics, such as catchment size or channel type), and the cost of sampling 150 to 300 reference sites each year (assuming a management agency assessed 50–100 sites/y) would be exorbitant. On the other hand, Hughes et al. (1994, p. 138) earlier had acknowledged that “Use of regional reference lakes or stream reaches (Hughes et al. 1986) to develop water resources criteria is hindered by ecoregion heterogeneity.” Hughes et al. (1994, p. 138) further stated that “…sites with substantial natural differences…in the same ecoregion should not have the same set of reference sites.”, and “If (subregionalization is impossible), reference sites for each type of natural gradient, substrate, or water quality should be selected.”

The fact that comparisons of biological assessment methods have been confounded by mixing the approach to predicting reference conditions with the type of index used (e.g., Reynoldson et al. 1997, Hawkins et al. 2010) is perhaps the main reason that consensus did not emerge from the index wars of the mid 1990s and early 2000s regarding how best to conduct ecological assessments (e.g., Gerritsen 1995, Norris 1995, Karr and Chu 2000, Norris and Hawkins 2000). Recent studies have demonstrated that the modeling approach is effective when applied to the development of MMIs (e.g., Oberdorff et al. 2001 [Fig. 1], 2002 [Fig. 1], Baker et al. 2005, Pont et al. 2006, Cao et al. 2007), and other investigators have started to evaluate the independent effects of prediction method and type of ecological index on the performance of ecological assessments. For example, Mazor et al. (2006) showed that RIVPACS or BEAST (i.e., Reynoldson et al. 1995) modeling improved assessment sensitivity, and both Hawkins (Hawkins 2006, Hawkins et al. 2010) and Davy-Bowker et al. (2006) have shown that modeling can improve the performance of different types of indices. Aroviita et al. (2009) recently showed that the performance of O/E_taxa indices derived from traditional RIVPACS-type models and a priori typologies were similar when indices were based on reference sites that spanned a limited geographic range, but that modeled indices performed better than typologies when geographic range increased.

Does the taxonomic group matter?—An implicit assumption regarding the general utility of any type of a priori classification in predicting reference-state biota is that the classification will be applicable to all taxonomic groups. Differential response of biota in different taxonomic groups to natural environmental gradients would require development of separate taxon-specific classifications to assess the condition of >1 taxonomic group. The degree to which different taxonomic groups show concordant patterns of variation across natural environmental gradients should provide insight regarding this assumption. Several studies have now addressed this issue (e.g., Paavola et al. 2003, 2006, Grenouillet et al. 2008, Infante et al. 2009, Virtanen et al. 2009). In general, these studies show that concordance is usually weak, although sometimes statistically significant. Fish, invertebrate, and plant taxa tend to respond most strongly to different natural environmental gradients and to different human-caused alterations to those environments (e.g., Carlisle et al. 2008, Mykrä et al. 2008a). The lack of strong concordance among taxa in their responses to both natural and human-influenced environmental gradients seriously constrains the application of regionalizations and typologies as a general tool in ecological assessments.

The issue of variable reference-site quality.—One of the most vexing issues that continue to complicate use of reference site data for predicting ecological benchmarks at test sites is that no sites are truly pristine any longer, and the degree of ecological alteration present at the least-disturbed sites can vary markedly both between and within regions (Stoddard et al. 2006, Herlihy et al. 2008). The consequence of using reference sites of variable quality is that assessed sites probably will be held to different standards. Thus, direct comparisons among sites is problematic at best and potentially difficult to justify to the water-policy regulated community (e.g., industry, agriculture). The issue is also problematic from a resource conservation perspective because as conditions progressively degrade over time, society's expectations regarding what is natural or acceptable can change. This shifting baseline syndrome has attracted considerable attention with respect to the conservation and management of marine resources (Pauly 1995, Dayton et al. 1998, Greenstein et al. 1998, Pinnegar and Engelhard 2008) and has similar implications for establishing expectations for freshwater ecosystems. The recent National Wadeable Stream Assessment (WSA) in the USA provides a clear illustration of how problematic this issue is (Herlihy et al. 2008). Furthermore, human-caused climate change will exacerbate the shifting baseline problem because even the most pristine ecosystems that currently exist will be affected. Even if we have already sampled many pristine ecosystems, climate change will confound our ability to assess the natural variability of these systems and, thus, will compromise our ability to specify accurately the true ranges of natural variability when conducting ecological assessments.

Theoretically, the reference condition of water bodies can be estimated by various types of statistical hindcasting. Such techniques are fundamental to the reconstruction of lake environmental conditions from fossil biotic assemblages (e.g., Birks et al. 1990 [Fig. 1], Birks 1998, Simpson et al. 2005). The use of these techniques in lakes is aided by the presence of data-rich sediment cores that can be dated and that allow either quantitative or qualitative identification of the assemblage structure present prior to anthropogenic alterations. The structure of these prealteration assemblages provides a direct estimate of the biotic reference condition for individual lakes, and the use of transfer functions (reviewed by Birks 1998) that relate biotic structure to environmental conditions across modern lakes (i.e., surface sediments) can be used to infer the prealteration water quality (e.g., pH, P concentrations, water temperature) at individual lakes.

Lack of fossil records generally precludes routine use of paleolimnological methods in flowing-water ecosystems (but see Thoms et al. 1999, Gell et al. 2005), but attempts have been made to hindcast reference conditions by adjusting statistically for the relationships observed between biota and landuse alteration and related stressors. In the WSA, an attempt was made to hindcast reference conditions to more natural and, hence, comparable benchmarks by regressing MMI values observed at reference sites against the degree of alteration occurring across those sites. In this case, Herlihy et al. (2008) used a synthetic stressor gradient derived from a principal component analysis to quantify the joint variation in several stressor variables. This attempt to hindcast appeared to be only partially successful in generating benchmarks of equivalent quality because streams in the heavily altered central portions of the USA were estimated to be in better ecological condition than were streams in the less-altered Eastern Highlands.

We found 5 other papers (Dodds and Oakes 2004, Baker et al. 2005, Kilgour and Stanfield 2006, Robertson et al. 2006, and Angradi et al. 2009) that describe conceptually similar approaches to hindcasting but that used different analytical techniques to estimate reference conditions. Dodds and Oakes (2004) used analysis of covariance with ecoregions as categories and % anthropogenic land use as covariates to predict background (reference) nutrient concentrations. Baker et al. (2005) used stepwise multiple linear regression (MLR) to predict fish, water-quality, and stream-habitat metrics from a suite of natural and anthropogenic catchment variables and then adjusted metric values to 0 levels of catchment alteration. Kilgour and Stanfield (2006) also used stepwise MLR of biotic and stream habitat variables on landuse variables to estimate reference conditions at levels of 0 land use. Robertson et al. (2006) used spatial regression-tree analyses to delineate zones of similar reference concentrations of P and suspended sediment. Angradi et al. (2009) used quantile regression to hindcast fish MMI values to 0 stress levels. In all of these examples, the authors acknowledged several potential problems in applying such models. Perhaps the most problematic issue is that of extrapolating beyond the range of calibration data, a “damned if you do, damned if you don't” situation. By extrapolating beyond the range of the calibration data, we risk serious prediction errors; by not extrapolating, we have no estimate of what historical reference conditions were like.

Chessman and Royal (2004) explored a different approach to the variable-quality reference site dilemma. These authors built on the environmental filters concept (Poff 1997), which assumes that key environmental features screen potential colonists based on the ecological traits (tolerances) possessed by each taxon. If the original or desired environmental reference conditions at a test site can be estimated, it should be possible to predict the set of taxa that should occur at a site based on knowledge of their environmental tolerances. A comparison of the observed taxa with the predicted taxa provides an assessment measure. This approach is conceptually appealing because it is grounded in ecological first principles and, therefore, should be widely applicable and transferable. However, that it is a first principles approach also might be its chief limitation because its performance will depend critically on how well we understand the adaptive traits of individual species (e.g., Lytle and Poff 2004, Poff et al. 2006) and how well we can describe environmental filters in a way relevant to how multiple species perceive them. This approach has yet to be tested thoroughly, but it might be the only approach with the potential to overcome the issues associated with statistical extrapolation described above. Other research has shown that ecological indices based on functional ecological traits can discriminate between human-stressed and reference sites (Gayraud et al. 2003) and that trait structure might be reasonably stable across reference sites and, thus, potentially much easier to quantify and predict than taxonomic composition (e.g., Bady et al. 2005, Statzner et al. 2005). However, Statzner et al. (2005) acknowledged that ecologists have not developed a strong predictive theory regarding the ecological reasons for the spatial and temporal variation in trait structure that has been observed among sites (see Dolédec and Statzner 2010).

Predicting thermal reference conditions

Temperature strongly influences the distributions of ectotherms (reviews by Allan 1995, Begon et al. 1996) through its effect on metabolic and growth rates (Newell and Minshall 1978, Merritt et al. 1982), phenology (Sweeney and Vannote 1981), and fecundity (Vannote and Sweeney 1980). Therefore, accurate and precise predictions of the thermal conditions that characterize reference sites should lead to better ecological assessments.

Stream temperature researchers have approached temperature prediction mainly with 2 of the approaches used by stream ecologists to predict overall reference conditions: single-site and site-specific predictions (Table 2). We found no examples in the literature that used regionalization to characterize the thermal environment of streams. Site-specific predictions can be further divided into 2 modeling approaches and 2 spatial domains: 1) physical single-site, 2) physical multisite, 3) empirical single-site, and 4) empirical multisite. Physical single-site models typically are based on energy balance equations, whereas empirical single-site models most often relate water temperature to air temperature at a nearby temperature station. Physical multisite models typically are designed to predict temperature at many unmeasured locations within a single catchment and require measurement and parameterization of several local environmental factors. Empirical multisite models typically relate stream temperature to landscape features known to affect or covary with stream temperature, e.g., latitude, drainage area, or elevation.

Typologies.—The typological approach has not been used extensively to predict stream temperatures (Table 2). Snelder and Biggs (2002) proposed a hierarchical classification of New Zealand streams based on their climatic variability (air temperature and precipitation) and source of flow. Their classification has been tested for its ecological (Snelder et al. 2004a, b) and hydrologic (Snelder et al. 2005) predictive power, but has not been tested for how well it predicts stream temperature. Nelitz et al. (2007) used classification and regression-tree analysis to classify 104 streams in British Columbia, Canada, into 6 summer maximum weekly average temperature (MWAT) classes based on catchment size, catchment elevation, and air temperature. For each class, they used Bayesian regression to predict the probable response of MWAT to forest management practices, such as road building. Another typological approach to characterize stream temperatures is to classify streams based on the thermal preferences of the biological assemblage found within them (e.g., Lyons 1996, Wehrly et al. 2003), i.e., the fauna serve as surrogates for temperature. However, this latter approach has no utility in predicting reference-condition temperature for ecological assessment purposes because of its obvious circularity.

Site-specific models.—Site-specific physical stream temperature models emerged in the 1960s (Edinger et al. 1968, Brown 1969) largely in response to the need to assess the effects of forest and catchment management practices on aquatic ecosystems (Brown 1970). This research has produced a significant body of literature (Table 2) and some software packages (e.g., SNTEMP by Theurer et al. 1984, CEQUEAU by Morin et al. 1987). The physical-based model of Theurer et al. (1984) was designed to predict both spatial and temporal variation in stream temperature within an entire stream network. Physical-based models have contributed significantly to our understanding of the processes that affect stream temperature (reviewed by Caissie 2006, Webb et al. 2008) but have not been used much in ecological assessments because they require detailed measurements of numerous fine-scale environmental variables, such as local wind speed, relative humidity, and ground reflectivity. However, Allen et al. (2007) developed a model (BASINTEMP) based on a simplified characterization of the main physical factors that influence stream heat budgets (solar radiation input, vegetation cover, groundwater input) that produces site-specific predictions within a geographical information system (GIS) framework. BASINTEMP predictions are constrained to locations within a single stream network, but this model was an important step toward development of physically based models that could be applied to larger regions.

Initial development of single-site, empirical models also occurred in the 1960s. The first models were based on simple sinusoidal functions fit to annual seasonal variation in monthly mean stream temperatures (Ward 1963). Later, linear and nonlinear regression models were developed that related stream temperature to air temperature recorded at a nearby weather station to create temporally continuous predictions of water temperature (e.g., Cluis 1972, Mohseni et al. 1998). These latter models have been used to examine the potential effects of climate warming on specific streams (Mohseni et al. 2003).

Less emphasis has been placed on developing models that predict how temperatures vary across streams (Table 2), and many of these models were developed primarily in support of ecological analyses. Miyake and Takeuchi (1951) regressed monthly mean temperature from 20 rivers in Japan against air temperature, and their model appears to be the first example of an empirical model capable of predicting at multiple sites. Over the last 30 y, several authors have used linear regression to model various measures of stream temperature (i.e., total annual degree days, daily, monthly, or annual means) as functions of latitude (Vannote and Sweeney 1980), air temperature (Walker and Lawson 1977, Smith 1981, Stefan and Preudhomme 1993, Eaton and Scheller 1996, Pilgrim et al. 1998), or elevation (Rahel and Nibbelink 1999). Still others have used fish distributions to determine elevation isoclines of fish occurrence, which are assumed to be predictive of stream temperature (Meisner 1990, Rieman and McIntyre 1995). In addition to latitude, air temperature, and elevation, more recent research has incorporated local and catchment factors into models (e.g., catchment area, catchment slope, soil characteristics, landcover) (Jones et al. 2006, Wehrly et al. 2006).

Statistical approaches other than linear regression also have been evaluated. Gardner et al. (2003) used a geostatistical technique (kriging) to interpolate temperatures between measured sites within a catchment. Guillemette et al. (2009) used canonical correlation analysis to plot 20 stations in environmental space (catchment and site physical attributes). Once stations were projected into environmental space, the monthly maximum stream temperatures were interpolated between stations to predict temperature at unmeasured sites based on their physical attributes. Wehrly et al. (2009) compared 3 statistical techniques (MLR, generalized additive models, and linear mixed models) to predict July mean stream temperatures from catchment and site characteristics and found little difference between methods. The root mean square error reported for the 3 models was 2.0 to 2.6°C, which are comparable to errors reported in other studies (e.g., Smith 1981, Pilgrim et al. 1998). Ordinary kriging also was tested but did not perform as well as the other techniques (RMSE  =  2.95°C; Wehrly et al. 2009).

Future research needs: challenges and opportunities.—Because temperature is so critical to stream biota, it is imperative that we accurately describe the thermal conditions that characterize reference streams. Many of the empirical multisite models we reviewed were developed within a single catchment, a small region, or in terrain, such as the central USA, that is topographically homogeneous. Our own analyses relating stream temperatures to GIS-derived catchment and local predictors show that complex topography, like that found in many parts of the western USA, can contribute greatly to the heterogeneity of local environmental conditions and, therefore, to errors associated with temperature prediction (RAH and CPH, unpublished data). Physical models have greatly improved our understanding of how local environmental conditions control stream temperature, and thoughtful inclusion of those factors into multisite, spatially explicit models should improve their performance and allow large-scale spatial prediction of stream temperature as functions of both natural and anthropogenic influences.

Predicting hydrologic reference conditions

Hydrologic regime is another important aspect of the habitat template that influences biotic assemblages. Thus, improving characterization of hydrologic reference conditions expected at sites should improve ecological assessments. Stream flow can be described in many ways, but magnitude, timing, and variability appear to be particularly important to biota (e.g., Gustard 1984, Poff et al. 1997). Hydrologists were motivated initially by a need for landscape-scale predictions of flood risk, and among scientists in the 4 subdisciplines reviewed here, hydrologists have applied the widest variety of techniques to make predictions. However, empirical models and simple physically based rainfall–runoff models with empirically estimated parameters have become the prevailing approaches to making site-specific predictions of flow because of a lack of flow data and the multiscale complexity inherent in catchment dynamics. In addition, modeling has expanded from predicting one aspect of flow to predicting the entire hydrograph over time as the need for predictions has moved from assessing flood potential to predicting available water supplies, hydropower potential, and ecological flows.

Regionalizations.—Predictions of hydrologic regime began in the 1930s and 1940s with delineations of regions with homogeneous flow characteristics, topography, and climate (Krasovskaia 1997). Our use of the term, regionalization, as the development and identification of geographic regions that are homogeneous with respect to physical-chemical attributes differs from how the term is used in the hydrology literature (i.e., development of a regionally applicable model). For consistency throughout our paper, we use the former meaning of the term. Mosley (1981) derived hydrologic regionalizations via cluster analyses of hydrologic data. Others (Singh 1971, Cunnane 1988) have predicted flow for regions with reasonably homogeneous flow characteristics by applying regionally representative flows to all catchments in those regions. The regionalization approach to flow prediction does not appear to be widely used (Table 2), but the approach is still used for prediction in some cases (e.g., Chen et al. 2006).

Typologies.—Juncker (1971) developed a global hydrologic typology to predict hydrologic regimes, but since then relatively few studies have explored this approach (Table 2). Most typologies classify catchments by their physical characteristics via cluster analysis (Acreman and Sinclair 1986). The Ecological Limits of Hydrologic Alteration framework (ELOHA; Henriksen et al. 2006, Apse et al. 2008) uses a typology to predict flow regime. Snelder et al. (2005) developed a flow regime typology that explained significantly more variation in flow among sites than did a regionalization approach.

Site-specific predictions.—Hydrologists recognize that most landscapes are too heterogeneous for effective regional classification, so they have focused on development of site-specific prediction methods that can be applied across regions (Table 2). Interpolation or mathematical modeling is used to predict flow at ungauged sites from data collected at gauged sites. Flow regimes of gauged sites have been interpolated to ungauged sites in 2 main ways (reviewed by Holmes et al. 2002): 1) spatial interpolation from data collected at those gauges that are closest to the ungauged site (e.g., Skoien et al. 2006) or 2) interpolation based on flow records collected at streams that are most similar in physical characteristics to the ungauged site (interpolation in environmental space; e.g., Acreman and Wiltshire 1989, Burn 1990). We found few studies that based predictions on interpolation approaches. Empirical and physical models were used more commonly, and both approaches build on a proposal by Nash (1960) to estimate unit hydrograph parameters from catchment characteristics via regression. The choice between empirical and physical models typically has depended on the level of detail required (NERC 1975). Stratifying natural variation among sites by applying a typology often improves predictions of both empirical and physical models (Tasker 1982, Burn and Boorman 1993), especially in hydroclimatically diverse regions (Sanborn and Bledsoe 2006).

Thomas and Benson (1970) built on earlier empirical models developed for different regions and regressed 71 different flow indices nation-wide on 21 physical catchment characteristics. Others expanded the use of empirical models by introducing weighted and generalized least squares regression (Stedinger and Tasker 1985), nonlinear models (Pandey and Nguyen 1999), ANNs (Nayebi et al. 2006), and Random Forests (Carlisle et al. 2009). Empirical models avoid some important problems associated with physical models, such as model structure uncertainty and difficulties in identifying unique parameters, and are considered more robust than physical models (Croke and Norton 2004).

In earlier studies, hydrologists developed process-based physical models to predict flow at ungauged sites by parameterizing models for ungauged sites with parameters from models developed at gauged sites. This approach accounted for 49% of the hydrology publications we examined. Originally, physical models were calibrated with data from all gauged catchments in a region, and then model parameters were statistically related to catchment attributes, such as climate, topography, land use, geology, soils, or composites of these variables (Sefton and Howarth 1998). These attributes could then be measured at an ungauged site and the statistical relationships could be used to estimate the parameters for the model at the ungauged site (Benson and Matalas 1967, Ross 1970, Heerdegen and Reich 1974, Jarboe and Haan 1974, NERC 1975). Various physical models have been used; these models range from the simple lumped conceptual rainfall–runoff models IHACRES (Sefton et al. 1995) and SYMHID (Peel et al. 2000) to the more complex spatially distributed Stanford Watershed (Ross 1970), HBV (Seibert 1999), and SWAT models (Heuvelmans et al. 2006). Fernandez et al. (2000) refined this approach by simultaneously maximizing the fit of model parameters to catchment characteristics and gauge data. Physical models also can be calibrated to ungauged sites by spatially interpolating parameter values from gauged sites, but this approach has produced mixed results (Merz and Bloschl 2004, 2005, McIntyre et al. 2005, Oudin et al. 2008). Other refinements include constraining parameter space during model calibration (Gotzinger and Bardossy 2007), relating model parameters to catchment attributes using weighted or sequential regression (Kay et al. 2006), and making predictions from ensembles of models (McIntyre et al. 2005). Increasingly, site-specific predictions are made with either complex physically based models that exploit spatially extensive fine-scale data (Ivanov et al. 2004, Immerzeel and Droogers 2008) or new techniques for estimating model parameters, like ANNs (Heuvelmans et al. 2006).

Future research needs: challenges and opportunities.—Since 1960, hydrologists have developed multiple techniques that potentially could be used to predict the reference hydrologic regime for any site within a given region. Recent research has focused on how to quantify and reduce the uncertainty associated with these predictions (Sivapalan et al. 2003) and how to apply predictions to ecological assessments (Henriksen et al. 2006, Sanborn and Bledsoe 2006, Kennen et al. 2008). Sivapalan et al. (2003) established the International Association of Hydrological Sciences (IAHS) Decade on Predictions in Ungauged Basins to focus research efforts on measuring and reducing prediction uncertainties, and several researchers have begun to address this challenge (e.g., Wagener and Wheater 2006, Zhang et al. 2008). Sanborn and Bledsoe (2006) used an empirical model to predict 84 streamflow metrics (with estimates of uncertainty) that describe flow regime at 150 ungauged sites across 3 states in the US. They also explained how these predictions could be used to: 1) improve ecological assessments by predicting flow characteristics affecting taxa distributions, 2) assess causes of impairment by quantifying flow disturbances, or 3) design a restoration that maximizes recruitment of cottonwood trees. Kennen et al. (2008) estimated parameters for a physical model from spatial data (i.e., topography, soils, impervious surface cover, precipitation, and temperature) and then predicted reference flow conditions. They compared reference flow to current flow conditions and related these differences in flow to macroinvertebrate assemblage composition. Advances in hydrologic modeling have made predictions of the hydrologic regime possible, and these predictions have great potential to improve models of biotic–environment relationships. We see no reason why these hydrologic models should not be used more frequently in the future to provide better understanding of the reference condition of streams.

Predicting geomorphic and bed sediment reference conditions

Structural habitat is a key factor that determines the spatial distribution of aquatic taxa and biotic assemblages. Here, we define structural habitat as the channel geomorphic characteristics and sediment size distributions that compose the physical environment upon and within which stream biota live (Minshall 1984). Stream geomorphic structure and sediment size are important to biota, so ecological assessments should benefit from their improved prediction at unmeasured sites. Studies of stream structural habitat often consider aspects of channel planform (e.g., straight, meandering, or braided), stream channel width, channel depth, channel slope, and streambed sediment size distributions. The physical characteristics of streams change with stream size (Leopold et al. 1964) and landscape setting (Lotspeich 1980).

Regionalizations.—We found no examples of attempts to classify whole regions with respect to either the geomorphic or sediment characteristics expected in streams (Table 2). However, some researchers have used regionalization as a first step towards a hierarchical classification of channel types (e.g., Maxwell et al. 1995).

Typologies.—Geomorphologists recognized early that streams and rivers show systematic patterns in structure that are amenable to classification (Gilbert 1877, Davis 1890). Numerous classification systems have been proposed (Table 2) based on stream size (Horton 1945, Strahler 1957), channel planform (Leopold and Wolman 1957), or the dominant geomorphic processes that form streams (Schumm 1977). More recently, Lotspeich (1980) proposed a hierarchical framework for catchments, and Frissell et al. (1986) refined this classification by adding finer nested levels of geomorphic units (catchment > stream system > valley segment > reach > pool/riffle > microhabitat). Much research in the 1990s focused on describing the structure of these units and the processes that create them, e.g., valley segments (Nanson and Croke 1992), reaches (Rosgen 1994, Montgomery and Buffington 1997), and pools and riffles (Hawkins et al. 1993). To the best of our knowledge, no typologies have been proposed that focus exclusively on differences in substrate size among reaches.

Several classification systems have been suggested as management tools for aquatic resources (e.g., Maxwell et al. 1995). These classification systems were designed to inform land managers about how different stream classes should respond to management activities (Downs 1995), were often created for a specific region (Paustian et al. 1992), and can include stream condition as part of the classification scheme (Rosgen 1996). Kondolf et al. (2003) provide a comprehensive discussion of geomorphic classification systems and their purposes and uses.

Site-specific predictions.—Physical models that treat sediment size as an independent variable to predict channel slope, sediment movement, and yield (Dade and Friend 1998) can be rearranged to predict sediment size at unmeasured locations. However, few studies have attempted to make site-specific predictions of sediment size from physical relationships (Table 2). Here, we briefly review those efforts and describe several recent attempts to develop empirical models that predict sediment characteristics for use in aquatic assessments.

Montgomery et al. (1998) suggested that Shields' equation (Shields 1936), which describes the shear stress needed to move the median sediment size (D₅₀) in a stream reach at bankfull discharge, can be rearranged to predict D₅₀ from bankfull depth and channel slope. Buffington et al. (2004) used an approach similar to analysis of covariance to show that certain geomorphic river classes (Montgomery and Buffington 1997) have smaller sediment sizes than predicted from Shields' equation. This analysis was used to adjust sediment size predictions in 3 river networks in Washington, USA. Buffington et al. (2004) used drainage area to estimate bankfull depth and a 10-m-resolution digital elevation model (DEM) to derive channel slope. Kaufmann et al. (2009) rearranged the Shields' equation to estimate relative bed stability (RBS) in 101 streams in the Pacific Coastal Range, USA. These authors related RBS to a gradient of land use and found that reference-condition streams tended to be more stable at high flows. In addition, Kaufmann et al. (2009) used MLR to predict RBS values from field and GIS-derived variables (R²  =  0.62). The selected predictor variables included basin lithology, a surrogate for unit stream power (A_ws^0.5S, where A_ws is basin area and S is the channel slope), and management-related factors (road density, a riparian condition index), results implying that reference-site sediment character depends largely on discharge (function of watershed area) and basin geology.

Several authors recently attempted to predict sediment size with empirical models that used GIS-derived predictors. Flores (2004) regressed sediment D₅₀ estimates from stream sites in Oregon and Colorado, USA, against DEM channel slope, drainage area, elevation, 2 catchment lithology variables, 2 valley confinement variables, and mean precipitation. His model explained 84% of the variation in D₅₀, but probably suffered from some overfitting, given the sample size (n  =  39) and number of predictors (8). Two other studies (Mugodo et al. 2006, Frappier and Eckert 2007) used a suite of GIS variables to predict various local habitat features including sediment characteristics (e.g., % sand, % cobble, % gravel). Both models suffered from large prediction errors. However, the modeled estimates of sediment size in Mugodo et al. (2006) predicted the distributions of 7 fish species as effectively as field measurements of sediment size, results implying that their modeled sediment sizes were predicted well enough to be ecologically useful.

Future research needs: challenges and opportunities.—Use of geomorphic typologies can aid in conducting inventories of aquatic resources (Brenden et al. 2007) and predicting the biotic assemblages expected at unvisited locations (Naiman 1998). Advances in GIS have allowed deployment of various typology schemes that account for a suite of habitat characteristics, including geomorphology, and that have promise for use in bioassessments (e.g., Snelder et al. 2004b).

Future progress in sediment size prediction will require expansion of the scale of investigations (i.e., larger sample sizes and geographic extents) and incorporation of local processes known to affect sediment size. However, incorporation of local processes at the spatial scale required for many ecological assessments might be challenging, especially for small streams where stochastic events, such as a single tree fall within the channel, can have a disproportionately large effect on the geomorphic structure when compared with downstream reaches (e.g., Swanson et al. 1976, Keller and Swanson 1979, Gooderham et al. 2007). Numerous studies have examined the factors controlling downstream variation in sediment size in individual rivers (e.g., Brierley and Hickin 1985, Hoey and Bluck 1999, Rice 1999). Recently, these papers have emphasized the importance of local natural controls that disrupt the idealized trends in the downstream reduction in sediment size (Ferguson et al. 2006). Local controls can include lateral sediment contributions from hill slopes (Benda and Dunne 1997, Bigelow et al. 2007) and tributaries (Rice 1998, reviewed by Benda et al. 2004), adjustments in flow and channel competence resulting from large woody debris inputs (Massong and Montgomery 2000, Gomi et al. 2006, Kaufmann et al. 2009; see Van Sickle and Gregory 1990, Sobota et al. 2006, Wohl and Jaeger 2009 for models of large wood recruitment to streams), fine sediment retention by small mountain lakes (Arp et al. 2007), and local geology (Mureşan 2009). From a practical application context, it is important to note that each of the natural processes listed above potentially could be mapped in a GIS. In addition, these processes have received much attention recently, and significant progress has been made toward understanding why, where, and how much each factor affects local stream sediment size. Additional progress toward predicting reference sediment characteristics will require explicit incorporation of these natural processes in modeling efforts.