There is limited documentation of cross-tenure collaborative weed management programs, and no consistent set of metrics for evaluating their performance. In this study, 12 weed management practitioners in southeast Australia participated in a qualitative social research project to discuss and document examples of cross-tenure collaborative weed management and critically reflect on whether existing metrics are suitable for evaluating the performance of their programs. Analysis of focus group discussions, project documentation, subsequent reflections, and review of the literature reveal that weed management practitioners, in Australia and elsewhere, mostly rely on metrics that measure weed management inputs, such as herbicides, labor, and costs. Metrics used to evaluate social outcomes focus on benefits for individuals rather than social relationships or achievement of equitable outcomes. Social research on collaborative governance and social science methods more broadly, such as social network analysis and collective narratives, could be used by weed management practitioners to better evaluate and explain social–ecological outcomes over time.
Introduction
For more than two decades, researchers and policy makers have argued that cross-tenure collaboration is needed to improve weed management outcomes. For example, Thorp and Lynch (1999) argued that weed management requires cross-tenure collaboration, because weeds easily spread across property and jurisdictional boundaries. They criticized the first Australian Weeds Strategy for failing to outline an approach for fostering effective working relationships wherein all interested parties, including private land managers and government agencies, are committed to achieving desired weed management outcomes. Yet the challenge of cross-tenure collaboration for weed management is not unique to Australia. Only a few years later, the U.S. Government Accountability Office (GAO 2005) undertook a review of nonagricultural, terrestrial weed management at local, state, and federal scales and found that effective weed management is hindered by a lack of collaboration among agencies. They recommended that all levels of government cooperate to control and eradicate weeds. However, the report provided little practical guidance on how to establish such partnerships and limited evidence of the benefits of such collaborations over time.
Recent academic literature has begun to gather empirical evidence on the different types of collaboration that underpin cross-tenure weed management programs and the benefits that accrue from such activities. For example, Graham (2019) identified a continuum of collaboration. At its most basic, collaboration involves adjoining land managers largely controlling weeds independently and coming together occasionally to share information on emerging weeds and their management. More involved forms of collaboration involved land managers and government agencies working together to develop shared weed management goals and then coordinating the timing and type of their weed control efforts (Graham 2019). This more involved form of collaboration, involving multiple private land managers and public land managers, is the focus of this article. Such cross-tenure collaboration is also referred to as tenure-blind (e.g., Turner et al. 2013) or nil-tenure (e.g., Low 2020) weed management.
Management Implications
Many metrics used to evaluate weed management programs focus on inputs, such as hours spent controlling weeds or amount of herbicide applied, rather than broader ecological, economic, or social outcomes. To ensure that collaborative cross-tenure programs deliver on the investment from multiple parties, continue to sustain interest over the long term, and provide better outcomes than would be achieved through single-agency programs, it is essential to measure the following before, during, and after the project.
At least one of the following ecological metrics should be measured: prevent-spread metrics that evaluate the spread of weeds; weed-led metrics that evaluate the density and extent of established weed populations; or asset-led metrics that evaluate the impact of weeds on environmental or agricultural assets. Such metrics should not be used in isolation. They need to be compared with the amount of effort invested to achieve those outcomes, for example, time and finances, and the extent to which collaboration was beneficial. Ecological metrics also need to account for the characteristics of the focus weed, landscape, and community.
Collaborative programs also require metrics that evaluate social networks, processes, and outcomes. Social network analysis enables evaluation of whether a project has cemented existing relationships and created new relationships with stakeholders who do not normally participate in traditional weed management programs. Participant evaluations can provide insights into social processes, for example, whether participants feel that there is transparency and whether they are being treated respectfully. Development of collective narratives can help evaluate the social outcomes, for example, whether the program has built knowledge and capacity to manage weeds over time, and whether there have been other benefits from collaborations, such as enhanced management of other natural resource management issues.
Research that documents the benefits of cross-tenure collaborative weed management is in its infancy. There is some evidence to suggest that cross-tenure collaboration is more likely to result in enhanced information sharing, reductions in weed densities and spread, lower weed management costs, and improvements in social capital (Graham 2013; Graham and Rogers 2017; Hershdorfer et al. 2007; Lien et al. 2021; Niemiec et al. 2016). Yet most studies focus on one or two case studies, and the indicators used to measure success are far from systematic or sustained. This means there is a lack of solid evidence about how collaborative weed management is established, limited operational guidance on how to manage weeds across tenures, and whether such approaches provide significant benefits compared with independent efforts. Given the considerable time and resources required to create and implement collaborative weed management programs, it is imperative that weed management practitioners and researchers have clear metrics to document and evaluate the relative success of such programs in terms of their ecological, economic, and social outcomes.
The aim of this article is to apply qualitative social science research and methods to weed management practice in the case study of southeast Australia to examine: What constitutes successful cross-tenure collaborative weed management, and what metrics are useful for evaluating the performance of cross-tenure collaborative weed management programs? It draws on 12 case studies of cross-tenure weed management to answer these questions.
Social science on collaborative governance from other fields of research and practice suggests that evaluation processes need to reflect the antecedent conditions, processes, and outcomes of cross-tenure collaborations (Ansell and Gash 2008; Emerson and Nabatchi 2015). Past meta-analyses have identified a clear set of process criteria for evaluating cross-tenure collaborative programs. These include having broadly shared visions, clear feasible goals (Conley and Moote 2003), good faith negotiation, trust building, mutual recognition of interdependence (Ansell and Gash 2008), and commitment (Emerson and Nabatchi 2015), among others. Such indicators are useful for evaluating participant satisfaction with collaborative processes (McKinney and Field 2008) and are largely applicable regardless of the specific public policy issue being addressed. There is less agreement about evaluation indicators for collaborative policy or management outcomes (Ansell and Gash 2008). This is in part because environmental and socioeconomic outcome criteria are more dependent on the specific issue, the place where the collaborative programs are being implemented, and the outcomes being sought by the community and key stakeholders (Barney et al. 2019; Cradock-Henry et al. 2017; Hill et al. 2015). Consideration also needs to be given to the practicalities involved in collecting data for each evaluation criterion over the short- and long-term (McKinney and Field, 2008). Practitioners are well-placed to evaluate the utility of performance metrics. The need for context-specific and practical outcome indicators explains the focus of this article on identifying and describing outcome metrics for collaborative weed management programs.
Evaluating the outcomes of cross-tenure collaboration requires considering whether a collaborative effort provides “a public purpose that could not otherwise be accomplished” (Emerson et al. 2012, 2), “the greatest benefits to the widest number of stakeholders” (Cradock-Henry et al. 2017, 2), or a “more effective, lasting outcome” (McKinney and Field 2008, 424). Attention also needs to be paid to the scale; composition, such as citizen-based, agency-based, or mixed partnership (Moore and Koontz 2003); and type of collaboration, such as self-initiating, independently convened or externally directed (Emerson and Nabatchi 2015); these three factors affect collaboration dynamics and outcomes.
In the weed management context, few studies compare the outcomes of collaborative programs. Hershdorfer et al. (2007) is a partial exception, because they evaluated whether four types of collaborative weed management programs—county, district, weed management area, and grassroots volunteer groups—provided outcomes that could not otherwise be achieved. They evaluated the outcomes of 11 single-agency and 31 multiple-agency weed management programs across four states in the United States. They found that weed programs involving multiple agencies and organizations, such as Cooperative Weed Management Areas, conducted significantly more monitoring but treated a significantly smaller area and proportion of weed infestations than programs run by a single government agency. There were no significant differences in the frequency of education and outreach activities between multiple- and single-agency programs. While Hershdorfer et al. (2007) considered the effectiveness of collaborative programs, they did not consider whether programs had equitable or lasting outcomes.
Other research has found that collaborative weed management programs involving land managers, government, research organizations, industry, and non-government organizations have resulted in reductions in weed population densities and spread in the United States (Ayer 1997), reduced use of herbicides, lower weed management costs across North America and globally (Ervin and Frisvold 2016), increased leveraging of limited resources (GAO 2005), improved access to important cultural sites in Australia (Bach et al. 2019), and enhanced social capital (McKiernan 2018). However, little attention has been given to the nature of collaboration in such programs despite the importance of context on the evolution and performance of cross-tenure collaboration (Emerson and Nabatchi 2015). There is a need for critical reflection of the metrics and notions of success that are used to evaluate the ecological, economic, and social outcomes of cross-tenure collaborative weed management programs (Anderson et al. 2003), and whether some metrics are more suitable for evaluating the performance of specific compositions and types of collaboration at diverse scales. The aim of this article is to provide such critical reflection.
Methods
This study forms part of an ongoing research project on how communities and public agencies collaboratively manage weeds, the benefits of such efforts, and how governments can better support cross-tenure collaborative initiatives. In this stage of the project, we engaged 12 weed management practitioners to discuss and co-analyze what cross-tenure collaboration and success mean in the context of weed management and develop practical outcome-focused metrics. The research began with two in-depth focus groups held in November 2020. An online format was used due to COVID-19 restrictions on face-to-face research. In the intervening period, focus group participants have been involved in the interpretation of the results and have provided additional resources on their collaborative weed management programs, such as project reports, to expand and support the analysis.
Thirty-one knowledgeable and experienced weed management practitioners were invited to attend the focus groups. Twelve (five women and seven men) were available and willing to participate, six joined each focus group. On average, the participants had more than 13 years of experience in weed management at the time of the focus groups, with length of experience ranging from 4 to 24 years. Focus groups are most effective when there are between six and eight participants, because this allows for everyone to contribute and for a diversity of opinions to be heard (Krueger and Casey 2001). In each focus group, there was at least one Landcare coordinator or chairperson (hereafter referred to as a “Landcare representative”), one local government weed officer, one practitioner working for a regional natural resource management (NRM)-focused state government organization, and one person who works for a state government agency. Thus, there were participants who lead weed management programs at a range of scales. Efforts were made to recruit Indigenous Australians involved in weed management in the region; however, none responded to the written and follow-up invitations to participate in the research. Further efforts have been successful in engaging Indigenous weed management practitioners in the ongoing project.
Ahead of the focus groups, participants were sent short but comprehensive summaries of weed management outcome metrics used by researchers and the goals of weed management included in policies across the case study region of southeast New South Wales (NSW), Australia. Participants were asked to identify and describe an example (local, regional, national, international) of successful cross-tenure collaborative weed management in which they had been actively involved. Follow-up questions during the focus groups invited participants to reflect on what evidence they used to evaluate the success of their examples.
The main questions guiding the discussions about the success of collaborations were:
What metrics do practitioners currently use to evaluate the outcomes of cross-tenure collaborative weed management programs?
What are the challenges to collecting data on these metrics?
What metrics would practitioners like to see included in future evaluations of cross-tenure collaborative weed management programs?
The small sample size and collaborative research process allowed for a focused exploration of practitioner knowledge in the field of weed management in the chosen location. Sketch et al. (2020) used a similar sample size of two focus groups over 2 days in their landowner-listening workshops in the United States, where they concluded that the workshops effectively bridge the gap between researchers and landowners. Similarly, our focus groups and subsequent communications between researchers and practitioners show how social research methods bridge gaps between basic or fundamental research (McLennan and Garvin 2012) and applied knowledge. Bringing together 12 weed management practitioners enabled us to describe and compare multiple examples of collaborative weed management, rather than focus on a single case study, which has been a critique of past collaborative governance research (Ansell and Gash 2008).
Approval to conduct the research was obtained from the University of Wollongong Human Research Ethics committee. Each participant provided consent to participate in the focus groups and then chose whether to participate in the ongoing research collaboration. Some analysis began during the focus groups, with the lead researcher synthesizing and feeding back the participants' key messages in real time, to ensure that everyone's views were recorded and that each participant had the opportunity to see and reflect on how his or her view was being documented. The focus groups were also digitally recorded, transcribed, and imported into the qualitative data analysis software NVivo for thematic coding (Braun and Clarke 2006) by the lead researcher. Once this analysis had been undertaken and shared, each participant was invited to contribute to the interpretation of the analysis and writing of the article. Thus, the results reflect the focus group discussions as well as multiple iterations of analysis since then.
In the “Results,” quotes from the focus groups have been anonymized. Each participant is identified by the type of organization he or she works for—Landcare, local council, or state government—with numbers used to differentiate between participants.
Case Study: Southeast New South Wales, Australia
The geographic focus of this study is the southeast region of NSW for two main reasons. First, the region is emblematic of complex rural landscapes in Australia and internationally. The northwestern part of the region is dominated by cropping enterprises, which transition into grazing enterprises along the southern tablelands. There are national parks, commercial hardwood native forests, and introduced softwood plantations throughout the region. Higher-rainfall coastal areas are dominated by dairy enterprises, and there is amenity migration to many parts of the region (McKiernan 2018). Second, a 2017 survey of weed management groups in rural NSW revealed many and diverse types of collaboration across the southeast (Graham et al. 2020). Thus, the region provides a rich opportunity to study diverse types of cross-tenure collaborative weed management, with findings of international relevance.
Results and Discussion
Focus group discussions revealed how weed management practitioners define and measure outcomes in two ways. The first was through accounts and explanations of cross-tenure collaborative weed management. The second was evident in descriptions of metrics that practitioners use to evaluate outcomes of weed management programs, and how they would like to see those metrics improve in future. The results are organized by these two discussions.
What Is Successful Cross-Tenure Collaborative Weed Management?
Each participant described a unique example of a successful cross-tenure collaborative weed management program (Table 1). Eight of the examples represent self-initiated collaborations, in which collaborators had preexisting shared interests and were inspired by a core set of stakeholders (as described by Emerson and Nabatchi 2015). Four examples are externally directed collaborations, in which participants are motivated to participate because of the authority or resources of an outside entity (as per Emerson and Nabatchi 2015). With respect to composition, three examples are citizen based, four are mixed partnerships, and five are agency based (as per Moore and Koontz 2003). With respect to scale, the projects were evenly split across locality, local government, regional, and state scales, with three projects operating at each scale. Table 1 captures variation across the examples of cross-tenure collaboration with respect to the type, composition, and scale of collaboration. The following descriptions of two local collaborations, one self-initiated agency-based (B) and one externally directed mixed partnership (D), focus on what was successful about each program.
Blackberry Forum Project (B)
The Blackberry Forum Project began when a local council hosted a forum to bring together multiple organizations, including Snowy Valleys Council (then the Tumut and Tumbarumba Shire Councils), Local Land Services, NSW National Parks and Wildlife Service, State Forests, Crown Lands, and adjacent local governments to discuss cross-tenure blackberry (Rubus fruticosus L. agg) control. An outcome of the forum was the signing of a memorandum of understanding by the participating organizations. The organizations then developed joint plans of management, pooled their financial resources, and jointly applied for grants to fund integrated weed control and other restoration and rehabilitation activities over a number of years. The result of such collaborative activities were five projects that enabled restoration, rehabilitation, and recovery of a number of former degraded natural landscapes and an endangered ecological plant community.
One project was the Wereboldera State Conservation Area (SCA) project (Wilkerson 2017). The 273.6-ha Wereboldera SCA hosts a unique collection of flora and fauna. The Wereboldera SCA project successfully restored the bushland and removed the weed threat. Project partners chemically treated or manually removed 150 ha of heavy blackberry infestations and 57 m3 of bridal creeper [Asparagus asparagoides (L.) Druce], the two target weed species. The project also removed 11 ha of boxthorn (Lycium ferocissimum Miers) infestations and treated two large infestations of Chilean needlegrass [Nassella neesiana (Trin. & Rupr.) Backworth]. Beyond weed control, the project facilitated the collection and disposal of more than 1,000 m3 of illegally dumped waste. The Blackberry Forum Project work in reducing blackberry biomass was a key contributing factor to successfully combatting bushfires when they spread into the SCA in 2019.
The Wereboldera SCA and surrounding areas have seen a return of native wildlife, creation of new recreational development opportunities, and reduced weed control costs for each partner organization. This whole-of-region management approach has provided benefits beyond weed management by creating a template for working with other industry service providers, such as the Rural Fire Service, Transgrid, Snowy Hydro, and Roads and Maritime Services, on other aspects of shared asset management.
Coordinated Aerial-Spraying Program (D)
In spring and autumn each year, the Snowy-Monaro Regional Council leads a works program with landholders and weed spray contractors to coordinate the broadscale aerial herbicide application for grassy and woody weeds on private and public lands. Council's biosecurity staff assist by identifying areas with widespread priority weeds; helping landholders to select suitable control methods and herbicides; contacting all adjoining landholders in the infested area to introduce the aerial spray opportunity; identifying and mapping the extent of weed infestations on each property; nominating suitable helicopter landing sites; carting water and transporting chemicals; and working with the pilot, loader, and landholders on the day to carry out the spray works. As the aerial application work is highly weather dependent, there are often delays between spray jobs, requiring ongoing communication between landholders and contractors.
The program is a cross-tenure collaboration success because it increases efficiency and effectiveness of weed control. The cost of aerial spraying is prohibitive for individual landholders. By identifying willing participants for large aerial-spraying programs, the council enables broader control of more weed infestations for more landholders. The program provides a sense of relief to participants by enabling them to tackle inaccessible areas of their properties. The council considers the program to be successful because it extends community participation and delivers more cost-effective and efficient weed control over the long term.
Evidence of Successful Collaborative Weed Management
Many themes evident in the two programs described were common in the remaining 10 case studies. Table 1 captures 7 overarching themes (‘Category’) and 21 subthemes (‘How the example demonstrates success’) evident in the descriptions.
Project duration. Project duration was often one of the first pieces of evidence used by practitioners to justify their selected example of cross-tenure collaborative weed management. For seven of the projects, 10 years was used as a measure of success (Table 1). An additional three projects were described as “long-term” because they spanned multiple years, but had not reached the 10-year time frame. The following quote reflects the importance of the decade-long metric of success.
We've been working through this program for over 10 years now . . . There's a short-term success, which is the plants will die in the next six months to a year. Beyond that, the long-term success is whether the landholder follows up . . . to rehabilitate that country. (Council Staff 1)
Project duration is particularly important for collaborative weed management programs because of the time required to establish and maintain new relationships, build capacity, and change entrenched practices.
Table 1.
Overview of 12 examples provided of successful cross-tenure weed management and the qualities (as indicated by X's) attributed to each project by the person who described the example.
Landscape scale. A project's functional area was key to identifying success. Nine projects were described as successful because they spanned a landscape scale—a river corridor, catchment, or larger scale, such as across a whole state. As one practitioner explained, achieving weed management across a larger scale is evidence of success, because weed managers often work at “cross-purposes. A weed that was being controlled in one area was being let go in another area” (State Government Staff 1). In this case, success meant getting agreement about priorities and coordinating work at a landscape scale.
We got agreement from the various managers that we would deal with the top end of the catchment and then work our way down the catchment progressively . . . With weeds it's really, really easy to get focused on your immediate problem in your local area. And to actually try and get that big picture accepted, and then people buying into it, is a very powerful thing. (State Government Staff 1)
For collaborative projects, the scale of the project needs to be well defined. If the project aims to cover too large an area, or the boundaries are too vague, then resources can be spread too thin, participants can lose motivation, and it can be difficult to achieve on-ground outcomes.
Ecological benefits. A measure of success used in nine programs involved evaluations of the extent to which weeds were treated and removed. These reductions were described with respect to the area from which weeds were removed, reduced weed densities, and fewer mature or flowering weeds. Some practitioners emphasized the value of removing weeds that have high potential for ecological impacts or are a priority for some land managers but not for others.
We had a 30% reduction in the number of reproductive plants found over the course of time, so we were obviously getting rid of the mature plants. One thing that was interesting, however, was that the number of non-reproductive plants actually increased over that time. That's not such a bad thing, in that it meant that there was [sic] a lot of seedlings germinating and we were actually getting in and getting those plants before they set seed themselves. (Council Staff 2)
Although not mentioned as often, half of the practitioners used rehabilitation of ecosystems as a measure of success: “Over this 10-plus year project, we were able to replant, revegetate, 18% of the New South Wales [Snowy] river and this is a 150 kilometre stretch” (State Government Staff 4). In collaborative cross-tenure programs, the ecological benefits may not accrue to all stakeholders equally; however, consistently measuring the ecological benefits can help stakeholders see the broader impacts of their efforts.
Economic benefits. Limited, short-term funding was considered to be a key limitation of weed management programs generally. Thus, successful projects were valued for their ability to secure long-term funding or leverage funding from multiple sources to ensure results were maintained over the longer term.
I was able to help make that levy a permanent levy, which is another really great thing to mean that there is permanent money available for the delivery of the program over a long period of time. Which is where I think a lot of the weed strategy falls down, is that there's a lot of really great weed removal happening, but without that long-term maintenance and long-term continual assessment of what is being done, [it can] very, very easily be undone. (Landcare Representative 1) Influencing government to put resources back into the community I think is pretty powerful. I don't know how you measure that, but maybe grants. We talked about having ongoing grants being a sign of a good program. (State Government Staff 3)
The sharing of financial resources and achieving economic security enables collaborative programs to evolve with the weed challenge and sustain stakeholder commitment.
Social benefits. The importance of social engagement for successful collaborative weed management initiatives was recognized in 11 projects. This involved reaching large numbers of diverse actors, including landholders, scientists, politicians, community groups, NGOs, and government agencies. Of particular importance was involving people “that we wouldn't normally reach” (State Government Staff 5).
One of the things we're proudest of is 74 of the private landholders that adjoin that river were able to participate and get the outcomes and the wins from it. (State Government Staff 4)
Beyond the number and type of people involved, practitioners focused on whether programs became self-sustaining and self-perpetuating, with community members directly or indirectly encouraging others to become involved over the long term.
We've been able to get a lot of recalcitrant people involved . . . If you were the last landholder left in that stretch, by diffusion you sort of ended up doing the work. I think pretty much every property has been worked on now, to the point where people further upstream in Queanbeyan-Palerang Council, they're doing work up there now. Which is great. So that diffusion is happening across that catchment more broadly as well and out of our catchment. (Council Staff 3)
Social relations are at the core of collaborative programs. The very establishment of new relationships not only means that weeds are more likely to be controlled, but also has flow-on benefits to other land management challenges, and maintenance of rural communities more broadly.
How is Successful Collaborative Weed Management Measured and Reported?
There were four categories of metrics that practitioners use to monitor the success of cross-tenure collaborative weed management programs over time: ecological, economic, social, and scientific (Table 2). While there was consistency in the overall outcomes that practitioners measure, there was little overlap in the specific metrics used. For example, reduction in weeds over time was measured by the presence of a weed, the number of plants, or the density of plants in an area.
Ecological Metrics
One practitioner described three types of ecological metrics. Prevent-spread metrics focus on preventing the spread of new and emerging weeds; these metrics center on surveillance to prevent a weed from becoming established. Weed-led metrics focus on established weeds and measuring whether they are being eradicated or contained. Asset-led metrics focus on reducing weeds' impacts on biodiversity or other assets, such as agricultural assets. Although only one participant used these descriptors, they effectively capture the three subcategories of ecological metrics that emerged during the analysis: surveillance, area treated effectively over time, and area regenerated (Table 2).
When describing specific ecological metrics, practitioners emphasized the importance of measuring the same ecological metrics over time, such as at yearly intervals. They also recognized there is limited benefit in measuring ecological outcomes without evaluating the amount of effort invested to achieve those outcomes.
We record all our inspections and the weed density and abundance, and location, etc. So we're able to compare year on year whether there's been a reduction in that weed on that site over time. For some weeds, that is occurring. For other weeds, it isn't. Despite the effort we're putting in. (Council Staff 2)
Table 2.
Metrics currently used by weed management practitioners to evaluate the success of weed management programs.
Economic Metrics
Money, labor, and herbicides were the three main resources measured to evaluate the economic performance of weed management programs. Practitioners emphasized that such economic metrics should not be interpreted in isolation from ecological outcomes. One practitioner highlighted the importance of considering the trade-offs of investing resources in some activities over others.
The other metric . . . I think is hugely important, is focused effort, which is to say whether it's the number of staff or the hours spent on the high priority programs . . . one of the really big battles in weed management is freeing your staff and resources up from the routine things, to actually doing the high priority things, such as the orange hawkweed, that you know will make a big strategic difference. (State Government Staff 1)
Social Metrics
Three types of social metrics were described: governance, community engagement, and psychological (Table 2). Here we adopt the definition of governance used by Lynn et al. (2001, 7) which includes “regimes of laws, rules, judicial decisions and administrative practices that constrain, prescribe and enable the provision of publicly supported goods and services.” Practitioners who discussed governance metrics emphasized that they were interested in measuring the number of people who proactively comply with legislation that requires them to undertake weed control work on their properties, rather than those who did so reactively in response to their interventions.
I try and measure success, to a certain degree, [by] whether—particularly when we do inspections and so forth—people have been proactive in their response to weed management . . . They've got a program in place, that they're aware of the weed that you're there talking to them about and want some control work done, and they've already gone out and implemented that. (Council Staff 2)
Those who discussed the community engagement metrics were not only interested in the number of people participating, but also the length of engagement and whether or not neighboring land managers were jointly participating in programs.
Two practitioners mentioned psychological metrics. For example, the land looks better from an aesthetic point of view or a weed management program provides a “relief” to land managers, because they have “figure[d] out how to manage the weeds on their property.” Neither psychological metric is formally measured; practitioners used these metrics informally to evaluate their programs.
Scientific Metrics
The nature of baseline data collected was integral to evaluating the success of weed management programs. This is because such data provide a reference point for evaluating the success of program activities. There are also distinct qualities that baseline data must have, such as accessibility and shareability, to make the program useful over the longer term and provide benefits to other weed management programs.
The Need for Meaningful Metrics
Practitioners felt that metrics needed to be contextual, cumulative and connected, and consistent in order to be considered “meaningful.”
Contextual metrics. Many practitioners were concerned that the metrics they use to meet institutional or funding requirements are premised on the assumption that bigger numbers are better and do not adequately take context into account. For example, some practitioners are required to record the number of properties inspected or number of kilometers of roadside sprayed but are not required to record the size of the properties, whether the roadside spraying was effective, or whether the work contributes to the goals outlined in local and regional weed management plans. Some practitioners were also concerned that they were asked to use the same metrics for different weeds, even though each weed requires tailored management strategies and has unique ecological responses.
People are asking for metrics that look good on paper and reports as opposed to what is actually there . . . We're asked to report how many weeds, or how many hectares have been effectively treated. There's no way to measure that because every weed has a different type of outcome base. The metrics are going to be different . . . And they're trying to fit it all into one template, which doesn't work. (Council Staff 4) Some of the other metrics we have to use . . . [are] community participation, and number of hours of people contributing to programs . . . Which [are] once again pretty meaningless. It's sort of a necessary evil for some of the funding grants and things like that. We have to meet certain hours. (State Government Staff 2)
Case studies were identified as one option for overcoming the problem of context-free metrics. In the second focus group, case studies were valued for their ability to incorporate local knowledge about how ecosystems respond to weed management, to record changes over time, and as a mechanism for community engagement and learning about weeds.
I think case studies are fantastic for community engagement on this and I guess if people have a good story to tell and it goes in, for instance, the Landcare newsletters . . . those stories about someone who has been doing some weed control, and the recovery of their property, and what they're seeing now in terms of interesting forbs and little wildflowers, I think they carry a lot of weight. People might say, “Oh, I hadn't really thought about that.” So those case studies I think can be important. (Landcare Representative 3)
Cumulative and connected metrics. Practitioners discussed the need for metrics to provide a holistic understanding of the impacts of weed management programs on the ecological functioning of the systems in which they operate. Metrics about the costs of weed management, the amount of herbicide applied, or the area sprayed are only useful if they are connected to measures that reflect ecological responses to the removal of weeds.
Practitioners also discussed the need for metrics that evaluate multiple efforts undertaken at the landscape scale. This means going beyond the project scale to connecting multiple weed and NRM programs: “Everybody has strategies and plans. Is there some way of measuring those collectively and the output? . . . It would be neat if we could do that.” (State Government Staff 3)
Consistency. Practitioners in the first focus group expressed a desire for greater consistency and standardization of the multiple internal and external databases that exist to record data about weed management and conservation programs. Many of the data collected that are relevant to their local context cannot be recorded in the statewide biosecurity database, resulting in a loss of data and missed opportunities for learning across the region and state. In addition, metrics often change over time, which means that it is difficult to establish long-term data sets and undertake longitudinal analyses.
Consistency I think is the key. We can't possibly compare management of areas when there's not enough consistency between the collection of baseline data, and the analysing of baseline data, and the reporting on what the effectiveness of the management is. (Landcare Representative 1)
In discussing the need for consistency, practitioners recognized that there are financial costs and expertise required to standardize metrics and data collection across organizations. Such costs need to be accounted for in future programs.
What Would Meaningful Metrics Look Like?
When practitioners described cross-tenure collaborative weed management programs in their own words, free from the restrictions of the metrics that they are required to document and report, there was agreement that success involves long-term projects, whose scale matches the goals of the program, and that deliver ecological, economic, and social benefits. Similarly, when practitioners discussed metrics they would like to use in future, they described metrics that would enable them to capture the benefits that weed management provides to social–ecological systems over the long term.
The focus group discussions reveal a mismatch between existing metrics and how practitioners define and would like to measure success. Current metrics mostly focus on weed management inputs, such as herbicides, labor, and costs. Few metrics help evaluate the psychological and social benefits (or impacts; Crowley et al. 2017) provided by weed management programs or capture the link between antecedent conditions, process performance, and outcomes (Emerson and Nabatchi 2015). The discussion that follows explores the extent to which social research on cross-tenure collaborations can inform the measurement of nested scales of social success and social–ecological outcomes in weed management and deliver rich narratives that explain the role of context on collaboration outcomes.
To begin, collaborative governance identifies that measuring the social success of collaborative programs requires understanding the system context in which the program operates, the dynamics of the collaboration, and what the collaboration produces (Emerson and Nabatchi 2015). Groce et al. (2019) describe these three elements as the social networks, social processes, and social outcomes, respectively, and distinguishes these from environmental outcomes. Emerson and Nabatchi (2015) also identify that metrics need to be tailored to the unit of analysis, that is, for whom the outcomes are being evaluated—for the participant or for the collaborative entity. There are many existing metrics that weed management practitioners could use to evaluate the impact of their programs for each element and unit of analysis.
Measuring the Social and Social–Ecological Success for the Collaborative Entity
Practitioners explained that they consider projects to be successful if the projects reach diverse actors not usually involved in weed management, and if the size of the network grows without their intervention (Table 1). Such impacts relate to the system context and can be measured using metrics from social network analysis undertaken at the start of a collaborative initiative and at multiple intervals thereafter. Social network analysis can help document the number of relationships built and strengthened (Bixler et al. 2016; Emerson and Nabatchi 2015) or the extent to which the network is centralized or heterogeneous (Groce et al. 2019).
Metrics for social processes can help evaluate whether collaborations achieve diverse and inclusive participation (Conley and Moote 2003), open communication, candid and reasoned discussion (Emerson and Nabatchi 2015), continual generation and inclusion of knowledge, and refinement of approaches based on new information (Bixler et al. 2016). With respect to social outcomes, metrics exist that can measure whether participants have gained shared knowledge or understanding (Conley and Moote, 2003) and the extent to which agreements have been reached (Emerson et al. 2012). Such process and outcome measures may help weed management practitioners to evaluate whether there are greater community understandings of weeds; improved capacity for, and uptake of, best management practices; and a clear strategy with coordinated priorities (Table 1).
Beyond social measures, practitioners sought metrics that can capture the social–ecological outcomes of their collaborative endeavors. Bixler et al. (2016) present a social–ecological performance evaluation framework that not only captures changes to social networks but also considers whether those networks result in more sustainable and effective outcomes that could not have been achieved without the existence of the network. This requires practitioners to consider whether cross-pollination is happening between projects and organizations that work on different resources, how social capacity overlaps with ecological need, whether social and ecological goals are being (concurrently) achieved, and whether the network is resilient to fluctuations in funding and politics.
Moore and Koontz (2003) found that composition of collaborative groups revealed important differences with respect to perceived accomplishments. Similarly, Emerson and Nabatchi (2015) argued that the applicable metrics are likely to vary with the type of collaboration but did not provide examples of which indicators are likely to be different or how. Here we found some differences in the reported accomplishments of cross-tenure collaborative weed management projects by composition and scale. Agency-based programs operating at regional or local government scales were the only ones to report coordinated priorities and aligned and nested projects as evidence of success in their programs (Table 1). Some agency-based programs at each scale (local government, regional, and state) used surveillance as an indicator of success. Agency-based programs did not use extent of area inspected as a measure of success at any scale. Citizen-based programs, all of which operated at the locality scale, did not use increased community understanding, impact on neighbors, or improved affordability for land managers as indicators of success. Only projects at the locality or local government scale used coordinated works as an indicator of success, and only projects at the local government scale used improved affordability and less impact on neighbors. For the remaining indicators, there were no clear differences across scale, composition, or type of group. Further research involving a larger number of case studies may be required to evaluate the use and suitability of specific metrics for different group compositions and types of collaboration and for different government arrangements elsewhere.
Measuring Social and Social–Ecological Outcomes for Participants
Almost all the metrics currently used by practitioners focus on the collaborative unit of analysis, rather than the perspectives of individual participants (Table 1). NRM evaluation research has identified that participants deem collaborative programs to be successful when they enhance working relationships, such as building trust, improving communication, and facilitating a better understanding of others' views and values (Barney et al. 2019; Leach et al. 2002; McKinney and Field 2008); are procedurally and distributively just, such as everyone who wants to participate in a program being able to do so, and whether there are avenues for participants' concerns to be addressed (McKinney and Field 2008); and whether individual needs are met (Frame et al. 2004). Such measures of participant satisfaction were not evident in the metrics currently being used or considered by practitioners, but could provide another avenue for evaluating the social processes of collaborative weed management programs and help show how outcomes for individuals relate to outcomes for collaborative entities.
Participants' perspectives can be used to evaluate whether ecological and economic outcomes have been achieved, especially where necessary baseline data and follow-up monitoring have not been conducted (Leach et al. 2002; McKinney and Field 2008). For example, McKinney and Field (2008) asked participants to reflect on whether programs deliver more effective and lasting outcomes than the “next best alternative.”
Some cautions need to be considered when using participant perceptions for evaluation. The “halo effect” occurs when participants overreport environmental improvements (Thomas and Koontz 2011). Participant evaluations may also change over time, such as through the “recall effect” (Emerson and Nabatchi 2015) and are related to their relationships with the collaborative group (Cradock-Henry et al. 2017). As a result, Leach et al. (2002) found that both perceived and “factual” data are necessary, because each captures different elements of partnerships' achievements. Alternatively, Emerson and Nabatchi (2015) recommend including the perspectives of multiple types of participants, such as individual and organizational participants, the recipients or beneficiaries of collaboration, as well as external funders. However, they also recognize that the more diverse the group of participants, the more likely there will be divergent perspectives on performance.
Going beyond Quantitative, Context-Free Measures
Practitioners identified case study descriptions as one way to provide project evaluations that reflect connected and collaborative successes in context. Such calls are consistent with broader trends in evaluation methods research and practice (Thomas and Koontz 2011), which is experiencing a reorientation toward methods that recognize and reflect complexity (Reynolds 2017) and context (Clement et al. 2020). Case studies have been recognized in other NRM evaluation research as being useful for explaining (chains of) causal mechanisms that link management interventions with environmental outcomes (Thomas and Koontz 2011). Case studies can also investigate whether more centralized approaches would have resulted in different outcomes (Thomas and Koontz 2011).
Researchers involved in collaborative management of other invasive species, such as pest animals, have identified narratives as a powerful tool that can not only aid evaluation and reflection, but can also help to build solidarity and a shared vision and fuel collaboration (Miller and Bridger 2019). Narratives can help give meaning to case studies and provide an avenue for expressing and recognizing emotions, values, and identities, fulfilling the practitioners' desire for “meaningful metrics.” Miller and Bridger (2019) provide examples of how to build and collect community narratives while being sensitive to divergent or alternative narratives. When narratives are collected in a sensitive way, they can help to provide context and appreciation for the significance of outcomes achieved, given the capacity and history of each community.
These social research methodologies align with the desire of practitioners for more contextual, cumulative and connected, and consistent metrics, and would also enable consideration of how participants (such as individual landholders) measure success and outcomes.
The practitioners in this study define successful collaborative weed management as decade-long programs that bring together diverse organizations and individuals to deliver landscape-scale coordination of weed control efforts, ecological rehabilitation, and self-perpetuating community participation. Yet practitioners are seldom asked to report on such diverse outcomes and rarely have enough resources provided to undertake project evaluations in a way that enables them to adapt their collaborative programs in real time. Instead, they are often asked to report on metrics that only focus on inputs or metrics that focus on ecological outcomes without considering associated economic and social outcomes.
Social science from other areas of NRM and collaborative governance suggests two main ways that practitioners can overcome such challenges. The first involves expanding the social metrics being used to encompass social networks, processes, and outcomes while concurrently evaluating whether ecological and economic outcomes are being achieved. The second involves bringing project participants into the evaluation process, either by providing individual perspectives on the ecological, economic, and social achievements of the program or by the development of collective narratives. Such approaches recognize that success is not only about weed management programs being run collaboratively, but that evaluation needs to be a collective process that bridges multiple scales, or units, of analysis. This way of evaluating collaborative weed management programs requires funding bodies and government departments to redesign their evaluation processes to ensure that they are meaningful for those developing, delivering, and participating in the programs. Such a redesign requires supporting research to determine whether the metrics identified by the practitioners in this study are useful for evaluating the relative benefits of collaborative cross-tenure weed management programs, and the extent to which they are broadly applicable across other regions of Australia and internationally.
Acknowledgments.
This research was conducted with the support of an Australian Research Council Discovery Early Career Research Award DE200100234. Thank you to Matthew Hammond for providing IT and logistical support throughout the focus groups. We would also like to thank the remaining focus group participants for their rich insights: David Pomery, Greg Thompson, Luke Pope, Jonathan Sanders, Mel Schroder, Sonya Reyes, and Tony Robinson. No conflicts of interest have been declared.
