The BFD | Overseer Not Fit for Purpose

Dr Samuel Dennis
grounded.nz

You may have noticed the recent headlines regarding the Overseer model (used by regional councils to determine farmer compliance with environmental regulation) being severely criticised in a recent independent review. “Major tool for managing farm pollution gets a fail from reviewers”, “Report finds shortcomings in Overseer tool”, “Damning report of Overseer programme prompts Hawke’s Bay to change approach”

This article explains both the situation and the implications for farmers.

The use of the Overseer model as a regulatory tool has long been questioned by farmers and scientists. The government has tasked regional councils with the job of regulating nutrient losses to water, but nutrient losses are notoriously difficult to measure or even estimate. The Overseer model however existed, was already used by the industry – and produced estimates of leaching losses. Actual numbers, in black and white. Just what the councils needed. So many councils latched onto Overseer as the obvious solution – arguably the only solution – regardless of concerns about appropriateness.

The Parliamentary Commissioner for the Environment reviewed Overseer in 2018 and concluded that there were serious reasons to be sceptical about its appropriateness for regulatory measurements. He recommended that the model be thoroughly and independently reviewed. That review was conducted over the past year, and the results have now been published.

The report firmly concludes that Overseer is not fit for purpose as a regulatory tool. In response, the government has clearly indicated a desire to either move away from it, radically reform it, or both.

I am very impressed with the quality of this report. The authors were well selected, I know some personally and others by reputation. Their analysis is fair and clearly explained.

The reviewers conclude that they cannot be confident that the model gives accurate absolute numbers. But more importantly, they also cannot be confident that it accurately indicates relative leaching losses. This is a more scathing criticism than I was anticipating (and far more scathing than any criticism I have ever made myself!).

I have generally assumed that if Overseer says leaching loss under one management system was 20 kgN/ha, and under a different system was 25 kgN/ha, we cannot trust the actual numbers, but we can be fairly confident that the second system has higher leaching losses than the first system. So we can still use Overseer to help find management systems that reduce leaching losses. This review concludes that we cannot even confidently use it for that purpose.

But There’s a Catch…

Farmers, don’t get excited. The reviewers believe Overseer is likely to underestimate nitrogen leaching losses, not overestimate them. You should expect your actual leaching losses to be higher than what Overseer is saying, NOT lower.

This is because Overseer only calculates nitrogen leaching in the form of nitrate. It ignores leaching of nitrogen as ammonium and organic N. It also ignores surface runoff of nitrogen in any form. If Overseer is replaced with a better tool that actually accounts for losses of N in these forms also, the assessed N losses will frequently be higher than what Overseer has said. Possibly a lot higher.

It also means that direct measurements of leaching, such as with the GroundTruth system, can be often expected to be higher than Overseer estimates.

This is an even greater problem because Overseer “has been used as a policy-making and policy implementing tool and for property-scale and catchment-scale nutrient management … in some fashion, by most regional councils.” (Government response). This means that if Overseer is underestimating leaching losses, then the limits set in existing regional rules are likely also too low. If a new tool is used for on-farm measurement, that increases estimated leaching loss, but is used to assess leaching against the existing rules in the plan, the comparison will be unfair and may unfairly penalise many farmers.

I well recall a certain Canterbury subcatchment where I was on an expert caucusing panel around rule development. In that catchment, the maximum permissible nitrogen loss to waterways from the whole catchment had been calculated using Overseer. I questioned how they had done this, as what actually matters is the concentration of N in waterways related to environmental outcomes, and Overseer knows nothing about fish. It ultimately appeared that ECan had simply decided what management they felt would be appropriate in that subcatchment, modelled the catchment in Overseer managed in this way, and then set the outputs of that modelling as the nutrient cap. This was completely circular reasoning with no relation to the real needs of the environment, and it was very concerning that the final limits could be far higher or lower than the environment required as the environment had not been adequately considered. Overseer has had a heavy influence on regional rules.

Changing from Overseer therefore also necessitates reviewing all regional rules. This essential step is however not being proposed by the government. This is a serious problem that could cause major further restrictions on farmers, potentially without actually helping the environment.

Summary of the Reasoning in the Report

Overseer was originally designed for fertiliser recommendations, in the 1980s and 1990s. A key goal was to make the model use input data that was readily available to any farmer. Also, at this time, much less data was available than is now available (e.g. daily climate data). As a result, the model began with an unusual structure and includes many simplifications. This novel structure is good for the original purpose of the model – it is not itself a problem.

Since that time, many further features have been added to the model – including nitrate leaching. However, these have been done by adding modules in a somewhat disjointed fashion. The resultant model is highly unusual when compared to other soil models, and is a patchwork of approaches that are not well integrated with each other and calculate the same thing in different ways in different places.

Some of the component modules are very good, and the reviewers are quite comfortable with them. Others are considered to be quite poor – and that includes the crucial calculations of leaching from urine patches.

Crucially, the model does not attempt to achieve a mass balance – it does not ensure than nitrogen inputs equal nitrogen outputs. This will be due to the patchwork nature of the workings of the model. But it seriously reduces confidence in nitrogen loss values.

The model also fails to account for some very crucial factors, such as real climate data (especially short-term events), organic matter changes over time, deep rooting plants, and different soil horizons. Part of this comes back to the fact that it is a steady state model, while the real environment is constantly changing.

Where to from Here

The government is proposing to:

Firstly, create “a new risk index tool, potentially using elements of Overseer (including the user interface)”.

I take this to mean they will basically tweak the Overseer user interface to replace the N loss numbers with “low”, “medium”, “high” or something like that. This is something that Overseer should have done years ago. It has been long obvious that if Overseer says leaching is 53 kgN/ha, that only means leaching is “probably high” – we can have no confidence in the actual number of 53. But by using a precise number, with no error values in the interface, Overseer has given the illusion of accuracy and certainty. This has encouraged bureaucrats to accept these numbers as if they were real, and to use the model inappropriately to determine whether farmers are above or below thresholds. That use has always been inappropriate – even the CEO of Overseer has stated “it’s not designed to be used to deliver an absolute pass/fail against an absolute value”.

If Overseer had used risk bands like this from the start, we might not even be in this situation today at all. Overseer would already be being used in a more limited and more appropriate fashion, where the errors in the model were less problematic.

This change will be a helpful measure in the short term.

Secondly, the government wishes to do one or more of the following things:

Develop “a next generation Overseer to address the issues raised by the Review Panel and ensure that it is fit for purpose as a tool to use in appropriate regulatory settings”
“greater use of controls on practices and inputs to manage nitrogen loss”
“a completely new approach to understanding and managing diffuse nutrient loss risk.”. Several examples are suggested, but they really sound like they’re quite unsure what to do and are open to any good ideas.

The government ambitiously aims to have “fit-for-purpose tools for use within regional plan rules” available within 12 months. I would be surprised if this is achievable.

In the meantime, the government states that “existing resource consents [using Overseer] must be administered and new applications processed by councils as the consent authority. The Review Report does not change that.”. There is therefore no immediate relief for any farmer already caught up in the consenting process. On the contrary, I can forsee compliance becoming even more complex, as councils are likely to continue to require Overseer budgets as per resource consent conditions, and simply add on any new tool on top of that – there may be a time when nitrogen must be accounted for in multiple ways.

The proposal to use more input controls is particularly concerning for farmers also, for obvious reasons.

The government also states “Where opportunities exist to use additional data, alternative modelling or for control to be exercised using means not dependent on Overseer, those opportunities should be considered in addition to, or as an alternative to, Overseer modelling. This will be particularly important where landscape factors and other evidence suggests that nitratenitrogen is not the dominant species of nitrogen loss at the farm and/or catchment scale.”

The GroundTruth System is one good option for such situations.

I would suggest that farmers who are not yet applying for consents consider very carefully whether to apply for one at this stage (and receive conditions including Overseer), or whether to hold off (as far as is legally permissible) to see what the future holds in order to receive conditions that are solely based on whatever new tools may be adopted. It is not at all clear which approach would be better, discuss this with your nutrient advisor.

Conclusion

It is good that the long-recognised problems with Overseer have finally been brought out into the open, and the government has taken notice.

However, this is actually nothing to get excited about. The next few years will actually see further complications in agricultural environmental compliance, not easing of it, as a result of this report. And it is not at all guaranteed that any such further complications will actually improve the real environment.

The government is not proposing to review existing legislation that used Overseer as a basis for compliance numbers, and that is a very serious omission with far-reaching implications.

There is a strong need for new approaches to leaching measurement, and I will be doing what I can to contribute to that.