The multimillion dollar computer model used by the Bay Program to determine how pollution — and cleanup efforts — will affect the Chesapeake is flawed and should not be used in decision making, a team of outside academics has concluded.
In a strongly worded report, the academic review team said that the Bay Program’s water quality model fails to accurately simulate key physical and biological functions in the Chesapeake, raising doubts about any conclusions drawn from the model.
“It is the opinion of this team that the Water Quality Model does not currently provide information suitable for major management decisions and that use of the model for such purposes should be suspended,” the review team said in a report presented in December to the Bay Program’s Scientific and Technical Advisory Committee (STAC).
Bay Program officials strongly dispute that assertion, noting that model results generally match up well to real-world conditions. “There is a good comfort level with the model as it has been used for management,” said Rich Batiuk, associate director for science with the EPA’s Bay Program Office.
At the same time, he agreed that “we have certainly seen room for improvement.” And in recent weeks the Bay Program has assembled three teams of well-known scientists from Bay-area universities to help review and revise the model because of issues raised by the review. “The credibility of the model is important,” Batiuk said.
In addition, STAC, which selected the review panel, is expected to make specific management recommendations to the Bay Program regarding its modeling efforts.
The review is a blow to one of the Bay Program’s most important decision-making tools. While officials and scientists both within and outside the Bay Program generally view problems identified by the review team as correctable, they say it will likely take most of the next year — and more money — to satisfactorily complete the refinements.
The credibility of the model is increasingly important because — unlike the past when nutrient reductions were largely voluntary — nutrient control efforts are increasingly being regulated, and are therefore more subject to legal challenges.
Right now, the Bay states are under a tight deadline to clean up the Bay by 2010 to avoid having to impose a potentially more harsh and prescriptive cleanup plan, known as a Total Maximum Daily Load, throughout the watershed to curb pollution.
The water quality model is considered by many to be the Bay Program’s most important tool in determining the amount of nutrient reductions — tributary by tributary — needed to meet its “clean Bay” goals. Many officials believe those goals, which for the first time will affect all six states in the watershed, will require far more nutrient reductions than have taken place in the past decade.
“The expectations and scrutiny of the model are only going to increase,” said Robert Magnien, head of the Maryland Department of Natural Resources’ Tidewater Ecosystem Assessment Division.
Magnien was one of the officials who called for a model review. Although the review team’s report may cause some short-term delays, Magnien said that identifying and fixing problems now will pay off in the long term because everyone from farmers to developers to wastewater treatment plant operators will be affected by — and need to have confidence in — decisions that stem from the model.
Magnien and others stress that the model is not used in isolation — officials also draw on years of research and monitoring data when making decisions. But the model is the tool relied on to predict how the Bay should respond to different management actions such as various levels of nutrient reductions, and it ultimately influences how tens of millions of dollars are spent.
“For quantitative predictions, we just don’t have any other tools but models to turn to,” Magnien said, “so we need to make them as accurate as possible.”
Over the years, the Bay Program has relied heavily on three large computer models to make decisions: A watershed model that calculates the amount of nutrients and sediment that enter the Bay; an air model that calculates the amount of atmospheric nitrogen deposition to the Chesapeake; and its watershed, and the water quality model, which calculates how loads of nutrients and sediments from the other two models affect the Bay. The academic team only reviewed the water quality model.
Development of the water quality model was begun by the U.S. Army Corps of Engineers in the 1980s. In its “virtual” Chesapeake, the model simulates the growth of algae, the vertical and horizontal movement of water and nutrients, the amount of oxygen available in different parts of the Bay and a host of other variables. It is generally considered the most complex and sophisticated model of any estuary in the world. The model is so complex that it takes some of the fastest supercomputers in the world 20 hours to run a scenario that shows how a particular management action would affect the Bay over a simulated 10-year period.
Because of its complexity, the review team did not evaluate the entire model but instead examined how well it simulated several important processes. The reviewers found what they considered to be serious errors.
They said the model erred in calculating the amount of primary production — photosynthesis by algae — which is created from nutrient inputs. Baywide, the model’s calculated productivity was only about half what is actually observed, and in some places production was underestimated by five times, the review team said.
Instead of a rich, productive estuary, they said the primary production in the model more closely simulated an “ocean desert” like the Sargasso Sea.
Algae production is an important factor in causing oxygen depletion in the Bay; when algae die and sink to the bottom, they are decomposed by bacteria in a process that depletes the water of oxygen.
The review team concluded that the model’s ability to relate nutrient inputs to low oxygen conditions was poor. The review team also found problems with other biological processes such as respiration — the rate at which organisms consume oxygen and recycle nutrients — that would affect oxygen concentrations throughout the Bay.
“Certainly, any model cannot embrace the full complexities of the natural system,” said Hugh Ducklow, a scientist with the Virginia Institute of Marine Science and a member of the review team. “But they never seem to have made an attempt to look critically at some of these processes like photosynthesis or respiration.”
The report also questioned the accuracy of the modeled circulation and water-column mixing. Despite those errors, the review team acknowledged that the model produced dissolved oxygen levels that appear to closely track observations when it is used to simulate real-world conditions.
That also raised concerns. “That means you’ve got a lot of other things wrong in your model, because you’re starting with the wrong amount of primary production in the model, and the wrong nutrient dynamics,” said Scott Nixon, a professor of oceanography at the University of Rhode Island and the chair of the review team.
“You may be getting something that looks right, but it looks right for all the wrong reasons,” Nixon said. “And if you are going to use the model in a forecasting mode, then you’ve got to be getting things right for the right reasons.”
Bay Program modelers tend to view the issue partly as a cultural clash, which pits academics — who use models to help analyze how complex systems work— against engineers, who use models to determine the impacts various management decisions will have on systems like the Chesapeake.
“This debate between scientific and engineering modeling has been going on for decades,” said Carl Cerco, the scientist with the Corps of Engineers who is primarily responsible for developing and operating the model. “I’m not surprised to see this.”
Outside scientists tend to agree, up to a point, that the review team failed to appreciate that the management needs of the model are different from academic needs. At the same time, they say the Bay Program aggravated the problem by failing to cooperate with the review, and on occasion supplying incorrect information to reviewers.
The result was a report that was harsh and, according to some, at times unfair. The review team, for example, contended that the model was lacking in peer-reviewed journal articles, when in fact seven have been published and two were in press. One paper, in the Journal of Environmental Engineering won the American Society of Civil Engineers Wesley M. Horner Award in 1995 for the best paper of the year.
Still, the review team noted — and Bay Program officials acknowledge — that no detailed documentation of the model had been published since 1994 despite major changes that have been made since then. Such documentation usually consists of reports hundreds of pages thick which describe in detail how a computer model works so it can be reviewed by outsiders.
Models, at best, offer a crude simulation of the natural world, and scientists both in and outside of the Bay Program agree that it is difficult to model primary production correctly. Few models have been successful in simulating such processes correctly without extraordinary effort.
Engineering models, like the Bay Program’s, routinely compensate elsewhere in the model to correct such miscalculations. Research models, in contrast, try to correct such errors by spending more time and effort trying to perfect natural process simulations.
Before being used to make forecasts, models are usually tested by simulating real-world conditions to see how well they match certain observed conditions, such as dissolved oxygen in the world.
Because of such runs, Bay Program officials say they have confidence in the model’s results to date. “As a whole, we have very good agreement between monitoring data and model results,” said Lewis Linker, modeling coordinator for the EPA’s Bay Program Office. “There is no way that you can get that agreement over a 10-year simulation without capturing the salient features of the system.”
But the failure to publish model documentation in the past five years makes it impossible for anyone to determine exactly what “tweaking” took place to help the model come up with relatively good numbers for dissolved oxygen when other important functions were off.
Scientists outside the review team generally agree that the questions raised are serious enough to warrant a closer look. “I do think the basic technical problems that the review team found are real, and not insubstantial problems,” said Donald Boesch, president of the University of Maryland’s Center for Environmental Science and a member of STAC.
Boesch said the issue presents an opportunity to forge a better relationship between the Bay Program and scientists who study the estuary. As a result, he said the Bay Program was taking a step in the right direction to assemble teams of scientists to work on questions raised by the review.
“I think we ought to try to address those issues, and try to use it as a way to get some of the scientists who have some knowledge involved more intimately,” Boesch said. “I’m encouraging our guys to pitch in and help. I think we can improve the model, and also improve the process [of bringing academic involvement to the model] as we do that.”
Batiuk, of the Bay Program, sees the exchange being two-way. Not only will modelers get more insights from the academic world, but scientists will understand better how the models work — and how they are used.
“We need some skeptics on the inside,” Batiuk said. Once they come in and “look under the hood” they can make recommendations and ultimately “sign off” on the inner workings of the models, he said. That will improve the model’s credibility when it is used for key nutrient reduction decisions about a year from now. “We’ve got to have an airtight case,” Batiuk said.
Ultimately, he and others acknowledge, it will come down to a policy decision of just how “right” the model must be to be used for decision making. “You can’t just hold this to the standard that it must replicate the system perfectly,” said the DNR’s Magnien. “It doesn’t have to be perfect. But it has to pass the test of being sufficient for guiding management decisions.”
In the longer term, the increased interaction with scientists may help resolve a broader concern raised by the review team about the direction of the Bay Program’s modeling efforts — the need for more outside input. The review team depicted the Bay Program as a “closed shop” in terms of modeling, with all the investment being made in a single Bay water quality model operated by the Corps of Engineers.
“They sole-sourced [the model] and got one answer from one point of view,” Ducklow said. He contrasted the Bay Program’s approach to modeling with approaches used in global warming or ocean circulation models where government agencies, academics, consultants and others develop and compare various models.
Bay Program officials view the notion of competing models as a management nightmare. They, too, draw comparisons with the global warming models — whose conflicting results have contributed to gridlock on dealing with climate change.
“We’re doing something in the Bay Program,” Linker said. “We’re making [nutrient] reductions, as opposed to the folks who are just talking about taking some carbon out of the atmosphere as a good idea.”
As an alternative, some suggest that the rapidly increasing power of desktop work stations will allow the water quality model to be more widely distributed. In effect, they see it becoming a “community model” in which a number of people can use and offer specific suggestions for the enhancement of the model.
Others believe the Bay Program should invest more in simpler models that can be used in conjunction with its more sophisticated models to test, compare and fine-tune results. “There is something to be said for not putting all of your eggs in one basket,” Boesch said.
And some, like Eileen Hofmann, a scientist at Old Dominion University and a member of the review team, suggest that even as the Bay Program works on resolving problems with the existing model, they should begin planning for a “next generation” model to replace it.
“They have learned a lot, they have a lot of knowledge and insight, and all of that can be brought into a second generation model,” she said. “You don’t lose what you’ve done, you just go forward from here.”
Bill Boicourt, a scientist with the University of Maryland’s Center for Environmental Science who is familiar with both the review team’s report and the water quality model, agreed that thought should go into the development of a new model but cautioned against writing off the existing model too soon.
“There needs to be some effort put into it, but they are not killer efforts that are necessary,” he said. “But we need to stop now and sharpen it. I think the model is definitely worth it.”