Follow The Money: How to interpret “peer review” science.

 

I regularly hear or read arguments from agencies compromising our natural heritage that such and such studies support their management decisions.  However, often the agencies overlook or ignore contrary science that does not support the policy or management decision.

To give them a break, the average district ranger or even specialists like wildlife biologists, fire managers, and others often do not have time to keep up with the latest science. So, recognize that they may not really know the “best” science.

Yet, the public, and often the media, naively accepts without question the assertions of public agencies as “unbiased” observers. One often hears agencies defend their statements and policies by suggesting if everyone is angry with their positions, they must be “doing something right.”

Well not everything is splitting the baby. The Earth is round, not flat or half way flat.  Gravity exists whether you believe it or not—just try jumping off a cliff. There are some things that are more accurate than others.

Both proponents and opponents of various public policies rely upon scientific studies to give credibility to their positions and garner confidence in their assertions. So how does one determine whose science is reliable? Most scientists do not purposefully distort their findings, but how they interpret their findings and what they emphasize can influence how people understand the outcomes.

There are several ways that I decide the relative veracity of scientific research and whether to grant authority to agency representatives—I follow the money.

The Upton Sinclair quote that “It is difficult to get a man to understand something when his salary depends upon his not understanding it” often is a good starting point for determining the accuracy of statements.

I know few foresters, for instance, who are opposed to logging. If you are a forester, your job depends on cutting trees.

The first thing I do is look at the occupation and affiliation of the spokesperson. Obviously if you are reading a study about the safety of smoking cigarettes and the authors work for tobacco companies, this would raise a yellow flag of caution. But it is not always as obvious that there are conflicts.

For instance, one of the ideas we hear surrounding livestock grazing is that grazing can prevent large wildfires by targeted removal of the fuel—grasses—that sustains fires.

There have been several studies that purport to show that “targeted” grazing can halt large fires. So, the first thing I do is look at who the authors are and where they are employed. The studies that suggest that livestock grazing—at least the ones I’m aware of—are all done by people in range departments.

Why is this important? Because if you are a professor or graduate student in a range program, your entire budget and survival as a professor and department depends on maintaining livestock grazing on public lands. Hence you have a vested interested in promoting livestock “benefits” whether real or imagined, and minimizing any known negative effects.

Next I look at the journal where the research was published. Not all journals are equal. Some are much more discriminating in the papers accepted for publication. Some journals also have unspoken biases. For instance, the Journal of Rangeland Ecology and Management published by the Society for Range Management is biased towards promoting livestock production.

You will find few papers in this journal that recommends removal of livestock as the best management option, even when the authors may document significant resource damage from livestock. They almost always recommend “proper grazing management” as the solution to problems, whether proper management can work under field conditions.

Beyond the journal publication, one then must look at the funding source for the study. If you are doing range studies in the western United States, most of your funding will be coming from either livestock organizations, and/or federal/state money appropriated to demonstrate why livestock grazing is a beneficial use of the landscape.

In all instances of the above examples, money and jobs dictates the perspective of the individual, and woe to the person who steps over the line and does not promote the accepted policy positions.

So, in the case of targeted grazing, you must read crucially the study methods, and conclusions. Most scientists have integrity—even those who are advocates of grazing, logging, oil extraction or whatever. They do not out-right lie or distort their findings.

What they do instead is restrict the kinds of questions they ask, how they set up their experimental design, how they interpret their findings and what they propose for solutions.

For instance, I studied wildlife biology in college. There was not one professor of mine that questioned or even raised the question whether hunting wildlife was appropriate and perhaps harmful to the long-term survival of the animals. (And yes, there is evidence that even “regulated” hunting can negatively impact wildlife).

We simply never discussed this idea because almost all wildlife biology professors get a substantial amount of their research funding from Fish and Game agencies.

As an example of the unexamined assumptions, one of the papers widely referenced by the BLM in its management plans to save sage grouse champions “targeted grazing” to reduce western range fires. The original study was based on the grazing of several small plots where the cattle were confined by fences and herding. While the grazed areas did have less fuels, the applicability of this management of confinement to large public lands allotments is questionable. The cost of such confinement would be prohibitive.

While the research might show that “targeted” grazing could perhaps reduce some wildfires, the practical application of this approach to wide-open public lands allotments is questionable.

Beyond the costs and the lack of landscape scale application, the study relied on “models” of fire behavior to conclude that grazing would reduce wildfires.

Models are notoriously imprecise. As the saying goes, what goes into the model affects what comes out.

One of the factors in their model was limiting the wind speed in the fire models.

Why is this important? Because nearly all large wildfires are driven by high winds. Under less than high winds, wildfires do not spread rapidly and are easy to control—whether grazed or not.

Models are better than nothing, but they are no substitute for empirical data. In other words, direct observation of how real wildfires interact with grazed landscapes.

Finally, in at least one of these studies, the authors admitted in the very last paragraph that their findings only applied to wildfires burning under low to moderate weather conditions.

One had to read the entire paper to find this one line which is a dead giveaway that targeted grazing is not likely to significantly influence the large wildfires burning across the West. These large wildfires all burn under extreme fire weather conditions.

The same caution applies to other science as well. Nearly all the science supporting thinning/logging to reduce high severity wildfire is done by forestry schools and/or researchers who work for federal or state forestry agencies like the US Forest Service.

For instance, the Oregon State University Forestry School gets 10% of its funding from a tax on logging, which alone would be enough incentive for the department to have a favorable perspective on logging issues, not to mention that timber industry dollars also directly fund some of the department’s research.

No one wants to bite the hand that feeds them.

I hasten to add that this does not mean all research done by forestry schools, nor are all professors in such departments minions for industry. Nevertheless, there are often unquestioned assumptions that permeates the science. A reasonable person would exercise caution in accepting all “peer reviewed” science as equally valid.

 

 

 

 

12 thoughts on “Follow The Money: How to interpret “peer review” science.

  1. There are examples of this bias, intended or not, all around us. In 1964 Robert Giles, my big game management professor covered this topic in detail and I found it intriguing. Runs the gammet from nonveiled advertising to the more veiled actions pointed out by G.W. Since 1964 I have developed an autoimmune system you might say that I can’t help but look for the bias in such statements. Statements from the NIH is one source that is virtually devoid of bias for obvious reasons. Excellent article G.W. and I have nothing to gain from saying so!

  2. Thank you, George, for reminding us that there are tools available to help us determine the validity and accuracy of these “peer reviewed” studies. Please confirm, then, that one way would be to inquire or search for the journals where the studies are published. Anything else?

  3. George, what journals would you recommend as being both unbiased and rigorous? What sources of funding would, in your opinion, allow for unbiased and rigorous reporting of results? Since a,most any funding agency could have a perceived conflict of interest, what are your recommendations for resolving this conflict? You’ve raised good questions, but I don’t see any suggestions on how to solve the problem, which would be helpful for advocacy.

    1. Can your State University Health Science Dept, Librarian.. They can explain it!!

  4. That good scientific studies can be replicated multiple times by independent researchers should be a given. Otherwise, unfortunately because of the “follow the money” influence, we are too often asked to believe in what has recently become known as “alternative facts.”

  5. I read a science journal where the article stated that GMO is safe. The author signed as being a biologist. I looked up other articles by the author. All of them challenged the nongmo community.

    This author may have been an animal biologist, knowing nothing of genetically modified organisms.
    And,all the articles he submitted being about GMOs signifies he may be hired to write the articles, and paid for getting them published..

    Critical thinking. That’s all it takes, is critical thinking.

    1. Watch the other side too. I’ll actually name an example, and encourage you to do the same, so I can judge if it’s genuine.
      https://en.wikipedia.org/wiki/S%C3%A9ralini_affair
      But my main defense is not who author or journal was, it’s read the paper. I note that GMO debates that are not in science papers, but mimic news coverage, are often chock-full of shills on both sides, and it’s often hard to tell who is funding it. On health topics I can often smell the money from big sugar too.

  6. Following the money here led to a misguided uproar.

    Wolves Not the Cause of Wyoming Elk Decline

    http://www.outdoorlife.com/blogs/newshound/2013/06/study-wolves-not-cause-wyoming-elk-decline

    The Study**
    Middleton and a coalition of biologists GPS collared wolves and elk west of Cody, Wyoming in and around Yellowstone National park. In a study area of about 1 million acres, they monitored interactions between predator and prey. Over three years they observed the animals in January, February, and March – when wolves typically put the most stress on elk. The study was funded by a variety of organizations and agencies including Rocky Mountain Elk Foundation, Boone & Crockett Club, U.S. Fish and Wildlife Service, and Safari Club International.

    I remember when this study first hit the blogosphere. Perhaps I have the original buried somewhere on my old computer, but I remember the outrage by some of the commenters, one or two well known anti wolf folks, who saw the article, did not read who funded the study, and jumped into the fire accusing NGO’s and Ralph Maughan for the said funding. It was hilarious.

    Also, the Middleton editorial in The NY Times

    https://www.nytimes.com/2014/03/10/opinion/is-the-wolf-a-real-american-hero.html?_r=0

    Where he debunked the whole Yellowstone tropic cascade theory…wolves changing the course of rivers… is derived from his study. This editorial is cherry picked for that one item (Mech and associates also have rightly picked the trophic cascade theory apart: correlation does not necessarily represent cause) by those who do not favor the presence of wolves, but conveniently omit the remainder of the editorials message.

  7. This author has a habit of not telling “the rest of the story”.

    On the west coast, there are three large colleges (Oregon State, University of Washington and Humboldt State) that have forestry and natural resource programs that do extensive research. Of course they do research that is pertinent to their curriculum, but together with the Pacific Northwest Research Station (funded by the Forest Service, BLM and other federal agencies) have been instrumental in researching old growth forests and the species that inhabit them. There is “no money” in discovering that northern spotted owls and marbled murrelets prefer “old growth forest” nor that large wood is critical habitat for aquatic and terrestrial species, among numerous other findings.

    Although I can’t speak for grazing practices, I led ~100 timber projects during my career with the BLM and I can tell you unequivocally sitting around the table with wildlife and fishery biologists, soils scientists, fuels specialists, botanists, silviculturists, road engineers and foresters, each one “fought” for their resource using the “best available science”, knowing full well, the projects completion depended upon it.

    Of course it’s easy to make generalizations such as the authors, but due to the diligent watch of environmental groups, the day of “shopping” for science that best meets timber production is long gone in the Pacific Northwest federal forests and I would presume in most other places. I would highly recommend anyone who is skeptical; read a federal timber sale EA and then decide if the agency just “followed the money”.

    A little cumbersome, but here is a link to all BLM projects and their NEPA documents.

    https://eplanning.blm.gov/epl-front-office/eplanning/nepa/nepa_register.do

  8. The problem with so-called “wildlife biology”, which is essentially re-branded game management, is deeper than funding sources. The primary purpose of the field, as established by Aldo Leopold, is to maintain huntable wildlife. Review of game management decisions by colleagues in the field of game management/wildlife biology may be peer review, but it is not objective science. An actual scientific review of wildlife policy would involve experts in established scientific fields such as mammalogy and conservation biology.

    1. I think there is pretty much fairly objective science. For example what happens to plants or animals when deer or predators are altered. What I agree with is that in deciding what to do, there are additional considerations that science can’t provide, that have to do with the value of things. It isn’t hypothesis testing, and it isn’t model selection, it’s decision theory. Depending on your field that extra piece might be called “the objective function” or the loss function. Allot of that can be quantified in terms of money, though that’s complicated, and some of it is really hard to quantify. Deer or salmon management near me would probably need an economist on the panel too.

Comments are closed.