The Nature Of Science Of Nature

I have judged many a science fair project in my

day. It is a learning experience, so adherence to

the steps of the scientific method taught in most

classes is O.K. But the students should realize

that there isn’t just one scientific method. This

young lady’s strategy was to take all the available

time and be the only project that got judged;

therefore, she's the winner.

A summary of the usual (but hopefully disappearing) lecture on the scientific method –

      1.     Define a problem

      2.     Research what is known

      3.     Hypothesize a cause

      4.     Test your hypothesis

      5.     Analyze your results

      6.     Support or refute your hypothesis

In the next few class periods, a few examples are identified and the class works through them. And there is always the "design an experiment to prove your hypothesis."

The question of the day:

Is this the only method to conduct science?

I don’t have room to go into all the things that are wrong with the process as it is enumerated above, but let’s hit a couple.

One big problem is the idea of supporting or refuting your hypothesis (hypo = under, and thesis = proposition; a supporting basis for an argument). True, your experiment may do one or the other, but it doesn’t have to. Many experiments end up with ambiguous results, especially if the scientist is doing his or her job.

I am using this picture because I couldn’t find a

way to work the quote into the post. But in truth,

science is like life, better results often come from

many trials. So... science is like life which is like 
science.

The design of the experiment should minimize any confounding effects that would render the results less than explicit. But, as the old saying goes, “You don’t know what you don’t know.” The middle of an experiment is often a learning moment. It is when you get into a study that you find the things that are going to make the study useless. Then you go back and design a better experiment.

Let’s say you have designed an experiment that will have a definitive result. What result are you going for? Most will say that you are trying to see if your hypothesis is supported. But anyone can design an experiment to give a desired result. Your hypothesis says “A” should occur if you do “B,” so you design an experiment to make sure “A” occurs. It doesn’t even have to be a conscious effort, your design will just tend toward that result.

A truly scientific experiment is designed to REFUTE your hypothesis. You design an experiment to prove your hypothesis is not true. If, under those, conditions, your hypothesis is still supported, then you really have something. If a scientist tries his/her best to refute their own hypothesis and can’t, the chances that the hypothesis is correct go way up. Then you report your results and let other scientists try to design an experiment to show your hypothesis is wrong. If they can’t do it either, then you are really onto something. Real scientists try to prove themselves wrong, not right. The big idea - you can never PROVE a hypothesis, you just have data to support it. But the next experiment may refute it. You can always design another experiment to test the hypothesis, so no hypothesis is ever proven absolutely. True science proceeds when you refute a hypothesis; only then can you make a concrete change to move closer to the truth.

Negative results and refuted hypotheses are the basis of science; 

too bad they get a bad rap.  Can you think of another profession 

where being wrong is your goal?

This leads to the next problem implied by the process outlined above. It is usually taught that negative data is a bad thing. Even people in the profession often downplay the importance of negative data; ie. data that does not give a result that is publishable.

Editors don't get excited over studies saying we tried this and it didn't work. But, an experiment that doesn’t work isn’t necessarily bad – you can learn a lot from it and so can other scientists. Unfortunately, journals don’t like to publish this data, so those who might learn from it don’t get to hear it.

This is one reason why it is important for scientists to have meetings and talk to one another personally, not to just write journal manuscripts and funding applications. Case in point, it turns out that studies that show new drugs aren't cure-alls, that they don't do what they say or don't do it as well as they say don't published. Ben Goldacre gave a recent TED talk on the subject, and has numbers to back up his assertions that negative data studies on drug efficacies hardly ever see the light of day.

In fact, negative data is the most common data and often the most useful. Refuting your hypothesis is a type of negative data. When faced with this result, you modify or discard your hypothesis and try again. You can design a thousand experiments that support your hypothesis and still not prove that your idea is the true mechanism - you may just not have thought of the experiment to disprove it yet. Like we discussed above, just one experiment that DISPROVES your hypothesis results in a step forward. Like Thomas Edison said, "If I find 1000 ways something won't work, I haven't failed. I am not discouraged, because every wrong attempt discarded is another step forward."

Negative data truly moves us forward, in fact failing is the only way we move forward. But this still leaves a problem with the way science is taught. Is there good data that is neither positive or negative? We tend to think of data only as that information that supports or refutes a hypothesis, but do you have to have a hypothesis?

This is the black walnut tree (Juglans nigrans).

The one in our front yard is about 70 feet tall and

produces over 200-300 kg (500-650 lb) of fruit.  
Black walnut dye comes from the husk, not the 
nut, and is yellow when immature and black 
when mature.

Consider an experiment I have been conducting for the last 13 years. We have a black walnut tree in our front yard, and I have been counting the number of black walnuts it drops every year since we moved in. I had no mechanism I was trying to define, I just wanted to know how many walnuts the tree produced.

Here is my data:

Year                        # of walnuts

2003                        3662

2004                        604

2005                        3508

2006                        368

2007                        4917

2008                        0                           

2009                        6265         

2010                        0             

2011                        6395
2012                        6
                                                                                       2013                        2140
                                                                                       2014                        159
                                                                                       2015                        1825

Now, we can ask a question and hypothesize a mechanism? What is responsible for the pattern in nut production, and why do the results keep diverging? Does that mean that my original observations aren’t science? True – you could say that I was answering a question about how many nuts the tree produces, but I did not have a hypothesis that I was trying to dispute. This is true science, but not the kind we teach in school. Is the change in the pattern during the middle years accounted for by weather? Do the fewer nuts later mean that the tree is dying?

Black walnut meats are expensive because
they are hard to get out of the shell. But the

shell is also economically important, used in

paints, oil wells, explosives, cosmetics, cleaning

and polishing agents, and jet blasting of metallic

and plastic surfaces.

Try this experiment at your school. Find a tree and start to count the nuts, or find a sapling and count the leaves each year. You can keep this experiment going over a number of years, with each class adding their data.

But the real learning is in defining the limits and possible confounding effects that could lead to errors. Did the tree lose a limb and therefore produce fewer nuts the next year? Was there an explosion in the squirrel population, and they stole all the nuts before you could count them? What was the weather over the time period you observed, could a change in weather account for a change in number? Is there another walnut tree too close, and the nuts are getting mixed up? Am I just getting better at finding and counting the nuts each year?

Squirrels are kind cute, if you forgive them for

carrying rabies. Here, he represents one source

of possible error in my black walnut counts. Can

you assume that every year they steal about the

same percentage of nuts before I can count them?

I have asked these questions and am observing multiple parameters to see if they account for the pattern in nut production. But there may also be a biologic reason, something to do with energy output versus opportunity to produce offspring. All these items can be investigated and used to better explain the observations.

Each might be considered a hypothesis – the weather affects nut production, so you try to show that different weather years had the same nut production – hypothesis refuted. The squirrel population exploded – talk to the local nature experts, if the number has been fairly constant- hypothesis refuted. There is an almost infinite number of possible confounding effects, and your class can come up and test as many as their brains can think up. Now that is a true scientific method!

Ben Goldacre (2012). What doctors don't know about the drugs they prescribe. TED MED 2012