Fix the PhD

Introduction:

From Spring- Fall 2012, I spent some time working as the Neurosciences Program representative to a student advisory board, working on curriculum reform for the Stanford Biosciences umbrella program. While I’m very proud of the work our committee did, particularly with implementing a new set of two week, intensive, hands-on practical minicourses, we ran into some fundamental obstacles when we considered how best to make grad school prepare students for a wide variety of career options. These obstacles existed not in Stanford itself, but in the structure of incentives for graduate schools as a whole. When the NSF put out a call for ideas to improve graduate education, I figured it was worth putting these thoughts into writing. The piece that follows was my submission, and generally outlines my feelings on what a science PhD program ought to look like for the 21^st century.

Broaden the Base of Excellence

In his 2013 State of the Union address, President Obama called for an education policy that increases “focus on science, technology, engineering and math, the skills today’s employers are looking for to fill the jobs that are there right now and will be there in the future.” Though a PhD in science or engineering can—and should—be valuable for any job that involves thinking hard about complex problems and coming up with innovative solutions, PhD programs generally do not train students with these options in mind. A substantial portion of trainees go into careers for which they have little or no training: as of 2011, the National Science Foundation reported that only half of the roughly 50,000 PhD graduates went on to academic postdocs, and that only 15% of the students that graduated five years earlier had become tenure-track faculty. If the underlying goal is innovation and economic growth, then we need to train people to thrive in non-academic-science positions where their scientific skills and expertise can help drive the economy.

PhDs could be starting companies that create jobs, communicating science to lawmakers, teaching the next generation of science students, or developing treatments in biotech and pharmaceuticals. But within the current framework of academic culture and funding incentives, most PhD training programs focus too narrowly on funneling students towards tenure-track academic science. We need to broaden the base of excellence in graduate training, maintaining high standards of quality while expanding the definition to include more careers that benefit from a scientific education.

In the long term, we need to fundamentally shift the culture of graduate school (for both faculty and students), from solely valuing the research that a student produces, to valuing the full range of meaningful contributions that they might choose to make with their training. But such a change cannot happen overnight—institutional factors, implicit biases, and the weight of tradition would not make it easy. In the short term, however, government could catalyze this transition, paving the way for scientific training programs that empower students to use their training for a variety of productive career options. If we change the way incentives are structured for the programs that train our PhDs, if we reward programs for training the best teachers and entrepreneurs alongside quality scientists, more fundamental change can follow.

Many graduate students, particularly those at top institutions, are funded by training grants. Some of these training grants are individual pre-doctoral fellowships, but some are also awarded at the level of a department or graduate program, such as the NSF IGERT program or the NIH T32. With funding for graduate students being a premium resource, the requirements for such grants, particularly institutional training grants, help to shape the structure and culture of training programs. Unfortunately, for many of these grants, the criteria by which a program or a student is judged (number of academic postdocs, number of publications in top journals) are far too narrow. Programs with institutional grants whose graduates later become leaders in industry, policy, or education are, at best, required to explain themselves, and at worst are actively discouraged. The first step in improving graduate education is for such grants to acknowledge that successful training means any career, research included, in which trainees use their skills and expertise to meaningfully contribute to society.

Some funding organizations are already getting on board with this definition. In 2011, the National Institute for General Medical Sciences put out a report calling for a comprehensive overhaul of their training grant criteria for success.  Based on conversations with stakeholders from all parts of the training process, from administrators to faculty to current trainees, the report suggests that the definition of success in scientific training needs to be broader. One version is particularly resonant: “For society, success is having a strong and diverse cadre of creative thinkers and innovative problem solvers.” To this end, the report suggests encouraging recipient programs to provide broad, flexible professional development training, and to encourage a focus on student development, rather than selection of talent alone. But why not go one step further? Funding institutions could actively encourage training programs that support any career path that thoroughly uses the skills and the training that a graduate education provides, and that make partnerships, within their universities and their communities, to offer training in the skills necessary to be competitive in today’s job market.

Institutional training grant applications currently require a description of how the program will provide professional development training to their students, usually focusing on scientific ethics and academic professional development, and individual predoctoral fellowships require statements of broader impacts. Why not also ask these institutions: how are you providing resources to students who want to acquire skills and knowledge outside the lab? How are your students exposed to a broad array of career options? Why not ask trainees on individual fellowships: how will you find the resources you need to succeed in the career of your choosing? Simply asking these kinds of questions incentivizes training institutions to experiment with how best to provide resources and encourage students to seek them out. Some programs might forge partnerships with biotechnology and pharmaceutical companies, or strengthen ties within their university to writing centers, public speaking centers, or career development centers. Others might, for example, set aside two weeks of every term as dedicated professional development time, offering courses in pedagogy, public speaking, interdisciplinary problem solving, or management. In turn, funding organizations can track which interventions get more graduates into jobs that use the skills they have acquired.

The hope, then, is that by changing the structural incentives at the highest level, we might begin the work of updating graduate school for the 21^st century. So that is the dream: a culture of graduate science education that empowers students to be the best in whatever science-related career that they choose, and that arms them with tools for thinking and interacting with others that make them valuable to employers across disciplines. Such a shift, beginning with a change in incentives, is what graduate education needs to train happy, healthy, empowered students who will develop into excellent researchers, leaders, innovators, and entrepreneurs.

Right now, many science PhD programs follow a similar pattern: take courses, perhaps act as a teaching assistant, take an oral and written qualifying exam, and then tackle a mentored project to generate data until you graduate. With those initial hurdles finished, the only measurement of success that a PhD student has—and the only one that has any traction in the current graduation school culture—is progress in the lab. From the qualifying exam to the thesis defense, the most important product of graduate school is the thesis itself, despite the fact that many students who go on to postdocs are not necessarily performing the same techniques or working in the same subfield. But what if students could measure their success not only by the progress of their research, but also by their progress towards the skillset they need for a rewarding career of their choice? What if taking a professional development course on public speaking, teaching, or management felt as valuable as running another experiment?

A PhD program for the 21^st century should focus on the trainee as the primary product of graduate school, and the culture of such a program should encourage a broad base of professional development and successful self-improvement to prepare trainees to meaningfully contribute to society. With such a program, a PhD could signify to any future employer that a graduate has a variety of marketable skills: critical thinking, making and evaluating evidence-based arguments, communicating complex concepts, identifying important problems from a body of background knowledge, and coming up with innovative ways to solve them. The students from such a program would benefit from a clear path to becoming leaders in whatever field they choose, from academic science, to industry, to policy, to teaching.

Admittedly, changing these incentives may require some work to measure and quantify, because the success of a program can no longer be boiled down to statistics about numbers of papers published in high-impact journals and number of students going on to academic postdocs. It will fall to the champions of 21^st-century graduate training to make decisions about what is sufficient breadth of career exposure, sufficient support for professional development, and about which career options are worth investing time and energy into supporting. Simply starting the conversation about re-defining success in graduate science education could be a priceless step towards improving the process, and is essential for building momentum for cultural and institutional change. Funding institutions, by virtue of the power they hold, are in a fantastic position to start these conversations and watch as their effects propagate. By taking those first steps out of the well-trodden path of purely academic definitions of success, funding institutions could become the trailblazers for bringing graduate education in line with the myriad of ways that intelligent, well-educated science trainees can contribute to society.

With immune privilege comes immune responsibility

I am not a neuroscientist. Having poked a few cockroaches with electrodes as an undergraduate, I left the brain behind for the glamorous world of immunology. But immunologists and neuroscientists alike are challenging the long-held belief that the brain is separated from the immune system, and are exploring the idea that good immunological health and good mental health are linked. The immunology community considers some organs “immune privileged” – that is to say that immune cells stay away from certain delicate organs because they tend to do more harm than good. When faced with a foreign particle, white blood cells (leukocytes) trigger an immune response typically involving heat, swelling, redness and pain, all of which together comprise inflammation. This is great for dealing with infection, but causes significant collateral damage to tissues, which may have important functions. For example, our eyes are so sensitive that any amount of inflammation would impair vision. Evolutionarily, vision was so important for survival that our ancestors were more likely to survive if their eyes were left alone by the immune system and so avoided any inflammation-induced damage. But with immune privilege comes immune responsibility; reduced surveillance by leukocytes allows bacteria, viruses and other pathogens easier access to the host so organs must maintain a balance between keeping infection out and preserving tissue function.

In the brain, we used to think that this balance erred on the side of caution – that blood vessels in the brain excluded leukocytes almost completely to prevent inflammation and preserve delicate brain tissue. Indeed, when the brain does experience inflammation, the outcomes are usually bad – diseases such as Parkinson’s and Multiple Sclerosis (MS) are associated with inflammation in the brain. However, it turns out that an immune presence in the brain is not all bad. Clinical trials in the early 2000s aimed to use immunosuppressive drugs to reduce the infiltration of T cells (a type of white blood cell) into the brains of MS patients, thus reducing inflammation and alleviating disease. While symptoms did improve in treated patients, a small number of people died of overwhelming viral replication [1]. The infection was caused by JC virus, a common polyomavirus, which we now know is normally kept in check by T cells continuously patrolling the brain and central nervous system (CNS). So even when inflammation is harmful to the brain, leukocytes still have an important role to play. They are keeping the CNS under active surveillance to suppress infection and keep the brain healthy.

Since the emergence of HIV/AIDS in the 1970s and 80s, we have become aware of many other brain and CNS diseases that rely on robust immunity to stay quiet. Many herpesviruses, for example, live in the CNS and are reactivated following the immunosuppression associated with HIV infection. These viruses are present in the vast majority of people, most of whom will never know they’re infected. It’s only when we lose the protection afforded by patrolling leukocytes that these infections get out of control and cause disease.

So it seems that the brain and CNS are not separated from the immune system altogether. Though migration into these tissues is limited, we rely on sentinel T cells and other leukocytes to keep latent neuronal infections in check. But does it work both ways? Do events in the brain affect the immune system? We all recognise the utility of neural connections with body systems – muscles, gut, skin, eyes. But the immune system? How can cells that move freely and continuously around the body establish connections with the brain? And why would they? There are likely to be many answers to this, but one reason for neural/immune crosstalk is surprisingly intuitive: stress.

Dr Firdaus Dhabhar, Associate Professor at the Stanford Center on Stress and Health, studies the interactions between psychological stress and immune function and has described his work in a recent TED talk. He and others have found that leukocytes respond to short-term psychological stress by changing their migration behaviour. In response to the stress-related hormones epinephrine, norepinephrine and corticosterone, leukocytes leave their usual hangouts (organs like the spleen and lymph nodes) and travel in the blood to sites of potential damage, like the skin. This makes sense when one considers that acute stress can often be followed by injury. The immune system is simply preparing for action. Just as our muscles tense and heart rate increases in case we have to run, immune cells head to the skin in case the source of stress has big teeth and a penchant for human sashimi. Should we survive the attack, we may need to repair the skin and fight infection. In the modern world, this translates to bulk relocation of leukocytes in anticipation of trauma such as surgery. Worrying about surgery the night before going under the knife gives patients a short period of psychological stress, which causes leukocytes to migrate to the skin. This means that they are in the right place to repair damage, so patients whose leukocytes relocate in this way experience a more rapid recovery than those who don’t show any change [2]. Having experienced short-term stress gives immunity a boost and this extends to other immune functions. Mice given a psychological stressor or made to do exercise before vaccination have increased responses to the vaccine [3]. By getting the mediators of vaccine responses (leukocytes) to the site of action before or soon after a shot, we allow these cells increased and/or prolonged contact with the vaccine, thus enhancing the response. This suggests a surprising approach to boosting immunity: a short, sharp shock before a shot may be just the thing to maximise vaccine efficacy.

Just as too much immune activity can harm the brain, excessive triggering of leaukocytes by the brain can harm immunity. In the short-term, once the source of stress is removed, stress hormone levels drop and the immune system returns to normal within a few days. However, long-term stress leads to more lasting changes and can have detrimental effects on immunity. Long-term stress has the combined effect of reducing effective immune responses, while at the same time exposing tissues to inflammation-induced damage. People subject to prolonged periods of stress, including people with depression, PTSD and those caring for family members with dementia, have fewer leukocytes in the blood but increased levels of inflammation-associated factors [4-6]. This seems counter-intuitive since immune cells are a source of inflammation. However, in places like the skin, contact with the outside world leads to constant bombardment with foreign particles. By leaving organs like the lymph nodes, which tightly control exposure to foreign particles, and moving into immunologically noisier tissues, leukocytes become more active but less mobile. These activated cells can still pump inflammatory molecules into the circulation but are no longer able to move around the body, thus making them less likely to find infection when it occurs. Immunologists now think that long-term stress has the paradoxical effect of increasing inflammation while reducing effective immunity because of its tendency to trap crucial immune cells in the periphery.

One path to good immunological health then, is to minimise long-term psychological stress and to sharpen acute stress just before an immune insult. So listen to your mother: relax, eat well and sleep well. And maybe get a friend to terrify you before your next tetanus shot.

References:

[1]        Clifford et al. (2010). Natalizumab-associated progressive multifocal leukoencephalopathy in patients with multiple sclerosis: lessons from 28 cases. Lancet Neurol. 9: 438–446.

[2]        Rosenberger et al. (2009). Surgical stress-induced immune cell redistribution profiles predict short-term and long-term postsurgical recovery. A prospective study. J Bone Joint Surg Am. 91 (12): 2783-94.

[3]        Dhabhar and Viswanathan (2005). Short-term stress experienced at time of immunization induces a long-lasting increase in immunologic memory. Am J Physiol Regul Integr Comp Physiol. 289 (3): R738-44.

[4]        Dhabhar et al. (2009). Low serum IL-10 concentrations and loss of regulatory association between IL-6 and IL-10 in adults with major depression. J Psychiatr Res. 43 (11): 962-9.

[5]        Rawdin et al. (2012). Dysregulated relationship of inflammation and oxidative stress in major depression. Brain Behav Immun. pii: S0889-1591(12)00497-7.

[6]        Aschbacher et al. (2013). Good stress, bad stress and oxidative stress: Insights from anticipatory cortisol reactivity. Psychoneuroendocrinology. pii: S0306-4530(13)00042-5.

Linky and the Brain: May 20, 2013

The past week has been all about maths for me. Well, not all about maths. There was quite a bit of coding (PHP is not my friend) and some experiments (I blocked ALL the acetylcholine receptors).

But special tribute must be paid to all the maths.

First, for those who didn't catch it, Neuro PhD Candidate Kelly Zalocusky posted a fabulous discussion on statistical reliability in neuroscience, reviewing recent work by Stanford Professor Dr. John Ioannidis that highlights the lack of statistical power in many published neuroscience articles. I highly recommend you read Kelly's article (found here). And, if once you're done reading Kelly's post, you have the irresitable urge to calculate the size of n your data needs to be statistically reliable, I recommend the book Power Analysis for Experimental Research: A Pratical Guide for the Biological, Medical and Social Sciences by R. Barker Bauesell and Yu-Fang Li. If you are a Stanford University affiliate, Lane Library has a digital copy (catalogue record here). Last Tuesday, I used the power charts in the t-test section to calculate the correct n I need to have full statistical power, given my pilot data.

From using math to study brains, to studying brains that are doing math. Just in, by a group of researchers at Oxford University - Shocks to the Brain Improve Mathmatical Abilities. This article initialy crossed my internet browser in the form of coverage in Scientific America, as reprinted from Nature. The "shock" in question: transcranial direct current stimulation. The "brain" - the prefrontal cortex. The "math" - arithmetic - "rote memorization of mathematical facts (such as 2 x 17 = 34) and more complicated calculations (for example, 32 – 17 + 5)". The "improvement" - increased response speed - both immediately after stimulation, and, 6 months later, when Oxford students who had received the stimulation were 28% faster than control compatriots. An in depth analysis of the findings/protocols/interpretations of this study would require me to write a longer post, so for the present I'll just link you all to the original article, published in Current Biology. 

And, to round out our maths trilogy, this morning Gizmodo posted two video's featuring a mathematician explaining math jokes. It's funny. Very funny. Cora Ames, I expect you to integrate this concept into an improv segment. (Maths jokes, Explained)

A few other (non-math related) links:

Science Seeker Awards - With special call out both Part 1 and Part II of The Crayolafication of the Brain (Part II won best psych/neuro post)

SfN Careers Youtube Channel highlights alternative career choices - video interviews with Society members whose career paths are not of the traditional academic flavor.

A meta-analysis of the use of literary puns in science article titles. Yeah, we scientists took English Lit in college, too.

Highlight on Neural Prosthetic Systems: NSF IGERT Video Competition

The talented graduate students of Krishna Shenoy's Neural Prosthetic Systems Lab (Dan, Cora, Eric, Niru and Sergey) have made a video for the NSF IGERT Video and Poster Competition. Here is a link to their video: http://posterhall.org/igert2013/posters/432

I think it's fantastic. If you enjoy it as much as I did, consider "voting" for it under the Public Choice tab (and if you're part of IGERT / MBC, you can also log in and vote under the "Community Choice" tab). The Shenoy Lab team would really appreciate it.

The NSF IGERT 2013 Video & Poster Competition

The competition features over 100 presentations each made by a student or a team of students nominated from different IGERT Ph.D. programs. The work presented often transcends traditional disciplinary boundaries and addresses complex research problems of significant scientific and societal importance. While the presenters have been asked to make their posters for a technical/scientific audience, their videos are to be geared toward a more general, non-technical/non-scientific audience. Source: NSF IGERT Competition Website

Competing Gustatory Interests: the author of this post was promised brownies in exchange for blogging the link.

Linguistic Disconnect between the Brain and Emotions

How many love songs have you heard that mention the brain? Heartache and heartbreak are the shorthand for romantic unhappiness, a racing heart indicates excitement, but no English expression I am aware of links the emotions to the brain. On the contrary, language assigns emotions to almost any organ except the one that is actually responsible for them. The heart gets assigned this responsibility most frequently, but common usage also credits the intestines for producing a person’s courage. After all, we say that a courageous person has guts, not that he or she has a powerful prefrontal cortex, which would be the physiologically correct statement. It’s not just English either. Spanish proverbs on courage center on the person’s kidneys, and a Russian, when annoyed, is likely to say, “You have touched my liver.” Why is our language so out of touch with a basic scientific fact? One reason is history. In his treatise “On the Sacred Disease” dated to 400 B.C., Hippocrates (1), the father of modern medicine, states “Men ought to know that from nothing else but the brain come joys, delights, laughter and sports, and sorrows, griefs, despondency, and lamentations.” And the excellent website for the PBS documentary titled The Secret History of the Brain (2) states that Galen, an eminent Roman physician, wrote of the brain as the source of temperament and emotion in 170 B.C. The ancient Greeks and Romans already knew about the role of the brain in causing emotions, but, like so much of the knowledge of antiquity, this was forgotten in the Middle Ages, especially because the Christian Church banned anatomical studies. And, by the way, until real-time imaging of the brain with fMRI came along, anatomical comparisons of the brain across different species provided some of the best evidence for the functions of the different organs. As Paul D. MacLean wrote in his 1967 article “The Brain in Relation to Empathy and Medical Education” in the Journal of Nervous and Mental Disease (3), the similarity of the limbic lobe across all mammals and its absence in reptiles led to the hypothesis that it is involved in emotion. This similarity also permitted anatomical experiments on animals that in conjunction with clinical data on human psychiatric patients provide the bulk of the evidence for the contemporary scientific understanding of emotion. However, all this occurred within the past two centuries, too recently to affect our language. Therefore, the modern European languages came into existence at a time when their speakers were ignorant of the true cause of emotions.

Also, our language reflects our intuition whereas many scientific findings, even basic ones, are counterintuitive. Your heart does race when you’re excited. It is easy to assume that this correlation is causation and much harder to understand the real neurobiology involved. Perhaps, there will one day be a time when there will be as many sayings about the limbic system as there are about the heart now. But for that to happen, the limbic system would need to be thoroughly investigated and explained to all people, so that it is as concrete and tangible as a heartbeat.

Footnotes

1. On the Sacred Disease, Hippocrates - Source
2. The Secret History of the Brain, PBS - Source
3. Maclean (1967). The Brain in Relation to Empathy and Medical Education. Journal of Nervous and Mental Disease, 144 (5): 374-382. Source (warning: paywall)

Why most published neuroscience findings are false

Stanford Professor Dr. John Ioannidis has made some waves over the last few years. His best-known work is a 2005 paper titled "Why most published research findings are false."(1) It turns out that Ioannidis is not one to mince words.

In the May 2013 issue of Nature Reviews Neuroscience, Ioannidis and colleagues specifically tackle the validity of neuroscience studies (2). This recent paper was more graciously titled "Power failure: why small sample size undermines the reliability of neuroscience," but it very easily could have been called "Why most published neuroscience findings are false."

Since these papers outline a pretty damning analysis of statistical reliability in neuroscience (and biomedical research more generally) I thought they were worth a mention here on the Neuroblog.

Ioannidis and colleagues rely on a measure called Positive Predictive Value or PPV, a metric most commonly used to describe medical tests. PPV is the likelihood that, if a test comes back positive, the result in the real world is actually positive. Let's take the case of a throat swab for a strep infection. The doctors take a swipe from the patient's throat, culture it, and the next day come back with results. There are four possibilities.

1. The test comes back negative, and the patient is negative (does not have strep). This is known as a "correct rejection".
2. The test comes back negative, even though the patient is positive (a "miss" or a "false negative").
3. The test comes back positive, even when the patient is negative (a "false alarm" or a "false positive").
4. The test correctly detects that a patient has strep throat (a "hit").

In neuroscience research, we hope that every published "positive" finding reflects an actual relationship in the real world (there are no "false alarms"). We know that this is not completely the case. Not every single study ever published will turn out to be true. But Ioannidis makes the argument that these "false alarms" come up much more frequently than we would like to think.

To calculate PPV, you need three other values:

1. the threshold of significance, or α, usually set at 0.05.
2. the power of the statistical test. If β is the "false negative" rate of a statistical test, power is 1 - β. To give some intuition--if the power of a test is 0.7, and there are 10 studies done that all are testing non-null effects, the test will only uncover 7 of them. The main result in Ionnadis's paper is an analysis of neuroscience meta-analyses published in 2011. He finds the median statistical power of the papers in these studies to be 0.2. More on that later.
3. the pre-study odds, or R. R is the prior on any given relationship tested in the field being non-null. In other words, if you had a hat full of little slips of paper, one for every single experiment conducted in the field, and you drew one out, R is the odds that that experiment is looking for a relationship that exists in the real world.

For those who enjoy bar-napkin calculations--those values fit together like this:

$latex PPV = ([1 - \beta] * R) / ([1 - \beta] * R + \alpha) $

Let's get back to our medical test example for a moment. Say you're working in a population where 1 in 5 people actually has strep (R = 0.25). The power of your medical test (1- β) is 0.8, and you want your threshold for significance to be 0.05. Then the test's PPV is (0.8 * 0.25)/ (0.8 * 0.25 + 0.05) = 0.8. This means that 80% of the times that the test claims the patient has strep, this claim will actually be true. If, however, the power of the test were only 0.2, as Ioannidis claims it is broadly across neuroscience, then the PPV drops to 50%. Fully half of the time, the test's results indicate a false positive.

In a clinical setting, epidemiological results frequently give us a reasonable estimate for R. In neuroscience research, this quantity might be wholly unknowable. But, let's start with the intuition of most graduate students in the trenches (ahem...at the benches?)...which is that 90% of experiments we try don't work. And some days, even that feels optimistic. If this intuition is accurate, then only 10% of relationships tested in neuroscience are non-null in the real world.

Using that value, and Ioannidis's finding that the average power in neuroscience is only 20%, we learn that the PPV of neuroscience research, as a whole, is (drumroll........) 30%.

If our intuitions about our research are true, fellow graduate students, then fully 70% of published positive findings are "false positives". This result furthermore assumes no bias, perfect use of statistics, and a complete lack of "many groups" effect. (The "many groups" effect means that many groups might work on the same question. 19 out of 20 find nothing, and the 1 "lucky" group that finds something actually publishes). Meaning—this estimate is likely to be hugely optimistic.

If we keep 20% power in our studies, but want a 50/50 shot of published findings actually holding true, the pre-study odds (R) would have to be 1 in 5.

To move PPV up to 75%, fully 3 in 4 relationships tested in neuroscience would have to be non-null.

1 in 10 might be pervasive grad-student pessimism, but 3 out of 4 is absolutely not the case.

So—how can we, the researchers, make this better? Well, the power of our analyses depends on the test we use, the effect size we measure, and our sample size. Since the tests and the effect sizes are unlikely to change, the most direct answer is to increase our sample sizes. I did some coffee-shop-napkin calculations from Ioannidis’s data to find that the median effect size in the studies included in his analysis is 0.51 (Cohen’s d). For those unfamiliar with Cohen’s d—standard intuition is that 0.2 is a “small” effect, 0.5 is a “medium” effect, and 0.8 constitutes a “large” effect. For those who are familiar with Cohen’s d…I apologize for saying that.

Assuming that the average effect size in neuroscience studies remains unchanged at 0.51, let’s do some intuition building about sample sizes. For demonstration’s sake, we’ll use the power tables for a 2-tailed t-test.

To get a power of 0.2, with an effect size of 0.51, the sample size needs to be 12 per group. This fits well with my intuition of sample sizes in (behavioral) neuroscience, and might actually be a little generous.

To bump our power up to 0.5, we would need an n of 31 per group.

A power of 0.8 would require 60 per group.

My immediate reaction to these numbers is that they seem huge—especially when every additional data point means an additional animal utilized in research. Ioannidis makes the very clear argument, though, that continuing to conduct low-powered research with little positive predictive value is an even bigger waste. I am happy to take all comers in the comments section, at the Oasis, and/or in a later blog post, but I will not be solving this particular dilemma here.

For those actively in the game, you should know that Nature Publishing Group is working to improve this situation (3). Starting next month, all submitting authors will have to go through a checklist, stating how their sample size was chosen, whether power calculations were done given the estimated effect sizes, and whether the data fit the assumptions of the statistics that are used. On their end, in an effort to increase replicability, NPG will be removing all limits on the length of methods sections. Perhaps other prominent publications would do well to follow suit.

Footnotes

1.  Ioannidis JPA (2005) Why Most Published Research Findings Are False. PLoS Med 2(8): e124. doi:10.1371/journal.pmed.0020124

2. Button et al (2013). Power Failure: why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience 14: 3665-376. doi:10.1038/nrn3475

3. The specific announcement detailing changes in submission guidelines, also the Nature Special on Challenges in Irreproducible Research