Reproducible research

# Reproducible research
## What are we doing and how can we improve?
### Claus Thorn Ekstrøm<br>UCPH Biostatistics<br>.small[<a href="mailto:ekstrom@sund.ku.dk">ekstrom@sund.ku.dk</a>]
### May 16th 2017<br>.small[Slides @ <a href="www.biostatistics.dk/talks/">biostatistics.dk/talks/</a>]

---

???

First slide: Reproducibility is key to science. If science is the best method that we have of figuring out how nature works, if our hypotheses and theories are to have any basis in reality, then the observations upon which those hypotheses and theories are based must be reproducible.

Reproducibility helps demonstrates a key difference between science and pseudoscience.

2011 was a year of horror for **social psychologists**.

Journal of personality and social psychologists. People can look into the future. Especially extrovert women confronted with erotic pictures. Rejected others - negative findings.

**Diederik Stabel** - falsified, students could not see data

---

???

Replication results from social sciences show that

Publication bias.

Chance findings.

---

background-image: url(pics/lancet.png)
background-size: auto 100%
class: middle, center

---

background-image: url(pics/penkowa.png)
background-size: 100% auto
class: middle, center

# Hansen and Penkowa

???

Why reproducibility is important

---

background-image: url(pics/soccer.png)
background-size: 100% auto
class: middle, center

---

# The research process

<div id="htmlwidget-e44379a9eaec74b5387b" style="width:100%;height:50%;" class="grViz html-widget"></div>
<script type="application/json" data-for="htmlwidget-e44379a9eaec74b5387b">{"x":{"diagram":"\ndigraph dot {\n\ngraph [layout = dot,\n       rankdir = LR,\n       bgcolor=\"#222222\",\n       size=2]\n\nnode [shape = circle,\n      style = filled,\n      fillcolor = grey,\n      fontcolor = White,\n      fontsize=20,\n      fontname=Helvetica,\n      label = \"\", \n      penwidth=4, \n      margin=0.05,\n      color=White]\na [label=\"Design\"]\nb [label=\"Collect\"]\nc [label=\"Analyze\"]\nd [label=\"Report\"]\ne [label=\"Publish\"]\n\nedge [color = White, penwidth=4]\na -> b -> c -> d\nd -> e [color=red, penwidth=4]\n}","config":{"engine":"dot","options":null}},"evals":[],"jsHooks":[]}</script>

???

The red line shows where peer review comes in. Total summary. We need to document the steps we did througout as shown in the soccer example.

---

# Statistical analysis example

> **Statistical analysis**
>
> .large[All of the data were analyzed with data processing software and figures with Microsoft excel 2007.]
>
> .pull-right[-- Tayefe *et al*, Advances in Bioresearch, 2014]

---

background-image: url(pics/term.png)
background-size: 100% auto
class: middle, center

---

# Terminology

**Reproducibility**

Given code/data/materials, can I get *the same* (=identical) numbers that you did?

**Replicability**

Given scientific protocol, can I get *the same* (=in agreement) result that you did in my own study?

---

---

# Nerd quiz

## 42

---

class: middle, center
background-image: url(pics/claus.png)
background-size: auto 100%

???

My most annoying collaborator.

He has done a lot of previous work that I would like to use.

PhD, 36000 lines of code. it's working ... but

He does not answer back when I email him!

---

# Reproducibility

## Tools

Already there

## Desire

---

# The public may not understand scientists ...

---

???

NEJM - Research parasites

---

???

EU parliament and European patent applications

---

# Poential problems

Incentives for individual success are focused on getting it **published**, not getting it **right**.

Publications favour novel results and positive results

No funding

"Research parasites"

---

# Replicability

## Tools

## Desire

Great!

---

# Workshop - part I

*  Groups of 2-3 people.
*  Spend 3 minutes each explaining who you are/what you do
*  Goal 1: we want to hear **good** stories about what we actually do 
*  Goal 2: where do you see **potential problems**, where we could do better.

<div id="htmlwidget-fc1f197fb7732f4f9ed4" style="width:100%;height:50%;" class="grViz html-widget"></div>
<script type="application/json" data-for="htmlwidget-fc1f197fb7732f4f9ed4">{"x":{"diagram":"\ndigraph dot {\n\ngraph [layout = dot,\n       rankdir = LR,\n       bgcolor=\"#901A1E\",\n       size=2]\n\nnode [shape = circle,\n      style = filled,\n      fillcolor = grey,\n      fontcolor = White,\n      fontsize=20,\n      fontname=Helvetica,\n      label = \"\", \n      penwidth=4, \n      margin=0.05,\n      color=White]\na [label=\"Design\"]\nb [label=\"Collect\"]\nc [label=\"Analyze\"]\nd [label=\"Report\"]\ne [label=\"Publish\"]\n\nedge [color = White, penwidth=4]\na -> b -> c -> d -> e\n}","config":{"engine":"dot","options":null}},"evals":[],"jsHooks":[]}</script>

---

# Tools for reproducibility

<div id="htmlwidget-5b1e6abe2d6fb6d6e63c" style="width:100%;height:50%;" class="grViz html-widget"></div>
<script type="application/json" data-for="htmlwidget-5b1e6abe2d6fb6d6e63c">{"x":{"diagram":"\ndigraph dot {\n\ngraph [layout = dot,\n       rankdir = LR,\n       bgcolor=\"#222222\",\n       size=2]\n\nnode [shape = circle,\n      style = filled,\n      fillcolor = grey,\n      fontcolor = White,\n      fontsize=20,\n      fontname=Helvetica,\n      label = \"\", \n      penwidth=4, \n      margin=0.05,\n      color=White]\na [label=\"Design\"]\nb [label=\"Collect\"]\nc [label=\"Analyze\"]\nd [label=\"Report\"]\ne [label=\"Publish\"]\n\nedge [color = White, penwidth=4]\na -> b -> c -> d -> e\n}","config":{"engine":"dot","options":null}},"evals":[],"jsHooks":[]}</script>

R - markdown
gitHub
Docker
Jupyter Notebooks
OSF

Dropbox, Google Drive, Amazon web services, evernote,

---

# R markdown

---

# R GitHub

---
class: center

# Jupyter Notebooks

---

# Open Science Framework

---

# Teaching

What are we doing - through supervision and courses - for

*  Ph.D.s and post docs? 
*  Bachelor and master students?

---

# Undergraduate courses

---

# Ph.D.

---

# Ph.D.

---

# RCR

RCR – A Danish textbook for courses in Responsible Conduct of Research

(See Chapter 5, in particular p. 54, Box 1, p. 63)

---

# Workshop - part II

*  Groups of 2-3 people.
*  Spend 2 minutes each explaining your role (student, teacher, courses, ...) and what you are involved in
*  Goal: Which curricula, courses, training programs should we focus on?
*  Goal 2: What should/could we change?

Small changes are better than no changes!

---

???

Reprodicible research is like brushing you teeth: slightly annoying, takes some time, but it's good for you and we just need to get into the habit