The life of a data scientist

Data scientists, according to interviews and expert estimates, spend from 50 percent to 80 percent of their time mired in this more mundane labor of collecting and preparing unruly digital data, before it can be explored for useful nuggets.

-- "For Big-Data Scientists, 'Janitor Work' Is Key Hurdle to Insight" - The New York Times, 2014

Beyond the hype

Many of the existing problems with small data are also applicable to big data. ...

The problems do not disappear because the data sizes becomes larger. They become worse.

State of the art?

Statistical analysis

All of the data were analyzed with data processing software and figures with Microsoft excel 2007.

-- Tayefe et al, Advances in Bioresearch, 2014

Reproducible research

What didn't we check?

Reproducible research

What didn't we check?

• Many studies cannot be replicated: time, money, unique
• New technologies increase data sizes
• Merge existing databases into megadatabases
• Genetic data for future analyses

Reproducible research

What didn't we check?

• Many studies cannot be replicated: time, money, unique
• New technologies increase data sizes
• Merge existing databases into megadatabases
• Genetic data for future analyses

You are your worst collaborator.

Using dataMaid interactively

check(toyData$var2)  # Individual check
check(toyData$var2, numericChecks = "identifyMissing")
visualize(toyData$var2)
summarize(toyData$var2)
summarize(toyData$var2, 
       numericSummaries = c("centralValue", "minMax"))

Extending dataMaid

isID <- function(v, nMax = NULL, ...) {
  out <- list(problem = FALSE, message = "")
  if (class(v) %in% setdiff(allClasses(), 
                            c("logical", "Date"))) {
    v <- as.character(v)
    lengths <- c(nchar(v))
    if (all(lengths > 10) & 
        length(unique(lengths)) == 1) {
      out$problem <- TRUE
      out$message <- "Warning: Seems to contain IDs."
    }  }
  out }

Adding the function

data("exampleData")
exampleData$names <- sapply(1:300, 
  function(i) { paste0(sample(LETTERS, size=10), 
                       collapse="") })
clean(exampleData,
  preChecks = c("isID"))