Counting NAs by column in R

Are you starting your data exploration? Do you want to have an easy overview of your variable NA percentage?

We create a function to benchmark different ways of achieving it:

library(microbenchmark)
library(tidyverse)

benchmark_count_na_by_column <- function(dataset){
  microbenchmark(
    # Summary table output
    dataset %>% summary(),
    # Numeric output
    colSums(is.na(dataset)),
    sapply(dataset, function(x) sum(is.na(x))),
    # List output
    dataset %>% map(~sum(is.na(.))),
    lapply( dataset, function(x) sum(is.na(x))),
    # Df output
    dataset %>%
      select(everything()) %>% 
      summarise_all(funs(sum(is.na(.)))),
    dataset %>% 
      summarise_each(funs(sum(is.na(.)))),
    # Tibble output
    dataset %>% map_df(~sum(is.na(.)))
  )
}

See the performance dealing with small datasets:

print(airquality %>% nrow()) # 153 rows

benchmark_count_na_by_column(airquality)

## Unit: microseconds
##funct             min       lq     mean     median      uq    max     neval  class
##summary()         1480.5   1582.60 1979.676 1897.30  2100.45 6403.2   100    table
##colSums()         24.4     38.45   47.854   44.70   53.90    152.4    100    integer
##sapply()          23.2     35.05   67.891   39.65   50.30    2494.8   100    integer
##map()             140.2    182.60  214.092  200.75  238.50   549.6    100    list
##lapply()          11.2     15.65   27.093   18.85   22.45    750.1    100    list
##summarise_all()   1996.9   2147.80 2650.223 2382.90 2798.55  8133.7   100    data.frame
##summarise_each()  2277.9   2497.05 2951.477 2898.40 3080.65  7977.2   100    data.frame
##map_df()          190.0    249.00  331.368  275.40  326.05   383      100    tbl_df

Let’s see how well them scale with 100000 rows dataset:

big_dataset %>% nrow() # 100 000 rows

benchmark_count_na_by_column(big_dataset)

## Unit: milliseconds
##funct             min        lq       mean      median        uq    max    neval class
##summary()         113.7535 129.35070 138.716624 133.14050 143.45920 252.0149 100   table
##colSums()         4.4280   5.31080  12.502741   5.65005  18.77570 124.8206   100   integer
##sapply()          2.2452   3.03095   6.788395   3.15310  15.04010  18.6061   100   integer
##map()             2.5950   3.28390   5.760602   3.38020   3.69445  19.4527   100   list
##lapply()          2.2018   2.95700   6.219106   3.03605   3.62860  19.5514   100   list
##summarise_all()   5.0982   5.85135  10.093431   6.05940   6.87070 127.5107   100   data.frame
##summarise_each()  5.7251   6.16980  10.191426   6.33065   6.72210 125.2943   100   data.frame
##map_df()          2.6913   3.42045   7.694863   3.56720   3.89715 122.2030   100   tbl_df