What is the tidyverse?

The tidyverse is an opinionated collection of R packages designed for data science. All packages share an underlying design philosophy, grammar, and data structures (tidyverse.org).

Packages and functions

Working with packages and functions

  1. Installing a package (once):
install.packages("dplyr")
  1. Loading a package (every time you want to use it):
library(dplyr)
  1. Now you can use functions from this package! (every time you want to use it)
dplyr::rename(iris, petal_length = Petal.Length)
    Sepal.Length Sepal.Width petal_length Petal.Width    Species
1            5.1         3.5          1.4         0.2     setosa
2            4.9         3.0          1.4         0.2     setosa
3            4.7         3.2          1.3         0.2     setosa
4            4.6         3.1          1.5         0.2     setosa
5            5.0         3.6          1.4         0.2     setosa
6            5.4         3.9          1.7         0.4     setosa
7            4.6         3.4          1.4         0.3     setosa
8            5.0         3.4          1.5         0.2     setosa
9            4.4         2.9          1.4         0.2     setosa
10           4.9         3.1          1.5         0.1     setosa
11           5.4         3.7          1.5         0.2     setosa
12           4.8         3.4          1.6         0.2     setosa
13           4.8         3.0          1.4         0.1     setosa
14           4.3         3.0          1.1         0.1     setosa
15           5.8         4.0          1.2         0.2     setosa
16           5.7         4.4          1.5         0.4     setosa
17           5.4         3.9          1.3         0.4     setosa
18           5.1         3.5          1.4         0.3     setosa
19           5.7         3.8          1.7         0.3     setosa
20           5.1         3.8          1.5         0.3     setosa
21           5.4         3.4          1.7         0.2     setosa
22           5.1         3.7          1.5         0.4     setosa
23           4.6         3.6          1.0         0.2     setosa
24           5.1         3.3          1.7         0.5     setosa
25           4.8         3.4          1.9         0.2     setosa
26           5.0         3.0          1.6         0.2     setosa
27           5.0         3.4          1.6         0.4     setosa
28           5.2         3.5          1.5         0.2     setosa
29           5.2         3.4          1.4         0.2     setosa
30           4.7         3.2          1.6         0.2     setosa
31           4.8         3.1          1.6         0.2     setosa
32           5.4         3.4          1.5         0.4     setosa
33           5.2         4.1          1.5         0.1     setosa
34           5.5         4.2          1.4         0.2     setosa
35           4.9         3.1          1.5         0.2     setosa
36           5.0         3.2          1.2         0.2     setosa
37           5.5         3.5          1.3         0.2     setosa
38           4.9         3.6          1.4         0.1     setosa
39           4.4         3.0          1.3         0.2     setosa
40           5.1         3.4          1.5         0.2     setosa
41           5.0         3.5          1.3         0.3     setosa
42           4.5         2.3          1.3         0.3     setosa
43           4.4         3.2          1.3         0.2     setosa
44           5.0         3.5          1.6         0.6     setosa
45           5.1         3.8          1.9         0.4     setosa
46           4.8         3.0          1.4         0.3     setosa
47           5.1         3.8          1.6         0.2     setosa
48           4.6         3.2          1.4         0.2     setosa
49           5.3         3.7          1.5         0.2     setosa
50           5.0         3.3          1.4         0.2     setosa
51           7.0         3.2          4.7         1.4 versicolor
52           6.4         3.2          4.5         1.5 versicolor
53           6.9         3.1          4.9         1.5 versicolor
54           5.5         2.3          4.0         1.3 versicolor
55           6.5         2.8          4.6         1.5 versicolor
56           5.7         2.8          4.5         1.3 versicolor
57           6.3         3.3          4.7         1.6 versicolor
58           4.9         2.4          3.3         1.0 versicolor
59           6.6         2.9          4.6         1.3 versicolor
60           5.2         2.7          3.9         1.4 versicolor
61           5.0         2.0          3.5         1.0 versicolor
62           5.9         3.0          4.2         1.5 versicolor
63           6.0         2.2          4.0         1.0 versicolor
64           6.1         2.9          4.7         1.4 versicolor
65           5.6         2.9          3.6         1.3 versicolor
66           6.7         3.1          4.4         1.4 versicolor
67           5.6         3.0          4.5         1.5 versicolor
68           5.8         2.7          4.1         1.0 versicolor
69           6.2         2.2          4.5         1.5 versicolor
70           5.6         2.5          3.9         1.1 versicolor
71           5.9         3.2          4.8         1.8 versicolor
72           6.1         2.8          4.0         1.3 versicolor
73           6.3         2.5          4.9         1.5 versicolor
74           6.1         2.8          4.7         1.2 versicolor
75           6.4         2.9          4.3         1.3 versicolor
76           6.6         3.0          4.4         1.4 versicolor
77           6.8         2.8          4.8         1.4 versicolor
78           6.7         3.0          5.0         1.7 versicolor
79           6.0         2.9          4.5         1.5 versicolor
80           5.7         2.6          3.5         1.0 versicolor
81           5.5         2.4          3.8         1.1 versicolor
82           5.5         2.4          3.7         1.0 versicolor
83           5.8         2.7          3.9         1.2 versicolor
84           6.0         2.7          5.1         1.6 versicolor
85           5.4         3.0          4.5         1.5 versicolor
86           6.0         3.4          4.5         1.6 versicolor
87           6.7         3.1          4.7         1.5 versicolor
88           6.3         2.3          4.4         1.3 versicolor
89           5.6         3.0          4.1         1.3 versicolor
90           5.5         2.5          4.0         1.3 versicolor
91           5.5         2.6          4.4         1.2 versicolor
92           6.1         3.0          4.6         1.4 versicolor
93           5.8         2.6          4.0         1.2 versicolor
94           5.0         2.3          3.3         1.0 versicolor
95           5.6         2.7          4.2         1.3 versicolor
96           5.7         3.0          4.2         1.2 versicolor
97           5.7         2.9          4.2         1.3 versicolor
98           6.2         2.9          4.3         1.3 versicolor
99           5.1         2.5          3.0         1.1 versicolor
100          5.7         2.8          4.1         1.3 versicolor
101          6.3         3.3          6.0         2.5  virginica
102          5.8         2.7          5.1         1.9  virginica
103          7.1         3.0          5.9         2.1  virginica
104          6.3         2.9          5.6         1.8  virginica
105          6.5         3.0          5.8         2.2  virginica
106          7.6         3.0          6.6         2.1  virginica
107          4.9         2.5          4.5         1.7  virginica
108          7.3         2.9          6.3         1.8  virginica
109          6.7         2.5          5.8         1.8  virginica
110          7.2         3.6          6.1         2.5  virginica
111          6.5         3.2          5.1         2.0  virginica
112          6.4         2.7          5.3         1.9  virginica
113          6.8         3.0          5.5         2.1  virginica
114          5.7         2.5          5.0         2.0  virginica
115          5.8         2.8          5.1         2.4  virginica
116          6.4         3.2          5.3         2.3  virginica
117          6.5         3.0          5.5         1.8  virginica
118          7.7         3.8          6.7         2.2  virginica
119          7.7         2.6          6.9         2.3  virginica
120          6.0         2.2          5.0         1.5  virginica
121          6.9         3.2          5.7         2.3  virginica
122          5.6         2.8          4.9         2.0  virginica
123          7.7         2.8          6.7         2.0  virginica
124          6.3         2.7          4.9         1.8  virginica
125          6.7         3.3          5.7         2.1  virginica
126          7.2         3.2          6.0         1.8  virginica
127          6.2         2.8          4.8         1.8  virginica
128          6.1         3.0          4.9         1.8  virginica
129          6.4         2.8          5.6         2.1  virginica
130          7.2         3.0          5.8         1.6  virginica
131          7.4         2.8          6.1         1.9  virginica
132          7.9         3.8          6.4         2.0  virginica
133          6.4         2.8          5.6         2.2  virginica
134          6.3         2.8          5.1         1.5  virginica
135          6.1         2.6          5.6         1.4  virginica
136          7.7         3.0          6.1         2.3  virginica
137          6.3         3.4          5.6         2.4  virginica
138          6.4         3.1          5.5         1.8  virginica
139          6.0         3.0          4.8         1.8  virginica
140          6.9         3.1          5.4         2.1  virginica
141          6.7         3.1          5.6         2.4  virginica
142          6.9         3.1          5.1         2.3  virginica
143          5.8         2.7          5.1         1.9  virginica
144          6.8         3.2          5.9         2.3  virginica
145          6.7         3.3          5.7         2.5  virginica
146          6.7         3.0          5.2         2.3  virginica
147          6.3         2.5          5.0         1.9  virginica
148          6.5         3.0          5.2         2.0  virginica
149          6.2         3.4          5.4         2.3  virginica
150          5.9         3.0          5.1         1.8  virginica

Go to Exercise 0 in datascience_exercises.Rmd

The Data Science workflow

Source: R 4 Data Science

tidyverse core packages

Afternoon workflow

  1. Read a file into R
  2. Clean the data:
    • Filter on relevant rows
    • Select only relevant columns
  3. Calculate a new column
  4. Rename a column
  5. Put the dataframe into a long (tidy) format
  6. Summarize the dataframe
  7. Visualize the dataframe
  8. Write the summarized dataframe to a file

Side Trip:
We’re Going To Antarctica…

Data: palmerpenguins

  • Observations about penguins from the Palmer Archipelago (Antarctica)
  • Package created by Allison Horst for worldwide use
  • species, island, culmen length and depth, flipper length, etc.

Importing Data

Reading data into R

R projects and working directories

When you start programming for yourself:

  • Create a folder dedicated to your project
  • Start a new R project: File > New Project > Existing Directory
  • An .RProj file will be created

Advantages:

  • Automatically set your working directory to that folder
  • Automatically retrieve only the history and objects from that R project
  • More reproducible (relative vs. absolute paths)
getwd()

readr: Read Rectangular Text Data

To read text data, you need:

  • location of the data (path)
  • delimiter (value separator) of the data, e.g. ,, ; or \t
  • function to use to read data into R
Read from file Write to file Value separator
read_delim() write_delim() catch-all: user needs to specify
read_csv() write_csv() comma separated
read_csv2() write_csv2() semicolon separated
read_tsv() write_tsv() tab separated

Reading non-text data file

For non-text data files, there are other R packages:

  • readxl: Excel files
  • haven: SPSS & STATA files
  • googlesheets4: Google Sheets
  • rvest: HTML files

Example: reading data

Read a file located in the same folder as your script:

raw_tsv_data <- read_tsv("my_raw_data.tsv")

Read a file located in a different folder:

raw_csv_data <- read_csv("data/raw/my_raw_data.csv")

Or from the web:

# Source: https://catalog.data.gov/dataset/electric-vehicle-population-data
electric_vehicles <- read_csv("https://data.wa.gov/api/views/f6w7-q2d2/rows.csv")

Bonus: use an in-built dataset from R

data(iris)        # Observational data on iris flowers
data(mtcars)      # Motor Trend Car Road Tests

data()            # Check all the available in-built datasets!

Go to Exercise 1, 2 and 3

Answers Exercise 1, 2 and 3

  1. Reading the penguins dataset into R.
penguins <- read_tsv("data/penguins.tsv")
  1. Reading penguins_isotopes.xlsx into R.
penguins_isotopes <- read_excel(path = "data/penguins_isotopes.xlsx")
  1. Writing penguins_isotopes to a csv file.
write_csv(penguins_isotopes,
          file = "data/penguins_isotopes.csv")

Selecting & filtering data

dplyr: Data Manipulation

dplyr contains functions for many types of data manipulation, such as:

  • filter(): select rows that meet one or several logical criteria
  • select(): select (or drop) columns
  • rename(): change column name
  • mutate(): transform column values or create new column
  • group_by(): group data on one or more columns
  • summarize(): reduces a group of data into a single row

Filter

Selects rows in your dataframe.

Use:

filter(your-dataframe, your-condition)

From the morning session: “From your dataframe df, return complete rows for everyone living in a country of your choice.”

df[df$country=="UK", ]        # Base R
filter(df, country == "UK")   # Tidyverse
  name age country
1  Ann  35      UK
2  Dan  51      UK

Select

Select or drop columns in your dataframe.

Basic use:

select(your-dataframe, col1, col2)  # select col1 and col2
select(your-dataframe, -col3)       # select all but col3

Use special selecting functions

select(your-dataframe, contains("col"))    # select cols containing "col"

From the morning session: “Return the columns name and age together.”

df[, c("name","age")]       # Base R
select(df, name, age)       # Tidyverse
   name age
1   Ann  35
2   Bob  22
3 Chloe  50
4   Dan  51

Go to exercises 4 and 5

Answers exercise 4

  1. Filter penguins to leave out the NAs.
penguins_subset <- filter(penguins, !is.na(Sex))

Or

penguins_subset <- filter(penguins, Sex == "MALE" | Sex == "FEMALE")

Or

penguins_subset <- filter(penguins, Sex %in% c("MALE", "FEMALE"))

Answers exercise 5

  1. Select the columns Individual_ID, Species, Sex, Island, Culmen_Depth_mm and Culmen_Length_mm
penguins_subset_2 <- select(penguins_subset, Individual_ID, Species,
                            Sex, Island, 
                            Culmen_Depth_mm, Culmen_Length_mm)

Or

penguins_subset_2 <- select(penguins_subset, Individual_ID, Species,
                            Sex, Island, 
                            starts_with("Culmen"))

Or

penguins_subset_2 <- select(penguins_subset, Individual_ID, Species,
                            Sex, Island, 
                            contains("Culmen"))

Mutate

Transform column values, or create new column.

Basic use:

mutate(your-dataframe, column_name = an_operation)

Add a new column:

df$old_age <- df$age + 20           # Base R
df <- mutate(df, old_age = age + 20) # Tidyverse

Rename

Renaming columns.

Basic use:

rename(your-dataframe, new_column_name = old_column_name)

Rename a column:

df$Old_Age <- df$old_age          # Base R
df <- rename(df, Old_Age = old_age) # Tidyverse
df
   name age country Old_Age
1   Ann  35      UK      55
2   Bob  22     USA      42
3 Chloe  50     USA      70
4   Dan  51      UK      71

Go to Exercise 6 and 7

Answers exercises 6 and 7

  1. Use mutate() to create a new column culmen_ratio.
penguins_subset_3 <- mutate(penguins_subset_2,
                            culmen_ratio = Culmen_Length_mm / Culmen_Depth_mm)
  1. Rename the columns Culmen_Length_mm and Culmen_Depth_mm.
penguins_subset_4 <- rename(penguins_subset_3,
                            length = Culmen_Length_mm, 
                            depth = Culmen_Depth_mm)

Piping Operations

magrittr: The Forward-Pipe Operator %>%

A key tidyverse component that chains all data science steps together:

%>%1

Why?

  • create an easily readable pipeline of chained commands
  • no nested function calls
  • no need to save intermediate R objects with <-
  • easily add and/or delete steps in your pipeline without breaking the code

Pipe operator: how it works

# Name the object that you use as initial input
my_new_df <- df %>%
  
  # Perform a function using that object as input
  filter(country == "UK") %>%
  
  # Add another operation
  select(name, age) %>%
  
  # And another, etc.
  mutate(old_age = age + 20)

Note:

  • df is only mentioned once at the beginning
  • final line of the operation does not get a %>%

Go to exercise 8

Answers exercise 8

Make a workflow that starts with the data penguins and subsequently applies your filter, select, mutate and rename operations.

penguins_subset_5 <- penguins %>%
  
  # Filter out NAs
  filter(!is.na(Sex)) %>%
  
  # Select only relevant columns
  select(Individual_ID, Species, Sex, Island, starts_with("Culmen")) %>%
  
  # Add a new columns culmen_ratio
  mutate(culmen_ratio =  Culmen_Length_mm / Culmen_Depth_mm) %>%
  
  # Rename Culmen measurement columns
  rename(length = Culmen_Length_mm,
         depth = Culmen_Depth_mm)

Tidy Data

Tidy Data

Tidy data sets are all alike; but every messy data set is messy in its own way (Wickham/Grolemund, 2017)

Tidy Data Principles: principles for structuring tabular data sets:

  1. Each variable must have its own column.
  2. Each observation must have its own row.
  3. Each value must have its own cell.

Our df - but extended

   name age country mood_wk1 mood_wk2
1   Ann  35      UK        4        2
2   Bob  22     USA        3        3
3 Chloe  50     USA        4        5
4   Dan  51      UK        2        4

Wide or long?

Wide! Why is this not tidy?

  • Values in column names
  • Multiple observations per row

Wide vs Long

Wide

   name age country mood_wk1 mood_wk2
1   Ann  35      UK        4        2
2   Bob  22     USA        3        3
3 Chloe  50     USA        4        5
4   Dan  51      UK        2        4
  • Values in column names
  • Multiple observations per row: all observations on 1 individual in 1 row
  • Not tidy

Long

# A tibble: 4 × 5
  name    age country  week  mood
  <chr> <dbl> <chr>   <dbl> <dbl>
1 Ann      35 UK          1     4
2 Ann      35 UK          2     2
3 Bob      22 USA         1     3
4 Bob      22 USA         2     3
  • Single observation (weight, mood) in a single row
  • No values in column names
  • Tidy!

Tidy data is a consistent way of storing data + most R functions work with vectors of values (columns). Tidyverse packages are designed to work with tidy data (dplyr, ggplot2, etc.)

tidyr: Tidy Messy Data

Do It Yourself:

  • pivot_longer(): lengthen data: more rows, fewer columns (long format, tidy)
  • pivot_wider(): widen data: fewer rows, more columns (wide format)

Basic use:

pivot_longer(your-dataframe, cols = c(col_to_pivot1, col_to_pivot2, etc),
            names_to = "name_of_measurement",
            values_to = "values_of_measurement")

Check ?pivot_longer() and Google! for examples and other function arguments.

In our example:

pivot_longer(df_ext, 
             cols = starts_with("mood_"),
             names_to = "week",
             values_to = "mood")

Go to Exercise 9

Source: Allison Hill

Answer exercise 9

Transform the dataframe from wide to long format using the function pivot_longer().

penguins_long <- penguins_subset_5 %>%
  pivot_longer(cols = c(length, depth),
               names_to = "culmen_element",
               values_to = "measurement")

Summarizing and combining data

group_by() and summarize()

Group by one or more columns and perform some summarizing operation.

Basic use:

group_by(your-dataframe, column1_to_group_by, column2_to_group_by, [etc])
summarize(your-dataframe, new_column = <some_summarizing_operation>)

Example 1:

df_ext %>% 
    group_by(country) %>% 
    summarize(count = n())
# A tibble: 2 × 2
  country count
  <chr>   <int>
1 UK          2
2 USA         2

Example 2

df_long %>% 
    group_by(week) %>% 
    summarize(avg_mood = mean(mood))
# A tibble: 2 × 2
   week avg_mood
  <dbl>    <dbl>
1     1     3.25
2     2     3.5 

Combining data (joins)

Read more about joins in R for Data Science.

Go to Exercises 10 and 11

Answers Exercises 10 and 11

  1. Use group_by() and summarize() to calculate the mean and standard deviation of all measurements, grouped by species and the type of measurement.
penguins_summary <- penguins_long %>%
  group_by(Species, culmen_element) %>%
  summarize(avg = mean(measurement),
            sd = sd(measurement))
  1. Merge penguins_summary and penguins_long.
penguins_join <- full_join(penguins_summary, penguins_long,
                           by = c("Species", "culmen_element"))

Data Visualization

ggplot2: Elegant Data Visualisations

  • ggplot2 is Hadley Wickham’s reimplementation of The Grammar of Graphics (Leland Wilkinson, 2005).
  • MANY functions to generate MANY graphs
  • Graphs built up from multiple layers

Our first plots - data

Remember df_ext?

df_ext
   name age country mood_wk1 mood_wk2
1   Ann  35      UK        4        2
2   Bob  22     USA        3        3
3 Chloe  50     USA        4        5
4   Dan  51      UK        2        4

And df_long?

df_long
# A tibble: 8 × 5
  name    age country  week  mood
  <chr> <dbl> <chr>   <dbl> <dbl>
1 Ann      35 UK          1     4
2 Ann      35 UK          2     2
3 Bob      22 USA         1     3
4 Bob      22 USA         2     3
5 Chloe    50 USA         1     4
6 Chloe    50 USA         2     5
7 Dan      51 UK          1     2
8 Dan      51 UK          2     4

Our first plot

First layer: the data

ggplot(df_ext)

Our first plot

Second layer: aesthetics

ggplot(df_ext) +
  aes(x = country)

Our first plot

Third layer: geom

ggplot(df_ext) +
  aes(x = country) +
  geom_bar()

Our first plot

Another layer: labels

ggplot(df_ext) +
  aes(x = country) +
  geom_bar() +
  labs(title="Countries in `df_ext`", 
       x ="Country", 
       y = "Count")

Our second plot

Turn week into character (to prevent ugly plotting) + plot data.

df_long %>%
  mutate(week = as.character(week)) %>%
  ggplot()

Our second plot

Add aesthetics.

df_long %>%
  mutate(week = as.character(week)) %>%
  ggplot() +
  aes(x = week, y = mood)

Our second plot

Add a geom.

df_long %>%
  mutate(week = as.character(week)) %>%
  ggplot() +
  aes(x = week, y = mood) + 
  geom_boxplot() 

Our second plot

Create a separate plot per country.

df_long %>%
  mutate(week = as.character(week)) %>%
  ggplot() +
  aes(x = week, y = mood) + 
  geom_boxplot() +
  facet_wrap(~ country)

Our second plot

Use a nicer theme.

df_long %>%
  mutate(week = as.character(week)) %>%
  ggplot() +
  aes(x = week, y = mood) + 
  geom_boxplot() +
  facet_wrap(~ country) +
  theme_minimal()

Our second plot

Larger text

df_long %>%
  mutate(week = as.character(week)) %>%
  ggplot() +
  aes(x = week, y = mood) + 
  geom_boxplot() +
  facet_wrap(~ country) +
  theme_minimal() +
  theme(text = element_text(size = 25))

Go to Exercise 12 - 13

Tip: Choose a visualization -> Get example code: https://www.data-to-viz.com/

Answers to Exercise 12

A scatterplot of Culmen_Length_mm against Flipper_Length_mm per Island.

ggplot(penguins, aes(x = Culmen_Length_mm, 
                     y = Flipper_Length_mm, 
                     color = Species)) + 
  geom_point() + 
  labs(x = "Culmen length (mm)",
       y = "Flipper length (mm)",
       title = "Culmen and Flipper length by Species") +
  theme_classic() +
  facet_wrap( ~ Island) + 
  geom_smooth(method = "lm")

Wrap-up

What have we learned this afternoon?

  • Packages vs. functions
  • Reading data into R: read_csv(), read_tsv(), read_excel()
  • Writing R objects to a csv file: write_csv
  • Wrangling data with filter(), select(), mutate(), rename()
  • Chaining operations together with %>%
  • Creating tidy data
  • Summarizing and combining datasets
  • Using ggplot2 to make some basic plots

Learn more

See also the What’s next page.

Where to find us

Thank you!

useR <- function(){
  print("Good luck and see you!")
}

useR()
[1] "Good luck and see you!"