---
title: "Wrangling Verbs (Notes)"
format:
  html:
    toc: true
    toc-depth: 2
    embed-resources: true
---

```{r include = FALSE}
# This chunk just sets up some styling (eg: default size of our images)
knitr::opts_chunk$set(
  collapse = TRUE, 
  warning = FALSE,
  message = FALSE,
  fig.height = 2.75, 
  fig.width = 4.25,
  fig.align = 'center')
```

<!-- NOTE: You'll have to complete Exercise 1 before being able to render this document. -->



::: {.callout-note title="Learning goals"}

Explore the following wrangling verbs: `select`, `mutate`, `filter`, `arrange`, `summarize`, `group_by`

:::





\
\
\
\


::: {.callout-note title="Additional resources"}

Watch:

-   [Intro to dplyr](https://www.youtube.com/watch?v=dqP0DejI7lw) (Lisa Lendway)
-   [Demonstration of dplyr](https://www.youtube.com/watch?v=PgZvq7ZPvEE&feature=youtu.be) (Lisa Lendway)

Read:

-   [Data Transformation](https://r4ds.hadley.nz/data-transform.html) (Wickham, Çetinkaya-Rundel, & Grolemund)
-   [A Grammar for Data Wrangling](https://mdsr-book.github.io/mdsr2e/ch-dataI.html) (Baumer, Kaplan, and Horton)

:::




\
\
\
\

## Unit 2 motivation

Recall the elections data by U.S. county:

```{r}
# Load tidyverse & data
library(tidyverse)
elections <- read.csv("https://mac-stat.github.io/data/election_2020_county.csv")
```

We've used data viz to explore some general patterns in the election outcomes. For example, a map!

```{r fig.width = 8}
# Get a background map
library(socviz)
data(county_map)

# Make a choropleth map
library(RColorBrewer)  # For the color scale
library(ggthemes) # For theme_map
elections %>% 
  mutate(county_fips = as.character(county_fips)) %>% 
  mutate(county_fips = 
           ifelse(nchar(county_fips) == 4, paste0("0", county_fips), county_fips)) %>% 
  ggplot(aes(map_id = county_fips, fill = cut(repub_pct_20, breaks = seq(0, 100, by = 10)))) +
    geom_map(map = county_map) +
    scale_fill_manual(values = rev(brewer.pal(10, "RdBu")), name = "% Republican") +
    expand_limits(x = county_map$long, y = county_map$lat)  + 
    theme_map() +
    theme(legend.position = "right") + 
    coord_equal()

```

Consider some fairly basic follow-up questions, each of which we cannot answer precisely (or sometimes even at all) using our data viz tools:

-   How many total people voted for the Democratic and Republican candidates in 2020?
-   What about in each state?
-   In just the state of Minnesota:
    -   Which counties had the highest and lowest Democratic vote in 2020?
    -   How did the Democratic vote in each county *change* from 2016 to 2020?

\
\
\
\

**GOALS OF UNIT 2: DATA WRANGLING**

We really cannot do anything with data (viz, modeling, etc) *unless* we can wrangle the data. The following is a typical quote. I agree with the 90% -- data wrangling isn't something we have to do before we can do data science, it *is* data science! But let's rethink the 10% -- data wrangling is a fun and empowering puzzle!

![](https://mac-stat.github.io/images/112/david_robinson.png)

The goals of Unit 2 are to explore how to:

-   Get data into the tidy shape / format we need for analysis. For example, we might want to:
    -   keep only certain observations
    -   define new variables
    -   reformat or "clean" existing variables
    -   combine various datasets
    -   process "string" or text data
-   Numerically (not just visually) explore and summarize various characteristics of the variables in our dataset.

\
\
\
\

**TIDYVERSE WRANGLING TOOLS**

We'll continue to use packages that are part of the `tidyverse`, which share a common general grammar and structure.

![](https://mac-stat.github.io/images/112/tidyverse.png)

\
\
\
\

## Warm-up

There are lots and lots of steps that can go into data wrangling, thus lots and lots of relevant R functions. BUT just 6 functions can get us very far. People refer to these as the 6 main **wrangling verbs** or functions:

-   why "verbs"? in the `tidyverse` grammar, functions serve as *action words*
-   the 6 verbs are all stored in the `dplyr` package within the `tidyverse`
-   each verb acts on a data frame and returns a data frame

\
\

| verb        | action                                            |
|:------------|:--------------------------------------------------|
| `arrange`   | **arrange** the *rows* according to some *column* |
| `filter`    | **filter** out or obtain a subset of the *rows*   |
| `select`    | **select** a subset of *columns*                  |
| `mutate`    | **mutate** or create a *column*                   |
| `summarize` | calculate a numerical **summary** of a *column*   |
| `group_by`  | **group** the *rows* by a specified *column*      |

\
\
\
\

**EXAMPLE 1**

```{r}
head(election)
```

Which verb would help us...

a.  keep *only* information about state names, county names, and the 2020 and 2016 Democratic support (not the 2012 results, demographics, etc)

b.  get *only* the data on Minnesota

c.  define a new variable which calculates the *change* in Democratic support from 2016 to 2020, using `dem_pct_20` and `dem_pct_16`

d.  sort the counties from highest to lowest Democratic support

e.  determine the total number of votes cast across all counties

\
\
\
\

**EXAMPLE 2: select columns**

To get a sense for the code structure, let's explore a couple verbs together. To start, let's simplify our dataset to include only some variables of interest. Specifically, `select()` only the *columns* relevant to state names, county names, and the 2020 and 2016 Democratic support:

```{r eval = FALSE}
# What's the first argument? The second?
select(elections, c(state_name, county_name, dem_pct_20, dem_pct_16))
```

Let's re-do this with the **pipe function** `%>%`:

```{r eval = FALSE}
elections %>% 
  select(state_name, county_name, dem_pct_20, dem_pct_16)
```

\
\
\
\

**THE PIPE FUNCTION**

`%>%` "passes" objects, usually datasets, to a function:

`object %>% function()` is the same `function(object)`

It improves code communication and human readability. For example, if we have many functions we want to use,

`object %>% function() %>% function2()` is the same `function2(function(object))`

The `%>%` like saying "and then" in English. 


*Note: You may see others use `|>` in R as the pipe. Both work in the same way.*


\
\
\
\

**EXAMPLE 3: filter rows**

Let's `filter()` out only the *rows* related to Minnesota (MN):

```{r eval = FALSE}
# Without a pipe
filter(elections, state_name == "Minnesota")
```

```{r eval = FALSE}
# With a pipe
elections %>% 
  filter(state_name == "Minnesota")
```

\
\
\
\

**SUBTLETIES: == vs =**

We use `a == b` to check whether `a` *matches exactly* `b`.

We use `a = b` to *define* that `a` is equal to `b`. We typically use `=` for this purpose inside a function, and `<-` for this purpose outside a function.

```{r}
# Ex: "=" defines x
x = 2
x
```

```{r}
# Ex: "==" checks whether x is/matches 3
x == 3
```

\
\
\
\

**EXAMPLE 4: filter and select**

Let's *combine* `select()` and `filter()` to create a new dataset with info about the county names, and 2020 and 2016 Democratic support among Minnesota counties.

```{r eval = FALSE}
# Without pipes
filter(select(elections, c(state_name, county_name, dem_pct_20, dem_pct_16)), state_name == "Minnesota")
```

```{r eval = FALSE}
# With pipes: all verbs in 1 row
elections %>% select(state_name, county_name, dem_pct_20, dem_pct_16) %>% filter(state_name == "Minnesota")
```

```{r eval = FALSE}
# With pipes: each verb in a new row
elections %>% 
  select(state_name, county_name, dem_pct_20, dem_pct_16) %>% 
  filter(state_name == "Minnesota")
```

```{r eval = FALSE}
# We can even do this with UNtidyverse code in "base" R
elections[elections$state_name == "Minnesota", c(1, 4, 8, 12)]
```

**REFLECT:**

Why will we typically use:

-   tidyverse code
-   the pipe function `%>%`
-   each verb on a new row

\
\
\
\

**EXAMPLE 5: order of operations**

Sometimes, the order of operations matters. For example: putting on socks, then shoes produces a different result than putting on shoes, then socks. Sometimes order doesn't matter. For example: Pouring cereal into a bowl, then adding milk produces the same result as pouring milk into a bowl, then adding cereal (though one order is obviously better than the other ;)) Above, we `select`ed some columns and then `filter`ed some rows:

```{r eval = FALSE}
elections %>% 
  select(state_name, county_name, dem_pct_20, dem_pct_16) %>% 
  filter(state_name == "Minnesota")
```

Would we get the same result if we reversed `select()` and `filter()`? Think first, then try it.

```{r}
# Try it

```

\
\
\
\

**EXAMPLE 6: Storing the results**

Typically:

-   We want to store our data wrangling results.
-   It's good practice to do so under a *new* name. We want to preserve, thus don't want to *overwrite*, the original data (especially if our code contains errors!!).

```{r}
# Store the results
mn <- elections %>% 
  select(state_name, county_name, dem_pct_20, dem_pct_16) %>% 
  filter(state_name == "Minnesota")

# Always check it out to confirm it's what you want it to be!
head(mn)
nrow(mn)
nrow(elections)
```

\
\
\
\

## Exercises

**GOALS**

Learn, practice, and identify when to use the 6 wrangling verbs: `select`, `mutate`, `filter`, `arrange`, `summarize`, `group_by`.

**DIRECTIONS**

-   Work together.
-   Use the pipe function `%>%`.

\
\

### Exercise 1: select practice {.unnumbered}

6 verbs: `select`, `filter`, `arrange`, `mutate`, `summarize`, `group_by`

Use `select()` to create a simplified dataset that we'll use throughout the exercises below.

-   Store this dataset as `elections_small`.
-   Only keep the following variables: state_name, county_name, total_votes_20, repub_pct_20, dem_pct_20, total_votes_16, dem_pct_16

```{r}
# Define elections_small
#___ <- ___ %>%
#  ___(___)

# Check out the first 6 rows to confirm your code did what you think it did!

```

\
\
\
\

### Exercise 2: filter demo {.unnumbered}

6 verbs: `select`, `filter`, `arrange`, `mutate`, `summarize`, `group_by`

Whereas `select()` selects certain *variables* or columns, `filter()` keeps certain *units of observation* or rows relative to their outcome on certain variables. To this end, we must:

-   Identify the variable(s) that are relevant to the filter.

-   Use a "logical comparison operator" to define which values of the variable to *keep*:

    | symbol           | meaning                   |
    |:-----------------|:--------------------------|
    | ==               | equal to                  |
    | !=               | not equal to              |
    | >               | greater than              |
    | >=              | greater than or equal to  |
    | <               | less than                 |
    | <=              | less than or equal to     |
    | %in% c(???, ???) | a list of multiple values |

-   Use quotes "" when specifying outcomes of interest for a categorical variable.

\
\

```{r}
# Keep only data on counties in Hawaii
#elections_small %>% 
#  filter(___ ___ ___)
```

```{r}
# What does this do?
elections_small %>% 
  filter(state_name %in% c("Hawaii", "Delaware"))
```

```{r}
# Keep only data on counties where the Republican got MORE THAN 93.97% of the vote in 2020
# THINK: What variable is relevant here?

```

```{r}
# Keep only data on counties where the Republican got AT LEAST 93.97% of the vote in 2020
# This should have 1 more row than your answer above

```

We can also filter with respect to 2 rules! Here, think what variables are relevant.

```{r}
# Keep only data on counties in Texas where the Democrat got more than 65% of the vote in 2020
# Do this 2 ways.
# Method 1: 2 filters with 1 condition each
#elections_small %>% 
#  filter(___) %>% 
#  filter(___)

# Method 2: 1 filter with 2 conditions
#elections_small %>% 
#  filter(___, ___)
```

\
\
\
\

### Exercise 3: arrange demo {.unnumbered}

6 verbs: `select`, `filter`, `arrange`, `mutate`, `summarize`, `group_by`

`arrange()` arranges or sorts the *rows* in a dataset according to a given column or variable, in ascending or descending order:

`arrange(variable)`, `arrange(desc(variable))`

```{r}
# Arrange the counties in elections_small from lowest to highest percentage of 2020 Republican support
# Print out just the first 6 rows
# elections_small %>% 
#   arrange(___) %>% 
#   head()
```

```{r}
# Arrange the counties in elections_small from highest to lowest percentage of 2020 Republican support
# Print out just the first 6 rows
```

\
\
\
\

### Exercise 4: mutate demo {.unnumbered}

6 verbs: `select`, `filter`, `arrange`, `mutate`, `summarize`, `group_by`

`mutate()` can either transform / mutate an existing variable (column), or define a new variable based on existing ones.

#### Part a {.unnumbered}

```{r}
# What did this code do?
elections_small %>% 
  mutate(diff_20 = repub_pct_20 - dem_pct_20) %>% 
  head()
```

```{r}
# What did this code do?
elections_small %>% 
  mutate(repub_votes_20 = round(total_votes_20 * repub_pct_20/100)) %>% 
  head()
```

```{r}
# What did this code do?
elections_small %>% 
  mutate(repub_win_20 = repub_pct_20 > dem_pct_20) %>% 
  head()
```

#### Part b {.unnumbered}

```{r}
# You try
# Define a variable that calculates the change in Dem support in 2020 vs 2016
#elections_small %>% 
#  mutate(_____) %>% 
#  head()
```

```{r}
# You try
# Define a variable that determines whether the Dem support was higher in 2020 than in 2016 (TRUE/FALSE)
#elections_small %>% 
#  mutate(_____) %>% 
#  head()
```

\
\
\
\

### Exercise 5: Pipe series {.unnumbered}

Let's now combine these verbs into a pipe series!

#### Part a {.unnumbered}

IMPORTANT: BEFORE running the below chunk, what do you think it will produce?

```{r}
elections_small %>% 
  filter(state_name == "Wisconsin",
         repub_pct_20 < dem_pct_20) %>% 
  arrange(desc(total_votes_20)) %>% 
  head()
```

#### Part b {.unnumbered}

IMPORTANT: BEFORE trying, what do you think will happen if you change the order of filter and arrange:

-   the results will be the same
-   we'll get an error
-   we won't get an error, but the results will be different

```{r}
# Now try it. Change the order of filter and arrange below.
elections_small %>% 
  filter(state_name == "Wisconsin",
         repub_pct_20 < dem_pct_20) %>% 
  arrange(desc(total_votes_20)) %>% 
  head()
```

#### Part c {.unnumbered}

So the order of `filter()` and `arrange()` did *not* matter -- rerranging them produces the same results. BUT what is one *advantage* of filtering before arranging?

#### Part d {.unnumbered}

IMPORTANT: BEFORE running the below chunk, what do you think it will produce?

```{r}
elections_small %>% 
  filter(state_name == "Delaware") %>% 
  mutate(repub_win_20 = repub_pct_20 > dem_pct_20) %>% 
  select(county_name, repub_pct_20, dem_pct_20, repub_win_20)
```

#### Part e {.unnumbered}

IMPORTANT: BEFORE trying, what do you think will happen if you change the order of mutate and select:

-   the results will be the same
-   we'll get an error
-   we won't get an error, but the results will be different

```{r}
# Now try it. Change the order of mutate and select below.
elections_small %>% 
  filter(state_name == "Delaware") %>% 
  mutate(repub_win_20 = repub_pct_20 > dem_pct_20) %>% 
  select(county_name, repub_pct_20, dem_pct_20, repub_win_20)
```

\
\
\
\

### Exercise 6: DIY pipe series {.unnumbered}

We've now learned 4 of the 6 wrangling verbs: select, filter, mutate, arrange. Let's practice combining these into pipe series. Here are some hot tips:

-   Before writing any code, translate the prompt: how many distinct wrangling steps are needed and what verb do we need in each step?
-   Add each verb one at a time -- don't try writing a whole chunk at once.

### Part a {.unnumbered}

Show just the counties in Minnesota and their Democratic 2020 vote percentage, from highest to lowest. Your answer should have just 2 columns.

```{r}

```

#### Part b {.unnumbered}

Create a new dataset named `mn_wi` that sorts the counties in Minnesota and Wisconsin from lowest to highest in terms of the *change* in Democratic vote percentage in 2020 vs 2016. This dataset should include the following variables (and only these variables): state_name, county_name, dem_pct_20, dem_pct_16, and a variable measuring the change in Democratic vote percentage in 2020 vs 2016.

```{r}
# Define the dataset
# Only store the results once you're confident that they're correct


# Check out the first 6 rows to confirm your results

```

#### Part c {.unnumbered}

Construct and discuss a plot of the county-level change in Democratic vote percent in 2020 vs 2016, and how this differs between Minnesota and Wisconsin.

```{r}

```

\
\
\
\

### Exercise 7: summarize demo {.unnumbered}

6 verbs: `select`, `filter`, `arrange`, `mutate`, `summarize`, `group_by`

Let's talk about the last 2 verbs. `summarize()` (or equivalently `summarise()`) takes an entire data frame as input and outputs a single row with one or more summary statistics. For each chunk below, indicate what the code does.

```{r}
# What does this do?
elections_small %>% 
  summarize(median(repub_pct_20))
```

```{r}
# What does this do?
elections_small %>% 
  summarize(median_repub = median(repub_pct_20))
```

```{r}
# What does this do?
elections_small %>% 
  summarize(median_repub = median(repub_pct_20), total_votes = sum(total_votes_20))
```

\
\
\
\

### Exercise 8: `summarize` + `group_by` demo {.unnumbered}

6 verbs: `select`, `filter`, `arrange`, `mutate`, `summarize`, `group_by`

Finally, `group_by()` groups the units of observation or rows of a data frame by a specified set of variables. Alone, this function doesn't change the appearance of our dataset or seem to do anything at all:

```{r}
elections_small %>% 
  group_by(state_name) %>% 
  head()
```

Though it does change the underlying structure of the dataset:

```{r}
# Check out the structure before and after group_by
elections_small %>% 
  class()

elections_small %>% 
  group_by(state_name) %>% 
  class()
```

Where it really shines is in partnership with `summarize()`.

```{r}
# What does this do?
# (What if we didn't use group_by?)
elections_small %>% 
  group_by(state_name) %>% 
  summarize(median_repub = median(repub_pct_20), total_votes = sum(total_votes_20)) 
```

REFLECT:\
Notice that `group_by()` with `summarize()` produces new data frame or tibble! But the units of observation are now states instead of counties within states.

\
\
\
\

### Exercise 9: DIY {.unnumbered}

Let's practice (some of) our 6 verbs: `select`, `filter`, `arrange`, `mutate`, `summarize`, `group_by` Remember:

-   Before writing any code, translate the given prompts: how many distinct wrangling steps are needed and what verb do we need in each step?
-   Add each verb one at a time.

#### Part a {.unnumbered}

NOTE: Part a is a challenge exercise. If you get really stuck, move on to Part b which is the same overall question, but with hints.

```{r}
# Sort the *states* from the most to least total votes cast in 2020

```

```{r}
# In 2020, what were the total number of votes for the Democratic candidate and the total number of votes for the Republican candidate in each *state*?

```

```{r}
# What states did the Democratic candidate win in 2020?

```

#### Part b {.unnumbered}

```{r}
# Sort the states from the most to least total votes cast in 2020
# HINT: Calculate the total number of votes in each state, then sort

```

```{r}
# In 2020, what were the total number of votes for the Democratic candidate and the total number of votes for the Republican candidate in each state?
# HINT: First calculate the number of Dem and Repub votes in each *county*
# Then group and summarize these by state

```

```{r}
# What states did the Democratic candidate win in 2020?
# HINT: Start with the results from the previous chunk, and then keep only some rows

```

\
\
\
\

### Exercise 10: Practice on new data {.unnumbered}

Recall the World Cup football/soccer data from TidyTuesday:

```{r}
world_cup <- read.csv("https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2022/2022-11-29/worldcups.csv")
```

You can find a codebook [here](https://github.com/rfordatascience/tidytuesday/tree/master/data/2022/2022-11-29). Use (some of) our 6 verbs (`select`, `filter`, `arrange`, `mutate`, `summarize`, `group_by`) and data viz to address the following prompts.

```{r}
# In what years did Brazil win the World Cup?

```

```{r}
# What were the 6 World Cups with the highest attendance?

```

```{r}
# Construct a univariate plot of goals_scored (no wrangling necessary)
# This provides a visual summary of how the number of goals_scored varies from World Cup to World Cup

```

```{r}
# Let's follow up the plot with some more precise numerical summaries
# Calculate the min, median, and max number of goals_scored across all World Cups
# NOTE: Visually compare these numerical summaries to what you observed in the plot

```

```{r}
# Construct a bivariate plot of how the number of goals_scored in the World Cup has changed over the years
# No wrangling necessary

```

```{r}
# Our above summaries might be a bit misleading.
# The number of games played at the World Cup varies.
# Construct a bivariate plot of how the typical number of goals per game has changed over the years


```

\
\
\
\

### Exercise 11: More viz review {.unnumbered}
Let's re-consider Minnesota and Wisconsin. Use `mn_wi_new` throughout this exercise to practice identifying what viz is appropriate in a given situation, and how to create that viz.
Challenge yourself to not just start typing things until something works -- first think about what plot would be appropriate.

```{r}
mn_wi_new <- elections %>%
  filter(state_name %in% c("Minnesota", "Wisconsin")) %>%
  mutate(repub_win_20 = (repub_pct_20 > dem_pct_20),
          repub_win_16 = (repub_pct_16 > dem_pct_16),
          ) %>%
  select(state_abbr, repub_pct_20, repub_pct_16, repub_pct_12, repub_win_20, repub_win_16)
head(mn_wi_new)
```

```{r}
# Construct a plot that illustrates the relationship of  repub_pct_20 with repub_pct_16 and repub_pct_12
```

```{r}
# Construct a plot that illustrates the relationship of  repub_pct_20 with repub_pct_16 in MN vs WI.
# Do this in 1 frame and add a linear trend
```

```{r}
# Construct a plot that illustrates the relationship of  repub_pct_20 with repub_pct_16 in MN vs WI.
# Do this in 2 separate frames -- 1 for MN and 1 for WI
# and include linear trends
```

```{r}
# Construct a plot that illustrates how many counties were
# won by the republican in 2020, combining MN and WI counties
```

```{r}
# Construct a plot that illustrates how many counties were
# won by the republican in 2020 in MN vs WI, i.e.
# the relationship between state_abbr and repub_win_20
```

```{r}
# Starting with the previous plot, and if you didn't already,
# tweak it so that we can directly compare the proportion
# of counties won by the republican in 2020 in MN and WI,
# ignoring the fact that MN has more counties
```
\
\
\
\
