---
title: "Overfitting (Notes)"
subtitle: "Stat 253"
author: "Your Name"
format:
  html:
    toc: true
    toc-depth: 2
    embed-resources: true
---



```{r include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE, 
  warning = FALSE,
  message = FALSE,
  error = TRUE,
  fig.height = 2.75, 
  fig.width = 4.25,
  fig.env='figure',
  fig.pos = 'h',
  fig.align = 'center')
```



# Learning Goals {-}

- Explain why training/in-sample model evaluation metrics can provide a misleading view of true test/out-of-sample performance
- Implement testing and training sets in R using the `tidymodels` package

\

# Notes {-}

## Overfitting {.unnumbered .smaller}


When we add more and more predictors into a model, it can become *overfit* to the noise in our sample data:

- our model loses the broader trend / big picture
- thus does not generalize to *new* data
- thus results in bad predictions and a bad understanding of the relationship among the new data points



**Preventing overfitting: training and testing**

- **In-sample** metrics, i.e. measures of how well the model performs on the same sample data that we used to build it, tend to be overly optimistic and lead to overfitting. 
- Instead, we should *build* and *evaluate*, or **train** and **test**, our model using different data.


\

## Future Reference: R Code for training/testing {.unnumbered .smaller}

:::{.callout-note}
This section is for **future** reference. 
It is a summary of code you'll learn below for creating and applying training and testing data. 
You don't need to (and in fact should not) run the code in this section---just use this as example code for future reference.
:::

Suppose we wish to build and evaluate a linear regression model of `y` vs `x1` and `x2` using our `sample_data`. 

```{r eval = FALSE}
# Load packages
library(tidyverse)
library(tidymodels)
```


**Split the sample data into training and test sets**    

```{r eval = FALSE}
# Set the random number seed
set.seed(___)

# Split the sample_data
# "prop" is the proportion of data assigned to the training set
# it must be some number between 0 and 1
data_split <- initial_split(sample_data, strata = y, prop = ___)

# Get the training data from the split
data_train <- data_split %>% 
  training()

# Get the testing data from the split
data_test <- data_split %>% 
  testing()
```


**Build a training model**    

```{r eval = FALSE}
# STEP 1: model specification
lm_spec <- linear_reg() %>% 
  set_mode("regression") %>% 
  set_engine("lm")

# STEP 2: model estimation using the training data
model_train <- lm_spec %>% 
  fit(y ~ x1 + x2, data = data_train)
```


**Use the training model to make predictions for the test data**        

```{r eval = FALSE}
# Make predictions
model_train %>% 
  augment(new_data = data_test)
```



**Evaluate the training model using the test data**

```{r eval = FALSE}
# Calculate the test MAE
model_train %>% 
  augment(new_data = data_test) %>% 
  mae(truth = y, estimate = .pred)
```


\
\


# Exercises (Part 2) {-}

## Directions {-}

- Open the Part 2 QMD file
- Same directions as before: 
  - Be kind to yourself/each other
  - Collaborate
- We will not discuss these exercises as a class. 
Be sure to ask questions as I walk around the room.

\

## Questions {.unnumbered .smaller}

The following exercises are inspired by Chapter 5.3.1 of ISLR.

```{r message = FALSE, warning = FALSE}
# Load packages & data
# NOTE: You might first need to install the ISLR package
library(tidyverse)
library(tidymodels)
library(ISLR)
data(Auto)
cars <- Auto %>% 
  dplyr::select(mpg, horsepower, year)
```

\

Let's use the `cars` data to compare three **linear regression models** of fuel efficiency in miles per gallon (`mpg`) by engine power (`horsepower`):

```{r}
# Raw data
cars_plot <- ggplot(cars, aes(x = horsepower, y = mpg)) + 
  geom_point()
cars_plot
```

```{r}
# model 1: 1 predictor (y = b0 + b1 x)
cars_plot + 
  geom_smooth(method = "lm", se = FALSE)
```

```{r}
# model 2: 2 predictors (y = b0 + b1 x + b2 x^2)
cars_plot + 
  geom_smooth(method = "lm", se = FALSE, formula = y ~ poly(x, 2))
```

```{r}
# model 3: 19 predictors (y = b0 + b1 x + b2 x^2 + ... + b19 x^19)
cars_plot + 
  geom_smooth(method = "lm", se = FALSE, formula = y ~ poly(x, 19))
```


\

**Goal**    

Let's evaluate and compare these models by **training** and **testing** them using different data. 


\

1. **155 review: set.seed()**       

Run the two chunks below *multiple* times each. Afterward, summarize what `set.seed()` does and why it's important to being able to *reproduce* a random sample.
    
```{r}
sample_n(cars, 2)
```
    
```{r}
set.seed(253)
sample_n(cars, 2)
```


\    

2.  **Training and test sets**       

Let's *randomly* split our original 392 sample cars into two separate pieces: select 80% of the cars to **train (build)** the model and the other 20% to **test (evaluate)** the model.
    
```{r}
# Set the random number seed
set.seed(8)
    
# Split the cars data into 80% / 20%
# Ensure that the sub-samples are similar with respect to mpg
cars_split <- initial_split(cars, strata = mpg, prop = 0.8)
```
  
```{r}
# Check it out
# What do these numbers mean?
cars_split
```
    
```{r}
# Get the training data from the split
cars_train <- cars_split %>% 
  training()
    
# Get the testing data from the split
cars_test <- cars_split %>% 
  testing()
```
    
```{r}
# The original data has 392 cars
nrow(cars)
    
# How many cars are in cars_train?
    
# How many cars are in cars_test?
    
```
    



\

3. **Reflect on the above code**   

a. Why do we want the training and testing data to be similar with respect to `mpg` (`strata = mpg`)? What if they *weren't*?
b. Why did we need all this new code instead of just using the *first* 80% of cars in the sample for training and the *last* 20% for testing?
    






\

4.  **Build the training model**   

```{r eval = FALSE}
# STEP 1: model specification
lm_spec <- linear_reg() %>% 
  set_mode("regression") %>% 
  set_engine("lm")

# STEP 2: model estimation using the training data
# Construct the 19th order polynomial model using the TRAINING data
model_19_train <- ___ %>% 
  ___(mpg ~ poly(horsepower, 19), data = ___)
```
    
```{r}
    
```






\

5. **Evaluate the training model** 

```{r eval = FALSE}
# How well does the TRAINING model predict the TRAINING data?
# Calculate the training (in-sample) MAE
model_19_train %>% 
  augment(new_data = ___) %>% 
  mae(truth = mpg, estimate = .pred)
```
    
```{r}
    
```
    
```{r eval = FALSE}
# How well does the TRAINING model predict the TEST data?
# Calculate the test MAE
model_19_train %>% 
  augment(new_data = ___) %>% 
  mae(truth = mpg, estimate = .pred)
```
    
```{r}

```






\

6. **Punchline**    
    The table below summarizes your results for `train_model_19` as well as the other two models of interest. (You should confirm the other two model results outside of class!) 
    
Model                         Training MAE   Testing MAE
---------------------------- ------------- -------------
`mpg ~ horsepower`                    3.78          4.00
`mpg ~ poly(horsepower, 2)`           3.20          3.49
`mpg ~ poly(horsepower, 19)`          2.99          6.59


    
Answer the following and reflect on why each answer *makes sense*:
    
a. *Within* each model, how do the training errors compare to the testing errors? (This isn't *always* the case, but is common.)    
b. What about the training and test errors for the third model suggest that it is *overfit* to our sample data?
c. Which model seems the best with respect to the *training* errors?    
d. Which model is the best with respect to the *testing* errors?    
e. Which model would you choose?
    

\

7.  **Final reflection**    
    a. The training / testing procedure provided a more honest evaluation and comparison of our model predictions. How might we *improve* upon this procedure? What problems can you anticipate in splitting our data into 80% / 20%?
    b. Summarize the key themes from today in your own words.
    


\

8. **STAT 155 REVIEW: data drill**    

a. Construct and interpret a plot of `mpg` vs `horsepower` and `year`.
b. Calculate the average `mpg`.
c. Calculate the average `mpg` for each `year`. HINT: `group_by()`
d. Plot the *average* `mpg` by `year`.


\

9. **Digging deeper (optional)**    

Check out the online solutions for exercise 6. Instead of calculating MAE from scratch for 3 different models, I wrote a function `calculate_MAE()` to automate the process. After picking through this code, adapt the function so that it also returns the $R^2$ value of each model.  


## Done!

- Render your notes.
- Check the solutions in the course website (Solution drop downs).
- If you finish all that during class, start your homework!


