---
title: "Model Evaluation Experiment (Notes)"
subtitle: "Stat 253"
author: "Your Name"
output: 
  html_document:
    toc: true
    toc_float: true
---

```{r include = FALSE}
# OPTIONAL: Set a more color blind friendly palette 
palette("Okabe-Ito")
scale_colour_discrete <- function(...) scale_colour_manual(values = palette())
scale_fill_discrete   <- function(...) scale_fill_manual(values = palette())

knitr::opts_chunk$set(
  collapse = TRUE, 
  warning = FALSE,
  message = FALSE,
  fig.height = 2.75, 
  fig.width = 4.25,
  fig.env='figure',
  fig.pos = 'h',
  fig.align = 'center')
```



# Learning Goals {-}

- Explain why training/in-sample model evaluation metrics can provide a misleading view of true test/out-of-sample performance
- Implement testing and training sets in R using the `tidymodels` package




# Small Group Discussion {-}


**Goal**

- Let's *build* and *evaluate* a predictive model of an adult's height ($y$) using some predictors $x_i$ (eg: age, height, etc).

- Since $y$ is *quantitative* this is a **regression task**.

- There are countless possible models of $y$ vs $x$. We'll utilize a **linear regression model**:

$$y = \beta_0 + \beta_1 x_1 + \cdots + \beta_p x_p + \varepsilon$$

- And after building this model, we'll **evaluate** it.





**Data:** Each group will be given a different sample of 40 adults.

```{r}
# Load packages needed for this analysis
library(tidyverse)
library(tidymodels)
```

```{r}
# Load your data: fill in the blanks at end of url with your number
# group 1 = 50
# group 2 = 143
# group 3 = 160
# group 4 = 174
# group 5 = 86
humans <- read.csv("https://bcheggeseth.github.io/253_spring_2024/data/bodyfat___.csv") %>% 
  filter(ankle < 30) %>% 
  rename(body_fat = fatSiri)
```

```{r}
# Check out a density plot of your y outcomes
ggplot(humans, aes(x = height)) + 
  geom_density()
```




**Model building:** Build a linear regression model of `height` (in) by `hip` circumference (cm).

```{r eval = FALSE}
# STEP 1: model specification
lm_spec <- ___() %>% 
  set_mode(___) %>% 
  set_engine(___)
```

```{r}

```

```{r eval = FALSE}
# STEP 2: model estimation
model_1 <- ___ %>% 
  ___(height ~ hip, data = humans)
```

```{r}

```

```{r}
# Check out the coefficients
# Do all groups have the same coefficients? Should they?

```



**Model evaluation:** How *good* is our model?

```{r}
# Calculate the R^2 for model_1

```

```{r eval = FALSE}
# Use your model to predict height for your subjects
# Just print the first 6 results
model_1 %>% 
  ___(new_data = ___) %>% 
  head()
```

```{r}

```

```{r eval = FALSE}
# Calculate the MAE, i.e. typical prediction error, for your model
model_1 %>% 
  augment(new_data = humans) %>% 
  ___(truth = ___, estimate = ___)
```

```{r}

```






**Reflection**

In addition to `hip` circumference, suppose we incorporated more predictors into our model of `height`. What would happen to $R^2$? To the MAE?




# In-Class Activity Part 1 - Exercises {-}

**DIRECTIONS**

- Be kind to yourself. You will make mistakes!
- Be kind to each other. Collaboration improves higher-level thinking, confidence, communication, community, & more. You are expected to:
    - actively contribute to discussion (don't work on your own)
    - actively include all other group members in discussion
    - create a space where others feel comfortable making mistakes & sharing their ideas (remember that you all have different experiences, both personal and academic)
    - stay in sync
- You will have **5 minutes** to complete exercise 1.

    

\
\


## Exercise 1: Select a model

Consider 3 different models of `height`, estimated below. As a group, use *your* data to choose which is the best predictive model of `height`. Calculate the MAE for this model.

```{r}
# height vs hip
model_1 <- lm_spec %>% 
  fit(height ~ hip, data = humans)
model_1 %>% 
  tidy()
```

```{r}
# height vs hip & weight
model_2 <- lm_spec %>% 
  fit(height ~ hip + weight, data = humans)
model_2 %>% 
  tidy()
```

```{r}
# height vs a lot of predictors (AND some interaction terms)
model_3 <- lm_spec %>% 
  fit(height ~ chest * age * weight * body_fat * abdomen + hip + thigh + knee + ankle + biceps + forearm + wrist, data = humans)
model_3 %>% 
  tidy()
```

```{r}
# Calculate the MAE for your model
___ %>% 
  augment(new_data = humans) %>% 
  mae(truth = height, estimate = .pred)
```





## Exercise 2: Share your results

Only when you're done with exercise 1:       

- Open this ["Top Model Competition" Google Doc](https://docs.google.com/spreadsheets/d/1DL1YadxFDV1o_pPHyST8u8QEyUKbjTRX1LH2NzhVtYU/edit?usp=sharing)).
- Record your team name.
- Record which model you chose (1, 2, or 3).
- Record the MAE for your model.
- **WAIT.** Don't keep going.














































































**DON'T PEAK**




























**SERIOUSLY**





























**REALLY**











































**Don't peak**

What do you know?! 40 new people just walked into the doctor's office and the doctor wants to predict their `height`:

```{r}
# Import the new data
new_patients <- read.csv("https://bcheggeseth.github.io/253_spring_2024/data/bodyfat182.csv") %>% 
  filter(ankle < 30) %>% 
  rename(body_fat = fatSiri)
```





## Exercise 3: Intuition

Consider using *your* model to predict `height` for these 40 *new* subjects. On average, do you think these predictions will be better or worse than for your original patients? Why?








## Exercise 4: How well does your model do in the real world?

Use *your* model to predict `height` for the *new* patients and calculate the typical prediction error (MAE). **Record this in the Google sheet.**
    
```{r}
___ %>% 
  augment(new_data = new_patients) %>% 
  mae(truth = height, estimate = .pred)
```









## Exercise 5: Reflection

In summary, which model seems best? What's the central theme here?




**Now open and save the Rmd for the Part 2. **



