Motivating Question

Question

The field of machine learning is most often associated with the building of predictive models, not inferential models. Specifically, the goal is to build a model which produces good predictions of our response variable \(y\), not one that necessarily lends itself to testing specific hypotheses about \(y\). In this case:

If we have access to a bunch of potential predictors \(x\), how can we decide which model to build?


Model selection methods

  1. Variable selection Identify a subset of predictors to use in our model of \(y\). Methods: best subset selection, backward stepwise selection, forward stepwise selection

  2. Shrinkage / regularization Shrink / regularize the coefficients of all predictors toward or to 0. Methods: LASSO, ridge regression, elastic net (a combination of LASSO & ridge)

  3. Dimension reduction Combine the predictors into a smaller set of new predictors. Methods: principal components regression