8.14 Vector Projection

The projection of a vector \(\mathbf{y}\) on a vector \(\mathbf{x}\) is \[\frac{\mathbf{y}^T\mathbf{x}}{|\mathbf{x}|_2^2}\mathbf{x} \]

The orthogonal projection of a vector \(\mathbf{y}\) on a the column space of matrix \(\mathbf{X}\) gives you the \(\hat{\mathbf{y}} = \mathbf{X}\boldsymbol\beta\) that minimizes \(||\mathbf{y} - \hat{\mathbf{y}}||\).

\[(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{y} \]