5.4 Autoregressive Models
The first model we will consider for a stationary process is an autoregressive model, which involves regressing current observations on values in the past. So this model regresses on itself (the ‘auto’ part of ‘autoregressive’).
5.4.1 AR(1) Model
One simple model for correlated errors is an autoregressive order 1 or AR(1) model. The model is stated
\[Y_t = \delta + \phi_1Y_{t-1} + W_t\] where \(W_t \stackrel{iid}{\sim} N(0,\sigma^2_w)\) is independent Gaussian white noise with mean 0 and constant variance, \(\sigma^2_w\). This model is weakly stationary if \(|\phi_1| < 1\).
Note: We will often think of \(\delta = 0\) because our error process is often on average 0.
Properties
Under this model, the expected value (mean) of \(Y_t\) is
\[E(Y_t) = \mu = \frac{\delta}{1-\phi_1}\]
and the variance is
\[Var(Y_t) = \frac{\sigma^2_w}{1-\phi_1^2}\]
The correlation between observations \(h\) lags (time periods) apart is
\[\rho_h = \phi^h_1\]
Derivations for these properties are available in the Appendix at the end of this chapter.
Sample ACF for AR(1): Decays to zero exponentially
Simulated Data Example
We will generate data from an AR(1) model with \(\phi_1 = 0.64\). For a positive \(\phi_1\), the autocorrelation exponentially decreases to 0 as the lag increases, \(\rho_h = (0.64)^h\).
# Simulate an ar(1) process
# x = 0.05 + 0.64*x(t-1) + e
# Create the vector x
x <- vector(length=1000)
#simulate the white noise errors
e <- rnorm(1000)
#Set the coefficient
beta <- 0.64
# set an initial value
x[1] <- 0.055
#Fill the vector x
for(i in 2:length(x))
{
x[i] <- 0.05 + beta*x[i-1] + e[i]
}
x <- ts(x)
plot(x)
acf(x)Now, we’ll try \(\phi_1 = -0.8\). For a negative \(\phi_1\), the autocorrelation exponentially decreases to 0 as the lag increases, but the signs of the autocorrelation alternate between positive and negative (\(\rho_h = (-0.8)^h\)).
# Simulate an ar(1) process
# x = 0.05 - 0.8*x(t-1) + e
# Create the vector x
x <- vector(length=1000)
#simulate the white noise errors
e <- rnorm(1000)
#Set the coefficient
beta <- -0.8
# set an initial value
x[1] <- 0.05
#Fill the vector x
for(i in 2:length(x))
{
x[i] <- 0.05 + beta*x[i-1] + e[i]
}
x <- ts(x)
plot(x)
acf(x)Remember that \(|\phi_1| < 1\) in order for an AR(1) process to be stationary (constant mean, constant variance, autocovariance only depends on lags).
5.4.2 Random Walk
If \(\phi_1 = 1\) and \(\delta = 0\) (a simplifying assumption), the AR(1) model becomes a well-known process called a random walk. The model for a random walk is
\[Y_t = Y_{t-1} + W_t\] where \(W_t \stackrel{iid}{\sim} N(0,\sigma^2_w)\).
If we start with \(t = 1\), then we start with random white noise,
\[Y_1 = W_1\]
The next observation is the past observation plus noise,
\[Y_2 = Y_1 + W_2 = W_1 + W_2\]
The next observation is the past observation plus noise,
\[Y_3 = Y_2 + W_3 = (W_1+W_2)+W_3\]
And so forth,
\[Y_n = Y_{n-1} + W_n = \sum_{i=1}^n W_i\]
So a random walk is the sum of random white noise random variables.
Properties
This random walk process is NOT weakly stationary because the variance is not constant as it is a function of time, \(t\),
\[Var(Y_t) = Var(\sum^t_{i=1} W_i) = \sum^t_{i=1} Var(W_i) = t\sigma^2_w\]
Simulated Data Example
When we generate many random walks, you’ll notice greater potential variability in values later in time. Additionally, you’ll see high autocorrelation at higher lags, indicating a high range of dependence. Two observations that are far in observation time are still highly correlated.
# Simulate one random walk process
# x = x(t-1) + e
#simulate the white noise errors
e <- rnorm(1000)
# cumulative sum based on recursive process above
x <- cumsum(e)
x <- ts(x)
plot(x)
acf(x)# Simulate many random walk processes
plot(1:1000,rep(1,1000),type='n',ylim=c(-100,100),xlab='t',ylab='x')
for(i in 1:100){
e <- rnorm(1000)
x <- cumsum(e)
x <- ts(x)
lines(ts(x))
}If we were to take a first-order difference of a random walk, \(Y_t - Y_{t-1}\), we could remove the trend and end up with independent white noise.
\[Y_t - Y_{t-1}= W_t\]
5.4.3 AR(p) Model
We could generalize the idea of an AR(1) model to allow an observation to be dependent on the past \(p\) previous observations. A autoregression process of order p is modeled as
\[Y_t = \delta + \phi_1Y_{t-1}+ \phi_2Y_{t-2}+\cdots + \phi_pY_{t-p} +W_t\] where \(\{W_t\}\) is independent Gaussian white noise, \(W_t \stackrel{iid}{\sim} N(0,\sigma^2_w)\). We will typically let \(\delta=0\), assuming we have removed the trend and seasonality before applying this model.
Properties
We’ll look at the expected value, variance, and covariance in the AR(p) as MA(\(\infty\)) section after discussing moving average models. We’ll also discuss when the AR(p) model is stationary. I know you can’t wait!
Simulated Data Example
Let’s see a few simulated examples and look at the autocorrelation functions.
set.seed(123)
## the 4 AR coefficients
ARp <- c(0.7, 0.2, -0.1, -0.3)
## empty list for storing models
AR.mods <- list()
## loop over orders of p
for (p in 1:4) {
## assume SD=1, so not specified
AR.mods[[p]] <- arima.sim(n = 10000, list(ar = ARp[1:p]))
}
## set up plot region
par(mfrow = c(4, 2))
## loop over orders of p
for (p in 1:4) {
plot.ts(AR.mods[[p]][1:50], ylab = paste("AR(", p, ")", sep = ""))
acf(AR.mods[[p]], lag.max = 12,main=paste("AR(", p, ")", sep = ""))
}Sample ACF for AR(p): Decay to zero