Lecture notes. Written from the material of Berkeley Stat 153, licensed CC BY 4.0. These are notes, not a transcript: the material has been reorganised and rewritten. This adaptation carries the same licence, and the original is linked above.

112. Parameter Estimation in AR(1)¶

What this covers¶

This chapter derives the likelihood for the AR(1) model $y_t = \phi_0 + \phi_1 y_{t-1} + \epsilon_t$ and asks whether its parameters can be estimated the way ordinary linear regression estimates $\beta_0,\beta_1$. It assumes the AR($p$) model, ordinary linear regression and its Gaussian likelihood, maximum likelihood estimation, and the Bayesian $t$-posterior for regression coefficients (from an earlier lecture and homework) are already familiar. The payoff is two competing definitions of "the AR(1) model" — one that simply fixes $y_1$ and never asks where it came from, one that treats every $y_t$ as generated by an infinite history of past shocks — and a first look at why $|\phi_1| < 1$ matters, ahead of a proper treatment of stationarity next lecture.

From AR($p$) to a regression problem¶

Recall the AR($p$) model: $$ y_t = \phi_0 + \phi_1 y_{t-1} + \dots + \phi_p y_{t-p} + \epsilon_t, \qquad t = p+1, \dots, n. $$ Stacking the equations for $t = p+1,\dots,n$ gives the matrix form $Y = X\beta + \epsilon$, with $$ Y = \begin{pmatrix} y_{p+1} \ y_{p+2} \ \vdots \ y_n \end{pmatrix}, \quad X = \begin{pmatrix} 1 & y_p & y_{p-1} & \cdots & y_1 \ 1 & y_{p+1} & y_p & \cdots & y_2 \ \vdots & \vdots & \vdots & & \vdots \ 1 & y_{n-1} & y_{n-2} & \cdots & y_{n-p} \end{pmatrix}, \quad \beta = \begin{pmatrix} \phi_0 \ \phi_1 \ \vdots \ \phi_p \end{pmatrix}, \quad \epsilon = \begin{pmatrix} \epsilon_{p+1} \ \epsilon_{p+2} \ \vdots \ \epsilon_n \end{pmatrix}. $$ So AR($p$) is literally an instance of the ordinary linear regression model $Y = X\beta+\epsilon$ — except that the "covariates" in $X$ and the "response" in $Y$ are both carved out of the same single sequence $y_1,\dots,y_n$, rather than coming from independent observational units. That single fact is what breaks the usual regression argument, and chasing down exactly where it breaks is the point of this chapter. To keep the algebra light, take $p=1$ throughout and return to general $p$ later: $$ y_t = \phi_0 + \phi_1 y_{t-1} + \epsilon_t, \qquad t = 2,\dots,n. $$

Detour: where does the regression likelihood actually come from?¶

The AR(1) equation looks exactly like a simple linear regression $y_i = \beta_0 + \beta_1 x_i + \epsilon_i$ with $i=t$, $x_i = y_{t-1}$. So before writing a likelihood for AR(1), it is worth being precise about which assumptions produce the familiar Gaussian likelihood in ordinary regression, so each one can be checked against the AR setting.

With data $(x_i,y_i)$, $i=1,\dots,m$, the likelihood is the joint density of the data, viewed as a function of $\theta = (\beta_0,\beta_1,\sigma)$. Since the model expresses $y_i$ in terms of $x_i$, it is natural to condition on the covariates first: $$ f_{x_1,y_1,\dots,x_m,y_m\mid\theta} = f_{y_1,\dots,y_m\mid x_1,\dots,x_m,\theta}(y_1,\dots,y_m)\; f_{x_1,\dots,x_m\mid\theta}(x_1,\dots,x_m). $$ Substituting $y_i = \beta_0+\beta_1 x_i+\epsilon_i$ turns the first factor into a density of the errors evaluated at the residuals $y_i - \beta_0 - \beta_1 x_i$, still conditional on $x_1,\dots, x_m,\theta$. From here, four separate assumptions are used, one at a time, to reach the familiar likelihood:

The model equation $y_i = \beta_0+\beta_1 x_i + \epsilon_i$ itself.
Independence of the errors and the covariates: $\epsilon_1,\dots,\epsilon_m$ independent of $x_1,\dots,x_m$ (given $\theta$). This is what lets the conditioning on $x_1,\dots,x_m$ be dropped from the error density.
$\epsilon_i \overset{\text{i.i.d}}{\sim} N(0,\sigma^2)$, which turns the error density into an explicit Gaussian product.
The covariates' density does not depend on $\theta$: $f_{x_1,\dots,x_m\mid\theta}$ is assumed free of $\theta$, so it is just a constant multiplicative factor and can be dropped when the likelihood is only needed up to proportionality.

Putting these together gives the familiar $$ \text{Likelihood} \;\propto\; \left(\frac{1}{\sqrt{2\pi}\sigma}\right)^m \exp\left(-\frac{1}{2\sigma^2}\sum_{i=1}^m (y_i-\beta_0-\beta_1 x_i)^2\right). $$ From here, frequentist inference maximizes this to get $\hat\beta = (X^TX)^{-1}X^Ty$ and $\hat\sigma_{\text{MLE}} = \sqrt{\text{RSS}/n}$, and then separately works out the sampling distribution of $\hat\beta,\hat\sigma_{\text{MLE}}$ — using the same four assumptions a second time. Bayesian inference instead combines the likelihood with the prior $\beta_0,\beta_1,\log\sigma \overset{\text{i.i.d}}{\sim}\text{unif}(-C,C)$ to get the posterior $$ \beta \mid \text{data} \sim t_{m-2,2}!\left(\hat\beta, \hat\sigma^2(X^TX)^{-1}\right), \qquad \hat\sigma := \sqrt{\text{RSS}/(m-2)}, $$ with a posterior for $\sigma$ derivable the same way (this was Homework One). The asymmetry to notice: once the likelihood is written down, Bayesian inference never touches the four assumptions again — only the likelihood function matters for the posterior. Frequentist inference uses the assumptions twice: once to build the likelihood and the MLEs, and again to find the MLEs' distribution. That asymmetry is exactly what will make AR(1) genuinely harder for one school than the other.

Why the same derivation fails for AR(1)¶

Check assumptions 2 and 4 against $y_t = \phi_0+\phi_1 y_{t-1}+\epsilon_t$ with $x_t = y_{t-1}$:

Assumption 4 fails outright. The density of $x_t = y_{t-1}$ does depend on $\theta$, because $y_{t-1}$ is itself generated by the AR equation one step earlier, $y_{t-1} = \phi_0+\phi_1 y_{t-2}+\epsilon_{t-1}$ — so $\phi_0,\phi_1,\sigma$ all shape the distribution of $y_{t-1}$.
Assumption 2 is not obviously true either. There is no a priori reason $\epsilon_t$ should be independent of $x_t = y_{t-1}, \dots$ (and, further back, of $y_1,\dots,y_{t-2}$), since those are earlier values of the very series $\epsilon_t$ helps generate.

So the regression argument cannot simply be recycled for AR(1). The likelihood has to be built from a different route, using a different — but analogous — set of assumptions.

Building the AR(1) likelihood by the chain rule¶

Rather than conditioning on covariates, factor the joint density of $y_1,\dots,y_n$ using the chain rule for conditional densities: $$ f_{y_1,\dots,y_n\mid\theta}(y_1,\dots,y_n) = f_{y_1\mid\theta}(y_1)\prod_{t=2}^n f_{y_t\mid y_1,\dots,y_{t-1},\theta}(y_t). $$ Substituting the model equation inside each conditional density turns $f_{y_t\mid y_1,\dots,y_{t-1},\theta}(y_t)$ into the density of $\epsilon_t$ (conditional on the past) evaluated at the residual $y_t - \phi_0-\phi_1 y_{t-1}$. Now assume $\epsilon_t$ is independent of $y_1,\dots,y_{t-1}$ for each $t=2,\dots,n$ — the AR analogue of the regression independence assumption, but stated about the driving noise rather than about covariates. This removes the conditioning: $$ f_{y_t\mid y_1,\dots,y_{t-1},\theta}(y_t) = f_{\epsilon_t}(y_t-\phi_0-\phi_1 y_{t-1}). $$ With $\epsilon_t\overset{\text{i.i.d}}{\sim}N(0,\sigma^2)$, $$ \text{Likelihood} = f_{y_1\mid\theta}(y_1)\left(\frac{1}{\sqrt{2\pi}\sigma}\right)^{n-1} \exp\left(-\frac{1}{2\sigma^2}\sum_{t=2}^n(y_t-\phi_0-\phi_1 y_{t-1})^2\right). \tag{5} $$ Only three assumptions went into this: the model equation, independence of $\epsilon_t$ from the past, and $\epsilon_t$ i.i.d. normal. Notice what was not assumed: nothing at all was said about the marginal distribution of $y_1$, because the model equation only constrains $t=2,\dots,n$ — it never puts $y_1$ on the left-hand side. So $f_{y_1\mid\theta}(y_1)$ sits in (5) unresolved, and what to do with it is the crux of the chapter.

Approach one: ignore the distribution of $y_1$¶

The simplest fix is to assume $f_{y_1\mid\theta}(y_1)$ does not depend on $\theta$ at all. It then becomes a constant factor, safe to drop since the likelihood is only needed up to proportionality: $$ \text{Likelihood} \;\propto\; \left(\frac{1}{\sqrt{2\pi}\sigma}\right)^{n-1} \exp\left(-\frac{1}{2\sigma^2}\sum_{t=2}^n(y_t-\phi_0-\phi_1 y_{t-1})^2\right). \tag{6} $$ This is exactly the ordinary-regression likelihood with $x_t=y_{t-1}$ — even though the route that produced it (chain rule plus noise independent of the past) is different from the route that produces the regression likelihood (covariate conditioning). Two consequences follow:

Bayesian inference is unchanged. With the same prior $\phi_0,\phi_1,\log\sigma \overset{\text{i.i.d}}{\sim}\text{unif}(-C,C)$, the posterior has exactly the regression form: $$ (\phi_0,\phi_1)\mid y_1,\dots,y_n \sim t_{n-3,2}!\left((\hat\phi_0,\hat\phi_1), \hat\sigma^2(X^TX)^{-1}\right), $$ where $\hat\phi_0,\hat\phi_1$ are just the least-squares estimates from regressing $y_t$ on $y_{t-1}$. The degrees of freedom are $n-3$, not $n-2$: the effective sample size is $n-1$ (since $t$ runs from $2$ to $n$) and $X$ has $2$ columns, so $(n-1)-2 = n-3$.
Frequentist inference is not. The MLEs of $\phi_0,\phi_1,\sigma$ under (6) coincide with the OLS estimates, since (6) is the regression likelihood. But finding the sampling distribution of those MLEs is a different problem than in ordinary regression, because $x_t=y_{t-1}$ is here a random quantity correlated with the rest of the series, not a fixed covariate. This has to be worked out asymptotically, as $n\to\infty$, using central limit theorems and laws of large numbers for dependent sequences (see Shumway and Stoffer, Time Series Analysis and Its Applications, 4th ed., Ch. 4 — referred to in the lecture but not derived there). The resulting inference is based on normal approximations, not exact $t$-distributions.

This is the first place Bayesian and frequentist inference genuinely diverge for AR models: Bayesian inference stays exact and no harder than ordinary regression, while frequentist inference needs asymptotic machinery just to describe the MLE's distribution. This is also why software that actually fits autoregressions — AutoReg in statsmodels, for instance — reports standard errors and $z$-scores rather than the $t$-scores a plain OLS regression of $y_t$ on $y_{t-1}$ would give: it is using the frequentist asymptotic formulas, not (6) treated as an exact regression likelihood. The two differ in principle, though usually the difference is negligible in practice.

Approach two: derive the distribution of $y_1$¶

A more careful fix is to work out what $f_{y_1\mid\theta}(y_1)$ actually should be, by imagining the model equation extended backward to times before the sample starts: $t = 1,0,-1,-2,\dots$. Substituting recursively, $$ \begin{aligned} y_1 &= \phi_0+\phi_1 y_0+\epsilon_1 \ &= \phi_0(1+\phi_1) + \phi_1^2 y_{-1} + \phi_1\epsilon_0+\epsilon_1 \ &= \phi_0(1+\phi_1+\phi_1^2) + \phi_1^3 y_{-2} + \phi_1^2\epsilon_{-1}+\phi_1\epsilon_0+ \epsilon_1, \end{aligned} $$ and continuing this way down to $t=-M$ for some large $M$, $$ y_1 = \phi_0\sum_{j=0}^M \phi_1^j + \phi_1^{M+1}y_{-M} + \sum_{j=0}^M\phi_1^j\epsilon_{1-j}. $$ This is not yet a distribution for $y_1$, since $y_{-M}$ is unknown. But if $|\phi_1|<1$, the coefficient $\phi_1^{M+1}$ shrinks geometrically as $M$ grows, so for large $M$ that term contributes almost nothing regardless of $y_{-M}$. Dropping it and letting $M\to\infty$, $$ y_1 \approx \phi_0\sum_{j=0}^\infty \phi_1^j + \sum_{j=0}^\infty\phi_1^j\epsilon_{1-j} = \frac{\phi_0}{1-\phi_1} + \sum_{j=0}^\infty\phi_1^j\epsilon_{1-j}. $$

The weight $\phi_1^j$ that the backward expansion places on the shock $\epsilon_{1-j}$, drawn for $\phi_1=0.7$. Because $|\phi_1|<1$ the weights decay geometrically, so truncating the recursion at a large $M$ and dropping the leftover $\phi_1^{M+1}y_{-M}$ term loses almost nothing — this is what lets $y_1$ be written as a convergent infinite sum of past shocks.

The remaining sum is an infinite sum of independent normal random variables, hence itself normal, with mean $0$ (each $\epsilon_{1-j}$ has mean $0$) and variance $$ \text{var}\left(\sum_{j=0}^\infty\phi_1^j\epsilon_{1-j}\right) = \sum_{j=0}^\infty\phi_1^{2j}\,\text{var}(\epsilon_{1-j}) = \sigma^2\sum_{j=0}^\infty\phi_1^{2j} = \frac{\sigma^2}{1-\phi_1^2}, $$ using $|\phi_1|<1$ so the geometric series converges. Hence $$ y_1 \sim N!\left(\frac{\phi_0}{1-\phi_1}, \frac{\sigma^2}{1-\phi_1^2}\right), \qquad f_{y_1\mid\theta}(y_1) = \frac{\sqrt{1-\phi_1^2}}{\sqrt{2\pi}\sigma} \exp\left(-\frac{1-\phi_1^2}{2\sigma^2}\left(y_1-\frac{\phi_0}{1-\phi_1}\right)^2\right). $$ Plugging this into (5) gives the full likelihood $$ \text{Likelihood} = \frac{\sqrt{1-\phi_1^2}}{\sqrt{2\pi}\sigma} \exp\left(-\frac{1-\phi_1^2}{2\sigma^2}\left(y_1-\frac{\phi_0}{1-\phi_1}\right)^2\right) \left(\frac{1}{\sqrt{2\pi}\sigma}\right)^{n-1} \exp\left(-\frac{1}{2\sigma^2}\sum_{t=2}^n(y_t-\phi_0-\phi_1 y_{t-1})^2\right). \tag{7} $$ This is more complicated than (6), and it is valid only when $|\phi_1|<1$ — outside that range the geometric series it rests on simply does not converge, and the whole derivation collapses.

Two different AR(1) models¶

The two approaches to $f_{y_1\mid\theta}(y_1)$ are not just two computational shortcuts to the same target — they amount to two different definitions of what "the AR(1) model" is.

Model A (conditional model, likelihood (6)): $$ y_t = \phi_0+\phi_1 y_{t-1}+\epsilon_t \ \ (t=2,\dots,n), \qquad \epsilon_t \text{ independent of } y_1,\dots,y_{t-1}, \qquad \epsilon_t\overset{\text{i.i.d}}{\sim}N(0,\sigma^2), $$ with $y_1$ simply treated as a fixed constant — its distribution is never modelled.

Model B (stationary model, likelihood (7)): take the backward expansion to its limit and define, for every integer $t$ (not just $t=2,\dots,n$), $$ y_t = \frac{\phi_0}{1-\phi_1} + \sum_{j=0}^\infty\phi_1^j\epsilon_{t-j}, \qquad t = \dots,-2,-1,0,1,2,3,\dots, \tag{8} $$ with $\epsilon_t\overset{\text{i.i.d}}{\sim}N(0,\sigma^2)$. Substituting $y_1$'s formula into $y_2 = \phi_0+\phi_1 y_1+\epsilon_2$ gives $$ y_2 = \phi_0+\phi_1\left(\frac{\phi_0}{1-\phi_1}+\sum_{j=0}^\infty\phi_1^j\epsilon_{1-j}\right)+ \epsilon_2 = \frac{\phi_0}{1-\phi_1}+\sum_{j=0}^\infty\phi_1^j\epsilon_{2-j}, $$ i.e. exactly formula (8) shifted to $t=2$; induction extends this to every $t$. So (8) is self-consistent: it automatically satisfies the AR equation $y_t=\phi_0+\phi_1 y_{t-1}+\epsilon_t$, and automatically makes $\epsilon_t$ independent of $y_{t-1},y_{t-2},\dots$ — both of the assumptions that Model A had to state separately fall out of the single formula (8) for free.

Model B is only well-defined when $|\phi_1|<1$: the infinite sum $\sum_j\phi_1^j\epsilon_{t-j}$ fails to converge when $|\phi_1|\ge1$, so the right-hand side of (8) makes no sense outside that range. Model B is the chapter's first example of a stationary model — the same distributional formula generates every $y_t$, with no special role for a starting time — and working out exactly what stationarity buys, and requires, is the subject of the next lecture.

Sources¶

Berkeley STAT 153, Fall 2025, Lecture Seventeen (Aditya Guntuboyina, October 28, 2025):
§1 Parameter Estimation in AutoRegressive Models — the AR($p$)-to-regression setup and matrix form.
§2 Detour: usual linear regression — the four assumptions behind the ordinary regression likelihood, and the Bayesian/frequentist asymmetry.
§3 Likelihood for AR(1) — the chain-rule derivation of (5), and both approaches (§3.1, §3.2) to $f_{y_1\mid\theta}(y_1)$, giving likelihoods (6) and (7).
§4 Two AR(1) Models — Model A and Model B (equation (8)), and the note that (8) requires $|\phi_1|<1$.
Referred to but not contained in this lecture: the asymptotic (frequentist) theory of AR parameter estimation, attributed to Shumway, R. H. and D. S. Stoffer, Time Series Analysis and Its Applications: With R Examples, 4th ed. (Springer), Ch. 4; the posterior for $\sigma$ in ordinary regression, said to have been derived in "Homework One"; and the $t$-posterior for regression coefficients, said to have been derived "previously" in the course.
The four source files are machine reconstructions of a PDF lecture-notes deck with no extractable text layer; the source material itself flags every displayed equation as unverified. Equations above have been checked for internal consistency (e.g. the degrees-of-freedom count, the induction step from $y_1$ to $y_2$) but not against the original PDF pages.

← 111. The Sinusoidal Model · Contents · 113. Broken-Stick Regression →