Lecture notes. Written from the material of Berkeley Stat 153, licensed CC BY 4.0. These are notes, not a transcript: the material has been reorganised and rewritten. This adaptation carries the same licence, and the original is linked above.

91. LSTM Forecasting for Time Series¶

What this covers¶

The lab asks what to do when a series is generated by a nonlinear recursion that a linear autoregressive model cannot represent, and answers it by fitting a recurrent neural network — an LSTM — instead. It assumes the reader already knows AR($p$) fitting and forecasting, and the usual way of judging a forecast by an error metric such as RMSE; it introduces the LSTM architecture, how it is trained and rolled forward for multi-step forecasting, when it beats AR and when it does not, and closes with both a simulated example and the classic Airline Passengers series.

A nonlinear benchmark¶

To see where AR breaks down, the lab first builds data for which the true model is known and is not linear. A series $y_t$ is simulated from

\[ y_t = 0.9 \sin(0.5\pi\, y_{t-p}) + \epsilon_t, \qquad t > p, \]

with $p = 85$ (called truelag in the code), Gaussian noise of standard deviation $0.01$, and the first $p$ values filled in linearly just to start the recursion off. Because $g(y) = 0.9\sin(0.5\pi y)$ takes values only in $[-0.9, 0.9]$, the whole noiseless series is trapped in that band no matter how far it runs — a fact that matters once linear forecasts are compared against it.

Since the generating function $g$ is known here (it normally would not be), the lab computes an oracle forecast: iterate the true recursion $y_t = g(y_{t-p})$ forward with no noise, using the already-observed values as the seed. This is not a model anyone could fit from data alone — it uses the true $g$ and the true $p$ — so it serves only as a benchmark: the best any method (AR, LSTM, RNN, GRU) could hope to match, since the noise itself is unpredictable.

Fitting AR(p) to nonlinear data¶

Fitting AutoReg with the correct lag $p = 85$ still fails, because the relationship between $y_t$ and $y_{t-p}$ is a bounded sine wave and AR(p) can only represent a linear one. The forecast "expands" relative to the oracle: unlike $g$, a linear recursion has no reason to stay inside $[-0.9, 0.9]$, so as the forecast horizon grows the AR(p) predictions drift away from the bounded true dynamics. Numerically, over the $1000$-step forecast horizon,

\[ \text{RMSE}_{\text{AR}} = 0.182, \qquad \text{median abs. error}_{\text{AR}} = 0.143. \]

The LSTM model¶

The alternative is a recurrent network that does not assume linearity. The lab uses a single-layer LSTM with one input feature (the previous value of the standardized series) and a hidden state of size $n_h$, followed by a linear layer mapping the hidden state back down to a scalar prediction at every time step — a "many-to-many" architecture, since a prediction is produced at each step of the input sequence, not just at the end.

In the code's own notation, the LSTM's hidden state $h_t$ is written $r_t$ and its cell state $c_t$ is written $s_t$. At each step the network reads the previous value $y_{t-1}$, updates $(r_t, s_t)$ from $(r_{t-1}, s_{t-1})$ and $y_{t-1}$, and produces $\hat y_t$ from $r_t$ through the linear layer. Unlike AR(p), which is told explicitly which $p$ past values to use, the LSTM's only handle on the past is whatever it compresses into the fixed-size hidden state — so $n_h$ plays a role loosely analogous to $p$, except the network decides for itself what to remember rather than being given a lag structure.

Training. The series is standardized (subtract the mean, divide by the standard deviation) before fitting, since neural networks train more reliably on data of order one. With batch size 1, the whole series is one training sequence: the network is trained to predict $y_t$ from $y_{t-1}$ at every $t$ simultaneously, minimizing mean squared error between the predicted sequence and the true one-step- ahead sequence, by full-batch gradient descent (Adam, 1000 epochs).

Forecasting. To forecast beyond the observed data, the network first runs once over the entire observed (standardized) series to obtain its final hidden and cell state — this is the "memory" it has accumulated. From there it forecasts recursively: feed the last observed value in, read off the next prediction, then feed that prediction back in as the next input, carrying the hidden and cell state forward at each step. This is exactly the same "generate one step at a time" logic that multi-step AR forecasting uses, but the state being carried forward is the learned hidden state rather than a window of $p$ raw lagged values.

Multi-step LSTM forecasting: the network's own prediction becomes the next input, while the hidden and cell state accumulated from the observed series is carried forward step by step.

Comparing AR and LSTM forecasts¶

With $n_h = 40$, the LSTM forecast tracks the oracle visibly better than AR(85) does over the same 1000-step horizon — yet its RMSE is worse:

\[ \text{RMSE}_{\text{AR}} = 0.182, \qquad \text{RMSE}_{\text{LSTM}} = 0.205, $$ $$ \text{median abs. error}_{\text{AR}} = 0.143, \qquad \text{median abs. error}_{\text{LSTM}} = 0.023. \]

The reconciliation is that $g$ has sharp transitions (steep parts of the sine), and RMSE — squaring errors — is dominated by a few large misses right at those transitions, even if the LSTM tracks the function well almost everywhere else. A more robust metric, such as the median absolute error, tells the more accurate story here: by that measure the LSTM is roughly six times better than AR. The general point survives beyond this example: which error metric is used can flip which model looks better, particularly when the truth has occasional sharp features.

The lab also flags that LSTM fits of this kind can be unstable: rerunning the same simulation with sig_noise changed from $0.01$ to $0$ or to $0.02$ changes the LSTM's forecast noticeably, which is worth keeping in mind when a single fit looks good.

Trend breaks the LSTM forecast¶

LSTM models of this kind do not extrapolate a trend well. The lab adds a linear drift $0.001 t$ to the same nonlinear simulated series and refits both models. AR(p), with the correct lag, forecasts this trending series fine. The LSTM, trained and rolled forward exactly as before, instead produces forecasts that flatten out as the horizon grows — it fails to keep extending the trend into the future. The lesson drawn is procedural: detrend before fitting an LSTM — e.g. fit a line to the series and work with the residuals — rather than expecting the network to learn to extrapolate a trend on its own.

Case study: airline passengers¶

The same recipe is applied to the classic Airline Passengers series — monthly counts of international airline passengers (in thousands), January 1949 to December 1960, $n = 144$ months, with a clear upward trend. The pipeline is:

Fit $y_t = \beta_0 + \beta_1 t + \text{residual}$ by ordinary least squares and take the residuals as the detrended series.
Standardize the residual series and fit an LSTM to it (here with a larger hidden size, $n_h = 200$), trained exactly as above by full-batch Adam over 1000 epochs.
Forecast the detrended residuals 60 months (five years) ahead by the same recursive feed-back procedure.
Undo both transformations: unstandardize the residual forecast, then add back the fitted linear trend $\hat\beta_0 + \hat\beta_1 t$ evaluated at the future time points, to get a forecast on the original passenger-count scale.

This mirrors the previous section's conclusion directly: the trend is handled by the simple linear model, and the LSTM is only asked to model what is left over — the detrended, presumably more nonlinear, residual structure.

Sources¶

All content, code, equations, and numerical results (loss values, RMSE, median absolute error) are from the converted Lab 14 notebook, berkeley-stat153, spring 2025 (Lab14.ipynb, CC BY 4.0), split across:
Introduction — the nonlinear simulation, oracle forecast, and AR(p) baseline
LSTM — the network definition, training loop, and recursive forecasting, compared against AR
Presence of Trend — the detrending experiment
Airline Passengers Dataset — the real-data case study
No slide deck or lecture transcript was supplied for this session; the material is a Jupyter notebook lab, and its markdown prose and code comments are the only exposition available. Figures referenced in the notebook (the simulated series, the forecast-comparison plots) were not reproduced in the converted source and are described here only insofar as the surrounding text and code make their content explicit.
RNN and GRU are named in the notebook as further methods to be compared against the oracle benchmark, but the supplied material does not contain code fitting either, only LSTM and AR.

← 90. Fitting MA(1)/AR(1) via PyTorch · Contents · 92. Change of Variables and MLE →