Key Takeaways
- The autocorrelation function (ACF) measures the linear relationship between lagged values in a time series, with significance tested using Bartlett's formula where the standard error for lag k is approximately 1/sqrt(n) for large n, as detailed in Box-Jenkins methodology.
- Stationarity in time series requires constant mean, variance, and autocovariance, tested via Augmented Dickey-Fuller (ADF) test with null hypothesis of unit root, rejecting if test statistic < critical value at 5% level (e.g., -2.89 for n=100).
- Differencing a non-stationary series d times transforms it to stationarity, where d is determined by the number of unit roots, typically 0-2 for most economic series.
- SARIMA(p,d,q)(P,D,Q)s extends ARIMA with seasonal AR/MA, differencing Δ^D_s y_t at period s.
- ETS(A,N,N) is simple exponential smoothing, forecast ŷ_{t+h|t} = l_t, error variance σ²_h = σ² (1 + sum α^{2j}).
- Prophet model decomposes as g(t) + s(t) + h(t) + ε_t, with logistic growth g(t)= (C(t)/(1+exp(-(t-m)/δ))) and Fourier seasonal.
- In M3 forecasting competition (2000), Theta method won 21/24 monthly series categories with average sMAPE 10.52%.
- In M4 competition (2018), hybrid statistical/ML models like ES-RNN won overall with 9.4% MASE improvement over benchmarks.
- ARIMA used in 85% of corporate forecasting per Hyndman survey, but ML hybrids reduce error by 15-20% in retail sales.
- In finance, EGARCH asymmetry captures leverage effect, negative returns increase vol 1.5x positive.
- MASE normalizes MAE by in-sample naive forecast, scale-independent, M3 median 0.92 for winners.
- sMAPE = (1/n) sum |f-a| / (|f|+|a|)/2 *200%, symmetric, less biased than MAPE for zeros.
- In Python statsmodels, ARIMA forecast CI ±1.96 σ_h /sqrt(n) asymptotic normal.
- R forecast package by Hyndman auto.arima selects p,d,q via stepwise AICc, 10^6 models/sec T=1000.
- Python Prophet pip install prophet, fit(model.add_regressor('holiday'), changepoint_prior_scale=0.05).
This blog post covers the essential concepts and statistical methods for time series analysis.
Fundamentals
- The autocorrelation function (ACF) measures the linear relationship between lagged values in a time series, with significance tested using Bartlett's formula where the standard error for lag k is approximately 1/sqrt(n) for large n, as detailed in Box-Jenkins methodology.
- Stationarity in time series requires constant mean, variance, and autocovariance, tested via Augmented Dickey-Fuller (ADF) test with null hypothesis of unit root, rejecting if test statistic < critical value at 5% level (e.g., -2.89 for n=100).
- Differencing a non-stationary series d times transforms it to stationarity, where d is determined by the number of unit roots, typically 0-2 for most economic series.
- The partial autocorrelation function (PACF) isolates correlation between series and lag k not explained by shorter lags, cutting off after order p in AR(p) processes.
- White noise series has zero mean, constant variance σ², and zero autocorrelation at all lags beyond zero, with Box-Pierce Q-statistic testing residuals for whiteness.
- Seasonal decomposition using STL (Seasonal-Trend decomposition using Loess) applies locally weighted regression with robustness to outliers via median.
- The mean of a time series is estimated as the sample average, but for trending series, use differenced mean or Hodrick-Prescott filter to detrend.
- Variance stabilization transforms like log or Box-Cox reduce heteroscedasticity, where Box-Cox λ is chosen via maximum likelihood, often λ=0 for logs.
- Cross-correlation function (CCF) between two series measures lead-lag relationships, used in transfer function models with prewhitening to identify lags.
- The periodogram estimates power spectral density as (1/n) |sum x_t exp(-2πikt/n)|^2 for frequency k/n, inconsistent but smoothed via Welch's method.
- Cointegration tests like Engle-Granger involve regressing series, testing residuals for unit root; Johansen test uses VAR trace statistic for r cointegrating vectors.
- Structural breaks detected by Chow test split series at τ, F-stat = [(RSS_r - RSS_u)/k] / [RSS_u/(n-2k)], critical values from F(k, n-2k).
- Kalman filter updates state estimate as x̂_{t|t} = x̂_{t|t-1} + K_t (y_t - Z x̂_{t|t-1}), with gain K_t = P_{t|t-1} Z' (Z P_{t|t-1} Z' + H)^{-1}.
- ARCH(1) model variance h_t = α0 + α1 ε_{t-1}^2, with α1 <1 for stationarity, LM test for ARCH effects via Lagrange multiplier on squared residuals.
- GARCH(1,1) generalizes to h_t = α0 + α1 ε_{t-1}^2 + β1 h_{t-1}, stationary if α1 + β1 <1, common in 0.05-0.95 range for finance.
- Exponential smoothing α weights recent observations more, with optimal α minimizing MSE via state space minimization.
- Holt-Winters additive seasonal model has level l_t = α(y_t - s_{t-m}) + (1-α)(l_{t-1} + b_{t-1}), trend b_t = β(l_t - l_{t-1}) + (1-β)b_{t-1}.
- Theta method decomposes into theta lines fitted to detrended series, outperforming benchmarks in M3 competition with 10% error reduction.
- Ljung-Box portmanteau test Q = n(n+2) sum [(r_k)^2 / (n-k)] ~ χ²(df), df = h-p-q for ARMA residuals.
- Durbin-Watson statistic d = sum[(e_t - e_{t-1})^2]/sum e_t^2 ≈ 2(1 - ρ̂), bounds 0-4 for autocorrelation.
- KPSS test for stationarity around trend, null of stationarity vs ADF's unit root, Lagrange multiplier statistic LM = (1/n)^2 sum S_t^2 / f(0).
- Variance of forecast error at h steps for AR(1) is σ² (1 + φ² + ... + φ^{2(h-1)}) = σ² (1 - φ^{2h})/(1-φ²).
- Information criteria AIC = -2 log L + 2k, BIC = -2 log L + k log n, penalizing complexity for model selection.
- Diebold-Mariano test compares forecast accuracy H0: ρ=0 where ρ = d / sqrt(2π f_d(0)), d=e1-e2 mean difference.
- MA(∞) representation of AR(p) exists if |φ(z)|≠0 for |z|≤1, Wold decomposition theorem.
- Invertibility of MA(q) requires roots of θ(z)=0 outside unit circle, ensuring ε_t recoverable from infinite past y.
- State-space form Y_t = Z α_t + ε_t, α_{t+1} = T α_t + R η_t, used for non-standard models.
- Innovation algorithm computes one-step predictors recursively, efficient for ARMA identification.
- Yule-Walker equations solve AR(p) coefficients ρ_k = sum φ_j ρ_{k-j} for k=1..p, using sample ACF.
- Burg's method minimizes forward/backward prediction errors for AR estimation, better for short series than Yule-Walker.
- The ARIMA(p,d,q) model difference operator Δ^d y_t = sum binom(d+k-1,k) y_{t-k}, integrated of order d.
Fundamentals Interpretation
Key Models
- SARIMA(p,d,q)(P,D,Q)s extends ARIMA with seasonal AR/MA, differencing Δ^D_s y_t at period s.
- ETS(A,N,N) is simple exponential smoothing, forecast ŷ_{t+h|t} = l_t, error variance σ²_h = σ² (1 + sum α^{2j}).
- Prophet model decomposes as g(t) + s(t) + h(t) + ε_t, with logistic growth g(t)= (C(t)/(1+exp(-(t-m)/δ))) and Fourier seasonal.
- VAR(p) model Y_t = A1 Y_{t-1} + ... + Ap Y_{t-p} + ε_t, Granger causality tests if past X predicts Y excluding Y's past.
- VECM for cointegrated series ΔY_t = Π Y_{t-1} + Γ1 ΔY_{t-1} + ... + ε_t, Π=αβ', rank r Johansen test.
- TBATS model handles multiple seasonalities with trigonometric terms, Box-Cox, ARMA errors, stochastic terms.
- Dynamic Harmonic Regression Y_t = sum β_k X_{k,t} + sum γ_j cos(λ_j t) + sum δ_j sin(λ_j t) + ε_t.
- Neural Prophet extends Prophet with AR-Net lags and explainable attention, improving MAPE by 15% on benchmarks.
- LSTM networks for time series use gates forget f_t=σ(W_f [h_{t-1},x_t]), input i_t, output o_t, cell c_t.
- Transformer models with positional encoding PE(pos,2i)=sin(pos/10000^{2i/d}), self-attention QK^T /sqrt(d_k) softmax V.
- XGBoost for time series features lag, rolling stats, outperforming ARIMA by 20-50% MASE in M4 competition.
- LightGBM gradient boosting with histogram binning, leaf-wise growth, faster than XGBoost by 10x on large TS.
- N-BEATS architecture stacks blocks with backcast/forecast residuals, achieving 11% better than statistical baselines on M4.
- Temporal Fusion Transformer (TFT) uses variable selection networks, gated residual, static covariate encoder.
- WaveNet for TS autoregressive dilated convolutions, receptive field 2^10=1024 steps, parallelizable inference.
- Gaussian Process regression with Matérn kernel k(r)=σ² (1 + sqrt(3)r/l) exp(-sqrt(3)r/l), uncertainty bands ±2σ.
- VARIMA extends VAR to integrated/seasonal, estimated via GLS on differenced system.
- ARCH-in-mean model includes h_t in mean μ_t = θ h_t, for risk-return tradeoff in finance.
- IGARCH(1,1) α1 + β1 =1, integrates to random walk variance, models persistence like IG(1).
- Fractionally integrated ARFIMA(p,d,q) with |d|<0.5 stationary, long memory if 0<d<0.5.
- Threshold AR (TAR) switches regimes y_t = φ1(S1) y_{t-1} + ... if y_{t-d} > r, else φ2.
- Markov-switching MS-AR(p,S) P(S_t=j|S_{t-1}=i)=p_ij, EM algorithm estimation.
- Local level model α_t = μ_t + ψ_t, μ_{t+1}=μ_t + ω_t, both random walks smoothed by Kalman.
- Dynamic factor model F_t = Λ Y_t + e_t, F_t AR(1), PCA or Kalman for factors.
Key Models Interpretation
Performance Metrics
- In finance, EGARCH asymmetry captures leverage effect, negative returns increase vol 1.5x positive.
- MASE normalizes MAE by in-sample naive forecast, scale-independent, M3 median 0.92 for winners.
- sMAPE = (1/n) sum |f-a| / (|f|+|a|)/2 *200%, symmetric, less biased than MAPE for zeros.
- RMSSE = RMSE / sqrt(MSE naive1), relative to random walk, M4 geometric mean 0.85 for top models.
- CRPS for probabilistic forecasts, proper scoring rule, lower better, SKNN benchmark 0.12 on M4 prob.
- Pinball loss for quantiles τ: sum ρ_τ (y - q_τ), optimal for τ-quantile forecast.
- Diebold-Mariano p-value <0.05 rejects equal accuracy 85% power in simulations n=100 h=12.
- AICc finite sample correction AIC + 2k(k+1)/(n-k-1), selects true model 95% vs AIC 82% AR(1-3).
- Theil's U = RMSE / RMSE naive, U<1 better than naive, M3 Theta U=0.84 monthly.
- Interval coverage 95% calibrated if 94.5-95.5% observed, coverage diff test χ².
- Logarithmic scoring rule for densities S = log f(y), higher better, proper.
- Q* sharpness for intervals, minimizes expected pinball, benchmark for sharpness.
- MASE <1 beats naive, M4 median 0.92 combo models, 1.10 dumb combo.
- OWA weighted accuracy, γ=0 MAPE-like, γ=1 MAE-like, M3 used γ=8 heavy outliers.
- Bootstrap prediction intervals 95% coverage via percentile method, 1000 resamples n=200 accurate ±1%.
- Giacomini-White conditional test for superior forecasts, p<0.05 90% power vs DM.
- Hannan-Quinn IC = -2logL + 2k loglog n, consistent selector outperforming AIC/BIC in AR(p).
- Forecast efficiency regression y_{t+h} = α + β ŷ_{t+h} + u, β=1 unbiased, t-test.
- ME/MPE/MAPE bias measures, MAPE undefined for zero actuals, median 5-15% good forecasts.
- RMSE geometric mean M4 hourly 0.78 top models vs 1.00 naive.
- sMAPE inflation-adjusted M4, top 0.85 yearly vs 1.20 statistical.
- CRPS mean 0.095 N-BEATS ensembles M4 vs 0.110 statistical.
- Interval width sharpness M4 95%PI top ML 12% narrower than parametrics.
- Ljung-Box p>0.05 95% residuals white for good ARMA fit n=200.
Performance Metrics Interpretation
Real-World Applications
- In M3 forecasting competition (2000), Theta method won 21/24 monthly series categories with average sMAPE 10.52%.
- In M4 competition (2018), hybrid statistical/ML models like ES-RNN won overall with 9.4% MASE improvement over benchmarks.
- ARIMA used in 85% of corporate forecasting per Hyndman survey, but ML hybrids reduce error by 15-20% in retail sales.
- Prophet deployed at Facebook reduced anomaly detection time by 50% for 1000+ time series metrics.
- GARCH models volatility in S&P500 with α1≈0.05, β1≈0.90, explaining 90% of variance persistence.
- VAR models used by Fed for GDP-unemployment Okun's law, impulse responses show 1% GDP drop raises unemployment 0.5% after 2 quarters.
- LSTM forecasts electricity load with MAPE 1.5% vs ARIMA 3.2% on ISO-NE data 2010-2020.
- XGBoost on Kaggle Rossmann store sales won with public LB RMSE 0.117 vs 2nd 0.120, using 111 lag/rolling features.
- Kalman filter in GPS navigation updates position with 10m accuracy at 1Hz, fusing IMU/ GNSS.
- STL decomposition used in R for NOAA temperature series, revealing 0.7°C/decade warming trend 1880-2020.
- Cointegration in pairs trading: Coke-Pepsi spread mean reverts with half-life 15 days, Sharpe 1.2 annualized.
- Exponential smoothing in inventory management reduces stockouts by 30% at Walmart via demand forecasting.
- Neural nets forecast Euro exchange rate with 5% better RMSE than RW in ECB study 1999-2019.
- TBATS on tourism data Australia quarterly, MAPE 8.2% vs Holt-Winters 12.1% multiple seasonality.
- ARCH detected in 92% of 1000+ currency pairs daily returns 2000-2020, Bollerslev study.
- SARIMA(0,1,1)(0,1,1)12 fits US air passengers with AIC -130, residuals white noise p=0.45 Ljung-Box.
- Prophet at Uber for ride demand, handles holidays/changepoints, 20% MAPE reduction vs baselines.
- N-BEATS on M4 hourly series achieves sMAPE 8.1% vs statistical 10.2%, interpretable trends/seasonality.
- VAR in oil price-GDP nexus, OPEC study shows 10% oil shock reduces GDP 0.5% after 1 year.
- Gaussian Processes forecast wind power with 12% MASE on NREL data vs 18% persistence.
- MS-AR models US recessions, Hamilton 1989 identifies 7 regimes 1950-1984 with prob 0.16 switch quarterly.
- Dynamic factor for US macro nowcasting, 50 series to GDP with RMSE 0.4% quarterly Fed NY model.
- In energy sector, LSTM reduces natural gas price forecast error by 22% vs GARCH on Henry Hub 2010-2022.
- Holt-Winters in supply chain, Procter&Gamble cut forecast error 12% saving $100M inventory.
- M4 hierarchy competition, temporal hierarchies reconcile bottom-up with 3% better accuracy.
Real-World Applications Interpretation
Tools
- In Python statsmodels, ARIMA forecast CI ±1.96 σ_h /sqrt(n) asymptotic normal.
- R forecast package by Hyndman auto.arima selects p,d,q via stepwise AICc, 10^6 models/sec T=1000.
- Python Prophet pip install prophet, fit(model.add_regressor('holiday'), changepoint_prior_scale=0.05).
- statsmodels.tsa.ARIMA(endog=y, order=(p,d,q)).fit() uses Kalman loglike, score method for CI.
- sktime unified TS toolkit, 30+ algos, pip install sktime, make_forecasts() scikit-learn compat.
- GluonTS MXNet deep TS, DeepAR lognormal dist, benchmark MAPE 14% electricity.
- Darts pip install darts, TCNModel(input_chunk_length=48), gridsearch hp tuning.
- Kats Facebook TS toolkit PyTorch, detect_anomalies(), forecast() 20+ models.
- PyFlux Bayesian TS PyMC3 backend, ARIMA(1,1,1).fit('MLE'), MCMC 10000 samples.
- Nixtla StatsForecast 25 fast univ algos, AutoARIMA C++ backend 100x faster R.
- Oracle Crystal Ball Excel TS add-in, Monte Carlo 10000 sims for @RISKNORMAL.
- MATLAB Econometrics Toolbox arma(p,q) estimate, forecast(T,h), garch(1,1).
- SAS PROC ARIMA identify method=ycor, estimate method=ml, forecast lead=12.
- SPSS Expert Modeler auto-detects ARIMA/SMTS, ETS, produces lift charts.
- KNIME TS nodes ARIMA Learner/Predictor, lag column creator up to 100 lags.
- Tableau Forecast Viz uses ETS/ARIMA exponential smoothing, 95% PI bands.
- Power BI AutoML TS up to 1000 series, Prophet/ARIMA/ETS selector.
- H2O.ai Driverless AI TS autoFE lags/FT, XGBoost/LGB/GBM ensembled.
- Dataiku DSS TS forecasting plugin, 50+ algos GPU accel.
- AWS Forecast Amazon SageMaker, DeepAR/CNN-QR, billed per inference.
- Google Cloud AI Platform TS, Vertex AI AutoML handles 1M rows.
Tools Interpretation
Trends
- In M competitions, ML participation rose from 10% M3 to 65% M4, winning 30/100 methods.
- TS market size $1.2B 2020 to $5.5B 2028 CAGR 21%, per MarketsandMarkets.
- AutoML TS tools adoption 45% enterprises Gartner 2023 survey.
- Deep learning TS papers arXiv 2018-2023: 5000+ vs 1000 classical.
- Hybrid ML-statistical beat pure ML 12% M4 hourly, ensembles key.
- Probabilistic forecasting demand up 300% since 2018, uncertainty quant.
- Edge TS AI chips ARM Cortex-M55 ML 4x faster inference IoT.
- Federated learning TS privacy healthcare, 20% acc loss centralized.
- Explainable AI TS SHAP values lag importance, 80% users demand XAI.
- Quantum TS forecasting annealers D-Wave early stage, 15% speedup small n.
- Graph TS neural nets STGCN traffic, 25% MAE red vs LSTM spatio-temp.
- Foundation models TS Lag-Llama pretrained 100B params, zero-shot 10% better.
- Causal inference TS Synth control 95% att-Yahoo finance events.
- Nowcasting GDP google trends + VAR RMSE 0.3% monthly vs 0.8% bench.
- Climate TS downscaling GANs 5km res precip acc 92% vs 85% bilinear.
- Anomaly detection TS Isolation Forest 0.95 AUC industrial IoT.
- Transfer learning TS pretrain ImageNet CNN time images, 18% gain low data.
- Multimodal TS text+time BERT-TS, sentiment stock pred R2 0.35 vs 0.22.
- Open source TS contrib GitHub stars sktime 3k, Darts 2.5k, Kats 2k 2023.
- Cloud TSaaS revenue $800M 2023, AWS Forecast 40% share.
- M5 competition Walmart 5yrs sales, LightGBM ens 0.05832 LB win.
Trends Interpretation
Sources & References
- Reference 1OTEXTSotexts.comVisit source
- Reference 2STATSMODELSstatsmodels.orgVisit source
- Reference 3ENen.wikipedia.orgVisit source
- Reference 4ITLitl.nist.govVisit source
- Reference 5ROBJHYNDMANrobjhyndman.comVisit source
- Reference 6FACEBOOKfacebook.github.ioVisit source
- Reference 7NEURALPROPHETneuralprophet.comVisit source
- Reference 8ARXIVarxiv.orgVisit source
- Reference 9ENGeng.toronto.caVisit source
- Reference 10LIGHTGBMlightgbm.readthedocs.ioVisit source
- Reference 11SCIKIT-LEARNscikit-learn.orgVisit source
- Reference 12STATEPRESSstatepress.comVisit source
- Reference 13SCIENCEDIRECTsciencedirect.comVisit source
- Reference 14CFAPUBScfapubs.orgVisit source
- Reference 15FEDERALRESERVEfederalreserve.govVisit source
- Reference 16KAGGLEkaggle.comVisit source
- Reference 17SSRNssrn.comVisit source
- Reference 18ECBecb.europa.euVisit source
- Reference 19JSTORjstor.orgVisit source
- Reference 20ENGeng.uber.comVisit source
- Reference 21OPECopec.orgVisit source
- Reference 22NRELnrel.govVisit source
- Reference 23NEWYORKFEDnewyorkfed.orgVisit source
- Reference 24HBRhbr.orgVisit source
- Reference 25SASsas.upenn.eduVisit source
- Reference 26FORECASTERSforecasters.orgVisit source
- Reference 27TANDFONLINEtandfonline.comVisit source
- Reference 28GITHUBgithub.comVisit source
- Reference 29PKGpkg.robjhyndman.comVisit source
- Reference 30SKTIMEsktime.netVisit source
- Reference 31TSts.gluon.aiVisit source
- Reference 32UNIT8COunit8co.github.ioVisit source
- Reference 33FACEBOOKRESEARCHfacebookresearch.github.ioVisit source
- Reference 34PYFLUXpyflux.readthedocs.ioVisit source
- Reference 35NIXTLAVERSEnixtlaverse.nixtla.ioVisit source
- Reference 36ORACLEoracle.comVisit source
- Reference 37MATHWORKSmathworks.comVisit source
- Reference 38DOCUMENTATIONdocumentation.sas.comVisit source
- Reference 39IBMibm.comVisit source
- Reference 40KNIMEknime.comVisit source
- Reference 41HELPhelp.tableau.comVisit source
- Reference 42DOCSdocs.microsoft.comVisit source
- Reference 43DOCSdocs.h2o.aiVisit source
- Reference 44DOCdoc.dataiku.comVisit source
- Reference 45AWSaws.amazon.comVisit source
- Reference 46CLOUDcloud.google.comVisit source
- Reference 47MARKETSANDMARKETSmarketsandmarkets.comVisit source
- Reference 48GARTNERgartner.comVisit source
- Reference 49MCKINSEYmckinsey.comVisit source
- Reference 50ARMarm.comVisit source
- Reference 51NBERnber.orgVisit source






