One natural question we should ask is whether stocks can be grouped by how they behave, rather than just by the sector labels someone assigned them decades ago. For example, here’s one of many things sector labels don’t capture: a tech company and a utility might sit in completely different industries, yet move in near-perfect lockstep during a market downturn. That’s why in this article, we’ll use KK-mean clustering to classify a universe of stocks based on their return and volatility characteristics. Of course, there are likely more robust ways of choosing factors for the task, e.g., including other features such…

Predicting asset returns is a difficult game where the usual rules of regression are stress-tested by noisy signals, correlated features, and the ever-present risk of overfitting to market microstructure. While ordinary least squares (OLS) gives us a clean starting point, penalized regression methods like Ridge, LASSO, and Elastic Net offer a principled way to impose structure on that noise. Note that due to the violation of assumptions, the performances of the models will suck a lot. The goal is therefore to understand how regularization changes what a model learns, what it discards, and how we should think about bias-variance tradeoff…

In the previous article, we introduced Hidden Markov Models (HMMs) as a way to capture volatility regime-switching in SPY returns. By decomposing returns into distinct states, i.e., low, medium, and high volatility, we’re able to uncover meaningful structure that a single continuous model like GARCH could not explicitly represent. However, HMMs assume that the market operates in a finite set of discrete regimes, forcing what is inherently a continuous process into a small number of fixed states. In reality, volatility and other latent dynamics tend to evolve smoothly over time, exhibiting gradual transitions rather than abrupt switches. To explore this…

Previously, we saw that modeling variance, rather than the mean, provides a much more effective way of capturing financial time series dynamics. GARCH models, in particular, are able to reproduce volatility clustering and persistence, making them a strong baseline for volatility modeling. However, GARCH comes with an important structural assumption, which is that volatility evolves smoothly as a deterministic function of past shocks and past variance. In reality, financial markets often appear to operate in distinct regimes, with periods of sustained low volatility punctuated by sudden transitions into high-volatility environments, such as during crises. So we simply ask: what if…

In the previous article, we fit AR, MA, ARMA, and ARIMA models to SPY log returns and watched them systematically fail in a very specific way. The residuals showed volatility clustering, the QQ plots showed fat tails, and ACF plot on squared residuals confirmed that the variance itself was autocorrelated. ARIMA models the conditional mean. The problem is in the conditional variance. That’s what ARCH and GARCH are built for, and that’s we’ll explore today. *** The data we’ll be using here is exactly the same as what we’ve used in the previous article — SPY log returns from the…

This is (hopefully) a beginner-friendly tutorial in attempting to model SPY data using linear time series models. Specifically, we take a look at basic properties of the data, the fitness of AR, MA, ARMA, and ARIMA models, and, spoiler alert, why they suck at the job. *** A good refresher from my article on linear time series models reminds us that unlike static data, the error term ε\varepsilon in time series data exhibits autocorrelation, which violates the assumption of independent ε\varepsilon for static models. Therefore, we use time series models because they’re capable of explicitly removing autocorrelation from the error…