I wanted to forecast daily volatility for a crypto-linked equity traded on the London Stock Exchange. Not direction; volatility. The goal was to produce a fair-value open and buy/sell levels each morning before the market opens, then get those delivered to my inbox as a structured briefing.

This post walks through the process end to end: gathering and cleaning data, building an evaluation framework, running experiments, selecting a champion model, and productionising the whole thing on AWS with daily notifications. The ML parts are more transferable than the specific instrument; the principles of walk-forward evaluation, scorekeeping, and simple-beats-fancy apply to most forecasting problems.

The instrument and what's predictable

The instrument is a blockchain-sector ETF. It tracks a basket of crypto-linked equities: miners, exchanges, infrastructure companies. Bitcoin is the primary driver. The fund has been trading since early 2022, so there's roughly four years of daily history to work with; not a lot, but enough to test a simple model.

The first useful thing established was that the trading day splits into two roughly equal halves that behave very differently:

  • The overnight gap: where it opens versus yesterday's close. This turns out to be largely mechanical; the open is a NAV catch-up to overnight crypto movement.
  • The intraday session range: how much it moves once open. This is the volatility component.

These are two separate prediction problems, not one. It was also confirmed early on that direction is noise for this instrument: you can forecast how much it will move, never which way. That shaped the whole project; the output is a range (fair-value open ± one daily-session sigma), never a directional call.

Data gathering and cleaning

The data comes from daily OHLCV bars (12 instruments including the target, Bitcoin, a US-listed peer ETF, and several crypto miners) plus 5-minute intraday bars for Bitcoin. Everything lives in S3 as typed Parquet files, appended daily by a Lambda that runs at 06:00 UTC.

The cleaning problem was more interesting than I expected. The ETF had severe data quality issues in its early history: 48% of trading days in 2022 had zero range (identical high and low), which is almost certainly bad data rather than a genuinely flat day. The 2023 data was noisy too. The instrument only became durably clean from late 2023 onwards.

The decision was to fix this at source rather than masking it in code. A one-time archival script moved the pre-clean data to an archive/ prefix in S3, and the retrain pipeline asserts clean data on every run. If bad data reappears, the pipeline fails loud rather than silently poisoning the model. This caught 63 zero-range days that would have dragged down the walk-forward evaluation.

Several of the miner stocks had similar issues: corporate pivots, SPAC mergers, and reused tickers meant the early history was for a fundamentally different company. Each instrument got a clean-start cutoff date, documented alongside the reason.

Building the scorekeeper before any model

Before fitting anything, we built a walk-forward evaluation harness and fixed the win conditions in advance. This is the part I think matters most and where most ML projects go wrong: deciding what counts as a win after you've seen the results.

The baseline is a simple EWMA (exponentially weighted moving average) forecast: "tomorrow's volatility is a smoothed version of recent volatility." To count as a genuine edge, a candidate model must:

  • Beat the EWMA bar by at least 5% on QLIKE (a loss function that penalises underestimating volatility harder than overestimating it, because being caught short is the costly mistake)
  • Pass a Diebold-Mariano statistical test (p < 0.10) confirming the win isn't luck
  • Hold up in every year, not just one lucky stretch

Pre-committing to these conditions is what makes the evaluation honest. Without them, it's too easy to fit a model, see a 3% improvement on one metric, and talk yourself into calling it a win.

The experiments

Three candidates were tested for the width (session range) model:

GARCH looked impressive at first: a 21% improvement over the baseline on QLIKE. But the improvement was fake. GARCH was systematically over-forecasting; it sat higher than reality, and QLIKE (which penalises under-forecasting more) rewarded that positioning. Once we corrected for the level trick, GARCH was actually worse than the simple baseline. Rejected.

This is a useful general lesson: a forecast's level and its shape are different things. A model that just predicts "high" all the time will score well on any loss function that penalises under-forecasting, without actually predicting anything useful.

Gradient-boosted trees (GBT), fed Bitcoin returns, miner returns, and a rich feature set, performed worse than the EWMA baseline. It overfit: chasing noise in the features rather than learning a durable signal. Yesterday's crypto and miner moves don't help predict today's range. Rejected.

HAR (Heterogeneous Autoregressive): yesterday's volatility, last week's average volatility, and last month's average volatility. Four parameters. It genuinely won: a modest but consistent improvement every year, beating the EWMA bar by roughly 28% on QLIKE (DM p ≈ 0.03). Champion.

Simple beat fancy. A 4-parameter linear model beat both GARCH and a feature-rich tree ensemble.

The overnight gap breakthrough

The centre model (predicting where the instrument opens) looked hopeless with daily data alone: an out-of-sample R² of about 0.12. The open seemed nearly random.

The breakthrough was timing. The instrument's overnight window straddles the middle of US trading hours (close at 16:30 London, reopen at 08:00). Daily Bitcoin bars can't line up with that window. Once we used 5-minute Bitcoin bars aligned to the exact overnight window, the out-of-sample R² jumped to roughly 0.55. The open tracks overnight Bitcoin almost one-for-one (coefficient ≈ 1.0, stable every year). It's a mechanical NAV catch-up that can be computed each morning from data available before the market opens.

Composing the levels

The two models compose into actionable levels:

fair-value open = previous close × exp(predicted gap), then buy/sell = fair-value open × exp(∓k·σ) where σ is the predicted daily-session volatility and k defaults to 1.0 (one-sigma bands). The bands can be tuned: wider for fewer but more confident signals, narrower for more active trading, and the two sides can be set asymmetrically.

HAR model Session range (width) Gap OLS Overnight gap (centre) Compose levels centre ± k·σ Daily levels FV open, buy, sell

Productionising on AWS

The production pipeline runs on AWS, deployed as a SAM/CloudFormation stack. Four Lambdas, scheduled via EventBridge, run each weekday morning before the London open:

  • 06:00 UTC: append yesterday's prices from yfinance to S3 (runs daily including weekends; crypto trades seven days a week)
  • 06:15 UTC: retrain both models on all available data, run walk-forward gates, promote the new artifact to S3 only if both gates pass
  • 06:30 UTC: produce today's forecast using the current artifact, write to DynamoDB
  • 06:40 UTC: run the briefing agent, deliver via SNS email
AWS (eu-west-1) EventBridge Append 06:00 UTC daily Retrain + gate 06:15 UTC weekdays S3 Parquet + artifacts Forecast 06:30 UTC weekdays DynamoDB Forecast rows Briefing agent 06:40 UTC weekdays SNS Email delivery

The retrain deserves some explanation. It re-fits both models on all available data every weekday, but only promotes the new artifact if it passes the walk-forward gate: the width model must beat the EWMA baseline by the pre-committed margin and pass the Diebold-Mariano test, and the centre model must clear an out-of-sample R² floor. If either gate fails, the current production artifact stays untouched. This means coefficients stay fresh (the models are recency-dominated, so new data matters) without any risk of regression.

Daily retrain runs in about 7 seconds on a 650-row history. The gate is the safety net that makes this cheap-and-frequent approach viable; without it, daily retraining would be reckless.

Each retrain result gets emailed via SNS with the new metrics, the delta versus the incumbent, and whether it was promoted or held. This makes it easy to spot drift or a model that's gradually losing its edge without having to check logs.

The briefing agent

The final Lambda in the chain is a briefing agent that produces a structured morning email. It follows a gather-compose-validate-deliver pipeline:

  • Gather: read the forecast row from DynamoDB and today's news articles from S3. The news is filtered by symbol relevance (Bitcoin and the fund's holdings score highest) and ranked so the model sees the most important stories first.
  • Compose: a single Bedrock call (Claude Sonnet) drafts a structured JSON briefing: forecast summary with the day's levels, top news articles with commentary, a direction call, and a forecast challenge section that pressure-tests the model's output against the news.
  • Validate: deterministic hard checks (do the price numbers match the forecast data? do the article links and titles exist in the source news?) plus an LLM soft check for grounding and consistency. Hard check failures always fail the briefing; the LLM can flag advisory notes but can't override a clean hard check. This prevents the validator from talking itself into a false negative.
  • Deliver: format and publish to SNS. If validation fails, the compose step retries once with the validation issues fed back as context. If it still fails, the briefing is sent anyway with an [UNVALIDATED] flag; a visible imperfect briefing is better than silence.

There's also a failure-mode handler: if the upstream forecast Lambda failed and no DynamoDB row exists, the briefing agent publishes a visible "forecast unavailable" warning rather than going dark. This was added after an incident where a failed forecast stayed hidden for a day because nothing was monitoring the gap.

Cost

The running cost is effectively zero. Four Lambdas running a few seconds each, PAY_PER_REQUEST DynamoDB writing roughly 250 rows a year, and about 100 MB of Parquet in S3. Everything fits within the free tier. The only meaningful cost is during development: SAM builds pull Docker images and the managed pandas layer weighs in at around 250 MB.

What I'd do differently

The research notebooks (01 through 07) evolved organically. If I were starting again I'd invest more upfront in making the walk-forward harness reusable across experiments; there's more copy-paste between notebooks than I'd like. The evaluation framework should have been a proper library from the start rather than growing into one.

I'd also be more aggressive about establishing data quality cutoffs early. The zero-range days in the early history weren't caught until they'd already dragged down several experiments. Fixing data at source rather than masking it in code was the right call, but I should have done more quality audit before any modelling, not during it.

What matters, briefly

You can forecast this instrument's daily range (modestly) and its open (well, because it's mechanically tied to overnight crypto), but never its direction. The discipline of pre-committing to win conditions, testing every model against the same bar, and requiring every-year consistency is what makes those claims credible. It correctly killed two impressive-looking models (GARCH and GBT) that were really just gaming the score or overfitting.

The one caveat worth keeping visible: this proves the forecasts are sound. It doesn't prove they would make money when traded because it doesn't include costs-and-slippage backtest results; that was the next question (and it was a no).