Towards Benchmarking Time Series Foundation Models on Native Scientific Data.

Benchmark structure

1. Observation sets, not matrices. Each sample, a plasma shot, ICU stay, or storm, is a set of tuples \((t_i, s_i, c_i, \mathbf{x}_i)\): continuous timestamp, signal type, channel, value. Scalars, 1D profiles, and 2D fields share one structure, missingness is the absence of an observation rather than an imputed value, and static metadata enters as pseudo-observations. Nothing is resampled on the way in.
2. Three scientific domains. MAST tokamak (2,889 plasma shots, 193 channels, 200 Hz to 500 kHz, roughly half a second each), MIMIC-IV ICU (74,829 stays, 10 channels, irregular hourly to four-hourly, 67 h median), and TC PRIMED (3,551 storms, 63 channels, 6 h tracks plus aperiodic satellite overpasses). Splits are 70/15/15, chronological for fusion and climate.
3. Four tasks. Each task is a context/query split over the observation set: forecasting, offline imputation, causal imputation (the same masked interval, but no access to the future), and cross-modal prediction. The model receives the context and the query coordinates \((t_i, s_i, c_i)\) and returns \(\hat{\mathbf{x}}_i\), so one adapter covers all four.
4. A five-level metric. Errors are normalised by training-set standard deviation, then averaged up a fixed hierarchy: signal-channel, sample, task, domain, benchmark. Every level contributes equally, so a 500 kHz magnetics channel cannot outweigh a four-hourly GCS reading, and 74,829 ICU stays cannot outweigh 2,889 shots. Predictions a model declines to make are scored at the training mean rather than dropped.

Key findings

1. Regridding is expensive and no single grid works. \(R\) is the NMAE under a grid condition over the NMAE at native resolution. On fusion fast signals at a 10 Hz grid, Chronos-Bolt reaches \(R = 5.61\), MOIRAI \(5.08\), and even LOCF (Last Observation Carried Forward) \(3.04\). No single grid can span five orders of magnitude of sampling rate. Climate spans a far narrower range and still degrades: Chronos-Bolt to 3.04 and MOIRAI to 2.65 at 48 h, once the context loses the sub-daily structure that storm-track forecasting relies on.
2. The cost is architecture-specific, and it reorders the results. On fusion fast signals Chronos-Bolt has the lowest native NMAE (0.182) and MOIRAI is second (0.205). At 10 Hz this changes: MOIRAI leads (0.615) and Chronos-Bolt falls to third (0.679), behind LOCF. A single model splits by signal speed too, Chronos-Bolt degrading to \(R = 5.61\) on fast fusion signals and \(1.71\) on slow ones. That is a degradation of roughly a factor of two across models at one grid and a factor of three across signal speeds within one model.
3. Past a point the context stops carrying information. On a 12 h healthcare grid, Chronos-Bolt, MOIRAI, and MOMENT all score exactly the mean baseline, and do not move at 24 h. Meaning that the grid has discarded enough temporal structure that no context-dependent model can beat predicting the training mean. Fast signals also degrade differently than slow ones, demonstrating that a global grid is not the right choice for scientific data.
4. This is a result about benchmark design, not a model ranking. Query set, horizon, and metric are identical across conditions, only the context representation changes. The mean baseline is context-independent, so \(R = 1.00\) for it throughout by construction, confirming the movement comes from the representation and not the evaluation.

Why it matters

Grid alignment is often treated as a preprocessing detail. It is applied once, upstream, and never reported as a result. Where instruments genuinely sample on a fixed schedule that is reasonable. In a tokamak, an ICU, or a satellite overpass record, it is a modelling decision taken on the model's behalf before anyone measures what it costs. Benchmarks that take that decision uniformly reward architectures built to consume its output (patch tokenisation, fixed context windows, value quantisation), and those architectures then define what benchmark performance means.

We do not claim the models tested would score better at native resolution. We claim that resampling is a modelling variable with a measurable cost, that the cost varies by roughly a factor of two across models at the same grid and a factor of three across signal speeds within one model, and that it is large enough to change which model wins. A benchmark that fixes the grid as a design constant cannot report any of that.

Scope and limitations

Three domains, one dataset each. Five models: two baselines (mean, LOCF) and three pre-trained foundation models (Chronos-Bolt-Small, MOIRAI-1.0-R-small, MOMENT-1-large). The grid sweep covers forecasting only; offline imputation, causal imputation, and cross-modal prediction are evaluated with the baselines alone, since the foundation models tested are causal forecasters that cannot condition on future observations. Point estimates only, scored with NMAE and NRMSE; no probabilistic scoring. MOMENT is fine-tuned once per domain with a capped training set, so its intervals capture test-unit sampling variability and not fine-tuning variance. No model in the sweep was designed to consume irregular timestamps, so the sweep measures what regridding costs existing architectures, not what a native-resolution architecture would gain.

Future work, not yet tested. The addition of architectures that ingest irregular timestamps directly, such as Neural ODE approaches. Building out the infrastructure with more datasets and models.