Infilling Global Surface Temperature#

Download this notebook: infilling_surface_temperature.ipynb

Temperature records have gaps: over the poles, over parts of Africa and the Southern Ocean, and, for satellites, wherever there is cloud. Climate scientists fill these gaps to map the field and to compute the global mean temperature. In this notebook we fill gaps with a GP, and we test the result.

To know how good the filled values are, we start from a field that is complete, remove cells ourselves, and compare the GP with the cells that we removed. We

fill gaps that are random, map the field, and get the global mean temperature with its uncertainty from joint posterior samples, and
show what goes wrong when cells are missing because of their values.

The notebook uses the gpjax.xarray module, which Working with Gridded Data introduces. It needs the optional extra: pip install "gpjax[xarray]".

from pathlib import Path

from jax import config
import jax.numpy as jnp
import jax.random as jr
from jaxtyping import install_import_hook
import matplotlib.pyplot as plt
import numpy as np
import xarray as xr

config.update("jax_enable_x64", True)

with install_import_hook("gpjax", "beartype.beartype"):
    import gpjax as gpx
    from gpjax.parameters import val
    from gpjax.xarray import (
        UnitSphere,
        from_xarray,
    )

key = jr.key(42)
rng = np.random.default_rng(0)
gpx.plotting.use_style()

'ledger'

Cells missing because of their values#

Real gaps are rarely random. A satellite cannot see the surface through cloud, a station fails in extreme weather, and the polar regions, which warm fastest, have the fewest observations. When the chance that a cell is missing depends on the value that is missing, the data are missing not at random.

We simulate this. We again aim to keep a quarter of the cells, but now the warmer a cell’s anomaly, the less likely it is to be kept.

standardised = ((truth - truth.mean()) / truth.std()).values
odds = np.exp(-1.5 * standardised)
keep_probability = np.clip(0.25 * odds / odds.mean(), 0.0, 1.0)
kept_selectively = rng.uniform(size=truth.shape) < keep_probability
selective = truth.where(kept_selectively).rename("tas").to_dataset()
print(f"{int(kept_selectively.sum())} of {truth.size} cells kept")

534 of 2592 cells kept

The steps are the same as before.

data_selective, spec_selective = from_xarray(
    selective, target="tas", inputs=["lat", "lon"], transforms=[UnitSphere()]
)
model_selective = fit_model(data_selective)
posterior_selective = model_selective.condition(data_selective)
prediction_selective = spec_selective.predict(
    lambda x: model_selective.likelihood(posterior_selective(x, covariance="diagonal")),
    truth_fine.to_dataset(),
    chunk_size=2048,
)

score(prediction_selective, selective)
plot_infill(prediction_selective, selective)

key, sample_key = jr.split(key)
means_selective = global_mean_samples(
    posterior_selective, spec_selective, selective, sample_key
)
report_global_mean(means_selective, selective)

RMSE, GP:                0.97 K
RMSE, mean of kept cells: 1.37 K
Within 2 sd:             81%
Truth:               0.64 K
Mean of kept cells:  0.06 K
Gaps filled by GP:   0.34 ± 0.05 K (2 sd)

../_images/4ff83d961a71b2d89f86d2583756126b827cf528f446aec02e654759082c9b2a.png

The kept cells are mostly the cold ones, so their mean is far too cold. The GP removes part of this bias, because it fills each gap from its neighbours and the warm regions still have some kept cells. But the result is still too cold, and the interval does not hold the truth: the GP is confidently wrong.

The reason is that a GP conditions only on the values it sees. Its prior has one constant mean, which it learns from the kept cells, so the cold kept cells pull that mean down. It also has no way to know that the missing cells are warm, because nothing in its inputs says so. Its uncertainty describes the spread of values that are consistent with the kept cells, not the error of the selection.

In real data we cannot see this bias, because we do not have the missing values. Useful steps are:

Add inputs that explain why cells are missing, such as cloud fraction or a covariate that is observed everywhere. If the chance of a gap depends only on the inputs, the gaps are missing at random given those inputs, and the GP can correct for them.
Model the observation process together with the field.
Test how sensitive the result is to different assumptions about the missing values, as we did here with a complete field.

Adding an input that explains the gaps#

If a variable that explains the gaps is observed everywhere, add it as an input. Standardise puts it on the same scale as the sphere coordinates:

data, spec = from_xarray(
    observed,
    target="tas",
    inputs=["lat", "lon", "cloud_fraction"],
    transforms=[UnitSphere(), Standardise(["cloud_fraction"])],
)

For a monthly record, Cyclic adds the seasonal cycle; see Working with Gridded Data.

References#

Kalnay, E., Kanamitsu, M., Kistler, R., Collins, W., Deaven, D., Gandin, L., Iredell, M., Saha, S., White, G., Woollen, J., Zhu, Y., Chelliah, M., Ebisuzaki, W., Higgins, W., Janowiak, J., Mo, K. C., Ropelewski, C., Wang, J., Leetmaa, A., Reynolds, R., Jenne, R. and Joseph, D. (1996). The NCEP/NCAR 40-year reanalysis project. Bulletin of the American Meteorological Society, 77(3), 437–471. doi:10.1175/1520-0477(1996)077<0437:TNYRP>2.0.CO;2

System configuration#

%reload_ext watermark
%watermark -n -u -v -iv -w -a 'Thomas Pinder'

Author: Thomas Pinder

Last updated: Sun, 04 Oct 2026

Python implementation: CPython
Python version       : 3.11.16
IPython version      : 9.17.1

gpjax     : 1.0.0
jax       : 0.10.2
jaxtyping : 0.3.11
logging   : 0.5.1.2
matplotlib: 3.11.2
numpy     : 2.4.6
xarray    : 2026.7.0

Watermark: 2.6.0

Infilling Global Surface Temperature#

A complete temperature field#

Cells missing at random#

Inputs on the sphere#

Fitting the model#

Predicting on the finer grid#

The global mean, with joint samples#

Cells missing because of their values#

Adding an input that explains the gaps#

References#

System configuration#