Gaussian Processes: Introduction

JuliaGPs packages integrate well with Turing.jl because they implement the Distributions.jl interface. You should be able to understand what is going on in this tutorial if you know what a GP is. For a more in-depth understanding of the JuliaGPs functionality used here, please consult the JuliaGPs docs.

In this tutorial, we will model the putting dataset discussed in Chapter 21 of Bayesian Data Analysis. The dataset comprises the result of measuring how often a golfer successfully gets the ball in the hole, depending on how far away from it they are. The goal of inference is to estimate the probability of any given shot being successful at a given distance.

Let’s download the data and take a look at it:

using CSV, DataFrames

df = CSV.read("golf.dat", DataFrame; delim=' ', ignorerepeated=true)
df[1:5, :]

5×3 DataFrame

Row	distance	n	y
	Int64	Int64	Int64
1	2	1443	1346
2	3	694	577
3	4	455	337
4	5	353	208
5	6	272	149

We’ve printed the first 5 rows of the dataset (which comprises only 19 rows in total). Observe it has three columns:

distance – how far away from the hole. I’ll refer to distance as d throughout the rest of this tutorial
n – how many shots were taken from a given distance
y – how many shots were successful from a given distance

We will use a Binomial model for the data, whose success probability is parametrised by a transformation of a GP. Something along the lines of: \[ \begin{aligned} f & \sim \operatorname{GP}(0, k) \\ y_j \mid f(d_j) & \sim \operatorname{Binomial}(n_j, g(f(d_j))) \\ g(x) & := \frac{1}{1 + e^{-x}} \end{aligned} \]

To do this, let’s define our Turing.jl model:

using AbstractGPs, LogExpFunctions, Turing

@model function putting_model(d, n; jitter=1e-4)
    v ~ Gamma(2, 1)
    l ~ Gamma(4, 1)
    f = GP(v * with_lengthscale(SEKernel(), l))
    f_latent ~ f(d, jitter)
    y ~ product_distribution(Binomial.(n, logistic.(f_latent)))
    return (fx=f(d, jitter), f_latent=f_latent, y=y)
end

putting_model (generic function with 2 methods)

We first define an AbstractGPs.GP, which represents a distribution over functions, and is entirely separate from Turing.jl. We place a prior over its variance v and length-scale l. f(d, jitter) constructs the multivariate Gaussian comprising the random variables in f whose indices are in d (plus a bit of independent Gaussian noise with variance jitter – see the docs for more details). f(d, jitter) has the type AbstractMvNormal, and is the bit of AbstractGPs.jl that implements the Distributions.jl interface, so it’s legal to put it on the right-hand side of a ~. From this you should deduce that f_latent is distributed according to a multivariate Gaussian. The remaining lines comprise standard Turing.jl code that is encountered in other tutorials and Turing documentation.

Before performing inference, we might want to inspect the prior that our model places over the data, to see whether there is anything obviously wrong. These kinds of prior predictive checks are straightforward to perform using Turing.jl, since it is possible to sample from the prior easily by just calling the model:

m = putting_model(Float64.(df.distance), df.n)
m().y

19-element Vector{Int64}:
 1275
  612
  404
  319
  230
  210
  201
  173
  169
  196
  163
  164
  132
  129
  153
  147
  138
  103
  111

We make use of this to see what kinds of datasets we simulate from the prior:

using Plots

function plot_data(d, n, y, xticks, yticks)
    ylims = (0, round(maximum(n), RoundUp; sigdigits=2))
    margin = -0.5 * Plots.mm
    plt = plot(; xticks=xticks, yticks=yticks, ylims=ylims, margin=margin, grid=false)
    bar!(plt, d, n; color=:red, label="", alpha=0.5)
    bar!(plt, d, y; label="", color=:blue, alpha=0.7)
    return plt
end

# Construct model and run some prior predictive checks.
m = putting_model(Float64.(df.distance), df.n)
hists = map(1:20) do j
    xticks = j > 15 ? :auto : nothing
    yticks = rem(j, 5) == 1 ? :auto : nothing
    return plot_data(df.distance, df.n, m().y, xticks, yticks)
end
plot(hists...; layout=(4, 5))

In this case, the only prior knowledge I have is that the proportion of successful shots ought to decrease monotonically as the distance from the hole increases, which should show up in the data as the blue lines generally go down as we move from left to right on each graph. Unfortunately, there is not a simple way to enforce monotonicity in the samples from a GP, and we can see this in some of the plots above, so we must hope that we have enough data to ensure that this relationship holds approximately under the posterior. In any case, you can judge for yourself whether you think this is the most useful visualisation that we can perform – if you think there is something better to look at, please let us know!

Moving on, we generate samples from the posterior using the default NUTS sampler. We’ll make use of ReverseDiff.jl, as it has better performance than ForwardDiff.jl on this example. See Turing.jl’s docs on Automatic Differentiation for more info.

using Random, ReverseDiff

m_post = m | (y=df.y,)
chn = sample(Xoshiro(123456), m_post, NUTS(; adtype=AutoReverseDiff()), 1_000, progress=false)

┌ Info: Found initial step size
└   ϵ = 0.2

Chains MCMC chain (1000×33×1 Array{Float64, 3}):

Iterations        = 501:1:1500
Number of chains  = 1
Samples per chain = 1000
Wall duration     = 152.45 seconds
Compute duration  = 152.45 seconds
parameters        = v, l, f_latent[1], f_latent[2], f_latent[3], f_latent[4], f_latent[5], f_latent[6], f_latent[7], f_latent[8], f_latent[9], f_latent[10], f_latent[11], f_latent[12], f_latent[13], f_latent[14], f_latent[15], f_latent[16], f_latent[17], f_latent[18], f_latent[19]
internals         = lp, n_steps, is_accept, acceptance_rate, log_density, hamiltonian_energy, hamiltonian_energy_error, max_hamiltonian_energy_error, tree_depth, numerical_error, step_size, nom_step_size

Summary Statistics
    parameters      mean       std      mcse    ess_bulk   ess_tail      rhat  ⋯
        Symbol   Float64   Float64   Float64     Float64    Float64   Float64  ⋯

             v    2.9156    1.3500    0.0471    888.9807   602.1180    0.9998  ⋯
             l    3.6597    0.8239    0.0542    204.5560   181.5073    1.0022  ⋯
   f_latent[1]    2.5427    0.0980    0.0031    976.8509   841.3983    1.0001  ⋯
   f_latent[2]    1.7005    0.0680    0.0020   1159.1773   697.0767    1.0007  ⋯
   f_latent[3]    0.9745    0.0782    0.0033    557.7282   696.2342    1.0014  ⋯
   f_latent[4]    0.4800    0.0756    0.0026    885.5237   663.8774    1.0018  ⋯
   f_latent[5]    0.1925    0.0778    0.0025    987.6531   661.5774    0.9995  ⋯
   f_latent[6]   -0.0136    0.0952    0.0046    432.1599   412.7216    0.9991  ⋯
   f_latent[7]   -0.2455    0.0876    0.0031    761.9793   545.9663    1.0007  ⋯
   f_latent[8]   -0.5048    0.0869    0.0031    775.3969   490.1728    0.9995  ⋯
   f_latent[9]   -0.7219    0.0959    0.0041    552.6868   565.7969    0.9998  ⋯
  f_latent[10]   -0.8629    0.0922    0.0036    649.0443   763.2700    0.9990  ⋯
  f_latent[11]   -0.9491    0.0923    0.0033    771.8466   597.0319    1.0026  ⋯
  f_latent[12]   -1.0394    0.1115    0.0042    694.0441   625.9025    1.0036  ⋯
  f_latent[13]   -1.1896    0.1189    0.0042    819.1409   609.3305    1.0007  ⋯
  f_latent[14]   -1.4028    0.1133    0.0040    815.2130   640.5742    1.0018  ⋯
  f_latent[15]   -1.6049    0.1162    0.0044    707.9191   627.7714    1.0017  ⋯
       ⋮            ⋮         ⋮         ⋮          ⋮          ⋮          ⋮     ⋱
                                                     1 column and 4 rows omitted

Quantiles
    parameters      2.5%     25.0%     50.0%     75.0%     97.5%
        Symbol   Float64   Float64   Float64   Float64   Float64

             v    1.0610    1.9583    2.6822    3.5928    5.9954
             l    2.3171    3.1385    3.5154    4.0461    5.7650
   f_latent[1]    2.3546    2.4762    2.5384    2.6043    2.7366
   f_latent[2]    1.5710    1.6567    1.7007    1.7474    1.8308
   f_latent[3]    0.8229    0.9203    0.9770    1.0285    1.1288
   f_latent[4]    0.3316    0.4335    0.4843    0.5317    0.6223
   f_latent[5]    0.0438    0.1411    0.1917    0.2408    0.3539
   f_latent[6]   -0.1982   -0.0776   -0.0141    0.0486    0.1723
   f_latent[7]   -0.4077   -0.3065   -0.2492   -0.1837   -0.0713
   f_latent[8]   -0.6685   -0.5595   -0.5041   -0.4463   -0.3391
   f_latent[9]   -0.9103   -0.7881   -0.7200   -0.6542   -0.5332
  f_latent[10]   -1.0388   -0.9249   -0.8638   -0.8013   -0.6808
  f_latent[11]   -1.1346   -1.0117   -0.9508   -0.8825   -0.7639
  f_latent[12]   -1.2519   -1.1170   -1.0424   -0.9591   -0.8230
  f_latent[13]   -1.4178   -1.2710   -1.1950   -1.1103   -0.9566
  f_latent[14]   -1.6482   -1.4767   -1.3980   -1.3272   -1.1679
  f_latent[15]   -1.8522   -1.6774   -1.6000   -1.5271   -1.3940
       ⋮            ⋮         ⋮         ⋮         ⋮         ⋮
                                                    4 rows omitted

We can use these samples and the posterior function from AbstractGPs to sample from the posterior probability of success at any distance we choose:

d_pred = 1:0.2:21
samples = map(generated_quantities(m_post, chn)[1:10:end]) do x
    return logistic.(rand(posterior(x.fx, x.f_latent)(d_pred, 1e-4)))
end
p = plot()
plot!(d_pred, reduce(hcat, samples); label="", color=:blue, alpha=0.2)
scatter!(df.distance, df.y ./ df.n; label="", color=:red)

We can see that the general trend is indeed down as the distance from the hole increases, and that if we move away from the data, the posterior uncertainty quickly inflates. This suggests that the model is probably going to do a reasonable job of interpolating between observed data, but less good a job at extrapolating to larger distances.