Debugging Models

This page describes the tools available for checking that a Turing model is defined correctly. It complements the Troubleshooting page, which explains specific error messages: the tools here help you find problems before (or without) hitting an error.

using Turing

Checking model validity with check_model

DynamicPPL.check_model evaluates a model once and checks it for common problems. It returns true if the check passes, and false (with warnings explaining the problem) otherwise.

Specifically, it checks for:

  • the same variable, or overlapping variables (such as x and x[1]), appearing on the left-hand side of more than one tilde statement;
  • NaN values on the left-hand side of observe statements;
  • a latent variable whose value was computed from a model input before its tilde statement, since that value is about to be overwritten and the variable was probably meant to be an observation;
  • with the keyword argument fail_if_discrete=true, the presence of discrete random variables, which gradient-based samplers such as HMC and NUTS cannot handle.

It is equally important to know what it does not catch. The model is evaluated a single time, with values drawn from the prior, so problems that only occur for some parameter values (for example, in a branch that this particular evaluation does not take) can be missed. It also checks only the structure of the model, not its statistical soundness: it cannot tell you that a prior is unreasonable or that parameters are not identifiable. The overwrite check needs ForwardDiff to be loaded, and is best effort even then, so it can miss a dependency that runs through an untaken branch, a condition, or code it cannot differentiate. Type instability and incorrect gradients are separate concerns, covered by the checks in the following sections.

For example, this model observes a NaN value, which usually indicates a bug in the data preparation:

using DynamicPPL: check_model

@model function gaussian(y)
    x ~ Normal()
    y ~ Normal(x)
end

check_model(gaussian(NaN))
┌ Warning: Encountered a NaN value on the left-hand side of an observe statement; this may indicate that your data contain NaN values.
└ @ DynamicPPL.DebugUtils ~/.julia/packages/DynamicPPL/KUCnw/src/debug_utils.jl:108

false

whereas the same model with valid data passes:

check_model(gaussian(1.5))
true

The overwrite check catches a different kind of mistake, where a value derived from the data is given a prior instead of being observed:

@model function overwritten(y)
    z = 2y
    z ~ Normal()
    return y ~ Normal(z)
end

check_model(overwritten(1.5))
┌ Warning: Variable z has a value derived from a model input before its tilde statement, but it is classified as latent and that value will be overwritten. It might be intended as an observation.
└ @ DynamicPPLInputProvenanceExt ~/.julia/packages/DynamicPPL/KUCnw/ext/DynamicPPLInputProvenanceExt.jl:164

true

Here z is computed from y and then immediately discarded, because z ~ Normal() samples a fresh value for it. Note that the check warns but still returns true, since this is legal Julia and legal DynamicPPL, just rarely what was meant.

Turing runs this check automatically at the start of sample (and the mode estimation functions). For gradient-based samplers, this includes the discrete-variable check. If the check reports a false positive for your model, you can disable it by passing check_model=false to sample.

The keyword argument error_on_failure=true makes check_model throw an error instead of returning false, which is useful in scripts and tests.

Checking type stability

Type instability is one of the most common reasons for a model being slower than expected. DynamicPPL.DebugUtils.model_warntype(model) works like @code_warntype, but for an entire model: types that the compiler cannot infer are marked in red in the output. The Performance Tips page has a worked example, along with advice on how to fix the instabilities it finds. There is also DynamicPPL.DebugUtils.model_typed, which returns the typed representation instead of printing it.

Checking gradients

If a model samples poorly with a gradient-based sampler such as NUTS, it is worth checking whether the AD backend computes correct gradients for it. DynamicPPL.TestUtils.AD.run_ad evaluates the gradient of a model with a given backend and, by default, tests it against a reference backend:

using ADTypes: AutoReverseDiff
using DynamicPPL.TestUtils.AD: run_ad
using ReverseDiff

run_ad(gaussian(1.5), AutoReverseDiff())
[ Info: Running AD on gaussian with ADTypes.AutoReverseDiff()
       params : [0.10340740115636878]
       actual : (-2.8184590552887068, [1.2931851976872624])
     expected : (-2.8184590552887068, [1.2931851976872624])

ADResult
  ├ model          : gaussian
  ├ adtype         : ADTypes.AutoReverseDiff()
  ├ value_actual   : -2.8184590552887068
  ├ value_expected : -2.8184590552887068
  ├ grad_actual    : [1.2931851976872624]
  ├ grad_expected  : [1.2931851976872624]
  └ params         : [0.10340740115636878]

If the gradient is incorrect, this errors. For debugging NaN or infinite gradients, see the Troubleshooting page.

Further help

This page covers the most commonly useful tools. DynamicPPL provides a few more, such as DynamicPPL.has_static_constraints; the full list is in the DynamicPPL API documentation.

If these tools do not surface the problem, feel free to open an issue, ideally with the smallest model that reproduces it.

Back to top