Debugging Models

This page describes the tools available for checking that a Turing model is defined correctly. It complements the Troubleshooting page, which explains specific error messages: the tools here help you find problems before (or without) hitting an error.

using Turing

Checking model validity with check_model

DynamicPPL.check_model evaluates a model once and checks it for common problems. It returns true if the check passes, and false (with warnings explaining the problem) otherwise.

Specifically, it checks for:

  • the same variable, or overlapping variables (such as x and x[1]), appearing on the left-hand side of more than one tilde statement;
  • NaN values on the left-hand side of observe statements;
  • with the keyword argument fail_if_discrete=true, the presence of discrete random variables, which gradient-based samplers such as HMC and NUTS cannot handle.

It is equally important to know what it does not catch. The model is evaluated a single time, with values drawn from the prior, so problems that only occur for some parameter values (for example, in a branch that this particular evaluation does not take) can be missed. It also checks only the structure of the model, not its statistical soundness: it cannot tell you that a prior is unreasonable or that parameters are not identifiable. Type instability and incorrect gradients are separate concerns, covered by the checks in the following sections.

For example, this model observes a NaN value, which usually indicates a bug in the data preparation:

using DynamicPPL: check_model

@model function gaussian(y)
    x ~ Normal()
    y ~ Normal(x)
end

check_model(gaussian(NaN))
┌ Warning: Encountered a NaN value on the left-hand side of an observe statement; this may indicate that your data contain NaN values.
└ @ DynamicPPL.DebugUtils ~/.julia/packages/DynamicPPL/Iktc1/src/debug_utils.jl:107

false

whereas the same model with valid data passes:

check_model(gaussian(1.5))
true

Turing runs this check automatically at the start of sample (and the mode estimation functions). For gradient-based samplers, this includes the discrete-variable check. If the check reports a false positive for your model, you can disable it by passing check_model=false to sample.

The keyword argument error_on_failure=true makes check_model throw an error instead of returning false, which is useful in scripts and tests.

Checking type stability

Type instability is one of the most common reasons for a model being slower than expected. DynamicPPL.DebugUtils.model_warntype(model) works like @code_warntype, but for an entire model: types that the compiler cannot infer are marked in red in the output. The Performance Tips page has a worked example, along with advice on how to fix the instabilities it finds. There is also DynamicPPL.DebugUtils.model_typed, which returns the typed representation instead of printing it.

Checking gradients

If a model samples poorly with a gradient-based sampler such as NUTS, it is worth checking whether the AD backend computes correct gradients for it. DynamicPPL.TestUtils.AD.run_ad evaluates the gradient of a model with a given backend and, by default, tests it against a reference backend:

using ADTypes: AutoReverseDiff
using DynamicPPL.TestUtils.AD: run_ad
using ReverseDiff

run_ad(gaussian(1.5), AutoReverseDiff())
[ Info: Running AD on gaussian with ADTypes.AutoReverseDiff()
       params : [-0.04921806067376239]
       actual : (-3.0391265749164753, [1.5984361213475249])
     expected : (-3.0391265749164753, [1.5984361213475249])

ADResult
  ├ model          : gaussian
  ├ adtype         : ADTypes.AutoReverseDiff()
  ├ value_actual   : -3.0391265749164753
  ├ value_expected : -3.0391265749164753
  ├ grad_actual    : [1.5984361213475249]
  ├ grad_expected  : [1.5984361213475249]
  └ params         : [-0.04921806067376239]

If the gradient is incorrect, this errors. For debugging NaN or infinite gradients, see the Troubleshooting page.

Further help

This page covers the most commonly useful tools. DynamicPPL provides a few more, such as DynamicPPL.has_static_constraints; the full list is in the DynamicPPL API documentation.

If these tools do not surface the problem, feel free to open an issue, ideally with the smallest model that reproduces it.

Back to top