TuringBenchmarking.jl

A useful package for benchmarking and checking Turing.jl models.

Example

julia> using TuringBenchmarking, Turing
julia> @model function demo(x)
           s ~ InverseGamma(2, 3)
           m ~ Normal(0, sqrt(s))
           for i in 1:length(x)
               x[i] ~ Normal(m, sqrt(s))
           end
       enddemo (generic function with 2 methods)
julia> model = demo([1.5, 2.0]);
julia> benchmark_model(
           model;
           # Check correctness of computations
           check=true,
           # Automatic differentiation backends to check and benchmark
           adbackends=[:forwarddiff, :reversediff, :reversediff_compiled, :zygote]
       )2-element BenchmarkTools.BenchmarkGroup:
  tags: []
  "evaluation" => 2-element BenchmarkTools.BenchmarkGroup:
	  tags: []
	  "linked" => Trial(370.000 ns)
	  "standard" => Trial(300.000 ns)
  "gradient" => 4-element BenchmarkTools.BenchmarkGroup:
	  tags: []
	  "ADTypes.AutoForwardDiff{0, Nothing}(nothing)" => 2-element BenchmarkTools.BenchmarkGroup:
		  tags: ["ForwardDiff"]
		  "linked" => Trial(541.000 ns)
		  "standard" => Trial(461.000 ns)
	  "ADTypes.AutoReverseDiff(true)" => 2-element BenchmarkTools.BenchmarkGroup:
		  tags: ["ReverseDiff [compiled]"]
		  "linked" => Trial(1.442 μs)
		  "standard" => Trial(1.442 μs)
	  "ADTypes.AutoReverseDiff(false)" => 2-element BenchmarkTools.BenchmarkGroup:
		  tags: ["ReverseDiff"]
		  "linked" => Trial(13.456 μs)
		  "standard" => Trial(13.145 μs)
	  "ADTypes.AutoZygote()" => 2-element BenchmarkTools.BenchmarkGroup:
		  tags: ["Zygote"]
		  "linked" => Trial(585.281 μs)
		  "standard" => Trial(565.234 μs)

API

TuringBenchmarking.extract_stan_data — Function
extract_stan_data(model::DynamicPPL.Model)

Return the data in model in a format consumable by the corresponding Stan model.

The Stan model requires the return data to be either

  1. A JSON string representing a dictionary with the data.
  2. A path to a data file ending in .json.
source
TuringBenchmarking.make_turing_suite — Method
make_turing_suite(model::Turing.Model; kwargs...)

Create default benchmark suite for model.

Keyword arguments

  • adbackends: a collection of adbackends to use, specified either as a type from

ADTypes.jl or using a Symbol. Defaults to ADTypes.AbstractADType[ADTypes.AutoForwardDiff{0, Nothing}(nothing), ADTypes.AutoReverseDiff(false), ADTypes.AutoReverseDiff(true), ADTypes.AutoZygote()].

  • run_once=true: if true, the body of each benchmark will be run once to avoid compilation to be included in the timings (this may occur if compilation runs longer than the allowed time limit).
  • check=false: if true, the log-density evaluations and the gradients will be compared against each other to ensure that they are consistent. Note that this will force run_once=true.
  • error_on_failed_check=false: if true, an error will be thrown if the check fails rather than just printing a warning, as is done by default.
  • error_on_failed_backend=false: if true, an error will be thrown if the evaluation of the log-density or the gradient fails for any of the backends rather than just printing a warning, as is done by default.
  • varinfo: the VarInfo to use. Defaults to DynamicPPL.VarInfo(model).
  • sampler: the Sampler to use. Defaults to nothing (i.e. no sampler).
  • context: the Context to use. Defaults to DynamicPPL.DefaultContext().
  • θ: the parameters to use. Defaults to rand(Vector, model).
  • θ_linked: the linked parameters to use. Defaults to randn(d) where d is the length of the linked parameters..
  • atol: the absolute tolerance to use for comparisons.
  • rtol: the relative tolerance to use for comparisons.

Notes

  • A separate "parameter" instance (DynamicPPL.VarInfo) will be created for each test. Hence if you have a particularly large model, you might want to only pass one adbackend at the time.
source