Turing AD tests
Turing.jl documentation | Turing.jl GitHub | Source code for these tests
This page is intended as a brief overview of how different AD backends perform on a variety of Turing.jl models. Note that the inclusion of any AD backend here does not imply an endorsement from the Turing team; this table is purely for information.
- The definitions of the models and AD types below can be found on GitHub.
- Numbers indicate the time taken to calculate the gradient of the log density of the model using the specified AD type, divided by the time taken to calculate the log density itself (in AD speak, the primal). Basically: smaller means faster.
- 'wrong' means that AD ran but the result was not correct. If this happens you should be very wary! Note that this is done by comparing against the result obtained using ForwardDiff, i.e., ForwardDiff is by definition always 'correct'.
- 'error' means that AD didn't run.
- Some of the 'wrong' or 'error' entries have question marks next to them. These will link to a GitHub issue or other page that describes the problem.
Results
| Model name \ AD type | EnzymeForward | EnzymeReverse | FiniteDifferences | ForwardDiff | Mooncake | ReverseDiff | ReverseDiffCompiled | Zygote |
|---|---|---|---|---|---|---|---|---|
| assume_beta | 2.655 | 3.290 | 23.485 | 1.703 | 7.048 | 29.247 | 3.055 | 771.909 |
| assume_dirichlet | 2.360 | 3.574 | 23.872 | 1.342 | 9.292 | 16.025 | 3.056 | 574.738 |
| assume_lkjcholu | 4.627 | 13.282 | 149.571 | 2.467 | 6.626 | 51.781 | 6.721 | error |
| assume_mvnormal | (?) error | 4.017 | 42.968 | 1.287 | 9.355 | 17.452 | 1.915 | 642.959 |
| assume_normal | 2.852 | 3.241 | 27.849 | 1.489 | 8.703 | 20.980 | 2.500 | 1169.839 |
| assume_submodel | 2.095 | 2.196 | 36.514 | 1.217 | 6.818 | 13.500 | 1.403 | 1169.978 |
| assume_wishart | (?) error | 14.094 | 44.427 | 1.023 | 10.629 | 18.971 | 4.089 | 256.474 |
| control_flow | 2.644 | 3.232 | 41.989 | 1.504 | 9.423 | 16.795 | wrong | 1099.246 |
| dot_assume | 2.870 | 2.249 | 78.368 | 1.362 | 5.268 | 23.561 | 2.464 | error |
| dot_assume_observe_index | wrong | wrong | 52.702 | 1.351 | 6.278 | 36.370 | 3.572 | error |
| dot_observe | 2.613 | 2.419 | 28.301 | 1.684 | 10.176 | 56.755 | 6.420 | error |
| dynamic_constraint | 2.473 | 2.551 | 36.296 | 1.270 | 7.313 | 32.790 | 2.767 | 1108.624 |
| multiple_constraints_same_var | 4.993 | 26.415 | 59.332 | 1.224 | 13.688 | 19.114 | 2.248 | error |
| n010 | 4.110 | 2.053 | 139.704 | 1.633 | 4.624 | 24.289 | 2.711 | error |
| n050 | 18.271 | 2.331 | 626.412 | 8.615 | 3.970 | 25.211 | 2.417 | error |
| n100 | 36.637 | 2.105 | 1240.029 | 13.967 | 4.094 | 25.355 | 2.652 | error |
| n500 | 182.968 | 2.275 | 6753.340 | 57.978 | 3.765 | 25.219 | 2.354 | error |
| observe_index | 2.745 | 2.603 | 28.236 | 1.657 | 9.714 | 59.427 | 5.765 | error |
| observe_literal | 2.869 | 2.974 | 26.849 | 1.467 | 9.053 | 33.167 | 3.784 | 1511.408 |
| observe_multivariate | 2.683 | 3.434 | 50.536 | 1.296 | 5.963 | 27.047 | 2.679 | error |
| observe_submodel | 2.437 | 2.262 | 25.983 | 1.386 | 7.619 | 26.870 | 2.764 | 2134.432 |
Manifest
The tests above were run with the following package versions:
| Package | Version |
|---|---|
| ADTypes | v1.14.0 |
| AbstractFFTs | v1.5.0 |
| AbstractMCMC | v5.6.0 |
| AbstractPPL | v0.11.0 |
| AbstractTrees | v0.4.5 |
| Accessors | v0.1.42 |
| Adapt | v4.3.0 |
| AliasTables | v1.1.3 |
| ArgCheck | v2.5.0 |
| ArgTools | v1.1.2 |
| ArnoldiMethod | v0.4.0 |
| Artifacts | v1.11.0 |
| BangBang | v0.4.4 |
| Base64 | v1.11.0 |
| Baselet | v0.1.1 |
| Bijectors | v0.15.6 |
| CEnum | v0.5.0 |
| ChainRules | v1.72.3 |
| ChainRulesCore | v1.25.1 |
| Chairmarks | v1.3.1 |
| ChangesOfVariables | v0.1.9 |
| CommonSolve | v0.2.4 |
| CommonSubexpressions | v0.3.1 |
| Compat | v4.16.0 |
| CompilerSupportLibraries_jll | v1.1.1+0 |
| CompositionsBase | v0.1.2 |
| ConsoleProgressMonitor | v0.1.2 |
| ConstructionBase | v1.5.8 |
| DataAPI | v1.16.0 |
| DataStructures | v0.18.22 |
| DataValueInterfaces | v1.0.0 |
| Dates | v1.11.0 |
| DefineSingletons | v0.1.2 |
| DensityInterface | v0.4.0 |
| DiffResults | v1.1.0 |
| DiffRules | v1.15.1 |
| DifferentiationInterface | v0.6.52 |
| Distributed | v1.11.0 |
| Distributions | v0.25.119 |
| DocStringExtensions | v0.9.4 |
| Downloads | v1.6.0 |
| DynamicPPL | v0.36.1 |
| Enzyme | v0.13.38 |
| EnzymeCore | v0.8.8 |
| Enzyme_jll | v0.0.174+0 |
| ExprTools | v0.1.10 |
| FileWatching | v1.11.0 |
| FillArrays | v1.13.0 |
| FiniteDifferences | v0.12.32 |
| ForwardDiff | v0.10.38 |
| FunctionWrappers | v1.1.3 |
| Functors | v0.5.2 |
| Future | v1.11.0 |
| GPUArraysCore | v0.2.0 |
| GPUCompiler | v1.4.0 |
| Graphs | v1.12.1 |
| HypergeometricFunctions | v0.3.28 |
| IRTools | v0.4.14 |
| Inflate | v0.1.5 |
| InitialValues | v0.3.1 |
| InteractiveUtils | v1.11.0 |
| InverseFunctions | v0.1.17 |
| IrrationalConstants | v0.2.4 |
| IteratorInterfaceExtensions | v1.0.0 |
| JLLWrappers | v1.7.0 |
| JSON | v0.21.4 |
| LLVM | v9.3.1 |
| LLVMExtra_jll | v0.0.35+0 |
| LazyArtifacts | v1.11.0 |
| LeftChildRightSiblingTrees | v0.2.0 |
| LibCURL | v0.6.4 |
| LibCURL_jll | v8.6.0+0 |
| LibGit2 | v1.11.0 |
| LibGit2_jll | v1.7.2+0 |
| LibSSH2_jll | v1.11.0+1 |
| LibTracyClient_jll | v0.9.1+6 |
| Libdl | v1.11.0 |
| LinearAlgebra | v1.11.0 |
| LogDensityProblems | v2.1.2 |
| LogExpFunctions | v0.3.29 |
| Logging | v1.11.0 |
| LoggingExtras | v1.1.0 |
| MacroTools | v0.5.16 |
| MappedArrays | v0.4.2 |
| Markdown | v1.11.0 |
| MbedTLS_jll | v2.28.6+0 |
| MicroCollections | v0.2.0 |
| Missings | v1.2.0 |
| MistyClosures | v2.0.0 |
| Mmap | v1.11.0 |
| Mooncake | v0.4.117 |
| MozillaCACerts_jll | v2023.12.12 |
| NaNMath | v1.1.3 |
| NetworkOptions | v1.2.0 |
| ObjectFile | v0.4.4 |
| OpenBLAS_jll | v0.3.27+1 |
| OpenLibm_jll | v0.8.5+0 |
| OpenSpecFun_jll | v0.5.6+0 |
| OrderedCollections | v1.8.0 |
| PDMats | v0.11.34 |
| Parsers | v2.8.3 |
| Pkg | v1.11.0 |
| PrecompileTools | v1.2.1 |
| Preferences | v1.4.3 |
| Printf | v1.11.0 |
| ProgressLogging | v0.1.4 |
| ProgressMeter | v1.10.4 |
| PtrArrays | v1.3.0 |
| QuadGK | v2.11.2 |
| Random | v1.11.0 |
| RealDot | v0.1.0 |
| Reexport | v1.2.2 |
| Requires | v1.3.1 |
| ReverseDiff | v1.16.1 |
| Richardson | v1.4.2 |
| Rmath | v0.8.0 |
| Rmath_jll | v0.5.1+0 |
| Roots | v2.2.7 |
| SHA | v0.7.0 |
| Scratch | v1.2.1 |
| Serialization | v1.11.0 |
| Setfield | v1.1.2 |
| SharedArrays | v1.11.0 |
| SimpleTraits | v0.9.4 |
| Sockets | v1.11.0 |
| SortingAlgorithms | v1.2.1 |
| SparseArrays | v1.11.0 |
| SparseInverseSubset | v0.1.2 |
| SpecialFunctions | v2.5.1 |
| SplittablesBase | v0.1.15 |
| StaticArrays | v1.9.13 |
| StaticArraysCore | v1.4.3 |
| Statistics | v1.11.1 |
| StatsAPI | v1.7.0 |
| StatsBase | v0.34.4 |
| StatsFuns | v1.5.0 |
| StructArrays | v0.7.1 |
| StructIO | v0.3.1 |
| SuiteSparse | |
| SuiteSparse_jll | v7.7.0+0 |
| TOML | v1.0.3 |
| TableTraits | v1.0.1 |
| Tables | v1.12.0 |
| Tar | v1.10.0 |
| TerminalLoggers | v0.1.7 |
| Test | v1.11.0 |
| Tracy | v0.1.4 |
| Transducers | v0.4.84 |
| UUIDs | v1.11.0 |
| Unicode | v1.11.0 |
| Zlib_jll | v1.2.13+1 |
| Zygote | v0.7.6 |
| ZygoteRules | v0.2.7 |
| libblastrampoline_jll | v5.11.0+0 |
| nghttp2_jll | v1.59.0+0 |
| p7zip_jll | v17.4.0+2 |