Turing.jl

Google Summer of Code 2022

2022-02-17T00:00:00+00:00

It is another year of the Google Summer of Code time, and we have compiled an updated list of exciting Turing projects! Projects that the Turing team would be interested in working with students on over the summer are listed below. This information is also cross-posted at Julia's Turing project page.

If you are interested in exploring any of these projects or have any questions, please reach out to the listed project mentors. You can find their contact information at turing.ml/team.

More real-world Bayesian models in Turing / Julia

Mentors: Kai Xu, Tor E. Fjelde, Hong Ge

Project difficulty: Medium

Project length: 175 hrs or 350 hrs

Description: There are many real-world Bayesian models out there, and they deserve a Turing / Julia implementation.

Examples include but not limited to

Forecasting (Prophet, datasets)
Recommender system (probabilistic matrix factorisation, dataset)
Ranking (TrueSkill, dataset)
Bayesian revenue estimation (example)
Political forecasting model (example)
Topic mining (latent Dirichlet allocation and new variants)
Multiple Annotators/Combining Unreliable Observations (Dawid and Skene, 1979)

For each model, we would consider the following tasks as part of a GSoC project:

Correctness test: correctness of the implementation can be tested by doing inference for prior samples, for which we know the ground truth latent variables.
Performance benchmark: this includes (i) time per MCMC step and (ii) time per effective sample; if the model is differentiable, a further break-down of (i) into (i.1) time per forward pass and (i.2) time per gradient pass are needed.
Real-world results: if available, the final step is to apply the model to a real-world dataset; if such an experiment has been done in the literature, consistency of inference results needs to be checked

Improving the integration between Turing and Turing's MCMC inference packages

Mentors: Cameron Pfiffer, Mohamed Tarek, David Widmann

Project difficulty: Easy

Project length: 175 hrs

Description: Turing.jl is based on a set of inference packages maintained by the TuringLang group. This project is about making use of improvements in DynamicPPL to create a generic integration between Turing.jl and the AbstractMCMC.jl sampling API. The ultimate goal is to remove or substantially reduce algorithm-specific glue code inside Turing.jl. The project would also involve improving data structures for storing model parameters in DynamicPPL.

Directed-graphical model support for the abstract probabilistic programming library

Mentors: Philipp Gabler, Hong Ge

Project difficulty: Hard

Project length: 350 hrs

Description: We want to have a very light-weight representation of probabilistic models of static graphs (similar to BUGS), which can serve as a representation target of other front-end DSLs or be dynamically built. The representation should consist of the model and node representations (stochastic and deterministic, perhaps hyperparameters) and conform to the AbstractPPL model interface, with basic functions (evaluation of density, sampling, conditioning; at later stages some static analysis like extraction of Markov blankets). The model should also contain the state of the variables and implement the AbstractPPL trace interface (dictionary functions, querying of variable names). The result should be able to work with existing sampling packages through the abstract interfaces.

A modular tape caching mechanism for ReverseDiff

Mentors: Qingliang Zhuo, Mohamed Tarek

Project difficulty: Medium

Project length: 175 hrs

Description: Tape caching often leads to significant performance improvements for gradient-based sampling algorithms (e.g. HMC/NUTS). Tape caching is only possible at the complete computational level for ReverseDiff at the moment. This project is about implementing a more modular, i.e. function-as-a-caching-barrier, tape caching mechanism for ReverseDiff.jl.

Benchmarking & improving performance of the JuliaGaussianProcesses libraries

Mentors: Theo Galy-Fajou, Will Tebbutt, ST John

Project difficulty: Medium

Project length: 350 hrs

Description: Although KernelFunctions.jl has extensive correctness testing, our performance testing is lacking. This project aims to resolve this, and resolve performance issues wherever they are found. The student would first need to extend our existing benchmarking coverage, and debug any obvious performance problems. The next phase of the work would be to construct end-to-end examples of KernelFunctions being used in practice, profile them to determine where performance problems lie, and fix them.

Iterative methods for inference in Gaussian Processes

Mentors: Will Tebbutt, S. T. John, Ross Viljoen

Project difficulty: Medium

Project length: 175 hrs

Description: There has recently been quite a bit of work on inference methods for GPs that use iterative methods rather than the Cholesky factorisation. They look quite promising, but no one has implemented any of these within the Julia GP ecosystem yet, but they should fit nicely within the AbstractGPs framework. If you're interested in improving the GP ecosystem in Julia, this project might be for you!

Approximate inference methods for non-Gaussian likelihoods in Gaussian Processes

Mentors: S. T. John, Ross Viljoen, Theo Galy-Fajou

Project difficulty: Hard

Project length: 350 hrs

Description: Adding approximate inference methods for non-Gaussian likelihoods which are available in other GP packages but not yet within JuliaGPs. The project would start by determining which approximate inference method(s) to implement---there's lots to do, and we're happy to work with a student on whichever method they are most interested in, or to suggest one if they have no strong preference.

GPU integration in the JuliaGPs ecosystem

Mentors: Ross Viljoen, Theo Galy-Fajou, Will Tebbutt

Project difficulty: Medium

Project length: 350 hrs

Description: This would involve first ensuring that common models are able to run fully on the GPU, then identifying and improving GPU-specific performance bottlenecks. This would begin by implementing a limited end-to-end example involving a GP with a standard kernel, and profiling it to debug any substantial performance bottlenecks. From there, support for a wider range of the functionality available in KernelFunctions.jl and AbstractGPs.jl can be added. Stretch goal: extension of GPU support to some functionality in ApproximateGPs.jl.

Google Summer of Code 2021

2021-02-10T00:00:00+00:00

It's about time for us to start thinking about projects we'd like to see at Turing.jl for the Google Summer of Code 2021! Below is a list of projects the Turing team would be interested in working with students on for the summer.

If you are interested in exploring any of these projects, please reach out to the listed project mentors. You can find their contact information at turing.ml/team.

MCMCChains improvements

Mentors: Cameron Pfiffer, Hong Ge

Project difficulty: Easy

Description: MCMCChains is a key component of the Turing.jl ecosystem. It is the package that determines how to analyze and store MCMC samples provided by packages like Turing. It's also used outside of Turing.

For this project, a student might improve the performance of the various statistical functions provided by MCMCChains, changing the back end to use a data storage format that maintains the shape of parameter samples, or improve the general plotting functionality of the package.

There's lots of fun little things to do for MCMCChains. Check out this meta-issue for more details and discussions.

Particle filtering methods

Mentors: Hong Ge, Cameron Pfiffer

Project difficulty: Medium

Description: Turing's support for particle sampling methods is slowing being improved with the addition of AdvancedPS.jl. If you're interested in implementing or improving particle sampling methods, this is a great project for you!

Nested Sampling

Mentors: Miles Lucas, Cameron Pfiffer, Hong Ge

Project difficulty: Hard

Description: NestedSamplers.jl is an excellent package which implements nested sampling methods. As of yet, it is not connected to Turing.jl. For this project, a student would connect the NestedSamplers.jl library to Turing.jl.

GPU acceleration

Mentors: Mohamed Tarek, Hong Ge, Kai Xu, Tor Fjelde

Project difficulty: Medium

Description: Turing's native GPU support is limited in that the Metropolis-Hastings and HMC samplers do not implement GPU sampling methods. This can and should be done -- GPU methods are awesome! If you are interested with working on parallelism and GPUs, this project is for you.

Students will work with the code at AdvancedMH or AdvancedHMC, depending on their interests.

Documentation and tutorial improvements

Mentors: Cameron Pfiffer, Martin Trapp

Project difficulty: Easy

Description: Turing's documentation and tutorials need a bit of an overhaul. Turing has changed significantly since the last time the documentation was written, and it's beginning to show. Students would use their knowledge of probabilistic programming languages and Turing to shore-up or rewrite documentation and tutorials.

Iterative Methods for Inference in Gaussian Processes

Mentors: Will Tebbutt, S. T. John, Theo Galy-Fajou

Project difficulty: Medium

Implement advanced variational Gaussian process models

Mentors: ST John, Will Tebbutt, Theo Galy-Fajou

Project difficulty: Easy to Medium

Description: Sparse variational Gaussian process models provide the flexibility to scale to large datasets, handle arbitrary (non-conjugate) likelihoods, and to be used as building blocks for composite models such as deep GPs. This project is about making such models more readily available within the Julia GP ecosystem - depending on your interests you can focus on making it easier for end users and providing good tutorials, or on the implementations of these models to give us the same or better performance as with established Python packages such as GPflow, integrating with Flux.jl, etc.

Google Summer of Code 2020

2020-09-11T00:00:00+00:00

As the 2020 Google Summer of Code comes to a close, the Turing team thought it would be a good opportunity to reflect on the work that was done by our superb students this summer.

Saranjeet Kaur's project focused primarily on expanding NestedSamplers.jl. NestedSamplers.jl now supports PolyChord-style nested sampling natively, which is an absolute delight. Saranjeet wrote about this here. She also provided a good tutorial on how to use NestedSamplers.jl here. The NestedSamplers.jl integration with Turing is still on-going -- integrating new samplers with Turing is one of the more difficult tasks. If you are interested to see the progress on this, check out the relevant pull request.

Arthur Lui's project was to provide a much-needed set of benchmarks of Bayesian nonparametric models between Turing and other PPLs. Arthur's work spawned a GitHub repository with good practices for benchmarking, as well as three blog posts with some (very cool!) statistics on Turing's performance:

Finally, Sharan Yalburgi (a returning GSoC student) completed an epic amount of work Turing's growing suite of Gaussian process tools. In particular, the GitHub organization JuliaGaussianProcesses was founded, and serves as an effort to build a robust Gaussian process framework for the Julia ecosystem. The framework consists of multiple GP related Julia packages:

KernelFunctions.jl provides kernel functions for GPs as well as efficient AD for these kernels. KernelFunctions.jl also supports multi-output GPs by providing necessary data abstractions and multi-output kernels.
AbstractGPs.jl defines GP abstractions and provides exact posteriors. It provides support for induced points based GP posteriors and for efficient sequential/online (sparse) GP updates.
GPLikelihoods.jl defines alternate likelihoods for Non-Gaussian GPs.
GPMLj.jl provides a Julia interface for GPFlow, a GP library written in Python using TensorFlow.

Special thanks to our three GSoC students for this summer, who all did excellent work. Additional thanks to Google for supporting open source software development and the Julia language!

Replication study: Estimating number of infections and impact of NPIs on COVID-19 in European countries (Imperial Report 13)

2020-05-14T00:00:00+00:00

The Turing.jl team is currently exploring possibilities in an attempt to help with the ongoing SARS-CoV-2 crisis. As preparation for this and to get our feet wet, we decided to perform a replication study of the Imperial Report 13, which attempts to estimate the real number of infections and impact of non-pharmaceutical interventions on COVID-19. In the report, the inference was performed using the probabilistic programming language (PPL) Stan. We have explicated their model and inference in Turing.jl, a Julia-based PPL. We believe the results and analysis of our study are relevant for the public, and for other researchers who are actively working on epidemiological models. To that end, our implementation and results are available here.

In summary, we replicated the Imperial COVID-19 model using Turing.jl. Subsequently, we compared the inference results between Turing and Stan, and our comparison indicates that results are reproducible with two different implementations. In particular, we performed 4 sets of simulations using the Imperial COVID-19 model. The resulting estimates of the expected real number of cases, in contrast to the recorded number of cases, the reproduction number \(R_t\), and the expected number of deaths as a function of time and non-pharmaceutical interventions (NPIs) for each Simulation are shown below.

Simulation (a): hypothetical Simulation from the model without data (prior predictive) or non-pharmaceutical interventions. Under the prior assumptions of the Imperial Covid-19 model, there is a very wide range of epidemic progressions with expected cases from almost 0 to 100% of the population over time. The black bar corresponds to the date of the last observation. Note that \(R_t\) has a different time-range than the other plots; following the original report, this shows the 100 days following the country-specific epidemic_start which is defined to be 31 days prior to the first date of 10 cumulative deaths, while the other plots show the last 60 days.

Simulation (b): future Simulation with non-pharmaceutical interventions kept in place (posterior predictive). After incorporating the observed infection data, we can see a substantially more refined range of epidemic progression. The reproduction rate estimate lies in the range of 3.5-5.6 before any intervention is introduced. The dotted lines correspond to observations, and the black bar corresponds to the date of the last observation.

Simulation (c): future Simulation with non-pharmaceutical interventions removed. Now we see the hypothetical scenarios after incorporating infection data, but with non-pharmaceutical interventions removed. This plot looks similar to Simulation (a), but with a more rapid progression of the pandemic since the estimated reproduction rate is bigger than the prior assumptions. The dotted lines correspond to observations, and the black bar corresponds to the date of the last observation.

Simulation (d): future Simulation with when lockdown is lifted two weeks before the last observation (predictive posterior). As a result there is a clear, rapid rebound of the reproduction rate. Comparing with Simulation (b) we do not observe an immediate increase in the number of expected cases and deaths upon lifting lockdown, but there is a significant difference in the number of cases and deaths in the last few days in the plot: Simulation (d) results in both greater number of cases and deaths, as expected. This demonstrates how the effects of lifting an intervention might not become apparent in the measurable variables, e.g. deaths, until several weeks later. The dotted lines correspond to observations, the black bar corresponds to the date of the last observation, and the red bar indicates when lockdown was lifted.

Overall, Simulation (a) shows the prior modelling assumptions, and how these prior assumptions determine the predicted number of cases, etc. before seeing any data. Simulation (b) predicts the trend of the number of cases, etc. using estimated parameters and by keeping all the non-pharmaceutical interventions in place. Simulation (c) shows the estimate in the case where none of the intervention measures are ever put in place. Simulation (d) shows the estimates in the case when the lockdown was lifted two weeks prior to the last observation while keeping all the other non-pharmaceutical interventions in place.

We want to emphasise that we do not provide additional analysis of the Imperial model yet, nor are we aiming to make any claims about the validity or the implications of the model. Instead we refer to Imperial Report 13 for more details and analysis. The purpose of this post is solely to add validation to the inference performed in the paper by obtaining the same results using a different probabilistic programming language (PPL) and by exploring whether or not Turing.jl can be useful for researchers working on these problems.

For our next steps, we're looking at collaboration with other researchers and further developments of this and similar models. There are some immediate directions to explore:

Incorporation of more sources of data, e.g. national mobility, seasonal changes and behavior changes in individuals.
How the assumptions incorporated into the priors and their parameters change resulting posterior.
The current model does not directly include recovery as a possibility and assumes that if a person has been infected once then he/she will be infectious until death. Number of recovered cases suffers from the same issues as the number of cases: it cannot be directly observed. But we can also deal with it in a similar manner as is done with number of cases and incorporate this into the model for a potential improvement. This will result in a plethora of different models from which we can select the most realistic one using different model comparison techniques, e.g. leave-one-out cross-validation (loo-cv).

Such model refinement can be potentially valuable given the high impact of this pandemic and the uncertainty and debates in the potential outcomes.

Acknowledgement We would like to thank the Julia community for creating such an excellent platform for scientific computing, and for the continuous feedback that we have received. We also thank researchers from Computational and Biological Laboratory at Cambridge University for their feedback on an early version of the post.

Google Summer of Code/Julia Summer of Code

2020-02-12T00:00:00+00:00

Last year, Turing participated in the Google Summer of Code (GSoC) through the Julia language organization. It was a fun time, and the project was better for it. Turing plans to participate in the upcoming GSoC, and we wanted to outline some potential projects and expectations we have for applicants.

If you are not aware, Google provides funds to students around the world to develop a project of their choice over the summer. Students receive funds from Google and spend three months on any open source project.

The Turing development team has prepared a list of possible projects that we have deemed valuable to the project and easy enough that it could feasibly be created in the three-month limit. This list is not exclusive -- if you have a good idea, you can write it up in your proposal, though it is recommend that you reach out to any of the Turing team on Julia's Slack (you can get an invite here) or Discourse. Messages on Discourse should be posted to the "Probabilistic programming" category -- we'll find you!

Possible project ideas:

Benchmarking. Turing's performance has been sporadically benchmarked against various other probabilistic programming languages (e.g. Turing, Stan, PyMC3, TensorFlow Prob), but a systemic approach to studying where Turing excels and where it falls short would be useful. A GSoC student would implement identical models in many PPLs and build tools to benchmark all PPLs against one another.
Nested sampling integration. Turing focuses on modularity in inference methods, and the development team would like to see more inference methods, particularly the popular nested sampling method. A Julia package (NestedSamplers.jl) but it is not hooked up to Turing and does not currently have a stable API. A GSoC student would either integrate that package or construct their own nested sampling method and build it into Turing.
Automated function memoization by model annotation. Function memoization is a way to reduce costly function evaluation by caching the output when the same inputs are given. Turing's Gibbs sampler often ends up rerunning expensive functions multiple times, and it would be a significant performance improvement to allow Turing's model compiler to automatically memoize functions where appropriate. A student working on this project would become intimately familiar with Turing's model compiler and build in various automated improvements.
Making Distributions GPU compatible. Julia's GPU tooling is generally quite good, but currently Turing is not able to reliably use GPUs while sampling because Distributions.jl is not GPU compatible. A student on this project would work with the Turing developers and the Distributions developers to allow the use of GPU parallelism where possible in Turing.
Static distributions. Small, fixed-size vectors and matrices are fairly common in Turing models. This means that sampling in Turing can probably benefit from using statically sized vectors and matrices from StaticArrays.jl instead of the dynamic normal Julia arrays. Beside the often superior performance of small static vectors and matrices, static arrays are also automatically compatible with the GPU stack in Julia. Currently, the main obstacle to using StaticArrays.jl is that distributions in Distributions.jl are not compatible with StaticArrays. A GSoC student would adapt the multivariate and matrix-variate distributions as well as the univariate distribution with vector parameters in Distributions.jl to make a spin-off package called StaticDistributions.jl. The student would then benchmark StaticDistributions.jl against Distributions.jl and showcase an example of using StaticDistributions.jl together with CuArrays.jl and/or CUDAnative.jl for GPU-acceleration.
GPnet extensions. One of Turing's satellite packages, GPnet, is designed to provide a comprehensive suite of Gaussian process tools. See this issue for potential tasks -- there's a lot of interesting stuff going on with GPs, and this task in particular may have some creative freedom to it.
Better chains and model diagnostics. One package that Turing (and many others) rely on heavily is MCMCChains.jl, a package designed to format, store, and analyze parameter samples generated during MCMC inference. MCMCChains is currently showing its age a little and has many bad design choices that need to be fixed. Alternatively, a student could construct a far more lightweight chain system.
Model comparison tools. Turing and its satellite packages do not currently provide a comprehensive suite of model comparison tools, a critical tool for the applied statistician. A student who worked on this project would implement various model comparison tools like LOO and WAIC, among others.
MLE/MAP tools. Maximum likelihood estimates (MLE) and maximum a posteriori (MAP) estimates can currently only be done by users through a clunky set of workarounds. A streamlined function like mle(model) or map(model) would be very useful for many of Turing's users who want to see what the MLE or MAP estimates look like, and it may be valuable to allow for functionality that allows MCMC sampling to begin from the MLE or MAP estimates. Students working on this project will work with optimization packages such as Optim.jl to make MLE and MAP estimation straightforward for Turing models.
Particle sampler improvements. Turing's development team has spent a lot of time and energy to make inference methods more modular, but Turing's particle samplers have not yet been modernized and spun off into a separate package. Two packages that resulted from this were AdvancedHMC for Hamiltonian MCMC methods, and AdvancedMH for Metropolis-Hastings style inference methods. A student who worked on this project would become very familiar with Turing's inference backend and with particle sampling methods. This is a good project for people who love making things efficient and easily extendable.

Other projects are welcome, but we do strongly recommend discussing any potential projects with members of the Turing team, as they will end up mentoring GSoC students for the duration of the project.

We're looking forward to what people are interested in!

Turing’s Blog

2019-12-14T00:00:00+00:00

All good open source projects should have a blog, and Turing is one such project. Later on, members of the Turing team may be populating this feed with posts on topics like

Interesting things you can do with Turing, or interesting things we have seen others do.
Development updates and major release announcements.
Research updates.
Explorations of Turing's internals.
Updates to Turing's satellite projects AdvancedHMC.jl or Bijectors.jl.

Stay tuned!