Member-only story
Introduction to Sampling Methods
Implementing inverse transform sampling, rejection sampling and importance sampling in Python
In this post we’ll discuss how to sample from a probability distribution. This is a common need, in an ML setting this is most frequently used to run inference for probabilistic models. However, due to the distributions being very complex, this is often intractable. Thus, main focus of this post is introducing approximate methods for this task, in particular using numerical sampling, known as Monte Carlo methods.
Still, for introductory purposes we’ll introduce inverse transform sampling first, which allows exact inference for arbitrary, tractable distributions. Then, we’ll shift our focus to approximate methods, allowing sampling or moment estimation for (near) arbitrary distributions: we start with rejection sampling and then move to importance sampling.
Note this post is the first in a series familiarising readers with [1], and this post in particular covers parts of Chapter 11: Sampling Methods.
Inverse Transform Sampling
The inverse transform sampling method allows sampling from any distribution for which we know how to calculate the inverse of the cumulative distribution function (CDF). It entails sampling y…
