Bayesian Inference Notes

1. Bayes Theorem

1.1. Bayes theorem

1.2. Prior predictive distribution

1.3. Posterior predictive distribution

2. Fundamental Distributions

3. Functions

3.1. Beta Function

Properties:

• $𝐵\left({𝑧}_1,{𝑧}_2\right)=𝐵\left({𝑧}_2,{𝑧}_1\right)$
• $𝐵\left({𝑧}_1,{𝑧}_2\right)=\frac{\mathrm{\Gamma }\left({𝑧}_1\right)\mathrm{\Gamma }\left({𝑧}_2\right)}{\mathrm{\Gamma }\left({𝑧}_1+{𝑧}_2\right)}$
• $𝐵\left(𝑚,𝑛\right)=\frac{\left(𝑚-1\right)!\left(𝑛-1\right)!}{\left(𝑚+𝑛-1\right)!}=\frac{𝑚+𝑛}{𝑚𝑛}/\left(\genfrac{}{}{0ex}{}{𝑚+𝑛}{𝑚}\right)$

4. Conjugate Prior

The idea of a conjugate prior is that for a given likelihood we choose a prior distribution such that, after observing data and applying Bayes’ theorem, the posterior distribution belongs to the same family as the prior.

That is, if $𝑝\left(𝜃\right)$ and $𝑝\left(𝜃|𝐷\right)$ have the same distributional form, then the prior is called a conjugate prior for the likelihood model.

This is useful because it makes Bayesian updating analytically tractable. Instead of performing difficult integration or numerical approximation, we can often derive the posterior parameters in closed form.

5. Conjugate Prior for Exponential Families

Note general exponential family:

For the data set $𝐷=\left\{{𝑦}_1,\dots ,{𝑦}_{𝑛}\right\}$ of i.i.d. observations, the likelihood is

So conjugate prior for that likelihood is

Posterior is

6. Proper and Improper Prior Distributions

A prior is called proper if it is a valid probability distribution:

And improper if

• If a prior is proper, so must the posterior.
• If a prior is improper, the posterior could be proper or improper.

In theory, all priors are acceptable, as long as the posterior is proper.

7. Fisher Information Matrix

8. Jeffreys’ Prior

9. Pivotal Quantities

For the binomial and other single-parameter models, different principles give (slightly) different noninformative prior distributions. But for two cases—location parameters and scale parameters—all principles seem to agree[1].

9.1. Location Parameter

9.2. Scale Parameter

10. Predictive Accuracy

People care about the accuracy in two different ways. First to assume that the model is all we known and check posterior predictions. The second is to compare several candidate models. Even if all of the models being considered have mismatches with the data, it can be informative to evaluate their predictive accuracy, compare them, and consider where to go next[2].

11. KL Divergence

12. Linear Algebra

12.1. Convex Combination

A subset $𝐴$ of a vector space $𝑉$ is said to be convex if $𝜆\overrightarrow{𝑥}+\left(1-𝜆\right)\overrightarrow{𝑦}$ for all vectors $\overrightarrow{𝑥},\overrightarrow{𝑦}\in 𝐴$, and all scalars $𝜆$ in $\left[0,1\right]$.

Via induction, this can be seen to be equivalent to the requirement that ${𝜆\overrightarrow{𝑥}}_1,{𝜆\overrightarrow{𝑥}}_2,\dots ,{𝜆\overrightarrow{𝑥}}_{𝑛}\in 𝐴$ for all vectors ${\overrightarrow{𝑥}}_1,{\overrightarrow{𝑥}}_2,\dots ,{\overrightarrow{𝑥}}_{𝑛}\in 𝐴$, and for all scalars ${𝜆}_1,{𝜆}_2,\dots ,{𝜆}_{𝑛}\ge 0$ such that $\sum {𝑘}_{𝑖}=1$.

13. Reversible Jump MCMC

Reversible Jump Markov chain Monte Carlo (RJMCMC) samples jointly over a family of candidate models $\mathrm{ℳ︀}=\left\{{𝑀}_{𝑘}:𝑘\in 𝐾\right\}$ and their parameters. Under model ${𝑀}_{𝑘}$, let ${\overrightarrow{𝜃}}_{𝑘}\in {\mathrm{\Theta }}_{𝑘}\subseteq {ℝ}^{{𝑛}_{𝑘}}$. For a fixed observed data set $𝐷$, the joint posterior is the product of the likelihood and the joint prior, $𝑝\left(𝑘,{\overrightarrow{𝜃}}_{𝑘}\right)=𝑝\left({\overrightarrow{𝜃}}_{𝑘}|𝑘\right)𝑝\left(𝑘\right)$[3]:

Unlike ordinary MCMC, the state space is the disjoint union ${⊍}_{𝑘\in 𝐾}\left(\left\{𝑘\right\}\times {\mathrm{\Theta }}_{𝑘}\right)$, so a jump between ${𝑀}_{𝑘}$ and ${𝑀}_{{𝑘}^{\prime }}$ may change the dimension of the parameter vector.

13.1. Sampler Configuration

An RJMCMC sampler is specified by the following components:

• a within-model kernel ${𝐾}_{𝑘}$ that leaves $𝜋\left({\overrightarrow{𝜃}}_{𝑘}|𝑘,𝐷\right)$ invariant
• jump-selection probabilities $𝑟\left({𝑘}^{\prime }|𝑘\right)$ for ${𝑘}^{\prime }\ne 𝑘$, satisfying ${\sum }_{{𝑘}^{\prime }\ne 𝑘}𝑟\left({𝑘}^{\prime }|𝑘\right)\le 1$
• auxiliary proposal densities ${𝑞}_{𝑘,{𝑘}^{\prime }}\left(\overrightarrow{𝑢}|{\overrightarrow{𝜃}}_{𝑘}\right)$
• a differentiable bijection ${𝑇}_{𝑘,{𝑘}^{\prime }}$ paired with its reverse ${𝑇}_{{𝑘}^{\prime },𝑘}$ ${𝑇}_{𝑘,{𝑘}^{\prime }}:{\mathrm{\Theta }}_{𝑘}\times {\mathrm{𝒰︀}}_{𝑘,{𝑘}^{\prime }}\to {\mathrm{\Theta }}_{{𝑘}^{\prime }}\times {\mathrm{𝒰︀}}_{{𝑘}^{\prime },𝑘}$, with the dimension-matching condition ${𝑛}_{𝑘}+{𝑑}_{𝑘,{𝑘}^{\prime }}={𝑛}_{{𝑘}^{\prime }}+{𝑑}_{{𝑘}^{\prime },𝑘}$, where ${𝑑}_{𝑘,{𝑘}^{\prime }}=dim{\mathrm{𝒰︀}}_{𝑘,{𝑘}^{\prime }}$.

Starting from $\left(𝑘,{\overrightarrow{𝜃}}_{𝑘}\right)$, select a destination model ${𝑘}^{\prime }$ with probability $𝑟\left({𝑘}^{\prime }|𝑘\right)$, draw $\overrightarrow{𝑢}\sim {𝑞}_{𝑘,{𝑘}^{\prime }}\left(\cdot |{\overrightarrow{𝜃}}_{𝑘}\right)$, and apply

The corresponding Jacobian factor is:

Accept the proposed state $\left({𝑘}^{\prime },{\overrightarrow{𝜃}}_{{𝑘}^{\prime }}\right)$ with probability:

The posterior’s normalizing constant cancels from this ratio. If the proposal is rejected, retain $\left(𝑘,{\overrightarrow{𝜃}}_{𝑘}\right)$. With the remaining probability $1-{\sum }_{{𝑘}^{\prime }\ne 𝑘}𝑟\left({𝑘}^{\prime }|𝑘\right)$, apply ${𝐾}_{𝑘}$ to update the parameters within the current model. Together, these updates preserve $𝜋\left(𝑘,{\overrightarrow{𝜃}}_{𝑘}|𝐷\right)$ as the stationary distribution.