Coin tossing

Toss a coin 5 times. Let \(H_1 =\) “first toss is heads” and let \(H_A =\) “all 5 tosses are heads”. Therefore \(\Pr(H_1 | H_A) = 1\) (if all five tosses are heads, then the first one must by definition also be heads) and \(\Pr(H_A | H_1) = \frac{1}{16}\) (\(\frac{1}{2^4} = \frac{1}{16}\)).

However we can also use Bayes’ theorem to calculate \(\Pr(H_1 | H_A)\) using \(\Pr(H_A | H_1)\). The terms we need are:

• \(\Pr(H_A | H_1) = \frac{1}{16}\)
• \(\Pr(H_1) = \frac{1}{2}\)
• \(\Pr(H_A) = \frac{1}{32}\)

So,

\[\Pr(H_A | H_1) = \frac{\Pr(H_A | H_1) \times \Pr(H_1)}{\Pr(H_A)} = \frac{\frac{1}{16} \times \frac{1}{2}}{\frac{1}{32}} = 1\]

False positive fallacy

A test for a certain rare disease is assumed to be correct 95% of the time:

• If a person has the disease, then the test results are positive with probability \(0.95\)
• If the person does not have the disease, then the test results are negative with probability \(0.95\)

A random person drawn from a certain population has probability \(0.001\) of having the disease. Given that the person just tested positive, what is the probability of having the disease?

• \(A = {\text{person has the disease}}\)
• \(B = {\text{test result is positive for the disease}}\)
• \(\Pr(A) = 0.001\)
• \(\Pr(B | A) = 0.95\)
• \(\Pr(B | A = 0) = 0.05\)

\[ \begin{align} \Pr(\text{person has the disease} | \text{test is positive}) &= \Pr(A|B) \\ & = \frac{\Pr(A) \times \Pr(B|A)}{\Pr(B)} \\ & = \frac{\Pr(A) \times \Pr(B|A)}{\Pr(A) \times \Pr(B|A) + \Pr(A = 0) \times(B | A = 0)} \\ & = \frac{0.001 \times 0.95}{0.001 \times 0.95 + 0.999 \times 0.05} \\ & = 0.0187 \end{align} \]

Even though the test is fairly accurate, a person who has tested positive is still very unlikely (less than 2%) to have the disease. Because the base rate of the disease in the population is so low, the vast majority of people taking the test are healthy and even with an accurate test most of the positives will be healthy people.¹

Example: Bernoulli random variable

Let \(X_1, \ldots, X_n \sim \text{Bernoulli} (p)\). Suppose we take the uniform distribution \(f(p) = 1\) as a prior. By Bayes’ theorem, the posterior has the form

\[ \begin{align} f(p | x^n) &\propto f(p) \Lagr_n(p) \\ &= p^s (1 - p)^{n - s} \\ &= p^{s + 1 - 1} (1 - p)^{n - s + 1 - 1} \end{align} \]

where \(s = \sum_{i=1}^n x_i\) is the number of successes. Importantly, a random variable has a Beta distribution with parameters \(\alpha\) and \(\beta\) if its density is

\[f(p; \alpha, \beta) = \frac{\Gamma(\alpha + \beta)}{\Gamma(\alpha) \Gamma(\beta)}p^{\alpha - 1} (1 - p)^{\beta - 1}\]

We can see that the posterior for \(p\) is a Beta distribution with parameters \(s + 1\) and \(n - s + 1\). That is,

\[f(p | x^n) = \frac{\Gamma(n + 2)}{\Gamma(s + 1) \Gamma(n - s + 1)}p^{(s + 1) - 1} (1 - p)^{(n - s + 1) - 1}\]

We write this as

\[p | x^n \sim \text{Beta} (s + 1, n - s + 1)\]

Notice that we have figured out the normalizing constant \(c_n = \frac{\Gamma(n + 2)}{\Gamma(s + 1) \Gamma(n - s + 1)}\) without actually doing the integral \(\int \Lagr_n(p) f(p) dp\). The mean of a \(\text{Beta}(\alpha, \beta)\) distribution is \(\frac{\alpha}{\alpha + \beta}\), so the Bayes estimator is

\[\bar{p} = \frac{s + 1}{n + 2}\]

We can rewrite the estimator as

\[\bar{p} = \lambda_n \hat{p} + (1 - \lambda_n) \tilde{p}\]

where \(\hat{p} = \frac{s}{n}\) is the MLE, \(\tilde{p} = \frac{1}{2}\) is the prior mean, and \(\lambda_n = \frac{n}{n + 2} \approx 1\). So we can think of the MLE as the estimate for \(p\) with a flat prior. If we have a non-flat prior, then our estimate \(\bar{p}\) is a weighted average between the prior and the MLE.

A 95% credible interval can be obtained by numerically finding \(a\) and \(b\) such that \(\int_a^b f(p | x^n) dp = 0.95\).

Suppose that instead of a uniform prior, we use the prior \(p \sim \text{Beta} (\alpha, \beta)\). If we repeat the calculations from before, we see that \(p | x^n \sim \text{Beta} (\alpha + s, \beta + n - s)\). The flat prior is the special case with \(\alpha = \beta = 1\). The posterior mean is

\[\bar{p} = \frac{\alpha + s}{\alpha + \beta + n} = \left( \frac{n}{\alpha + \beta + n} \right) \hat{p} + \left( \frac{\alpha + \beta}{\alpha + \beta + n} \right) p_0\]

where \(p_0 = \frac{\alpha}{\alpha + \beta}\) is the prior mean.

Example: coin tossing

Now let’s look at a coin tossing problem. There are three types of coins with different probabilities of landing heads when tossed.

• Type \(A\) coins are fair, with \(p = 0.5\) of heads
• Type \(B\) coins are bent, with \(p = 0.6\) of heads
• Type \(C\) coins are bent, with \(p = 0.9\) of heads

Suppose I have a drawer containing 5 coins: 2 of type \(A\), 2 of type \(B\), and 1 of type \(C\). I reach into the drawer and pick a coin at random. Without showing you the coin I flip it once and get heads. What is the probability it is type \(A\)? Type \(B\)? Type \(C\)?

Example: Bernoulli random variable

Let \(X_1, \ldots, X_n \sim \text{Bernoulli} (p)\) and \(f(p) = 1\) so that \(p | X^n \sim \text{Beta} (s + 1, n - s + 1)\) with \(s = \sum_{i=1}^n x_i\). Let \(\psi = \log \left( \frac{p}{1 - p} \right)\) (i.e. the log-odds). If we wanted to calculate the PMF and CDF of \(\psi | x^n\), we could do a lot of calculus and analytic math to solve for these equations.² Alternatively, we can approximate the posterior for \(\psi\) without doing any calculus.

1. Draw \(P_1, \ldots, P_B \sim \text{Beta} (s + 1, n - s + 1)\).
2. Let \(\psi_i = \log \left( \frac{P_i}{1 - P_i} \right)\), for \(i = 1, \ldots, B\)

Now \(\psi_1, \ldots, \psi_B\) are IID draws from \(h(\psi | x^n)\). A histogram of these values provides an estimate of \(h(\psi | x^n)\).

Improper priors

Let \(X \sim N(\theta, \sigma^2)\) with \(\sigma\) known. Suppose we adopt a flat prior \(f(\theta) \propto c\) where \(c > 0\) is a constant. Note that \(f(\theta) d\theta = \infty\), so this is not a probability density in the usual sense (otherwise it would integrate to 1). Such a prior is called an improper prior. However, we can still carry out Bayes’ theorem and compute the posterior density by multiplying the prior and the likelihood:

\[f(\theta) \propto \Lagr_n(\theta) f(\theta) = \Lagr_n(\theta)\]

This gives \(\theta | X^n \sim N(\bar{X}, \sigma^2 / n)\) and the resulting point and interval estimators agree exactly with their frequentist counterparts. In general, improper priors are not a problem as long as the resulting posterior is a well-defined probability distribution.

Flat priors are not invariant

Let \(X \sim \text{Bernoulli} (p)\) and suppose we use the flat prior \(f(p) = 1\). This flat prior represents our lack of knowledge about \(p\) before the experiment. Now let \(\psi = \log(p / (1 - p))\). This is a transformation of \(p\) and we can compute the resulting distribution for \(\psi\)

\[f_\Psi (\psi) = \frac{e^\psi}{(1 + e^\psi)^2}\]

which is not flat. But if we are ignorant of \(p\), then we are also ignorant about \(\psi\) so we should use a flat prior for \(\psi\). This is a contradiction. In short, the notion of a flat prior is not well defined because a flat prior on a parameter does not imply a flat prior on a transformed version of the parameter. Flat priors are not transformation invariant.

Example: comparing two binomials

Suppose we have \(n_1\) control patients and \(n_2\) treatment patients and that \(X_1\) control patients survive while \(X_2\) treatment patients survive. We want to estimate \(\tau = g(p_1, p_2) = p_2 - p_1\). Then,

\[X_1 \sim \text{Binomial} (n_1, p_1) \, \text{and} \, X_2 \sim \text{Binomial} (n_2, p_2)\]

If \(f(p_1, p_2) = 1\), the posterior is

\[f(p_1, p_2 | x_1, x_2) \propto p_1^{x_1} (1 - p_1)^{n_1 - x_1} p_2^{x_2} (1 - p_2)^{n_2 - x_2}\]

Notice that

\[f(p_1, p_2 | x_1, x_2) = f(p_1 | x_1) f(p_2 | x_2)\]

where

\[f(p_1 | x_1) \propto p_1^{x_1} (1 - p_1)^{n_1 - x_1} \, \text{and} \, f(p_2 | x_2) \propto p_2^{x_2} (1 - p_2)^{n_2 - x_2}\]

which implies that \(p_1\) and \(p_2\) are independent under the posterior. Also

\[ \begin{align} p_1 | x_1 &\sim \text{Beta} (x_1 + 1, n_1 - x_1 + 1) \\ p_2 | x_2 &\sim \text{Beta} (x_2 + 1, n_2 - x_2 + 1) \end{align} \]

If we simulate

\[ \begin{align} P_{1,1}, \ldots, P_{1,B} &\sim \text{Beta} (x_1 + 1, n_1 - x_1 + 1) \\ P_{2,1}, \ldots, P_{2,B} &\sim \text{Beta} (x_2 + 1, n_2 - x_2 + 1) \end{align} \]

Then \(\tau_b = P_{2,b} - P_{1,b}, \, b = 1, \ldots, B\) is a sample from \(f(\tau | x_1, x_2)\).

Critique of Bayesian inference

1. The subjective prior is subjective. There is no single method for choosing a prior, so different (well-intentioned) people will produce different priors and therefore arrive at different posteriors and conclusions. If you accept the premise of subjective priors, you still need good information to create a well-defined prior distribution.
2. Philosophically, some object to assigning probabilities to hypotheses as hypotheses do not constitute outcomes of repeatable experiments in which one can measure long-term frequency. Rather, a hypothesis is either true or false, regardless of whether one knows which is the case.
   • A coin is either fair or unfair
   • Treatment 1 is either better or worse than treatment 2
   • The sun will or will not come up tomorrow
   • I will either win or not win the lottery
3. For many parametric models with large samples, Bayesian and frequentist methods give approximately the same inferences. Since frequentist methods are historically more common and easier to estimate, there is no reason to go through the steps of Bayesian inference.
4. Bayesian inference depends entirely on the likelihood function. In high dimensional and nonparametric methods, the likelihood function may not yield accurate inferences.

Defense of Bayesian inference

1. The probability of hypotheses is exactly what we need to make decisions. When the doctor tells me a screening test came back positive for a disease, what I really want to know is the probability of the hypothesis “I’m sick”.
2. Bayes’ theorem is logically rigorous (once we obtain a prior).
3. By testing different priors we can see how sensitive our results are to the choice of prior.
4. It is easy to communicate a result framed in terms of probabilities of hypotheses (try explaining the result of a null hypothesis test to a layperson).
5. Priors can be defended based on the assumptions made to arrive at it.
6. Evidence derived from the data is independent of notions about “data more extreme” that depend on the exact experimental setup.
7. Data can be used as it comes in. We don’t have to wait for every contingency to be planned for ahead of time.