When an error bar can't warn you

Bridge sampling estimates the marginal likelihood of a Bayesian model, and reports a Monte Carlo standard error (MCSE) alongside the estimate. In Bridge Sampling Diagnostics, Aki Vehtari and I show that this error estimate has a ceiling: however badly the estimate fails, the MCSE can’t exceed $\sqrt{\log 3} \approx 1.05$.

The figure runs bridge sampling on a Gaussian posterior whose answer is known, so you can compare what the MCSE reports with how far off the estimate really is. Each point averages 20 runs.

What you’re seeing

With the standard proposal, a Gaussian fitted with the sample covariance of the draws, the two lines agree until the sample covariance gets too noisy for the dimension. Then the reported error flattens below the ceiling while the actual error keeps climbing. More draws push the breakdown further out, but slowly. The MCSE does cross 0.3 shortly before it breaks down, which is why the paper recommends trusting it only below that.

The hybrid proposal combines the sample covariance with the inverse covariance of the posterior’s gradients, taking their midpoint in the affine-invariant geometry of covariance matrices. For a Gaussian posterior that midpoint is the true covariance, which is why its lines stay low and together here. Real posteriors aren’t Gaussian, but across the 41 posteriors in the paper the hybrid proposal kept the actual error within a factor of 1.4 of the reported one.

How it’s computed

The posterior is $\mathcal{N}(0, \Sigma)$ with variances spread from 0.1 to 10 and its normalising constant dropped. Each run draws independent samples, fits the proposal on half of them, and uses the other half, plus as many proposal draws, for the estimate. The estimator uses the bridge function of Meng and Wong (1996), starts from the median log ratio as the bridgesampling R package does, and computes the MCSE with the formula from the paper. Everything runs in your browser.

The code for the paper’s experiments is on GitHub.