Probability and Statistics: A Geometric Foundation
This lecture includes interactive SymPy cells that verify key results symbolically. SymPy is a Python library for symbolic mathematics. Loading it fetches a WebAssembly Python runtime (approx. 15 MB, cached after first load). You can also load it on demand from any code cell below.
This article includes interactive NumPy cells that verify the numerical examples in each section. NumPy runs in a WebAssembly Python runtime (approx. 5 MB, cached after first load). You can also load it on demand from any code cell below.
1. Probability as measure
Probability is usually first presented as rules for computing with fractions: count the favourable outcomes, divide by the total, apply Bayes' formula. This works for coins and dice and fails for continuous quantities, infinite sample spaces and the convergence of random variables. Probability is a theory of measurement. The question "what is the probability of an event?" asks for the size of a set. Answering it requires the infrastructure of the Lebesgue integral: -algebras, measures and measurable functions.
This section rebuilds the foundations from measure theory. The law of large numbers and the central limit theorem then become theorems about integrals, and the passage from discrete to continuous distributions needs only a change of reference measure. The Radon-Nikodym derivative at the end of this section makes that change precise and begins the geometric perspective of the rest of the post.
A probabilistic model needs two structures: a specification of the observable events and an assignment of sizes to them.
Let be a non-empty set. A collection of subsets of is a -algebra if it satisfies:
- .
- If , then .
- If , then .
The pair is called a measurable space.
Condition (iii) is closure under countable unions. It separates -algebras from algebras of sets and is the condition needed to take limits. Without it convergence of events is undefined and no asymptotic statement can be made.
Every set admits at least two sigma-algebras: the trivial sigma-algebra , in which only the certain and impossible events are observable, and the power set , in which every subset is observable. For uncountable spaces like , the power set is too large to support a useful measure, and we use the Borel sigma-algebra , generated by the open intervals. The choice of sigma-algebra encodes what we can observe: a coarser sigma-algebra means less information.
Let be a measurable space. A function is a probability measure if:
- .
- Countable additivity: for any sequence of pairwise disjoint sets in ,
The triple is called a probability space.
The axioms are Kolmogorov's (1933). They say nothing about equally likely outcomes, frequencies or subjective beliefs. They say only that probability is a normalised countably additive set function. Every interpretation of probability (frequentist, Bayesian, decision-theoretic) commits to what means and the framework is agnostic among them.

A random variable assigns a numerical value to each outcome.
Let be a measurable space. A random variable is a measurable function , where is the Borel sigma-algebra on . That is, for every Borel set ,
Condition [eq:measurability] makes the question "what is the probability that falls in ?" well posed: the preimage belongs to so can evaluate it. A function that is not measurable can produce sets of undefined probability. This pathology arises when probabilities are assigned to all subsets of (the Vitali construction).
Let be a random variable on the probability space . The expectation of is defined as the Lebesgue integral of with respect to :
constructed in three stages. First, for a simple function with , define . Second, for a non-negative measurable , define the integral as the supremum over all simple functions bounded above by . Third, for a general , decompose where and , and set , provided at least one of these is finite.
Expectation is the Lebesgue integral with respect to a probability measure. Every theorem about Lebesgue integrals (dominated convergence, monotone convergence, Fubini-Tonelli) applies at once to expectations.
Countable additivity gives probabilities of limits as limits of probabilities.
Let be a probability space. If is an increasing sequence of events with , then
Symmetrically, if is a decreasing sequence with , then .
Proof
Define and for . The sets are pairwise disjoint, and . By countable additivity [eq:countable-additivity],
where by finite additivity (a consequence of countable additivity) and the fact that .
The decreasing case follows by applying the increasing case to the complements and using .
The simplest non-trivial probability space illustrates these definitions.
Take and (the power set). For a parameter , define by
This determines on all of by additivity: and . The resulting probability space is the Bernoulli space .
The identity function defined by is trivially measurable (every subset of is in ). Its expectation is
and its variance is
The Bernoulli space is minimal and contains the structure of every model: a sample space, a -algebra encoding what we can observe, a measure encoding our beliefs, and a random variable converting outcomes to numbers.
1.1. Change of reference by the Radon-Nikodym derivative
So far we have worked with a single probability measure . In practice we constantly compare two measures on the same measurable space: a prior and a posterior, a model and the data-generating distribution, a risk-neutral and a physical measure. The Radon-Nikodym derivative makes such comparisons precise.
Let and be -finite measures on . We say is absolutely continuous with respect to , written , if for every ,
When , the Radon-Nikodym theorem ([radon-nikodym-thm]) guarantees the existence of a measurable function such that for all ,
The function , unique -almost everywhere, is called the Radon-Nikodym derivative of with respect to and is written .
Let and be -finite measures on with . Then there exists a measurable function such that
The function is unique -almost everywhere.
Proof
We sketch von Neumann's proof via the Riesz representation theorem. Assume first that and are finite. The -finite case follows by partitioning into countably many sets of finite measure and summing the densities.
Define , so that both and are absolutely continuous with respect to . For any , the map
defines a bounded linear functional on (bounded because by and Cauchy-Schwarz). By the Riesz representation theorem, there exists such that
Taking for gives . Since for all , we have -almost everywhere. Similarly, implies -a.e.
On the set where , we can write , which rearranges to and so . On the identity reads for . Hence , and by . Setting on and on yields the desired density. For uniqueness let and both satisfy the identity. Then for every , and taking and gives -a.e.
The Radon-Nikodym derivative is the infinitesimal ratio of -volume to -volume. When is the Lebesgue measure on , the derivative is the probability density function of a first course in statistics. A density is a relative quantity: it depends on the reference measure and changes with it. This relativity is the first sign that probability has a geometric character. The Fisher metric of Section 4 makes it explicit.
1.2. Families of distributions as curves
Consider the family of all Bernoulli distributions parametrised by . Each value of defines a probability measure on , and these measures live in the probability simplex
The Bernoulli family traces a curve in this simplex: . The simplex contains all distributions on a finite sample space and parametric families are curves or surfaces within it. The curve passes through every interior point of so the Bernoulli family is a full parametrisation of the open simplex. Larger sample spaces give higher-dimensional simplices in which parametric families are typically lower-dimensional submanifolds.

Entropy locates the point of the curve that is "most uncertain".
The Shannon entropy of is
This function is maximised at , where bit, and vanishes at and (certainty). High entropy means an unpredictable result and zero entropy a determined one. In the simplex picture, the maximum-entropy point sits at the midpoint of , equidistant from both vertices.
from sympy import symbols, log, simplify, Rational
p = symbols('p', positive=True)
# Expectation of Ber(p)
EX = 0*(1-p) + 1*p
print(f"E[X] = {EX}")
# Variance
VarX = simplify(EX*(1-EX)) # E[X^2] - (E[X])^2 = p - p^2
print(f"Var(X) = {VarX}")
# Shannon entropy (base 2)
H = simplify(-p*log(p, 2) - (1-p)*log(1-p, 2))
print(f"H(p) = {H}")
# Value at p = 1/2
print(f"H(1/2) = {H.subs(p, Rational(1,2))} bits")Bernoulli expectation, variance, and entropy
2. Estimation theory
Statistics asks what observed data say about the mechanism that generated them. Answering requires a formal statistical model and a theory of estimation.
2.1. Statistical models and likelihood
A statistical model formalises a "family of possible data-generating processes" by indexing probability measures with a parameter.
A statistical model is a family of probability measures on a measurable space , indexed by a parameter space . If each has a density with respect to a common dominating measure , the model is called a parametric model.
The parametric assumption that the family is smoothly indexed by a finite-dimensional parameter makes classical estimation theory tractable. The dominating measure is typically Lebesgue measure (for continuous models) or counting measure (for discrete models). It allows us to work with densities.
Given a parametric model, an observation singles out a function on the parameter space: how "compatible" is each with what we saw?
Given observed data , the likelihood function is
the density evaluated at , viewed as a function of . The log-likelihood is
For i.i.d. observations , the log-likelihood is
The likelihood is not a probability distribution over . It does not integrate to one and has no probabilistic interpretation on the parameter space (that requires a prior, deferred to the Bayesian section). It scores compatibility. The log-likelihood is preferred in practice because products become sums and sums are numerically stable.
Maximum likelihood chooses the parameter value that makes the observed data most likely.
The maximum likelihood estimator (MLE) is
When the log-likelihood is differentiable, the MLE solves the score equation:
The notation appears throughout estimation theory. Take a real-valued function defined on a set .
The maximum of is the largest value attained:
The argmax of is the point (or set of points) at which that maximum is attained:
returns how large the function gets; returns where it gets there. When the maximiser is unique (as it typically is for log-likelihoods of regular models) denotes that element.
The MLE is a function of the data and hence a random variable. Its consistency, efficiency and asymptotic normality are established below.
2.2. Sufficient statistics
Some functions of the data compress it without losing information about the parameter and others discard relevant information. Sufficiency makes this precise.
A statistic is sufficient for if the conditional distribution of given does not depend on . Intuitively captures all the information in the data about : once is known the remaining randomness in is noise.
Sufficiency is often impractical to verify from the conditional distribution. The factorisation criterion is checkable.
A statistic is sufficient for if and only if the density factors as
for some functions and , where does not depend on .
Proof
We give the proof for discrete ; the general case needs conditional densities and follows the same pattern.
If is sufficient, then . The first factor does not depend on (by sufficiency); call it . The second depends on only through ; call it .
If , then
which does not depend on .
2.3. Exponential families
Exponential families contain most standard parametric models and have geometric structure.
A parametric family is an exponential family if the density can be written
where is the natural parameter, is the sufficient statistic, is the log-partition function (the normaliser), and is the base measure.
By the factorisation theorem is sufficient for . The log-partition function is convex (it is the logarithm of a moment-generating function) and its derivatives are the moments of the sufficient statistic.
For , write
So (log-odds), , , and .
For , the density is . Rewrite as
where , , , and
The log-partition function generates the cumulants of the sufficient statistic.
For an exponential family with log-partition function ,
Proof
The normalisation condition is
so . Differentiating both sides with respect to :
Dividing by yields . Differentiating again gives the covariance:
Since is convex, the Hessian is positive semi-definite, consistent with the covariance matrix being non-negative definite.
2.4. Fisher information
The curvature of the log-likelihood at the true parameter value measures the information the data contain about . The Fisher information formalises it.
The Fisher information at is
The quantity is the score function. Since (the score has mean zero under the model), the Fisher information equals the variance of the score: . Under regularity conditions (interchange of differentiation and integration),
For a multi-parameter model with , the Fisher information matrix has entries
For , the log-density is . The score is
Since , we obtain
Here diverges as or : near certainty each observation is maximally informative.
For with , the Fisher information matrix is
The diagonal structure reflects the orthogonality of the location and scale parameters. Reparametrising to yields .
from sympy import symbols, log, diff, simplify, Function
p, x = symbols('p x')
# Log-likelihood for single Bernoulli observation
logp = x*log(p) + (1-x)*log(1-p)
score = diff(logp, p)
print(f"Score: {score}")
# Fisher info: -E[d^2/dp^2 log p] = -E[second derivative]
d2 = diff(logp, p, 2)
# E[x] = p, E[1-x] = 1-p
I_p = simplify(-d2.subs(x, p)) # E[d2] by replacing x with E[x]=p
print(f"I(p) = {I_p}")Fisher information for the Bernoulli family
from sympy import symbols, log, diff, simplify, pi, sqrt, exp, integrate, oo
x = symbols('x', real=True)
mu = symbols('mu', real=True)
sigma = symbols('sigma', positive=True)
# Log-density of N(mu, sigma^2)
logp = -log(sigma*sqrt(2*pi)) - (x - mu)**2 / (2*sigma**2)
density = exp(logp)
# Score components
s_mu = diff(logp, mu)
s_sig = diff(logp, sigma)
print(f"Score (mu): {simplify(s_mu)}")
print(f"Score (sigma): {simplify(s_sig)}")
# Fisher info: I_ij = E[s_i * s_j], integrated against the density
E = lambda f: simplify(integrate(f * density, (x, -oo, oo)))
print(f"I_11 = E[s_mu^2] = {E(s_mu**2)}")
print(f"I_12 = E[s_mu s_sigma] = {E(s_mu*s_sig)}")
print(f"I_22 = E[s_sigma^2] = {E(s_sig**2)}")Fisher information matrix for the Normal family
The Fisher information has a geometric interpretation developed in Section 4.
The Fisher information measures the curvature of the log-likelihood at . A sharply peaked log-likelihood (high ) means the data is informative about ; a flat log-likelihood (low ) means the data tells us little. For the Bernoulli family, diverges at the boundary of the parameter space: near and a small change in is easy to detect. For the Normal family the Fisher information matrix defines a two-dimensional geometry on the plane. Section 4 shows that this matrix is a Riemannian metric. Its Riemann curvature is a different object from the curvature of the log-likelihood and is computed in Section 4.5.
2.5. Mean squared error and the Cramer-Rao bound
Comparing estimators requires a decomposition of estimation error into systematic error (bias) and random fluctuation (variance).
For an estimator of :
- Bias: .
- Mean squared error: .
An estimator is unbiased if for all .
Reducing bias (e.g. by shrinkage) may increase variance and conversely. Among unbiased estimators the variance has a sharp lower bound.
An unbiased estimator is efficient if
i.e. it achieves the Cramer-Rao lower bound with equality.
For any unbiased estimator of based on i.i.d. observations,
Proof
Consider a single observation. The score satisfies and . For any unbiased estimator with , differentiate both sides with respect to :
Since , this gives , i.e. (using ). By the Cauchy-Schwarz inequality:
Therefore . For i.i.d. observations, the total Fisher information is and so .

In an exponential family the MLE of achieves the Cramer-Rao bound exactly. Outside exponential families the bound may be unattainable by every unbiased estimator. The MLE still achieves it asymptotically.
2.6. Asymptotic efficiency of the MLE
The MLE is the best estimator for large samples in the following sense.
Under regularity conditions (the parameter space is open, the support of does not depend on , and ), the MLE is consistent () and asymptotically normal:
In particular the MLE is asymptotically efficient: its variance achieves the Cramer-Rao lower bound in the limit.
Proof sketch
Taylor-expand the score around the true parameter . At the MLE, the score vanishes:
By the law of large numbers, . By the central limit theorem, . Solving for :
Uniform convergence of the Taylor remainder follows from the regularity conditions and makes the argument rigorous.
The Fisher information bounds estimation precision (Cramer-Rao) and the MLE attains the bound for large samples. The asymptotic covariance of the MLE is the inverse of the Fisher information matrix. Section 4 identifies the Fisher information matrix as a Riemannian metric on the statistical manifold. The asymptotic covariance is therefore the inverse metric.
3. Inference and testing
Estimation says what to believe about a parameter and testing says what can be excluded. Hypothesis testing is a formal framework for decisions under uncertainty and follows from the likelihood theory above.
The question changes from "what is the best estimate of ?" to "is a particular value of compatible with the data?" Answering it needs a decision procedure with quantified error rates.
3.1. Testing framework
A hypothesis test consists of:
- A null hypothesis and an alternative hypothesis , where and partition the parameter space ,
- A test statistic , and
- A critical region such that we reject if .
A procedure that divides the sample space into "reject" and "fail to reject" regions makes two kinds of error. Testing theory controls them.
The size (Type I error rate) of a test is
The Type II error rate is
The power function is
A good test has small (rarely rejects a true null) and large power (reliably detects true alternatives). Shrinking the critical region reduces but also reduces power. The Neyman-Pearson framework fixes and maximises power.

The p-value summarises the evidence against in a single number.
The p-value is the probability, under , of observing a test statistic at least as extreme as the one realised:
For a simple and a continuously distributed , the p-value is uniformly distributed on under . For a discrete it is stochastically larger than uniform.
Uniformity says that under the p-value falls below any threshold with probability exactly . Hence rejecting when is a test of size (at most in the discrete case).
3.2. Optimal tests
Which test of a given size has the greatest power? The Neyman-Pearson lemma answers this for simple hypotheses.
For testing the simple hypothesis against the simple alternative at size , the most powerful test rejects when
where is chosen so that .
Proof
Let denote the Neyman-Pearson test (reject when ) and let be any other test of size . We need to show .
Consider the integral
In the region , we have and , so since .
In the region , we have and , so
since and .
Therefore the integral is non-negative. Expanding:
The right-hand side is since has size . Hence , and the Neyman-Pearson test is most powerful.
The likelihood ratio is the sufficient statistic for distinguishing two simple hypotheses. contains all the information relevant to the decision.

The Neyman-Pearson lemma handles simple vs simple hypotheses. For composite alternatives (e.g., ) the model must satisfy a stronger condition.
If a family has monotone likelihood ratio in , then the uniformly most powerful (UMP) test of against at size rejects when , where satisfies .
Proof
Fix any . By the Neyman-Pearson lemma, the most powerful test of vs rejects when
for some threshold . Since the family has monotone likelihood ratio in , the ratio is non-decreasing in . Therefore
for some that depends only on (through ) and not on . Since the rejection region does not depend on the specific alternative , the test is UMP against the entire composite alternative . Monotone likelihood ratio also makes the power function non-decreasing. So the size over the composite null is attained at and equals .
A one-parameter exponential family whose natural parameter is increasing has monotone likelihood ratio in . One-sided UMP tests therefore exist for all standard one-parameter exponential family models.
3.3. Large-sample testing
When exact UMP tests are unavailable (as with two-sided or multi-parameter problems) the likelihood ratio test is an asymptotically optimal alternative.
For testing , where is an -dimensional subspace of the -dimensional parameter space , the likelihood ratio statistic
converges in distribution to under as .
Proof sketch
Taylor-expand the log-likelihood around the unrestricted MLE :
where the first-order term vanishes because . Therefore
By the asymptotic normality of the MLE, . Write . It is asymptotically . To leading order is the -orthogonal projection of onto the tangent space of . In the coordinates of this is an orthogonal projection onto an -dimensional subspace, and is the squared norm of the component of in the -dimensional complement. That squared norm follows a distribution.
Wilks' theorem gives an asymptotic null distribution that depends only on the dimensions of the full and restricted parameter spaces and not on the true parameter value. This universality makes the likelihood ratio test the default in most applied settings.
3.4. Confidence regions
Confidence regions connect testing and estimation.
A confidence region for is a random set such that
Confidence regions are often built from quantities with parameter-free distributions.
A pivot is a function of the data and the parameter whose distribution does not depend on . If has known distribution , then
is a confidence region.
Tests and confidence regions are dual.
There is a duality between hypothesis tests and confidence regions:
Proof
By construction, if and only if we fail to reject at level . When is the true parameter, the test rejects with probability at most . Therefore
This holds for every , so is a valid confidence region.
Every test at level generates a confidence region (by inverting the acceptance region), and every confidence region generates a family of tests (by checking membership).
3.5. Examples
Consider testing against with coin flips. The exact binomial test rejects if or , where and are chosen so the total rejection probability under is at most .
The likelihood ratio approach rejects when
For moderate the two tests give similar (but not identical) rejection regions. The exact test controls size precisely; the LRT relies on the approximation from Wilks' theorem.
With known variance , the z-test uses the pivot
and rejects when .
With unknown variance we replace with the sample standard deviation and obtain
That follows a -distribution rests on a structural fact about the Normal: and are independent by Cochran's theorem (the orthogonal decomposition of the sample space into the mean direction and its complement). The confidence interval is the inversion of this test and illustrates the confidence-test duality.
For jointly, the confidence ellipse is centred at and shaped by the observed Fisher information. In the standard Euclidean metric on -space this ellipse treats a step of size in the same whether or .
The Fisher metric of the Normal family differs. It is
which says that -space is curved. At small (where is large) a step of fixed coordinate size traverses far more statistical distance than the same step at large . Correct confidence regions account for this curvature.
The confidence ellipses above are computed using the Euclidean distance in space. The Fisher information matrix shows this space is curved. In the Euclidean metric a step of size in is the same whether or . In the Fisher metric a step at small (where is large) covers far more statistical distance.
The toggle above shows the difference: Fisher-metric confidence regions are larger at large , where a coordinate step covers little Fisher distance, and smaller at small . Its curvature is everywhere (Section 4.5). Only the scale of the metric varies with .
3.6. Computational verification
The coverage probability of a confidence interval is a frequentist guarantee: over repeated samples the interval should contain the true parameter at least of the time. Simulation checks it directly.
import numpy as np
np.random.seed(42)
p_true, n, alpha = 0.6, 30, 0.05
z = 1.96
n_sim = 10000
covers = 0
for _ in range(n_sim):
x = np.random.binomial(n, p_true)
p_hat = x / n
se = np.sqrt(p_hat * (1 - p_hat) / n)
lo, hi = p_hat - z * se, p_hat + z * se
if lo <= p_true <= hi:
covers += 1
print(f"Wald CI coverage: {covers/n_sim:.4f} (nominal: {1-alpha:.2f})")
print(f"(n={n}, p={p_true}, {n_sim} simulations)")Monte Carlo verification of Wald interval coverage
The Wald interval undercovers for moderate and extreme . This motivates alternatives such as the Wilson interval.
The power curve of a test visualises how detection probability varies with the true parameter value. At the power equals the size of the test (at most , and below it for a discrete statistic); away from it should increase towards 1.
import numpy as np
from math import comb
n = 20
alpha = 0.05
# Two-sided exact binomial test of H0: p = 0.5
def pmf(k, p):
return comb(n, k) * p**k * (1 - p)**(n - k)
def pvalue(k):
lower = sum(pmf(j, 0.5) for j in range(k + 1))
upper = sum(pmf(j, 0.5) for j in range(k, n + 1))
return min(1.0, 2 * min(lower, upper))
reject = [k for k in range(n + 1) if pvalue(k) <= alpha]
p_values = np.linspace(0.02, 0.98, 49)
# Power = P(reject H0 | true p)
power = np.array([sum(pmf(k, p) for k in reject) for p in p_values])
print("p power")
for p, pw in zip(p_values[::5], power[::5]):
print(f"{p:.2f} {pw:.4f}")
print(f"\nPower at p=0.5: {power[24]:.4f} (size of the test, at most alpha={alpha})")
print(f"Power at p=0.8: {power[39]:.4f}")Power curve for the Bernoulli two-sided test
The power curve is symmetric about (since the test is two-sided) and increases monotonically as moves away from the null. The rate of increase depends on : larger samples yield steeper power curves and concentrate the "uncertain" region into a narrower band around .
4. Geometric synthesis
The preceding sections spoke informally of the "curvature" of the log-likelihood, the "geometry" of the parameter space and the "shape" of confidence regions. This section makes these precise. The parameter space of a statistical model has a natural Riemannian structure, the Fisher metric, which encodes the intrinsic geometry of statistical inference.
4.1. Manifold structure
A smooth manifold of dimension is a second-countable Hausdorff topological space together with a collection of pairs (called charts), where each is an open subset of and is a homeomorphism onto an open subset of , such that:
- The charts cover : .
- For any two charts with , the transition map is smooth ().
The parameter space of a regular statistical model is a smooth manifold: openness in provides a single global chart, and the geometric viewpoint also allows parameter spaces with nontrivial topology (circles, spheres) and curved parameter sets such as the cone of positive-definite matrices.
A tangent vector at is an equivalence class of smooth curves with , where two curves are equivalent if
in any chart containing . The tangent space is the set of all tangent vectors at ; it is a real vector space of dimension . In local coordinates , every tangent vector can be written
where the Einstein summation convention is in force: repeated upper and lower indices are summed from to .
4.2. Riemannian structure
A manifold on its own has no notion of distance or angle. That structure is imposed by a metric.
A Riemannian metric on is a smooth assignment of a positive-definite inner product on each tangent space . In local coordinates,
where are smooth functions and the matrix is symmetric and positive-definite at every . The pair is called a Riemannian manifold.
In statistics such a metric is the Fisher information.
Let be a regular statistical model. The Fisher information matrix defines a Riemannian metric on the statistical manifold by
This is a -tensor field on , called the Fisher-Rao metric or the Fisher metric. The arc length element is
It remains to verify that this definition yields a Riemannian metric. The property to check is positive-definiteness.
On a regular statistical model, is positive-definite at every .
Proof
For any tangent vector , we compute
This is the expectation of a squared random variable, so . Equality holds if and only if almost surely under . For a regular model the score functions are linearly independent as random variables (this is part of the regularity conditions guaranteeing that the Fisher matrix is nonsingular). Therefore implies .
The Fisher metric is geometric in that it does not depend on the coordinate system.
Under a smooth reparametrisation , the Fisher matrix transforms as
This is the transformation law of a -tensor.
Proof
By the chain rule applied to the score function,
Substituting into the definition of the Fisher metric in the new coordinates:
where the Jacobian factors exit the expectation because they are deterministic functions of the parameters.
4.3. Geometry of the Levi-Civita connection
A Riemannian metric determines a canonical way to differentiate vector fields along curves: the Levi-Civita connection. Its components in local coordinates are the Christoffel symbols.
The Christoffel symbols of the Levi-Civita connection are
where denotes the entry of the inverse matrix of .
A geodesic is a curve that parallel-transports its own tangent vector. In local coordinates, it satisfies the system of second-order ODEs
Geodesics are the curves of zero acceleration and generalise straight lines to curved spaces. The geodesic distance between two points is the infimum of the arc lengths of all smooth curves connecting them.
The Riemann curvature tensor is defined in local coordinates by
For a two-dimensional manifold, the Gaussian curvature is the single scalar
The sign of determines the local geometry: (spherical), (flat), (hyperbolic).
4.4. Fisher metric as infinitesimal KL divergence
The Fisher metric is the leading-order term in the Taylor expansion of the Kullback-Leibler divergence.
The Kullback-Leibler divergence (or relative entropy) between distributions and is
It is non-negative ( iff ) and asymmetric and violates the triangle inequality. Hence it is no metric in the topological sense.
For an infinitesimal displacement in parameter space,
The Fisher metric is the Hessian of the KL divergence at zero separation.
Proof
Taylor-expand the log-likelihood around :
Substituting into the definition of KL divergence:
The first term vanishes because (the score has mean zero). For the second term the Bartlett identity gives . Therefore

The KL divergence is asymmetric and so is no distance. Its symmetrised version has the same leading-order expansion . Every -divergence with smooth at 1 has local expansion . Hence the Fisher metric is canonical: all such notions of statistical distance agree with it infinitesimally up to the constant .
4.5. Worked examples on familiar models
Bernoulli manifold
The Bernoulli family has Fisher metric
We show that this one-dimensional Riemannian manifold is isometric to a semicircle of unit radius. Define the Fisher-Rao coordinate , so that . Then
Substituting:
In the coordinate the metric is flat: . As ranges over , ranges over . The total geodesic length is , and the geodesic distance between and is
Geometrically, parametrises a semicircular arc of radius by arc length: the embedding traces out a semicircle of unit radius in with total arc length .
The Bernoulli family equipped with the Fisher metric is isometric to the open semicircular arc of unit radius, via the map .
Proof
The computation in the example above shows that the map pulls back the flat metric to the Fisher metric . Since this map is a smooth bijection from to with smooth inverse , it is a diffeomorphism. A diffeomorphism that preserves the metric is an isometry. The interval with the flat metric is isometric to a semicircular arc of unit radius (parametrised by arc length).
Normal manifold
The Normal family is a two-dimensional statistical manifold with coordinates . Its Fisher metric is
This is (up to a constant rescaling) the Poincare half-plane metric, the standard model of two-dimensional hyperbolic geometry.
We compute the Christoffel symbols, geodesics and curvature of this surface.
Christoffel symbols. The inverse metric is , , . Applying the Christoffel formula with coordinates , all metric components depend only on and so . The nonzero derivatives are:
Computing each Christoffel symbol (all unlisted ones vanish):
from sympy import symbols, Matrix, simplify, Rational, diff
mu, sigma = symbols('mu sigma', positive=True)
# Fisher metric for N(mu, sigma): g = diag(1/sigma^2, 2/sigma^2)
g = Matrix([[1/sigma**2, 0], [0, 2/sigma**2]])
g_inv = g.inv()
coords = [mu, sigma]
print("Fisher metric g:")
print(g)
print("\nChristoffel symbols:")
for k in range(2):
for i in range(2):
for j in range(i, 2):
val = 0
for l in range(2):
val += Rational(1,2) * g_inv[k,l] * (
diff(g[j,l], coords[i]) +
diff(g[i,l], coords[j]) -
diff(g[i,j], coords[l])
)
val = simplify(val)
if val != 0:
print(f" Gamma^{k+1}_{{{i+1}{j+1}}} = {val}")Christoffel symbols for the Normal statistical manifold
Gaussian curvature. Using , the component has :
(All other -terms vanish because the relevant Christoffel symbols are zero.) Therefore:
Lowering the index: . The determinant of the metric is . The Gaussian curvature is:
The Normal statistical manifold has constant negative Gaussian curvature . This is hyperbolic geometry: triangles have angle deficit, parallel geodesics diverge and circles grow exponentially.
from sympy import symbols, Matrix, simplify, Rational, diff
mu, sigma = symbols('mu sigma', positive=True)
g = Matrix([[1/sigma**2, 0], [0, 2/sigma**2]])
g_inv = g.inv()
coords = [mu, sigma]
# Compute Gamma^k_ij
def christoffel(k, i, j):
val = 0
for l in range(2):
val += Rational(1,2) * g_inv[k,l] * (
diff(g[j,l], coords[i]) + diff(g[i,l], coords[j]) - diff(g[i,j], coords[l]))
return simplify(val)
# Riemann tensor R^1_212
R1_212 = (diff(christoffel(0,1,1), coords[0])
- diff(christoffel(0,0,1), coords[1]))
for m in range(2):
R1_212 += christoffel(0,0,m)*christoffel(m,1,1) - christoffel(0,1,m)*christoffel(m,0,1)
R1_212 = simplify(R1_212)
# R_1212 = g_1l R^l_212
R_1212 = simplify(g[0,0]*R1_212)
detg = simplify(g.det())
K = simplify(R_1212 / detg)
print(f"R^1_212 = {R1_212}")
print(f"R_1212 = {R_1212}")
print(f"det(g) = {detg}")
print(f"Gaussian curvature K = {K}")Gaussian curvature of the Normal statistical manifold
Geodesics. With and the geodesic equations for the Normal manifold are:
These are the geodesic equations of the Poincare half-plane (with the coordinate rescaling reducing to the standard form). The solutions are of two types: vertical lines (corresponding to pure changes in variance) and half-ellipses centred on the -axis ( boundary). These are semicircles in the coordinates . In statistical terms the geodesic between two Normal distributions curves upward through higher-variance regions, because at larger the distributions are more spread out and therefore closer together in Fisher distance.

4.6. Bayesian-frequentist synthesis
At large sample sizes Bayesian and frequentist inference agree, and the Fisher metric governs both.
Let for some interior point , and let be a prior that is continuous and positive at . Under standard regularity conditions the posterior distribution satisfies
Geometrically: the posterior concentrates in a Fisher-metric ball of radius around the MLE. The Fisher metric determines the shape of the ball and the prior does not.
At large Bayesian credible sets and frequentist confidence sets coincide and the Fisher geometry determines both.
The Jeffreys prior is the prior distribution
where is the Fisher metric. It is the Riemannian volume form of the statistical manifold, and it is invariant under reparametrisation: if , then .
Jeffreys prior defines "uniform" on a curved space. On a Riemannian manifold the natural volume element is . Integrating the constant function against it gives the Riemannian volume of a region. Jeffreys prior assigns equal probability to equal volumes of statistical distance. In coordinates where is constant this agrees with equal volumes of coordinate distance. In other coordinates it does not, and the factor corrects for the distortion of volume. The Bernoulli manifold is flat and still has a non-uniform Jeffreys prior in the coordinate .
Bernoulli. The Fisher metric is , so and
This is the density of the distribution, also known as the arcsine distribution. It concentrates mass near and . These are the regions where Fisher information is largest and a unit of covers the most Fisher length. In the coordinate the Jeffreys prior is uniform on .
Normal. The Fisher metric is , so and
It is improper (does not integrate to a finite value) and yields a proper posterior once the sample contains two distinct observations. Jeffreys himself preferred the independence prior for this model. That prior is also the Berger-Bernardo reference prior and gives the usual intervals.
The visualisation above shows the Bernstein-von Mises theorem. At small the prior matters and the Bayesian credible interval (gold) and frequentist confidence interval (blue) diverge. At large both converge to the same interval centred at the MLE with width determined by the Fisher information. Both frameworks respect the Fisher metric. The frequentist approach conditions on and averages over data and the Bayesian approach conditions on data and integrates over . The Fisher metric governs the precision of both.
4.7. Looking ahead
The Fisher metric begins information geometry. The full theory, developed by Amari and others, introduces -connections: a one-parameter family of connections on the statistical manifold. The Levi-Civita connection is the member. The and connections (the e-connection and m-connection) are dual with respect to the Fisher metric, and exponential families are dually flat with respect to this pair. This dual structure governs distance, divergence, projection and the geometry of model selection. The sequel develops it.
References
- Kolmogorov, A. N. (1933). Grundbegriffe der Wahrscheinlichkeitsrechnung. Springer. English translation: Foundations of the Theory of Probability, 2nd ed., Chelsea, 1956. DOI
- Fisher, R. A. (1922). On the Mathematical Foundations of Theoretical Statistics. Philosophical Transactions of the Royal Society A, 222(594-604), 309-368. DOI
- Fisher, R. A. (1925). Theory of Statistical Estimation. Mathematical Proceedings of the Cambridge Philosophical Society, 22(5), 700-725. DOI
- Neyman, J. and Pearson, E. S. (1933). On the Problem of the Most Efficient Tests of Statistical Hypotheses. Philosophical Transactions of the Royal Society A, 231(694-706), 289-337. DOI
- Rao, C. R. (1945). Information and the Accuracy Attainable in the Estimation of Statistical Parameters. Bulletin of the Calcutta Mathematical Society, 37, 81-91. Link
- Cramer, H. (1946). Mathematical Methods of Statistics. Princeton University Press. Publisher
- Amari, S. and Nagaoka, H. (2000). Methods of Information Geometry. Translations of Mathematical Monographs, Vol. 191, AMS. DOI
- Amari, S. (2016). Information Geometry and Its Applications. Applied Mathematical Sciences, Vol. 194, Springer. DOI
- Lehmann, E. L. and Casella, G. (1998). Theory of Point Estimation, 2nd ed. Springer. DOI
- Lehmann, E. L. and Romano, J. P. (2005). Testing Statistical Hypotheses, 3rd ed. Springer. DOI
- van der Vaart, A. W. (1998). Asymptotic Statistics. Cambridge University Press. DOI
- Jeffreys, H. (1946). An Invariant Form for the Prior Probability in Estimation Problems. Proceedings of the Royal Society A, 186(1007), 453-461. DOI
- Le Cam, L. (1986). Asymptotic Methods in Statistical Decision Theory. Springer. DOI
- do Carmo, M. P. (1992). Riemannian Geometry. Birkhauser. DOI
- Efron, B. (1975). Defining the Curvature of a Statistical Problem (with Applications to Second Order Efficiency). Annals of Statistics, 3(6), 1189-1242. DOI
- Kullback, S. and Leibler, R. A. (1951). On Information and Sufficiency. Annals of Mathematical Statistics, 22(1), 79-86. DOI