Semantic Segmentation: Resolving a Domain and Typing its Pieces

Point a camera at a street and ask a computer which parts of the picture are road. The answer wanted is a label on every pixel, road or otherwise, so that the boundary between road and pavement falls exactly where it falls in the world. That task is semantic segmentation: given a scene, mark out its parts and say what each part is. It is what a surgical planner does to a scan when it outlines a tumour, what a survey does to a lidar cloud when it separates a wall from the vegetation growing over it, and what a car does to a camera frame many times a second.

The word semantic carries the second half of the task. A method may find every boundary in a scene correctly and still answer nothing, since knowing where one region stops differs from knowing that the region is a road. This article separates the two questions and argues that almost everything else follows from how a method separates them.

Section 1 assembles the mathematics the argument uses, from partitions and group actions through the graph Laplacian, differential forms and Riemannian metrics, assuming a first course in linear algebra and one in calculus. A reader holding that material already can start at section 2.

Semantic segmentation is usually presented as a catalogue of architectures. The catalogue grows every month, and its entries differ in ways incidental to what they compute. Underneath every one of them sit two operations, and the mathematics is in how each method performs them.

The first operation resolves a domain into pieces. The domain may be an image, a point cloud, a triangulated surface, a volume, or a sequence of any of these; the pieces may be pixels, primitives, faces, or regions. This is a question about representation: what the domain is made of, and what structure, if any, relates one element to the next.

The second operation types each piece, assigning it a label drawn from a semantic vocabulary. This is a question about inference: a classifier, an energy minimisation, a prompted foundation model, or a learned metric.

Reading the field through those two operations puts methods that share no code and no vocabulary on a single axis. It also produces one observation that neither side of the literature states, given in section 2.5: the primal-dual mesh pair on which discrete exterior calculus is built is the same combinatorial object a modern mesh-segmentation network consumes as its input graph, and the two disciplines diverge on where the dual vertices are placed, for reasons that follow from what each of them does with the pair.

Both operations, and the problem of holding them steady over an ordered index, are visible in one object. The scene below is a single scan position of a terrestrial lidar survey of the Great Mosque of Kilwa Kisiwani. It carries no connectivity, so it is resolved into pieces by a graph built over the returns, typed from the geometry of each point's neighbourhood, and tracked across the scanner's own sweep. Every later section of this article is a question about one of those three steps.

1. Preliminaries

The mathematics used in this article is assembled here, starting from vectors and matrices, eigenvalues and eigenvectors, partial derivatives and multiple integrals. Everything past that point is built below, each construction followed by a worked instance of it. Fuller treatments of the geometric material are in Manifolds and in the Analysis on Manifolds series; the discrete operators of section 2.5 are developed at length in the numerical analysis article.

1.1. Partitions, equivalence and quotients

Segmentation produces a partition, and every representation in section 2 differs only in which partitions it admits.

A function on passes to the quotient exactly when it is constant on classes, in which case it factors as for a unique on . A label assigned to pieces rather than to points is a function on the quotient, which is the shape of the typing map in section 1.9.

1.2. Group actions and invariance

Declaring two objects to have the same type usually means declaring some family of transformations irrelevant. Groups record which.

Three groups recur below. The special Euclidean group consists of rotations composed with translations of ; the scalings act by ; and a reparameterisation group acts on the domain of a parameterised object, by permutations for a labelled point set and by diffeomorphisms for a curve or a surface. The symbol in section 2.7 marks a semidirect product, the group built on the product set when acts on by automorphisms, with the twisted law

1.3. Graphs, the Laplacian and the Rayleigh quotient

A weighted graph on vertices is recorded by a symmetric matrix with the weight of the edge between and , and zero diagonal. The degree of vertex is , collected in .

The identity is the whole content of the matrix: the quadratic form is a weighted sum of squared differences across edges, so is positive semidefinite, and since the differences across edges vanish on a constant vector. Minimising under a normalisation is minimising a Rayleigh quotient , whose stationary points are the eigenvectors of and whose stationary values are the eigenvalues. Restricting to vectors orthogonal to discards the trivial minimiser and leaves the eigenvector of the second-smallest eigenvalue. Replacing the normalisation by replaces the eigenproblem by the generalised one , which is the same statement taken in the inner product weighted by degree.

1.4. Functionals, and what a boundary costs

A functional assigns a number to a function, and a variational method chooses the function minimising it. The functionals below have three kinds of term: a fidelity term measuring departure from the observation, a smoothness term measuring how fast the approximation varies, and a penalty on the set where the approximation is allowed to jump. In the plane that set is a family of arcs and its size is total arc length; on a graph it is a set of edges and its size is the total weight severed, which is the of section 2.2. Both measure a one-dimensional set sitting inside a larger domain, which is what puts the continuum penalty and the graph cut on one footing.

1.5. Simplices, chains and cochains

Fixing an ordering of the -simplices of makes the chain space the vector space with those simplices as its basis; a -chain is a formal sum of oriented -simplices with integer or real coefficients. The cochain space is its dual, a -cochain being a linear functional on chains, and the pairing is written . The boundary operator sends a simplex to the alternating sum of the simplices obtained by dropping one vertex, stated in general in section 2.5. In the chosen ordering it is a matrix with entries in , and the discrete theory is linear algebra over these bases.

1.6. Differential forms, Stokes and the Hodge star

A differential -form is what a -dimensional integral integrates. A 0-form is a function, evaluated at points; a 1-form is integrated along curves; a 2-form is integrated over surfaces. The wedge is antisymmetric, , which is how a form records orientation. The exterior derivative raises degree by one and specialises to gradient, curl and divergence in , and satisfies .

The fundamental theorem of calculus is the case , Green's theorem and the classical Stokes theorem the case in the plane and on a surface in space, and the divergence theorem the case . Section 2.5 takes this identity as the definition of the discrete exterior derivative, which is why the theorem holds there by construction.

The Hodge star needs an inner product and an orientation, and matches each -form with the -form spanning the complementary directions. In with the standard metric,

so a 1-form and a 2-form carry the same three numbers, and the curl of a vector field can be read as either. The discrete star of section 2.5 is the same correspondence, with the metric entering through the measured volumes of primal and dual cells. Exterior algebra, forms, integration and Stokes are developed in Analysis on Manifolds II through V.

1.7. Manifolds, metrics and square-integrable maps

A diffeomorphism is a smooth bijection with smooth inverse, and an immersion is a smooth map whose derivative is injective at every point, which is the condition making a parameterised curve or surface a copy of its domain inside the ambient space. When a group acts on a manifold freely and properly, the orbit space of section 1.2 is again a manifold, which is what makes the shape space of section 2.7 a manifold away from its singular orbits. Charts, tangent spaces, metrics and geodesics are treated in Manifolds.

Function spaces enter as the infinite-dimensional case. The space consists of maps with , carrying the inner product and norm

The distance is measured along straight segments, since the space is linear, and an isometry is a map preserving distances. Transporting a problem along an isometry leaves every distance, geodesic and average unchanged, which is the force of the transform in section 2.7.

1.8. Gaussian noise and the size of a maximum

A random vector in has mean and covariance . Isotropic noise has covariance , the same variance in every direction and no correlation between coordinates. Averaging independent copies gives covariance , so the noise scale falls as while a signal repeated across the copies survives.

The threshold argument of section 4.1 needs one further fact. For independent standard Gaussians, a union bound over the coordinates gives

which is small once passes . The largest of many independent Gaussians therefore sits near , growing slowly in the number of coordinates.

1.9. The two operations

Fix a domain and a finite label set . A segmentation of is a pair: a partition

into pieces, and a typing map . Semantic segmentation asks for both. Instance segmentation asks for the partition and for identity in place of a class, so that two pieces receive the same type exactly when they belong to the same object. Panoptic segmentation asks for both notions at once.

Stated that way, the definition carries content only through two choices. The first is which partitions are admissible, which is fixed by the representation: a pixel grid admits any subset of pixels, a triangulated surface admits any subset of faces, and a Gaussian point cloud admits any subset of primitives. The second is what makes a typing map good, which is fixed by the inference procedure. The rest of this article works through the spectrum of answers to each, in increasing structure.

2. Resolving the domain

2.1. Pixels, and the variational ancestor

The pixel grid is where the subject starts, and it carries the least structure of anything here: a rectangular lattice with four- or eight-neighbour adjacency. Every partition of the lattice is admissible, so the representation constrains nothing and the whole burden falls on the criterion.

The classical criterion is variational. Mumford and Shah^[1] approximate an observed image on a region by a function that is differentiable away from a set of arcs , penalising three quantities at once (p. 579):

where is "the total length of the arcs making up" the discontinuity set, and the smaller is, the better the pair segments . The segmentation is the minimising , and that fixes the shape of nearly everything downstream: the boundary is the object being solved for, found by trading its own length against the error it saves. Every later method that regularises a label field with a smoothness term is paying a discretised version of the price.

Restricting to a constant on each connected component of collapses the functional to

(p. 580), which is the cartoon limit: fit a constant per region, pay for the total boundary length. Every superpixel method and every region-merging heuristic is an approximate minimiser of this.

2.2. Similarity graphs and the spectral relaxation

The discrete counterpart replaces the domain by a weighted graph , one vertex per pixel, with measuring the similarity of pixels and . A partition into and costs the total weight it severs, . Minimising that quantity directly favours slicing off single vertices, so Shi and Malik^[2] normalise it by how strongly each side attaches to the whole graph, giving what they call a disassociation measure,

Exact minimisation is NP-complete, and remains so on grid graphs, a result they attribute to Papadimitriou (p. 890). Their contribution is what to do about it, and the derivation runs as follows.

Write for the affinity matrix, for the degree of vertex , and . Encode the partition in an indicator , with when (p. 890). Since and are the indicators of the two sides, the cut and the two association terms are all quadratic forms in , and substituting

turns the criterion into a ratio of quadratic forms (p. 890),

The numerator is the quadratic form of the graph Laplacian , since , which is the total weight the partition severs, weighted by how far apart it drives the two sides. The constraint removes the constant vector, which is the trivial minimiser.

What remains is discrete only through taking two values. Dropping that, and allowing , leaves a Rayleigh quotient, and its stationary points solve the generalised eigenvalue system

The constant vector is the eigenvector for , so the constraint is met by every other eigenvector, and the minimiser subject to it is the eigenvector of the second-smallest eigenvalue. That vector is thresholded to produce the partition, and the procedure recurses on each side.

Relaxation is where the guarantee goes. The real-valued minimiser bounds the combinatorial one from below and need not be near it, so the eigenvector carries no approximation guarantee for the combinatorial problem. What the method achieves and how it fails both follow from that one step.

Two features of this are permanent. The matrix is the graph Laplacian, so image segmentation was placed inside spectral graph theory in 2000 and has stayed there; and the whole construction depends on the similarity graph rather than on the grid, so nothing in it cares that the vertices were pixels. Every later graph-based method enters through that generality. Graph cuts in the sense of Boykov and Jolly^[3] take the other branch at the same fork, keeping the combinatorial problem exact and solving a maximum-flow problem on a graph augmented with terminal vertices, at the price of a criterion that must be expressible as a flow. What that buys is stated sharply in their own terms: the method "provides a globally optimal solution for an N-dimensional segmentation when the cost function is clearly defined", so that "imperfections of a globally optimal solution are directly related to the definition of the cost function" (p. 105). Separating a wrong answer caused by the criterion from a wrong answer caused by the search is what the spectral relaxation surrenders.

2.3. Points with no adjacency

At the opposite extreme from a mesh sits a representation with no connectivity at all. In 3D Gaussian splatting^[4] a scene is a finite multiset of anisotropic primitives, each carrying a mean, a covariance, an opacity and a view-dependent colour, optimised so that the rasterised image matches a set of photographs. There is no adjacency structure whatsoever: the scene is a bag of primitives, and any subset of it is a candidate piece.

Segmenting such a scene means producing an assignment

from primitive indices to instance indices. Two families of answer exist. The first projects a class-agnostic 2D segmenter over many rendered views and merges the per-view masks into 3D by consensus and spatial proximity. The second, Gaussian Grouping^[5], attaches an identity feature to every primitive and learns it jointly with the geometry, so that falls out of a clustering in feature space at the end of training rather than out of a merge heuristic.

The absence of connectivity is a design choice, and its payoff is topological. A representation that carries adjacency has to maintain it, and events that change the topology of the scene, an object splitting or a structure collapsing, invalidate the adjacency wholesale. A bag of primitives has nothing to invalidate. The cost is paid on the other side: with no adjacency there is no cheap smoothness prior, so every regularity the partition has must be recovered from geometry.

2.4. Meshes as two-dimensional simplicial complexes

A triangulated surface is a simplicial complex, and everything in the next section follows from that.

Written that way it is a two-dimensional simplicial complex, with , and its 0-, 1- and 2-simplices. Segmentation on a mesh usually types faces rather than vertices, since a face carries a normal, an area, and a patch of texture, and a labelled face set is what a downstream geometric pipeline wants.

Typing faces calls for a notion of which faces are near which. The construction that supplies it is the dual graph: one node per face, one edge for each pair of faces sharing an edge of the mesh. LMSeg^[6] places the dual node at the barycentre of its face, calls the result the barycentric dual graph, and runs a message-passing network over it, with each node carrying the face normal, its position, and its colour.

The message-passing scheme itself contains one construction that stands on its own. Aggregating a neighbourhood of features requires a choice among maximum, mean and sum, and the choice is usually made by hand and fixed. LMSeg replaces it by a temperature-scaled softmax over a scalar score attached to each neighbour, giving a one-parameter family

whose weights tend to as , recovering mean aggregation, and concentrate on the highest-scoring neighbour as and on the lowest as , recovering maximum and minimum. Learning by backpropagation turns a discrete architectural decision into a continuous parameter, and interpolates between the aggregators previously chosen by hand.

2.5. The dual graph, and where discrete exterior calculus meets it

The barycentric dual graph of the previous section is the object on which discrete exterior calculus is built. The correspondence between the two uses is exact up to one choice, and that choice decides which operations the pair supports.

Discrete exterior calculus assigns discrete -forms to -simplices, so that a discrete form is a cochain, a linear functional on chains, with read as the integral of over . Everything else follows from duality with the boundary operator.

Two consequences arrive immediately, and Desbrun, Hirani, Leok and Marsden^[7] note both. Since , the dual statement holds. And the defining identity, read from right to left, is the discrete generalised Stokes theorem,

which is true by definition in this framework. The theorem that costs a chapter in the smooth theory is the definition of the derivative in the discrete one.

The dual complex enters with the Hodge star. In the smooth setting the star maps -forms to -forms; in the discrete setting the dual of a -simplex is an -cell, which is the hint that primal and dual meshes are both needed.

Fixing an ordering of the simplices makes all of this matrix algebra. Let be the incidence matrix of -simplices against -simplices, carrying the signs of the alternating sum. The defining identity says that is its transpose,

so the exterior derivative costs a transpose and a sparse multiply, and is inherited from rather than proved again. The Hodge star is diagonal in the same ordering,

which is the equal-averages definition read off entry by entry, and the Hodge Laplacian assembles from the two as .

That diagonal is where the comparison bites. The volumes and appear in the definition, so the Hodge star depends on the geometry of the dual cells, and therefore on where the dual vertices are placed. Desbrun and coauthors place them at circumcentres, and rest their theory on that placement: earlier work used barycentric subdivision or barycentric duals, and they show that circumcentric duals admit a theory carrying both differential forms and vector fields. Delaunay triangulation of the primal vertices enters that setting for a reason internal to the theory, since it guarantees that the circumcentre of a simplex lies inside that simplex.

A message-passing network reads none of that. It consumes the dual graph as combinatorics, one node per face and one edge per shared edge, and the position attached to a node is an input feature like any other. Placement at the barycentre is then the obvious choice, being cheap, always interior to the triangle, and defined without regard to the shape of the triangle. The primal-dual pair is the same object on both sides of the comparison, and the two disciplines part company at the placement of the dual vertices: a discrete Hodge star measures dual cells and so cannot choose freely, whereas a learned aggregation only traverses them and so can.

The consequence runs in one direction. Anyone wishing to run discrete differential operators over a segmented mesh, to compute a discrete curvature or a Hodge decomposition on the labelled regions, inherits the DEC constraint and cannot reuse the barycentric dual graph on which the segmentation network was trained. The two representations look identical in a diagram and are not interchangeable in a computation.

2.6. Producing the mesh

A mesh has to come from somewhere, and the choice of meshing algorithm determines which partitions the segmentation can express, since a boundary that fails to follow an edge of the mesh is unrepresentable.

Delaunay triangulation is the workhorse where the input is already a set of surface points: LMSeg's landscape dataset is built by triangulating ground-classified lidar returns. It carries the property noted above, that circumcentres stay inside their simplices, which is why it recurs in the numerical literature for reasons unrelated to segmentation.

Where the input is a volumetric or implicit field rather than a point set, marching cubes^[8] extracts an isosurface by classifying the eight corners of each voxel against a threshold and emitting a triangulation from a lookup table indexed by that classification. Eight corners each inside or outside give ways a surface can cut the cube, and the authors reduce those to 14 patterns by complementary and rotational symmetry, recovering the full 256 by permutation (p. 165). The whole algorithm is that table plus linear interpolation along the cut edges, which is why it has outlived every architecture built on top of it. Where the input is an oriented point cloud, Poisson surface reconstruction^[9] recovers the surface as a level set of an indicator function whose gradient best matches the sample normals, which turns reconstruction into a Poisson equation.

Simplification is the reverse operation, and quadric error metrics^[10] are the standard tool: each vertex carries a quadratic form measuring squared distance to its incident planes, edge contractions are ordered by the error they induce, and the mesh is coarsened greedily. LMSeg considers this pooling strategy and declines it, on the ground that greedy sequential contraction suits a GPU poorly, using random sub-sampling under an edge-similarity criterion instead. The classical algorithm is serial, and the surrounding computation is parallel.

2.7. Shape space as a quotient manifold

Every representation so far is a discretisation of a domain. The last one is different in kind: it makes the space of shapes itself the object of study, and it gives typing a rigorous meaning that the learned methods approximate operationally.

The construction is due in its landmark form to Kendall^[11], who writes the shape space of labelled points in as the quotient by the similarity group (p. 83), and identifies the planar case outright: (p. 86). The shape space of points in the plane is complex projective space, so shape is a projective notion and carries no linear structure. The review by Choi and coauthors^[12] states the construction in the generality above. Taking with the permutations recovers Kendall's classical shape spaces; taking a space of immersions of a manifold with gives the space of unparameterised curves and surfaces on which elastic shape analysis is built.

The quotient says what a type is. Two objects have the same shape exactly when they lie in the same fibre of the quotient map, and the quotient records precisely which transformations were declared irrelevant. A typing procedure that is invariant to pose and scale is a procedure that factors through , and one that does not is answering a different question.

Quotients of this kind are nonlinear, so comparing shapes means choosing a metric. The elastic family measures a deformation by splitting it into components normal and tangential to the curve, weighting the two by parameters and , and taking the geodesic distance

over paths in the space of curves joining to . The choice , singles out the square-root velocity metric, and with it the result that makes the whole framework computable.

Geodesic distance in a curved, infinite-dimensional shape space becomes the ordinary distance after one pointwise transform, so geodesics become straight lines and averaging becomes averaging. The transform is due to Srivastava, Klassen, Joshi and Jermyn^[13], who define it on a curve as , with every coming from a curve unique up to translation, with the unit-length curves landing on a unit hypersphere in that space (p. 1416). It is what makes elastic shape analysis computable.

Two further metrics behave differently under the situation that troubles mesh representations. The continuous Procrustes distance compares surfaces by minimising over area-preserving diffeomorphisms and rigid motions jointly, which removes the need for landmarks. Varifold norms represent a shape as a measure on , position paired with tangent plane, and compare two such measures through a reproducing kernel inner product

Discarding orientation, which the theory of currents retains, buys robustness to change in topology, and the varifold is the one representation on this axis that survives an object splitting in two.

3. Typing the pieces

3.1. Superfacets, hand-crafted features, and a smoothing pass

The pre-deep-learning pipeline for mesh segmentation has three stages, and its shape survives in methods that have discarded every one of its components. Faces are grouped into superfacets by region growth under a geometric similarity criterion; each superfacet is summarised by hand-crafted geometric and photometric features; a random forest or decision tree assigns a class to each summary; and a Markov random field smooths the result over the adjacency structure.

The final stage is the one that connects back to section 2.1. Writing for the label of superfacet , the smoothing pass minimises

with the classifier's posterior, the adjacency of the superfacets, and one when the labels differ. Set beside the functional of section 2.1, the correspondence is term by term: the first sum is the fidelity , and the second is , since a label change across an edge is a piece of boundary and is the length it contributes. The smoothness term of the continuum problem is missing, because a piecewise-constant label field has no gradient to penalise off the boundary, which is exactly the cartoon limit of section 2.1. A Markov random field over superfacets is a discretised Mumford-Shah energy, and the variational criterion reappears as a post-processing step in a pipeline that has otherwise nothing to do with it.

3.2. Class-agnostic segmenters, and the factorisation they force

Prompted foundation segmenters such as SAM^[14] changed the structure of the problem rather than its accuracy alone. Such a model is class-agnostic: given a point, a box, or a coarse mask, it returns a boundary, and it never returns a label.

Typing therefore factors into two independent problems. Locating the boundary is delegated to a large pretrained model with no notion of the label vocabulary. Deciding what lies inside the boundary is then a much cheaper signal, and can be a text prompt, a class-activation map, a bounding box from a detector, or a projection of votes from other views. The factorisation is what makes the point-cloud methods of section 2.3 possible: a 3D scene has no pretrained segmenter, so the boundary problem is solved in 2D, view by view, and only the labels are lifted.

The practical consequence is that the two halves fail independently, and diagnosing a segmentation means asking which half went wrong. A correct boundary with a wrong class and a right class with a boundary in the wrong place are different faults with different remedies, and a single mean-intersection-over-union figure hides the difference.

3.3. Message passing over a relational structure

Learned typing on a structured domain is vertex classification by message passing, and the domain enters only through the relational structure the messages traverse. LMSeg types mesh faces by propagating over face adjacency in the dual graph; a hypergraph network types abstract instances by propagating over hyperedges that encode relations of higher arity than a pair. The operation is the same at both levels of the representation stack.

A normalised hypergraph Laplacian, and spectral clustering on it, predate the learned methods by two decades. A message-passing network parameterises the propagation and learns it from labels in place of taking eigenvectors of a fixed operator. The gain is that the propagation can depend on features. The constant across both is the relational structure, which the modeller chooses and which carries most of the domain knowledge.

3.4. Identity as a point in a metric space

Tracking demands a notion of type that a softmax over a fixed vocabulary cannot supply, since the question is whether two detections are the same object rather than which of the classes describes each. The answer is to embed and to learn the metric. IGGT4D^[15] attaches a feature to each pixel, forms per-instance prototypes as means over the instance mask, and trains with hinged pull and push terms

with , applied both within a view and across views, so that prototypes of the same object in different frames are pulled together and distinct prototypes are pushed apart. A type is then a point in embedding space, and two detections share a type when they fall within the pull margin of one prototype.

The vocabulary is unbounded under this definition, which is the property tracking needs, and the margins and replace the decision boundary a classifier would have learned.

4. Consistency across frames

Everything above resolves and types a single domain. Applying it independently to each frame of a sequence produces a segmentation that flickers, since nothing ties frame to frame . The mechanisms that supply the tie are more varied than the phrase spatiotemporal segmentation suggests, and four of them are distinct.

4.1. Any ordered index will do

The cheapest mechanism starts from a change of index. A 3D medical volume, sliced along one axis, is a sequence of images indexed by slice number, and every temporal-coherence technique applies verbatim with read as slice index. Stated generally: coherence arguments need an ordered index and are indifferent to whether that index counts time or counts space.

Gu and coauthors^[16] exploit this in weakly supervised volumetric segmentation. Model the class-activation map of slice as a latent signal in noise, with zero-mean and isotropic in . Averaging slices reduces the noise covariance to . The averaging requires no retraining and no additional data, since the frames were slices of one object throughout.

The threshold argument follows from a concentration bound. For iid standard Gaussian noise in dimensions the maximum concentrates,

a standard fact^[17]. A peak of strength is distinguishable from the noise maximum when , and after averaging slices the requirement relaxes to

The factor is what makes the raw per-slice threshold demanding on a large image, and the is what the ordering buys. The statistical argument then becomes a combinatorial rule: a frame is flagged unreliable when its own activation peak and the aggregated peak fail to contain each other,

and a flagged frame borrows its neighbour's mask.

4.2. Causal attention and a constant-time codebook

The second mechanism is architectural: it fixes what a method is permitted to see. Offline 4D reconstruction attends over a whole fixed set of views; replacing that with a causal mask, so that frame attends only to frames , is the geometric analogue of autoregressive language modelling, and it is what turns reconstruction into streaming.

Causality alone is insufficient, since clustering instance features across a sequence offline is quadratic in its length. IGGT4D maintains instead a codebook of instance centres and accumulated pixel counts, and fuses each new frame by area-weighted update

which costs per frame and is a running weighted mean, the same update as Welford's online average with a data-dependent weight.

Set that update against section 2.7. Both fit a representative without revisiting the past: one takes a running average in a flat feature space, the other fits a geodesic in a curved shape space. The streaming constraint is the same constraint in both settings, and the geometry of the space is what decides how much work meeting it takes.

4.3. Temporal resolution allocated by distance

The third mechanism spends resolution unevenly along the time axis. Recent frames are informative at fine spatial detail and distant frames are informative only in aggregate, so a fixed window over all reference frames wastes computation on the distant ones. MSTF-Net^[18] sets the window scale by temporal distance, reshaping a reference feature map at scale increasing in ,

trading spatial resolution for receptive field as the reference frame recedes, and fusing the resulting context tokens with the query from the current frame.

4.4. Dense correspondence and topology change

The fourth mechanism establishes correspondence directly, and it comes in two strengths that fail in different circumstances.

The weak form aligns two known states with a single rigid transform, estimating an element of by minimising a Chamfer distance and reading a per-instance displacement from the residual. It assumes nothing about what happened between the two states, which is what makes it usable when the states are a before and an after with no continuous path between them.

The strong form asks for pointwise correspondence across a whole time-ordered family, through diffeomorphic registration in the LDDMM tradition. It gives far more, a dense deformation field with an inverse, and it demands far more: the topology has to be preserved throughout, since a diffeomorphism cannot split a component in two. The review of shape-space methods flags high curvature, incomplete sampling and segmentation error as the situations where dense correspondence degrades, and change in topology is the limiting case of all three.

This is the general form of the argument that appears piecemeal across the representations above. Adjacency, dense correspondence and diffeomorphic linking all assume the topology holds still. Where it does not, the varifold representation of section 2.7 and the connectivity-free point representation of section 2.3 are the two branches that survive, from opposite ends of the structure axis.

4.5. Missing channels

One practical remark: consistency over time is usually demanded of a sensor rig that does not always deliver. Typing that depends on several modalities has to survive the loss of one, and the standard treatment freezes a pretrained encoder, trains a residual auxiliary encoder injected through zero-initialised convolutions, and drops each modality stochastically during training so that no branch becomes indispensable^[19]. There is little mathematics in it, and a segmentation that assumes depth is always present will fail in the field for reasons unrelated to any of the above.

5. Reading a method off the two spectra

Ordering the representations by how much structure they impose puts them on one axis: a bag of primitives with no adjacency, a pixel lattice, a similarity graph, a simplicial complex with its dual, and a quotient manifold of shapes. Structure buys locality, which is what makes smoothness priors and message passing possible, and it is paid for at events that invalidate the structure, of which change in topology is the sharpest.

Typing runs a parallel spectrum: energy minimisation over a fixed vocabulary, a classifier over hand-crafted summaries, a learned propagation over a relational structure, a prompted class-agnostic boundary with the label supplied separately, and a learned metric in which identity is proximity to a prototype. The last of these abandons the fixed vocabulary, which is what tracking requires.

The two spectra are close to independent, and most published methods are one choice from each list with an architecture wrapped around the pair. Those two choices usually predict where a method will fail.

The one place where the lists touch a classical formalism directly is section 2.5. The primal-dual mesh pair is common to discrete exterior calculus and to graph-based mesh segmentation, and the disciplines diverge on the placement of the dual vertices, because a discrete Hodge star measures the dual cells while a message-passing network merely traverses them. A fuller treatment of the DEC machinery, including the coboundary as a matrix transpose and the Hodge Laplacian, is in the numerical analysis article.

References

1. Mumford, D. and Shah, J. (1989). Optimal approximations by piecewise smooth functions and associated variational problems. Communications on Pure and Applied Mathematics, 42(5), 577–685. DOI
2. Shi, J. and Malik, J. (2000). Normalized cuts and image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(8), 888–905. DOI
3. Boykov, Y. Y. and Jolly, M.-P. (2001). Interactive graph cuts for optimal boundary and region segmentation of objects in N-D images. Proceedings Eighth IEEE International Conference on Computer Vision (ICCV 2001), 105–112. DOI
4. Kerbl, B., Kopanas, G., Leimkühler, T. and Drettakis, G. (2023). 3D Gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4). arXiv:2308.04079
5. Ye, M., Danelljan, M., Yu, F. and Ke, L. (2024). Gaussian Grouping: segment and edit anything in 3D scenes. ECCV 2024. arXiv:2312.00732
6. Huang, Z., Khoshelham, K., Gunditj Mirring Traditional Owners Corporation and Tomko, M. (2024). LMSeg: an end-to-end geometric message-passing network on barycentric dual graphs for large-scale landscape mesh segmentation. arXiv:2407.04326
7. Desbrun, M., Hirani, A. N., Leok, M. and Marsden, J. E. (2005). Discrete exterior calculus. arXiv:math/0508341
8. Lorensen, W. E. and Cline, H. E. (1987). Marching cubes: a high resolution 3D surface construction algorithm. ACM SIGGRAPH Computer Graphics, 21(4), 163–169. DOI
9. Kazhdan, M., Bolitho, M. and Hoppe, H. (2006). Poisson surface reconstruction. Proceedings of the Fourth Eurographics Symposium on Geometry Processing, 61–70. DOI
10. Garland, M. and Heckbert, P. S. (1997). Surface simplification using quadric error metrics. Proceedings of SIGGRAPH 97, 209–216. DOI
11. Kendall, D. G. (1984). Shape manifolds, Procrustean metrics, and complex projective spaces. Bulletin of the London Mathematical Society, 16(2), 81–121. DOI
12. Choi, G. P. T. et al. (2026). Learning the geometry of data: a mathematical review of shape space analysis. arXiv:2606.17022
13. Srivastava, A., Klassen, E., Joshi, S. H. and Jermyn, I. H. (2011). Shape analysis of elastic curves in Euclidean spaces. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(7), 1415–1428. DOI
14. Kirillov, A. et al. (2023). Segment Anything. arXiv:2304.02643
15. Zou, Z. et al. (2026). IGGT4D: streaming 4D instance-grounded geometry transformer. arXiv:2607.19228
16. Gu, R. et al. (2026). Robust activation map rectification for weakly supervised volumetric segmentation: temporal coherence as a free lunch. arXiv:2607.19877
17. Vershynin, R. (2018). High-Dimensional Probability: An Introduction with Applications in Data Science. Cambridge University Press.
18. Wang, C. et al. (2026). MSTF-Net: UAV-oriented multi-spectral video segmentation. arXiv:2607.15628
19. Zhu, X. et al. (2026). Toward reliable RGB-D semantic segmentation: handling missing modalities via condition dropout. arXiv:2607.20326