Can a neural embedding space preserve temporal structure while remaining effective for both date estimation and object retrieval — simultaneously?
While time evolves linearly, neural embeddings are chaotic and multi-dimensional. TDIR asks: can we decompose a single embedding into independent temporal and categorical subspaces — so the year of one image can be surgically transplanted onto another, label-free?
Disentangle the space — drag the slider or press play
We propose Temporally Decomposable Image Representations (TDIR) — the first formal treatment of temporal decomposability in visual embeddings. We prove conditions under which decomposition holds, that orthogonality between subspaces emerges from joint optimisation without being explicitly imposed, and characterise the error when conditions are only partially met.
Beyond geometry, TDIR enables transitive operations: the year of one image can be extracted and injected into another with no label supervision — enabling the first zero-label composed historical image retrieval.
Around 5–10% of archival material is photographic. Unlike text, photographs encode temporal information implicitly — in the design of a car, the cut of a jacket, the typography of a poster. Archivists infer dates from objects; TDIR makes this machine-readable.
In archival science, the probative value of an image is its capacity to uniquely illustrate a phenomenon. Objects evolve at different temporal rhythms. A tire or a pair of shorts are strong temporal signals; a building or a wheel are ambiguous. TDIR learns which objects are reliable "time capsules."
TDIR supports three retrieval modes of increasing expressiveness. Select each to see the geometric operation on the embedding space.
TDIR rests on two definitions, two propositions, a lemma, a theorem, and its corollary — each with a geometric illustration inline. No appendix needed.
A representation is temporally independent from object classes when:
Category centroids carry no date information; date centroids carry no category information.
The year subspace is temporally transitive if for any triple (A, B, C):
Necessary for zero-label inference: a year residual from any image acts as a consistent temporal operator when transplanted.
The ideal decomposition v = f(x) = µc + µy requires three proxy sets, each with a distinct role at training and inference time.
Under Definition 1:
The total information in the joint embedding decomposes into two independent sums — no cross-information between categorical and temporal components.
The proof is a two-line cancellation. Step through it — read the diagram as an information map: the total area is everything the proxies tell us about (category, year). Watch the cross-terms vanish, leaving two disjoint pools.
So v = f(x) is temporally decomposable: knowing both proxies captures everything v encodes about (c, y), with no shared (overlapping) information left between the temporal and categorical parts.
Under Definition 1, the category and temporal centroids are orthogonal:
Independence → zero covariance → Pythagorean variance decomposition.
As I(ε;c,y) grows, µc picks up a projection onto µy — but Theorem 1 shows joint optimisation drives this projection back to zero.
In real data, residual noise ε entangles year and category. But:
δ is a data-dependent constant — it does not depend on the proxy vectors µc, µy. The orthogonality constraint ⟨µc, µy⟩ = 0 can be satisfied without affecting this bound.
An embedding splits as v = µc + µy + ε. You are free to rotate the orthogonal (µc, µy) frame to satisfy ⟨µc,µy⟩ = 0 — but that only re-shuffles information between the two proxies. The residual δ = I(ε; c,y) is fixed by the data alone, so the proxies can be learned freely without touching the bound.
Part I (exact). If I(ε;c,y) = 0:
Part II (asymptotic). If I(ε;c,y) ≠ 0:
Orthogonality is an emergent property of the joint optimisation — it does not need to be imposed explicitly.
Decompose µc = µ̃c + αµyA + βµyB, where (α,β) is its projection onto the year subspace. Drag µc: watch the two year residuals rA, rB drift away from the year proxies they are supposed to match, and the year loss climb. Only at α=β=0 do the residuals land exactly on µyA, µyB — that is the unique minimum.
The right panel is the actual loss surface Q(α,β) = 2α²‖µyA‖² + 2β²‖µyB‖² + 4αβ⟨µyA,µyB⟩ = 2‖α µyA + β µyB‖². It is a positive-definite bowl with a single minimum at α=β=0 — but only while the year proxies are linearly independent (the Lemma). Force them parallel and the bowl flattens into a valley: the minimum is no longer unique. Double-click the left panel to reset.
The shared displacement vectors ky cannot simultaneously compensate projections α ≠ α' across two categories — since ε is not a control variable, the unique globally consistent solution is α = α' = 0 for all c, c'.
The swapping trick forces residuals from different categories to be displaced by the same ky. This makes the optimizer implicitly require all category proxies to be orthogonal to all year proxies — not just category by category. It's the key to scalability.
Two categories (blue, red) must both have their year-A residual mapped onto the same target µyB by the same displacement kyB. If their projections α, α' differ, a single kyB cannot hit the target for both. The only globally consistent solution is α = α' = 0.
Set both sliders to 0 to see both residuals land exactly on µyB with the shared displacement — the consistent optimum.
Click each step — or press Play — to step through Algorithm 1. The canvas shows the geometric operation on the embedding space at each point. Color-coded to match the pseudocode.
At inference, 𝒦y is discarded. The three retrieval modes follow from f(x) ≈ µc + µy. Each is a different geometric operation on the learned embedding space.
Starting from DEW (1M natural-scene photographs, 1930–1999, one year per image), we apply OWLv2 with a 30% threshold — yielding 7.2M detections across 586 classes with year-level annotation. Top-50 categories used for evaluation (4.4M detections).
For each inference mode, retrieve top-K=10 images for every query. A date retrieval is correct if within the same lustrum (5-year window). Random baseline for date: 7.14% (70yr / 14 bins).
| Backbone | Label+Label | Image+Label | Image+Image | |||
|---|---|---|---|---|---|---|
| Obj P@K | Date P@K | Obj P@K | Date P@K | Obj P@K | Date P@K | |
| CLIP (arithmetic) | .152 | .115 | .387 | .076 | .389 | .102 |
| CLIP (prompting) | .415 | .105 | .372 | .079 | .520 | .081 |
| VGG + TDIR | .397 | .712 | .388 | .579 | .669 | .234 |
| ResNet + TDIR | .489 | .803 | .444 | .665 | .721 | .306 |
| ConvNeXt + TDIR | .443 | .824 | .416 | .664 | .725 | .393 |
| ViT + TDIR | .413 | .863 | .355 | .708 | .646 | .429 |
The paper ablates exactly one component at a time, so this is a single choice: pick what to remove and the bars compare it against the full model (faded bars). Notably, removing the swapping trick hurts Label+Label too — confirming Corollary 1 (implicit orthogonality regularisation).
Object precision remains stable across source years, target years, and gaps — the category subspace is insulated from temporal injection. Date precision follows a U-shaped curve over target year and degrades monotonically with gap.
Propositions 1 & 2 bound the decomposition error. Theorem 1 proves orthogonality emerges from joint optimisation. Corollary 1 shows the swapping trick enforces this cross-category.
DEW extended with 7.2M detections across 50 categories. Image+Image mode transplants the year of one image onto the content of another — no labels required.
A single temporal displacement across all categories may not reflect uneven historical evolution. Additive transplantation loses fidelity at large temporal distances.
Replace additive operator with a non-linear mechanism. Explore category-specific temporal subspaces. Extend to zero-shot evaluation over unseen categories.
The geometry of a well-structured embedding is not merely a tool for retrieval — it is a formal language for historical reasoning. Every µc + µy is a sentence: "Show me this object from this era."