A large-scale, OCR-free benchmark for historical document retrieval — 400K fragments across three centuries of Spain's official gazette, where the text is too damaged to trust.
Historical archives hold millions of pages whose OCR is noisy, partial, or simply wrong. Retrieval that leans only on transcribed text breaks down exactly where heritage collections are richest.
Most historical-document benchmarks chase word spotting — finding exact matches of query words inside a page. FAS aims higher, at topic spotting: fetching the fragment that means what a natural-language query says, even when the characters underneath are degraded. That demands visual insight, not just transcription.
The damaged condition of the documents, the absence of reliable ground truth, and a tangle of languages and scripts make text-only retrieval unfeasible. FAS reframes the task as multi-modal matching between a human-written query and the pixels of a document fragment.
FAS draws on the BOE, Spain's Official State Gazette — roughly a million annotated summary–document pairs, distilled into a curated 400K-fragment benchmark with train, test, and a fixed distractor anchor.
The temporal breadth is the point. Where most retrieval datasets cluster in a narrow window, FAS stretches from the XVII century to the mid-twentieth, capturing enormous variation in layout, script, and condition — and a real correlation: legibility trends upward with modernity, though plenty of modern pages remain poorly legible. That captured variance is what makes a retrieval system robust.
Each bar is a historical-document retrieval dataset, drawn across the years it covers. Most cluster in a narrow window; FAS (gold) stretches across roughly three centuries while remaining a topic/retrieval corpus — not just dates or writers.
Datasets built for word spotting tend to sit on a single decade or century; those probing writer or date can span a millennium but aren't about content retrieval. FAS occupies the rare middle ground the paper argues for: wide temporal variance and a topic-retrieval task — the variance that makes deployed systems robust to drifting layouts, scripts, and legibility.
To verify that a query truly belongs to a fragment, FAS aligns their named entities — and the OCR ones are garbled (E s p a ñ a, Int.ervirelonai). A Hungarian assignment over Levenshtein distance pairs each clean entity to its noisy counterpart. Add your own words to either side and watch the alignment.
E s p a ñ a → espana matches España at distance 0. The thick green edge is the minimum-total-distance assignment; leftover OCR entities stay unclaimed, flagging invalid fragments.This is the noise filter behind FAS's labels. Because fragment presence is human-verified and invalid regions per page are few, aligning named entities by edit distance — rather than trusting raw OCR strings — keeps spurious query–document pairs out of the ground truth. Try adding a badly mangled entity on the right: if nothing clean is close enough, it simply goes unmatched.
With ground truth in hand, the actual benchmark is text-to-image retrieval: encode a query and rank document fragments by semantic similarity. Add your own queries and see which fragment each one is matched to under a one-to-one assignment.
This mirrors the evaluation in the paper: the correct query should be the nearest neighbour of its fragment among a set of distractors. A retrieval is a hit when the true pair shows the minimum distance.
The same matched pairs power two evaluations. Flip between them.
Both reduce to the same objective: encode query and fragment so the true pair is the nearest neighbour among a distractor set. Minimise the distance to the correct match while pushing every distractor away.
A CLIP-pretrained ViT (fine-tuned with metric/contrastive losses) versus an OCR + Sentence-BERT pipeline. The headline numbers favour OCR — until you look only at the least legible quarter of documents.
| Approach | Loss | Acc@1 | Acc@5 | Acc@10 | AvgRank ↓ | Acc@1low | Acc@5low |
|---|---|---|---|---|---|---|---|
| Topic Spotting · text → image | |||||||
| ViT-B/32 | Triplet | .473 | .639 | .695 | 52.74 | .289 | .419 |
| ViT-B/32 | SimCLR | .509 | .672 | .723 | 44.12 | .296 | .443 |
| ViT-B/32 | CLIP | .460 | .657 | .719 | 32.33 | .264 | .447 |
| OCR + sBERT | — | .657 | .785 | .837 | 16.46 | .124 | .304 |
| Information Extraction · image → text | |||||||
| ViT-B/32 | Triplet | .523 | .640 | .692 | 64.00 | .326 | .421 |
| ViT-B/32 | SimCLR | .522 | .675 | .726 | 43.05 | .341 | .458 |
| ViT-B/32 | CLIP | .482 | .662 | .725 | 32.99 | .326 | .464 |
| OCR + sBERT | — | .623 | .723 | .760 | 44.19 | .073 | .149 |
Read the last two columns. On the worst-legibility quartile, the picture flips: OCR + Sentence-BERT collapses (Acc@1low as low as .073) while the vision baselines hold (.26–.34). When characters survive, transcription wins easily; when they don't, pixels carry the meaning. The implication is a hybrid — visual when text is unavailable, textual otherwise.
GradCam on the ViT's visual tokenizer shows it does attend to words, and even tokenizes them toward their stems (Infantería → <infanter>, Cuerpo → <cuerp>) — a sign of robustness to noise. Common keywords are a necessary, not sufficient, cue.
But the more revealing finding is temporal. Probing the ViT's internal representations for date estimation, the deeper the layer, the lower the error — down to an 11–15 year mean absolute error — and it stays low even as the fragment is heavily blurred. The model isn't reading dates off the text; it's reading them off the layout. That temporal bias, learned from era-specific page design, actively helps retrieval in low-legibility regimes.