one schema crate, two analyzers: bilingual full-text search with embedded tantivy

you can see the word. it’s right there on the page — filósofos, third line of a plutarch translation i typed out by hand. you paste it into the search box, hit enter, and the archive tells you it has never heard of it. no error, no red box, no “did you mean.” a clean, confident, empty result set — byte-for-byte the same empty set you’d get for a word that genuinely isn’t in the corpus.

the index knew the word. i had watched it index the word. the reader just asked for it in a dialect the writer never spoke.

i keep a personal digital archive of classical texts — latin and greek originals with their public-domain spanish and english translations, built as a small gift to the commons rather than a business. rust, axum, postgres for the source of truth, and an embedded tantivy ↗ index for full-text search. the companion piece to this one, clean and verified are different claims , was about the postgres side: modelling may we publish this and how faithful is this text as two orthogonal columns. this is the search side i teased at the end of it — and the two things about tantivy that bit me hard enough to be worth writing down.

both bites grow from the same root: a search index is two programs that have to agree, and nothing forces them to. one program writes the index (my ingest cli). another reads it (the web server). if they disagree about a single detail — the name of a field, the exact analyzer behind that field — the disagreement doesn’t crash. it just quietly returns nothing.

one schema, two binaries

the first disagreement to kill is the easy one: field names. a tantivy schema is just an ordered list of fields, each with a type and some options, and both sides have to construct the same list. the tempting move is to build it twice — once in the cli, once in the web server — because it’s a dozen lines and who’s going to get a dozen lines wrong.

me. i am going to get a dozen lines wrong, six months from now, in exactly one of the two places.

so the schema doesn’t live in either binary. it lives in its own crate that both depend on:

crates/search-schema/src/lib.rs

//! Shared Tantivy search-index schema.
//!
//! Single source of truth for both the CLI (ingest) and the web server
//! (query). Keeping the schema in one place removes the risk that one side
//! adds a field and the other silently reads a stale index.

/// Folded text of the Latin/Greek **original** of a passage.
pub const FIELD_TEXT: &str = "text";
/// Folded + Spanish-stemmed text of a passage's **Spanish** translation.
pub const FIELD_TEXT_ES: &str = "text_es";
// ...
/// Distinct normalized lemmas of a Latin passage, space-joined.
pub const FIELD_LEMMA: &str = "lemma";

that’s the whole reason the crate exists. build_schema() is defined once, register_tokenizers() is defined once, the field-name constants are defined once, and both binaries use them. this is the kind of boring structural guarantee that pulled me off go and onto rust : one crate, two binaries, and the compiler flatly refuses to let them disagree about whether the field is called text_es or text_spanish. a shared constant is a contract a machine enforces, which — per that companion post — is the only kind of contract i actually trust.

but field names were never the bite. the bite was the second kind of disagreement, the one the type system can’t see.

the tokenizer tantivy stores by name

here’s the thing the tantivy docs ↗ tell you if you read closely, and that i learned the other way around: an index does not store its analyzers. it stores their names.

when you index a text field, tantivy writes a string into the index metadata — "fold", say — meaning “this field was tokenized with the analyzer registered under the name fold.” it does not store what that analyzer does. the lowercasing, the accent-folding, the stemming — none of it is persisted. it’s a name and a promise. when the reader opens the index and queries that field, it looks up whatever analyzer is currently registered under "fold" in its own process and runs the query terms through that.

which means the writer and the reader each have to register an analyzer under that name, out of band, and the two had better be byte-for-byte identical. nothing checks this. the field name is in the shared schema; the analyzer behind the name is not. register a subtly different "fold" on the reader — or, in my case, forget to register it at all and let tantivy fall back to its default tokenizer — and every accented or inflected query silently returns nothing, because the query term got folded one way at index time and a different way (or not at all) at search time. filósofos went into the index as filosofos and came back out of the query box as filósofos, and those two strings never meet.

so registration is a function in the shared crate, with a comment i wrote in blood:

crates/search-schema/src/lib.rs

pub const FOLD_TOKENIZER: &str = "fold";
pub const ES_TOKENIZER: &str = "es";

/// Register the two custom analyzers on an [`Index`].
///
/// Tantivy stores a field's tokenizer *by name* in the index metadata, not the
/// implementation, so this MUST be called on every `Index` handle that writes
/// (ingest) or queries (web) the folded fields — otherwise indexing and
/// querying disagree and accented/inflected terms silently return nothing.
pub fn register_tokenizers(index: &Index) {
    let fold = TextAnalyzer::builder(SimpleTokenizer::default())
        .filter(RemoveLongFilter::limit(MAX_TOKEN_BYTES))
        .filter(LowerCaser)
        .filter(AsciiFoldingFilter)
        .build();
    let spanish = TextAnalyzer::builder(SimpleTokenizer::default())
        .filter(RemoveLongFilter::limit(MAX_TOKEN_BYTES))
        .filter(LowerCaser)
        .filter(AsciiFoldingFilter)
        .filter(Stemmer::new(Language::Spanish))
        .build();
    index.tokenizers().register(FOLD_TOKENIZER, fold);
    index.tokenizers().register(ES_TOKENIZER, spanish);
}

and the discipline is: this function gets called on every Index handle, on both sides, immediately after opening. the writer:

crates/cli/src/commands/ingest.rs

let index = Index::create_in_dir(&tmp_index_path, schema)?;
register_tokenizers(&index);
let mut writer = index.writer(50_000_000)?;

the reader:

crates/web/src/search/mod.rs

let index = Index::open_in_dir(index_path)?;
register_tokenizers(&index);
let reader = index
    .reader_builder()
    .reload_policy(ReloadPolicy::OnCommitWithDelay)
    .try_into()?;

miss either call and you don’t get an error. you get an archive that cheerfully reports it contains nothing. the failure is silent, and silent is the worst adjective a bug can wear.

why latin gets folded and spanish gets stemmed

look back at that register_tokenizers and you’ll notice it builds two analyzers, not one, and the difference between them is the single most important decision in the whole search stack.

the fold analyzer is accent-folding only: split on whitespace and punctuation, drop over-long tokens, lowercase, then ASCII-fold ↗ so a query for cesar matches César and virtud matches virtúd. crucially it does not stem. it leaves filósofos and filósofo as distinct tokens.

the es analyzer is that same chain plus one more filter: a Snowball stemmer ↗ set to Language::Spanish ↗ , which collapses filósofos, filósofo and filosofía toward a shared stem so a search for one finds the others.

and the reason there are two, rather than one clever multilingual analyzer, is this: you cannot stem latin with a spanish stemmer. a spanish stemmer knows spanish morphology — that -os is a plural ending, that -ción is a suffix. point it at omnes homines qui sese student praestare and it will happily hack latin endings off according to spanish rules and produce garbage stems that match nothing a latin reader would ever type. so the corpus is split across two fields on the way in: originals (latin, greek) go to a text field analyzed fold-only; spanish translations go to a separate text_es field analyzed fold-plus-stemmer. greek gets fold-only too — the ASCII folder is latin-script only and doesn’t touch greek diacritics at all (a deferred problem i’m honest about below), but at least it doesn’t actively corrupt them.

the writer routes each passage to the field its language belongs in; the reader scopes each query to the field whose analyzer fits. same split, both sides, from the same shared crate. that is the entire point.

64 bytes, because greek

one small tuning that only exists because this corpus isn’t english. every analyzer chain starts with a RemoveLongFilter ↗ , which drops tokens longer than a given number of bytes — a cheap guard so that a 4,000-character run of OCR sludge doesn’t become a single monstrous term. the obvious limit is something like 40 bytes; no real word is 40 characters.

except tantivy counts bytes, and ancient greek isn’t ascii — with its accents and breathing marks it lives in a corner of unicode where a character is two to three bytes in UTF-8. a 13-character greek word is already ~40 bytes, so a 40-byte limit would silently eat legitimate greek vocabulary:

crates/search-schema/src/lib.rs

/// `RemoveLongFilter` byte limit. Greek is multi-byte in UTF-8 (≈3 bytes/char),
/// so 40 bytes would drop legitimate ~13-char Greek words; 64 keeps them while
/// still discarding pathological tokens.
const MAX_TOKEN_BYTES: usize = 64;

64 bytes keeps the real greek words and still discards the pathological ones. it’s a one-constant fix, but it’s the kind of thing you only find by testing in the actual languages you serve — there’s a unit test in the crate that pushes a long greek word through the filter specifically to prove it survives. the english-shaped default would have thrown the word away and, you guessed it, returned nothing for it.

lemma: as a query operator

the last field in the schema is the one i’m proudest of, and it’s the reason the append-last discipline matters. latin is heavily inflected: uirtus, uirtutem, uirtute, uirtutis are all one word wearing different case endings. accent-folding won’t unite them and the spanish stemmer would mangle them. so there’s a separate lemma field: for each latin passage i resolve every word form to its dictionary headword through a lexicon i built, and store the distinct lemmas space-joined.

that field was added to the schema after the index already existed in the wild, which is where this comment comes from:

crates/search-schema/src/lib.rs

builder.add_text_field(FIELD_TEXT, analyzed_text(FOLD_TOKENIZER));
builder.add_text_field(FIELD_TEXT_ES, analyzed_text(ES_TOKENIZER));
// ...
// appended LAST so the existing fields keep their ordinals and an index
// built before this field stays readable until the next full reindex.
builder.add_text_field(FIELD_LEMMA, analyzed_text(FOLD_TOKENIZER));
builder.build()

tantivy identifies fields by their ordinal position, so a field appended at the end doesn’t disturb the ordinals of the fields before it — an index built before lemma existed stays readable until i choose to do a full rebuild. add a field in the middle and every older index becomes a garbage reinterpretation of itself. so: new fields go last, always.

the field is only useful if you can query it, so the query parser carves out a lemma: operator by hand before handing the remainder to tantivy:

crates/web/src/search/mod.rs

fn extract_lemma_filters(query_str: &str) -> (String, Vec<String>) {
    let mut free: Vec<&str> = Vec::new();
    let mut lemmas: Vec<String> = Vec::new();
    for token in query_str.split_whitespace() {
        if let Some(value) = token.strip_prefix("lemma:") {
            let norm = lemma::normalize(value);
            if !norm.is_empty() {
                lemmas.push(norm);
            }
        } else {
            free.push(token);
        }
    }
    (free.join(" "), lemmas)
}

each lemma:<form> token is pulled out of the raw query and normalized through the same shared normalizer the lexicon was built with — NFD-strip the diacritics, lowercase, fold j→i and v→u — so lemma:VIRTVS, lemma:virtus and lemma:Virtūs all reduce to the stored key uirtus. then it becomes an exact-match term on the lemma field:

crates/web/src/search/mod.rs

for lemma in &lemma_filters {
    let term = Term::from_field_text(self.f_lemma, lemma);
    clauses.push((
        Occur::Must,
        Box::new(TermQuery::new(term, IndexRecordOption::Basic)),
    ));
}

so lemma:uirtus finds uirtutem in a sentence that never literally contains it, and habet lemma:gloria ANDs a lemma match against free text. that normalizer is, once again, a single function in a single crate that both the offline lexicon build and the runtime query path call. i’ll spare you the third rendition of the sermon.

the honest part

the blog rule around here is that i don’t get to pretend. so — three warts, in ascending order of how much they bother me.

the snippet sanitizer allows exactly one tag. tantivy’s snippet generator wraps matched terms in <b> for highlighting, and i splice that html straight into the results page — which is an XSS hole the width of the corpus if i trust it. so every snippet passes through a hand-rolled sanitizer that permits <b> and </b> and strips every other tag down to its text content:

crates/web/src/search/mod.rs

fn sanitize_snippet_html(html: &str) -> String {
    // ...
    if c == '<' {
        // collect the full tag, then:
        let lower = tag.to_lowercase();
        if lower == "<b>" || lower == "</b>" {
            result.push_str(&tag);
        }
        // all other tags are stripped (content preserved, tag removed)
    }
    // ...
}

it’s deliberately dumb — a real allowlist parser would be more robust — but the dumbness is the feature: there is exactly one thing it lets through and it’s four characters long. i’ll take a sanitizer i can prove correct by reading it over a clever one i can’t.

the index is a derived artifact, and i had to keep reminding myself of that. postgres is the source of truth. the tantivy index is a throwaway projection of it, rebuilt from scratch into a temp directory and swapped into place atomically, keeping exactly one generation of backup:

crates/cli/src/commands/ingest.rs

// write to a temporary directory, then atomic rename to avoid
// leaving a broken index if the process crashes during rebuild
let tmp_index_path = index_path.with_extension("_building");
// ... build the fresh index into tmp_index_path ...

let prev_path = index_path.with_extension("prev");
if prev_path.exists() {
    std::fs::remove_dir_all(&prev_path)?;
}
if index_path.exists() {
    std::fs::rename(index_path, &prev_path)?;
}
std::fs::rename(&tmp_index_path, index_path)?;

nothing in the index is authored. delete the whole thing and cli ingest --index-only rebuilds it from the database in a couple of minutes. that’s liberating — i can change the analyzer chain, the field list, the lemma resolution, anything, and the fix is “rebuild,” not “migrate.” but it’s also a discipline i had to enforce against my own instinct to treat the search index as if it held state. it holds no state. the moment it seems to, i have a bug.

the by-name tokenizer coupling still fails silently, and i haven’t fixed the class of bug — only this instance. the shared crate means my cli and my web server register the same analyzers today. but nothing at the tantivy level verifies, on open, that the analyzers registered in this process match the ones the index was written with. there’s no checksum of the analyzer chain stored in the metadata to compare against. i’ve centralized registration so the two sides can’t drift by accident, but the failure mode — wrong analyzer, zero results, no error — is still one forgotten function call away, and the only thing between me and it is a discipline and a couple of round-trip tests. if tantivy stored a hash of the analyzer behind each field name and refused to query on mismatch, this whole post’s cold open would have been a loud error instead of a silent afternoon of what the hell did i do. it doesn’t, so i wrote the tests and the comment in blood and moved on.

none of these are bugs. they’re the shape of embedding a search engine whose internals you don’t own: you inherit its contracts, including the ones it enforces at runtime instead of compile time, and your job is to build the compile-time guardrails it didn’t.

lessons learned

• a search index is a distributed system with two nodes living in one repo. the writer and the reader are separate programs that have to agree, and the agreement you can make structural — field names, in a shared crate — you must, precisely because it throws the agreement you can’t make structural into sharp relief. one shared build_schema() is worth more than any amount of care.
• read what your index persists, not what you assume it persists. tantivy stores tokenizers by name, not by implementation. the most productive hour i spent on search was the one where i stopped assuming the index was self-describing and actually traced what a field’s metadata contains. it contains a promise, and a promise needs both parties in the room.
• tune for the languages you actually serve. a 40-byte token limit and a single multilingual analyzer are both fine — for english. greek is multi-byte and latin morphology isn’t spanish morphology, and every english-shaped default in the stack was quietly wrong for my corpus in a way that returned zero results instead of an error. test in the real alphabets.

next: the latin lemmatizer that lemma: leans on — building a multi-million-form inflection lexicon out of a public-domain dictionary file from the 1990s, and why the hardest part wasn’t the morphology but deciding what counts as the same word.

the code isn’t public yet. i’m at github.com/krtffl ↗ if you want to argue with any of this.