← All posts

Shrinking a 3 GB SQLite dictionary to ~10 MB with an FST

The tsk case: finite state transducers in Rust, shared Finnish suffixes, and why SQLite FTS was only a stepping stone.

Tags
DatabasesNewsPerformance
Shrinking a 3 GB SQLite dictionary to ~10 MB with an FST
Contents

In brief

The author of the tsk dictionary app replaced a 3 GB SQLite database with an FST (finite state transducer) binary of about 10 MB — roughly 300× less on disk. The story is about choosing the right data structure, not defaulting to “just use a database.”

What happened

The project is a mobile Finnish–English dictionary. Prefix search started with an in-memory trie: fine for simple languages, not for Finnish’s rich inflection.

A temporary fix was SQLite with full-text search (FTS). Pros: fast to ship, familiar SQL. Cons: users had to download gigabytes of data — unacceptable on low-end devices.

The next step was Rust and FST: an automaton that compactly encodes the lexicon by sharing suffixes and repeated morpheme parts. Instead of a flat list of strings, the structure compresses the dictionary the way morphological lexicons do.

The tsk v2 line ends up around 20 MB on disk instead of a multi-gigabyte dump, while keeping the search behavior they need.

Why it matters

Not every “word lookup” problem equals SQLite or Elasticsearch. When data is static, well structured, and read as one blob from disk or memory, specialized structures (trie, FST, suffix arrays) often beat a general-purpose DB on size and latency.

Takeaways for backend and tooling:

  • FTS in SQLite is a great prototype and a poor final distribution format when weight matters.
  • Morphology and agglutination break naive tries; you need structures that exploit shared endings.
  • Rust + native FST crates is a practical path if the team can step outside “everything in Postgres.”

In practice

If you have a similar problem (glossary, hints, autocomplete over a static corpus):

  1. Treat distribution size as seriously as API latency.
  2. Prototype on SQLite/Postgres, but plan a read-only binary export for production.
  3. Evaluate FST / double-array trie and libraries in your language ecosystem.
  4. Measure memory on target devices, not only on a dev laptop.

For web apps with frequent corpus updates, FST is harder to increment — this case is about a rarely updated dictionary.

Takeaway

Replacing SQLite with FST is not “Rust magic” but the right data model for the domain: static lexicon, prefix search, minimal weight. Read the original if you design glossaries, suggestions, or offline corpora — especially with rich morphology.