How SeekStorm is built

Dual engine architecture for hybrid search

SeekStorm doesn't retrofit vector search onto a lexical index, or bolt keyword search onto a vector database. It runs two separate, first-class, native index architectures under one roof — an inverted index for lexical relevance and an ANN index for vector similarity — and lets a query planner decide how to combine them.

Both engines share a document store and a document ID space, but keep separate storage layouts, indexing, search, and scoring. The query planner exposes 6 dedicated QueryModes and FusionTypes, selectable automatically or manually, and returns the active mode for explainability. Results from both engines are merged with Reciprocal Rank Fusion (RRF) — the user only ever sees a single, unified index.

Sharded index: a processor as a miniature data center

Search engines have long used sharding to split an index across multiple servers — a shard per machine — to scale index size, indexing speed, and query throughput past the limit of one box. SeekStorm moves the same principle down a level: instead of a shard per server, it uses a shard per processor core, enabling intra-query parallelism.

This concept is lock-free and prevents synchronization losses. It allows to fully utilize every core on a CPU during indexing and search.

Documents are partitioned across shards during indexing; a query touches every shard in parallel, and partial results are aggregated into the final top-k. Because each shard is independent, there's no cross-core synchronization or locking — every core stays fully utilized. Both the lexical and vector indices use this leveled, sharded layout: documents accumulate in RAM and, once a shard reaches 64k documents, that level is committed to disk and becomes immutable.

Sharding raises one subtlety: BM25 scores depend on values averaged across an entire shard (document count, average document length), so raw scores aren't directly comparable between shards. SeekStorm re-scores the aggregated top-k using global document frequencies and average document length collected from every shard, so ranking stays consistent regardless of how documents were distributed.

The result, measured indexing 5 million Wikipedia documents: roughly 4–6× faster indexing and 3× shorter query latency compared to a non-sharded index, reaching 40,000 docs/sec indexed and 10,000 queries/sec at 0.25 ms average latency on a single laptop CPU.

N-gram indexing for faster phrase search

A phrase query like "the who" is expensive with a plain inverted index: both terms are frequent, their posting lists are long, and the engine has to intersect two long lists and then check that the positions are actually adjacent. SeekStorm instead indexes bigrams and trigrams of frequent terms directly, moving that intersection and phrase check from query time to index time.

Frequent terms — what other engines call stop words — are the ones worth combining into N-grams: there are only a handful of them, so the extra index size stays small, while their posting lists are the longest and most expensive to intersect. Restricting N-grams to adjacent frequent terms keeps the index cost negligible while accelerating exactly the queries — like "the who" or "to be or not to be" — that would otherwise be slowest.

Measured impact: phrase queries saw mean latency improve by 2.18×, p99.9 tail latency by 7.63×, and some individual phrase queries sped up by up to 3 orders of magnitude — with BM25 scores staying almost identical to single-term indexing.

Lexical search

The steps below cover the lexical engine specifically — the inverted-index side of the dual engine. The vector engine has its own, separate search path, covered in Vector search below.

On the lexical side, SeekStorm processes posting lists Document-at-a-Time (DaaT) rather than Term-at-a-Time, so it never materializes long intermediate result lists in RAM and can stream results — which is what makes huge indices tractable. Intersection and union of roaring-bitmap posting lists are SIMD-accelerated (AVX2 / NEON), combined with galloping intersection and a faster-than-Block-Max WAND for early termination, plus N-gram lookups for frequent-term phrases.

Vector search: sharded & leveled IVF index

The vector engine reuses the same sharding and leveling principle as the lexical engine — one shard per processor core, one level per 64k committed documents — but builds a different kind of sub-index on top of it: an IVF (Inverted File) index, clustered with K-Medoid (PAM — Partition Around Medoids) rather than K-Means.

PAM picks actual data points as cluster centers (medoids) instead of averaging vectors into synthetic centroids the way K-Means does. That tends to produce better-quality clusters, but PAM is more expensive to compute than K-Means at scale. By clustering only within a single 64k-document level of a single shard — never the whole index at once — SeekStorm keeps the clustering cost per sub-index manageable while still gaining K-Medoid's better cluster quality, without giving up the scalability that comes from sharding.

At query time, Approximate Nearest Neighbor Search (ANNS) probes the nearest medoids in each level's IVF sub-index (the nprobe parameter controls the recall/latency trade-off), in parallel across shards and levels — the same fan-out-and-aggregate pattern used for lexical search. Field filters are applied directly during vector search rather than as a post-search step, so filtered queries don't waste candidates on documents that would be discarded anyway.

On-disk storage layout

Every SeekStorm index is a self-contained directory tree, with no external dependencies or database engines. At every commit (auto or manual), the index is serialized to disk for persistence. Every index directory follows the same predictable hierarchy: API keys at the top, indices below each key, shards below each index, and levels (64k-document batches) below each shard. Levels are collected in RAM and committed to disk — becoming immutable — once the threshold is reached. It's a plain directory tree, so backup and restore is just a file copy.

The index can be kept fully in RAM for zero disk access at search time, or memory-mapped for minimal RAM use at slightly higher latency — both modes share the identical file format, so an existing index can switch access mode at any time. Posting lists are roaring-bitmap compressed; term positions are delta-compressed and VINT-encoded; documents are Zstandard-compressed JSON.