‹ back to the set

Albex

rust · wasm · webgpu · 33 kB core

Document search that runs entirely in the browser — no server, no telemetry, no network call after load. Streaming parsers, accent-insensitive fuzzy matching, zero allocator, zero backend. Your documents are processed locally and never transmitted anywhere.

source ↗

Overview

Albex is a full-text document search engine shipped as a 33 KB WebAssembly binary. It runs entirely in the browser: streaming parsers turn eleven file formats into an index, a four-stage pipeline answers queries in under 5 ms, and nothing the user drops ever leaves the machine — no upload, no telemetry, no network call after the initial .wasm fetch.

Albex exposes one primary class — AlbexEngine — that wraps the WebAssembly module and orchestrates indexing, search, persistence, and the adaptive runtime. Additional opt-in helpers (AlbexEngineWorker, AlbexPool, TieredStore, BloomGpu) are exported from subpath entry points so consumers only pay for what they use.

Install, import, use. The WASM binary travels with the npm package and your bundler (Vite, Webpack 5+, Next, esbuild, Rollup, Parcel 2, Bun, Deno) resolves it automatically through import.meta.url. No assets to copy. No paths to remember.

33 KB

main wasm (baseline)

~50 KB

total bundle gzipped (no PDF)

11

supported formats

0

runtime dependencies

How it works — three steps, no server

01 · Drop

The file enters the sandbox. No network, no disk — the sandbox boundary is the trust boundary.

02 · Index

Streaming parsers feed a static BSS pool. Zero-copy through a 64 KB scratchpad.

03 · Search

Bloom skips, Bitap finishes. O(N log k) with a min-heap top-K.

Features

Zero backend

Entire engine lives in a 33 KB WASM binary. No server, no cloud, no network call after load.

Fuzzy & typo-tolerant

Bitap (Shift-Or / Wu-Manber) up to 3 edits. Finds "clausula" even when typed "claúsula".

Accent-insensitive

Latin-1 + Latin-A fold. ES / FR / DE / IT / PT / PL / CZ / TR transparently.

Query DSL

Phrase, OR, fuzzy, mixed. "a b" | c syntax. Up to 4 tokens per simple query.

11 formats

DOCX · XLSX · PDF · HTML · MD · JSON · CSV · EML · RTF · TXT · XML. Heavy ones stream; the lite ones (CSV, EML, RTF) handle BOM, base64, cp1252.

Zero allocator

Static BSS region only. No heap fragmentation, no GC pressure, no OOM mid-search.

Adapts to the host

3 tier binaries (mini / std / pro) × SIMD variants. Auto-picked from deviceMemory and WASM probes.

Cooperative search

searchCooperative() with frameBudgetMs yields to scheduler.yield() between slices. UI thread keeps a chance to paint.

Worker pool

AlbexPool shards documents across N workers. Map-reduce search merges global top-K.

WebGPU pre-filter

WGSL compute shader runs Bloom in parallel for large corpora. Experimental; opt-in via gpu auto-select.

Tiered storage

TieredStore evicts cold docs to OPFS, promotes on demand. Search archives that exceed RAM.

Snapshot v2

Per-document content hash persisted. Save the index to OPFS in milliseconds; reload across sessions with dedup intact.

OCR companion

@albex/ocr drops Tesseract.js next to the engine. Six languages auto-loaded by demand. Scanned PDFs become searchable.

Hybrid PDF mode

Opt-in: native PDFs get their embedded images OCR'd too. For contracts with scanned signatures, reports with screenshots.

The OCR companion

@albex/ocr wraps Tesseract.js as an opt-in companion. Scanned PDFs and images become searchable. The core stays 33 KB — Tesseract loads only the first time you call enableOcr(engine), then cached forever in IndexedDB. Six languages auto-loaded by demand. Still 100 % local, zero backend.

33 KB

core stays untouched

+ 3.5 MB

Tesseract.js (lazy on first OCR call)

~1.8 MB

per language model (cached forever)

0 bytes

initial cost until enableOcr() fires

What's new in v0.3.0 (2026-05-30)

Scanned-PDF OCR

Image-only PDFs are now searchable through the optional @albex/ocr companion. Tesseract.js, lazy by language, zero cost when not enabled.

Hybrid PDF mode

Opt-in alwaysExtractEmbeddedImages OCRs embedded images on top of vector text. For contracts with scanned signatures, reports with screenshot tables.

Snapshot v2

Per-document content hash now persisted. Content-hash dedup survives the save/load round trip. v1 snapshots still load.

Parser-crash recovery

When pdf-extract traps on an unusual PDF, the engine falls back to lopdf-only image extraction. Many "unsupported" PDFs become searchable through OCR.

Hardened lite parsers

CSV strips the UTF-8 BOM. EML decodes base64 and quoted-printable bodies, walks nested multipart. RTF reads \'XX (cp1252) and \uN ? Unicode escapes.

Honest API rename

searchStreamsearchCooperative. The old name implied incremental streaming the method never did. Deprecated alias preserved until 0.4.0.

Install

No backend required. No data leaves the browser. One package, one command:

install · shell
npm install albex
# or
pnpm add albex
# or
deno add npm:albex

To add the optional OCR companion for scanned PDFs and images:

install with ocr · shell
npm i albex @albex/ocr

npm entry points

Albex ships multiple subpath exports so each feature can be tree-shaken independently. Import only what you use.

Import pathSurfacePulls in
albexAlbexEngine, types, errors, profile helpersmain engine code, profile detector, persistence layer, resource manager
albex/workerAlbexEngineWorkermain-thread wrapper that proxies to a Worker
albex/worker-runtime(runs inside a Worker)Worker-side handler; reference via new URL(..., import.meta.url)
albex/poolAlbexPool, AlbexPoolOptionspool coordinator that orchestrates N worker shards
albex/tieredTieredStore, TieredStoreOptionshot/warm tier manager with OPFS persistence of original blobs
albex/gpuBloomGpu, packBloomsFromChunksstandalone WebGPU runtime + WGSL shader for Bloom scan
noteThe library ships six WASM variants of the main engine (3 capacity tiers × baseline/SIMD) plus a lazy 1 MB PDF module. The zero-config path loads the std-baseline binary, which works on every device. If you want runtime tier auto-selection (mini/std/pro picked from navigator.deviceMemory) serve the variants yourself and pass wasmBaseUrl. See the architecture section for the rationale.

Quick start

Up in under a minute:

index.ts · typescript
import { AlbexEngine } from "albex";
const engine = await AlbexEngine.create();
for (const file of input.files) {
  await engine.indexFile(file);
}
const hits = engine.search('"clausula novena" | rescisión');
for (const h of hits) {
  console.log(h.documentName, h.location, h.score, h.snippet);
}

Or the explicit constructor form, with options:

explicit lifecycle · typescript
import { AlbexEngine } from "albex";

const engine = new AlbexEngine();
await engine.init();

await engine.indexFile(myFile);
const results = engine.search('contrato', { windowed: true });

Query syntax

One small query language, five patterns:

PatternMeaningExample match
wordSingle fuzzy token — up to 3 character edits (auto-adjusted by query length)The word in context
a b cAND: all tokens must appear in the same chunk (proximity scored)found a then b then c
"a b"Phrase: tokens must appear in order and adjacentmatched a b exactly
a | bOR: union of two independent searches, merged by scorehas a or b
"a b" | cMix phrase and ORhas a b or just c

Supported formats

What goes in — eleven formats, each with a parser tuned to how the format behaves in the wild:

ExtensionFormatHow it is parsed
.docxWord documentNative Rust/WASM streaming XML parser (word/document.xml); paragraph + table extraction
.xlsxExcel workbookNative Rust/WASM parser; shared strings + inline strings streaming
.pdfPDF documentSeparate albex_pdf.wasm module (~1 MB), loaded lazily on the first PDF
.html .htmHTML<script>/<style> stripped, paragraphs at block-level boundaries
.md .markdownMarkdownCommonMark markers stripped (code fences, headings, links, lists)
.jsonJSONRecursive walk; every string key and leaf indexed
.csvCSVRFC 4180 lite; one row per chunk (location = row number)
.emlEmail (MIME)MIME-lite: From/To/Subject + first text/plain body part
.rtfRTFControl words and groups stripped, text runs preserved
.txtPlain textDirect UTF-8 pass-through, split on double newlines
.xmlXMLTag-stripped, entity-decoded

API reference

Constructor

new AlbexEngine(opts: AlbexOptions) → AlbexEngine

Constructs the engine. The WASM module is NOT loaded yet — call init() before any other method. The constructor only validates options and stores them.

constructor · typescript
import { AlbexEngine } from "albex";

// Zero config — the WASM binary ships with the package and your bundler
// (Vite, Webpack 5+, Next, esbuild, Rollup, Parcel 2, Bun) resolves it
// automatically through `import.meta.url`. No assets to copy, no URL to
// configure, no path to remember.
const engine = new AlbexEngine();
await engine.init();

// Optional overrides — only if you want tier auto-selection or a CDN.
// new AlbexEngine({ wasmBaseUrl: "/assets" })  // serve the 6 variants yourself
// new AlbexEngine({ wasmUrl: "https://cdn.example.com/albex_wasm.wasm" })

AlbexOptions

FieldTypeDescription
wasmUrl?stringExplicit URL to the .wasm binary. Overrides every other option. Useful when serving from a custom CDN.
wasmBaseUrl?stringBase directory containing tier variants (albex_wasm_<tier>[_simd].wasm). Required only if you want runtime tier auto-selection. Leave undefined to fall back to the bundled std-baseline binary.
pdfWasmUrl?stringOverride for albex_pdf.wasm. By default the bundled module is resolved via import.meta.url and loaded lazily on first PDF.
tier?'auto'|'mini'|'std'|'pro'Capacity tier. auto requires wasmBaseUrl because the bundler cannot know which of the 6 binaries to copy. Without wasmBaseUrl the engine loads std by default.
simd?'auto'|'on'|'off'SIMD variant policy. Only effective when wasmBaseUrl is set.
gpu?'auto'|'on'|'off'WebGPU pre-filter policy. Default: auto (enabled when corpus > gpuThreshold).
gpuThreshold?numberMinimum chunk count to engage WebGPU. Default: 20 000.

Lifecycle

engine.init() → Promise<void>

Resolves the WASM URL (using wasmUrl or wasmBaseUrl + tier auto-detection), fetches the binary, instantiates it, runs initial setup, and subscribes the engine to the global ResourceManager. Throws AlbexInitError on fetch or instantiation failure.

engine.reset() → void

Clears every indexed document and search result. The engine is immediately ready to index a fresh corpus. The WASM module instance is preserved (no re-fetch).

engine[Symbol.dispose]() → void

TC39 explicit-resource-management hook. Resets state, unsubscribes from the resource manager, destroys the GPU device (if any), and nulls out internal references so the WASM instance becomes unreachable for GC. Use with using engine = new AlbexEngine(...) when available.

after init · typescript
await engine.init();
// engine.tier         → 'mini' | 'std' | 'pro'
// engine.simdEnabled  → boolean
// engine.gpuEngaged   → true after the first search that uses WebGPU

Indexing

engine.indexFile(file: File) → Promise<IndexedDocument>

Detects the format from the extension, parses the file, and streams text into the WASM index. Content is hashed (FNV-1a 64-bit) before indexing — if a document with the same hash already exists, the previous entry is returned and no work is done. Throws AlbexUnsupportedFormatError or AlbexParseError on failures. Supported: .docx, .xlsx, .pdf, .md, .html/.htm, .json, .csv, .eml, .rtf, .txt, .xml.

indexFile · typescript
const input = document.querySelector('input[type=file]');

input.addEventListener('change', async () => {
  for (const file of input.files) {
    const doc = await engine.indexFile(file);
    console.log(`${file.name}: ${doc.chunks} chunks, hash=${doc.contentHash}`);
  }
});

// Idempotent: re-indexing the same file is a no-op and returns the existing
// IndexedDocument (matched by FNV-1a content hash).

IndexedDocument

FieldTypeDescription
namestringOriginal file name from the File object.
extstringLowercase extension without leading dot.
chunksnumberNumber of chunks produced from this document.
indexTimeMsnumberWall-clock time spent indexing.
textBytesnumberBytes of indexed text contributed by this document.
docIdnumberStable identifier within the engine. Persists across compact().
contentHashstring64-bit FNV-1a hex hash of the source bytes. Used for dedup.

Search

engine.search(query: string, opts?: SearchOptions) → SearchResult[]

Synchronous full-corpus search. Drives the pipeline: parse query → Bloom filter → Bitap fuzzy match → rich scoring → min-heap top-K. Results are returned already sorted by score, descending. May invoke the WebGPU pre-filter automatically when opts.gpu permits and chunk count exceeds gpuThreshold.

engine.searchCooperative(query: string, opts?: SearchOptions) → AsyncIterable<SearchResult>

Cooperative variant of search(). The corpus is processed in slices; between slices the engine yields to the browser scheduler via scheduler.yield() (or requestAnimationFrame fallback). UI stays responsive during 50 ms+ scans. Use this in any interactive search box.

search · typescript
// Synchronous fast path — best for small corpora and tests.
const results = engine.search('"contrato marco" | rescisión', { windowed: true });
for (const r of results) {
  console.log(`[${r.score}] ${r.documentName} · ${r.snippet}`);
}
searchCooperative · typescript
// Cooperative streaming search — main thread stays at 60 fps.
for await (const r of engine.searchCooperative('contrato', { frameBudgetMs: 8 })) {
  renderResult(r);   // render incrementally as results arrive
}

SearchOptions

FieldTypeDescription
windowed?booleanReturn cropped snippets with ASCII ellipsis markers instead of full chunks.
before?numberBytes of context before the match (default 60).
after?numberBytes of context after the match (default 120).
frameBudgetMs?numberSlice duration for searchCooperative() before yielding to the scheduler. Default 8 ms.

SearchResult

FieldTypeDescription
documentNamestringFile name as registered by indexFile().
locationnumberParagraph index (DOCX/TXT/MD/…) or page number (PDF, 1-based).
scorenumberComposite relevance score 0–1000 (higher is better).
snippetstringChunk text, optionally windowed with "... " / " ..." sentinels.
matchStartnumberByte offset of the primary token start in snippet.
matchEndnumberByte offset of the primary token end (exclusive).
matchesMatchSpan[]All matched token spans in query order. Length 1–4.

Tuning knobs

engine.setMaxErrors(n: 0 | 1 | 2 | 3) → void

Maximum edit distance for fuzzy match. 0 = exact only. Engine auto-shrinks for short queries. engine.setMaxErrors(1);

engine.setThreshold(n: number) → void

Minimum score (0–1000) below which results are dropped. Default 250. engine.setThreshold(400);

engine.setMaxResults(n: number) → void

Cap on number of returned results. 1–200. Default 50. engine.setMaxResults(100);

engine.setLanguage(lang: 'off' | 'es') → void

Enable lightweight Spanish stemming on query tokens. Indexed text is never stemmed, so snippets stay faithful to the source. engine.setLanguage('es'); // "contratos" now matches "contrato"

Incremental updates — remove · replace · compact

Indexing is idempotent (content-hash dedup is automatic). Documents can be removed individually without rebuilding the entire index. Reclaim storage on demand with compact().

engine.removeDocument(idOrName: string) → boolean

Tombstone a document. Subsequent searches skip its chunks. Storage is reclaimed by compact(). id accepts the file name or contentHash returned by indexFile.

engine.replaceDocument(name: string, newFile: File) → Promise<IndexedDocument>

Atomic remove + re-index. Bypasses dedup so re-indexing the same bytes after a remove works. await engine.replaceDocument('contract.pdf', newVersionFile);

engine.compact() → void

Reclaim storage from tombstoned documents. Rewrites internal arrays in place. Doc IDs of survivors are preserved.

incremental · typescript
engine.removeDocument('contract-2024-03.pdf');
engine.removeDocument(doc.contentHash);  // by hash also works

engine.removeDocument('old.pdf');
engine.compact();  // bytes freed; subsequent indexFile sees real headroom

Persistence — snapshots on OPFS & IndexedDB

The full engine state can be serialised to a binary blob and restored later. OPFS is preferred (zero-copy writes); IndexedDB is the universal fallback. A 16 MB snapshot typically restores in tens of milliseconds — far faster than re-parsing the documents.

engine.save(name: string) → Promise<void>

Serialise the index to a binary snapshot in OPFS (preferred) or IndexedDB. await engine.save('my-corpus');

engine.load(name: string) → Promise<boolean>

Restore a previously saved snapshot. Returns true on success, false if missing or header mismatched.

engine.loadOrInit(name: string) → Promise<boolean>

Convenience wrapper: load if it exists, otherwise reset() and start clean. await engine.loadOrInit('my-corpus');

engine.deleteSnapshot(name: string) → Promise<void>

Remove a snapshot from storage. await engine.deleteSnapshot('old-corpus');

engine.listSnapshots() → Promise<string[]>

List the names of all snapshots saved in the current origin. const names = await engine.listSnapshots();

load fallback · typescript
if (!await engine.load('my-corpus')) {
  console.log('No snapshot yet; starting fresh');
}

Introspection — getStats() & getLastSearchStats()

engine.getStats() → EngineStats

Snapshot of current engine state: document count, chunk count, memory usage, loaded tier, capacity caps.

engine.getLastSearchStats() → SearchStats | null

Bloom/Bitap pipeline counters from the most recent search call. Useful for debugging relevance and detecting performance regressions.

EngineStats

FieldTypeDescription
documentsnumberActive (non-tombstoned) document count.
chunksnumberTotal indexed chunks.
textUsednumberBytes of indexed text currently stored.
textCapacitynumberMaximum text bytes the loaded tier can hold.
wasmMemoryBytesnumberTotal WASM linear memory size (BSS + grown pages).
tier'mini'|'std'|'pro'|nullTier of the loaded binary. null before init().
maxChunksnumberCompile-time chunk capacity for the loaded tier.
maxDocsnumberCompile-time document capacity for the loaded tier.

SearchStats

FieldTypeDescription
querystringVerbatim query string.
timeMsnumberEnd-to-end search time.
resultsnumberNumber of results above the threshold.
bloomTestednumberChunks tested against the Bloom filter.
bloomPassednumberChunks that passed Bloom (subset of tested).
bitapMatchednumberChunks confirmed by Bitap (subset of passed).

Adaptive runtime — profile detection & helpers

The library exposes pure helpers from the same package so consumers can read the device profile, override tier selection, or build their own UI around resource state. None of them require an initialised engine.

detectProfile(opts?: { fresh?: boolean }) → Promise<DeviceProfile>

Probe the host capabilities (cores, memory, WASM features, WebGPU, storage budget, network, battery). Result cached in sessionStorage. Pass fresh: true to bypass the cache.

pickTier(profile: DeviceProfile) → 'mini' | 'std' | 'pro'

Pure heuristic: <=1 GB → mini, 2–4 GB → std, >=8 GB → pro, null (Safari) → std.

pickWorkerCount(profile: DeviceProfile) → number

cores/2 clamped [1, 8]. Falls to 1 when battery is reported below 20 % and discharging.

shouldUseGpu(profile: DeviceProfile, chunkCount: number, threshold?: number) → boolean

true when WebGPU is available AND chunk count crosses the threshold (default 20 000).

profile helpers · typescript
import { detectProfile } from 'albex';
const profile = await detectProfile();
console.log(profile.memoryGB, profile.wasm.simd);

const tier = pickTier(profile);
const workers = pickWorkerCount(profile);
if (shouldUseGpu(profile, engine.getStats().chunks)) { /* … */ }

DeviceProfile

FieldTypeDescription
coresnumbernavigator.hardwareConcurrency.
memoryGBnumber | nullnavigator.deviceMemory (capped at 8 GB by spec; null on Safari).
wasm.simdbooleanWebAssembly v128 supported (validated via probe module).
wasm.bulkMemorybooleanBulk memory ops supported.
wasm.threadsbooleanThreads supported AND page cross-origin isolated.
webgpubooleannavigator.gpu present.
coopCoepbooleancrossOriginIsolated === true.
storage{ quotaBytes, usageBytes }navigator.storage.estimate() result.
net{ effectiveType, saveData }Connection info if reported (Chrome only).
battery{ level, charging } | nullBattery state if available.
visiblebooleandocument.visibilityState at probe time.

Off-main-thread — AlbexEngineWorker

new AlbexEngineWorker(opts: AlbexWorkerOptions) → AlbexEngineWorker

Drop-in replacement for AlbexEngine that runs the entire engine inside a Web Worker. Surface is identical except every method returns a Promise. Files are transferred via postMessage with transferable ArrayBuffers — no copy. Import from albex/worker. The runtime script is exported separately as albex/worker-runtime and must be referenced via new URL(..., import.meta.url).

worker · typescript
import { AlbexEngineWorker } from 'albex/worker';

// Same zero-config pattern as the main engine — the worker runtime URL is
// the only thing you must point at (so the bundler can spawn it).
const engine = new AlbexEngineWorker({
  workerUrl: new URL('albex/worker-runtime', import.meta.url),
});

await engine.init();
const results = await engine.search('contrato', { windowed: true });

Parallelisation — AlbexPool

new AlbexPool(opts: AlbexPoolOptions) → AlbexPool

Orchestrates N worker shards. Documents are sharded round-robin across workers. Searches broadcast to every shard and the coordinator merges top-K results preserving the global descending score order. Default worker count is half of hardwareConcurrency clamped to [1, 8]; battery-aware (drops to 1 on low battery). Import from albex/pool.

pool · typescript
import { AlbexPool } from 'albex/pool';

const pool = new AlbexPool({
  workerUrl: new URL('albex/worker-runtime', import.meta.url),
  workers: 'auto',   // cores/2 clamped [1, 8]
});
await pool.init();

await pool.indexFile(fileA);                   // sharded round-robin
await pool.indexFile(fileB);
const results = await pool.search('contrato'); // map-reduce across shards

Large corpora — TieredStore

new TieredStore(engine: AlbexEngine, opts?: TieredStoreOptions) → TieredStore

Adds hot/warm tiers behind the engine. When the engine reaches evictThreshold of capacity, the LRU document is removed and its original blob remains in OPFS. promote(name) brings it back by re-indexing from the persisted blob. Import from albex/tiered.

tiered · typescript
import { AlbexEngine, TieredStore } from 'albex';

const engine = new AlbexEngine();
await engine.init();

const store = new TieredStore(engine, { evictThreshold: 0.85, hotFloor: 4 });
await store.init();

await store.indexFile(file);          // persists original blob in OPFS
await store.promote('older-doc.pdf'); // brings warm doc back to engine

GPU acceleration — BloomGpu (advanced)

new BloomGpu() → BloomGpu

Standalone WebGPU Bloom-scan accelerator. AlbexEngine instantiates one automatically when opts.gpu permits and the corpus exceeds gpuThreshold (default 20 000 chunks). You only need to touch this class directly to integrate the WGSL shader into a non-Albex pipeline. Import from albex/gpu.

Typed error hierarchy

Every error thrown by Albex extends AlbexError. Switch on the kind discriminator (which survives structuredClone across worker boundaries) or use instanceof against the subclasses.

ClasskindThrown when
AlbexError(base)Base class for every Albex error. Carries kind discriminator.
AlbexInitErrorinitWASM fetch failed, init() not called, or PDF module not initialised.
AlbexUnsupportedFormatErrorunsupported_formatFile extension is not in the supported list. Carries ext field.
AlbexParseErrorparseA parser (DOCX/XLSX/PDF/JSON/…) failed. Carries format field.
AlbexCapacityErrorcapacityScratchpad write would exceed buffer size, or hard cap reached.
error handling · typescript
import {
  AlbexError, AlbexInitError, AlbexParseError,
  AlbexUnsupportedFormatError, AlbexCapacityError,
} from 'albex';

try {
  await engine.indexFile(file);
} catch (e) {
  if (e instanceof AlbexUnsupportedFormatError) {
    console.warn(`Unsupported: .${e.ext}`);
  } else if (e instanceof AlbexParseError) {
    console.warn(`Failed to parse ${e.format}:`, e.message);
  } else if (e instanceof AlbexCapacityError) {
    console.warn('Engine full — upgrade tier or use TieredStore');
  } else throw e;
}

Architecture

Albex is a full-text search engine that lives entirely in the user's browser. That sentence is the thesis. Every other choice — the algorithms, the memory model, the absence of an allocator — is a corollary of it. Four axioms follow:

  1. Zero server. No back-end ever. Documents never transit the network.
  2. Human latency. Searches must answer within ~100 ms — the perception threshold for "instant".
  3. Reduced footprint. Bytes shipped to the browser are a tax on every page load.
  4. Adapts to the machine. The same package runs on a Chromebook and a Mac Pro, picking what each can support.

The layer map

The codebase is split into six layers. Dependencies flow strictly upward: a layer never calls down into the one below it. This is what lets the algorithm layer be testable without WASM, and the adaptive layer be evolved without touching Rust.

0 · Pure algorithms

Rust no_std · core/ — Bloom filter, Bitap matcher, light Spanish stemmer, Unicode fold tables. No heap, no allocator, no I/O. Testable on host without WASM.

1 · Streaming parsers

Rust no_std · ingest/ — XML byte state machines for DOCX and XLSX. Same constraints as layer 0. New Rust formats live here.

2 · WASM shell

Rust · wasm/ — BSS arrays, C ABI exports, EngineState bookkeeping, resumable search, tier feature flags. Pegs layers 0 and 1 to a JS-facing interface.

3 · PDF module (separate)

Rust + std · pdf-wasm/ — pdf-extract wrapper. Its own WASM binary (~1 MB) loaded lazily on first PDF — never paid by users who do not touch PDFs.

4 · TypeScript orchestrator

TS · src/ — AlbexEngine API, format indexers in TS, query parser, typed errors, persistence layer, content hashing.

5 · Adaptive capabilities

TS · src/{profile,resource-manager,pool/,gpu/,tiered-store}.ts — Profile detection, resource awareness, worker pool, WebGPU runtime, tiered storage. Strictly opt-in; the base engine works without any of it.

The search pipeline — from query to ranked results in four stages

01 · Parse query

Detect simple / phrase / OR. Up to 4 tokens. Optional stemming. Cost: < 5 µs.

02 · Bloom filter

AND + compare per chunk against the pattern Bloom. Cost: 2 instructions / chunk.

03 · Bitap fuzzy

Bit-parallel match on the chunks that passed Bloom. Cost: ~10 ns / chunk.

04 · Score + top-K

Rich scoring (accuracy + WB + TF + position + proximity + IDF) into a min-heap. Cost: O(log K).

The cost ladder is deliberate: the cheapest filter runs first on every chunk; the expensive ones only on what survives. A typical search tests 100 000 chunks but runs the Bitap matcher on under 1 % of them. The Bloom filter is the gatekeeper that keeps total search latency under 10 ms even on big corpora.

The pipeline itself is resumable: searchBegin sets up token state, searchSlice(N) processes N chunks and returns done=0/1. That lets searchCooperative yield to the scheduler between slices (frame budgeting) without blocking a paint. It honours an optional GPU candidate mask and a tombstone bitset, then funnels survivors into scoring and the top-K heap.

Bloom filter — 64-bit probabilistic gate

A Bloom filter answers "could the pattern be here?" in two machine instructions: an AND and a compare. We use a single u64 per chunk, with each character mapped to one of 64 buckets by c & 0x3F. A chunk passes iff (chunkBloom & queryBloom) == queryBloom.

The hash function is intentionally trivial. A cryptographic hash would give better distribution but cost 10–100× more CPU. The Bloom runs on every chunk in the corpus — sometimes 100 000 times per query — so raw speed wins over perfect distribution.

For typical European text the filter rejects 80–95 % of chunks before the more expensive Bitap stage. False positives are unavoidable but harmless: Bitap discards them at no extra cost.

noteThe character Bloom is only sound for exact tokens — a fuzzy substitution can introduce a character absent from the chunk. Since 0.6.0 it is applied only when a token matches with zero errors; a 256-bit trigram q-gram signature carries the fuzzy case. Trigrams are far rarer than characters, so the trigram filter shrinks the Bitap candidate set ~10× on prose, and it stays sound under fuzzy matching via the q-gram lemma (an occurrence with e errors keeps ≥ N − 3e of a token's exact trigrams). Signatures are rebuilt from the text pool after compaction and snapshot restore.

Bitap — bit-parallel fuzzy match

The Shift-Or / Wu-Manber algorithm keeps the matching state inside a single u64 register. Each text byte advances the state with a shift + OR + AND — no backtracking, no per-character allocations, no branches dependent on the data.

We extend it to k errors (substitution, insertion, deletion) by maintaining k+1 parallel registers. Albex caps k at 3: more produces noise on short queries and multiplies the linear cost. Pattern length is capped at 64 bytes — the width of the register. Longer queries are truncated with a truncated flag the host can surface in its UI. The error budget is adaptive: tokens of ≤5 characters match exactly, ≤8 allow 1 error, longer allow up to 3.

Scoring and top-K

Each survivor gets a capped sum (0–1000) of components: base accuracy (−200 per average edit), word-boundary bonus, term frequency, document position, phrase proximity (close + in order), and an approximate IDF from per-Bloom-bit document frequency. Weights are hand-tuned.

Results stream into a fixed-size min-heap: insert is O(log K), and when full a new hit only displaces the current worst. A final heap-pop yields the K best in descending order — O(k log k), versus the O(k²) selection sort an earlier version used (27× faster than insertion sort in practice).

Accent folding and stemming

fold_utf8_char maps each code point to a lowercase ASCII base: Latin-1 Supplement and Latin Extended-A (Polish, Czech, Turkish…) collapse to a/c/e/n/o/s/u/z. Accented search is free — acción and accion fold to the same bytes. The same fold runs at index and query time.

The optional Spanish stemmer is a handful of high-impact Snowball-Spanish suffix rules, inlined (~150 bytes vs the full 6 KB state machine). Applied only to query tokens, never to indexed text, so snippets stay faithful — recall improves because the query reduces to a shared prefix that still matches the original word.

Memory model — BSS-only, no allocator

The default wasm32-unknown-unknown target has no OS to call for memory. Pulling in std would drag in dlmalloc or wee_alloc, inflate the binary, add allocation overhead, and introduce a failure mode (OOM) that is hard to surface cleanly to JavaScript.

Instead, every storage region is a zero-initialised static array — TEXT_POOL (16 MB), CHUNKS, NAME_POOL, CHUNK_SIG. Because the BSS segment is described in the binary but not stored, a .wasm with 16 MB of static arrays still weighs ~33 KB on disk. At runtime the WebAssembly linear memory grows to accommodate the layout (~20 MB for std tier), but this grow happens once at instantiation — no per-call allocation, no fragmentation, no OOM during a search.

The trade-off: capacity is a fixed ceiling. Exceeding it is a silent stop, not an error. That is acceptable because the user already chose a tier; a "give me more space" feature would mean an allocator, which would mean fragmentation and unpredictable latency.

The JS ↔ WASM bridge

The entire JS↔WASM data plane is one 64 KB shared byte buffer — the scratchpad. The dance is always: getBuffer(size) → a pointer; JS writes bytes there; JS calls the action (setPattern(len), feedXmlBytes(len), search()); WASM reads the scratchpad, processes, stores results; for output it writes back and JS reads it. No JSON across the boundary — just offsets and lengths. That is what keeps the boundary cost near zero.

At init() the exports contract is validated before the engine returns: memory is a real WebAssembly.Memory, every required export exists, and the ABI version matches. A mismatched binary fails loudly here instead of crashing later inside indexFile.

Adaptive runtime — same package, different machines

Albex ships six variants of the main WASM binary (three capacity tiers, each with and without SIMD) plus the lazy PDF module. The browser fetches one variant per init() call, picked from the device profile.

TierMax docsMax chunksMax textRule (deviceMemory)Working setBinary
mini3225 0004 MB≤ 1 GB~5 MB33 KB
std128100 00016 MB2–7 GB~20 MB33 KB
pro1 024800 000128 MB≥ 8 GB~160 MB33 KB

SIMD sits on top of every tier: the runtime probes for WebAssembly v128 by attempting to validate a tiny module that uses it. If it validates, the engine fetches albex_wasm_<tier>_simd.wasm — a binary built with -C target-feature=+simd128. The inner loop of the Bloom batch is branchless precisely so LLVM can auto-vectorise it.

noteWhy "8 GB" appears as a ceiling everywhere: the W3C navigator.deviceMemory spec caps reported memory at 8 GB for privacy. A Mac Pro with 192 GB reports 8 just like an iPad with 16 GB — they cannot be distinguished. The pro tier (128 MB pool) is the largest default because the working set must still fit comfortably in a browser tab — Chrome starts flagging pages as "high memory" around 500 MB. For genuine archive-scale corpora the right answer is the TieredStore (LRU eviction with OPFS-persisted blobs) or the AlbexPool sharded across N workers, not a tier with a 1 GB pool.

WebGPU pre-filter — Bloom on the GPU when it pays off

The Bloom check on 100 000 chunks is embarrassingly parallel: each test is independent. A WGSL compute shader can run all 100 000 of them in a single dispatch. The result is a packed bitset of candidates; the CPU then runs Bitap only on those.

Engaging the GPU has a fixed cost (upload + dispatch + readback, ~3–5 ms). Below ~20 000 chunks the CPU wins. Above it, the GPU gives a 5–10× speedup. The default threshold is exactly that break-even point, but it is exposed as gpuThreshold for tuning.

The candidate mask is single-shot: it applies to the next searchBegin and is automatically cleared at the end of the last slice. That prevents stale masks from silently corrupting an unrelated next search if anything ever goes wrong in the JS layer.

PDF and OCR path

The PDF module is a separate WASM binary fetched only on the first PDF — users who never touch PDFs never pay for it. It wraps pdf-extract/lopdf to pull vector text per page. Built with panic=abort, so a malformed PDF traps the instance; the host catches that, discards the poisoned instance, and (if OCR is wired) falls back to extracting embedded page images via lopdf — recovering content pdf-extract could not.

The OCR orchestrator (@albex/ocr, wired via engine.attachOcr(adapter)) manages a pool of Tesseract.js workers and feeds them the JPEG/JPEG2000 image XObjects the PDF module extracts per page. Two modes: scanned-PDF (image-only files) and hybrid (OCR on top of vector text). Best-effort: one failed image never stops the page, one failed page never stops the document. A lightweight script / letter-frequency heuristic guesses the language so the orchestrator loads the right Tesseract traineddata — picking the wrong model is the single biggest hit to OCR accuracy. Tesseract itself (~3.5 MB, LSTM engine compiled to WASM) is loaded lazily via await import("tesseract.js") only when enableOcr() has been called and a scanned image needs recognising.

Rust + TypeScript — when each language wins

Some parsers live in Rust; others live in TypeScript. The rule is not "Rust for important things". It is: which primitive of the host environment wins for this operation?

TaskLanguageWhy
DOCX parsingRustStreaming XML state machine. Files reach hundreds of MB; loading the DOM would be infeasible.
XLSX parsingRustSame as DOCX. Shared strings tables get huge in real-world workbooks.
PDF extractionRustpdf-extract dependency is mature and large. Worth its own module loaded lazily.
Bloom + Bitap hot loopRustBit-parallel arithmetic. WASM v128 wins clearly over JS bitwise.
Markdown / HTML / RTFTSFiles fit in memory; V8 regex is JIT-compiled C++ — faster than a hand-written WASM parser.
JSON parsingTSJSON.parse is C++ inside the JS engine — no Rust port can beat it.
Content hash FNV-1aTSOne-shot per file, not a hot path. JS bitwise is fine at 100 MB/s.
WebGPU shaderWGSLGPU language. No alternative.
OPFS / IndexedDBTSBrowser APIs; unreachable from WASM.
Worker orchestrationTSnew Worker() is a JS API; postMessage protocol lives where the workers live.

DOCX, XLSX and PDF go to Rust because the source files can be hundreds of megabytes and streaming a state machine is the only way to handle them. Markdown, HTML, JSON, CSV, EML and RTF go to TypeScript because they fit comfortably in memory and V8 gives us String.prototype.replace with JIT-compiled regex, which beats any hand-written WASM parser for documents of typical size.

noteThis page summarises the parts that come up most often in conversation. For the full pseudocode, layout tables and side-by-side cost comparisons, check the in-repo Maintainer's guide (MAINTAINER.md) and Technical deep-dive (TECHNICAL.md).

Storage & identity

Three primitives keep the engine honest across sessions and across repeated uploads: OPFS for snapshots, IndexedDB as the fallback, and a 64-bit content hash for deduplication. None of them are configurable surface area — they're internal contracts you should understand when reasoning about what the engine actually does.

OPFS & IndexedDB — why both

When you call engine.save("my-corpus"), the engine serialises the entire index (chunks, document table, text pool, Bloom filters) to a single binary blob and writes it to browser storage. engine.load("my-corpus") reads it back and memcpy's it into the BSS arrays. A 16 MB corpus restores in tens of milliseconds — orders of magnitude faster than re-parsing 50 DOCXs.

APIAvailable sinceWrite 16 MBWhy we picked it
OPFSChrome 102 · Safari 15.2 · Firefox 111~20 msZero-copy FileSystemWritableFileStream.write(Uint8Array)
IndexedDBAll browsers since 2015~80 msStructured-clone of the Uint8Array

The router lives in src/persistence.ts and detects navigator.storage.getDirectory at runtime. If present → OPFS path. Otherwise → IndexedDB. Both paths satisfy the same contract (savePersisted / loadPersisted / deletePersisted / listPersisted), so calling code is identical.

Two distinct uses of OPFS

The persistence backend serialises the engine state: chunks, Bloom filters, doc table. TieredStore uses OPFS differently — it stores the original file blobs, so an evicted document can be promoted back without asking the user to re-pick the file. Two stores, two purposes, same OPFS API.

FNV-1a 64-bit — why this hash

Every call to engine.indexFile(file) hashes the raw bytes before any parsing. If a document with that hash already lives in the index, the engine returns the existing entry and skips the work. This makes indexFile idempotent and safe to call repeatedly — drag the same DOCX twice, the index stays clean.

HashThroughput in JSOutputTrade-off
FNV-1a 64-bit~100 MB/s16 hex charsNon-cryptographic; ~10⁻¹⁵ collision probability at 128 docs.
SHA-256~30 MB/s64 hex charsCryptographic; 3× slower, no value added for our use case.
MurmurHash3~150 MB/sVariableSlightly faster; less portable across language ecosystems.

Why non-cryptographic is fine here

We don't need adversarial collision resistance. The threat model for a content hash in a deduplication store is "the user accidentally dragged the same file twice", not "an attacker is crafting two distinct files that hash the same to confuse the index". If an attacker could exploit a deliberate collision, the worst case would be that their second file does not get indexed — there is no path to elevation of privilege, data leak or denial of service.

SHA-256 would cost 3× the throughput for zero added safety in this scenario. We chose throughput. The same logic is used by Git for object identification (well, until SHA-256 transition, but Git's case IS adversarial), by SQLite for ROWID generation, by countless databases for dedup.

Where the hash surfaces

Two places. First, internally during indexFile — the dedup check. Second, on the returned IndexedDocument.contentHash field. The host can pass that hash to engine.removeDocument(hash) as a stable identifier that survives rename: if your user uploads same-contract-renamed.pdf and you want to remove the previously-uploaded contract.pdf that contains the same bytes, the hash matches even though the names don't.

Capacity & numbers

Default limits for the std tier (mini and pro scale ×0.25 and ×8 respectively):

16 MB

text pool · std tier (4 MB mini · 128 MB pro)

100k

chunk capacity · std tier (25k mini · 800k pro)

128 docs

document limit · std tier (32 mini · 1 024 pro)

64 chars

query length — Bitap u64 register width

Performance figures

< 5 ms

typical query, full corpus

80–95 %

chunks rejected by Bloom before Bitap

27×

min-heap top-K vs insertion sort

5–10×

GPU speedup above 20 000 chunks

OperationCostNotes
Parse query< 5 µssimple / phrase / OR detection, up to 4 tokens
Bloom test2 instructions / chunkone AND + one compare per chunk, up to 100 000 per query
Bitap match~10 ns / chunkonly on chunks that passed Bloom (< 1 % of corpus typically)
Score + top-KO(log K) insertfixed-size min-heap; final pop O(k log k)
GPU engage overhead~3–5 msupload + dispatch + readback; break-even at ~20 000 chunks
Snapshot write, 16 MB (OPFS)~20 mszero-copy FileSystemWritableFileStream
Snapshot write, 16 MB (IndexedDB)~80 msstructured-clone of the Uint8Array
Content hash (FNV-1a, JS)~100 MB/sone-shot per file, not a hot path
Snapshot restore, 16 MBtens of msmemcpy into BSS arrays — orders of magnitude faster than re-parsing
performanceThe search complexity is O(N log k): the Bloom gate is linear over chunks at two instructions each, and only the top-K heap pays the logarithmic factor. The memory layout is fixed at instantiation — TEXT_POOL 16 MB + CHUNKS 3.2 MB + scratchpad + doc names for the std tier, ~20 MB working set total.

Guides — recipes by use case

From install to advanced setups: cooperative search, worker pool, big-corpus tiering, and integration in React / Angular. Every recipe starts with npm install albex.

ESM / TypeScript

index + search · typescript
import { AlbexEngine } from "albex";

// 1. Construct + init. The WASM ships with the package and your bundler
//    resolves it through import.meta.url — Vite, Webpack 5+, Next, esbuild,
//    Rollup, Parcel 2, Bun all handle this automatically. No assets to copy.
const engine = new AlbexEngine();
await engine.init();

// 2. Index files from drag-and-drop or <input type="file">.
const input = document.querySelector("input[type=file]");
input.addEventListener("change", async () => {
  for (const file of input.files) {
    const doc = await engine.indexFile(file);
    console.log(`Indexed ${doc.name}: ${doc.chunks} chunks`);
  }
});

// 3. Search.
const results = engine.search('"contrato marco" | rescisión', { windowed: true });
for (const r of results) {
  console.log(`[${r.score}] ${r.documentName} — ${r.snippet}`);
}
advanced · typescript
// Tweak relevance.
engine.setMaxErrors(1);     // tighter fuzziness
engine.setThreshold(400);   // only return strong hits
engine.setMaxResults(100);
engine.setLanguage("es");   // light Spanish stemming on queries

// Inspect what was loaded.
console.log(engine.tier);          // 'std' by default
console.log(engine.simdEnabled);   // boolean
console.log(engine.gpuEngaged);    // true once first search uses WebGPU

// Want tier auto-selection? Serve the 6 binaries yourself and pass wasmBaseUrl.
// The engine then picks mini/std/pro based on navigator.deviceMemory.
//   new AlbexEngine({ wasmBaseUrl: '/assets', tier: 'auto', simd: 'auto' })

Cooperative search

cooperative · typescript
// Cooperative search — yields to the scheduler between slices so the UI
// thread keeps a chance to paint during long scans. NOTE: results arrive
// in one batch after the search completes; the async iterator exists so
// callers can break early. True incremental streaming is on the backlog.

for await (const r of engine.searchCooperative(query, { frameBudgetMs: 8 })) {
  renderResultCard(r);
  if (results.length >= 50) break;   // stop early — engine cleans up
}
search box with debounce · typescript
// Wire it into a search input with debounce.
let abortToken = 0;
input.addEventListener("input", async () => {
  const token = ++abortToken;
  const query = input.value.trim();
  resultsEl.innerHTML = "";
  if (!query) return;

  for await (const r of engine.searchCooperative(query, { frameBudgetMs: 8 })) {
    if (token !== abortToken) break;   // newer query started — stop
    appendResult(r);
  }
});

Persistence

save · typescript
// First visit: index the user's documents normally.
for (const file of files) {
  await engine.indexFile(file);
}

// Save a snapshot to OPFS (preferred) or IndexedDB (fallback).
await engine.save("my-corpus");
restore · typescript
// On subsequent visits: load instantly instead of re-indexing.
const restored = await engine.loadOrInit("my-corpus");
if (restored) {
  console.log(`Restored ${engine.getStats().documents} documents`);
} else {
  console.log("No snapshot yet, starting fresh");
}

// List all snapshots in this origin.
const names = await engine.listSnapshots();

// Delete one explicitly.
await engine.deleteSnapshot("old-corpus");

Web Worker

worker · typescript
// Run the engine in a Web Worker. Same surface, every call returns a
// Promise. Main thread never blocks on indexing or search.
import { AlbexEngineWorker } from "albex/worker";

// Only the worker runtime URL is required so the bundler spawns it.
// The main WASM is resolved automatically inside the worker.
const engine = new AlbexEngineWorker({
  workerUrl: new URL("albex/worker-runtime", import.meta.url),
});

await engine.init();
await engine.indexFile(file);
const results = await engine.search("contrato", { windowed: true });
worker · advanced · typescript
// Streaming search works through the Worker too.
for await (const r of engine.searchCooperative("contrato", { frameBudgetMs: 8 })) {
  render(r);
}

// Dispose when done (TC39 Symbol.dispose).
engine[Symbol.dispose]();

Worker pool

pool · typescript
// AlbexPool shards documents across N workers. Searches map-reduce.
// Best for large corpora on multi-core machines.
import { AlbexPool } from "albex/pool";

const pool = new AlbexPool({
  workerUrl: new URL("albex/worker-runtime", import.meta.url),
  workers:   "auto",   // = cores/2, clamped [1, 8]
});

await pool.init();
console.log("Shards:", pool.workerCount);

// Documents are sharded round-robin.
for (const file of files) {
  await pool.indexFile(file);
}

// Search broadcasts to every shard; coordinator merges top-K.
const results = await pool.search("contrato", { windowed: true });
pool · advanced · typescript
// Aggregate stats across shards.
const stats = await pool.getStats();
console.log(`${stats.documents} docs across ${pool.workerCount} shards`);

// Remove documents — the pool finds the shard that owns it.
await pool.removeDocument("old-contract.docx");

// Compact every shard at once.
await pool.compact();

// Stream results from a pooled search.
for await (const r of pool.searchCooperative("contrato")) {
  render(r);
}

Big corpora — TieredStore

tiered · typescript
// TieredStore keeps hot documents in the engine and warm ones in
// OPFS. When the engine fills past evictThreshold, the LRU document
// is removed from RAM but its original blob stays persisted.
import { AlbexEngine, TieredStore } from "albex";

const engine = new AlbexEngine();
await engine.init();

const store = new TieredStore(engine, {
  evictThreshold: 0.85,   // start evicting when 85 % full
  hotFloor: 4,            // never evict the last 4 docs
});
await store.init();

for (const file of files) {
  await store.indexFile(file);   // persists blob + adds to engine
}
tiered · advanced · typescript
// Bring a warm document back into the engine on demand.
const promoted = await store.promote("old-contract.pdf");
if (promoted) {
  // Now searchable in the engine again.
  engine.search("clausula");
}

// Inspect the tier balance.
const stats = store.getTierStats();
console.log(`${stats.hot} hot · ${stats.warm} warm · ${stats.totalBytes} bytes`);

// Forget a document entirely (engine + OPFS).
await store.forget("expired.pdf");

React

DocSearch.jsx · react
import { useEffect, useRef, useState } from "react";
import { AlbexEngine } from "albex";

export function DocSearch() {
  const engineRef = useRef(null);
  const [hits, setHits] = useState([]);
  const [ready, setReady] = useState(false);

  useEffect(() => {
    const e = new AlbexEngine();
    e.init().then(() => {
      engineRef.current = e;
      setReady(true);
    });
    return () => engineRef.current?.[Symbol.dispose]();
  }, []);

  async function handleFile(ev) {
    for (const file of ev.target.files) {
      await engineRef.current.indexFile(file);
    }
  }

  function handleSearch(q) {
    setHits(engineRef.current?.search(q, { windowed: true }) ?? []);
  }

  if (!ready) return <p>Loading…</p>;
  return (
    <div>
      <input type="file" multiple onChange={handleFile} />
      <input onChange={e => handleSearch(e.target.value)} />
      {hits.map(h => (
        <div key={h.documentName + h.location}>
          <b>{h.documentName}</b> — {h.snippet}
        </div>
      ))}
    </div>
  );
}

Angular

search.component.ts · angular
import { Component, OnInit, OnDestroy } from "@angular/core";
import { AlbexEngine } from "albex";

@Component({
  selector: "app-search",
  template: `
    <input type="file" multiple (change)="onFiles($event)" />
    <input (input)="onQuery($event)" placeholder="Search…" />
    <div *ngFor="let h of hits">
      <b>{{ h.documentName }}</b> — {{ h.snippet }}
    </div>
  `,
})
export class SearchComponent implements OnInit, OnDestroy {
  private engine!: AlbexEngine;
  hits: any[] = [];

  async ngOnInit() {
    this.engine = new AlbexEngine();
    await this.engine.init();
  }

  ngOnDestroy() {
    this.engine?.[Symbol.dispose]();
  }

  async onFiles(event: Event) {
    const files = (event.target as HTMLInputElement).files!;
    for (const file of Array.from(files)) {
      await this.engine.indexFile(file);
    }
  }

  onQuery(event: Event) {
    const q = (event.target as HTMLInputElement).value;
    this.hits = this.engine.search(q, { windowed: true });
  }
}