Overview
Albex is a full-text document search engine shipped as a 33 KB WebAssembly binary. It runs entirely in the browser: streaming parsers turn eleven file formats into an index, a four-stage pipeline answers queries in under 5 ms, and nothing the user drops ever leaves the machine — no upload, no telemetry, no network call after the initial .wasm fetch.
Albex exposes one primary class — AlbexEngine — that wraps the WebAssembly module and orchestrates indexing, search, persistence, and the adaptive runtime. Additional opt-in helpers (AlbexEngineWorker, AlbexPool, TieredStore, BloomGpu) are exported from subpath entry points so consumers only pay for what they use.
Install, import, use. The WASM binary travels with the npm package and your bundler (Vite, Webpack 5+, Next, esbuild, Rollup, Parcel 2, Bun, Deno) resolves it automatically through import.meta.url. No assets to copy. No paths to remember.
33 KB
main wasm (baseline)
~50 KB
total bundle gzipped (no PDF)
11
supported formats
0
runtime dependencies
How it works — three steps, no server
01 · Drop
The file enters the sandbox. No network, no disk — the sandbox boundary is the trust boundary.
02 · Index
Streaming parsers feed a static BSS pool. Zero-copy through a 64 KB scratchpad.
03 · Search
Bloom skips, Bitap finishes. O(N log k) with a min-heap top-K.
Features
Zero backend
Entire engine lives in a 33 KB WASM binary. No server, no cloud, no network call after load.
Fuzzy & typo-tolerant
Bitap (Shift-Or / Wu-Manber) up to 3 edits. Finds "clausula" even when typed "claúsula".
Accent-insensitive
Latin-1 + Latin-A fold. ES / FR / DE / IT / PT / PL / CZ / TR transparently.
Query DSL
Phrase, OR, fuzzy, mixed. "a b" | c syntax. Up to 4 tokens per simple query.
11 formats
DOCX · XLSX · PDF · HTML · MD · JSON · CSV · EML · RTF · TXT · XML. Heavy ones stream; the lite ones (CSV, EML, RTF) handle BOM, base64, cp1252.
Zero allocator
Static BSS region only. No heap fragmentation, no GC pressure, no OOM mid-search.
Adapts to the host
3 tier binaries (mini / std / pro) × SIMD variants. Auto-picked from deviceMemory and WASM probes.
Cooperative search
searchCooperative() with frameBudgetMs yields to scheduler.yield() between slices. UI thread keeps a chance to paint.
Worker pool
AlbexPool shards documents across N workers. Map-reduce search merges global top-K.
WebGPU pre-filter
WGSL compute shader runs Bloom in parallel for large corpora. Experimental; opt-in via gpu auto-select.
Tiered storage
TieredStore evicts cold docs to OPFS, promotes on demand. Search archives that exceed RAM.
Snapshot v2
Per-document content hash persisted. Save the index to OPFS in milliseconds; reload across sessions with dedup intact.
OCR companion
@albex/ocr drops Tesseract.js next to the engine. Six languages auto-loaded by demand. Scanned PDFs become searchable.
Hybrid PDF mode
Opt-in: native PDFs get their embedded images OCR'd too. For contracts with scanned signatures, reports with screenshots.
The OCR companion
@albex/ocr wraps Tesseract.js as an opt-in companion. Scanned PDFs and images become searchable. The core stays 33 KB — Tesseract loads only the first time you call enableOcr(engine), then cached forever in IndexedDB. Six languages auto-loaded by demand. Still 100 % local, zero backend.
33 KB
core stays untouched
+ 3.5 MB
Tesseract.js (lazy on first OCR call)
~1.8 MB
per language model (cached forever)
0 bytes
initial cost until enableOcr() fires
What's new in v0.3.0 (2026-05-30)
Scanned-PDF OCR
Image-only PDFs are now searchable through the optional @albex/ocr companion. Tesseract.js, lazy by language, zero cost when not enabled.
Hybrid PDF mode
Opt-in alwaysExtractEmbeddedImages OCRs embedded images on top of vector text. For contracts with scanned signatures, reports with screenshot tables.
Snapshot v2
Per-document content hash now persisted. Content-hash dedup survives the save/load round trip. v1 snapshots still load.
Parser-crash recovery
When pdf-extract traps on an unusual PDF, the engine falls back to lopdf-only image extraction. Many "unsupported" PDFs become searchable through OCR.
Hardened lite parsers
CSV strips the UTF-8 BOM. EML decodes base64 and quoted-printable bodies, walks nested multipart. RTF reads \'XX (cp1252) and \uN ? Unicode escapes.
Honest API rename
searchStream → searchCooperative. The old name implied incremental streaming the method never did. Deprecated alias preserved until 0.4.0.
Install
No backend required. No data leaves the browser. One package, one command:
install · shellnpm install albex
# or
pnpm add albex
# or
deno add npm:albex
To add the optional OCR companion for scanned PDFs and images:
install with ocr · shellnpm i albex @albex/ocr
npm entry points
Albex ships multiple subpath exports so each feature can be tree-shaken independently. Import only what you use.
| Import path | Surface | Pulls in |
|---|---|---|
albex | AlbexEngine, types, errors, profile helpers | main engine code, profile detector, persistence layer, resource manager |
albex/worker | AlbexEngineWorker | main-thread wrapper that proxies to a Worker |
albex/worker-runtime | (runs inside a Worker) | Worker-side handler; reference via new URL(..., import.meta.url) |
albex/pool | AlbexPool, AlbexPoolOptions | pool coordinator that orchestrates N worker shards |
albex/tiered | TieredStore, TieredStoreOptions | hot/warm tier manager with OPFS persistence of original blobs |
albex/gpu | BloomGpu, packBloomsFromChunks | standalone WebGPU runtime + WGSL shader for Bloom scan |
mini/std/pro picked from navigator.deviceMemory) serve the variants yourself and pass wasmBaseUrl. See the architecture section for the rationale.Quick start
Up in under a minute:
index.ts · typescriptimport { AlbexEngine } from "albex";
const engine = await AlbexEngine.create();
for (const file of input.files) {
await engine.indexFile(file);
}
const hits = engine.search('"clausula novena" | rescisión');
for (const h of hits) {
console.log(h.documentName, h.location, h.score, h.snippet);
}
Or the explicit constructor form, with options:
explicit lifecycle · typescriptimport { AlbexEngine } from "albex";
const engine = new AlbexEngine();
await engine.init();
await engine.indexFile(myFile);
const results = engine.search('contrato', { windowed: true });
Query syntax
One small query language, five patterns:
| Pattern | Meaning | Example match |
|---|---|---|
word | Single fuzzy token — up to 3 character edits (auto-adjusted by query length) | The word in context |
a b c | AND: all tokens must appear in the same chunk (proximity scored) | found a then b then c |
"a b" | Phrase: tokens must appear in order and adjacent | matched a b exactly |
a | b | OR: union of two independent searches, merged by score | has a or b |
"a b" | c | Mix phrase and OR | has a b or just c |
Supported formats
What goes in — eleven formats, each with a parser tuned to how the format behaves in the wild:
| Extension | Format | How it is parsed |
|---|---|---|
.docx | Word document | Native Rust/WASM streaming XML parser (word/document.xml); paragraph + table extraction |
.xlsx | Excel workbook | Native Rust/WASM parser; shared strings + inline strings streaming |
.pdf | PDF document | Separate albex_pdf.wasm module (~1 MB), loaded lazily on the first PDF |
.html .htm | HTML | <script>/<style> stripped, paragraphs at block-level boundaries |
.md .markdown | Markdown | CommonMark markers stripped (code fences, headings, links, lists) |
.json | JSON | Recursive walk; every string key and leaf indexed |
.csv | CSV | RFC 4180 lite; one row per chunk (location = row number) |
.eml | Email (MIME) | MIME-lite: From/To/Subject + first text/plain body part |
.rtf | RTF | Control words and groups stripped, text runs preserved |
.txt | Plain text | Direct UTF-8 pass-through, split on double newlines |
.xml | XML | Tag-stripped, entity-decoded |
API reference
Constructor
new AlbexEngine(opts: AlbexOptions) → AlbexEngine
Constructs the engine. The WASM module is NOT loaded yet — call init() before any other method. The constructor only validates options and stores them.
import { AlbexEngine } from "albex";
// Zero config — the WASM binary ships with the package and your bundler
// (Vite, Webpack 5+, Next, esbuild, Rollup, Parcel 2, Bun) resolves it
// automatically through `import.meta.url`. No assets to copy, no URL to
// configure, no path to remember.
const engine = new AlbexEngine();
await engine.init();
// Optional overrides — only if you want tier auto-selection or a CDN.
// new AlbexEngine({ wasmBaseUrl: "/assets" }) // serve the 6 variants yourself
// new AlbexEngine({ wasmUrl: "https://cdn.example.com/albex_wasm.wasm" })
AlbexOptions
| Field | Type | Description |
|---|---|---|
wasmUrl? | string | Explicit URL to the .wasm binary. Overrides every other option. Useful when serving from a custom CDN. |
wasmBaseUrl? | string | Base directory containing tier variants (albex_wasm_<tier>[_simd].wasm). Required only if you want runtime tier auto-selection. Leave undefined to fall back to the bundled std-baseline binary. |
pdfWasmUrl? | string | Override for albex_pdf.wasm. By default the bundled module is resolved via import.meta.url and loaded lazily on first PDF. |
tier? | 'auto'|'mini'|'std'|'pro' | Capacity tier. auto requires wasmBaseUrl because the bundler cannot know which of the 6 binaries to copy. Without wasmBaseUrl the engine loads std by default. |
simd? | 'auto'|'on'|'off' | SIMD variant policy. Only effective when wasmBaseUrl is set. |
gpu? | 'auto'|'on'|'off' | WebGPU pre-filter policy. Default: auto (enabled when corpus > gpuThreshold). |
gpuThreshold? | number | Minimum chunk count to engage WebGPU. Default: 20 000. |
Lifecycle
engine.init() → Promise<void>
Resolves the WASM URL (using wasmUrl or wasmBaseUrl + tier auto-detection), fetches the binary, instantiates it, runs initial setup, and subscribes the engine to the global ResourceManager. Throws AlbexInitError on fetch or instantiation failure.
engine.reset() → void
Clears every indexed document and search result. The engine is immediately ready to index a fresh corpus. The WASM module instance is preserved (no re-fetch).
engine[Symbol.dispose]() → void
TC39 explicit-resource-management hook. Resets state, unsubscribes from the resource manager, destroys the GPU device (if any), and nulls out internal references so the WASM instance becomes unreachable for GC. Use with using engine = new AlbexEngine(...) when available.
await engine.init();
// engine.tier → 'mini' | 'std' | 'pro'
// engine.simdEnabled → boolean
// engine.gpuEngaged → true after the first search that uses WebGPU
Indexing
engine.indexFile(file: File) → Promise<IndexedDocument>
Detects the format from the extension, parses the file, and streams text into the WASM index. Content is hashed (FNV-1a 64-bit) before indexing — if a document with the same hash already exists, the previous entry is returned and no work is done. Throws AlbexUnsupportedFormatError or AlbexParseError on failures. Supported: .docx, .xlsx, .pdf, .md, .html/.htm, .json, .csv, .eml, .rtf, .txt, .xml.
const input = document.querySelector('input[type=file]');
input.addEventListener('change', async () => {
for (const file of input.files) {
const doc = await engine.indexFile(file);
console.log(`${file.name}: ${doc.chunks} chunks, hash=${doc.contentHash}`);
}
});
// Idempotent: re-indexing the same file is a no-op and returns the existing
// IndexedDocument (matched by FNV-1a content hash).
IndexedDocument
| Field | Type | Description |
|---|---|---|
name | string | Original file name from the File object. |
ext | string | Lowercase extension without leading dot. |
chunks | number | Number of chunks produced from this document. |
indexTimeMs | number | Wall-clock time spent indexing. |
textBytes | number | Bytes of indexed text contributed by this document. |
docId | number | Stable identifier within the engine. Persists across compact(). |
contentHash | string | 64-bit FNV-1a hex hash of the source bytes. Used for dedup. |
Search
engine.search(query: string, opts?: SearchOptions) → SearchResult[]
Synchronous full-corpus search. Drives the pipeline: parse query → Bloom filter → Bitap fuzzy match → rich scoring → min-heap top-K. Results are returned already sorted by score, descending. May invoke the WebGPU pre-filter automatically when opts.gpu permits and chunk count exceeds gpuThreshold.
engine.searchCooperative(query: string, opts?: SearchOptions) → AsyncIterable<SearchResult>
Cooperative variant of search(). The corpus is processed in slices; between slices the engine yields to the browser scheduler via scheduler.yield() (or requestAnimationFrame fallback). UI stays responsive during 50 ms+ scans. Use this in any interactive search box.
// Synchronous fast path — best for small corpora and tests.
const results = engine.search('"contrato marco" | rescisión', { windowed: true });
for (const r of results) {
console.log(`[${r.score}] ${r.documentName} · ${r.snippet}`);
}
searchCooperative · typescript
// Cooperative streaming search — main thread stays at 60 fps.
for await (const r of engine.searchCooperative('contrato', { frameBudgetMs: 8 })) {
renderResult(r); // render incrementally as results arrive
}
SearchOptions
| Field | Type | Description |
|---|---|---|
windowed? | boolean | Return cropped snippets with ASCII ellipsis markers instead of full chunks. |
before? | number | Bytes of context before the match (default 60). |
after? | number | Bytes of context after the match (default 120). |
frameBudgetMs? | number | Slice duration for searchCooperative() before yielding to the scheduler. Default 8 ms. |
SearchResult
| Field | Type | Description |
|---|---|---|
documentName | string | File name as registered by indexFile(). |
location | number | Paragraph index (DOCX/TXT/MD/…) or page number (PDF, 1-based). |
score | number | Composite relevance score 0–1000 (higher is better). |
snippet | string | Chunk text, optionally windowed with "... " / " ..." sentinels. |
matchStart | number | Byte offset of the primary token start in snippet. |
matchEnd | number | Byte offset of the primary token end (exclusive). |
matches | MatchSpan[] | All matched token spans in query order. Length 1–4. |
Tuning knobs
engine.setMaxErrors(n: 0 | 1 | 2 | 3) → void
Maximum edit distance for fuzzy match. 0 = exact only. Engine auto-shrinks for short queries. engine.setMaxErrors(1);
engine.setThreshold(n: number) → void
Minimum score (0–1000) below which results are dropped. Default 250. engine.setThreshold(400);
engine.setMaxResults(n: number) → void
Cap on number of returned results. 1–200. Default 50. engine.setMaxResults(100);
engine.setLanguage(lang: 'off' | 'es') → void
Enable lightweight Spanish stemming on query tokens. Indexed text is never stemmed, so snippets stay faithful to the source. engine.setLanguage('es'); // "contratos" now matches "contrato"
Incremental updates — remove · replace · compact
Indexing is idempotent (content-hash dedup is automatic). Documents can be removed individually without rebuilding the entire index. Reclaim storage on demand with compact().
engine.removeDocument(idOrName: string) → boolean
Tombstone a document. Subsequent searches skip its chunks. Storage is reclaimed by compact(). id accepts the file name or contentHash returned by indexFile.
engine.replaceDocument(name: string, newFile: File) → Promise<IndexedDocument>
Atomic remove + re-index. Bypasses dedup so re-indexing the same bytes after a remove works. await engine.replaceDocument('contract.pdf', newVersionFile);
engine.compact() → void
Reclaim storage from tombstoned documents. Rewrites internal arrays in place. Doc IDs of survivors are preserved.
engine.removeDocument('contract-2024-03.pdf');
engine.removeDocument(doc.contentHash); // by hash also works
engine.removeDocument('old.pdf');
engine.compact(); // bytes freed; subsequent indexFile sees real headroom
Persistence — snapshots on OPFS & IndexedDB
The full engine state can be serialised to a binary blob and restored later. OPFS is preferred (zero-copy writes); IndexedDB is the universal fallback. A 16 MB snapshot typically restores in tens of milliseconds — far faster than re-parsing the documents.
engine.save(name: string) → Promise<void>
Serialise the index to a binary snapshot in OPFS (preferred) or IndexedDB. await engine.save('my-corpus');
engine.load(name: string) → Promise<boolean>
Restore a previously saved snapshot. Returns true on success, false if missing or header mismatched.
engine.loadOrInit(name: string) → Promise<boolean>
Convenience wrapper: load if it exists, otherwise reset() and start clean. await engine.loadOrInit('my-corpus');
engine.deleteSnapshot(name: string) → Promise<void>
Remove a snapshot from storage. await engine.deleteSnapshot('old-corpus');
engine.listSnapshots() → Promise<string[]>
List the names of all snapshots saved in the current origin. const names = await engine.listSnapshots();
if (!await engine.load('my-corpus')) {
console.log('No snapshot yet; starting fresh');
}
Introspection — getStats() & getLastSearchStats()
engine.getStats() → EngineStats
Snapshot of current engine state: document count, chunk count, memory usage, loaded tier, capacity caps.
engine.getLastSearchStats() → SearchStats | null
Bloom/Bitap pipeline counters from the most recent search call. Useful for debugging relevance and detecting performance regressions.
EngineStats
| Field | Type | Description |
|---|---|---|
documents | number | Active (non-tombstoned) document count. |
chunks | number | Total indexed chunks. |
textUsed | number | Bytes of indexed text currently stored. |
textCapacity | number | Maximum text bytes the loaded tier can hold. |
wasmMemoryBytes | number | Total WASM linear memory size (BSS + grown pages). |
tier | 'mini'|'std'|'pro'|null | Tier of the loaded binary. null before init(). |
maxChunks | number | Compile-time chunk capacity for the loaded tier. |
maxDocs | number | Compile-time document capacity for the loaded tier. |
SearchStats
| Field | Type | Description |
|---|---|---|
query | string | Verbatim query string. |
timeMs | number | End-to-end search time. |
results | number | Number of results above the threshold. |
bloomTested | number | Chunks tested against the Bloom filter. |
bloomPassed | number | Chunks that passed Bloom (subset of tested). |
bitapMatched | number | Chunks confirmed by Bitap (subset of passed). |
Adaptive runtime — profile detection & helpers
The library exposes pure helpers from the same package so consumers can read the device profile, override tier selection, or build their own UI around resource state. None of them require an initialised engine.
detectProfile(opts?: { fresh?: boolean }) → Promise<DeviceProfile>
Probe the host capabilities (cores, memory, WASM features, WebGPU, storage budget, network, battery). Result cached in sessionStorage. Pass fresh: true to bypass the cache.
pickTier(profile: DeviceProfile) → 'mini' | 'std' | 'pro'
Pure heuristic: <=1 GB → mini, 2–4 GB → std, >=8 GB → pro, null (Safari) → std.
pickWorkerCount(profile: DeviceProfile) → number
cores/2 clamped [1, 8]. Falls to 1 when battery is reported below 20 % and discharging.
shouldUseGpu(profile: DeviceProfile, chunkCount: number, threshold?: number) → boolean
true when WebGPU is available AND chunk count crosses the threshold (default 20 000).
import { detectProfile } from 'albex';
const profile = await detectProfile();
console.log(profile.memoryGB, profile.wasm.simd);
const tier = pickTier(profile);
const workers = pickWorkerCount(profile);
if (shouldUseGpu(profile, engine.getStats().chunks)) { /* … */ }
DeviceProfile
| Field | Type | Description |
|---|---|---|
cores | number | navigator.hardwareConcurrency. |
memoryGB | number | null | navigator.deviceMemory (capped at 8 GB by spec; null on Safari). |
wasm.simd | boolean | WebAssembly v128 supported (validated via probe module). |
wasm.bulkMemory | boolean | Bulk memory ops supported. |
wasm.threads | boolean | Threads supported AND page cross-origin isolated. |
webgpu | boolean | navigator.gpu present. |
coopCoep | boolean | crossOriginIsolated === true. |
storage | { quotaBytes, usageBytes } | navigator.storage.estimate() result. |
net | { effectiveType, saveData } | Connection info if reported (Chrome only). |
battery | { level, charging } | null | Battery state if available. |
visible | boolean | document.visibilityState at probe time. |
Off-main-thread — AlbexEngineWorker
new AlbexEngineWorker(opts: AlbexWorkerOptions) → AlbexEngineWorker
Drop-in replacement for AlbexEngine that runs the entire engine inside a Web Worker. Surface is identical except every method returns a Promise. Files are transferred via postMessage with transferable ArrayBuffers — no copy. Import from albex/worker. The runtime script is exported separately as albex/worker-runtime and must be referenced via new URL(..., import.meta.url).
import { AlbexEngineWorker } from 'albex/worker';
// Same zero-config pattern as the main engine — the worker runtime URL is
// the only thing you must point at (so the bundler can spawn it).
const engine = new AlbexEngineWorker({
workerUrl: new URL('albex/worker-runtime', import.meta.url),
});
await engine.init();
const results = await engine.search('contrato', { windowed: true });
Parallelisation — AlbexPool
new AlbexPool(opts: AlbexPoolOptions) → AlbexPool
Orchestrates N worker shards. Documents are sharded round-robin across workers. Searches broadcast to every shard and the coordinator merges top-K results preserving the global descending score order. Default worker count is half of hardwareConcurrency clamped to [1, 8]; battery-aware (drops to 1 on low battery). Import from albex/pool.
import { AlbexPool } from 'albex/pool';
const pool = new AlbexPool({
workerUrl: new URL('albex/worker-runtime', import.meta.url),
workers: 'auto', // cores/2 clamped [1, 8]
});
await pool.init();
await pool.indexFile(fileA); // sharded round-robin
await pool.indexFile(fileB);
const results = await pool.search('contrato'); // map-reduce across shards
Large corpora — TieredStore
new TieredStore(engine: AlbexEngine, opts?: TieredStoreOptions) → TieredStore
Adds hot/warm tiers behind the engine. When the engine reaches evictThreshold of capacity, the LRU document is removed and its original blob remains in OPFS. promote(name) brings it back by re-indexing from the persisted blob. Import from albex/tiered.
import { AlbexEngine, TieredStore } from 'albex';
const engine = new AlbexEngine();
await engine.init();
const store = new TieredStore(engine, { evictThreshold: 0.85, hotFloor: 4 });
await store.init();
await store.indexFile(file); // persists original blob in OPFS
await store.promote('older-doc.pdf'); // brings warm doc back to engine
GPU acceleration — BloomGpu (advanced)
new BloomGpu() → BloomGpu
Standalone WebGPU Bloom-scan accelerator. AlbexEngine instantiates one automatically when opts.gpu permits and the corpus exceeds gpuThreshold (default 20 000 chunks). You only need to touch this class directly to integrate the WGSL shader into a non-Albex pipeline. Import from albex/gpu.
Typed error hierarchy
Every error thrown by Albex extends AlbexError. Switch on the kind discriminator (which survives structuredClone across worker boundaries) or use instanceof against the subclasses.
| Class | kind | Thrown when |
|---|---|---|
AlbexError | (base) | Base class for every Albex error. Carries kind discriminator. |
AlbexInitError | init | WASM fetch failed, init() not called, or PDF module not initialised. |
AlbexUnsupportedFormatError | unsupported_format | File extension is not in the supported list. Carries ext field. |
AlbexParseError | parse | A parser (DOCX/XLSX/PDF/JSON/…) failed. Carries format field. |
AlbexCapacityError | capacity | Scratchpad write would exceed buffer size, or hard cap reached. |
import {
AlbexError, AlbexInitError, AlbexParseError,
AlbexUnsupportedFormatError, AlbexCapacityError,
} from 'albex';
try {
await engine.indexFile(file);
} catch (e) {
if (e instanceof AlbexUnsupportedFormatError) {
console.warn(`Unsupported: .${e.ext}`);
} else if (e instanceof AlbexParseError) {
console.warn(`Failed to parse ${e.format}:`, e.message);
} else if (e instanceof AlbexCapacityError) {
console.warn('Engine full — upgrade tier or use TieredStore');
} else throw e;
}
Architecture
Albex is a full-text search engine that lives entirely in the user's browser. That sentence is the thesis. Every other choice — the algorithms, the memory model, the absence of an allocator — is a corollary of it. Four axioms follow:
- Zero server. No back-end ever. Documents never transit the network.
- Human latency. Searches must answer within ~100 ms — the perception threshold for "instant".
- Reduced footprint. Bytes shipped to the browser are a tax on every page load.
- Adapts to the machine. The same package runs on a Chromebook and a Mac Pro, picking what each can support.
The layer map
The codebase is split into six layers. Dependencies flow strictly upward: a layer never calls down into the one below it. This is what lets the algorithm layer be testable without WASM, and the adaptive layer be evolved without touching Rust.
0 · Pure algorithms
Rust no_std · core/ — Bloom filter, Bitap matcher, light Spanish stemmer, Unicode fold tables. No heap, no allocator, no I/O. Testable on host without WASM.
1 · Streaming parsers
Rust no_std · ingest/ — XML byte state machines for DOCX and XLSX. Same constraints as layer 0. New Rust formats live here.
2 · WASM shell
Rust · wasm/ — BSS arrays, C ABI exports, EngineState bookkeeping, resumable search, tier feature flags. Pegs layers 0 and 1 to a JS-facing interface.
3 · PDF module (separate)
Rust + std · pdf-wasm/ — pdf-extract wrapper. Its own WASM binary (~1 MB) loaded lazily on first PDF — never paid by users who do not touch PDFs.
4 · TypeScript orchestrator
TS · src/ — AlbexEngine API, format indexers in TS, query parser, typed errors, persistence layer, content hashing.
5 · Adaptive capabilities
TS · src/{profile,resource-manager,pool/,gpu/,tiered-store}.ts — Profile detection, resource awareness, worker pool, WebGPU runtime, tiered storage. Strictly opt-in; the base engine works without any of it.
The search pipeline — from query to ranked results in four stages
01 · Parse query
Detect simple / phrase / OR. Up to 4 tokens. Optional stemming. Cost: < 5 µs.
02 · Bloom filter
AND + compare per chunk against the pattern Bloom. Cost: 2 instructions / chunk.
03 · Bitap fuzzy
Bit-parallel match on the chunks that passed Bloom. Cost: ~10 ns / chunk.
04 · Score + top-K
Rich scoring (accuracy + WB + TF + position + proximity + IDF) into a min-heap. Cost: O(log K).
The cost ladder is deliberate: the cheapest filter runs first on every chunk; the expensive ones only on what survives. A typical search tests 100 000 chunks but runs the Bitap matcher on under 1 % of them. The Bloom filter is the gatekeeper that keeps total search latency under 10 ms even on big corpora.
The pipeline itself is resumable: searchBegin sets up token state, searchSlice(N) processes N chunks and returns done=0/1. That lets searchCooperative yield to the scheduler between slices (frame budgeting) without blocking a paint. It honours an optional GPU candidate mask and a tombstone bitset, then funnels survivors into scoring and the top-K heap.
Bloom filter — 64-bit probabilistic gate
A Bloom filter answers "could the pattern be here?" in two machine instructions: an AND and a compare. We use a single u64 per chunk, with each character mapped to one of 64 buckets by c & 0x3F. A chunk passes iff (chunkBloom & queryBloom) == queryBloom.
The hash function is intentionally trivial. A cryptographic hash would give better distribution but cost 10–100× more CPU. The Bloom runs on every chunk in the corpus — sometimes 100 000 times per query — so raw speed wins over perfect distribution.
For typical European text the filter rejects 80–95 % of chunks before the more expensive Bitap stage. False positives are unavoidable but harmless: Bitap discards them at no extra cost.
Bitap — bit-parallel fuzzy match
The Shift-Or / Wu-Manber algorithm keeps the matching state inside a single u64 register. Each text byte advances the state with a shift + OR + AND — no backtracking, no per-character allocations, no branches dependent on the data.
We extend it to k errors (substitution, insertion, deletion) by maintaining k+1 parallel registers. Albex caps k at 3: more produces noise on short queries and multiplies the linear cost. Pattern length is capped at 64 bytes — the width of the register. Longer queries are truncated with a truncated flag the host can surface in its UI. The error budget is adaptive: tokens of ≤5 characters match exactly, ≤8 allow 1 error, longer allow up to 3.
Scoring and top-K
Each survivor gets a capped sum (0–1000) of components: base accuracy (−200 per average edit), word-boundary bonus, term frequency, document position, phrase proximity (close + in order), and an approximate IDF from per-Bloom-bit document frequency. Weights are hand-tuned.
Results stream into a fixed-size min-heap: insert is O(log K), and when full a new hit only displaces the current worst. A final heap-pop yields the K best in descending order — O(k log k), versus the O(k²) selection sort an earlier version used (27× faster than insertion sort in practice).
Accent folding and stemming
fold_utf8_char maps each code point to a lowercase ASCII base: Latin-1 Supplement and Latin Extended-A (Polish, Czech, Turkish…) collapse to a/c/e/n/o/s/u/z. Accented search is free — acción and accion fold to the same bytes. The same fold runs at index and query time.
The optional Spanish stemmer is a handful of high-impact Snowball-Spanish suffix rules, inlined (~150 bytes vs the full 6 KB state machine). Applied only to query tokens, never to indexed text, so snippets stay faithful — recall improves because the query reduces to a shared prefix that still matches the original word.
Memory model — BSS-only, no allocator
The default wasm32-unknown-unknown target has no OS to call for memory. Pulling in std would drag in dlmalloc or wee_alloc, inflate the binary, add allocation overhead, and introduce a failure mode (OOM) that is hard to surface cleanly to JavaScript.
Instead, every storage region is a zero-initialised static array — TEXT_POOL (16 MB), CHUNKS, NAME_POOL, CHUNK_SIG. Because the BSS segment is described in the binary but not stored, a .wasm with 16 MB of static arrays still weighs ~33 KB on disk. At runtime the WebAssembly linear memory grows to accommodate the layout (~20 MB for std tier), but this grow happens once at instantiation — no per-call allocation, no fragmentation, no OOM during a search.
The trade-off: capacity is a fixed ceiling. Exceeding it is a silent stop, not an error. That is acceptable because the user already chose a tier; a "give me more space" feature would mean an allocator, which would mean fragmentation and unpredictable latency.
The JS ↔ WASM bridge
The entire JS↔WASM data plane is one 64 KB shared byte buffer — the scratchpad. The dance is always: getBuffer(size) → a pointer; JS writes bytes there; JS calls the action (setPattern(len), feedXmlBytes(len), search()); WASM reads the scratchpad, processes, stores results; for output it writes back and JS reads it. No JSON across the boundary — just offsets and lengths. That is what keeps the boundary cost near zero.
At init() the exports contract is validated before the engine returns: memory is a real WebAssembly.Memory, every required export exists, and the ABI version matches. A mismatched binary fails loudly here instead of crashing later inside indexFile.
Adaptive runtime — same package, different machines
Albex ships six variants of the main WASM binary (three capacity tiers, each with and without SIMD) plus the lazy PDF module. The browser fetches one variant per init() call, picked from the device profile.
| Tier | Max docs | Max chunks | Max text | Rule (deviceMemory) | Working set | Binary |
|---|---|---|---|---|---|---|
mini | 32 | 25 000 | 4 MB | ≤ 1 GB | ~5 MB | 33 KB |
std | 128 | 100 000 | 16 MB | 2–7 GB | ~20 MB | 33 KB |
pro | 1 024 | 800 000 | 128 MB | ≥ 8 GB | ~160 MB | 33 KB |
SIMD sits on top of every tier: the runtime probes for WebAssembly v128 by attempting to validate a tiny module that uses it. If it validates, the engine fetches albex_wasm_<tier>_simd.wasm — a binary built with -C target-feature=+simd128. The inner loop of the Bloom batch is branchless precisely so LLVM can auto-vectorise it.
navigator.deviceMemory spec caps reported memory at 8 GB for privacy. A Mac Pro with 192 GB reports 8 just like an iPad with 16 GB — they cannot be distinguished. The pro tier (128 MB pool) is the largest default because the working set must still fit comfortably in a browser tab — Chrome starts flagging pages as "high memory" around 500 MB. For genuine archive-scale corpora the right answer is the TieredStore (LRU eviction with OPFS-persisted blobs) or the AlbexPool sharded across N workers, not a tier with a 1 GB pool.WebGPU pre-filter — Bloom on the GPU when it pays off
The Bloom check on 100 000 chunks is embarrassingly parallel: each test is independent. A WGSL compute shader can run all 100 000 of them in a single dispatch. The result is a packed bitset of candidates; the CPU then runs Bitap only on those.
Engaging the GPU has a fixed cost (upload + dispatch + readback, ~3–5 ms). Below ~20 000 chunks the CPU wins. Above it, the GPU gives a 5–10× speedup. The default threshold is exactly that break-even point, but it is exposed as gpuThreshold for tuning.
The candidate mask is single-shot: it applies to the next searchBegin and is automatically cleared at the end of the last slice. That prevents stale masks from silently corrupting an unrelated next search if anything ever goes wrong in the JS layer.
PDF and OCR path
The PDF module is a separate WASM binary fetched only on the first PDF — users who never touch PDFs never pay for it. It wraps pdf-extract/lopdf to pull vector text per page. Built with panic=abort, so a malformed PDF traps the instance; the host catches that, discards the poisoned instance, and (if OCR is wired) falls back to extracting embedded page images via lopdf — recovering content pdf-extract could not.
The OCR orchestrator (@albex/ocr, wired via engine.attachOcr(adapter)) manages a pool of Tesseract.js workers and feeds them the JPEG/JPEG2000 image XObjects the PDF module extracts per page. Two modes: scanned-PDF (image-only files) and hybrid (OCR on top of vector text). Best-effort: one failed image never stops the page, one failed page never stops the document. A lightweight script / letter-frequency heuristic guesses the language so the orchestrator loads the right Tesseract traineddata — picking the wrong model is the single biggest hit to OCR accuracy. Tesseract itself (~3.5 MB, LSTM engine compiled to WASM) is loaded lazily via await import("tesseract.js") only when enableOcr() has been called and a scanned image needs recognising.
Rust + TypeScript — when each language wins
Some parsers live in Rust; others live in TypeScript. The rule is not "Rust for important things". It is: which primitive of the host environment wins for this operation?
| Task | Language | Why |
|---|---|---|
| DOCX parsing | Rust | Streaming XML state machine. Files reach hundreds of MB; loading the DOM would be infeasible. |
| XLSX parsing | Rust | Same as DOCX. Shared strings tables get huge in real-world workbooks. |
| PDF extraction | Rust | pdf-extract dependency is mature and large. Worth its own module loaded lazily. |
| Bloom + Bitap hot loop | Rust | Bit-parallel arithmetic. WASM v128 wins clearly over JS bitwise. |
| Markdown / HTML / RTF | TS | Files fit in memory; V8 regex is JIT-compiled C++ — faster than a hand-written WASM parser. |
| JSON parsing | TS | JSON.parse is C++ inside the JS engine — no Rust port can beat it. |
| Content hash FNV-1a | TS | One-shot per file, not a hot path. JS bitwise is fine at 100 MB/s. |
| WebGPU shader | WGSL | GPU language. No alternative. |
| OPFS / IndexedDB | TS | Browser APIs; unreachable from WASM. |
| Worker orchestration | TS | new Worker() is a JS API; postMessage protocol lives where the workers live. |
DOCX, XLSX and PDF go to Rust because the source files can be hundreds of megabytes and streaming a state machine is the only way to handle them. Markdown, HTML, JSON, CSV, EML and RTF go to TypeScript because they fit comfortably in memory and V8 gives us String.prototype.replace with JIT-compiled regex, which beats any hand-written WASM parser for documents of typical size.
MAINTAINER.md) and Technical deep-dive (TECHNICAL.md).Storage & identity
Three primitives keep the engine honest across sessions and across repeated uploads: OPFS for snapshots, IndexedDB as the fallback, and a 64-bit content hash for deduplication. None of them are configurable surface area — they're internal contracts you should understand when reasoning about what the engine actually does.
OPFS & IndexedDB — why both
When you call engine.save("my-corpus"), the engine serialises the entire index (chunks, document table, text pool, Bloom filters) to a single binary blob and writes it to browser storage. engine.load("my-corpus") reads it back and memcpy's it into the BSS arrays. A 16 MB corpus restores in tens of milliseconds — orders of magnitude faster than re-parsing 50 DOCXs.
| API | Available since | Write 16 MB | Why we picked it |
|---|---|---|---|
| OPFS | Chrome 102 · Safari 15.2 · Firefox 111 | ~20 ms | Zero-copy FileSystemWritableFileStream.write(Uint8Array) |
| IndexedDB | All browsers since 2015 | ~80 ms | Structured-clone of the Uint8Array |
The router lives in src/persistence.ts and detects navigator.storage.getDirectory at runtime. If present → OPFS path. Otherwise → IndexedDB. Both paths satisfy the same contract (savePersisted / loadPersisted / deletePersisted / listPersisted), so calling code is identical.
Two distinct uses of OPFS
The persistence backend serialises the engine state: chunks, Bloom filters, doc table. TieredStore uses OPFS differently — it stores the original file blobs, so an evicted document can be promoted back without asking the user to re-pick the file. Two stores, two purposes, same OPFS API.
FNV-1a 64-bit — why this hash
Every call to engine.indexFile(file) hashes the raw bytes before any parsing. If a document with that hash already lives in the index, the engine returns the existing entry and skips the work. This makes indexFile idempotent and safe to call repeatedly — drag the same DOCX twice, the index stays clean.
| Hash | Throughput in JS | Output | Trade-off |
|---|---|---|---|
| FNV-1a 64-bit | ~100 MB/s | 16 hex chars | Non-cryptographic; ~10⁻¹⁵ collision probability at 128 docs. |
| SHA-256 | ~30 MB/s | 64 hex chars | Cryptographic; 3× slower, no value added for our use case. |
| MurmurHash3 | ~150 MB/s | Variable | Slightly faster; less portable across language ecosystems. |
Why non-cryptographic is fine here
We don't need adversarial collision resistance. The threat model for a content hash in a deduplication store is "the user accidentally dragged the same file twice", not "an attacker is crafting two distinct files that hash the same to confuse the index". If an attacker could exploit a deliberate collision, the worst case would be that their second file does not get indexed — there is no path to elevation of privilege, data leak or denial of service.
SHA-256 would cost 3× the throughput for zero added safety in this scenario. We chose throughput. The same logic is used by Git for object identification (well, until SHA-256 transition, but Git's case IS adversarial), by SQLite for ROWID generation, by countless databases for dedup.
Where the hash surfaces
Two places. First, internally during indexFile — the dedup check. Second, on the returned IndexedDocument.contentHash field. The host can pass that hash to engine.removeDocument(hash) as a stable identifier that survives rename: if your user uploads same-contract-renamed.pdf and you want to remove the previously-uploaded contract.pdf that contains the same bytes, the hash matches even though the names don't.
Capacity & numbers
Default limits for the std tier (mini and pro scale ×0.25 and ×8 respectively):
16 MB
text pool · std tier (4 MB mini · 128 MB pro)
100k
chunk capacity · std tier (25k mini · 800k pro)
128 docs
document limit · std tier (32 mini · 1 024 pro)
64 chars
query length — Bitap u64 register width
Performance figures
< 5 ms
typical query, full corpus
80–95 %
chunks rejected by Bloom before Bitap
27×
min-heap top-K vs insertion sort
5–10×
GPU speedup above 20 000 chunks
| Operation | Cost | Notes |
|---|---|---|
| Parse query | < 5 µs | simple / phrase / OR detection, up to 4 tokens |
| Bloom test | 2 instructions / chunk | one AND + one compare per chunk, up to 100 000 per query |
| Bitap match | ~10 ns / chunk | only on chunks that passed Bloom (< 1 % of corpus typically) |
| Score + top-K | O(log K) insert | fixed-size min-heap; final pop O(k log k) |
| GPU engage overhead | ~3–5 ms | upload + dispatch + readback; break-even at ~20 000 chunks |
| Snapshot write, 16 MB (OPFS) | ~20 ms | zero-copy FileSystemWritableFileStream |
| Snapshot write, 16 MB (IndexedDB) | ~80 ms | structured-clone of the Uint8Array |
| Content hash (FNV-1a, JS) | ~100 MB/s | one-shot per file, not a hot path |
| Snapshot restore, 16 MB | tens of ms | memcpy into BSS arrays — orders of magnitude faster than re-parsing |
Guides — recipes by use case
From install to advanced setups: cooperative search, worker pool, big-corpus tiering, and integration in React / Angular. Every recipe starts with npm install albex.
ESM / TypeScript
index + search · typescriptimport { AlbexEngine } from "albex";
// 1. Construct + init. The WASM ships with the package and your bundler
// resolves it through import.meta.url — Vite, Webpack 5+, Next, esbuild,
// Rollup, Parcel 2, Bun all handle this automatically. No assets to copy.
const engine = new AlbexEngine();
await engine.init();
// 2. Index files from drag-and-drop or <input type="file">.
const input = document.querySelector("input[type=file]");
input.addEventListener("change", async () => {
for (const file of input.files) {
const doc = await engine.indexFile(file);
console.log(`Indexed ${doc.name}: ${doc.chunks} chunks`);
}
});
// 3. Search.
const results = engine.search('"contrato marco" | rescisión', { windowed: true });
for (const r of results) {
console.log(`[${r.score}] ${r.documentName} — ${r.snippet}`);
}
advanced · typescript
// Tweak relevance.
engine.setMaxErrors(1); // tighter fuzziness
engine.setThreshold(400); // only return strong hits
engine.setMaxResults(100);
engine.setLanguage("es"); // light Spanish stemming on queries
// Inspect what was loaded.
console.log(engine.tier); // 'std' by default
console.log(engine.simdEnabled); // boolean
console.log(engine.gpuEngaged); // true once first search uses WebGPU
// Want tier auto-selection? Serve the 6 binaries yourself and pass wasmBaseUrl.
// The engine then picks mini/std/pro based on navigator.deviceMemory.
// new AlbexEngine({ wasmBaseUrl: '/assets', tier: 'auto', simd: 'auto' })
Cooperative search
cooperative · typescript// Cooperative search — yields to the scheduler between slices so the UI
// thread keeps a chance to paint during long scans. NOTE: results arrive
// in one batch after the search completes; the async iterator exists so
// callers can break early. True incremental streaming is on the backlog.
for await (const r of engine.searchCooperative(query, { frameBudgetMs: 8 })) {
renderResultCard(r);
if (results.length >= 50) break; // stop early — engine cleans up
}
search box with debounce · typescript
// Wire it into a search input with debounce.
let abortToken = 0;
input.addEventListener("input", async () => {
const token = ++abortToken;
const query = input.value.trim();
resultsEl.innerHTML = "";
if (!query) return;
for await (const r of engine.searchCooperative(query, { frameBudgetMs: 8 })) {
if (token !== abortToken) break; // newer query started — stop
appendResult(r);
}
});
Persistence
save · typescript// First visit: index the user's documents normally.
for (const file of files) {
await engine.indexFile(file);
}
// Save a snapshot to OPFS (preferred) or IndexedDB (fallback).
await engine.save("my-corpus");
restore · typescript
// On subsequent visits: load instantly instead of re-indexing.
const restored = await engine.loadOrInit("my-corpus");
if (restored) {
console.log(`Restored ${engine.getStats().documents} documents`);
} else {
console.log("No snapshot yet, starting fresh");
}
// List all snapshots in this origin.
const names = await engine.listSnapshots();
// Delete one explicitly.
await engine.deleteSnapshot("old-corpus");
Web Worker
worker · typescript// Run the engine in a Web Worker. Same surface, every call returns a
// Promise. Main thread never blocks on indexing or search.
import { AlbexEngineWorker } from "albex/worker";
// Only the worker runtime URL is required so the bundler spawns it.
// The main WASM is resolved automatically inside the worker.
const engine = new AlbexEngineWorker({
workerUrl: new URL("albex/worker-runtime", import.meta.url),
});
await engine.init();
await engine.indexFile(file);
const results = await engine.search("contrato", { windowed: true });
worker · advanced · typescript
// Streaming search works through the Worker too.
for await (const r of engine.searchCooperative("contrato", { frameBudgetMs: 8 })) {
render(r);
}
// Dispose when done (TC39 Symbol.dispose).
engine[Symbol.dispose]();
Worker pool
pool · typescript// AlbexPool shards documents across N workers. Searches map-reduce.
// Best for large corpora on multi-core machines.
import { AlbexPool } from "albex/pool";
const pool = new AlbexPool({
workerUrl: new URL("albex/worker-runtime", import.meta.url),
workers: "auto", // = cores/2, clamped [1, 8]
});
await pool.init();
console.log("Shards:", pool.workerCount);
// Documents are sharded round-robin.
for (const file of files) {
await pool.indexFile(file);
}
// Search broadcasts to every shard; coordinator merges top-K.
const results = await pool.search("contrato", { windowed: true });
pool · advanced · typescript
// Aggregate stats across shards.
const stats = await pool.getStats();
console.log(`${stats.documents} docs across ${pool.workerCount} shards`);
// Remove documents — the pool finds the shard that owns it.
await pool.removeDocument("old-contract.docx");
// Compact every shard at once.
await pool.compact();
// Stream results from a pooled search.
for await (const r of pool.searchCooperative("contrato")) {
render(r);
}
Big corpora — TieredStore
tiered · typescript// TieredStore keeps hot documents in the engine and warm ones in
// OPFS. When the engine fills past evictThreshold, the LRU document
// is removed from RAM but its original blob stays persisted.
import { AlbexEngine, TieredStore } from "albex";
const engine = new AlbexEngine();
await engine.init();
const store = new TieredStore(engine, {
evictThreshold: 0.85, // start evicting when 85 % full
hotFloor: 4, // never evict the last 4 docs
});
await store.init();
for (const file of files) {
await store.indexFile(file); // persists blob + adds to engine
}
tiered · advanced · typescript
// Bring a warm document back into the engine on demand.
const promoted = await store.promote("old-contract.pdf");
if (promoted) {
// Now searchable in the engine again.
engine.search("clausula");
}
// Inspect the tier balance.
const stats = store.getTierStats();
console.log(`${stats.hot} hot · ${stats.warm} warm · ${stats.totalBytes} bytes`);
// Forget a document entirely (engine + OPFS).
await store.forget("expired.pdf");
React
DocSearch.jsx · reactimport { useEffect, useRef, useState } from "react";
import { AlbexEngine } from "albex";
export function DocSearch() {
const engineRef = useRef(null);
const [hits, setHits] = useState([]);
const [ready, setReady] = useState(false);
useEffect(() => {
const e = new AlbexEngine();
e.init().then(() => {
engineRef.current = e;
setReady(true);
});
return () => engineRef.current?.[Symbol.dispose]();
}, []);
async function handleFile(ev) {
for (const file of ev.target.files) {
await engineRef.current.indexFile(file);
}
}
function handleSearch(q) {
setHits(engineRef.current?.search(q, { windowed: true }) ?? []);
}
if (!ready) return <p>Loading…</p>;
return (
<div>
<input type="file" multiple onChange={handleFile} />
<input onChange={e => handleSearch(e.target.value)} />
{hits.map(h => (
<div key={h.documentName + h.location}>
<b>{h.documentName}</b> — {h.snippet}
</div>
))}
</div>
);
}
Angular
search.component.ts · angularimport { Component, OnInit, OnDestroy } from "@angular/core";
import { AlbexEngine } from "albex";
@Component({
selector: "app-search",
template: `
<input type="file" multiple (change)="onFiles($event)" />
<input (input)="onQuery($event)" placeholder="Search…" />
<div *ngFor="let h of hits">
<b>{{ h.documentName }}</b> — {{ h.snippet }}
</div>
`,
})
export class SearchComponent implements OnInit, OnDestroy {
private engine!: AlbexEngine;
hits: any[] = [];
async ngOnInit() {
this.engine = new AlbexEngine();
await this.engine.init();
}
ngOnDestroy() {
this.engine?.[Symbol.dispose]();
}
async onFiles(event: Event) {
const files = (event.target as HTMLInputElement).files!;
for (const file of Array.from(files)) {
await this.engine.indexFile(file);
}
}
onQuery(event: Event) {
const q = (event.target as HTMLInputElement).value;
this.hits = this.engine.search(q, { windowed: true });
}
}