LavinHash

01Overview

LavinHash is a high-performance fuzzy hashing library that generates compact fingerprints for file similarity detection. Unlike cryptographic hashes that change completely with any modification, fuzzy hashes allow computing similarity scores between files.

A cryptographic hash (MD5, SHA-256) is deliberately brittle: flip one bit of the input and the digest changes beyond recognition. A fuzzy hash is deliberately elastic: nearby inputs produce nearby fingerprints, so two fingerprints can be compared — the result is a percentage, not a yes/no.

Key features

02DLAH: Dual-Layer Adaptive Hashing

The DLAH algorithm analyzes files in two orthogonal dimensions, combining them to produce a robust similarity metric resistant to both structural and content modifications.

Layer 1 — structural fingerprinting (30% weight)

Captures the file's topology using Shannon entropy analysis. This layer detects structural changes like data reorganization, compression changes, or block-level modifications.

1. Divide input into fixed-size blocks (default: 256 bytes)
2. For each block, calculate Shannon entropy: H(X) = −Σ p(x) log₂ p(x)
3. Quantize entropy to 4-bit nibbles (0–15 range)
4. Concatenate nibbles to form the structural vector
5. Compare vectors using Levenshtein distance (edit distance)

Example: 256-byte block → entropy 4.83 → quantized to nibble 12 (0xC)

Layer 2 — content-based hashing (70% weight)

Extracts semantic features using a rolling hash over a sliding window. This layer detects content similarity even when data is moved, inserted, or partially modified.

1. Initialize BuzHash with a 64-byte window
2. Slide the window byte-by-byte, computing the rolling hash
3. When hash ≡ 0 (mod M), extract a feature (adaptive trigger)
4. Insert the feature into an 8192-bit Bloom filter using 3 hash functions
5. Compare Bloom filters using Jaccard similarity: |A ∩ B| / |A ∪ B|

The modulus is selected adaptively, adjusting feature density to file size:

M = min(file_size / 256, 8192)

Combined similarity score

Δ(A, B) = α · S_structural + (1 − α) · S_content

Where:

• α = 0.3 (structural weight, configurable)
• S_structural = normalized Levenshtein similarity
• S_content = Jaccard similarity of Bloom filters
• Result ∈ [0, 100] (percentage similarity)

Reading the score

Every comparison decomposes into two sub-scores plus the weighted combination — similarity = 0.3 × structural + 0.7 × content:

03System architecture

Core library (Rust)

src/
├── lib.rs              # Public API and FFI exports
├── algo/
│   ├── entropy.rs      # Shannon entropy (SIMD optimized)
│   ├── buzhash.rs      # BuzHash rolling hash
│   └── bloom.rs        # Fixed 8192-bit Bloom filter
├── model/
│   └── fingerprint.rs  # Fingerprint struct (repr(C))
└── utils/
    └── mem.rs          # Zero-copy FFI helpers

Binary wire format

Fingerprints serialize to a compact, versioned binary layout:

04Performance characteristics

Optimization techniques

• SIMD entropy calculation: AVX2 intrinsics for parallel processing of 8 blocks
• Rayon parallelization: automatic multi-threading for files > 1 MB
• Cache-friendly design: the Bloom filter fits in L1/L2 cache (1 KB)
• Zero-copy FFI: no memory duplication across language boundaries
• Lazy evaluation: iterator-based processing, minimal allocations

05Getting started

Install

npm install lavinhash

Cargo, Maven, and Go module coordinates for the other bindings are listed per language in guides.

Quick start

Hash a set of files, then fold over the fingerprints to find the closest match — here, detecting malware variants:

// Node.js / TypeScript - Functional Pipeline Approach
import { hash, compare, findDelta } from 'lavinhash';
import { readFile } from 'fs/promises';
import { pipeline } from 'stream/promises';

// Functional composition for file analysis
const analyzeFiles = async (paths: string[]) => {
  const fingerprints = await Promise.all(
    paths.map(async (path) => ({
      path,
      data: new Uint8Array(await readFile(path)),
      fingerprint: null as any
    }))
  );

  return fingerprints.map(({ path, data }) => ({
    path,
    fingerprint: hash(data),
    metadata: { size: data.length, timestamp: Date.now() }
  }));
};

// Compare with functional fold (reduce)
const findMostSimilar = (target: any, candidates: any[]) =>
  candidates.reduce((best, current) => {
    const sim = compare(target.fingerprint, current.fingerprint);
    return sim > best.similarity
      ? { candidate: current, similarity: sim }
      : best;
  }, { candidate: null, similarity: 0 });

// Usage: Detect malware variants
const [target, ...database] = await analyzeFiles([
  'unknown_sample.exe',
  'known_trojan_v1.exe',
  'known_trojan_v2.exe'
]);

const match = findMostSimilar(target, database);
console.log(`Match: ${match.candidate.path} (${match.similarity}%)`);

06API reference

The whole surface is three functions and a configuration builder. Signatures below use the Node.js / TypeScript binding; the Rust core takes an extra optional config argument (hash(&data, Option<&Config>), compare(&a, &b, Option<&Config>)).

Delta fields

07Implementation guides

Language-specific guides for integrating LavinHash into your projects.

Node.js / TypeScript

npm install lavinhash

import { hash, compare, findDelta } from 'lavinhash';
import { readFile } from 'fs/promises';

// Functional approach with pipe composition
const pipe = (...fns) => (x) => fns.reduce((v, f) => f(v), x);

const toUint8Array = (buffer: Buffer) => new Uint8Array(buffer);
const computeHash = (data: Uint8Array) => ({ data, fingerprint: hash(data) });

// Process single file
const processFile = pipe(toUint8Array, computeHash);

// Compare files functionally
const compareFiles = async (path1: string, path2: string) => {
  const [file1, file2] = await Promise.all([
    readFile(path1).then(processFile),
    readFile(path2).then(processFile)
  ]);

  const similarity = compare(file1.fingerprint, file2.fingerprint);
  const delta = findDelta(
    file1.fingerprint, file2.fingerprint,
    file1.data, file2.data
  );

  return {
    similarity,
    changes: { total: delta.totalChanges, added: delta.bytesAdded, deleted: delta.bytesDeleted }
  };
};

// Usage
const result = await compareFiles('v1.bin', 'v2.bin');
console.log(`Similarity: ${result.similarity}%`);
console.log(`Delta: +${result.changes.added} -${result.changes.deleted}`);

Advanced: streaming and clustering

// Advanced: Streaming pipeline with backpressure
import { hash, compare, Config } from 'lavinhash';
import { createReadStream } from 'fs';
import { pipeline } from 'stream/promises';
import { Transform } from 'stream';

// Custom configuration with builder pattern
const createConfig = () =>
  new Config()
    .withAlpha(0.4)
    .withWindowSize(128);

// Stream-based processing for large files
class HashTransform extends Transform {
  constructor(private config: Config) {
    super({ objectMode: true });
  }

  _transform(chunk: { path: string; data: Uint8Array }, _, callback) {
    const fingerprint = hash(chunk.data, this.config);
    callback(null, { ...chunk, fingerprint });
  }
}

// Functional similarity matrix computation
const computeSimilarityMatrix = (fingerprints: any[]) =>
  fingerprints.map((fp1, i) =>
    fingerprints.slice(i + 1).map((fp2) => ({
      pair: [i, i + fingerprints.indexOf(fp2, i + 1)],
      similarity: compare(fp1, fp2)
    }))
  ).flat();

// Find clusters using transitive closure
const findClusters = (matrix: any[], threshold = 80) =>
  matrix
    .filter(({ similarity }) => similarity >= threshold)
    .reduce((clusters, { pair: [a, b] }) => {
      const cluster = clusters.find(c => c.includes(a) || c.includes(b));
      return cluster ? (cluster.push(a, b), clusters) : [...clusters, [a, b]];
    }, [] as number[][]);

// Usage
const files = ['file1.bin', 'file2.bin', 'file3.bin'];
const results = await processFiles(files);
const matrix = computeSimilarityMatrix(results.map(r => r.fingerprint));
const clusters = findClusters(matrix);

Rust

[dependencies]
lavinhash-core = "0.1.0"

use lavinhash_core::{hash, compare};

fn main() {
    // Hash a file
    let data = std::fs::read("file.txt")?;
    let fingerprint = hash(&data, None)?;

    // Compare two files
    let data2 = std::fs::read("file2.txt")?;
    let fp2 = hash(&data2, None)?;

    let similarity = compare(&fingerprint, &fp2, None);
    println!("Similarity: {}%", similarity);
}

Java

<dependency>
    <groupId>com.lavinhash</groupId>
    <artifactId>lavinhash-jni</artifactId>
    <version>0.1.0</version>
</dependency>

import com.lavinhash.*;

public class Example {
    public static void main(String[] args) {
        // Hash a file
        byte[] data = Files.readAllBytes(Paths.get("file.txt"));
        Fingerprint fp = LavinHash.hash(data);

        // Compare two files
        byte[] data2 = Files.readAllBytes(Paths.get("file2.txt"));
        Fingerprint fp2 = LavinHash.hash(data2);

        int similarity = LavinHash.compare(fp, fp2);
        System.out.println("Similarity: " + similarity + "%");
    }
}

Go

go get github.com/lavinhash/lavinhash-go

package main

import (
    "fmt"
    "os"
    lavin "github.com/lavinhash/lavinhash-go"
)

func main() {
    // Hash a file
    data, _ := os.ReadFile("file.txt")
    fp, _ := lavin.Hash(data)

    // Compare two files
    data2, _ := os.ReadFile("file2.txt")
    fp2, _ := lavin.Hash(data2)

    similarity := lavin.Compare(fp, fp2)
    fmt.Printf("Similarity: %d%%\n", similarity)
}

08Production use cases

Worked example — malware variant detection

A complete classifier: hash the unknown sample, compare it against a fingerprint database, and treat scores at or above 70% as a family match.

// Functional malware classification with monadic error handling
import { readFile } from 'fs/promises';
import { hash, compare } from 'lavinhash';

type MalwareFamily = { name: string; fingerprint: any; severity: string };
type ClassificationResult =
  | { type: 'match'; family: string; similarity: number; severity: string }
  | { type: 'unknown'; candidates: Array<{ family: string; similarity: number }> };

// Pure function composition
const classifyMalware = (database: MalwareFamily[]) =>
  (unknownSample: Uint8Array): ClassificationResult => {
    const unknownFP = hash(unknownSample);

    const matches = database
      .map(({ name, fingerprint, severity }) => ({
        family: name,
        similarity: compare(unknownFP, fingerprint),
        severity
      }))
      .sort((a, b) => b.similarity - a.similarity);

    const [best, ...rest] = matches;

    return best.similarity >= 70
      ? { type: 'match', ...best }
      : { type: 'unknown', candidates: matches.slice(0, 3) };
  };

// Database as immutable data structure
const malwareDB: MalwareFamily[] = [
  { name: 'Trojan.Emotet', fingerprint: fp1, severity: 'critical' },
  { name: 'Ransomware.WannaCry', fingerprint: fp2, severity: 'critical' },
  { name: 'Backdoor.Cobalt', fingerprint: fp3, severity: 'high' }
];

// Pipeline with async/await
const analyzeSample = async (path: string) => {
  const data = new Uint8Array(await readFile(path));
  const classifier = classifyMalware(malwareDB);
  return classifier(data);
};

// Usage with pattern matching
const result = await analyzeSample('suspicious.exe');
const message = result.type === 'match'
  ? `⚠️  ${result.family} detected (${result.similarity}%, ${result.severity})`
  : `Unknown sample (top: ${result.candidates[0].family} at ${result.candidates[0].similarity}%)`;
console.log(message);

09Implementation details

Rust core library

use lavinhash_core::{hash, compare, Config};
use std::fs;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Read files
    let data1 = fs::read("file1.bin")?;
    let data2 = fs::read("file2.bin")?;

    // Generate fingerprints
    let fp1 = hash(&data1, None)?;
    let fp2 = hash(&data2, None)?;

    // Compare
    let similarity = compare(&fp1, &fp2, None);
    println!("Similarity: {}%", similarity);

    // Custom configuration
    let config = Config::new()
        .with_alpha(0.4)  // Adjust structural weight
        .with_window_size(128);
    let fp3 = hash(&data1, Some(&config))?;

    Ok(())
}

Memory safety guarantees

• No unsafe code except in the FFI boundary (fully audited)
• All allocations tracked, no memory leaks
• Validated with Miri (Rust interpreter for undefined behavior detection)
• ASAN/MSAN clean on all platforms

Cross-platform determinism

Identical files produce identical fingerprints across all platforms:

• Linux x86_64, ARM64
• Windows x86_64
• macOS x86_64, ARM64 (M1/M2)
• WebAssembly (wasm32)