Marc Rejohn Castillano 5cb6561924 added ruflo

2026-04-09 19:01:53 +08:00

17 KiB

Raw Blame History

@claude-flow/memory

High-performance memory module for Claude Flow V3 - AgentDB unification, HNSW indexing, vector search, self-learning knowledge graph, and hybrid SQLite+AgentDB backend (ADR-009).

Features

150x-12,500x Faster Search - HNSW (Hierarchical Navigable Small World) vector index for ultra-fast similarity search
Hybrid Backend - SQLite for structured data + AgentDB for vectors (ADR-009)
Auto Memory Bridge - Bidirectional sync between Claude Code auto memory and AgentDB (ADR-048)
Self-Learning - LearningBridge connects insights to SONA/ReasoningBank neural pipeline (ADR-049)
Knowledge Graph - PageRank + label propagation community detection over memory entries (ADR-049)
Agent-Scoped Memory - 3-scope agent memory (project/local/user) with cross-agent knowledge transfer (ADR-049)
Vector Quantization - Binary, scalar, and product quantization for 4-32x memory reduction
Multiple Distance Metrics - Cosine, Euclidean, dot product, and Manhattan distance
Query Builder - Fluent API for building complex memory queries
Cache Manager - LRU caching with configurable size and TTL
Migration Tools - Seamless migration from V2 memory systems

Installation

npm install @claude-flow/memory

Quick Start

import { HNSWIndex, AgentDBAdapter, CacheManager } from '@claude-flow/memory';

// Create HNSW index for vector search
const index = new HNSWIndex({
  dimensions: 1536,  // OpenAI embedding size
  M: 16,             // Max connections per node
  efConstruction: 200,
  metric: 'cosine'
});

// Add vectors
await index.addPoint('memory-1', new Float32Array(embedding));
await index.addPoint('memory-2', new Float32Array(embedding2));

// Search for similar vectors
const results = await index.search(queryVector, 10);
// [{ id: 'memory-1', distance: 0.05 }, { id: 'memory-2', distance: 0.12 }]

API Reference

HNSW Index

import { HNSWIndex } from '@claude-flow/memory';

const index = new HNSWIndex({
  dimensions: 1536,
  M: 16,                    // Max connections per layer
  efConstruction: 200,      // Construction-time search depth
  maxElements: 1000000,     // Max vectors
  metric: 'cosine',         // 'cosine' | 'euclidean' | 'dot' | 'manhattan'
  quantization: {           // Optional quantization
    type: 'scalar',         // 'binary' | 'scalar' | 'product'
    bits: 8
  }
});

// Add vectors
await index.addPoint(id: string, vector: Float32Array);

// Search
const results = await index.search(
  query: Float32Array,
  k: number,
  ef?: number  // Search-time depth (higher = more accurate)
);

// Search with filters
const filtered = await index.searchWithFilters(
  query,
  k,
  (id) => id.startsWith('session-')
);

// Remove vectors
await index.removePoint(id);

// Get statistics
const stats = index.getStats();
// { vectorCount, memoryUsage, avgSearchTime, compressionRatio }

AgentDB Adapter

import { AgentDBAdapter } from '@claude-flow/memory';

const adapter = new AgentDBAdapter({
  dimension: 1536,
  indexType: 'hnsw',
  metric: 'cosine',
  hnswM: 16,
  hnswEfConstruction: 200,
  enableCache: true,
  cacheSizeMb: 256
});

await adapter.initialize();

// Store memory
await adapter.store({
  id: 'mem-123',
  content: 'User prefers dark mode',
  embedding: vector,
  metadata: { type: 'preference', agentId: 'agent-1' }
});

// Semantic search
const memories = await adapter.search(queryVector, {
  limit: 10,
  threshold: 0.7,
  filter: { type: 'preference' }
});

// Cross-agent memory sharing
await adapter.enableCrossAgentSharing({
  shareTypes: ['patterns', 'preferences'],
  excludeTypes: ['secrets']
});

Cache Manager

import { CacheManager } from '@claude-flow/memory';

const cache = new CacheManager({
  maxSize: 1000,
  ttlMs: 3600000,  // 1 hour
  strategy: 'lru'
});

// Cache operations
cache.set('key', value);
const value = cache.get('key');
const exists = cache.has('key');
cache.delete('key');
cache.clear();

// Statistics
const stats = cache.getStats();
// { size, hits, misses, hitRate }

Query Builder

import { QueryBuilder } from '@claude-flow/memory';

const results = await new QueryBuilder()
  .semantic(queryVector)
  .where('agentId', '=', 'agent-1')
  .where('type', 'in', ['pattern', 'strategy'])
  .where('createdAt', '>', Date.now() - 86400000)
  .orderBy('relevance', 'desc')
  .limit(20)
  .execute();

Migration

import { MemoryMigration } from '@claude-flow/memory';

const migration = new MemoryMigration({
  source: './data/v2-memory.db',
  destination: './data/v3-memory.db'
});

// Dry run
const preview = await migration.preview();
console.log(`Will migrate ${preview.recordCount} records`);

// Execute migration
await migration.execute({
  batchSize: 1000,
  onProgress: (progress) => console.log(`${progress.percent}%`)
});

Quantization Options

// Binary quantization (32x compression)
const binaryIndex = new HNSWIndex({
  dimensions: 1536,
  quantization: { type: 'binary' }
});

// Scalar quantization (4x compression)
const scalarIndex = new HNSWIndex({
  dimensions: 1536,
  quantization: { type: 'scalar', bits: 8 }
});

// Product quantization (8x compression)
const productIndex = new HNSWIndex({
  dimensions: 1536,
  quantization: { type: 'product', subquantizers: 8 }
});

Auto Memory Bridge (ADR-048)

Bidirectional sync between Claude Code's auto memory files and AgentDB. Auto memory is a persistent directory (~/.claude/projects/<project>/memory/) where Claude writes learnings as markdown. MEMORY.md (first 200 lines) is loaded into the system prompt; topic files are read on demand.

Quick Start

import { AutoMemoryBridge } from '@claude-flow/memory';

const bridge = new AutoMemoryBridge(memoryBackend, {
  workingDir: '/workspaces/my-project',
  syncMode: 'on-session-end', // 'on-write' | 'on-session-end' | 'periodic'
  pruneStrategy: 'confidence-weighted', // 'confidence-weighted' | 'fifo' | 'lru'
});

// Record an insight (stores in AgentDB + optionally writes to files)
await bridge.recordInsight({
  category: 'debugging',
  summary: 'HNSW index requires initialization before search',
  source: 'agent:tester',
  confidence: 0.95,
});

// Sync buffered insights to auto memory files
const syncResult = await bridge.syncToAutoMemory();

// Import existing auto memory files into AgentDB (on session start)
const importResult = await bridge.importFromAutoMemory();

// Curate MEMORY.md index (stays under 200-line limit)
await bridge.curateIndex();

// Check status
const status = bridge.getStatus();

Sync Modes

Mode	Behavior
`on-write`	Writes to files immediately on `recordInsight()`
`on-session-end`	Buffers insights, flushes on `syncToAutoMemory()`
`periodic`	Auto-syncs on a configurable interval

Insight Categories

Category	Topic File	Description
`project-patterns`	`patterns.md`	Code patterns and conventions
`debugging`	`debugging.md`	Bug fixes and debugging insights
`architecture`	`architecture.md`	Design decisions and module relationships
`performance`	`performance.md`	Benchmarks and optimization results
`security`	`security.md`	Security findings and CVE notes
`preferences`	`preferences.md`	User and project preferences
`swarm-results`	`swarm-results.md`	Multi-agent swarm outcomes

Key Optimizations

Batch import - bulkInsert() instead of individual store() calls
Pre-fetched hashes - Single query for content-hash dedup during import
Async I/O - node:fs/promises for non-blocking writes
Exact dedup - hasSummaryLine() uses bullet-prefix matching, not substring
O(1) sync tracking - syncedInsightKeys Set prevents double-write race
Prune-before-build - Avoids O(n^2) index rebuild loop

Utility Functions

import {
  resolveAutoMemoryDir,  // Derive auto memory path from working dir
  findGitRoot,           // Walk up to find .git root
  parseMarkdownEntries,  // Parse ## headings into structured entries
  extractSummaries,      // Extract bullet summaries, strip metadata
  formatInsightLine,     // Format insight as markdown bullet
  hashContent,           // SHA-256 truncated to 16 hex chars
  pruneTopicFile,        // Keep topic files under line limit
  hasSummaryLine,        // Exact bullet-prefix dedup check
} from '@claude-flow/memory';

Types

import type {
  AutoMemoryBridgeConfig,
  MemoryInsight,
  InsightCategory,
  SyncDirection,
  SyncMode,
  PruneStrategy,
  SyncResult,
  ImportResult,
} from '@claude-flow/memory';

Self-Learning Bridge (ADR-049)

Connects insights to the @claude-flow/neural learning pipeline. When neural is unavailable, all operations degrade to no-ops.

Quick Start

import { AutoMemoryBridge, LearningBridge } from '@claude-flow/memory';

const bridge = new AutoMemoryBridge(backend, {
  workingDir: '/workspaces/my-project',
  learning: {
    sonaMode: 'balanced',
    confidenceDecayRate: 0.005,   // Per-hour decay
    accessBoostAmount: 0.03,      // Boost per access
    consolidationThreshold: 10,   // Min insights before consolidation
  },
});

// Insights now trigger learning trajectories automatically
await bridge.recordInsight({
  category: 'debugging',
  summary: 'Connection pool exhaustion on high load',
  source: 'agent:tester',
  confidence: 0.9,
});

// Consolidation runs JUDGE/DISTILL/CONSOLIDATE pipeline
await bridge.syncToAutoMemory(); // Calls consolidate() first

Standalone Usage

import { LearningBridge } from '@claude-flow/memory';

const lb = new LearningBridge(backend, {
  // Optional: inject neural loader for custom setups
  neuralLoader: async () => {
    const { NeuralLearningSystem } = await import('@claude-flow/neural');
    return new NeuralLearningSystem();
  },
});

// Boost confidence when insight is accessed
await lb.onInsightAccessed('entry-123'); // +0.03 confidence

// Apply time-based decay
const decayed = await lb.decayConfidences('default'); // -0.005/hour

// Find similar patterns via ReasoningBank
const patterns = await lb.findSimilarPatterns('connection pooling');

// Get learning statistics
const stats = lb.getStats();
// { totalTrajectories, activeTrajectories, completedTrajectories,
//   totalConsolidations, accessBoosts, ... }

Confidence Lifecycle

Event	Effect	Range
Insight recorded	Initial confidence from source	0.1 - 1.0
Insight accessed	+0.03 per access	Capped at 1.0
Time decay	-0.005 per hour since last access	Floored at 0.1
Consolidation	Neural pipeline may adjust	0.1 - 1.0

Knowledge Graph (ADR-049)

Pure TypeScript knowledge graph with PageRank and community detection. No external graph libraries required.

Quick Start

import { AutoMemoryBridge, MemoryGraph } from '@claude-flow/memory';

const bridge = new AutoMemoryBridge(backend, {
  workingDir: '/workspaces/my-project',
  graph: {
    similarityThreshold: 0.8,
    pageRankDamping: 0.85,
    maxNodes: 5000,
  },
});

// Graph builds automatically on import
await bridge.importFromAutoMemory();

// Curation uses PageRank to prioritize influential insights
await bridge.curateIndex();

Standalone Usage

import { MemoryGraph } from '@claude-flow/memory';

const graph = new MemoryGraph({
  pageRankDamping: 0.85,
  pageRankIterations: 50,
  pageRankConvergence: 1e-6,
  maxNodes: 5000,
});

// Build from backend entries
await graph.buildFromBackend(backend, 'my-namespace');

// Or build manually
graph.addNode(entry);
graph.addEdge('entry-1', 'entry-2', 'reference', 1.0);
graph.addEdge('entry-1', 'entry-3', 'similar', 0.9);

// Compute PageRank (power iteration)
const ranks = graph.computePageRank();

// Detect communities (label propagation)
const communities = graph.detectCommunities();

// Graph-aware ranking: blend vector score + PageRank
const ranked = graph.rankWithGraph(searchResults, 0.7);
// alpha=0.7 means 70% vector score + 30% PageRank

// Get most influential insights for MEMORY.md
const topNodes = graph.getTopNodes(20);

// BFS traversal for related insights
const neighbors = graph.getNeighbors('entry-1', 2); // depth=2

Edge Types

Type	Source	Description
`reference`	`MemoryEntry.references`	Explicit cross-references between entries
`similar`	HNSW search	Auto-created when similarity > threshold
`temporal`	Timestamps	Entries created in same time window
`co-accessed`	Access patterns	Entries frequently accessed together
`causal`	Learning pipeline	Cause-effect relationships

Performance

Operation	Result	Target
Graph build (1k nodes)	2.78 ms	<200 ms
PageRank (1k nodes)	12.21 ms	<100 ms
Community detection (1k)	19.62 ms	—
`rankWithGraph(10)`	0.006 ms	—
`getTopNodes(20)`	0.308 ms	—
`getNeighbors(d=2)`	0.005 ms	—

Agent-Scoped Memory (ADR-049)

Maps Claude Code's 3-scope agent memory directories for per-agent knowledge isolation and cross-agent transfer.

Quick Start

import { createAgentBridge, transferKnowledge } from '@claude-flow/memory';

// Create a bridge for a specific agent scope
const agentBridge = createAgentBridge(backend, {
  agentName: 'my-coder',
  scope: 'project', // 'project' | 'local' | 'user'
  workingDir: '/workspaces/my-project',
});

// Record insights scoped to this agent
await agentBridge.recordInsight({
  category: 'debugging',
  summary: 'Use connection pooling for DB calls',
  source: 'agent:my-coder',
  confidence: 0.95,
});

// Transfer high-confidence insights between agents
const result = await transferKnowledge(sourceBackend, targetBridge, {
  sourceNamespace: 'learnings',
  minConfidence: 0.8,   // Only transfer confident insights
  maxEntries: 20,
  categories: ['debugging', 'architecture'],
});
// { transferred: 15, skipped: 5 }

Scope Paths

Scope	Directory	Use Case
`project`	`<gitRoot>/.claude/agent-memory/<agent>/`	Project-specific learnings
`local`	`<gitRoot>/.claude/agent-memory-local/<agent>/`	Machine-local data
`user`	`~/.claude/agent-memory/<agent>/`	Cross-project user knowledge

Utilities

import {
  resolveAgentMemoryDir,  // Get scope directory path
  createAgentBridge,       // Create scoped AutoMemoryBridge
  transferKnowledge,       // Cross-agent knowledge sharing
  listAgentScopes,         // Discover existing agent scopes
} from '@claude-flow/memory';

// Resolve path for an agent scope
const dir = resolveAgentMemoryDir('my-agent', 'project');
// → /workspaces/my-project/.claude/agent-memory/my-agent/

// List all agent scopes in a directory
const scopes = await listAgentScopes('/workspaces/my-project');
// [{ agentName: 'coder', scope: 'project', path: '...' }, ...]

Performance Benchmarks

Operation	V2 Performance	V3 Performance	Improvement
Vector Search	150ms	<1ms	150x
Bulk Insert	500ms	5ms	100x
Memory Write	50ms	<5ms	10x
Cache Hit	5ms	<0.1ms	50x
Index Build	10s	800ms	12.5x

ADR-049 Benchmarks

Operation	Actual	Target	Headroom
Graph build (1k nodes)	2.78 ms	<200 ms	71.9x
PageRank (1k nodes)	12.21 ms	<100 ms	8.2x
Insight recording	0.12 ms/each	<5 ms/each	41.0x
Consolidation	0.26 ms	<500 ms	1,955x
Confidence decay (1k)	0.23 ms	<50 ms	215x
Knowledge transfer	1.25 ms	<100 ms	80.0x

TypeScript Types

import type {
  // Core
  HNSWConfig, HNSWStats, SearchResult, MemoryEntry,
  QuantizationConfig, DistanceMetric,

  // Auto Memory Bridge (ADR-048)
  AutoMemoryBridgeConfig, MemoryInsight, InsightCategory,
  SyncDirection, SyncMode, PruneStrategy,
  SyncResult, ImportResult,

  // Learning Bridge (ADR-049)
  LearningBridgeConfig, LearningStats,
  ConsolidateResult, PatternMatch,

  // Knowledge Graph (ADR-049)
  MemoryGraphConfig, GraphNode, GraphEdge,
  GraphStats, RankedResult, EdgeType,

  // Agent Scope (ADR-049)
  AgentMemoryScope, AgentScopedConfig,
  TransferOptions, TransferResult,
} from '@claude-flow/memory';

Dependencies

agentdb - Vector database engine
better-sqlite3 - SQLite driver (native)
sql.js - SQLite driver (WASM fallback)
@claude-flow/neural - Optional peer dependency for self-learning (graceful fallback when unavailable)

@claude-flow/neural - Neural learning integration (SONA, ReasoningBank, EWC++)
@claude-flow/shared - Shared types and utilities
@claude-flow/hooks - Session lifecycle hooks for auto memory sync

License

MIT

17 KiB Raw Blame History

@claude-flow/memory

Features

Installation

Quick Start

API Reference

HNSW Index

AgentDB Adapter

Cache Manager

Query Builder

Migration

Quantization Options

Auto Memory Bridge (ADR-048)

Quick Start

Sync Modes

Insight Categories

Key Optimizations

Utility Functions

Types

Self-Learning Bridge (ADR-049)

Quick Start

Standalone Usage

Confidence Lifecycle

Knowledge Graph (ADR-049)

Quick Start

Standalone Usage

Edge Types

Performance

Agent-Scoped Memory (ADR-049)

Quick Start

Scope Paths

Utilities

Performance Benchmarks

ADR-049 Benchmarks

TypeScript Types

Dependencies

Related Packages

License

17 KiB

Raw Blame History