Marc Rejohn Castillano 5cb6561924 added ruflo

2026-04-09 19:01:53 +08:00

11 KiB

Raw Permalink Blame History

Issue: npm install fails with @xenova/transformers version conflict

Problem Description

Users report installation failure when running:

npm i -g claude-flow@alpha

Error:

npm error code ETARGET
npm error notarget No matching version found for @xenova/transformers@^3.2.0.
npm error notarget In most cases you or one of your dependencies are requesting
npm error notarget a package version that doesn't exist.

Root Cause Analysis

Version Mismatch

Requested version: @xenova/transformers@^3.2.0 (doesn't exist)
Latest available version: 2.17.2
Our package.json: Correctly specifies ^2.17.2

Source of Conflict - IDENTIFIED! ✅

The problem is in claude-flow@alpha package:

User runs: npm i -g claude-flow@alpha

Dependency chain:
└── claude-flow@2.7.28 (alpha tag)
    └── agentic-flow@^1.8.10
        ├── @xenova/transformers@^2.17.2 ✅ (correct)
        └── agentdb@^1.4.3
            └── NO @xenova/transformers ❌ (missing!)

The issue:

claude-flow@alpha (v2.7.28) depends on agentic-flow@^1.8.10
agentic-flow@1.8.10 depends on agentdb@^1.4.3
agentdb@1.4.3 does NOT have @xenova/transformers in dependencies
But agentdb@1.5.0+ DOES include @xenova/transformers@^2.17.2

Why the error occurs:

User installs claude-flow@alpha globally
npm resolves to agentic-flow@1.8.10 → agentdb@1.4.3
agentdb@1.4.3 is OLD and missing transformers dependency
Somewhere in the dependency resolution, npm gets confused about version requirements
Reports non-existent ^3.2.0 (likely npm registry corruption or cache issue)

Published versions:

✅ agentdb@1.4.3 - No transformers (OLD)
✅ agentdb@1.5.0+ - Has transformers@^2.17.2 (FIXED)
✅ agentdb@1.6.1 - Has transformers@^2.17.2 (CURRENT)
✅ agentic-flow@1.8.10 - Has transformers@^2.17.2 (used by claude-flow@alpha)
✅ agentic-flow@1.9.1 - Has transformers@^2.17.2 (LATEST, not published as alpha)

Current Usage in Codebase

Files using @xenova/transformers:

packages/agentdb/src/controllers/EmbeddingService.ts:32 - Dynamic import
agentic-flow/src/agentdb/controllers/EmbeddingService.ts:32 - Dynamic import
agentic-flow/src/router/providers/onnx.ts:33 - Dynamic import
agentic-flow/src/reasoningbank/utils/embeddings.ts:6 - Static import

Key observation: Most imports are dynamic (await import()) except embeddings.ts which uses static import.

Impact

Severity: HIGH - Blocks all global installations of claude-flow@alpha
Affected package: claude-flow@alpha (v2.7.28)
Dependency chain: claude-flow@alpha → agentic-flow@1.8.10 → agentdb@1.4.3 (missing transformers)
User impact: Cannot install claude-flow@alpha globally
Root package: This repository (agentic-flow) needs to coordinate fix with claude-flow
Workaround: Update claude-flow@alpha to use agentic-flow@^1.9.0 which depends on newer agentdb

Proposed Solution

IMMEDIATE FIX: Update claude-flow@alpha dependency (REQUIRED)

The claude-flow package needs to be updated:

// claude-flow/package.json
{
  "dependencies": {
    "agentic-flow": "^1.9.0"  // Change from ^1.8.10 to ^1.9.0
  }
}

This ensures it pulls agentdb@1.6.x which has the transformers dependency.

Action required:

Update claude-flow repository to bump agentic-flow to ^1.9.0
Publish new claude-flow@alpha version
Test global installation: npm i -g claude-flow@alpha

Strategy 1: Make @xenova/transformers Optional (LONG-TERM, RECOMMENDED)

Convert @xenova/transformers to an optional peer dependency with lazy loading:

Benefits:

✅ Users only install ONNX libraries when actually needed
✅ Reduces package size (23MB+ model downloads)
✅ Avoids dependency conflicts
✅ Faster installation for users not using embeddings
✅ Graceful fallback to hash-based embeddings

Implementation:

1. Update package.json dependencies

packages/agentdb/package.json:

{
  "dependencies": {
    "@modelcontextprotocol/sdk": "^1.20.1",
    "chalk": "^5.3.0",
    "commander": "^12.1.0",
    "hnswlib-node": "^3.0.0",
    "sql.js": "^1.13.0",
    "zod": "^3.25.76"
  },
  "optionalDependencies": {
    "better-sqlite3": "^11.8.1",
    "@xenova/transformers": "^2.17.2"
  },
  "peerDependenciesMeta": {
    "@xenova/transformers": {
      "optional": true
    }
  }
}

agentic-flow/package.json:

{
  "dependencies": {
    "@anthropic-ai/claude-agent-sdk": "^0.1.5",
    "@anthropic-ai/sdk": "^0.65.0",
    "agentdb": "^1.4.3",
    // ... other deps
  },
  "optionalDependencies": {
    "@xenova/transformers": "^2.17.2"
  },
  "peerDependenciesMeta": {
    "@xenova/transformers": {
      "optional": true
    }
  }
}

2. Update static import to dynamic

agentic-flow/src/reasoningbank/utils/embeddings.ts:

// BEFORE (line 6):
import { pipeline, env } from '@xenova/transformers';

// AFTER:
let transformersModule: any = null;

async function loadTransformers() {
  if (!transformersModule) {
    try {
      transformersModule = await import('@xenova/transformers');
      return transformersModule;
    } catch (error) {
      console.warn('[Embeddings] @xenova/transformers not installed. Using hash-based fallback.');
      console.warn('[Embeddings] For semantic embeddings: npm install @xenova/transformers');
      return null;
    }
  }
  return transformersModule;
}

// Update line 11-12:
async function configureTransformers() {
  const transformers = await loadTransformers();
  if (transformers?.env) {
    transformers.env.backends.onnx.wasm.proxy = false;
    transformers.env.backends.onnx.wasm.numThreads = 1;
  }
}

// Update initializeEmbeddings() function:
async function initializeEmbeddings(): Promise<void> {
  // ... existing npx detection code ...

  const transformers = await loadTransformers();
  if (!transformers) {
    console.log('[Embeddings] Transformers not available - using hash-based embeddings');
    isInitializing = false;
    return;
  }

  try {
    embeddingPipeline = await transformers.pipeline(
      'feature-extraction',
      'Xenova/all-MiniLM-L6-v2',
      { quantized: true }
    );
    console.log('[Embeddings] Local model ready! (384 dimensions)');
  } catch (error: any) {
    console.error('[Embeddings] Failed to initialize:', error?.message || error);
    console.warn('[Embeddings] Falling back to hash-based embeddings');
  } finally {
    isInitializing = false;
  }
}

3. Clean up lock files

# Remove lock files that may have stale dependencies
rm -f packages/agentdb/package-lock.json
rm -f agentic-flow/package-lock.json
rm -f package-lock.json

# Regenerate with correct versions
npm install

4. Add installation documentation

README.md addition:

## Optional Dependencies

### Semantic Embeddings (@xenova/transformers)

For semantic similarity search with ReasoningBank and AgentDB, install:

```bash
npm install @xenova/transformers

Without this package: Hash-based embeddings are used (fast, deterministic, no semantic meaning) With this package: True semantic embeddings (slower, requires 23MB model download, better accuracy)

The package will work without @xenova/transformers installed, using a fast hash-based fallback.


### Strategy 2: Fix Transitive Dependency (Alternative)

**Investigation steps:**
```bash
# Find which dependency requires 3.2.0
npm ls @xenova/transformers
npm explain @xenova/transformers

# Check for outdated dependencies
npm outdated

If found, update the problematic dependency:

npm update <problematic-package>

Strategy 3: Use npm overrides (Quick Fix)

package.json:

{
  "overrides": {
    "@xenova/transformers": "^2.17.2"
  }
}

This forces all dependencies to use version 2.17.2.

Recommended Implementation Order

✅ Immediate: Add npm overrides to force correct version (Strategy 3)
✅ Next release: Implement optional dependencies (Strategy 1)
✅ Investigation: Identify and update problematic transitive dependency (Strategy 2)

Testing Plan

Unit Tests

✅ Test EmbeddingService with transformers installed
✅ Test EmbeddingService without transformers (fallback)
✅ Test ReasoningBank embeddings with/without transformers
✅ Verify hash-based fallback accuracy

Integration Tests

# Test global installation
npm uninstall -g claude-flow
npm install -g claude-flow@alpha

# Test with transformers
npm install @xenova/transformers
npx claude-flow reasoningbank test

# Test without transformers
npm uninstall @xenova/transformers
npx claude-flow reasoningbank test

Performance Validation

Measure installation time (with vs without transformers)
Measure embedding generation speed (semantic vs hash)
Memory usage comparison

Migration Guide

For Users

Current behavior:

npm i -g claude-flow@alpha  # FAILS with version error

After fix:

npm i -g claude-flow@alpha  # ✅ Works, uses hash-based embeddings

# Optional: Install for semantic embeddings
npm install -g @xenova/transformers

For Developers

No API changes required. All existing code continues to work:

computeEmbedding() - Automatically uses available method
computeEmbeddingBatch() - Same behavior
Error handling already exists for transformer failures

To Modify:

packages/agentdb/package.json - Move to optionalDependencies
agentic-flow/package.json - Move to optionalDependencies
agentic-flow/src/reasoningbank/utils/embeddings.ts - Convert static import to dynamic

Already Correct (using dynamic imports):

packages/agentdb/src/controllers/EmbeddingService.ts
agentic-flow/src/agentdb/controllers/EmbeddingService.ts
agentic-flow/src/router/providers/onnx.ts

Success Criteria

✅ npm install -g claude-flow@alpha completes successfully
✅ Package works without @xenova/transformers (hash fallback)
✅ Package works with @xenova/transformers (semantic embeddings)
✅ Installation time reduced by ~5 seconds (no 23MB download)
✅ Package size reduced by ~50MB (ONNX models optional)
✅ All existing tests pass
✅ Documentation updated with optional dependency info

Timeline

Investigation: 30 minutes (identify transitive dependency)
Implementation: 2 hours (package.json changes, dynamic imports, tests)
Testing: 1 hour (installation, functionality, performance)
Documentation: 30 minutes (README, migration guide)
Total: ~4 hours

Priority: HIGH

This blocks all global installations and should be fixed immediately.

Labels: bug, dependencies, installation, high-priority Milestone: v1.9.2 (patch release) Assignee: @ruvnet

11 KiB Raw Permalink Blame History