906 lines
21 KiB
Markdown
906 lines
21 KiB
Markdown
# Requesty.ai Integration - Testing Strategy
|
|
|
|
## Testing Overview
|
|
|
|
This document outlines a comprehensive testing strategy for the Requesty.ai integration, covering unit tests, integration tests, end-to-end scenarios, and validation criteria.
|
|
|
|
## Test Categories
|
|
|
|
| Category | Scope | Duration | Automation |
|
|
|----------|-------|----------|------------|
|
|
| Unit Tests | Proxy functions | 30 min | Jest/Vitest |
|
|
| Integration Tests | CLI → Proxy → API | 60 min | Manual + Script |
|
|
| E2E Tests | Full user workflows | 45 min | Manual |
|
|
| Performance Tests | Latency, throughput | 30 min | Script |
|
|
| Security Tests | API key handling | 15 min | Manual |
|
|
| **Total** | | **3 hours** | |
|
|
|
|
---
|
|
|
|
## 1. Unit Tests
|
|
|
|
### Test File: `src/proxy/anthropic-to-requesty.test.ts`
|
|
|
|
#### 1.1 Proxy Initialization
|
|
|
|
```typescript
|
|
describe('AnthropicToRequestyProxy - Initialization', () => {
|
|
it('should initialize with default configuration', () => {
|
|
const proxy = new AnthropicToRequestyProxy({
|
|
requestyApiKey: 'test-key'
|
|
});
|
|
|
|
expect(proxy.requestyBaseUrl).toBe('https://router.requesty.ai/v1');
|
|
expect(proxy.defaultModel).toBe('openai/gpt-4o-mini');
|
|
});
|
|
|
|
it('should accept custom base URL', () => {
|
|
const proxy = new AnthropicToRequestyProxy({
|
|
requestyApiKey: 'test-key',
|
|
requestyBaseUrl: 'https://custom.requesty.ai/v1'
|
|
});
|
|
|
|
expect(proxy.requestyBaseUrl).toBe('https://custom.requesty.ai/v1');
|
|
});
|
|
|
|
it('should accept custom default model', () => {
|
|
const proxy = new AnthropicToRequestyProxy({
|
|
requestyApiKey: 'test-key',
|
|
defaultModel: 'anthropic/claude-3.5-sonnet'
|
|
});
|
|
|
|
expect(proxy.defaultModel).toBe('anthropic/claude-3.5-sonnet');
|
|
});
|
|
});
|
|
```
|
|
|
|
#### 1.2 Format Conversion (Anthropic → OpenAI)
|
|
|
|
```typescript
|
|
describe('AnthropicToRequestyProxy - Request Conversion', () => {
|
|
let proxy: AnthropicToRequestyProxy;
|
|
|
|
beforeEach(() => {
|
|
proxy = new AnthropicToRequestyProxy({
|
|
requestyApiKey: 'test-key'
|
|
});
|
|
});
|
|
|
|
it('should convert simple message', () => {
|
|
const anthropicReq = {
|
|
model: 'openai/gpt-4o-mini',
|
|
messages: [
|
|
{ role: 'user', content: 'Hello' }
|
|
]
|
|
};
|
|
|
|
const openaiReq = proxy.convertAnthropicToOpenAI(anthropicReq);
|
|
|
|
expect(openaiReq.model).toBe('openai/gpt-4o-mini');
|
|
expect(openaiReq.messages).toHaveLength(2); // system + user
|
|
expect(openaiReq.messages[1].content).toBe('Hello');
|
|
});
|
|
|
|
it('should convert system prompt (string)', () => {
|
|
const anthropicReq = {
|
|
model: 'openai/gpt-4o-mini',
|
|
system: 'You are a helpful assistant',
|
|
messages: [{ role: 'user', content: 'Hi' }]
|
|
};
|
|
|
|
const openaiReq = proxy.convertAnthropicToOpenAI(anthropicReq);
|
|
|
|
expect(openaiReq.messages[0].role).toBe('system');
|
|
expect(openaiReq.messages[0].content).toContain('You are a helpful assistant');
|
|
});
|
|
|
|
it('should convert system prompt (array)', () => {
|
|
const anthropicReq = {
|
|
model: 'openai/gpt-4o-mini',
|
|
system: [
|
|
{ type: 'text', text: 'You are helpful' },
|
|
{ type: 'text', text: 'Be concise' }
|
|
],
|
|
messages: [{ role: 'user', content: 'Hi' }]
|
|
};
|
|
|
|
const openaiReq = proxy.convertAnthropicToOpenAI(anthropicReq);
|
|
|
|
expect(openaiReq.messages[0].content).toContain('You are helpful');
|
|
expect(openaiReq.messages[0].content).toContain('Be concise');
|
|
});
|
|
|
|
it('should convert tools to OpenAI format', () => {
|
|
const anthropicReq = {
|
|
model: 'openai/gpt-4o-mini',
|
|
messages: [{ role: 'user', content: 'Read file' }],
|
|
tools: [{
|
|
name: 'Read',
|
|
description: 'Read a file',
|
|
input_schema: {
|
|
type: 'object',
|
|
properties: {
|
|
file_path: { type: 'string' }
|
|
},
|
|
required: ['file_path']
|
|
}
|
|
}]
|
|
};
|
|
|
|
const openaiReq = proxy.convertAnthropicToOpenAI(anthropicReq);
|
|
|
|
expect(openaiReq.tools).toHaveLength(1);
|
|
expect(openaiReq.tools[0].type).toBe('function');
|
|
expect(openaiReq.tools[0].function.name).toBe('Read');
|
|
expect(openaiReq.tools[0].function.parameters).toEqual(anthropicReq.tools[0].input_schema);
|
|
});
|
|
});
|
|
```
|
|
|
|
#### 1.3 Format Conversion (OpenAI → Anthropic)
|
|
|
|
```typescript
|
|
describe('AnthropicToRequestyProxy - Response Conversion', () => {
|
|
it('should convert text response', () => {
|
|
const openaiRes = {
|
|
id: 'chatcmpl-123',
|
|
model: 'openai/gpt-4o-mini',
|
|
choices: [{
|
|
index: 0,
|
|
message: {
|
|
role: 'assistant',
|
|
content: 'Hello, how can I help?'
|
|
},
|
|
finish_reason: 'stop'
|
|
}],
|
|
usage: {
|
|
prompt_tokens: 10,
|
|
completion_tokens: 20,
|
|
total_tokens: 30
|
|
}
|
|
};
|
|
|
|
const anthropicRes = proxy.convertOpenAIToAnthropic(openaiRes);
|
|
|
|
expect(anthropicRes.id).toBe('chatcmpl-123');
|
|
expect(anthropicRes.role).toBe('assistant');
|
|
expect(anthropicRes.content).toHaveLength(1);
|
|
expect(anthropicRes.content[0].type).toBe('text');
|
|
expect(anthropicRes.content[0].text).toBe('Hello, how can I help?');
|
|
expect(anthropicRes.stop_reason).toBe('end_turn');
|
|
expect(anthropicRes.usage.input_tokens).toBe(10);
|
|
expect(anthropicRes.usage.output_tokens).toBe(20);
|
|
});
|
|
|
|
it('should convert tool_calls response', () => {
|
|
const openaiRes = {
|
|
id: 'chatcmpl-123',
|
|
model: 'openai/gpt-4o-mini',
|
|
choices: [{
|
|
message: {
|
|
role: 'assistant',
|
|
content: null,
|
|
tool_calls: [{
|
|
id: 'call_abc123',
|
|
type: 'function',
|
|
function: {
|
|
name: 'Read',
|
|
arguments: '{"file_path": "/test.txt"}'
|
|
}
|
|
}]
|
|
},
|
|
finish_reason: 'tool_calls'
|
|
}],
|
|
usage: { prompt_tokens: 10, completion_tokens: 20, total_tokens: 30 }
|
|
};
|
|
|
|
const anthropicRes = proxy.convertOpenAIToAnthropic(openaiRes);
|
|
|
|
expect(anthropicRes.content).toHaveLength(1);
|
|
expect(anthropicRes.content[0].type).toBe('tool_use');
|
|
expect(anthropicRes.content[0].id).toBe('call_abc123');
|
|
expect(anthropicRes.content[0].name).toBe('Read');
|
|
expect(anthropicRes.content[0].input).toEqual({ file_path: '/test.txt' });
|
|
expect(anthropicRes.stop_reason).toBe('tool_use');
|
|
});
|
|
|
|
it('should convert mixed content response', () => {
|
|
const openaiRes = {
|
|
id: 'chatcmpl-123',
|
|
model: 'openai/gpt-4o-mini',
|
|
choices: [{
|
|
message: {
|
|
role: 'assistant',
|
|
content: 'Let me read that file',
|
|
tool_calls: [{
|
|
id: 'call_abc123',
|
|
type: 'function',
|
|
function: {
|
|
name: 'Read',
|
|
arguments: '{"file_path": "/test.txt"}'
|
|
}
|
|
}]
|
|
},
|
|
finish_reason: 'tool_calls'
|
|
}],
|
|
usage: { prompt_tokens: 10, completion_tokens: 20, total_tokens: 30 }
|
|
};
|
|
|
|
const anthropicRes = proxy.convertOpenAIToAnthropic(openaiRes);
|
|
|
|
expect(anthropicRes.content).toHaveLength(2); // tool_use + text
|
|
expect(anthropicRes.content[0].type).toBe('tool_use');
|
|
expect(anthropicRes.content[1].type).toBe('text');
|
|
});
|
|
});
|
|
```
|
|
|
|
#### 1.4 Error Handling
|
|
|
|
```typescript
|
|
describe('AnthropicToRequestyProxy - Error Handling', () => {
|
|
it('should handle invalid API key', async () => {
|
|
const proxy = new AnthropicToRequestyProxy({
|
|
requestyApiKey: 'invalid-key'
|
|
});
|
|
|
|
await expect(proxy.handleRequest(validRequest, mockRes))
|
|
.rejects.toThrow('Invalid API key');
|
|
});
|
|
|
|
it('should handle rate limit errors', async () => {
|
|
// Mock 429 response
|
|
global.fetch = jest.fn().mockResolvedValue({
|
|
ok: false,
|
|
status: 429,
|
|
headers: new Map([['Retry-After', '60']]),
|
|
text: async () => 'Rate limit exceeded'
|
|
});
|
|
|
|
await expect(proxy.handleRequest(validRequest, mockRes))
|
|
.rejects.toThrow('Rate limit exceeded');
|
|
});
|
|
|
|
it('should handle model not found', async () => {
|
|
const req = {
|
|
model: 'invalid/model',
|
|
messages: [{ role: 'user', content: 'Test' }]
|
|
};
|
|
|
|
await expect(proxy.handleRequest(req, mockRes))
|
|
.rejects.toThrow('Model not found');
|
|
});
|
|
});
|
|
```
|
|
|
|
### Unit Test Coverage Goals
|
|
|
|
- [ ] Proxy initialization: 100%
|
|
- [ ] Request conversion: 95%
|
|
- [ ] Response conversion: 95%
|
|
- [ ] Tool calling: 100%
|
|
- [ ] Error handling: 90%
|
|
- [ ] **Overall: >90%**
|
|
|
|
---
|
|
|
|
## 2. Integration Tests
|
|
|
|
### 2.1 CLI to Proxy Integration
|
|
|
|
```bash
|
|
#!/bin/bash
|
|
# tests/integration/requesty-cli.sh
|
|
|
|
echo "=== Requesty CLI Integration Tests ==="
|
|
|
|
# Test 1: Basic provider detection
|
|
echo "Test 1: Provider detection via flag"
|
|
npx agentic-flow --agent coder \
|
|
--task "Say hello" \
|
|
--provider requesty \
|
|
--model "openai/gpt-4o-mini" | grep "Requesty"
|
|
|
|
# Test 2: Provider detection via env var
|
|
echo "Test 2: Provider detection via USE_REQUESTY"
|
|
USE_REQUESTY=true npx agentic-flow --agent coder \
|
|
--task "Say hello" | grep "Requesty"
|
|
|
|
# Test 3: API key validation (missing key)
|
|
echo "Test 3: Missing API key error"
|
|
REQUESTY_API_KEY= npx agentic-flow --agent coder \
|
|
--task "Test" \
|
|
--provider requesty 2>&1 | grep "REQUESTY_API_KEY required"
|
|
|
|
# Test 4: Proxy startup
|
|
echo "Test 4: Proxy starts on port 3000"
|
|
npx agentic-flow --agent coder \
|
|
--task "Test" \
|
|
--provider requesty 2>&1 | grep "http://localhost:3000"
|
|
|
|
# Test 5: Model override
|
|
echo "Test 5: Model override works"
|
|
npx agentic-flow --agent coder \
|
|
--task "Test" \
|
|
--provider requesty \
|
|
--model "anthropic/claude-3.5-sonnet" 2>&1 | grep "claude-3.5-sonnet"
|
|
```
|
|
|
|
### 2.2 Proxy to Requesty API Integration
|
|
|
|
```bash
|
|
#!/bin/bash
|
|
# tests/integration/requesty-api.sh
|
|
|
|
echo "=== Requesty API Integration Tests ==="
|
|
|
|
# Test 1: Chat completions endpoint
|
|
curl -X POST http://localhost:3000/v1/messages \
|
|
-H "Content-Type: application/json" \
|
|
-H "x-api-key: test-key" \
|
|
-d '{
|
|
"model": "openai/gpt-4o-mini",
|
|
"messages": [{"role": "user", "content": "Hello"}]
|
|
}'
|
|
|
|
# Test 2: Tool calling
|
|
curl -X POST http://localhost:3000/v1/messages \
|
|
-H "Content-Type: application/json" \
|
|
-H "x-api-key: test-key" \
|
|
-d '{
|
|
"model": "openai/gpt-4o-mini",
|
|
"messages": [{"role": "user", "content": "Read file.txt"}],
|
|
"tools": [{
|
|
"name": "Read",
|
|
"input_schema": {
|
|
"type": "object",
|
|
"properties": {"file_path": {"type": "string"}},
|
|
"required": ["file_path"]
|
|
}
|
|
}]
|
|
}'
|
|
|
|
# Test 3: Health check
|
|
curl http://localhost:3000/health
|
|
```
|
|
|
|
### 2.3 End-to-End User Workflows
|
|
|
|
#### Workflow 1: Simple Code Generation
|
|
|
|
```bash
|
|
npx agentic-flow --agent coder \
|
|
--task "Create a Python function that adds two numbers" \
|
|
--provider requesty \
|
|
--model "openai/gpt-4o-mini"
|
|
```
|
|
|
|
**Expected:**
|
|
- Proxy starts
|
|
- Request sent to Requesty
|
|
- Python function generated
|
|
- Exit code 0
|
|
|
|
#### Workflow 2: File Operations with Tools
|
|
|
|
```bash
|
|
npx agentic-flow --agent coder \
|
|
--task "Create a file hello.py with a hello world function" \
|
|
--provider requesty
|
|
```
|
|
|
|
**Expected:**
|
|
- Tool calling works
|
|
- File created: `hello.py`
|
|
- Function is valid Python
|
|
- Exit code 0
|
|
|
|
#### Workflow 3: Research with Streaming
|
|
|
|
```bash
|
|
npx agentic-flow --agent researcher \
|
|
--task "Explain machine learning in simple terms" \
|
|
--provider requesty \
|
|
--model "anthropic/claude-3.5-sonnet" \
|
|
--stream
|
|
```
|
|
|
|
**Expected:**
|
|
- Streaming output (real-time)
|
|
- Coherent explanation
|
|
- Exit code 0
|
|
|
|
#### Workflow 4: Multi-Step Task
|
|
|
|
```bash
|
|
npx agentic-flow --agent coder \
|
|
--task "Create a REST API with Express.js - include routes for GET /users and POST /users" \
|
|
--provider requesty \
|
|
--model "openai/gpt-4o"
|
|
```
|
|
|
|
**Expected:**
|
|
- Multiple files created
|
|
- Valid Express.js code
|
|
- Includes route handlers
|
|
- Exit code 0
|
|
|
|
### Integration Test Matrix
|
|
|
|
| Test ID | Component | Input | Expected Output | Status |
|
|
|---------|-----------|-------|-----------------|--------|
|
|
| INT-01 | CLI Detection | `--provider requesty` | Proxy starts | [ ] |
|
|
| INT-02 | CLI Detection | `USE_REQUESTY=true` | Proxy starts | [ ] |
|
|
| INT-03 | CLI Validation | Missing API key | Error message | [ ] |
|
|
| INT-04 | Proxy Startup | Start proxy | Port 3000 listening | [ ] |
|
|
| INT-05 | Proxy Health | GET /health | 200 OK | [ ] |
|
|
| INT-06 | API Request | Simple chat | Valid response | [ ] |
|
|
| INT-07 | Tool Calling | Request with tools | Tool use response | [ ] |
|
|
| INT-08 | Streaming | Stream flag | SSE events | [ ] |
|
|
| INT-09 | Model Override | Different model | Model changed | [ ] |
|
|
| INT-10 | Error Handling | Invalid key | 401 error | [ ] |
|
|
|
|
---
|
|
|
|
## 3. Model-Specific Tests
|
|
|
|
### Test Different Model Families
|
|
|
|
#### 3.1 OpenAI Models
|
|
|
|
```bash
|
|
# GPT-4o Mini (fast, cheap)
|
|
npx agentic-flow --agent coder \
|
|
--task "Create a hello function" \
|
|
--provider requesty \
|
|
--model "openai/gpt-4o-mini"
|
|
|
|
# GPT-4o (premium quality)
|
|
npx agentic-flow --agent coder \
|
|
--task "Create a complex sorting algorithm" \
|
|
--provider requesty \
|
|
--model "openai/gpt-4o"
|
|
|
|
# GPT-4 Turbo
|
|
npx agentic-flow --agent researcher \
|
|
--task "Research quantum computing" \
|
|
--provider requesty \
|
|
--model "openai/gpt-4-turbo"
|
|
```
|
|
|
|
#### 3.2 Anthropic Models
|
|
|
|
```bash
|
|
# Claude 3.5 Sonnet
|
|
npx agentic-flow --agent coder \
|
|
--task "Write a file parser" \
|
|
--provider requesty \
|
|
--model "anthropic/claude-3.5-sonnet"
|
|
|
|
# Claude 3 Haiku (fast)
|
|
npx agentic-flow --agent coder \
|
|
--task "Simple function" \
|
|
--provider requesty \
|
|
--model "anthropic/claude-3-haiku"
|
|
```
|
|
|
|
#### 3.3 Google Models
|
|
|
|
```bash
|
|
# Gemini 2.5 Flash (FREE)
|
|
npx agentic-flow --agent coder \
|
|
--task "Create a calculator" \
|
|
--provider requesty \
|
|
--model "google/gemini-2.5-flash"
|
|
|
|
# Gemini 2.5 Pro
|
|
npx agentic-flow --agent researcher \
|
|
--task "Analyze AI trends" \
|
|
--provider requesty \
|
|
--model "google/gemini-2.5-pro"
|
|
```
|
|
|
|
#### 3.4 DeepSeek Models
|
|
|
|
```bash
|
|
# DeepSeek Chat V3 (cheap)
|
|
npx agentic-flow --agent coder \
|
|
--task "Create API endpoint" \
|
|
--provider requesty \
|
|
--model "deepseek/deepseek-chat-v3"
|
|
|
|
# DeepSeek Coder
|
|
npx agentic-flow --agent coder \
|
|
--task "Write Python script" \
|
|
--provider requesty \
|
|
--model "deepseek/deepseek-coder"
|
|
```
|
|
|
|
#### 3.5 Meta/Llama Models
|
|
|
|
```bash
|
|
# Llama 3.3 70B
|
|
npx agentic-flow --agent coder \
|
|
--task "Create function" \
|
|
--provider requesty \
|
|
--model "meta-llama/llama-3.3-70b-instruct"
|
|
|
|
# Llama 3.3 8B (fast)
|
|
npx agentic-flow --agent coder \
|
|
--task "Simple task" \
|
|
--provider requesty \
|
|
--model "meta-llama/llama-3.3-8b-instruct"
|
|
```
|
|
|
|
### Model Test Matrix
|
|
|
|
| Model | Provider | Tools | Stream | Expected Time | Status |
|
|
|-------|----------|-------|--------|---------------|--------|
|
|
| gpt-4o-mini | OpenAI | ✓ | ✓ | <5s | [ ] |
|
|
| gpt-4o | OpenAI | ✓ | ✓ | <10s | [ ] |
|
|
| claude-3.5-sonnet | Anthropic | ✓ | ✓ | <15s | [ ] |
|
|
| claude-3-haiku | Anthropic | ✓ | ✓ | <5s | [ ] |
|
|
| gemini-2.5-flash | Google | ✓ | ✓ | <3s | [ ] |
|
|
| gemini-2.5-pro | Google | ✓ | ✓ | <10s | [ ] |
|
|
| deepseek-chat-v3 | DeepSeek | ✓ | ✓ | <8s | [ ] |
|
|
| llama-3.3-70b | Meta | ✓ | ✓ | <12s | [ ] |
|
|
|
|
---
|
|
|
|
## 4. Performance Tests
|
|
|
|
### 4.1 Latency Testing
|
|
|
|
```bash
|
|
#!/bin/bash
|
|
# tests/performance/latency.sh
|
|
|
|
echo "=== Requesty Performance Tests ==="
|
|
|
|
for i in {1..10}; do
|
|
start=$(date +%s%N)
|
|
|
|
npx agentic-flow --agent coder \
|
|
--task "Say hello" \
|
|
--provider requesty \
|
|
--model "openai/gpt-4o-mini" > /dev/null
|
|
|
|
end=$(date +%s%N)
|
|
duration=$(( (end - start) / 1000000 ))
|
|
|
|
echo "Request $i: ${duration}ms"
|
|
done
|
|
```
|
|
|
|
**Success Criteria:**
|
|
- Average latency < 3000ms
|
|
- P95 latency < 5000ms
|
|
- No timeouts
|
|
|
|
### 4.2 Concurrent Requests
|
|
|
|
```bash
|
|
#!/bin/bash
|
|
# tests/performance/concurrent.sh
|
|
|
|
echo "=== Concurrent Request Test ==="
|
|
|
|
for i in {1..5}; do
|
|
npx agentic-flow --agent coder \
|
|
--task "Task $i" \
|
|
--provider requesty &
|
|
done
|
|
|
|
wait
|
|
echo "All concurrent requests completed"
|
|
```
|
|
|
|
**Success Criteria:**
|
|
- All requests complete successfully
|
|
- No proxy crashes
|
|
- No rate limit errors (or proper retry)
|
|
|
|
### 4.3 Large Context Test
|
|
|
|
```bash
|
|
# Test with large system prompt
|
|
npx agentic-flow --agent coder \
|
|
--task "$(cat large-context.txt)" \
|
|
--provider requesty \
|
|
--model "google/gemini-2.5-pro" # Large context window
|
|
```
|
|
|
|
**Success Criteria:**
|
|
- Request succeeds
|
|
- No context truncation errors
|
|
- Response is relevant
|
|
|
|
---
|
|
|
|
## 5. Security Tests
|
|
|
|
### 5.1 API Key Handling
|
|
|
|
**Test:** Verify API key is not logged
|
|
|
|
```bash
|
|
# Run with verbose logging
|
|
VERBOSE=true npx agentic-flow --agent coder \
|
|
--task "Test" \
|
|
--provider requesty 2>&1 | grep -c "requesty-"
|
|
```
|
|
|
|
**Expected:** 0 matches (only prefix should be logged)
|
|
|
|
### 5.2 API Key Validation
|
|
|
|
```bash
|
|
# Test invalid key format
|
|
REQUESTY_API_KEY="invalid" npx agentic-flow --agent coder \
|
|
--task "Test" \
|
|
--provider requesty
|
|
```
|
|
|
|
**Expected:** Error message about invalid key
|
|
|
|
### 5.3 Environment Isolation
|
|
|
|
```bash
|
|
# Verify proxy doesn't leak API key to client
|
|
curl http://localhost:3000/v1/messages \
|
|
-H "Content-Type: application/json" \
|
|
-d '{"model": "openai/gpt-4o-mini", "messages": [{"role": "user", "content": "Hello"}]}' \
|
|
| grep -c "requesty-"
|
|
```
|
|
|
|
**Expected:** 0 matches (API key should not be in response)
|
|
|
|
---
|
|
|
|
## 6. Error Handling Tests
|
|
|
|
### 6.1 Invalid API Key
|
|
|
|
```bash
|
|
REQUESTY_API_KEY="requesty-invalid" npx agentic-flow --agent coder \
|
|
--task "Test" \
|
|
--provider requesty
|
|
```
|
|
|
|
**Expected:**
|
|
- HTTP 401 Unauthorized
|
|
- Clear error message
|
|
- Exit code 1
|
|
|
|
### 6.2 Rate Limiting
|
|
|
|
```bash
|
|
# Send many requests quickly
|
|
for i in {1..50}; do
|
|
npx agentic-flow --agent coder \
|
|
--task "Test $i" \
|
|
--provider requesty &
|
|
done
|
|
wait
|
|
```
|
|
|
|
**Expected:**
|
|
- Some requests succeed
|
|
- Some requests retry (429 errors)
|
|
- All eventually complete or fail gracefully
|
|
|
|
### 6.3 Model Not Found
|
|
|
|
```bash
|
|
npx agentic-flow --agent coder \
|
|
--task "Test" \
|
|
--provider requesty \
|
|
--model "invalid/model-123"
|
|
```
|
|
|
|
**Expected:**
|
|
- HTTP 404 or 400
|
|
- Clear error message
|
|
- Exit code 1
|
|
|
|
### 6.4 Network Timeout
|
|
|
|
```bash
|
|
# Mock network timeout (requires proxy modification for testing)
|
|
REQUESTY_TIMEOUT=1 npx agentic-flow --agent coder \
|
|
--task "Long task" \
|
|
--provider requesty
|
|
```
|
|
|
|
**Expected:**
|
|
- Timeout error
|
|
- Retry attempt
|
|
- Clear error message
|
|
|
|
---
|
|
|
|
## 7. Compatibility Tests
|
|
|
|
### 7.1 Cross-Platform
|
|
|
|
Test on multiple OS:
|
|
|
|
- [ ] Linux (Ubuntu 22.04)
|
|
- [ ] macOS (Ventura+)
|
|
- [ ] Windows (WSL2)
|
|
|
|
### 7.2 Node.js Versions
|
|
|
|
Test with different Node versions:
|
|
|
|
- [ ] Node.js 18.x
|
|
- [ ] Node.js 20.x
|
|
- [ ] Node.js 22.x
|
|
|
|
### 7.3 Package Managers
|
|
|
|
Test with different package managers:
|
|
|
|
- [ ] npm
|
|
- [ ] yarn
|
|
- [ ] pnpm
|
|
|
|
---
|
|
|
|
## 8. Regression Tests
|
|
|
|
### 8.1 Existing Functionality
|
|
|
|
Verify Requesty doesn't break existing providers:
|
|
|
|
```bash
|
|
# Test Anthropic (should still work)
|
|
npx agentic-flow --agent coder \
|
|
--task "Test" \
|
|
--provider anthropic
|
|
|
|
# Test OpenRouter (should still work)
|
|
npx agentic-flow --agent coder \
|
|
--task "Test" \
|
|
--provider openrouter
|
|
|
|
# Test Gemini (should still work)
|
|
npx agentic-flow --agent coder \
|
|
--task "Test" \
|
|
--provider gemini
|
|
|
|
# Test ONNX (should still work)
|
|
npx agentic-flow --agent coder \
|
|
--task "Test" \
|
|
--provider onnx
|
|
```
|
|
|
|
**Success Criteria:**
|
|
- All providers still work
|
|
- No interference between providers
|
|
- Proxy ports don't conflict
|
|
|
|
---
|
|
|
|
## 9. Acceptance Criteria
|
|
|
|
### Must Pass (MVP)
|
|
|
|
- [ ] Unit tests: >90% coverage
|
|
- [ ] At least 5 models tested successfully
|
|
- [ ] Tool calling works (1+ model)
|
|
- [ ] Streaming works (1+ model)
|
|
- [ ] Error handling for invalid API key
|
|
- [ ] No regressions in existing providers
|
|
- [ ] Documentation complete
|
|
|
|
### Should Pass (V1)
|
|
|
|
- [ ] 10+ models tested
|
|
- [ ] Performance tests pass
|
|
- [ ] Security tests pass
|
|
- [ ] Cross-platform tested
|
|
- [ ] Concurrent requests work
|
|
- [ ] Rate limiting handled
|
|
|
|
### Nice to Have (Future)
|
|
|
|
- [ ] 20+ models tested
|
|
- [ ] Load testing (100+ concurrent)
|
|
- [ ] Automated test suite
|
|
- [ ] CI/CD integration
|
|
|
|
---
|
|
|
|
## 10. Test Execution Plan
|
|
|
|
### Day 1: Unit Tests
|
|
- Write test files
|
|
- Run unit tests
|
|
- Fix any failures
|
|
- Achieve >90% coverage
|
|
|
|
### Day 2: Integration Tests
|
|
- Test CLI integration
|
|
- Test proxy integration
|
|
- Test API integration
|
|
- Document results
|
|
|
|
### Day 3: Model Testing
|
|
- Test 5+ models
|
|
- Test tool calling
|
|
- Test streaming
|
|
- Document results
|
|
|
|
### Day 4: Final Validation
|
|
- Run all tests
|
|
- Fix any regressions
|
|
- Update documentation
|
|
- Sign off
|
|
|
|
---
|
|
|
|
## 11. Bug Reporting Template
|
|
|
|
When filing bugs, use this template:
|
|
|
|
```markdown
|
|
## Bug Report: Requesty Integration
|
|
|
|
**Test Category:** [Unit/Integration/E2E/Performance/Security]
|
|
**Test ID:** [e.g., INT-07]
|
|
|
|
**Description:**
|
|
[Clear description of the issue]
|
|
|
|
**Steps to Reproduce:**
|
|
1. Step 1
|
|
2. Step 2
|
|
3. Step 3
|
|
|
|
**Expected Behavior:**
|
|
[What should happen]
|
|
|
|
**Actual Behavior:**
|
|
[What actually happened]
|
|
|
|
**Environment:**
|
|
- OS: [e.g., Ubuntu 22.04]
|
|
- Node.js: [e.g., 20.10.0]
|
|
- agentic-flow: [e.g., 1.3.0]
|
|
- Requesty Model: [e.g., openai/gpt-4o-mini]
|
|
|
|
**Logs:**
|
|
[Paste relevant logs]
|
|
|
|
**Screenshots:**
|
|
[If applicable]
|
|
```
|
|
|
|
---
|
|
|
|
## 12. Test Results Dashboard
|
|
|
|
### Summary
|
|
|
|
| Category | Total Tests | Passed | Failed | Skipped | Coverage |
|
|
|----------|-------------|--------|--------|---------|----------|
|
|
| Unit | 0 | 0 | 0 | 0 | 0% |
|
|
| Integration | 0 | 0 | 0 | 0 | N/A |
|
|
| E2E | 0 | 0 | 0 | 0 | N/A |
|
|
| Performance | 0 | 0 | 0 | 0 | N/A |
|
|
| Security | 0 | 0 | 0 | 0 | N/A |
|
|
| **Total** | **0** | **0** | **0** | **0** | **0%** |
|
|
|
|
*(To be filled during testing)*
|
|
|
|
---
|
|
|
|
## Conclusion
|
|
|
|
This testing strategy ensures comprehensive validation of the Requesty integration across all critical dimensions: functionality, performance, security, and compatibility. By following this plan, we can confidently release the Requesty provider to users.
|
|
|
|
**Total Testing Time:** ~3 hours
|
|
**Test Coverage Goal:** >90%
|
|
**Model Coverage Goal:** 5+ models (MVP), 10+ models (V1)
|