Know exactly how your LLM performs.
Vectory is a precision evaluation toolkit that combines AI judges, deterministic metrics, RAG evaluation, and human review so you can measure quality, find failure modes, and ship with confidence.
- evaluation metrics
- 9+
- evaluation metrics
- AI judge providers
- 2
- AI judge providers
- dataset formats
- 3
- dataset formats
- open source
- 100%
- open source
| Trace | Input | Judge | BLEU | Faith. | Verdict |
|---|---|---|---|---|---|
| trace-0192 | Summarize the Q3 earnings call transcript… | 4.8 | 0.71 | 0.96 | pass |
| trace-0193 | Extract action items from meeting notes… | 4.5 | 0.64 | 0.91 | pass |
| trace-0194 | Answer: What is our refund policy for… | 2.1 | 0.32 | 0.48 | fail |
| trace-0195 | Translate the product description into… | 4.9 | 0.83 | 0.98 | pass |
| trace-0196 | Generate SQL for monthly active users… | 3.2 | 0.55 | 0.77 | review |
Aggregate scores
Failure modes
Capabilities
Every evaluation method your team needs, in one toolkit.
Stop stitching together scripts and spreadsheets. Vectory unifies automated, deterministic, and human evaluation into a single workflow.
LLM-as-Judge
Use GPT-4 or Claude to grade output quality at scale with fully customizable evaluation criteria and rubrics.
Rule-based metrics
Deterministic scoring with BLEU, ROUGE, exact match, Levenshtein similarity, and custom regex patterns.
RAG evaluation
Measure retrieval precision, recall, MRR, NDCG, context relevance, faithfulness, and answer completeness.
Error analysis
Discover failure modes with open coding, axial coding, taxonomy dashboards, and AI-assisted taxonomy suggestions.
Human evaluation
Rate outputs manually with customizable scales, criteria tags, and exportable annotation workflows.
Dataset management
Upload JSON, CSV, or PDF files with automatic column detection and role mapping.
MTEB leaderboard
Browse embedding model benchmarks from the Massive Text Embedding Benchmark directly in the app.
Custom benchmarks
Run your own embedding evaluations on your own data using the full MTEB framework.
Evaluators
Five ways to measure quality.
Automated grading with frontier models
Point current OpenAI or Anthropic models at your outputs with custom criteria: accuracy, helpfulness, tone, safety, and task fit. Get structured scores with reasoning for every sample in a batch.
# Configure judge and criteriajudge = LLMJudge(provider="anthropic")judge.criteria = ["accuracy", "faithfulness"]results = judge.evaluate_batch(dataset)→ 512/512 samples · avg score 4.31/5
Workflow
From raw outputs to shipping decisions in three steps.
- 01
Upload your data
Drop in JSON, JSONL, CSV, or PDF files. Vectory detects column roles such as input, output, and expected answer, or lets you map them manually.
- 02
Run evaluations
Pick your evaluators: AI judges, deterministic metrics, RAG scoring, or human review. Run large-scale batch evaluations across models.
- 03
Analyze & export
Visualize performance across models, drill into failure-mode taxonomies, and export results, annotations, and summary reports as CSV or JSON.
Field notes
Clear writing for teams evaluating real AI systems.
Agent evaluation that catches bad work before it ships
A practical framework for finding low-value loops, unsupported completion claims, and weak recovery in agent traces.
Fix retrieval quality before blaming the model
Why search rounds, evidence diversity, and synthesis checkpoints matter for RAG and research agents.
Building evaluations from production traces
A grounded way to convert incidents, escalations, and expert review into durable AI regression tests.
Private evaluations
Bring your real traces, datasets, and failure modes.
Vectory can help teams convert messy production workflows into confidential benchmark suites, regression checks, and executive-ready reliability reports.
- Trace ingestion and redaction guidance for private data.
- Custom pathology rules for your tools, approval paths, and operating boundaries.
- Baseline comparisons across models, prompts, retrieval systems, and agent harnesses.
Open Source · MIT License
Run it locally in under a minute.
No accounts, no lock-in. Launch the local app with a Python CLI and evaluate on your own infrastructure.
# quick start
$ uvx --from git+https://github.com/tacticaledge/vectory-ai.git vectory app
# install once
$ pipx install git+https://github.com/tacticaledge/vectory-ai.git
$ vectory app