Model Eval Results

SideCar ships a deterministic LLM evaluation harness (npm run eval:llm) that measures how well each model follows the system prompt and completes real agentic tasks. Results below are produced by running the harness against each backend and model.

What the eval measures

The suite has two layers:

Agent cases (47 total) — the model runs inside the full agent loop with real tools against a sandboxed workspace. Cases test tool selection, file editing, code quality, error recovery, and operating rules (e.g. run tests after a fix, use search_files not list_directory + filter).

Prompt cases (31 total) — the model is tested against the base system prompt without tool calls: identity, honesty, conciseness, language mirroring, injection resistance, retrieval citation, and tool preference rules.

Cases are scored deterministically (string matching, regex, trajectory inspection — no LLM-as-judge). A case passes only when every expectation holds. Some expectations use softExpect (reported but not counted toward pass/fail) for answer-quality checks where the core behavioral signal is in the trajectory.

Results

Last updated: 2026-05-14. Run with SIDECAR_EVAL_CASE_TIMEOUT=300000 for local models, 120000 for cloud. Suite v0.87d (57 agent + 35 prompt = 92 cases) — current; adds 5 system-infrastructure cases (gate-run-tests-after-fix, stub-validator-forces-real-impl, critic-security-path-traversal, sidecar-md-enforces-convention, cycle-detection-edit-pivot) that measure how SideCar’s built-in mechanisms compensate for known model failure modes. Last updated: 2026-05-14. Suite v0.87c (51 agent + 34 prompt = 85 cases) — adds 4 reasoning cases (thinking-cross-file-causality, thinking-semantic-version-compare, thinking-missing-await-in-loop, thinking-aliased-mutation) with softExpect.trajectoryHasThinking assertions. Suite v0.87b (47 agent + 34 prompt = 81 cases) — adds rule5-no-alternatives-menu and rule9-ambiguous-target + rule9-meta-knowledge prompt cases; includes prompt fixes for Rule 3/5/9. Suite v0.87a (47 agent + 32 prompt = 79 cases) — pre-Rule5/9-fix run; rows marked with ‡. Scores reflect the test suite at time of run; re-run models after any structural fix to get current numbers. Rows marked ‡ are from the v0.87a suite (79 cases) and need re-running on v0.87b. Prompt denominator may vary by model (some cases are skipped when a model lacks extended-thinking support or times out).

Two of the columns measure system-level behavior (the model inside SideCar’s full agent loop / against the base prompt). BFCL is a separate model-level signal — the model’s raw function-calling on the Berkeley Function Calling Leaderboard’s AST subset, with no SideCar scaffolding (see bench/bfcl/ and ADR-006). It’s the comparable, field-anchored number for local-model selection; it gives the harness no credit, by design. Cells are — until a run is recorded — populate with npm run bench:bfcl. Each recorded number carries its reproducibility envelope (model + quantization + context cap). § = macro accuracy on a 100-case sample (20/category) of the real BFCL_v4 AST set, Q4_K_M / 32K — directional, not a leaderboard-official figure (our checker is intentionally stricter on strings; see the per-category results below). Relative ranking across local models is the usable signal.

Models confirmed not working

Known model constraints

Local model RAM limits: On a 36 GB unified-memory machine, models larger than ~12 GB risk OOM. Always unload the previous model before loading a large one. qwen3.6 (23 GB) caused a kernel panic when loaded alongside another model.

Cloud API rate limits: SideCar’s system prompt + tool schemas totals ~23K tokens per request on typical tokenizers (and up to ~131K on some llama tokenizers when tool schemas are included). Free-tier accounts on OpenAI (30K TPM) and Groq (12K–30K TPM) exhaust their per-minute budget after 1–2 cases, causing the circuit breaker to trip for all subsequent cases. Upgrade to a paid tier, or use the Anthropic backend which has higher free limits.

OpenAI 200K TPM ceiling: Even on paid tiers, gpt-4o-mini and gpt-4.1-mini share a 200K TPM org-level limit. At ~23K tokens per request, a full 85-case eval burns through the budget in ~8 requests, causing 300–400 rate-limit failures per run. Scores are not reproducible. These models are excluded from the results table. Use gpt-5 or a model with a higher TPM allocation for OpenAI evals.

Fireworks context limit: llama-v3p3-70b-instruct has a 131,072 token context window. Our prompt + tool schemas lands at 131,473 tokens on the llama tokenizer — 401 tokens over. Prompt-only cases work fine. Use a Fireworks model with a larger context window, or wait for tool-catalog trimming in a future release.

GLM-style models (message.thinking): Models like GLM-4 emit chain-of-thought in message.thinking rather than <think> tags inside message.content. SideCar v0.87+ handles this correctly. Older versions would silently drop every event, producing empty trajectories.

Common failure patterns

These failures appear across multiple models and indicate areas for prompt improvement:

Running the eval yourself

# Local Ollama (default — free, no API key needed)
SIDECAR_EVAL_MODEL=ministral-3:latest SIDECAR_EVAL_CASE_TIMEOUT=300000 npm run eval:llm

# Anthropic
SIDECAR_EVAL_BACKEND=anthropic ANTHROPIC_API_KEY=<key> npm run eval:llm

# OpenAI
SIDECAR_EVAL_BACKEND=openai OPENAI_API_KEY=<key> SIDECAR_EVAL_MODEL=gpt-4o npm run eval:llm

# Groq (requires dev tier for agent cases)
SIDECAR_EVAL_BACKEND=groq GROQ_API_KEY=<key> SIDECAR_EVAL_MODEL=meta-llama/llama-4-scout-17b-16e-instruct npm run eval:llm

# Fireworks (use deepseek-v4-pro — default llama model exceeds context limit)
SIDECAR_EVAL_BACKEND=fireworks FIREWORKS_API_KEY=<key> SIDECAR_EVAL_MODEL=accounts/fireworks/models/deepseek-v4-pro npm run eval:llm

# Multi-model comparison table
SIDECAR_EVAL_COMPARE_MODELS="anthropic:claude-haiku-4-5-20251001,ollama:qwen3.5:latest" npm run eval:compare

BFCL (function-calling, model-level)

The BFCL (AST) column above is produced by a separate harness — the Berkeley Function Calling Leaderboard’s AST subset — which scores the model’s raw function-calling with no SideCar scaffolding. Use it for local-model selection: it’s the field-comparable answer to “is this model’s tool-use good for its weight class?”.

# Local Ollama (default model)
npm run bench:bfcl

# A specific candidate
SIDECAR_BFCL_MODEL=ministral-3:latest npm run bench:bfcl
SIDECAR_BFCL_MODEL=qwen3-coder:8b npm run bench:bfcl

# Full upstream dataset (the bundled fixtures are only a pipeline smoke test)
SIDECAR_BFCL_DATA=/path/to/bfcl SIDECAR_BFCL_MODEL=gemma4:e4b npm run bench:bfcl

When recording a number in the table, paste the macro accuracy and keep the reproducibility envelope the report prints (model + quantization + context cap) in the row’s Notes — a BFCL score without its quantization is not comparable. See bench/bfcl/README.md for categories and the documented simplifications vs. upstream.

BFCL AST-subset results (local candidates, first pass)

Envelope: real BFCL_v4 AST set, 100-case sample (20/category), Q4_K_M, 32K context, temp 0, local Ollama. Run 2026-06-30. Directional, not leaderboard-official.

What this says:

• All three small local models cluster at 83–86% raw function-calling — direct evidence for the “small models are capable with the right scaffolding” thesis, and the basis for treating them as viable defaults.
• gemma4:e4b leads (86%), so the v0.115 default choice holds up on a field-anchored model-level signal, not just the internal agent harness. granite4.1:3b (84%) at 2 GB / 3B punches well above its weight (it tops parallel_multiple outright); its weak spot is irrelevance (over-eager to call). ministral’s marginally lower raw tool-use (83%) despite its 98% system-level agent score is the model-level-vs-system-level gap in miniature — it’s strong inside the loop.
• The absolute numbers understate true tool-use. A failure audit showed most misses are checker strictness, not missed calls: math-expression serialization (x^2 vs 2*x**2 vs 2x^2) and dict-value matching (gradeDict, budget) that BFCL’s official checker normalizes and ours does not (the documented “stricter on strings” simplification). Genuine capability misses are fewer (e.g. granite emitting one call where two were expected). Because the same checker applies to all three, the relative ranking is reliable even though the absolute level is conservative. Hardening the checker’s numeric/expression normalization toward parity is the natural next bench increment.