Same Information. Frozen Portfolios. Real Scores.

What CapitalBench Measures

CapitalBench evaluates one narrow question: given the same market information and asset list, which portfolio does each model allocate for a one-week or one-month round? It is a benchmark for comparing model portfolios, not a trading system or investment recommendation engine.

Models may use internal learned knowledge and general market priors. They do not have to behave like blank slates. The controlled part is the externally supplied information: every model receives the same prompt, briefing, asset list, and optional mechanical market-data table.

Test Steps

1. Create the test directory. Each test lives under rounds/<round_id>/ with its rules, prompt, briefing, asset list, price files, hashes, and run folders.
2. Prepare the model-facing material. Operators write or import the factual briefing, exact prompt, and saved asset list.
3. Validate the asset list. Non-cash ETF tickers must pass Tiingo EOD validation before a public test is saved.
4. Add optional return context. A mechanical full-asset-list return table may be generated from adjusted closes and sorted by option order.
5. Save the inputs. capitalbench hash-round writes SHA-256 hashes for the test inputs before submissions are collected.
6. Collect model calls before the deadline. Public runs use one call per model. Consistency checks use repeated calls and a different run_id.
7. Validate submissions. Raw provider responses are preserved. Only schema-valid submissions move into parsed submissions and scoring.
8. Fetch prices and score after the test ends. Starting prices can be collected while a test is pending; ending prices are collected only after the test period is over.
9. Publish normalized artifacts. Public results, hashes, reports, and leaderboard rows are synced to the website read model when Supabase sync is configured.

Test Files

The local test directory is the proof source of truth. Supabase stores normalized published copies for the website, but the canonical record remains the hashed test files.

Research And Briefing Rules

Deep research output is stored as proof material first. The only research artifact copied into the model prompt is research/final_briefing.md, which becomes test-level briefing.md. Market fact reports, source ledgers, and briefing audit reports remain proof-only.

The model-facing briefing should include facts, dates, values, forecasts labeled as forecasts, scheduled catalysts, and source-reported uncertainties. It should not include opinion, interpretation, scenario analysis, "why it matters" commentary, affected-market mapping, recommendations, or option rankings.

capitalbench import-research \
  --round rounds/<id> \
  --market-fact-report market_fact_report.md \
  --audit-report briefing_audit_report.md \
  --final-briefing final_briefing.md \
  --research-cutoff-utc "YYYY-MM-DDTHH:MM:SSZ"

Asset List Policy

Public tests declare an asset-list version in the test manifest and save that exact option file before model calls. New tests use the current configured universe shown on the Asset List page; as of Universe v2.1, that list has 70 choices. Earlier tests stay tied to the exact list the models saw, including v1.5 with 40 choices and v2.0 with 65 choices.

The model sees readable option ids, names, public symbols, asset classes, categories, groups, risk buckets, and exposure descriptions. Internal fields and provider-specific data-fetching fields are kept out of the prompt.

All non-cash options are public tickers that must validate against Tiingo EOD data before the test is saved. CASH has no ticker and is skipped during Tiingo validation. Because prompt context may include 7-day, 30-day, six-month, and one-year trailing returns, the pre-test data check should cover the full lookback window needed for the test.

capitalbench validate-universe \
  --round rounds/<id> \
  --start-date YYYY-MM-DD \
  --end-date YYYY-MM-DD

Submission Format

Each model must return one JSON or YAML object matching the round's declared submission format. Invalid raw responses remain preserved, but they are not scored and cannot enter a public benchmark result.

Public runs require replicate_index: 1, replicate_count: 1, and is_official_score: true. Consistency runs use repeated replicate indexes for each model and require is_official_score: false.

Model Execution Policy

• Every model receives the same prompt, briefing, asset list, and saved market-data artifact.
• Temperature is set to 0 where supported.
• Tools, browsing, web search, code execution, and external retrieval are disabled at the API level where supported.
• Live market data and intentional use of post-cutoff facts, prices, news, or events are not allowed.
• Reasoning or thinking is set to the lowest provider-supported setting that still allows valid structured output.
• Hidden reasoning tokens are recorded when exposed but are not treated as directly comparable across providers.
• Provider token usage, latency, and cost are logged when available.

capitalbench run-round \
  --round rounds/<id> \
  --models configs/models.local.yaml \
  --run-id official-YYYYMMDD \
  --run-type official \
  --allow-real-api-calls

Retry Policy

A public retry is allowed only when no valid allocation can be parsed because of infrastructure or format failure: malformed JSON, truncated response, provider transport or API failure, or schema output failure.

A retry is not allowed because of the selected asset, confidence value, or rationale quality. Failed raw responses must remain in the run artifacts and must stay ineligible for public scoring.

Pricing And Scoring

CapitalBench scores valid submissions against local price files. Adjusted close is preferred. If only close is supplied, scoring may continue but records a warning in the result artifacts. Tiingo fetching is strict about dates and requires rows matching the manifest start and end dates. Final automated scoring refreshes both start and end adjusted closes together after the round window ends, so ETF distribution adjustments are on one post-window price basis.

Cash is treated as a zero return unless cash prices are explicitly supplied. Portfolio tests use the weighted realized return of the submitted portfolio. Latest-test tables show raw portfolio return, S&P 500 return, Portfolio Minus S&P 500, and regret. Combined track scorecards use CapitalBench Score, where 100 means matching the highest-returning scored option, 0 means no net return, and negative values preserve losses. Overall scores divide summed model returns by summed oracle returns.

Benchmark Comparison Sets are the fair headline ranking system. A set has a fixed model roster and includes only resolved rounds where every model in that roster has an official result. If one model misses a round, that round is excluded from the set for everyone.

All-available history remains useful context, but it is not the headline fair ranking when model histories differ. Weekly comparison sets become the Current Benchmark at 6 shared resolved rounds. Monthly comparison sets become the Current Benchmark at 3 shared resolved rounds.

Interim tracking can be generated from reusable price snapshots after the start date and before the final end date. These rows are shown on round pages as provisional progress versus S&P 500 only; they do not finish a round, populate the final benchmark result, or create rank/regret metrics.

capitalbench fetch-prices \
  --round rounds/<id> \
  --run-id <run_id> \
  --entry-date YYYY-MM-DD \
  --exit-date YYYY-MM-DD \
  --full-universe

Benchmark Comparison Sets

Benchmark Result Definitions

Fairness Controls

• Freeze manifest.yaml, briefing.md, options.yaml, prompt.md, and any prompt-facing market_data/ artifact before collection.
• Publish hashes.json so readers can verify that inputs did not change after model calls.
• Do not give one model extra context or a different asset list.
• Do not revise a completed round after seeing model choices just to create more varied allocations.
• Preserve invalid raw submissions instead of deleting them.
• Use the same starting and ending price source for every option.
• Identify exactly one public model-call run for each round's public benchmark result.
• Exclude mock, provider-smoke, failed, incomplete, and retrospective runs from public benchmark results.
• Add newly released models only to future rounds. Do not rerun old rounds after outcomes may be knowable.

Audit Packets And Website Sync

The CLI stores exact provider text in local raw_responses/ sidecars, normalized payloads in submissions/raw/, validated submissions in submissions/parsed/, and SHA-256 paths in run_log.jsonl. Raw provider text and private smoke-test output are excluded from the public repo; reports, validation summaries, result CSVs, and public hashes are generated from the sanitized artifacts.

When the Supabase URL and server-side service-role credentials are configured, publish and scoring commands can sync normalized public rows to Supabase for website rendering. The public frontend uses only the Supabase anon key. Service-role keys, provider keys, and market-data keys are never exposed in built assets.

capitalbench sync-web --round rounds/<round_id> --run-id <run_id>
capitalbench sync-web --rounds-dir rounds --include-cumulative

Limitations

CapitalBench measures one prompt, one asset list, and one time window at a time. It does not model taxes, transaction costs, slippage, liquidity, dividends, or real-world execution constraints. A one-week or one-month result can be dominated by noise, and a test where many models choose the same asset can be fair but low-discrimination as a ranking event.

The framework is useful for reproducible comparison, but it should not be read as proof that a model has durable investing skill. The benchmark output is research, not financial advice.