Same Information. Frozen Portfolios. Real Scores.

What CapitalBench Measures

CapitalBench evaluates one narrow question: given the same market information and asset list, which portfolio does each model allocate for a one-week or one-month round? It is a benchmark for comparing model portfolios, not a trading system or investment recommendation engine.

Models may use internal learned knowledge and general market priors. They do not have to behave like blank slates. The controlled part is the externally supplied information: every model receives the same prompt, briefing, asset list, and optional mechanical market-data table.

Direct Model Evaluation

CapitalBench gives every model the same frozen market brief and asset choices once, then scores one valid investment decision. The model does not operate as an agent, browse, run code, retrieve live data, call other models, or revise a valid answer through follow-up prompts.

CapitalBench prepares the briefing and market data before the call and scores the result afterward. Those benchmark operations are separate from the model itself, which receives no tools during the decision.

This measures the model's direct response to a controlled briefing. It does not measure the best result an agentic system might achieve with tools, more compute, or repeated attempts.

Portfolio V2.0 Decision Rules

New portfolio rounds use Portfolio V2.0. Before allocating, each model reviews 6 to 8 allowed assets. The list must include the S&P 500 and cover at least four predefined economic-exposure groups. Rejected finalists stay in the audit record alongside selected assets.

The final portfolio still contains 1 to 5 holdings in 5% increments and totals 100%. Only its realized return is scored. Candidate estimates are saved for audit and forecast calibration; they do not create a second score.

Test Steps

1. Create the test directory. Each test lives under rounds/<round_id>/ with its rules, prompt, briefing, asset list, price files, hashes, and run folders.
2. Prepare the model-facing material. Operators write or import the factual briefing, exact prompt, and saved asset list.
3. Validate the asset list. Non-cash ETF tickers must pass Tiingo EOD validation before a public test is saved.
4. Generate the decision context. V2 uses a complete horizon-specific table of returns, benchmark-relative movement, volatility, drawdown, volume, beta, correlation, and 52-week position, sorted by option order.
5. Save the inputs. capitalbench hash-round writes SHA-256 hashes for the test inputs before submissions are collected.
6. Collect model calls before the deadline. Public runs use one call per model. Consistency checks use repeated calls and a different run_id.
7. Validate submissions. Raw provider responses are preserved. Only schema-valid submissions move into parsed submissions and scoring.
8. Fetch prices and score after the test ends. Starting prices can be collected while a test is pending; ending prices are collected only after the test period is over.
9. Publish normalized artifacts. Public results, hashes, reports, and leaderboard rows are synced to the website read model when Supabase sync is configured.

Test Files

The local test directory is the proof source of truth. Supabase stores normalized published copies for the website, but the canonical record remains the hashed test files.

Research And Briefing Rules

Deep research output is stored as proof material first. The only research artifact copied into the model prompt is research/final_briefing.md, which becomes test-level briefing.md. Market fact reports, source ledgers, and briefing audit reports remain proof-only.

The model-facing briefing should include facts, dates, values, forecasts labeled as forecasts, scheduled catalysts, and source-reported uncertainties. It should not include opinion, interpretation, scenario analysis, "why it matters" commentary, affected-market mapping, recommendations, or option rankings.

capitalbench import-research \
  --round rounds/<id> \
  --market-fact-report market_fact_report.md \
  --audit-report briefing_audit_report.md \
  --final-briefing final_briefing.md \
  --research-cutoff-utc "YYYY-MM-DDTHH:MM:SSZ"

Asset List Policy

Public tests declare an asset-list version in the test manifest and save that exact option file before model calls. New tests use the current configured universe shown on the Asset List page; as of Universe v2.1, that list has 70 choices. Earlier tests stay tied to the exact list the models saw, including v1.5 with 40 choices and v2.0 with 65 choices.

The model sees readable option ids, names, public symbols, asset classes, categories, groups, risk buckets, and exposure descriptions. Internal fields and provider-specific data-fetching fields are kept out of the prompt.

All non-cash options are public tickers that must validate against Tiingo EOD data before the test is saved. CASH has no ticker and is skipped during Tiingo validation. Because prompt context may include 7-day, 30-day, six-month, and one-year trailing returns, the pre-test data check should cover the full lookback window needed for the test.

capitalbench validate-universe \
  --round rounds/<id> \
  --start-date YYYY-MM-DD \
  --end-date YYYY-MM-DD

Submission Format

Each model must return one JSON or YAML object matching the round's declared submission format. Invalid raw responses remain preserved, but they are not scored and cannot enter a public benchmark result.

Public runs require replicate_index: 1, replicate_count: 1, and is_official_score: true. Consistency runs use repeated replicate indexes for each model and require is_official_score: false.

Model Execution Policy

• Every model receives the same prompt, briefing, asset list, and saved market-data artifact.
• Temperature is set to 0 where supported.
• Models receive no tools, browsing, web search, code execution, or external retrieval during the decision.
• Live market data and intentional use of post-cutoff facts, prices, news, or events are not allowed.
• Reasoning or thinking is set to the lowest provider-supported setting that still allows valid structured output.
• Official scores do not use extra model calls to improve or select a valid answer.
• Hidden reasoning tokens are recorded when exposed but are not treated as directly comparable across providers.
• Provider token usage, latency, and cost are logged when available.

capitalbench run-round \
  --round rounds/<id> \
  --models configs/models.local.yaml \
  --run-id official-YYYYMMDD \
  --run-type official \
  --allow-real-api-calls

Retry Policy

A public retry is allowed only when no valid allocation can be parsed because of infrastructure or format failure: malformed JSON, truncated response, provider transport or API failure, or schema output failure.

A retry is not allowed because of the selected asset, confidence value, or rationale quality. Failed raw responses must remain in the run artifacts and must stay ineligible for public scoring.

Pricing And Scoring

CapitalBench scores valid submissions against local price files. Adjusted close is preferred. If only close is supplied, scoring may continue but records a warning in the result artifacts. Tiingo fetching is strict about dates and requires rows matching the manifest start and end dates. Final automated scoring refreshes both start and end adjusted closes together after the round window ends, so ETF distribution adjustments are on one post-window price basis.

Cash is treated as a zero return unless cash prices are explicitly supplied. Portfolio tests use the weighted realized return of the submitted portfolio. Latest-test tables show raw portfolio return, S&P 500 return, Portfolio Minus S&P 500, and regret. Combined track scorecards use CapitalBench Score, where 100 means matching the highest-returning scored option, 0 means no net return, and negative values preserve losses. Overall scores divide summed model returns by summed oracle returns.

Benchmark Comparison Sets are the fair headline ranking system. A set has a fixed model roster and includes only resolved rounds where every model in that roster has an official result. If one model misses a round, that round is excluded from the set for everyone.

All-available history remains useful context, but it is not the headline fair ranking when model histories differ. Weekly comparison sets become the Current Benchmark at 6 shared resolved rounds. Monthly comparison sets become the Current Benchmark at 3 shared resolved rounds.

Interim tracking can be generated from reusable price snapshots after the start date and before the final end date. These rows are shown on round pages as provisional progress versus S&P 500 only; they do not finish a round, populate the final benchmark result, or create rank/regret metrics.

capitalbench fetch-prices \
  --round rounds/<id> \
  --run-id <run_id> \
  --entry-date YYYY-MM-DD \
  --exit-date YYYY-MM-DD \
  --full-universe

Benchmark Comparison Sets

Model Behavior Labels And Pills

Behavior profiles are a post-publication descriptive layer. They summarize how a model allocated; they do not change a round's return, score, rank, eligibility, or benchmark-set membership.

The strongest qualifying risk or exposure signal supplies the caption modifier. Peer-relative construction, turnover, or Portfolio Difference supplies its allocation-style noun. Four fixed roles keep the pills comparable: signature, construction, tempo, and current positioning or lifecycle. Weekly/monthly conflicts and sufficiently sampled reversals under a newer methodology explicitly qualify the confidence claim.

Realized returns and ranks, free-form rationale wording, briefing prose, and ineligible, invalid, retrospective, or pilot runs cannot assign a behavior label. Structured candidate ledgers, forecasts, confidence, and key risks appear only as separately labeled decision-process context.

Portfolio Difference leaves the measured model out of the comparison portfolio. A score of 42 means about 42% of allocation would need to change to match the group portfolio. It compares outputs; it does not prove copying, influence, or intentional contrarian reasoning. Both horizons are required for a combined value.

Recent-winner tilt does not assign the archetype. It reports whether a frozen portfolio favored assets that had already outperformed before the decision cutoff. S&P 500 and cash receive the neutral score of 50; peer context uses the median of the other models in the same round. A combined value is published only when both monthly and weekly observations are available.

Benchmark Result Definitions

Fairness Controls

• Freeze manifest.yaml, briefing.md, options.yaml, prompt.md, and any prompt-facing market_data/ artifact before collection.
• Publish hashes.json so readers can verify that inputs did not change after model calls.
• Do not give one model extra context or a different asset list.
• Do not revise a completed round after seeing model choices just to create more varied allocations.
• Preserve invalid raw submissions instead of deleting them.
• Use the same starting and ending price source for every option.
• Identify exactly one public model-call run for each round's public benchmark result.
• Exclude mock, provider-smoke, failed, incomplete, and retrospective runs from public benchmark results.
• Add newly released models only to future rounds. Do not rerun old rounds after outcomes may be knowable.

Audit Packets And Website Sync

The CLI stores exact provider text in local raw_responses/ sidecars, normalized payloads in submissions/raw/, validated submissions in submissions/parsed/, and SHA-256 paths in run_log.jsonl. Raw provider text and private smoke-test output are excluded from the public repo; reports, validation summaries, result CSVs, and public hashes are generated from the sanitized artifacts.

When the Supabase URL and server-side service-role credentials are configured, publish and scoring commands can sync normalized public rows to Supabase for website rendering. The public frontend uses only the Supabase anon key. Service-role keys, provider keys, and market-data keys are never exposed in built assets.

capitalbench sync-web --round rounds/<round_id> --run-id <run_id>
capitalbench sync-web --rounds-dir rounds --include-cumulative

Limitations

CapitalBench measures one prompt, one asset list, and one time window at a time. It does not model taxes, transaction costs, slippage, liquidity, dividends, or real-world execution constraints. A one-week or one-month result can be dominated by noise, and a test where many models choose the same asset can be fair but low-discrimination as a ranking event.

The framework is useful for reproducible comparison, but it should not be read as proof that a model has durable investing skill. The benchmark output is research, not financial advice.