Same frozen inputs
Every system receives the same approved evaluation packet.
PRIVATE CAPITALBENCH EVALS
Private AI capital allocation evals.
CapitalBench independently evaluates your model, agent, or workflow using the same frozen market brief, portfolio constraints, and outcome window as leading AI systems.
Launch price: $3,000 USD · Typical completion: 10-14 calendar days · Fixed scope
You receive a private comparative scorecard, consistency analysis, failure-mode review, machine-readable results, and a complete audit packet.
Private by default · No source code or model weights required · No real capital traded
Proof points
Current comparators
Compare against up to four supported frontier models.
Real outcomes
Official performance resolves using actual market prices.
Consistency testing
Repeated runs reveal unstable or highly variable decisions.
Auditable files
Inputs, outputs, timestamps, hashes, prices, and calculations are preserved.
Decision evidence
Make a consequential model decision with evidence.
A model can produce an impressive investment thesis and still make unstable, overconcentrated, expensive, or poorly calibrated allocation decisions.
A Private CapitalBench Eval turns one specific model configuration into a documented decision record. It shows what the system selected, how consistently it selected it, what risk it took, how it compared with frontier alternatives, and how the allocation performed after the market window closed.
Choose the right model
Compare your current system with frontier alternatives under identical conditions.
Validate a release
Test the exact model, prompt, workflow, and configuration you are considering shipping.
Diagnose behavior
Identify concentration, instability, invalid outputs, excessive risk-taking, weak diversification, or poor cost-performance trade-offs.
Support internal claims
Give product leaders, investors, research teams, and customers evidence that is more defensible than screenshots or selected examples.
Standard sprint scope
One defined system. One controlled evaluation. No moving goalposts.
| Scope item | Standard sprint |
|---|---|
| Client system | One model, agent, workflow, or configuration |
| Comparator roster | Up to four currently supported frontier models |
| Official evaluation | One live weekly CapitalBench allocation round |
| Consistency evaluation | Five separate non-official repeated runs per system |
| Asset universe | Current CapitalBench public ETF and cash universe |
| Portfolio format | One to five holdings, fixed allocation increments, 100% total |
| Information access | Same frozen brief, prompt, universe, and market-data packet |
| Base evaluation mode | Browsing, tools, and live retrieval disabled |
| Performance window | Approximately one week |
| Final output | Report, scorecard, data export, audit bundle, and readout |
| Confidentiality | Private unless the client approves publication in writing |
Evaluation modes
Controlled mode versus production mode
A sprint may include both modes, but their scores are never mixed into one misleading comparison.
Controlled Benchmark Mode
The system receives the same frozen external information as every comparator. Browsing, search, code execution, live prices, and external retrieval are disabled where technically possible.
This mode isolates allocation judgment and creates the fairest comparison with the public CapitalBench benchmark.
Production Workflow Mode
The system runs using its normal retrieval, tools, prompts, or internal workflow.
Production-mode results are reported separately because a tool-enabled agent should not be placed on the same headline ranking as a closed-capability model receiving only frozen inputs.
Evaluation Plan
The rules are approved and frozen before any official run.
Every buyer receives a short Evaluation Plan before execution.
What gets frozen in the Evaluation Plan
| Field | What is frozen |
|---|---|
| System under test | Model name, version, endpoint, prompt configuration, and relevant settings |
| Primary business question | The decision the company wants the evaluation to support |
| Comparator roster | Exact comparator models and versions |
| Evaluation mode | Controlled or production workflow |
| Input packet | Prompt, briefing, market data, asset universe, and schema |
| Portfolio constraints | Holdings limit, allocation increments, cash rules, and total weight |
| Official-run policy | Number of official calls and allowed retry conditions |
| Consistency protocol | Number of repeats and settings |
| Market window | Entry rule, exit rule, benchmark, and date fallback |
| Metrics | Performance, consistency, validity, behavior, latency, and cost fields |
| Success thresholds | Any agreed internal release or acceptance thresholds |
| Confidentiality | Private, anonymized, or approved for public publication |
| Retention | Credential, raw-output, and report retention periods |
Nothing material is changed after the official outputs have been collected. If a rule must change because of a technical issue, the change is documented and approved before the market outcome is known.
Process
From system access to final report
Scope review
CapitalBench confirms the system, business question, access method, comparator roster, and whether the evaluation can be run fairly.
- Client time
- 15-20 minutes
- Output
- fit confirmation and fixed scope
Plan and connection
A short Evaluation Plan freezes the tested configuration, inputs, constraints, comparators, retry rules, and technical access path.
- Client time
- 60-90 minutes total
- Output
- approved plan and verified connection
Official and consistency runs
One official allocation determines the live outcome score. Repeated runs measure stability, invalid outputs, and allocation dispersion.
- Client time
- no live meeting required after approval
- Output
- frozen outputs, run log, and hashes
Market resolution and scoring
The frozen portfolios resolve against actual market prices, then CapitalBench calculates performance, risk, concentration, validity, cost, latency, and consistency.
- Client time
- review questions only
- Output
- scorecard, attribution, and evidence files
Report and readout
The final packet includes the decision report, machine-readable exports, audit bundle, failure-mode register, and a 60-minute readout.
- Client time
- one 60-minute readout
- Output
- report, audit packet, and follow-up window
Metrics and scoring
Performance is only one part of the scorecard.
| Metric | What it answers |
|---|---|
| Portfolio return | How did the frozen allocation perform during the defined outcome window? |
| S&P 500 difference | Did it outperform or underperform the benchmark, and by how much? |
| CapitalBench Score | How did the return compare with the highest-returning eligible option in the same window? |
| Valid-output rate | Did the system reliably return a complete, schema-valid allocation? |
| Allocation consistency | Did repeated runs produce similar holdings and weights? |
| Concentration | How much capital was placed in the largest holding or narrowest theme? |
| Risk appetite | How strongly did the portfolio lean toward growth, momentum, cyclicality, defense, cash, or other risk profiles? |
| Cost and latency | How much time and provider usage were required for valid responses where available? |
See calculation details
Portfolio return = sum of each holding's weight multiplied by its realized return.
Portfolio minus S&P 500 = portfolio return minus S&P 500 return.
CapitalBench Score = portfolio return relative to the hindsight-best eligible asset return in that same window.
Regret = hindsight-best eligible asset return minus portfolio return.
Consistency = similarity of holdings and weights across repeated identical tasks.
Performance, consistency, cost, and latency remain visible as separate fields. CapitalBench does not hide materially different trade-offs inside one unexplained composite grade.
Deliverables
Everything you receive
Approved Evaluation Plan
The exact system, comparator roster, inputs, constraints, retry policy, metrics, timeline, confidentiality setting, and scoring rules.
PDF or MarkdownExecutive Evaluation Report
A decision-focused report explaining the result, what it means, major risks, limitations, and recommended next actions.
12-20 page PDFComparative Scorecard
Side-by-side results for the client system and every comparator, including performance, consistency, validity, behavior, latency, and cost where available.
PDF table, CSV, and JSONConsistency and Reliability Analysis
Repeated-run evidence covering portfolio overlap, weight dispersion, frequent holdings, invalid responses, confidence dispersion, and official-run representativeness.
Report section and data exportFailure-Mode Register
A prioritized record of observed weaknesses, severity, evidence, business impact, recommended action, and retest condition.
Report tableAudit Packet and Exports
Prompt, frozen briefing, asset universe, schema, hashes, raw-output references, parsed submissions, price files, calculations, CSV, and JSON.
Downloadable bundle Sample report preview
See the evidence before you buy
The ungated sample report uses public CapitalBench round CB-2026-05-28-1W and presents it in the private-report format.
Executive summary
Preview of the private report section, using public benchmark evidence and marked scope limits.
Comparative scorecard
Preview of the private report section, using public benchmark evidence and marked scope limits.
Allocation attribution
Preview of the private report section, using public benchmark evidence and marked scope limits.
Consistency analysis
Preview of the private report section, using public benchmark evidence and marked scope limits.
Risk and concentration
Preview of the private report section, using public benchmark evidence and marked scope limits.
Audit-file index
Preview of the private report section, using public benchmark evidence and marked scope limits.
Security and data handling
Designed to evaluate the system without taking control of it.
The standard sprint does not require source code, model weights, production databases, customer records, brokerage credentials, fund holdings, or access to real capital.
| Access method | How it works | Best for |
|---|---|---|
| Customer-hosted endpoint | The client creates a temporary, restricted credential for a non-production or isolated endpoint. | Proprietary models and agents |
| Provider API access | CapitalBench runs the named model through an approved provider account or temporary client-provided key. | Commercial foundation models |
| Customer-executed runner | CapitalBench supplies the frozen packet and execution instructions. The client returns timestamped raw outputs and logs. | Strict internal security policies |
Customer-executed results are clearly identified because CapitalBench did not directly control execution. They may still be useful internally, but they provide weaker independent execution assurance.
Private by default
Client identity, system details, prompts, outputs, scores, and reports are not published without written approval.
Purpose-limited access
Credentials are used only to execute the agreed evaluation.
No training use
CapitalBench does not use private client inputs or outputs to train its own models.
Credential deletion
Temporary credentials are removed after execution is complete.
No sensitive production data required
The standard sprint uses a CapitalBench market packet and does not require customer PII or live customer records.
NDA available
A mutual NDA may be completed before private technical details or credentials are shared.
Independence policy
Payment buys the evaluation, not a favorable result.
CapitalBench is paid to design, execute, and document the agreed evaluation. Payment does not determine the score, model ranking, interpretation of valid results, or whether failures appear in the final report.
Inputs and rules are frozen before official execution. Failed and invalid attempts remain part of the record. Results are not replaced because a model selected an undesirable asset or performed poorly.
The client controls confidentiality and publication. It does not control the underlying calculation after the Evaluation Plan has been approved.
No pay-to-rank No private answer shopping No retroactive scoring changes No publication without client approval
Pricing
Fixed scope. Transparent price.
Private CapitalBench Eval Sprint
Request Scope Review $3,000 USD
Launch pricing for the first five completed private engagements.
Included
- One client system or configuration
- Up to four frontier comparator models
- One official live weekly evaluation
- Five consistency runs per system
- Current CapitalBench asset universe
- Performance, risk, concentration, and consistency scoring
- Cost and latency analysis where available
- 12-20 page private report
- CSV and JSON exports
- Complete audit packet
- 60-minute executive readout
- Seven days of written follow-up
Payment terms
50% after the Evaluation Plan and SOW are approved. 50% when the final report and audit packet are delivered.
Cost details
Comparator model API costs within the agreed scope are included. Usage charges incurred through the client's own endpoint or provider account remain the client's responsibility. Applicable taxes are additional.
Future standard price
Standard price after the first five completed engagements: $5,000 USD
No payment is required to submit a request. CapitalBench confirms fit, access method, comparator roster, timeline, and fixed scope before issuing an invoice.
Add-ons and extended scope
| Add-on | Price |
|---|---|
| Additional client model, prompt, or configuration | $750 |
| Additional live weekly round | $1,250 |
| One-month outcome track | $2,000 |
| Custom asset universe or scoring rubric | From $1,500 |
| Production workflow mode with tools or retrieval | From $2,500 |
| Additional executive or technical workshop | $500 |
| Ongoing private monitoring | From $2,500/month |
Every add-on is agreed in writing before execution. CapitalBench does not add unapproved hourly charges.
Fit
Built for teams putting AI near investment decisions
AI finance and research products
Evaluate the model behind an AI analyst, investment copilot, portfolio assistant, or research workflow.
Fintech and wealthtech companies
Compare vendors or configurations before committing engineering effort to a production integration.
Funds and investment research teams
Evaluate an internal model, agent, or research workflow without connecting CapitalBench to brokerage systems or real capital.
Model providers and evaluation teams
Measure domain-specific allocation behavior that generic knowledge and reasoning benchmarks do not directly reveal.
Investors and diligence teams
Independently inspect performance claims made by a company building an AI investment product.
Limitations
What a responsible evaluation can and cannot prove
A one-week market result can be affected by noise, unusual events, and the chosen asset universe. It is a real outcome, but it is not by itself proof of persistent investment skill.
The result applies only to the tested model version, configuration, input packet, constraints, and outcome window. A model update or workflow change may produce different behavior.
CapitalBench therefore reports performance alongside consistency, validity, concentration, risk, cost, latency, and explicit limitations. Companies seeking stronger evidence should run multiple weekly and monthly rounds.
FAQ
Complete FAQ
Scope and fit
What is a Private CapitalBench Eval Sprint?
It is a fixed-scope evaluation of one AI model, agent, workflow, or configuration on a controlled capital-allocation task. The system is compared with frontier models using frozen inputs and an agreed scoring protocol.
What business decision is it designed to support?
Common decisions include selecting a model vendor, validating a new release, comparing a fine-tune with its base model, diagnosing unstable behavior, establishing a regression baseline, or substantiating a limited performance claim.
What kinds of systems can be evaluated?
API-accessible foundation models, fine-tuned models, proprietary models, RAG systems, investment research agents, prompt workflows, and third-party AI products may be evaluated when a stable execution method can be established.
Can you evaluate two versions of our system?
The standard sprint includes one client configuration. A second model, prompt, tool setup, or version can be added for $750.
Can you evaluate a system that does not produce portfolios today?
Usually, provided it can return the required structured output. CapitalBench supplies the schema and runs a non-scored validation test before the official evaluation.
Do you need access to our source code or model weights?
No. The standard evaluation requires only a working execution method, such as an API endpoint, temporary provider credential, or customer-executed runner.
Do you need our brokerage account or real capital?
No. CapitalBench evaluates simulated allocations and market outcomes. It does not place trades or connect to brokerage accounts.
Method and fairness
How are comparator models selected?
The buyer and CapitalBench agree on up to four currently supported comparators before execution. The exact provider, model identifier, and version information available at the time are recorded in the Evaluation Plan.
Does every model receive the same information?
In Controlled Benchmark Mode, every system receives the same approved prompt, factual briefing, asset universe, market-data packet, portfolio constraints, and response schema.
Can a model browse the web or use tools?
Not in the standard Controlled Benchmark Mode. Tools and live retrieval are disabled where technically possible. A tool-enabled Production Workflow Mode is available as an add-on and is reported separately.
What happens if a model returns invalid output?
The raw attempt is retained and marked invalid. A retry is permitted only under the pre-approved policy for transport, truncation, malformed structured output, or similar infrastructure failure. A model is not retried merely because its allocation appears weak.
Do you choose the best answer from repeated runs?
No. The live outcome score uses one official frozen response. Repeated runs are used only for separately reported consistency analysis.
Can the rules change after the market result is known?
No material scoring or input rule is changed after the official responses are collected. Any necessary pre-outcome amendment is documented and approved.
How are prices selected?
The Evaluation Plan states the market-data source, entry rule, exit rule, benchmark, adjusted-price treatment, and fallback for non-trading days or unavailable data before execution.
Why use a live market window instead of only historical data?
Live execution reduces the risk that models already contain or infer the outcome. Historical replay may be useful for diagnostics, but it is labelled separately and is not presented as equivalent to a prospective live result.
Is one week enough to determine which model is best?
No. One round is a defined decision event, not proof of persistent skill. It can still expose differences in allocation, consistency, concentration, validity, latency, and behavior. Multiple rounds provide stronger evidence.
Will we receive a pass or fail?
Only when measurable thresholds are agreed before execution. Otherwise, the report provides a comparative scorecard and a recommendation such as ship, iterate, switch, or gather more evidence.
Data and confidentiality
Are the evaluation and results private?
Yes. Private is the default. The client's identity, system, prompts, outputs, scores, and report are not published without written approval.
Will CapitalBench sign an NDA?
A mutual NDA can be completed before confidential technical details or access credentials are exchanged.
Will our data be used for training?
CapitalBench does not use private client inputs or outputs to train its own models. Any third-party API handling remains subject to the named provider's terms.
How long is private data retained?
The recommended default is 30 days after final delivery for raw private artifacts, unless the SOW specifies another period. Temporary credentials should be deleted immediately after execution.
Are you SOC 2 or ISO 27001 certified?
CapitalBench does not currently represent itself as SOC 2 or ISO 27001 certified. The standard sprint is designed not to require PII, production customer data, source code, or brokerage access. Customer-hosted execution is available where internal policy requires it.
Can the evaluation run inside our environment?
A customer-executed runner or customer-hosted endpoint can be used. The report clearly identifies who controlled execution and any resulting assurance limitations.
Results and publication
What happens if our system performs poorly?
The result remains part of the private report. CapitalBench does not replace it with a more favorable run. The report focuses on why the system behaved as it did and what should change before a retest.
Can we publish the result?
Yes, with written agreement on the exact system version, evaluation date, scope, limitations, and wording. Publication is optional.
Can we use the CapitalBench name or logo in marketing?
Only with written approval. Any public statement must identify the tested version and evaluation window and must not imply certification, durable outperformance, or a broader endorsement than the evidence supports.
Does paying for the sprint guarantee a favorable result?
No. Payment covers the evaluation work, report, and delivery. It does not purchase a ranking, score, testimonial, badge, or endorsement.
Does CapitalBench publish negative private evaluations?
Not without client approval. Private results remain private, but they are not removed from the report delivered to the client.
Pricing and delivery
What is included in the $3,000 price?
One client configuration, four comparators, one live weekly round, consistency testing, a report, CSV and JSON exports, an audit packet, a 60-minute readout, and seven days of written follow-up.
Are model API costs included?
Comparator model usage within the agreed scope is included. Charges incurred through the client's own endpoint or provider account remain the client's responsibility.
How long does the sprint take?
A standard weekly sprint typically takes 10-14 calendar days after access and scope are approved. Market holidays, data exceptions, or client delays may extend delivery.
How much time does our team need to provide?
Usually two to three hours in total: one kickoff, technical connection support, Evaluation Plan approval, and the final readout.
When is payment due?
Fifty percent is due after the SOW and Evaluation Plan are approved. The remaining fifty percent is due when the report and audit packet are delivered.
What is the cancellation policy?
Before technical setup begins, the deposit is refundable. After the Evaluation Plan is approved and official execution begins, the deposit is non-refundable. If CapitalBench cannot complete the agreed scope, fees for undelivered work are refunded.
Is there an ongoing subscription?
No subscription is required for the sprint. Additional rounds or ongoing monitoring are optional.
What happens after the sprint?
The client can implement the recommendations, test a revised configuration, add monthly rounds, or establish ongoing private monitoring. Any next engagement is separately scoped.
Legal boundary
Is this investment advice?
No. CapitalBench evaluates AI-system behavior using simulated allocations and market data. It does not recommend that the client or any other person trade the evaluated portfolio.
Is this a regulatory or model-risk certification?
No. The report is research and evaluation evidence, not regulatory approval, legal advice, fiduciary advice, a security audit, or a guarantee of production suitability.
Request scope review
Your AI may sound convincing. Find out how it allocates.
Submit the system you want evaluated and the decision you need to make. CapitalBench will confirm fit, access method, comparator roster, timing, and fixed scope before you pay.
No payment at submission. Private by default. NDA available before technical access.
Intake form
Request Scope Review
The next step is a fit review followed by a proposed Evaluation Plan and fixed SOW. No payment or system access is requested until scope is agreed.
Procurement
Procurement and billing
Service type Fixed-scope professional model evaluation
Contract Mutual NDA and SOW available
Billing currency USD
Payment Invoice and supported electronic payment methods
Primary contact [email protected]
Security inquiries [email protected]
Contracting legal entity Specified in the SOW before invoice issuance
Business jurisdiction Specified in the SOW before invoice issuance
Billing address Provided on the invoice and SOW
Tax information Provided on the invoice where applicable
Legal contact [email protected]
CapitalBench provides AI-model evaluation and research services. Results apply only to the tested system, version, configuration, inputs, constraints, and evaluation window. Results do not constitute investment advice, a recommendation to trade, certification, regulatory approval, endorsement, or a guarantee of future performance. Short market windows may be affected by noise and unusual events. The applicable SOW governs confidentiality, data handling, permitted use, payment, and delivery.