Submit Evaluation

• Target Repository: PenguinHarness Benchmark
• Specification: RFC-0002 Evaluation

Submission Workflow

1. Learn the RFC Data Structure: Read RFC-0002 Evaluation to understand the schema of meta_info.yaml, summary.yaml, and attempt-<n>.yaml under experiments/<leaderboard-id>/<experiment-id>/ (where train-<n> and val-<n> must be followed by a number, and test must never be followed by a number).
2. Open an Issue: Open an Issue in PenguinHarness Benchmark Issues describing the target Benchmark, backbone model, Harness, and RSI algorithm setup.
3. Claim in Issue Comments (Optional): Leave a comment on the Issue to claim the experiment (claiming is optional and prevents duplicate evaluation runs).
4. Submit a PR: Run the evaluation, organize experiments/<leaderboard-id>/<experiment-id>/ according to RFC-0002 (with status: "draft" in meta_info.yaml), and open a Pull Request linked to the Issue.
5. Maintainer Review & Merge: Wait for a maintainer to verify the experiment metadata, score statistics, and cost calculation; once approved, status is updated to "published" and the PR is merged.

Recommended Evaluation Guidelines & Examples

• Recommended Setup for Self-Evolving (RSI) Algorithms: When evaluating self-evolution algorithms, we recommend using the DeepSeek Flash model together with the GDPevo (GDPevo-essential) dataset:
  • GDPevo-essential comprises six independent sub-benchmarks: court-clerk-operations, coverage-review, engineering-portfolio-analytics, financial-control, portfolio-management, and service-ticket-resolution.
  • Because each sub-benchmark is trained and evaluated independently (never combined into a single joint training run), each sub-benchmark must be submitted as its own separate <experiment-id> folder (see penguin-harness-benchmark#18), and then aggregated across the six sub-benchmarks via RFC-0005 Leaderboard.
• Recommended Setup for General Benchmarks: For general Benchmarks (such as penguinharness-benchmark-sec-a through sec-e), we recommend running 3 repeated attempts (n_attempt: 3) with DeepSeek Flash and reporting score_mean, score_std, and optionally pass@k.
• Model Version Requirement: When using models such as deepseek-flash, always include the specific version suffix in model_name inside meta_info.yaml (for example, deepseek-flash-1017).

Format Examples

1. Issue Example

Title: [Evaluation] <experiment-id>: deepseek-flash-1017 + <Algorithm> on GDPevo-essential

### Experiment Setup
- **Experiment ID**: `<experiment-id>`
- **Harness**: `PenguinHarness` (`v0.2.16`)
- **Model**: `deepseek/deepseek-flash-1017` (`thinking_level: high`)
- **RSI Algorithm**: `<plugin-id>` (or none)
- **Train / Validation Data**: `GDPevo-essential` `*-train` splits (e.g., `penguinharness-benchmark-gdpevo-crm-train`)
- **Test Data**: `GDPevo-essential` `*-test` splits (e.g., `penguinharness-benchmark-gdpevo-crm-test`)

2. Issue Claim Comment Example (Optional)

I am claiming this evaluation experiment and plan to submit the results PR by YYYY-MM-DD.

3. PR Example

Title: feat(evaluation): add <experiment-id>

Closes #<issue-number>

### Results Summary
- **Experiment ID**: `<experiment-id>`
- **Model**: `deepseek/deepseek-flash-1017`
- **Test Phase (`test`) Score**: `<score_mean> ± <score_std>`, `pass@k` (optional): `<pass@k>` (`n_attempt: 3`)
- **Single-Experiment Cost / Duration**: `$<cost>` / `<duration_ms> ms`

### Checklist
- [x] Directory layout and YAML fields conform to RFC-0002 Evaluation
- [x] `train-<n>` / `val-<n>` phases have numeric suffixes, `test/` has no numeric suffix, and `status: "draft"` is set in `meta_info.yaml`
- [x] `model_name` includes the explicit version suffix (e.g., `deepseek-flash-1017`) and cost is calculated using MMSP model list prices