# RFC-0005 Leaderboard Source: https://club.penguin.ooo/rfcs/rfc-0005-leaderboard/ - Feature Name: `leaderboard` - Start Date: 2026-10-30 - RFC PR: [#23](https://github.com/Prism-Shadow/penguin-harness-community/pull/23) - Updated by: [#25](https://github.com/Prism-Shadow/penguin-harness-community/pull/25), [#27](https://github.com/Prism-Shadow/penguin-harness-community/pull/27) A Leaderboard aggregates one or more [RFC-0002 Evaluation](https://club.penguin.ooo/rfcs/rfc-0002-evaluation/) experiments into statistical charts, identified by ``. Leaderboard files store only experiment ID references and expressions—never precomputed numeric scores. We also use the open-source [PenguinHarness Benchmark](https://github.com/Prism-Shadow/penguin-harness-benchmark) repository to store all Leaderboards. Inspired by [Terminal-Bench (tbench.ai)](https://www.tbench.ai/), a single Leaderboard configuration file drives three chart views: - **Ranking Table (`Table`)**: Sorted by evaluated score from high to low by default and rendered with a horizontal bar chart + number. - **Pareto Frontier (`Pareto`)**: Compares score against resource usage, supporting switching the resource axis in the UI between total `cost` (default), total `duration` (`duration_ms`), or total `token`s. - **Waffle Grid (`Waffle`)**: Directly reads `source` and `score` from `attempt-.yaml` in the referenced experiments to display per-task scores. ## Directory Structure Each Leaderboard is stored in its own `leaderboards//` directory as a single `leaderboard.yaml` file, and its associated experiments are stored under `experiments///`: ```text penguin-harness-benchmark/ ├── leaderboards/ │ └── / │ └── leaderboard.yaml └── experiments/ └── / └── / ``` ## Adding a Leaderboard To add a new Leaderboard to the repository, create a `leaderboards/` directory and add a `leaderboard.yaml` configuration file inside it: ```yaml leaderboard_id: "gdpevo-essential" # Required. Unique leaderboard ID (must match the folder name) status: "draft" # Required. Status: draft when submitted, published after review title: "GDPevo-Essential Leaderboard" # Required. English title title_zh: "GDPevo-Essential 排行榜" # Optional. Chinese title author: # Required. Author and submitter metadata name: "Prism-Shadow" # Required. Author or team name email: "author@example.com" # Optional. Contact email homepage: "https://github.com/Prism-Shadow" # Optional. Author homepage URL values: # Required. List of entry expressions - op: "mean" args: - { experiment_id: "gdpevo-essential-court-clerk-operations-skillopt-deepseek-flash-1017-20261030", phase: "test", metric: "score_mean" } - { experiment_id: "gdpevo-essential-coverage-review-skillopt-deepseek-flash-1017-20261030", phase: "test", metric: "score_mean" } - { experiment_id: "gdpevo-essential-engineering-portfolio-analytics-skillopt-deepseek-flash-1017-20261030", phase: "test", metric: "score_mean" } - { experiment_id: "gdpevo-essential-financial-control-skillopt-deepseek-flash-1017-20261030", phase: "test", metric: "score_mean" } - { experiment_id: "gdpevo-essential-portfolio-management-skillopt-deepseek-flash-1017-20261030", phase: "test", metric: "score_mean" } - { experiment_id: "gdpevo-essential-service-ticket-resolution-skillopt-deepseek-flash-1017-20261030", phase: "test", metric: "score_mean" } - op: "mean" args: - { experiment_id: "gdpevo-essential-court-clerk-operations-baseline-deepseek-flash-1017-20261028", phase: "test", metric: "score_mean" } - { experiment_id: "gdpevo-essential-coverage-review-baseline-deepseek-flash-1017-20261028", phase: "test", metric: "score_mean" } - { experiment_id: "gdpevo-essential-engineering-portfolio-analytics-baseline-deepseek-flash-1017-20261028", phase: "test", metric: "score_mean" } - { experiment_id: "gdpevo-essential-financial-control-baseline-deepseek-flash-1017-20261028", phase: "test", metric: "score_mean" } - { experiment_id: "gdpevo-essential-portfolio-management-baseline-deepseek-flash-1017-20261028", phase: "test", metric: "score_mean" } - { experiment_id: "gdpevo-essential-service-ticket-resolution-baseline-deepseek-flash-1017-20261028", phase: "test", metric: "score_mean" } ``` ### `leaderboard.yaml` Specification Required fields: - `leaderboard_id` — Unique identifier for the Leaderboard. Must match the `` directory name and contain only letters, digits, and hyphens `-`. - `status` — Current status: defaults to `"draft"` upon submission and becomes `"published"` after passing review. - `title` — English display title for the Leaderboard. - `author` — Author metadata object, where `author.name` (author or team name) is required. - `values` — List of expression objects, where each item corresponds to one entry on the Leaderboard: - **Leaf Reference**: `{ experiment_id, phase, metric }` reads the specified metric (defaulting to `"score_mean"`) from `experiments////summary.yaml`. A single-benchmark entry can use a leaf object directly. - **Composite Expression**: Combines multiple leaf references across sub-benchmarks via `op` (`"mean"`, `"sum"`, `"sub"`, `"div"`, or `"weighted_mean"`) and `args`. - **Consistency & Automatic Classification**: **All experiments referenced within a single expression must share the exact same `harness_name`, `model_name`, `algorithm`, and `phase`, differing only in Benchmark**. The renderer automatically extracts `(harness_name, model_name, algorithm, phase)` from `experiments///meta_info.yaml` and `phase` to display and classify each entry. Optional fields: - `title_zh` — Chinese display title for the Leaderboard. - `author.email` — Contact email address of the author. - `author.homepage` — Author homepage link (can default to a GitHub profile URL). ### Score Calculation Formulas Given a linear expression $Y = \sum_{g=1}^{G} w_g X_g$ across $G$ sub-benchmark experiment phases (e.g., $w_g = \frac{1}{G}$ for `op: "mean"`), with per-attempt scores $S_{g,1}, \dots, S_{g,n}$ ($n = \text{n\_attempt}$) and mean $\bar{S}_g$: 1. **Expression Mean ($\bar{Y}$)**: $$\bar{Y} = \sum_{g=1}^{G} w_g \bar{S}_g$$ 2. **Expression Sample Standard Deviation ($\text{std}(Y)$, $n > 1$)**: With $Y_j = \sum_{g=1}^{G} w_g S_{g,j}$: $$\text{std}(Y) = \sqrt{\frac{1}{n - 1} \sum_{j=1}^{n} \left(Y_j - \bar{Y}\right)^2}$$ Or from independent sub-benchmark `score_std` values ($s_g$): $$\text{std}(Y) = \sqrt{\sum_{g=1}^{G} w_g^2 s_g^2}$$