Workflow
workflow/README.md
All community contributions follow a unified collaboration workflow:
- Read the Specification: Review the corresponding RFC specification or repository guide to understand the required directory layout and schema.
- Open an Issue: Check existing Issues or open a new Issue in the target repository.
- Claim in Issue Comments (Optional): Leave a comment on the Issue to claim the task (claiming is optional and primarily prevents duplicate work).
- Submit a PR: Open a Pull Request linking the Issue once the work is complete.
- Maintainer Review & Merge: Wait for a maintainer to review and merge the PR into the main branch.
Recommended Evaluation & Testing Setup
To keep experimental costs low and results directly comparable across the community, we recommend the following guidelines:
- Submit Benchmark: When verifying a newly formatted Benchmark, we recommend running repeated attempts (e.g.,
n_attempt: 3) with the DeepSeek Flash model, optionally reportingpass@k, following the built-in examplepenguinharness-benchmark-sec-a. - Submit Evaluation: For self-evolving (RSI) algorithms, we recommend evaluating with the DeepSeek Flash model on the
GDPevo-essentialdataset (covering six domain task groups—CRM, ERP, Finance, Healthcare, Legal, and DataAnalysis—training on each group's*-trainsplit, evaluating on its paired*-testsplit, and averaging across task groups). - Submit Algorithm (Plugin): Keep the Plugin implementation decoupled from specific datasets (containing only the algorithm method), and test it with DeepSeek Flash on
GDPevo-essentialafter implementation to verify whether it improves over the baseline. - Model Version Requirement: When using models such as
deepseek-flash, always include the explicit version suffix inmodel_name(for example,deepseek-flash-1017).
Contribution Tracks
| Track | Target Repository | Specification | Workflow Guide |
|---|---|---|---|
| 01 Submit Benchmark | penguin-harness-benchmark | RFC-0001 Benchmark | Submit Benchmark |
| 02 Submit Evaluation | penguin-harness-benchmark | RFC-0002 Evaluation | Submit Evaluation |
| 03 Submit Algorithm | penguin-harness-plugins | RFC-0003 Plugin | Submit Algorithm |
| 04 Contribute to PenguinHarness | penguin-harness | — | Contribute to PenguinHarness |
| 05 Contribute to Awesome RSI | awesome-rsi | — | Contribute to Awesome RSI |