Integrate the SWE-bench software engineering benchmark into penguin-harness-benchmark with standardized container environments and grading scripts (per RFC-0001).
This task is currently in progress:
Integrate FDE-Gym (arXiv:2610.22350 — 95 end-to-end Forward-Deployed Engineer customer delivery tasks across 20 scenarios and 18 sectors evaluating ambiguous requirement clarification, unstated constraint discovery, and live system delivery) into penguin-harness-benchmark following RFC-0001.
Reply /claim under this task's GitHub Issue to claim it:
Build 50 standardized benchmark tasks across five domains (PenguinHarness Benchmark Sec A–E) with unified package manifests and grading rubrics (per RFC-0001).
This task has been completed and merged (+50 EXP settled):
Integrate the 60-task GDPevo Essential dataset (30 continuous task-group training tasks and 30 independent test tasks across 6 business scenarios) into penguin-harness-benchmark (per RFC-0001).
Build biomedical repeated-measure prediction benchmark tasks evaluating patient-level holdout validation, acoustic feature predictive uplift, and conditional uncertainty intervals (per RFC-0001).
Using DeepSeek-V4.1-Flash-1017 as the base model, evaluate Codex before and after Recursive Self-Improvement (RSI) across the six GDPEVO Essential scenarios, submitting full execution traces to the GDPEVO Essential leaderboard (per RFC-0002).
On PenguinHarness, measure GPT-6's multi-round self-evolution gains and generalization across standard RSI methods on the six GDPEVO Essential scenarios, contributing traces to the GDPEVO Essential leaderboard (per RFC-0002).
Evaluate DeepSeek v4.1 across four RSI methods (SkillAdam, SkillOpt, TRACE, OpenSkill) and a shared baseline on all six GDPevo Essential scenarios with three test repetitions (per RFC-0002).
Benchmark DeepSeek Harness, PenguinHarness, Pi, and Codex using DeepSeek v4.1 across Sec A–E benchmarks (600 scored test runs) and publish the cross-harness leaderboard (per RFC-0002 & RFC-0005).
Implement and integrate RSIAgent (arXiv:2609.15364), coordinating Curriculum, Actor, and Verifier agents for broad-then-deep exploration and frozen causal memory.
Implement and integrate Regularized RSI (RRSI, arXiv:2609.24972) with annealed proposer budget, anti-overfitting critic, and token pruner.
Develop a Knowledge Base plugin for PenguinHarness supporting document ingestion, hybrid retrieval, inline citations, and trace-driven knowledge accumulation.
Develop an Ontology plugin for PenguinHarness providing typed entity-relation-action modeling and trace-driven ontology refinement.
Optimize PenguinHarness context compaction to write structured hand-off files before summarization and guide the next context on line-by-line Trace querying.
Build an engaging, high-playability self-evolution case study on PenguinHarness with a fully reproducible environment, pre- and post-evolution execution traces, and step-by-step reproduction instructions.
This link goes to a site outside Shared RSI Community and opens in a new tab:
Scan the QR code with WeChat to join