Obinka Research / Approach
Make the change and its evidence inspectable.
Research into self improvement needs a clear account of the system, the change and the test. The approach below describes how we intend to frame that work. It is a proposed standard, not a statement that these controls are already implemented.
The proposed method
-
01
Begin with a defined task
Set out the task, the baseline and what an acceptable outcome would be. State the conditions under which the result will be judged.
-
02
Name what can change
A prompt, a tool, a harness and a model's weights are different objects of change. Explain which is involved and which remains fixed.
-
03
Keep a record of the search
Record the candidates, the feedback used to select them and the resources spent. Failed attempts help explain both the result and the cost of reaching it.
-
04
Assess beyond the selection task
Where the question concerns transfer, assess it on tasks that did not guide candidate selection. Explain what the system could see and how repeated use of feedback was handled.
-
05
Report the limits
Publish the scope of the comparison, relevant uncertainty and the conditions in which the finding may not apply.
-
06
Connect the finding to the question
An experiment may support a narrow result while leaving the wider research question open. State both clearly.
What do we mean by recursive self improvement?
For this work, the recursive part concerns a system changing the methods it uses to make further improvements. The exact mechanism matters. We want each study to say what changed, how it affected later work and how that effect was tested.
We do not assume that each successive change will help, or that progress will continue without limit. Those would be claims requiring their own evidence.
What a research record should tell you
Each published record should carry the following information.
- The question and scope
- The system and baseline
- The permitted change
- The task and evaluation method
- The result and its uncertainty
- Resource use and failed attempts
- Limitations
- Authorship, versions and available artefacts
Further reading
Primary work that informs the agenda
External research, attributed to its authors. These works are not Obinka results, and citing them does not imply endorsement of this site's agenda by their authors.
-
Self-Taught Optimizer (STOP): Recursively Self-Improving Code Generation
An example of improving the scaffold that calls a language model, with the underlying models unchanged.
-
Darwin Gödel Machine: Open-Ended Evolution of Self-Improving Agents
An agent-code self-modification study with a candidate archive and empirical evaluation on coding tasks.
-
The Ladder: A Reliable Leaderboard for Machine Learning Competitions
Shows how repeated evaluation against a leaderboard can overfit the held-out data behind it.
-
AI Agents That Matter
Identifies evaluation shortcomings in agent work, including narrow comparisons, inadequate holdouts and reproducibility.