CLE Paper

Documentation

Task design, evaluation protocol, result definitions, and resources.

Overview

CLE evaluates experience-driven learning in professional workflows. Its 120 tasks span 12 occupational domains and 89 professional roles. Each task forms a five-level ladder, for a total of 600 levels.

Tasks
120
Domains
12
Professional Roles
89
Levels
600
Figure 1. CLE at a glance: 12 domains, 89 professional roles, and completion and improvement results across 17 agent configurations.
Figure 1.CLE at a glance. View Full-Size Figure ↗

Task Structure

Levels are linked through earlier artefacts, tools, decisions, or professional knowledge. Passing a level unlocks the next one. The working environment and agent’s harness state persist as the task progresses.

Tools & Skills
Reuse procedures and tool-use knowledge in later assignments.
Data Dependency
Use information and artefacts from earlier levels as inputs.
Strategy
Apply effective approaches and avoid recurring mistakes.
User Context
Retain preferences and constraints from prior interactions.
Domain Knowledge
Apply professional rules and conventions to new requirements.
Figure 2.Task statistics by level across the 120 tasks: mean checklist items, mean task-brief length in words, and the number of tasks whose specification or rubric at that level refers to files from an earlier level. Bars in the first two rows are scaled to the row’s largest value. Bars in the last row are scaled to all 120 tasks. Data: paper Table 6 and Figure 9.
MeasureL1L2L3L4L5
Checklist items mean18.7422.0725.8330.9036.83
Brief length mean words246271300375466
Tasks referencing earlier-level files of 120None73 60.8%75 62.5%75 62.5%84 70.0%

The environments include 23,981 input files, 77 file extensions, 19 stateful mock applications, and 1,024 APIs. Five complete tasks, with their instructions and input files, are on the Tasks page.

Evaluation Protocol

An evaluation agent checks submitted artefacts and application state against 16,125 human-authored rubric criteria. A level passes only when every checklist item is satisfied.

Figure 3.One level under each submission budget.

Trial 1K = 1

  1. Submit
  2. Rubric Check
  3. No feedback, no second submission
  4. Pass next level
    Fail task ends

Trial 2K = 2

  1. Submit
  2. Rubric Check
  3. if failedDiagnostic Feedback
  4. Revise
  5. Rubric Check
  6. Pass next level
    Fail task ends

Feedback and a revision follow only a failed first submission.

Trial 1 · K = 1

One Submission, No Feedback

One submission per level, without diagnostic feedback.

Trial 2 · K = 2

Feedback and Revision

A failed first submission receives diagnostic feedback and one opportunity to revise.

Feedback identifies unmet rubric categories. During evaluation, the agent does not receive the rubrics or reference answers. The public task showcase is for inspection. Using up the submission budget ends the task, and unreached levels count as failed.

Model parameters stay fixed in the reported experiments. Agents can retain and update their harness state across the task stream.

Metrics

Completion
Levels cleared divided by 600, reported as a percentage.
cleared / 600
L1 to L5
Share of the 120 tasks whose first k levels are all cleared. The denominator includes tasks that never reached level k.
first k cleared / 120
Full-Task Completion
The L5 pass rate: tasks completed end to end, divided by 120.
L5
Δ Completion
Trial 2 completion minus Trial 1 completion, in percentage points (pp).
T2 − T1

Trial differences include both feedback and an additional submission. They do not isolate the causal effect of a memory mechanism or learning strategy.

View the Complete Leaderboard →

Resources & Availability

Project Repository (README)GitHub ↗Hugging Face OrganizationVisit ↗Research ManuscriptPDF ↗Results, Both TrialsCSV ↓Structured Result DataJSON ↓Ten TasksBrowse →

The Tasks page includes thirteen tasks with their original instructions and inspectable input files. Reference answers and evaluation materials require approved access.

Citation

Use the provisional manuscript citation below. Publication details will be updated when a final reference is available.

BibTeX
@misc{cle2026continual,
  title  = {Continual Learning Exam},
  author = {{CLE Team}},
  year   = {2026},
  note   = {Manuscript}
}