Problem
LLM agents suffer from "Tabula Rasa" syndrome — every domain-specific quirk must be rediscovered through trial and error on each invocation. ERL (Experiential Reflective Learning, arXiv:2603.24639, ICLR 2026 MemAgents Workshop) fixes this by building a persistent pool of structured heuristics distilled from past task trajectories.
Pipeline:
- After a task completes (success or failure), an LLM reflects on the trajectory and extracts "Trigger-Action" heuristics — structured rules like "IF tool X returns error Y under condition Z, THEN try approach W"
- Heuristics are stored in a persistent pool
- For each new task, an LLM decomposes the task into potential sub-tasks, scores each stored heuristic for relevance (considering similarity, diversity of experience coverage, and informativeness), and injects the top-k (paper uses 20) into the agent's system prompt
- Periodically evaluate and prune harmful/stale heuristics
Key empirical findings: heuristics transfer far better than raw trajectories; LLM-based retrieval outperforms both random and embedding-based selection; learning from failures vs. successes impacts different task types differently (failures help more for tool-use, successes help more for reasoning). Achieved 56.1% on Gaia2 (+7.8% over ReAct baseline).
Why it matters for Hermes
Hermes's tqmemory stores notes (lessons, decisions, patterns, handoffs) but these are human-authored or agent-authored at the note level — coarse-grained, narrative. ERL's approach of extracting fine-grained, structured "Trigger-Action" heuristics from every task trajectory is a different and more actionable granularity. The evolution pipeline observes recurring failure patterns (agents over-decomposing simple tasks, premature completions, redundant tool calls) but currently has no automated mechanism to distill these into reusable guidance.
The paper's finding that LLM-based retrieval beats embedding-based selection is important — it means the heuristic pool needs semantic scoring, not just vector similarity, to surface the right guidance.
Proposed Solution
- Heuristic extraction pass. After each task (or pipeline stage) completes, run a reflection prompt that extracts 1–3 structured "Trigger-Action" heuristics from the trajectory. These are distinct from tqmemory notes (reflective) — they are operational rules with explicit triggers and actions.
- Dedicated heuristic store. Store heuristics keyed by trigger conditions (tool name, error type, task type).
- LLM-scored retrieval at task start. At task start, an LLM scores the heuristic pool against the current task description and injects the top-k relevant heuristics into the system prompt.
- Effectiveness tracking and pruning. Periodically evaluate heuristic effectiveness (did tasks that used heuristic X succeed more often?) and prune harmful or stale ones.
Cost mitigation: the paper reports ~40% API cost overhead. Mitigate by only running reflection on failed or borderline tasks, not every task.
Acceptance Criteria
Source
Priority
Priority score: 1.12 (Impact: High, Effort: Medium). Parameter-free (no fine-tuning). Directly automates what Hermes's evolution pipeline currently does manually. The closest academic analog to Hermes's evolution pipeline — but automated, per-task, and persistent.
Dedup note
No existing issue addresses automated heuristic extraction from task trajectories, Trigger-Action heuristic pools, or post-task reflection distillation. tqmemory notes (lessons, decisions, patterns) are reflective/narrative at the note level; ERL operates at the fine-grained operational-rule level with explicit trigger conditions and actions. Distinct from #1696 (SKILL-KD — contrastive distillation from teacher/student trajectory pairs, which produces skill patches, not operational heuristics). ERL is the task-level complement to MUSE-Autoskill's skill-level lifecycle.
Problem
LLM agents suffer from "Tabula Rasa" syndrome — every domain-specific quirk must be rediscovered through trial and error on each invocation. ERL (Experiential Reflective Learning, arXiv:2603.24639, ICLR 2026 MemAgents Workshop) fixes this by building a persistent pool of structured heuristics distilled from past task trajectories.
Pipeline:
Key empirical findings: heuristics transfer far better than raw trajectories; LLM-based retrieval outperforms both random and embedding-based selection; learning from failures vs. successes impacts different task types differently (failures help more for tool-use, successes help more for reasoning). Achieved 56.1% on Gaia2 (+7.8% over ReAct baseline).
Why it matters for Hermes
Hermes's tqmemory stores notes (lessons, decisions, patterns, handoffs) but these are human-authored or agent-authored at the note level — coarse-grained, narrative. ERL's approach of extracting fine-grained, structured "Trigger-Action" heuristics from every task trajectory is a different and more actionable granularity. The evolution pipeline observes recurring failure patterns (agents over-decomposing simple tasks, premature completions, redundant tool calls) but currently has no automated mechanism to distill these into reusable guidance.
The paper's finding that LLM-based retrieval beats embedding-based selection is important — it means the heuristic pool needs semantic scoring, not just vector similarity, to surface the right guidance.
Proposed Solution
Cost mitigation: the paper reports ~40% API cost overhead. Mitigate by only running reflection on failed or borderline tasks, not every task.
Acceptance Criteria
Source
research/2026-08-07.md, finding chore(actions)(deps): bump the actions-minor-patch group across 1 directory with 4 updates #1: ERL: Experiential Reflective Learning.Priority
Priority score: 1.12 (Impact: High, Effort: Medium). Parameter-free (no fine-tuning). Directly automates what Hermes's evolution pipeline currently does manually. The closest academic analog to Hermes's evolution pipeline — but automated, per-task, and persistent.
Dedup note
No existing issue addresses automated heuristic extraction from task trajectories, Trigger-Action heuristic pools, or post-task reflection distillation. tqmemory notes (lessons, decisions, patterns) are reflective/narrative at the note level; ERL operates at the fine-grained operational-rule level with explicit trigger conditions and actions. Distinct from #1696 (SKILL-KD — contrastive distillation from teacher/student trajectory pairs, which produces skill patches, not operational heuristics). ERL is the task-level complement to MUSE-Autoskill's skill-level lifecycle.