Memory Integration

[noctuid]

To upvote this issue, give it a thumbs up. See this list for the most upvoted issues.

Memory

I think one of the biggest features missing from eca is memory integration. Hermes agent, for example, has a memory plugin system that allows adding different memory providers.

I think these are the most notable memory systems that are open source and can run locally:

• Hindsight
• Honcho
• OpenViking

Out of these, Hindsight stands out to me for these reasons:

• Hindsight is #1 on BEAM (a much more difficult benchmark than older ones like locomo and longmemeval)
• hindsight's website is mostly documentation and only has a small link to their cloud option vs. honcho where it's not even immediately obvious there is a non-paid option (also not clear to me if there is any functionality only in the paid option)
• hindsight is extremely easy to setup locally, just run uvx hindsight-embed@latest (no complex installation or docker setup)
• nice dashboard

Memory Best Practices (especially for Hindsight)

I don't have time at the moment to do this for myself (might eventually), but if someone else decides they want to add memory support, I can provide some information on best practices because hermes' implementation is currently flawed.

• Should support ambient store/recall and manual tools for store/recall (it should be configurable which are enabled)
• Ambient store should automatically store every turn in the memory system. Ideally it is configurable when the LLM will actually run (e.g. every n turns or only on session switch/close)
• Ambient recall should be ephemerally injected just before sending the user message to the LLM. It should be given the current turn's user message (up to some char limit). Hermes currently prefetches based on the previous turn's user message, but this can cause irrelevant memories to surface and potentially confuse the agent. Hindsight's recall is low latency compared to LLM API calls, so I do not believe there is a use case for this
• Forking and resuming are edge cases that need to be taken into account immediately. ECA's rollback might be something to consider how to interact with.

Hindsight specific:

• Hindsight has a best practices guide
• Most notable best practices:
  • Don't pre-summarize
  • context field should be set to describe the nature and source of the content. The way I'm doing this in pi is by having a configurable prefix and a truncated session title, which will be the manually set title or part of the first message, e.g. "pi: research hindsight"
  • Sessions should use their stable session id as the document_id. Hindsight now has an update_mode=append that can be used to add the latest turn to an existing document.
  • timestamp - can be session start time
• tags - used for filtering at recall time; For automatically retained information, I'm currently tagging with parent:<parent id>, session:<session id>, harness:pi (user configurable constants tag), cwd:<cwd>
• Forking/resume can be handled if session id is unique for forks by only appending new turns to the document with that session id (if you retain full document of a fork, you will get duplicate information stored)

I have a pi plugin for hindsight that also:

• After every message queues to disk and then sends them as json strings to hindsight (currently hindsight doesn't have a way to delay processing; queueing to disk is also useful if hindsight is down, as can flush later)
• Allows pruning certain fields before sending into hindsight (may or may not be a good idea, but it reduces token usage)
• Allows upserting the entirety of old sessions to hindsight (to seed with existing memories instead of needing to start fresh; also allows changing pruning settings or reseed after changing retain mission)

The configuration for how to prune session messages currently looks like this:

{
  "retainContent": {
    "assistant": ["text", "thinking", "toolCall"],
    "user": ["text"],
    // remove all tool results and only include user messages and assistant reasoning
    "toolResult": []
  },
  "strip": {
    // assumes pi's /tree is not being used so messages are ordered and ids not relevant
    "topLevel": ["type", "id", "parentId"],
    "message": ["api", "provider", "model", "usage", "cost", "stopReason", "timestamp", "responseId"]
  }
}

I'm not sure at this point whether or not it makes sense to include full code reads/edits. Hindsight has a "retain mission" which is a prompt you can use to determine what actually gets stored. Currently I am more interested in storing memories about architectural decisions, library/linter/testing preferences, etc. I'm just totally pruning tools for token savings. The alternative would be to just send everything and let the retain mission determine what it needs to extract.

[ericdallo]

I will take a closer look later on this, but would love to hear @zikajk opinion here meanwhile

[zikajk]

@ericdallo
Actually, just today I was checking out Hindsight and thinking I really need to give it a shot.

It’s an area I want to explore more deeply. Theoretically, it’s no problem to implement basic vector and full-text search in CLJ, but the question is whether it wouldn’t be more worth it to have a solid integration with Hindsight or something else. So I don't have a firm opinion yet, but it’s something I’ve been wanting to dive into soon. :-)

[ericdallo]

nice, your help on that subject would be really appreciated @zikajk ! thanks

[zikajk]

@noctuid Have you tried using hindsight as an MCP? Does it make sense? It occurred to me that the rollback could be handled via a hook that detects what was rolled back and then sends an instruction to the LLM to remove set of items from memory. Since that’s fairly complicated, it could be offered as an ECA plugin.

[noctuid]

You could use it as an mcp, but it only makes sense for manual retain, recall, and reflect. The mcp also has a bunch of tools I would normally have disabled (e.g. bank management doesn't need to be done regularly, and I definitely wouldn't want to give the LLM a delete bank tool).

The most important functionality I think is automatic retain/recall. The manual tools make sense in some cases if the agent decides something is important and needs to be stored or remembered. You could use the mcp for those tools, but for automatic retain/recall, I'm using the hindsight API (through their typescript library). On message end, I queue a modified message to be appended to the document for the current session. On session end or session switch (and maybe I'll add compact), I actually call the retain API.

The way hindsight works, I don't think you can easily remove specific parts of a document. Instead of removing you would upsert the full session information for that document id with the rolled back information removed. The other approach I've been thinking about taking in pi for this is to just not remove anything and include the rollback as extra information. In pi at least, every message has an id and parent id, so if I /tree back to an earlier message, the latest message will have a parent id from an older message. With the way hindsight chunking works (might be split just before rollback), it may not be so straightforward for the LLM to notice this. I might insert an explicit rollback/tree message into the queue. It does seem like potentially useful to still keep information here (what happened and why did we have to rollback? answer could be a useful memory).