01
A test of the one thing a second brain must do
A second brain is only as good as what it gives back. Not what it stores: what it can find, and use, the day you ask. That's surprisingly hard to measure, which is why LongMemEval matters.
LongMemEval is a benchmark for long-term memory in chat assistants, published at ICLR 2025 by Di Wu, Hongwei Wang, Wenhao Yu, Yuwei Zhang, Kai-Wei Chang and Dong Yu (the paper). It has 500 questions. Each one sits on top of a long history of earlier conversations between a person and an assistant, and asks about one detail buried somewhere in it.
In its "S" set, the one most memory systems report, each question comes with about 48 earlier chats: roughly 115,000 tokens, or close to 490 messages. The detail you need might be one sentence in one of them.
02
Five kinds of remembering
The questions test five abilities, the same ones you'd want from a person with a good memory:
- A fact you mentioned once. "What's my cat's name?", asked months after you said it in passing.
- Adding up across chats. "How many bikes do I own now?", when you mentioned them one at a time.
- Things that changed. Your record in the volleyball league last month, not the one from spring.
- Dates and time. "How many days between planting the herbs and the first harvest?"
- Knowing when it was never said. An honest "you didn't tell me that" instead of a confident guess.
Some questions also check your preferences and what the assistant itself said earlier, so it can't remember only your side of the conversation.
03
Why it's hard, even for big AI models
You might think a model that can read 100,000 tokens at once solves this: just paste in the whole history. The paper tested exactly that. One of the strongest AI models of the time, handed the full history for every question, got 60.6% right. Given only the few conversations that held the answer, it got 87.0%.
That gap is the point. Memory isn't a big enough window; it's finding the one sentence that matters, knowing which version of a fact is newest, and doing the date math. That's the job of a memory system, and it's what Recallr is built around.
04
How Recallr scores: 86.7%
Recallr answered 104 of 120 questions right, 86.7%. We ran it exactly as it ships: each question's earlier chats were loaded into a fresh account as past chats, then the question was asked in a new chat. Every answer was graded blind, with the benchmark's own answer-check prompts and grader model.
Two honest notes. We ran 120 of the 500 questions, an even 20 from each of the six kinds, from the cleaned version of the set the authors released. And Recallr answered with a fast, low-cost everyday model, one of the AI models it uses.
| System | Score | AI model that answered | Source |
|---|---|---|---|
| Mastra Observational Memory | 94.9% | a small general model | Mastra research, Feb 2026 |
| Hindsight | 91.4% | a larger backbone model | Latimer et al., arXiv, Dec 2025 |
| Recallr | 86.7% | a fast everyday model | This test |
| EmergenceMem | 86.0% | an internal model | Emergence AI blog, June 2025 |
| Supermemory | 85.2% | a large general model | Supermemory research |
| Zep | 71.2% | an older general model | Rasmussen et al., arXiv, Jan 2025 |
| An AI model reading the whole history | 60.6% | an older general model | Wu et al., ICLR 2025 (the benchmark's paper) |
Read it as a range, not a race: every system answered with a different AI model, and the others ran all 500 questions. What it shows is that Recallr, a consumer app, remembers in the same range as the memory systems AI developers build agents on, and far better than a big model reading everything (60.6%).
05
What it remembered
A few of Recallr's right answers, word for word. The people and their earlier chats are the test's own.
"How many bikes do I currently own?"
Four. In your October 2 message you said you'd just bought a new hybrid bike, so you now have a road bike, mountain bike, commuter bike and that hybrid - up from the three you told me about back on February 15.
"How many times have I tried making a Negroni at home since my friend Emma showed me how to make it?"
Ten times - you told me that on Thursday, October 15, 2026. (Back in June you'd said it was five, so you've doubled it since.)
"What is my current highest score in Ticket to Ride?"
132 points - you hit that on Oct 4, 2026, when you mentioned it during the Scythe game-night chat. That beat your earlier 124 from Oct 1.
06
What this means if you use Recallr
Tell it once, and ask months later. Recallr remembers your past chats, everything you save (articles, videos, posts, PDFs, photos) and what you ask it to remember, and it says where an answer came from.
We run the test again whenever a new version of Recallr ships, and the benchmark page always shows the version you're using. If you want to try it on your own memory, Recallr is free to start.