Streaming responses

Recall memory before a provider stream, deliver tokens immediately, and ingest only the completed exchange afterward.

Estimated time: 10 minutes

Level: Intermediate

Overview

Streaming changes response delivery, not memory semantics. Recall must finish before the provider request, and ingestion should use the complete assistant text after the stream closes.

When to use this

First-token latency matters to the user experience.

The application already controls a provider streaming API.

Memory recall should still influence the complete answer.

Completed exchanges can be ingested after delivery.

Streaming lifecycle

Recall first

Start provider stream

Collect final text

Ingest completed turn

Step 1: Recall before streaming

recall.ts

const memory = await agent.recall(userMessage, {
  tokenBudget: 1000,
  limit: 6,
});

Step 2: Stream and collect

stream.ts

let completed = "";
const stream = provider.stream({
  system: memory.systemPrompt,
  message: userMessage,
});

for await (const chunk of stream) {
  completed += chunk;
  sendToClient(chunk);
}

Step 3: Ingest the completed exchange

Use the lower-level MemoGrafter API when the application owns the stream and explicit session ID. Do not ingest partial chunks as separate assistant turns.

ingest.ts

await memo.ingest(
  [
    { role: "user", content: userMessage },
    { role: "assistant", content: completed },
  ],
  sessionId,
);

Full example

streaming-handler.ts

const memory = await memo.graftByRelevance(sessionId, userMessage, {
  topK: 4,
});

let assistantText = "";
for await (const chunk of provider.stream({
  system: memory.systemPrompt,
  message: userMessage,
})) {
  assistantText += chunk;
  response.write(chunk);
}

await memo.enqueueIngest(
  [
    { role: "user", content: userMessage },
    { role: "assistant", content: assistantText },
  ],
  sessionId,
);
response.end();

Best practices

Start the provider stream only after memory selection is complete.

Collect exactly the text delivered to the user.

Define whether interrupted streams are discarded or stored with explicit status metadata.

Use queue mode when post-stream extraction would delay request completion.

Keep provider streaming and MemoGrafter execution on the server.

Common mistakes

Ingesting every token or chunk as a separate message.

Starting the stream before recall finishes.

Calling invoke() and separately streaming the same response.

Dropping the final assistant text before ingestion.

Assuming an accepted queue job is already visible to recall.