I Gave My Search Agent Memory and a Team. Then I Checked If That Was Actually True

Part 2 of my agent build series

Let me set the scene again.

Part 1 shipped. One tool, one loop, no memory — and I was honest about all three limits in the post. A few days later I ended up reading through Google's writeup on how they built Dev Signal, an internal multi-agent system that goes from "scan Reddit for trending questions" to "draft a technical blog post" to "remember your writing style for next time." Root orchestrator, three specialist agents, and a long-term memory layer sitting underneath all of it.

I closed the tab and immediately started drafting a title in my head: "I gave Lumina memory and a team of agents."

Same instinct as last time. Same problem as last time — I hadn't written a line of it yet.

So instead of the announcement post, here's the actual plan — architecture, code, and the parts of Google's pattern I'm not stealing as-is because their own comment section already poked holes in them.

![Lumina multi-agent architecture: a supervisor node routes to search, verify, and synth agents sharing short-term state, with the synth agent reading and writing long-term memory in a vector store] Image description

Infrastructure and Model Setup

Lumina today is Bun + TypeScript. The multi-agent version is a separate Python service sitting next to it — LangGraph for the graph, LangChain for the memory tools, same OpenRouter model underneath so I'm not paying for two providers.

# lumina_agents/graph.py
from langgraph.graph import StateGraph, END
from langchain_openai import ChatOpenAI
from typing import TypedDict, Annotated
import operator

llm = ChatOpenAI(
    model="anthropic/claude-sonnet-4-6",
    base_url="https://openrouter.ai/api/v1",
    temperature=0,
)

class AgentState(TypedDict):
    messages: Annotated[list, operator.add]
    query: str
    sources: list
    verified_claims: list
    final_answer: str

AgentState is the short-term working memory every node reads and writes. It's the thing that replaces the ad-hoc object I was passing through agent-runner.ts by hand in the TS version — LangGraph threads it through the graph for me instead.

Memory Ingestion Logic

The goal isn't "store everything Lumina ever sees." It's: capture the handful of things a user actually corrects or repeats — preferred source recency, terse vs. detailed answers — and make those available on the next session without re-asking.

Long-term Memory

A vector store, written to explicitly, never automatically:

# lumina_agents/memory.py
from langchain_chroma import Chroma
from langchain_openai import OpenAIEmbeddings
from langchain_core.tools import tool

embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
memory_store = Chroma(
    collection_name="lumina_user_memory",
    embedding_function=embeddings,
    persist_directory="./memory_db",
)

@tool
def save_preference(user_id: str, preference: str) -> str:
    """Save an explicit user preference to long-term memory."""
    memory_store.add_texts(
        texts=[preference],
        metadatas=[{"user_id": user_id}],
    )
    return f"Saved: {preference}"

@tool
def load_preferences(user_id: str, query: str, k: int = 3) -> list[str]:
    """Retrieve the most relevant stored preferences for this user."""
    results = memory_store.similarity_search(
        query, k=k, filter={"user_id": user_id}
    )
    return [r.page_content for r in results]

Two tools, same split Google's pattern uses — one for writing, one for reading. The difference is save_preference only fires when a node explicitly calls it after an unambiguous signal ("always sort by recency"), not automatically at the end of every turn. Given the retrieval-precision problem below, I'd rather have less memory that's trustworthy than more memory I have to second-guess.

Short-term Memory

This is just AgentState. No separate service, no persistence — it resets when the graph run ends:

def verify_node(state: AgentState) -> AgentState:
    # reads what search_node already put in state, no DB round-trip
    sources = state["sources"]
    overlap = check_source_overlap(state["query"], sources)
    return {"verified_claims": overlap}

The boundary matters for the same reason it did in Google's writeup: state answers "what happened this run," memory answers "what do I know about this user across every run." Conflating them is how you end up either re-asking the same question every session or leaking one session's context into a completely different conversation.

Specialist 1: Search Agent

This one's not new — it's Lumina's existing web_search tool, ported over as a node instead of the whole loop:

# lumina_agents/agents/search_agent.py
from langchain_core.tools import tool
from lumina_agents.tools.tavily import tavily_search

@tool
def web_search(query: str, depth: str = "basic") -> list[dict]:
    """Search the web and return numbered, deduped results."""
    results = tavily_search(query, depth=depth)
    return [{"index": i + 1, "title": r["title"], "url": r["url"], "content": r["content"]}
            for i, r in enumerate(results)]

def search_node(state: AgentState) -> AgentState:
    response = llm.bind_tools([web_search]).invoke(state["messages"])
    if response.tool_calls:
        results = web_search.invoke(response.tool_calls[0]["args"])
        return {"sources": results}
    return {}

Same citation-numbering rule I learned the hard way in Part 1 applies here — the index gets assigned the moment a result comes back, not whenever the model gets around to referencing it.

Specialist 2: Verify Agent

New. Lumina v0 trusted whatever came back first. This node checks whether the sources actually agree before anything gets synthesized into an answer:

# lumina_agents/agents/verify_agent.py
def verify_node(state: AgentState) -> AgentState:
    sources = state["sources"]
    prompt = f"""Given these sources: {sources}
Identify which claims are supported by 2+ independent sources.
Flag any claim supported by only one source as UNVERIFIED."""
    response = llm.invoke(prompt)
    return {"verified_claims": response.content}

Deliberately dumb prompt, on purpose — the point of this node existing is that "did I check" becomes a visible step in the graph instead of an implicit assumption baked into the synthesis prompt.

Specialist 3: Synth Agent

Writes the final answer, and is the only node that reads from long-term memory — because tone and format preferences only matter at the point where you're producing output, not while you're gathering it:

# lumina_agents/agents/synth_agent.py
def synth_node(state: AgentState, user_id: str) -> AgentState:
    prefs = load_preferences.invoke({"user_id": user_id, "query": state["query"]})
    prompt = f"""Verified findings: {state['verified_claims']}
User preferences on record: {prefs}
Write the final answer, following those preferences where they apply."""
    response = llm.invoke(prompt)
    return {"final_answer": response.content}

The Root Orchestrator

The supervisor is the piece I'm most cautious about. An LLM deciding "route this to search vs. verify vs. synth" is classification with a probabilistic model attached — it will misroute sometimes, and that's not a bug to patch, it's a property to design around. So the routing function checks explicit signal first and only falls back to the model for genuinely ambiguous cases:

# lumina_agents/graph.py (continued)
def route(state: AgentState) -> str:
    if not state.get("sources"):
        return "search"
    if not state.get("verified_claims"):
        return "verify"
    if state.get("verified_claims") and not state.get("final_answer"):
        return "synth"
    return END

graph = StateGraph(AgentState)
graph.add_node("search", search_node)
graph.add_node("verify", verify_node)
graph.add_node("synth", synth_node)
graph.set_entry_point("search")
graph.add_conditional_edges("search", route)
graph.add_conditional_edges("verify", route)
graph.add_conditional_edges("synth", route)

app = graph.compile()

No LLM call in route() at all, for now. State shape decides the next node deterministically. I'll only reach for LLM-based routing if a real case shows up that state shape can't disambiguate — not by default, because it's less code to write upfront.

What I'm Not Copying As-Is

Google's own comment section did the work of stress-testing this pattern before I had to, and two points are worth taking seriously instead of shipping the architecture and hoping:

Embedding similarity isn't a precision instrument. One commenter had actually measured it — semantic search struggled to cleanly separate stylistic preferences that were near-opposite of each other. A load_preferences call can hand back the wrong preference with a confident-looking score. Nothing pulled from memory changes the final answer's substance without at least a recency check behind it.
Unstructured content flowing into long-term memory is a prompt-injection surface. If save_preference ever gets called on text the model read off the open web instead of something the user explicitly typed, that's an open door. For now, only user-authored turns can trigger a memory write — search results never do.

Summary

Three specialists — search, verify, synth — routed by explicit state instead of LLM judgment, with a short-term AgentState for in-run handoffs and a long-term vector store for anything a user explicitly wants remembered across sessions. Not the full Dev Signal pattern, and not meant to be — the parts I kept are the ones that held up under scrutiny; the parts I changed are the ones that didn't.

Code's still at github.com/Saurabhsing21/Lumina — the memory branch goes up once this is running end-to-end, not before.

If you've run embedding-based memory retrieval in production and it held up better than I'm expecting, or you've got a cleaner way to handle explicit-vs-LLM routing — tell me in the comments. I check daily.

#ai #agents #opensource #llm #langgraph #buildinpublic