Scope-Aware Memory Access Control for Multi-Agent Systems

The problem

When you have one agent, memory is simple: everything the agent knows is in one bucket. But the moment you have two agents working together, you need to answer a question that most tutorials skip: who can see what?

Imagine a research crew: a LiteratureReviewer and a MethodAnalyst working on the same paper. The reviewer discovers a useful finding and stores it in memory. Should the analyst be able to see it? Probably yes — that's shared context. But the reviewer also has notes about its internal reasoning process and dead-end searches. Should the analyst see those too? Probably not — that's noise.

This is the scope problem, and it gets harder with more agents, longer-running tasks, and production reliability requirements.

3-tier memory scope architecture: global scope at top accessible to all agents, shared scope in the middle accessible within a crew, and private scopes at bottom isolated per agent

The three tiers: global (system-wide), shared (crew-wide), private (agent-only). Solid arrows show direct access; dashed arrows show read-only paths.

The three-tier model

In Aegis Memory, I implemented three tiers of memory scope:

Private scope — memories that belong to a single agent. Internal reasoning steps, failed attempts, scratch calculations. Only the owning agent can read or write.

Shared scope — memories visible to all agents in a crew or team. Findings, decisions, agreed-upon context. Any agent in the group can read; writing is controlled by the agent that created the memory.

Global scope — memories visible across all teams and sessions. Organisational knowledge, persistent facts, system-wide configuration. Typically written by supervisor agents or system processes.

Key insight: The hard part isn't implementing the three tiers — it's deciding which tier a memory belongs to at write time. If you get the defaults wrong, agents either over-share (noisy context) or under-share (repeated work).

Implementation

Each memory entry in Aegis has a scope field and an owner_agent_id. The access control check runs at query time:

def can_access(agent_id: str, memory: Memory) -> bool:
    if memory.scope == MemoryScope.GLOBAL:
        return True
    if memory.scope == MemoryScope.SHARED:
        return same_crew(agent_id, memory.owner_agent_id)
    if memory.scope == MemoryScope.PRIVATE:
        return agent_id == memory.owner_agent_id
    return False

Simple enough. But the real complexity is in the query path — when an agent asks for relevant memories, the system needs to search across all accessible scopes efficiently without scanning private memories of other agents.

In pgvector, this means the scope filter has to be part of the vector similarity query, not a post-filter. Post-filtering means you might retrieve 10 results from agent B's private memory, throw them all away, and return nothing — even though there were 10 great shared memories that ranked 11th through 20th.

SELECT content, embedding <=> query_embedding AS distance
FROM memories
WHERE (
    scope = 'global'
    OR (scope = 'shared' AND crew_id = %(crew_id)s)
    OR (scope = 'private' AND agent_id = %(agent_id)s)
)
ORDER BY embedding <=> query_embedding
LIMIT %(k)s;

What broke

The first thing that broke was performance. The compound WHERE clause with OR conditions prevented PostgreSQL from using the HNSW index efficiently. At 10K memories the query was fine. At 100K it was noticeably slower. The fix was a partial index per scope tier — three smaller indexes instead of one large one with a complex filter.

The second thing that broke was defaults. I initially defaulted all memories to private scope. This was "safe" but meant agents in a crew couldn't see each other's work unless the developer explicitly set scope at every write. In practice, nobody did, and multi-agent crews performed worse than single agents because they were doing redundant work.

The fix: default to shared scope within a crew, private scope for internal reasoning traces. This required distinguishing between "observation" memories (findings, facts — shared by default) and "trace" memories (reasoning steps, tool call logs — private by default).

Query flow diagram showing how scope filters are applied at the HNSW index scan level using three partial indexes, one per scope tier, before merging and ranking results

The query path: scope filter builds a WHERE clause, three partial HNSW indexes are scanned in parallel, results are merged and ranked by vector distance. Agent B's private memories are never touched.

What I'd change

If I were rebuilding this from scratch, I'd add a fourth tier: team scope, sitting between shared and global. In a hierarchical multi-agent system (think: Engineering department and Research department under a VP agent), some memories should be visible within a department but not across departments. The current three-tier model doesn't handle this cleanly — you either make it shared (visible to everyone in the org) or private (visible only to the creating agent).

I'd also invest earlier in a memory effectiveness tracker. Right now, scope decisions are made at write time and never revisited. But some private memories turn out to be useful to other agents, and some shared memories are just noise. A voting or feedback mechanism — where agents can flag memories as helpful or unhelpful — would let the system auto-promote useful private memories to shared, and demote noisy shared memories.

The takeaway

Scope-aware memory access control is not optional for production multi-agent systems. Without it, agents either drown in irrelevant context or can't access the information they need. The three-tier model (private / shared / global) is a good starting point, but the real engineering is in the defaults, the query performance, and the feedback loops that let the system improve over time.

If you're building multi-agent memory, start here. And if you want to use this directly, Aegis Memory is open source.