All insightsObsidianLogseq

Who Owns What Your AI Remembers: Personal, Agent and Organizational Memory

Perspective8 min read

An AI that remembers is only safe in a business when it's clear who owns each memory and who can see it. This is how we think about it: three long-term owners, layers that match how the business works, and permissions checked whenever memory is loaded.

Personal and agent memory feeding nested layers of organizational memory, which reach a session through a permission check.

Most businesses treat AI memory as one bucket: the assistant either remembers or it doesn't. That works for one person. We think it breaks the moment more than one person, client or agent is involved.

There are plenty of ways to divide memory, and no single right one. The split we've found most useful is by owner. The person owns how they work, and it's private by default. The agent owns what it has learned about doing its job. The organization owns official knowledge, arranged in layers like company, client and project. Session context is separate again: the short-term working memory of one conversation.

In our experience, storage is rarely the hard part. The hard part is deciding who can see what, how a lesson learned in one place stays there, and how a personal note becomes an official fact.

We split memory four ways

Three kinds of memory are long-term and one is short-term.

  • Personal (long-term). Mostly how one person works: preferences, shortcuts and style. Sometimes drafts of company knowledge they're still working out. Private by default; the person chooses what to share.
  • Agent (long-term). What an AI agent has learned about doing its job: lessons, corrections and worked examples. It makes the agent better at the work over time.
  • Organizational (long-term). The company's official knowledge, arranged as nested layers. The outer layer is the company, with official knowledge and hard rules. Inside it, for firms that serve clients, a layer per client, and inside that, a layer per project. Each inner layer inherits the rules above it, and its access can only narrow.
  • Session (short-term). The working memory of one conversation or task, full or compacted. It's built fresh each time from the three long-term memories.

Most memory should stay where it is

The connections matter more than the categories. We'd keep them few and deliberate:

  • Most personal memory never moves: a person's way of working stays theirs. Only drafts of company knowledge, such as a fact, rule or template someone worked out, can be proposed to organizational memory, and they become official once someone vets them.
  • Agent memory flows into organizational memory only when a person promotes a lesson as general.
  • All three long-term memories feed the session through one permission check, which loads only what the person in that session is allowed to see.
Long-term and short-term memory
How long-term memory feeds a sessionLong-term memory holds personal memory, agent memory and organizational memory. Organizational memory is nested: a project inside a client inside the company. Personal drafts and agent lessons can be proposed to the organization. All long-term memory reaches the short-term session only through a permission check.LONG-TERMSHORT-TERMPERSONALhow one personworksAGENTlessons fromthe workCOMPANYCLIENTPROJECTaccess narrows inwardSESSIONbuilt fresh

Personal memory belongs to the person

We think personal memory should belong to the person, not the company. The company doesn't inherit it, in the same way a file on someone's laptop isn't on the shared drive. People choose what to share and keep the rest private. When someone leaves, what they shared stays; the rest goes with them or is deleted.

It's also where company knowledge usually starts. People work out a fact, rule or template locally, then propose it. Once an owner vets it, it becomes an official item in the organization's memory. Until then it's a draft, and an AI relying on it should say so. One guardrail: a personal draft about a client still carries that client's permissions, so private never becomes a back door.

The best-known version of personal memory today is an Obsidian-style vault: one person's linked Markdown notes, increasingly with an AI agent pointed at them. Logseq, an open-source alternative built around an outline of linked notes, works the same way. Both work well for one person because one person sets the conventions, sees everything and decides what's true.

That doesn't scale beyond them. A shared vault has no permissions finer than the folder, no difference between a draft and an official fact, no layers for clients or projects, and no owner to settle conflicts. So it either stays personal or turns into an unvetted wiki. We'd have personal memory borrow the vault's ease, and leave sharing to the organizational layer.

Organizational memory should follow the shape of the business

We'd arrange organizational knowledge in layers that match how the business actually works: company, then client, then project for a firm that serves clients; company, then team or product, then project for one that doesn't. Each layer inherits the rules and permissions of the layer above it.

Lower layers can narrow access but never widen it. A deal team is a subset of the people on a client. Rules come in two strengths. Hard rules set high up, such as compliance or a client's "never do this", can't be overridden lower down. Preferences, such as tone and format, can, with the most specific layer winning.

Real organizations aren't perfect trees. Projects span clients, and people sit on several teams. Our practical answer is to use the layers to pass rules down, and the access lists a business already keeps, such as CRM teams, matter lists and deal teams, to decide who sees what.

How the layers might look by business type

These are the layers we'd start with. Every business will draw them a little differently.

Business typeTypical layersHard rules that sit high upWhere it goes wrong
Law, accounting and consulting firmsFirm > client > matterPrivilege, confidentiality, ethical wallsOne client's facts surfacing in another client's work
Marketing and creative agenciesAgency > client > brand > campaignBrand guidelines, approval rulesCompeting clients' strategy bleeding across
Private equity and investment firmsFirm > fund > deal or portfolio companyMNPI handling, marketing rules, deal-team confidentialityA live deal's details reaching people outside the deal team
Product and software companiesCompany > product or team > projectSecurity policy, customer-data handlingStale or conflicting knowledge rather than leaks
Operations-heavy businesses such as manufacturing, retail and logisticsCompany > region or site > line or storeSafety procedures, regulatory SOPsThe wrong site's procedure applied, or local fixes never made official
Regulated sectors such as health and financial servicesCompany > department > case or accountPrivacy law, retention, auditRegulated data ending up in layers without the right controls

The pattern we see: firms that serve clients need walls between layers first. Firms that don't need a good path from personal knowledge to official knowledge first.

Agent lessons are the leak nobody watches

Agents get better by remembering what worked: corrections, edge cases and examples of good output. That memory is valuable, and we think it's the most overlooked way information leaks. A lesson an agent learns on one client changes how it works for the next, even if no fact ever crosses over.

Our rule is that lessons stay on the layer where they were learned, and become general only when a person reviews and promotes them.

Permissions belong in retrieval, not in the prompt

This is the point we feel most strongly about. Permissions have to be checked when memory is retrieved, not written into the AI's instructions. If a person can't see a client's files, the AI working for them should never load that client's memory. Telling a model "don't mention client A" isn't a control; not giving it client A is.

Anything built from more than one layer carries every layer's restrictions. A summary drawn from two clients is visible only to people who can see both. Compacted conversation history counts too: it's a summary, and it can carry one client's details into another's session if it isn't labelled.

Most businesses won't start from a clean slate

Everything above assumes a clean slate, where memory is built up as people work. Few organizations start there. Their knowledge already sits in shared drives, wikis, email, CRMs and ticketing systems, each with its own permissions.

We wouldn't move it all before starting. Connect the AI to those systems, leave them as the source, and check each system's own permissions at retrieval, just as you would for memory. Then build organizational memory alongside them. When the same question keeps being answered from a document, or two documents disagree, an owner turns the answer into an official item on the right layer, with a link to its source. Over time, memory holds the vetted facts and rules, and the old systems hold the records behind them.

Two cautions. Permissions in older systems are often wider than anyone intended, so an AI that searches everything a person can technically open will surface things nobody meant them to find. Review access before connecting it. And a document isn't official just because it exists: old drafts and superseded versions should be treated as drafts, not facts.

Get the owners right first

We don't think there's one correct way to divide memory, and the details will differ from business to business. But storing memory is the easy part. Decide what belongs to people, to agents and to the organization; arrange the layers to match how the business works; and check permissions every time memory is loaded. Get those right, and an AI that remembers becomes more useful with every person, client and agent you add, rather than more risky.