RAG is an authorization problem: filter before you retrieve

The standard retrieval-augmented generation pipeline is four steps: ingest documents, embed them, retrieve the semantically closest chunks, hand them to a model. It works, and inside a single-user application it is fine.

Point it at a company’s email, files, calendars, and project tracker and it becomes a system that answers questions using documents the person asking is not allowed to read. A chunk being semantically relevant says nothing about whether the reader is authorized. Those are unrelated properties, and the default pipeline only checks one of them.

The connector authentication underneath this has its own post, and the two are related: a delegated token is where a user’s real permissions come from.

This is a design writeup for a permission-aware connector platform I have been building, not a postmortem. The architecture is implemented in parts and I will mark what is settled versus what is still design.

The failure mode in one sentence

Somebody asks “what is the budget for the Henderson project”, the vector search returns the most relevant chunk, and the most relevant chunk is a message from the CEO to the CFO.

Nothing malfunctioned. Retrieval did its job perfectly. The system has no concept that documents have owners.

The tempting fix, and why it is wrong

The obvious repair is to filter after retrieval:

semantic search  ->  take top 10  ->  drop the unauthorized ones  ->  LLM

I have seen this in enough codebases to treat it as the default mistake. Three things are wrong with it.

It silently degrades results. If eight of the top ten are unauthorized, the user gets two, and neither the user nor you can tell whether that is because the answer does not exist or because it was censored on the way out. Quality becomes a function of the asker’s permissions in a way nobody can debug.

It leaks through the side. Result counts, latency, and “no results found” versus “here are two weak matches” are all observable. A determined user learns that a document about the Henderson budget exists and is above their level, which is often the sensitive bit.

And it puts the security control in the last place in the chain, where a refactor deletes it. The filter is a .filter() call somewhere after the ranking. Nothing structurally prevents someone from removing it during a performance pass six months from now.

The correct ordering inverts it:

permission filter  ->  semantic ranking  ->  rerank  ->  LLM context
        ^
        the candidate set never contains anything unauthorized

Retrieval operates on a set that was legal before ranking started. The model never sees a document it should not, because the document was never a candidate.

What that costs you

This is the part usually left out, so let me be direct: permission-first retrieval is harder to make fast.

Vector databases are built to search a large index quickly. Restricting that index per user, where the restriction is a set membership test against a permission model that lives in another system, is not what they optimise for. You are asking for filtered nearest-neighbour search over a filter that changes per request.

Two workable approaches, depending on how your permissions are shaped:

Precomputed access scopes. Resolve each user’s permissions into a compact set of identifiers stored alongside each chunk (team ids, resource ids, a visibility class). The filter becomes a cheap set intersection the vector store can push down into its own index rather than a join across systems.

Two-stage candidates with strict post-validation. Retrieve a wider candidate set with a coarse filter that is cheap and conservative, then validate each survivor properly before it reaches the model. This is only safe if the second stage is authoritative and fails closed. It is the pragmatic option, and it is a performance variant of permission-first retrieval, not a return to filter-afterwards. The difference is that no unauthorized chunk ever reaches the context window, and the coarse filter is not permitted to be the only check.

What every chunk has to carry

If you take one implementable thing from this: ingestion has to preserve far more than text and a vector. The metadata is the security model.

The one people skip is the last-synced timestamp, and it is the one that makes the whole thing auditable. Without it you cannot answer “how out of date was the permission decision this answer was based on”, which is the first question anyone serious will ask.

Revocation is the hard part

Permissions are not static. Someone leaves a team, a document is unshared, a project is locked down. Your index was built when the old permissions were true.

An ingest-once pipeline has no way to learn about any of that. The document sits in the vector store, correctly embedded, permanently retrievable by whoever could read it at ingest time. This is the failure that will not show up in testing and will show up in an audit.

What it takes to handle:

Permission re-synchronisation as a first-class job, not a side effect of content sync. Content changes and access changes are separate events arriving at different times. Tombstones so deleted source objects become non-retrievable immediately rather than at the next full reindex. Versioned access control, so you can tell whether a cached decision predates a change. Cache invalidation keyed on permission changes, since caching a retrieval result caches an authorization decision. And audit logs on sensitive reads, because eventually somebody asks what the system showed to whom.

A useful framing that changed how I built this: a permission is not metadata attached to a document. It is a claim with a timestamp, and claims expire.

Grounding is a security feature

Every generated answer cites its sources: system, object, retrieval time, and a link back.

The obvious reason is trust. The better reason is that attribution is how permission bugs get caught. If an answer arrives with no source, a leak is invisible. If it arrives with a link to a document, the person reading it can tell you they should not be able to see that, and now you have a bug report instead of an incident you never learned about.

Unattributed output is unauditable output.

Reads and writes are different systems

The platform also takes actions in the source systems, and I keep that path structurally separate from retrieval.

A read that returns the wrong document is a leak, which is bad and recoverable. A write that goes to the wrong place sends an email, changes someone’s task, or modifies a shared document, and there is no undo. So writes declare themselves as a capability rather than being implied, use narrower scopes than reads, require confirmation or approval, are idempotent so a retry does not double them, preview their external effect before executing, and are fully audited.

Read connectors run on a schedule. Write actions do not run on a schedule.

Status, honestly

The connector and sync layer, the normalized schema with permission metadata, and the queue-based synchronisation are built. Permission-first retrieval is implemented against the primary sources. Full revocation handling across every connector, and the evaluation work to prove retrieval quality did not suffer from permission-first ordering, are in progress. I have no production leak-rate numbers to show you and I am not going to invent any.

The general lesson

The framing that took me longest to arrive at, and which I would give to anyone building this:

Retrieval quality and retrieval authorization are separate problems. A pipeline that treats authorization as a post-processing step on a quality result has chosen the wrong primitive.

You see the same mistake elsewhere. Search that filters results after ranking. An API that fetches a record and then checks ownership. A dashboard that loads everything and hides rows in the browser. In each case the data left its security boundary before the boundary was consulted, and the only thing standing between that and a breach is a line of code that a refactor can move.

Filter first. Rank what is left.