Skip to content
Open WebUI Alternative

How Open WebUI RAG works, and the open source path past retrieval

Open WebUI's RAG reads your documents, web pages and knowledge bases and answers questions from them. Here is how to set it up, where retrieval stops, and where Kortix, the open-source AI Operating System, runs the work instead.

Try Kortix

What RAG in Open WebUI does

Open WebUI is a self-hosted chat interface that sits over your models. Its retrieval-augmented generation (RAG) is the step that puts your own material in front of the model before it answers, so the answer is grounded in your documents.

Open WebUI's RAG documentation defines the technique and lists the sources. RAG "is a technique that enhances the conversational capabilities of chatbots by incorporating context from diverse sources," and it "works by retrieving relevant information from a wide range of sources such as local and remote documents, web content, and even multimedia sources like YouTube videos." The retrieved text is then combined with a RAG template and prefixed to the prompt.

Three kinds of source feed that retrieval:

  • Documents you upload to the instance.
  • Web content, pulled in from a URL that starts with #.
  • Knowledge bases, the named collections that group documents so a model can search them.

A knowledge base is the reusable unit. Open WebUI's knowledge-base guide describes it as the place to "ask questions across PDFs, documents, and other files," where you "upload your files, attach the collection to a model or conversation, and ask a question." At chat time, Open WebUI "embeds the user's query with its configured RAG embedding model, searches the external database by vector, and feeds the top matches into the prompt."

Document extraction runs before any of that, turning a file into text the retriever can chunk. Open WebUI's document-extraction page covers "PDFs (both text-based and scanned)," "images containing text," and "handwritten documents." An uploaded image or video is usually kept as-is for a vision model to read, rather than sent through the extraction engine.

How RAG is set up in Open WebUI

Four steps, each from Open WebUI's own documentation.

  1. Open Workspace > Knowledge, click Create, and give the collection a name and a description. This is where a document set is grouped so it can be reused.
  2. Add files to the knowledge base and wait for processing to finish. Extraction runs first, so a scanned PDF, an image with text or a handwritten scan becomes searchable text before it is embedded.
  3. Set the embedding model. Open WebUI embeds the query with its configured RAG embedding model. If the knowledge base points at an external vector database, the vectors in that store must come from the same model, with matching dimensions, or the results are meaningless.
  4. Reference the knowledge base in a chat with #, or attach it to a model in Workspace > Models > Edit, then start a conversation with that model. Attaching a knowledge base scopes the model to those documents; leaving it unscoped lets the model discover everything the user can reach.

Two retrieval modes sit behind the attachment. Focused Retrieval is the default and injects the chunks the query scores as relevant. Full Context injects the complete file with no chunking.

The limit: retrieval answers, it does not do the work

RAG answers a question from documents. It does not edit the document, run the migration the document describes, or send the reply the thread is waiting for. Open WebUI's own guide draws the line at the answer: the AI "searches them by meaning, not just keywords, and cites where it found the answer."

Open WebUI has reached past the chat window in places. Its own repository describes a terminal companion that lets a model "analyze data, run scripts, fix errors, and produce files directly in chat" (github.com/open-webui/open-webui). That companion is separate from the retrieval pipeline, which still returns passages for a model to answer from.

That answer is the same job whether the front end is Open WebUI, AnythingLLM or Onyx: retrieve the passage, hand it to a model, return a sentence. When the ask is finished work, the retrieval step is the wrong tool. Producing a report from the sources, applying a patch to the repository, or drafting and sending the reply is a different layer, and that layer is where Kortix sits.

Kortix is the open-source AI Operating System: your agents, their skills, your company memory and every connector live in one git repo you own, and each session runs on its own cloud computer. An agent reads the sources and produces the work, then lands it as a change request a person reviews before it reaches main. Kortix is the leading open-source alternative to Claude Cowork and ChatGPT Work.

If you are choosing between the two front ends, the Kortix vs Open WebUI page runs the comparison lane by lane. Teams weighing retrieval tools should read Open WebUI vs AnythingLLM, and the alternatives index covers the rest of the field.

02

Where the work lands

kortix docstext
A project is a git repo with kortix.yaml at its root.A session runs one agent in its own sandbox. The sandbox clones the repo and works on a new branch.The agent commits, pushes the session branch, and opens a change request against the project's default branch.A person with the project.gitops.merge permission reviews the change request and merges it into the default branch.
source: https://kortix.com/docs

The path from a session to main, quoted from the Kortix docs.

03

Open WebUI RAG questions

  1. 01

    What documents can you upload to Open WebUI?

    Open WebUI's document-extraction page covers "PDFs (both text-based and scanned)," "images containing text," and "handwritten documents." An uploaded image or video is normally stored as-is for a vision model rather than extracted.

  2. 02

    Does Open WebUI store your files?

    In the built-in pipeline, files live in the instance you run, stored locally or on S3, Google Cloud Storage or Azure Blob Storage (repository). With an external knowledge source, Open WebUI's RAG docs say "no copy of the documents is stored in Open WebUI," and your documents, embeddings and indexing stay in your own store.

  3. 03

    Do you need a separate vector database?

    No. Open WebUI embeds and searches documents itself, and carries ChromaDB and PGVector as officially maintained stores. A separate one is an option: an external knowledge source can point at Qdrant, Milvus or pgvector, a setting the docs still mark experimental.

  4. 04

    Does RAG read a whole document or only parts?

    Focused Retrieval is the default and injects the most relevant chunks. Full Context injects the complete content of a file into every message, with no chunking.

  5. 05

    When is RAG the wrong tool?

    When the ask is a deliverable rather than an answer. Retrieval hands the model a passage to summarize; it does not run the process the document describes or send the reply. Kortix, the open-source AI Operating System, runs agents that produce that work and land it as a change request a person reviews.

Give the agents the documents and the job.

Kortix is open source, and self-host is free.

Try Kortix