Retrieval-Augmented Generation (RAG)
A model generates an answer and that answer is enhanced by retrieving relevant information from a collection of documents.
Even when a Large Language Model (LLM) has a sufficiently large context window, feeding a huge amount of data into the model all at once can still cause confusion or dilute its attention. In addition, if the original documents contain poorly written content or conflicting viewpoints, they can directly affect the quality of the final output.
Therefore, we need to split large amounts of text into smaller chunks.
There is no single best way to split text into chunks. Common approaches include:
- Character/Token-based splitting: Divide the text into chunks based on a fixed number of characters or tokens.
- Semantic/Structural splitting: Split the text based on words, paragraphs, or even the chapter structure of a document.
Semantic Navigation: Embeddings
After splitting the text into chunks, the next step is to find the chunks that are most relevant to the user’s query. This is where embeddings come into play.
An embedding is a numerical representation of the semantic meaning of text. We pass each chunk to a dedicated Embedding Model—usually a relatively small and fast model—which analyzes its meaning and converts it into a vector in a high-dimensional space. This vector consists of a series of floating-point numbers.
Key characteristics of embeddings include:
- Fixed dimensions: Whether the input is a short phrase of just a few words or an 800-word article, the same embedding model produces a vector with exactly the same number of dimensions. For example, Nomic Embed Text produces a 768-dimensional vector.
- Vector comparison: As long as the vectors have the same dimensions, mathematical algorithms such as cosine similarity can be used to calculate their semantic similarity and identify the text chunks that are closest to the user’s query.
- Embedding models cannot be mixed directly: Different models generate vectors with different dimensions and different semantic spaces, so their vectors cannot be directly compared. Once you switch to a different embedding model, all existing data in the system must be embedded again.
Where the Data Lives: Vector Databases
When the number of chunks grows to thousands or even millions, simply storing them in local JSON files or separate text files creates serious scalability and query-performance issues. This is where a Vector Database becomes useful.
Examples include Chroma, Milvus, and relational databases with vector extensions, such as PostgreSQL with pgvector.
For each chunk, the database needs to store two important pieces of information:
- Embedding (vector): Used for efficient semantic similarity search.
- Original text: Used later when constructing the prompt and providing context to the LLM.
In theory, some techniques can attempt to reconstruct text from vectors. However, in practical applications, storing the original text together with its vector is far more efficient.
The Complete RAG Workflow
The core goal of Retrieval-Augmented Generation (RAG) is very simple:
Find the text chunks most relevant to a question, add them to the prompt, and give the model background information that it would not otherwise have.
【Preparation: Building the Index】
Documents → Split into Chunks → Embeddings → Store in Vector Database
(The database stores both the vectors and the original text.)
【Real-Time Query: When the User Asks a Question】
- Embed the Query: Generate an embedding for the user’s query.
- Compare Vectors: Compare the query vector with the vectors stored in the database.
- Select Top-K Results: Retrieve the top 5 or 10 chunks with the highest similarity scores.
- Retrieve the Original Text: Obtain the original text associated with those vectors.
- Construct the Prompt: Add instructions such as “Please answer the question based on the following information:” and include the retrieved text chunks.
- Generate the Answer: Send the complete prompt, including the retrieved context, to the LLM to generate the final response.
RAG can help reduce AI hallucinations, build private enterprise knowledge bases, improve cost efficiency and productivity, and support data access control.
In the next article, we will explore how to combine Ollama with RAG applications, including PageAssist and AnythingLLM.