Implementing RAG with AnythingLLM
Large Language Models (LLMs) can answer a wide range of questions. However, when a question involves specific documents or private information, relying solely on the model’s existing knowledge may not provide accurate answers. RAG (Retrieval-Augmented Generation) is a common approach to this problem. It retrieves relevant information from documents and provides it to the LLM as a basis for generating an answer.
AnythingLLM is a tool that allows users to build local AI knowledge bases and RAG applications. It can also work with Ollama to run Large Language Models locally.
In this article, we will demonstrate how to implement RAG with AnythingLLM and Ollama. We will start by installing and configuring AnythingLLM, then set up the LLM, embedding model, and vector database. Finally, we will upload TXT and PDF documents and test how the model answers questions based on the contents of these documents.
1.Download and Install AnythingLLM
First, download and install AnythingLLM.
After the installation is complete, launch AnythingLLM. You will see the initial interface.
2. Initialize and Configure AnythingLLM
First, click Customization → UI Preference on the left-hand side.
In the Display Language field on the right, you can select the interface language for AnythingLLM. In this article, we use English.
Configure the LLM
Next, click LLM under AI Providers on the left-hand side.
Here, you can select the software used to run the LLM as well as the LLM model. In this article, we use Ollama as the LLM provider and Gemma4 as the model.
Configure the Embedding Model
Next, click Embedder under AI Providers.
Here, you can select the Embedding Provider and Embedding Model.
In this article, we use the default AnythingLLM Embedder provided by AnythingLLM, with nomic-embed-text-vt as the embedding model.
Configure the Vector Database
You can find Vector Database above Embedder.
A Vector Database is mainly used to store embeddings. In this article, we use the default LanceDB provided by AnythingLLM.
3. Upload a Plain Text File (TXT) and Test It
After completing the basic configuration, we can start uploading documents.
As shown in the figure below, click the “+” button in the chat toolbar and select the TXT file you want to upload.
In this article, we use the example file “Hua Company.txt.” The contents of the TXT file are shown below.
We ask the LLM:
Who is the CEO of Hua Company?
The model will answer the question based on the contents of the TXT document we just uploaded.
4. Upload a Complex Document (PDF) and Test It
In addition to plain text files, we can also use PDF documents for testing.
First, as shown in the figure below, click the Upload button.
Then, select the PDF file you want to upload.
After selecting the file, you also need to click “Move to Workspace” to actually add the selected PDF document to the workspace.
As shown in the figure below, the PDF has been successfully uploaded to the workspace.
For this test, we use a PDF document containing Lanner Electronics ‘s first-quarter 2026 revenue information.
Next, we test the system by asking:
What was Lanner Electronics ‘s revenue in the first quarter of 2026?
The model will answer the question based on the contents of the PDF document we just uploaded.
This demonstrates how AnythingLLM can allow an LLM to answer questions based on documents uploaded by the user.
5. Conclusion
Through this implementation, we can see how AnythingLLM works with Ollama, embeddings, and a vector database to process uploaded documents and allow an LLM to answer questions based on their contents.