How to Run Local Large Language Models (Local LLMs) with Ollama in C#
Large Language Models (LLMs) have become an essential part of modern software development. While cloud-based AI services such as OpenAI and Azure OpenAI are popular, many developers and organizations are also exploring local LLMs for reasons including privacy, offline capability, and reduced API costs.
In this article, we’ll learn how to run a local language model using Ollama and integrate it into a C# (.NET) application with Microsoft.Extensions.AI and OllamaSharp.
Why Run LLMs Locally?
Running an LLM locally offers several advantages:
- Keep sensitive data on your own machine
- No API usage fees
- Work without an Internet connection
- Lower latency for local applications
- Easier experimentation with different open-source models
Popular open-source models supported by Ollama include:
- Gemma
- Llama
- Qwen
- Mistral
- Phi
What is Ollama?
Ollama is a lightweight runtime that allows developers to download and run open-source LLMs locally.
Instead of calling a cloud AI service, your application communicates with a REST API running on your own computer.
C# Application -> Microsoft.Extensions.AI -> OllamaSharp -> Ollama REST API -> Local LLM (Gemma / Llama / Qwen)
Prerequisites
Before writing any code, install Ollama.
Download Ollama from:
After installation, pull a model.
For example:
ollama pull gemma4
Then verify that Ollama is running.
ollama list
By default, Ollama exposes its REST API at:
http://localhost:11434
Create a .NET Console Application
Create a new project.
Install the required NuGet packages.
dotnet add package Microsoft.Extensions.AI
Alternatively, install it by selecting NuGet Package Manager in the project.
What is Microsoft.Extensions.AI?
Microsoft.Extensions.AI is an AI abstraction layer library introduced by Microsoft. It provides unified APIs and interfaces that allow developers to use the same codebase to interact with different large language models, such as Ollama, OpenAI, and Azure OpenAI.
dotnet add package OllamaSharp
Alternatively, install it by selecting NuGet Package Manager in the project.
Calling a Local LLM
The following example sends a prompt to a locally running Gemma model.
Example output:
Notice that we interact with the model through the IChatClient interface rather than calling the REST API directly.
This abstraction makes it easier to switch between different AI providers in the future.
Using ChatMessage
Although sending a plain string works well for simple examples, real AI applications typically exchange structured messages.
Using ChatMessage provides additional context by identifying the role of each participant in the conversation.
What is ChatRole
Each ChatMessage must specify a role. The four most common roles are:
| Role | Description |
| User | User input |
| Assistant | AI response |
| System | Instructions that define the AI’s behavior |
| Tool | Designed to support Function Calling (tool calling) |
Why Use Microsoft.Extensions.AI?
One of the biggest advantages of Microsoft’s AI abstraction library is portability.
Your application depends on the IChatClient interface instead of a vendor-specific implementation.
Today you might use:
- Ollama
Later you could switch to:
- Azure OpenAI
- OpenAI
- GitHub Models
- Other providers
with minimal changes to your application.
This approach follows the same design philosophy used throughout modern .NET development, where applications depend on abstractions rather than concrete implementations.
Conclusion
Running LLMs locally is becoming an increasingly attractive option for developers and enterprises.
With Ollama, OllamaSharp, and Microsoft.Extensions.AI, integrating a local AI model into a C# application requires only a few lines of code.
Whether you’re building an internal chatbot, an AI assistant, or experimenting with Retrieval-Augmented Generation (RAG), local LLMs provide a flexible and cost-effective foundation.