Running large language models (LLMs) locally has gained traction for development, privacy, and offline testing. Ollama streamlines this process, letting you run models like Llama 3 or Mistral right on your machine.
By using Podman on Fedora Linux, you can isolate Ollama within a container. This keeps your system tidy and makes it simple to spin up, manage, and tear down your AI development sandbox.
What is Ollama?
Ollama is an open-source framework for running, creating, and sharing large language models. It bundles model weights, configurations, and data into a single management system. Running it in a container sidesteps the hassle of local dependencies, Python environments, or complicated GPU driver setups on your host OS.
Verify or Install Podman
Podman comes pre-installed in Fedora Workstation. If it’s missing, you can easily install it with dnf:
$ sudo dnf install podman -y
On Fedora Silverblue, Podman is built into the immutable base system, so no additional installation is needed.
To confirm the installation and check that everything is working, run:
$ podman --version
Step 1: Create a Persistent Volume for Your Models
LLM weights are enormous—often 4 GB to 40 GB or more, depending on the model. To prevent re-downloading them each time you restart the container, set up a persistent Podman volume to stash them on your host disk:
$ podman volume create ollama_storage
Step 2: Run the Ollama Container
Now, launch the Ollama container. This command pulls the official image, attaches the volume we just created, and maps port 11434 to your host.
$ podman run -d \ -v ollama_storage:/root/.ollama \ -p 11434:11434 \ --name ollama \ ollama/ollama
Note on Hardware Acceleration
The command above runs Ollama on your CPU. If you’re on Fedora Workstation or Silverblue and want to leverage an Nvidia GPU for faster acceleration, install the Nvidia Container Toolkit and add this flag:
--device nvidia.com/gpu=all
Step 3: Download and Run an AI Model
With the container running in the background, you can interact with it via Podman’s exec command. Let’s pull and run Llama 3, a capable, lightweight model ideal for local development:
$ podman exec -it ollama ollama run llama3
The first time you run this, Podman downloads the model weights into your ollama_storage volume. Once done, you’ll land in an interactive prompt:
>>> Send a message (/? for help) >>> Tell me a fun fact about Fedora Linux. Fedora Linux is named after the iconic felt hat worn by the Red Hat shadowman logo! It started as a community project to provide extra packages for Red Hat Linux. >>> To exit the interactive prompt, simply type /bye.
Step 4: Interact with the Local API
Since we forwarded port 11434 to the host, you can also talk to your Ollama instance through its built-in REST API. Just open a terminal and send a curl request:
curl http://localhost:11434/api/generate -d '{
"model": "llama3",
"prompt": "Why use containers?",
"stream": false
}'
This returns a structured JSON response with your answer, making it easy to connect your local model to web apps, scripts, or IDE extensions.
Checking Container Status
To keep an eye on your running AI instance, use standard Podman commands, starting with:
$ podman ps
You can also peek at the logs to verify the API server is listening:
$ podman logs ollama
When your session is over and you want to free up memory, stop the container:
$ podman stop ollama
To completely remove the container, use:
$ podman rm ollama
Note: Your downloaded models are safely stored in the ollama_storage volume and will reattach instantly the next time you launch the container.
Conclusion
Running Ollama with Podman on Fedora Linux or Silverblue gives you a clean, containerized environment for building and testing LLM-powered apps entirely offline. It avoids polluting your host, neatly stores large models in a named volume, and treats your AI stack like any other microservice.
