Background and what did not work
I wish to just experiment. I am using an nVidia 2080Ti on an X58 motherboard with an Intel i7-920 CPU. It is an old and reliable setup.
I do not wish to install apps that alter my operating system. This includes Docker. Additionally, Docker requires further complex setup to avoid running it as root. Downloaded .debs are a security risk and usually block debian distribution upgrades.
FlatHub FlatPak apps are acceptable. I want to install and run AI as a separate, non-privileged user.
I am running the programmes below as a separate user to keep them sandboxed
What did not work
- Fooocus required conflicting versions of Python dependencies.
- Jan.ai produced a blank screen, whether running from a FlatPak or AppInage.
- StabilityMatrix kept crashing.
- Open Web UI in python - needs an older version of Python (3.11)
- Open WebUI desktop - the download link for the AppImage is broken
What does work
Manual download of Ollama
To set up Ollama on your Debian system without using Docker, follow these steps:
- Download Ollama Software:
- cd Downloads
- mkdir ollama
- cd ollama
- wget https://ollama.com/download/ollama-linux-amd64.tar.zst
- tar xf ollama-linux-amd64.tar.zst
- Run Ollama service:
- cd bin/
- ./ollama serve
- Check if Ollama is running by visiting:
- http://localhost:11434
- Download one or more models (run in a new terminal in the ollama/bin directory). These are the models that I have tried:
- ./ollama pull deepseek-r1:7b
- ./ollama pull llama3.2-vision:11b
- ./ollama pull qwen2.5-coder:14b-instruct-q4_K_M (this is too big)
- ./ollama pull qwen2.5-coder:7b-instruct-q4_K_M (in fact, gemma seems better)
- ./ollama pull gemma2:9b (worked better for me for R coding than qwen2.5-coder)
- ./ollama pull gemma4:e2b
- Run model and use command prompt
- ./ollama run deepseek-r1:7b
- Or just to explore the TUI
- ./ollama
- According to Google Gemini, I can ./ollama run llama3.2-vision 'Then, simply drag and drop your image file into the terminal window and type your question.' This needs testing.
- Delete ollama models with rm
- ./ollama rm qwen2.5-coder:14b-instruct-q4_K_M
This method is straightforward and effective for running Ollama on your system without Docker, making it a simple choice compared to other tools that encountered issues.
ThunderAI
ThunderAI extension downloadable from https://services.addons.thunderbird.net/EN-us/thunderbird/addon/thunderai/
Set an environment variable when running ollama server:
- OLLAMA_ORIGINS=moz-extension://* ./ollama serve
Gemma 2 may be a better model for this purpose
- ./ollama pull gemma2:9b
Easy Diffusion
Easy Diffusion has worked. I used its start.sh script.
Alpaca
Alpaca installed from flatpak.Recommended vision-capable model:
- /ollama pull llama3.2-vision:11b
Experimental
- AnythingLLM looks good: https://anythingllm.com/
- Github incognide looks fascinating, but is a .deb
- Page assist: https://github.com/n4ze3m/page-assist?ref=itsfoss.com - no, I want to upload documents.
