Llama.cpp

A general purpose llm server that projects can connect to.

Network

Create the ai docker network, applications can access the OpenAI spec server through http://llm-server:8080/v1.

Setup

Start service

docker compose up

Download model(s)

cd /pwspool/software/ai/llama_cpp
sudo wget https://huggingface.co/bartowski/Meta-Llama-3.1-8B-Instruct-GGUF/resolve/main/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf -O Llama-3.1-8B-Instruct-Q4_K_M.gguf