Local LLM Deployment: Data-Private AI in an SME

Summary: A local LLM (Large Language Model) deployment is the alternative to cloud-based AI services like ChatGPT, Claude, and Gemini — your organisation runs open-source models (Llama 3, Mistral, Qwen, Phi) on its own server. The main motivation for SMEs: data privacy. Customer records, contract text, financial reports — sent to a cloud AI, these carry KVKK and commercial-confidentiality risk. With a local LLM, the data never leaves the office. Hardware: 16 GB RAM is enough for 7B-parameter models (slow on CPU, comfortable on GPU); 70B+ models need a corporate-class GPU server. Tools like Ollama and LM Studio have brought setup down to minutes; with Open WebUI, a ChatGPT-style interface is up in another minute.

The "we should leverage AI" claim is taking over SMEs; but the question "is it safe to put customer data into ChatGPT?" is a legitimate worry. Cloud LLMs come with risks around contractual / non-contractual data use, training-set inclusion, and cross-border transfer. A local LLM removes all of that: the model lives on your server; data stays put. Performance is a touch lower, but privacy is uncontested.

In this article we cover local-LLM rollout at SME scale, hardware / software options, and practical use scenarios. Target audience: IT managers, SMEs handling sensitive data, and decision-makers who say "let's get into AI, but keep privacy intact".

Why a Local LLM?

Cloud-LLM Concerns

• Data usage: how do OpenAI, Anthropic, and Google use your data? Reading the contract is essential
• Training-set inclusion: some models learn from user data
• Cross-border transfer: KVKK Article 9, explicit consent required
• Third-party compliance: picking an EU-aligned provider
• Cost: at heavy use the monthly bill climbs
• API access: stops working during an internet outage

Local-LLM Benefits

• Data doesn't leave the office (KVKK-aligned)
• Works during an internet outage
• Unlimited usage (up to hardware capacity)
• Customisation (fine-tuning) is possible
• A one-off hardware investment

Local-LLM Limits

• Performance is behind the cloud "frontier" models
• Hardware investment required
• Operational responsibility sits with you
• Software updates are manual

Open-Source Models

Open-source LLMs commonly used at SME scale:

Model	Parameters	Size	Turkish support
Llama 3.1 8B	8 billion	~5 GB	Medium
Llama 3.3 70B	70 billion	~40 GB	Good
Mistral 7B	7 billion	~4 GB	Medium
Mixtral 8x7B	47 billion (effective 13B)	~26 GB	Good
Qwen 2.5 (Alibaba)	7B–72B	Varies	Very good
Phi-3 (Microsoft)	3.8B	~2 GB	Medium
Gemma 2 (Google)	9B–27B	Varies	Medium
Cohere Aya	8B–35B	Varies	Very good (multilingual)

Pragmatic SME starting point: Llama 3.1 8B or Qwen 2.5 7B — Turkish output is good enough and the hardware is affordable.

Hardware Requirements

What it takes to run an LLM.

Parameters vs RAM / VRAM

Model size	Minimum RAM (CPU)	GPU VRAM (fast)
3B parameters (Phi-3)	8 GB	4–6 GB
7B parameters (Llama 8B)	16 GB	10–12 GB
13B parameters	32 GB	16–20 GB
30B parameters	64 GB	32–40 GB
70B parameters	128 GB	80+ GB (A100/H100)

CPU vs GPU

• CPU only: slow but works (e.g. 8B model: 5–10 tokens/second)
• Apple M-series: excellent (unified memory) — a MacBook Pro M3 Max can run a 70B model
• NVIDIA RTX 4090 (24 GB): the consumer SME pick, up to 13B
• NVIDIA A100 (40–80 GB): corporate; runs a 70B model comfortably
• NVIDIA H100: premium; overkill at SME scale

Typical SME Setups

Scenario	Hardware	Approximate cost
Individual testing (CPU)	32 GB RAM PC	~30,000 TL
Small team (single GPU)	RTX 4090 + 64 GB RAM	~150,000 TL
Production (single A100)	A100 80 GB server	~500,000 TL
Premium (multi-GPU)	2x A100 server	~1,000,000+ TL

For an SME, RTX 4090 + 64 GB RAM is a sufficient starting point.

The Software Stack

Ollama — the Easiest Start

Ollama is an open-source, cross-platform LLM runtime.

Install (Linux / Mac):

curl -fsSL https://ollama.ai/install.sh | sh
ollama pull llama3.1:8b
ollama run llama3.1:8b

Install (Windows):

• Download the official installer and run it
• PowerShell: ollama pull llama3.1:8b

What you get:

• A single command to download and run a model
• REST API (for application integration)
• Automatic GPU / CPU optimisation
• Multiple models in parallel
• An active community

LM Studio — a Visual UI

LM Studio is a GUI for Windows / Mac / Linux.

• User-friendly, no coding required
• Hugging Face model search / download
• Built-in chat UI
• API server mode
• Ideal on the SME user side

vLLM — Production-Grade Serving

vLLM is high-performance LLM serving.

• High throughput via PagedAttention
• Multi-GPU support
• Built for production
• For higher-demand SME settings

Open WebUI — a ChatGPT-Style Interface

Open WebUI is a chat UI on top of Ollama.

docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway \
  -v open-webui:/app/backend/data \
  --name open-webui --restart always \
  ghcr.io/open-webui/open-webui:main

Open http://localhost:3000 in the browser — a ChatGPT-style interface.

llama.cpp — the Lightest Option

llama.cpp is written in C++ and very lean.

• CPU-optimised
• Quantisation (4-bit, 5-bit, 8-bit)
• Runs even on older hardware
• Command line

Quantisation — Shrinking the Model

Original LLMs are stored at 16-bit float (FP16). Quantisation reduces their size.

Quantisation Levels

Type	Size	Quality	SME fit
FP16 (original)	100%	100%	Production
Q8_0	50%	99%	Excellent
Q5_K_M	35%	97%	Recommended
Q4_K_M	25%	95%	SME standard
Q3_K_M	20%	90%	Quality-limited
Q2_K	15%	80%	Experimentation only

Typical SME pick: Q4_K_M — excellent size / quality balance.

A 70B-parameter model is 140 GB at FP16; at Q4_K_M it's ~40 GB — combined with an RTX 4090 + system RAM, it becomes runnable.

SME Use Cases

Where a local LLM typically pays off in an SME:

1. Customer-Email Assistance

Analyse customer questions, draft responses. Data never leaves the organisation.

2. Contract Summaries

Summarise a new contract via the LLM and highlight risky clauses. Sensitive content stays inside the building.

3. Document Search (RAG)

Embed company documents, let the LLM answer in context (Retrieval-Augmented Generation). Customer data never leaves.

4. Coding Assistance

Help developers — tools like Cursor and Continue support local LLMs.

5. Content Writing

Marketing copy, social posts, blog drafts. Keep brand / customer data out.

6. Call-Centre Summaries

Summarise customer-call transcripts. Sensitive content stays protected.

7. CV Screening (HR)

Analyse candidate CVs, score matches. KVKK alignment satisfied.

RAG (Retrieval-Augmented Generation)

How to make a local LLM use company data it doesn't natively know.

RAG Architecture

[Company documents] → [Embedding model] → [Vector DB]
                                              ↓
                  [User question] → [Vector search]
                                              ↓
                              [Relevant docs + question] → [LLM]
                                              ↓
                                       [Context-grounded answer]

SME RAG Stack

• Embedding model: all-MiniLM-L6-v2, mxbai-embed-large
• Vector DB: Qdrant, Chroma, Weaviate
• Orchestration: LangChain, LlamaIndex
• LLM: Llama 3.1 8B (Ollama)
• UI: Open WebUI, or custom

This stack stands up at SME scale in 1–2 weeks and gives a "ChatGPT" experience over your own documents.

Performance Expectations

Typical LLM performance in an SME environment:

Hardware	8B Q4 tokens/s	70B Q4 tokens/s
Apple M3 Max 64 GB	30–50	7–10
RTX 4090 24 GB	60–100	15–25 (offloading)
A100 80 GB	80–150	60–100
Pure CPU (32 GB DDR5)	5–15	1–3

Human reading speed is ~5 tokens/second, so SME hardware delivers acceptable UX.

The Security Angle

Security considerations for a local-LLM rollout:

LLM Security

• Where did the model come from? (Official Ollama, Hugging Face Verified)
• Is the model signed? Did you verify the hash?
• Could the training set be poisoned?

API Security

• Ollama's port 11434 is open by default — tighten it
• API authentication (Open WebUI adds this)
• Reverse proxy + HTTPS (Nginx, Caddy)

Prompt Injection

• A user may try to get the model to reveal other people's data
• Harden the system prompt
• Output filtering

Audit Log

• Who asked what, what was answered
• May be requested in a KVKK audit
• Open WebUI supports this out of the box

What Yamanlar Bilişim Offers

Our local-LLM support areas at SME scale:

• "Is a local LLM right for us?" assessment
• Hardware-selection advisory
• Ollama / LM Studio / vLLM rollout
• Open WebUI or custom UI deployment
• RAG architecture (vector DB, embeddings)
• Turkish-model fine-tuning
• KVKK compliance documentation
• Annual model-performance review

Frequently Asked Questions

• Low usage, sensitive data: local LLM (hardware amortises)
• Heavy usage, sensitive data: local LLM (clear win)
• Low usage, non-sensitive data: cloud LLM (simplicity)
• Heavy usage, non-sensitive data: cloud + cost controls

At SME scale, a typical ~150,000 TL of hardware drops below cloud costs within 1–2 years under heavy usage.

• Since data doesn't leave the office, "cross-border transfer" isn't an issue
• No third-party data processor (contract chains shorten)
• Data doesn't enter the model's training set
• Audit log stays inside the organisation

You still need information notices, explicit consent, and access controls.

Conclusion

A local LLM deployment lets your SME tap into AI without sacrificing data privacy. Modern tools like Ollama and LM Studio have brought setup down to minutes; an affordable hardware investment like RTX 4090 + 64 GB RAM runs 8B–13B models comfortably. For anything KVKK-sensitive — customer data, contract text, financial reports — putting a local LLM in front of it is the safest path. A RAG architecture makes a ChatGPT-style experience over your company documents perfectly achievable at SME scale.

Yamanlar Bilişim provides local-LLM selection, hardware planning, and RAG architecture services sized to your needs — pointing your AI investment toward a data-privacy-aligned, KVKK-friendly, long-term economical path.