> ## Documentation Index
> Fetch the complete documentation index at: https://docs.agents-squads.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Local LLM Setup with Ollama

> Run AI agent squads locally with Ollama - complete privacy, no API costs

Run your AI agent squads entirely on your machine using Ollama and local LLMs. Zero API costs, complete privacy, works offline.

<Info>
  **Best for:** Privacy-sensitive projects, offline development, reducing API costs, experimenting without usage limits.
</Info>

## Why Local LLMs?

| Benefit       | Description                              |
| ------------- | ---------------------------------------- |
| **Privacy**   | Code never leaves your machine           |
| **Cost**      | Zero API costs after hardware investment |
| **Offline**   | Works without internet connection        |
| **No limits** | No rate limits or quotas                 |
| **Control**   | Choose and customize your models         |

***

## Quick Start

### 1. Install Ollama

<Tabs>
  <Tab title="macOS">
    ```bash theme={null}
    brew install ollama
    ```
  </Tab>

  <Tab title="Linux">
    ```bash theme={null}
    curl -fsSL https://ollama.com/install.sh | sh
    ```
  </Tab>

  <Tab title="Windows">
    Download from [ollama.com/download](https://ollama.com/download)
  </Tab>
</Tabs>

### 2. Pull a Model

```bash theme={null}
# Recommended for coding tasks
ollama pull qwen2.5-coder:14b

# Or for general tasks
ollama pull llama3.2:latest

# List installed models
ollama list
```

### 3. Start Ollama Server

```bash theme={null}
ollama serve
# Server runs at http://localhost:11434
```

### 4. Install Squads CLI

```bash theme={null}
npm install -g squads-cli
squads init
```

### 5. Configure for Local LLM

Create or update your agent to use Ollama:

```markdown theme={null}
# .agents/squads/local/code-reviewer.md

---
provider: ollama
model: qwen2.5-coder:14b
---

# Code Reviewer

## Purpose
Review code changes for bugs, security issues, and improvements.

## Instructions
1. Read the provided code diff
2. Identify potential issues
3. Suggest improvements
4. Format as actionable feedback
```

### 6. Run Your Agent

```bash theme={null}
squads run local/code-reviewer --execute
```

***

## Recommended Models

### For Coding

| Model                 | Size | VRAM | Best For           |
| --------------------- | ---- | ---- | ------------------ |
| `qwen2.5-coder:14b`   | 14B  | 10GB | Complex code tasks |
| `qwen2.5-coder:7b`    | 7B   | 6GB  | General coding     |
| `codellama:13b`       | 13B  | 10GB | Code completion    |
| `deepseek-coder:6.7b` | 6.7B | 5GB  | Budget coding      |

### For General Tasks

| Model          | Size | VRAM | Best For             |
| -------------- | ---- | ---- | -------------------- |
| `llama3.2:8b`  | 8B   | 6GB  | Balanced performance |
| `mistral:7b`   | 7B   | 6GB  | Fast responses       |
| `mixtral:8x7b` | 47B  | 32GB | High quality         |
| `phi3:14b`     | 14B  | 10GB | Reasoning tasks      |

<Tip>
  **Start with `qwen2.5-coder:7b`** if you have 8GB+ VRAM. It offers the best balance of speed and capability for code-related agent tasks.
</Tip>

***

## Configuration Options

### Squad-Level Default

Set Ollama as default for an entire squad:

```yaml theme={null}
# .agents/squads/local/SQUAD.md
---
name: local
mission: Privacy-first local development

providers:
  default: ollama
  model: qwen2.5-coder:7b
---
```

### Agent-Level Override

Override for specific agents:

```markdown theme={null}
---
provider: ollama
model: mixtral:8x7b
temperature: 0.3
---

# Deep Analyzer

Uses larger model for complex analysis tasks.
```

### Environment Variables

```bash theme={null}
# .env
OLLAMA_HOST=http://localhost:11434
OLLAMA_MODEL=qwen2.5-coder:7b
```

***

## LM Studio Alternative

[LM Studio](https://lmstudio.ai/) provides a GUI for running local models with OpenAI-compatible API.

### Setup

1. Download LM Studio from [lmstudio.ai](https://lmstudio.ai/)
2. Download a model (e.g., `TheBloke/CodeLlama-13B-GGUF`)
3. Start the local server (runs on port 1234)

### Configure Squads

```yaml theme={null}
# .agents/squads/local/SQUAD.md
---
providers:
  default: openai  # LM Studio uses OpenAI-compatible API
  base_url: http://localhost:1234/v1
  model: local-model
---
```

***

## Performance Tips

### Hardware Requirements

| Model Size | Minimum VRAM | Recommended |
| ---------- | ------------ | ----------- |
| 7B         | 6GB          | 8GB         |
| 13B        | 10GB         | 12GB        |
| 30B        | 20GB         | 24GB        |
| 70B        | 40GB         | 48GB        |

### Optimization Settings

```bash theme={null}
# Increase context window (uses more VRAM)
ollama run qwen2.5-coder:7b --num-ctx 8192

# Use GPU layers (faster inference)
OLLAMA_NUM_GPU=999 ollama serve
```

### Quantization

Lower precision = faster + less VRAM:

```bash theme={null}
# Q4 quantization (smallest, fastest)
ollama pull qwen2.5-coder:7b-q4_0

# Q8 quantization (balanced)
ollama pull qwen2.5-coder:7b-q8_0
```

***

## Hybrid Setup

Use local LLMs for development, cloud for production:

```yaml theme={null}
# .agents/squads/engineering/SQUAD.md
---
providers:
  # Local for development
  development:
    provider: ollama
    model: qwen2.5-coder:7b

  # Cloud for production
  production:
    provider: anthropic
    model: claude-sonnet-4
---
```

Switch with environment:

```bash theme={null}
# Development (local)
SQUADS_ENV=development squads run engineering

# Production (cloud)
SQUADS_ENV=production squads run engineering
```

***

## Troubleshooting

### Model Not Loading

```bash theme={null}
# Check Ollama is running
curl http://localhost:11434/api/tags

# Restart Ollama
ollama serve
```

### Out of Memory

```bash theme={null}
# Use smaller model
ollama pull qwen2.5-coder:3b

# Or use CPU-only (slower but works)
OLLAMA_NUM_GPU=0 ollama serve
```

### Slow Responses

1. Use quantized models (`-q4_0` suffix)
2. Reduce context window (`--num-ctx 4096`)
3. Ensure GPU acceleration is enabled
4. Close other GPU-intensive applications

***

## Related

<CardGroup cols={2}>
  <Card title="Multi-LLM Usage" icon="layer-group" href="/guides/multi-llm">
    Mix local and cloud providers
  </Card>

  <Card title="Token Economics" icon="coins" href="/guides/token-economics">
    Compare local vs cloud costs
  </Card>
</CardGroup>