Local AI
I run Qwen3.5-35B-A3B locally as part of my homelab. The project is about making a large sparse MoE model useful on constrained hardware by measuring memory movement, PCIe transfers, GPU behavior, and generation speed.
How it works
- Qwen3.5-35B-A3B runs through llama.cpp on a down-tuned RTX 3060
- The sparse MoE architecture reduces active computation, while PCIe transfer is the main bottleneck
- FastAPI streams responses and exposes inference metrics
- Prometheus and Grafana track latency and GPU behavior
- Tool calls can perform bounded infrastructure diagnostics
Current performance
- Approximately 101 generated tokens per second
- Tuned voltage and power profile on the RTX 3060
- PCIe bandwidth is the current limiting factor
- No hosted-model API dependency
Retrieval experiments
I experimented with document chunking, embeddings, Qdrant, semantic retrieval, and cross-encoder reranking. Those components are not running in the current service. I keep them documented as experiments because they established practical familiarity with the RAG pipeline without presenting it as production infrastructure.
Stack
- llama.cpp local inference
- FastAPI and server-sent events
- Prometheus and Grafana
- Linux, Docker, and NVIDIA GPU monitoring
- Experimental: Qdrant, embeddings, and cross-encoder reranking