CASE STUDY · LLM SYSTEMS

A production chatbot with no cloud API in sight

An open-source-LLM, multi-modal chatbot — multi-document RAG, multi-agent responses, and chart generation from a fine-tuned LLM at 87% accuracy — designed and deployed on local GPUs.

A local GPU server weaving threads of light into a constellation-like graph of document nodes

At a glance

role
R&D ML Engineer, Ninestars (designed & deployed)
runs on
local GPUs, open-source LLMs
capabilities
multi-doc RAG · multi-agent · graph generation
graph-gen LLM
87% accuracy (fine-tuned)

Problem

Document intelligence raises a question users ask immediately: “can I just talk to my documents?” Answering it with a cloud LLM API is easy; answering it when everything must run on local hardware with open-source models — no data leaving the premises — is an engineering problem across the whole stack: models, retrieval, serving, and GPU budget.

Approach & architecture

I designed and deployed the system end to end on local GPUs:

  • Open-source LLMs as the reasoning core — chosen and served locally rather than via API.
  • Multi-document RAG — retrieval-augmented generation grounded across multiple documents at once.
  • Multi-agent responses — specialized agents composing answers rather than one monolithic prompt.
  • Graph generation — a dedicated LLM, fine-tuned specifically to write chart-generation code, so users get visualizations, not just prose. It reached 87% accuracy.

The hard part

Local-first is a constraint multiplier: every capability (multi-modality, retrieval quality, agent orchestration, chart generation) had to fit open-source models and finite local GPU capacity. Fine-tuning a dedicated model for graph-generation code — instead of hoping a general model gets plotting code right — is the kind of trade that constraint forces, and it’s what made the feature dependable enough to measure at 87%.

Result

A deployed multi-modal chatbot running entirely on local infrastructure: multi-document RAG, multi-agent responses, and reliable chart generation from a fine-tuned 87%-accuracy LLM.

Stack

  • Ollama
  • LLaMA-Factory
  • LangChain
  • FAISS
  • Python
  • local GPU serving

stack mapping from skills inventory