The Open-Source AI Stack for Startups — Zero Cost, Full Power

TLDR

• • A production-ready AI stack for a startup costs $0 in software licenses — every layer has a capable open-source option.
• • The total infrastructure cost runs $50-$200/month on commodity cloud (compute + storage only).
• • Six categories to cover: model serving, agents, orchestration, RAG/memory, observability, and deployment.
• • The tradeoff is engineering time vs. cost. For teams with one competent engineer, this stack is manageable and saves $5K-$30K/year in SaaS fees.

Why This Matters Now

Two years ago, "build your own AI stack" meant six months of ML infrastructure work before shipping a single product feature. The open-source ecosystem has compressed that dramatically.

Today you can deploy a production LLM application in a weekend using entirely open-source tools — model serving, agent orchestration, vector storage, observability, the whole stack. The tools are stable, actively maintained, and used at scale by companies you've heard of.

This isn't about avoiding OpenAI or Anthropic's APIs — those are still the right choice for many workloads. It's about owning your infrastructure stack around those APIs and not paying SaaS margins for capabilities that have open-source equivalents.

For a seed-stage startup, the difference between "all SaaS" and "open-source stack" is $5K-$30K per year in vendor fees. For a Series A company running production AI workloads, it's often $50K-$200K. The math is simple once you have a team that can run the stack.

Layer 1: Model Serving

For inference via API (OpenAI/Anthropic BYOK):

No open-source needed here. You're paying per token. The tools in other layers connect to these APIs. Cost: usage-based, typically $0.001-$0.015 per 1K tokens depending on model.

For local/self-hosted models:

Ollama (54K+ stars) is the default choice for running models locally. One command to pull and run Llama 3.3, Mistral, Qwen, or Phi. Expose as an OpenAI-compatible API. Works on Mac, Linux, Windows. Zero configuration.

ollama pull llama3.3
ollama serve  # OpenAI-compatible endpoint at localhost:11434

vLLM (44K+ stars) for production GPU inference. Continuous batching, PagedAttention, 24x higher throughput than naive serving. Runs on a single A10G (24GB VRAM) for 7-13B models at production scale.

Cost: EC2 g5.xlarge (~$1.01/hr for an A10G) handles 50-100 req/min for a 7B model. For most startups, self-hosted inference is cheaper than API calls at volumes above 10M tokens/month.

Layer 2: Agent Frameworks

LangGraph (11K+ stars) for stateful agent workflows. Graph-based orchestration with cycles, branching, and persistent state. The production-ready choice when you need complex multi-step agent logic with controllable execution flow.

CrewAI (29K+ stars) for multi-agent teams. Define roles, assign tasks, coordinate agents. The pattern: specialist agents (researcher, writer, reviewer) working in sequence or parallel. Easier to reason about than raw LangGraph for human-defined workflows.

Pydantic AI (9K+ stars) for type-safe agent development. If you're building agents in Python and care about validation and correctness, Pydantic AI wraps the model layer with type enforcement. Cleaner than LangChain for production code.

For Claude specifically:

Claude Agent SDK — Anthropic's official SDK with full agent support including hooks, tools, and multi-turn context management. Free, well-documented, actively maintained.

Layer 3: Orchestration and Workflow

Prefect (17K+ stars) for workflow orchestration. Schedule and monitor your AI pipelines — data ingestion, embedding jobs, report generation. The open-source version runs locally or on your own infrastructure with a clean UI.

n8n (56K+ stars) for no-code/low-code workflow automation. Connect AI agents to external tools — Slack, Gmail, Notion, GitHub — without writing integration code. Self-hosted with Docker in 5 minutes.

docker run -it --rm --name n8n -p 5678:5678 -v n8n_data:/home/node/.n8n n8nio/n8n

Cost: n8n Community Edition is free forever for self-hosted use. Handles up to ~10K workflow executions/month on a $20/month VPS without issue.

Layer 4: RAG and Memory

Chroma (18K+ stars) for vector storage. Embedded Python library or client-server mode. No infrastructure required for development. Persists to disk. For production workloads under 10M vectors, Chroma on a small VPS outperforms managed vector DBs on cost.

Qdrant (22K+ stars) for production-scale vector search. Rust-based, high throughput, filterable metadata. The choice for production workloads where you need performance guarantees. Docker image, simple REST/gRPC API.

LlamaIndex (39K+ stars) for the RAG pipeline itself — document ingestion, chunking, embedding, retrieval. Handles 50+ document formats out of the box. Pairs with any vector store.

Typical RAG stack for a startup:

Documents → LlamaIndex (parsing + chunking) → OpenAI text-embedding-3-small 
→ Qdrant (storage + retrieval) → LangGraph (agent that calls retrieval) → Claude API

Monthly cost at 100K document chunks: Qdrant on a $10/month VPS, embedding via OpenAI at ~$0.02 per 1M tokens. Total under $15/month for the RAG infrastructure.

Layer 5: Observability

LangFuse (12K+ stars) for LLM observability. Trace every LLM call — input, output, latency, cost, model version. Runs as a Docker compose stack. Essential for debugging production issues and tracking spend.

# docker-compose.yml excerpt
services:
  langfuse-server:
    image: langfuse/langfuse:2
    ports: ["3000:3000"]
    environment:
      DATABASE_URL: postgresql://postgres:postgres@db:5432/langfuse

Prometheus + Grafana for infrastructure metrics. Standard stack, nothing AI-specific. Track model latency distributions, error rates, queue depths, and cost per workflow run.

The observability layer is the one startups most often skip until they have a production incident. Don't. Two hours of setup with LangFuse will pay back in the first debugging session.

Layer 6: Deployment

Coolify (39K+ stars) for self-hosted Heroku/Vercel-style deployments. Deploy Docker containers from git repos on your own VPS. Free alternative to Railway/Render that runs on a $5/month Hetzner server.

Kamal for zero-downtime Docker deployments to any cloud or bare metal. Built by Basecamp, used in production at Hey.com. Simpler than Kubernetes for teams that don't have a dedicated DevOps engineer.

For AI workloads specifically: Modal is worth mentioning as a paid-but-cheap option ($0.0001/GB-second) for serverless GPU functions. Not open-source, but the pricing model fits startup AI workloads better than EC2 reservations.

The Full Stack, Assembled

Here's the complete reference stack with costs:

| Layer | Tool | Stars | Monthly Cost | |-------|------|-------|-------------| | Model inference (local) | Ollama + vLLM | 54K + 44K | $50-150 compute | | Agent framework | LangGraph or CrewAI | 11K + 29K | $0 | | Workflow orchestration | n8n | 56K | $0 (self-hosted) | | Vector storage | Qdrant | 22K | $10 VPS | | RAG pipeline | LlamaIndex | 39K | $0 | | LLM observability | LangFuse | 12K | $0 (self-hosted) | | Deployment | Coolify | 39K | $0 (self-hosted) | | Total | | | $60-160/month |

This excludes the model API cost (OpenAI/Anthropic) if you're using external APIs. For a startup doing 5M tokens/day on Claude Haiku, that's ~$150/month. The total all-in infrastructure cost is $200-$350/month for a production AI application.

Compare to the equivalent SaaS stack: Pinecone ($70-$700/month) + LangChain Plus ($40/month) + Datadog ($200/month) + Railway/Heroku ($50-$200/month) = $360-$1,140/month before you've written a line of product code.

The Real Tradeoff

The cost savings are real. So is the engineering time cost.

Running this stack yourself means owning upgrades, monitoring, incident response, and capacity planning. For a team with zero dedicated infrastructure engineering time, the SaaS options are worth paying for — the alternative is your product engineers debugging Qdrant disk capacity at 2am.

The sweet spot: startups with at least one engineer who's comfortable with Docker, basic Linux administration, and reading GitHub issues. That person can run this stack part-time and save the company real money at early stage.

The order of operations that works: start with SaaS tools while you validate the product. When you hit $10K/month in AI infrastructure spend, audit what you can replace with self-hosted alternatives. Most teams find Qdrant and LangFuse are the first replacements that pay back immediately.

The open-source ecosystem in 2026 has closed the capability gap with the SaaS tools. The choice is now primarily operational: do you have the bandwidth to run the stack, or do you pay the margin to have someone else run it for you?

Both are legitimate answers. The cost of the self-hosted option is no longer "6 months of infrastructure work." It's "a weekend and a Hetzner server."