llm-serving

Here are 6 public repositories matching this topic...

aws-samples / sample-genai-on-eks-starter-kit

A comprehensive toolkit for deploying production-ready Generative AI infrastructure on Amazon EKS. Includes pre-configured components for: 🚀 AI Gateway (LiteLLM) 🤖 LLM Serving (vLLM, SGLang, Ollama) 📊 Vector Databases, 🔍 Embedding Models (TEI) 📈 Observability (Langfuse, Phoenix) etc. Fast-track your GenAI deployment with Kubernetes

kubernetes aws terraform ai-agents amazon-eks ai-platform vector-database ai-engineering generative-ai llmops llm-serving gpu-inference vllm llm-inference langfuse ai-gateway agentic-ai mcp-server nvidia-dynamo

Updated Sep 19, 2026
JavaScript

lora-sys / nano-vllm-interactive-guide

Star

面向开发者与初学者的 nano-vLLM 交互式源码教程 - 通过 13 个 HTML 互动实验 + 13 章中文教程理解大模型推理引擎

tutorial chinese interactive-tutorial kv-cache llm-serving vllm llm-inference pagedattention continuous-batching nano-vllm

Updated Aug 23, 2026
JavaScript

AmirrezaFarnamTaheri / qwen2api

Star

FastAPI-based OpenAI-compatible API server and streaming proxy for local Qwen2 models.

python inference fastapi openai-api llm-serving qwen2

Updated Aug 6, 2026
JavaScript

elinx / llm-mem-calculator

Star

Interactive KV cache memory calculator for LLMs — supports MLA, GQA, hybrid attention, sliding window, and linear attention architectures. Estimate GPU memory for serving any model at any context length.

calculator gpu-memory kv-cache llm llm-serving vllm