Generate, train, and validate small specialist models. One YAML, one command.
Small specialist models handle the high-volume, repetitive tasks in your AI workflows (classification, routing, extraction) so your LLMs can focus on what they're best at. Slemify automates the path to a validated, production-ready model: a CPU-trained encoder classifier for routing and labeling, or a stock generative SLM (GGUF) served on CPU and grounded by RAG for free-form reasoning. How you deploy that model is up to you.
slemify deploy --config expert.yamlSlemify picks the right model family from project.task:
task: generation— a causal LM served stock on CPU (GGUF/llama.cpp): Slemify downloads the base model, converts it to GGUF, and quantizes it. No fine-tuning; knowledge comes from RAG at serving time. For reasoning and free-form output.task: classification— a frozen encoder + a lightweight head, trained and served entirely on CPU in seconds. For routing, intent, and labeling.task: scoring— the same encoder-head family with a regression head, trained and served on CPU. Returns a single number in [0,1]. For risk/quality/confidence guardrails.task: extraction— a CPU-trained token tagger that pulls typed entity spans out of free-form text. Returns a list of{type, text}spans. For entity/field extraction from tickets, logs, and messages.task: embedding— a domain-tuned text embedding model, contrastively fine-tuned and served on CPU (ONNX). Returns a vector. For retrieval (RAG) over your own corpus.
Not every task needs a frontier model. Most agentic AI systems have "hot spots". repetitive sub-tasks that run thousands of times a day with the same pattern. These are ideal for a specialized SLM:
| Task Type | Example | Why SLM |
|---|---|---|
| Classification | Alert triage, intent routing, document categorization | Same pattern, different inputs. Fast, predictable output. |
| Scoring | Risk/quality/confidence guardrails on a config, answer, or request | One number in [0,1] decides auto-approve vs escalate. Cheap on every request. |
| Extraction | Pull structured fields from logs, invoices, clinical notes | Rigid output schema. Doesn't need world knowledge. |
| Routing | Pick which tool/API/agent handles a request | Binary or multi-class decision. Sub-100ms matters. |
| Validation | Safety checks, compliance gates, format verification | Rule-based logic baked into weights. Runs on every request. |
The criteria: high repetition, low semantic variation, structured output. If the task looks the same every time with different inputs, an SLM can do it faster and cheaper than a general-purpose LLM. often with higher accuracy for that specific task.
Slemify doesn't replace LLMs. It adds a fast, cheap layer alongside them.
[Request] → [SLM Router] → high confidence → [SLM Result] → done (50ms, $0)
→ low confidence → [LLM Fallback] → done (3s, $0.01)
The inference endpoint exposes an OpenAI-compatible API (/v1/chat/completions). Any agent, orchestrator, or application can call it directly via HTTP. Set llm_endpoint in your config to any OpenAI-compatible API (vLLM, llama.cpp, Bedrock proxy) for LLM fallback. The SLM handles 70-90% of requests at fixed cost. The LLM handles the rest.
expert.yaml → [DATA] → [TRAINING] → [SERVING + VALIDATION]
│ │ │
Ingest + CPU train, or Deploy model,
Synthetic convert+quantize run eval report,
via Bedrock (generation) generate HTML
- Data. Ingests your raw data from S3. For trained tasks, Bedrock generates synthetic training pairs from your source content and you verify them before training. Generation is served stock, so it skips synthetic data.
- Training. Encoder tasks fit a head or contrastively tune an encoder on CPU in seconds to minutes. Generation has nothing to fine-tune, so this stage downloads the base model, converts it to GGUF, and quantizes it on CPU. Either way the output is uploaded to S3.
- Serving + Validation. Deploys the model on a live endpoint, runs the evaluation dataset through it, and generates an HTML report with accuracy, latency, and cost projections.
The output is a GGUF model file in S3 and a production readiness report. The serving deployment that Slemify creates is production-quality and serves as a reference for your own infrastructure. You can use it as-is, adapt it, or serve the GGUF with any compatible runtime (llama.cpp, vLLM, Ollama). See the Serving deep dive for deployment guidance and best practices.
- EKS cluster with Karpenter
- S3 bucket for data and artifacts
- AWS credentials with Bedrock access
kubectlconfigured for your cluster
apiVersion: slemify/v1
project:
name: k8s-autoscaling-triage
task: classification
domain: >
Classify Kubernetes autoscaling support queries into a routing
category. Each message is classified into exactly one category:
karpenter_config, keda_config, hpa_config, pdb_disruption,
spot_interruption, multi_resource, or noise for off-topic messages.
labels:
routing:
- karpenter_config
- keda_config
- hpa_config
- pdb_disruption
- spot_interruption
- multi_resource
- noise
model:
base: "" # encoder model ID (a text encoder for classification)
head: logistic # classifier head: logistic | linear | mlp
data:
bucket: slemify-data
path: k8s-autoscaling/data/
sources:
- path: queries/
type: raw
synthetic:
model: eu.anthropic.claude-sonnet-4-6
pairs: 1200
evaluation:
model: eu.anthropic.claude-sonnet-4-6
pairs: 150
sources:
- path: eval-queries/
type: raw
training:
spot: trueaws s3 sync ./data/queries s3://slemify-data/k8s-autoscaling/data/queries/
aws s3 sync ./data/eval-queries s3://slemify-data/k8s-autoscaling/data/eval-queries/slemify deploy --config expert.yamlSlemify handles data processing, synthetic pair generation, training, quantization, and validation. The resulting GGUF model is uploaded to S3. You then deploy it in your own infrastructure using the reference deployment as a starting point.
slemify report --config expert.yamlDownloads the HTML report from S3 and opens it in your browser. The report includes accuracy metrics, latency benchmarks, SLM vs LLM comparison, and cost projections.
| Task Type | Training Examples | Notes |
|---|---|---|
| Classification (routing, triage) | 200-500 | Binary or multi-class. Clear categories. |
| Scoring (risk, quality, confidence) | 500-1,200 | Regression target in [0,1]. Spread examples across the full range. |
| Extraction (fields from text) | 500-1,000 | More examples = better edge case coverage. |
These apply to the trained (encoder-family) tasks. task: generation is served stock and grounded by RAG, so it needs no training data. Quality matters more than quantity. 500 well-curated instruction-response pairs beat 10,000 noisy ones. Bedrock generates synthetic examples from your source data, so you don't need to write them all by hand.
| Item | Cost |
|---|---|
| Model prep (CPU: encoder train, or download + convert + quantize for generation) | <~$1 |
| Synthetic data for trained tasks (Bedrock) | ~$10-50 |
| Total to produce a model | ~$10-50 |
Generation no longer uses a GPU: it is downloaded, converted to GGUF, and quantized on CPU, so it has no training cost and no synthetic-data cost. Inference cost depends on how you deploy. The reference deployment (llama.cpp on CPU Spot) runs at ~$117/mo per replica. Throughput scales linearly: 3 replicas = 3x throughput at 3x cost. No rate limits, no per-token charges. See the Serving deep dive for cost comparisons across CPU, GPU, and LLM API options.
This isn't only a $/token argument. Many production Kubernetes clusters report single-digit percent GPU utilization, because latency-insensitive and structurally simple work (routing, classification, validation, embedding) ends up parked on the same expensive pool as the generation workloads that actually need it. Moving that work to CPU isn't just cheaper per call, it frees up GPU capacity for the model that genuinely needs it. For the four encoder-family tasks (classification, scoring, extraction, embedding), there's effectively no volume threshold to clear: CPU training and serving are cheap at any scale, so the usual self-hosting break-even math (which only favors self-hosting past a real volume threshold, often millions of tokens/day) doesn't apply — that math is specific to serving a generative model, and Slemify sidesteps it by not fine-tuning generation in the first place.
- K8s Autoscaling Auditor. Tiered SLM system: a triage classifier routes queries, an 8B auditor produces structured reasoning about Karpenter/KEDA/HPA misconfigurations
- K8s Autoscaling Risk Scorer. A
task: scoringencoder-head model that rates a config change's operational risk 0.0–1.0 on CPU — a cheap guardrail that auto-approves low-risk changes and escalates high-risk ones to the auditor - K8s Autoscaling Retriever. A
task: embeddingmodel contrastively fine-tuned on in-domain (question, document) pairs — a domain-tuned RAG retriever that beats a stock encoder on recall, trained and served on CPU - Support Ticket Extractor. A
task: extractiontoken tagger that pulls service, error, version, and environment entities out of free-form support tickets on CPU — and a worked demonstration of when extraction earns ML over a regex baseline (open-vocabulary prose) and when it doesn't (structured configs)
Technical docs covering the design decisions, best practices, and research behind each pipeline stage. Written for Platform Engineers.
- Getting Started. End-to-end tutorial: build a multi-agent K8s expert from scratch
- Data Stage. Raw data quality, synthetic generation, label taxonomy, verification
- Training Stage. Encoder-head and embedding training on CPU, the stock generation convert/quantize path, model sizing, quantization
- Serving Stage. Reference deployment, CPU inference, autoscaling guidance
- Report Stage. Accuracy measurement, SLM vs LLM comparison, cost projections
The pipeline runs on Kubernetes (EKS). The output is a GGUF model in S3.
- Karpenter. CPU nodes for training, conversion, and the reference deployment (no GPU in the pipeline)
- llama.cpp. GGUF conversion and quantization, plus CPU inference (used in the reference deployment and validation report)
- Pod Identity. IAM access to S3 and Bedrock, no static credentials
- Systems Manager. Remote container builds via SSM, no SSH keys or open ports required
The reference serving deployment (llama.cpp on CPU) is included for validation and as a starting point. You can serve the GGUF model with any compatible runtime: llama.cpp, vLLM, Ollama, or any tool that reads GGUF files.
| Command | Description |
|---|---|
slemify deploy |
Run the full pipeline |
slemify deploy --stage training --no-wait |
Submit a stage and exit |
slemify status my-project |
Show pipeline progress |
slemify status my-project -o json |
Machine-readable status for agents |
slemify validate |
Validate config without deploying |
slemify report |
Download and open the accuracy report in the browser |
slemify report --output my-report.html |
Save report to a custom path |
slemify report --no-open |
Download without opening the browser |
slemify build |
Build container images to ECR |
Q: When should I use an SLM vs just calling an LLM API?
A: If the task is repetitive, structured, and runs more than ~1,000 times/day, or if data can't leave your VPC. Below that volume, an LLM API is simpler and fine. This threshold applies to the encoder-family tasks (classification, scoring, extraction, embedding) — training and serving them on CPU costs cents regardless of volume, so there's little downside to starting early. task: generation is a different calculation: you're comparing a self-hosted CPU (or GPU) deployment against an LLM API's per-token price, and that comparison only favors self-hosting past real volume (industry self-hosting break-even estimates for generative models commonly land in the millions of tokens/day). Below that, keep generation on the LLM API even if you've already adopted Slemify for routing/classification around it.
Q: Can a 3B model really match a frontier LLM? A: For general tasks, no. For YOUR specific structured task with YOUR categories, a fine-tuned 3B model matches or beats general-purpose LLMs. Salesforce's xLAM-2-8B beat GPT-4o and Claude 3.5 at tool calling on the Berkeley Function-Calling Leaderboard. Specialization beats size.
Q: Does fine-tuning always improve quality? When doesn't it help? A: No, and Slemify is deliberate about this. Fine-tuning helps most when the model has to learn something it doesn't already know, and it backfires when the model already has the skill and only lacks the facts:
- Generation — not fine-tuned, on purpose. For a knowledge task the base model can already reason and write; what it lacks is your facts, and RAG supplies those at serving time better than training does. We tested fine-tuning the generative auditor in the k8s example and it made answers worse, so Slemify serves generation stock (download, convert to GGUF, quantize) and grounds it with RAG. (Adapting weights to a lower-precision quantization grid — QAT/QAD — is a different kind of fine-tuning and would be its own future task.)
- Classification / scoring — the head (a router taxonomy, a risk rubric) doesn't exist in any pretrained model, so it must be trained. These tasks always benefit; the question is just whether you have enough data.
- Embedding — domain-tuning a retriever measurably helps when your corpus uses vocabulary or relationships a general encoder hasn't specialized in (in our k8s example, recall@1 improved ~12 points over stock).
- Extraction — domain-dependent, and the example shows both sides. Pulling entities from open-vocabulary prose (support tickets) a trained tagger beats a regex/gazetteer baseline by a wide margin (F1 0.63 → 0.89, driven by open-vocab service/error names). But pulling fields from structured text (k8s YAML configs) a plain parser already wins, so there a trained model adds nothing. The extractor example documents both.
- Reranking — not a Slemify task, on purpose. A strong general-purpose cross-encoder is already excellent at judging (query, document) relevance, and fine-tuning it reliably needs curated hard negatives (human-labeled "looks relevant but isn't"). We tested it: synthesizing those over an overlapping technical corpus produces false negatives that degrade a good model (NDCG@5 0.85 → 0.58 on a fair eval). Since fine-tuning doesn't help, Slemify doesn't do it — running a stock cross-encoder reranker on CPU with no GPU is a serving pattern, shown in the k8s-autoscaling demo, not a model Slemify builds.
The reports always show the metric against an honest baseline (stock-vs-tuned for embedding, a trivial baseline for scoring, a regex/memorization baseline for extraction) so you can see whether training actually helped on your data — not just trust that it did.
Q: How much context can the router/classifier handle?
A: The default text encoder caps at ~512 tokens (roughly 350-400 words) and silently truncates beyond that — but that's usually fine, because a router only needs the decision-relevant slice, not the whole input. Feed it the question, the latest turn, or one retrieved chunk at a time, and scale by sending it less, more often. If the signal is genuinely buried in long text, trim or summarize to the relevant part first, or point model.base at a longer-context encoder. See the k8s-autoscaling routing example and the serving deep dive for the right-tool / wrong-tool guide.
Q: What about RAG?
A: SLMs and RAG solve different problems, and Slemify now covers both sides. RAG retrieves relevant context for knowledge questions; the retrieval step itself is an embedding model, which you can domain-tune with task: embedding. The classification/scoring tasks handle routing and guardrails where you don't need retrieval, you need a fast decision. They work well together: an encoder classifier routes the query, a domain-tuned embedding model retrieves the knowledge, and a generative SLM writes the answer.
Q: Can I use a different base model?
A: Yes. For task: generation, any HuggingFace causal LM that llama.cpp's GGUF converter supports. For encoder-head tasks (task: classification, task: scoring) and task: embedding, any sentence-transformers text encoder. task: extraction (v1) is the exception — its feature-based token tagger uses no encoder, so model.base is omitted. The auto-sizer adjusts infrastructure based on the task and model size.
Q: What happens during a Spot interruption? A: The encoder-family training jobs run in seconds to minutes on CPU, so an interrupted job simply re-runs. The generation convert job (download + GGUF + quantize) is a one-shot, bandwidth-heavy run, so Slemify pins it to on-demand capacity to avoid a mid-run reclaim forcing a full re-download. Serving runs on Spot and is replaced automatically.
Slemify includes an agent skill compatible with Claude Code, OpenAI Codex, Gemini CLI, and Cursor. The skill teaches AI coding agents how to identify SLM opportunities in your system, design the agent's role, write the expert.yaml config, run the pipeline, and interpret results.
Install in Claude Code:
/plugin install slemify@<your-repo>Or reference the skill directly:
"Use the Slemify skill to identify which of my LLM calls could be replaced with a specialized SLM."
The skill includes templates for two patterns:
- Router Agent (
task: classification): a CPU encoder classifier for fast routing and intent decisions - Analyst Agent (
task: generation, 7-8B): structured reasoning grounded by RAG
- Small Language Models are the Future of Agentic AI (NVIDIA, 2025). Position paper arguing SLMs under 10B parameters can handle 60-80% of agentic AI tasks
- xLAM: Large Action Models (Salesforce). 8B model that beat GPT-4o at tool calling, proving specialization beats size
- Forbes: Don't Default to the Biggest AI Model. 40-70% of agentic AI invocations can use SLMs
- Hallucination Propensity in Small Models. Research on knowledge mismatch between fine-tuning data and base model knowledge
- llama.cpp. GGUF conversion, quantization, and CPU inference engine
- Model Context Protocol. How SLMs expose tools to AI assistants
- Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks (Reimers & Gurevych, 2019). The sentence-embedding approach behind Slemify's encoder-head and embedding tasks
- EKS Best Practices: AI/ML CPU Inference. When CPU inference is appropriate and how to tune it