Self-Hosted Models

Bring any OpenAI-compatible model into the ByteAsk TUI and drive it exactly like a built-in cloud model. It shows up in the /model picker under a self/ prefix, streams, and calls tools. Two flavors:

• Your own server — a model on a DGX box, a vLLM/TGI/SGLang cluster, Ollama, LM Studio, llama.cpp, or a LiteLLM proxy.
• A cloud provider with your own key — OpenRouter, Groq, Together, DeepSeek, Mistral, xAI, Cerebras, or Fireworks, via a one-word preset.

Either way, the traffic goes directly from your machine to the provider and never through ByteAsk's servers.

Your own server: your compute, never billed

Prompts and completions for a self-hosted model go directly from your machine to your server. They never touch ByteAsk's servers, are never metered by ByteAsk, and don't count against your plan or credit.

Cloud providers: your key, they bill you

A cloud provider preset (OpenRouter, Groq, …) runs on that provider's cloud under your API key. ByteAsk doesn't bill you and never sees the traffic — but the provider does bill your account, and an agent that loops through many tool calls can consume real credits. This is different from a self-hosted server, which costs nothing to run a request against.

---

How it works

The ByteAsk engine speaks only the OpenAI Responses API, and your server usually speaks Chat Completions. ByteAsk bridges that locally:

• The BYOK sidecar (a loopback-only Python process on your machine) becomes a two-tier router. Cloud models (gpt-*, claude-*, gemini-*) route as usual; a self/<alias> request is sent direct to your endpoint.
• If your server exposes /v1/responses natively, ByteAsk passes the request through unchanged. Otherwise it translates Responses ⇄ Chat Completions on the fly. ByteAsk auto-detects which, so you never have to know your server's wire protocol.
• The endpoint lives in ~/.byteask/byok-config.json (mode 0600). The engine never sees your endpoint URL or key.

Because it all runs on your laptop, ByteAsk reaches private endpoints (LAN / VPN / on-prem DGX) that a cloud service never could.

Requires Python 3

Self-hosted models use the same local sidecar as BYOK, so python3 must be on your PATH. If the helper isn't installed yet, run byteask --update.

---

Quick start

Register a model and pick it:

# vLLM / TGI / SGLang / any OpenAI-compatible server
byteask models add my-model --url http://your-host:8000

# Ollama (defaults to http://localhost:11434)
byteask models add local-llama --ollama

byteask models add probes your server's /v1/models, lets you pick the model id, auto-detects the wire protocol, and registers the endpoint. Then select it in the TUI:

/model

Your model appears as self/my-model. You can also select it directly:

byteask --model self/my-model "refactor the parser"

Verify it end-to-end:

byteask models test my-model

---

Cloud providers (bring your own key)

Instead of pasting a provider's URL, name it with --provider and ByteAsk fills in the endpoint. You supply your own key:

byteask models add or --provider openrouter --key sk-or-...

List the known providers any time:

byteask models providers

Provider	--provider id	Key env var
OpenRouter	openrouter	OPENROUTER_API_KEY
Groq	groq	GROQ_API_KEY
Together AI	together	TOGETHER_API_KEY
DeepSeek	deepseek	DEEPSEEK_API_KEY
Mistral	mistral	MISTRAL_API_KEY
xAI (Grok)	xai	XAI_API_KEY
Cerebras	cerebras	CEREBRAS_API_KEY
Fireworks	fireworks	FIREWORKS_API_KEY
Ollama / LM Studio / llama.cpp / vLLM	ollama / lmstudio / llamacpp / vllm	(none — local)

Supplying the key. ByteAsk looks for the key in this order: --key on the command line → the provider's environment variable (e.g. $OPENROUTER_API_KEY) → a hidden prompt if you're at a terminal. In a script with no key anywhere, the command stops and tells you which env var to set.

Picking a model. Aggregators like OpenRouter list hundreds of models. At a terminal, ByteAsk shows the first 20 and lets you type a substring to filter (or the exact id). In a non-interactive script you must pass --model, so nothing binds to an arbitrary default.

# interactive: pick from the filtered list
byteask models add or --provider openrouter --key sk-or-...

# scripted: name the model explicitly
byteask models add or --provider openrouter --key sk-or-... --model anthropic/claude-sonnet-4.6

Key rotation

The key is read once and stored in ~/.byteask/byok-config.json. Rotating the environment variable later does not change a stored endpoint — to update a key, just run byteask models add again with the same alias (it overwrites).

---

byteask models commands

models providers

byteask models providers

Lists the built-in cloud and local provider presets you can pass to --provider.

models add

byteask models add <alias> --provider <id> [OPTIONS]     # a known provider
byteask models add <alias> --url <endpoint> [OPTIONS]    # your own server

Option	Description
--provider ID	A known provider preset (openrouter, groq, deepseek, …). Fills the URL for you. See byteask models providers.
--url URL	The OpenAI-compatible root of your server (e.g. http://dgx-01.lan:8000). Required unless --provider or --ollama. Overrides a preset's URL if you pass both.
--model ID	The model id the server expects (e.g. Qwen2.5-Coder-32B-Instruct). If omitted, ByteAsk lists the models and lets you pick — required in a non-interactive script for a cloud provider.
--key KEY	Bearer API key. Required for cloud providers; many self-hosted servers don't need one. Can also come from the provider's env var.
--wire auto|responses|chat	Wire protocol. Default auto (probe and pin). responses = pass-through; chat = translate.
--ctx N	Context window in tokens (used for the picker and auto-compaction). Probed from the server when possible.
--tools auto|native|none	Tool-calling tier. Default auto (start native, models test reports if it doesn't round-trip).
--name "Display Name"	Friendly name shown in /model.
--ollama	Preset for Ollama: defaults --url http://localhost:11434, no key, --wire auto.
--ca-bundle PATH	Path to a CA bundle for a private/corporate TLS certificate.
--insecure	Skip TLS verification (use only for trusted internal endpoints — prints a warning).
--header "K=V"	Add a custom HTTP header to every request (repeatable).

The alias is what you pick in /model; the model_id is sent verbatim to your server, so decoupling the two avoids "model not found" errors with long open-weight ids.

models list

byteask models list            # aliases, wire, tool tier, host
byteask models list --check    # also probe each endpoint for reachability

models test

byteask models test <alias>

Runs a plain-English capability check against your endpoint and prints a matrix: is it reachable, is the model id valid, does streaming work, and does a tool call round-trip. This is the fastest way to diagnose a misconfigured server.

models remove

byteask models remove <alias>

Removes the endpoint. If it was your active model, ByteAsk switches you back to a safe cloud default.

---

Server setup notes

Most servers work out of the box. A couple of things to know:

• Tool calling depends on the model and the server. Wire format and streaming "just work"; whether a model can act as a coding agent depends on its tool-calling ability. For vLLM, tool calls require the server to be started with a tool-call parser enabled, e.g. --enable-auto-tool-choice --tool-call-parser hermes (or llama3_json). If it isn't, tool calls come back as plain text — byteask models test detects this and tells you.
• Pick a tool-trained model for agentic work (Qwen2.5-Coder, Llama 3.1+, DeepSeek-V3, Mistral-Large). Base/non-tool models can still chat; set --tools none for them.
• Set --ctx honestly. If the registered context window overstates your server's real limit, long conversations can fail mid-turn. ByteAsk probes max_model_len from /v1/models where available.

apply_patch quality on open models

ByteAsk edits files with an apply_patch tool. On open-weight models served over Chat Completions, apply_patch is the known weak spot — patch quality varies by model. Plain shell commands, file reads, and other tools are unaffected. For the most reliable editing on a self-hosted model, prefer a strong code model and review diffs carefully.

---

Registry file

Endpoints are stored in ~/.byteask/byok-config.json (mode 0600), alongside your BYOK keys:

~/.byteask/byok-config.json

{
  "endpoints": {
    "my-dgx-qwen": {
      "base_url": "http://dgx-01.lan:8000",
      "model_id": "Qwen2.5-Coder-32B-Instruct",
      "wire": "chat",
      "context_window": 32768,
      "tools": "auto",
      "display_name": "Qwen2.5-Coder 32B (my DGX)"
    }
  }
}

You normally never edit this by hand — use byteask models — but it's plain JSON if you need to inspect it. The custom rows are re-merged into your /model catalog on every launch and every byteask --update, so a CLI update never drops them.