Skip to main content

Your Own Models

Bring any OpenAI-compatible model into the ByteAsk TUI and drive it exactly like a built-in cloud model. It shows up in the /model picker under a self/ prefix, streams, and calls tools. Two flavors:

  • Your own server - a model on a DGX box, a vLLM/TGI/SGLang cluster, Ollama, LM Studio, llama.cpp, or a LiteLLM proxy.
  • A cloud provider with your own key - OpenRouter, Groq, Together, DeepSeek, Mistral, xAI, Cerebras, or Fireworks, via a one-word preset.

Either way, the traffic goes directly from your machine to the provider and never through ByteAsk's servers.

Your own server: your compute, never billed

Prompts and completions for a model on your own server go directly from your machine to your server. They never touch ByteAsk's servers, are never metered by ByteAsk, and don't count against your plan or credit.

Cloud providers: your key, they bill you

A cloud provider preset (OpenRouter, Groq, …) runs on that provider's cloud under your API key. ByteAsk doesn't bill you and never sees the traffic - but the provider does bill your account, and an agent that loops through many tool calls can consume real credits. This is different from your own server, which costs nothing to run a request against.

How it works​

The ByteAsk engine speaks only the OpenAI Responses API, and your server usually speaks Chat Completions. ByteAsk bridges that locally:

  • The BYOK sidecar (a loopback-only Python process on your machine) becomes a two-tier router. Cloud models (gpt-*, claude-*, gemini-*) route as usual; a self/<alias> request is sent direct to your endpoint.
  • If your server exposes /v1/responses natively, ByteAsk passes the request through unchanged. Otherwise it translates Responses ⇄ Chat Completions on the fly. ByteAsk auto-detects which, so you never have to know your server's wire protocol.
  • The endpoint lives in ~/.byteask/byok-config.json (mode 0600). The engine never sees your endpoint URL or key.

Because it all runs on your laptop, ByteAsk reaches private endpoints (LAN / VPN / on-prem DGX) that a cloud service never could.

Requires Python 3

Your own models use the same local sidecar as BYOK, so python3 must be on your PATH. If the helper isn't installed yet, run byteask --update.

Quick start​

Register a model and pick it:

# vLLM / TGI / SGLang / any OpenAI-compatible server
byteask models add my-model --url http://your-host:8000

# Ollama (defaults to http://localhost:11434)
byteask models add local-llama --ollama

byteask models add probes your server's /v1/models, lets you pick the model id, auto-detects the wire protocol, and registers the endpoint. Then select it in the TUI:

/model

Your model appears as self/my-model. You can also select it directly:

byteask --model self/my-model "refactor the parser"

Verify it end-to-end:

byteask models test my-model

Cloud providers (bring your own key)​

Instead of pasting a provider's URL, name it with --provider and ByteAsk fills in the endpoint. You supply your own key:

byteask models add or --provider openrouter --key sk-or-...

List the known providers any time:

byteask models providers
Provider--provider idKey env var
OpenRouteropenrouterOPENROUTER_API_KEY
GroqgroqGROQ_API_KEY
Together AItogetherTOGETHER_API_KEY
DeepSeekdeepseekDEEPSEEK_API_KEY
MistralmistralMISTRAL_API_KEY
xAI (Grok)xaiXAI_API_KEY
CerebrascerebrasCEREBRAS_API_KEY
FireworksfireworksFIREWORKS_API_KEY
Ollama / LM Studio / llama.cpp / vLLMollama / lmstudio / llamacpp / vllm(none - local)

Supplying the key. ByteAsk looks for the key in this order: --key on the command line → the provider's environment variable (e.g. $OPENROUTER_API_KEY) → a hidden prompt if you're at a terminal. In a script with no key anywhere, the command stops and tells you which env var to set.

Picking a model. Aggregators like OpenRouter list hundreds of models. At a terminal, ByteAsk shows the first 20 and lets you type a substring to filter (or the exact id). In a non-interactive script you must pass --model, so nothing binds to an arbitrary default.

# interactive: pick from the filtered list
byteask models add or --provider openrouter --key sk-or-...

# scripted: name the model explicitly
byteask models add or --provider openrouter --key sk-or-... --model <provider/model-id>
Key rotation

The key is read once and stored in ~/.byteask/byok-config.json. Rotating the environment variable later does not change a stored endpoint - to update a key, just run byteask models add again with the same alias (it overwrites).

byteask models commands​

models providers​

byteask models providers

Lists the built-in cloud and local provider presets you can pass to --provider.

models add​

byteask models add <alias> --provider <id> [OPTIONS]     # a known provider
byteask models add <alias> --url <endpoint> [OPTIONS]    # your own server
OptionDescription
--provider IDA known provider preset (openrouter, groq, deepseek, …). Fills the URL for you. See byteask models providers.
--url URLThe OpenAI-compatible root of your server (e.g. http://dgx-01.lan:8000). Required unless --provider or --ollama. Overrides a preset's URL if you pass both.
--model IDThe model id the server expects (e.g. Qwen2.5-Coder-32B-Instruct). If omitted, ByteAsk lists the models and lets you pick - required in a non-interactive script for a cloud provider.
--key KEYBearer API key. Required for cloud providers; many servers you run yourself don't need one. Can also come from the provider's env var.
--wire auto|responses|chatWire protocol. Default auto (probe and pin). responses = pass-through; chat = translate.
--ctx NContext window in tokens (used for the picker and auto-compaction). Probed from the server when possible.
--tools auto|native|noneTool-calling tier. Default auto (start native, models test reports if it doesn't round-trip).
--name "Display Name"Friendly name shown in /model.
--ollamaPreset for Ollama: defaults --url http://localhost:11434, no key, --wire auto.
--ca-bundle PATHPath to a CA bundle for a private/corporate TLS certificate.
--insecureSkip TLS verification (use only for trusted internal endpoints - prints a warning).
--header "K=V"Add a custom HTTP header to every request (repeatable).

The alias is what you pick in /model; the model_id is sent verbatim to your server, so decoupling the two avoids "model not found" errors with long open-weight ids.

models list​

byteask models list            # aliases, wire, tool tier, host
byteask models list --check    # also probe each endpoint for reachability

models test​

byteask models test <alias>

Runs a plain-English capability check against your endpoint and prints a matrix: is it reachable, is the model id valid, does streaming work, and does a tool call round-trip. This is the fastest way to diagnose a misconfigured server.

models remove​

byteask models remove <alias>

Removes the endpoint. If it was your active model, ByteAsk switches you back to a safe cloud default.

Server setup notes​

Most servers work out of the box. A couple of things to know:

  • Tool calling depends on the model and the server. Wire format and streaming "just work"; whether a model can act as a coding agent depends on its tool-calling ability. For vLLM, tool calls require the server to be started with a tool-call parser enabled, e.g. --enable-auto-tool-choice --tool-call-parser hermes (or llama3_json). If it isn't, tool calls come back as plain text - byteask models test detects this and tells you.
  • Pick a tool-trained model for agentic work (Qwen2.5-Coder, Llama 3.1+, DeepSeek-V3, Mistral-Large). Base/non-tool models can still chat; set --tools none for them.
  • Set --ctx honestly. If the registered context window overstates your server's real limit, long conversations can fail mid-turn. ByteAsk probes max_model_len from /v1/models where available.
apply_patch quality on open models

ByteAsk edits files with an apply_patch tool. On open-weight models served over Chat Completions, apply_patch is the known weak spot - patch quality varies by model. Plain shell commands, file reads, and other tools are unaffected. For the most reliable editing on a model you run yourself, prefer a strong code model and review diffs carefully.

Registry file​

Endpoints are stored in ~/.byteask/byok-config.json (mode 0600), alongside your BYOK keys:

~/.byteask/byok-config.json
{
  "endpoints": {
    "my-dgx-qwen": {
      "base_url": "http://dgx-01.lan:8000",
      "model_id": "Qwen2.5-Coder-32B-Instruct",
      "wire": "chat",
      "context_window": 32768,
      "tools": "auto",
      "display_name": "Qwen2.5-Coder 32B (my DGX)"
    }
  }
}

You normally never edit this by hand - use byteask models - but it's plain JSON if you need to inspect it. The custom rows are re-merged into your /model catalog on every launch and every byteask --update, so a CLI update never drops them.