--- title: "Embedding Models for Virtuoso Vector Index" description: "Learn about the embedding models supported by Virtuoso's vector index, including the bundled local ONNX model (IBM granite-embedding-30m-english), remote HTTP providers (OpenAI, Ollama, LM Studio), API key management, model registration, and how to choose the right model for your workload." url: urn:osdi:this dateModified: 2026-10-30T15:45:34Z language: en --- One page of the Vector Index documentation, MERGED by build_vector_index_docs.py from this template and the markdown in fixture/vector-index/. Do not edit the generated file: change this template or the generator and rebuild. It opens on the documentation header (dxshell.py), not on the site hero. [Home](urn:osdi:site)/[Virtuoso](urn:osdi:sitevirtuoso/)/[Vector index](urn:osdi:sitevirtuoso/docs/vector-index/)/Embedding models for the Virtuoso vector index # Embedding models for the Virtuoso vector index Learn about the embedding models supported by Virtuoso's vector index, including the bundled local ONNX model (IBM granite-embedding-30m-english), remote HTTP providers (OpenAI, Ollama, LM Studio), API key management, model registration, and how to choose the right model for your workload. Start [Overview](urn:osdi:sitevirtuoso/docs/vector-index/) [Quickstart](urn:osdi:sitevirtuoso/docs/vector-index/quickstart/) [Tutorial: FAQ semantic search](urn:osdi:sitevirtuoso/docs/vector-index/tutorial-faq-semantic-search/) Build [SQL vector index how-to](urn:osdi:sitevirtuoso/docs/vector-index/howto-sql-vector-index/) [SPARQL vector index how-to](urn:osdi:sitevirtuoso/docs/vector-index/howto-sparql-vector-index/) [Search RDF literals (knnSimilar)](urn:osdi:sitevirtuoso/docs/vector-index/knn-similar/) Configure Embedding models [Index parameters](urn:osdi:sitevirtuoso/docs/vector-index/index-parameters/) [Similarity metrics](urn:osdi:sitevirtuoso/docs/vector-index/similarity-metrics/) Operate [VVec Console (web UI)](urn:osdi:sitevirtuoso/docs/vector-index/vvec-console/) [Troubleshooting](urn:osdi:sitevirtuoso/docs/vector-index/troubleshooting/) Reference [API reference](urn:osdi:sitevirtuoso/docs/vector-index/api-reference/) An embedding model converts text into a fixed-length float vector (an "embedding") so that semantically similar texts land close together in vector space. Virtuoso's vector index is **model-bound**: every index is created against a named model in `DB.DBA.ONNX_MODELS`, and that model is used to embed both your stored data and your query text. Choosing and managing the right model is therefore central to getting good search results. This reference covers: - **Overview**: What embedding models are and the two categories Virtuoso supports - **The ONNX_MODELS table**: The model registry, its schema, and how to inspect it - **Local ONNX models**: The bundled granite model, bootstrap, load, and embed functions - **Remote HTTP embedding providers**: OpenAI, Ollama, LM Studio, and custom endpoints - **API key management**: Storing provider keys - **Registering a provider model**: The registration procedure and examples - **Choosing a model**: A comparison of latency, cost, quality, privacy, and dimensions - **Using a custom ONNX model**: Exporting and registering your own ONNX model Note This is a model-lifecycle reference. For end-to-end index creation and querying, see [SQL vector index how-to](urn:osdi:sitevirtuoso/docs/vector-index/howto-sql-vector-index/) or [SPARQL vector index how-to](urn:osdi:sitevirtuoso/docs/vector-index/howto-sparql-vector-index/). --- ## Overview An embedding model is a neural network that maps a piece of text to a vector of real numbers — for example, a 384-dimensional float vector for the bundled granite model. Texts that mean similar things produce vectors that are close together under a distance metric such as cosine similarity. The Virtuoso vector index stores these vectors in a native HNSW graph and uses the same model to embed query text at search time, so the query vector can be matched against the stored vectors. Because the index is bound to a single model, the model determines three things up front: - **Output dimension** — the length of every vector (e.g. 384 for the bundled granite model, 1536 for OpenAI `text-embedding-3-small`). The index's `dimensions` must match exactly, and you cannot change models after the index is created. - **Quality** — how well the embeddings capture semantic similarity. Larger or more recent models generally produce better rankings. - **Operational characteristics** — latency, cost, privacy, and memory footprint, which differ sharply between local and remote models. Virtuoso supports two categories of embedding models: | Category | How it runs | Network | Example | | --- | --- | --- | --- | | **Local ONNX** | On the Virtuoso host, in helper processes the `vonnx` plugin starts, using ONNX Runtime | None | Bundled `granite-384-local` | | **Remote HTTP provider** | HTTP call to an OpenAI-compatible endpoint | One request per text, or per batch of texts | OpenAI, Ollama, LM Studio | Both categories are represented as rows in the same `DB.DBA.ONNX_MODELS` table and are referenced by model name from `CREATE VECTOR INDEX` (SQL) or `create vector index` (SPARQL). The difference is in how the row is populated and how embeddings are produced at runtime. --- ## The ONNX_MODELS table `DB.DBA.ONNX_MODELS` is the central model registry. Every model a vector index can use — local or remote — has a row here. You do not normally populate this table by hand; the bundled local model is auto-provisioned at startup, and remote models are registered through a stored procedure. ### Auto-provisioning at startup When Virtuoso starts with the `vonnx` plugin loaded, the bundled `granite-384-local` model is automatically registered and loaded. This happens inside the bootstrap routine `ONNX_BOOTSTRAP_MODELS()`, which: 1. Ensures the `ONNX_MODELS` table exists (creating it if necessary). 2. Registers `granite-384-local`. It looks for `granite-embedding-30m-english/model.onnx` and `vocab.json` (with `merges.txt` beside it) under `[ONNX] ModelRoot` — or at `[ONNX] GraniteModelPath` and `GraniteVocabPath` when those are set — and for the helper program at `[ONNX] HelperPath`, falling back to the plugin `LoadPath` and the `DirsAllowed` directories. The paths are resolved again at every start, so a moved installation still finds them. If a file cannot be found, the server log names it and says what to configure, and an existing `granite-384-local` row is marked `unavailable`. 3. Loads every registered model and stores each local model's handle in the `handle` column. The server log reports `ONNX bootstrap refreshed N model(s)`. After startup, the bundled model is ready to use with no extra configuration. The relevant `virtuoso.ini` settings, as `make install` writes them: INI ``` [ONNX] ModelRoot = /onnx-models ;HelperPath = /onnx_helper [Plugins] LoadPath = Load17 = plain, vonnx ``` With `HelperPath` commented out, the helper is found in the plugin `LoadPath`. Relative paths are resolved against the server root — the directory the server runs in, which holds its database — so use absolute paths when in doubt. ### Schema | Column | Type | Description | | --- | --- | --- | | `model_id` | INTEGER IDENTITY | Primary key; SQL vector indexes refer to their model by this id | | `model_name` | VARCHAR, unique | The name used in `USING 'model_name'` | | `description` | VARCHAR | Free text. For the bundled model, `local bootstrap model`, or the reason it is unavailable | | `model_path` | VARCHAR | Local models: path of the `.onnx` file, resolved against `ModelRoot`, `LoadPath` and `DirsAllowed` at every load. Remote models: the provider's own model name | | `model_type` | VARCHAR | `text_embedding` for a local ONNX model; `openai`, `ollama`, `lmstudio` or `openai_compatible` for a remote one; `unavailable` for a bundled model whose files are missing | | `input_names` | VARCHAR | Comma-separated input tensor names, passed as `input`, `input2` and `input3`. Empty means `input_ids,attention_mask` | | `input_shapes` | VARCHAR | Not used | | `output_name` | VARCHAR | Output tensor to read (`pooled_output` for the bundled model) | | `output_shape` | VARCHAR | Output shape, e.g. `1,384`. Its last number is the model's dimension: a vector index over a local model needs it, and `CREATE VECTOR INDEX` takes `dimensions` from it when the option is omitted | | `vocab_path` | VARCHAR | Tokenizer file: a WordPiece `vocab.txt`, or a byte-level BPE `vocab.json` with `merges.txt` beside it | | `max_seq_len` | INTEGER | Tokens per text; longer text is truncated. Seeded with 512 for the bundled model and never reset at start-up | | `provider_endpoint` | VARCHAR | HTTP endpoint URL (remote models only) | | `provider_api_key_name` | VARCHAR | Registry name of the API key (remote models only) | | `provider_timeout` | INTEGER | HTTP timeout in seconds, 1–300 (remote models only) | | `provider_dimensions` | INTEGER | Expected embedding dimension (remote models only); a response of any other length is rejected | | `provider_insecure` | INTEGER | The `_insecure` flag the model was registered with, passed to the HTTP client; default `0` | | `provider_max_batch` | INTEGER | Inputs per embedding request (remote models only). `null` uses the per-provider default — see [Request batching](#request-batching) | | `provider_consecutive_failures` | INTEGER | Failed requests since the last success (remote models only) | | `provider_last_error` | VARCHAR | The last failure's error message (remote models only) | | `provider_last_error_at` | DATETIME | When it happened | | `intra_threads` | INTEGER | ONNX Runtime intra-op threads per helper (local models). `null` means 1 — see [Helper thread counts](#helper-thread-counts) | | `inter_threads` | INTEGER | ONNX Runtime inter-op threads per helper (local models). `null` means 1 | | `handle` | INTEGER | The loaded model's handle (local models only; `null` for remote models and for a model not loaded yet) | | `created_at`, `updated_at` | DATETIME | Row timestamps | Tip For local ONNX models, `handle` is non-null when the model is loaded. For remote HTTP models, `handle` is `null` and the `provider_*` columns carry the connection details instead. ### How to inspect the table SQL ``` SELECT model_name, model_type, handle, output_shape FROM DB.DBA.ONNX_MODELS ORDER BY model_name; ``` Example output after a default startup: | model_name | model_type | handle | output_shape | | --- | --- | --- | --- | | `granite-384-local` | `text_embedding` | 1 | `1,384` | To see the full picture including remote provider details: SQL ``` SELECT model_name, model_type, handle, output_shape, provider_endpoint, provider_api_key_name, provider_timeout, provider_dimensions FROM DB.DBA.ONNX_MODELS ORDER BY model_name; ``` Important The `provider_api_key_name` column stores the **registry name** of a key, not the key value itself. Secret material is never stored in this table. See [API key management](#api-key-management). --- ## Local ONNX models A local ONNX model runs on the Virtuoso host itself. The `vonnx` plugin wraps ONNX Runtime and runs each loaded model in a pool of helper processes that the server starts and talks to over pipes. There are no network calls and no text ever leaves the machine. ### The bundled granite-384-local model Virtuoso ships with a single bundled embedding model: | Property | Value | | --- | --- | | Model name | `granite-384-local` | | Base model | IBM `granite-embedding-30m-english` (r1.1), Apache 2.0 | | Architecture | RoBERTa encoder, 6 layers, 30M parameters | | Output dimensions | 384, L2-normalized (CLS pooling, in the graph) | | Tokenizer | Byte-level BPE, cased, 50,265 tokens and 50,000 merges | | Max sequence length | 512 tokens (the registry seeds `max_seq_len` with 512) | | File size | 60,458,629 bytes — weights stored as bf16, the precision IBM publishes; ONNX Runtime widens them to fp32 when it loads the model | | SHA-256 of `model.onnx` | `a07c196ffe53ab19a8b68280a9d80b2ab2ba98924e48e5f102b4558b738cfc58` | | Loaded by | `ONNX_BOOTSTRAP_MODELS()` at startup | This is a general-purpose **English** embedding model. It is small, free to run, and needs no setup, which makes it the default for development and for production workloads where keeping data on the host is a priority. Its files — `model.onnx`, `vocab.json` and `merges.txt` — are installed under `onnx-models/granite-embedding-30m-english/` in the hosting directory, with a README that records how the model was exported and checked, and a `SHA256SUMS` file. `model.onnx` is not kept in the source repository. A build with `--enable-onnxruntime` downloads it from OpenLink's model repository (`https://nexus.openlinksw.com/repository/onnx-models`) when it is missing and checks every file against `SHA256SUMS`; a mismatch fails the build. A tree that already holds a verified copy builds without the network. ### ONNX_BOOTSTRAP_MODELS() `ONNX_BOOTSTRAP_MODELS()` is called automatically during server startup (after the `vonnx` plugin loads). It creates the `ONNX_MODELS` table if it does not exist, registers the bundled `granite-384-local` row, and loads every registered local model so their handles are ready immediately. You normally never call this procedure yourself. It is safe to call it again manually if you need to re-bootstrap after a table was damaged, but in normal operation startup handles it. ### ONNX_LOAD_MODEL `ONNX_LOAD_MODEL(model_path, options)` loads an ONNX model file from disk into ONNX Runtime and returns an **integer handle** that identifies it. For a registered model, the bootstrap and `ONNX_MODEL_REFRESH` store that handle in the `handle` column of `ONNX_MODELS`. This is the low-level load primitive used by the bootstrap and by custom-model registration, and it is **DBA-only**. The `options` string tells the runtime how to feed the model — see [Using a custom ONNX model](#using-a-custom-onnx-model) for the format. A file already loaded with the same options returns the existing handle rather than a new one. ### ONNX_UNLOAD_MODEL `ONNX_UNLOAD_MODEL(handle)` (DBA-only) closes a handle and stops its helper processes; it returns 1. It refuses while calls are in flight on that handle (`Model N has M in-flight calls`). Unload a handle before reloading a model file you replaced on disk — see [ONNX_MODEL_REFRESH](#onnx_model_refresh). ### ONNX_EMBED_TEXT `ONNX_EMBED_TEXT(handle, text)` is the lower-level BIF that produces an embedding from an already-loaded model. You pass the **handle** (the integer returned by `ONNX_LOAD_MODEL`), not the model name, so you must look up the handle first: SQL ``` SELECT ONNX_EMBED_TEXT( (SELECT handle FROM DB.DBA.ONNX_MODELS WHERE model_name = 'granite-384-local'), 'hello world' ); -- Expect: a 384-element float vector ``` Note `ONNX_EMBED_TEXT` is a **BIF** — call it **unqualified** (no `DB.DBA.` prefix). It only works for local ONNX models (those with a non-null `handle`). Its batch form, `ONNX_EMBED_BATCH(handle, texts)`, takes a vector of strings and returns a vector of embeddings, up to 32 texts per call by default (`batch too large (33 > 32)` beyond that). ### ONNX_EMBED_TEXT_FOR_MODEL `DB.DBA.ONNX_EMBED_TEXT_FOR_MODEL(model_name, text)` is the recommended wrapper. You pass the **model name** instead of a handle, and the procedure resolves the model, checks whether it is local or remote, and dispatches accordingly. A local model that is registered but not loaded yet is loaded on first use. SQL ``` -- Local model (uses the ONNX handle): SELECT DB.DBA.ONNX_EMBED_TEXT_FOR_MODEL('granite-384-local', 'hello world'); -- Remote model (issues an HTTP call to the provider): SELECT DB.DBA.ONNX_EMBED_TEXT_FOR_MODEL('openai-small', 'hello world'); ``` This is the procedure you should call when you want to embed text manually — for example, to precompute a query vector and reuse it across multiple columns with a remote provider. During normal vector queries you do not call it yourself; the index embeds your query text automatically with its bound model. `DB.DBA.ONNX_EMBED_BATCH_FOR_MODEL(model_name, texts)` is its batch form: it takes a vector of strings, returns a vector of embeddings in the same order, and splits the work for you — into requests of `provider_max_batch` inputs for a remote model, and into calls of 32 texts for a local one. Tip For a **remote** provider, each call to `ONNX_EMBED_TEXT_FOR_MODEL` is one HTTP request. When a query projects both `VECTOR_DISTANCE` and `X_SIMILAR`, embed once with this procedure and pass the resulting vector to avoid duplicate API calls. ### ONNX_MODEL_REFRESH `DB.DBA.ONNX_MODEL_REFRESH(model_name)` loads a registered local model with the options its `ONNX_MODELS` row describes — paths resolved afresh, plus `input_names`, `output_name`, `vocab_path`, `max_seq_len`, `intra_threads` and `inter_threads` — stores the resulting handle in the row, and returns it: SQL ``` SELECT DB.DBA.ONNX_MODEL_REFRESH('granite-384-local'); ``` Use it after you change the row's load settings. Because a file already loaded with the same options returns the existing handle, a refresh alone does **not** pick up a model file you replaced on disk — it returns the old handle, and the old weights keep serving. Unload the old handle first: SQL ``` SELECT ONNX_UNLOAD_MODEL ((SELECT handle FROM DB.DBA.ONNX_MODELS WHERE model_name = 'my-custom-model')); SELECT DB.DBA.ONNX_MODEL_REFRESH ('my-custom-model'); -- a new handle ``` Vectors from different weights are not comparable: if the new file is a different model, rebuild the indexes that use it. ### Performance characteristics | Characteristic | Value | | --- | --- | | Inference location | Helper processes on the Virtuoso host (no network) | | Single-text embedding latency | **~5 ms** measured (median of 200 calls; min 4 ms, mean 4.5 ms), bundled granite, an eight-word query text, including the round trip to the helper process | | Batch throughput | **~68 texts/s** measured at embedding concurrency 8 (batches of 32) refreshing an index over 4,000 FiQA passages — 58.2 s with `build = 'bulk'`, 59.4 s with the default build; granite reads up to 512 tokens of each passage | | Helper process RSS | **265–266 MB** freshly loaded and idle; **547–620 MB** after one batch of 32 long passages, as ONNX Runtime's arenas grow with sequence length | | Helpers per model | **Two** at start. A SPARQL `refresh vector index` raises the pool to its embedding concurrency (`__vec_embed_concurrency`, default 4), and a pool does not shrink while the server runs. Each loaded handle has its own pool | | Threads per helper | 1 intra-op, 1 inter-op by default; see [Helper thread counts](#helper-thread-counts) | Note The single-text figure is what a **text query** pays before its ANN search, and the search itself costs about **0.95 ms** — so embedding is roughly five times the search it feeds and still most of a text query's latency. Batch throughput is the number that matters for building an index; see Embedding concurrency. All figures on this page: 10-core arm64 (Apple M1 Pro), measured 2026-10-10. ### Helper thread counts Each ONNX helper process runs its ONNX Runtime session with **one intra-op and one inter-op thread by default**. That is deliberate: a helper pool gets its parallelism from running several helpers at once, and an intra-op pool per helper on top of that would oversubscribe the machine and compete with Virtuoso's own threads. Set them per model with the `intra_threads` / `inter_threads` load options: SQL ``` SELECT ONNX_LOAD_MODEL ( '/path/to/model.onnx', 'input=input_ids;input2=attention_mask;output=pooled_output;vocab=/path/to/vocab.json;max_seq_len=512;intra_threads=6'); ``` Note the options are separated by **semicolons**, not commas. Give the full set of options a model needs: a load without `max_seq_len`, for example, truncates every text at the helper's default of 128 tokens. Note The thread counts are part of the model's identity, like `max_seq_len` and the batching options. `ONNX_LOAD_MODEL` returns the existing handle only when the file is already loaded with the same options; loading the bundled model again with its startup options plus `intra_threads=6` returns a **new** handle, whose helpers run six threads, and the registered model keeps its one. That new handle serves only the calls that pass it — `ONNX_EMBED_TEXT (h, …)`, `ONNX_EMBED_BATCH (h, …)`. A vector index, `ONNX_SUMMARIZE` and the `…_FOR_MODEL` functions embed through the model's `ONNX_MODELS` row: set the threads there instead, and reload it. For a registered model — the bundled one included — the row carries them: SQL ``` UPDATE DB.DBA.ONNX_MODELS SET intra_threads = 4 WHERE model_name = 'granite-384-local'; SELECT DB.DBA.ONNX_MODEL_REFRESH ('granite-384-local'); ``` The refresh loads a new model and moves the row's `handle` to it; the old one stays loaded until `ONNX_UNLOAD_MODEL` or a restart. Re-registration at start-up keeps the setting. Measured through the row, `ONNX_EMBED_TEXT_FOR_MODEL` on 200 FiQA passages one at a time (704 characters on average): 20.6 a second with the column empty, 66.2 with `intra_threads = 4` (two rounds each, alternating; 10-core arm64, 2026-10-10). **Measured**, bundled granite on a 10-core arm64 machine (Apple M1 Pro), aggregate helper CPU while embedding batches of 32 FiQA passages, 2026-10-10: | configuration | aggregate helper CPU | | --- | --- | | one serial caller, `intra_threads=1` | 100% (pinned to one core) | | one serial caller, `intra_threads=6` | 545% | | 8 concurrent callers, pool of 8, `intra_threads=1` | 793% | Raise `intra_threads` on a handle you call directly when a **single call's latency** matters and there is no concurrency to exploit. For building an index, raise `__vec_embed_concurrency` instead — process level concurrency scales better than intra-op threading, and the two combine. Important The helpers' memory — about 265 MB each at rest, two or more per loaded model, and more while they embed long texts — is on top of Virtuoso's own buffer pool (`NumberOfBuffers * 8 KB`). On small instances, budget for this and add swap — see the deployment guidance in [SQL vector index how-to](urn:osdi:sitevirtuoso/docs/vector-index/howto-sql-vector-index/). --- ## Remote HTTP embedding providers Instead of (or in addition to) a local ONNX model, you can register external embedding APIs. Virtuoso presents them through a unified **OpenAI-compatible** interface: every provider speaks the same request and response shape, so the only differences are the endpoint URL, the model name, the dimensions, and whether a key is required. ### Built-in providers | Provider | `_provider` value | Default endpoint | Key required | | --- | --- | --- | --- | | OpenAI | `openai` | `https://api.openai.com/v1/embeddings` | Yes | | Ollama | `ollama` | `http://127.0.0.1:11434/v1/embeddings` | No | | LM Studio | `lmstudio` | `http://127.0.0.1:1234/v1/embeddings` | No | | OpenAI-compatible | `openai_compatible` | You supply it | As needed | - **OpenAI** is a hosted commercial API. You must supply an API key, and usage is billed. - **Ollama** and **LM Studio** are local model servers you run yourself. They default to loopback endpoints and need no key, but the server process must be running and must have the model loaded. - **`openai_compatible`** is the catch-all for any service that implements the OpenAI embeddings request/response contract — for example, a self-hosted vLLM, TEI, or LiteLLM gateway. You provide the endpoint explicitly; registering one without an endpoint fails with `An endpoint is required for openai_compatible models`. Note Ollama and LM Studio must be started with their OpenAI-compatible embeddings endpoint enabled. Consult each project's documentation for the exact startup flags. ### Endpoint security - **HTTPS, except on a private network.** An endpoint must use `https://`; plain `http://` is accepted only for a loopback or private-network host (`localhost`, `127.*`, `10.*`, `172.16`–`172.31.*`, `192.168.*`, link-local and IPv6 private addresses). The host is classified by its literal text, so any other host name counts as public. Registration and every request check it:Output`*** Error 42000: [Virtuoso Driver][Virtuoso Server]HTTP embedding endpoint http://embeddings.example.com/v1/embeddings must use https:// -- plain http is allowed only to a loopback or private-network host (refusing to send an API key in cleartext to a public host).: ONNXP` - **Redirects are not followed**, so a key is never re-sent to another host. ### API key management Provider API keys are stored in the Virtuoso registry, never in the `ONNX_MODELS` table. The table holds only the **registry name** of a key. #### DB.DBA.ONNX_API_KEY_SET SQL ``` DB.DBA.ONNX_API_KEY_SET(key_name, api_key); ``` `DB.DBA.ONNX_API_KEY_SET` stores a key under a registry name of your choosing. It is **DBA-only**. The conventional name for the OpenAI key is `ONNX_API_KEY_OPENAI`, which is the name the `openai` provider auto-defaults to when you register a model without specifying a key. SQL ``` DB.DBA.ONNX_API_KEY_SET('ONNX_API_KEY_OPENAI', ''); ``` #### DB.DBA.ONNX_PROVIDER_API_KEY_SET SQL ``` DB.DBA.ONNX_PROVIDER_API_KEY_SET(provider, api_key); ``` `DB.DBA.ONNX_PROVIDER_API_KEY_SET` is a convenience wrapper (also DBA-only). It maps a provider name to its conventional registry name, stores the key there, and returns the name it used: | provider | registry name | | --- | --- | | `openai` | `ONNX_API_KEY_OPENAI` | | `ollama` | `ONNX_API_KEY_OLLAMA` | | `lmstudio` | `ONNX_API_KEY_LMSTUDIO` | | `openai_compatible` | `ONNX_API_KEY_OPENAI_COMPATIBLE` | SQL ``` DB.DBA.ONNX_PROVIDER_API_KEY_SET('openai', ''); -- Equivalent to: DB.DBA.ONNX_API_KEY_SET('ONNX_API_KEY_OPENAI', ''); ``` Only the `openai` provider falls back to its conventional name when a model is registered without one. For the other providers, pass the registry name as `_api_key_name` when you register the model, or no key is sent. #### Key visibility and safety Caution Keys are **not echoed** in error messages or in the server log. When a provider rejects a request — a `401`, say — the caller gets a generic error that points to the server log, and the log line records the provider, the model, the endpoint and the provider's status line, not the key. Keep the key-setting procedures DBA-only; do not grant execute rights on them to application users. ### Registering a provider model Remote models are registered with `DB.DBA.ONNX_REGISTER_HTTP_EMBED_MODEL` (DBA-only). The procedure creates (or replaces) a row in `ONNX_MODELS` with the provider connection details. It does not contact the provider. #### Signature SQL ``` DB.DBA.ONNX_REGISTER_HTTP_EMBED_MODEL ( _model_name, -- alias used in USING '...' for CREATE VECTOR INDEX _provider, -- 'openai', 'ollama', 'lmstudio', or 'openai_compatible' _remote_model, -- the provider's own model name (e.g. 'text-embedding-3-small') _dimensions, -- expected output dimension; must match the provider model _endpoint := NULL, -- null = use the provider's default endpoint _api_key_name := NULL, -- registry NAME of the key, NOT the key itself _timeout := 30, -- HTTP timeout in seconds, 1 to 300 _description := NULL, -- optional free-text description _insecure := 0); -- passed to the HTTP client's insecure option ``` Important The `_api_key_name` parameter is the **registry name** of the key (for example `'ONNX_API_KEY_OPENAI'`), **not** the key value. For the `openai` provider, passing `null` auto-defaults to `ONNX_API_KEY_OPENAI` — so if you stored your key with the conventional name, you can omit it entirely. If you pass the literal key string by mistake, it is looked up as a registry name, nothing usable is found, and the provider rejects the request with a `401`. #### Examples SQL ``` -- 1. OpenAI text-embedding-3-small (1536 dimensions) -- Store the key first (DBA-only): DB.DBA.ONNX_API_KEY_SET('ONNX_API_KEY_OPENAI', ''); DB.DBA.ONNX_REGISTER_HTTP_EMBED_MODEL ( 'openai-small', 'openai', 'text-embedding-3-small', 1536, NULL, 'ONNX_API_KEY_OPENAI', 30, 'OpenAI text-embedding-3-small'); -- 2. Local Ollama (no key needed; default endpoint) DB.DBA.ONNX_REGISTER_HTTP_EMBED_MODEL ( 'ollama-mxbai', 'ollama', 'mxbai-embed-large', 1024, NULL, NULL, 30, 'Ollama mxbai-embed-large'); -- 3. Local LM Studio (no key needed; default endpoint) DB.DBA.ONNX_REGISTER_HTTP_EMBED_MODEL ( 'lmstudio-bge', 'lmstudio', 'bge-large-en-v1.5', 1024, NULL, NULL, 30, 'LM Studio BGE large'); -- 4. Any OpenAI-compatible custom endpoint (key as needed) DB.DBA.ONNX_API_KEY_SET('ONNX_API_KEY_MYHOST', ''); DB.DBA.ONNX_REGISTER_HTTP_EMBED_MODEL ( 'my-remote', 'openai_compatible', 'bge-large-en', 1024, 'https://my-host.example.com/v1/embeddings', 'ONNX_API_KEY_MYHOST', 30, 'Self-hosted BGE via LiteLLM'); ``` After registration, reference the model by its `_model_name` alias in DDL: SQL ``` CREATE VECTOR INDEX DOCS_VEC ON DB.DBA.DOCS (content) USING 'openai-small' WITH (dimensions = 1536, distance_metric = 'cosine', hnsw_m = 32, ef_construction = 400); ``` Tip For high-dimensional models such as 1536-dim OpenAI embeddings, raise `hnsw_m` to 32 and `ef_construction` to 400 for better graph quality. See [Choosing a model](#choosing-a-model) and [index parameters](urn:osdi:sitevirtuoso/docs/vector-index/index-parameters/). ### HTTP request details When the server needs an embedding from a remote model, it performs the following exchange with the provider endpoint: **Request** — an HTTP `POST` to the configured endpoint with a JSON body. A single text sends a string; a batch sends an array (see [Request batching](#request-batching)): JSON ``` { "model": "text-embedding-3-small", "input": ["first text to embed", "second text to embed"] } ``` **Authorization** — when the model has a key name, Virtuoso sends an `Authorization: Bearer ` header, where `` is the value stored in the registry under the `provider_api_key_name` of the model row. Store the key before the model is first used; a provider that requires one rejects the requests (`401`) otherwise. A model registered with no key name (Ollama or LM Studio, typically) sends no `Authorization` header. **Response parsing** — the server reads the JSON response and extracts every `data[].embedding`, placing each vector by the `index` the provider reports rather than by the order the items arrive. This is the OpenAI embeddings response shape, which all supported providers implement. Providers are not required to answer in order, and pairing by arrival would silently attach one text's vector to another text, so a response whose item count does not match the request, or which repeats or omits an index, fails with an error instead. **Dimension validation** — the server checks that the length of every returned embedding matches the `_dimensions` value registered for the model, and fails the call rather than storing a mismatched vector: Output ``` HTTP embedding dimension 3 at batch position 0 does not match registered model dimension 4 ``` **Retries and errors** — a `429`, `500`, `502`, `503` or `504`, a timeout and a connection error are retried, up to three attempts in all, waiting for a numeric `Retry-After` (at most 30 seconds) or else 0.5 s and then 1 s. Any other status fails at once. The caller then gets a generic error such as `HTTP embedding provider request failed after 1 attempt(s); see the server log for the provider response.`, the server log gets one line per failed attempt, and the model row's `provider_consecutive_failures`, `provider_last_error` and `provider_last_error_at` record the failure until the next success. ### Request batching A backfill or refresh embeds texts in batches of `__vec_embed_batch_size` (32 by default) and sends each batch as **one request per `provider_max_batch` inputs**, rather than one request per text. With the defaults, 57,600 documents take 1,800 requests instead of 57,600; with `__vec_embed_batch_size` raised to 128 — OpenAI's default request size — they take 450. Defaults are deliberately well below each provider's documented ceiling: | provider | default inputs per request | | --- | --- | | `openai` | 128 | | `ollama`, `lmstudio` | 32 | | `openai_compatible` | 32 | OpenAI accepts 2048 inputs per request but also caps a request at 300k tokens, so a 2048-input batch of ordinary documents breaks the token limit long before the input limit — the win is in going from 1 to ~100, not in reaching 2048. Local servers hold the whole batch in memory, so they get a smaller default. Override per model when a gateway or self-hosted server has a different limit: SQL ``` UPDATE DB.DBA.ONNX_MODELS SET provider_max_batch = 64 WHERE model_name = 'my-openai-compatible-model'; ``` Note A retry re-sends the whole chunk. These endpoints are stateless and pure, so re-embedding a text is wasted work but never incorrect. Note A text query against a remote-backed index embeds its query text over HTTP before it searches, so query latency includes the provider round-trip (typically hundreds of milliseconds for a hosted API). Depending on the query's shape that can be more than one request: measured, a `WHERE X_SIMILAR (…) > x` filter made one, while `ORDER BY VECTOR_DISTANCE (…)` made two. Passing a vector embedded once with `ONNX_EMBED_TEXT_FOR_MODEL` makes exactly one. Stored documents are embedded at index-build and maintenance time, not at query time. --- ## Choosing a model The right model depends on your priorities across latency, cost, quality, privacy, and vector dimensions. Use this comparison to decide. ### Comparison | Factor | Local granite (bundled) | OpenAI (small) | Ollama | LM Studio | | --- | --- | --- | --- | --- | | **Latency per embed** | ~5 ms for a short text, measured (helper process on the host) | ~300–800 ms (network) | ~10–100 ms (localhost) | ~10–100 ms (localhost) | | **Cost** | Free | Paid (per-token API billing) | Free (self-hosted) | Free (self-hosted) | | **Quality** | Good (English) | Generally higher | Varies by model | Varies by model | | **Privacy** | Data stays on the host | Data sent to OpenAI API | Data stays on its host | Data stays on its host | | **Dimensions** | 384 | 1536 | Model-dependent (e.g. 1024) | Model-dependent (e.g. 1024) | | **Setup** | None (auto-provisioned) | API key required | Run Ollama server | Run LM Studio server | | **Network dependency** | None | Required | None (localhost) | None (localhost) | The remote latencies are typical ranges, not measurements; measure from your own host. ### Factors in detail - **Latency** — Local ONNX embedding of a short text takes about 5 ms (measured), including the round trip to the helper process on the same host. A remote provider adds a network round-trip: hosted OpenAI is the slowest (hundreds of milliseconds from a typical data center), while a local Ollama or LM Studio server on the same host is faster but still slower than the bundled model's helper. This matters most for interactive query latency, since each query embeds the query text. - **Cost** — The bundled granite model and any self-hosted Ollama/LM Studio model are free to run (you own the hardware). OpenAI is a paid API billed per token. - **Quality** — The bundled model has 30M parameters. Larger models — OpenAI's `text-embedding-3-small`, or larger self-hosted BGE or `mxbai-embed-large` checkpoints — can rank better, especially on longer or more technical text; measure on your own data before switching, since a model change means rebuilding every index that uses it. - **Privacy** — With a local ONNX model, your text never leaves the Virtuoso host. With Ollama or LM Studio, text goes to that server — over loopback when it runs on the same host. With OpenAI, your text is sent to a third-party API. If your data is sensitive or regulated, prefer a local model. - **Dimensions** — Dimension count affects index memory and HNSW build quality. 384-dim vectors from the bundled model are compact; 1536-dim OpenAI vectors are larger and benefit from higher `hnsw_m` / `ef_construction`. The dimensions you choose are fixed for the life of the index. Some models support **Matryoshka Representation Learning (MRL)**, which allows you to truncate vectors to a lower dimension while retaining most of the semantic quality — for example, OpenAI `text-embedding-3-small` (1536-dim native) can be reduced to 256 or 512 dimensions via `WITH (reduce = 'truncate', dimensions = 256)`. See [Index parameters: Matryoshka reduction](urn:osdi:sitevirtuoso/docs/vector-index/index-parameters/#matryoshka-dimension-reduction). ### Recommendation Tip **Start with the local `granite-384-local` model.** It is auto-provisioned, free, fast, and keeps your data on the host — enough to validate your schema, queries and recall, and for many production workloads. Consider a larger model — hosted, such as OpenAI `text-embedding-3-small`, or self-hosted through Ollama, LM Studio or another OpenAI-compatible server — when an evaluation on your own data shows it ranks better, and you can accept its latency, cost and (for a hosted API) the data-egress trade-off. Remember that a vector index is bound to one model for its lifetime. To change models, drop the index, register or confirm the new model, and recreate the index with the new model name and dimensions. --- ## Using a custom ONNX model You are not limited to the bundled granite model. You can register any ONNX text-embedding model that meets the requirements below. This is useful when you need a different language, a larger model for better quality, or a quantized model for a memory-constrained host. ### Model file requirements - **Format**: an ONNX graph (`.onnx`). - **Inputs**: two int64 tensors of shape `[batch, sequence]` — token ids and attention mask, by default named `input_ids` and `attention_mask`. The helper tokenizes the text and fills both. - **Output**: **one pooled vector per input**, shape `[batch, dimension]`. The helper does no pooling and no normalisation: it returns the output tensor's row as it is, so a graph whose output is `last_hidden_state` (one vector per token) does not work. A stock Hugging Face ONNX export has exactly that output; [the export tool](#registering-a-custom-model) adds the pooling. - **A tokenizer the helper implements**: a WordPiece `vocab.txt` (BERT-style, uncased), or a byte-level BPE `vocab.json` with `merges.txt` in the same directory (RoBERTa- and GPT-2-style). The file extension decides which: a path ending in `.json` is BPE. SentencePiece tokenizers are not supported. - **A maximum sequence length**: longer text is truncated to it. The helper accepts 2–8192 tokens and defaults to 128 when none is given. ### How text is normalised before tokenisation This section applies to **WordPiece** models (`vocab.txt`). A byte-level BPE model — the bundled granite among them — is cased: the helper strips leading and trailing whitespace and then tokenises exactly as Hugging Face's RoBERTa tokenizer does, which the test suite checks against Hugging Face's own token ids. For WordPiece, the server normalises text the way BERT's own tokenizer does with `do_lower_case=True` — lowercase, then decompose and drop combining marks — so that `cafe`, `CAFE`, `café` and `CAFÉ` all reach the model as the same token. Text that arrives already decomposed (`cafe` + U+0301) normalises identically. Because lowercasing is always applied, the WordPiece path suits uncased vocabularies. Covered: | block | what it gives you | | --- | --- | | ASCII | case folding | | U+00C0–U+024F Latin-1 Supplement, Latin Extended-A and -B | Western and Central European accents | | U+1E00–U+1EFF Latin Extended Additional | Vietnamese | | U+0370–U+04FF Greek and Cyrillic | case folding | | U+0300–U+036F combining marks | dropped, so already-decomposed text behaves the same | For these, Virtuoso produces the same token ids — and so the same embedding — as `transformers` does on the same vocabulary. **Not covered**, so worth knowing before you index such a corpus: - **CJK.** BERT inserts spaces around CJK codepoints so each character becomes its own token; Virtuoso does not. In practice this matters less than it sounds with an English vocabulary: `bert-base-uncased` holds only 488 CJK tokens, so `transformers` itself tokenises most Chinese to `[UNK]` and Virtuoso's answer is already close to it. It becomes a real limitation with a CJK-capable vocabulary — which in practice means a multilingual checkpoint, and those usually want SentencePiece rather than WordPiece. - **Greek final sigma.** A word-final capital sigma lowercases to U+03C2 in Python and to U+03C3 here; a per-codepoint table cannot see word position. Both are in the vocabulary, so a word ending in sigma may tokenise one piece differently. - **Full Unicode NFD.** Only the blocks above decompose. ### Registering a custom model Place the `.onnx` model file and the tokenizer files on the Virtuoso host where the server process can read them. Then register the model in `ONNX_MODELS` and load it, or load it directly with `ONNX_LOAD_MODEL`. Important Readable by the server process is necessary but **not sufficient**. `ONNX_LOAD_MODEL` requires DBA, and the model path — and every other path in the `options` string — must be either under `[ONNX] ModelRoot` / the plugin `LoadPath`, or inside `DirsAllowed`. A path outside all of them is refused before ONNX Runtime sees it: Output ``` *** Error 42000: [Virtuoso Driver][Virtuoso Server]FA003: Access to '/srv/elsewhere/model.onnx' is denied due to access control in ini file ``` The simplest placement is a directory under `ModelRoot`, alongside the bundled model. The `options` string of `ONNX_LOAD_MODEL` tells the runtime how to feed and read the model. The format is a semicolon-separated list of `key=value` pairs: Output ``` input=input_ids;input2=attention_mask;output=pooled_output;vocab=/path/to/vocab.txt;max_seq_len=256 ``` | Option key | Meaning | | --- | --- | | `input` | Name of the token-ids input tensor | | `input2` | Name of the attention-mask input tensor | | `output` | Name of the output tensor to read the embedding from | | `vocab` | Path of the tokenizer vocabulary: `vocab.txt` (WordPiece) or `vocab.json` (byte-level BPE, with `merges.txt` beside it) | | `max_seq_len` | Maximum input sequence length in tokens, 2–8192 (default 128); longer text is truncated | | `intra_threads`, `inter_threads` | ONNX Runtime threads per helper (default 1 each) — see [Helper thread counts](#helper-thread-counts) | Important The exact input and output tensor names depend on how the model was exported. Inspect the model with Netron or `onnxruntime`'s session API to find the correct names before constructing the `options` string. A wrong tensor name will cause the load or the embed call to fail. The export tool that ships with the plugin, `virtuoso-export-embedding-model` (installed in `bin/`), turns a Hugging Face sentence-transformers model into this form. It reads the model's own pooling and normalisation configuration rather than assuming them, wraps the transformer in them, exports a graph with the inputs `input_ids` and `attention_mask` and the output `pooled_output`, and copies the tokenizer files beside it. `--dry-run` reads the configuration only — no PyTorch, and no network for a local directory: Shell ``` virtuoso-export-embedding-model ./granite-embedding-30m-english /dev/null --dry-run # plan: pooling=cls normalize=1 inferred=0 inputs=input_ids,attention_mask output=pooled_output max_seq_len=512 tokenizer=bpe ``` A real export needs `torch` and `transformers`, and verifies its output against the source model. For a model published in bf16, `--bf16-weights` halves the file without changing a single output value; the bundled model is stored that way. When it finishes, the tool prints the `ONNX_LOAD_MODEL` call and the registration for the files it wrote. Register the model with its `output_shape` as well — a vector index needs the dimension from it: SQL ``` INSERT INTO DB.DBA.ONNX_MODELS (model_name, model_path, model_type, input_names, output_name, output_shape, vocab_path, max_seq_len) VALUES ('my-custom-model', '/opt/models/my-model/model.onnx', 'text_embedding', 'input_ids,attention_mask', 'pooled_output', '1,384', '/opt/models/my-model/vocab.json', 512); -- Load it; returns the handle: SELECT DB.DBA.ONNX_MODEL_REFRESH ('my-custom-model'); -- Verify it produces the expected dimension (384 here): SELECT length (DB.DBA.ONNX_EMBED_TEXT_FOR_MODEL ('my-custom-model', 'hello world')); ``` Without `output_shape`, the model embeds text but cannot back a vector index: `CREATE VECTOR INDEX … USING 'my-custom-model'` fails with `VEC13: Model my-custom-model does not declare an output shape, so this index needs WITH (dimensions = ).`, and even with `dimensions` given, the backfill fails with `Vector index DOCS_C_VEC: its model my-custom-model is not available: VEC52`. Set the column and the index builds: SQL ``` UPDATE DB.DBA.ONNX_MODELS SET output_shape = '1,384' WHERE model_name = 'my-custom-model'; ``` To replace the model file later, unload the old handle before refreshing — see [ONNX_MODEL_REFRESH](#onnx_model_refresh). Tip The embedding model is good for more than search. Sentence embeddings give you **extractive summarisation** with no extra model — embed each sentence, take the document centroid, keep the closest. `DB.DBA.ONNX_SUMMARIZE (text, keep, model_name)` does exactly that, with `keep` defaulting to 3 sentences and the model to `granite-384-local`, and returns the kept sentences in their original order: SQL ``` SELECT DB.DBA.ONNX_SUMMARIZE ('Virtuoso stores RDF triples in a column-wise quad table. It can index the text of those triples with a vector index. The weather in Boston was cold yesterday. A vector index answers semantic queries with SPARQL or SQL. Queries embed their text with the same model as the data.', 2); -- It can index the text of those triples with a vector index. A vector index answers semantic queries with SPARQL or SQL. ``` --- ## Next steps - **[SQL vector index how-to](urn:osdi:sitevirtuoso/docs/vector-index/howto-sql-vector-index/)** — Create, query, and maintain a relational SQL vector index bound to one of these models - **[SPARQL vector index how-to](urn:osdi:sitevirtuoso/docs/vector-index/howto-sparql-vector-index/)** — Use vector indexes with RDF data and the `bif:vector_search` magic predicate - **[Index parameters](urn:osdi:sitevirtuoso/docs/vector-index/index-parameters/)** — HNSW build and query parameters that control the accuracy/speed trade-off ## Related content **Reference:** - [ONNX Runtime](https://github.com/microsoft/onnxruntime) — Microsoft ONNX Runtime project - [granite-embedding-30m-english](https://huggingface.co/ibm-granite/granite-embedding-30m-english) — The bundled embedding model on Hugging Face - [OpenAI embeddings docs](https://platform.openai.com/docs/guides/embeddings) — OpenAI embeddings API reference [PreviousVector search over RDF literals (knnSimilar)](urn:osdi:sitevirtuoso/docs/vector-index/knn-similar/) [NextIndex parameters and tuning for the Virtuoso vector index](urn:osdi:sitevirtuoso/docs/vector-index/index-parameters/) On this page 1. [Overview](#overview) 2. [The ONNX_MODELS table](#the-onnx_models-table) 3. [Local ONNX models](#local-onnx-models) 4. [Remote HTTP embedding providers](#remote-http-embedding-providers) 5. [Choosing a model](#choosing-a-model) 6. [Using a custom ONNX model](#using-a-custom-onnx-model) 7. [Next steps](#next-steps) 8. [Related content](#related-content)