Embeddings are sold as "search that understands meaning", and the demo is
always the same: type a paraphrase, get the right document, marvel. This
guide measures instead. We took the 990 customer-support tickets from the
synthetic dataset guide,
each with a known intent, embedded them with multilingual-e5-small over
our OpenAI-compatible embeddings endpoint,
and asked three questions with numbers attached:
- When a ticket is the query, is the nearest other ticket about the same thing? And is that better than a keyword index would do?
- When the query is a short search phrase in English, Romanian, German, Spanish or Chinese, against an English corpus?
- What do the two standard fixes, hybrid retrieval and an LLM reranker, add, and what do they cost?
The scoring proxy is strict on purpose: a retrieved ticket counts as a hit only if it carries the same intent label as the query. A ticket about a lost phone case retrieved for "where is my phone case" is a miss if the retrieved customer's actual problem was logging in. That is harsher than a human judge would be, and it is the same for every method compared here.
| precision@1 | precision@5 | |
|---|---|---|
| ticket as query, English, 989 candidates | ||
embeddings, query:/passage: prefixes | 72.6% | 60.2% |
| embeddings, no prefixes | 75.5% | 62.4% |
| TF-IDF keyword search | 78.7% | 68.9% |
| hybrid (reciprocal rank fusion of both) | 75.3% | 64.0% |
| short queries, 5 languages, 300 queries | ||
| TF-IDF, average of es / zh | 8.3% | 10.2% |
| embeddings, average of 5 languages | 48.3% | 42.2% |
| embeddings + 27B reranker on the top 10 | 74.3% | 50.9% |
Read the two halves separately. In the top half, a keyword index beats a small embedding model on its home turf. In the bottom half, the keyword index scores at chance (one in twelve intents is 8.3%) the moment the query is not in the corpus's language, the embedding model does not, and a reranker turns "usable" into "good".
The setup
Corpus: 990 generated tickets, 12 intents, 42 words on average, English.
Embedded with passage: in front, in batches of 64, one request in
flight:
| documents | 990 |
| input tokens | 60,617 (61 per ticket) |
| requests | 16 |
| wall time, sequential batches of 64 | 56 s (17.7 documents/s) |
| cost at $0.02 per million tokens | $0.0012 |
Per request the endpoint accepts up to 256 texts; with 16 requests in flight the four CPU workers behind it at the time of writing deliver roughly 80 documents a second. A million tickets of this size is about 61 million tokens, $1.20, and a few hours on one key.
from openai import OpenAI
import numpy as np
client = OpenAI(base_url="https://paraloncloud.com/v1", api_key=KEY)
def embed(texts, prefix=""):
out = []
for i in range(0, len(texts), 64):
r = client.embeddings.create(model="multilingual-e5-small", input=[prefix + t for t in texts[i:i+64]])
out += [d.embedding for d in r.data]
m = np.array(out, dtype=np.float32)
return m / np.linalg.norm(m, axis=1, keepdims=True) # unit length: cosine = dot product
P = embed(tickets, "passage: ")
q = embed(["where is my order"], "query: ")
top = np.argsort(-(q @ P.T)[0])[:10]
Nothing else is needed for a corpus of this size: a numpy matrix and a dot product. A vector database becomes worth its setup somewhere past a few hundred thousand vectors, not before.
Finding 1: same language, keyword search is as good or better
With each ticket as the query against the other 989, TF-IDF over unigrams and bigrams put a same-intent ticket first 78.7% of the time; the embedding model 75.5%. Reciprocal rank fusion of the two, the usual "hybrid search" recipe, landed between them.
This is not surprising once you look at what the two methods key on. A ticket that says "cancel", "refund" or "wrong size" shares those exact tokens with every other ticket of the same intent, and a keyword index is built to find exactly that. A 118-million-parameter embedding model compresses the ticket into 384 numbers that carry topic (a board game, a jacket, a laptop bag) about as strongly as they carry what the customer wants done, and the scoring here is about the latter. The intent proxy punishes that, and a human would too: a search for damaged cookware should return damaged cookware, not every ticket that mentions cookware.
If your corpus and your queries are in one language and your users type the words that appear in the documents, a keyword index is the baseline to beat, and a small embedding model does not beat it. Measure before you replace one with the other.
Finding 2: the E5 prefixes did not help here
intfloat/multilingual-e5-small is trained with query: and
passage: prefixes, and the model card asks for them. On this task they
cost three points of precision@1 (72.6% with, 75.5% without). The
prefixes tell the model that queries and passages are different kinds of
text; when the query is a passage, as in this leave-one-out test, that
is wrong information. For the short search phrases in the next section
the prefixes were kept, since there the query genuinely is a query. The
endpoint documentation recommends the
prefixes; take that as "measure with and without on your own data",
which is a two-line change in the script below.
Finding 3: across languages, keyword search is at chance and embeddings are not
The realistic search box is short and in whatever language the person speaks. We had the chat model write 300 search queries, 25 per intent, five each in English, Romanian, German, Spanish and Chinese ("Wo ist meine Kleidung?", "买的衣服物流停滞了怎么办", "stop delivery of my new cookware set"), and ran them against the English corpus.
| query language | TF-IDF p@1 | embeddings p@1 | embeddings p@5 |
|---|---|---|---|
| English | 55.0% | 60.0% | 49.0% |
| German | 18.3% | 58.3% | 51.0% |
| Spanish | 8.3% | 50.0% | 45.7% |
| Chinese | 8.3% | 46.7% | 38.0% |
| Romanian | 20.0% | 26.7% | 27.3% |
The keyword index has no way to connect "Kleidung" to "clothes"; its Spanish and Chinese scores are what random guessing gives. The embedding model connects them well enough to put a same-intent ticket first about half the time in German, Spanish and Chinese. Romanian is this model's weak language, at roughly half the German figure; a larger multilingual model would likely close that gap, and it is the first thing to test if Romanian matters to you.
The hybrid did not help across languages either: it averaged 41.0% p@1, below the embeddings alone, because fusing in a keyword ranking that is random adds noise, not signal.
By intent, the queries the embedding model handled worst were complaint
(16.8% p@5), product_question (20.0%) and wrong_item (28.0%), the
categories defined by what the customer wants rather than by a distinctive
vocabulary. cancel_order, account_login and
shipping_address_change, each with its own words, sat at 68 to 73%.
That is the same shape the judge guide in this series (16 September)
finds in this taxonomy from the other direction.
Finding 4: a reranker over the top ten fixes most of it
Retrieval gets ten candidates cheaply; a chat model can read ten short
tickets and a query and say which three match. We sent the embedding
top-10 for each of the 300 queries to qwen3.8-27b, with the answer
constrained to three indices by a JSON schema.
| query language | embeddings p@1 | + reranker p@1 | + reranker p@5 |
|---|---|---|---|
| English | 60.0% | 83.3% | 57.7% |
| German | 58.3% | 80.0% | 63.0% |
| Chinese | 46.7% | 78.3% | 46.7% |
| Spanish | 50.0% | 66.7% | 50.7% |
| Romanian | 26.7% | 63.3% | 36.3% |
| reranker cost, per query | |
|---|---|
| input / output tokens | 447 / 20 |
| price at list ($0.12 / $1.70 per million) | $0.000088, or $0.088 per 1,000 queries |
| latency, median | 1.63 s |
| failed requests | 0 of 300 |
Precision@1 rose by 17 to 37 points in every language, and Romanian more than doubled: the reranker reads Romanian fine, so it only needed the right ticket to be somewhere in the ten. Precision@5 moved less, because the reranker was asked for three and the rest of the five stayed in embedding order; ask it for five if that is the number you show.
The cost is the interesting part. At list price the reranking step is nine cents per thousand queries and 1.6 seconds. Embedding the query itself is about 0.2 seconds and a few thousandths of a cent. So a search that is cross-lingual, meaning-aware and right three times out of four at the top slot costs about a tenth of a cent per query, all from one key on one endpoint.
SCHEMA = {"type": "object",
"properties": {"ranked": {"type": "array", "minItems": 3, "maxItems": 3,
"items": {"type": "integer", "minimum": 1, "maximum": 10}}},
"required": ["ranked"]}
def rerank(query, candidates):
listing = "\n".join(f"[{i+1}] {t[:300]}" for i, t in enumerate(candidates))
r = client.chat.completions.create(
model="qwen3.8-27b",
messages=[{"role": "system", "content": "You rank customer-support tickets by how well they match a search query, "
"which may be in another language than the tickets. Return the numbers of the 3 best matches, best first."},
{"role": "user", "content": f"Query: {query}\n\nTickets:\n{listing}"}],
response_format={"type": "json_schema", "json_schema": {"name": "rank", "schema": SCHEMA}},
extra_body={"chat_template_kwargs": {"enable_thinking": False}},
temperature=0, max_tokens=40)
return [candidates[i - 1] for i in json.loads(r.choices[0].message.content)["ranked"]]
What to do with this
- One language, literal users: keyword index first. Add embeddings when you can show they help on your own queries, not because they are the modern thing.
- Several languages, or users who describe rather than name: embeddings, then rerank. Retrieve ten to twenty with the embedding, hand them to the chat model, show its order. Both calls go to the same endpoint with the same key.
- Score it the way this guide did. A few hundred queries with a known right answer and a keyword baseline take an afternoon and settle the argument. The script below prints every table above for your corpus.
- The reranker is where quality lives. The embedding model's job is to put the right document in the top ten cheaply; it did that far more often than it put it first. Spend the model budget on the reranker.
Scripts and data
search.py: embeds the corpus with and without prefixes, runs the leave-one-out and the multilingual tests against a TF-IDF baseline, prints precision@1 and @5, tokens, time and cost.hybrid_rerank.py: reciprocal-rank-fusion hybrid and the 27B reranker over the top 10, per language, with the reranker's cost and latency.gen_queries.pyand the 300 queries it produced,queries_300.json.- The corpus, 990 generated tickets with intent labels:
support_tickets_990.jsonl; put it next to the scripts.
pip install numpy scikit-learn requests
PRL_KEY=prlc_... python3 search.py
PRL_KEY=prlc_... python3 hybrid_rerank.py
The full run, embeddings and reranks and query generation, spent about
five cents of credit. A free key from the Console carries a
250,000-token trial, which covers indexing this corpus and a few hundred
reranked queries; after that it is the per-token price on
a premium key. n8n users get the same endpoint through the Embeddings
OpenAI node with the credential's Base URL set to
https://paraloncloud.com/v1; an n8n guide follows on 19 September.
Frequently asked questions
Why a 384-dimension model and not a larger one? Because it runs on the CPU nodes of the network at 6 ms per text, which is what makes the index a tenth of a cent. A 1024-dimension model such as bge-m3 ran about eight times slower on the same CPU in our checks (46 ms against 6 ms per text) and would be a GPU model at a GPU price; we did not measure its retrieval quality here. The reranker is the cheaper place to buy precision, and it is already measured above.
Does the prefix advice from the model card apply to me? Only your data can say; here it cost three points on ticket-to-ticket search and was kept for phrase queries. The script measures both in one run.
How do I store the vectors? For under a few hundred thousand documents, a numpy array on disk and a dot product. Beyond that, any vector store that takes 384-dimension float vectors; the endpoint returns plain JSON arrays, so nothing about it is specific to one store.
Can the reranker use a smaller model? qwen3-3b is on the same
endpoint at a quarter of the input price. We did not measure it as a
reranker; the script takes the model name from one line, so it is a
five-minute experiment.
