Best embedding model for RAG: how to choose by size, dimensions, languages and cost

There is no single best embedding model for RAG. The right one depends on your languages, chunk length, vector size, where the model must run and what you pay per million tokens. The MTEB leaderboard narrows the field, but a short recall test on your own documents makes the final call.

What an embedding model decides in a RAG system

An embedding model turns each chunk of your documents, and each user query, into a vector. Retrieval then returns the chunks whose vectors sit closest to the query vector. If the model places a question far from the paragraph that answers it, the generator never sees that paragraph, whatever language model you put after it.

The choice also fixes things that are expensive to change later. The vector size sets your storage and index memory. The input limit sets how long a chunk can be before text is cut off. The vector space itself belongs to that model: Google's Gemini embeddings guide says gemini-embedding-001 and gemini-embedding-2 produce incompatible spaces, so upgrading means re-embedding all existing data. Expect the same whenever you change models.

Decision table: the questions that settle it

Answer these before you open a leaderboard. The model names are examples drawn from the official docs cited in this guide, not rankings.

How to read the MTEB leaderboard for the best embedding model for RAG

The MTEB leaderboard (opens in a new tab) is a Hugging Face Space run by the MTEB project. It ranks embedding models on public benchmarks, such as MTEB(eng, v2). The scores are useful for building a shortlist. They are less useful for picking a winner.

The original MTEB paper (opens in a new tab) covered 8 embedding tasks, 58 datasets and 112 languages, and benchmarked 33 models. Its authors found that no particular text embedding method dominates across all tasks. The follow-up MMTEB paper expanded this to over 500 evaluation tasks across 250+ languages. It reported that the best publicly available model was multilingual-e5-large-instruct, at 560 million parameters, while larger LLM-based models led only on some languages and task categories.

Three habits make the leaderboard more useful:

• Look at retrieval results, not only the overall average. MTEB also scores classification, clustering, reranking, STS and bitext mining, which matter less for RAG.
• Pick the benchmark that matches your languages. An English score says little about Portuguese queries.
• Filter out models too large for your hardware. The Sentence Transformers docs give this advice and add that leaderboard leaders do not necessarily do well on your tasks.

Open models you can run yourself with Sentence Transformers

Sentence Transformers is a Python library for loading and running embedding models locally. Its pretrained models page (opens in a new tab) names all-mpnet-base-v2 as the best-quality general model of its all-* set, and says all-MiniLM-L6-v2 is 5 times faster while still offering good quality. Those models were trained on more than 1 billion training pairs.

The all-MiniLM-L6-v2 model card lists 384 dimensions, truncation after 256 word pieces and an apache-2.0 license. It is intended as a sentence and short paragraph encoder, so long chunks lose their tail.

For several languages, the same page lists paraphrase-multilingual-MiniLM-L12-v2 and paraphrase-multilingual-mpnet-base-v2, trained on parallel data for 50+ languages. LaBSE supports 109 languages and is aimed at finding translation pairs rather than general search.

Running a model yourself means no per-token bill and no text leaving your machines. You pay in hardware, and in the work of hosting and scaling the encoder.

Hosted embedding APIs and their pricing units

Hosted APIs charge per input token. There is no output price for embeddings.

OpenAI's embeddings guide (opens in a new tab) lists text-embedding-3-small at 1536 dimensions and text-embedding-3-large at 3072, both with an 8192-token input limit. The guide reports MTEB scores of 62.3% and 64.6%, against 61.0% for the older text-embedding-ada-002. OpenAI's pricing page lists $0.02, $0.13 and $0.10 per 1M input tokens for the three models.

Google's Gemini embeddings guide (opens in a new tab) lists two models. gemini-embedding-2 is multimodal (text, images, video, audio and PDFs), covers over 100 languages and accepts 8,192 tokens. gemini-embedding-001 is text-only with a 2,048-token limit. The Gemini pricing page lists gemini-embedding-2 text at $0.20 per 1M tokens on the paid tier and $0.10 through Batch, with a free tier. Image, audio and video inputs have their own rates.

Estimate the re-embedding bill

An illustrative corpus of 100 million tokens would cost $2 to embed with text-embedding-3-small, $13 with text-embedding-3-large, and $20 with gemini-embedding-2 text at the standard rate. These figures are simple multiplication from the list prices above. Every model switch repeats that bill, plus the time to rebuild the index.

Small on-device models such as EmbeddingGemma

Google's EmbeddingGemma page (opens in a new tab) now leads with EmbeddingGemma 2. It is a 740M parameter model that maps text, images, audio and video into one 768-dimensional space. Text and source code inputs can reach 8,192 tokens across over 100 languages. Google says it runs locally and offline on consumer GPUs or CPUs, and releases it with open weights under the Apache 2.0 License.

The first version is still documented. EmbeddingGemma 1 has 308M parameters, a 2K token context and, per Google, can run on less than 200MB of RAM with quantization. Its Hugging Face card, google/embeddinggemma-300m, notes that activations do not support float16; use float32 or bfloat16 instead.

For a RAG system, the appeal is privacy and zero per-token cost: documents and queries are embedded on the user's own hardware. The 2K context of version 1 means chunks must stay short.

Dimensions, storage and re-embedding cost

Vector size drives storage. As illustrative arithmetic, assuming 32-bit floats at 4 bytes each, a 3072-dimension vector takes 12,288 bytes. One million chunks then need about 12.3 GB of raw vectors before any index overhead. At 768 dimensions, or 3,072 bytes per vector, the same chunks need about 3.1 GB.

Several models let you shorten vectors instead of switching models. OpenAI's dimensions parameter cuts text-embedding-3-large from 3072 to, for example, 1024 for stores with a 1024-dimension limit. OpenAI's guide reports that a text-embedding-3-large vector shortened to 256 still outperforms a full 1536-dimension text-embedding-ada-002 vector on MTEB. Gemini accepts output_dimensionality from 128 to 3072 and recommends 768, 1536 or 3072. EmbeddingGemma truncates to 128, 256, 512 or 768 through Matryoshka Representation Learning.

Changing the dimension later still means re-embedding, since stored vectors cannot grow back.

Test on your own documents before committing

Build a small evaluation set: real queries, each paired with the chunk that answers it. Then measure how often each candidate model puts that chunk in its top results. This script uses calls shown in the Sentence Transformers docs; the data is illustrative.

from sentence_transformers import SentenceTransformer

# Illustrative data: replace with your own chunks and real user queries.
chunks = [
    "Refunds are issued to the original card within 14 days.",
    "Orders over $50 ship free inside the country.",
    "You can change your delivery address until the order is packed.",
]
queries = [  # (query, index of the chunk that answers it)
    ("how long until I get my money back", 0),
    ("is shipping free", 1),
    ("can I update where my parcel goes", 2),
]

k = min(3, len(chunks))
for name in ["sentence-transformers/all-MiniLM-L6-v2",
             "sentence-transformers/all-mpnet-base-v2"]:
    model = SentenceTransformer(name)
    doc_emb = model.encode(chunks)
    query_emb = model.encode([q for q, _ in queries])
    scores = model.similarity(query_emb, doc_emb)
    hits = sum(gold in scores[i].topk(k).indices.tolist()
               for i, (_, gold) in enumerate(queries))
    print(name, f"recall@{k} = {hits / len(queries):.2f}")

Use a few hundred queries rather than three, drawn from search logs or support tickets. Chunk the documents the way production will. Add a hosted model to the loop through its own SDK, and compare recall alongside cost per million tokens and vector size.