Model selection
Every embedder and reranker on the shortlist is scored on a public Czech retrieval pack before the DealBench dev split decides. Only licences that permit commercial use can become a pick; others are measured and shown as research-only. Scores are the slice's MTEB main metric, in points.
Pick rule. Highest mean main score over the stage's slices (all slices for embedders, reranking slices for rerankers), among models with a licence that allows commercial use and a full (not subsampled) score on every one of them; ties go to the smaller model. No pick is made with fewer than two such contenders. Provisional until confirmed on the DealBench dev split. Generated 2026-10-01T00:57:24+00:00.
Dense embedders
decided by the mean over these slices| Model | Licence | Hosting | VRAM | belebele-cs-q-en-doc nDCG@10 | wiki-cs-rerank MAP | Mean | Status |
|---|---|---|---|---|---|---|---|
| bge-m3 Baseline (D1); no query instruction. | MIT commercial | self-hosted | 1.1 GB | 93.0 | 88.8 | 90.9 | not scored on belebele-ces-ces, belebele-en-q-cs-doc, webfaq-ces, wiki-cs |
| qwen3-emb-8b | Apache-2.0 commercial | self-hosted | 15.1 GB | — | — | — | not scored on belebele-ces-ces, belebele-cs-q-en-doc, belebele-en-q-cs-doc, webfaq-ces, wiki-cs, wiki-cs-rerank |
| nemotron-3-embed-8b No published Czech evaluation. | OpenMDW-1.1 commercial | self-hosted | 15.9 GB | — | — | — | not scored on belebele-ces-ces, belebele-cs-q-en-doc, belebele-en-q-cs-doc, webfaq-ces, wiki-cs, wiki-cs-rerank |
| pplx-embed-4b Instruction-free by design (model card). | MIT commercial | self-hosted | 8 GB | — | — | — | not scored on belebele-ces-ces, belebele-cs-q-en-doc, belebele-en-q-cs-doc, webfaq-ces, wiki-cs, wiki-cs-rerank |
| harrier-27b Gemma-3 backbone; needs the H100 NVL (bee). | MIT commercial | self-hosted | 54 GB | — | — | — | not scored on belebele-ces-ces, belebele-cs-q-en-doc, belebele-en-q-cs-doc, webfaq-ces, wiki-cs, wiki-cs-rerank |
| octen-8b | Apache-2.0 commercial | self-hosted | 15.1 GB | — | — | — | not scored on belebele-ces-ces, belebele-cs-q-en-doc, belebele-en-q-cs-doc, webfaq-ces, wiki-cs, wiki-cs-rerank |
| granite-311m CPU tier. | Apache-2.0 commercial | self-hosted | 0.6 GB | — | — | — | not scored on belebele-ces-ces, belebele-cs-q-en-doc, belebele-en-q-cs-doc, webfaq-ces, wiki-cs, wiki-cs-rerank |
| e-infra-qwen3-emb-4b Hosted fallback on the e-INFRA CZ API; exact served revision unknown. | Apache-2.0 commercial | e-INFRA API | — | — | — | — | not scored on belebele-ces-ces, belebele-cs-q-en-doc, belebele-en-q-cs-doc, webfaq-ces, wiki-cs, wiki-cs-rerank |
| voyage-4-large | commercial API commercial | API | — | — | — | — | not scored on belebele-ces-ces, belebele-cs-q-en-doc, belebele-en-q-cs-doc, webfaq-ces, wiki-cs, wiki-cs-rerank |
| gemini-embedding-001 Vertex EU; backend not implemented yet. | commercial API commercial | API | — | — | — | — | not scored on belebele-ces-ces, belebele-cs-q-en-doc, belebele-en-q-cs-doc, webfaq-ces, wiki-cs, wiki-cs-rerank |
* scored on a query subsample, which cannot decide a pick.
Rerankers
decided by the mean over these slices| Model | Licence | Hosting | VRAM | wiki-cs-rerank MAP | Mean | Status |
|---|---|---|---|---|---|---|
| bge-reranker-v2-m3 | Apache-2.0 commercial | self-hosted | 1.1 GB | 96.5 * | 96.5 | only a query subsample on wiki-cs-rerank |
| qwen3-rr-4b Prompt format from the model card / vLLM example; served via seq-cls override. | Apache-2.0 commercial | self-hosted | 8 GB | — | — | not scored on wiki-cs-rerank |
| nemotron-rr-1b vLLM needs the score chat template from the model card, else scores are silently wrong. | OpenMDW-1.1 + Llama 3.2 commercial | self-hosted | 2.5 GB | — | — | not scored on wiki-cs-rerank |
| jina-rr-v3.5 research-only (non-commercial licence). | CC-BY-NC-4.0 research-only | self-hosted | 1.2 GB | — | — | licence does not allow commercial use |
| querit-4b Optional wildcard; custom loading, not wired to a backend yet. | Apache-2.0 commercial | self-hosted | — | — | — | not scored on wiki-cs-rerank |
| e-infra-qwen3-rr-4b Hosted fallback on the e-INFRA CZ API. | Apache-2.0 commercial | e-INFRA API | — | — | — | not scored on wiki-cs-rerank |
| cohere-rerank-v3.5 | commercial API commercial | API | — | — | — | not scored on wiki-cs-rerank |
* scored on a query subsample, which cannot decide a pick.
Slices
| Slice | Kind | Description | Published reference |
|---|---|---|---|
| belebele-cs-q-en-doc | retrieval | Czech questions over English passages (Belebele eng_Latn-ces_Latn) | BAAI/bge-m3 nDCG@10 93.0 |
| wiki-cs-rerank | reranking | Rerank 1 positive among 9 hard negatives (Czech Wikipedia) | BAAI/bge-m3 MAP 88.7 |
Compute
Open models run as offline batch jobs on MetaCentrum (PBS Pro, Singularity, vLLM). Their outputs are imported into the request cache, so rerunning the evaluation needs no GPU access. Latency for API stacks is measured from Prague; for open models on the node that runs them.
Computational resources were provided by the e-INFRA CZ project (ID:90254), supported by the Ministry of Education, Youth and Sports of the Czech Republic.