← Back to progress
Selecting a model for multilingual semantic search.
LA Referencia evaluated different alternatives for representing queries and metadata by meaning. The selection considered retrieval quality, as well as local execution, vector size, and integration with Solr and VuFind.
Starting point
What is an embedding model?
It is a model that transforms text into numerical representations. These representations make it possible to compare a query with titles or abstracts by meaning, even when they do not share the same words.
This does not translate the document. The model places related texts in a shared mathematical space so they can be compared.
Selection criteria
Quality was one criterion, but not the only one.
The selected alternative needed to operate within a regional infrastructure and be applicable to a collection of millions of records.
Cross-language retrieval
Connect queries and documents expressed in different languages, vocabularies, and writing systems.
Local execution
Run an open model without mandatory dependence on external commercial services.
Integration
Fit into the existing Apache Solr, VuFind, and indexing architecture.
Scale
Keep model size and vector dimensions manageable when generating, storing, and querying millions of representations.
Alternatives analyzed
From a broad exploration to six comparable references.
The initial evaluation examined approximately thirty configurations from different families, sizes, and quantizations. The consolidated comparison brought together six models with different profiles.
≈30initial configurations
→34languages in the initial framework
→6consolidated references
| Model | Parameters | Dimensions | Cross-lingual MRR | Role in the process |
|---|
Ollama, LM Studio, and OpenRouter were used as environments or services for running and comparing configurations. They are not alternative models in this table.
How they were evaluated
Two scenarios for answering different questions.
A pilot showed how the approach worked within VuFind and Solr. A controlled corpus made it possible to compare models using the same documents, queries, and metrics.
A
Applied pilot
LA Referencia index in VuFind
- Records
- 30,063
- Languages
- 9
- Queries
- 36
- Topics
- 4
Compared keyword, semantic, and hybrid search using hits@10 and nDCG@10.
B
Controlled comparison
Multilingual OpenAlex corpus
- Documents
- 50,000
- Languages
- 10
- Known-item queries
- 19,200
- Thematic queries
- 1,440
Compared MRR, Recall@1, Recall@10, and nDCG@10 under consistent conditions.
Pilot results
Semantic representation expanded retrieval across languages.
In the 30,063-record pilot index, the evaluation reviewed how many relevant documents appeared among the top ten results. These figures refer to the evaluated snapshot, not the complete collection.
When the words do not match
The change was especially visible across writing systems.
For queries in Chinese and Japanese, keyword search found no relevant documents among the pilot’s top ten results; semantic search retrieved 9.8 and 9.3 respectively. In Arabic, hybrid search reached 9.5.
Implementation decision
The highest-scoring model was not automatically the selected model.
Qwen3-Embedding achieved the highest cross-lingual quality. For the first implementation, vector volume, model size, and integration experience were also considered.
Highest cross-lingual score
Qwen3-Embedding-8B
- Cross-lingual MRR
- 0.948
- Dimensions
- 4,096
- Parameters
- 8.8 billion
- Vector components
- 4× BGE-M3
Selected model · first year
BGE-M3
- Cross-lingual MRR
- 0.801
- Recall@10
- 0.897
- Dimensions
- 1,024
- Parameters
- 568 million
A choice for getting started and continuing to measure.
BGE-M3 was selected because it provides an appropriate balance among multilingual retrieval, local execution, vector dimensions, model size, and integration with Solr and VuFind. Qwen3-Embedding remains the higher-quality reference and EmbeddingGemma a compact alternative for later evaluations.
From A4 to Demo A2
The selected model was then applied to the complete collection.
The A4 result was incorporated into the A2 multilingual semantic search demo. There, BGE-M3 was applied to LA Referencia’s complete available collection, and the beta was published so users can explore it and provide feedback on the results and search experience.
How to read the evidence
Methodology and scope.
Evaluation period: January–June 2026. The initial framework included approximately thirty configurations and 34 languages; the final scenarios used 9 and 10 languages respectively.
MRR summarizes the position of the correct document; Recall@1 and Recall@10 indicate whether it appears first or among the first ten; hits@10 counts relevant documents on the first page; nDCG@10 also considers ranking quality.
Relationship with A2: the public beta over the complete collection belongs to Demo A2 and shows the later application of the model selected in A4.