AnyRAG

We tested more than 40 RAG setups in Persian and English. Pick the one that fits your project.

What Reaches the Reader: A Comparison of Retrieval-Augmented Generation Setups in Persian and English

Arash Nasr EsfahaniUniversity of Tehran

Find your RAG in one minute

Which setup fits your project, what changes the choice, and why.

The explainer video is being prepared.

Where RAG loses its answers

The question

  • RAG (retrieval-augmented generation) finds passages; an LLM, the reader, writes the answer.
  • Search matches keywords (BM25), meaning (dense retrieval), or both.
  • Which search, ranking and budget choices matter in Persian and English?

The benchmark

  • 156 questions on 155 real reports, books and statutes
  • Six search pools, three difficulty tiers
  • Half Persian, half English, with annotated gold evidence
  • 90 answerable test questions

What was compared

  • Fixed in advance: four whole pipelines at two evidence budgets
  • Post hoc: search channels, chunk size, rewriting, embeddings, reranking, LLM selection, budgets, agents

The answer

Most wrong answers start before the reader: the evidence is not retrieved, or is cut from the pack.

Post hoc. The test fixed in advance missed its pass mark.

Evidence budget
Retrieved text the reader sees, in characters; 6,000 is the tight budget.
Evidence pack
The text the reader actually gets, cut to the budget.
Evidence bundle
Passages that together answer a question; a complete pack holds one.
Gold evidence
Passages marked as evidence; used only to score retrieval.
Test fixed in advance
Committed before any test answer was seen; not registered externally.
Post hoc
Run after the test answers were known; not confirmatory.

A Persian and English benchmark with gold evidence

  • Each question searches its whole pool
  • Examples below are development questions
T1

Local fact

One passage, list or table region answers it.

Whose consent is needed before a prosecution under the Bribery Act 2010 can be brought in England and Wales?

LAW-en

در سال ۱۴۰۰ دیوار چند کاربر را مسدود کرد و این تعداد چند برابر سال ۱۳۹۹ بود؟

In 1400, how many users did Divar block, and how many times the 1399 figure was that?

RPT-fa

T2

Within one document

One document suffices, but no single region does.

Wise's FY2024 annual report gives its total customer balances and, in the Wise Business section, the business deposits it holds. Roughly what share of total customer balances were business deposits?

RPT-en

در سال ۱۴۰۰ تقریباً چند نفر در دیوار بدون پرداخت هزینه آگهی گذاشتند؟

Roughly how many people posted ads on Divar free of charge in 1400?

RPT-fa

T3

Across documents

Every sufficient bundle spans two or more documents.

Turkcell and Vodafone Turkey both had their 2G licences extended to 30 April 2029. What fee did each operator agree for the extension, and how did the way they paid it differ?

RPT-en

حداقل سنی که برای متقاضی امتیاز روزنامه یا نشریه تعیین شده بود، از لایحهٔ قانونی مطبوعات ۱۳۳۴ تا قانون مطبوعات ۱۳۶۴ چه تغییری کرد؟

How did the minimum age set for an applicant for a newspaper or periodical licence change from the 1334 [1955] Press Bill to the 1364 [1985] Press Law?

LAW-fa

Each pool: 3 development and 5 test questions per tier, plus one unanswerable per split. Totals: 60 development, 96 test.

Six search pools

PoolDocumentsCharactersSources
Reports fa974.76 MAnnual and activity reports, 42 Iranian companies
Reports en1516.64 MAnnual or integrated reports, 11 international companies
Books fa95.70 MModern Persian prose, Wikisource (Kasravi, Pirnia, Bahar …)
Books en116.26 MProject Gutenberg non-fiction (Darwin, Mill, Tocqueville, Douglass …)
Statutes fa131.18 MIranian statutes, Wikisource (Civil Code, Commercial Codes, Direct Taxes Acts …)
Statutes en103.52 MEU regulations (GDPR, DSA, AI Act …) and UK Acts
Total15538.06 M58,218 chunks (indexed text pieces) of at most 1,000 characters
  • Born-digital text or checked transcriptions only; no scans.
  • Persian and English use different documents, so language and source are mixed.

How evidence is annotated

  1. Requirementsatomic points a correct answer must meet;
  2. Evidence passagesexact quotes, located by character position;
  3. Evidence bundlessets of passages that together suffice.

حداقل سنی که برای متقاضی امتیاز روزنامه یا نشریه تعیین شده بود، از لایحهٔ قانونی مطبوعات ۱۳۳۴ تا قانون مطبوعات ۱۳۶۴ چه تغییری کرد؟

press-bill-1334

درخواست کننده اجازه باید دارای شرایط ذیل باشد: … ب - سن او از سی سال کمتر نباشد.

The applicant must meet the following conditions: … b. be not younger than thirty.

press-law-1364

شخص حقیقی متقاضی صاحب امتیاز باید دارای شرایط زیر باشد: … ۲- دارا بودن حداقل ۲۵ سال.

A natural person applying for a licence must meet the following conditions: … 2. be at least 25 years old.

Expected answer 30 years under the 1334 bill, 25 years under the 1364 law: the minimum age fell by 5 years.

Fig. 1 A development question (T3) whose bundle spans two documents. Neither passage answers alone.

How the questions were built

  • Drafted by Claude Opus 5.5 agents; checked by a verifier and a review agent
  • Persian wording reviewed by a native speaker
  • 32 of 156 checked by hand: 31 accepted, 1 fixed

How a RAG pipeline answers

  • Search the pool, order or select candidates, pack them into the budget
  • All four pipelines share chunks, prompts and readers
The 3D view of a pipeline loads here.
The 2D view of a pipeline loads here.
Fig. 2 From documents to reader. Recorded runs on three development questions.

Four pipelines, fixed in advance

Pipeline What it does LLM tokens, × Hybrid Median latency
HybridbaselineBM25 (keywords) plus dense retrieval (meaning, bge-m3), fused by rank.1.001.8 s
HRHybrid + rerankerHybrid plus a cross-encoder reranker: a small model that rescores each passage. No LLM tokens.1.013.2 s
Lex-RAGour methodFuses BM25, dense and learned-sparse scores. One LLM call picks a bundle per question part; ranked passages fill the rest.2.04–2.324.0 s
SetR-p+filladapted selectorOur SetR adaptation: one chain-of-thought call picks a set from Hybrid's candidates; Hybrid fills the rest. Not the published system.3.39–3.675.5 s

6,000 characters, 96 test questions. Latency on a shared laptop GPU: only the order is reliable.

Readers and scoring

  • Primary reader: GLM-5.3-flash, also used for both selection calls.
  • Five readers from other model families answer the same packs.
  • A strict LLM judge: every required point met, nothing wrong or unsupported.

Choose your RAG

Answer a few questions; get a setup backed by the paper.

The chooser loads here.

Post hoc, 90 test questions. Your data may differ: test on your own questions.

Use it in Claude Code or Codex

An agent skill that reads your codebase and recommends a RAG setup, using the same rules and evidence as the chooser.

  • It scans your repo read-only: document languages, file types and size, plus any vector DB, embedding model, chunk size, reranker, GPU or LLM provider.
  • It asks only what it can't infer.
  • It returns a setup, the evidence behind it, and file-by-file changes to your existing stack.

Claude Code

git clone --depth 1 https://github.com/ArashNasrEsfahani/AnyRAG.git && mkdir -p ~/.claude/skills && cp -r AnyRAG/skills/choose-rag ~/.claude/skills/

Codex

git clone --depth 1 https://github.com/ArashNasrEsfahani/AnyRAG.git && mkdir -p ~/.codex/skills && cp -r AnyRAG/skills/choose-rag ~/.codex/skills/

Then, in your project, ask: Which RAG setup should I use for this project?

Its advice holds within the study's scope: Persian and English, 155 documents, one primary reader.

What the results show

Correct answers of 90 (primary reader, first judge). A range like 62–63 includes answers the judge could not score.

The main result

Lex-RAG is our method; Hybrid and Hybrid + reranker are the baselines.
Pipeline6,000 characters15,500 characters
Hybrid62–6368
Hybrid + reranker69–7169–70
Lex-RAG68–6967
SetR-p+fill6366–67

Test fixed in advance: not met

  • Pass mark: at least 5 of 90 above both baselines at 6,000 characters.
  • Result: 5 to 7 more than Hybrid, 0 to 3 fewer than Hybrid + reranker.
  • Neither primary test was significant. Everything below is post hoc.
Fig. 3 Where answers are lost
The figure loads here.

Gold evidence only: 83 of 90. Lost in retrieval: 15 to 18. Lost in packing: 11 to 15 more.

Fig. 4 How much text to show
The figure loads here.

Selection adds 15 to 16 answers at 3,000 characters (one run), 2 to 4 from 10,000. Plain hybrid climbs from about 50 to 72.

Fig. 5 Which parts of the pipeline matter
The figure loads here.

Hybrid 62 to 63, dense alone 53, BM25 alone 48. 2,000-character chunks lose 13 to 20. Rewriting and a second embedding model did not help.

Searching harder

  • Iterative search agent (hybrid index, reranker): 77 of 90, the best without annotations.
  • Up to five searches; about four times the tokens; median 10.5 s.
  • Its 6 to 8 lead over the reranker (69 to 71) is within the noise.

Reading whole documents

  • Helps only when the right documents are known.
  • Gold documents: 79 correct.
  • Top BM25 documents, cut to the same length: 24.

Persian and English

Two results do not depend on the documents: token cost and cross-language questions.

Token cost

1.5–1.8×

  • Persian costs 1.46 to 1.80× the tokens per character for five of six readers; Qwen3 235B 2.99×.
  • Set budgets in tokens: 6,000 Persian characters ≈ 10,000 English (primary reader).

Memory gap

20 and 6 of 45

  • No evidence: 20 of 45 English and 6 of 45 Persian correct.
  • Asked in the other language: still 18 and 4 or 5.
  • The gap follows the source texts, not the question language.

Questions in the other language

37 → 2

  • BM25 top 40: a complete bundle for only 2 and 8 of 45 (was 37 and 29).
  • Dense retrieval keeps 35 and 28 (was 38 and 33).
  • Reranked pipeline: 53 to 54 correct, down from 69 to 71.
Fig. 6 Token cost by reader; retrieval with translated questions
The figure loads here.

Persian costs more tokens under every tokenizer tested. Across languages BM25 collapses and dense holds: keep a dense channel. Strong

Persian spelling

  • Spelling varies: Arabic or Persian letters, non-joiners, three digit systems.
  • Without normalising, BM25 loses 5 to 8 of 45 complete bundles; dense at most one.
  • Normalise before BM25. Tentative, retrieval only

How far to trust this

One benchmark, one primary reader, LLM judges, 90 answerable questions.

The rest is post hoc

  • Other results reuse test questions whose answers were already seen.
  • Their p-values are descriptive; the advice needs a fresh test.

90 questions detect about 11 answers

  • 80% power only for gaps of about 11 answers (about 8 within one language).
  • Detecting a gap of 5 needs about 400 to 460 questions.

Judges and the human review disagree

  • A second judge agrees on 91% (kappa 0.76) but reorders the middle pipelines.
  • The human review: 30 questions, blind to the method, by the author alone.
  • Its significant +8 for Lex-RAG over Hybrid fell to +5 (corrected p = .36) with all four reviewed.

The reader is noisy, about ±2

  • Same packs rerun: standard deviation about 2 answers; single cells moved up to six.
Fig. 7 Pipeline order across reruns, two judges and the human review
The figure loads here.

The order shifts with run, judge and review. Treat small gaps as ties.

Paper, code and data

  • Reviewer copy of the paper on request
ResourceStatusLicence
PaperWhat Reaches the ReaderIn preparation–
Codepipelines, experiments, test runner, evaluation and analysis scriptsGitHubArashNasrEsfahani/AnyRAGMIT
Benchmark156 questions with requirements, gold evidence and bundlesHugging FaceArashNasr/anyrag-benchmarkCC BY 4.0
Resultsper-question records, verdicts and token countsHugging FaceArashNasr/anyrag-resultsCC BY 4.0
Large artifactsjudge caches, raw logs, indexes and embeddingsOn request–

Documents are not redistributed; a manifest with file hashes lets you rebuild each pool.

Cite

@misc{nasresfahani2026anyrag,
  author = {Nasr Esfahani, Arash},
  title  = {What Reaches the Reader: A Comparison of
            Retrieval-Augmented Generation Setups in Persian and English},
  year   = {2026},
  note   = {In preparation}
}