Pyserini: Prebuilt Indexes

Pyserini provides a number of pre-built Lucene indexes. To list what's available in code:

from pyserini.search.lucene import LuceneSearcher
LuceneSearcher.list_prebuilt_indexes()

from pyserini.index.lucene import IndexReader
IndexReader.list_prebuilt_indexes()

It's easy initialize a searcher from a pre-built index:

searcher = LuceneSearcher.from_prebuilt_index('robust04')

You can use this simple Python one-liner to download the pre-built index:

python -c "from pyserini.search.lucene import LuceneSearcher; LuceneSearcher.from_prebuilt_index('robust04')"

The downloaded index will be in ~/.cache/pyserini/indexes/.

It's similarly easy initialize an index reader from a pre-built index:

index_reader = IndexReader.from_prebuilt_index('robust04')
index_reader.stats()

The output will be:

{'total_terms': 174540872, 'documents': 528030, 'non_empty_documents': 528030, 'unique_terms': 923436}

Note that unless the underlying index was built with the -optimize option (i.e., merging all index segments into a single segment), unique_terms will show -1. Nope, that's not a bug.

Below is a summary of the pre-built indexes that are currently available. Detailed configuration information for the pre-built indexes are stored in pyserini/prebuilt_index_info.py.

Standard Lucene Indexes

: Lucene index of the MS MARCO V1 document corpus. (Lucene 9)
: Lucene index of the MS MARCO V1 document corpus ('slim' version). (Lucene 9)
: Lucene index of the MS MARCO V1 document corpus ('full' version). (Lucene 9)
: Lucene index of the MS MARCO V1 document corpus with doc2query-T5 expansions. (Lucene 9)
: Lucene index (+docvectors) of the MS MARCO V1 document corpus with doc2query-T5 expansions. (Lucene 9)
: Lucene index of the MS MARCO V1 segmented document corpus. (Lucene 9)
: Lucene index of the MS MARCO V1 segmented document corpus ('slim' version). (Lucene 9)
: Lucene index of the MS MARCO V1 segmented document corpus ('full' version). (Lucene 9)
: Lucene index of the MS MARCO V1 segmented document corpus with doc2query-T5 expansions. (Lucene 9)
: Lucene index (+docvectors) of the MS MARCO V1 segmented document corpus with doc2query-T5 expansions. (Lucene 9)
: Lucene index of the MS MARCO V1 passage corpus. (Lucene 9)
: Lucene index of the MS MARCO V1 passage corpus ('slim' version). (Lucene 9)
: Lucene index of the MS MARCO V1 passage corpus ('full' version). (Lucene 9)
: Lucene index of the MS MARCO V1 passage corpus with doc2query-T5 expansions. (Lucene 9)
: Lucene index (+docvectors) of the MS MARCO V1 passage corpus with doc2query-T5 expansions. (Lucene 9)
: Lucene index of the MS MARCO passage corpus with four extra preprocessed fields for LTR. (Lucene 8)
: Lucene index of the MS MARCO document per-passage corpus with four extra preprocessed fields for LTR. (Lucene 8)
: Lucene index of the MS MARCO document segmented corpus with four extra preprocessed fields for LTR. (Lucene 8)
: Lucene index of the MS MARCO V2 document corpus. (Lucene 9)
: Lucene index of the MS MARCO V2 document corpus ('slim' version). (Lucene 9)
: Lucene index of the MS MARCO V2 document corpus ('full' version). (Lucene 9)
: Lucene index of the MS MARCO V2 document corpus with doc2query-T5 expansions. (Lucene 9)
: Lucene index (+docvectors) of the MS MARCO V2 document corpus with doc2query-T5 expansions. (Lucene 9)
: Lucene index of the MS MARCO V2 segmented document corpus. (Lucene 9)
: Lucene index of the MS MARCO V2 segmented document corpus ('slim' version). (Lucene 9)
: Lucene index of the MS MARCO V2 segmented document corpus ('full' version). (Lucene 9)
: Lucene index of the MS MARCO V2 segmented document corpus with doc2query-T5 expansions. (Lucene 9)
: Lucene index (+docvectors) of the MS MARCO V2 segmented document corpus with doc2query-T5 expansions. (Lucene 9)
: Lucene index of the MS MARCO V2 passage corpus. (Lucene 9)
: Lucene index of the MS MARCO V2 passage corpus ('slim' version). (Lucene 9)
: Lucene index of the MS MARCO V2 passage corpus ('full' version). (Lucene 9)
: Lucene index of the MS MARCO V2 passage corpus with doc2query-T5 expansions. (Lucene 9)
: Lucene index (+docvectors) of the MS MARCO V2 passage corpus with doc2query-T5 expansions. (Lucene 9)
: Lucene index of the MS MARCO V2 augmented passage corpus. (Lucene 9)
: Lucene index of the MS MARCO V2 augmented passage corpus ('slim' version). (Lucene 9)
: Lucene index of the MS MARCO V2 augmented passage corpus ('full' version). (Lucene 9)
: Lucene index of the MS MARCO V2 augmented passage corpus with doc2query-T5 expansions. (Lucene 9)
: Lucene index (+docvectors) of the MS MARCO V2 augmented passage corpus with doc2query-T5 expansions. (Lucene 9)
: Lucene flat index of BEIR (v1.0.0): TREC-COVID
: Lucene flat index of BEIR (v1.0.0): BioASQ
: Lucene flat index of BEIR (v1.0.0): NFCorpus
: Lucene flat index of BEIR (v1.0.0): NQ
: Lucene flat index of BEIR (v1.0.0): HotpotQA
: Lucene flat index of BEIR (v1.0.0): FiQA-2018
: Lucene flat index of BEIR (v1.0.0): Signal-1M
: Lucene flat index of BEIR (v1.0.0): TREC-NEWS
: Lucene flat index of BEIR (v1.0.0): Robust04
: Lucene flat index of BEIR (v1.0.0): ArguAna
: Lucene flat index of BEIR (v1.0.0): Webis-Touche2020
: Lucene flat index of BEIR (v1.0.0): CQADupStack-android
: Lucene flat index of BEIR (v1.0.0): CQADupStack-english
: Lucene flat index of BEIR (v1.0.0): CQADupStack-gaming
: Lucene flat index of BEIR (v1.0.0): CQADupStack-gis
: Lucene flat index of BEIR (v1.0.0): CQADupStack-mathematica
: Lucene flat index of BEIR (v1.0.0): CQADupStack-physics
: Lucene flat index of BEIR (v1.0.0): CQADupStack-programmers
: Lucene flat index of BEIR (v1.0.0): CQADupStack-stats
: Lucene flat index of BEIR (v1.0.0): CQADupStack-tex
: Lucene flat index of BEIR (v1.0.0): CQADupStack-unix
: Lucene flat index of BEIR (v1.0.0): CQADupStack-webmasters
: Lucene flat index of BEIR (v1.0.0): CQADupStack-wordpress
: Lucene flat index of BEIR (v1.0.0): Quora
: Lucene flat index of BEIR (v1.0.0): DBPedia
: Lucene flat index of BEIR (v1.0.0): SCIDOCS
: Lucene flat index of BEIR (v1.0.0): FEVER
: Lucene flat index of BEIR (v1.0.0): Climate-FEVER
: Lucene flat index of BEIR (v1.0.0): SciFact
: Lucene multifield index of BEIR (v1.0.0): TREC-COVID
: Lucene multifield index of BEIR (v1.0.0): BioASQ
: Lucene multifield index of BEIR (v1.0.0): NFCorpus
: Lucene multifield index of BEIR (v1.0.0): NQ
: Lucene multifield index of BEIR (v1.0.0): HotpotQA
: Lucene multifield index of BEIR (v1.0.0): FiQA-2018
: Lucene multifield index of BEIR (v1.0.0): Signal-1M
: Lucene multifield index of BEIR (v1.0.0): TREC-NEWS
: Lucene multifield index of BEIR (v1.0.0): Robust04
: Lucene multifield index of BEIR (v1.0.0): ArguAna
: Lucene multifield index of BEIR (v1.0.0): Webis-Touche2020
: Lucene multifield index of BEIR (v1.0.0): CQADupStack-android
: Lucene multifield index of BEIR (v1.0.0): CQADupStack-english
: Lucene multifield index of BEIR (v1.0.0): CQADupStack-gaming
: Lucene multifield index of BEIR (v1.0.0): CQADupStack-gis
: Lucene multifield index of BEIR (v1.0.0): CQADupStack-mathematica
: Lucene multifield index of BEIR (v1.0.0): CQADupStack-physics
: Lucene multifield index of BEIR (v1.0.0): CQADupStack-programmers
: Lucene multifield index of BEIR (v1.0.0): CQADupStack-stats
: Lucene multifield index of BEIR (v1.0.0): CQADupStack-tex
: Lucene multifield index of BEIR (v1.0.0): CQADupStack-unix
: Lucene multifield index of BEIR (v1.0.0): CQADupStack-webmasters
: Lucene multifield index of BEIR (v1.0.0): CQADupStack-wordpress
: Lucene multifield index of BEIR (v1.0.0): Quora
: Lucene multifield index of BEIR (v1.0.0): DBPedia
: Lucene multifield index of BEIR (v1.0.0): SCIDOCS
: Lucene multifield index of BEIR (v1.0.0): FEVER
: Lucene multifield index of BEIR (v1.0.0): Climate-FEVER
: Lucene multifield index of BEIR (v1.0.0): SciFact
: Lucene index for Mr.TyDi v1.1 (Arabic).
: Lucene index for Mr.TyDi v1.1 (Bengali).
: Lucene index for Mr.TyDi v1.1 (English).
: Lucene index for Mr.TyDi v1.1 (Finnish).
: Lucene index for Mr.TyDi v1.1 (Indonesian).
: Lucene index for Mr.TyDi v1.1 (Japanese).
: Lucene index for Mr.TyDi v1.1 (Korean).
: Lucene index for Mr.TyDi v1.1 (Russian).
: Lucene index for Mr.TyDi v1.1 (Swahili).
: Lucene index for Mr.TyDi v1.1 (Telugu).
: Lucene index for Mr.TyDi v1.1 (Thai).
: Lucene index for MIRACL v1.0 (Arabic).
: Lucene index for MIRACL v1.0 (Bengali).
: Lucene index for MIRACL v1.0 (English).
: Lucene index for MIRACL v1.0 (Spanish).
: Lucene index for MIRACL v1.0 (Persian).
: Lucene index for MIRACL v1.0 (Finnish).
: Lucene index for MIRACL v1.0 (French).
: Lucene index for MIRACL v1.0 (Hindi).
: Lucene index for MIRACL v1.0 (Indonesian).
: Lucene index for MIRACL v1.0 (Japanese).
: Lucene index for MIRACL v1.0 (Korean).
: Lucene index for MIRACL v1.0 (Russian).
: Lucene index for MIRACL v1.0 (Swahili).
: Lucene index for MIRACL v1.0 (Telugu).
: Lucene index for MIRACL v1.0 (Thai).
: Lucene index for MIRACL v1.0 (Chinese).
: Lucene index for MIRACL v1.0 (German).
: Lucene index for MIRACL v1.0 (Yoruba).
: Lucene index for CIRAL v1.0 (Hausa).
: Lucene index for CIRAL v1.0 (Somali).
: Lucene index for CIRAL v1.0 (Swahili).
: Lucene index for CIRAL v1.0 (Yoruba).
: Lucene index of the CACM corpus. (Lucene 9)
: Lucene index of TREC Disks 4 & 5 (minus Congressional Records), used in the TREC 2004 Robust Track. (Lucene 9)
: Lucene index of English Wikipedia for BERTserini
: Lucene index of Chinese Wikipedia for BERTserini
: Lucene index for TREC-COVID Round 5: abstract index
: Lucene index for TREC-COVID Round 5: full-text index
: Lucene index for TREC-COVID Round 5: paragraph index
: Lucene index for TREC-COVID Round 4: abstract index
: Lucene index for TREC-COVID Round 4: full-text index
: Lucene index for TREC-COVID Round 4: paragraph index
: Lucene index for TREC-COVID Round 3: abstract index
: Lucene index for TREC-COVID Round 3: full-text index
: Lucene index for TREC-COVID Round 3: paragraph index
: Lucene index for TREC-COVID Round 2: abstract index
: Lucene index for TREC-COVID Round 2: full-text index
: Lucene index for TREC-COVID Round 2: paragraph index
: Lucene index for TREC-COVID Round 1: abstract index
: Lucene index for TREC-COVID Round 1: full-text index
: Lucene index for TREC-COVID Round 1: paragraph index
: Lucene index for TREC 2019 CaST
: Lucene index of Wikipedia with DPR 100-word splits
: Lucene index of Wikipedia with DPR 100-word splits (slim version, document text not stored)
: Lucene index of Wikipedia snapshot used as KILT's knowledge source.
: Lucene index of wiki-all-6-3-tamber from castorini/odqa-wiki-corpora
: Lucene index for HC4 v1.0 (Persian). (Lucene 9)
: Lucene index for HC4 v1.0 (Russian). (Lucene 9)
: Lucene index for HC4 v1.0 (Chinese). (Lucene 9)
: Lucene index for NeuCLIR 2022 corpus (Persian). (Lucene 9)
: Lucene index for NeuCLIR 2022 corpus (Russian). (Lucene 9)
: Lucene index for NeuCLIR 2022 corpus (Chinese). (Lucene 9)
: Lucene index for NeuCLIR 2022 corpus (official English translation from Persian). (Lucene 9)
: Lucene index for NeuCLIR 2022 corpus (official English translation from Russian). (Lucene 9)
: Lucene index for NeuCLIR 2022 corpus (official English translation from Chinese). (Lucene 9)
: Lucene index for AToMiC Text v0.2.1 small setting on validation set (Lucene 9)
: Lucene index for AToMiC Text v0.2.1 base setting on validation set (Lucene 9)
: Lucene index for AToMiC Text v0.2.1 large setting on validation set (Lucene 9)
: Lucene index for AToMiC Images v0.2 small setting on validation set (Lucene 9)
: Lucene index for AToMiC Images v0.2 base setting on validation set (Lucene 9)
: Lucene index for AToMiC Images v0.2 large setting on validation set (Lucene 9)

Lucene Impact Indexes

: Lucene impact index of the MS MARCO V1 passage corpus enoded by SLIM trained with BM25 negatives. (Lucene 9)
: Lucene impact index of the MS MARCO V1 passage corpus enoded by SLIM trained with cross-encoder distillation and hardnegative mining. (Lucene 9)
: Lucene impact index of the MS MARCO V1 passage corpus for uniCOIL. (Lucene 9)
: Lucene impact index of the MS MARCO V1 passage corpus for uniCOIL (noexp). (Lucene 9)
: Lucene impact index of the MS MARCO passage corpus encoded by DeepImpact. (Lucene 9)
: Lucene impact index of the MS MARCO passage corpus encoded by uniCOIL-TILDE. (Lucene 9)
: Lucene impact index of the MS MARCO passage corpus encoded by distill-splade-max. (Lucene 9)
: Lucene impact index of the MS MARCO passage corpus encoded by SPLADE++ CoCondenser-EnsembleDistil. (Lucene 9)
: Lucene impact index (with docvectors) of the MS MARCO passage corpus encoded by SPLADE++ CoCondenser-EnsembleDistil. (Lucene 9)
: Lucene impact index (with text) of the MS MARCO passage corpus encoded by SPLADE++ CoCondenser-EnsembleDistil. (Lucene 9)
: Lucene impact index of the MS MARCO passage corpus encoded by SPLADE++ CoCondenser-SelfDistil. (Lucene 9)
: Lucene impact index (with docvectors) of the MS MARCO passage corpus encoded by SPLADE++ CoCondenser-SelfDistil. (Lucene 9)
: Lucene impact index (with text) of the MS MARCO passage corpus encoded by SPLADE++ CoCondenser-SelfDistil. (Lucene 9)
: Lucene impact index of the MS MARCO V1 segmented document corpus for uniCOIL, with title/segment encoding. (Lucene 9)
: Lucene impact index of the MS MARCO V1 segmented document corpus for uniCOIL (noexp), with title/segment encoding. (Lucene 9)
: Lucene impact index of the MS MARCO V2 passage corpus for uniCOIL. (Lucene 9)
: Lucene impact index of the MS MARCO V2 passage corpus for uniCOIL (noexp). (Lucene 9)
: Lucene impact index of the MS MARCO V2 segmented document corpus for uniCOIL, with title prepended. (Lucene 9)
: Lucene impact index of the MS MARCO V2 segmented document corpus for uniCOIL (noexp) with title prepended. (Lucene 9)
: Lucene impact index of BEIR (v1.0.0): TREC-COVID encoded by SPLADE-distill CoCodenser-medium
: Lucene impact index of BEIR (v1.0.0): BioASQ encoded by SPLADE-distill CoCodenser-medium
: Lucene impact index of BEIR (v1.0.0): NFCorpus encoded by SPLADE-distill CoCodenser-medium
: Lucene impact index of BEIR (v1.0.0): NQ encoded by SPLADE-distill CoCodenser-medium
: Lucene impact index of BEIR (v1.0.0): HotpotQA encoded by SPLADE-distill CoCodenser-medium
: Lucene impact index of BEIR (v1.0.0): FiQA-2018 encoded by SPLADE-distill CoCodenser-medium
: Lucene impact index of BEIR (v1.0.0): Signal-1M encoded by SPLADE-distill CoCodenser-medium
: Lucene impact index of BEIR (v1.0.0): TREC-NEWS encoded by SPLADE-distill CoCodenser-medium
: Lucene impact index of BEIR (v1.0.0): Robust04 encoded by SPLADE-distill CoCodenser-medium
: Lucene impact index of BEIR (v1.0.0): ArguAna encoded by SPLADE-distill CoCodenser-medium
: Lucene impact index of BEIR (v1.0.0): Webis-Touche2020 encoded by SPLADE-distill CoCodenser-medium
: Lucene impact index of BEIR (v1.0.0): CQADupStack-android encoded by SPLADE-distill CoCodenser-medium
: Lucene impact index of BEIR (v1.0.0): CQADupStack-english encoded by SPLADE-distill CoCodenser-medium
: Lucene impact index of BEIR (v1.0.0): CQADupStack-gaming encoded by SPLADE-distill CoCodenser-medium
: Lucene impact index of BEIR (v1.0.0): CQADupStack-gis encoded by SPLADE-distill CoCodenser-medium
: Lucene impact index of BEIR (v1.0.0): CQADupStack-mathematica encoded by SPLADE-distill CoCodenser-medium
: Lucene impact index of BEIR (v1.0.0): CQADupStack-physics encoded by SPLADE-distill CoCodenser-medium
: Lucene impact index of BEIR (v1.0.0): CQADupStack-programmers encoded by SPLADE-distill CoCodenser-medium
: Lucene impact index of BEIR (v1.0.0): CQADupStack-stats encoded by SPLADE-distill CoCodenser-medium
: Lucene impact index of BEIR (v1.0.0): CQADupStack-tex encoded by SPLADE-distill CoCodenser-medium
: Lucene impact index of BEIR (v1.0.0): CQADupStack-unix encoded by SPLADE-distill CoCodenser-medium
: Lucene impact index of BEIR (v1.0.0): CQADupStack-webmasters encoded by SPLADE-distill CoCodenser-medium
: Lucene impact index of BEIR (v1.0.0): CQADupStack-wordpress encoded by SPLADE-distill CoCodenser-medium
: Lucene impact index of BEIR (v1.0.0): Quora encoded by SPLADE-distill CoCodenser-medium
: Lucene impact index of BEIR (v1.0.0): DBPedia encoded by SPLADE-distill CoCodenser-medium
: Lucene impact index of BEIR (v1.0.0): SCIDOCS encoded by SPLADE-distill CoCodenser-medium
: Lucene impact index of BEIR (v1.0.0): FEVER encoded by SPLADE-distill CoCodenser-medium
: Lucene impact index of BEIR (v1.0.0): Climate-FEVER encoded by SPLADE-distill CoCodenser-medium
: Lucene impact index of BEIR (v1.0.0): SciFact encoded by SPLADE-distill CoCodenser-medium

Faiss Indexes

: Faiss FlatIP index of the MS MARCO passage corpus encoded by aggretriever-cocondenser encoder.
: Faiss FlatIP index of the MS MARCO passage corpus encoded by aggretriever-distilbert encoder.
: Faiss FlatIP index of the MS MARCO passage corpus encoded by the ANCE MS MARCO passage encoder
: Faiss FlatIP index of the MS MARCO passage corpus encoded by the distilbert-dot-margin_mse-T2-msmarco encoder
: Faiss FlatIP index of the MS MARCO passage corpus encoded by distilbert-dot-tas_b-b256-msmarco encoder
: Faiss FlatIP index of the MS MARCO passage corpus encoded by the SBERT MS MARCO passage encoder
: Faiss FlatIP index of the MS MARCO passage corpus encoded by TCT-ColBERT
: Faiss HNSW index of the MS MARCO passage corpus encoded by TCT-ColBERT
: Faiss FlatIP index of the MS MARCO passage corpus encoded by the tct_colbert-v2 passage encoder
: Faiss FlatIP index of the MS MARCO passage corpus encoded by the tct_colbert-v2-hn passage encoder
: Faiss FlatIP index of the MS MARCO passage corpus encoded by the tct_colbert-v2-hnp passage encoder
: Faiss FlatIP index of the MS MARCO document corpus encoded by TCT-ColBERT-V2-HNP
: Faiss FlatIP index of the MS MARCO document corpus encoded by the ANCE MaxP encoder
: Faiss FlatIP index of the MS MARCO document corpus encoded by TCT-ColBERT
: Faiss FlatIP index of the MS MARCO document corpus encoded by TCT-ColBERT-V2-HNP
: Faiss index for BEIR v1.0.0 (TREC-COVID) corpus encoded by Contriever encoder.
: Faiss index for BEIR v1.0.0 (BioASQ) corpus encoded by Contriever encoder.
: Faiss index for BEIR v1.0.0 (NFCorpus) corpus encoded by Contriever encoder.
: Faiss index for BEIR v1.0.0 (NQ) corpus encoded by Contriever encoder.
: Faiss index for BEIR v1.0.0 (HotpotQA) corpus encoded by Contriever encoder.
: Faiss index for BEIR v1.0.0 (FiQA-2018) corpus encoded by Contriever encoder.
: Faiss index for BEIR v1.0.0 (Signal-1M) corpus encoded by Contriever encoder.
: Faiss index for BEIR v1.0.0 (TREC-NEWS) corpus encoded by Contriever encoder.
: Faiss index for BEIR v1.0.0 (Robust04) corpus encoded by Contriever encoder.
: Faiss index for BEIR v1.0.0 (ArguAna) corpus encoded by Contriever encoder.
: Faiss index for BEIR v1.0.0 (Webis-Touche2020) corpus encoded by Contriever encoder.
: Faiss index for BEIR v1.0.0 (CQADupStack-android) corpus encoded by Contriever encoder.
: Faiss index for BEIR v1.0.0 (CQADupStack-english) corpus encoded by Contriever encoder.
: Faiss index for BEIR v1.0.0 (CQADupStack-gaming) corpus encoded by Contriever encoder.
: Faiss index for BEIR v1.0.0 (CQADupStack-gis) corpus encoded by Contriever encoder.
: Faiss index for BEIR v1.0.0 (CQADupStack-mathematica) corpus encoded by Contriever encoder.
: Faiss index for BEIR v1.0.0 (CQADupStack-physics) corpus encoded by Contriever encoder.
: Faiss index for BEIR v1.0.0 (CQADupStack-programmers) corpus encoded by Contriever encoder.
: Faiss index for BEIR v1.0.0 (CQADupStack-stats) corpus encoded by Contriever encoder.
: Faiss index for BEIR v1.0.0 (CQADupStack-tex) corpus encoded by Contriever encoder.
: Faiss index for BEIR v1.0.0 (CQADupStack-unix) corpus encoded by Contriever encoder.
: Faiss index for BEIR v1.0.0 (CQADupStack-webmasters) corpus encoded by Contriever encoder.
: Faiss index for BEIR v1.0.0 (CQADupStack-wordpress) corpus encoded by Contriever encoder.
: Faiss index for BEIR v1.0.0 (Quora) corpus encoded by Contriever encoder.
: Faiss index for BEIR v1.0.0 (DBPedia) corpus encoded by Contriever encoder.
: Faiss index for BEIR v1.0.0 (SCIDOCS) corpus encoded by Contriever encoder.
: Faiss index for BEIR v1.0.0 (FEVER) corpus encoded by Contriever encoder.
: Faiss index for BEIR v1.0.0 (Climate-FEVER) corpus encoded by Contriever encoder.
: Faiss index for BEIR v1.0.0 (SciFact) corpus encoded by Contriever encoder.
: Faiss index for BEIR v1.0.0 (TREC-COVID) corpus encoded by Contriever encoder that has been fine-tuned with MS MARCO passage.
: Faiss index for BEIR v1.0.0 (BioASQ) corpus encoded by Contriever encoder that has been fine-tuned with MS MARCO passage.
: Faiss index for BEIR v1.0.0 (NFCorpus) corpus encoded by Contriever encoder that has been fine-tuned with MS MARCO passage.
: Faiss index for BEIR v1.0.0 (NQ) corpus encoded by Contriever encoder that has been fine-tuned with MS MARCO passage.
: Faiss index for BEIR v1.0.0 (HotpotQA) corpus encoded by Contriever encoder that has been fine-tuned with MS MARCO passage.
: Faiss index for BEIR v1.0.0 (FiQA-2018) corpus encoded by Contriever encoder that has been fine-tuned with MS MARCO passage.
: Faiss index for BEIR v1.0.0 (Signal-1M) corpus encoded by Contriever encoder that has been fine-tuned with MS MARCO passage.
: Faiss index for BEIR v1.0.0 (TREC-NEWS) corpus encoded by Contriever encoder that has been fine-tuned with MS MARCO passage.
: Faiss index for BEIR v1.0.0 (Robust04) corpus encoded by Contriever encoder that has been fine-tuned with MS MARCO passage.
: Faiss index for BEIR v1.0.0 (ArguAna) corpus encoded by Contriever encoder that has been fine-tuned with MS MARCO passage.
: Faiss index for BEIR v1.0.0 (Webis-Touche2020) corpus encoded by Contriever encoder that has been fine-tuned with MS MARCO passage.
: Faiss index for BEIR v1.0.0 (CQADupStack-android) corpus encoded by Contriever encoder that has been fine-tuned with MS MARCO passage.
: Faiss index for BEIR v1.0.0 (CQADupStack-english) corpus encoded by Contriever encoder that has been fine-tuned with MS MARCO passage.
: Faiss index for BEIR v1.0.0 (CQADupStack-gaming) corpus encoded by Contriever encoder that has been fine-tuned with MS MARCO passage.
: Faiss index for BEIR v1.0.0 (CQADupStack-gis) corpus encoded by Contriever encoder that has been fine-tuned with MS MARCO passage.
: Faiss index for BEIR v1.0.0 (CQADupStack-mathematica) corpus encoded by Contriever encoder that has been fine-tuned with MS MARCO passage.
: Faiss index for BEIR v1.0.0 (CQADupStack-physics) corpus encoded by Contriever encoder that has been fine-tuned with MS MARCO passage.
: Faiss index for BEIR v1.0.0 (CQADupStack-programmers) corpus encoded by Contriever encoder that has been fine-tuned with MS MARCO passage.
: Faiss index for BEIR v1.0.0 (CQADupStack-stats) corpus encoded by Contriever encoder that has been fine-tuned with MS MARCO passage.
: Faiss index for BEIR v1.0.0 (CQADupStack-tex) corpus encoded by Contriever encoder that has been fine-tuned with MS MARCO passage.
: Faiss index for BEIR v1.0.0 (CQADupStack-unix) corpus encoded by Contriever encoder that has been fine-tuned with MS MARCO passage.
: Faiss index for BEIR v1.0.0 (CQADupStack-webmasters) corpus encoded by Contriever encoder that has been fine-tuned with MS MARCO passage.
: Faiss index for BEIR v1.0.0 (CQADupStack-wordpress) corpus encoded by Contriever encoder that has been fine-tuned with MS MARCO passage.
: Faiss index for BEIR v1.0.0 (Quora) corpus encoded by Contriever encoder that has been fine-tuned with MS MARCO passage.
: Faiss index for BEIR v1.0.0 (DBPedia) corpus encoded by Contriever encoder that has been fine-tuned with MS MARCO passage.
: Faiss index for BEIR v1.0.0 (SCIDOCS) corpus encoded by Contriever encoder that has been fine-tuned with MS MARCO passage.
: Faiss index for BEIR v1.0.0 (FEVER) corpus encoded by Contriever encoder that has been fine-tuned with MS MARCO passage.
: Faiss index for BEIR v1.0.0 (Climate-FEVER) corpus encoded by Contriever encoder that has been fine-tuned with MS MARCO passage.
: Faiss index for BEIR v1.0.0 (SciFact) corpus encoded by Contriever encoder that has been fine-tuned with MS MARCO passage.
: Faiss index for Mr.TyDi v1.1 (Arabic) corpus encoded by mDPR passage encoder pre-fine-tuned on NQ.
: Faiss index for Mr.TyDi v1.1 (Bengali) corpus encoded by mDPR passage encoder pre-fine-tuned on NQ.
: Faiss index for Mr.TyDi v1.1 (English) corpus encoded by mDPR passage encoder pre-fine-tuned on NQ.
: Faiss index for Mr.TyDi v1.1 (Finnish) corpus encoded by mDPR passage encoder pre-fine-tuned on NQ.
: Faiss index for Mr.TyDi v1.1 (Indonesian) corpus encoded by mDPR passage encoder pre-fine-tuned on NQ.
: Faiss index for Mr.TyDi v1.1 (Japanese) corpus encoded by mDPR passage encoder pre-fine-tuned on NQ.
: Faiss index for Mr.TyDi v1.1 (Korean) corpus encoded by mDPR passage encoder pre-fine-tuned on NQ.
: Faiss index for Mr.TyDi v1.1 (Russian) corpus encoded by mDPR passage encoder pre-fine-tuned on NQ.
: Faiss index for Mr.TyDi v1.1 (Swahili) corpus encoded by mDPR passage encoder pre-fine-tuned on NQ.
: Faiss index for Mr.TyDi v1.1 (Telugu) corpus encoded by mDPR passage encoder pre-fine-tuned on NQ.
: Faiss index for Mr.TyDi v1.1 (Thai) corpus encoded by mDPR passage encoder pre-fine-tuned on NQ.
: Faiss index for Mr.TyDi v1.1 (Arabic) corpus encoded by mDPR passage encoder pre-fine-tuned on MS MARCO.
: Faiss index for Mr.TyDi v1.1 (Bengali) corpus encoded by mDPR passage encoder pre-fine-tuned on MS MARCO.
: Faiss index for Mr.TyDi v1.1 (English) corpus encoded by mDPR passage encoder pre-fine-tuned on MS MARCO.
: Faiss index for Mr.TyDi v1.1 (Finnish) corpus encoded by mDPR passage encoder pre-fine-tuned on MS MARCO.
: Faiss index for Mr.TyDi v1.1 (Indonesian) corpus encoded by mDPR passage encoder pre-fine-tuned on MS MARCO.
: Faiss index for Mr.TyDi v1.1 (Japanese) corpus encoded by mDPR passage encoder pre-fine-tuned on MS MARCO.
: Faiss index for Mr.TyDi v1.1 (Korean) corpus encoded by mDPR passage encoder pre-fine-tuned on MS MARCO.
: Faiss index for Mr.TyDi v1.1 (Russian) corpus encoded by mDPR passage encoder pre-fine-tuned on MS MARCO.
: Faiss index for Mr.TyDi v1.1 (Swahili) corpus encoded by mDPR passage encoder pre-fine-tuned on MS MARCO.
: Faiss index for Mr.TyDi v1.1 (Telugu) corpus encoded by mDPR passage encoder pre-fine-tuned on MS MARCO.
: Faiss index for Mr.TyDi v1.1 (Thai) corpus encoded by mDPR passage encoder pre-fine-tuned on MS MARCO.
: Faiss index for Mr.TyDi v1.1 (Arabic) corpus encoded by mDPR passage encoder pre-fine-tuned on NQ.
: Faiss index for Mr.TyDi v1.1 (Bengali) corpus encoded by mDPR passage encoder pre-fine-tuned on NQ.
: Faiss index for Mr.TyDi v1.1 (English) corpus encoded by mDPR passage encoder pre-fine-tuned on NQ.
: Faiss index for Mr.TyDi v1.1 (Finnish) corpus encoded by mDPR passage encoder pre-fine-tuned on NQ.
: Faiss index for Mr.TyDi v1.1 (Indonesian) corpus encoded by mDPR passage encoder pre-fine-tuned on NQ.
: Faiss index for Mr.TyDi v1.1 (Japanese) corpus encoded by mDPR passage encoder pre-fine-tuned on NQ.
: Faiss index for Mr.TyDi v1.1 (Korean) corpus encoded by mDPR passage encoder pre-fine-tuned on NQ.
: Faiss index for Mr.TyDi v1.1 (Russian) corpus encoded by mDPR passage encoder pre-fine-tuned on NQ.
: Faiss index for Mr.TyDi v1.1 (Swahili) corpus encoded by mDPR passage encoder pre-fine-tuned on NQ.
: Faiss index for Mr.TyDi v1.1 (Telugu) corpus encoded by mDPR passage encoder pre-fine-tuned on NQ.
: Faiss index for Mr.TyDi v1.1 (Thai) corpus encoded by mDPR passage encoder pre-fine-tuned on NQ.
: Faiss index for Mr.TyDi v1.1 (Arabic) corpus encoded by mDPR passage encoder pre-fine-tuned on NQ.
: Faiss index for Mr.TyDi v1.1 (Bengali) corpus encoded by mDPR passage encoder pre-fine-tuned on NQ.
: Faiss index for Mr.TyDi v1.1 (English) corpus encoded by mDPR passage encoder pre-fine-tuned on NQ.
: Faiss index for Mr.TyDi v1.1 (Finnish) corpus encoded by mDPR passage encoder pre-fine-tuned on NQ.
: Faiss index for Mr.TyDi v1.1 (Indonesian) corpus encoded by mDPR passage encoder pre-fine-tuned on NQ.
: Faiss index for Mr.TyDi v1.1 (Japanese) corpus encoded by mDPR passage encoder pre-fine-tuned on NQ.
: Faiss index for Mr.TyDi v1.1 (Korean) corpus encoded by mDPR passage encoder pre-fine-tuned on NQ.
: Faiss index for Mr.TyDi v1.1 (Russian) corpus encoded by mDPR passage encoder pre-fine-tuned on NQ.
: Faiss index for Mr.TyDi v1.1 (Swahili) corpus encoded by mDPR passage encoder pre-fine-tuned on NQ.
: Faiss index for Mr.TyDi v1.1 (Telugu) corpus encoded by mDPR passage encoder pre-fine-tuned on NQ.
: Faiss index for Mr.TyDi v1.1 (Thai) corpus encoded by mDPR passage encoder pre-fine-tuned on NQ.
: Faiss index for MIRACL v1.0 (Arabic) corpus encoded by mDPR passage encoder pre-fine-tuned on MS MARCO.
: Faiss index for MIRACL v1.0 (Bengali) corpus encoded by mDPR passage encoder pre-fine-tuned on MS MARCO.
: Faiss index for MIRACL v1.0 (English) corpus encoded by mDPR passage encoder pre-fine-tuned on MS MARCO.
: Faiss index for MIRACL v1.0 (Spanish) corpus encoded by mDPR passage encoder pre-fine-tuned on MS MARCO.
: Faiss index for MIRACL v1.0 (Persian) corpus encoded by mDPR passage encoder pre-fine-tuned on MS MARCO.
: Faiss index for MIRACL v1.0 (Finnish) corpus encoded by mDPR passage encoder pre-fine-tuned on MS MARCO.
: Faiss index for MIRACL v1.0 (French) corpus encoded by mDPR passage encoder pre-fine-tuned on MS MARCO.
: Faiss index for MIRACL v1.0 (Hindi) corpus encoded by mDPR passage encoder pre-fine-tuned on MS MARCO.
: Faiss index for MIRACL v1.0 (Indonesian) corpus encoded by mDPR passage encoder pre-fine-tuned on MS MARCO.
: Faiss index for MIRACL v1.0 (Japanese) corpus encoded by mDPR passage encoder pre-fine-tuned on MS MARCO.
: Faiss index for MIRACL v1.0 (Korean) corpus encoded by mDPR passage encoder pre-fine-tuned on MS MARCO.
: Faiss index for MIRACL v1.0 (Russian) corpus encoded by mDPR passage encoder pre-fine-tuned on MS MARCO.
: Faiss index for MIRACL v1.0 (Swahili) corpus encoded by mDPR passage encoder pre-fine-tuned on MS MARCO.
: Faiss index for MIRACL v1.0 (Telugu) corpus encoded by mDPR passage encoder pre-fine-tuned on MS MARCO.
: Faiss index for MIRACL v1.0 (Thai) corpus encoded by mDPR passage encoder pre-fine-tuned on MS MARCO.
: Faiss index for MIRACL v1.0 (Chinese) corpus encoded by mDPR passage encoder pre-fine-tuned on MS MARCO.
: Faiss index for MIRACL v1.0 (German) corpus encoded by mDPR passage encoder pre-fine-tuned on MS MARCO.
: Faiss index for MIRACL v1.0 (Yoruba) corpus encoded by mDPR passage encoder pre-fine-tuned on MS MARCO.
: Faiss index for MIRACL v1.0 (Arabic) corpus encoded by mDPR passage encoder pre-fine-tuned on MS MARCO.
: Faiss index for MIRACL v1.0 (Bengali) corpus encoded by mDPR passage encoder pre-fine-tuned on MS MARCO.
: Faiss index for MIRACL v1.0 (English) corpus encoded by mDPR passage encoder pre-fine-tuned on MS MARCO.
: Faiss index for MIRACL v1.0 (Spanish) corpus encoded by mDPR passage encoder pre-fine-tuned on MS MARCO.
: Faiss index for MIRACL v1.0 (Persian) corpus encoded by mDPR passage encoder pre-fine-tuned on MS MARCO.
: Faiss index for MIRACL v1.0 (Finnish) corpus encoded by mDPR passage encoder pre-fine-tuned on MS MARCO.
: Faiss index for MIRACL v1.0 (French) corpus encoded by mDPR passage encoder pre-fine-tuned on MS MARCO.
: Faiss index for MIRACL v1.0 (Hindi) corpus encoded by mDPR passage encoder pre-fine-tuned on MS MARCO.
: Faiss index for MIRACL v1.0 (Indonesian) corpus encoded by mDPR passage encoder pre-fine-tuned on MS MARCO.
: Faiss index for MIRACL v1.0 (Japanese) corpus encoded by mDPR passage encoder pre-fine-tuned on MS MARCO.
: Faiss index for MIRACL v1.0 (Korean) corpus encoded by mDPR passage encoder pre-fine-tuned on MS MARCO.
: Faiss index for MIRACL v1.0 (Russian) corpus encoded by mDPR passage encoder pre-fine-tuned on MS MARCO.
: Faiss index for MIRACL v1.0 (Swahili) corpus encoded by mDPR passage encoder pre-fine-tuned on MS MARCO.
: Faiss index for MIRACL v1.0 (Telugu) corpus encoded by mDPR passage encoder pre-fine-tuned on MS MARCO.
: Faiss index for MIRACL v1.0 (Thai) corpus encoded by mDPR passage encoder pre-fine-tuned on MS MARCO.
: Faiss index for MIRACL v1.0 (Chinese) corpus encoded by mDPR passage encoder pre-fine-tuned on MS MARCO.
: Faiss index for MIRACL v1.0 (Chinese) corpus encoded by mDPR passage encoder pre-fine-tuned on MS MARCO.
: Faiss index for MIRACL v1.0 (Chinese) corpus encoded by mDPR passage encoder pre-fine-tuned on MS MARCO.
: Faiss index for MIRACL v1.0 (Arabic) corpus encoded by mDPR passage encoder pre-fine-tuned on MS MARCO, then fine-tuned in-language with MIRACL.
: Faiss index for MIRACL v1.0 (Bengali) corpus encoded by mDPR passage encoder pre-fine-tuned on MS MARCO, then fine-tuned in-language with MIRACL.
: Faiss index for MIRACL v1.0 (English) corpus encoded by mDPR passage encoder pre-fine-tuned on MS MARCO, then fine-tuned in-language with MIRACL.
: Faiss index for MIRACL v1.0 (Spanish) corpus encoded by mDPR passage encoder pre-fine-tuned on MS MARCO, then fine-tuned in-language with MIRACL.
: Faiss index for MIRACL v1.0 (Persian) corpus encoded by mDPR passage encoder pre-fine-tuned on MS MARCO, then fine-tuned in-language with MIRACL.
: Faiss index for MIRACL v1.0 (Finnish) corpus encoded by mDPR passage encoder pre-fine-tuned on MS MARCO, then fine-tuned in-language with MIRACL.
: Faiss index for MIRACL v1.0 (French) corpus encoded by mDPR passage encoder pre-fine-tuned on MS MARCO, then fine-tuned in-language with MIRACL.
: Faiss index for MIRACL v1.0 (Hindi) corpus encoded by mDPR passage encoder pre-fine-tuned on MS MARCO, then fine-tuned in-language with MIRACL.
: Faiss index for MIRACL v1.0 (Indonesian) corpus encoded by mDPR passage encoder pre-fine-tuned on MS MARCO, then fine-tuned in-language with MIRACL.
: Faiss index for MIRACL v1.0 (Japanese) corpus encoded by mDPR passage encoder pre-fine-tuned on MS MARCO, then fine-tuned in-language with MIRACL.
: Faiss index for MIRACL v1.0 (Korean) corpus encoded by mDPR passage encoder pre-fine-tuned on MS MARCO, then fine-tuned in-language with MIRACL.
: Faiss index for MIRACL v1.0 (Russian) corpus encoded by mDPR passage encoder pre-fine-tuned on MS MARCO, then fine-tuned in-language with MIRACL.
: Faiss index for MIRACL v1.0 (Swahili) corpus encoded by mDPR passage encoder pre-fine-tuned on MS MARCO, then fine-tuned in-language with MIRACL.
: Faiss index for MIRACL v1.0 (Telugu) corpus encoded by mDPR passage encoder pre-fine-tuned on MS MARCO, then fine-tuned in-language with MIRACL.
: Faiss index for MIRACL v1.0 (Thai) corpus encoded by mDPR passage encoder pre-fine-tuned on MS MARCO, then fine-tuned in-language with MIRACL.
: Faiss index for MIRACL v1.0 (Chinese) corpus encoded by mDPR passage encoder pre-fine-tuned on MS MARCO, then fine-tuned in-language with MIRACL.
: Faiss index for MIRACL v1.0 (Arabic) corpus encoded by mContriever passage encoder pre-fine-tuned on MS MARCO.
: Faiss index for MIRACL v1.0 (Bengali) corpus encoded by mContriever passage encoder pre-fine-tuned on MS MARCO.
: Faiss index for MIRACL v1.0 (English) corpus encoded by mContriever passage encoder pre-fine-tuned on MS MARCO.
: Faiss index for MIRACL v1.0 (Spanish) corpus encoded by mContriever passage encoder pre-fine-tuned on MS MARCO.
: Faiss index for MIRACL v1.0 (Persian) corpus encoded by mContriever passage encoder pre-fine-tuned on MS MARCO.
: Faiss index for MIRACL v1.0 (Finnish) corpus encoded by mContriever passage encoder pre-fine-tuned on MS MARCO.
: Faiss index for MIRACL v1.0 (French) corpus encoded by mContriever passage encoder pre-fine-tuned on MS MARCO.
: Faiss index for MIRACL v1.0 (Hindi) corpus encoded by mContriever passage encoder pre-fine-tuned on MS MARCO.
: Faiss index for MIRACL v1.0 (Indonesian) corpus encoded by mContriever passage encoder pre-fine-tuned on MS MARCO.
: Faiss index for MIRACL v1.0 (Japanese) corpus encoded by mContriever passage encoder pre-fine-tuned on MS MARCO.
: Faiss index for MIRACL v1.0 (Korean) corpus encoded by mContriever passage encoder pre-fine-tuned on MS MARCO.
: Faiss index for MIRACL v1.0 (Russian) corpus encoded by mContriever passage encoder pre-fine-tuned on MS MARCO.
: Faiss index for MIRACL v1.0 (Swahili) corpus encoded by mContriever passage encoder pre-fine-tuned on MS MARCO.
: Faiss index for MIRACL v1.0 (Telugu) corpus encoded by mContriever passage encoder pre-fine-tuned on MS MARCO.
: Faiss index for MIRACL v1.0 (Thai) corpus encoded by mContriever passage encoder pre-fine-tuned on MS MARCO.
: Faiss index for MIRACL v1.0 (Chinese) corpus encoded by mContriever passage encoder pre-fine-tuned on MS MARCO.
: Faiss index for MIRACL v1.0 (German) corpus encoded by mContriever passage encoder pre-fine-tuned on MS MARCO.
: Faiss index for MIRACL v1.0 (Yoruba) corpus encoded by mContriever passage encoder pre-fine-tuned on MS MARCO.
: Faiss FlatIP index of Wikipedia encoded by the DPR doc encoder trained on multiple QA datasets
: Faiss FlatIP index of Wikipedia encoded by the DPR doc encoder trained on NQ
: Faiss binary index of Wikipedia encoded by the BPR doc encoder trained on NQ
: Faiss FlatIP index of Wikipedia encoded by the ANCE-multi encoder
: Faiss FlatIP index of Wikipedia DPR encoded by the retriever model from 'Distilling Knowledge from Reader to Retriever for Question Answering' trained on NQ
: Faiss FlatIP index of Wikipedia DPR encoded by the retriever model from 'Distilling Knowledge from Reader to Retriever for Question Answering' trained on TriviaQA
: Faiss FlatIP index of wiki-all-6-3-tamber encoded by a 2nd iteration DPR model trained on multiple QA datasets
: Faiss HNSW index of the CAsT2019 passage corpus encoded by the tct_colbert-v2 passage encoder

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

prebuilt-indexes.md

prebuilt-indexes.md

Pyserini: Prebuilt Indexes

Standard Lucene Indexes

Lucene Impact Indexes

Faiss Indexes

Files

prebuilt-indexes.md

Latest commit

History

prebuilt-indexes.md

File metadata and controls

Pyserini: Prebuilt Indexes

Standard Lucene Indexes

Lucene Impact Indexes

Faiss Indexes