I read:
Christopher D. Manning, Prabhakar Raghavan, Hinrich
Schutze, “Introduction to Information Retrieval”. Cambridge University
Press. 2008. Chapter 4-5.
From 4.2: Large collections where index too large for memory but index is on a single machine
use blocked sort-based indexing (BSBI)
Single-pass in-memory indexing (SPIMI) - better (from 4.3)
From 4.4: Distributed indexing
term-partitioned index
Map Reduce - (parser and inverter)
From 5.1: Must use lossless compression.
From 5.2: Methods for Dictionary Compression:
store the whole dictionary as a string to avoid wasting space
group terms into blocks and store pointer
From 5.3: Methods for Postings file compression
variable encoding to store gaps between results
variable byte codes
γ codes
Question
Is it better to update an index to handle changes to documents or recreate on a schedule?
No comments:
Post a Comment