Thursday, January 26, 2012

Week 3 - Index Construction and Compression - Reading Note

I read:
Christopher D. Manning, Prabhakar Raghavan, Hinrich Schutze, “Introduction to Information Retrieval”. Cambridge University Press. 2008. Chapter 4-5.

From 4.2: Large collections where index too large for memory but index is on a single machine
    use blocked sort-based indexing (BSBI)
    Single-pass in-memory indexing (SPIMI) - better (from 4.3)

From 4.4: Distributed indexing
    term-partitioned index
    Map Reduce - (parser and inverter)

From 5.1: Must use lossless compression.

From 5.2: Methods for Dictionary Compression:
    store the whole dictionary as a string to avoid wasting space
    group terms into blocks and store pointer

From 5.3: Methods for Postings file compression
    variable encoding to store gaps between results
        variable byte codes
        γ codes

Question
Is it better to update an index to handle changes to documents or recreate on a schedule?

No comments:

Post a Comment