I read:
Christopher D. Manning, Prabhakar Raghavan, Hinrich Schutze, “Introduction to Information Retrieval”. Cambridge University Press. 2008. Section 1.2, Chapter 2, and Chapter 3.
The example process for creating an inverted index in Section 1.2 is a basic example but I think that it will prove to be a naive implementation.
In chapter 2, there are several mentions of doing tasks by hand; such as in machine learning tokenizing, filtering stop words, and synonym lists for token normalization. It is surprising to me that these systems require such careful hand-tuning but I suppose it makes sense given the crazy rules present in language.
From 3.1, Search trees > Hashing for Dictionaries.
From 3.2, Special Indexes for General Wildcard queries:
- permuterm index: rotations of each term (simple but space inefficient)
- k-Gram index: all k character long chunks of term (more complex and requires post-filtering of results)
From 3.3.3 - 3.3.4, isolated-term: k-Gram indexes used for spelling correction use the number of shared k-grams in the two terms.
From 3.3.5, Context-sensitive: use query frequency to correct.
Question
I'm curious how these techniques relate to real-world search engines. What of this is Google using?
No comments:
Post a Comment