Thursday, March 29, 2012

Week 11 - Multilingual and Parallel Retrieval - Reading Note

This week I read:
  1. Oard, D.W. and Diekema, A.R.  (1998)  Cross-Language Information Retrieval. Annual Review of Information Science and Technology, 33. 223-256.
  2. Daqing He, Jianqiang Wang.  (2008)  Cross-Language Information Retrieval. In Ayse Goker, John Davies, Margaret Graham (eds) Information Retrieval, Wiley Publisher.
  3. IES chapter 14 parallel information retrieval http://www.ir.uwaterloo.ca/book/14-parallel-information-retrieval.pdf
I was unable to obtain a copy of 1 online.
I was unable to obtain a copy of 2 online.

From 3, query processing can be parallelized.

  • index partitioning - document vs. term partitioning - term partitioning allows for greater scalability by allowing the index to stay in memory - hybrid approach partitions by document and then by term
  • replication - helps provide reliability and fault tolerance in addition to scaling benefits

MapReduce can be used to parallelize other tasks such as index building, link analysis, etc. MapReduce is able to handle massive collections of processors and huge data sets. Maps and Reduces can be sharded into smaller sets.


No comments:

Post a Comment