Search Engine - Search Index - (Postings|Inverted) (Index|File) - Natural Language Processing

Text Mining


An inverted index is an index data structure storing a mapping from:

  • token (content), such as words or numbers,
  • to its locations (in a database file, document or a set of documents)
  • In text search, a forward index maps documents in a data set to the tokens they contain. This is also called the natural relationship in which documents list token (terms).
  • An inverted index supports the inverse mapping where it can list, for a term, the documents that contain it.

Given a set of documents, an inverted index is a dictionary where each word is associated with a list of the document identifiers in which that word appears.


Aysnchronous Update

The full-text index is updated asynchronously (crawled) rather than being maintained transactionally.


map reduce without reduce function as we will get already an (iterator|list) structure of the tweets.


tweet1, ("I love pancakes for breakfast") \\ 
tweet2, ("I dislike pancakes") \\ 
tweet3, ("What should we eat for breakfast?") 
tweet4, ("I love to eat") 

Desired output:

"pancakes", (tweet1, tweet2)
"breakfast", (tweet1, tweet3)
"eat", (tweet3, tweet4)
"love" (tweet1, tweet4)


Reduce running time of comparison

An inverted index is a data structure that allow to avoid making quadratically the running time of token comparisons. It maps each token in the dataset to the list of documents that contain the token. So, instead of comparing, record by record, each token to every other token to see if they match, the inverted indices is used to look up records that match on a particular token.



To not include a Web Page in a search index, the page need to have the noindex directive in a header or meta tag


  • No index for all bot
<meta name="robots" content="noindex">
<meta name="googlebot" content="noindex">


To include a Web Page in a search index, the page need to have the index directive in a header or meta tag


  • index for all bot
<meta name="robots" content="index">
<meta name="googlebot" content="index">

Google Index

Search Engine - Google Index

Documentation / Reference

Discover More
Data System Architecture
Log - Server

A log server is an application that is aimed to: receive log via a collector analyse them report on them and send alert if needed They are mostly search engine where: the words are stored...

Lucene Lucene is a text search engine library. The following application are Lucene application (ie build on it): * Solr * Elastic Search * New Relic Logs * ... The text data model of...
Map Reduce One Picture
Map Reduce (MR) Framework

Map reduce is a distributed execution . The MapReduce programming model (and a corresponding system) was proposed in a 2004 paper from a team at Google as a simpler abstraction for processing very large...
Text Mining
NLP - Forward index

In text search, a forward index is an index that maps documents in a data set to the tokens they contain. This is also called the natural relationship. inverted index
Text Mining
Natural Language Processing - Index

in NLP The index are created during the tokenization.
Search Engine - Bot

A search engine bot is a bot that crawl the web to build a index queried by the search engine Bing (Ip Range) ...
Search Engine - Search Index

A search index is an index of token (word) to web page A search engine query it in order to return result. It's structure is inverted index meaning that it maps word to URL (page) The search index is...
Robots Useragent
Web - Robots (Wanderers | Crawlers | Spiders)

This page is in a web context. Web Robots (also known as Web Wanderers, Crawlers, or Spiders), are crawler program that scan the web generally in order to: create an search engine. See or seo...
Text Mining
What are models of text in NLP? (Natural Language, Text Modeling)

This page talks model creation for natural language text. ie how to store and represent text ? Let's say that you want to search in a list of documents, documents that are similar on 2 dimensions,...

Share this page:
Follow us:
Task Runner