Information Retrieval

Chapter 5:
Retrieval Evaluation
 IR Evaluation

• It is known that measuring or evaluating the

performance and accuracy of the system is very
important after IR system is designed.

• According to (Singhal, 2001), there are two main things

to measure in IR system; these are: effectiveness of the
system and its efficiency
 Cont..
• Effectiveness:-Power to be effective; the quality of being able to
bring about an effect
• How is a system capable of retrieving relevant documents from
the collection?

about user satisfaction

• Efficiency:- The ratio of the output to the input of any system
• Skillfulness in avoiding wasted time and effort

It is about time, space

 …cont

 To measure ad hoc (informal) information retrieval

effectiveness in the standard way, we need a test collection
consisting of three things:
1. A document collection
2. A test suite (set)of information needs, expressible as
queries
3. A set of relevance judgments, standardly a binary
assessment of either relevant or non relevant for each
query-document pair
 Document collection

• Specific questions that might be considered when gathering

documents include:

1. How many items should be gathered?

2. What items should be sampled to create the document
collection?
3. What about copyright constraints?
Example (N=128)
 ….cont

The standard approach to information retrieval system

evaluation revolves around the notion of relevant and
non relevant documents.
With respect to a user information need, a document in
the test collection is given a binary classification as
either relevant or non relevant.
This decision is referred to as the gold standard or
ground truth judgment of relevance.
 Mind Break
A document is relevant if it addresses the stated
information need, not because it just happens to
contain all the words in the query.

How ?
 Types of Evaluation Strategies

•System-centered studies
– Given documents, queries, and relevance judgments
• Try several variations of the system
• Measure which system returns the “best” hit list

•User-centered studies
– Given several users, and at least two retrieval systems
• Have each user try the same task on both systems
• Measure which system works the “best” for users information need
 Performance measures (Recall, Precision, etc.)

• The two most frequent and basic measures for

information retrieval effectiveness are :
1. Precision and
2. Recall.
 Precision

Precision (P) is the fraction of retrieved documents that

are relevant
The ability to retrieve top-ranked documents
that are mostly relevant.
Precision is percentage of retrieved documents
that are relevant to the query (i.e. number of retrieved
documents that are relevant).
Precision Formula
 Recall
Recall (R) is the fraction of relevant documents that are
retrieved

– The ability of the search to find all of the

relevant items in the corpus

– Recall is percentage of relevant documents

retrieved from the database in response to users query.
Recall Formula
 Question

• When do you think the precision/recall has value

100% ? Or sometimes we can get the value of
precision and recall 100% or one. How can we
justify this value?
Example
 Examples
An IR system returns 8 relevant documents, and 10
non relevant documents. There are a total of 20
relevant documents in the collection.
a. What is the precision of the system on this search,
and
b. what is its recall?
c. What is F-measure?
 R- Precision

Precision at the R-th position in the ranking of results for a

query, where R is the total number of relevant documents.

It requires having a set of known relevant documents, from

which we calculate the precision of the top relevant
documents returned
– Calculate precision after R documents are seen
– Can be averaged over all queries
Example
Example 2:
 Exercise
• Given a query q, for which the relevant documents are d1,
d6, d10, d15, d22, d26, an IR system retrieves the following
ranking: d6, d2, d11, d3, d10, d1, d14, d15, d7, d23.
• compute the precision and recall for this ranking at each
retrieved document.
Cont..
 Cont..
• The average precision over positions 1, 5, 6, and 8
where relevant documents were found is
(1.0+0.40+0.50+0.50)/6=0.40. The R-precision is the
precision at position 6, which is 3/6=0.50.
total retrieved
 Problems with both precision and recall
 Number of irrelevant documents in the collection is not
taken into account.
 Recall is undefined when there is no relevant document
in the collection.
 Precision is undefined when no document is retrieved.
Other measures
 Noise = retrieved irrelevant docs / retrieved docs
 Silence/Miss = non-retrieved relevant docs / relevant
docs

Noise = 1 – Precision; Silence = 1 – Recall

 F-measure

• A single measure that trades off precision versus

recall is the F measure, which is the weighted
harmonic mean of precision and recall:
• One measure of performance that takes into accounts
both recall and precision. Harmonic mean of recall
and precision:
 Exercise
• The following list of Rs and Ns represents relevant (R) and
non relevant (N) returned documents in a ranked list of 20
documents retrieved in response to a query from a collection
of 10,000 documents. The top of the ranked list (the document
the system thinks is most likely to be relevant) is on the left of
the list. This list shows 6 relevant documents. Assume that
there are 8 relevant documents in total in the collection.
RRNNNNNNRNRNNNRNNNNR
 Questions
• Calculate the following:

a) What is the precision of the system on the top 20?

b) What is recall?
c) What is p@10?
d) What is the F-measure on the top 20?
e) Assume that these 20 documents are the complete result set
of the system. What is the MAP for the query?
f) Noise
g) Silence
 Difficulties in Evaluating IR System

 IR systems essentially facilitate communication between a

user and document collections
 Relevance is a measure of the effectiveness of
communication
– Effectiveness is related to the relevancy of retrieved
items.
– Relevance: relates to problem, information need,
query and a document or surrogate
 ……..cont

 Relevance judgments is made by

– The user who posed the retrieval problem

– An external judge

– Is the relevance judgment made by users and external

person the same?

 Relevance judgment is usually:

 ……….cont

– Subjective: Depends upon a specific user’s judgment.

– Situational: Relates to user’s current needs.
– Cognitive: Depends on human perception and
behavior.
– Dynamic: Changes over time.
 Information Retrieval

Chapter 6:
Query Languages and Operations
 Introduction
• Information is the main value of Information Society.

• Depending on the particular application scenario and on the

type of information that has to be managed and searched,
different techniques need to be devised.

• The dictionary definition of query is a set of instructions passed

to a database to retrieve particular data.
 Cont….

• A query is the formulation of a user information need.

• A query is composed of keywords and the documents

containing such keywords are searched for popular and
Intuitive, Easy to express, Allow fast ranking.
 Cont…
Query language (QL) refers to any computer programming language
that requests and retrieves data from database and information
systems by sending queries.

• Query Languages: A source language consisting of procedural

operators that invoke functions to be executed.
 Keyword-based queries

 Queries are combinations of words.

 The document collection is searched for documents that

contain these words.

 Word queries are intuitive, easy to express and provide fast

ranking.
 popular Keyword-based queries are
1. Single-word queries:
 A query is a single word
 Simplest form of query.
 All documents that include this word are retrieved.

 Documents may be ranked by the frequency of this word in the

document.
 Con’ted
2. phrase queries:
A query is a sequence of words treated as a single unit. Also
called “literal string” or “exact phrase” query, Phrase is usually
surrounded by quotation marks,

All documents that include this phrase are retrieved, Usually,

separators (commas, colons, etc.) and “trivial words” (e.g., “a”,
“the”, or “of”) in the phrase are ignored,
 Conted
In effect, this query is for a set of words that must appear in
sequence, Allows users to specify a context and thus gain
precision.

Example: “United States of America”.

 Con’ted
3. Multiple-word queries:

A query is a set of words (or phrases).

Two interpretations:
• A document is retrieved if it includes any of the query words.
• A document is retrieved if it includes each of the query
words.
 Cont..
Documents may be ranked by the number of query words they
contain: A document containing n query words is ranked higher
than a document containing n-1 query words.

Documents containing all the query words are ranked at the top.
Documents containing only one query word are ranked at
bottom.
Frequency counts may still be used to break ties among documents
that contain the same query words.
 Cont…
4. Proximity queries:
Restrict the distance within a document between two search
terms.
Important for large documents in which the two search words
may appear in different contexts.

Proximity specifications limit the acceptable occurrences and

hence increase the precision of the search.
 Cont….
General Format: Word1 within m units of Word2. Unit may be
character, word, paragraph, etc.
Example:
• nuclear within 0 paragraphs of cleanup

Finds documents that discuss “nuclear” and “cleanup” in the

same paragraph.

• united within 5 words of american

 Structural queries

 So far, we assumed documents that are entirely free of

structure.
 Structured documents would allow more powerful queries.

 Queries could combine text queries with structural queries:

queries that relate to the structure of the document.

 Mixing contents and structure in queries:

 Cont…
• Contents words, phrases, or patterns and

• Structural constraints containment, proximity, or other

restrictions on structural elements
 Example
• Example: Retrieve documents that contain a page in which the
phrase “terrorist attack” appears in the text and a photo whose
caption contains the phrase “World Trade Center”.

• The corresponding query could be: same page (“terrorist

attack”, photo (caption (“World Trade Center”))).
 Types

 Three main structures

 Fixed structure
Hypertext structure
Hierarchical structure
 Fixed structure
 Document is divided to a fixed set of fields, much like a filled
form.

 Fields may be associated with types, such as date.

 Each field has text and fields cannot nest or overlap.

 Queries (multiple-words, Boolean, proximity, patterns, etc.) are

targeted at particular fields.
Hypertext structure
 Hierarchical structure

 Intermediate model between fixed structure and hypertext.

 The “anarchic” hypertext network is restricted to a hierarchical
structure.
 The model allows recursive decomposition of documents.

 Queries may combine Regular text queries, which are targeted

at particular areas (the target area is defined by a “path
expression”) and Queries on the structure itself; for example
“retrieve documents with at least 5 sections
Cont…..
 Relevance feedback

 After initial retrieval results are presented, allow the user to

provide feedback on the relevance of one or more of the
retrieved documents.
 The system use this feedback information to reformulate the
query and Produce new results based on reformulated query.
After that allows more interactive, multi-pass process.
 RF
 The idea of relevance feedback (RF) is to involve the user in
RELEVANCE FEEDBACK the retrieval process so as to
improve the final result set.

 In particular, the user gives feedback on the relevance of

documents in an initial set of results.
 The basic procedure is:
 The user issues a (short, simple) query.
 The system returns an initial set of retrieval results.

 The user marks some returned documents as relevant or

non relevant.

 The system computes a better representation of the

information need based on the user feedback.

 The system displays a revised set of retrieval results.

Architecture
THE END OF:

Chapter 5 and 6