Association Rules & Sequential Patterns: Road Map

 
Association Rules & Sequential

Patterns
Web Text Mining
Road map
 Basic concepts of Association Rules

 Apriori algorithm
 Different data formats for mining
 Mining with multiple minimum supports
 Mining class association rules
 Sequential pattern mining
 Summary
Web Text Mining: Association Mining 2
  1
 
Association rule mining

 Proposed by Agrawal et al in 1993.
 It is an important data mining model studied
extensively by the database and data mining
community.
 Assume all data are categorical.
 No good algorithm for numeric data.
 Initially used for Market Basket Analysis to find
how items purchased by customers are related.
Bread → Milk [sup = 5%, conf = 100%]
The model: data
 I = {i1, i2, …, im}: a set of items.

 Transaction t :
 t a set of items, and t ⊆ I.
 Transaction Database T: a set of transactions

T = {t1, t2, …, tn}.
  2
 
Transaction data: supermarket data

 Market basket transactions:
t1: {bread, cheese, milk}
t2: {apple, eggs, salt, yogurt}
… …
tn: {biscuit, eggs, milk}
 Concepts:
 An item: an item/article in a basket
 I: the set of all items sold in the store
 A transaction: items purchased in a basket; it may
have TID (transaction ID)
 A transactional dataset: A set of transactions
Transaction data: a set of documents

 A text document data set. Each document
is treated as a “bag” of keywords
doc1: Student, Teach, School
doc2: Student, School
doc3: Teach, School, City, Game
doc4: Baseball, Basketball
doc5: Basketball, Player, Spectator
doc6: Baseball, Coach, Game, Team
doc7: Basketball, Team, City, Game
  3
 
The model: rules

 A transaction t contains X, a set of items
(itemset) in I, if X ⊆ t.
 An association rule is an implication of the
form:
X → Y, where X, Y ⊂ I, and X ∩Y = ∅
 An itemset is a set of items.

 E.g., X = {milk, bread, cereal} is an itemset.
 A k-itemset is an itemset with k items.
 E.g., {milk, bread, cereal} is a 3-itemset
Rule strength measures

 Support: The rule holds with support sup in T
(the transaction data set) if sup% of
transactions contain X ∪ Y.
 sup = Pr(X ∪ Y).
 Confidence: The rule holds in T with
confidence conf if conf% of tranactions that
contain X also contain Y.
 conf = Pr(Y | X)
 An association rule is a pattern that states
when X occurs, Y occurs with certain
probability.
  4
 
Support and Confidence

 Support count: The support count of an
itemset X, denoted by X.count, in a data set
T is the number of transactions in T that
contain X. Assume T has n transactions.
 Then,
( X ! Y ).count
support =
n
( X ! Y ).count
confidence =
X .count
Goal and key features
 Goal: Find all rules that satisfy the user-

specified minimum support (minsup) and
minimum confidence (minconf).
 Key Features
 Completeness: find all rules.
 No target item(s) on the right-hand-side
 Mining with data on hard disk (not in memory)
  5
 
t1: Beef, Chicken, Milk

An example t2: Beef, Cheese
t3: Cheese, Boots
t4: Beef, Chicken, Cheese
t5: Beef, Chicken, Clothes, Cheese, Milk
 Transaction data t6: Chicken, Clothes, Milk
 Assume: t7: Chicken, Milk, Clothes
minsup = 30%
minconf = 80%
 An example frequent itemset:
{Chicken, Clothes, Milk} [sup = 3/7]
 Association rules from the itemset:
Clothes → Milk, Chicken [sup = 3/7, conf = 3/3]
… …
Clothes, Chicken → Milk, [sup = 3/7, conf = 3/3]
Transaction data representation
 A simplistic view of shopping baskets,

 Some important information not considered.
E.g,
 the quantity of each item purchased and
 the price paid.
  6
 
Many mining algorithms

 There are a large number of them!!
 They use different strategies and data structures.
 Their resulting sets of rules are all the same.
 Given a transaction data set T, and a minimum support and
a minimum confident, the set of association rules existing in
T is uniquely determined.
 Any algorithm should find the same set of rules
although their computational efficiencies and
memory requirements may be different.
 We study only one: the Apriori Algorithm
Road map

 Summary
  7
 
The Apriori algorithm

 The best known algorithm
 Two steps:
 Find all itemsets that have minimum support
(frequent itemsets, also called large itemsets).
 Use frequent itemsets to generate rules.
 E.g., a frequent itemset

{Chicken, Clothes, Milk} [sup = 3/7]
and one rule from the frequent itemset
Clothes → Milk, Chicken [sup = 3/7, conf = 3/3]
Step 1: Mining all frequent itemsets

 A frequent itemset is an itemset whose support
is ≥ minsup.
 Key idea: The apriori property (downward
closure property): any subsets of a frequent
itemset are also frequent itemsets
ABC ABD ACD BCD
AB AC AD BC BD CD
A B C D
  8
 
The Algorithm
 Iterative algo. (also called level-wise search):
Find all 1-item frequent itemsets; then all 2-item
frequent itemsets, and so on.
 In each iteration k, only consider itemsets that
contain some k-1 frequent itemset.
 Find frequent itemsets of size 1: F1
 From k = 2
 Ck = candidates of size k: those itemsets of size k
that could be frequent, given Fk-1
 Fk = those itemsets that are actually frequent, Fk
⊆ Ck (need to scan the database once).
Dataset T TID Items

Example – minsup=0.5 T100 1, 3, 4
Finding frequent itemsets T200 2, 3, 5
T300 1, 2, 3, 5
T400 2, 5
itemset:count
1. scan T  C1: {1}:2, {2}:3, {3}:3, {4}:1, {5}:3
 F1 : {1}:2, {2}:3, {3}:3, {5}:3
 C 2: {1,2}, {1,3}, {1,5}, {2,3}, {2,5}, {3,5}
2. scan T  C2: {1,2}:1, {1,3}:2, {1,5}:1, {2,3}:2, {2,5}:3, {3,5}:2
 F2 : {1,3}:2, {2,3}:2, {2,5}:3, {3,5}:2
 C3: {2, 3,5}
3. scan T  C3: {2, 3, 5}:2  F3: {2, 3, 5}
  9
 
Details: ordering of items
 The items in I are sorted in lexicographic

order (which is a total order).
 The order is used throughout the algorithm in
each itemset.
 {w[1], w[2], …, w[k]} represents a k-itemset w
consisting of items w[1], w[2], …, w[k], where
w[1] < w[2] < … < w[k] according to the total
order.
Details: the algorithm

Algorithm Apriori(T)
C1 ← init-pass(T);
F1 ← {f | f ∈ C1, f.count/n ≥ minsup}; // n: no. of transactions in T
for (k = 2; Fk-1 ≠ ∅; k++) do
Ck ← candidate-gen(Fk-1);
for each transaction t ∈ T do
for each candidate c ∈ Ck do
if c is contained in t then
c.count++;
end
end
Fk ← {c ∈ Ck | c.count/n ≥ minsup}
end
return F ← U k Fk;
  10
 
Apriori candidate generation

 The candidate-gen function takes Fk-1 and
returns a superset (called the candidates)
of the set of all frequent k-itemsets. It has
two steps
 join step: Generate all possible candidate
itemsets Ck of length k
 prune step: Remove those candidates in Ck
that cannot be frequent.
Candidate-gen function
Function candidate-gen(Fk-1)
Ck ← ∅;
forall f1, f2 ∈ Fk-1
with f1 = {i1, … , ik-2, ik-1}
and f2 = {i1, … , ik-2, i’k-1}
and ik-1 < i’k-1 do
c ← {i1, …, ik-1, i’k-1}; // join f1 and f2
Ck ← Ck ∪ {c};
for each (k-1)-subset s of c do
if (s ∉ Fk-1) then
delete c from Ck; // prune
end
end
return Ck;
  11
 
An example
 F3 = {{1, 2, 3}, {1, 2, 4}, {1, 3, 4},
{1, 3, 5}, {2, 3, 4}}
 After join
 C4 = {{1, 2, 3, 4}, {1, 3, 4, 5}}
 After pruning:
 C4 = {{1, 2, 3, 4}}
because {1, 4, 5} is not in F3 ({1, 3, 4, 5} is removed)
Step 2: Generating rules from frequent

itemsets
 Frequent itemsets ≠ association rules
 One more step is needed to generate
association rules
 For each frequent itemset X,
For each proper nonempty subset A of X,
 Let B = X - A
 A → B is an association rule if
 Confidence(A → B) ≥ minconf,
support(A → B) = support(A∪B) = support(X)

confidence(A → B) = support(A ∪ B) / support(A)
  12
 
Generating rules: an example

 Suppose {2,3,4} is frequent, with sup=50%
 Proper nonempty subsets: {2,3}, {2,4}, {3,4}, {2}, {3}, {4}, with
sup=50%, 50%, 75%, 75%, 75%, 75% respectively
 These generate these association rules:
 2,3 → 4, confidence=100%
 2,4 → 3, confidence=100%
 3,4 → 2, confidence=67%
 2 → 3,4, confidence=67%
 All rules have support = 50%
Generating rules: summary

 To recap, in order to obtain A → B, we need
to have support(A ∪ B) and support(A)
 All the required information for confidence
computation has already been recorded in
itemset generation. No need to see the data
T any more.
 This step is not as time-consuming as
frequent itemsets generation.
  13
 
On Apriori Algorithm
Seems to be very expensive

 Level-wise search
 K = the size of the largest itemset
 It makes at most K passes over data
 In practice, K is bounded (10).
 The algorithm is very fast. Under some conditions,

all rules can be found in linear time.
 Scale up to large data sets
More on association rule mining

 Clearly the space of all association rules is
exponential, O(2m), where m is the number of
items in I.
 The mining exploits sparseness of data, and
high minimum support and high minimum
confidence values.
 Still, it always produces a huge number of
rules, thousands, tens of thousands, millions,
...
  14
 
Road map

 Summary
Different data formats for mining

 The data can be in transaction form or table
form
Transaction form: a, b
a, c, d, e
a, d, f
Table form: Attr1 Attr2 Attr3
a, b, d
b, c, e
 Table data need to be converted to
transaction form for association mining
  15
 
From a table to a set of transactions

Table form: Attr1 Attr2 Attr3
a, b, d
b, c, e
⇒ Transaction form:
(Attr1, a), (Attr2, b), (Attr3, d)
(Attr1, b), (Attr2, c), (Attr3, e)
Road map

 Summary
  16
 
Problems with the association mining

 Single minsup: It assumes that all items in
the data are of the same nature and/or
have similar frequencies.
 Not true: In many applications, some items
appear very frequently in the data, while
others rarely appear.
E.g., in a supermarket, people buy food processor
and cooking pan much less frequently than they
buy bread and milk.
Rare Item Problem

 If the frequencies of items vary a great deal,
we will encounter two problems
 If minsup is set too high, those rules that involve
rare items will not be found.
 To find rules that involve both frequent and rare
items, minsup has to be set very low. This may
cause combinatorial explosion because those
frequent items will be associated with one another
in all possible ways.
  17
 
Multiple minsups model

 The minimum support of a rule is expressed in
terms of minimum item supports (MIS) of the items
that appear in the rule.
 Each item can have a minimum item support.
 By providing different MIS values for different
items, the user effectively expresses different
support requirements for different rules.
 To prevent very frequent items and very rare items
from appearing in the same itemsets, we introduce
a support difference constraint.
maxi∈s{sup{i} − mini∈s{sup(i)} ≤ ϕ,
Minsup of a rule
 Let MIS(i) be the MIS value of item i. The

minsup of a rule R is the lowest MIS value of
the items in the rule.
 I.e., a rule R: a1, a2, …, ak → ak+1, …, ar
satisfies its minimum support if its actual
support is ≥
min(MIS(a1), MIS(a2), …, MIS(ar)).
  18
 
An Example
 Consider the following items:

bread, shoes, clothes
The user-specified MIS values are as follows:
MIS(bread) = 2% MIS(shoes) = 0.1%
MIS(clothes) = 0.2%
The following rule doesn’t satisfy its minsup:
clothes → bread [sup=0.15%,conf =70%]
The following rule satisfies its minsup:
clothes → shoes [sup=0.15%,conf =70%]
Downward closure property

 In the new model, the property no longer
holds
E.g., Consider four items 1, 2, 3 and 4 in a
database. Their minimum item supports are
MIS(1) = 10% MIS(2) = 20%
MIS(3) = 5% MIS(4) = 6%
{1, 2} with support 9% is infrequent, but {1, 2, 3}

and {1, 2, 4} could be frequent.
  19
 
On multiple minsup rule mining

 Multiple minsup model subsumes the single
support model.
 It is a more realistic model for practical
applications.
 The model enables us to found rare item rules
yet without producing a huge number of
meaningless rules with frequent items.
 By setting MIS values of some items to 100% (or
more), we effectively instruct the algorithms not
to generate rules only involving these items.
Road map

 Summary
  20
 
Mining class association rules (CAR)

 Normal association rule mining does not
have any target.
 It finds all possible rules that exist in data,
i.e., any item can appear as a consequent or
a condition of a rule.
 However, in some applications, the user is
interested in some targets.
 E.g, the user has a set of text documents from
some known topics. He/she wants to find out what
words are associated or correlated with each
topic.
Problem definition
 Let T be a transaction data set consisting of n
transactions.
 Each transaction is also labeled with a class y.
 Let I be the set of all items in T, Y be the set of all
class labels and I ∩ Y = ∅.
 A class association rule (CAR) is an implication of
the form
X → y, where X ⊆ I, and y ∈ Y.
 The definitions of support and confidence are the
same as those for normal association rules.
  21
 
An example
 A text document data set
doc 1: Student, Teach, School : Education
doc 2: Student, School : Education
doc 3: Teach, School, City, Game : Education
doc 4: Baseball, Basketball : Sport
doc 5: Basketball, Player, Spectator : Sport
doc 6: Baseball, Coach, Game, Team : Sport
doc 7: Basketball, Team, City, Game : Sport
 Let minsup = 20% and minconf = 60%. The following are two
examples of class association rules:
Student, School → Education [sup= 2/7, conf = 2/2]
game → Sport [sup= 2/7, conf = 2/3]
Mining algorithm
 Unlike normal association rules, CARs can be mined
directly in one step.
 The key operation is to find all ruleitems that have
support above minsup. A ruleitem is of the form:
(condset, y)
where condset is a set of items from I (i.e., condset
⊆ I), and y ∈ Y is a class label.
 Each ruleitem basically represents a rule:
condset → y,
 The Apriori algorithm can be modified to generate
CARs
  22
 
Multiple minimum class supports

 The multiple minimum support idea can also be
applied here.
 The user can specify different minimum supports to
different classes, which effectively assign a different
minimum support to rules of each class.
 For example, we have a data set with two classes,
Yes and No. We may want
 rules of class Yes to have the minimum support of 5% and
 rules of class No to have the minimum support of 10%.
 By setting minimum class supports to 100% (or
more for some classes), we tell the algorithm not to
generate rules of those classes.
 This is a very useful trick in applications.
Road map

 Summary
  23
 
Summary
 Association rule mining has been extensively studied
in the data mining community.
 So is sequential pattern mining
 There are many efficient algorithms and model
variations.
 Other related work includes
 Multi-level or generalized rule mining
 Constrained rule mining
 Incremental rule mining
 Maximal frequent itemset mining
 Closed itemset mining
 Rule interestingness and visualization
 Parallel algorithms
 …
  24

Association Rules & Sequential Patterns: Road Map

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Association Rules & Sequential Patterns: Road Map

Uploaded by

Copyright:

Available Formats

 

Association Rules & Sequential

 Basic concepts of Association Rules

Web Text Mining: Association Mining 2

Association rule mining

Bread → Milk [sup = 5%, conf = 100%]

Web Text Mining: Association Mining 3

The model: data

 I = {i1, i2, …, im}: a set of items.

 Transaction Database T: a set of transactions

Web Text Mining: Association Mining 4

Transaction data: supermarket data

Transaction data: a set of documents

Web Text Mining: Association Mining 6

The model: rules

 An itemset is a set of items.

Web Text Mining: Association Mining 7

Rule strength measures

Support and Confidence

Goal and key features

 Goal: Find all rules that satisfy the user-

Web Text Mining: Association Mining 10

t1: Beef, Chicken, Milk

Web Text Mining: Association Mining 11

Transaction data representation

 A simplistic view of shopping baskets,

Web Text Mining: Association Mining 12

Many mining algorithms

Web Text Mining: Association Mining 13

 Basic concepts of Association Rules

Web Text Mining: Association Mining 14

The Apriori algorithm

 E.g., a frequent itemset

Web Text Mining: Association Mining 15

Step 1: Mining all frequent itemsets

Web Text Mining: Association Mining 16

Web Text Mining: Association Mining 17

Dataset T TID Items

Web Text Mining: Association Mining 18

Details: ordering of items

 The items in I are sorted in lexicographic

Web Text Mining: Association Mining 19

Details: the algorithm

Web Text Mining: Association Mining 20

Apriori candidate generation

Web Text Mining: Association Mining 21

Web Text Mining: Association Mining 22

Web Text Mining: Association Mining 23

Step 2: Generating rules from frequent

support(A → B) = support(A∪B) = support(X)

Generating rules: an example

Web Text Mining: Association Mining 25

Generating rules: summary

Web Text Mining: Association Mining 26

Seems to be very expensive

 K = the size of the largest itemset

 It makes at most K passes over data

 In practice, K is bounded (10).

 The algorithm is very fast. Under some conditions,

Web Text Mining: Association Mining 27

More on association rule mining

Web Text Mining: Association Mining 28