Professional Documents
Culture Documents
Chapter 4 - Syntax Analysis
Chapter 4 - Syntax Analysis
Role of parser
Context free grammars
Top down parsing
Bottom up parsing
Parser generators
The role of parser
token Rest of
Source Lexical Parse tree Intermediate
program Analyzer Parser Front representation
getNext End
Token
Symbol
table
Uses of grammars
E -> E + T | T
T -> T * F | F
F -> (E) | id
E -> TE’
E’ -> +TE’ | Ɛ
T -> FT’
T’ -> *FT’ | Ɛ
F -> (E) | id
Error handling
Common programming errors
Lexical errors
Syntactic errors
Semantic errors
Logical errors
Error handler goals
Report the presence of errors clearly and accurately
Recover from each error quickly enough to detect
subsequent errors
Add minimal overhead to the processing of correct programs
Error-recover strategies
Panic mode recovery
Discard input symbol one at a time until one of
designated set of synchronization tokens is found
Phrase level recovery
Replacing a prefix of remaining input by some string
that allows the parser to continue
Error productions
Augment the grammar with productions that generate
the erroneous constructs
Global correction
Choosing minimal sequence of changes to obtain a
globally least-cost correction
Context free grammars
Terminals
Nonterminals
expression -> expression + term
Start symbol expression -> expression – term
productions expression -> term
term -> term * factor
term -> term / factor
term -> factor
factor -> (expression)
factor -> id
Derivations
Productions are treated as rewriting rules to generate
a string
Rightmost and leftmost derivations
E -> E + E | E * E | -E | (E) | id
Derivations for –(id+id)
E => -E => -(E) => -(E+E) => -(id+E)=>-(id+id)
Parse trees
-(id+id)
E => -E => -(E) => -(E+E) => -(id+E)=>-(id+id)
Ambiguity
For some strings there exist more than one parse tree
Or more than one leftmost derivation
Or more than one rightmost derivation
Example: id+id*id
Elimination of ambiguity
Elimination of ambiguity (cont.)
Idea:
A statement appearing between a then and an else
must be matched
Elimination of left recursion
A grammar is left recursive if it has a non-terminal A
+
such that there is a derivation A=> Aα
Top down parsing methods cant handle left-
recursive grammars
A simple rule for direct left recursion elimination:
For a rule like:
A -> A α|β
We may replace it with
A -> β A’
A’ -> α A’ | ɛ
Left recursion elimination (cont.)
There are cases like following
S -> Aa | b
A -> Ac | Sd | ɛ
Left recursion elimination algorithm:
Arrange the nonterminals in some order A1,A2,…,An.
For (each i from 1 to n) {
For (each j from 1 to i-1) {
Replace each production of the form Ai-> Aj γ by the production
Ai -> δ1 γ | δ2 γ | … |δk γ where Aj-> δ1 | δ2 | … |δk
are all current Aj productions
}
Eliminate left recursion among the Ai-productions
}
Left factoring
Left factoring is a grammar transformation that is useful
for producing a grammar suitable for predictive or top-
down parsing.
Consider following grammar:
Stmt -> if expr then stmt else stmt
| if expr then stmt
On seeing input if it is not clear for the parser which
production to use
We can easily perform left factoring:
If we have A->αβ1 | αβ2 then we replace it with
A -> αA’
A’ -> β1 | β2
Left factoring (cont.)
Algorithm
For each non-terminal A, find the longest prefix α
common to two or more of its alternatives. If α<> ɛ,
then replace all of A-productions A->αβ1 |αβ2 | …
| αβn | γ by
A -> αA’ | γ
A’ -> β1 |β2 | … | βn
Example:
S -> iE t S | i E t S e S | a
E -> b
Introduction
A Top-down parser tries to create a parse tree from the
root towards the leafs scanning input from left to right
It can be also viewed as finding a leftmost derivation for
an input string
Example: id+id*id
E -> TE’ E E E E E E
lm lm lm lm lm
E’ -> +TE’ | Ɛ T E’ T E’ T E’ T E’ T E’
T -> FT’
F T’ F T’ + T E’
T’ -> *FT’ | Ɛ F T’ F T’
F -> (E) | id id id Ɛ id Ɛ
Recursive descent parsing
Consists of a set of procedures, one for each
nonterminal
Execution begins with the procedure for start symbol
A typical procedure for a non-terminal
void A() {
choose an A-production, A->X1X2..Xk
for (i=1 to k) {
if (Xi is a nonterminal
call procedure Xi();
else if (Xi equals the current input symbol a)
advance the input to the next symbol;
else /* an error has occurred */
}
}
Recursive descent parsing (cont)
General recursive descent may require backtracking
The previous code needs to be modified to allow
backtracking
In general form it cant choose an A-production easily.
So we need to try all alternatives
If one failed the input pointer needs to be reset and
another alternative should be tried
Recursive descent parsers cant be used for left-
recursive grammars
Example
S->cAd
A->ab | a Input: cad
S S S
c A d c A d c A d
a b a
First and Follow
First() is set of terminals that begins strings derived from
If α=>ɛ
* then ɛ is also in First(α)
In predictive parsing when we have A-> α|β, if First(α)
and First(β) are disjoint sets then we can select
appropriate A-production by looking at the next input
Follow(A), for any nonterminal A, is set of terminals a that
can appear immediately after A in some sentential form
If we have S *=> αAaβ for some αand βthen a is in
Follow(A)
If A can be the rightmost symbol in some sentential form,
then $ is in Follow(A)
Computing First
To compute First(X) for all grammar symbols X, apply
*
following rules until no more terminals or ɛ can be
added to any First set:
1. If X is a terminal then First(X) = {X}.
2. If X is a nonterminal and X->Y1Y2…Yk is a production for
some k>=1, then place a in First(X) if for some i, a is in
First(Yi) and ɛ is in all of First(Y1),…,First(Yi-1) that is
Y1…Yi-1 => *ɛ. If ɛ is in First(Yj) for j=1,…,k then add ɛ to
First(X).
3. If X -> ɛ is a production then add ɛ to First(X)
Example!
Computing Follow
To compute Follow(A) for all nonterminals A, apply
following rules until nothing can be added to any
follow set:
1. Place $ in Follow(S) where S is the start symbol
2. If there is a production A-> αBβ then everything in
First(β) except ɛ is in Follow(B).
3. If there is a production A-> αB or a production
A->αBβ where First(β) contains ɛ, then
everything in Follow(A) is in Follow(B)
1. Example!
LL(1) Grammars
Predictive parsers are those recursive descent parsers needing
no backtracking
Grammars for which we can create predictive parsers are called
LL(1)
The first L means scanning input from left to right
The second L means leftmost derivation
And 1 stands for using one input symbol for lookahead
A grammar G is LL(1) if and only if whenever A-> α|βare two
distinct productions of G, the following conditions hold:
For no terminal a do αandβ both derive strings beginning with a
At most one of α or βcan derive empty string
*
If α=> ɛ then βdoes not derive any string beginning with a
terminal in Follow(A).
Construction of predictive
parsing table
For each production A->α in grammar do the
following:
1. For each terminal a in First(α), add A->α in M[A,a].
2. If ɛ is in First(α), then for each terminal b in
Follow(A), add A-> ɛ to M[A,b]. If ɛ is in First(α)
and $ is in Follow(A), add A-> ɛ to M[A,$] as well
1. If after performing the above, there is no production
in M[A,a] then set M[A,a] to error
Example F
First
{(, id}
Follow
{+, *, ), $}
E -> TE’ {(, id} {+, ), $}
E’ -> +TE’ | Ɛ T
E {(, id} {), $}
T -> FT’ {+, ɛ}
T’ -> *FT’ | Ɛ E’ {), $}
T’ {*, ɛ} {+, ), $}
F -> (E) | id
Input Symbol
Non -
terminal id + * ( ) $
E E -> TE’ E -> TE’
Input Symbol
Non -
terminal a b e i t $
S S -> a S -> iEtSS’
S’ S’ -> Ɛ S’ -> Ɛ
S’ -> eS
E E -> b
Non-recursive predicting parsing
a + b $
Predictive
parsing output
stack X
Y program
Z
$
Parsing
Table
M
Predictive parsing algorithm
Set ip point to the first symbol of w;
Set X to the top stack symbol;
While (X<>$) { /* stack is not empty */
if (X is a) pop the stack and advance ip;
else if (X is a terminal) error();
else if (M[X,a] is an error entry) error();
else if (M[X,a] = X->Y1Y2..Yk) {
output the production X->Y1Y2..Yk;
pop the stack;
push Yk,…,Y2,Y1 on to the stack with Y1 on top;
}
set X to the top stack symbol;
}
Example
id+id*id$
Example
EE -> TE’ E -> TE’ synch synch
id id F id T*F
id F id
id
Shift-reduce parser
The general idea is to shift some symbols of input to the
stack until a reduction can be applied
At each reduction step, a specific substring matching the
body of a production is replaced by the nonterminal at the
head of the production
The key decisions during bottom-up parsing are about when
to reduce and about what production to apply
A reduction is a reverse of a step in a derivation
The goal of a bottom-up parser is to construct a derivation
in reverse:
E=>T=>T*F=>T*id=>F*id=>id*id
Handle pruning
A Handle is a substring that matches the body of a
production and whose reduction represents one step
along the reverse of a rightmost derivation
Stack Input
… if expr then stmt else …$
Reduce/reduce conflict
stmt -> id(parameter_list)
stmt -> expr:=expr
parameter_list->parameter_list, parameter
parameter_list->parameter
parameter->id
expr->id(expr_list)
expr->id
expr_list->expr_list, expr
Stack Input
expr_list->expr
… id(id ,id) …$
LR Parsing
The most prevalent type of bottom-up parsers
LR(k), mostly interested on parsers with k<=1
Why LR parsers?
Table driven
Can be constructed to recognize all programming language
constructs
Most general non-backtracking shift-reduce parsing method
Can detect a syntactic error as soon as it is possible to do so
Class of grammars for which we can construct LR parsers are
superset of those which we can construct LL parsers
States of an LR parser
States represent set of items
An LR(0) item of G is a production of G with the dot at
some position of the body:
For A->XYZ we have following items
A->.XYZ
A->X.YZ
A->XY.Z
A->XYZ.
If A->α.Bβ is in closure(I) and B->γ is a production then add the item B->.γ to
clsoure(I).
Example:
I0=closure({[E’->.E]}
E’->E E’->.E
E -> E + T | T E->.E+T
T -> T * F | F E->.T
T->.T*F
F -> (E) | id T->.F
F->.(E)
F->.id
Constructing canonical LR(0)
item sets (cont.)
Goto (I,X) where I is an item set and X is a grammar
symbol is closure of set of all items [A-> αX. β] where
[A-> α.X β] is in I
I1
Example E’->E.
E E->E.+T
I0=closure({[E’->.E]}
E’->.E I2
E->.E+T T
E’->T.
E->.T T->T.*F
T->.T*F I4
T->.F ( F->(.E)
F->.(E) E->.E+T
E->.T
F->.id T->.T*F
T->.F
F->.(E)
F->.id
Closure algorithm
SetOfItems CLOSURE(I) {
J=I;
repeat
for (each item A-> α.Bβ in J)
for (each prodcution B->γ of G)
if (B->.γ is not in J)
add B->.γ to J;
until no more items are added to J on one round;
return J;
GOTO algorithm
SetOfItems GOTO(I,X) {
J=empty;
if (A-> α.X β is in I)
add CLOSURE(A-> αX. β ) to J;
return J;
}
Canonical LR(0) items
Void items(G’) {
C= CLOSURE({[S’->.S]});
repeat
for (each set of items I in C)
for (each grammar symbol X)
if (GOTO(I,X) is not empty and not in C)
add GOTO(I,X) to C;
until no new set of items are added to C on a round;
}
E’->E
E -> E + T | T
Example acc
$
T -> T * F | F
F -> (E) | id
I6 I9
E->E+.T
I1
+
T->.T*F T E->E+T.
E’->E. T->.F
T->T.*F
E E->E.+T
F->.(E)
F->.id
I0=closure({[E’->.E]} I2
E’->.E T I7
F I10
E->.E+T E’->T. * T->T*.F
F->.(E)
T->T.*F T->T*F.
E->.T
T->.T*F id id F->.id
T->.F I5
F->.(E)
F->.id ( F->id. +
I4
F->(.E)
I8 I11
E->.E+T
E->.T
E E->E.+T )
T->.T*F F->(E.) F->(E).
T->.F
F->.(E)
F->.id
I3
T>F.
Use of LR(0) automaton
Example: id*id
Line Stack Symbols Input Action
(1) 0 $ id*id$ Shift to 5
(2) 05 $id *id$ Reduce by F->id
(3) 03 $F *id$ Reduce by T->F
(4) 02 $T *id$ Shift to 7
(5) 027 $T* id$ Shift to 5
(6) 0275 $T*id $ Reduce by F->id
(7) 02710 $T*F $ Reduce by T->T*F
(8) 02 $T $ Reduce by E->T
(9) 01 $E $ accept
LR-Parsing model
INPUT a1 … ai … an $
LR Parsing Output
Sm
Program
Sm-1
…
$
ACTION GOTO
LR parsing algorithm
let a be the first symbol of w$;
while(1) { /*repeat forever */
let s be the state on top of the stack;
if (ACTION[s,a] = shift t) {
push t onto the stack;
let a be the next input symbol;
} else if (ACTION[s,a] = reduce A->β) {
pop |β| symbols of the stack;
let state t now be on top of the stack;
push GOTO[t,A] onto the stack;
output the production A->β;
} else if (ACTION[s,a]=accept) break; /* parsing is done */
else call error-recovery routine;
}
Example (0) E’->E
(1) E -> E + T
(2) E-> T
STATE ACTON GOTO
(3) T -> T * F
id + * ( ) $ E T F (4) T-> F
0 S5 S4 1 2 3 (5) F -> (E) id*id+id?
(6) F->id
1 S6 Acc
Line Stac Symbol Input Action
2 R2 S7 R2 R2
k s
3 R R7 R4 R4 (1) 0 id*id+id$ Shift to 5
4
(2) 05 id *id+id$ Reduce by F->id
4 S5 S4 8 2 3
(3) 03 F *id+id$ Reduce by T->F
5 R R R6 R6
(4) 02 T *id+id$ Shift to 7
6 6
(5) 027 T* id+id$ Shift to 5
6 S5 S4 9 3
(6) 0275 T*id +id$ Reduce by F->id
7 S5 S4 10
(7) 02710 T*F +id$ Reduce by T-
8 S6 S11 >T*F
Action
=
Shift 6
2 Reduce R->L
More powerful LR parsers
Canonical-LR or just LR method
Use lookahead symbols for items: LR(1) items
Results in a large collection of items
LALR: lookaheads are introduced in LR(0) items
Canonical LR(1) items
In LR(1) items each item is in the form: [A->α.β,a]
An LR(1) item [A->α.β,a] is valid for a viable prefix γ if
*
there is a derivation S=>δAw=>δαβw, where
rm
Γ= δα
Either a is the first symbol of w, or w is ε and a is $
Example:
*
S=>aaBab=>aaaBab
S->BB rm
SetOfItems Goto(I,X) {
initialize J to be the empty set;
for (each item [A->α.Xβ,a] in I)
add item [A->αX.β,a] to set J;
return closure(J);
}
void items(G’){
initialize C to Closure({[S’->.S,$]});
repeat
for (each set of items I in C)
for (each grammar symbol X)
if (Goto(I,X) is not empty and not in C)
add Goto(I,X) to C;
until no new sets of items are added to C;
}
Example
S’->S
S->CC
C->cC
C->d
Canonical LR(1) parsing table
Method
Construct C={I0,I1, … , In}, the collection of LR(1) items for G’
State i is constructed from state Ii:
If [A->α.aβ, b] is in Ii and Goto(Ii,a)=Ij, then set ACTION[i,a] to
“shift j”
If [A->α., a] is in Ii, then set ACTION[i,a] to “reduce A->α”
If {S’->.S,$] is in Ii, then set ACTION[I,$] to “Accept”
If any conflicts appears then we say that the grammar is not
LR(1).
If GOTO(Ii,A) = Ij then GOTO[i,A]=j
All entries not defined by above rules are made “error”
The initial state of the parser is the one constructed from the
set of items containing [S’->.S,$]
Example
S’->S
S->CC
C->cC
C->d
LALR Parsing Table
For the previous example we had:
I4
C->d. , c/d
I47
C->d. , c/d/$
I7
C->d. , $
E->E*E 0 S3 S2 1
1 S4 S5 Acc
E->(E) 2 S3 S2 6
E->id 3 R4 R4 R4 R4
4 S3 S2 7
5 S3 S2 8