ConnectionbetweenCFGsandpush-downautomata ResolvingConﬂictsinParserGeneration LR(k)Analysis GeneralPrinciplesofBottom-UpSyntaxAnalysis EducationalObjectives: Bottom-UpSyntaxAnaysis

(1)

Implementation of Parsers Bottom-Up Syntax Analysis

Bottom-Up Syntax Anaysis

Educational Objectives:

• General Principles of Bottom-Up Syntax Analysis

• LR(k) Analysis

• Resolving Conflicts in Parser Generation

• Connection between CFGs and push-down automata

(2)

Basic Ideas: Bottom-Up Syntax Analysis

• Bottom-Up Analysis is more powerful than top-down analysis, since production is chosen at the end of the analysis while in top-down analysis the production is selected up front.

• LR: Read input from left (L) and search for right derivations (R)

Ina Schaefer Context-Free Analysis 60

(3)

Principles of LR Parsing

1. Reduce from sentence to axiom

2. Construct sentential forms from prefixes in (N ∪ T)^∗ and input rests in T^∗. Prefixes are right sentential forms of grammar. Such prefixes are called viable prefixes. This prefix property has to hold invariantly to avoid dead ends.

3. Reductions are always made at the left-most possible position.

(4)

Viable Prefix

Definition

Let S ⇒^∗_rm βAu ⇒rm βαu a right sentential form of Γ.

Then α is called handle or redex of the right sentential for βαu.

Each prefix of βα is a viable prefix of Γ.

(5)

Regularity of Viable Prefixes

Theorem

The language of viable prefixes of a grammar Γ is regular.

Proof.

Cf. Wilhelm, Maurer Thm. 8.4.1 and Corrollary 8.4.2.1. (pp. 361, 362), Essential proof steps are illustrated in the following by construction of LR − DFA(Γ).

(6)

LR Parsing: Examples

• Consider Γ₁

! S → aCD

! C → b

! D → a|b

Analysis of aba can lead to an dead end. (cf. Lecture).

Considering viable prefixes can avoid this.

(7)

LR Parsing: Examples (2)

• Consider Γ₂

! S → E#

! E → a|(E)|EE

Analysis of ( ( a ) ) ( a) # (cf. Lecture) Stack can manage prefixes already read.

(8)

LR Parsing: Examples (3)

• Consider Γ₃

! S → E#

! E → E + T|T

! T → ID

Analysis of ID + ID + ID # (cf. Lecture)

(9)

LR Parsing: Shift and Reduce

Schematic syntax tree for input xay with a ∈ T, x, y ∈ T^∗ and start symbol S:

x a y

! a

Schematischer Syntaxbaum zur Eingabe xay mit a in T, x,y in T* und Startsymbol S:

x a y

! = "#

x a y

!

Lesezeiger

Schiebe Schritt (shift): Reduktionsschritt (reduce):

"$ =>

80

© A. Poetzsch-Heffter, TU Kaiserslautern 26.04.2007

x a y

! a

Lesezeiger

Schematischer Syntaxbaum zur Eingabe xay mit a in T, x,y in T* und Startsymbol S:

x a y

! = "#

Lesezeiger x a y

!

Lesezeiger

Schiebe Schritt (shift): Reduktionsschritt (reduce):

"$ =>

80

x a y

! a

Lesezeiger

Schematischer Syntaxbaum zur Eingabe xay mit a in T, x,y in T* und Startsymbol S:

x a y

! = "#

Lesezeiger x a y

!

Lesezeiger

Schiebe Schritt (shift): Reduktionsschritt (reduce):

"$ =>

Read Pointer

(10)

LR Parsing: Shift and Reduce (2)

Shift step:

80

x a y

!a

Lesezeiger

Schematischer Syntaxbaum zur Eingabe xay mit a in T, x,y in T* und Startsymbol S:

x a y

! = "#

Lesezeiger x a y

!

Lesezeiger

Schiebe Schritt (shift): Reduktionsschritt (reduce):

"$=>

80

x a y

! a

Lesezeiger

Schematischer Syntaxbaum zur Eingabe xay mit a in T, x,y in T* und Startsymbol S:

x a y

! = "#

Lesezeiger x a y

!

Lesezeiger

Schiebe Schritt (shift): Reduktionsschritt (reduce):

"$ =>

80

x a y

!a

Lesezeiger

a in T, x,y in T* und Startsymbol S:

x a y

! = "#

Lesezeiger x a y

!

Lesezeiger

Schiebe Schritt (shift): Reduktionsschritt (reduce):

"$=>

Read Pointer

Reduce step:

80

x a y

! a

Lesezeiger

Schematischer Syntaxbaum zur Eingabe xay mit a in T, x,y in T* und Startsymbol S:

x a y

! = "#

Lesezeiger x a y

!

Lesezeiger

Schiebe Schritt (shift): Reduktionsschritt (reduce):

"$ =>

© A. Poetzsch-Heffter, TU Kaiserslautern 80 26.04.2007

x a y

! a

Lesezeiger

Schematischer Syntaxbaum zur Eingabe xay mit a in T, x,y in T* und Startsymbol S:

x a y

! = "#

Lesezeiger x a y

!

Lesezeiger

Schiebe Schritt (shift): Reduktionsschritt (reduce):

"$ =>

x a y

! a

Lesezeiger

Schematischer Syntaxbaum zur Eingabe xay mit a in T, x,y in T* und Startsymbol S:

x a y

! = "#

Lesezeiger x a y

!

Lesezeiger

Schiebe Schritt (shift): Reduktionsschritt (reduce):

"$ =>

Read Pointer

(11)

LR Parsing: Shift and Reduce (3)

Main Problems:

• Reductions can only be performed if remaining prefix is still a viable prefix.

• When to shift? When to reduce? Which production to use?

Solution:

For each grammar Γ construct LR − DFA(Γ) automaton (also called LR(0) automaton), that describes the viable prefixes.

(12)

Construction of LR-DFA

Let Γ =( T,N,Π,S) be a CFG.

• For each non-terminal n ∈ N, construct Item Automaton

• Build union of item automata: Start state is the start state of item automaton for S, Final states are final states of item automata

• Add # transitions from each state which contains the position point in front for a non-terminal A to the starting state of the item

automaton of A

If all states of the LR-DFA automaton are considered as final states, the accepted language is the language of viable prefixes.

(13)

Construction of LR-DFA: Example

Γ₃: S → E#, E →_!⁵: S E + T|TE # , E , T → ID E + T | T , T ID

Beispiel: (Konstruktion eines LR-DEA)

Konstruktion des LR-DEA für

[S .Ê ^#] ^[S Ê.^{# ]} ^[S Ê#.^]

[E .^E+T]

[E .^{T ]}

[T .^{ID ]}

[E E+.^T] ^{[E E+T}.^]

[E T.^]

[T ID.^]

E #

E + T

ID

[E E.^+T]

T

"

" "

"

Deterministisch machen liefert folgenden Automaten:

(14)

Construction of LR-DFA: Example (2)

Determinisation:

83

[S .^E ^#]

[S E.^{# ]}

[S E#.^]

[E .^E+T]

[E .^{T ]}

[T .^{ID ]}

[E E+.^T]

[E E+T.^]

[E T.^] ^[T ^ID.^]

E #

+

T

ID ^Fehler

T

[E E.^+T]

bezeichnet Fehlerkanten

q⁰

q¹ q²

q³

q⁴ q⁵

q⁶

Die zuverlässigen Präfixe maximaler Länge:

E# , T , ID , E+ID , E+T

[T .^{ID ]}

ID

Bemerkungen:

• Im Beispiel enthält jeder Endzustand genau eine vollständig gelesene Produktion. Dies ist im Allg.

nicht so.

• Enthält ein Endzustand mehrere vollständig gelesene Produktionen spricht man von einem reduce/reduce- Konflikt.

• Enthält ein Endzustand eine vollständig gelesene und eine unvollständig gelesene Produktion

mit einem Terminal nach dem Positionspunkt, spricht man von einem shift/reduce-Konflikt.

q⁷

Error

Error Transitions

Viable prefixes of maximal length: E#, T, ID, E + ID, E + T

(15)

Construction of LR-DFA: Example (3)

Remarks:

• In the example, each final state contains one completely read production, this is in general not the case.

• If a final state contains more than one completely read productions, we have a reduce/reduce conflict.

• If a final state contains a completely read and an uncompletely read production with a terminal after the position point, we have a shift/reduce conflict.

(16)

Analysis with LR-DFA

Analysis of ID + ID + ID # with LR-DFA (the viable prefix is underlined)

Analyse von ID + ID + ID # mit dem LR-DEA, unterstrichen ist jeweils der zuverlässige Präfix:

ID + ID + ID # <=

T + ID + ID # <=

E + ID + ID # <=

E + T + ID # <=

E + ID # <=

E + T # <=

E # <=

S

Beispiel: (Analyse mit LR-DEA)

Beachte:

• Die Satzformen bestehen immer aus einem zuverlässigen Präfix und der Resteingabe.

• Verwendet man nur den LR-DEA

zur Analyse muss man nach jeder Reduktion die Satzform von Anfang an lesen.

deshalb: verwende Kellerautomaten zur Analyse

(17)

Analysis with LR-DFA (2)

Note:

• The sentential forms always consist of a viable prefix and a remaining input.

• If an LR-DFA is used, after each reduction the sentential form has to be read from the beginning.

Thus: Use pushdown automaton for analysis.

(18)

LR pushdown automaton

Definition

Let Γ =( N, T,Π,S) be a CFG. The LR-DFA pushdown automaton for Γ contains:

• a finite set of state Q (the states of the LR-DFA(Γ))

• a set of actions Act = {shift,accept,error} ∪ red(Π), where red(Π) contains an action reduce(A → α) for each production A → α.

• an action table at : Q → Act.

• a successor table succ : P × (N ∪ T) → Q with P = {q ∈ Q |at(q) = shift}

(19)

LR pushdown automaton (2)

Remarks:

• The LR-DFA pushdown automaton is a variant of pushdown automata particularly designed for LR parsing.

• States encode the read left context.

• If there are no conflicts, the action table can be directly constructed from the LR-DFA:

! accept: final state of item automaton of start symbol

! reduce: all other final states

! error: error state

! shift: all other states

(20)

Execution of Pushdown Automaton

• Configuration: Q^∗ × T^∗ where variable stack denotes the

sequence of states and variable inr denotes the remaining input

• Start configuration: (q₀,input), where q₀ is the start state of the LR-DFA

• Interpretation Procedure:

(stack, inr) := (q0,input);

do {

step(stack,inr);

} while ( at(top(stack)) != accept

and at(top(stack)) ! = error );

if (( at (top(stack)) == error) return error;

with

(21)

Execution of Push-Down Automaton (2)

void step ( var StateSeq stack, var SymbolSeq inr) { State tk: = top(stack);

switch ( at(tk) ) { case shift:

stack: = push ( succ (tk,top(inr)), keller);

inr := tail(inr);

break;

case reduce A -> a:

stack := mpop( length(a) ,stack);

stack := push (succ(top(stack), A), stack);

break;

} }

(22)

LR push down automaton: Example

LR-DFA with states q₀, . . . , q₇ for grammar Γ₃ Action Table

87

Beispiel: (LR-Kellerautomat zu

!⁵

)

Aktionstabelle:

q

⁰

schieben q

¹

schieben q

²

akzeptieren q

³

schieben

q

⁴

reduzieren E E+T q

⁵

reduzieren E T q

6

reduzieren T ID q

7

fehler

Nachfolgertabelle:

ID + # E T q

⁰

q

⁶

q

⁷

q

⁷

q

¹

q

⁵

q

¹

q

⁷

q

³

q

²

q

⁷

q

⁷

q

²

q

³

q

⁶

q

⁷

q

⁷

q

⁷

q

⁴

q

⁴

q

⁵

q

6

q

7

LR-DEA mit Zuständen q

0

– q

7

(siehe Beipiel oben)

Rechnung zu Eingabe ID + ID + ID # :

Keller Eingaberest Aktion q

0

q

⁰

q

⁶

q

⁰

q

⁵

q

⁰

q

1

q

⁰

q

¹

q

³

q

⁰

q

¹

q

³

q

⁶

q

⁰

q

¹

q

³

q

⁴

q

⁰

q

1

q

0

q

1

q

3

q

0

q

1

q

3

q

6

q

0

q

1

q

3

q

4

q

0

q

1

q

0

q

1

q

2

ID + ID + ID # schieben

+ ID + ID # reduzieren T ID + ID + ID # reduzieren E T + ID + ID # schieben

ID + ID # schieben

+ ID # reduzieren T ID + ID # reduzieren E E+T + ID # schieben

ID # schieben

# reduzieren T ID

# reduzieren E E+T

# schieben akzeptieren

shift accept

error reduce

shift shift reduce reduce

Successor Table

Beispiel: (LR-Kellerautomat zu _!

⁵

)

Aktionstabelle:

q

⁰

schieben q

¹

schieben q

²

akzeptieren q

³

schieben

q

⁴

reduzieren E E+T q

⁵

reduzieren E T q

⁶

reduzieren T ID q

⁷

fehler

Nachfolgertabelle:

ID + # E T q

⁰

q

⁶

q

⁷

q

⁷

q

¹

q

⁵

q

¹

q

⁷

q

³

q

²

q

⁷

q

⁷

q

²

q

³

q

⁶

q

⁷

q

⁷

q

⁷

q

⁴

q

⁴

q

⁵

q

⁶

q

⁷

LR-DEA mit Zuständen q

⁰

– q

⁷

(siehe Beipiel oben)

Rechnung zu Eingabe ID + ID + ID # :

Keller Eingaberest Aktion q

0

q

⁰

q

⁶

q

⁰

q

⁵

q

⁰

q

1

q

⁰

q

¹

q

³

q

⁰

q

¹

q

³

q

⁶

q

⁰

q

¹

q

³

q

⁴

q

⁰

q

1

q

⁰

q

¹

q

³

q

⁰

q

¹

q

³

q

⁶

q

⁰

q

¹

q

³

q

⁴

q

⁰

q

1

q

⁰

q

¹

q

2

ID + ID + ID # schieben

+ ID + ID # reduzieren T ID + ID + ID # reduzieren E T + ID + ID # schieben

ID + ID # schieben

+ ID # reduzieren T ID + ID # reduzieren E E+T + ID # schieben

ID # schieben

# reduzieren T ID

# reduzieren E E+T

# schieben akzeptieren

(23)

LR push down automaton: Example (2)

Computation for Input ID + ID + ID #

Beispiel: (LR-Kellerautomat zu

!⁵

)

Aktionstabelle:

q⁰ schieben q1 schieben q2 akzeptieren q3 schieben

q⁴ reduzieren E E+T q⁵ reduzieren E T q6 reduzieren T ID q7 fehler

Nachfolgertabelle:

ID + # E T q⁰q⁶q⁷q⁷q¹q⁵ q1 q7 q3 q2 q7 q7

q2

q3 q6 q7 q7 q7 q4

q⁴ q⁵ q6

q7

LR-DEA mit Zuständen q⁰ – q⁷ (siehe Beipiel oben)

Rechnung zu Eingabe ID + ID + ID # :

Keller Eingaberest Aktion q0

q0 q6

q0 q5

q0 q1

q0 q1 q3

q⁰q¹ q³q⁶ q0 q1 q3 q4

q0 q1

q0 q1 q3

q0 q1 q3 q6

q⁰q¹ q³q⁴ q q

ID + ID + ID # schieben

+ ID + ID # reduzieren T ID + ID + ID # reduzieren E T + ID + ID # schieben

ID + ID # schieben

+ ID # reduzieren T ID + ID # reduzieren E E+T + ID # schieben

ID # schieben

# reduzieren T ID

# reduzieren E E+T

# schieben

Stack Input Rest Action

shift shift shift

shift shift

shift reduce reduce reduce reduce reduce reduce

(24)

LR-DFA Construction

Questions:

• Does LR-DFA construction work for all unambiguous grammars?

• For which grammars does the construction work?

• How can the construction be generalized / made more expressive?

(25)

Example LR-DFA

LR-DFA for Γ₆: S → E#, E → T + E|T, T → ID|N(), N → ID

Fragen:

• Funktioniert die obige Konstruktion für alle eindeutigen Grammatiken?

• Für welche Grammatiken funktioniert sie?

• Wie kann man sie verallgemeinern/mächtiger machen?

Beispiel:

LR-DEA für !^{6 :}

S E # , E T+E | T , T ID | N( ) , N ID

[S .^E ^#]

[S E.^{# ]} _[S _E#.^]

[E .^T+E]

[E .^{T ]}

[T .^{ID ]}

[ T N(.^{) ]}

E

#

+ T

ID T

q⁰

q¹ q² q³

q⁴

q⁵

q⁶

ID [T .^{N( ) ]}

[N .^{ID ]}

[E T.^+E]

[E T. ^] ^{[E T+}.^E]

[E .^{T ]}

[E .^T+E]

[T .^{N( ) ]}

[N .^{ID ]}

[T .^{ID ]}

[E T+E.^]

E

[T ID.^]

q¹⁰

[N ID.^]

N

[ T N.^{( ) ]} ^{[ T N( )}. ^]

N

( )

q⁸

q⁷ q⁹

error

(26)

LR Parsing Conflicts

2 Kinds of Conflicts:

• Shift/Reduce Conflicts (q₄ in example)

• Reduce/Reduce Conflicts (q₆ in example)

(27)

LR Parsing Theory

Definition

Let Γ =( N,T,Π,S) be a CFG and k ∈ N. Γ is an LR(k) grammar if for any two right derivations

S ⇒^∗_rm αAu ⇒^rm αβu S ⇒^∗_rm γBv ⇒rm αβw it holds that:

If prefix(k,u) = prefix(k, w) then α = γ, A = B and v = w

(28)

LR Parsing Theory (2)

Remarks:

• While for LL grammars the selection of the production depends on the non-terminal to be derived, for LR grammars it depends on the complete left context.

• For LL grammars, the look ahead considers the language to be generated from the non-terminal. For LR grammars, the look ahead considers the language generated from not yet read non-terminals.

(29)

Characterization of LR(0)

Theorem

A reduced CFG Γ is LR(0) if-and-only-if the LR-DFA(Γ) contains no conflicts.

Proof.

cf. Lecture

(30)

Characterization of LR(0) (2)

Example: Application of LR(0)-Chracterization Show (using the above theorem) that Γ₅ is LR(0).

Γ₅:

• S → A|B

• A → aAb|0

• B → aBbb|1

(31)

Expressiveness of LR(k)

• For each context-free language L with prefix property

(i.e. ∀v,w ∈ L: v is no prefix of w), there exists an LR(0) grammar.

• Grammar Γ₅ is not LL(k), but LR(0).

• Methods for LR(1) can be generalized to LR(k), SLR(k) and LALR(k).

(32)

Resolving Conflicts by Look Ahead

• Compute look ahead sets from (N ∪ T)^≤^k for items. The look ahead set of an item approximates the set of prefixes of length k with which the input rest at this item can start.

• If the look ahead sets at an item are disjoint, then the action to be executed (shift, reduce) can be determined by k symbols look

ahead.

• For an item, select the action whose look ahead set contains the prefix of the input rest. Action table has to be extended.

• For computation of look ahead sets, there are different methods.

(33)

Common Methods for Look Ahead Computation

• SLR(k) uses LR-DFA and FOLLOW_k of conflicting items for look ahead

• LALR(k) - look ahead LR - uses LR-DFA with state-dependent look ahead sets

• LR(k) integrates computation of look ahead sets in automata construction (LR(k) automaton)

(34)

SLR Grammars

Definition (SLR(1) grammar)

Let Γ =( N,T,Π,S) be a CFG and LA([A → α.]) = FOLLOW₁(A). A state LR-DEA(Γ) has an SLR(1) conflict if there exists two different reduce items with LA([A → α.]) ∩ LA([B → β.]) )= ∅ or two items

[A → α.] and [B → α.aβ] with a ∈ LA([A → a]).

Γ is SLR(1) if there is no SLR(1) conflict.

(35)

SLR Grammars (2)

Example: Γ₆ is an SLR(1) grammar

• S → E#

• E → T + E|T

• T → ID|N()

• N → ID

Consider the conflicts between [E → T.] and [E → T. + E] and between [T → ID.] and [N → ID.]

FOLLOW₁(E) ∩ {+} = {#} ∩{ +} = ∅

FOLLOW₁(T) ∩ FOLLOW₁(N) = {#,+} ∩{ (} = ∅

(36)

SLR Grammars (3)

Example: Γ₇ (simplifed C expressions) is not an SLR(1) grammar

• S → E#

• E → L = R|R

• L → ∗R|ID

• R → L

(37)

SLR Grammars (4)

LR-DFA for Γ₇

Beispiel: (nicht SLR(1)-Sprache)

Betrachte folgende Grammatik für vereinfachte C-Ausdrücke:

!^{7 :}S E # , E L = R | R , L *R | ID , R L Der zugehörige LR-DEA:

[S .E# ]

[S E.# ] [S E#. ]

[E .L=R]

[E .R]

[E L .=R]

[E L= .R]

[E L=R.]

[E R.]

[R .L]

[R L .]

[L .*R]

[L .ID]

[L * .R]

[L *R.]

[L ID.]

[R .L]

[L .*R]

[L .ID]

[R .L]

[L .*R]

[L .ID]

[R L .]

R E

L

*

= ID

#

R

* ID L

ID

R L

*

Der einzige Zustand mit einem Konflikt enthält die Items [E L .=R] und [R L .] mit

Only conflict in items [E → L. = R] and [R → L.] with

(38)

Construction of LR(1) Automata

LR(1) automaton contains items [A → α.β, V] with V ⊆ T where

• α is on top of the stack

• the input rest is derivable from βc with c ∈ V, i.e.

V ⊆ FOLLOW₁(A).

(39)

Construction of LR(1) Automata (2)

LR(1) automaton for Γ₇. Conflict is resolved, as {=} ∩{ #} = ∅.

Konstruktion von LR(1)-Automaten:

Items der Form [ A !.", V ] mit V T

und der Bedeutung, dass ! auf dem Keller liegt und der Anfang des Eingaberests aus"c ableitbar ist mit c in V. D.h. V FOLLOW1(A) .

U

S .E#

S E.#

S E#.

E .L=R #

E .R #

E L .=R # E L= .R #

E L=R. # E R. #

R .L #

R L . #

L .*R #,=

L .ID #,=

L * .R #,=

L *R. #,=

L ID. #

R .L # L .*R # L .ID #

R .L #,=

L .*R #,=

L .ID #,=

R L . #

E R

L

*

=

ID

#

R L

ID

R L

* L * .R # R .L # L .*R # L .ID #

L *R. # L ID. #,= *

R L . #,=

R L

(40)

LALR(1) Automata

• LALR(1) Automata are constructed from LR(1) automata by

merging states in which items only differ in look ahead sets. Look ahead sets for equal items are conjoint. The resulting automaton has the same states as the LR-DFA.

• However, LALR(1) automata can be generated more efficiently.

(41)

LALR(1) Automata (2)

LR(1) automaton for Γ₇.

LALR(1)-Automaten:

Aus dem LR(1)-Automaten erhält man den LALR(1)- Automaten durch Zusammenlegen der Zustände, in denen sich die Items nur in der Vorausschaumenge unterscheiden. Die Vorausschaumengen zu gleichen Items werden dabei vereinigt. Der resultierende Automat hat die gleichen Zustände wie der LR-DEA..

S .E#

S E.#

S E#.

E .L=R #

E .R #

E L .=R # E L= .R #

E L=R. # E R. #

R .L #

R L . #

L .*R #,=

L .ID #,=

L * .R #,=

L *R. #,=

R .L # L .*R # L .ID #

R .L #,=

L .*R #,=

E R

L

*

=

ID

#

ID

R

L

* L ID. #,=

R L . #,=

L q0

q1

q2 q3

q4

q5

q6

q7

q8

q9

(42)

Grammar Classes

Zusammenhang der Grammatikklassen:

Wilhelm, Maurer:

• aus Kap. 8, Abschnitt 8.4.1 bis einschl. 8.4.5, S. 353 – 383.

mehrdeutige Grammatiken eindeutige Grammatiken

LR(k) LR(1)

LALR(1) SLR(1)

LR(0) LL(k)

LL(1)

LL(0)

unambiguous grammars

ambiguous grammars

(43)

Literature

Recommended Reading for Bottom-Up Analysis:

• Wilhelm, Maurer: Chapter 8, Sections 8.4.1 - 8.4.5, pp. 353 - 383

(44)

Parser Generators

Educational Objectives

• Usage of Parser Generators

• Characteristics of Parser Generators

(45)

JavaCUP Parser Generator

• CUP - Constructor of Useful Parsers

http://www2.cs.tum.edu/projects/cup/

• Java-based Generator for LALR-Parsers

• JFlex can be used to generate according scanner.

• Running JavaCUP:

java -jar java-cup-11a.jar options inputfile

(46)

Structure of JavaCUP Specification

package JavaPackageName;

import java_cup.runtime.*;

/* User supplied code for scanner, actions, ... */

/* Terminals (tokens returned by the scanner). */

terminal TerminalDecls;

/* Non-terminals */

non terminal NonTerminalDecls;

/* Precedences */

precedence [left | right | nonassoc ] TerminalList;

/* Grammar */

start with non-terminalName;

non_terminalName :: = prod_1 | ... | prod_n ;

(47)

Example: JavaCUP Specification for Γ

₇

import java_cup.runtime.*;

/* Terminals (tokens returned by the scanner). */

terminal ID, EQ, MULT;

/* Non terminals */

non terminal S, E, L, R;

/* The grammar */

start with S;

S ::= E;

E ::= L EQ R | R;

(48)

Structure of Generated Parser Code

• Output Files parser.java and sym.java

• Tables for LALR Automaton

! Production table: provides the symbol number of the left hand side non-terminal, along with the length of the right hand side, for each production in the grammar,

! Action table: indicates what action (shift, reduce, or error) is to be taken on each lookahead symbol when encountered in each state

! Reduce-goto table: indicates which state to shift to after reduce

(49)

Usage of Generated Parser

• Parser calls scanner with scan() method when a new terminal is needed

• Initialising Parser with new Scanner

parser parser_obj = new parser(new my_scanner());

• Usage of Parser:

Symbol parse_tree = parser_obj.parse();

(50)

Error Handling

Educational Objectives:

• Problems and Principles of Error Handling

• Techniques of Error Handling for Context-Free Analysis

(51)

Error Handling

Principles of Error Handling

Error handling is required in all analysis phases and at runtime. One distinguishes

• lexical errors

• parse errors (in context-free analysis)

• errors in name and type analysis

• runtime errors (cannot be avoided in most cases)

• logical errors (behavioural errors)

First 2 (3) kinds of errors are syntactic errors. We only consider error handling in context-free analysis.

Specification of error handling results basically from language

(52)

Requirements for error handling

• Errors should be localized as exactly as possible.

(Problem: Error is not detected at error position.)

• As many errors at possible should be detected at once, but only real errors and no errors as consequences.

• Errors are not always unique, i.e. it is not clear in general how to correct an error: class int { Int a; .... } or int a = 1-;

• Error handling should not slow down analysis of correct programs.

Therefore, error handling is non-trivial and depends on the source language to be analysed.

(53)

Error Handling

Error Handling in Context-Free Analysis

1. Panic Error Handling

Mark synchronizing terminal symbols, e.g. end or ;

If parser reaches error state, all symbols up to next synchronizing symbol are skipped and the stack is corrected as if the production with the synchronizing symbol was read correctly.

! Pros: easy to implement, termination guaranteed

! Cons: large parts of the program can be skipped or misinterpreted

! Example: Incorrect Input a : = b *** c;

Read until ; correct stack and reuse as if statement has been accepted

(54)

Error Handling in Context-Free Analysis (2)

2. Error Productions

Extend grammar with productions describing typical error

situations, so called error productions. Error messages can be directly associated with error productions.

! Pros: easy to implement, termination guaranteed

! Cons: extended grammar can belong to more general grammar class, knowledge of typical error situations is necessary

! Example: Typical error in PASCAL if ... then A := E; else ...

Error Production:

Stmt → if Expr then Stmt^∗ ; else Stmt^∗

(55)

Error Handling

Error Handling in Context-Free Analysis (3)

3. Production-Local Error Correction

Goal is local correction of input such that analysis can be

resumed. Local means that it is tried to correct the input for the current production.

! Pros: flexible and powerful technique

! Cons: problematic if errors occur earlier than they can be detected, operations for corrections can lead to nonterminating analysis

(56)

Error Handling in Context-Free Analysis (4)

4. Global Error Correction

Attempt to get a correction that is as good as possible by altering the read input or the look ahead input.

Idea: Define distance or quality measure on inputs. For each incorrect input, look for a syntactically correct input that is best according to the used measure.

! Pros: very powerful technique

! Cons: analysis effort can be rather high, implementation is complex and poses risk of non-termination.

(57)

Error Handling

Error Handling in Context-Free Analysis (5)

5. Interactive Error Correction

In modern programming languages, syntactic analysis is often already supported by editors. In this case, editor marks error positions.

! Pros: quick feedback, possible error positions are shown directly, interaction with programmer possible

! Cons: editing can be disturbed, analysis must be able to handle incomplete programs

The presented techniques can also be combined. For selection of technique, programming language syntax is important. Error

handling also depends on grammar class and implementation techniques used for parser.

(58)

Burke-Fischer Error Handling

Example of global error correction technique

• Procedure: Use correction window of n symbols before symbol at which error was detected. Check all possible variations of symbol sequence in correction window that can be obtained by insertion, exchange or modification of a symbol at any position.

• Quality Measure: Choose variation that allows longest continuation of parsing procedure

• Implementation: Work with two stack automata, one represents the configuration at the beginning of the correction window, the other one the configuration at the end of the correction window. In an error case, the automaton running behind can be used to

resume at the old position and to test the computed variations.

(59)

Error Handling

Literature

Recommended Reading: Wilhelm, Maurer: Chapter 8, Sections 8.3.6 and 8.4.6 (general understanding sufficient)

(60)

Concrete and Abstract Syntax

Educational Objectives

• Connection of parsing to other phases of program processing and translation

• Differences between abstract and concrete syntax

• Language concepts for describing syntax trees

• Syntax tree construction

(61)

Concrete and Abstract Syntax

Connection of Parsers to other Phases

1. Parser directly controls following phases 2. Concrete Syntax Tree as Interface

3. Abstract Syntax Tree as Interface

(62)

Direct Control by Parser

• Example: Recursive Descent: Parser calls other actions after each derivation/reduction step

• Pros:

! simple (if realisable)

! flexible

! efficient (especially memory efficient)

• Cons:

! non-modular, no clear interfaces

! not suitable for global aspects of translation

! following phases depend on parsing

! cannot be used with every parser generator

(63)

Abstract Syntax vs. Concrete Syntax

Definition (Concrete Syntax)

The concrete syntax of a programming languages determines the actual text representation of the programs (incl. key words,

separators).

If Γ is the CFG used for parsing a program P in a certain language, the syntax tree of P according to Γ is the concrete syntax tree of P.

Definition (Abstract Syntax)

The abstract syntax of a programming language describes the tree

structure of programs in a form that is sufficient and suitable for further processing.

A tree for representing a program P according to the abstract syntax of

(64)

Abstract Syntax

• abstraction from keywords and separators

• operator precedences are represented in tree structure (different non-terminals are not necessary)

• better incorporation of symbol information

• simplifying transformations Remarks:

• The abstract syntax of a language is often not specified in the language report.

• The abstract syntax usually also comprises information about source code positions.

(65)

Example: Concrete vs. Abstract Syntax

Concrete Syntax: Γ₂

• S → E#

• E → T + E |T

• T → F ∗ T |F

• F → (E) |ID Abstract Syntax

• Exp = Add | Mult Ident

• Add (Exp left, Exp right)

• Mult (Exp left, Exp right)

(66)

Example: Concrete vs. Abstract Syntax (2)

Text: (a + b) ∗ c

Concrete Syntax Tree

Textrepräsentation: ( a + b ) * c

Konkreter Syntaxbaum: Abstrakter Syntaxbaum:

S Mult

T

E #

Mult

Add c

F F

T

a b

E

E T

F F T

( ID ID ) * ID

( ID + ID ) * ID a b c

113

Abstract Syntax Tree

Textrepräsentation: ( a + b ) * c

Konkreter Syntaxbaum: Abstrakter Syntaxbaum:

S Mult

T

E #

Mult

Add c

F F

T

a b

E

E T

F F T

( ID ID ) * ID

( ID + ID ) * ID a b c

113

(67)

Concrete Syntax Tree as Interface

Token Stream

Parser

(with Tree Construction)

Concrete Syntax Tree

Further Language Processing

• Counters disadvantages of direct control by parser

• Advantages over Abstract Syntax

! No additional specification of abstract syntax required

(68)

Abstract Syntax Tree as Interface

Token Stream

Parser

(with Transforming Tree Construction)

Abstract Syntax Tree

Further Language Processing

• Advantages over Concrete Syntax

! Simpler, more compact tree representation

! Simplifies later phases

! Often implemented by programming or specification language as mutable data structure

(69)

Abstract Syntax: Specification and Tree Construction

• For representing abstract syntax trees, we use order-sorted terms.

• The sets and types of these terms are described by type declarations.