Compiler and Language Processing Tools Summer Term 2009 Introduction Dr.-Ing. Ina Schaefer

(1)

Compiler and Language Processing Tools

Summer Term 2009 Introduction

Dr.-Ing. Ina Schaefer

Software Technology Group TU Kaiserslautern

Ina Schaefer Compilers 1

(2)

Introduction

Outline

1. Introduction

Overview and Application Domains Tasks of Language-Processing Tools Examples

2. Language Processing

Terminology and Requirements Compiler Architecture

3. Compiler Construction

(3)

Introduction Overview and Application Domains

Language Processing Tools

• Processing of Source Texts in Source Languages

• Analysis of Source Texts

• Translation to Target Languages

(4)

Language Processing Tools (2)

Typical Source Languages

• Programming Languages: C, C++, C#, Java, ML, Smalltalk, Prolog, Script Languages (JavaScript), bash

• Languages for Configuration Management: make, ant

• Application and Tool-Specific Languages: Excel, JFlex, CUPS

• Specification Languages: Z, CASL, Isabelle/HOL

• Formatting and Data Description Languages: LaTeX, HTML, XML

• Design and Architecture Description Languages: UML, SDL, VHDL, Verilog

(5)

Language Processing Tools (3)

Typical Target Languages

• Assembly and Machine Languages

• Programming Language

• Data and layout Description Languages

• Languages for Printer Control

(6)

Language Processing Tools (4)

Language Implementation Tasks

• Tool Support for Language Processing

• Integration into Existing Systems

• Connection to Other Systems

(7)

Application Domains

• Programming Environments

I Context-sensitive Editors, Class Browers

I Graphical Programming Tools

I Pre-Processors

I Compilers

I Interpreters

I Debuggers

I Run-time Environments (loading, linking, execution, memory management)

(8)

Application Domains (2)

• Generation of Programs from Design Documents (UML)

• Program Comprehension, Re-engineering

• Design and Implementation of Application-specific Languages

I Robot Control

I Simulation Tools

I Spread Sheets, Active Documents

• Web Technology

I Analysis of Web Sites

I Active Websites (with integrated functionality)

I Abstract Platforms, e.g. JVM, .NET

I Optimization of Caching

(9)

Related Fields

• Formal Languages, Language Specification and Design

• Programming and Specification Languages

• Programming, Software Engineering, Sotware Generation, Software Architecture

• System Software, Computer Architecture

(10)

Introduction Tasks of Language-Processing Tools

Tasks of Language-Processing Tools

Analyser Translation Interpreter Source Code Source Code

Target Code Analysis

Results

Source Code

Input Data

Output Data

Analysis, Translation and Interpretation are often combined.

(11)

Tasks of Language-Processing Tools (2)

1. Translation

I Compiler implements Analysis and Translation

I OS and Real Machine implement Interpretation Pros:

I Most efficient solution

I One interpreter for all programming languages

I Prerequisite for other solutions

(12)

Tasks of Language-Processing Tools (3)

2. Direct Interpretation

I Interpreter implements all tasks.

I Examples: Java Script, Command Line Languages (bash)

I Pros: No translation necessary (but analysis at run-time)

(13)

Tasks of Language-Processing Tools (4)

3. Abstract and Virtual Machines

I Compiler implements Analysis and Translation to Abstract Machine Code

I Abstract Machine works as Interpreter

I Examples: Java/JVM, C# ˙NET Pros:

• Platform independent (portability, mobile code)

• Self-modifing programs possible

4. Other Combinations

(14)

Introduction Examples

Example: Analysis

17.04.2007 © A. Poetzsch-Heffter, TU Kaiserslautern 8

package b1_1

;

class Weltklasse extends Superklasse implement BesteBohnen {Qualifikation studieren ( Arbeit schweiss) { return new

Qualifikation ();}}

Beispiel: (Analyse)

javac-Analysator

Superklasse.class Qualifikation.class Arbeit.class

BesteBohnen.class

...

b1_1/Weltklasse.java:4: '{' expected.

extends Superklasse

^ 1 error

(15)

Example: Translation

17.04.2007 © A. Poetzsch-Heffter, TU Kaiserslautern 9 package b1_1;

class Weltklasse extends Superklasse implements BesteBohnen {

Qualifikation studieren ( Arbeit schweiss ) { return new Qualifikation();

}}

Beispiel 1: (Übersetzung)

javac

Superklasse.class Qualifikation.class Arbeit.class

BesteBohnen.class

...

Compiled from Weltklasse.java class b1_1/Weltklasse

extends ... implements ... { b1_1/Weltklasse();

b1_1.Qualifikation studieren(...);

}

Method b1_1/Weltklasse() ...

Method b1_1.Qualifikation studieren(...) ...

(16)

Example: Translation (2)

Result of Translation_17.

04.200710© A. Poetzsch-Heffter, TU Kaiserslautern Beispiel 1:(Fortsetzung)

Compiled from Weltklasse.java class b1_1/Weltklasse

extends b1_1.Superklasse implements b1_1.BesteBohnen { b1_1/Weltklasse();

b1_1.Qualifikation studieren(b1_1.Arbeit);

}

Method b1_1/Weltklasse() 0 aload_0

1 invokespecial #6 <Method b1_1.Superklasse()>

4 return

Method b1_1.Qualifikation studieren(b1_1.Arbeit) 0 new #2 <Class b1_1.Qualifikation>

3 dup

4 invokespecial #5 <Method b1_1.Qualifikation()>

7 areturn

(17)

Example 2: Translation

17.04.2007 © A. Poetzsch-Heffter, TU Kaiserslautern 11 int main() {

printf("Willkommen zur Vorlesung!");

return 0;

}

gcc

.file "hello_world.c"

.version "01.01"

gcc2_compiled.:

.section .rodata .LC0:

.string "Willkommen zur Vorlesung!"

.text .align 16 .globl main

.type main,@function main:

pushl %ebp movl %esp,%ebp subl $8,%esp ...

(18)

Example 2: Translation (2)

Result of Translation

Beispiel 2: (Fortsetzung)

.file "hello_world.c"

.version "01.01"

gcc2_compiled.:

.section .rodata .LC0:

.string "Willkommen zur Vorlesung!"

.text .align 16 .globl main

.type main,@function main:

pushl %ebp movl %esp,%ebp subl $8,%esp addl $-12,%esp pushl $.LC0 call printf addl $16,%esp xorl %eax,%eax jmp .L2 .p2align 4,,7 .L2:

movl %ebp,%esp popl %ebp ret .Lfe1:

.size main,.Lfe1-main

.ident "GCC: (GNU) 2.95.2 19991024 (release)"

(19)

Example 3: Translation

\documentclass{article}

\begin{document}

\vspace*{7cm}

\centerline{\Huge\bf It‘s groovy}

\end{document}

groovy.tex (104 bytes)

...

groovy.dvi (207 bytes, binary)

%!PS-Adobe-2.0

%%Creator: dvips(k) 5.86 ...

%%Title: groovy.dvi ...

groovy.ps (7136 bytes)

latex

dvips

(20)

Example: Interpretation

Beispiel: (Interpretation)

...

14 iload_1 15 iload_2 16 idiv 17 istore_3 ...

.class-Datei

Eingabedaten

Ausgabedaten ...

14 iload_1 15 iload_2 16 idiv 17 istore_3 ...

Java Virtual Machine (JVM)

Input Data

Output Data .class File

(21)

Example: Combined Technique

Java Implementation with Just-In-time (JIT) Compiler

Kombinierte Implementierungstechnik:

Java-Implementierung mit JIT-Übersetzer

Java-Überset- zungseinheit

javac

Analysator Übersetzer

Eingabedaten

Java Byte Code .class-Datei

Ausgabedaten JIT-Übersetzer

JVM

Maschinencode reale Maschine/Hardware (JIT=Just in time) Beispiel: (Kombinierte Technik)

Java Source Code Unit

Analyzer Translator

Input Data

Output Data .class file

JIT Translator

Machine Code Real Machine / Hardware

(22)

Language Processing Terminology and Requirements

Language Processing: The Translation Task

Translator Source Code

Error Message or Target Code

• Translator(in a broader sense):

Analysis, Optimization and Translation

• Souce Code:

Input (String) for Translator in Syntax of Source Language (SL)

• Target Code:

Output (String) of Translator in Syntax of Target Language (TL)

(23)

Phases of Language Processing

• Analysis of Input:

I Program Text

I Specification

I Diagrams

• Dependant on Target of Implementation

I Transformation (XSLT, Refactoring)

I Pretty Printing, Formatting

I Semantic Analysis (Program Comprehension)

I Optimization

I (Actual) Translation

(24)

Compile Time vs. Run-time

• Compile Time: during Run-time of Compiler/Translator Static: All Information/Aspects known at Compile Time, e.g.

I Type Checks

I Evaluation of Constant Expressions

I Relative Adresses

• Run Time: during Run-time of Compiled Program

Dynamic: All Information that are not statically known, e.g.

I Allocation of Dynamic Arrays

I Bounds Check of Arrays

I Dynamic Binding of Methods

I Memory Management of Recursive Procedures

Fordynamic aspectsthat cannot be handled atcompile time, the compiler generates code that handles these aspects atruntime.

(25)

What is a good compiler?

(26)

Requirements for Translators

• Error Handling (Static/Dynamic)

• Efficient Target Code

• Choice: Fast Translation with Slow Code vs. Slow Translation with Fast Code

• Semantically Correct Translation

(27)

Semantically Correct Translation

Intuitive Definition: Compiled Program behaves according to Language Definition of Source Language.

Formal Definition:

• semSL: SL_Program×SL_Data→SL_Data

• semTL: TL_Program×TL_Data→TL_Data

• compile: SL_Program→TL_Program

• code: SL_Data→TL_Data

• decode: TL_Data→SL_Data Semantic Correctness:

semSL(P,D) = decode(semTL(compile(P), code(D)))

(28)

Language Processing Compiler Architecture

Compiler Architecture

Scanner Source Code

as String

Token Stream

Parser

Name and Type Analysis

Translator

Code Generator Syntax

Tree

Decorated Syntax Tree

(Close to SL)

Intermediate Language

Target Code as String

Attribution &

Optimization

Attribution &

Optimization

Peep Hole Optimization Analysis

Synthesis

(29)

Properties of Compiler Architectures

• Phases represent Concepts.

• Phases can be interleaved.

• Concrete Layout of Phases depends on Source Language, Target Language and Concrete Design Decisions.

• Phase vs.Pass(Phase can comprise more than one pass.)

• Separate Translation of Program Parts (Interface information must be accessible.)

• Combination with other Architecture Decisions:

Common Intermediate Language

(30)

Common Intermediate Language

Source Language 1

Source Language 2

Source Language n

Intermediate Language

Target Language 1

Target Language 2

Target Language m ...

...

(31)

Dimensions of Compiler Construction

• Programming Languages

I Sequential Procedural, Imperative, OO-Languages

I Functional, Logical Languages

I Parallel Languages/Language Constructs

• Target Languages/Machines

I Code for Abstract Machines

I Assembler

I Machine Languages (CISC, RISC, ...)

I Multi Processor Architectures

I Memory Hierarchy

• Translation Tasks: Analysis, Optimization, Synthesis

• Construction Techniques and Tools: Bootstrapping, Generators

• Portability, Specification, Correctness

(32)

Compiler Construction

Compiler Construction Techniques

1. Stepwise Construction

I Construction with compiler for different language

I Construction with compiler for different machine

I Bootstrapping

2. Compiler - Compiler: Tools for Compiler Generation

I Scanner Generators (regular expressions)

I Parser Generators (context-free grammars)

I Attribute Evaluation Generators (attribute grammar)

I Code Generator Generators (machine specification)

I Interpreter Generators (semantics of language)

I Other Phase-specific Tools 3. Special Programming Techniques

I General Technique: syntax-driven

I Special Technique: recursive descend

(33)

Stepwise Construction

Programming typically depends on existing compiler for implementation language. For compiler construction, this does not hold in general.

Source, target and implementation languages of compilers can be denoted in T-Diagrams.

17.04.2007 © A. Poetzsch-Heffter, TU Kaiserslautern 22

Konstruktionstechniken:

Programmierung basiert üblicherweise auf Existenz eines Übersetzers für die Implementierungssprache.

Bei Übersetzerkonstruktion kann davon im Allg. nicht ausgegangen werden; stehe

für einen Übersetzer der in Sprache PS geschrieben ist und QS in ZS übersetzt.

QS

PS

ZS

A. Schrittweise Konstruktion:

1. Konstruktion mit Übersetzer für andere Sprache:

Gesucht: QS-Compiler, Ziel MS, läuft auf MS Annahme: C-Compiler existiert auf Plattform mit

Maschinensprache MS

C

MS

MS QS

C

MS QS

MS

MS zu entwickeln

existiert

durch Übersetzung

T-diagram denotes compiler from QS to ZS written in PS.

(34)

Construction with compiler for different language

• Given: SL Compiler

• Construct: Compiler for machine language (MS) in MS

• Suppose: C Compiler exists on platform with machine language

Konstruktionstechniken:

Programmierung basiert üblicherweise auf Existenz eines Übersetzers für die Implementierungssprache.

Bei Übersetzerkonstruktion kann davon im Allg. nicht ausgegangen werden; stehe

für einen Übersetzer der in Sprache PS geschrieben ist und QS in ZS übersetzt.

QS PS

ZS

A. Schrittweise Konstruktion:

1. Konstruktion mit Übersetzer für andere Sprache:

Gesucht: QS-Compiler, Ziel MS, läuft auf MS Annahme: C-Compiler existiert auf Plattform mit

Maschinensprache MS

C

MS MS QS

C

MS QS

MS MS zu entwickeln

existiert

durch Übersetzung

SL SL

existing

to be developed by

translation

(35)

Construction with compiler for different machine

• Construct: C Compiler forM₁inM₁

• Suppose: C Compiler exists forM₂inM₂

• Method: ConstructCross Compilerfirst First Step

2. Konstruktion mit Übersetzer für anderen Rechner:

Gesucht: C-Compiler auf für

Annahme: C-Compiler auf für existiert Methode: realisiere zunächst Cross-Compiler

C C

C

MS1

MS₂ MS₁

MS₂

MS₁ MS₂

MS1

MS₂

Cross-Compiler

C C

C

C MS1

MS1

MS₂

1.Schritt:

2.Schritt:

MS₁

2. Konstruktion mit Übersetzer für anderen Rechner:

Gesucht: C-Compiler auf für

Annahme: C-Compiler auf für existiert Methode: realisiere zunächst Cross-Compiler

C C

C

MS1

MS₂ MS₁

MS₂

MS₁ MS₂

MS1

MS₂

Cross-Compiler

C C

C

C MS1

MS1

MS₂

1.Schritt:

2.Schritt:

MS1

Cross Compiler

(36)

Construction with compiler for different machine (2)

Second Step

2. Konstruktion mit Übersetzer für anderen Rechner:

Gesucht: C-Compiler auf für

Annahme: C-Compiler auf für existiert Methode: realisiere zunächst Cross-Compiler

C C

C

MS

1

MS

₂

MS

1

MS

₂

MS

1

MS

₂

MS

1

MS

1

MS

₂

MS

₂

Cross-Compiler

C C

C

C MS

1

MS

1

MS

₂

1.Schritt:

2.Schritt:

MS

1

2. Konstruktion mit Übersetzer für anderen Rechner:

Gesucht: C-Compiler auf für

Annahme: C-Compiler auf für existiert Methode: realisiere zunächst Cross-Compiler

C C

C

MS

1

MS

₂

MS

₁

MS

₂

MS

₁

MS

₂

MS

1

MS

1

MS

₂

MS

₂

Cross-Compiler

C C

C

C MS

1

MS

1

MS

₂

1.Schritt:

2.Schritt:

MS

1

Cross Compiler

(37)

Bootstrapping

• Construct: QS Compiler forMSinMS

• Suppose: yet no compiler exists

• Method:

1. Construct partial languageQS_i ofQSsuch that

QS₀⊂QS₁⊂QS₂⊂. . .⊂QS

2. ImplementQS₀Compiler forMSinMS 3. ImplementQS_i+1Compiler forMSinQS_i 4. CreateQS_i₊₁Compiler forMSinMS

(38)

Bootstrapping (2)

3. Bootstrapping:

Gesucht: QS-Compiler auf MS für MS Annahme: kein Compiler verfügbar Methode:

1. Entwerfe Teilsprachen von QS mit

2. Implementiere QS -Compiler für MS in MS 3. Implementiere QS -Compiler für MS in QS 4. Erzeuge QS -Compiler für MS in MS

QS₁

QSi

QS

durch Erweiterung

QS MS

2

UQS₂

UQS₁

UQS₀

0

i+1

i

0

i+1

QS0 MS MS

MS QS₁ MS QS2

QS1

MS

MS QS2 MS QS

QS MS

MS MS QS

durch Übersetzung von Hand

manually by extension

by translation

(39)