Flat fileFlat file

(1)

XML Databases

1. Introduction, 27.10.08

Silke Eckstein Andreas Kupfer

Institut für Informationssysteme

Technische Universität Braunschweig http://www.ifis.cs.tu-bs.de

(2)

1.1 Motivation

1.2 Relational Databases – Repetition 1.3 Why use XML?

1.4 XML & Databases

1. Introduction

1.4 XML & Databases 1.5 XML Fundamentals

1.6 Organisational matters 1.7 Overview

1.8 References

2 XML Databases – Silke Eckstein – Institut für Informationssysteme – TU Braunschweig

(3)

"If I invent another programming language, its name will contain the letter X“

1.1 Motivation

its name will contain the letter X“

(N. Wirth, Software Pioniere Konferenz, Bonn 2001)

XML Databases – Silke Eckstein – Institut für Informationssysteme – TU Braunschweig 3

(4)

• Within the last 10 years XML has become the

de facto standard for data exchange over the web

– Examples:

• The latest office documents

• SVG graphics files

1.1 Motivation

• SVG graphics files

• Lots of conguration files

• Some WebCMSs store page contents in XML format

• Mpeg7 is a standard for describing media meta data in XML format

• . . .

– In order to see examples of XML-structured documents, browse through your computer's file system and check for file contents starting with "<?xml "!

(5)

• Why is XML relevant from DB perspective?

– XML is becoming the data "format"

• Amount of XML is ever increasing

• DBMS are good at handling GBs and TBs of data

– Accepted model for semi-structured data

1.1 Motivation

– Accepted model for semi-structured data

• Overcome limitations of structured data

• Extend usefulness of DBMS

– DB technology is not limited to DBMS

• Apps servers, application integration

XML Databases – Silke Eckstein – Institut für Informationssysteme – TU Braunschweig 5 [Fisch05]

(6)

Aim of this lecture

Give answers to the following questions:

• What (additional) concepts do we need in order

1.1 Motivation

• What (additional) concepts do we need in order to store XML data in a RDBMS?

• What concepts are crucial in order to build native XML-DBMS systems?

(7)

1.1 Motivation

1.2 Relational Databases – Repetition 1.3 Why use XML?

1.4 XML & Databases

Outline

1.4 XML & Databases 1.5 XML Fundamentals

1.6 Organisational matters 1.7 Overview

1.8 References

(8)

What is a Database?

• A database (DB) is a collection of related data

– Represents some aspects of the real world

• Universe of Discourse (UoD)

1.2 Relational Databases

Universe of Discourse (UoD)

– Data is logically coherent

– Is provided for an intended group of users and applications

XML Databases – Silke Eckstein – Institut für Informationssysteme – TU Braunschweig 8 [EN06, 1.1]

(9)

What is a Database Management System?

• A database management system (DBMS) is a

collection of programs to maintain a database, i.e.

for

1.2 Relational Databases

for

– Definition of Data and Structure – Physical Construction

– Manipulation

– Sharing/Protecting

– Persistence/Recovery

(10)

Why not use the File System?

• File management systems are physical interfaces

1.2 Relational Databases

F i l Account

Data

App 1

Customer Letters

l e S y s t e m Data

Customer Data

Loans

1

App 2

Balance Sheets Money Transfer

(11)

File Systems

• Advantages

– Fast and easy access

• Disadvantages

1.2 Relational Databases

• Disadvantages

– Uncontrolled redundancy – Inconsistent data

– Limited data sharing and access rights – Poor enforcement of standards

– Excessive data and access paths maintenance

(12)

• Databases are logical interfaces

– Controlled redundancy

– Data consistency & integrity constraints – Integration of data

1.2 Relational Databases

Integration of data

– Effective and secure data sharing – Backup and recovery

• However…

– More complex

– More expensive data access

(13)

• Databases control redundancy

– Same data used by different applications/tasks is only stored once

– Access via a single interface provided by DBMS

1.2 Relational Databases

– Redundancy only purposefully used to speed up data access (e.g. materialized views)

• Databases are well-structured

– Catalog (data dictionary) contains all meta-data – Defines the structure of the data in the database

XML Databases – Silke Eckstein – Institut für Informationssysteme – TU Braunschweig 13 [EN06, 1.6.1, 1.3]

(14)

• Databases aim at efficient manipulation of data

– Physical tuning allows for good data allocation – Indexes speed up search and access

– Query plans are optimized for improved performance

1.2 Relational Databases

Query plans are optimized for improved performance

• Example: Simple Index

AccNo type balance

1278945 saving € 312.10

2437954 saving € 1324.82

4543032 checking € -43.03

5539783 saving € 12.54

7809849 checking € 7643.89

8942214 checking € -345.17

9134354 saving € 2.22

9543252 saving € 524.89

AccNo 1278945 5539783 9134354

Index File

Data File

(15)

• Isolation between applications and data

– Database employs data abstraction by providing data models

– Applications work only on the conceptual representation of data

1.2 Relational Databases

representation of data

• Data is strictly typed (Integer, Timestamp, VarChar,…)

• Details on where data is actually stored and how it is accessed is hidden by the DBMS

• Applications can access and manipulate data by invoking abstract operations (e.g. SQL Select statements)

– DBMS-controlled parts of the file system are strongly protected against outside manipulation (tablespaces)

(16)

• Example: Schema is changed and table-space moved without an application noticing

1.2 Relational Databases

Application

SELEC T AccNo FROM account WHERE balance>0

DBMS

AccNo balance

1278945 € 312.10 2437954 € 1324.82 4543032 € -43.03 5539783 € 12.54

Disk 1 Disk 2

(17)

1.2 Relational Databases

Application

SELEC T AccNo FROM account WHERE balance>0

• Example: Schema is changed and table-space moved without an application noticing

DBMS

AccNo balance

1278945 € 312.10 2437954 € 1324.82 4543032 € -43.03 5539783 € 12.54

AccNo type balance

1278945 saving € 312.10 2437954 saving € 1324.82 4543032 checking € -43.03 5539783 saving € 12.54

Disk 1 Disk 2

(18)

• Databases support multiple views of the data

– Views provide a different perspective of the DB

• A user’s conceptual understanding or

task-based excerpt of all data (e.g. aggregations)

1.2 Relational Databases

task-based excerpt of all data (e.g. aggregations)

• Security considerations and access control (e.g. projections)

– For the application, a view does not differ from a table – Views may contain subsets of a DB and/or contain

virtual data

• Virtual data is derived from the DB (mostly by simple SQL statements, e.g. joins over several tables)

• Can either be computed at query time or materialized upfront

(19)

• Sharing of data and support for atomic multi- user transactions

– Multiple user and applications may access the DB at the same time

– Concurrency control is necessary for maintaining

1.2 Relational Databases

– Concurrency control is necessary for maintaining consistency

– Transactions need to be atomic and isolated from each other

(20)

• Persistence of data and disaster recovery

– Data needs to be persistent and accessible at all times – Quick recovery from system

crashes without data loss

1.2 Relational Databases

– Recovery from natural

desasters ( fire, earthquakes,…)

(21)

1.1 Motivation

1.2 Relational Databases – Repetition 1.3 Why use XML?

1.4 XML & Databases

Outline

1.4 XML & Databases 1.5 XML Fundamentals

1.6 Organisational matters 1.7 Overview

1.8 References

(22)

• Bioinformatics example:

• Presentation and processing of database query results

– Flat file

1.3 Why use XML?

– Flat file – Web page – HTML text – XML text

– Search in TRANSPATH database for molecule "TLR4"

(23)

Key

Originator Molecule name Species

Links to other DBs

F la t f ile

Links to other DBs Gene Ontology

references

Reactions the molecule participates in

Publications

F la t f ile

(24)

Key

Links to other DBs

W e b p a g e

24

references

Reactions the molecule participates in

Publications

W e b p a g e

XML Databases – Silke Eckstein – Institut für Informationssysteme – TU Braunschweig

(25)

Key

Originator

Molecule name

H T M L

Species

H T M L

(26)

Key

Links to other DBs

X M L

references

X M L

(27)

• Flat files

• HTML

• Little layout information

• Suitable for presentation only to a limited extent

• Can be parsed, but cumbersome

• Only layout information

1.3 Why use XML?

• HTML

• Solution

• Only layout information

• Good for presentation

• Automatic processing difficult

• Just as generation of other presentation formats

• Separation of layout and content

(28)

• What is XML?

– The Extensible Markup Language (XML) is the

universal format for structured documents and data on the Web.

– Base specifications:

1.3 Why use XML?

– Base specifications:

• XML 1.0, W3C Recommendation Feb '98

• XML 1.1 (2nd Ed.), W3C Recommendation Aug '06

• Namespaces, W3C Recommendation Jan '99

(29)

• What is XML now then?

– XML is semi-structured text

– XML is a tag-based markup-language (like HTML)

• eXtensible Markup Language

– XML was designed to exchange data

1.3 Why use XML?

– XML was designed to exchange data – XML tags are not predefined

• Tags are defined in a separate schema

– XML is designed to be self-descriptive – XML is a W3C Recommendation

– XML became highly popular due to its simplicity and flexibility

(30)

• XML Data Example

1.3 Why use XML?

<Buch>

<Autor id="1234567890">Rainer Eckstein</Autor>

<Autor id="1234568723">Silke Eckstein</Autor>

<Titel>XML und Datenmodellierung</Titel>

<Untertitel>XML-Schema ...</Untertitel>

<Verlag id="3-89864">dpunkt.Verlag</Verlag>

− Syntax, no abstract model

− Documents, elements and attributes

− Tree-based, nested, hierarchically organized structure

</Buch>

(31)

1.1 Motivation

1.2 Relational Databases – Repetition 1.3 Why use XML?

1.4 XML & Databases

Outline

1.4 XML & Databases 1.5 XML Fundamentals

1.6 Organisational matters 1.7 Overview

1.8 References

(32)

1.4 XML & Databases

• Database world

– 1970 relational databases

– 1990 nested relational model and object

• Documents world

– 1974 SGML (Structured Generalized Markup

Language)

– 1990 HTML (Hypertext

[Fisch05] 32

model and object oriented databases

– 1995 semi-structured databases

– 1990 HTML (Hypertext Markup Language)

– 1992 URL (Universal Resource Locator) Data + documents = information

1996 XML (Extensible Markup Language) URI (Universal Resource Identifier)

XML Databases – Silke Eckstein – Institut für Informationssysteme – TU Braunschweig

(33)

• Information systems have different degrees of data structure rigidness

– Structured, e.g., relational databases

• Structure explicitly specified in schema

• Every tuple in a table has the same attributes and domains

1.4 XML & Databases

• Every tuple in a table has the same attributes and domains

• Queries can take advantage of structure

– Unstructured, e.g., information retrieval systems

• Often just full text with no or only limited structure information

• Properties of data usually unknown

• Queries difficult to evaluate

(34)

• But there is also something in between

– Semi-structured, e.g., XML

• Structure of data follows a template, but still allows for a degree of flexibility

• Data instances following the same schema

1.4 XML & Databases

• Data instances following the same schema may have a different structure

• Often, complex relationships between data are allowed (associations, inheritance, sub-classing, aggregation, etc.)

• Queries often involve those relationships

(35)

• Relational data

– Killer Application:

Banking

– Invented as a

mathematically clean abstract data model

• XML

– 1st killer application:

Publishing industry

– Invented as a syntax for data, only later an abstract data model

1.4 XML & Databases

abstract data model

– Philosophy: schema first, then data

data model

– Philosophy: data and schemas should not be correlated, data can exist with or without schema, or with multiple schemas

(36)

• Relational data

– Never had a standard syntax for data

– Strict rules for data normalization, flat

• XML

– Standard syntax existed

– No data normalization, flexibility is a must,

1.4 XML & Databases

normalization, flat tables

– Order is irrelevant,

textual data supported but not primary goal

flexibility is a must, nesting is good

– Order may be very

important, textual data support a primary goal

(37)

• Data-Centric XML

– XML is used to store or transport regularly structured and fine grained data

– Data can be mapped to relational tables with some tricks

1.4 XML & Databases

tricks

– Is often designed to be pro- cessed by machines

Table Columns

Aggregated Columns? Foreign Keys?

Another table?

(38)

• Document-Centric XML

– Just loosely structured with a lot of unstructured text – Often intended to for

human consumption – Querying and proc-

1.4 XML & Databases

– Querying and processing quite difficult – Advantages of rela-

tional DBs don’t pay of

– Additional IR techni- ques advantageous

(39)

• XML documents thus can store all kinds of data

• Thus, is an XML document already a database?

– Generally speaking… yes. But a crappy one!

– For allowing effective XML use, we additionally need

1.4 XML & Databases

• Storage schemes for efficiently storing even huge documents

• Query Languages

• Schema Languages

• Support for data integrity and transactions (ACID)

• Support for data security

• Programming Interfaces

• … and all the other thing we know from real DBMS systems

(40)

• Many of these requirements can be fulfilled by specialized standards and technologies

– Storage:

• XML document on the file system

– Queries:

1.4 XML & Databases

– Queries:

• Simple queries with XPath

• Complex queries with XQuery

– Schemas:

• Simple schemas with DTD

• Complex schemas XML-Schema (XSD)

– Programming Interfaces:

• Provided by various implementations of SAX, DOM, STAX, …

(41)

• Still, those isolated technologies are not yet a real DBMS

• The topic of XML Databases deals with integrating them into a fully functional DBMS

1.4 XML & Databases

integrating them into a fully functional DBMS

• Two options

– Integrating XML support into RDMS systems

• Especially suited for data-centric XML

– Building native XML-DBMS systems

• Suited for data-centric and document centric XML

(42)

• What are XML supporting RDBMS?

– Maps XML data into relational tables

– Main problem: How to create an efficient and meaningful mapping?

• What are native XML databases?

– „Native“ is a marketing term

1.4 XML & Databases

– „Native“ is a marketing term – Common Agreement:

• Native XML DBs works with a logical model of the XML document (not directly with the data)

– i.e. nodes, attributes, types, tree structure, CDATA entries, …

• XML is the primary form of storage

• Are not limited to a particular storage model (could use a relational DB, an object DB, file system, etc)

– Main problem: How to query and store effieciently?

(43)

• Example (very simple):

1.2 XML & Databases

id airline origin destination 1 ABC Air Dallas Fort Worth

Flights Relational Mapping

id departure arrival flight_ref

1 09:15 09:16 1

2 11:15 11:16 1

3 13:15 13:16 1

Flight

Native Mapping

id parent name value

1 null Flights null

2 1 Airline ABC Air

3 1 Origin Dallas

4 1 Destination Fort Worth

5 1 Flight Null

6 4 Departure 09:15

• RDBMS with XML support

1.4 XML & Databases

• Native XML-DBMS systems

(45)

1.1 Motivation

1.2 Relational Databases – Repetition 1.3 Why use XML?

1.4 XML & Databases

Outline

1.4 XML & Databases 1.5 XML Fundamentals

1.6 Organisational matters 1.7 Overview

1.8 References

(46)

1.5 XML Fundamentals

• Reasons for the XML success:

– XML is a general data representation format – XML is human readable

– XML is machine readable

– XML is internationalized (UNICODE) – XML is internationalized (UNICODE) – XML is platform independent

– XML is vendor independent

– XML is endorsed by the World Wide Web Consortium – XML is not a new technology

– XML is not only a data representation format, it’s a full infrastructure of technologies

[Fisch05] XML Databases – Silke Eckstein – Institut für Informationssysteme – TU Braunschweig 46

(47)

• W3C: World Wide Web Consortium

– Established in 1994

– Initiator: Tim Berners-Lee

– Over 400 member organizations from more

1.5 XML Fundamentals

Over 400 member organizations from more than 40 countries

– Mission:

• " To lead the World Wide Web to its full potential by developing protocols and guidelines that

ensure long-term growth for the Web."

(48)

1.5 XML Fundamentals

• W3C Process

Source: Mario Jeckle, www.jeckle.de

(49)

1.5 XML Fundamentals

• Structure of XML documents

– XML prolog

– Document Type Definition (DTD) – Document Instance

<Bücher>

– Have to be well-formed

<Bücher>

<Buch>

<Autor id="1234567890">Rainer Eckstein</Autor>

<Autor id="1234568723">Silke Eckstein</Autor>

<Titel>XML und Datenmodellierung</Titel>

<Untertitel>XML-Schema ...</Untertitel>

</Buch>

</Bücher>

(50)

1.5 XML Fundamentals

• Document Type Definition

<!DOCTYPE Bücher [

<!ELEMENT Bücher (Buch)* >

<!ELEMENT Buch (Autor+, Titel, Untertitel?, Verlag >

<!ELEMENT Autor (#PCDATA) >

<!ATTLIST Autor

– Validity

<!ATTLIST Autor

id ID #REQUIRED email CDATA #IMPLIED

>

<!ELEMENT Titel (#PCDATA) >

<!ELEMENT Untertitel (#PCDATA) >

<!ELEMENT Verlag (#PCDATA)>

]>

(51)

1.5 XML Fundamentals

• XML Schema

<xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema">

<xsd:element name="Bücher">

<xsd:complexType>

<xsd:sequence>

<xsd:element name="Buch" maxOccurs="unbounded" minOccurs="0" >

<xsd:complexType>

<xsd:sequence>

<xsd:element name="Autor" maxOccurs="unbounded" >

<xsd:complexType>

<xsd:simpleContent>

<xsd:extension base="string">

<xsd:attribute name="id" type="ID"/>

<xsd:attribute name="email" type="string"/>

</xsd:extension>

</xsd:simpleContent>

</xsd:complexType>

</xsd:element>

...

</xsd:schema>

(52)

• Misunderstanding about XML

– “Data is self-describing.”

– Tags don’t hold semantics, they only hold the

1.5 XML Fundamentals

structure of the information

– The interpretation of the tags is in the application that handles the data, not in the tags themselves.

(53)

• XML as a family of technologies

– XML Information Set – XML Schema

– XML Query

– The Extensible Stylesheet Transformation Language (XSLT) – XLink, XPointer

1.5 XML Fundamentals

– XLink, XPointer – XML Forms

– XML Protocol – XML Encryption – XML Signature – Others

– … almost all the pieces needed for a good XML-based information hub

(54)

1.5 XML Fundamentals

XML Databases – Silke Eckstein – Institut für Informationssysteme – TU Braunschweig 54 Source: Mario Jeckle, www.jeckle.de

(55)

• Overview of XML Technologies

– W3C Standards

• Data: XML, Namespaces, Infoset, Schema

• Communication: SOAP, Encryption, WSDL, UDDI

• Processing: Xpath, XSLT, Xquery, Xupdate, Xquery Text

1.5 XML Fundamentals

• Processing: Xpath, XSLT, Xquery, Xupdate, Xquery Text

• Integration: RDF, OWL

– Other Standards

• Vertical domains: RosettaNet, ebXML, SBML, GML

• Workflow: BPEL

• Interfaces: DOM, SAX, JAXP, SQL/XML

(56)

1.1 Motivation

1.2 Relational Databases – Repetition 1.3 Why use XML?

1.4 XML & Databases

Outline

1.4 XML & Databases 1.5 XML Fundamentals

1.6 Organisational matters 1.7 Overview

1.8 References

(57)

• Who is who ?

– Silke Eckstein

• (Lecture, exams)

– Andreas Kupfer

1.6 Organisational matters

• (Tutorial)

– Regine Dalkıran

• (Office)

– Wolf-Tilo Balke

• (Head)

• In case of questions, don't hesitate to ask us.

(58)

• Lectures:

– Monday, 9:45 – 11:15, (IZ 131, lecture) – Monday, 11:30 – 12: 15, (IZ 131, tutorial)

• Office hours:

1.6 Organisational matters

• Office hours:

– Silke Eckstein: Tuesday, 12:30 – 13:30, IZ 232 – Andreas Kupfer: Friday, 10:30 – 11:30, IZ 213

• Course homepage:

– http://infbsdb1.idb.cs.tu-bs.de/eckstein/xmldatabases – lecture notes, links, latest news etc.

(59)

• Assignments:

– Presentations as well as programming – Details will be announced

• Credits: 4

1.6 Organisational matters

• Credits: 4

• Exams: Oral

– Master students: agree on certain week in Feb./Mar.

– Diploma students: on appointment

Please contact R. Dalkiran (regine.dalkiran at tu-braunschweig.de) for an exam appointment.

(60)

1. Introduction 2. XML Basics

3. Schema definition

4. XML query languages I 5. Mapping relational data

8. XML query languages II 9. XML storage I

10. XML storage - index 11. XML storage - native

1.7 Overview

5. Mapping relational data to XML

6. SQL/XML

7. XML processing

11. XML storage - native 12. Updates / Transactions 13. Systems

14. XML Benchmarks

(61)

• http://www.w3.org/ [W3C]

• XML in a Nutshell [HM04]

– Harold & Means

– O'Reilly, 2004, ISBN 0596007647

1.8 References

– O'Reilly, 2004, ISBN 0596007647

• Beginning XML Databases [Pow07]

– Gavin Powell

– Wiley & Sons, 2007, ISBN 0471791202

• XML und Datenbanken [Sch02]

– Harald Schöning

– Hanser, 2002, ISBN 3446220089

(62)

• XQuery: Grundlagen und fortgeschrittene Methoden [LS04]

– Lehner & Schöning

– Dpunkt-Verlag, 2004, ISBN 3898642666

• XML & Datenbanken. Konzepte, Sprachen und

1.8 References

• XML & Datenbanken. Konzepte, Sprachen und Systeme [KM02]

– Klettke & Meyer

– Dpunkt-Verlag, 2002, ISBN 3898641481

• Peter Fischer, "XML und Datenbanken", Lecture, ETH Zürich, WS 05/06 [Fisch05]

(63)

• Fundamentals of Database Systems [EN06]

– Elmasri & Navathe

– Addison Wesley, 2006, ISBN 032141506X

1.8 References

(64)