Flat fileFlat file

(1)

XML Databases

1. Introduction, 28.10.09

Silke Eckstein Andreas Kupfer

Institut für Informationssysteme

Technische Universität Braunschweig http://www.ifis.cs.tu-bs.de

(2)

1.1 Motivation

1.2 Relational Databases – Repetition 1.3 Why use XML?

1.4 XML & Databases

1. Introduction

1.4 XML & Databases 1.5 Overview

1.6 XML Fundamentals

1.7 Organisational matters 1.8 References

(3)

"If I invent another programming language, its name will contain the letter X“

1.1 Motivation

its name will contain the letter X“

(N. Wirth, Software Pioniere Konferenz, Bonn 2001)

XML Databases – Silke Eckstein – Institut für Informationssysteme – TU Braunschweig 3

(4)

• Within the last 10 years XML has become the

de facto standard for data exchange over the web

– Examples:

• The latest office documents

• SVG graphics files

1.1 Motivation

• SVG graphics files

• Lots of conguration files

• Some WebCMSs store page contents in XML format

• Mpeg7 is a standard for describing media meta data in XML format

• . . .

– In order to see examples of XML-structured documents, browse through your computer's file system and check for

(5)

• Why is XML relevant from DB perspective?

– XML is becoming the data "format"

• Amount of XML is ever increasing

• DBMS are good at handling GBs and TBs of data

– Accepted model for semi-structured data

1.1 Motivation

– Accepted model for semi-structured data

• Overcome limitations of structured data

• Extend usefulness of DBMS

– DB technology is not limited to DBMS

• Apps servers, application integration

XML Databases – Silke Eckstein – Institut für Informationssysteme – TU Braunschweig 5 [Fisch05]

(6)

Aim of this lecture

Give answers to the following questions:

• How to query XML data?

1.1 Motivation

• How to query XML data?

• How to produce XML out of relational data?

• How to store XML data in a database system?

(7)

1.1 Motivation

1. Introduction

7 XML Databases – Silke Eckstein – Institut für Informationssysteme – TU Braunschweig

(8)

What is a Database?

• A database (DB) is a collection of related data

– Represents some aspects of the real world

• Universe of Discourse (UoD)

1.2 Relational Databases

Universe of Discourse (UoD)

– Data is logically coherent

– Is provided for an intended group of users and applications

(9)

What is a Database Management System?

• A database management system (DBMS) is a

collection of programs to maintain a database, i.e.

for

1.2 Relational Databases

for

– Definition of Data and Structure – Physical Construction

– Manipulation

– Sharing/Protecting

– Persistence/Recovery

XML Databases – Silke Eckstein – Institut für Informationssysteme – TU Braunschweig 9 [EN06, 1.1]

(10)

Why not use the File System?

• File management systems are physical interfaces

1.2 Relational Databases

F i l Account

Data

App 1

Customer Letters l

e S y s t Data

Customer Data

1

App 2

Money Transfer

(11)

• Databases are logical interfaces

– Controlled redundancy

– Data consistency & integrity constraints – Integration of data

1.2 Relational Databases

Integration of data

– Effective and secure data sharing – Backup and recovery

• However…

– More complex

– More expensive data access

(12)

• Databases aim at efficient manipulation of data

– Physical tuning allows for good data allocation – Indexes speed up search and access

– Query plans are optimized for improved performance

1.2 Relational Databases

Query plans are optimized for improved performance

• Example: Simple Index ^AccNo ^type ^balance

1278945 saving € 312.10

2437954 saving € 1324.82

4543032 checking € -43.03

5539783 saving € 12.54

7809849 checking € 7643.89

8942214 checking € -345.17

AccNo 1278945 5539783 9134354

Index File

Data File

(13)

• Isolation between applications and data:

Schema is changed and table-space moved without an application noticing

1.2 Relational Databases

Application

SELEC T AccNo FROM account WHERE balance>0

DBMS

AccNo balance

1278945 € 312.10 2437954 € 1324.82 4543032 € -43.03 5539783 € 12.54

Disk 1 Disk 2

(14)

1.2 Relational Databases

Application

SELEC T AccNo FROM account WHERE balance>0

• Isolation between applications and data:

Schema is changed and table-space moved without an application noticing

DBMS

AccNo balance

1278945 € 312.10 2437954 € 1324.82 4543032 € -43.03

AccNo type balance

1278945 saving € 312.10 2437954 saving € 1324.82

Disk 1 Disk 2

(15)

1.1 Motivation

1. Introduction

(16)

• Flat files

• HTML

• Little layout information

• Suitable for presentation only to a limited extent

• Can be parsed, but cumbersome

• Only layout information

1.3 Why use XML?

• HTML • Only layout information

• Good for presentation

• Automatic processing difficult

• Just as generation of other presentation formats

(17)

• Bioinformatics example:

• Presentation and processing of database query results

– Flat file

1.3 Why use XML?

– Flat file – Web page – HTML text – XML text

– Search in TRANSPATH database for molecule "TLR4"

(18)

Key

Originator Molecule name Species

Links to other DBs

F la t f ile

Links to other DBs Gene Ontology

references

Reactions the molecule participates in

Publications

F la t f ile

(19)

Key

Links to other DBs

W e b p a g e

19

references

Reactions the molecule participates in

Publications

W e b p a g e

XML Databases – Silke Eckstein – Institut für Informationssysteme – TU Braunschweig

(20)

Key

Originator

Molecule name

H T M L

Species

H T M L

(21)

Key

Links to other DBs

X M L

references

X M L

(22)

• What is XML?

– The Extensible Markup Language (XML) is the

universal format for structured documents and data on the Web.

– Base specifications:

1.3 Why use XML?

– Base specifications:

• XML 1.0 (5^th Ed.), W3C Recommendation Nov '08

• XML 1.1 (2^nd Ed.), W3C Recommendation Aug '06

• Namespaces in XML 1.1, W3C Recommendation Aug '06

(23)

• What is XML now then?

– XML is semi-structured text

– XML is a tag-based markup-language (like HTML)

• eXtensible Markup Language

– XML was designed to exchange data

1.3 Why use XML?

– XML was designed to exchange data – XML tags are not predefined

• Tags are defined in a separate schema

– XML is designed to be self-descriptive – XML is a W3C Recommendation

– XML became highly popular due to its simplicity and flexibility

(24)

• XML Data Example

1.3 Why use XML?

<Buch>

<Autor id="1234567890">Rainer Eckstein</Autor>

<Autor id="1234568723">Silke Eckstein</Autor>

<Titel>XML und Datenmodellierung</Titel>

<Untertitel>XML-Schema ...</Untertitel>

<Verlag id="3-89864">dpunkt.Verlag</Verlag>

− Syntax, no abstract model

− Documents, elements and attributes

− Tree-based, nested, hierarchically organized structure

</Buch>

(25)

1.1 Motivation

1. Introduction

(26)

1.4 XML & Databases

• Database world

– 1970 relational databases

– 1990 nested relational model and object

• Documents world

– 1974 SGML (Structured Generalized Markup

Language)

– 1990 HTML (Hypertext model and object

oriented databases

– 1995 semi-structured databases

– 1990 HTML (Hypertext Markup Language)

– 1992 URL (Universal Resource Locator) Data + documents = information

(27)

• Information systems have different degrees of data structure rigidness

– Structured, e.g., relational databases

• Structure explicitly specified in schema

• Every tuple in a table has the same attributes and domains

1.4 XML & Databases

• Every tuple in a table has the same attributes and domains

• Queries can take advantage of structure

– Unstructured, e.g., information retrieval systems

• Often just full text with no or only limited structure information

• Properties of data usually unknown

• Queries difficult to evaluate

(28)

• But there is also something in between

– Semi-structured, e.g., XML

• Structure of data follows a template, but still allows for a degree of flexibility

• Data instances following the same schema

1.4 XML & Databases

• Data instances following the same schema may have a different structure

• Often, complex relationships between data are allowed (associations, inheritance, sub-classing, aggregation, etc.)

• Queries often involve those relationships

(29)

• Relational data

– Killer Application:

Banking

– Invented as a

mathematically clean abstract data model

• XML

– 1st killer application:

Publishing industry

– Invented as a syntax for data, only later an abstract data model

1.4 XML & Databases

abstract data model

– Philosophy: schema first, then data

data model

– Philosophy: data and schemas should not be correlated, data can exist with or without schema, or with multiple schemas

(30)

• Relational data

– Never had a standard syntax for data

– Strict rules for data normalization, flat

• XML

– Standard syntax existed

– No data normalization, flexibility is a must,

1.4 XML & Databases

normalization, flat tables

– Order is irrelevant,

textual data supported but not primary goal

flexibility is a must, nesting is good

– Order may be very

important, textual data support a primary goal

(31)

• Data-Centric XML

– XML is used to store or transport regularly structured and fine grained data

– Data can be mapped to relational tables with some tricks

1.4 XML & Databases

tricks

– Is often designed to be pro- cessed by machines

Table Columns

Aggregated Columns? Foreign Keys?

Another table?

(32)

• Document-Centric XML

– Just loosely structured with a lot of unstructured text – Often intended to for

human consumption – Querying and proc-

1.4 XML & Databases

– Querying and processing quite difficult – Advantages of rela-

tional DBs don’t pay of

– Additional IR techni-

(33)

• XML documents thus can store all kinds of data

• Thus, is an XML document already a database?

– Generally speaking… yes. But a crappy one!

– For allowing effective XML use, we additionally need

1.4 XML & Databases

• Storage schemes for efficiently storing even huge documents

• Query Languages

• Schema Languages

• Support for data integrity and transactions (ACID)

• Support for data security

• Programming Interfaces

• … and all the other thing we know from real DBMS systems

(34)

• Many of these requirements can be fulfilled by specialized standards and technologies

– Storage:

• XML document on the file system

– Queries:

1.4 XML & Databases

– Queries:

• Simple queries with XPath

• Complex queries with XQuery

– Schemas:

• Simple schemas with DTD

• Complex schemas XML-Schema (XSD)

– Programming Interfaces:

(35)

• Still, those isolated technologies are not yet a real DBMS

• The topic of XML Databases deals with integrating them into a fully functional DBMS

1.4 XML & Databases

integrating them into a fully functional DBMS

• Two options

– Integrating XML support into RDMS systems

• Especially suited for data-centric XML

– Building native XML-DBMS systems

• Suited for data-centric and document centric XML

(36)

• What are XML supporting RDBMS?

– Maps XML data into relational tables

– Main problem: How to create an efficient and meaningful mapping?

• What are native XML databases?

– „Native“ is a marketing term

1.4 XML & Databases

– „Native“ is a marketing term – Common Agreement:

• Native XML DBs works with a logical model of the XML document (not directly with the data)

– i.e. nodes, attributes, types, tree structure, CDATA entries, …

• XML is the primary form of storage

• Are not limited to a particular storage model (could use a relational DB, an object DB, file system, etc)

(37)

• Example (very simple):

1.4 XML & Databases

id airline origin destination 1 ABC Air Dallas Fort Worth

Flights Relational Mapping

id departure arrival flight_ref

1 09:15 09:16 1

2 11:15 11:16 1

3 13:15 13:16 1

Flight

Native Mapping

id parent name value

1 null Flights null

2 1 Airline ABC Air

3 1 Origin Dallas

4 1 Destination Fort Worth

5 1 Flight Null

6 4 Departure 09:15

1.4 XML & Databases

• Native XML-DBMS systems

(39)

• SQL/XML

– Datatype XML with

1.4 XML & Databases

<City>

<Name>

Braunschweig

</Name>

<Zip>38100</Zip>

<Zip>38106</Zip>

<State>

Niedersachsen

</State>

</City>

<City>

Storing XML documents inside

the database as values of type XML

Generating XML documents using SQL/XML functions

– Datatype XML with belonging functions – Mapping between

SQL and XML

– Embedding XQuery in SQL

XML Databases – Silke Eckstein – Institut für Informationssysteme – TU Braunschweig 39 [Tür08]

SQL database XML datatype SQL XQuery

Mapping between SQl and XML

(40)

• SQL/XML defines the following:

– The XML data type as a regular SQL type.

– Functions converting XML type values to and from non-XML data types.

1.4 XML & Databases

– The function XMLVALIDATE for XML Schema validation.

– XML publishing functions.

– Functions to embed XPath and XQuery in SQL statements.

(41)

1.1 Motivation

1. Introduction

(42)

Introduction and Basics 1. Introduction

2. XML Basics

3. Schema Definition 4. XML Processing Querying XML

Producing XML

9. Mapping relational data to XML

Storing XML 10. XML storage

1.5 Overview

Querying XML

5. XPath & SQL/XML Queries

6. XQuery Data Model 7. XQuery

XML Updates

10. XML storage

11. Relational XML storage 12. Storage Optimization Systems

13. Technology Overview

(43)

• Example: DTD and valid XML encoding academic titles

1.5 Introduction & Basics

<?xml version="1.1"?>

<!DOCTYPE academic [

<!ELEMENT academic (Prof?,

(Dr, (rernat|emer|phil)*)?, Firstname, Middlename*,

<academic>

<Prof/><Dr/><emer/>

<Firstname>

Don

XML Databases – Silke Eckstein – Institut für Informationssysteme – TU Braunschweig 43 [Scholl07]

Firstname, Middlename*, Lastname) >

<!ELEMENT Prof EMPTY >

<!ELEMENT Dr EMPTY >

<!ELEMENT rernat EMPTY >

<!ELEMENT emer EMPTY >

<!ELEMENT phil EMPTY >

<!ELEMENT Firstname (#PCDATA) >

<!ELEMENT Middlename (#PCDATA) >

<!ELEMENT Lastname (#PCDATA) >

]>

Don

</Firstname>

<Middlename>

E

</Middlename>

<Lastname>

Knuth

</Lastname>

</academic>

(44)

1.5 Introduction & Basics

Source: Mario Jeckle, www.jeckle.de

(45)

• XML Processors

– Virtually all XML applications operate on the logical tree view which is provided to them through an XML Processor (i.e., the XML parser):

1.5 Introduction & Basics

– How is the XML processor supposed to

communicate the XML tree structure to the application . . . ?

XML Databases – Silke Eckstein – Institut für Informationssysteme – TU Braunschweig 45 [Gru08]

? x m l

< ...

XML Processor

XML Application

< ...

(46)

• XPath

– XPath defines a family of 12 axes allowing for flexible navigation within the node hierarchy of an XML tree.

– XPath axes semantics

• ○○○○ marks the context node

1.5 Querying XML – XPath

• ○○○○ marks the context node

• @ marks attribute nodes,

• ● represents any other node kind (inner ● nodes are

element nodes).

@

@ @@

@

(47)

• XMLELEMENT referencing the database

– Can be used directly from an SQL statement

1.5 Querying XML – SQL/XML

SELECT XMLELEMENT( NAME "City",

XMLCOMMENT ( "Example 3" ),

XMLATTRIBUTES( "State", "Zip" AS "PLZ" ),

"City" )

• Creates

XML Databases – Silke Eckstein – Institut für Informationssysteme – TU Braunschweig 47 [Kud07]

"City" ) FROM Cities WHERE …;

<City STATE="Niedersachsen" PLZ="38100">

<!– Example 3 -->

Braunschweig

</City>

(48)

• History of XQuery

1.5 Querying XML – XQuery

(49)

• XQuery Update Facility: New XQuery expressions

1.5 XML Updates

XQuery expressions

ExprSingle ::= FLWORExpr

| QuantifiedExpr

| TypeswitchExpr

XML Databases – Silke Eckstein – Institut für Informationssysteme – TU Braunschweig 49 [Scholl07]

| TypeswitchExpr

| IfExpr

| InsertExpr

| DeleteExpr

| RenameExpr

| ReplaceExpr

| TransformExpr

| OrExpr

(50)

• Why map relational database contents to XML?

– Interoperability

• We may want to use (parts of) our RDB contents in many

different application contexts (XML as data interchange format)

– Reconstruction

1.5 Producing XML

– Reconstruction

• We might have stored (parts of) our XML documents in an RDBMS in the first place (RDBMS as XML store)

– Dynamic XML contents

• We may use RDBMS queries to retrieve dynamic XML contents (cf. dynamic Web sites)

Wrapping

(51)

• Storage approaches for XML documents

– Text-based

• Storage as character data

– Model-based

1.5 Storing XML

• Generic storage of the graph structure

• Storage of the DOM

– Schema-based

• Mapping to (object-)relational databases

– Deriving the database schema from the XML structure – Using user defined mapping procedures

(52)

• XPath Accelerator encoding

– XML fragment f and its skeleton tree

1.5 Storing XML

– Pre/post encoding of f : table accel

(53)

• "pureXML and relational hybrid database"

1.5 Systems

XML Databases – Silke Eckstein – Institut für Informationssysteme – TU Braunschweig 53 [IBM06a]

(54)

1.1 Motivation

1. Introduction

(55)

1.5 XML Fundamentals

• Reasons for the XML success:

– XML is a general data representation format – XML is human readable

– XML is machine readable

– XML is internationalized (UNICODE) – XML is internationalized (UNICODE) – XML is platform independent

– XML is vendor independent

– XML is endorsed by the World Wide Web Consortium – XML is not a new technology

– XML is not only a data representation format, it’s a full infrastructure of technologies

[Fisch05] XML Databases – Silke Eckstein – Institut für Informationssysteme – TU Braunschweig 55

(56)

• W3C: World Wide Web Consortium

– Established in 1994

– Initiator: Tim Berners-Lee

– Over 400 member organizations from more

1.5 XML Fundamentals

Over 400 member organizations from more than 40 countries

– Mission:

• " To lead the World Wide Web to its full potential by developing protocols and guidelines that

ensure long-term growth for the Web."

(57)

1.5 XML Fundamentals

• W3C Process

Source: Mario Jeckle, www.jeckle.de

(58)

1.5 XML Fundamentals

• Structure of XML documents

– XML prolog

– Document Type Definition (DTD) – Document Instance

<Bücher>

<Buch>

<Autor id="1234567890">Rainer Eckstein</Autor>

<Autor id="1234568723">Silke Eckstein</Autor>

<Titel>XML und Datenmodellierung</Titel>

<Untertitel>XML-Schema ...</Untertitel>

</Buch>

</Bücher>

(59)

1.5 XML Fundamentals

• Document Type Definition

<!DOCTYPE Bücher [

<!ELEMENT Bücher (Buch)* >

<!ELEMENT Buch (Autor+, Titel, Untertitel?, Verlag >

<!ELEMENT Autor (#PCDATA) >

<!ATTLIST Autor

– Validity

<!ATTLIST Autor

id ID #REQUIRED email CDATA #IMPLIED

>

<!ELEMENT Titel (#PCDATA) >

<!ELEMENT Untertitel (#PCDATA) >

<!ELEMENT Verlag (#PCDATA)>

]>

(60)

1.5 XML Fundamentals

• XML Schema

<xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema">

<xsd:element name="Bücher">

<xsd:complexType>

<xsd:sequence>

<xsd:element name="Buch" maxOccurs="unbounded" minOccurs="0" >

<xsd:complexType>

<xsd:sequence>

<xsd:element name="Autor" maxOccurs="unbounded" >

<xsd:complexType>

<xsd:simpleContent>

<xsd:extension base="string">

<xsd:attribute name="id" type="ID"/>

<xsd:attribute name="email" type="string"/>

</xsd:extension>

</xsd:simpleContent>

</xsd:complexType>

(61)

• Misunderstanding about XML

– “Data is self-describing.”

– Tags don’t hold semantics, they only hold the

1.5 XML Fundamentals

structure of the information

– The interpretation of the tags is in the application that handles the data, not in the tags themselves.

(62)

1.5 XML Fundamentals

(63)

• Overview of XML Technologies

– W3C Standards

• Data: XML, Namespaces, Infoset, Schema

• Communication: SOAP, Encryption, WSDL, UDDI

• Processing: Xpath, XSLT, Xquery, Xupdate, Xquery Text

1.5 XML Fundamentals

• Processing: Xpath, XSLT, Xquery, Xupdate, Xquery Text

• Integration: RDF, OWL

– Other Standards

• Vertical domains: RosettaNet, ebXML, SBML, GML

• Workflow: BPEL

• Interfaces: DOM, SAX, JAXP, SQL/XML

(64)

1.1 Motivation

1. Introduction

(65)

• Who is who ?

– Silke Eckstein

• (Lecture, exams)

– Andreas Kupfer

1.6 Organisational matters

• (Tutorial)

– Regine Dalkıran

• (Office)

– Wolf-Tilo Balke

• (Head)

• In case of questions, don't hesitate to ask us.

(66)

• Lectures:

– Wednesday, 9:45 – 11:15, (IZ 131, lecture) – Wednesday, 11:30 – 12: 15, (IZ 131, tutorial)

• Office hours:

1.6 Organisational matters

• Office hours:

– Silke Eckstein: Tuesday, 12:30 – 13:30, IZ 232 – Andreas Kupfer: Friday, 10:30 – 11:30, IZ 213

• Course homepage:

– http://www.ifis.cs.tu-bs.de/teaching/ws-0910/xmldb

(67)

• Assignments:

– Presentations as well as programming – Details will be announced

• Credits: 4

1.6 Organisational matters

• Credits: 4

• Exams: Oral

– Master students: First week in March – Diploma students: on appointment

Please contact R. Dalkiran (regine.dalkiran at tu-braunschweig.de) for an exam appointment.

(68)

• http://www.w3.org/ [W3C]

• XML in a Nutshell [HM04]

– Harold & Means

– O'Reilly, 2004, ISBN 0596007647

1.8 References

– O'Reilly, 2004, ISBN 0596007647

• Beginning XML Databases [Pow07]

– Gavin Powell

– Wiley & Sons, 2007, ISBN 0471791202

• XML und Datenbanken [Sch02]

– Harald Schöning

(69)

• XQuery: Grundlagen und fortgeschrittene Methoden [LS04]

– Lehner & Schöning

– Dpunkt-Verlag, 2004, ISBN 3898642666

• XML & Datenbanken. Konzepte, Sprachen und

1.8 References

• XML & Datenbanken. Konzepte, Sprachen und Systeme [KM02]

– Klettke & Meyer

– Dpunkt-Verlag, 2002, ISBN 3898641481

• Peter Fischer, "XML und Datenbanken", Lecture, ETH Zürich, WS 05/06 [Fisch05]

(70)

• Fundamentals of Database Systems [EN06]

– Elmasri & Navathe

– Addison Wesley, 2006, ISBN 032141506X

1.8 References

(71)

• Now, or ...

• Room: IZ 232

• Office our: Tuesday, 12:30 – 13:30 Uhr

Questions, Ideas, Comments

• Office our: Tuesday, 12:30 – 13:30 Uhr or on appointment

• Email: eckstein@ifis.cs.tu-bs.de