1.1 Motivation 1.1 Motivation 1.1 Motivation 1.1 Motivation XML Databases 1. Introduction

(1)

XML Databases

1. Introduction, 28.10.09

Silke Eckstein Andreas Kupfer

Institut für Informationssysteme Technische Universität Braunschweig http://www.ifis.cs.tu-bs.de

1.1 Motivation

1.2 Relational Databases – Repetition 1.3 Why use XML?

1.4 XML & Databases 1.5 Overview

1.6 XML Fundamentals 1.7 Organisational matters 1.8 References

2

1. Introduction

XML Databases – Silke Eckstein – Institut für Informationssysteme – TU Braunschweig

"If I invent another programming language, its name will contain the letter X“

(N. Wirth, Software Pioniere Konferenz, Bonn 2001)

XML Databases – Silke Eckstein – Institut für Informationssysteme – TU Braunschweig 3

1.1 Motivation

• Within the last 10 years XML has become the de facto standard for data exchange over the web

–Examples:

•The latest office documents

•SVG graphics files

•Lots of conguration files

•Some WebCMSs store page contents in XML format

•Mpeg7 is a standard for describing media meta data in XML format

•. . .

–In order to see examples of XML-structured documents, browse through your computer's file system and check for file contents starting with "<?xml "!

1.1 Motivation

• Why is XML relevant from DB perspective?

–XML is becoming the data "format"

•Amount of XML is ever increasing

•DBMS are good at handling GBs and TBs of data –Accepted model for semi-structured data

•Overcome limitations of structured data

•Extend usefulness of DBMS

–DB technology is not limited to DBMS

•Apps servers, application integration

1.1 Motivation

Aim of this lecture

Give answers to the following questions:

• How to query XMLdata?

• How to produce XML out of relational data?

• How to store XML data in a database system?

1.1 Motivation

(2)

1.1 Motivation

7

1. Introduction

What is a Database?

• A database (DB) is a collection of related data –Represents some aspects of the real world

•Universe of Discourse (UoD) –Data is logically coherent

–Is provided for an intended group of usersand applications

XML Databases – Silke Eckstein – Institut für Informationssysteme – TU Braunschweig 8 [EN06, 1.1]

1.2 Relational Databases

What is a Database Management System?

• A database management system (DBMS) is a collection of programs to maintain a database, i.e.

for

–Definition of Data and Structure –Physical Construction

–Manipulation –Sharing/Protecting –Persistence/Recovery

1.2 Relational Databases

Why not use the File System?

• File management systems are physicalinterfaces

1.2 Relational Databases

F i l e S y s t e m Account

Data

Customer Data

Loans

App 1

App 2

Balance Sheets Customer Letters

Money Transfer

• Databases are logicalinterfaces –Controlled redundancy

–Data consistency & integrity constraints –Integration of data

–Effective and secure data sharing –Backup and recovery

• However…

–More complex

–More expensive data access

1.2 Relational Databases

• Databases aim at efficientmanipulation of data –Physical tuning allows for good data allocation –Indexes speed up search and access

–Query plans are optimized for improved performance

• Example: Simple Index

1.2 Relational Databases

AccNo type balance

1278945 saving € 312.10

2437954 saving € 1324.82

4543032 checking € -43.03

5539783 saving € 12.54

7809849 checking € 7643.89

8942214 checking € -345.17

9134354 saving € 2.22

9543252 saving € 524.89

AccNo 1278945 5539783 9134354 Index File

Data File

(3)

• Isolationbetween applications and data:

Schema is changed and table-space moved without an application noticing

1.2 Relational Databases

Application

DBMS

AccNo balance 1278945 € 312.10 2437954 € 1324.82 4543032 € -43.03 5539783 € 12.54

Disk 1 Disk 2

SELEC T AccNo FROM account WHERE balance>0

1.2 Relational Databases

Application

DBMS

AccNo balance 1278945 € 312.10 2437954 € 1324.82 4543032 € -43.03 5539783 € 12.54

AccNo type balance

1278945 saving € 312.10 2437954 saving € 1324.82 4543032 checking € -43.03 5539783 saving € 12.54

Disk 1 Disk 2

SELEC T AccNo FROM account WHERE balance>0

• Isolationbetween applications and data:

Schema is changed and table-space moved without an application noticing

1.1 Motivation

15

1. Introduction

• Flat files

• HTML

• Solution

• Little layout information

• Suitable for presentation only to a limited extent

• Can be parsed, but cumbersome

• Only layout information

• Good for presentation

• Automatic processing difficult

• Just as generation of other presentation formats

• Separation of layout and content

1.3 Why use XML?

• Bioinformatics example:

• Presentation and processing of database query results

–Flat file –Web page –HTML text –XML text

–Search in TRANSPATH database for molecule "TLR4"

1.3 Why use XML?

Key Originator Molecule name Species Links to other DBs Gene Ontology

references Reactions the molecule

participates in Publications

Flat file

(4)

19

references Reactions the molecule

participates in Publications

Web page

XML Databases – Silke Eckstein – Institut für Informationssysteme – TU Braunschweig XML Databases – Silke Eckstein – Institut für Informationssysteme – TU Braunschweig 20

Key Originator

Molecule name

Species

HTML

references

XML

• What is XML?

–The Extensible Markup Language (XML) is the universal format for structured documents and data on the Web.

–Base specifications:

•XML 1.0 (5^thEd.), W3C Recommendation Nov '08

•XML 1.1 (2^ndEd.), W3C Recommendation Aug '06

•Namespaces in XML 1.1, W3C Recommendation Aug '06

XML Databases – Silke Eckstein – Institut für Informationssysteme – TU Braunschweig 22 [Fisch05]

1.3 Why use XML?

• What is XML now then?

–XML is semi-structured text

–XML is a tag-based markup-language (like HTML)

•eXtensible Markup Language

–XML was designed to exchange data –XML tags are not predefined

•Tags are defined in a separate schema –XML is designed to be self-descriptive –XML is a W3C Recommendation

–XML became highly popular due to its simplicityand flexibility

1.3 Why use XML?

• XML Data Example

−Syntax, no abstract model

−Documents, elements and attributes

−Tree-based, nested, hierarchically organized structure

1.3 Why use XML?

<Buch>

<Autor id="1234567890">Rainer Eckstein</Autor>

<Autor id="1234568723">Silke Eckstein</Autor>

<Titel>XML und Datenmodellierung</Titel>

<Untertitel>XML-Schema ...</Untertitel>

<Verlag id="3-89864">dpunkt.Verlag</Verlag>

</Buch>

(5)

1.1 Motivation

25

1. Introduction

XML Databases – Silke Eckstein – Institut für Informationssysteme – TU Braunschweig [Fisch05] 26

1.4 XML & Databases

• Database world –1970 relational

databases

–1990 nested relational model and object oriented databases –1995 semi-structured

databases

• Documents world –1974 SGML (Structured

Generalized Markup Language)

–1990 HTML (Hypertext Markup Language) –1992 URL (Universal

Resource Locator) Data + documents = information 1996 XML (Extensible Markup Language)

URI (Universal Resource Identifier)

• Information systems have different degrees of data structure rigidness

–Structured, e.g., relational databases

•Structure explicitly specified in schema

•Every tuple in a table has the same attributes and domains

•Queries can take advantage of structure

–Unstructured, e.g., information retrieval systems

•Often just full text with no or only limited structure information

•Properties of data usually unknown

•Queries difficult to evaluate

1.4 XML & Databases

• But there is also something in between –Semi-structured,e.g., XML

•Structure of data follows a template, but still allows for a degree of flexibility

•Data instances following the same schema may have a different structure

•Often, complex relationships between data are allowed (associations, inheritance, sub-classing, aggregation, etc.)

•Queries often involve those relationships

1.4 XML & Databases

• Relational data

–Killer Application:

Banking –Invented as a

mathematically clean abstract data model –Philosophy: schema first,

then data

• XML

–1st killer application:

Publishing industry –Invented as a syntax for

data, only later an abstract data model

–Philosophy: data and schemas should not be correlated, data can exist with or without schema, or with multiple schemas

1.4 XML & Databases

• Relational data –Never had a standard

syntax for data –Strict rules for data

normalization, flat tables

–Order is irrelevant, textual data supported but not primary goal

• XML

–Standard syntax existed

–No data normalization, flexibility is a must, nesting is good –Order may be very

important, textual data support a primary goal

1.4 XML & Databases

(6)

• Data-Centric XML

–XML is used to store or transport regularly structured and fine grained data

–Data can be mapped to relational tables with some tricks

–Is often designed to be pro- cessed by machines

1.4 XML & Databases

Table Columns

Aggregated Columns? Foreign Keys?

Another table?

• Document-Centric XML

–Just loosely structured with a lot of unstructured text –Often intended to for

human consumption –Querying and proc-

essing quite difficult –Advantages of relational DBs don’t pay of

–Additional IR techni- ques advantageous

1.4 XML & Databases

• XML documents thus can store all kinds of data

• Thus, is an XML document already a database?

–Generally speaking… yes. But a crappy one!

–For allowing effective XML use, we additionally need

•Storage schemes for efficiently storing even huge documents

•Query Languages

•Schema Languages

•Support for data integrity and transactions (ACID)

•Support for data security

•Programming Interfaces

•… and all the other thing we know from real DBMS systems

1.4 XML & Databases

• Many of these requirements can be fulfilled by specialized standards and technologies

–Storage:

•XML document on the file system –Queries:

•Simple queries with XPath

•Complex queries with XQuery –Schemas:

•Simple schemas with DTD

•Complex schemas XML-Schema (XSD) –Programming Interfaces:

•Provided by various implementations of SAX, DOM, STAX, …

1.4 XML & Databases

• Still, those isolated technologies are not yet a real DBMS

• The topic of XML Databases deals with integrating them into a fully functional DBMS

• Two options

–Integrating XML support into RDMS systems

•Especially suited for data-centric XML –Building native XML-DBMSsystems

•Suited for data-centric and document centric XML

1.4 XML & Databases

• What are XML supporting RDBMS?

–Maps XML data into relational tables

–Main problem: How to create an efficient and meaningful mapping?

• What are nativeXML databases?

–„Native“ is a marketing term –Common Agreement:

•Native XML DBs works with a logical model of the XML document (not directly with the data)

–i.e. nodes, attributes, types, tree structure, CDATA entries, …

•XMLis the primaryform of storage

•Are not limited to a particular storage model (could use a relational DB, an object DB, file system, etc)

–Main problem: How to query and store effieciently?

1.4 XML & Databases

(7)

• Example (very simple):

1.4 XML & Databases

id airline origin destination 1 ABC Air Dallas Fort Worth

id departure arrival flight_ref

1 09:15 09:16 1

2 11:15 11:16 1

3 13:15 13:16 1

Flights

Flight

Relational Mapping

Native Mapping

id parent name value

1 null Flights null

2 1 Airline ABC Air

3 1 Origin Dallas

4 1 Destination Fort Worth

5 1 Flight Null

6 4 Departure 09:15

1.4 XML & Databases

• SQL/XML

–Datatype XML with belonging functions –Mapping between

SQL and XML

–Embedding XQuery in SQL

XML Databases – Silke Eckstein – Institut für Informationssysteme – TU Braunschweig 39 [Tür08]

1.4 XML & Databases

SQL database XML datatype SQL XQuery

<City>

<Name>

Braunschweig

</Name>

<Zip>38100</Zip>

<Zip>38106</Zip>

<State>

Niedersachsen

</State>

</City>

<City>

Storing XML documents inside

the database as values of type XML

Generating XML documents using SQL/XML functions

Mapping between SQl and XML

• SQL/XML defines the following:

–The XML data type as a regular SQL type.

–Functions converting XML type values to and from non-XML data types.

–The function XMLVALIDATE for XML Schema validation.

–XML publishing functions.

–Functions to embed XPath and XQuery in SQL statements.

1.4 XML & Databases

1.1 Motivation

1. Introduction

Introduction and Basics 1. Introduction

2. XML Basics 3. Schema Definition 4. XML Processing Querying XML 5. XPath & SQL/XML

Queries

6. XQuery Data Model 7. XQuery

XML Updates 8. XML Updates & XSLT

Producing XML 9. Mapping relational data

to XML Storing XML 10. XML storage

11. Relational XML storage 12. Storage Optimization Systems

13. Technology Overview

1.5 Overview

(8)

• Example: DTD and valid XML encoding academic titles

XML Databases – Silke Eckstein – Institut für Informationssysteme – TU Braunschweig 43 [Scholl07]

1.5 Introduction & Basics

<?xml version="1.1"?>

<!DOCTYPE academic [

<!ELEMENT academic (Prof?,

(Dr, (rernat|emer|phil)*)?, Firstname, Middlename*, Lastname) >

<!ELEMENT Prof EMPTY >

<!ELEMENT Dr EMPTY >

<!ELEMENT rernat EMPTY >

<!ELEMENT emer EMPTY >

<!ELEMENT phil EMPTY >

<!ELEMENT Firstname (#PCDATA) >

<!ELEMENT Middlename (#PCDATA) >

<!ELEMENT Lastname (#PCDATA) >

]>

<academic>

<Prof/><Dr/><emer/>

<Firstname>

Don

</Firstname>

<Middlename>

E

</Middlename>

<Lastname>

Knuth

</Lastname>

</academic>

1.5 Introduction & Basics

Source: Mario Jeckle, www.jeckle.de

• XML Processors

–Virtually all XML applications operate on the logical tree view which is provided to them through an XML Processor (i.e., the XML parser):

–How is the XML processor supposed to

communicate the XML tree structureto the application . . . ?

XML Databases – Silke Eckstein – Institut für Informationssysteme – TU Braunschweig 45 [Gru08]

1.5 Introduction & Basics

? x m l

< ...

XML Processor

XML Application

<...

• XPath

–XPath defines a family of 12 axes allowing for flexible navigation within the node hierarchy of an XML tree.

–XPath axes semantics

•○○○○marks the context node

•@ marks attribute nodes,

•●represents any other node kind (inner ●nodes are element nodes).

1.5 Querying XML – XPath

@

@ @@

@

• XMLELEMENTreferencing the database –Can be used directly from an SQL statement

•Creates

XML Databases – Silke Eckstein – Institut für Informationssysteme – TU Braunschweig 47 [Kud07]

1.5 Querying XML – SQL/XML

SELECT XMLELEMENT( NAME "City",

XMLCOMMENT ( "Example 3" ),

XMLATTRIBUTES( "State", "Zip" AS "PLZ" ),

"City" ) FROM Cities WHERE …;

<City STATE="Niedersachsen" PLZ="38100">

<!– Example 3 -->

Braunschweig

</City>

• History of XQuery

1.5 Querying XML – XQuery

(9)

• XQuery Update Facility: New XQuery expressions

1.5 XML Updates

XQuery expressions

ExprSingle ::= FLWORExpr

| QuantifiedExpr

| TypeswitchExpr

| IfExpr

| InsertExpr

| DeleteExpr

| RenameExpr

| ReplaceExpr

| TransformExpr

| OrExpr

• Why map relational database contents to XML?

–Interoperability

•We may want to use (parts of) our RDB contents in many different application contexts (XML as data interchange format) –Reconstruction

•We might have stored (parts of) our XML documents in an RDBMS in the first place (RDBMS as XML store) –Dynamic XML contents

•We may use RDBMS queries to retrieve dynamic XML contents (cf. dynamic Web sites)

–Wrapping

•Everybody likes XML …, so why don't we give it to them?

1.5 Producing XML

• Storage approaches for XML documents –Text-based

•Storage as character data –Model-based

•Generic storage of the graph structure

•Storage of the DOM –Schema-based

•Mapping to (object-)relational databases –Deriving the database schema from the XML structure –Using user defined mapping procedures

1.5 Storing XML

• XPath Accelerator encoding –XML fragment fand its skeleton tree

–Pre/postencoding of f: table accel

1.5 Storing XML

• "pureXML and relational hybrid database"

1.5 Systems

1.1 Motivation

1. Introduction

(10)

1.5 XML Fundamentals

• Reasons for the XML success:

–XML is a general data representation format –XML is human readable

–XML is machine readable

–XML is internationalized (UNICODE) –XML is platform independent –XML is vendor independent

–XML is endorsed by the World Wide Web Consortium –XML is not a new technology

–XML is not onlya data representation format, it’s a full infrastructure of technologies

55 [Fisch05] XML Databases – Silke Eckstein – Institut für Informationssysteme – TU Braunschweig

• W3C: World Wide Web Consortium –Established in 1994

–Initiator: Tim Berners-Lee

–Over 400 member organizations from more than 40 countries

–Mission:

•" To lead the World Wide Web to its full potential by developing protocols and guidelines that ensure long-term growth for the Web."

1.5 XML Fundamentals

57

1.5 XML Fundamentals

XML Databases – Silke Eckstein – Institut für Informationssysteme – TU Braunschweig Source: Mario Jeckle, www.jeckle.de

• W3C Process

1.5 XML Fundamentals

• Structure of XML documents

–XML prolog

–Document Type Definition (DTD) –Document Instance

–Have to be well-formed

58 XML Databases – Silke Eckstein – Institut für Informationssysteme – TU Braunschweig

<Bücher>

<Buch>

<Autor id="1234567890">Rainer Eckstein</Autor>

<Autor id="1234568723">Silke Eckstein</Autor>

<Titel>XML und Datenmodellierung</Titel>

<Untertitel>XML-Schema ...</Untertitel>

<Verlag id="3-89864">dpunkt.Verlag</Verlag>

</Buch>

</Bücher>

1.5 XML Fundamentals

• Document Type Definition

–Validity

<!DOCTYPE Bücher [

<!ELEMENT Bücher (Buch)* >

<!ELEMENT Buch (Autor+, Titel, Untertitel?, Verlag >

<!ELEMENT Autor (#PCDATA) >

<!ATTLIST Autor

id ID #REQUIRED email CDATA #IMPLIED

>

<!ELEMENT Titel (#PCDATA) >

<!ELEMENT Untertitel (#PCDATA) >

<!ELEMENT Verlag (#PCDATA)>

]>

1.5 XML Fundamentals

• XML Schema

<xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema">

<xsd:element name="Bücher">

<xsd:complexType>

<xsd:sequence>

<xsd:element name="Buch" maxOccurs="unbounded" minOccurs="0" >

<xsd:complexType>

<xsd:sequence>

<xsd:element name="Autor" maxOccurs="unbounded" >

<xsd:complexType>

<xsd:simpleContent>

<xsd:extension base="string">

<xsd:attribute name="id" type="ID"/>

<xsd:attribute name="email" type="string"/>

</xsd:extension>

</xsd:simpleContent>

</xsd:complexType>

</xsd:element>

...

</xsd:schema>

(11)

• Misunderstanding about XML –“Data is self-describing.”

–Tags don’t hold semantics, they only hold the structure of the information

–The interpretation of the tags is in the application that handles the data, not in the tags themselves.

1.5 XML Fundamentals

Source: Mario Jeckle, www.jeckle.de

• Overview of XML Technologies –W3C Standards

•Data: XML, Namespaces, Infoset, Schema

•Communication: SOAP, Encryption, WSDL, UDDI

•Processing: Xpath, XSLT, Xquery, Xupdate, Xquery Text

•Integration: RDF, OWL –Other Standards

•Vertical domains: RosettaNet, ebXML, SBML, GML

•Workflow: BPEL

•Interfaces: DOM, SAX, JAXP, SQL/XML

1.5 XML Fundamentals

1.1 Motivation

64

1. Introduction

• Who is who ? –Silke Eckstein

•(Lecture, exams) –Andreas Kupfer

•(Tutorial) –Regine Dalkıran

•(Office) –Wolf-Tilo Balke

•(Head)

• In case of questions, don't hesitate to ask us.

1.6 Organisational matters

• Lectures:

–Wednesday, 9:45 – 11:15, (IZ 131, lecture) –Wednesday, 11:30 – 12: 15, (IZ 131, tutorial)

• Office hours:

–Silke Eckstein: Tuesday, 12:30 – 13:30, IZ 232 –Andreas Kupfer: Friday, 10:30 – 11:30, IZ 213

• Course homepage:

–http://www.ifis.cs.tu-bs.de/teaching/ws-0910/xmldb –lecture notes, links, latest news etc.

1.6 Organisational matters

(12)

• Assignments:

–Presentations as well as programming –Details will be announced

• Credits: 4

• Exams: Oral

–Master students: First week in March –Diploma students: on appointment

Please contact R. Dalkiran (regine.dalkiran at tu-braunschweig.de) for an exam appointment.

67

1.6 Organisational matters

• http://www.w3.org/[W3C]

• XML in a Nutshell [HM04]

–Harold & Means

–O'Reilly, 2004, ISBN 0596007647

• Beginning XML Databases [Pow07]

–Gavin Powell

–Wiley & Sons, 2007, ISBN 0471791202

• XML und Datenbanken [Sch02]

–Harald Schöning

–Hanser, 2002, ISBN 3446220089

68

1.8 References

• XQuery: Grundlagen und fortgeschrittene Methoden [LS04]

–Lehner & Schöning

–Dpunkt-Verlag, 2004, ISBN 3898642666

• XML & Datenbanken. Konzepte, Sprachen und Systeme [KM02]

–Klettke & Meyer

–Dpunkt-Verlag, 2002, ISBN 3898641481

• Peter Fischer, "XML und Datenbanken", Lecture, ETH Zürich, WS 05/06 [Fisch05]

69

1.8 References

• Fundamentals of Database Systems [EN06]

–Elmasri & Navathe

–Addison Wesley, 2006, ISBN 032141506X

1.8 References

• Now, or ...

• Room: IZ 232

• Office our: Tuesday, 12:30 – 13:30 Uhr or on appointment

• Email: eckstein@ifis.cs.tu-bs.de

1.1 Motivation 1.1 Motivation 1.1 Motivation 1.1 Motivation XML Databases 1. Introduction

XML Databases

1. Introduction

1.1 Motivation

1.1 Motivation

1.1 Motivation

1.1 Motivation

1. Introduction

1.2 Relational Databases

1.2 Relational Databases

1.2 Relational Databases

1.2 Relational Databases

1.2 Relational Databases

1.2 Relational Databases

1.2 Relational Databases

1. Introduction

1.3 Why use XML?

1.3 Why use XML?

1.3 Why use XML?

1.3 Why use XML?

1.3 Why use XML?

1. Introduction

1.4 XML & Databases

1.4 XML & Databases

1.4 XML & Databases

1.4 XML & Databases

1.4 XML & Databases

1.4 XML & Databases

1.4 XML & Databases

1.4 XML & Databases

1.4 XML & Databases

1.4 XML & Databases

1.4 XML & Databases

1.4 XML & Databases

1.4 XML & Databases

1.4 XML & Databases

1.4 XML & Databases

1. Introduction

1.5 Overview

1.5 Introduction & Basics

1.5 Introduction & Basics

1.5 Introduction & Basics

1.5 Querying XML – XPath

1.5 Querying XML – XQuery

1.5 XML Updates

1.5 Producing XML

1.5 Storing XML

1.5 Storing XML

1.5 Systems

1. Introduction

1.5 XML Fundamentals

1.5 XML Fundamentals

1.5 XML Fundamentals

1.5 XML Fundamentals

1.5 XML Fundamentals

1.5 XML Fundamentals

1.5 XML Fundamentals

1.5 XML Fundamentals

1.5 XML Fundamentals

1. Introduction

1.6 Organisational matters

1.6 Organisational matters

1.6 Organisational matters

1.8 References

1.8 References

1.8 References

Questions, Ideas, Comments