Multimedia Databases

(1)

Multimedia Databases

Wolf-Tilo Balke

Institut für Informationssysteme

Technische Universität Braunschweig http://www.ifis.cs.tu-bs.de

(2)

• Texture-Based Image Retrieval

– Low Level Features

• Tamura Measure, Random Field Model

– High-Level Features

• Fourier-Transform, Wavelets

Multimedia Databases– Wolf-Tilo Balke – Institut für Informationssysteme – TU Braunschweig 2

Previous Lecture

(3)

4 Multiresolution Analysis and Shape-based Features

4.1 Multiresolution Analysis 4.2 Shape-based Features

- Thresholding - Edge detection - Morphological

Operators

4 Shape-based Features

(4)

• In the case of images with many pixels (high

resolution) wavelet transforms provide high- dimensional equation systems

• Calculation of long feature vectors by solving linear equations?

– Far too expensive!

Multimedia Databases – Wolf-Tilo Balke – Institut für Informationssysteme – TU Braunschweig 4

4.1 Multiresolution Analysis

(5)

• The wavelet transform of an image can be computed using fast wavelet transform algorithms in linear time

– It can be calculated by the repetition of two steps:

• Converting the image into a representation with reduced resolution (pixel count)

• Storing the image information lost by this transformation (which provides the wavelet coefficients)

• The underlying technology is called Multiresolution Analysis

4.1 Multiresolution Analysis

(6)

• Idea:

– Consider the image in different resolutions

– The image signal is composed of “raster" parts and detail parts

– Therefore: representation of the image by

blocks of detailed information from which the image can be restored in stages

– Example:

4.1 Multiresolution Analysis

More detail

More

detail …

(7)

• Forming the average in different resolutions

• Summarize blocks of pixels by using their average as one pixel

• „Averaging and Downsampling“

4.1 Image Resolution

(8)

• Basic idea

– V_k: Pixel raster of the original image – V_k–1: Raster of a lower resolution,

therefore it has less pixels than V_k

– The process will continue down to V₀, which consists of only one pixel

– It still has to be defined for each V_i how the intensities of pixels are obtained from the

intensities of pixels belonging to V_i-1’s coarser raster

4.1 Multiresolution Analysis

. . .

(9)

• Usually

– The intensity of a pixel in the grid V_i-1 is the mean of a set of corresponding pixels in the grid V_i

– V₀ has then as intensity the average intensity of the output image V_k

– V_i-1 is calculated from V_i by halving the number of pixels in width or height

4.1 Multiresolution Analysis

(10)

4.1 Example

…

300  200 150  200

150  100 75  100

75  50

(11)

• For each pixel (x, y), in the original image there is in each raster V

_i

a pixel p

_i

(x, y) derived from (x, y) through repeated averaging

– Let f_i(x, y) be the intensity of the pixel p_i(x, y) in raster V_i

– For each pixel (x, y) of the original image and each i we have:

f_i(x, y) = f_i-1(x, y) + d_i-1(x, y)

– By using the detail information d_i(x, y) we can reconstruct the intensity of the pixel (x, y) in the original image f_i(x, y):

4.1 Multiresolution Analysis

(12)

• Details are often described by the differences of averages (“Averaging and Differencing”):

4.1 Multiresolution Analysis

Differences:

Advantage: In images, usually neighbor pixels are similar, thus the differences are often 0. Only strong intensity differences are contained in the compressed image.

(13)

• These two steps correspond to the application of filters in the signal processing:

– High pass filter: only receives signal components with high frequency (= baby wavelets of higher order) – Low pass filter: only receiving signal components

with low frequency (= baby wavelets of low order)

4.1 Example

X(ω)

ω

X_LP(ω)

ω X_HP(ω)

ω

(14)

• High-pass filter extract the image details, low-pass filter, the averages

• Four possible applications of both filters, to reduce the image size, both vertically and

horizontally by half:

– HH, HL, LH, LL (sub-band)

• Save the results of the high pass filter for the subsequent reconstruction of the image

4.1 Multiresolution Analysis

(15)

4.1 Multiresolution Analysis

Filtering and Downsampling

(16)

• Various resolutions

4.1 Multiresolution Analysis

LL LH

HL HH

The total number of pixels in each step is the same, i.e. no loss of information!

(17)

• Feature array

– Save the expected value and standard deviation of Wavelet coefficients at each resolution

– E.g., three-stage resolution is a 20-dimensional feature array

4.1 Multiresolution Analysis

(18)

• Shape-based retrieval

– Occurring shapes contribute significantly to the similarity of images

• In contrast to the purely visual impression made by colors or textures, shapes often carry deeper semantic information

• A displayed item, is often independent of color, however usually identical items have an identical shape

4.2 Shape-based Features

(19)

4.2 Example: Chair

(20)

• Combination of simple shape-features (round, elliptical, triangular, square, trapezoid, ...) with

other features (color, texture, etc.) brings better retrieval

– "Round object in a red-orange image" may be a search for a sunset ...

4.2 Basic Idea

(21)

• Even more complicated: "Find all coats of arms containing crosses"

4.2 Basic Idea

(22)

• Fundamental problems

– How to recognize the shape of things in images?

– Is a semantic mapping always possible?

– How do we describe shapes with features?

– Which shapes are similar and how do you compare different shapes?

4.2 Shape-based Retrieval

(23)

• Shape segmentation is a fundamental problem

– Which shapes are displayed in the image?

– All of them?

– All important?

– Only foreground motive?

4.2 Segmentation

(24)

• What represents shape and what does not?

• Is the shape homogeneous in colors or textures?

4.2 Segmentation

(25)

• Are all parts of the shape visible?

• Is the sun round?

• Segmentation?

4.2 Segmentation

(26)

• Can the segmentation be done automatically?

– At least semi-automatically?

• Not in early versions of multimedia retrieval!

– E.g.: IBM's QBIC Image Classifier

4.2 Automatic Segmentation

(27)

4.2 IBMs QBIC Prototype

(28)

• Manually or semi-automatic with Flood Fill ("seeded region growing")

4.2 Segmentation in QBIC

input image masked marked shape auto-unmask

(29)

• Problem

– Only segmentation of monochrome surfaces – 'End' forms

4.2 Segmentation in QBIC

input image masked marked shape auto-unmask

(30)

• Many research projects in multimedia retrieval have been working on the topic (e.g., Blobworld, Photobook)

– There are solutions, however mostly for special cases

4.2 Automatic Segmentation

(31)

• Due to segmentation problems, shape features were removed from all commercial databases

– IBM's QBIC

→ DB2 Image Extender (set) – Virage Retrieval Engine

→ Oracle Multimedia → Oracle Intermedia – Excalibur Technologies

→ Informix Image Foundation DataBlade

4.2 Automatic Segmentation

(32)

• In principle, a form is defined through the outer perimeter

4.2 Automatic Segmentation

(33)

• How do you get this outline?

– Segmentation of areas with the same brightness, color and/or texture

– Edge detection (differences in brightness, gradient, watersheds, etc.)

– Filling in the spaces with morphological operators (dilation and erosion)

– Segmentation of the outline as closed curve (polygon, splines, ...)

– And a large number of other procedures ...

4.2 Automatic Segmentation

(34)

• Usually applied for gray value images

• Idea:

– Important objects can be

differentiated from the background because of their different brightness range

– A certain threshold separates the regions

4.2 Thresholding

(35)

• Supposition:

– Thematically related areas have similar gray values – Can be clearly separated from the background

4.2 Thresholding

(36)

• Fixed threshold

– A fixed threshold is applied to each image

– Enough for example in the case of binary images

• Flexible Threshold

– Depending on the gray value histogram – New threshold for each image

– Often, the histogram is first smoothed but without moving the peaks

4.2 Thresholding

(37)

• ISODATA algorithm (Ridler and Calvard, 1978)

– Divide the gray value histogram into two parts

– Calculate the expectation values of the gray values in the left and right part

– Compute a new threshold as the average of the two expected values

– Iteratively compute the new expected values and a new threshold (until the threshold no longer changes significantly)

4.2 Thresholding

(38)

• Triangle algorithm (Zack and others, 1977)

– Connect the

highest peak in the histogram with the

highest brightness value – Maximize the distance

to the connecting line – Threshold is minimum,

shifted by some constant value

4.2 Thresholding

(39)

• Medical segmentation

4.2 Application example

(40)

• There are also area-based algorithms, which evaluate thresholds of individual image areas to segment an image

– Applicability depends strongly on each image collection

4.2 Thresholding

(41)

• Area based algorithms, especially for color images

4.2 Thresholding

Segmentation with Edge Flow (Ma and Manjunath, 1997)

(42)

• Advantage

– Very simple procedure

• Disadvantage

– Determination of the "right" Thresholds

– Supposition: strong color or gray value change between foreground object and background – Problem: decomposition of complex objects

4.2 Thresholding

(43)

• Strong color change between the foreground object and background?

4.2 Thresholding

(44)

• Not the area of the foreground objects will be detected, but borders of such areas

• The goal is a closed curve around an image object

• Usually, maxima of the first and second derivative of the brightness function are considered

• Gradient and Laplace operator

4.2 Edge Detection

(45)

• How to determine the gradient at (x, y)?

• Problem: gradients require differentiable

(continuous) functions; we only have discrete supporting points

• Two common solutions:

– (1) Estimate a differentiable function from the available supporting points and use these

(e.g., via Fourier transformation)

– (2) Estimate the course of this function for each pixel from its immediate neighborhood (e.g., Sobel filter);

often much faster than (1)

4.2 Edge Detection

(46)

• Gradient-based method

– Calculate the magnitude of the gradient at each point (e.g., Sobel filter)

– Edges denote high gradient

– Then use an threshold algorithm to separate the edges from regions

4.2 Edge Detection

(47)

• Advantage

– More simple filter

• Disadvantage

– Very susceptible to noise (one possibility would be performing noise reduction before applying the Sobel filter)

– Blurred or merging contours

4.2 Edge Detection

(48)

• Zero crossing of second derivative (“Laplacian Zero-Crossing”)

• Is particularly used in "noisy" images with blurred edges

• The behavior of the gradient is studied starting from an ideal edge

4.2 Edge Detection

(49)

• Idea: zero passage of the second derivative shows the maximum of the gradient

4.2 Edge Detection

(50)

• Unlike the gradient procedure, it is not expected that every point with a sufficiently high gradient value to be assigned to the edge, but only the points on the zero-crossing

• Applying a smoothing filter (normally Gaussian filter) before calculating the derivative prevents the susceptibility to noise

4.2 Edge Detection

(51)

• Important: Only real zero crossings, not zero points

• Mark all pixels with zero crossings and multiply them by the "strength" of the edge (e.g.,

magnitude of the gradient)

• Again, we can bring thresholding in performing segmentation

4.2 Edge Detection

(52)

• Example:

4.2 Edge Detection

(53)

• Comparison between gradient procedure and zero crossing technique:

4.2 Edge Detection

original gradient zero crossing

(54)

• Sobel and Zero-crossing filters in Matlab

– Transform image to gray scale values – sobel = edge(img, ‘sobel’)

– zeroc = edge(img, ‘zerocross’)

4.2 Edge Detection

(55)

• Watershed transformation

• Supposition: surfaces are defined by minimal gray values and their zone of influence

• Idea: “Flooding" a surface judging by the

minimum gray value, so that different surfaces do not connect

• Gray values can be seen as topographical surfaces or "Mountains"

4.2 Watersheds

(56)

• Example: Flood regions based on the minimum gray values

4.2 Watersheds

(57)

• For image segmentation:

Watershed transformation of the gradient :

4.2 Watersheds

original gradient water

separation

segmentation

(58)

• Advantage

– Enclosed and correct bordering

• Disadvantage

– Difficult to implement efficiently – Over –

segmentation

4.2 Watersheds

(59)

• Supposition

– Regions are bordered by a predominantly closed curve ("salient boundary")

• Method

– Based on a curve ("snake") iterate towards the best possible separation

– Minimize the energy of the snake curve

• Internal energy: curvature and continuity

• External energy: image energy (gradient)

4.2 Active Contour

(60)

4.2 Example

(61)

• Advantage

– Fits also "fuzzy" edges

• Disadvantage

– Complexity of the curve increases with the accuracy of contour

– Where does the initial snake curve come from?

4.2 Active Contour

(62)

• Problem:

– Noise can make shape recognition difficult

• Goal:

– Make the contours of surfaces easily recognizable and easy to describe

• Solution:

– Apply morphological operators as a preprocessing step

4.2 Morphological Operators

(63)

• Morphological operators are binary

neighborhood operations for changing the surfaces

– Pixels are removed or added to the object edges by such operations

– These operations are controlled by an operator mask (the "structure element")

4.2 Morphological Operators

(64)

• Basic operators

– Dilation – “inflating”, adding pixels to the area

– Erosion – “shrinking”, removing pixels from the area

• Typical structural elements are symmetric areas

of a pixel

4.2 Morphological Operators

(65)

• Dilation

– The structural element will be applied on all pixels of the source image

– The structural element defines a neighborhood around each pixel – In the dilated image the black pixels,

are exactly the pixels which had a black pixel

anywhere in their neighborhood in the original image – Effects:

→ enlarging areas

→ connecting objects with small distance

4.2 Morphological Operators

(66)

• Example of a dilation with various structural elements

4.2 Morphological Operators

Original pixel New pixel after dilation

(67)

• Erosion

– The structural element is again applied to every pixel of the source image

– The structural element again defines neighborhoods

– In the resulting image the white pixels,

are exactly the pixels which had a white pixel in their neighborhood

– Effects:

→ small spots disappear

→ breaking up areas with small connections

4.2 Morphological Operators

(68)

• Example

4.2 Morphological Operators

Original

Dilation

Erosion

(69)

• Opening - erosion followed by dilation

– Elimination of thin and small objects – Breaking up thinly connected areas – Smoothing of edges

• Closing - dilation followed by erosion

– Small holes are filled – Joining close objects – Smoothing of edges

4.2 Morphological Operators

(70)

• Example

4.2 Morphological Operators

Original

Opening

Closing

(71)

• Advantages

– Using morphological operators for image processing makes it easier to obtain good shapes

• Disadvantages

– Gray values of the areas must be uniform – Precise control is relatively difficult

4.2 Morphological Operators

(72)

• Multiresolution Analysis

• Shape-based Features

- Thresholding - Edge detection

- Morphological Operators

This Lecture

(73)

• Query by Visual Examples

• Shape-based Features

– Chain Codes

– Fourier Descriptors – Moment Invariants