State of the Art
Q- Learning
3.3 Discussion
Approach Typeof feedback Incremental Explicit time
represent.
Time intervals Relations Background knowledge Represent./ Input Algorithm(s) Learning result
Supervised inductive logic programming
FOIL [Qui90] superv. - - - Relational data
Sequential covering with information-based estimate
∼Horn clauses
Progol [Mug95] superv. - - - Prolog Horn clauses &
Mode declarations
Sequential covering with inverse entailment
First-order rules
TILDE [BD98] superv. - - - Horn clause logic &
refinement modes
Top-down induction of decision trees
First-order decision trees
Cohen and Hirsh [CH94]
superv. - - - Description
Logic
Least common subsumer
Concept descriptions Frequent pattern and association rule mining
Apriori(TID) [AS94]
unsup. - - - - - Transactions Level-wise candidate generation and statistical evaluation
Large itemsets / association rules
AprioriSome / AprioriAll / DynamicSome [AS95]
unsup. - - - - Transactions with timestamps
Level-wise candidate generation and statistical evaluation
(Maximal) frequent sequential patterns GSP [SA96] unsup. - - - - Transactions
with timestamps, item hierarchy
Frequent itemset candidate generation and statistical evaluation
Generalized frequent sequential patterns MFS
[ZKYC01]
unsup. - - - - Transactions with timestamps
Sample-based estimate itemset mining, candidate generation from estimate
Frequent sequential patterns
GSP+ / MFS+
[ZKCY02, KZYC05]
unsup. - - - Transactions
with timestamps,
re-moved/added transactions
Incremental update of
sequential patterns
Frequent sequential patterns
PrefixSpan and extensions [PHMA+01, PHP+01, HY06b]
unsup. () () - - Transactions with timestamps
Prefix-projected pattern growth
Frequent sequential patterns (with time gaps) SPADE
[Zak01, PZOD99]
unsup. () - - - Transactions with timestamps
Lattice search strategies and join operations
Frequent sequential patterns
Table 3.2: Survey of the different learning approaches (1/3)
Approach Typeof feedback Incremental Explicit time
represent.
Time intervals Relations Background knowledge Represent./ Input Algorithm(s) Learning result
SPIRIT [GRS99]
unsup. - - - - Transactions with timestamps
Extension of GSP by regular expressions (RE)
Freq.
sequential patterns satisfying the REs
MINEPI / WINEPI [MTV97]
unsup. - - - - Event
sequences
Frequent episode generation and statistical evaluation
Frequent episodes
H¨oppner [H¨op01] / Winarko and Roddick [WR05]
unsup. - - - Labeled time
intervals
Level-wise candidate generation and statistical evaluation
Temporal patterns with interval relations Tatavarty and
Bhatnagar [TB05]
unsup. - - - Multivariate
time series
Substring pattern mining, clustering, temporal relation mining
Frequent substrings and temporal dependencies WARMR
[DT99]
unsup. - - - Prolog /
Deductive relational database
Level-wise atomset candidate
generation and statistical evaluation
Frequent atomsets
MIDOS [Wro97]
unsup. - - -
Multi-relational database
Top-down, general-to-specific search
Conjunction of first-order literals SPADA
[ML01]
unsup. - - - Prolog /
Deductive relational database
Frequent atomset candidate generation and statistical evaluation
Spatial association rules
Kaminka et al.
[KFCV03]
unsup. - - - Event
sequences
Trie-based frequency counts
& statistical dependency methods
Frequent event sequences
De Amo et al.
[dFGL04]
unsup. - - () - Transactions with timestamps and group IDs
GSP-based Frequent multi-sequence patterns SPIRIT-LOG
[MJ03]
unsup. - - - Logical event
sequences, regular expression constraints
Level-wise candidate generation
Logical sequences satisfying the REs
Table 3.3: Survey of the different learning approaches (2/3)
data. However, in RL it is also not the case that explicit representations of patterns are learned but policies how an agent should behave in certain situations.
The probabilistic approaches are especially valuable when uncertain information has to be handled. The basic Bayesian networks have been extended in different
Approach Typeof feedback Incremental Explicit time
represent.
Time intervals Relations Background knowledge Represent./ Input Algorithm(s) Learning result
MineSeqLog [LD04, Lee06]
unsup. - - Logical event
sequences, background knowledge
Level-wise candidate generation with optimal refinement operator
Set of max.
(min.) patterns (not) satisfying anti-monoton.
(monoton.) constraints Similarity-based approaches
RIBL [EW96] unsup. / superv.
- - - Function-free
Horn clauses
Similarity Measure Instance collection Bergmann &
Stahl [BS98]
unsup. / superv.
- - - -
Object-oriented case repr. & class hierarchy
Similarity Measure Instance collection
SimRank [JW01]
unsup. / superv.
- - - Linked objects Similarity Measure Instance collection LAUD [AP01] unsup. /
superv.
- - - Feature term
graph
Similarity Measure Instance collection Reinforcement learning
TD, Q-Learning, Sarsa [Sut88, RN94, WD92]
reinf. - - - States,
Actions, Policy, Reward
TD, Sarsa, Q-Learning
Value-function, action-value function, Q-table
RRL [DDB98] reinf. - - Relational
States, Actions, Policy, Reward
Q-Learning &
TILDE
First-order decision tree for Q-function Probability-based approaches
1BC/1BC2 [FL04]
superv. - - - - First-order data without background knowledge
Top-down refinement &
statistics
First-order Bayesian classifier HMM, Kalman
filter, DBN; cf.
[RN03]
- - - - Evidence
variables
Particle filtering etc.
Probabilistic Model DPRM
[SDW03]
- - - Observed
relational objects’
attributes
Adapted
Rao-Blackwellised Particle Filtering
Dynamic Probabilistic Relational Model Artificial neural networks
FF ANN / Recurrent ANN
unsup. / superv.
- - - - Numerical
feature vectors
Backpropagation etc.
Trained neural network with adapted weights
Table 3.4: Survey of the different learning approaches (3/3)
ways that probabilistic reasoning over time (e.g., Hidden Markov Models, Kalman Filters, and Dynamic Bayesian Networks) and relational data can be handled (e.g., Probabilistic Relational Models). A recent extension even unites aspects of the two extension with the Dynamic Probabilistic Relational Model. However, these
ap-proaches are not suited to explicitely create the intended patterns which combine relations with different validity intervals, interval relations and generalizations about classes of objects in patterns.
Artificial neural networks can be used for supervised and unsupervised learning and with recurrent networks it is also possible to learn temporal patterns. Different approaches to knowledge extraction exist in order to create explicit understandable rules about the learned networks. However, the networks with their numeric unit representations cannot be used to represent relations between objects adequately.
Altogether it appears to be promising to combine different ideas from the fields of inductive logic programming and association rule mining. In the latter case, the extensions to sequential and relational association rule mining are particularly of interest. Thus, the approach presented in the next chapter is based on ideas of approaches of these fields.
In Section 3.2.2, different formalisms for the representation of action and time have been presented: the situation calculus, the STRIPS representation, and the interval temporal logic. Objects, properties, and relational data can be represented by all three formalisms. It is also possible to represent background knowledge in the different logical representations. The representation of temporal relations bet-ween the validity of predicates and the duration of actions or events can only be done by the interval temporal logic. The other two approaches assume changes bet-ween two world states to be done by single instantaneous actions with no temporal extension. For the situation calculus there exist extensions to handle concurrent actions (e.g., [Pin94]), but it is still not possible to precisely represent the temporal interrelationship between the actions.
Allen and Ferguson [AF94, p. 1-2] list properties of actions and events they feel to be essential:
1. “Actions and events take time.”
2. “The relationship between actions and events and their effects is complex.”
3. “Actions and events may interact in complex ways when they overlap or occur simultaneously.”
4. “External changes in the world may occur no matter what actions an agents plans to do, and may interact with the planned actions.”
5. “Knowledge of the world is necessarily incomplete and unpredictable in detail, thus prediction can only be done on the basis of certain assumptions.”
The temporal interval logic has the highest expressiveness w.r.t. representing actions and events with temporal extensions and their complex interactions. It meets
all defined requirements and shall thus be the basic formalism for the representation of dynamic scenes in this thesis.
The different approaches to motion description provide a number of interesting ideas for the actual representation of dynamic scenes. Such a representation is highly dependent on the domain, e.g., for the choice of qualitative classes for speed or distances, regions (e.g., lanes in traffic domains or the penalty area in soccer), and background knowledge. In many cases, a mapping of quantitative data to qualitative data must be performed. All the presented approaches need such a qualitative abstraction. The qualitative motion description by Miene [Mie04] appears to be the most elaborated approach as besides well-known qualitative spatial representations also qualitative information about dynamics is generated by the monotonicity-based interval generation in a way that, e.g., acceleration of objects and approaching of object pairs can be recorded. This representation is based on Allen’s temporal interval logic and is applied to analyze and interpret dynamic scenes in (robotic) soccer which is also the evaluation domain in this thesis.