Besides learning a completely new task, real-world situations often require the abil-ity to adapt an already learned task to changing conditions without learning the acquired motion repertoire from scratch, as covered by H4.2. To investigate this issue, the humanoid robot Affetto (Section 5.2) learns to solve a drumming scenario (Figure 3.13) with varying positions of the drum as evaluated in Section 3.4.3. Then, the environment changes in a way such that the drum cannot be observed directly and the robot has to perceive the drum position through a mirror, located beside the workspace (Figure 4.15). Further potential changes in the scenario include the replacement of the original (possibly faulty) sensor by a newer/intact one, a changed position of the robot which would be otherwise static, or another modified point of view on the scenery. Relearning the complete task in the high-dimensional space of actions would be highly ineffective if instead the already acquired knowledge could be adapted and reused.
The field investigating such principles is called transfer learning [Pan and Yang, 2010;Salaken et al.,2017], in which the main goal is to reuse as much as possible of the previous knowledge for the new situation. Recently, a promising transfer learning approach has been proposed for classification in myoelectric prosthesis control under electrode shift [Paaßen et al.,2018]. This approach allows to transfer the classification model between two settings, without assuming a continuous drift, by optimizing a mapping of the input features directly for the target task.
Param. Task
Skill Learning:
Transfer Learner
Reward Function
Current Task
Instance Enconding
Motion Primitive PS
Generalization
Execution Adaptation
�
Demonstration
-Gradient
∆ Readout Mismatch
Figure 4.13: Illustration of the Transfer Learning approach. Based on human demon-strations, a transfer mappingψ is updated according to the gradient of the parame-terized skill.
In this section, the generalization of the transfer learning approach for a regression model is presented and applied for adaptation of a previously learned skill of a humanoid robot towards changing task conditions.
The remaining of this chapter is structured as follows. First, relevant related work on transfer learning,Section 4.3.1, and the proposed transfer learning algorithm are introduced. Section 4.3.2 illustrates the method for an artificial example while Section 4.3.3 describes the main experiment where the proposed transfer learning
task.
Related Work The literature differentiates between different types of changing conditions [Pan and Yang, 2010]: Changes in the task and changes in the data domain. In this work, the latter case is considered where the task to be performed stays the same, while the data domain changes. In particular, the general assumption is that enough data are available in an old scenario, the so called source domain, but the goal is to solve the task in the new target domain, where only very few data are available. These types of problems are also referred to as transductive transfer learning [Pan and Yang,2010] or as domain adaptation [Ben-David et al., 2006]. More formally, data instances from the source domain will be referred to as τ ∈ T =RE and to instances from the target domain as ˆτ ∈Tˆ =REˆ.
A popular set of methods in this area are related to the concept of importance sampling, one example being the kernel mean matching algorithm [Huang et al., 2007]. Those methods introduce weights for the data points in the source space and utilize them for learning a new supervised model to improve the performance in the target space. A central assumption is that the conditional distributions in both data spaces are the same: pTˆ(θ|τ) = pT(θ|τ) [Pan and Yang, 2010]. This strong assumption, however, does not hold in this scenario where the input space is changed strongly and thus the conditional distribution changes as well.
Another set of transfer learning methods aims to solve the transfer problem by finding a common latent space for the source and target domain [Pan and Yang, 2010; Bl¨obaum et al.,2015]. However, these methods assume the availability of only unlabeled data in the target space and, thus, do not make use of any supervised information if existing. Other work, such as Procrustes Analysis [Wang and Mahade-van, 2008], requires correspondence information between some samples from both domains which is unavailable for the drumming task.
Transfer learning has been applied in robotic settings, like reinforcement learn-ing [Taylor and Stone,2009]. For the purpose of multi-robot transfer learning [Helwa and Schoellig,2017;Malekzadeh et al.,2014b], i.e. for learning a skill for a robot from another robot. A further application is inter-task learning, e.g. transfer knowledge of multiple acquired tasks to solve more complex new tasks [Fachantidis et al.,2012].
Those settings are, however, different from the presented ones because they consider only changes in the input but not in the output as learning is based on kinesthetic teaching to adapt for changing task configurations.
4.3.1 Transfer learning for nonlinear regression with the ELM For formalizing transfer learning, this work follows the main idea from [Paaßen et al., 2018,2016], which is to learn a mapping that transforms the novel target data in such a way, that the original model is applicable again. In contrast to [Paaßen et al., 2018,2016], implementation aims at a regression model and is evaluated in a robotic scenario.
While in principle this technique is applicable to any supervised machine learning model with a differentiable cost function, demonstration is performed in this case on the regression model ELM. Given a training data setD={(τj,θj)|j= 1, . . . , Ntr} in the source domain, the ELM optimizes the cost
Ntr
X
j=1
XF i=1
θij−PSi(τj)2
(4.15)
with respect to the parameters Wout, where PSi(.) is defined in Equation 3.2. This results in a learned function PS(τ), applicable to instances from the source domain τ. A further discussion on ELMs and its learning methods is given in Section 2.2.2.
For the proposed transfer learning approach, the same cost function is utilized, but this time instances from the target domain are taken as input ˆD={( ˆτj,θj)|j= 1, . . . ,Nˆtr}, with ˆNtr ≪Ntr. Furthermore, the transfer mapping is defined asψ(ˆτ) which is applied to the input ˆτ. Thereby, ψ(.) realizes a mapping from the target to the source domain and learning its parameters comprises the main part of the transfer learning step. In many application, it is reasonable to assume a linear transformation of the form ψ(ˆτ) = Ψˆτ +b, where Ψ∈ RE×Eˆ and b ∈ RE. The transfer learning problem finally is
minΨ,b Nˆtr
X
j=1
XF i=1
θij−PSi(ψ( ˆτj))2
+γkΨ˜k2. (4.16)
Thereby, ˜Ψconstitutes the matrixΨaugmented by an additional column containing the values ofbwhile λis a weighting for the l2 regularization.
Then, finding a minimum of this problem with respect to the parameters of ψ(.) constitutes the transfer learning step and the Broyden-Fletcher-Goldfarb-Shanno (BFGS) algorithm is employed for optimization.
4.3.2 Experiments I:
A Toy Data Example
The Proposed transfer learning scheme is demonstrated for a toy data set first, before it is applied to a robotic setting in the next section. For evaluation, 20 data points are sampled from the function
R27→R:x7→(x1+ 1)3+ (2(x2+ 1)3)/10, (4.17) where 14 randomly selected points are utilized for training an ELM. The means squared error (MSE) is 0.00 on the training and 0.007 on the remaining testing data.
The trained model together with the data is shown in Figure 4.14a. In order to simulate a systematic disturbance on the data, 20 new data points are sampled with an applied rotation of 180◦. The resulting target data together with the original ELM
0 -2 5 10
Output Dim
15 20
2 Input Dim. 1
1 0
Input Dim. 2 -1 0 2 -2
Data ELM estimate
(a)
0 -2 5 10
Output Dim
15 20
2 Input Dim. 1
1 0
Input Dim. 2 -1 0 2 -2
Data ELM estimate
(b)
0 -2 5 10
Output Dim
15 20
2 Input Dim. 1
1 0
Input Dim. 2 -1 0 2 -2
Data ELM estimate
(c)
Figure 4.14: Illustration of the proposed transfer learning approach on toy data:
the figures always show data (green circles) and the predictive function of the ELM (trained on the source data). (a) Source data; (b) Target data; (c) Target data after
transfer learning.
is shown inFigure 4.14b. The prediction MSE is 585.772, due to the transformation of the new data.
For adaptation to the transformation of the data, five target data points are selected randomly and are used for training a transfer mapping with the proposed transfer learning algorithm. Using these transferred target data the algorithm can employ the original ELM to evaluate the quality of the transfer by calculating the MSE. Repeating this transfer step 100 times with different random training points yields the averaged MSE 0.001(±0.001) for the points used to train the transfer and 0.129(±0.381) for the other points (standard deviations in brackets). An example run is shown in Figure 4.14c. The median error of 0.035(±0.381) reveals outliers caused by local minima that disturb the gradient descent, therefore, the solution of Ninit = 10 repetitions (random initializations) giving the lowest MSE is selected in the robotic setup.
4.3.3 Experiments II:
Drumming Through Mirror on Humanoid Robot
This chapter aims at the evaluation of transfer learning for complex robot skills. The upper body of the humanoid robot Affetto has to play a drum positioned on a table in front of the robot, as shown inFigure 3.13. For the Transfer Learning condition of the experiments, the Affetto robot is not allowed to observe the drum directly and has to learn a new parameterized skillPS. As shown inc Figure 4.15a, the robot is commanded to rotate its upper body into the direction of a mirror. As before, the marker position of the drum is extracted by blob detection. The rotation angle of the upper body is fixed, the task parameterization τˆ = (ximg, yimg)⊤6=τ is given by the perceived location of the reflection of the marker in the mirror. Accordingly, there is a considerable difference in the mapping PS(ˆc τ)6= PS(ˆτ), so that relearning of PS(ˆc τ) becomes necessary.
(a) (b)
Figure 4.15: Task parameterization of the modified perception in the drumming scenario.
Transfer Learning with Mirror
To solve this modified task, four learning schemes have been evaluated: i) the modi-fication of the parameter space is ignored and the previously acquired parameterized skill PS is evaluated, as in Section 3.4.3; ii) relearning the task from scratch is performed in the same way as inSection 3.4.3; iii) the parameterized skill obtained inSection 3.4.3is reused and training is continued with new human demonstration samples by incremental learning. Thereby ignoring the modification of the parameter space; iv) application of Transfer Learning as proposed in Section 4.3.1. Human demonstrations are utilized to estimate ˜Ψby application of Equation 4.16.
Let ˆD = {(τˆk,θk)|k = 1, . . . ,Nˆtr} be the new data set for transfer learning.
Training is performed on ˆNtr = 6 human demonstrations for drum positions dis-tributed in the workspace of the robot. Each learner is incrementally trained with 3-5 randomly selected samples of ˆD and generalization performance is evaluated for 6 randomly selected unseen drum positions. The experiment is repeated ten times and the results of the evaluation can be seen inFigure 4.16.
Thereby, a baseline is given by the evaluation of the previously learned skill PS(ˆτ) (i) resulting in a low performance due to the modifications of the task. Continued
training of PS(ˆτ) (iii) with new samples is also not able to adapt to the new task situation. A significantly better performance can be reached by transfer learning (iv) in comparison to relearning from scratch (ii).
4.3.4 Discussion
In this section, a novel transfer learning algorithm was presented, that aims at domain adaptation problems with a few labeled instances from the target domain and without correspondence information between the source and target space.
Evaluation of the method was performed on a toy data set for illustration and
proach significantly outperformed two baselines and a retrained model and supported hypothesis H4.3
3 4 5 Training Demonstrations
0 0.2 0.4 0.6 0.8 1
Success rate [%] orig. ELM
Transfer Learner new ELM cont. Training 95% conf. int.
(a)
orig. ELM Transfer Learner
new ELM cont. Training cont. Training
new ELM Transfer Learner
orig. ELM *** *** *
*** *** ***
*** ***
* ***
3 Samples:
orig. ELM Transfer Learner
new ELM cont. Training
*** ***
*** ** ***
*** ** **
*** **
4 Samples:
orig. ELM Transfer Learner
new ELM cont. Training
*** *** .
*** ** ***
*** ** ***
. *** ***
5 Samples:
(b)
Figure 4.16: (a) Evaluation of the Transfer Learning approach against three test conditions: No update of the ELM for new situations, learning of a new ELM and continued training of the previous ELM. (b) Significance analysis of results for 3,4 and 5 presented training samples. Confidence interval is based on evaluation of 10 repetitions with 6 random unseen drum positions.
C o m p l i a nt & S o f t R o b o t s
Chapter Overview This chapter tackles the improvement of low-level control of highly compliant robotic systems by a combination of machine learning and classical control methods: First, an improved low-level control of pneumatic robots is presented that integrates an equilibrium model of the actuator. The inverse equilibrium model represents simplified properties of the dynamics of the robot, i.e. in case velocity and acceleration are zero. Second, an active compliant control mode that allows for kinesthetic teaching of highly compliant robots is proposed. Experimental evaluation was performed on a highly compliant, continuum soft robot and the humanoid robot platform Affetto. Further, the applicability of the proposed control mode for industrial light-weight robots is elaborated.
This Chapter is Partially Based on:
❼ Queißer, J. F., K. Neumann, M. Rolf, R. F. Reinhart, and J. J. Steil
2014. An active compliant control mode for interaction with a pneumatic soft robot. In2014 IEEE/RSJ International Conference on Intelligent Robots and Systems, Pp. 573–579
❼ Rolf, M., K. Neumann, J. F. Queißer, F. Reinhart, A. Nordmann, and J. J.
Steil
2015. A multi-level control architecture for the bionic handling assistant. Ad-vanced Robotics, 29(13: SI):847–859
❼ Balayn, A., J. F. Queißer, M. Wojtynek, and S. Wrede
2016. Adaptive handling assistance for industrial lightweight robots in sim-ulation. In IEEE International Conference on Simulation, Modeling, and Programming for Autonomous Robots (SIMPAR), Pp. 1–8
93
❼ Malekzadeh, M. S., J. F. Queißer, and J. J. Steil
2017b. Imitation learning for a continuum trunk robot. In Proceedings of the 25. European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning. ESANN 2017, M. Verleysen, ed., Pp. 335–
340. Ciaco