This chapter illustrates how the uncompromising application of framework technology leads to the construction of ANN software with appropriate flexibility. The implementation of such a framework corroborates that a sound object-oriented design of neural network components delivers the expected benefits, which could not be provided by a conventional solution.
An important goal of the component framework development was its usefulness in different application domains and reusability for different tools. The CANN components have been used for different purposes in different systems as expected. An early version of the CNM ANN component was applied to perform credit rating for the Quelle retail company. The very good results achieved in this work were one of the main motivations for constructing the CANN simulation environment. The numerical results of this work are not public domain.
The developed ANN framework, specially the optimized CNM component (see Chapter 6) was the base for the implementation of the AIRA data mining tool (http://www.godigital.com.br). AIRA has been applied mainly in the area of personalization of web sites.
The CANN simulation environment has been applied to weather forecast. In the work described in (da Rosa et. Al, 2001a and 2001b), it is applied to rare event weather forecasting
at airport terminals. Rare events are difficult to forecast because, by definition, the experts do not have extensive experience with such events. Those events may make it difficult for an aircraft to land or take off, causing many problems like traffic controllers acting under stress situations, the aircraft having to land at another airport, etc.
Even in an early stage, the CANN simulation environment has been used as a simulation tool for implementing a VMI solution (Vendor Managed Inventory) for an E-Business company dedicated to implementing B2B solutions for supermarkets and its suppliers (http://www.mercador.com).
The framework design has provided flexibility and reliability to those systems. The CANN framework components expanded from a classificatory system with only one learning algorithm to the possibility of implementing many different learning algorithms. The design permits the straightforward application of the different ANN models to different ANN domain problems. Different data sources are easily coupled with the domain problem at hand and applied to the ANN learning and testing processes. The design also allowed the framework to add other implementation facilities such as parallelization and distribution. This plays an important role to overcome the limitations of hardware that the ANN learning may face. The Java implementation also permits the necessary platform independence.
The current design and implementation of CANN may be considered as a generic decision-making system based on neural networks. An ambitious goal would be to enhance the framework further, so that other decision support problems, like forecasting, can be supported. Also ambitious would be to allow the implementation of other learning mechanisms that do not rely only on neural networks, such as machine learning algorithms.
Currently, most excellent frameworks are products of a more or less chaotic development process, often carried out in the realm of research-like settings. In the realm of designing and implementing CANN hot-spot-analysis was particularly helpful for coming up with a suitable framework architecture more quickly. An explicit capturing of flexibility requirements indeed contributes to a more systematic framework development process.
4 A N N P a r a l l e l I m p l e m e n t a t i o n
One of the main concerns of implementing ANN is to take care of performance aspects. The software solution shall be carefully developed in the sense that it does its best to guarantee the ANN performance while performing learning and testing tasks. To improve the ANN performance, it is necessary to use the hardware platform as much as possible. The first step in this direction in the realm of the CANN project was to implement parallelism in the neural network learning and testing mechanisms. With such an implementation, machines with more than one processor (for instance 2 to 4 processors) improve the network performance.
The parallel program design in ANN is an important aspect to be considered while developing a software solution such as the CANN framework. Thus, the goals of exploring parallel software implementation in this work are twofold:
• To explore the possibility of having a generic parallel solution for the CANN framework.
• To propose and implement a parallel solution for the CNM.
The first goal is focused on exploring the state of the art in parallel implementations for ANN in order to understand the best solution for the simulation environment as a whole.
The second goal is specific to the CNM which is an ANN model were parallelism still was not explored in detail, being a contribution to this model state of the art.
4.1 Introduction
The first step, in order to better understand how to implement parallel ANN solutions, is to take a look at structuring approaches for implementing parallelism in ANNs. Kock (1996) proposes how to break the ANN structure from large- to fine-grained pieces in order to run them in parallel:
• Training session parallelism – Train a given ANN simultaneously with different learning parameters in the same training examples. Typically different sessions are placed on different processors.
• Training example parallelism – Implementation of simultaneous learning in different training examples within the same session. A training set is split into a number of subsets and the corresponding number of network instances are trained simultaneously. The weights of each network instance are accumulated separately and at the end of the process they are brought together in one network. The different training subsets are distributed on the different processors.
• Layer parallelism – It provides concurrent computation for layers. Layers are pipelined so that learning examples are fed through the network in a way that each layer works in a different training example. The different layers are distributed on the different processors.
• Node parallelism – The ANN neurons perform weighed input summation and other computation in parallel.
• Weight parallelism – It refines node parallelism allowing the simultaneous calculation of each weighed input. This form of parallelism can be implemented in most of the ANN models.
• Bit serial parallelism – Each bit of the numerical calculations is processed in parallel. It is a hardware dependent solution.
Implementing a generic solution for the CANN framework means that any implemented ANN shall naturally have a parallel solution. The mentioned implementations shall be considered in a generic solution.
In general, there are four ways of implementing ANN algorithms: adapting or extending a preprogrammed simulator; developing the solution from scratch using a general purpose language; developing the solution using a ANN library based on a general-purpose language; and using an ANN specification language. Simulators in general lack flexibility and extensibility in terms of ANN parallel implementations so that the developers have to make their hands-on implementation. Parallel implementations of ANN simulation kernels are rare because a parallelization of the complete kernel is difficult. When doing so, the kernels have restricted functionality being applied to specific ANN architectures or models.
In the early nineties a major challenge of ANN development was to achieve maximum performance on parallel machines performing ANN tasks. The common strategy of these developments on general-purpose parallel computers (SIMD architectures like MasPar and MIMD architectures like IBM SP-2) was to speed up the processing using special characteristics of the target ANN architectures. That means implementing specific solutions to specific ANN models from scratch. Special-purpose parallel neurocomputer architectures like CNAPS (Hammerstrom, 1990) or Synapse-1 (Ramacher, 1992) were also used. Such implementations were done using machine-dependent languages or libraries. The programmer is responsible for choosing how to partition the ANN structures in order to better use the available processors. This may not be optimal and may collapse if any ANN architectural change has to be done.
Another alternative is to use libraries for the simulation of ANN like SESAME (Linden et al. 1993), MUME (Jabri et al. 1993) or PDP++ (Dawson et al. 1997). Such libraries are based on general-purpose languages and contain facilities for constructing ANN architectures and simulate them. They are neither suited for neurocomputers nor for parallel computers due to the underlying sequential programming language.
Trying to avoid this, some specific languages for building ANN solutions were created.
One example is CuPit (Prechelt, 1994). This language was specifically designed to express neural network learning algorithms. Its programs can be compiled into efficient code for parallel machines. Its advantages in terms of software design are the high expressiveness, clarity and ease of programming for implementing ANN learning algorithms. As the language is domain specific, it can result in more efficient code because it applies optimizations unavailable to compilers for general-purpose parallel languages. However, when compared to solutions built in sequential languages such as C/C++, the resulting code may be less efficient. Furthermore, this language does not support characteristics of object-oriented languages such as inheritance and neurocomputers were not supported as target architectures.
Another more recent language for building ANN parallel solutions is the EpsiloNN (Strey, 1999). The EpsiloNN (Efficient parallel simulation of Neural Networks) was built to efficiently simulate ANNs on different parallel computer machines such as SIMD parallel computers and neurocomputers. Object-oriented concepts such as classes and inheritance are applied to describe the ANN architectures. Similarly to the CuPit language, EpsiloNN main building blocks for the ANN construction are neurons, synapses and networks.
EpsiloNN does not rely on polymorphism and dynamic binding because they can only be
analyzed at runtime and forbid the computation of an optimal mapping at compile-time. The ANN specification is transformed into an appropriate simulation source code. It generates C ANSI code for sequential computers and adequate C dialects for parallel computers or neurocomputers.
Neural networks in general execute few kinds of operations: local operations on neurons or synapses; reduction operations such as summing over all weighed incoming synapses; and broadcast operations such as applying a global parameter to all neurons. Those described operations may happen on local objects such as the neurons or on groups of them such as the incoming synapses of a neuron. Such kinds of operations lead to two nested levels of parallelism: Weight parallelism and Node parallelism. The already cited languages for implementing ANN algorithms CuPit-2 and EpsiloNN implement Weight and Node parallelism having achieved very good results. The CANN solution goes in the same direction by implementing a generic solution to Weight parallelism that is explained in the next section.