A successful artificial neural network project (ANN), like project cycles in other disciplines, constitute a number of phases, namely; problem definition and formulation, system design, realization, verification, implementation, and system maintenance phase. The last two phases (system implementation and maintenance) involves embedding the obtained networks in an appropriate working system e.g. hardware or a packaged program that can be installed to run in a computer. This paper is only confined to the first four steps of the project cycle. Figure 1 below shows various stages of an ANN project cycle and the study scope.

Figure 1.

The project cycle of an ANN project, based on [5]

The overall view of this phase, together with the rationale has been partially covered in the first two chapters. The outstanding part is specific problem definition and formulation which entails explaining the kind of data available and what was required out of it. The problem involved two non-linear, non-stationery, univariate vectors of wind speeds and directions collected over a period of 2 years (from 1.11.2009 up until 30.10.2011). The data sampling intervals was 10 minutes. It was taken at a height of 60 m from the ground in the municipal region of Puumala. Puumala municipality is strategically situated along a 3,000 km shoreline at the southern Saviona region of Eastern Finland. Its location makes it prone to offshore winds that can be harnessed for wind power. The fundamental end results of the project were to construct the three common types of ANNs namely; feed forward, cascade feed forward and Jordan Elman neural networks for wind speeds and directions forecasting, and to test the networks by comparing and assessing their mean square error (MSE) and sum squared error (SSE) as the convergence criteria, during training and upon forecasting. Procedurally, the models were used in making a one step ahead hourly forecasts with 10 minute intervals, daily forecasts with hourly averages, weekly forecasts with half-daily averages and monthly forecasts with daily averages and the convergence criteria also measured for this forecasting step and the results presented and discussed.

System design phase usually starts with data collection, and pre-processing, which can be done within or outside the computation environment. Selection of simulation parameters is the second process before model construction begins. The data used herein was provided by Lappeenranta University of Technology (LUT), and granted the author with permission to use as part of this paper. System design therefore began from data pre-processing i.e. data averaging, subdivision of data into training, validation and testing sets, normalization (scaling) and backward/forward shifting in time into various lagged variables, in a process referred to as ‘sliding window technique’ used as inputs/outputs of the networks.

Wind speed and direction vectors of length (104,043) were periodically averaged into the required time periods. To get hourly data, 6-ten minute measurements were averaged. Similarly to obtain daily means of wind speeds and directions, 24-hourly averages were taken. Averaging is followed by normalization of the vector. There are a number of ways to normalize data; here we used the reciprocal which scales the data to a range of 0 to 1, before subdividing into three parts; 70% for training, 15% for validation and 15% for system testing. Lagged variables (sliding windows) were then created conforming to the desired inputs and outputs; for hourly forecasts, six 10-minute interval outputs were required, for daily forecasts 24 outputs of hourly intervals, weekly interval required 7 outputs of daily averaged values, and monthly interval 30 outputs of daily averages.

In general three classes of models were constructed; the feed forward neural networks (FFNN), Jordan Elman neural networks (JENN), and Cascaded feed forward neural networks (CFNN), (Figures 2–5). For each class of models above, lagged variables of wind speeds and directions were separately used as inputs to the networks. Four sub models were then constructed corresponding to the forecast horizons (hourly, daily, weekly and monthly), making a total of 24 models built.

Figure 2.

FFNN used for hourly forecasts

Figure 3.

JENN used for half-day forecasting

Figure 4.

Cascade feed forward neural network used for weekly forecasting

Figure 5.

Cascade feed forward neural network used for monthly forecasting

To make them comparable, authenticable and more realistic, models of the same network topologies were constructed and used for the same forecast horizon, e.g. for hourly forecasting, a model with 12 inputs, 2 hidden neurons and 6 outputs, (denoted as 12:2:6), was used throughout for all model types (JENN, FFNN and CFNN). For daily forecasting: 24:2:12, weekly forecasting: 28:21:14 and monthly forecasts were performed with the largest model with a topology of 60:20:30.

The most interesting, challenging and critical phase of the study is to build the models. Tens of parameters are usually controlled during modelling with neural networks. However, not all of them have significant effects on the network’s generalization ability. As a result, a number of modelling parameters are selected depending on the forecast horizon, degree of accuracy required, the speed at which the results are needed, among other factors. In most cases, applications used for modelling have inbuilt default settings e.g. MATLAB has readily available codes for quick modelling. In order to achieve a more meaningful model however, the modeller has to diligently select the parameters and optimize them according to some set rules and/or past experience. Noted parameters that influence network results are; the data size partitioning i.e. into training, validation and testing, type of data normalization used, input/output representation, network weight initialization, the learning rate, momentum coefficient, transfer function, convergence criteria, number of training cycles (epochs), hidden layer sizes, the training algorithm etc. For the current study, the following modelling parameters were considered

In this study, the default normalization function was disabled to give room for custom defined normalization and denormalization; continuous, normalized variables between 0 and 1 were used as inputs and outputs representing wind speeds and directions, before denormalization to their original formats.

The transfer functions used for this study were arrived at by trial and error methods starting from the presumption that data was scaled to the range of 0 to 1 and thus a sigmoidal transfer functions which possesses the distinctive properties of continuity and differentiability on the range (−∞, +∞) was necessary, an essential requirement of Back propagation learning [5]. A prior consideration was also given for the fact that a combination of hyperbolic transfer functions for both the hidden and the output layers yielded better recognition results [6].

Nagendra and Khare in their study suggest that the rules failed to yield the ‘optimal’ size of hidden layer, inferring that the best way to obtaining the required hidden layer size is by iteratively adjusting the size while measuring the error during neural network testing [7]. In this study the neural networks should ideally be able to learn and ‘understand’ the fluid statics/dynamics of the atmosphere e.g. the effects of longitudinal and transverse wind velocity gradients, atmospheric temperature and pressure among other factors and assign appropriate weights to accurately forecast the future values. The final sizes of the hidden layer was arrived at by continuously iterating, while measuring the convergence criteria i.e. sum squared error (SSE) and mean squared error (MSE) during evaluation of the network. SSE and MSE were evaluated for one point per ‘sliding window’ and for ‘one step ahead’ forecasts and compared.

Different training algorithms are good for different purposes, the predictive ability (which is the current subject), has been tested by Ghaffari and team, who concluded that the order of predictive ability of a network trained using above group of training algorithms is IBP, BBP followed by LM, QP and lastly GA. [8]. In this study, Bayesian regulation (BR) Back propagation algorithm was used for all the models. Lavenberg-Marquardt (LM) was also tried but it proved to take too long training time than expected. Both LM and BP training algorithms are implemented in MATLAB and can be invoked by a single command. Many training algorithms suffer from the problem of over fitting, a phenomenon in ANN, caused by overtraining, resulting in memorization of input/output, rather than basing them on the internal factors determined by the weights generated. This causes the network to respond poorly when presented with new data that was not used during training, thus losing the object orientedness, an important aspect of the network, also referred to as generalization. Bayesian regulation seems to train successfully has an inbuilt ability to get rid of this problem through automatic early stopping once the error starts to propagate. [9].

Several main techniques are currently used to get rid of premature saturation, a phenomenon that has been known to cause over fitting and affect network convergence [10], [11]. Nguyen and Widrow had suggested that initializing adaptive weights over a large number of training problems achieved major improvements in learning efficiency [12]. Moallem and Ayoughi proposed three methods; increasing the number of hidden neurons, Weigend weight regularization and renewing saturated terms by adding anti-saturating terms [13]. Network weight initialization involves assigning predetermined optimum initial values for the weights to all existing connection links that help the network to converge faster. For the current study, Nguyen and Widrow weight initialization algorithm was used. In this algorithm, weight bias initialization values are picked between the intervals located randomly in the predetermined region i.e. −1 and 1. Nguyen and Widrow suggested that, if H is the number of units in the first layer, Wbi = 0.7H. Wi are chosen between -1 and 1 and the weights, w are assigned so that w = -Wi/Wbi, simply put as the uniform random values between -1 and 1 and is implemented in MATLAB [14] as a script file [15].

A high learning rate is detrimental to the network as it poses a risk of overshooting while a slow learning rate takes too much time for the network to converge. The learning rate can be constant throughout, as was done in this study or can be made adaptive i.e. to vary with time, η(t). In the case of adaptive parameter, it can be made high in the beginning of the training or rather when the search is far away from the minimum; and smaller as the search reaches minimum. This parameter rate can be anything between 0 and 10. In all the networks created for this paper, the learning rates ranging from 0.01 to 3 gave satisfactory results.

A high μ is likely to reduce the risk of getting trapped in the local minima. However, it runs the risk of overshooting just as a high learning rate does. This value, just like the learning rate, can be made adaptive, i.e. μ(t). It is set relatively high when the search is far away from the solution and lower as the search approaches the true minimum, depending on the error gradient [16]. For this project, the momentum coefficient between 0.0 and 1.0, as suggested by [17], produced satisfactory results.

An epoch is defined as a single presentation of each input/output data on the training set [16]. Epochs are set as one of the training parameters and are important in gauging the training time taken by a neural network to reach convergence and also to set the goal that determines the extent to which the network should be trained. For this study, the training epochs were set by trial and error, with a range of 100 to 1000. At most 1000 epochs for all the models built produced satisfactory results. The use of Bayesian regulation training algorithm also was used as a tool for setting the stopping time, making epoch setting just a supporting criterion.

This is the stage that this study is focused on, as it clearly distinguishes the variation in the original data to the predicted. It was made part of the modelling stage by supplying the model with the range of original data set, from the 70–85th percentile and comparing the model output to the last 85th to 100 percentile of the same data. The convergence criteria were then measured by determining the two statistical properties, i.e. the MSE and the SSE between the forecast and the target results, which were compared and reported for each model built. In the next section the quantitative results of the study were presented graphically and by tabulation.

In this study the mean square errors (MSE) and the sum square errors (SSE) were used to gauge the performance of the networks. The mean square error is the average of all the squares of individual errors between the model and the real measurements, and is given by:

where N is the number of samples, x_i and y_i are measured and predicted values.

The sum square error (SSE) is the total summation of the individual squares of errors without averaging, and it gives an indication of the total magnitude of the error between the models and the measured results. SSE is given by:

In addition, MSE and SSE are useful in making comparisons between several models with same sets of data and same observations, N. In the event that more than one model is compared, one important indicator obtained is how better a model is, compared to the others. As seen, both MSE and SSE are dependent on the number of observations and so the quantities (orders) of errors are only significant, relative to those of other models and have units same as the square of the variable under question (m²/s² for wind speed and sq. degree (o²) for directions).

Tables below show the results of the models based on both MSE and SSE on training and upon simulation with totally new inputs, not used during training. Here we compare a column on the model outputs, to the corresponding column on the measured data. This is referred to as 1-point per sliding window. A 1- point per sliding window extends for the entire column length. Plotting a column on the target matrix versus a corresponding column on the model output matrix measures the generalization ability of the model with increasing forecast horizon on the long-term. The results for this exercise are shown in Tables 1–4. Tables 1 and 2 were used to assess long-term generalization ability for wind speed forecasting; similarly Tables 3 and 4 were used to test the generalization ability for wind directions on a long-term basis.

The short term usability of the models was assessed by measuring the relative error between the model output rows and the measured data, referred to as a sliding window. A sliding window is simply one set of inputs and outputs to a neural network model, e.g. for hourly forecasting with 10-minute interval data, a row of six model outputs are compared to the corresponding row in the real measured data matrix. Plotting and comparing the rows cutting across the model output matrix to those of the target matrix is what was referred to, as sample whole sliding window. This measures the generalization ability of the model on a short term basis, also commonly referred to as one-step-ahead forecasting. The results are presented in Tables 5 and 6. Table 5 assesses the generalization ability of the models when used for forecasting wind speeds; Table 6 presents same equivalent results for wind directions forecasting.

The core needs determines the criteria applied by the modeller in choosing between various types of models. A number of criteria used in this study to assist in making that choice are identified as the degree of accuracy needed, the forecast horizon for which the model is designed, and whether the model is usable for long-term or short-term forecasting. In this case, long-term forecasting can be hourly forecasts for a relatively long period of time e.g. several months ahead. With the kind of results presented in section 3.1 and 3.2 therefore, one can tell which model type has the lowest statistical error compared to other models, during training and upon verification i.e. with new inputs. It is also possible to tell which model is best suited for which forecast horizon, and which one is good enough for long/short term forecasting for both wind speeds and directions. Tables 7 and 8 summarize the obtained results, specifically answering the above important questions regarding the models.

Sample plots for predicted and forecasted values versus measured data for two groups of networks models (FFNN and CFNN) are presented below for hourly, weekly and monthly forecast horizons. Two important terminologies are emphasized predicted and forecasted results; the difference between predicted and forecasted variables should be noted. In statistical modelling, predicted variable usually refers to the output of data used for training, i.e. assessing how well the training data fits to the model output. Forecasting is the expected results into the future from a predictive model, for inputs that were not used during training of the model. Samples of predicted, measured, and forecasted results for hourly, weekly and monthly horizons are shown in Figures 6–11.

Figure 10.

Comparing model outputs and the measured monthly wind directions upon training

Figure 11.

Comparing model outputs and the measured monthly wind directions upon forecasting

NB: The measured data on training is a part of the first 70% and on verification is part of the last 15% of the original data; therefore they are not the same set of data.