The original dataset that was used during this research consists of an Excel-file with two sheets. This file contains records in the form of time series of energy delivery and consumption data from 33 dwellings of a neighbourhood called Groene Mient in The Netherlands. The data was extracted from the smart meters installed in these dwellings. The smart meters collected energy delivery and consumption records from 11-07-2017 till 31-05-2019. The data exists of the amount of kWh (kilowatt-hour) taken from the net, and the amount of kWh going back to the net minus the consumption of the residents. This is measured and stored every 15 minutes. The smart meter measures the use of energy in kWh. The dwelling characteristics are manually added to the dataset by their rightful owners. These characteristics divide into three parts. The first one is the type of heating system installed in the dwelling. This can either be represented as ‘E’ (net energy), ‘WP’ (heating pump) and ‘Zon’ (thermal solar panel). The second one is the number of solar panels. These can vary between 8 and 17. The third one is the number of inhabitants. This can vary between 1 and 4. Dwelling characteristics come in many varieties, however due to constrains in the data availability and confidentiality reasons, only the three characteristics mentioned above were made available by the Groene Mient and used during this research. Furthermore, it is good to address that the only energy source for producing the renewable energy comes for the solar panels installed on each dwelling. Additional needed energy is collected from the Dutch energy supplier.

In the data cleaning phase, the Python-scripts pre-processed the rows containing missing records and accumulated consumption data so that correct learning models could be created in the next phase. The dataset consists of different data depending on each dwelling because the smart meters were installed at different times. Therefore, the dataset was reduced for the dwellings which had fewer data compared to the other dwellings. This process cleaned out five dwellings from the dataset, which means that every machine learning model was built based on the 28 dwellings that were left in the dataset.

Apart from the smart meters data, weather data has been gathered from the Koninklijk Nederlands Meteorologisch Instituut (KNMI) [18]. This dataset helps us to build the models providing more data about the environment which influence directly to the energy delivery and consumption. The data consists of the temperature, duration of sunshine, global radiation and cloud cover indexes. Moreover, dummy variables, variables that are created from values within the dataset, have been created based on the timestamps, including are the following: hour of the day, day of the week, day of the month, week of the month, month and season.

In order to let the algorithms find the underlying patterns to correctly predict the targets, it is necessary to perform a transformation of the original dataset by grouping the energy delivery and consumption into series of daily and weekly data. Depending on which aggregated data an algorithm is trained with, it always predicted better with aggregated data instead of data per 15 minutes. For the weather data, the mean has been computed of every variable during the week, and for the dummy variables, the mode has been calculated. In the case of the Recurrent Neural Network, this grouping was not necessary to do.

Machine learning is an evolving branch of computational algorithms that are designed to emulate human intelligence by learning from the surrounding environment. They are considered the working horse in the new era of the so-called big data. Machine learning can be divided according to the nature of the data labelling into supervised, unsupervised, and semi-supervised. Supervised learning is applicable for estimating an unknown (input, output) mapping from known (input, output) samples, where the output is labelled (e.g. classification and regression). While in unsupervised learning, only input samples are given to the learning system (e.g. clustering and estimation of probability density function) [23].

The focus was on supervised learning because it allows to classify different targets, including which type of heating system, number of solar panels and number of inhabitants is the most suitable for a certain dwelling by using the available datasets. This is done by using classification algorithms, which classify data into two or more classes.

The following machine learning algorithms have been used in order to classify based on the data:

Logistic Regression. Logistic Regression is a supervised machine learning algorithm which can be used to solve classifications problems. Logistic Regression uses a linear equation to calculate a value. The computed value can be anywhere between negative infinity and positive infinity. The output needs to be between 0 and 1 to make the classification. To scale the output to a value between 0 and 1, the sigmoid function is used where ‘z’ is the output value [24]:

When using Logistic Regression, a threshold is specified that indicates at which value the output will be put into one class versus the other class. A threshold with the value 0.5 means that a class with a 50% (or greater) probability will be classified as class2 and a class with a probability less than 50% as class1. Logistic Regression can be divided into binary (where the model classifies the data in two classes) and multi-class classification (where the model classifies the data in two or more classes). Depending on the data, multi-class classification was used since the prediction for each label could be divided in two or more classes.

The model is fed with 9744 rows of aggregated data per day which contains delivery, consumption and KNMI data. Logistic Regression was used to classify what kind of heating system is used, how many solar panels are installed and how many inhabitants are in this dwelling. For every prediction, a different model is used. To improve the accuracy of the model, the hyperparameter ‘C’ can be changed. For small values of C, the regularization strength is increased which will creates simple models that underfit the data. For large values of C vice versa.

K-Nearest Neighbors. K-Nearest Neighbors (KNN) can be used for both classification and regression predictive problems and is a supervised machine learning algorithm. The goal of the algorithm is to learn a function so that given an unseen observation x, h(x) can confidently predict the corresponding output y.

In the classification setting, KNN essentially boils down to forming a majority vote between the K most similar instances to a given “unseen” observation. The similarity is defined according to a distance metric between two data points. The KNN classifier is commonly based on the Euclidean distance between a test sample and the specified training samples. Let x_i be an input sample with p features (x_i1, x_i2, ..., x_ip), n be the total number of input samples (i=1, 2, ..., n). The Euclidean distance between sample x_i and x_j is defined as:

The model is fed with 2912 rows of aggregated data per week which contains delivery, consumption and KNMI data. KNN was used to classify what kind of heating system is used, how many solar panels are installed and how many inhabitants are in this dwelling. For every prediction, a different model is used. To improve the accuracy of the model, the hyperparameter ‘n_neighbors’, which determines how many neighbours have to be checked to set the class for the new sample; and ‘p’, which represents the power parameter for the Minkowski metric can be changed [25].

Recurrent Neural Network. Recurrent Neural Network (RNN) is one type of neural networks which are designed to detect and recognize patterns in sequences of data. The dataset from Groene Mient is suitable for RNN, because it contains time and sequential data. That means that the dataset contains temporal dimensions, which is useful for RNN’s. Furthermore, the dataset contains different sizes of sequences with intervals of 15 minutes. The powerful tool of RNN is that they have a certain type of memory, and that is also part of the human brain condition. As they have feedforward connections, they can use the outputs as part of the next input moment after moment [26].

Long short-term memory. Long Short-Term Memory (LSTM) is a recurrent neural network (RNN) architecture that has shown to outperform traditional RNNs on numerous temporal processing task [26].

LSTM is a variant of the RNN maintaining the error more constantly; therefore, they are used to long time series data memorization in order to continue learning over many more time steps than the traditional RNN. One relevant point of the LSTM is that they have a forget gate; using this feature, the network can forget some low-quality patterns and start over others. Basically, LSTM performs better than other recurrent neural networks when the goal is to learn from very long-term data sequences. The ability to forget, remember and update the information make better adjustments one step ahead of RNNs. The input data shape of the LSTM must be three dimensional. The first dimension measures the batch size, the second one measure the time-steps, and the last one dimension measures the number of units in one input sequence [27].

The network is fed with 8925 samples with 92 data records (96 should be the optimum, because there are 92 data records per day, but the entire dataset has no possibility to divide it by 96, so the closest number is 92) and ten units (each data record has ten features to train delivery, consumption, KNMI data and dummy variables).

The architecture of the models in Figure 3 and Figure 4 are composed of three layers. The model in Figure 2 contains an additional dense layer, to intermediately calculate outputs that can be dropped by the dropout layer.

Figure 2

LSTM architecture for classifying the heating system

Figure 4

The initial layer consists of a LSTM layer, containing recurrently connected blocks that connect the inputs with multiplicative units. Following is the dropout layer that attempts to remove noise from the data, with the goal to prevent the model from overfitting by forcing the model to learn the pattern in different ways. 

This means that their contribution to the activation of downstream neurons is temporally removed on the forward pass and any weight updates are not applied to the neuron on the backward pass) [28]. Finally, the output is another Dense layer using a “softmax” activation function composed of 3 nodes (depends on the number of classes to predict). The outputs of the models are described based on the One-Hot-Encoding (binary encoding). The heating system outputs are encoded as following: [1,0,0] = ”E”, [0,1,0] = “WP”, [0,0,1] = “Zon”. The outputs of the number of inhabitants are encoded as following: [1,0,0,0] = 1 inhabitant, [0,1,0,0] = 2 inhabitants, [0,0,1,0] = 3 inhabitants, [0,0,0,1] = 4 inhabitants. The outputs of the number of solar panels was encoded as following: [1,0,0] = 8-10 solar panels, [0,1,0] = 11-13 solar panels, [0,0,1] = 14-17.

The rectified linear activation function (RELU) is a piecewise linear function that will output the input directly if it is positive. Otherwise, it will output zero defined as following [24]:

By assigning a “softmax” activation function (Figure 5), a generalization of the logistic function, on the output layer of the neural network (or a “softmax” component in a component-based network) for categorical target variables, the outputs can be interpreted as posterior probabilities. This is useful in classification as it gives a certainty measure on classifications.

Figure 5

The "softmax" activation function used in the output "Dense" layer

All the created models using the machine learning algorithms were validated by using Cross Validation. By using Cross Validation, it provides a possibility to assess how the results of a statistical analysis model will generalize to an independent dataset. This is done by splitting up the dataset in a training- and test set with an 80/20 ratio. This is a common approach within machine learning for splitting up data in different sets and is also described as a high-performance measure [29].

The algorithms were trained by the training data. After the training process, the algorithms were validated using the remaining (validation) data. This way, the models are validated on data that it has never seen before. The validation set contains monthly data for the years 2017, 2018 and 2019, where the data is aggregated by a whole day. By extracting the whole day and applying it to the predicted model, the model can give an output of the predicted target. Knowing what the targets are from each dwelling in the validation set, it is possible to validate the predictions with the true targets.

In order to evaluate the models, evaluation metrics were used to produce classification reports that indicates which of the models work the best for this specific problem. The confusion matrix (Figure 6) contains information about actual and predicted classifications done by the classification models. Performance of these models is commonly evaluated using the test data.

Figure 6

The confusion matrix used to determine the performance of the algorithms

The outputs in the confusion matrix have the following meaning:

The following metrics can be calculated with the outputs of the confusion matrix:

• Logistic Regression;

• Support Vector Machine;

• K-Nearest Neighbors;

• Recurrent Neural Network.

• a is the number of correct predictions where the actual value is negative;

• b is the number of incorrect predictions where the actual value is negative;

• c is the number of incorrect predictions where the actual value is positive, and

• d is the number of correct predictions where the actual value is positive.

• The accuracy (AC) is the proportion of the total number of predictions that were correct: a + d / (a + b + c + d);

• The recall or true positive rate (TP) is the proportion of positive cases that were correctly identified: d / (c + d);

• The false positive rate (FP) is the proportion of negatives cases that were incorrectly classified as positive: b / (a + b);

• The true negative rate (TN) is defined as the proportion of negatives cases that were classified correctly: a / (a + b);

• The false negative rate (FN) is the proportion of positives cases that were incorrectly classified as negative: c / (c + d);

• The precision (P) is the proportion of the predicted positive cases that were correct: d / (b + d).