A Comparative Assessment of Regularized Regression Techniques for Modeling the Mechanical Properties of Rubberized Concrete

2.1. Dataset Acquisition

This research aims to quantify rubberized concrete's elastic modulus and compressive and splitting tensile strengths. The dataset was assembled in an experimental database from a previous study to construct numerical models of rubberized concrete properties across a wide range of rubber constitutions [21]. The descriptive statistics of the dataset's characteristics are summarized and organized in Fig. (2). The contents of rubber particles are directly related to the amount of coarse and fine aggregates (Fig. 3), which would raise several issues during the development of a multiple linear regression model.

2.2. Regression Analysis Methods

In statistical analysis, regression techniques are often applied to estimate the risk of a probable outcome [36]. Overfitting occurs when typical regression approaches are utilized for a set of candidate variables to construct a model, generating an overestimation of the model's performance when using the included variables to describe observed variability [37]. The algorithm tends to underperform when it comes to predicting high-risk events. These issues may be addressed using different (penalized or regularization) regression methods [38]. Multiple linear regression employs many explanatory variables to estimate the result of a response variable. Achen [39] discussed its mathematical algorithm, as shown in Eq. 1.

(1)

Where,

Y = [y₁, K, y_n]^T vector of the dependent variables;

matrix of the independent variables for n measurements and k inputs;

β = [β₁, K, β_K]^T vector of model's coefficients to be predicted;

ε = [ε₁, K, ε_K]^T vector of random error;

Ridge regression is a procedure for predicting multivariable regression coefficients when strongly correlated observations are present. As a result, this method is more likely to estimate RBC characteristics from its components better than multivariable linear regression. by considering Eq. 1, the standard least-squares solution of the β coefficients is shown in Eq. 2, in which is the best linear unbiased estimate of β.

(2)

On the other hand, Hoerl and Kennard [40] developed a ridge regression model to address the issue of multicollinearity by computing coefficients β using Eq. 3 for strongly correlated independent variables.

(3)

expresses the ridge estimator, α > 0 is the complexity parameter that regulates shrinkage and ensures and I_P represents the identity matrix.

Besides, Ridge regression handles some of the drawbacks in traditional least-squares. It imposed a penalty on the size of the coefficients to reduce a penalized residual sum of squares, as given in Eq. 4. Hence, ridge regression employs the squared euclidean norm 1₂ regularization procedure to penalize the coefficient vector.

(4)

Lasso regression is a linear predictor of dispersed coefficients. According to its disposition to select solutions with fewer non-zero coefficients, it is helpful in some settings, effectively decreasing the number of characteristics on which the delivered result is reliant [41]. Mathematically, it composes of a linear algorithm with an added regularization term. The objective function to minimize is indicated in Eq. 5, where α is a constant value and denotes the absolute norm 1₁ of the coefficient vector.

(5)

ElasticNet regression is a linear method trained with both 1₁ and 1₂ norm regularization of the coefficients to enable learning a scattered algorithm in which few weights are non-zero such as Lasso, while keeping the Ridge’s regularization features [42]. On the other hand, ElasticNet is beneficial when several properties are correlated. The objective function to minimize is determined in Eq. 6, where ρ is a parameter inserted to control the convex combination of 1₁ and 1₂.

(6)

Bayesian regression is a method used to develop linear regression models by utilizing Bayesian inference where the errors ε_i : N (0, ) are independent and normally distributed. Additionally, it has a prior probability distribution of variables with the likelihood function given in Eq. 7 to find the posterior probability distribution.

(7)

In Bayesian ridge regression (BR), on the other hand, the regression problem is modeled using a probability distribution to avoid multicollinearity (Zhang et al., 2019). It is possible to write the objective function of this model, as shown in Eq. 8. The priors over parameter α and the precision λ are defined using gamma distributions and are predicted jointly with β in the model’s fitting stage.

(8)

Stochastic gradient descent (SGD) is an easy and effective method for fitting linear regression models. It is currently considered a beneficial method for fitting models with many features. The loss gradient is predicted for each sample at once and then updated with a decreasing strength schedule. Within the prediction of this model, three types of regularizers can be adopted, including 1₁ , 1₂, and ElasticNet. However, the SGD model in this study adopts the ElasticNet in the penalty function for estimating the mechanical properties of RBC.

Huber regression is a method of developing linear regression that is sensitive to outliers by optimizing β and σ values for the samples’ squared loss when and for their absolute loss when . This step ensures that the errors in the model are highly affected by the existing outliers. The objective function used for this type of model is defined in Eq. 9.

(9)

Where, H_ε is given by the following equation:

(10)

Quantile regression is a type of linear regression that predicts the medium quantiles of a given y conditional on X and uses the ordinary least squares to predict the conditional mean. Accordingly, its output is the q^th quantile for any value of q ranging between 0 and 1. The objective function used in this technique is defined in Eq. 11.

(11)

Where, PB_q is the pinball (linear) loss given by the following mathematical expressions:

(12)

Indeed, the pinball loss herein is linear for residuals only, and thus, this technique is better for handling outliers than the squared error prediction of the mean.

2.3. Model Development and Hyperparameters Tuning

The chosen hyperparameter values have a significant influence on regression analysis results. This article utilized a grid search procedure with k-fold cross-validation to improve the method hyperparameters during the training phase. Fig. (4) illustrates a proposed methodology for concocting regression models in which the dataset is divided into two groups: 70% training and 30% testing data. The appropriate parameter is selected using a cross-validation method with 10-folds repeats. The final tuned algorithm implementation is evaluated by comparing the outcomes of different scoring parameters on the test dataset after the hyperparameters of each technique have been decided.

2.4. Models' Performance Evaluation

Statistical measures and visual representations are adopted to analyze regression techniques' performance. The goodness-of-fit was checked using the coefficient of determination, by using Eq. 13. The root mean square error (RMSE), by Eq. 14, and mean absolute error (MAE), by Eq. 15, were used for the error analysis.

(13)

(14)

(15)

x_i is the measured value, is the mean of the measured values, y_i is the predicted value, is the mean of the predicted values, and n is the number of observations.

3.1. Compressive Strength

Indeed, the compressive strength of concrete is a critical parameter for designing structures [43, 44]. This section is intended to evaluate the applicability of the abovementioned eight different regression approaches in terms of suitability and accuracy in modeling RBC compressive strength. The estimation results for the training and testing datasets are similar across most models, as shown in Fig. (5). The residual plots of the prediction outcomes are depicted in Fig. (6), in which most regression models tend to provide similar findings with slight variations among them. Table 1 shows the model's performance, and it is clear that the MLR model had the highest coefficient of determination compared to other techniques for the training dataset while having a lesser one when it came to the testing dataset. Also, the Quantile and Huber methods produced the best MAE values in the training case, while the worst was for the testing scenario. This observation is attributed to overfitting issues in these models. In contrast, the ElasticNet and SGD models showed the lowest error values in training but had the highest ones in the testing case, which means that these models are the best regarding the overfitting issue. The RMSE of ElasticNet and SGD was 8.41% and 9.01% lower than the MLR on the testing dataset, but the MAE was reduced by 6.1% and 8.8%, respectively.

3.2. Splitting Tensile Strength

The potential of regularized regression techniques will be compared in this section to predict the splitting tensile strength of RBC. Fig. (7) highlights the analysis results, and Fig. (8) depicts the residual plots of the outputs. Similar to the preceding part, Figs. (7 and 8) demonstrate a moderate shift in the results of the estimating model. The coefficients of determination listed in Table 2 for the training and testing datasets are significantly high regardless of the used regression model. The errors analysis of the algorithms has shown that the SGD generated the lowest values with an 8.7%, 20%, and 2.5% reduction in the RMSE, MAE, and Max error, respectively, compared to the MLR. In contrast, the outcomes of the Ridge, Lasso, and ElasticNet were identical, and the Quantile and Huber methods had the worst capabilities for both the training and testing cases. Thus, it can be seen that the SGD is the best predictor for the splitting tensile strength of RBC.

3.3. Modulus of Elasticity

Figs. (9 and 10) show the outcomes of the prediction methods for the modulus of elasticity of RBC mixtures and the models' residual diagrams, respectively. Indeed, the findings show that most models have a comparable performance, which is in line with the observations of the earlier part. Moreover, the fitting rates of the investigated models and their error analyses are indicated in Table 3. Unlike the splitting tensile strength, it can be seen that the maximum coefficient of determination for the testing dataset was 0.9, while the lowest was 0.77, and was measured in the Quantile model. Additionally, it was shown that the best results were obtained using the SGD model, in which its RMSE and MAE were decreased by 9.4% and 7.4%, respectively, compared to the MLR model. In contrast, the performances of the Ridge, Lasso, and ElasticNet were almost identical, and the Quantile had minor outcomes in the training and testing cases. It means that the SGD model is a suitable approach for rapidly estimating the modulus of elasticity of RBC mixtures compared to other techniques.

3.4. Comparison of the Regression Analysis Methods

This section is intended to compare and benchmark the performance of the investigated regression models for both the training and testing datasets. The assessment results are shown in Fig. (11) for the compressive strength of RBC, in Fig. (12) for the splitting tensile strength, and in Fig. (13) for the modulus of elasticity of the concrete mixtures. Indeed, it can be seen that for the case of RBC's compressive strength, most models had lower coefficients of determination in the training dataset while achieving higher ones in the testing scenario. This observation is also seen in the cases of the splitting tensile strength and modulus of elasticity of RBC, and can be attributed to the overfitting issues that the MLR model faces while the penalized techniques can control. In addition, the results show that the Ridge, Lasso, ElasticNet, BR, and SGD approaches yield better outcomes compared to the MLR method, with the SGD being the best among them. In contrast, the Quantile and Huber methods have significantly worse behavior than the MLR approach for all of the mechanical properties' estimation cases.