Gaussian Process-based Bayesian Optimization and Shape Transformation of Benchmark Functions

Gaussian process-based Bayesian optimization (GPBO) finds application in various fields for approximate optimization of parameters. Because the search performance depends on the shape of the black-box function, users of GPBO should know these details. Therefore, we provide some experiment results of the relationship between GPBO search performance and the shape of the black-box function. We adopted “Easom,” “Ackley,” “Bukin N.6,” “Beale,” “Rosenbrock,” and “Goldstein–Price,” which are benchmark functions for optimization problems. Moreover, we adopted logarithmic and range-transformed functions to provide deeper insight.


Introduction
Gaussian process-based Bayesian optimization (GPBO) is an important approximate optimization method often used for hyper-parameter tuning in machine learning approaches, such as neural network [1,2], support vector machine [3,4], random forest [5].In general, the shapes of the black-box function affect GPBO search performance.Therefore, we investigate the relationship between function shapes and search performance using the following black-box functions.
Easom function: Ackley function: Bukin N.6 function: Beale function: Rosenbrock function: Goldstein-Price function: The above benchmark functions were used for verifying optimization methods [6].Input domain [x 1 x 2 ] ⊤ ∈ Ψ, output scale, location of global minimum x * are given in Table 1.The output scales vary depending on the black-box function.However, depending on the adopted function, the GPBO search may not work well.Therefore, we also adopted logarithmictransformed functions (LT), defined as where, a is a coefficient for avoiding log e 0. In addition, we adopted range transformation from -1 to 1 (RT) as a shape transformation function.RT is defined as This study uses GPBO for the black-box functions (no transformation, LT, and RT).These shapes are illustrated in Figure 1.
Moreover, we set a = 0.01 as the coefficient of LT.For initialization, the algorithm randomly generated four initial solutions; next, we performed 75 GPBO iterations.The algorithm ran with 100 random seeds to remove the randomness effect and stabilize the results.
The results are provided in Table 2.In Easom and Ackley functions, GPBO search performances are high, regardless of shape transformations.However, in the Easom function, finding an optimal solution is time-consuming.Therefore, changing the shape to the Easom function using RT may not be desirable.
In Bukin N.6, Beale, Rosenbrock, and Goldstein-Price functions, GPBO could not find an optimal solution when no transformation was applied, probably because of the large output scale.Therefore, some transformation is needed to search for optimal solutions.Our results suggest that Bukin N.6 and Beale functions achieve desired results when adopting RT and LT, respectively.Moreover, we confirm that RT and LT are desirable for the Rosenbrock and Goldstein-Price functions.The experiments revealed that the desirability of shape transformations to find optimal solutions efficiently differs depending on the black-box functions.

Conclusion
We reported the relationships between the GPBO search performance and the shapes of blackbox functions.Because the appropriate transformation depends on the shape of the original function, it is essential to consider the shape of the black-box function for the search problem when using GPBO.Table 2. Rates of the optimal solution, rates of neighborhoods of optimal solutions, number of BO steps to reach the optimal solution (rates, means, and standard deviations in 100 trials).A neighborhood is defined as a region of radius less than one, centered on the optimal solution.

Figure 1 .
Figure 1.Shapes of black-box functions.The first column represents original functions, second column represents LT functions, and third column represents zoomed figures.

Table 1 .
Input domains, output scales, and global minimums of each function.Shapes of functions are shown in Figure1.