1. Proceedings
  2. I3M
  3. 2020
  4. MAS
  5. 10.46354/i3m.2020.mas.008

Hyper-parameter handling for gaussian processes in efficient global optimization

  • a Bernhard Werth , 
  • b Johannes Karder , 
  • c Andreas Beham , 
  • d Stefan Wagner  
  • a,b Heuristic and Evolutionary Algorithms Laboratory, University of Applied Sciences Upper Austria, Softwarepark 13, 4232 Hagenberg, Austria
  • b,c,d Josef Ressel Center for Adaptive Optimization in Dynamic Environments, University of Applied Sciences Upper Austria, Softwarepark 13, 4232 Hagenberg, Austria
  • b Institute for Formal Models and Verification, Johannes Kepler University, Altenberger Straße 69, 4040 Linz, Austria
Cite as
Werth B., Karder J., Beham A., Wagner S. (2020). Hyper-parameter handling for gaussian processes in efficient global optimization. Proceedings of the 19th International Conference on Modeling & Applied Simulation (MAS 2020), pp. 60-67. DOI: https://doi.org/10.46354/i3m.2020.mas.008

Abstract

In simulation-based optimization, a common issue with many meta-heuristic algorithms is the limited computational budget. Performing a simulation is usually considerably more time-consuming than evaluating a closed mathematical function. Surrogate-assisted algorithms alleviate this problem by using representative models of the simulation which can be evaluated much faster. One of the most promising surrogate-assisted optimization approaches is Efficient Global Optimization, which uses Gaussian processes as surrogate-models. In this paper, the importance of carefully chosen hyper-parameters for Gaussian process kernels and a way of self-configuration is shown. Based on properties of the training set, e.g. distances between observed points, observed target values, etc., the hyper-parameters of the used kernels are initialized and bounded accordingly. With these initial values and bounds in mind, hyper-parameters are then optimized, which results in improved Gaussian process models that can be used for regression. The goal is to provide an automated way of hyper-parameter initialization, which can be used when building Kriging models in surrogate-assisted algorithms, e.g. Efficient Global Optimization (EGO). Obtained results show that applying the proposed hyper-parameter initialization and bounding can increase the performance of EGO in terms of either convergence speed or achieved objective function value.

References

  1. Affenzeller, M., Beham, A., Vonolfen, S., Pitzer, E., Winkler, S. M., Hutterer, S., Kommenda, M., Kofler, M., Kronberger, G., and Wagner, S. (2015). Simulation-Based Optimization with HeuristicLab: Practical Guidelines and Real-World Applications, pages 3–38. Springer International Publishing, Cham.
  2. Bartz-Beielstein, T. (2016). A survey of model-based methods for global optimization. Bioinspired Optimization Methods and Their Applications, pages 1–18.
  3. Berkenkamp, F., Schoellig, A. P., and Krause, A. (2019). No-regret bayesian optimization with unknown hyperparameters. Journal of Machine Learning Research, 20(50):1–24.
  4. Bull, A. D. (2011). Convergence rates of efficient global optimization algorithms.
  5. Byrd, R. H., Lu, P., Nocedal, J., and Zhu, C. (1995). A limited memory algorithm for bound constrained optimization. SIAM Journal on scientific computing,
  6. Parihar S.S., 2014. Mathematical Modeling on Dropwise Condensation. International Journal of Current Engineering and Scientific Research 2014: 30-35.
  7. Sakakura K., Yamamoto S., 2006. Numerical and experimental predictions of heterogeneous condensate flow of moist air in cooled pipe. International Journal of Heat and Fluid Flow 27:220-228.
  8. Palar, P. S. and Shimoyama, K. (2019). Efficient global optimization with ensemble and selection of ker16(5):1190–1208.
  9. Haftka, R. T., Villanueva, D., and Chaudhuri, A. (2016). Parallel surrogate-assisted global optimization with expensive functions – a survey. Structural and Multidisciplinary Optimization, 54(1):3–13.
  10. Jones, D. R., Schonlau, M., and Welch, W. J. (1998). Efficient Global Optimization of Expensive Black-Box Functions. Journal of Global Optimization, 13(4):455–492. nel functions for engineering design. Structural and Multidisciplinary Optimization, 59(1):93–116
  11. Picheny, V., Wagner, T., and Ginsbourger, D. (2013). A benchmark of kriging-based infill criteria for noisy optimization. Structural and Multidisciplinary Optimization, 48(3):607–626.
  12. Rasmussen, C. E. and Williams, C. K. I. (2005). Gaussian Processes for Machine Learning. The MIT Press.
  13. Regis, R. G. (2019). A survey of surrogate approaches for expensive constrained black-box optimization. In World Congress on Global Optimization, pages 37–47. Springer
  14. Tran, N., Schneider, J.-G., Weber, I., and Qin, A. (2020). Hyper-parameter optimization in classification: Todo or not-to-do. Pattern Recognition, 103:107245. 
  15. van Stein, B., Wang, H., Kowalczyk, W., and Bäck, T. (2018). A novel uncertainty quantification method for efficient global optimization. In International Conference on Information Processing and Management of Uncertainty in Knowledge-Based Systems, pages 480–491. Springer
  16. Wolpert, D. H. (2002). The Supervised Learning No-FreeLunch Theorems, pages 25–42. Springer London, London.
  17. Xiao, Y., Wang, H., Zhang, L., and Xu, W. (2014). Two methods of selecting gaussian kernel parameters for one-class svm and their application to fault detection. Knowledge-Based Systems, 59:75–84.
  18. Yang, K., van der Blom, K., Bäck, T., and Emmerich, M. (2019). Towards single-and multiobjective bayesian global optimization for mixed integer problems. In AIP Conference Proceedings, volume 2070, page 020044. AIP Publishing LLC.
  19. Zaefferer, M. and Bartz-Beielstein, T. (2016). Efficient global optimization with indefinite kernels. In Handl, J., Hart, E., Lewis, P. R., López-Ibáñez, M., Ochoa, G., and Paechter, B., editors, Parallel Problem Solving from Nature – PPSN XIV, pages 69–79, Cham. Springer International Publishing.