The emergence of large-scale applications in High-Performance Computing (HPC) and Artificial Intelligence (AI) is driving an increasing demand for computational resources. From specialized Application-Specific Integrated Circuits (ASIC) to AI accelerators for neural-network training, computing architectures are becoming more and more complex, leading to important challenges in their designs. Before actually manufacturing the chips, designs are simulated virtually to evaluate and validate their performance. A key task for designers is to identify the hardware parameters that lead to the best Performance-Power-Area (PPA) trade-offs through a series of simulations. Given the combinatorial explosion of the design space and the time-consuming nature of design simulations, all the difficulty for designers lies in how to effectively explore the design space to reach optimal PPA, a challenge known as design space exploration (DSE) problem. In this thesis, we are interested in the automation of design space exploration using operations research and machine learning techniques. While traditional design space explorations are often conducted manually based on past experiences and tribal knowledge, the limitations of human analysis in complex decision-making are pushing automated DSE techniques to the forefront. We reviewed, from a broader perspective, different automated DSE techniques based on optimization algorithms, highlighting their advantages and drawbacks. In the context of time-consuming design evaluations, we decided to leverage Bayesian Optimization (BO) as one of the most suitable exploration algorithms. We detailed the different components that make Bayesian optimization work from its standard single-objective version to its multi-objective one. The main contributions of this thesis are the proposal and implementation of two automated DSE algorithms based on multi-objective Bayesian optimization and have been illustrated for PPA optimization of the system-level design of HPC Arm Neoverse V1 Multi-Processor System-on-Chip (MPSoC) architecture. The first contribution, utilizing random-forest-based Bayesian optimization, aims to provide a reliable baseline that surpasses the performance of the previous genetic-algorithm-based automated DSE developed in our laboratory. The second contribution which is at the heart of this work focuses on developing an advanced BO-based DSE algorithm. To achieve so, we take inspiration from the results of NeurIPS 2020 black-box optimization challenge. By leveraging the key conclusions and insights of the challenge, we developed a new exploration algorithm named SEBO for Stage-Enhanced Bayesian Optimization by improving the three main stages of the optimization process. These enhancements feature an intelligent initial sampling for informative surrogate model initialization, a localized ensemble surrogate modeling for scalable and accurate objective predictions, and a parallel hypervolume-based acquisition function for parallel coherent design suggestions. SEBO has been compared to other state-of-the-art exploration algorithms and demonstrated its ability to return a more qualitative and diversified Pareto front within a reasonable exploration time, ultimately making a strong automated design space exploration technique.