Intrusion Detection Systems (IDS) are critical for securing modern communication networks, especially as cyber threats become increasingly complex. However, existing evaluation methodologies for machine learning-based IDS lack standardisation, often overlook best practices, and primarily focus on performance within specific datasets without addressing broader issues. This thesis addresses these limitations by first defining a comprehensive theoretical framework for evaluating ML-based IDS. Building on this theoretical foundation, we introduce FREIDA, a tool that implements the framework, emphasising completeness, reliability, and reproducibility. FREIDA integrates both traditional IDS evaluation methods and machine learning best practices, with a particular focus on the critical relationship between data selection and evaluation choices. Our approach also extends the evaluation process to include assessments of robustness against adversarial attacks and privacy considerations, providing a more holistic evaluation of IDS resilience. Through the formalisation and implementation of our evaluation framework, we aim to standardise IDS evaluation methods and promote the development of resilient and adaptive intrusion detection systems for next-generation networks.