Federated Learning (FL) has emerged as a promising paradigm for collaborative machine learning, enabling multiple participants to train shared models while preserving data privacy. Despite it's potential, FL faces significant challenges including communication overhead, data heterogeneity, model heterogeneity, and performance degradation under non-independent and identically distributed (non-IID) data conditions. These challenges are particularly critical in sensitive domains such as healthcare, finance, and mobile computing, where both privacy preservation and model accuracy are essential. This thesis addresses these challenges through the integration of knowledge distillation (KD) techniques into FL frameworks. KD which transfers knowledge from complex teacher models to simpler student models, offers a powerful approach to mitigate communication costs and handle heterogeneous environments while maintaining model performance. The first contribution introduces FedFB, a communication-efficient FL framework designed for medical imaging applications. FedFB employs online ensemble KD with an auxiliary attention-convolution module to enable effective collaboration under non-IID data. The framework demonstrates substantial communication overhead reduction while maintaining high classification accuracy in fetal brain ultrasound analysis. The second contribution proposes FedAK, a semi-supervised one-shot FL framework that combines feature-level attention mechanisms with KD. Unlike traditional multi-round FL approaches, FedAK requires only a single communication round, significantly reducing network traffic. The attention-based aggregation module effectively handles model heterogeneity and generates informative ensemble features for training a global student model. Overall, through extensive experimental validation across multiple benchmark datasets and real-world applications, this thesis demonstrates that KD-based approaches can effectively address the fundamental challenges of FL. The proposed frameworks provide practical solutions that improve communication efficiency, robustness to data and model heterogeneity, and overall learning performance in medical imaging settings.