Enhancing Medical Image Segmentation for Skin Lesions by Deep Learning-Based Boundary Optimization

Abstract

Skin cancer continues to be among the most widespread forms of cancer globally, making early and precise detection essential for improving survival rates and treatment outcomes. Identifying and accurately outlining pigmented skin lesions in dermoscopic images is a critical step in diagnosis. However, traditional manual evaluation depends heavily on the expertise of dermatologists, making it time-consuming and often inconsistent due to subjective judgment. With the rise of deep learning, automated analysis of dermoscopic images has significantly improved, offering more reliable and scalable solutions. Despite this progress, challenges still exist, particularly in detecting lesions with unclear boundaries, irregular shapes, and low contrast. To overcome these limitations, this study introduces Boundary-Optimized TransUNet (BOTU-Net), a hybrid model that combines convolutional neural networks with Transformer-based learning. This architecture enhances the standard TransUNet by integrating an Edge-Aware Attention Module (EAAM), which helps the model focus more effectively on boundary details. The model is evaluated using the HAM10000 dataset, which includes over 10,000 dermoscopic images across seven categories of pigmented skin lesions. To address class imbalance and improve robustness across different skin tones, the study employs several techniques, including Canny edge feature integration, data augmentation, SMOTE-Tomek resampling, and transfer learning. Experimental results based on five-fold cross-validation and independent testing demonstrate that BOTU-Net outperforms baseline models. By enabling earlier and more accurate detection of skin cancer, it can contribute to better patient care and improved accessibility across diverse populations.