Pulmonary Embolism (PE) has diverse manifestations with different etiologies such as venous thromboembolism, septic embolism, and paradoxical embolism. In this study, a novel attention-based multi-task model is proposed for PE segmentation and detection from Computed Tomography Pulmonary Angiography (CTPA) images. A Y-Net architecture is used to implement this model, which facilitates segmentation and classification jointly, improving performance and efficiency. It is leveraged with Multi Head Attention (MHA), which allows the model to focus on important regions of the image while suppressing irrelevant information, improving the accuracy of the segmentation and detection tasks. The proposed PE-YNet model is tested with two public datasets, achieving a maximum mean detection and segmentation accuracy of 99.89% and 99.83%, respectively, on the CAD-PE challenge dataset. Similarly, it also achieves a detection accuracy of 99.75% and a segmentation accuracy of 99.81% on the FUMPE dataset. Additionally, sensitivity analysis also shows a high sensitivity of 0.9885 for the localization error ɛ = 0 for the CAD-PE dataset, demonstrating the model's robustness against false predictions compared to state-of-the-art models. Further, this model also exhibits lower inference time, size, and memory usage compared to representative models. An automated PE-YNet tool can assist physicians with PE diagnosis, treatment, and prognosis monitoring in the clinical management of CoVID-19.