My research areas include:
My recent research focus is on building multimodal conversational agents that can reason about and interact seamlessly with humans in real time. Additionally, I am interested in the real-world applications of computer vision and machine learning algorithms in healthcare and remote sensing.
I am looking for exceptional candidates for various positions at MBZUAI: PostDocs, Research Engineers, PhD students (for fall 2025 intake), and Research Interns. PostDoc applicants should have strong experience in Multimodal Models and LLM/VLMs. Research engineers must demonstrate strong development skills through past projects (a background in LLM/VLM and Generative AI for healthcare is a plus). Research Interns and PhD candidates should possess a strong academic background in BS/MS programs, focusing on computer vision/machine learning, with relevant coursework and projects. Previous publications in top conferences such as CVPR, ECCV, ICCV, NeurIPS, ICLR, or ICML are desirable. If you are interested, please send your CV and GitHub profile link to me
I have had the privilege of collaborating with exceptional students, colleagues, and collaborators. Below is a partial list:
1. Sahal Shaji Sahal Shaji Mullappilly (PhD@MBZUAI, 2023- , MSc @MBZUAI, 2021-2023)
2. Jose Renato Restom Viera (PhD@MBZUAI, 2023- , MSc @MBZUAI, 2021-2023)
3. Daniya Abdul Kareem (PhD@MBZUAI, 2021- -)
4. Mohammad Ahmad Eid Mohamed Almansoori (PhD@MBZUAI, 2023- )
5. Yevheniia Kryklyvets (MSc@MBZUAI, 2023- )
6. Fawaghy Ahmed Saeed Mohamed Alshahmy (MSc@MBZUAI, 2023- )
7. Tooba Tahreem Sheikh (MSc@MBZUAI, 2023-)
8. Amrin Kareem (MSc@MBZUAI, 2022- 2024)
9. Mohammed Fazli Imam (MSc@MBZUAI, 2022- 2024)
10. Hosam Mahmoud Engendy (MSc@MBZUAI, 2022- 2024)
11. Shamma Sultan Saeed Alsaedi (MSc@MBZUAI, 2022- 2024)
1. Mustansar Fiaz ( Postdoc@ MBZUAI, Now with IBM Research)
2. Mohammad Khaled Almansoori (MSc@MBZUAI 2021-2023, Now with Abu Dhabi Police,)
3. Yahia Dalbah ( MSc@MBZUAI 2021-2023, Now with SAAB, UAE)
4. Sara Pieri (MSc@MBZUAI 2021-2023, Joined PhD at INRIA , France)
5. Abhishek Singh Gehlot (MSc@MBZUAI 2021-2023, Now with Shinobi Security, UAE)
6. Aidana (MSc@MBZUAI 2021-2023, Now with Abu Dhabi University, UAE)
7. Dhanalaxmi Gaddam (MSc@MBZUAI 2021-2023, Now with DP World, UAE)
8. Ankan Kumar (RA @MBZUAI 2020-2023, Now with University of Edinburgh, UK)
9. Amandeep Kumar (RA @MBZUAI 2022-2024, Now with John Hopkins, USA)
10. Guolei Sun (SE at IIAI, Now with ETH Zurich)
11. Jiale Cao (Visiting Postdoc at IIAI, now with Tianjin University, China)
My recent research focus on developing LLMs and LMMs for different applications. Few example projects are listed below:
This recent work published at ECCV 2024 is motivated by the need for integrating reasoning and grounding capabilities for intelligent perception systems, with potential for various applications such as robotics. For instance, an intelligent system should know where to hold a ‘kettle’ or which part of a ‘bottle’ to open in a fine-grained manner without the user explicitly naming the object or its parts. In this project, we introduced a novel problem setting called reasoning-based part segmentation/grounding on 3D point clouds. This task involves generating a part segmentation mask for a 3D object based on implicit textual queries that require complex reasoning. Our PARIS3D model leverages its reasoning capabilities to
(i) understand implicit instructions, such as referencing properties of an object or its parts without explicit articulation, and
(ii) explain or justify its responses, whether in generated text or predicted segmentation masks. To support future research on this important problem, we have introduced and open-sourced an evaluation benchmark and a dataset named RPSeg3D, comprising 2624 3D objects and over 60k instructions with part segmentation mask.
[Papr Link][Project Page/Code]
Many existing open-source and closed-source large multimodal models lack explicit grounding capability to accurately segment referred object regions. To address this limitation, we introduced the first model that can seamlessly output text responses intertwined with pixel-precise grounding segmentation masks by taking textual and optional visual prompts as input. The unique structure of this model supports a wide range of tasks, including grounded conversation generation (GCC), referring expression segmentation, image and region-level captioning, and vision-language conversations. To facilitate future research on this challenging problem, we propose a densely annotated Grounding-anything Dataset (GranD) that encompasses 7.5M unique concepts grounded in a total of 810M regions with segmentation masks.
[Paper], [Project Page/Code]
This project introduces an inclusive Large Multilingual Multimodal Model called Palo. Palo offers visual reasoning capabilities in 10 major languages, including English, Chinese, Hindi, Spanish, French, Arabic, Bengali, Russian, Urdu, and Japanese, that span a total of ~5B people (65% of the world population).
This project explores the challenge of designing accurate yet efficient Small Language Models (SLMs) for on-device processing on resource-constrained mobile devices. Our focus is to develop fully transparent, open-source, mobile-friendly LLMs with enhanced energy efficiency, a low memory footprint, and high response efficiency
Climate change is one of the most significant challenges we face together as a society. Creating awareness and educating policy makers the wide-ranging impact of climate change is an essential step towards a sustainable future. We propose a light-weight Arabic Mini-ClimateGPT that is built on an open-source LLM and is specifically fine-tuned on a conversational-style instruction tuning curated Arabic dataset Clima500-Instruct with over 500k instructions about climate change and sustainability.This project was later extended to Jais-Cliamte and Presented at COP28
[Paper] [Project Page]
(i) BiMediX
[Paper] [Project Page] [HF Demo] [Telegram Live Demo]
In this project, we introduce the first bilingual medical mixture of experts LLM designed for seamless interaction in both English and Arabic. Our model facilitates a wide range of medical interactions in English and Arabic, including multi-turn chats to inquire about additional details such as patient symptoms and medical history, multiple-choice question answering, and open-ended question answering. This model achieve state-of-the-art performance on multiple Arabic and English medical LLM evaluation benchmarks.
Our contributions are as follows:
In this work, we propose a few-shot colorectal tissue image generation method for addressing the scarcity of histopathological training data for rare cancer tissues. Our few-shot generation method, named XM-GAN, takes one base and a pair of reference tissue images as input and generates high-quality yet diverse images.
In this project, we develop novel segmentation approaches for 2D and 3D medical and microsopic images.
Selected Publications
1. Semi-supervised Medical Image Segmentation, MICCAI 2024,
[Paper], [Project Page/Code]
2. Microscopic Image segmentation, BMVC 2023 (Oral)
[Paper] [Project Page]
3. 3D Medical Image Segmentation, IEEE TAI, 2024
The proposed vMixer framework exploits explicit local and global volumetric features to better learn the shape-boundary details of the organs. We also provide an extensive study on the selection of architectural design that is adapted for 3-D medical segmentation from 2-D vision literature for better boundary localization. Finally, we exploit the transfer learning capabilities of the proposed vMixer where training data are limited.
[Paper], [Project Page/Code]
4. 3D Medical Image Segmentation, ISBI 2024 (Oral),
[Paper], [Project Page/Code]
Here we propose a Directional Window Attention based method for medical image segmentation
This project focus on developing novel architectuers for different visual recogntion applications such as image classification, object detection, person search, remote sensing change detection, and instance segmentation on images, videos, radar data and 3D point clouds.
Representative Publications:
1. High quality object detection and instance segmentation, CVPR 2020.
2. Real-time Instace Segmentation on Images and Videos, ECCV 2020
3. Video Instance Segmentation, ECCV 2022
4. Computationally Efficient Architectures for Mobile devices, ECCV Workshop, 2022
5. Radar Object Detection, WACV 2023
5. Person search
CVPR 2022 [Code] WACV 2024 Paper [Code]
6. Visual Recognition Architectures for Federated Learning, NeurIPS 2023
The objective of this project is to investigate novel learning approaches for practical deployment of visual recognition approaches, such as open-world learning, and learning with limited supervision (eg. Few-shot learning, zero-shot learning).
1. Semi-supervised Open-World Object Detection in Natural Images and Satellite Imagery AAAI 2024
2. Open-world 3D Indoor Instance Segmentation, NeurIPS 2023
3. Open-world Video Instance Segmentation, IJCV 2024
[Paper] [Project Page]
4. Few-shot Semantic Segmentation, ICML 2024
[Paper] [Project Page]
5. Object Counting with Image-level Supervision, CVPR 2019
For full list, visit: https://scholar.google.com/citations?user=bZ3YBRcAAAAJ&hl=en
• Ankan Kumar Bhunia, Salman Khan, Hisham Cholakkal, Rao Anwer, Fahad Khan, ”System and method for handwriting generation”, US Patent Granted, Num.:11756244, 2023.
• Hisham Cholakkal, G. Sun, FS Khan and L. Shao, ”Object counting and instance segmentation using neural network architectures with image-level supervision”, US Patent Granted.
• Hisham Cholakkal, J. Cao, R.M Anwer, FS Khan, Y. Pang, L. Shao, ”Dense and Discriminative Neural Network Architectures for Improved Object Detection and Instance Segmentation”, US Patent Granted, Num.:11244188, 2022.
• Hisham Cholakkal, Sanath Narayan, Arjun Jain, Shuaib Ahmed, Amit Bhatkal, M.B.R Reddy, Apurbaa Mallik, ”Method for identifying a hand pose in a vehicle”, US Patent Granted, Number 11887396, 2024.
• M Maaz, A Shaker, Hisham Cholakkal, S Khan, SW Zamir, RM Anwer, FS Khan, ”Sys-
tem and method for efficiently amalgamated CNN-transformer architecture for Mobile Visiob Applications”, US Patent App.18/078,657, 2024.
• Amandeep Kumar, Ankan Kumar Bhunia, Hisham Cholakkal, Sanath Narayan, Rao Muhammad Anwer, Fahad Khan, ”System and method of cross-modulated dense local fusion for few-shot image generation”, US Patent App. Number 17983952, 2024.
• Mustansar Fiaz, Hisham Cholakkal, Sanath Narayan, Rao Muhammad Anwer, Fahad Khan, ”System and method for attention-aware relation mixer for person search”, US Patent App. Number 17983741, 2023.
• Omkar Thawakar, Sanath Narayan, Hisham Cholakkal, Rao M. Anwer, Muhammad Haris, Salman Khan, Fahad Khan, ”System and method for video instance segmentation via multi-scale spatio-temporal split attention transformer”, US Patent App. Number 17983841, 202
Course Contents: The course content includes the following: (i) Introduction to fundamental image processing concepts such as filtering, edge detection, and color planes (ii) Introduction to computer vision concepts such as corner detection, blob detection, and SIFT (iii) Introduction to camera and optics, homogeneous coordinate system, camera parameters, extrinsic and intrinsic parameters, stereo matching, and 3D vision, (iv) Machine learning and deep learning fundamentals, and learning-based computer vision algorithms, (v) Introduction to human action recognition, human pose estimation, and other advanced computer vision techniques.
Course Offerings: This is a core course for the first semester MSc and PhD Computer Vision students at MBZUAI. From Fall 2023, this course is offered only for the MSc students. I have offered this course in the following semesters at MBZUAI:
Course Contents: The course covers the following topics: fundamentals of visual recognition, CNN-based visual recognition architectures, transformer-based visual recognition architectures, single-stage object detection, two-stage object detection, end-to-end object detection using transformers, remote sensing object detection, segmentation architectures, two-stage instance segmentation, single-stage instance segmentation, and video tracking.
Course Offerings: This is a core course for second-semester MSc and PhD Computer Vision students at MBZUAI. Starting from Spring 2024, this course is offered only to MSc students. I have offered this course in the following semesters at MBZUAI:
Course Contents: The course has two parts. The first part aims to build the essential computer vision background for first-semester PhD students, while the second part introduces advanced computer vision concepts. The first part covers an introduction to convolutional neural networks, vision transformers, and object detection and segmentation using convolutional neural networks and transformers. The second part introduces foundation models, vision-language models, fundamentals of large language models and large multimodal models, the segment anything model, efficient computer vision, remote sensing change detection, and image generation and diffusion models.
Course Offerings: This is a core course first semester PhD Computer Vision students at MBZUAI. I have offered this course in the following semesters.
I am looking for exceptional candidates for various positions at MBZUAI: PostDocs, Research Engineers, PhD students (for fall 2025 intake), and Research Interns. PostDoc applicants should have strong experience in Multimodal Models and LLM/VLMs. Research engineers must demonstrate strong development skills through past projects (a background in LLM/VLM and Generative AI for healthcare is a plus). Research Interns and PhD candidates should possess a strong academic background in BS/MS programs, focusing on computer vision/machine learning, with relevant coursework and projects. Previous publications in top conferences such as CVPR, ECCV, ICCV, NeurIPS, ICLR, or ICML are desirable. If you are interested, please send your CV and GitHub profile link to me
