Generative AI for smarter CCAM: enhancing perception, decision-making, and validation (CCAM Partnership)

Expected Outcome: Project results are expected to contribute to all the following expected outcomes: Availability and integration of advanced, trustworthy, energy-efficient perception systems, exploiting technological advancements of Generative AI (GenAI) to enhance situational awareness and support safe decision-making; Enhanced Vulnerable Road User (VRU) safety, based on elevated, more temper-proof perception and understanding of their behaviour and intention predictions; Enhanced robustness of CCAM systems - both on-board and on the infrastructure side - in critical situations due to their training, virtual testing and validation in scenarios generated by GenAI, complementing existing scenario databases for the testing and validation of CCAM systems; Enhanced understanding of the relevance and limitations of using GenAI for CCAM; Tools and harmonised approaches for the use of GenAI in mobility technology development, training and validation, as well as for systemic applications such as traffic management and remote control, integrating them into existing approaches. Scope: Pilots and demonstrations using Level 3 and 4 vehicle services face major challenges in perception and decision making, highlighting the necessity for low-latency solutions that enhance responsiveness and situational awareness in real-time operating conditions. This is especially relevant for driving in more complex environments like urban areas, where environmental variance is higher and where new scenarios can be regularly encountered. Furthermore, there is the need to limit the latency, bandwidth and energy use for on-board calculations, as well as the need to enhance the security, privacy and reliability (e.g. scene understanding and prediction of near-future scenario development). For rapid decision-making in interactions with VRUs, this is essential for implementing CCAM-enabled solutions and ensuring scalability. At the same time, developments of sector-agnostic technologies show advancements -such as GenAI- that can be beneficial for CCAM. First exploratory steps can be expected from a project funded under HORIZON-CL5-2023-D6-01-02 [1] regarding the potential in the virtual generation of edge cases, which could be used for the development, training, virtual testing and validation of CCAM systems. Further advancements in GenAI applications specifically for the CCAM domain need to be developed, trained and validated [2] . Thus, proposed actions shall include approaches to exploit further technological advancements for CCAM. Major steps are needed to advance to highly advanced,…