On Optimizing Multimodal Jailbreaks for Spoken Language Models

This paper demonstrates that Spoken Language Models (which process both speech and text) can be effectively attacked by simultaneously manipulating both audio and text inputs together, rather than just one modality alone. The researchers developed JAMA, a technique that optimizes adversarial perturbations across both modalities at the same time, and showed it achieves 1.5 to 10 times higher attack success rates than previous single-modality attacks. The findings highlight that protecting these AI systems requires considering both input types together, not separately.