Machine Learning-Based Prototyping Of Graphical User Interfaces For Mobile Apps
Abstract
Mobile app developers spend significant time manually crafting GUI layouts from verbal requirements, a tedious process prone to iteration. While image-to-code tools and voice assistants exist, no system directly transforms spoken design instructions into production-ready mobile interfaces. Voice-to-GUI3 introduces a novel pipeline that converts natural voice commands into structured Android XML through integrated speech recognition, transformer-based semantic analysis, layout tree construction, and automated code synthesis. From instructions like “login screen with email field, password, blue submit button”, the framework extracts UI elements, infers spatial relationships, andgenerates valid LinearLayout hierarchies in seconds. Testing on a custom voice-annotated prototype dataset shows strong component recognition and 67% faster prototyping compared to manual design workflows. This work bridges speech processing, natural language UI understanding, and software automation, enabling rapid interface creation from spoken ideas alone.