Journal of Digital Engineering and Business Management

Scholarly, Peer-Reviewed, and Fully Refereed Open Access Multidisciplinary Quarterly Research Journal.

ISSN (Online) : 3107-7897 | D3 Publishers | editor@jdebm.com | jdebmjournal@gmail.com
📢 Now Accepting Papers for Upcoming Issue | ISSN: 3107-7897 | Contact: editor@jdebm.com | Visit: www.jdebm.com

Machine Learning-Based Prototyping Of Graphical User Interfaces For Mobile Apps

Authors: J. Swathi, S. Keerthana, A. Pavani, B. Nikhitha, J. Ramya Sri

Abstract

Mobile app developers spend significant time manually crafting GUI layouts from verbal requirements, a tedious process prone to iteration. While image-to-code tools and voice assistants exist, no system directly transforms spoken design instructions into production-ready mobile interfaces. Voice-to-GUI3 introduces a novel pipeline that converts natural voice commands into structured Android XML through integrated speech recognition, transformer-based semantic analysis, layout tree construction, and automated code synthesis. From instructions like “login screen with email field, password, blue submit button”, the framework extracts UI elements, infers spatial relationships, andgenerates valid LinearLayout hierarchies in seconds. Testing on a custom voice-annotated prototype dataset shows strong component recognition and 67% faster prototyping compared to manual design workflows. This work bridges speech processing, natural language UI understanding, and software automation, enabling rapid interface creation from spoken ideas alone.

Keywords

Voice-driven UI synthesis; Mobile GUI automation; Transformer-based layout generation; Speech-to-layout conversion; Android XML synthesis; Semantic interface parsing; Hierarchical layout modeling; Hands-free prototyping; AI interface design.

Article Information

Volume: 2
Issue: 1
Published Date: 21/01/2026
DOI: https://doi.org/10.5281/zenodo.19091942
Scroll to Top