| dc.description.abstract | Generative artificial intelligence (AI) has demonstrated transformative potential in creative and technical fields, yet its application to engineering design remains underdeveloped. Unlike domains where AI outputs can be directly consumed, engineering design demands integration across heterogeneous tools, multimodal data, and highly structured workflows. This thesis develops and evaluates AI-driven approaches for enabling copilot-style systems that assist engineers throughout the early stages of design, where decisions have the greatest impact on cost and performance. We identify and address three central challenges: the need for user control over abstract generative processes, the scarcity of high-quality engineering datasets, and the complexity of integrating AI into diverse design toolchains. Our first contribution is Sketch2Prototype, a multi-stage framework that transforms conceptual sketches into text, images, and manufacturable 3D meshes. Evaluated on a dataset of 1,087 sketches, the system produces more diverse and manufacturable prototypes than direct sketch-to-3D methods, while enabling iterative refinement through a controllable intermediate text stage. Our second contribution is VideoCAD, a synthetic dataset of over 41,000 annotated CAD modeling videos—up to twenty times longer in action horizon than prior UI agent datasets—capturing pixel-precise, long-horizon interactions in a professional CAD environment. We benchmark state-of-the-art behavior cloning models and large language models on VideoCAD, and introduce VideoCADFORMER, a transformer-based architecture that achieves superior performance on long-horizon CAD action prediction. Finally, we present VisionCAD, a fine-tuned Large Language Model (LLM) that constructs CAD Generation code from point cloud and image data, trained with a dataset of over two million image, point cloud, and CADQuery triplets. Together, these contributions demonstrate that generative AI, multimodal learning, and large-scale dataset generation can be combined to accelerate design exploration, improve manufacturability, and integrate seamlessly into engineering workflows. By addressing both the data and workflow bottlenecks, this work lays the foundation for AI copilots that enhance productivity, creativity, and precision in engineering design. | |
