Enhancing LLM Capability to Generate a Problem Statement in Mission Engineering Using RAG
College
College of Engineering & Technology (Batten)
Department
EMSE
Graduate Level
Doctoral
Graduate Program/Concentration
PhD/Systems Engineering
Presentation Type
Oral Presentation
Abstract
The first phase of the Mission Engineering (ME) process, as outlined in the Mission Engineering Guide (MEG), is critical for defining the Mission Problem or Opportunity. This phase involves establishing the mission's purpose, formulating investigative questions, and identifying decision needs to guide subsequent analysis and system integration(DoD MEG 2.0, 2023). However, traditional approaches to problem definition often rely heavily on manual processes, which can be time-intensive and prone to gaps in knowledge representation.
This research explores the application of Refined Augmented Generation (RAG) to enhance the capabilities of large language models (LLMs) in generating precise and actionable problem statements during this critical ME phase. By leveraging RAG's ability to combine retrieval-based grounding with generative AI capabilities, I aim to dynamically incorporate real-time, domain-specific knowledge from curated databases and external sources into the problem-framing process. This approach ensures that problem statements are not only data-driven but also aligned with operational realities and decision-makers needs.
RAG is an advanced AI framework that combines the generative capabilities of LLMs with real-time retrieval of external, domain-specific, or up-to-date knowledge, enhancing the accuracy, relevance, and contextual understanding of AI-generated responses by grounding them in factual data. RAG integrates retrieval and generation by fetching relevant information from external sources such as databases, documents, or the web before generating a response, ensuring outputs are informed by accurate and current data rather than relying solely on the static knowledge embedded in the LLM. This approach minimizes "hallucinations," where LLMs generate plausible but incorrect information by grounding responses in retrieved facts. Additionally, RAG enables AI systems to provide real-time and domain-specific knowledge by accessing updated external knowledge bases, allowing specialization in areas such as law or medicine without requiring extensive retraining (Lewis et al., 2020).
Human-Centered Design (HCD) is a design approach that prioritizes user needs and experiences, focusing on understanding and addressing core problems rather than symptoms (JND, 2019). It involves iterative testing and refinement to ensure solutions meet user requirements effectively (Gillingham et al., 2023). Key principles include empathy, problem-solving, and user collaboration throughout the design process (Univ. of Min./CCAPS, 2024).
The proposed methodology integrates RAG with Human-Centered Design (HCD) principles to ensure that end-user inputs and stakeholders' collaboration are central to defining mission purpose and investigative questions. The expected outcomes include a structured framework for using RAG-enhanced LLMs in ME workflows, improved efficiency in generating problem statements, and enhanced alignment between operational requirements and system capabilities. This research contributes to advancing mission engineering practices by demonstrating how AI-driven tools can streamline early-phase processes while maintaining a focus on usability, adaptability, and mission success.
Keywords
Mission Engineering, Problem statement, Refine augmented generation, LLMs, Human-centered design, AI-driven problem framing, Decision needs analysis, Real-time knowledge access
Enhancing LLM Capability to Generate a Problem Statement in Mission Engineering Using RAG
The first phase of the Mission Engineering (ME) process, as outlined in the Mission Engineering Guide (MEG), is critical for defining the Mission Problem or Opportunity. This phase involves establishing the mission's purpose, formulating investigative questions, and identifying decision needs to guide subsequent analysis and system integration(DoD MEG 2.0, 2023). However, traditional approaches to problem definition often rely heavily on manual processes, which can be time-intensive and prone to gaps in knowledge representation.
This research explores the application of Refined Augmented Generation (RAG) to enhance the capabilities of large language models (LLMs) in generating precise and actionable problem statements during this critical ME phase. By leveraging RAG's ability to combine retrieval-based grounding with generative AI capabilities, I aim to dynamically incorporate real-time, domain-specific knowledge from curated databases and external sources into the problem-framing process. This approach ensures that problem statements are not only data-driven but also aligned with operational realities and decision-makers needs.
RAG is an advanced AI framework that combines the generative capabilities of LLMs with real-time retrieval of external, domain-specific, or up-to-date knowledge, enhancing the accuracy, relevance, and contextual understanding of AI-generated responses by grounding them in factual data. RAG integrates retrieval and generation by fetching relevant information from external sources such as databases, documents, or the web before generating a response, ensuring outputs are informed by accurate and current data rather than relying solely on the static knowledge embedded in the LLM. This approach minimizes "hallucinations," where LLMs generate plausible but incorrect information by grounding responses in retrieved facts. Additionally, RAG enables AI systems to provide real-time and domain-specific knowledge by accessing updated external knowledge bases, allowing specialization in areas such as law or medicine without requiring extensive retraining (Lewis et al., 2020).
Human-Centered Design (HCD) is a design approach that prioritizes user needs and experiences, focusing on understanding and addressing core problems rather than symptoms (JND, 2019). It involves iterative testing and refinement to ensure solutions meet user requirements effectively (Gillingham et al., 2023). Key principles include empathy, problem-solving, and user collaboration throughout the design process (Univ. of Min./CCAPS, 2024).
The proposed methodology integrates RAG with Human-Centered Design (HCD) principles to ensure that end-user inputs and stakeholders' collaboration are central to defining mission purpose and investigative questions. The expected outcomes include a structured framework for using RAG-enhanced LLMs in ME workflows, improved efficiency in generating problem statements, and enhanced alignment between operational requirements and system capabilities. This research contributes to advancing mission engineering practices by demonstrating how AI-driven tools can streamline early-phase processes while maintaining a focus on usability, adaptability, and mission success.
Comments
References