Diversity meets Relevancy: Multi-Agent Knowledge Probing for Industry 4.0 Applications

Industrial data scientists modeling an asset's condition need to build domain understanding by asking questions about a given asset. Some example asset questions are what failure modes can it experience, under which operating conditions, and how the manufacturer and weather affect. Traditionally, the main source of domain information comes from Subject Matter Experts (SMEs) and Failure Modes and Effects Analysis (FMEA) documents which are not always available and may not be detailed enough to cover different external factors (e.g., operating mode, manufacturer, weather). Now that Large Language Models (LLMs) have became a commodity, this gives us a big opportunity to leverage them to bridge this gap. Inspired by other's work on LLM knowledge probing, we present a Multi-Agent System (MAS) specialized on aiding industrial data scientists guide their modeling decisions. One challenge we address is the generated linguistic diversity and question relevance, which we optimize by using popular information diversity metrics and a grounded relevancy classifier. We continuously monitor the set of newly generated instruction sets at the end of each round, compare the linguistic diversity against common baselines and show high generated knowledge coverage on the downstream FMEA task. We also conduct user studies to validate the quality of the questions. We finally present the real-world implications of providing diverse asset specific information to aid data scientist's modeling decisions through our deployed MAS. Through the deployed system, we show its generalizability to different assets and extendibility to more downstream tasks like work order scheduling, failure mode sensor analysis and machine learning model recipes generation.