Automated Single-Turn Solution Recommendation System for Software IT Support Tickets

Clients wishing to implement generative AI in the domain of IT Support and AIOps face several critical issues: lengthy, vague, and insufficient customer input; model domain coverage; and hallucination. Customers often submit lengthy or incomplete support tickets not suitable for downstream tasks such as retrieval and question-answering in their raw form. For solution recommendation generation itself, clients might choose not to use larger proprietary models such as GPT-4 due to cost and privacy concerns and so are limited to smaller models with potentially less domain coverage that do not generalize to the client's specific technical domain. Retrieval augmented generation is a common solution that addresses both of these issues: a retrieval system first retrieves the necessary domain knowledge which a smaller generative model leverages as context for generation. We present a system developed for a client in the IT Support domain for support case solution recommendation that combines 1) an encoder-only model for classification 2) a generative large language model for query generation 3) retrieval augmented generation (RAG) for answer generation and 4) ensemble generative large language model and k-precision for evaluating the answer groundedness to combat hallucination. The paper describes architecture details, data collection and annotation, development journey and preliminary validations, deployment evaluation, and finally lessons learned.