Refining LLM-based COBOL-to-Java Translation via Natural Language Summary Augmentation

COBOL remains widely used in mainframe environments, making the migration of legacy applications to modern languages such as Java essential for improving maintainability, reducing technical debt, and addressing knowledge loss as experienced developers retire. Large Language Models (LLMs) show promise for automating this migration, but their effectiveness is limited by COBOL’s domain-specific syntax and the scarcity of high-quality training data. We propose enhancing COBOL-to-Java translation by augmenting source code with natural language summaries, enabling LLMs to leverage their stronger natural language understanding capabilities. We evaluate this approach on two complementary datasets: an open-source CodeNet benchmark (727 samples) and a curated enterprise-grade benchmark (571 samples). We further develop routing strategies that dynamically select between baseline and summary-augmented pipelines to balance translation quality and computational cost. Results with different LLMs show substantial improvements for low-quality baseline translations: summary augmentation improves 36% of eligible CodeNet samples and 50% of low-scoring enterprise benchmark samples, demonstrating effectiveness on the most challenging cases. Additionally, for enterprise benchmark, a simple threshold-rule routing strategy achieves up to 8.75% improvement in translation quality with 0.7 additional LLM calls per sample. Overall, this work presents a practical and scalable method for enterprise COBOL-to-Java modernization, reducing manual review and refinement effort.