Development of an Automatic Summarization System based on Large Language Models for Annual Report Analysis

Abstract

The increasing interest in stock market investment in Indonesia has highlighted a significant challenge for retail investors: the difficulty of analyzing lengthy and complex corporate annual reports. These documents, essential for fundamental analysis, are often hundreds of pages long and contain detailed narrative sections that require considerable time and effort to comprehend. This research addresses this issue by developing an automatic summarization system using a Large Language Model (LLM) to generate concise and insightful summaries of such reports. The primary objective was to develop and evaluate an LLM-based system specifically adapted for the structure and content of annual reports. The method involved creating a tailored dataset comprising 2,008 narrative text excerpts and their corresponding manual summaries sourced from the annual reports of companies listed on the Indonesia Stock Exchange (IDX). The open-source Llama-3.2-3B-Instruct model was then fine-tuned using the Parameter-Efficient Fine-Tuning (PEFT) technique, specifically Low-Rank Adaptation (LoRA). The research results demonstrated a significant improvement in the model's performance after fine-tuning. Quantitative evaluation using ROUGE metrics showed a relative increase of 18.63% in ROUGE-1, 44.45% in ROUGE-2, and 33.83% in ROUGE-L compared to the base model. Qualitative analysis confirmed that the fine-tuned model was capable of generating informative and relevant summaries aligned with the context of annual report analysis. In conclusion, this study demonstrates that fine-tuning LLMs with document-specific data is an effective approach for specialized tasks such as annual report summarization.