Performance evaluation of large language models for automated knowledge graph generation

Cloud systems generate large and heterogeneous raw data logs containing information related to infrastructure, application, and security. Transforming these logs into RDF triples enables their integration into knowledge graphs, improving interpretability, root-cause analysis, and cross-service reasoning. Large Language Models (LLMs) offer a promising approach to automate RDF knowledge graph generation. However, their effectiveness on complex cloud logs remains largely unexplored. In this paper, we evaluate multiple LLM architectures and prompting strategies for automated RDF extraction using a controlled framework with two pipelines for systematically processing semi-structured log data. The extraction pipeline integrates multiple LLMs to identify relevant entities and relationships, automatically generating subject-predicate-object triples. These outputs are evaluated using a dedicated validation pipeline with both syntactic and semantic metrics to assess accuracy, completeness, and quality. Due to the lack of public ground-truth datasets, we created a reference Log-to-KG dataset from OpenStack logs using manual annotation and ontology-driven methods, enabling objective baseline. Few-Shot learning achieved the best overall performance in our evaluation, with LLaMA achieving a 99.35% F1 score and 100% valid RDF output while Qwen, NuExtract, and Gemma also perform well under Few-Shot prompting, with Chain-of-Thought approaches maintaining similar accuracy. One-Shot prompting offers a lighter but effective alternative, while Zero-Shot and strategies such as Single-Prompt Tree-of-Thought, and Generate-Multiple-Then-Vote showed consistently lower performance. The top-performing LLMs, LLaMA and Qwen, with One-Shot and Few-Shot prompting were evaluated on a different dataset, to assess transferability, performance stability and efficiency in terms of latency, token generation, and GPU memory usage. Our work highlights the importance of contextual examples and prompt design for accurate RDF extraction and reveal model-specific limitations across LLM architectures offering practical guidelines and a foundation for future knowledge graphs extraction from unstructured data research.