Data, Retrieval & Knowledge
A large language model framework extracts high-quality knowledge graph triplets without parameter tuning and handles larger schemas than earlier methods, benchmark tests show
In a 2024 paper presented at the Conference on Empirical Methods in Natural Language Processing, researchers at the National University of Singapore describe a three-phase method for building knowledge graphs from text. Knowledge graphs store facts as triplets: a subject, a relationship, and an object. The method first extracts open information, then defines a schema, the set of allowed types and relationships for the graph, and then canonicalizes the facts into consistent entries. On three knowledge graph construction benchmarks, the framework produced high-quality triplets without any parameter tuning, meaning its learned settings were not adjusted for the task, and worked with much larger schemas than earlier methods.
What problem does the paper address?
Earlier methods often needed to include the whole schema in the instructions given to a large language model so the model would produce valid triplets. When a schema grew larger and more complex, it could exceed the model's context window, the amount of text the model can handle at once. In some situations no fixed schema exists at all. The authors wanted a method that could still build a useful graph with a short schema it created itself.
How does the three-phase method work?
The authors call the framework Extract-Define-Canonicalize. Its first phase performs open information extraction, pulling possible facts from text without requiring a fixed schema. The second phase defines a schema, and the third phase canonicalizes the extracted facts so the same thing is recorded the same way. The method can use a target schema when one is available. When no fixed schema exists, it builds a schema automatically and then standardizes the extracted facts with that schema. A trained component also retrieves schema elements that are relevant to the input text, and the authors report that this improves extraction performance.
What did the benchmark tests show?
On three knowledge graph construction benchmarks, the method extracted high-quality triplets without any parameter tuning. It also handled significantly larger schemas than prior methods.
What else should a reader know?
What is a knowledge graph triplet?
A triplet is a small statement with three parts: a subject, a relationship, and an object. For example, a graph could store the fact that a drug treats a disease as one triplet.
When does the method create its own schema?
The method creates a schema automatically when no fixed schema is provided. If a target schema is available, it can use that schema instead.
Why was the retrieval component added?
The authors added a trained component that selects schema elements relevant to the input text. They report that this improved how well the language model extracted facts.
Limits and what's next
The evidence comes from three knowledge graph construction benchmarks, as the abstract describes them. The abstract does not report testing on the broader real-world text that motivated the work, and it reports no independent replication. The abstract also does not say how many documents or which domains the benchmarks cover. This is one study, and its result is tied to those benchmarks.
Source: Bowen Zhang and Harold Soh, “Extract, Define, Canonicalize: An LLM-based Framework for Knowledge Graph Construction,” Conference on Empirical Methods in Natural Language Processing (EMNLP), 2024.