Tagging metadata and tracking SQL lineage manually is often tedious and prone to mistakes in data engineering. Although essential for compliance and data governance, these tasks usually involve lengthy manual checks of datasets, table structures, and SQL code.
Thankfully, advancements in large language models (LLMs) such as GPT-4 provide a smarter and more efficient solution. This guide helps beginner data engineers learn how to use LLMs with tools like OpenMetadata, dbt, Trino, and Python APIs to automate metadata tagging (like identifying PII) and lineage tracking for SQL changes. Let’s explore the details.