A Blueprint for Automated ETL Documentation with an Active Metadata Utility Framework (AMUF): A Jenkins-Based Approach for Non- Containerized Environments

Abstract

The effectiveness of modern data strategy hinges on complex ETL pipelines, yet our ability to document these systems remains a critical failure point. Manual documentation is consistently neglected, leading to brittle data operations, eroded trust, and significant governance risks. This paper tackles this challenge head-on by exploring the paradigm of automated documentation driven by active metadata. We first establish the theoretical groundwork in data lineage and metadata manage- ment. We then propose a novel conceptual model, the Active Metadata Unification Framework (AMUF), to serve as an archi- tectural blueprint for a robust documentation ecosystem. Using a formal Multi-Criteria Decision Analysis (MCDA), we evaluate the key tools that underpin this framework—dbt, OpenLineage, Amundsen, and DataHub—assessing their architectural merits and limitations. From this analysis, we derive a practical, phased implementation guide. Our conclusion is clear: a federated system combining transformation-integrated documentation with a centralized, graph-based catalog offers the most potent solution. We close by outlining the next frontier of research: leveraging AI for semantic inference and predictive governance.