AI-powered ETL optimization: Recent advancements in self-tuning data pipelines
Santa Clara University, USA.
Review
Open Access Research Journal of Engineering and Technology, 2025, 08(02), 035-042.
Article DOI: 10.53022/oarjet.2025.8.2.0047
Publication history:
Received on 18 March 2025; revised on 26 April 2025; accepted on 29 April 2025
Abstract:
This article explores the transformation of Extract, Transform, Load (ETL) processes through artificial intelligence innovations, focusing on self-optimizing data pipelines that dynamically adjust execution parameters without human intervention. As global data volumes expand exponentially, traditional manual optimization approaches have become inadequate, prompting the development of intelligent alternatives. The article examines major advancements, including predictive resource allocation that anticipates processing needs before bottlenecks occur, adaptive scheduling algorithms that optimize job sequencing based on historical patterns, intelligent data partitioning strategies that automatically adjust to distribution characteristics, and sophisticated anomaly detection models that identify potential failures preemptively. These AI-driven technologies significantly reduce processing times, decrease operational costs, and enhance reliability across enterprise data environments while minimizing manual configuration requirements. The article also discusses emerging directions in reinforcement learning techniques and explainable AI that promise to further revolutionize ETL optimization.
Keywords:
Self-Tuning ETL; Predictive Resource Allocation; Adaptive Scheduling Algorithms; Intelligent Data Partitioning; Anomaly Detection
Full text article in PDF:
Copyright information:
Copyright © 2025 Author(s) retain the copyright of this article. This article is published under the terms of the Creative Commons Attribution Liscense 4.0