Intelligent Data Pipeline Orchestration: Combining Predictive Modeling, LLMs, and RL for Autonomous Data Engineering Systems

The contemporary data engineering systems face challenges of operational complexity, such as unpredictable system failures, resource competition, and inadequate automation. This paper suggests a framework for intelligent orchestration of data processing pipelines through using machine learning for predicting system failure, reinforcement learning for scheduling tasks, and large language models for interpreting and automating corrective actions. A dataset of 1,000 samples, each with 18 network and behavioral features, is employed to train and test four supervised learning models, namely, Random Forest, Support Vector Machine (SVM), Gradient Boosting, and Logistic Regression. Random Forest trained on SMOTE-balanced dataset performs with the accuracy and macro F1-score of 94.7% and approximately 0.95, respectively. The developed three-layer architecture provides a possible approach to constructing self-healing data processing pipelines with intelligent scheduling mechanisms. Thus, the findings confirm the efficiency of the predictive layer as a key element of autonomous data processing pipelines.