Title: Automated ETL Pipeline: Batch Processing and CSV to Parquet Optimization

Title: Automated ETL Pipeline: Batch Processing and CSV to Parquet Optimization Overview: A high-performance data engineering solution designed to automate the ingestion and transformation of large-scale datasets. The project focuses on building a scalable ETL (Extract, Transform, Load) pipeline that streamlines the transition from legacy storage to modern, cloud-optimized formats. Key Technical Highlights: Batch Processing Automation: Developed a Python-driven system using the OS and Pandas libraries to automatically detect, process, and migrate multiple datasets in bulk. Storage Architecture Optimization: Engineered a seamless conversion from CSV to Apache Parquet, leveraging columnar storage to reduce disk footprint by -70% and drastically improve I/O performance. Data Integrity & Pipeline Safety: Implemented robust error-handling and directory management to ensure reliable data flow and prevent processing failures Big Data Readiness: Focused on high-speed data engines (fostporquet) to ensure the pipeline is capable of handling complex analytical workloads