● Design and implement complex data pipelines using Apache Airflow, ensuring scalability, reliability, and efficiency.
● Work closely with data scientists to integrate AI and machine learning models into production pipelines.
● Manage and optimize data workflows and schedules, ensuring timely and accurate data processing.
● Develop and maintain data lake and data warehouse architectures, supporting a wide range of data types and formats.
● Implement best practices for data governance, security, and compliance throughout the data pipeline.
● Monitor, troubleshoot, and optimize performance of data pipelines and infrastructure.
● Stay abreast of emerging technologies and techniques in data engineering and AI, recommending improvements to our processes and systems.