Skills and Experience:
- 5+ years of experience in data engineering or a similar role.
- At minimum a bachelor’s degree in computer science, information technology, or a related field.
- In-depth experience with the Microsoft Azure Platform (Data Factory, ADLS, etc.).
- Advanced knowledge of Azure Databricks and Delta Lake.
- Knowledgeable in the medallion architecture and other best practices.
- Experience with relational databases (e.g., SQL Server).
- Advanced in coding (e.g., PySpark, and SQL).
- Experience with version control (Git) and Unit Testing.
- Ability to translate business requirements to code.
- Proficient in articulating technical solutions to non-technical stakeholders.
- Experience with agile teams and Scrum.
As a Senior Data Engineer at LIPTON Teas and Infusions, you will play a pivotal role in shaping and maintaining our advanced data infrastructure. Your expertise in ETL processes, data ingestion, and processing will ensure the reliability and robustness of our data foundation. You will collaborate closely with BI developers, data scientists, product owners, and senior stakeholders to create impactful data & analytics solutions. Additionally, you will implement best practices to enhance data quality and manage metadata, while also leveraging software engineering principles such as version control and CI/CD to drive continuous improvement. Your innovative approach and attention to cost management in our cloud environment will support our mission of data excellence.
Key Responsibilities:
- Leading the development of data infrastructure to extract, transform, and load (ETL/ELT) data.
- Supporting BI developers and data scientists to build data & analytics solutions.
- Working with product owners and (senior) stakeholders to clarify business requirements.
- Responsible for the ingestion and processing enterprise-wide data through our medallion architecture.
- Closely collaborate with Data Engineering Lead to build and enhance the data engineering framework.
- Managing and optimizing periodic data refreshes through data pipelines (scheduled jobs).
- Designing and implementing data management practices to improve data quality and meta data.
- Leveraging software engineering best practices such as version control (Git) and CI/CD (DevOps).
- Continuously strengthening our data foundation through experimentation and innovation.
- Monitoring the cost associated with the cloud environment, data processing, and data computation.
