Data Engineering Lead
COMPANY OVERVIEW:
The company’s expertise is in transformation, data science, and change management that helps make businesses more efficient and effective, improve customer relationships and enhance revenue growth. Instead of focusing on multi-year, resource and time-intensive platform designs or migrations, they look deeper at the entire value chain to integrate strategies with impact.
This division serves the insurance, healthcare, banking, capital markets, utilities, retail and e-commerce, travel, transportation and logistics industries.
POSITION OVERVIEW:
The Data Engineer Lead will work with a team of people to provide data-driven, action-oriented solutions to business problems through statistical data mining, cutting edge analytics techniques and a consultative approach. Leveraging proprietary methodology and best-of-breed technology, the company takes an industry-specific approach to transform clients’ decision making and embed analytics more deeply into their business processes. The global footprint of nearly 5,000 data scientists and analysts assist client organizations with complex risk minimization methods, advanced marketing, pricing and CRM strategies, internal cost analysis, and cost and resource optimization within the organization.
Data Engineering Lead will be responsible for designing a unified data layer and reporting solution as a DE SME and guide the offshore team at various junctures to ensure timely delivery at best quality. The job holder will be working with technical and business stakeholders at different levels in understanding requirements and building end-to-end data warehousing solutions. Incumbent will be working in an onshore-offshore model to deliver agile, scalable, and efficient data solutions.
RESPONSIBILITIES:
- Lead a team of data engineers, analysts & BI developers as a DE SME to design and build efficient & scalable data warehouses for reporting purposes, compliant with best practices like MDM, RDM etc.
- Support the team as DE SME to apply the best approaches for large scale data movement, capture data changes, apply incremental data load strategies, exception handling and support development of reusable ETL modules.
- Gather available details & documentation on existing data feeds by collaborating with business and technical stakeholders; Support the team in building the feeds prioritization, standardization framework and data quality (DQ) plan.
- Oversee and contribute to the creation and maintenance of relevant data artifacts (data lineages, source to target mappings, high level designs, interface agreements, etc.).
- Work with data feeds from multiple geographies and understand their privacy & security requirements, standard contractual clauses (SCC), data sharing agreements and support the team creating amendments as per best practices.
- Mentor the team wherever required and verify that design best practices as well as coding and architectural guidelines, standards, and frameworks are adhered to by offering guidance, communicating risk, and addressing roadblocks as they arise.
- Serves as the primary interface between senior client management and senior leadership (VPs and SVPs)
- Provides thought leadership and delivers business insights to identify and resolve complex issues critical to clients’ success
- Manages communication between senior partners and clients to update project progress & solicit feedback on project deliverables
PREFERRED PROFILE:
- 10+ years IT experience in software development for data systems at scale (building data warehouse, data marts, data lakes, ETL, analytic solutions, etc.)
- 5+ years of hands-on experience in data warehouse architecture design and modelling to integrate data from diverse sources.
- At least 5 years of hands-on experience on various ETL tools (Talend – Preferred, Glue, Informatica etc.), Data Warehousing services (Amazon Redshift – Preferred, Snowflake etc.) and other cloud services like S3/Blob, Lambda/Functions etc.
- At least 4 years of experience in leading projects end-to-end and working directly with stakeholders in navigating complexities to enable successful delivery of scalable data pipelines in cloud supporting various applications.
- At least 4 years of proven track record of leading end to end projects and globally distributed teams (onshore-offshore model)
- Strong understanding of CI/CD principles and experience of CI/CD tools like GIT, Jenkins etc.
- Experience with orchestration tools like Airflow to automate and manage complex data pipelines and workflows.
- Strong experience on SQL on a variety of platforms (such as MS SQL, Athena, Redshift) and Python. Good to have experience on PySpark to develop and execute distributed data processing pipelines.
- Experience in encryption of datasets in data warehouses and exposure to data privacy & security regulations.
- Familiarity with infrastructure as a code for cloud resource management as code (CloudFormation, Terraform, AWS CDK).
- Demonstrable superior problem solving, interpersonal and people management skills. Organized and able to effectively communicate progress throughout project lifecycles.
- Bachelor’s or Master’s (preferred) degree in in a quantitative or technical field such as Computer Science, Information Technology, Computer Engineering or equivalent
LOCATION: New York City
Job ID# 1663529
Artemis invites you to subscribe to our free Job Alerts and “The Hunt” Blog for free insights on hiring and career development.
Artemis Referral Bonus – $1000! If you know someone for this job, please join our Referral Bonus Program.