Looking for a Senior Data Engineer in the USA

Primary Responsibilities of the Senior Data Engineer:

• Data Pipeline Development: Develop and maintain data pipelines that extract, transform, and load (ETL) data from various sources into a centralized data storage system, such as a data warehouse or data lake. Ensure the smooth flow of data from source systems to destination systems while adhering to data quality and integrity standards.

• Data Integration: Integrate data from multiple sources and systems, including databases, APIs, log files, streaming platforms, and external data providers. Handle data ingestion, transformation, and consolidation to create a unified and reliable data foundation for analysis and reporting.

• Data Transformation and Processing: Develop data transformation routines to clean, normalize, and aggregate data. Apply data processing techniques to handle complex data structures, handle missing or inconsistent data, and prepare the data for analysis, reporting, or machine learning tasks.

• Contribute to common frameworks and best practices in code development, deployment, and automation/orchestration of data pipelines.

• Implement data governance in line with company standards

• Partner with Data Analytics and Product leaders to design best practices and standards for developing and productionalizing analytic pipelines.

• Partner with Infrastructure leaders on architecture approaches to advance the data and analytics platform, including exploring new tools and techniques that leverage the cloud environment (Azure, Snowflake, others)

• Monitoring and Support: Monitor data pipelines and data systems to detect and resolve issues promptly. Develop monitoring tools, alerts, and automated error handling mechanisms to ensure data integrity and system reliability.

You will be rewarded and recognized for your performance in an environment that will challenge you and give you clear direction on what it takes to succeed in your role, as well as providing development for other roles you may be interested in.

Required Qualifications of the Senior Data Engineer:

Extensive experience designing data solutions including data modeling. Extensive hands-on experience developing data processing jobs (PySpark / SQL) that demonstrate a strong understanding of software engineering principles. Experience orchestrating data pipelines using technology like ADF, Airflow etc Experience working with both real-time and batch data, knowing the strengths and weaknesses of both and when to apply one over another. Experience building data pipelines on either AWS, Azure or GCP, following best practices in Cloud deployments

Experience working with Hive /HBase / Presto

Fluent in SQL (any flavor), with experience using Window functions and more advanced features. Understanding of DevOps tools, Git workflow and building CI/CD pipelines Ability to work with business and technical audiences on business requirement meetings, technical white boarding exercises, and SQL coding/debugging sessions.

Experience supporting big data pipelines.

Experience applying data governance controls within a highly regulated environment.

Preferred Qualifications of the Senior Data Engineer:

Bachelor's Degree or higher in Database Management, Information Technology, Computer Science or similar Experience working in projects with agile/scrum methodologies. Familiar with Azure Data Factory or Apache Airflow Familiar with Azure Databricks or Snowflake

Experience with shell scripting languages

Well versed in Python, in fulfilling multiple general-purpose use-cases, and not limited to developing data APIs and pipelines. Experience with Apache Spark and related Big Data stack and technologies Experience working with Apache Kafka, building appropriate producer/consumer apps. Familiarity with production quality ML and/or AI model development and deployment. Experience working with Kubernetes and Docker, and knowledgeable about cloud infrastructure automation and management (e.g., Terraform)