karthick

Data Engineer

Virtusa

Jul 2022 - Present

Developed and maintained end-to-end ELT/ETL workflows using Python, PySpark, and SnapLogic to process large-scale enterprise datasets. Spearheaded data ingestion strategies using Fivetran to automate zero-code data pipelines from operational databases into centralized cloud data platforms. Architected and managed Databricks notebooks for robust batch and near-real-time data processing, utilizing Delta Lake for ACID compliance. Designed and optimized data loading strategies, schema layouts, and cluster scaling configurations in Snowflake, improving query performance and reducing warehouse costs. Built custom Python-based automated data validation and reconciliation scripts to guarantee data integrity between PostgreSQL source systems and target cloud warehouses. Leveraged AWS services including S3 for storage, Glue for cataloging, and Athena/Redshift for analytics to support scalable data architectures. Collaborated with cross-functional architecture teams to troubleshoot complex Python transformation logic, data skew issues, and SQL performance bottlenecks. Monitored production pipeline health, AWS resource utilization, and data SLAs using ITRS Geneos and CloudWatch, ensuring 99.9% system uptime and high operational reliability.