Nikhitha Guggilla

Data Engineer

Wipro

May 2022 - Jun 2025

Developed scalable Azure Data Factory pipelines to ingest structured, semi-structured, and unstructured data from multiple source systems into ADLS Gen2. Implemented incremental data loading using watermark logic and metadata tables to process only newly inserted and updated records. Implemented and developed PySpark transformation logic in Azure Databricks for data cleaning, validation, deduplication, joins, aggregations, and business rule implementation using SQL and window functions. Worked extensively on Medallion Architecture using Bronze, Silver, and Gold layers for organizing raw, cleaned, and business-ready data. Implemented Delta Lake merge/upsert logic to handle incremental processing and late-arriving data efficiently. Performed source-to-target reconciliation, duplicate validation, and null checks to ensure data quality. Worked on Spark optimization techniques including partition pruning, repartition/coalesce strategies, broadcast joins, caching optimization, and shuffle reduction to improve processing performance and reduce execution time by nearly 40%. Handled real-time production issues such as duplicate data, schema drift, data skew, OOM issues, late-arriving data, and pipeline failures. Used Self-Hosted Integration Runtime (SHIR) to connect Azure services with on-premise systems. Monitored ADF pipelines and Databricks jobs and resolved production support issues. Participated in Agile sprint ceremonies including sprint planning, daily stand-ups, backlog discussions, and retrospective meetings.