I am a Data Engineer , experienced in handling vast amounts of data,creating robust data pipelines , been working in Fin-Tech since graduation and have good knowledge of banking systems and core technologies that are used behind the scenes.
Working as a data-engineer at fintech startup. Gained Experience in working with cutting edge technologies being used in fintech currently like aws , apache hudi , spark. Good hands-on experience in working behind the scenes of a digital payments company.
• Developed extensible testing framework for verifying core logic of transaction processing, credit card features and data transformation withing the data lake.
• Developed framework for load and throughput testing of ETL pipelines, bulk inserting thousands of records, which provided meaningful insights in meeting SLAs and pipeline scalability.
• Built 10 + highly scalable, robust real time ETL pipelines using spark-structured streaming and AWS services like DynamoDB, Kinesis, Glue, S3 and Apache HUDI which provided upserts to the data lake in near real-time.
• Performed extensive research into Spark and HUDI configurations and tackled challenging problems like merging data with evolved schemas in data lakes.
• Developed schema management framework which helped to manage 30 + schemas and their respective versions, decoupled complex schemas which made data transfer to down streams considerably easier and faster
• Built highly scalable and adaptive end-to-end monthly outbound data transfer framework transferring terabyte level data and orchestrated using AWS CloudWatch and triggers.
• Migrated ETL pipelines to ELT type, reducing overheads in transferring reporting data.
• Building AWS products using service catalogue, modelling AWS resources as infrastructure-as-code using CloudFormation and deployments through Jenkins.
Key Skills: Python, Pyspark, Spark Structured Streaming, Apache HUDI, AWS-Glue, Step Functions
CGPA: 8.6 , GPA: 3.55
I enjoy most of my time being outdoors. In the winter and monsoons, I am an avid hiker and novice mountain climber. During the warmer months here in Mumbai, I enjoy swimming, biking, free climbing.
When forced indoors, I follow a number of sci-fi and fantasy genre movies and television shows, I am an aspiring chef, and I spend a large amount of my free time exploring the latest technology advancements in the front-end web development and data world. D Sozey.