Software Developer II
Hashedin By Deloitte (Contractor at Thomson Reuters US office)
Stack : AWS (Step Function, Cloudwatch, Athena, Glue, Lambda), Python (Boto3, Pyspark), Postgres, Jfrog, Java(Spring Boot)
Worked on the backend of an e-commerce website using Java Spring Boot and Netflix Eureka as a part of training.
- Implemented using microservice architecture, created the cart and orders microservice.
- Deployed the Microservices and the database on GCP.
Worked on Ingest and Publishing for organization authority in Thomson Reuters.
- Modified Ingest Step Function to increase max concurrency for ingestion glue jobs, increasing the efficiency of Ingest step function and reducing the execution time significantly.
- Performed data validation and preprocessing to standardize the data ingested in the tables.
- Created Glue jobs to generate reports (naics/country/sic/partition) of the organizations and stored it in s3 to feed it to the API facing the customers as well as notifying them when a new report was generated through SNS.
- Wrote test cases and brought test coverage from 7% to 80% for the whole Gcs publishing repo. Found a bug which was causing the publishing glue job to fail constantly and was being worked on by the colleagues. I Pointed out the exact issue and got it fixed.
- Spearheaded the implementation of spark executor logs for all Glue jobs, addressing a long-standing issue adopted across Org Authority Team repositories.
- Engineered organization ID mappings in PostgreSQL via PySpark, optimizing performance and transitioning to a shared library for streamlined use.
Led a 4-member team in integrating Dragos on an AWS Outpost server with Greengrass, developing an end-to-end data pipeline for insightful AWS Quicksight reports, utilized as a POC for clients like Shell and Koch.
JAN 2022 - JUL 2023
Platform Engineer
WABTEC corporation
Stack : Terraform, AWS (EC2, Eventbridge, EMR, Athena, Glue, Lambda), Python (Boto3, Pyspark), HiveQL
Migrated the current infrastructure to IaC (Infrastructure as Code) using Terraform & Chef.
- New setup now allows various features such as spinning up Redshift, EMR & Kubernetes clusters.
- All applications now run on Chef bootstrapped servers created using terraform with additional support for AWS auto-scaling and AWS load balancing (both ALB & ELB).
POC of next gen Data Lake architecture using Terraform. PoC was a success and was adopted for org wide usage.
- Ran Glue crawler and EMR jobs (pyspark and hiveql queries in master node) to benchmark efficiency on 125 GB data.
- Ran Glue jobs & Athena queries on the schema created by Glue crawler, documented the steps to be used in the new setup.
- Increased the availability of the system by automating fallback to backup clusters when emr-targets are unhealthy (AWS Lambda, network-LB, Boto3).
- Created an email based notification service on a serverless architecture (Lambda, EventBridge and SNS) to notify the team when an EMR cluster terminates and a backup cluster gets activated.
JAN 2021 - JAN 2022
Software Engineering Intern
R Systems International, Noida
Stack : Python (Django), MySQL, JS/Bootstrap5
Report Manager - A web-app to automate the record of ticket-approvals. Team now doesn’t have to store emails in excel sheets manually, excel/pdf/zip files were created on demand. Emails were parsed and stored in mysql tagged with subject for easy querying.
Bitlocker - Inventory management app that keeps track of the company assets given to the employees.
Internship, May ’19 - Jul ’19
Software Engineering Intern
Riders Easyrath, Gurgaon
Stack : ElectronJS, Bootstrap5, SQL/Websql
Created the front end and windows exec (electronJS) of POS (Point Of Sale) website, worked on state management using client side cookies (WebSQL, Js). Used webSql to store the data offline and synced it with the database when the user came online.
Internship, Jun ’20 - Jul ’20
Software Engineering Intern
Cogmac Industries, Noida
Stack : Python (BeautifulSoup4), VueJS, HTML/CSS
Created tools for web scraping, managing java heap, login/signup page & automated file downloads.
Internship, May ’18 - Jul ’18