Data Engineering Solution for Fintech Company
Background
An Indian fintech company sought to integrate Spark structured streaming and make modifications into their pre-existing spark pipeline for real-time data access and batch process. Additionally, they aimed to implement a high speed deduplication API with fuzzy, phonetic and exact match in AWS Lambda, with data stored in DynamoDB.
A suite of AWS services including Lambda, Glue, Kinesis, DynamoDB, Redshift, StepFunctions, APIGateway, S3, and Elasticsearch were utilised.
Spark Scala was incorporated for real-time data streaming and batch processing.
Its capability to perform fuzzy, phonetic, and exact matches ensured a highly accurate and efficient data filtering process.
Strategic Solution
Benefits
- Significant performance improvements in real-time data access.
- The high-speed deduplication API minimized data redundancy and ensure data integrity.
- By implementing solutions in AWS, the company benefited from the scalability of the cloud, leading to optimized costs.
