Data Engineering Solution for Fintech Company


An Indian fintech company sought to integrate Spark structured streaming and make modifications into their pre-existing spark pipeline for real-time data access and batch process. Additionally, they aimed to implement a high speed deduplication API with fuzzy, phonetic and exact match in AWS Lambda, with data stored in DynamoDB.

A suite of AWS services including Lambda, Glue, Kinesis, DynamoDB, Redshift, StepFunctions, APIGateway, S3, and Elasticsearch were utilised.

Spark Scala was incorporated for real-time data streaming and batch processing.

Its capability to perform fuzzy, phonetic, and exact matches ensured a highly accurate and efficient data filtering process.

Strategic Solution


  • Significant performance improvements in real-time data access.
  • The high-speed deduplication API minimized data redundancy and ensure data integrity.
  • By implementing solutions in AWS, the company benefited from the scalability of the cloud, leading to optimized costs.

Help us know you better

  • Analyze & assess 1 application, with upto 300 data elements
  • Identify the business cases that's tied with data
  • Assess Current Data Strategy (if any)
  • Define a Data Platform Architecture & Strategy
  • Define a road map & ROI trajectory
  • Access to our Implementation Methodology

Data & AI Strategic Assessment

A strategic consulting engagement to help you with a data and AI strategy, to drive and grow your business 3x times quicker.   |   +919656730556