A major payment processing firm based out of the United States wished to build a cloud-based big data strategy that the enterprise, as well as external users, could leverage to build better analytics and make better business decisions. One of the key components and challenges in the build process was migration of on premise data to the cloud.
Client Challenges and Requirements
- Migration of 15+ TB of uncompressed data between geographically separated locations
- Maintaining on-premise cluster performance so as to have a balance between data migration load and current user workloadÂ
- On-premise data in different file formats and compressions
- Storage, data type and other compatibility changes between on premise and Cloud
- Keeping migrated historical data in sync with on-premise cluster
Bitwise Solution
Schema conversion, data extraction, ingestion and data load through GCP Migration agent
Bitwise Cloud Data Migration Framework for Trigger based data load to ingest historical data once available in cloud storage bucket
In-built DQ, validation checks and Bitwise Cloud Data Migration Framework’s full and sample validation utilities to validate migrated historical data to ensure accuracy
Implemented throttling and elasticity for optimal load on source database and maximum extraction performance through Bitwise Cloud Data Migration Framework
CDC implementation through GCP Migration Agent to keep migrated historical data in sync with on premise cluster
Dedicated 1gbps network connection between on premise Hadoop cluster and Google cloud to avoid network congestion
Tools & Technologies We Used
Unix
Google BigQuery Data Transfer Service
Shell Script
Gcloud SDK
BigQuery
Bitwise Cloud Data Migration Framework
Key Results
Successful migration of historical data ensuring data integrity
Automated Data validation resulted in 30% reduction in migration time
Ease of scalability and enhancements
