Client Challenges and Requirements
Our proprietary ETL migration accelerator automates the migration of any ETL code into the XML format, which can be directly executed on Hadoop. The solution was designed to provide a unified developer experience for engineers throughout the company, whether they work on MapReduce, Spark, Storm, Flink or SQL, as well as accelerate data transformation on the data lake (or in other Big Data environments). This was achieved by implementing an easy-to-use developer productivity tool and GUI, which allows developers to interact with a drag-and-drop environment to create
new transformations or update/edit existing jobs, and visualize existing workloads to
facilitate ongoing management.
This approach enabled the company to realize their initial goals, as well as create a suite of extensions and tools to provide IDE-like functionality for ETL by utilizing Cascading so that it offers an abstraction layer over Hadoop, which is ideal for creating complex data transformations.
Ensures future-proof business investment by retaining code and business logic at first-principal-abstraction layer using XML
Creates a low learning curve for existing ETL developers
Provides all the benefits of underlying Hadoop platform including cost efficiency, fault tolerant, distributed processing, data integration, etc.
Eliminates the expensive lock-in costs from the legacy ETL platform
Creates a reusable framework that can be used to develop and deploy data transformations agnostic of platform
Enables fast, easy transitions between execution environments or the adoption of future environments
Provides a framework for ETL/ELT that is not tied to a single platform or development language