Hadoop to Databricks
Hadoop to Databricks migration
The short answer
Novuz converts Hive and Impala SQL, the shell and Spark scripts around them and their Oozie or cron schedules into Databricks SQL, PySpark and Databricks Workflows with fixed rules. Every object ships with a reconciliation report, and the software runs inside your own cloud account.
Risks
What breaks in a Hadoop migration
These are the five problems we see most often in Hadoop estates.
- Hive and Impala SQL lean on behavior that differs on Databricks: implicit casts, NULL handling in joins and partition pruning that the query never states.
- Partitioned tables, INSERT OVERWRITE patterns and dynamic partitions carry load logic that must be rebuilt for Delta, or reruns double-count.
- User-defined functions in Java or Python, often undocumented, sit inside queries that look like plain SQL.
- Orchestration lives in Oozie coordinators, cron and shell wrappers that pass parameters and dates nobody wrote down.
- The data moves easily; the logic does not. Teams that copy the files first find out at validation that the numbers no longer match.
Coverage
What Novuz converts
Final coverage is confirmed during the assessment.
| Hadoop object | Databricks target |
|---|---|
| Hive and Impala tables (managed, external, partitioned) | Delta tables in Unity Catalog, partitioning or liquid clustering agreed in the assessment |
| Views | Databricks SQL views |
| HiveQL and Impala SQL scripts | Databricks SQL or PySpark |
| INSERT OVERWRITE and dynamic partition loads | Idempotent MERGE or replaceWhere loads |
| User-defined functions | SQL functions or Python UDFs, flagged where a rewrite is needed |
| Shell and Spark scripts | Notebooks with parameters and control flow preserved |
| Oozie workflows and coordinators, cron | Databricks Workflows with the original dependencies and schedules |
| Data on HDFS | Bulk and incremental data migration to cloud storage |
Worked example
What a typical estate looks like
An illustrative Hadoop estate at a regional bank. These figures are an example, not a customer result.
- 2,800 HiveQL and Impala scripts, of which 600 are wrapped in shell scripts that pass parameters
- 350 Oozie workflows and 90 coordinators
- 7,500 tables and 1,200 views across 60 databases
- 180 user-defined functions, about a third with no source in the repository
- 400 TB on HDFS
- Complexity split: 50% low, 35% medium, 15% high
- Assessment output: a fixed price for the full estate and a recommended first domain of about 1,500 objects
Proof
Delivered on Databricks
Delivered estates of 1,000 to 1,600 objects typically go from code received to reconciled output in 4 to 8 weeks, with every object reconciled.¹ Novuz is a Databricks validated technology partner, with three Databricks-commissioned enterprise migrations delivered. The same engine and validation process applies to Hive and Impala.
Read the case studiesPricing
Pricing for Hadoop to Databricks
Pricing follows object count and complexity, not hours or terabytes. Many teams time the assessment to land before the next Hadoop support renewal or hardware refresh.
Estate assessment
$3,000. Three business days. Credited in full.2
First Domain
$25,000 to $40,000. 5 to 6 weeks on one domain of up to 1,500 components. Equivalence guarantee in writing.2
Full estate
Fixed from the assessment. Typically $40,000 to $200,000 for 1,500 to 10,000 objects, or an enterprise license for larger estates.2
Honest fit
Who should not choose Novuz
Novuz is not the right choice for estates under about 300 objects; for Oracle-only or mainframe estates; for teams with no target and no date inside 18 months; or for buyers who want staff augmentation rather than software.
Where Novuz runs does not rule you out. Most regulated teams run it inside their own environment, and it can also run in a Novuz-hosted environment with the same fixed price, the same reconciliation report and the same guarantee.
FAQ
Frequently asked questions
Which Hadoop components does Novuz convert?
Hive and Impala SQL, the shell and Spark scripts around them, user-defined functions, and Oozie or cron schedules, to Databricks SQL, PySpark and Databricks Workflows. Final coverage for your estate is confirmed in the assessment, object by object.
What about user-defined functions with no source code?
The assessment lists every function a query calls and whether its source exists. Functions with source are converted or mapped to built-in functions. Functions without source are flagged with the queries that use them, so your team decides whether to rebuild, replace or retire them before anyone prices the work.
How does Novuz handle Oozie and cron schedules?
Novuz reads the workflow and coordinator definitions and the cron entries, including the parameters and dates the shell wrappers pass. It rebuilds them as Databricks Workflows with the same dependencies and schedules, and each converted job is reconciled against the original output on the same input data.
Does Novuz migrate the data on HDFS as well as the code?
Yes. Novuz migrates data in bulk and incrementally, and validates it with the same reconciliation checks used for code. Most teams move the data for a domain first, then run converted logic against it, so every comparison uses the same inputs.
Can we move to Microsoft Fabric instead of Databricks?
Yes. The same conversion and reconciliation apply with Fabric as the target. The assessment agrees the target and the patterns before anything is converted.
Does Novuz run in our cloud account?
Yes. Novuz deploys as a container or VM inside your AWS, Azure or Google Cloud account, next to your Databricks workspace. Your code and data never leave your environment. No outbound connection is required; if you choose a hosted model for the resolver, only the edge-case fragments go to it, under your own agreement.
How is this different from Lakebridge or a rebuild?
Lakebridge’s analyzer counts the Hive, Impala and Oozie estate for free, but its converters do not list Hive or Impala, and Databricks’ Hadoop guidance treats the move as a rebuild. Novuz converts Hive and Impala by fixed rules, prices the estate before work starts, and ships a reconciliation report for every object. The Hadoop options page sets out each route from the vendors’ published material.
What does a First Domain cover?
A First Domain is one domain of up to 1,500 components and 25 TB, converted, reconciled and cutover-ready in your environment in 5 to 6 weeks, for $25,000 to $40,000, credited against the full estate. You agree the objects and the reconciliation threshold up front. If the threshold is missed at the readout, the equivalence guarantee waives or credits the fee in full.
- 1 ¹ Typical across delivered estates, counted under the rules on the Methodology page.
- 2 Prices in USD as of the review date. Full terms on the Pricing page.
Written by Ted Ghose, Co-founder and CTO, Novuz
Last reviewed:
Find out what your logic debt is costing you.
Every object counted, lineage, dead code and a fixed price in three business days. $3,000, credited.
Not sure which door? The scorecard takes ten minutes and tells you.