HIVE-29419: Provide a Hive-specific docker image for Tez AM by abstractdog · Pull Request #6435 · apache/hive

abstractdog · 2026-04-15T15:52:01Z

What changes were proposed in this pull request?

Make hive image able to start a TezAM in LLAP mode that can assign tasks to the LLAP daemons.

Why are the changes needed?

Because it's the next step to have a fully distributed, Dockerized environment for Hive.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Manually, steps are below.

Needs Hive 4.3.0-SNAPSHOT jars that contain recent changes (specifically HIVE-29477):

# assuming that you're standing on hive master or rather the code of this PR:
mvn clean install -DskipTests -Pdist # this will take a long, sorry, we need a snapshot jar
cp packaging/target/apache-hive-4.3.0-SNAPSHOT-bin.tar.gz packaging/cache

start cluster:

export HIVE_VERSION=4.3.0-SNAPSHOT
export POSTGRES_LOCAL_PATH=...your_postgres_driver_path...

# hive: 4.3.0-SNAPSHOT is mandatory
# tez: 1.0.0-SNAPSHOT is mandatory to override released tez jars with the currently unreleased 1.0.0 tez jars that can talk to unmanaged tez sessions
./build.sh -hive 4.3.0-SNAPSHOT -hadoop 3.4.1 -tez 0.10.5 -tez-snapshot 1.0.0-SNAPSHOT

./start-hive.sh --llap
docker compose --profile llap logs -f

test:

 beeline -u 'jdbc:hive2://localhost:10000/' -e "DROP table IF EXISTS iceberg_table; CREATE TABLE iceberg_table (id BIGINT) STORED BY iceberg; INSERT INTO iceberg_table VALUES(1);"

see logs that queries go through tezam and daemons:

tezam         | 2026-04-15T16:08:05,906 INFO  DAGAppMaster - Starting DAG submitted via RPC: INSERT INTO iceberg_table VALUES(1) (Stage-1)

...

llapdaemon-1  | 2026-04-15T16:08:06,368  INFO [Task-Executor-0 (1776269274583_0000_1_00_000000_0)] impl.LlapTaskReporter: Registered counters for fragment: 1776269274583_0000_1_00_000000_0 vertexName: Map 1

...

hiveserver2   | 2026-04-15T16:08:07,267  INFO [HiveServer2-Background-Pool: Thread-117] SessionState: Status: DAG finished successfully in 1.23 seconds

very important test case is that the container layout implemented here in docker-compose is compatible with the already existing and working hs2+llapdeamon setup (no tezam), which is confirmed as:

 ./stop-hive.sh --cleanup
 
export POSTGRES_LOCAL_PATH=~/.m2/repository/org/postgresql/postgresql/42.7.3/postgresql-42.7.3.jar
./build.sh -hive 4.2.0 -hadoop 3.4.1 -tez 0.10.5
./start-hive.sh --llap
docker compose --profile llap logs -f

beeline -u 'jdbc:hive2://localhost:10000/' -e "DROP table IF EXISTS iceberg_table; CREATE TABLE iceberg_table (id BIGINT) STORED BY iceberg; INSERT INTO iceberg_table VALUES(1);"

in which case tezam will simply fail to start, and the cluster works exactly the same way as post-HIVE-29411
be aware of the difference, in case of:

./build.sh -hive 4.2.0 -hadoop 3.4.1 -tez 0.10.5

the tez zookeeper-based registry and external sessions code are simply not present, that's why - regardless of the config - here it can work

sonarqubecloud · 2026-04-16T07:35:17Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

deniskuzZ · 2026-04-17T10:16:30Z

FYI, there is a docker profile to create new Hive image: -Pdocker

deniskuzZ · 2026-04-17T10:18:35Z

    </property>
+    <property>
+        <name>hive.server2.tez.initialize.default.sessions</name>
+        <value>false</value>


why not initialize ?

it's a yarn-based thing that HS2 initializes a few sessions on startup
in the standalone mode, HS2 has no control over them, just discovers them via ZK

deniskuzZ · 2026-04-17T10:20:29Z

+    <!--
+      A registry namespace prefix is a hardcoded prefix for Tez external sessions.
+      The actual tez.am.registry.namespace value is appended to this prefix.
+      Once hive can use the registry client that Tez provides (ZkAMRegistryClient), this property will be removed.


is there a ticket for that? needs Tez upgrade?

for the release we already have: https://issues.apache.org/jira/browse/TEZ-4701
now I created: https://issues.apache.org/jira/browse/HIVE-29573

deniskuzZ · 2026-04-17T10:26:20Z

@@ -0,0 +1,5 @@
+log4j.rootLogger=INFO, console


maybe, simply [tez-log4j.properties]

deniskuzZ · 2026-04-17T10:27:33Z

+    </property>
+    <property>
+        <name>hive.llap.daemon.umbilical.port</name>
+        <value>33333</value>


default is 0, do we even need to touch it?

right, it works but needs a minor fix in LlapTaskCommunicator to handle the default properly:

- conf.get(HiveConf.ConfVars.LLAP_TASK_UMBILICAL_SERVER_PORT.varname) + HiveConf.getVar(conf, HiveConf.ConfVars.LLAP_TASK_UMBILICAL_SERVER_PORT)

otherwise I get:

tezam | 2026-04-17T12:23:28,727 INFO AbstractService - Service org.apache.hadoop.hive.llap.tezplugins.LlapTaskCommunicator failed in state STARTED tezam | java.lang.NullPointerException: Cannot invoke "String.split(String)" because the return value of "org.apache.hadoop.conf.Configuration.get(String)" is null tezam | at org.apache.hadoop.hive.llap.tezplugins.LlapTaskCommunicator.startRpcServer(LlapTaskCommunicator.java:261) tezam | at org.apache.tez.dag.app.TezTaskCommunicatorImpl.start(TezTaskCommunicatorImpl.java:140) tezam | at org.apache.hadoop.hive.llap.tezplugins.LlapTaskCommunicator.start(LlapTaskCommunicator.java:237) tezam | at org.apache.tez.dag.app.ServicePluginLifecycleAbstractService.serviceStart(ServicePluginLifecycleAbstractService.java:41) tezam | at org.apache.hadoop.service.AbstractService.start(AbstractService.java:195) tezam | at org.apache.tez.dag.app.TaskCommunicatorManager.serviceStart(TaskCommunicatorManager.java:165) tezam | at org.apache.hadoop.service.AbstractService.start(AbstractService.java:195) tezam | at org.apache.tez.dag.app.DAGAppMaster$ServiceWithDependency.start(DAGAppMaster.java:1857) tezam | at org.apache.tez.dag.app.DAGAppMaster$ServiceThread.run(DAGAppMaster.java:1878)

good catch, fixing it now

deniskuzZ · 2026-04-17T10:43:23Z

+  tezam:
+    profiles:
+      - llap
+    image: apache/hive:${HIVE_VERSION}


is it Ok to reuse hive image for TezAM? IDK just asking. is is the same downstream?

I think it's ok to reuse, actually, we build both (Hive specific TezAM image + Tez AM image in Tez project), see the motivation on HIVE-29419
downstream we have only a hive image for TezAM, but upstream, the Tez project needs to have its own, TEZ-4682, most probably mimics the Tez Container mode (DAGAppMaster + TezChiild), not the LLAP

deniskuzZ · 2026-04-17T10:52:06Z

+
+      # LLAP daemon discovery
+      HIVE_ZOOKEEPER_QUORUM: zookeeper:2181
+      LLAP_SERVICE_HOSTS: '@llap0'


is the env var correct? not HIVE_LLAP_DAEMON_SERVICE_HOSTS ? see https://github.com/apache/hive/pull/6435/changes#diff-d389e28124c68afe85354e9f947d6d9ec8f7cfb9d49c46b39891ab5da7a7be7cR68

good catch, it's not correct, the only reason why it works is that I take care of it one level deeper in entrypoint:
https://github.com/apache/hive/pull/6435/changes#diff-b7d5fbeab2c6af92616ab371fa3f237867d60d7c81072045342cb44a8981bf90R45

fixing this

deniskuzZ · 2026-04-17T10:56:08Z

+ARG TEZ_SNAPSHOT_REPO_URL=https://repository.apache.org/content/repositories/snapshots
+
+# When snapshot jars are included, client version must match the snapshot version.
+ENV TEZ_CLIENT_VERSION=${TEZ_SNAPSHOT_VERSION:-$TEZ_VERSION}


is this used somewhere? can't find any usages here

this is needed or another config to disable client version check in TezAM, otherwise we get:

tezam | 2026-04-17T12:50:58,952 INFO DAGAppMaster - Comparing client version with AM version, clientVersion=Unknown, AMVersion=1.0.0-SNAPSHOT tezam | 2026-04-17T12:50:58,953 ERROR DAGAppMaster - Incompatible versions found, clientVersion=Unknown, AMVersion=1.0.0-SNAPSHOT

I'm fine with disabling the check:
https://github.com/apache/tez/blob/2ca44b4ba3839ff4c2c2ab2ec95e34d687f61c09/tez-dag/src/main/java/org/apache/tez/dag/app/DAGAppMaster.java#L455-L476

in which case tez site xml has to contain:

<property> <name>tez.am.disable.client-version-check</name> <value>true</value> </property>

it's fine to keep TEZ_CLIENT_VERSION, but the comment should be changed then to:

Client version check is enabled by default in Tez AM, which is picked up from TEZ_CLIENT_VERSION env var.

to reflect that this is not because of the optional snapshot override

wdyt?

if version check makes sense, let's keep the env var

deniskuzZ · 2026-04-17T11:00:23Z

 HIVE_VERSION=
 HADOOP_VERSION=
 TEZ_VERSION=
+TEZ_SNAPSHOT_VERSION=


could we keep one required input in [build.sh]: -tez
If it ends with -SNAPSHOT, treat it as snapshot channel automatically - otherwise release (for example 0.10.5)

a release tarball must always be fetched (as the structural foundation, mostly because it contains all the dependencies for Tez), and snapshot jars are an optional overlay, placed on top to override the release-version tez jars while leaving lib/ intact
a snapshot version alone cannot form a self-contained, runnable Tez installation, as we don't have SNAPSHOT tez tarballs downloadable

Do we need to support SNAPSHOT overlays? Currently, the Docker image only allows released versions.

short term, yes, otherwise the whole TezAM initiative cannot be tried out as here
Tez 1.0.0 could be months (or more) away, and I would like to be able to proceed with cloud-native initiative in the meantime
released/published Docker images must not contain SNAPSHOT jars, I agree, but a temporary Hive image built with Tez 1.0.0-SNAPSHOT can be utilized for distributed/perf test even in the next 1-2 months

would it make sense splitting this PR in 2 parts so we could revert the SNAPSHOT workaround once Tez 1.0.0 is released?

maybe a cleaner approach could be to build a custom Tez Docker image on top of the nightly Tez image, extending it with the required Hive jars.

asf-ci-hive added tests pending tests unstable and removed tests pending labels Apr 15, 2026

HIVE-29419: Provide a Hive-specific docker image for Tez AM

08f94a2

abstractdog force-pushed the HIVE-29419-tez-am-image branch from 7240e71 to 08f94a2 Compare April 16, 2026 06:15

asf-ci-hive added tests pending and removed tests unstable labels Apr 16, 2026

asf-ci-hive added tests unstable and removed tests pending labels Apr 16, 2026

deniskuzZ reviewed Apr 17, 2026

View reviewed changes

Conversation

abstractdog commented Apr 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

sonarqubecloud Bot commented Apr 16, 2026

Quality Gate passed

Uh oh!

deniskuzZ commented Apr 17, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

abstractdog Apr 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

deniskuzZ Apr 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

deniskuzZ Apr 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

abstractdog Apr 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

deniskuzZ Apr 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

deniskuzZ Apr 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

abstractdog Apr 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

deniskuzZ Apr 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

deniskuzZ Apr 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

abstractdog commented Apr 15, 2026 •

edited

Loading

abstractdog Apr 17, 2026 •

edited

Loading

deniskuzZ Apr 17, 2026 •

edited

Loading

deniskuzZ Apr 17, 2026 •

edited

Loading

abstractdog Apr 17, 2026 •

edited

Loading

deniskuzZ Apr 17, 2026 •

edited

Loading

deniskuzZ Apr 17, 2026 •

edited

Loading

abstractdog Apr 17, 2026 •

edited

Loading

deniskuzZ Apr 17, 2026 •

edited

Loading

deniskuzZ Apr 17, 2026 •

edited

Loading