Gerrit-init issues when running k8s-gerrit in HA mode.

Samuel Idowu

unread,

Apr 25, 2025, 12:05:02 PM (14 days ago) Apr 25

to Repo and Gerrit Discussion

We are trying to migrate from a VM-based Gerrit setup to Kubernetes. We have followed the Gerrit operator guide here, but we are consistently encountering this error when enabling the primary HA setup by setting the refDB values to Zookeeper with a connection string. We have tried this on both a minikube setup and on AWS EKS, but the issue is the same.

Whenever we enable the refDB, the Gerrit cluster pod, specifically the gerrit-init container, runs into the following error. Does anyone have any suggestions on how to overcome this issue?

k logs -n gerrit gerrit-0
-c gerrit-init
[2025-04-25 08:50:45,068] INFO Requiring plugins (ClusterMode: HIGH_AVAILABILITY): ['healthcheck', 'high-availability', 'zookeeper-refdb']
[2025-04-25 08:50:45,069] INFO Requiring libs (ClusterMode: HIGH_AVAILABILITY): ['high-availability', 'global-refdb']
[2025-04-25 08:50:45,073] INFO Removed plugin delete-project.jar
[2025-04-25 08:50:45,073] INFO Removed plugin download-commands.jar
[2025-04-25 08:50:45,076] INFO Installing plugin healthcheck from container to /var/gerrit/plugins/healthcheck.jar.
[2025-04-25 08:50:45,076] DEBUG SHA1 of file '/var/plugins/healthcheck.jar' is 9f1f45952a412ad9445192f11fd7f474278fea65
[2025-04-25 08:50:45,077] DEBUG SHA1 of file '/var/gerrit/plugins/healthcheck.jar' is 9f1f45952a412ad9445192f11fd7f474278fea65
[2025-04-25 08:50:45,079] INFO Installing plugin high-availability from container to /var/gerrit/plugins/high-availability.jar.
[2025-04-25 08:50:45,088] DEBUG SHA1 of file '/var/plugins/high-availability.jar' is 4817182afb052fd993455c3f671aab484f862359
[2025-04-25 08:50:45,101] DEBUG SHA1 of file '/var/gerrit/plugins/high-availability.jar' is 4817182afb052fd993455c3f671aab484f862359
[2025-04-25 08:50:45,103] INFO Installing plugin zookeeper-refdb from container to /var/gerrit/plugins/zookeeper-refdb.jar.
[2025-04-25 08:50:45,112] DEBUG SHA1 of file '/var/plugins/zookeeper-refdb.jar' is ad4d66dddaf4cc524c21e1049af06fb8609c12e4
[2025-04-25 08:50:45,126] DEBUG SHA1 of file '/var/gerrit/plugins/zookeeper-refdb.jar' is ad4d66dddaf4cc524c21e1049af06fb8609c12e4
[2025-04-25 08:50:45,128] INFO Installing plugin high-availability from container to /var/gerrit/lib/high-availability.jar.
[2025-04-25 08:50:45,136] DEBUG SHA1 of file '/var/plugins/high-availability.jar' is 4817182afb052fd993455c3f671aab484f862359
[2025-04-25 08:50:45,149] DEBUG SHA1 of file '/var/gerrit/lib/high-availability.jar' is 4817182afb052fd993455c3f671aab484f862359
[2025-04-25 08:50:45,151] INFO Installing plugin global-refdb from container to /var/gerrit/lib/global-refdb.jar.
[2025-04-25 08:50:45,151] DEBUG SHA1 of file '/var/plugins/global-refdb.jar' is f35976daad3cfd8111afe76e5b534aff6f468483
[2025-04-25 08:50:45,152] DEBUG SHA1 of file '/var/gerrit/lib/global-refdb.jar' is f35976daad3cfd8111afe76e5b534aff6f468483
[2025-04-25 08:50:45,152] INFO Installing packaged plugin download-commands.
[2025-04-25 08:50:45,155] INFO Installing packaged plugin delete-project.
[2025-04-25 08:50:51,249] INFO Installed Gerrit version: gerrit version 3.11.1-657-g998b07ccfc; Provided Gerrit version: gerrit version 3.11.1-657-g998b07ccfc).
[2025-04-25 08:50:51,249] INFO Plugins were installed or updated. Initializing.
[2025-04-25 08:50:51,249] INFO Existing gerrit.config found.
Exception in thread "main" com.google.inject.ProvisionException: Unable to provision, see the following errors:

1) [Guice/NullInjectedIntoNonNullable]: null returned by binding at BaseInit$1.configure(BaseInit.java:280)
but the 2nd parameter repoManager of NoteDbSchemaVersionManager.<init>(NoteDbSchemaVersionManager.java:39) is not @Nullable
at BaseInit$1.configure(BaseInit.java:280)
at NoteDbSchemaVersionManager.<init>(NoteDbSchemaVersionManager.java:39)
\_ for 2nd parameter repoManager
at ZkInit.versionManager(ZkInit.java:56)
\_ for field versionManager
while locating ZkInit

Learn more:
https://github.com/google/guice/wiki/NULL_INJECTED_INTO_NON_NULLABLE

1 error

======================
Full classname legend:
======================
BaseInit$1: "com.google.gerrit.pgm.init.BaseInit$1"
NoteDbSchemaVersionManager: "com.google.gerrit.server.schema.NoteDbSchemaVersionManager"
ZkInit: "com.googlesource.gerrit.plugins.validation.dfsrefdb.zookeeper.ZkInit"
========================
End of classname legend:
========================

at com.google.inject.internal.InternalProvisionException.toProvisionException(InternalProvisionException.java:251)
at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:74)
at com.google.inject.internal.InjectorImpl.injectMembers(InjectorImpl.java:1109)
at com.googlesource.gerrit.plugins.validation.dfsrefdb.zookeeper.ZkInit.run(ZkInit.java:72)
at com.google.gerrit.pgm.init.InitPlugins.initPlugins(InitPlugins.java:180)
at com.google.gerrit.pgm.init.InitPlugins.run(InitPlugins.java:99)
at com.google.gerrit.pgm.init.SitePathInitializer.run(SitePathInitializer.java:92)
at com.google.gerrit.pgm.init.BaseInit.run(BaseInit.java:115)
at com.google.gerrit.pgm.util.AbstractProgram.main(AbstractProgram.java:62)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at com.google.gerrit.launcher.GerritLauncher.invokeProgram(GerritLauncher.java:251)
at com.google.gerrit.launcher.GerritLauncher.mainImpl(GerritLauncher.java:147)
at com.google.gerrit.launcher.GerritLauncher.main(GerritLauncher.java:92)
at Main.main(Main.java:30)
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "/var/tools/gerrit-initializer/__main__.py", line 18, in <module>
main()
File "/var/tools/gerrit-initializer/main.py", line 96, in main
args.func(args)
File "/var/tools/gerrit-initializer/main.py", line 43, in _run_init
GerritInitHA(_parse_gerrit_config(), config).execute()
File "/var/tools/gerrit-initializer/initializer/tasks/init.py", line 188, in execute
init_process = subprocess.run(
^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/subprocess.py", line 573, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['java', '-jar', '/var/war/gerrit.war', 'init', '--no-auto-start', '--batch', '--dev', '-d', '/var/gerrit']' returned non-zero exit status 1.

Samuel Idowu

unread,

Apr 25, 2025, 12:34:44 PM (14 days ago) Apr 25

to Repo and Gerrit Discussion

Looking at this a bit deeper. I found out that an older version of Gerrit k8s worked well without this issue. This version worked as expected. v0.1-762-gb3ed5ecea2-3.11.0-354-g2b112f1817

Can someone point me to the change logs for the k8s-gerrit, and how do we work with a more recent version, or is this a known issue?

Thomas Dräbing

unread,

Apr 28, 2025, 10:54:56 AM (11 days ago) Apr 28

to Samuel Idowu, Repo and Gerrit Discussion

Hi Samuel,

this looks like an issue with the zookeeper-refdb plugin, which does not seem to work properly with the latest Gerrit version.

I will have a look.

Best,

Thomas

--
--
To unsubscribe, email [email protected]
More info at https://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To view this discussion visit https://groups.google.com/d/msgid/repo-discuss/14ebc71d-ff3b-4397-801d-e69a43228052n%40googlegroups.com.

Samuel Idowu

unread,

Apr 28, 2025, 11:51:30 AM (11 days ago) Apr 28

to Repo and Gerrit Discussion

Oh, good to know. Thanks for the update.

/BR, Sam.

Luca Milanesio

unread,

Apr 28, 2025, 1:55:55 PM (11 days ago) Apr 28

to Repo and Gerrit Discussion, Luca Milanesio

> On 28 Apr 2025, at 09:51, Samuel Idowu <[email protected]> wrote:
>
> Oh, good to know. Thanks for the update.
>
> /BR, Sam.
>
> On Monday, April 28, 2025 at 9:54:56 AM UTC+2 Thomas Dräbing wrote:
> Hi Samuel,
>
> this looks like an issue with the zookeeper-refdb plugin, which does not seem to work properly with the latest Gerrit version.

I believe it has something to do with the Guice optional injections: the versionManager is flagged with @Inject(optional = true).

Luca.

> To view this discussion visit https://groups.google.com/d/msgid/repo-discuss/8ba4c505-37a4-4bb2-a7a9-26ce77227bc8n%40googlegroups.com.

Luca Milanesio

unread,

Apr 28, 2025, 2:02:54 PM (11 days ago) Apr 28

to Repo and Gerrit Discussion, Luca Milanesio

> On 28 Apr 2025, at 11:55, Luca Milanesio <[email protected]> wrote:
>
>
>
>> On 28 Apr 2025, at 09:51, Samuel Idowu <[email protected]> wrote:
>>
>> Oh, good to know. Thanks for the update.
>>
>> /BR, Sam.
>>
>> On Monday, April 28, 2025 at 9:54:56 AM UTC+2 Thomas Dräbing wrote:
>> Hi Samuel,
>>
>> this looks like an issue with the zookeeper-refdb plugin, which does not seem to work properly with the latest Gerrit version.
>
>
> I believe it has something to do with the Guice optional injections: the versionManager is flagged with @Inject(optional = true).

Yes, see [1]: whenever we use an optional injection, we also have to flag the field as nullable.
It sounds strange though that this shows up only now?

Luca.

[1] https://github.com/google/guice/wiki/NULL_INJECTED_INTO_NON_NULLABLE

Luca Milanesio

unread,

Apr 28, 2025, 2:38:07 PM (11 days ago) Apr 28

to Repo and Gerrit Discussion, Luca Milanesio

> On 28 Apr 2025, at 12:02, Luca Milanesio <[email protected]> wrote:
>
>
>
>> On 28 Apr 2025, at 11:55, Luca Milanesio <[email protected]> wrote:
>>
>>
>>
>>> On 28 Apr 2025, at 09:51, Samuel Idowu <[email protected]> wrote:
>>>
>>> Oh, good to know. Thanks for the update.
>>>
>>> /BR, Sam.
>>>
>>> On Monday, April 28, 2025 at 9:54:56 AM UTC+2 Thomas Dräbing wrote:
>>> Hi Samuel,
>>>
>>> this looks like an issue with the zookeeper-refdb plugin, which does not seem to work properly with the latest Gerrit version.
>>
>>
>> I believe it has something to do with the Guice optional injections: the versionManager is flagged with @Inject(optional = true).
>
> Yes, see [1]: whenever we use an optional injection, we also have to flag the field as nullable.
> It sounds strange though that this shows up only now?

Fixed with [2], @Thomas added you as reviewer.

HTH

Luca.

[2] https://gerrit-review.googlesource.com/c/plugins/zookeeper-refdb/+/471284

Samuel Idowu

unread,

Apr 29, 2025, 10:03:54 AM (10 days ago) Apr 29

to Repo and Gerrit Discussion

Great! Thanks for the quick fix.

Reply all

Reply to author

Forward