Page tree
Skip to end of metadata
Go to start of metadata

Table of Contents

Nordix Gerrit 

This page contains instructions to upgrade Nordix Gerrit:  https://gerrit.nordix.org/

Before any upgrade please always refer to Gerrit Release Notes, for example https://www.gerritcodereview.com/3.5.html

Gerrit upgrade process with migration to NoteDb.

In order to make sure the Gerrit DB schema changes are correctly updated during the upgrade the following upgrade path will be used:

The full Gerrit backup will be done before the upgrade. The plan is to make a offline Gerrit upgrade in the next maintenance window. 

No new gerrit plugins will be installed during the upgrade process. The following plugins will be upgraded:

  • deleteproject - Provides the ability to delete a project.
  • download-commands - This plugin defines commands for downloading changes in different download schemes (for downloading via different network protocols).
  • gerrit-oauth-provider - This plugin enables Gerrit to use OAuth2 protocol for authentication. 

Upgrade to 2.14.20

Based on the Gerrit 2.14 Release Notes the below steps will are required in order to upgrade to 2.14.20 release:

  1. Apply DB schema changes.

    $ java -jar gerrit-2.14.20.war init -d /data/gerrit

    Rebuild index before starting Gerrit:

    java -jar /data/gerrit/bin/gerrit.war reindex -d /data/gerrit
  2. Updated primary key on JdbcAccountPatchReviewStore. Manually drop and recreate the primary key as follows:

    $ ssh -p 29418 infra@gerrit.nordix.org gerrit gsql  
    
    
      # drop the key
      ALTER TABLE account_patch_reviews
      DROP CONSTRAINT primary_key_account_patch_reviews;
      # recreate the key
      ALTER TABLE account_patch_reviews
      ADD CONSTRAINT primary_key_account_patch_reviews
      PRIMARY KEY (change_id, patch_set_id, account_id, file_name);
  3. Update gerrit-oauth-provider plugin

    ssh -i ~/.ssh/nordixinfra -p <port_number> 29418 infra@<gerrit> gerrit plugin install https://github.com/davido/gerrit-oauth-provider/releases/download/v2.14.6.2/gerrit-oauth-provider.jar


Upgrade to 2.15.16

Based on the Gerrit 2.15 Release Notes the below steps will are required in order to upgrade to 2.15.16 release:

  1. Apply DB schema changes.

    $ java -jar gerrit-2.15.16.war init -d /data/gerrit

    Rebuild index before starting Gerrit:

    java -jar /data/gerrit/bin/gerrit.war reindex -d /data/gerrit

Note:

  1. Support for draft changes and draft patch sets has been completely removed.
  2. Almost all account data is now stored in NoteDb.
  3. NoteDb for Changes.  With this release, the new Gerrit storage backend, NoteDb, is officially supported for storing change metadata and is the default storage backend for new installations. For existing sites, migration to the new backend may be done either offline or online in a running server. Please refer Gerrit 2.15 Release Notes for details.

Upgrade to 2.16.11.1

Based on the Gerrit 2.16 Release Notes the below steps will are required in order to upgrade to 2.16.11 release:

  1. Apply DB schema changes.

    $ java -jar gerrit.war init -d /data/gerrit

    Rebuild index before starting Gerrit:

    java -jar /data/gerrit/bin/gerrit.war reindex -d /data/gerrit
  2. Update gerrit-oauth-provider plugin

    ssh -i ~/.ssh/nordixinfra -p <port_number> 29418 infra@<gerrit> gerrit plugin install https://github.com/davido/gerrit-oauth-provider/releases/download/v2.16.1/gerrit-oauth-provider.jar


  3. Data migration from ReviewDb to NoteDB. In my opinion we should migrate our Nordix Gerrit from SQL to NoteDB as recommended by Gerrit Release Notes.
    Although ReviewDb is still technically available on Gerrit v2.16, the upgrade to NoteDb is strongly recommended. In the next version of Gerrit (v3.0), ReviewDb will not be available.

    Please take a look at Gerrit Code Review - NoteDb Backend for it's advantages and offline migration process overview.  

  4. Install DeleteProject plugin:

    ssh -i ~/.ssh/nordixinfra -p <port_number> 29418 infra@<gerrit> gerrit plugin install https://gerrit-ci.gerritforge.com/view/Plugins-stable-2.16/job/plugin-delete-project-bazel-stable-2.16/lastSuccessfulBuild/artifact/bazel-bin/plugins/delete-project/delete-project.jar

Note:

  1. New plugin  codemirror-editor, is available, which uses CodeMirror to provide a rich code editing experience in PolyGerrit.

  2. From 2.16 GWT UI is deprecated and will be removed in a future version.

  3. Support for Velocity templates removed
  4. Push to refs/changes is deprecated
  5. Git clients older than 2.x are not supported anymore

Nordix Gerrit test site after the upgrade

Please take a look and test the copy of the Nordix gerrit site after upgrade to 2.16.11.1: https://gerrittest.ddns.net/#/dashboard/self

Issues during testing

  1. Long start-up of gerrit after upgrade to 2.14.20. (Server : Started @298535ms)
    It looks like during the test we hit a problem with a lack of entropy in the test system. Detail of the problem described in: Upgraded from gerrit 2.13.5 to 2.13.6, hangs during start

    FIX:
    The problem was fixed by installing haveged entropy source which improved startup time (Server : Started @8868ms):

    $ sudo apt install haveged
    $ sudo systemctl status haveged.service


  2. After migration to NoteDB backend we can't remove the Postgresql from gerrit server as accountPatchReviewDb is stored externally - Please refer database in Gerrit 2.16.11.1 for details.
  3. The gitweb is failing after upgrade to 2.16.11.1

    [2019-09-17 09:17:14,681] [Gitweb-ErrorLogger] ERROR com.google.gerrit.httpd.gitweb.GitwebServlet : CGI: Can't locate CGI.pm in @INC (you may need to install the CGI module) (@INC contains: /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.26.1 /usr/local/share/perl/5.26.1 /usr/lib/x86_64-linux-gnu/perl5/5.26 /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl/5.26 /usr/share/perl/5.26 /usr/local/lib/site_perl /usr/lib/x86_64-linux-gnu/perl-base) at /usr/share/gitweb/gitweb.cgi line 15.
    CGI: BEGIN failed--compilation aborted at /usr/share/gitweb/gitweb.cgi line 15.
    [2019-09-17 09:17:14,684] [HTTP-72] ERROR com.google.gerrit.httpd.gitweb.GitwebServlet : Non-zero exit status (2) from /usr/share/gitweb/gitweb.cgi

    FIX:
    The problem was fixed by installing Perl session support to CGI::Application

    $ sudo apt install libcgi-application-plugin-session-perl/bionic
  4. There was a Java Exception during migration to NoteDB but looking at the logs the migration was successful:

    Collecting projects:    175
    Reindexing changes: projects: 20% (35/175), 88% (1100/1243) (-)[2019-09-17 08:25:19,315] [Index-Batch-1] WARN  com.google.gerrit.server.change.ChangeKindCacheImpl : Cannot check trivial rebase of new patch set 3af44a96b56d216fa418ef9eb05722c5d856e1a0 in onap/doc
    java.util.concurrent.ExecutionException: org.eclipse.jgit.errors.MissingObjectException: Missing blob 5c384fb2888029c2babb859c30318749e1ce828c
    	at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:531)
    	at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:492)
    	at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:83)
    	at com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:196)
    	at com.google.common.cache.LocalCache$Segment.getAndRecordStats(LocalCache.java:2312)
    	at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2278)
    	at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2154)
    	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2044)
    	at com.google.common.cache.LocalCache.get(LocalCache.java:3952)
    	at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4871)
    	at com.google.gerrit.server.cache.h2.H2CacheImpl.get(H2CacheImpl.java:131)
    	at com.google.gerrit.server.change.ChangeKindCacheImpl.getChangeKind(ChangeKindCacheImpl.java:350)
    	at com.google.gerrit.server.ApprovalCopier.getForPatchSet(ApprovalCopier.java:186)
    	at com.google.gerrit.server.ApprovalCopier.getForPatchSet(ApprovalCopier.java:144)
    	at com.google.gerrit.server.ApprovalCopier.getForPatchSet(ApprovalCopier.java:128)
    	at com.google.gerrit.server.ApprovalsUtil.byPatchSet(ApprovalsUtil.java:403)
    	at com.google.gerrit.server.query.change.ChangeData.currentApprovals(ChangeData.java:632)
    	at com.google.gerrit.server.index.change.ChangeField.lambda$static$19(ChangeField.java:477)
    	at com.google.gerrit.index.FieldDef.get(FieldDef.java:138)
    	at com.google.gerrit.index.Schema$1.apply(Schema.java:184)
    	at com.google.gerrit.index.Schema$1.apply(Schema.java:179)
    	at com.google.common.collect.Iterators$6.transform(Iterators.java:785)
    	at com.google.common.collect.TransformedIterator.next(TransformedIterator.java:47)
    	at com.google.common.collect.Iterators$5.computeNext(Iterators.java:638)
    	at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141)
    	at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:136)
    	at com.google.gerrit.lucene.AbstractLuceneIndex.toDocument(AbstractLuceneIndex.java:310)
    	at com.google.gerrit.lucene.LuceneChangeIndex.replace(LuceneChangeIndex.java:208)
    	at com.google.gerrit.lucene.LuceneChangeIndex.replace(LuceneChangeIndex.java:103)
    	at com.google.gerrit.server.index.change.ChangeIndexer.indexImpl(ChangeIndexer.java:234)
    	at com.google.gerrit.server.index.change.ChangeIndexer.index(ChangeIndexer.java:204)
    	at com.google.gerrit.server.index.change.AllChangesIndexer$ProjectIndexer.index(AllChangesIndexer.java:243)
    	at com.google.gerrit.server.index.change.AllChangesIndexer$ProjectIndexer.lambda$call$0(AllChangesIndexer.java:228)
    	at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
    	at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
    	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
    	at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
    	at java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:742)
    	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
    	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
    	at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
    	at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
    	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
    	at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485)
    	at com.google.gerrit.server.index.change.AllChangesIndexer$ProjectIndexer.call(AllChangesIndexer.java:228)
    	at com.google.gerrit.server.index.change.AllChangesIndexer$ProjectIndexer.call(AllChangesIndexer.java:200)
    	at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
    	at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57)
    	at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
    	at com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:83)
    	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
    	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    	at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:646)
    	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    	at java.lang.Thread.run(Thread.java:748)
    Caused by: org.eclipse.jgit.errors.MissingObjectException: Missing blob 5c384fb2888029c2babb859c30318749e1ce828c
    	at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:170)
    	at com.google.gerrit.server.git.InMemoryInserter$Reader.open(InMemoryInserter.java:128)
    	at org.eclipse.jgit.lib.ObjectReader$Filter.open(ObjectReader.java:550)
    	at org.eclipse.jgit.merge.ResolveMerger.getRawText(ResolveMerger.java:1083)
    	at org.eclipse.jgit.merge.ResolveMerger.contentMerge(ResolveMerger.java:873)
    	at org.eclipse.jgit.merge.ResolveMerger.processEntry(ResolveMerger.java:822)
    	at org.eclipse.jgit.merge.ResolveMerger.mergeTreeWalk(ResolveMerger.java:1331)
    	at org.eclipse.jgit.merge.ResolveMerger.mergeTrees(ResolveMerger.java:1279)
    	at org.eclipse.jgit.merge.ResolveMerger.mergeImpl(ResolveMerger.java:388)
    	at org.eclipse.jgit.merge.Merger.merge(Merger.java:266)
    	at org.eclipse.jgit.merge.Merger.merge(Merger.java:219)
    	at org.eclipse.jgit.merge.ThreeWayMerger.merge(ThreeWayMerger.java:129)
    	at com.google.gerrit.server.change.ChangeKindCacheImpl$Loader.call(ChangeKindCacheImpl.java:246)
    	at com.google.gerrit.server.change.ChangeKindCacheImpl$Loader.call(ChangeKindCacheImpl.java:170)
    	at com.google.gerrit.server.cache.h2.H2CacheImpl.lambda$get$1(H2CacheImpl.java:141)
    	at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4876)
    	at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3528)
    	at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2277)
    	... 52 more
    Reindexing changes: projects: 100% (175/175), 100% (1243/1243), done    

Gerrit Upgrade to 3.5.0.1

As of today 30th, March 2022 v3.4.2 is deployed on gerrit.nordix.org, upgrading from 3.4.2 to 3.5.0.1 can be done by executing the single command mentioned below. This release also doesn’t contain schema changes.

reference: https://www.gerritcodereview.com/3.5.html

  1. Download

    Download
    wget https://gerrit-releases.storage.googleapis.com/gerrit-3.5.0.1.war
  2. Stopping Gerrit Service

    sudo systemctl stop gerrit.service
  3. Installation

    Install
    java -jar gerrit-3.5.0.1.war init -d /data/gerrit

    After executing above command the installation process will ask multiple inputs

    The default repositories directory is "git" however there is also unused repositories directory is present at /home/infra/repos

    Before submitting any values please match values with /data/gerrit/etc/gerrit.config and /data/gerrit/etc/secure.config

    The the mail/stmp config values need to default

    The installation process will also present default values from last upgrade.

    Enabled plugins can be found under following directory

    /data/gerrit/plugins

  4. Indexing

    Indexing
      java -jar gerrit-3.5.0.1.war reindex --index changes -d /data/gerrit

    Indexing for this upgrade took 1 hour and 30 minutes.

  5. Start Gerrit Service

    sudo systemctl start gerrit.service
  6. Post Upgrade

    Navigate to https://gerrit.nordix.org/
    Login using google auth failed and showed "Server error"

    tail -f /data/gerrit/logs/error_log
    
    java.lang.NoClassDefFoundError: Could not initialize class com.github.scribejava.core.extractors.OAuth2AccessTokenJsonExtractor

    The above error is caused by and outdated plugin jar in /data/gerrit/plugins/oauth.jar
    which is downloaded from 
    https://gerrit-ci.gerritforge.com/view/Plugins-stable-3.5/job/plugin-oauth-bazel-master-stable-3.5/
    and placed into plugins directory and after restarting gerrit service following functionalities are tests 

    • Login and logout using Google Auth
    • Clone Repository
    • Browse Repository Source
Write a comment...