Software Upgrade

The maintenance release mr9.5 and newer uses the new upgrade approach commonly known as 'A/B Upgrade'

ngcp-upgrade requires a special partitioning schema of disk subsystem. The server has 3 separate partitions:

ngcp-data - it stores data files, like databases files, logs, etc.

ngcp-root and ngcp-fallback - they are equal size and contain the software, OS files and NGCP packages. One of them is the current root '/' partition, the another one is mounted as /ngcp-fallback directory.

See more details in The default disk partitions

During the upgrade the second partition is formatted and the target version is installed into it. After the reboot the system will be started from this partition and previous one becomes the /ngcp-fallback directory.

Please, pay particular attention that this partitioning schema is mandatory and if your system doesn’t have it - create it beforehand. You can reinstall node from the peer and re-join the cluster. WARNING: This is the only supported upgrade schema. The old, in-place upgrade is not supported and technically not possible.

Release Notes

The Sipwise C5 version mr9.5.1 has the following important changes:

  • Network interfaces were renamed from eth* to neth* [TT#105151]

  • Add possibility to play to the caller an emulated ringback tone after pre-call announcement. See Emulated Ringback Tone for more details [TT#120551]

  • Add possibility to play an announcement in Auto-Attendant before terminating the call in case of selection timeout [TT#119455]

  • [PRO/Carrier] Enable 'kamailio.proxy.skip_pbx_loop' config.yml option by default [TT#105874]

  • Add support of 'SO_REUSEPORT' socket option (TCP/IP stack). Reusing socket descriptors already bound to listening at LB, for initiating new TCP connections from behalf of Sipwise C5. Applies also to TLS. [TT#124801]

  • Add support of TCP transport, as well as TLS for outbound registrations. [TT#124801]

  • Extend DTX jitter buffer to all other codecs supported for transcoding. [TT#122401]

  • Add option to use generic comfort noise generator instead of the native AMR SID noise generator to handle AMR SID/DTX. [TT#122401]

  • Providing Kamailio’s secsipid modules, which implement the IETF extensions for Secure Telephone Identity (RFC8224, RFC 8588), known as STIR and SHAKEN [TT#105600]

  • A/B Upgrade schema, ngcp-upgrade installs a new release to the second 'ngcp-fallback' partition [TT#104381]

  • New alarm based SNMP traps (in addition to the existing event based ones). See External Monitoring Using SNMP for more details. [TT#108956]

Some important technical points for those interested:

  • Removes support for obsolete legacy binary Apple APNS protocol [TT#120601]

  • CSC mixed mode to use Perl and JS CSC interfaces in parallel in mr9.5 LTS [TT#122026]

  • The soundset announcements announce_before_call_setup, announce_before_cf, announce_before_recording and announce_to_callee have been moved from 'early_rejects' section to the new 'early_media' section [TT#120551]

  • User preference "force_inbound_calls_to_peer" has now always priority over CFNA [TT#83601] The new priority only applies if CALLEE has "force_inbound_calls_to_peer" set to "force_offline*" or CALLER has "force_outbound_calls_to_peer" set to "force_offline*"

  • [PRO/Carrier] 'ngcp-approx-snapshots' is a new tool to sync Approx caches between LAB and PROD systems [TT#116404]

  • To avoid MAC address changes on virtual interfaces (e.g. with network bonding), the system is running with 'MACAddressPolicy=none' (see /etc/systemd/network/99-default.link) [TT#123500]

  • Applied further systemd services security hardenings (haveged + ngcp-rate-o-mat) [TT#76552]

  • Automatic manual page database updates have been disabled for speed reasons (if you need 'apropos(1)' or 'whatis(1)' tools, invoke systemctl restart man-db.service) [TT#122950]

  • [PRO/Carrier] The git remote origin of /etc/ngcp-config is set to shared_name:/mnt/glusterfs/ngcpcfg-share on all nodes. Shared_name is sp for PRO and web01 for Carrier. No longer relevant /mnt/glusterfs/ngcpcfg-share directory on non-mgmt Carrier nodes get automatically deleted during upgrades [TT#123200]

  • [PRO/Carrier] As part of upgrade, the new GlusterFS v9.2 will be installed to current Debian/buster system on both nodes in pair (active and standby) prior the upgrade to Debian/bullseye [TT#104381]

Some major software related changes/updates [TT#118659]:

  • Debian/bullseye v11 (updated from Debian/buster, v10)

  • Linux kernel v5.10 (updated from v4.19)

  • GCC v10 (updated from v8)

  • MariaDB v10.5 (updated from v10.3)

  • [PRO/Carrier] GlusterFS v9.2 (updated from v5.5)

  • Perl v5.32 (updated from v5.28)

  • Python v3.9 (updated from v3.7)

  • Python2 is no longer shipped ('python' binary is no longer available, use 'python3' in custom scripts instead)

  • OpenSSL v1.1.1k (updated from v1.1.1d)

  • OpenSSH v8.4p1 (updated from v7.9p1)

  • Kamailio v5.5.1 (updated from v5.4.5)

Deprecation notice:

  • Perl-based CSC (Customer Self-Care) interface has been deprecated in mr6.5 LTS and will be removed in mr10.x. Please ensure JS-based CSC is in use [TT#122032]

  • With the introduction of a new ngcp-rest-api service that re-uses port 2443, that was previously used by the legacy SOAP interface, the ossbss section in config.yml is going to be removed in mr10.x as most of the parameters there are not used any longer, and ones that are still in use will be moved to the corresponding existing or new sections [TT#124400]

  • The InfluxDB-based monitoring backend is deprecated and is fully replaced with the Prometheus-based backend. InfluxDB will be removed in mr10.0 [TT#129100]

  • The feature 'SIP over Websockets' is deprecated and will be removed in mr10.0 [TT#122032]

Please find the complete changelog in our release notes on our WEB site.

Overview

The Sipwise C5 software upgrade procedure to mr9.5.1 will perform several fundamental tasks:

  • format fallback partition

  • install Debian to fallback partition

  • install NGCP packages to fallback partition

  • copy current configuration to fallback partition

  • upgrade the NGCP configuration schema in fallback partition

  • update grub configuration so after reboot the current fallback partition becomes the active one

  • reboot to new partition. This steps needs to be taken care manually during maintenance window

  • upgrade the NGCP database schema

Please make sure to have remote access to the system via out-of-band management (like IPMI, iLO, IMM, iDRAC, KVM, etc)

Sipwise C5 CARRIER is a PRO-style system that has "A" and "B" sets of nodes with specific roles. The number of nodes can differ between installations and must be clarified before the upgrade at the planning stage.

The software upgrade is usually performed by Sipwise engineers according to the following steps:

  • create the software upgrade plan

  • execute pre-upgrade steps: patchtt, customtt, backups

  • make all "B" nodes (on CARRIER) or the sp2 node (on PRO) active

  • ensure that all "A" nodes (on CARRIER) or the sp1 node (on PRO) are standby

  • perform the software upgrade on all "A" nodes (on CARRIER) or the sp1 node (on PRO)

  • schedule and make services switchover to all "A" nodes (on CARRIER) or the sp1 node (on PRO)

  • ensure that "A" nodes (on CARRIER) or the sp1 node (on PRO) perform well (otherwise, perform a switch back)

  • perform the software upgrade on all "B" nodes (on CARRIER) or the sp2 node (on PRO)

  • perform the system post-upgrade testing and cleanup

The only allowed software upgrade path is the one described above. The nodes sp1/s2 (or a/b) MUST be used as described in this document. All the other theoretically possible upgrade scenarios can lead to unpredictable results.

Planning a software upgrade

Confirm the following information:

  • which system should be upgraded (LAB/LIVE, country, etc.)

  • the date and time schedule for each of the steps above (keeping the time zone in mind)

  • a confirmed timeframe for the upgrade operation (allowed switchover timeframe)

  • the basic functionality test (BFT) to be executed before the start of the software upgrade and after the switchovers to ensure that the new release does not show critical issues (the BFT scenario should be prepared by the customer engineers)

  • actions to be taken if the software upgrade operation cannot be completed within the defined maintenance window

  • contact persons and ways of communication in case of emergency

  • ensure that the customer and/or Sipwise engineers have access to the virtual consoles of the servers: KVM, iDRAC, AMM

Pre-upgrade checks

It is recommended to execute the preparatory steps in this chapter a few days before the actual software upgrade. They do not cause a service downtime, so it is safe to execute them during peak hours.

Log into the C5 standby management node

This should be web01a on CARRIER and sp1 on PRO.

Use the static server IP address so you can switch between the nodes.

Run the terminal multiplexer under the sipwise user (to reuse the Sipwise .screenrc settings that are convenient for working in multiple windows):

screen -S my_screen_name_for_ngcp_upgrade

Become root inside your screen session:

sudo -s

Check the overall system status

Check the overall system status:

ngcp-status --all

Make sure that the cluster health status is OK: Check the nodes in parallel, using the clish command:

  • ngcp-clish "ngcp version summary" - ensure that all cluster nodes have correct/expected from version

  • ngcp-clish "ngcp version package installed ngcp-ngcp-pro" ensure that the metapackages version is equal to the ngcp version above

  • ngcp-clish "ngcp version package check" - ensure that all nodes have the identical Debian package installed

Software must be identical on all nodes (before and after the upgrade!)

  • ngcp-clish "ngcp cluster ssh connectivity" - check SSH connectivity from the current node to all other nodes

  • ngcp-clish "ngcp cluster ssh crossconnectivity" - check SSH cross-connectivity from all nodes to all other nodes

  • ngcp-clish "ngcp service summary" - all required services must be running on corresponding nodes

  • ngcp-clish "ngcp cluster status" - active node(s) (with all services running) must print "active", the other(s) must print "standby"

  • ngcp-clish "ngcp status collective-check" - all checks must be OK

  • ngcp-clish "ngcp show date" - date and time must be in sync on all the servers

  • ngcp-clish "ngcp show dns-servers" - ensure that the DNS configuration is consistent among the nodes

  • ngcp-clish "ngcp cluster replication" - check all MySQL replications statuses on all nodes.

to exit from 'ngcp-clish' press Ctrl+Z (or type exit): 
# ngcp-clish
Entering 'clish-enable' view (press Ctrl+Z to exit)...
# exit
#

Check access to license server and license validity

Check from within the system (better from all nodes, for extra safety) that the license server is accessible from the network point of view, and that these commands do not end with timeouts or HTTP errors:

ping -c 3 -w 5 license.sipwise.com

curl --head https://license.sipwise.com/

Also ensure that:

  • /proc/ngcp/check contains the string "ok" (if not, check logs)

  • and that there are no errors or important warnings in /ngcp-data/logs/licensed.log (/var/log/ngcp/licensed.log in systems before mr6.5).

Evaluate and update custom modifications

For the below steps, investigate and make sure you understand why the custom modifications were introduced and if they are still required after the software upgrade. If the custom modifications are not required anymore, remove them (e.g. if a bug was fixed in the target release and the existing patch becomes irrelevant).

Create tickets to Sipwise developers to make relevant custom modifications part of the product in future releases. This allows you to get rid of the customtt files one day.

If you directly change the working configuration (e.g. add custom templates or change the existing ones) for some reason, then the system must be thoroughly tested after these changes have been applied. Continue with the software upgrade preparation only if the results of the tests are acceptable.

Find the local changes to the template files:

ngcp-customtt-diff-helper

The script will also ask you if you would like to download the templates for your target release. To download the new templates separately, execute:

ngcp-customtt-diff-helper -d

In the tmp folder provided by the script, you can review the patchtt files or merge the current customtt with the new tt2 templates, creating the new customtt.tt2 files. Once you do this, archive the new patchtt/customtt files to reapply your custom modifications after the software upgrade:

ngcp-customtt-diff-helper -t

Find all available script options with the "-h" parameter.

Check system integrity

Log into all the servers.

Open separate windows for all the servers inside your "screen" session. (Press Ctrl+a + c to open a new window, Ctrl+a + a or Ctrl+a + [0-9] to change the window. Ctrl+a + " shows the list of all your windows. Use Ctrl+a + A to change the window names to corresponding hosts).

Changes made directly in tt2 templates will be lost after the software upgrade. Only custom changes made in customtt.tt2 or added by patchtt.tt2 files will be kept. Hence, check the system for locally modified tt2 files on all nodes:

ngcp-status --integrity

Check the configuration framework status

Check the configuration framework status on all nodes. All checks must show the "OK" result and there must be no actions required:

ngcpcfg status

On a CARRIER, check the replication on both central DB servers and on ports 3306 and 3308 of all the proxy servers. Ensure that all the proxy nodes replicate the read-only DB (127.0.0.1:3308) from the db01a node. Otherwise, discuss a special plan to address your particular configuration.

On a PRO, check the replication on both nodes.

The result must always show:

Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Seconds_Behind_Master: 0

Test the cluster failover to see if everything works fine as well on "B" nodes (on a CARRIER) or the second node (on a PRO). On all the standby nodes execute:

ngcp-make-active

Create two test subscribers or use the credentials for existing ones. Register subscribers with the platform and perform a test call to ensure that call routing and media flow are working fine.

Run "apt-get update" on all nodes and ensure that you do not have any warnings and errors in the output.

If the installation uses locally specified mirrors, then the mirrors must be switched to the Sipwise APT repositories (at least for the software upgrade). Otherwise, the public Debian mirrors may not provide packages for old Releases anymore or at least provide outdated ones!

Check access to deb.sipwise.com

Ensure that both management nodes have access to deb.sipwise.com by executing the following commands on a management node:

ngcp-approx-cache --check --node sp1

ngcp-approx-cache --check --node sp2

The checks must show only 200 OK results. If you see cannot connect!, Received 404 or any other error, check these possible causes:

  • A node does not have a connection to deb.sipwise.com

  • The deb.sipwise.com whitelist does not have the node’s public IP (IPv6) address.

License check

The Sipwise C5 — starting from mr6.5.1 release — enforce software licensing restrictions in form of a regular comparison of the licensed services and capacities against the actual usage patterns of the platform. In case some functionalities are enabled but not licensed, an error in syslog will be reported and the impacted services will be automatically deactivated.

Before proceeding with the upgrade, please take some time to check that all the modules not licensed are actually disabled in config.yml file. To verify if they are enabled execute the following commands:

ngcpcfg get sems.prepaid.enable
ngcpcfg get sems.prepaid.inew.enable
ngcpcfg get sems.sbc.xfer.enable
ngcpcfg get pbx.enable
ngcpcfg get pushd.enable
ngcpcfg get intercept.enable
ngcpcfg get voisniff.admin_panel
ngcpcfg get voisniff.daemon.li_x1x2x3.enable
ngcpcfg get voisniff.daemon.start
ngcpcfg get tpcc.enable
ngcpcfg get websocket.enable

If the output of one of the commands is 'yes' but the module is not licensed, you have to deactivate it. For example, in case of pre-paid billing module execute:

ngcpcfg set /etc/ngcp-config/config.yml sems.prepaid.enable=no
ngcpcfg apply 'Disable prepaid module'
ngcpcfg push all
Please, pay particular attention to pre-paid billing module because it is enabled by default.

Check that the packages ngcp-system-tools-pro, ngcp-ngcpcfg and ngcp-ngcpcfg-ha are installed at the latest version

For the upgrade to this version it is important that these packages are installed at their latest available version, to not cause problems during the upgrade.

To achieve this, execute the following commands: In management node, refresh the list of packages available:

ngcp-approx-cache-helper -a

In every node, upgrade the packages:

apt-get update
apt-get install ngcp-system-tools-pro ngcp-ngcpcfg ngcp-ngcpcfg-ha

Pre-upgrade steps

Sipwise C5 can be upgraded to mr9.5.1 from previous LTS release, or any non-LTS release since the previous LTS-release. The script ngcp-upgrade will find all the possible destination releases for the upgrade and makes it possible to choose the proper one.
If there is an error during the upgrade, the ngcp-upgrade script will request you to solve it. Once you’ve fixed the problem, execute ngcp-upgrade again and it will continue from the previous step.

The upgrade script will ask you to confirm that you want to start. Read the given information carefully, and if you agree, proceed with y.

The upgrade process will take several minutes, depending on your network connection and server performance. After everything has been updated successfully, it will finally ask you to reboot your system. Confirm to let the system reboot (it will boot with an updated kernel).

ngcp-upgrade options

The following options in ngcp-upgrade can be specially useful in some instances of upgrade:

  • --step-by-step: confirm before proceeding to next step. With this option the upgrade operation is performed confirming every step before execution, with the possibility to instruct to continue without confirming further steps until the end (if confirmation is only needed for some steps at the beginning).

  • --pause-before-step STEP_NAME: pause execution before step, given by the name of the script (e.g. "backup_mysql_db"). This option can be useful in several scenarios, for example:

    • to help to debug problems or work around known problems during upgrades. In this case the operator can pause at a given step known to be problematic or right before a problematic set, perform some manual checks or changes, then continue the upgrade until another step (with confirmation like with the recent option --step-by-step), or continue without stop until the end

    • another use might be to help to speed up upgrades when it involves several nodes: they can all proceed in parallel when it’s known to be safe to do so; then perform some parts in lock-step (some nodes waiting until others finish with some stage); then continue in parallel until the end

  • --skip-db-backup: This will speed-up the process in cases where it’s deemed unnecessary, and this is very likely in the upgrade of nodes other than the first.

Preparing for maintenance mode

Sipwise C5 introduces Maintenance Mode with its mr5.4.1 release. The maintenance mode of Sipwise C5 will disable some background services (for instance: ngcp-mediator) during the software upgrade. It thus prevents the system from getting into an inconsistent state while the upgrade is being performed. You can activate maintenance mode by applying a simple configuration change as described later.

  • Pull pending configuration (if any):

ngcpcfg pull

  • Enable maintenance mode:

ngcpcfg set /etc/ngcp-config/config.yml "general.maintenance=yes"

  • Apply configuration changes by executing:

ngcpcfg apply 'Enabling maintenance mode before the upgrade to mr9.5.1'
ngcpcfg push all

Download new package metadata into the approx cache (on the standby node only)

Customers with far-sighted software upgrade policies usually have pre-production installations to test the services in their environment before upgrading the production platform. In this case, the approx cache should be updated on both platforms simultaneously to synchronize the package versions between them, hence consider carefully before executing this step.

To download the latest packages metadata into the approx cache, execute the following command on the standby management node. This action ensures that all nodes within the platform have identical packages after the software upgrade:

ngcp-approx-cache-helper --update --skip-all-repos --extra-ngcp-release mr9.5.1 --extra-debian-release bullseye

Upgrading Sipwise C5 CARRIER

Log in to all nodes and execute the checks from Pre-upgrade checks again. This will ensure that nothing was broken since the preparation steps were finished. Also, execute ngcpcfg show and ngcpcfg status to check the latest configuration changes.

Perform the BFT test.

To upgrade Sipwise C5 CARRIER to mr9.5.1 release, execute the following commands on the standby management "A" node:

Upgrading ONLY the first standby management node "A" (web01a/db01a)

Sometimes the DB and MGMT roles are assigned to the same host. This is OK.
Do NOT execute the software upgrade on web01a and db01a in parallel!

Run the following command node to install the package responsible for upgrading Sipwise C5 to a newer release:

ngcp-prepare-upgrade mr9.5.1
Don’t worry, ngcp-upgrade-carrier does not exist, ngcp-upgrade-pro is used in Carrier too.
Do not use "ngcpcfg apply/build" after executing the steps from the above section, otherwise the changes will be overwritten and you will have to redo these steps. The same applies to similar sections below.

Run the upgrade script on the standby node as root:

ngcp-upgrade

There is "stop" step in upgrade scenario, upgrader stops there. At this time the new system in fallback partition is ready so you can reboot the server to boot from mr9.5.1 system.

After reboot run ngcp-upgrade again with the same parameters so upgrade will continue with post-upgrade steps.

Custom configuration templates

Merge/add the custom configuration templates if needed.

Apply the changes to configuration templates:

ngcpcfg apply 'apply customtt/patchtt for new the release mrX.X on xxx01a'

Send the new templates to the shared storage and the other nodes

ngcpcfg push --nobuild --noapply all
Do NOT execute ngcpcfg push --shared-only at this stage, as it will affect further upgrades due to noticed outdated local ngcpcfg storage. If you did so, run ngcpcfg push --nobuild --noapply all once again to pull ngcpcfg changes on all the nodes from glusterfs.

Upgrading the standby database node "A" (db*a)

If the DB and MGMT roles are assigned to the same host, then skip this step as you have already upgraded the standby MGMT node "A" above.

Run the following commands to upgrade the standby DB node "A" (select the same release version as above and follow the on-screen recommendations):

ngcp-prepare-upgrade mr9.5.1
ngcp-upgrade
It is important to upgrade db01a node before upgrading any proxy nodes. Otherwise, the "local" MySQL (127.0.0.1:3308) on proxy nodes may become out of sync in case the new release has _not_replicated.up DB statements.

There is "stop" step in upgrade scenario, upgrader stops there. At this time the new system in fallback partition is ready so you can reboot the server to boot from mr9.5.1 system.

After reboot run ngcp-upgrade again with the same parameters so upgrade will continue with post-upgrade steps.

Upgrading other standby nodes "A" (lb*a/prx*a)

Run the below commands selecting the same release version and follow the on-screen recommendations:

ngcp-prepare-upgrade mr9.5.1
ngcp-upgrade

You can reboot the server after the 'stop' step reached by ngcp-upgrade and after server is back online, run ngcp-upgrade with the same parameters.

Useful options in ngcp-upgrade

The following options in ngcp-upgrade can be useful for this phase of upgrades, because it is very likely that the backup was already performed:

  • --skip-db-backup: This will speed-up the process in cases where it’s deemed unnecessary.

See a more detailed description of the options in: ngcp-upgrade options

Promote ALL standby nodes "A" to active.

Ensure that all standby nodes "A" are: * upgraded to the new release (check /etc/ngcp_version or use ngcp-clish) * have been rebooted (run 'ngcp-status' on each standby node)

Prior to the promotion, the DB has to be updated with information from the new configuration. This should be done as close as possible to the activation of the upgraded nodes, to minimize the changes to the active service with the old release.

Execute on one of the standby nodes as root, for example on db01a:

MYSQL_VALUES_UPDATE_BEFORE_SWITCHOVER=true /etc/ngcp-config/templates/etc/ngcp-provisioning-tools/mysql_values.cfg.services

On all "A" nodes run:

ngcp-make-active

Ensure that the "A" nodes became active, by executing the 'ngcp-status' and 'ngcp-clish' commands described above.

Ensure that ALL "B" nodes are standby now!

Upgrading ALL standby nodes "B" (web*b/db*b/lb*b/prx*b)

Run the following commands selecting the same release version and following the on-screen recommendations:

ngcp-prepare-upgrade mr9.5.1
ngcp-upgrade

You can reboot the server after the 'stop' step reached by ngcp-upgrade and after server is back online, run ngcp-upgrade with the same parameters.

You can upgrade all standby "B" nodes simultaneously (including the ones with the mgmt and db roles).

Useful options in ngcp-upgrade

The following options in ngcp-upgrade can be useful for this phase of upgrades:

  • --step-by-step: confirm before proceeding to next step.

  • --pause-before-step STEP_NAME: pause execution before step, given by the name of the script (e.g. "backup_mysql_db").

See a more detailed description of the options in: ngcp-upgrade options

Upgrading Sipwise C5 PRO

Make sure you are prepared to spend about two hours upgrading the system. Note that a short service downtime is possible during the services switchover to the upgraded node.

Start with the software upgrade on the standby sp1 node. Then, switch the services over to the upgraded node and upgrade the other (now standby) sp2 node, as described in the steps below.

Upgrade the first PRO node

Execute the upgrade script on the standby node as root:

ngcp-prepare-upgrade mr9.5.1
ngcp-upgrade

At this time the new system in fallback partition is ready so you can reboot the server to boot from mr9.5.1 system.

After reboot run ngcp-upgrade again with the same parameters so upgrade will continue with post-upgrade steps.

Useful options in ngcp-upgrade

The following options in ngcp-upgrade can be useful for this phase of upgrades:

  • --step-by-step: confirm before proceeding to next step.

  • --pause-before-step STEP_NAME: pause execution before step, given by the name of the script (e.g. "backup_mysql_db").

See a more detailed description of the options in: ngcp-upgrade options

The custom modification handling (optional)

Introduce the custom changes to configuration templates if needed (by adding corresponding patchtt/customtt files). Apply the changes to configuration templates and send them to the shared storage and the other node:

ngcpcfg apply 'Added custom changes when the first node was upgraded'
ngcpcfg push --nobuild --noapply
Starting with mr9.0 release, sems instances has been unified under the name of sems-b2b. All the configuration files of sems are now stored in a single folder: '/etc/sems-b2b'. All the existing sems customizations must be moved under the corresponding new ngcp template folder_/etc/ngcp-config/templates/etc/sems-b2b/_ and adapted accordingly in order to have effect. For example, if the file /etc/ngcp-config/templates/etc/ngcp-sems/etc/reg_agent.conf.customtt.tt2 was created to configure a peering outbound registration, after the upgrade the customization must be moved to /etc/ngcp-config/templates/etc/sems-b2b/etc/reg_agent.conf.customtt.tt2.

Promote the upgraded standby node to active

Prior to the promotion, the DB has to be updated with information from the new configuration. This should be done as close as possible to the activation of the upgraded node, to minimize the changes to the active service with the old release.

Execute on the current standby node as root:

MYSQL_VALUES_UPDATE_BEFORE_SWITCHOVER=true /etc/ngcp-config/templates/etc/ngcp-provisioning-tools/mysql_values.cfg.services
ngcp-make-active

Upgrade the second PRO node

Go to the new standby node. Run the upgrade script as root:

ngcp-prepare-upgrade mr9.5.1
ngcp-upgrade

You can reboot the server after the 'stop' step reached by ngcp-upgrade and after server is back online, run ngcp-upgrade with the same parameters.

Useful options in ngcp-upgrade

The following options in ngcp-upgrade can be useful for this phase of upgrades, because it is very likely that the backup was already performed in the upgrade of the first node:

  • --skip-db-backup: This will speed-up the process in cases where it’s deemed unnecessary.

See a more detailed description of the options in: ngcp-upgrade options

Post-upgrade steps

Disabling maintenance mode

In order to disable the maintenance mode, do the following:

  • Pull outstanding ngcpcfg changes (if any):

ngcpcfg pull

  • Disable the maintenance mode:

ngcpcfg set /etc/ngcp-config/config.yml "general.maintenance=no"

  • Apply the changes to configuration templates:

ngcpcfg apply 'Disable the maintenance mode after the upgrade to mr9.5.1'
ngcpcfg push all

Post-upgrade checks

When everything has finished successfully, check that replication is running. Check ngcp-status --all. Finally, do a basic functionality test. Check the web interface, register two test subscribers and perform a test call between them to ensure call routing works.

You can find a backup of some important configuration files of your existing installation under /ngcp-data/backup/ngcp-mr9.5.1-\* (where \* is a place holder for a timestamp) in case you need to roll back something at any time. A log file of the upgrade procedure is available at /ngcp-data/ngcp-upgrade/$FROM-mr9.5.1/logs/.

Applying the Latest Hotfixes

If your current release is already the latest or you prefer to be on the LTS release, we still suggest applying the latest hotfixes and critical bug fixes.

Execute all steps as described in Pre-upgrade checks. They include the system checks, customtt/patchtt preparation and others. It is important to execute all the steps from the above chapter.

On a CARRIER it is suggested to promote B-nodes to active and start the update with A-nodes.

Update the approx cache on the standby management node

The main goal of the following command is to download the new packages into the approx cache. So all the nodes in the cluster will get identical packages.

ngcp-approx-cache --auto --node localhost

Apply hotfixes on the standby management node

ngcp-update

Recheck or update the custom configuration templates

Merge/add the custom configuration templates if needed.

Apply the changes to configuration templates:

ngcpcfg apply 'apply customtt/patchtt after installing the latest packages'

Send the new templates to the shared storage and the other nodes.

ngcpcfg push --nobuild --noapply all

Apply hotfixes on all other standby nodes (CARRIER-only)

ngcp-update

Promote the standby nodes to active

Execute on the standby nodes as root:

ngcp-make-active

Check in a minute that the nodes became active:

ngcp-check-active

Apply hotfixes on the second node

ngcp-update

Execute the final checks as described in the Post-upgrade checks section.