Novell is now a part of Micro Focus

Setting Up an IPSec VPN Tunnel between Nortel and an NBM 3.8.4 Server

Novell Cool Solutions: AppNote
By Chendil Kumar

Digg This - Slashdot This

Posted: 8 Nov 2006


Nortel* Contivity is a popular VPN product providing remote access and branch office connectivity to resources in the corporate network. Novell Border Manager is a VPN solution from Novell providing Remote Access and Branch office connection. Border Manager inter-operates with several other VPN products for both site-to-site connection and remote access.

This document describes the configuration required for interoperability between Nortel Contivity Server (V04_90.264) and NBM 3.8 VPN Server in the Pre-Shared Secret (PSS) Mode. It does not discuss the NAT features of IPSec or the configuration in Certificate Mode.

Figure 1 - Nortel/NBM VPn configuration


  • Both the servers must have the same Perfect Forward Secrecy (PFS) enabled. That is, PFS should be set to ON on both the server or OFF on both the servers.
  • The protected networks configured in both servers must match.
  • PHASE 2 (IPSEC) Encryption and Authentication Algorithms must also match.

Nortel Contivity Configuration

Defining Networks

You must define Networks for your local LAN and Remote LAN(s) before you define the Nortel Branch Office tunnel. If you have not already defined Networks, do the following:

1. Log in to the management console of the Nortel switch.

2. Select Profiles > Networks.

3. Specify a name for your local LAN. In this example, we have specified Nortel Lan.

4. Click Create.

Figure 2 - Setting up the Nortel LAN subnet

5. Enter the following information for the new subnet:

  • IP address: Specify the Network number
  • Mask: Specify the Subnet mask

6. Click Add.

7. Click Close.

IPSec and IKE Configuration

1. Log in to the management console of the Nortel switch.

2. Select Profiles > Branch Office.

3. Select a group to which you want your Branch office to belong, from the Group drop-down list.

4. Click Configure to edit the group. The Edit Group page is displayed.

Figure 3 - Edit Group page

5. Click Configure in the IPSec section.

6. Edit the required IPSec and IKE settings on this page.

7. Select IKE Encryption and authentication algorithms, IPSec Encryption and authentication algorithms, Rekey Timeout, and PFS.

Note: Click Configure in the Connectivity section and ensure that the value for Idle Time-out is specified as 00:00:00. If not, the Nortel Contivity kills the tunnel after 15 minutes of inactivity. This disables the inactivity feature: i.e., even if there is no data transfer through the tunnel, the tunnel will not be killed.

5. Click OK to save the configuration.

Nortel Branch Office Configuration

1. Select Profiles > Branch Office.

2. Click Add in the Connections section to add a new connection. The Add Connection page is displayed.

Figure 4 - Add Connection page

3. Fill in the following information:

  • Connection Name: Specify a name for the connection.
  • Control Tunnel: Select Disabled from the drop-down list, as we do not configure it as the control tunnel.
  • Tunnel Type: Specify the type of tunnel as IPSec, as we are configuring an IPSec tunnel.
  • Connection Type: Specify if your connection type is peer-to-peer, initiator, or Responder.

4. Click OK. The Connection Configuration page is displayed.

Figure 5 - Connection Configuration page

5. In the Connection section, select the Enable check box to enable the tunnel.

6. Enter the following information in the Endpoints section:

  • Local IP address: Select the public IP address of Nortel Contivity from the drop-down list.
  • Remote IP address: Specify the public IP address of Novell BorderManager.

7. In the Filter section, select Permit All from the drop-down list, for all traffic between Novell BorderManger and the Nortel box.

8. Select Text Pre-Shared Key from the Authentication drop-down list.

9. Specify the Pre-Shared Key value in the text boxes.

10. Because the networks are known and are not changed, select the IP configuration as Static. If you choose Dynamic, the routing protocol automatically determines the accessible networks based on information that is entered on the LAN Interfaces of the Nortel Contivity.

11. In the Local Networks field, select the local LAN you created in the "Defining Networks" section above. For our example, we'll use "Nortel Lan."

12. Click Add in the Remote Network section to add the network that is behind Novell Border Manager. The Configure Remote Network page is displayed.

Figure 6 - Configure Remote Network page

13. Enter the following information in the Remote Network section:

  • IP address: Specify the Network address.
  • IP Mask: Specify the subnet mask.
  • Cost: Specify the cost to reach the network.
  • Enabled: Select this check box to enable the added remote network.

14. Click OK.

15. Apply the changes in the Connection Configuration Page.

Novell Border Manager Configuration

Basic Server Configuration

1. In iManager, select NBM VPN Configuration > NBM VPN Server Configuration.

2. Click Add. The New VPN server Configuration page is displayed.

3. Select a server from tree and fill in the following fields:

  • Server Address: Specify the IP address and the subnet mask of the server in this section. For example, use /
  • Tunnel Address: Specify the IP address and the subnet mask of the tunnel. For example, use / for the Tunnel Address
  • Key Life Time: Specify the IKE Rekey life time. This should be the same for both Nortel and NBM Servers.
  • Perfect Forward Secrecy: Check or uncheck the PFS option, depending on what you have configured in Nortel Gateway. This should be the same for both Nortel and NBM Servers.

Leave the default values in the other fields unchanged.

4. Click OK to complete the configuration.

5. Select NBM Server Configuration, then select the site-to-site check box.

6. Click the Master radio button.

7. Click Details. An Issuer Certificate, which was automatically created, is displayed.

8. Check the Subject Name and then browse for the server certificate.

9. Click the Certificate Subject Name to display it.

10. Provide the Protected Network of the NBM 3.8 server in the Protected Networks list (in this case it would be /

Adding Nortel as a Member to the Server

1. In iManager select NBM VPN Configuration > VPN Site-to-Site Configuration.

Figure 7 - VPN Site-to-Site Configuration in iManager

2. Go to the Member Lists tab, then click Add.

3. Provide the IP Address and the subnet mask of the Nortel Switch, and provide one tunnel IP Address to the Nortel server in the same network as the NBM 3.8 server (in our case, /

4. Select the Non-Border Manager VPN checkbox.

5. Select PSS as the Authentication Method.

6. Specify the pre-shared key in the PSS Key field. You must specify the same pre-shared key you specified for the Nortel Switch. Note: The keys are case-sensitive.

7. In the Protected IP Networks and Hosts section, click Add.

8. Specify / This is the Protected Networks list of the Nortel Switch.

9. Click OK.

Adding Third-Party Traffic Rules in NBM 3.8

1. In iManager, select NBM VPN Configuration > NBM VPN Server configuration.

2. Go to the Third Party Traffic Rules tab, then click New.

3. Specify a name for the traffic rule in the Name field.

Figure 8 - Traffic rule name

4. Expand the 3rd Party Server Configuration panel, then select the IP address of Nortel Switch from the 3rd Party Server Gateway Address drop-down list.

5. Select the Only Use IP List option from the Rule Applies To radio button.

6. Click Add.

7. Specify the Public IP Address as given in the Nortel Switch (Third Party Server). For our example, the network IP Address is, and the Subnet Mask is

8. Expand the NBM Server Protected Network List Panel.

9. Select the Only Use IP List radio button under Rule Applies To.

10. Click Add and provide the network IP Address as it is given in the Nortel Switch (Third Party Server). In our case, the network is

11. Expand Define Action.

12. Select Encrypt, then select Encryption key Lifetime by time.

13. Specify the IPSec lifetime value. Select the Configuration and Authentication Algorithm. This should exactly match the Transform set given in the Nortel Switch.

14. Click Apply, then click OK.

Starting the Tunnel

You can start the tunnel either from Novell Border Manager or from the Contivity, as the tunnel was set up as a "peer-to-peer" on the Contivity side.

To start the tunnel,

1. In Contivity, select Profiles > Branch Office.

2. Click Test. It attempts to establish a session with the Novell Border Manager Gateway.

Common Problems / Troubleshooting

1. Problem: No Proposal Chosen.
Possible Cause: Contivity does not have the proposals enabled.
Solution: Make sure that the Contivity has the required proposals enabled in Profiles > Branch Office> /Base. It should match what is configured in Nortel.

2. Problem: Secret Mismatch
Possible Cause: The Pre-Shared Keys do not match.
Solution: Make sure the Pre-Shared Keys match.

3. Problem: Tunnel "dies" after a while.
Possible Cause: Idle Timeout is not set appropriately.
Solution: Make sure you have set the Idle Timeout on the Connectivity section to 00:00:00.

Error Logs and Messages

Border Manager messages are stored in the following locations:

  • IKE Logs
  • CSAudit logs

Nortel Contivity provides the following types of messages:

  • Status
  • Event Log
  • System Log
  • Security Log


Novell Border Manager 3.8.4 successfully interoperates with the Nortel Contivity Switch, for IPSec Site-To-Site VPN in PSS Mode. The Nortel Contivity Switch version V04_90.264 was used for the test.

Novell Cool Solutions (corporate web communities) are produced by WebWise Solutions.

© Copyright Micro Focus or one of its affiliates