Novell is now a part of Micro Focus

AppNote: Interoperability of Cisco PIX 500 and NBM 3.8 VPN

Novell Cool Solutions: AppNote
By Sreekanth Settipalli

Digg This - Slashdot This

Posted: 28 Oct 2004

Cool Solutions AppNote: Cisco PIX 500 Series Firewall - Interoperability with NBM 3.8 VPN

S Sreekanth
Senior Software Engineer, Novell

Abstract: This AppNote describes in detail the configuration steps to be followed for setting up an IPsec tunnel between the Novell BorderManager 3.8 VPN Gateway and the Cisco PIX 500 Series Firewall.


This AppNote discusses the configuration of Novell BorderManager 3.8 VPN and Cisco PIX 515 Firewall to run a simple virtual private network (VPN) tunnel between them, over the Internet or any public network using IP Security (IPsec). IPsec is a combination of open standards that provides data confidentiality, data integrity, and data origin authentication between IPsec peers.

In this example, a pre-shared key is used to join the protected networks behind them. The joined networks are the 10.10.10.x private network inside the Novell BorderManager 3.8, and 20.20.20.x private network inside the Cisco PIX 515 Firewall. An IPsec tunnel between the two gateways enables the traffic between the protected networks to be transmitted securely over the public Internet.

This AppNote is intended for network administrators who plan to set up an encrypted channel between the gateways, thereby saving the cost of having expensive leased lines between the corporate branch offices.


IPsec negotiation can be broken down into five steps, including two Internet Key Exchange (IKE) phases:

  1. An IPsec tunnel is initiated by interesting traffic. Traffic is considered interesting when it is traveling between the IPsec peers.
  2. In IKE Phase 1, the IPsec peers negotiate the established IKE Security Association (SA) policy. Once the peers are authenticated, a secure tunnel is created using Internet Security Association and Key Management Protocol (ISAKMP).
  3. In IKE Phase 2, the IPsec peers use the authenticated and secure tunnel to negotiate IPsec SA transforms. The negotiation of the shared policy determines how the IPsec tunnel will be established.
  4. The IPsec tunnel is created and data is transferred between the IPsec peers based on the IPsec parameters configured in the IPsec transform sets.
  5. The IPsec tunnel terminates when the IPsec SAs are deleted or when their lifetime expires.

Note: IPsec negotiation between the two peers will fail if the SAs on both of the IKE phases do not match on the peers.


It is assumed you are familiar with installation and configuration of Novell BorderManager 3.8 VPN and Cisco PIX 500 Firewall. This document does not discuss the installation or network configuration of the gateways.

It is also assumed that traffic from inside the PIX and inside the NBM to the Internet (represented here by the 137.65.X.X networks) is flowing prior to beginning this configuration.

Components Used

The information in this document is based on the following software and hardware versions:

  • Cisco PIX 500 Series Firewall Platform 515
  • Cisco PIX Version 6.2(3) or later
  • Novell BorderManager 3.8 VPN
  • Novell BorderManager 3.8 Support Pack 2 or later

The information in this document is based on devices in a specific lab environment. All these devices used in this document started with a cleared (default) configuration. If you are working in a live network, ensure that you understand the potential impact of any command before using it. The information in this document may also apply to other models of the Cisco PIX 500 Series.

Network Diagram

This document uses the network setup shown in the diagram below.

Figure 1: Network Setup


  • NBM - Novell BorderManager
  • IKE - Internet Key Exchange
  • IPsec - Internet Protocol Security
  • ESP - Encapsulation Security Payload
  • SA - Security Association
  • VPN - Virtual Private Network
  • PSK - Pre-Shared Key
  • DH - Diffie-Hellman

A: Configuring the Cisco PIX 515

The Cisco PIX 515 can be configured by performing the following steps, each of which is explained in detail below:

  1. Configuring the Interfaces
  2. Configuring IKE for Pre-shared Keys
  3. Configuring IPsec
  4. Configuring Network Address Translation (NAT)
  5. Configuring PIX System Options

Task A1 - Configuring the Interfaces

Configure the public (outside) and the private (inside) interfaces. In this example, the public IP address ( is assigned by the DHCP server, and the private interface is configured as

config# ip address outside dhcp
config# ip address inside

Task A2 - Configuring IKE for Pre-shared Keys

Enable IKE on the IPsec terminating interface by using the isakmp enable command. In this scenario, the outside (public) interface is the IPsec terminating interface.

config# isakmp enable outside

Define the IKE policies that will be used during the IKE negotiations by using the isakmp policy command. When using this command, you must assign a priority level so that the policies are uniquely identified. In this case, the priority of 9 is assigned to the policy.

The policy is also set to do the following things:

  • Use a pre-shared key
  • Use the SHA hashing algorithm for data authentication
  • Use DES for the Encapsulating Security Payload (ESP)
  • Use Diffie-Hellman group1
  • Set the Security Association (SA) lifetime to 28800 seconds

config# isakmp policy 9 authentication pre-share
config# isakmp policy 9 encryption des
config# isakmp policy 9 hash sha
config# isakmp policy 9 group 1
config# isakmp policy 9 lifetime 28800

Finally, configure the pre-shared key (in this example, 'novell') and assign a peer address by using the isakmp key command. The same pre-shared key must match on the NBM and Cisco peers when using pre-shared keys.

config# isakmp key novell address netmask 

Task A3 - Configuring IPsec

IPsec is initiated when one of the peers receives traffic that is destined for the other peer's inside network. This traffic is deemed interesting - it needs to be protected by IPsec. An access list is used to determine which traffic will initiate the IKE and IPsec negotiations. The access list shown below permits traffic to be sent from the 20.20.20.x network, via the IPsec tunnel, to the 10.10.10.x network. The access list on the NBM's configuration will mirror this access list.

config# access-list 90 permit ip

The IPsec transform set defines the security policy the peers will use to protect the data flow. The IPsec transform is defined by using the crypto ipsec transform-set command. A unique name (in this example, 'mytransform') must be chosen for the transform set, and up to three transforms can be selected to define the IPsec security protocols. The following configuration only uses two transforms: esp-sha-hmac and esp-des.

config# crypto ipsec transform-set mytransform esp-des esp-sha-hmac

Crypto maps serve the purpose of setting up IPsec SAs for the encrypted traffic. To create a crypto map, you must assign a map name (in this case, 'toNBM') and a sequence number ('20'), and define the crypto map parameters. The crypto map shown below uses IKE to establish IPsec SAs. It encrypts anything that matches access-list 90. It also has a set peer (in this example, NBM Public IP address) and uses the 'mytransform' transform-set to enact its security policy for traffic.

config# crypto map toNBM 20 ipsec-isakmp
config# crypto map toNBM 20 match address 90
config# crypto map toNBM 20 set peer
config# crypto map toNBM 20 set transform-set mytransform
config# crypto map toNBM 20 set security-association lifetime seconds 7200 kilobytes 4608000

After defining the crypto map, apply it to the IPsec terminating interface (in this scenario, 'outside').

config# crypto map toNBM interface outside

Task A4 - Configuring Network Address Translation (NAT)

The following command tells the PIX not to NAT any traffic deemed as interesting for IPsec. Thus all traffic that matches the access-list command statements will be exempt from the NAT services.

config# nat (inside) 0 access-list 90

Task A5 - Configuring the PIX System Options

Because all inbound sessions must be explicitly permitted by an access list or a conduit, the sysopt connection permit-ipsec command is used to permit all inbound IPsec-authenticated cipher sessions. With IPsec protected traffic, the secondary conduit check could be redundant and cause the tunnel creation to fail. The sysopt command tunes various PIX Firewall security and configuration features.

config# sysopt connection permit-ipsec

Sample Configuration File

The following configuration file is extracted from the Cisco PIX 515 firewall configured for the setup.

PIX Configuration
: Saved
: Written by enable_15 at 14:31:03.847 UTC Thu Oct 21 2004
PIX Version 6.2(3)
nameif ethernet0 outside security0
nameif ethernet1 inside security100
enable password 9Dr33fLjHLsNxGE6 encrypted
passwd 9Dr33fLjHLsNxGE6 encrypted
hostname pixfirewall
fixup protocol ftp 21
fixup protocol http 80
fixup protocol h323 h225 1720
fixup protocol h323 ras 1718-1719
fixup protocol ils 389
fixup protocol rsh 514
fixup protocol rtsp 554
fixup protocol sqlnet 1521
fixup protocol sip 5060
fixup protocol skinny 2000
no fixup protocol smtp 25
fixup protocol sip udp 5060
access-list acl_in permit ip any any
access-list 90 permit ip

pager lines 24
logging on
logging buffered errors
interface ethernet0 auto
interface ethernet1 auto
mtu outside 1500
mtu inside 1500
ip address outside dhcp
ip address inside
ip audit info action alarm
ip audit attack action alarm
pdm history enable
arp timeout 14400
nat (inside) 0 access-list 90
nat (inside) 1 0 0
static (inside,outside) netmask 0 0
access-group acl_in in interface inside
conduit permit gre any any
timeout xlate 3:00:00
timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 rpc 0:10:00 h323
0:05:00 sip 0:30:00 sip_media 0:02:00
timeout uauth 0:05:00 absolute
aaa-server TACACS+ protocol tacacs+
aaa-server RADIUS protocol radius
aaa-server LOCAL protocol local
no snmp-server location
no snmp-server contact
snmp-server community public
snmp-server enable traps
tftp-server outside post_vpn_config_update.txt
floodguard enable

!- Ipsec Configuration

sysopt connection permit-ipsec
no sysopt route dnat
crypto ipsec transform-set mytransform esp-des esp-sha-hmac
crypto map toNBM 20 ipsec-isakmp
crypto map toNBM 20 match address 90
crypto map toNBM 20 set peer
crypto map toNBM 20 set transform-set mytransform
crypto map toNBM 20 set security-association lifetime seconds 7200
kilobytes 4608000
crypto map toNBM interface outside

!- IKE Configuration

isakmp enable outside
isakmp key novell address netmask
isakmp policy 9 authentication pre-share
isakmp policy 9 encryption des
isakmp policy 9 hash sha
isakmp policy 9 group 1
isakmp policy 9 lifetime 28800

telnet inside
telnet inside
telnet timeout 20
ssh timeout 5
terminal width 80
: end

B: Configuring Novell BorderManager 3.8 VPN

Complete the following tasks to configure the Novell BorderManager 3.8 VPN, using the iManager configuration tool from a Web browser.

Task B1: Adding the NBM server to the VPN Server List

  1. Select NBM VPN Configuration > NBM VPN Server Configuration > VPN Server List > Add.
  2. Enter the server name with context and click Next. Do not select the Site to Site or Client to Site role at this point.
  3. Enter the NBM server public address and subnet mask.
  4. Enter the Tunnel Address and subnet mask. The Tunnel Address is not used with third-party interoperability, but it is required for completing the configuration.
  5. Make sure that the Key Life Time is set to 480 minutes (IKE Key Life Time) and that Perfect Forward Secrecy is disabled.
  6. Click OK. The entire certificate-related configuration will be created automatically.

After this step, the NBM server (PRV-BM.novell, in this example) is added to VPN Server List (path: NBM VPN Configuration > NBM VPN Server Configuration > VPN Server List). Note: Key Life Time and Perfect Forward Secrecy values should be the same as the ones configured in the Cisco PIX.

Figure 2: Adding the NBM server.

Task B2 - Creating Site to Site Configuration

  1. Select NBM VPN Configuration > NBM VPN Server Configuration > VPN Server List.
  2. Click the server name you created in Task B1.
  3. Select Site to Site Role as Master.
  4. Click Create.

  5. Figure 3: Selecting NBM server as Master.

  6. Select the subject name of the server certificate and add the protected local network (10.10.10.x) behind the NBM server.
  7. Disable IP RIP.
  8. Save the configuration by clicking Apply and then OK.

After this step, Site to Site service (VPNS2SPRV-BM.novell, in this example) is created with NBM server as the master in the Site to Site Member list. This list is found under NBM VPN Configuration > VPN Site To Site Configuration > Member List.

Figure 4: Creating Site to Site configuration

Task B3: Adding Cisco PIX Firewall as a slave to the Site To Site Member List

  1. Select VPN Site To Site Configuration > VPNS2S Service > Member Lists > Site To Site Member List > Add.
  2. Select Server Name as "toCiscoPIX" with Server Address as the Cisco peer address.
  3. Make sure the Tunnel Address and subnet mask are entered, even though they are not used for third-party interoperability.
  4. Select Non-BorderManager VPN with Authentication Method as PSS (Pre-Shared Secret).
  5. Enter the same PSS key that was entered in the Cisco VPN configuration.
  6. Add the protected remote network (20.20.20.x) behind the Cisco Server.
  7. Click Apply.

Figure 5: Adding the Cisco Server as a Slave Member in Site to Site configuration.

The Site to Site Member List will have the NBM server as the master server and the Cisco server as the slave server.

Task B4: Configuring Third-party Traffic Rules

  1. Select VPN Site To Site Configuration > VPNS2S Service > 3rd Party Traffic Rules > New.
  2. Enter the traffic rule name (cisco_allow_10net_to_20net, in this example) and make sure the rule is enabled.
  3. Select Cisco PIX Server Address as the third-party Server Gateway Address.
  4. Add the third-party (Cisco PIX) server protected network (20.20.20.x, in this example).
  5. Add the NBM server protected network (10.10.10.x, in this example).
  6. Define the action as Encrypt with IPsec Key Life Time as 120 minutes, Encryption as DES, and Authentication as HMAC-SHA1.
  7. Click Apply and then OK.

Note: Make sure the IPsec Proposal values entered for Key Life Time, Authentication and Encryption match the ones entered in the Cisco PIX configuration. The default rule says to deny all the packets. So, make sure that the new rule you add comes above the default rule in the list of third-party traffic rules, to avoid packet denial.

Figure 6: Configuring Third-party Traffic Rules (cont'd.)

Figure 7: Configuring Third-party Traffic Rules (concluded)

TroubleShooting Cisco PIX Firewall

Show, clear, and debug Commands

This section provides information you can use to confirm your configuration is working properly. Note: The clear commands must be performed in configuration mode. Before issuing debug commands, please see Important Information on Debug Commands.

  • show crypto ipsec sa - Displays the current status of the IPsec security associations and is useful in determining if traffic is being encrypted.
  • show crypto isakmp sa - Shows the current state of the IKE security associations.
  • clear crypto isakmp sa - (from configuration mode) Clears all active IKE connections.
  • clear crypto ipsec sa - (from configuration mode) Deletes all IPsec security associations.
  • debug crypto ipsec - Shows if peers are negotiating the IPsec portion of the VPN connection.
  • debug crypto isakmp - Shows if the peers are negotiating the ISAKMP portion of the VPN connection.
  • debug crypto engine - Displays debug messages about crypto engines, which perform encryption and decryption.


The configuration of the IPsec tunnel between Novell BorderManager and Cisco PIX Firewall is discussed in this document. The results described in this document are derived from test scenarios. Novell recommends that you always verify configuration changes on a simulated test network before you deploy directly into a production environment.

Additional Information

For NBM 3.8 VPN configuration and administration:

For NBM 3.8 VPN Troubleshooting:

For NBM 3.8 Monitoring:

For Cisco PIX IPsec Configuration and Troubleshooting:

Novell Cool Solutions (corporate web communities) are produced by WebWise Solutions.

© Copyright Micro Focus or one of its affiliates