Configuring OpenSwan IPSec Server


The intent of this article is to walk through the installation, configuration, and general debugging of OpenSwan based IPSec tunnels. Though primarily focused on Ubuntu & Debian systems, non-package management portions should apply generally.

Installation & Initial Configuration

We install using apt-get or aptitude.

% apt-get install openswan

For a basic PSK (Private Shared Key) configuration, there are two main files we need to modify. The first is the configuration file, /etc/ipsec.conf. The second is the ipsec secrets file where the PSK is stored in /etc/ipsec.secrets.

The ipsec.conf file

There are two main sections to the ipsec configuration file. The configuration setup section & the connection section.

The configuration setup section

The configuration setup section begins by declaring

config setup

. The 'virtual_private 'option declares which subnets should be allowed through the tunnel and which should be excluded. Typically, you'll wish to exclude any networks that overlap with your private LAN.

Networks are allowed by specifying the internet protocol version and the CIDR of the allowed network. Networks are excluded by prepending the bang symbol, '!', to the CIDR.

Assuming a private network of on a /24 network, the setup configuration would look something like...

config setup

If either end of the tunnel is being nat'd, it may be necessary to add the nat_traversal work around to the config setup section.


The plutoopts specified create a log for each peer that connects. The protostack specifies which kernel protocol to use. 'auto' is generally fine. However, if you wish to apply firewall rules to traffic coming over the tunnel, you may wan to look into the KLIPS protocol as it creates a unique interface for tunnel traffic, greatly reducing the complexity of your firewall scripts.

The connection configuration section

The connection configuration section begins with the declaration of the 'conn' keyword followed by an arbitrary connection label. An example configuration might look like...

conn State-Secrets
    aggrmode=no    left=
    right=    leftsubnet=

We'll walk through each of the above connection options. It is important to note that though the protocol allows for client negotiation of most of these parameters with the server, in practice I've found explicitly setting them to conform to the remote servers specifications produces the most reliable results. If you are acting as the client, without control of the server, obtaining a thorough specification document that lists phase1 and phase2 algorithms can be quite the time saver. Such a document for the above configuration might look like

Configuration Table
Configuration Option Configuration Value
Athentication Method Pre Shared Key
Phase 1 Encryption Scheme IKE
Phase 1 DH Group 2
Encryption Algorithm AES/256
Hashing Algorithm SHA-1
Main or Aggressive Mode Main Mode
Phase 1 Lifetime 1440
Phase 2 Encapsulation ESP
Phase 2 Encryption Algorithm AES/256
Phase 2 Authentication Algorithm SHA
Phase 2 Perfect Forward Secrecy No
Phase 2 Lifetime 3600
Key Exchange for Subnets Yes

Let's walk through each of the options, look at their corresponding value from our document, and clarify its use.


This declares the type of connection to be formed. Since we're configuring an ipsec tunnel, we specify 'tunnel' (I know, very original).


This option declares the type of authentication scheme to use. IPSec supports several different authentication schemes. 'secret' is used due to our specification document specifying a type of "Pre Shared Key".


There's no corresponding document value for this setting. It specifies that the ipsec tunnel should be started and routes created when the ipsec daemon itself starts.


This value comes from our documents PFS setting.


This is a combination of several values in our document. It specifies the phase 1 encryption scheme, the hashing algorithm, and the diffie-hellman group. The modp1024 is for Diffie-Hellman 2. Why 'modp' instead of dh? DH2 is a 1028 bit encryption algorithm that modulo's a prime number, e.g. modp1028. See RFC 5114 for details or the wiki page on diffie hellmann, if interested.

The bang symbol, !, specifies the connection shoudl be formed in strict mode. Technically, this is the default. However, at some point (version wise) it was not being properly applied and it doesn't hurt.


This is a combination of several values in our document. It specifies the phase 2 encryption scheme, the hashing algorithm, and the diffie-hellman group just like the ike parameter.

For further information on valid parameters for the ike and phase2alg variables, please see the ipsec.conf man page.


Enables or disables Aggressive Mode. Agressive Mode is almost never needed and 'no' is the default. However, I've included it for clarity.


The 'left' and &39;right' arguments specify details about the two end points of the tunnel. Which is used for which endpoint is arbitrary, but many used left to denote the 'local' endpoint and the right to denote the 'remote' endpoing for mnemonic reasons. This should be the ip of the peer we're attempting to connect to.


These are the subnets that are exposed by the tunnel. Again, left and right is arbitrary. However, consistency is obviously required. Make sure the subnet declaration matches its corresponding left/right endpoing declaration.


Configuration is normally the easy portion of setting up an ipsec tunnel, it's normally the debugging that takes up the majority of time. Particularly if dealing with heterogenous peers.

Pertinent log files

These are the log files most likely to contain important information about why a particular tunnel is not succeeding.

  • The /var/log/auth.log is where logs on the authentication transactions are, stored.
  • The /var/log/syslog is the system log, it's always good to check here for any startup or general errors.
  • /var/log/pluto/peer/a/b/c/d/a.b.c.d.log file is the per peer pluto log. You can find some of the startup and shutdown information in here.


The main utilities to use are

  • tcpdump
  • ip
  • ipsec


Since the traffic is encrypted, you can only track the "flow" of the ipsec transaction to ensure all packets are reaching their destination and routing is working as expected. However, seeing the packet flow can be very informative.


The ip tool is used to ensure that the needed routes have been crated. Alternatively, you could use something like netstat.


Read the man page for complete information, but the most used switch is the status command. We configured our example connection for 'auto'. To retrieve the status of all auto configured connections, you execute

% ipsec auto --status

When debugging, I find it very useful to poll the output of this command in one second intervales. For example,

% for (( ;; ));do ipsec auto --status; sleep 1; done

The astute may notice the 'watch' command was made for this. I experienced glitches on some systems with ipsec status and the watch command. YMMV.

The ipsec.secrets file

The ipsec.secrets file takes the following format

leftid1 rightid1 : PSK "preshared key1"
leftid2 rightid2 : PSK "preshared key2"

The secrets file can have multiple entries. Of note, the combination of the leftid1 + rightid1 must be unique for each tunnel in order for the PSK lookup to succeed.

If you're only managing a single ipsec tunnel, using the the special catch all %any can work well. Such a config would look like this

%any %any : PSK "preshared key"

'Gotchas' to keep an eye out for

There are a few problems that seem to come up over and over and over again. I cover those here in brief along with ways to eliminate them as suspects. . . or at least ways to confirm if not completely eliminate.

Mismatched Peer Identities

This comes up quite a bit. Keep an eye out for messages from the peer like 'no proposal found for peer' or connections that make it through phase 1, but fail to complete phase 2. Another way to verify is by inspecting the status output. You'll find a line like this:<>[+S=C]...<>[+S=C]===; erouted; eroute owner: #6

That line indicates that the ips of the endpoints are being used as the peer id. It also declares the subnet being exposed. The standard default for peer id&39;s is the ip of the peer. However, this is not required. Nor is it uncommon for it to be overridden by a remote admin. If you suspect a peer id mismatch, the remote peer admin should be able to provide you with her routers configuration.

Peer id's must match.

Mismatched configurations

This is pretty vague. In the end, virtually all connection problems boil down to mismatched configurations. However, it still bears mentioning as it is eminently worth it to go over configurations repeatedly to ensure they match up.

Share on: TwitterFacebookGoogle+Email

Comments !