For the past four days I was trying to configure 2 instances of OpenESB instances with SSL Mutual authentication. I initially thought it is straightforward that I just need to import their respective certificates in the truststore of their counterparts. But it proved more difficult than that. This blog entry is my notes on one of the hurdle I faced and how I resolved it.

What is Mutual Authentication?
In General, SSL authentication is used in websites to authenticate the server. Which means that the server has to authenticate to the client by sending a digital certificate signed by a well known trusted CA. If the CA is trusted by the browser, the website is safe and you will see the address bar highlighed in green.

In Mutual Authentication, in addition to server authentication, the client also has to present its certificate to the server. The server verifies it by checking if it is signed by a trusted CA and if it is tampered. If both server and client authenticated themselves, then SSL authentication is a success.

How does mutual authentication actually works in Java?
Keystores and truststores are the important pieces to get this work. Both keystores and truststores are storage files for private keys, public keys and certificates. But there is a big difference.
Keystore: Typically, it stores the identity information about the subject. In our case we need to store the server’s certificate along with its private key and certificate chain.
Truststore: This holds the various certificates of Certifying Authorities. It need not store an individual’s certificate or a server’s certificate. Although technically they can be stored.

In a SSL Mutual Authentication scenario, these are the overall steps that take place during SSL handshake:

1. Client initiates the request.
2.  The server sends its own certificate which is found from its keystore. (I still need to figure out how it identifies a single certificate as its own certificate if there are a bunch of certificates in its keystore)
3.  The client verifies its certificate if it can be trusted. If the server’s certificate or its CA’s certificate are found in truststore, then the server is authenticated.
4.  If client authentication is enabled at server side, the server requests’s for client’s certificate.
5.  The client sends its own certificate which is found from its keystore.
6.  The server verifies the client’s certificate if it can be trusted. If the client’s certificate or its CA’s certificate are found in its truststore, then the client is authenticated.

So, to make the SSL authentication mechanism simpler we can simply import the server’s certificate in client’s truststore and vice versa. This works best in test environment or in protected intranet. But in real world scenario this will not be the case. The server may have to deal with multiple unknown clients. And the clients may have to deal with unknown servers. That is where Certifiying Authorities help us. If the CA is trusted, the server/client is trusted.

Problem faced:
Default keystores from glassfish works. But genuine certificates did not work.
By default glassfish comes with a keystore and truststore. Both the keystore and truststore have a certificate each with alias of s1as. I exported the certificate using keytool and imported into the other server’s truststore. It works great. Then I obtained two certificates from aces-3.orc.com/ca. One for server and one for client. I converted the certificate into x509 format using openssl. Then I imported server’s certficate in server’s keystore and client’s truststore. And imported client’s certificate in client’s keystore and client’s truststore. This did not work.

When I send a request from client to server, I get these errors:

In client’s log:

Caused by: HTTP transport error: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake Caused by: Remote host closed connection during handshake Caused by: SSL peer shut down incorrectly

In server’s log:

javax.net.ssl.SSLHandshakeException: no cipher suites in common

Turn on SSL debugging:
After googling for the above errors and solutions, I decided to debug SSL handshake. To turn on SSL debuggin, all I had to do is set the system property at startup javax.net.debug=all.

More information about SSL debugging is available here: http://docs.oracle.com/javase/1.5.0/docs/guide/security/jsse/ReadDebug.html
After looking into the logs and comparing the cipher suites printed, I still had no clue why the above error was thrown. From the log I can clearly see there are many cipher suites in common between both server and client.

After investigating for few hours by comparing the default keystore and truststore with what I created, I found the problem.

The problem was with Server’s keystore. I had imported the server’s certificate in keystore using keytool.
When I print the details from keystore, this is what I got:

C:\openesb-standalone-0.0.1-SNAPSHOT-Server>keytool -list -keystore serverkeystore.jks
 Enter keystore password:
 Keystore type: JKS
 Keystore provider: SUN

Your keystore contains 1 entry

serveridentity, 23 Dec, 2013, trustedCertEntry,
Certificate fingerprint (MD5): A1:EF:1E:0C:53:29:7C:7F:23:42:85:B2:50:B5:50:2F

Upon comparing this with the default certificate in keystore.jks, I recongnized that the entry type is different. It had PrivateKey, instead of trustedCertEntry. So what I imported is wrong. I have to import the certificate’s PrivateKey

Importing a certificate with its private key:
To import a certificate with its private key we need to package the certificate and its private key along with certificate chain in a PKCS12 file. Fortunately I had written a tool sometimeback to do this. Once you have a PKCS12 file, you can convert it into keystore using the keytool command : “keytool -importkeystore”
The full command I used is :

keytool -importkeystore -deststorepass changeit -destkeypass changeit -destkeystore logicoy-id-with-ca-chain.p12.jks -srckeystore

After specifying the server keystore as logicoy-id-with-ca-chain.p12.jks, the above error in server disappeared.

Similarly import the PKCS12 certificate chain into client’s keystore. Import the CA’certificates into both server and client’s truststore. Now both the server and client trust each other because their respective certificates are issued by the CA’s they trust.

I faced several other problems during this configuration, but above was the important one that gave a breakthrough. So the best method to follow always is

Keystore: Store the individual/server’s certificate along with its private key and certificate chain

TrustStore: Store the CA’s certificate. If there is a certificate chain, convert it into PKCS#7 file and store it together.

Following the above rule will make it easier to configure this setup.