Oracle SOA Suite: Two-way SSL with TLS1.2 made easy (slightly less complicated)

Transport layer security is not an easy topic. Many blogs have been written about this already. Surprisingly though, I did not find a single blog which was more or less complete and provided me with everything I needed to know to get this working on SOA Suite 12.2.1. In this blog I try to make the topic more easy to understand and provide a complete end to end example.

Suppose you only want an implementation and do not care much about the explanation, you can skip the 'Some basics' section, only execute the commands in bold in the 'Lets get started!' section and the steps in the 'WebLogic and SOA Suite' section. Do take into consideration any existing SSL related configuration on your own system.

Some basics

SSL/TLS

SSL stands for Secure Sockets Layer. SSL is the predecessor of TLS. SSL should be considered insecure since in October 2014 the POODLE attack was announced. TLS currently has 4 versions. TLS 1.0, 1.1, 1.2 and 1.3. 1.3 is not widely supported/adopted yet. SSL/TLS provide integrity checks, security and authentication.

Identity

A server which hosts traffic on a port which has SSL/TLS enabled, has an identity keystore. This identity keystore contains a private key and a public key/certificate. The public key/certificate can safely be given to other parties. With websites when visiting an HTTPS website (HTTP with SSL enabled), the public key is send to you. The other party / client can use the public key to encrypt messages meant for the server. The only one who can decrypt the messages is the one having the private key of the server. This is usually only the server.

Trust

Can you trust a server? You can use a certificate authority to create a signed public key. If someone trust the certificate authority, that someone also automatically trusts the signed key. With websites you often see a green lock when a certain website uses HTTPS with a public certificate signed by a (by your webbrowser) trusted certificate authority.

Usually a truststore is used to store trusted certificate authorities or specific trusted certificates. If you have many servers in your application landscape, it is recommended to use a certificate authority since it is cumbersome to load every public key of every server in every truststore. Trusting a single certificate authority makes things a lot easier.

Certificate authority

A certificate authority has a private key which it can use to sign a so-called certificate signing request. From this certificate signing request you can create a signed public key.

Certain companies such as Google and Microsoft provide certain checks to confirm someones identity before providing them with a signed public key. You can pay these companies to provide those checks and give you a signed certificate. Most of these companies are trusted certificate authorities by default in several OSs and browsers. This way for a website for example, you do not have to make changes on a client for your certificate to be trusted.

If you run several servers within your internal company network, you often do not require these external checks. You can create your own certificate authority private key and create a signed public key yourself. This certificate authority is not trusted by default so you should trust the public certificate of your self-signed certificate authority in order establish trust.

Cipher

A cipher is an algorithm for encryption and decryption. With SSL, during the handshake phase (the phase which establishes an SSL session), a cipher is determined. The client usually provides a list of the ciphers it supports and the server chooses which one to use. During an SSL handshake you can see in logfiles which cipher is chosen.

Lets get started!

I used 2 SOA Suite 12.2.1.2 installations (complete, no quickstart) in 2 different VM's for this example. soaserver1 and soaserver2. I used a host-only network with fixed IP's in VirtualBox and added IP/hostname mappings in the hosts files of the two servers.

Create a self-signed certificate autority

A blog explaining the topic on creating your own certificate authority can be found on here. This is just my short summary with some corrections. Do read it for some easy to understand background information.

This simple example uses OpenSSL. OpenSSL is installed by default on most Linux environments and can also be installed on other OSs.

First create a private key for your certificate authority:

openssl genrsa -des3 -out rootCA.key 2048

I create an RSA key and protect it with the DES3 cipher algorithm based on a password. I want my key to have a length of 2048 bytes. You can also choose for ECC keys. They can be smaller when comparing to RSA keys to provide the same level of protection. ECDSA (Elliptic Curve Digital Signature Algorithm) ciphers use ECC keys. Keep this key private! It allows you to sign public keys (see later in this post) and create trust.

Next I self-sign this generated key. This creates a public signed key for the certificate authority. I can load this key in truststores to achieve trust for keys which are signed with this certificate:

openssl req -x509 -new -nodes -key rootCA.key -sha256 -days 1024 -out rootCA.pem -subj '/CN=Conclusion/OU=Integration/O=AMIS/L=Nieuwegein/ST=Utrecht/C=NL' -extensions v3_ca

Lets break this down:

• req: do a request
• x509: this defines the format of the key to be generated. In the x509 standard, several pieces of metadata can be stored with the certificate and the certificate authority structure is also part of the x509 standard. Read more here.
• new: generate a new key
• nodes: this is actually 'no DES'. My public key does not need to be protected with a password.
• key: specifies the private key to sign
• sha256: secure hash algorithm. Hashing is used to provide data integrity functionality. Creating a hash of a transmission allows you to check at a later time if the transmission has been tampered with.
• days: specifies the validity of the generated certificate
• subj: provides some metadata for the certificate
• extensions v3_ca: this adds a metadata field to the certificate indicating that it is a certificate of a certificate authority. If this extension is not added, certain validations might fail

You can use the certificate autority private key and certificate as server identity but you shouldn't. This will give certain validation errors because of the 'extensions v3_ca'.

Create server identity keys

Next we create a private key which will be used as identity of the WebLogic server

openssl genrsa -des3 -out soaserver1.key 2048

After we have created this private key, we can create a certificate signing request for this private key

openssl req -new -key soaserver1.key -out soaserver1.csr -subj '/CN=soaserver1/OU=Integration/O=AMIS/L=Nieuwegein/ST=Utrecht/C=NL'

This is pretty similar as to what we have done for the certificate authority. However  mind the subj clause here. The common name should match the server hostname. This will be used later for verification of the identity of the server by the client. In order to allow two-way SSL, I added the server hostname to IP mapping to every servers hosts file. In an enterprise you would use a DNS (domain name system) for this since you do not want to maintain every mapping in every server locally.

Next sign the certificate using the information in the private key and certificate of the certificate authority.

openssl x509 -req -in soaserver1.csr -CA rootCA.pem -CAkey rootCA.key -CAcreateserial -out soaserver1.crt -days 1024 -sha256

This is very similar to signing the certificate authority certificate. Mind that a validity with a longer period than the validity of the certificate authority key is of course useless. Createserial creates a new file rootCA.srl. This serial number is unique for every signed certificate. You should save it so at a later time you can check if a certificate has been tampered with.

The next time you sign a certificate, you can use:

openssl x509 -req -in soaserver1.csr -CA rootCA.pem -CAkey rootCA.key -CAserial rootCA.srl -out soaserver1.crt -days 1024 -sha256

This will increase the previous serial with 1, making sure it is unique. 

Creating an identity keystore

Now you have a signed certificate and a private key. Time to make a Java keystore (JKS) which can be used in WebLogic server and SOA Suite and other pieces of Java.

openssl pkcs12 -export -in soaserver1.crt -inkey soaserver1.key -chain -CAfile rootCA.pem -name "soaserver1" -out soaserver1.p12

keytool -importkeystore -deststorepass Welcome01 -destkeystore soaserver1identity.jks -srckeystore soaserver1.p12 -srcstoretype PKCS12

The above steps; 

• creating a private key
• creating a certificate signing request
• signing the certificate with the private key of the certificate authority
• creating an identity keystore

 need to be done for every server.

Creating a trust keystore

Here you can pick the fruits of the above work of using a certificate authority to sign your server private keys. You can use the certificate authority certificate in a truststore and every key signed with the certificate is trusted. You do not need to load every specific server certificate into every truststore the server needs access to. Creating a truststore is easy and you can do this once and use the same trust.jks file in all your servers.

keytool -import -alias rootCA -file rootCA.pem -keystore trust.jks -storepass Welcome01

WebLogic and SOA Suite

It is interesting to notice the incoming WebLogic configuration differs from the SOA Suite outgoing configuration. This is of course not surprising since a server usually only has a single identity, but an integration product like SOA Suite should able to interact with multiple protected external resources, maybe requiring different ciphers and keys for each of them. Also SOA Suite in the past (I'm not sure if that is still the case) could run on IBM WebSphere instead of WebLogic Server. Thus I can understand Oracle chose to provide a more generic implementation of SSL in the SOA Suite than the WebLogic specific one.

WebLogic

The WebLogic server configuration is pretty straightforward. In this example I'm only looking at SSL for incoming and outgoing messages for SOA Suite. The WebLogic specific configuration is only relevant for incoming connections. Basically the steps are as followed:

• Enable SSL for the managed server
• Specify keystores for identity and trust
• Configure incoming SSL specifics
• Force the server to use TLS1.2

Enable SSL for the managed server

First Enable the listen port for SSL. In WebLogic console, environment, servers, specify your server, configuration, general and indicate 'SSL Listen port enabled'. You can also specify the SSL port here.

Specify the keystores for identity and trust

 In WebLogic console, environment, servers, specify your server, configuration, keystores. You can specify the identity and trust keystores you have created during the above steps.

Configure incoming SSL specifics

 In WebLogic console, environment, servers, specify your server, configuration, SSL. You can specify the identity key used for the server and several checks which can be done when establishing the SSL connection.

Some important settings: 

• BEA Hostname verifier. This indicates the CN field in the certificate is checked against the server hostname.
• Client certs requested and enforced. If set, Two-Way SSL will be used and the client won't be able to connect unless it presents a certificate.
• Buildin SSL Validation and Cert Path Validators. This checks the certificate chain.

It is important to understand what these checks do. A host name verifier ensures the host name in the URL to which the client connects matches the host name in the digital certificate that the server sends back as part of the SSL connection. This helps prevent man in the middle attacks where the client might connect to a different URL.

The below situation is something you won't prevent even with this checks. I could connect without problems with the soaserver2 WebLogic server from soaserver1 with the certificate of soaserver2. Also when using the private key of soaserver1 as identity on soaserver2, soaserver2 would not complain about this. FireFox would though and most likely also other clients.

Force TLS1.2

If you want to force WebLogic / SOA Suite to use TLS 1.2 you can specify the following JVM parameters in the setDomainEnv.sh file.

-Dweblogic.security.SSL.minimumProtocolVersion=TLSv1.2 -Dhttps.protocols=TLSv1.2

SOA Suite

The SOA Suite configuration is a bit more elaborate in that it requires configuration in different places of which not all can be done from the GUI.

The steps which need to be performed are:

• Specify the identity store used by SOA
• Create a keystore password credential in the credential store
• Configure composite to use two-way SSL

Specify identity store

First you have to specify the identity store which the SOA Suite will use for outbound connections. You can find this setting by going to SOA, soa-infra, SOA Administration, Common Properties, (scroll down), 'More SOA Infra Advanced Configuration Properties...'

Here you have to specify the servers identity keystore. In my case /home/oracle/certs/soaserver1identity.jks.

Create a keystore password credential in the credential store

Next you have to specify the keystore password. If you forget to do this, you will encounter errors like:

On the client:
<May 7, 2017, 12:58:43,939 PM CEST> <Error> <oracle.integration.platform.blocks.soap> <BEA-000000> <Unable to create SSL Socket Factory>

On the server:
[2017-05-07T12:26:02.364+02:00] [soa_server1] [NOTIFICATION] [] [oracle.integration.platform.common.SSLSocketFactoryManagerImpl] [tid: [ACTIVE].ExecuteThread: '25' for queue: 'weblogic.kernel.Default (self-tuning)'] [userId: <anonymous>] [ecid: cd874e6b-9d05-4d97-a54d-ff9a3b8358e8-00000098,0] [APP: soa-infra] [partition-name: DOMAIN] [tenant-name: GLOBAL] Could not obtain keystore location or password

You can set the keystore password by going to your domain, Security, Credentials. You can create a credential map SOA with a keyname/user of KeyStorePassword with the password you have used for your keystore. It will use the same password as key password. If it differs you can use the KeyPassword credential store entry.

Configure composite to use two-way SSL

This step is easy. You have to add a binding property to your reference which indicates you want to use two-way SSL.

In the composite.xml file on your reference you can add:

<property name="oracle.soa.two.way.ssl.enabled">true</property>

This causes the composite binding to use the identity (with the credential store password) for outbound SSL specified in the previously configured MBean.

You should of course also not forget to set the endpoint to a port which hosts HTTPS and indicate in the URL that it should use HTTPS to call this endpoint. In my example I've overridden the URL in the EM. Be aware though that overriding the endpoint URL might still cause the original endpoint to be called when the overridden endpoint is not accessible (if for example the SSL connection has issues).

Some useful tips

If you want to debug SSL connections, the following tips might help you.

FireFox

It might appear strange to use a webbrowser to test SSL connections, but which piece of software uses SSL more than a browser? FireFox is very clear in its error messages what has gone wrong which greatly helps with debugging. FireFox uses its own certificate store and can provide certificates to login to a server. You can configure them from FireFox, Preferences, Advanced, Certificates, View Certificates. Here you can import client certificates such as the p12 files you have generated in an earlier step.

This provides for a very easy way to check whether a server can be accessed with SSL and if the server certificate has been correctly generated / set-up. FireFox also extensively checks certificates to provide the green-lock icons people are quite familiar with.

In this case I have SSL enabled for soaserver1 on port 7002. I open https://soaserver1:7002 in FireFox (do not forget the HTTPS part). Since I have enabled 'Client certs requested and enforced' in the WebLogic SSL configuration, it will ask me for a client key.

In this case you can check whether a client certificate will be trusted. When opening https://soaserver1:7002 and you get a 404 message, the WebLogic server is responding to you after the SSL handshake has succeeded.

In FireFox you can tweak the cipher suites which are used. Read about this here. Do mind that SSL connections can be cached and FireFox can remember to send specific keys. If you run FireFox on soaserver1 and open a link on soaserver1, Wireshark (read below) will not detect traffic on the same interface which is used to access soaserver2.

Wireshark

Use Wireshark to monitor connections/handshakes.

• You can confirm the SSL/TLS version being used
• You can see the number of messages which have crossed the wire (allows you to distinguish retries of for example a handshake fails)
• Allows you to decrypt SSL traffic (if you have the private key)
• It allows you to confirm an SSL connection is actually being set up. If you do not see it in Wireshark, no message has been send and the connection build-up fails on the client. This for example happens when the SOA, KeyStorePassword entry has not been set in the SOA Suite credential store.

SSL debug logging

If you want to see what is happening with your SSL connection, it is very helpful to provide some JVM switches in setDomainEnv.

-Dweblogic.security.SSL.verbose -Djavax.net.debug=all -Dssl.debug=true 

You can also enable WebLogic SSL debugging in WebLogic console. Open a server and enable weblogic.security.SSL

Portecle

Portecle is a handy and freely available tool if you want to manage keystores and look at key details. 

Which service is called?

Suppose you have process A on server X which calls process B on server Y. For testing you first deploy process B on server X and use the WSDL of process B locally from process A. Next you override the endpoint to refer to the SSL port of server Y. What happens if the SSL connection cannot be established? By default, there are 3 retries after which the process falls back to using the endpoint as specified in the WSDL file. When testing it might seem the call from process A on X to B on Y works but it is actually a local call because the local call is the fallback for the remote call. In this case you should confirm an instance of B is created on Y.

You can also use the following binding properties in the composite.xml to tweak the failover behavior:

<property name="oracle.webservices.endpoint.failover">false</property>                              
<property name="endpointURI" many="false" override="may">HTTPENDPOINT</property>

Finally

Performance impact

Using SSL of course has a performance impact. 1-way SSL is faster than 2-way SSL. Using encryption is slower than not using encryption. Key length and cipher suites also play a major role in how fast your SSL connection will be. I have not measured the precise cost of the different options, but you should consider what you need and what you are willing to pay for it in terms of performance impact.

• One way SSL allows the client to verify the server identity (certificate, hostname). The server provides the client with a public key but not the other way around.
• Two way SSL also allows the server to verify the client. The client also needs to provide a public key.

SSL verifies host identities, keys, certificate chains. It does not allow you to provide (specific user) application authentication or authorization. You could do it with SSL but it would require giving every user a specific certificate. There are better ways to do that such as WS-Security, SAML or OAuth.

Entropy

If you use a server which has a lot of SSL connections, the random number generator is asked often for a new random number. Random numbers are generated by using entropy (a measure of randomness/disorder), which is a limited resource, especially in virtualized environments.

There is a setting which allows WebLogic server to recycle random numbers at the cost of security (the random number generator becomes predictable). Read more about that here.

-Djava.security.egd=file:/dev/./urandom

Oracle does not recommend using this recycling mechanism in production environments since if you can predict the random number generator, you have introduced a security vulnerability which can be exploited. Next to speeding up SSL connections, your server startup will most likely also be improved.

CRLs

I've not talked about a lot of things such as certificate revocation lists (CRLs). These lists contain keys which have been compromised. Compromised means the private key of a certificate authority has become public. Using the private CA key, someone is able to create new certificates which are being trusted by people who trust the CA. If a person can do such a thing, he is able to gain access to systems. Remember private keys can also be used to decrypt traffic? This is of course an issue on the internet but also when you have your own certificate authority. More generally speaking, if a private key is compromised, all trust should be revoked since you cannot count anymore on that for example a server is the sole owner of the key and is the only one who can decrypt traffic.

JDBC and SSL

Read more about this in the whitepaper here. It requires Oracle Advanced Security (OAS), which is an Oracle Database Enterprise Edition option. The US Government does not allow double encryption (you can imagine why..). If you configure Oracle Advanced Security to use SSL encryption and another encryption method concurrently, then the connection fails. See SSL Usage issues here.

Other things

I have not talked about securing the connection between managed servers in a cluster and between the NodeManager and managed servers. You can read more about that here. Do mind though that using trust can be more efficient than specifically putting every public key in every trust-store. Especially when you have many servers.