Network issues with Coherence

I recently had to enable the normal unicast Coherence cluster required by the Oracle SOA suite on a set of Windows servers which was completely separated by local firewalls. It took a good bit of trial and error to get it to work, so I thought I would sum up what I learned during the endevaour.

Error Messages

Initially we saw a lot of deployment and Coherence errors with timeouts and connection issues like these:

[oracle.integration.platform.blocks.cluster.CoherenceClusterInterfaceImpl]: Constructor threw exception; nested exception is Timeout during service start: ServiceInfo(Id=0, Name=Cluster, Type=Cluster......

[2014-10-08T13:22:14.221+02:00] [soa_server02] [WARNING] [] [Coherence] [tid: Logger@1932191207] [userId: <anonymous>] [ecid: 0000KZjU_CZEKQG5Mzt1ie1KDHoj000002,0] [APP: soa-infra] 2014-10-08 13:22:14.221/375.070 Oracle Coherence GE <Warning> (thread=Cluster, member=n/a): Delaying formation of a new cluster; IpMontor failed to verify the reachability of senior Member(Id=1, Timestamp=2014-10-08 13:18:54.344, Address=, MachineId=3060, Location=site:,machine:exampleerver1,process:4864, Role=WeblogicServer); if this persists it is likely the result of a local or remote firewall rule blocking either ICMP pings, or connections to TCP port 7

<08-10-2014 15:48:30 CEST> <Error> <Coherence> <BEA-000000> <2014-10-08 15:48:30.468/517.608 Oracle Coherence GE <Error> (thread=Cluster, member=n/a): Detected soft timeout) of {WrapperGuardable Guard{Daemon=IpMonitor} Service=ClusterService{Name=Cluster, State=(SERVICE_STARTED, STATE_JOINED), Id=0, Version=3.7.1, OldestMemberId=1}}> 
<08-10-2014 15:48:30 CEST> <Error> <Coherence> <BEA-000000> <2014-10-08 15:48:30.499/517.639 Oracle Coherence GE <Error> (thread=Recovery Thread, member=n/a): Full Thread Dump

<08-10-2014 15:48:56 CEST> <Error> <oracle.integration.platform.blocks.deploy.coordinator> <BEA-000000> <Cluster communication initialization failed.  If you are using multicast, please make sure multicast is enabled on your network and that there is no interference on the address in use.  Please see the documentation for more details. Timeout during service start: ServiceInfo(Id=0, Name=Cluster, Type=Cluster

Needed Ports

During the normal configuration of Coherence you define, among other things, what port it should run on. What isn't apparent is that Coherence actually uses two ports. The one you configure + The one above. So if you configure it to use 8088, it also needs 8089. Ensure that both ports are open on TCP and UDP both. Besides that, the firewall needs to allow communication on port 7 (Echo) and to allow ICMP ping packages. This is used to keep the Coherence cluster alive.

Echo Service

Even though the Coherence documentation states that port 7 just needs to be open. I actually had to have a proper echo service respond. In windows this is done by adding the feature "Simple TCP/IP Services".

It took a bit of research to figure all of this out, so I hope it helps others having problems with Coherence clustering.

String concatenation in bash problems

Today I spent several hours figuring out how to do string concatenation in a bash script.

I was thinking that it should be quite easy just do

STRING_2= World

And of course it turns out that it is just that easy. However my problem was that I was using Notepad++ to edit the files (I know, bash on windows?). Without going into the discussion, let's just say it makes sense for my setup :)

But of course I got strange results, it seemed that the second string would be written "on top" of the first string. I finally figured out what the problem was and of course it was because the files was formatted with Windows line breaks and not Unix line breaks. I'm writing this because I need to remember to check these things every time I see strange behavior.

Now I just have to figure out how to force Notepad++ to default to using Unix line breaks.