Apache Geode

How Communication Works

Geode uses a combination of TCP and UDP unicast and multicast for communication between members. You can change the default behavior to optimize communication for your system.

Client/server communication and gateway sender to gateway receiver communication uses TCP/IP sockets. The server listens for client communication at a published address and the client establishes the connection, sending its location. Similarly, the gateway receiver listens for gateway sender communication and the connection is established between sites.

In peer systems, for general messaging and region operations distribution, Geode uses either TCP or UDP unicast. The default is TCP. You can use TCP or UDP unicast for all communications or you can use it as the default but then can target specific regions to use UDP multicast for operations distribution. The best combination for your installation depends in large part on your data use and event messaging.

TCP

TCP (Transmission Control Protocol) provides reliable in-order delivery of the system messages. TCP is more appropriate than UDP if the data is partitioned, if the cluster is small, or if network loads are unpredictable. TCP is preferable to UDP unicast in smaller clusters because it implements more reliable communications at the operating system level than UDP and its performance can be substantially faster than UDP. As the size of the cluster increases, however, the relatively small overhead of UDP makes it the better choice. TCP adds new threads and sockets to every member, causing more overhead as the system grows.

Note: Even when Geode is configured to use UDP for messaging, Geode uses a TCP connection when attempting to detect failed members. See Failure Detection and Membership Views for more details. In addition, the TCP connection’s ping is not used for keep alive purposes; it is only used to detect failed members. See TCP/IP KeepAlive Configuration for TCP keep alive configuration.

UDP Unicast and Multicast

UDP (User Datagram Protocol) is a connectionless protocol which uses far fewer resources than TCP. Adding another process to the cluster incurs little overhead for UDP messaging. UDP on its own is not reliable however, and messages are restricted in size to 64k bytes or less, including overhead for message headers. Large messages must be fragmented and transmitted as multiple datagram messages. Consequently, UDP is slower than TCP in many cases and unusable in other cases if network traffic is unpredictable or heavily congested.

UDP is used in Geode for both unicast and multicast messaging. Geode implements retransmission protocols to ensure proper delivery of messages over UDP.

UDP Unicast

UDP unicast is the alternative to TCP for general messaging. UDP is more appropriate than TCP for unicast messaging when there are a large number of processes in the cluster, the network is not congested, cached objects are small, and applications can give the cache enough processing time to read from the network. If you disable TCP, Geode uses UDP for unicast messaging.

For each member, Geode selects a unique port for UDP unicast communication. You can restrict the range used for the selection by setting membership-port-range in the gemfire.properties file. Example:

membership-port-range=1024-60000

Note: In addition to UDP port configuration, the membership-port-range property defines the TCP port used for failure detection. See the Reference for a description of the Geode property.

UDP Multicast

Your options for general messaging and for default region operations messaging is between TCP and UDP unicast. You can choose to replace the default with UDP multicast for operations distribution of some or all of your regions. For every region where you want to use multicast, you set an additional attribute on the region itself.

When multicast is enabled for a region, all processes in the cluster receive all events for the region. Every member receives each message for the region and has to unpack it, schedule it for processing, and then process it, all before discovering whether it is interested in the message. Multicasting is suitable, therefore, for regions that are of general interest in the cluster, where most or all members have the region defined and are interested in receiving most or all messages for the region. Multicasting should not be used for regions that are of little general interest in the cluster.

Multicast is most appropriate when the majority of processes in a cluster are using the same cache regions and need to get updates for them, such as when the processes define replicated regions or have their regions configured to receive all events.

Even if you use multicast for a region, Geode will send unicast messages when appropriate. If data is partitioned, multicast is not a useful option. Even with multicast enabled, partitioned regions still use unicast for almost all purposes.