Forming a Cluster#

As noted earlier, a cluster consists of a set of one or more nodes, all running grb_rs. This section explains how to form a cluster. Multi-node clusters provide additional capabilities relative to single-node clusters. For Compute Server, a multi-node cluster will automatically balance computational load among the various member nodes. For distributed algorithms, a multi-node cluster enables various algorithms to distribute work among multiple machines. This section begins by discussing the different types of nodes that are needed to support both Compute Server and distributed algorithms. Next, we will explain the grouping feature that can be used to create subsets of nodes to process some jobs. Finally, we will discuss the dynamic nature of a cluster. The system administrator can ask individual nodes to start or stop processing jobs, which makes it possible to smoothly add or remove nodes from a cluster to simplify maintenance or to scale processing capacity up or down.

Connecting Nodes#

Every Remote Services cluster starts with a single node. The steps for starting Remote Services on a single node, either as a standard process or as a service, were covered in earlier sections.

Before adding nodes into your cluster, you first need to make sure that the cluster token (property CLUSTER_TOKEN in the configuration file) has the same value in each node and in the Cluster Manager. For better security, we recommend that you change the predefined value of the token by generating a new one and pasting the same value into each node configuration file. You can generate a new token with the following command:

> grb_rs token
GRBTK-6o4xujs59WJO5508nmaNwc1TtjZJAL1UcwN4vTD4qK4nata8oLr9GnubyXrLTkggc/aw2A==

Adding nodes with a Cluster Manager#

If you have started a Cluster Manager, you add additional nodes using the exact same command you used to add the first node. You do this by providing the Cluster Manager address. The Cluster Manager acts as a registry of nodes of your cluster, and the nodes will then connect between themselves.

> grb_rs --manager=http://mymanager:61080 --port=61000

The MANAGER property can also be set through the configuration file:

MANAGER=http://mymanager:61080
PORT=61000

You won’t have the opportunity to provide command-line options when starting grb_rs as a service, so your only option in this case is to provide this information through the configuration file.

If you wish to start multiple grb_rs processes on the same machine for testing purposes (this is not recommended for production use), you will need to make sure each instance of grb_rs is started on a different port and using a different data directory. The command grb_rs init will help you by copying the default configuration and the data directory into a current directory.

For example, to start two nodes on the same machine with a hostname of myserver:

  1. In a first terminal window, create a new directory node1,

  2. Change your current directory to node1 and run grb_rs init

  3. Start the first node:

    grb_rs --manager=http://mymanager:61080 --port=61000

  4. In a second terminal window, create a new directory node2,

  5. Change your current directory to node2 and run grb_rs init

  6. Start the second node on a different port and connect to the Cluster Manager:

    grb_rs --manager=http://mymanager:61080 --port=61001

Adding nodes to a Self-Managed Cluster#

If you have not started a Cluster Manager, nodes must be connected to each other. Once you’ve started a single-node cluster, you can add nodes using the —-join flag to grb_rs or the JOIN configuration property. For example, if you’ve already started a cluster on the default port of server1, you would run the following command on the new node (call it server2) to create a two-node cluster:

> grb_rs --join=server1

In the log output for server2, you should see the result of the handshake between the servers:

info  : Node server1, transition from JOINING to ALIVE

Similarly, the log output of server1 will include the line:

info  : Node server2, added to the cluster

If you are using a non-default port, you can specify the target node port as part of the node URL in the —-join flag. You can specify the port of the current node using the —-port flag. You can use different ports on different machines, but it is a good practice to use the same one (port 61000 is typically a good choice). The command would look like this:

> grb_rs --join=server1:61000 --port=61000

The JOIN property can also be set through the configuration file:

JOIN=server1:61000
PORT=61000

Again, you won’t have the opportunity to provide command-line options when starting grb_rs as a service, so your only option in this case is to provide this information through the configuration file.

Once you’ve created a multi-node cluster, you can add additional nodes by doing a JOIN with any member node. Furthermore, the —-join flag or the JOIN property can take a comma-separated list of node names, so a node can still join a cluster even if one of the member nodes is unavailable. Note that when a list of nodes is specified, the joining node will try to join with all of the specified nodes at the same time. Joining is an asynchronous process, so if some target nodes are not reachable, the joining node will retry before giving up on joining. If all of the nodes are reachable, they will all join and form a single cluster.

If you wish to start multiple grb_rs processes on the same machine for testing purposes (this is not recommended for production use), you will need to make sure each instance of grb_rs is started on a different port and using a different data directory. The command grb_rs init will help you by copying the default configuration and the data directory into a current directory.

For example, to start two nodes on the same machine with a hostname of myserver:

  1. In a first terminal window, create a new directory node1,

  2. Change your current directory to node1 and run grb_rs init

  3. Start the first node:

    grb_rs --port=61000

  4. In a second terminal window, create a new directory node2,

  5. Change your current directory to node2 and run grb_rs init

  6. Start the second node on a different port and join the first node:

    grb_rs --port=61001 --join=myserver:61000

Checking the status of your cluster#

You can use grbcluster to check the status of the cluster:

> grbcluster nodes
ID       ADDRESS       STATUS TYPE    LICENSE PROCESSING #Q #R JL IDLE %MEM  %CPU
b7d037db server1:61000 ALIVE  COMPUTE VALID   ACCEPTING  0  0  10 <1s  61.42 9.72
eb07fe16 server2:61000 ALIVE  COMPUTE VALID   ACCEPTING  0  0  8  <1s  61.42 8.82

The nodes of the cluster constantly share information about their status. Each node can be in one of the following states:

ALIVE

The node is up and running.

DEGRADED

The node failed to respond to recent communications. The node could return to the ALIVE state if it becomes reachable again. The node will stay in this state until a timeout (controlled by the configuration property DEGRADED_TIMEOUT), at which point it is considered as FAILED

FAILED

The node has been in DEGRADED state for too long, and has been flagged as FAILED. A node will remain in the FAILED state for a short time, and it will eventually be removed from the cluster. If the node comes back online, it will not re-join the cluster automatically.

JOINING

The node is in the process of joining the cluster.

LEAVING

The node left the cluster. It will stay in that state for a short time period before being removed from the cluster.

Compute Servers and Distributed Workers#

A Remote Services cluster is a collection of nodes of two different types:

COMPUTE

A Compute Server node supports the offloading of optimization jobs. Features include load balancing, queueing and concurrent execution of jobs. A Compute Server license is required on the node. A Compute Server node can also act as a Distributed Worker.

WORKER

A Distributed Worker node can be used to execute part of a distributed algorithm. A license is not necessary to run a Distributed Worker, because it is always used in conjunction with a manager (another node or a client program) that requires a license. A Distributed Worker node can only be used by one manager at a time (i.e., the job limit is always set to 1).

By default, grb_rs will try to start a node in Compute Server mode and the node license status will be INVALID if no license is found. In order to start a Distributed Worker, you need to set the WORKER property in the grb_rs.cnf configuration file (or the —-worker command-line flag):

WORKER=true

Once you form your cluster, the node type will be displayed in the TYPE column of the output of grbcluster nodes:

> grbcluster nodes
ID       ADDRESS       STATUS TYPE    LICENSE PROCESSING #Q #R JL IDLE %MEM  %CPU
b7d037db server1:61000 ALIVE  COMPUTE VALID   ACCEPTING  0  0  10 19m  15.30 5.64
735c595f server2:61000 ALIVE  COMPUTE VALID   ACCEPTING  0  0  10 19m  10.45 8.01
eb07fe16 server3:61000 ALIVE  WORKER  VALID   ACCEPTING  0  0  1  <1s  11.44 2.33
4f14a532 server4:61000 ALIVE  WORKER  VALID   ACCEPTING  0  0  1  <1s  12.20 5.60

The node type cannot be changed once grb_rs has started. If you wish to change the node type, you need to stop the node, change the configuration, and restart the node. You may have to update your license as well.

Distributed Optimization#

When using distributed optimization, distributed workers are controlled by a manager. There are two ways to set up the manager:

  • The manager can be a job running on a Compute Server. In this case, a job is submitted to the cluster and executes on one of the COMPUTE nodes as usual. When the job reaches the point where distributed optimization is requested, it will also request some number of workers (see parameters DistributedMIPJobs, ConcurrentJobs, or TuneJobs). The first choice will be WORKER nodes. If not enough are available, it will use COMPUTE nodes. The workload associated with managing the distributed algorithm is quite light, so the initial job will act as both the manager and the first worker.

  • The manager can be the client program itself. The manager does not participate in the distributed optimization. It simply coordinates the efforts of the distributed workers. The manager will request distributed workers (using the WorkerPool parameter), and the cluster will first select the WORKER nodes. If not enough are available, it will use COMPUTE nodes as well.

In both cases, the machine where the manager runs must be licensed to run distributed algorithms (you should see a DISTRIBUTED= line in your license file).

It is typically better to use the Compute Server itself as the distributed manager, rather than the client machine. This is particularly true if the Compute Server and the workers are physically close to each other, but physically distant from the client machine. In a typical environment, the client machine will offload the Gurobi computations onto the Compute Server, and the Compute Server will then act as the manager for the distributed computation.

Grouping#

With the Remote Services grouping feature, you can define a subset of the nodes in your cluster as a group, and then submit jobs specifically to that group. This can be quite useful when some nodes in the cluster are different from others. For example, some nodes may have more memory or faster CPUs. Using this feature, you can force jobs to only run on the appropriate type of machines. If all nodes of the requested group are at capacity, jobs will be queued until a member of that group is available.

In order to define a group, you will need to add the GROUP property to the grb_rs.cnf configuration file and give a name to the group:

GROUP=group1

The groups are static and can only be changed in the node configuration file. If you wish to change the group of a node, you will need to stop the node, edit the configuration file, and restart the node. A node can only be a member of one group.

The grbcluster nodes command displays the assigned group for each node (in the GRP column):

> grbcluster nodes
ID       ADDRESS       STATUS TYPE    GRP     LICENSE PROCESSING #Q #R JL IDLE %MEM  %CPU
b7d037db server1:61000 ALIVE  COMPUTE group1  VALID   ACCEPTING  0  0  10 19m  15.30 5.64
735c595f server2:61000 ALIVE  COMPUTE group1  VALID   ACCEPTING  0  0  10 19m  10.45 8.01
eb07fe16 server3:61000 ALIVE  WORKER  group2  VALID   ACCEPTING  0  0  1  <1s  11.44 2.33
4f14a532 server4:61000 ALIVE  WORKER  group2  VALID   ACCEPTING  0  0  1  <1s  12.20 5.60

You can submit an optimization job to a given group by using the GROUP property of the client license file (see set up a client license). You can also set the CSGROUP parameter in the programming interface.

The value of this parameter can be a single group to target a subset of nodes as explained. It can also be a list of groups, and you can also specify a priority for each group. Here is an example to submit a job to the group1 nodes with priority 10, and to group2 with priority 50.

group1:10,group2:50

Note that if a group is not specified for a submitted job, the job can run on any nodes of any group.

Processing State and Scaling#

Each node of the cluster can be in one of three processing states:

ACCEPTING

The node is accepting new jobs.

DRAINING

The node is not accepting new jobs, but it is still processing existing jobs.

STOPPED

The node is not accepting new jobs and no jobs are running.

A node always starts in the ACCEPTING state. If you need to perform maintenance on a node, or if you want the node to leave the cluster in a clean state for other reasons, the system administrator can issue the node stop command:

> grbcluster node stop --server=server1:61000

If jobs are currently running, the state will change to DRAINING until the jobs finish, at which point it will change to STOPPED. If no jobs are running, the state will change to STOPPED immediately. In the DRAINING or STOPPED states, new jobs are rejected on that node. However, the node is still a member of the cluster and all of the other features, commands, and APIs are still active.

Once a node is in the STOPPED state, you can safely remove it from the cluster (to perform maintenance, shut it down, etc.). To return it to the cluster and resume job processing, run the node start command:

> grbcluster node start --server=server1:61000

The flag —-server is used to target a specific node in the cluster. Adding the —-all flag requests that the command (e.g., start or stop) be applied to all nodes in the cluster.

By using the start and stop with a cluster of Compute Servers, you can effectively scale your cluster up or down, depending on the current cluster workload:

  • You can scale down the cluster by stopping the processing on some nodes.

  • You can scale up the cluster by starting new nodes or resuming processing on some nodes. As soon as a node starts or resumes processing, it will pick up jobs from the current queue or wait for new jobs to be submitted.