Forming a Cluster¶
As noted earlier, a cluster consists of a set of one or more nodes, all
running grb_rs
. This section explains how to form a cluster.
Multi-node clusters provide additional capabilities relative to
single-node clusters. For Compute Server, a multi-node cluster will
automatically balance computational load among the various member nodes.
For distributed algorithms, a multi-node cluster enables various
algorithms to distribute work among multiple machines. This section
begins by discussing the different
types of nodes that are needed to support both
Compute Server and distributed algorithms. Next, we will explain the
grouping feature that can be used to create
subsets of nodes to process some jobs. Finally, we will discuss the
dynamic nature of a cluster. The system administrator can ask individual
nodes to start or stop processing jobs,
which makes it possible to smoothly add or remove nodes from a cluster
to simplify maintenance or to scale processing capacity up or down.
Connecting Nodes¶
Every Remote Services cluster starts with a single node. The steps for starting Remote Services on a single node, either as a standard process or as a service, were covered in earlier sections.
Before adding nodes into your cluster, you first need to make sure that
the cluster token (property CLUSTER_TOKEN
in the configuration file)
has the same value in each node and in the Cluster Manager. For better
security, we recommend that you change the predefined value of the token
by generating a new one and pasting the same value into each node
configuration file. You can generate a new token with the following
command:
> grb_rs token
GRBTK-6o4xujs59WJO5508nmaNwc1TtjZJAL1UcwN4vTD4qK4nata8oLr9GnubyXrLTkggc/aw2A==
Adding nodes with a Cluster Manager¶
If you have started a Cluster Manager, you add additional nodes using the exact same command you used to add the first node. You do this by providing the Cluster Manager address. The Cluster Manager acts as a registry of nodes of your cluster, and the nodes will then connect between themselves.
> grb_rs --manager=http://mymanager:61080 --port=61000
The MANAGER
property can also be set through the configuration file:
MANAGER=http://mymanager:61080
PORT=61000
You won’t have the opportunity to provide command-line options when
starting grb_rs
as a service, so your only option in this case is to
provide this information through the configuration file.
If you wish to start multiple grb_rs
processes on the same machine
for testing purposes (this is not recommended for production use), you
will need to make sure each instance of grb_rs
is started on a
different port and using a different data directory. The command
grb_rs init
will help you by copying the default configuration and
the data directory into a current directory.
For example, to start two nodes on the same machine with a hostname of
myserver
:
In a first terminal window, create a new directory
node1
,Change your current directory to
node1
and rungrb_rs init
Start the first node:
grb_rs --manager=http://mymanager:61080 --port=61000
In a second terminal window, create a new directory
node2
,Change your current directory to
node2
and rungrb_rs init
Start the second node on a different port and connect to the Cluster Manager:
grb_rs --manager=http://mymanager:61080 --port=61001
Adding nodes to a Self-Managed Cluster¶
If you have not started a Cluster Manager, nodes must be connected to
each other. Once you’ve started a single-node cluster, you can add nodes
using the —-join
flag to grb_rs
or the JOIN
configuration
property. For example, if you’ve already started a cluster on the
default port of server1
, you would run the following command on the
new node (call it server2
) to create a two-node cluster:
> grb_rs --join=server1
In the log output for server2
, you should see the result of the
handshake between the servers:
info : Node server1, transition from JOINING to ALIVE
Similarly, the log output of server1
will include the line:
info : Node server2, added to the cluster
If you are using a non-default port, you can specify the target node
port as part of the node URL in the —-join
flag. You can specify the
port of the current node using the —-port
flag. You can use
different ports on different machines, but it is a good practice to use
the same one (port 61000 is typically a good choice). The command would
look like this:
> grb_rs --join=server1:61000 --port=61000
The JOIN
property can also be set through the configuration file:
JOIN=server1:61000
PORT=61000
Again, you won’t have the opportunity to provide command-line options
when starting grb_rs
as a service, so your only option in this case
is to provide this information through the configuration file.
Once you’ve created a multi-node cluster, you can add additional nodes
by doing a JOIN
with any member node. Furthermore, the —-join
flag or the JOIN
property can take a comma-separated list of node
names, so a node can still join a cluster even if one of the member
nodes is unavailable. Note that when a list of nodes is specified, the
joining node will try to join with all of the specified nodes at the
same time. Joining is an asynchronous process, so if some target nodes
are not reachable, the joining node will retry before giving up on
joining. If all of the nodes are reachable, they will all join and form
a single cluster.
If you wish to start multiple grb_rs
processes on the same machine
for testing purposes (this is not recommended for production use), you
will need to make sure each instance of grb_rs
is started on a
different port and using a different data directory. The command
grb_rs init
will help you by copying the default configuration and
the data directory into a current directory.
For example, to start two nodes on the same machine with a hostname of
myserver
:
In a first terminal window, create a new directory
node1
,Change your current directory to
node1
and rungrb_rs init
Start the first node:
grb_rs --port=61000
In a second terminal window, create a new directory
node2
,Change your current directory to
node2
and rungrb_rs init
Start the second node on a different port and join the first node:
grb_rs --port=61001 --join=myserver:61000
Checking the status of your cluster¶
You can use grbcluster
to check the status of the cluster:
> grbcluster nodes
ID ADDRESS STATUS TYPE LICENSE PROCESSING #Q #R JL IDLE %MEM %CPU
b7d037db server1:61000 ALIVE COMPUTE VALID ACCEPTING 0 0 10 <1s 61.42 9.72
eb07fe16 server2:61000 ALIVE COMPUTE VALID ACCEPTING 0 0 8 <1s 61.42 8.82
The nodes of the cluster constantly share information about their status. Each node can be in one of the following states:
- ALIVE
The node is up and running.
- DEGRADED
The node failed to respond to recent communications. The node could return to the
ALIVE
state if it becomes reachable again. The node will stay in this state until a timeout (controlled by the configuration propertyDEGRADED_TIMEOUT
), at which point it is considered asFAILED
- FAILED
The node has been in
DEGRADED
state for too long, and has been flagged asFAILED
. A node will remain in theFAILED
state for a short time, and it will eventually be removed from the cluster. If the node comes back online, it will not re-join the cluster automatically.- JOINING
The node is in the process of joining the cluster.
- LEAVING
The node left the cluster. It will stay in that state for a short time period before being removed from the cluster.
Compute Servers and Distributed Workers¶
A Remote Services cluster is a collection of nodes of two different types:
- COMPUTE
A Compute Server node supports the offloading of optimization jobs. Features include load balancing, queueing and concurrent execution of jobs. A Compute Server license is required on the node. A Compute Server node can also act as a Distributed Worker.
- WORKER
A Distributed Worker node can be used to execute part of a distributed algorithm. A license is not necessary to run a Distributed Worker, because it is always used in conjunction with a manager (another node or a client program) that requires a license. A Distributed Worker node can only be used by one manager at a time (i.e., the job limit is always set to 1).
By default, grb_rs
will try to start a node in Compute Server mode
and the node license status will be INVALID
if no license is found.
In order to start a Distributed Worker, you need to set the WORKER
property in the grb_rs.cnf
configuration file (or the —-worker
command-line flag):
WORKER=true
Once you form your cluster, the node type will be displayed in the
TYPE
column of the output of grbcluster nodes
:
> grbcluster nodes
ID ADDRESS STATUS TYPE LICENSE PROCESSING #Q #R JL IDLE %MEM %CPU
b7d037db server1:61000 ALIVE COMPUTE VALID ACCEPTING 0 0 10 19m 15.30 5.64
735c595f server2:61000 ALIVE COMPUTE VALID ACCEPTING 0 0 10 19m 10.45 8.01
eb07fe16 server3:61000 ALIVE WORKER VALID ACCEPTING 0 0 1 <1s 11.44 2.33
4f14a532 server4:61000 ALIVE WORKER VALID ACCEPTING 0 0 1 <1s 12.20 5.60
The node type cannot be changed once grb_rs
has started. If you wish
to change the node type, you need to stop the node, change the
configuration, and restart the node. You may have to update your license
as well.
Distributed Optimization¶
When using distributed optimization, distributed workers are controlled by a manager. There are two ways to set up the manager:
The manager can be a job running on a Compute Server. In this case, a job is submitted to the cluster and executes on one of the
COMPUTE
nodes as usual. When the job reaches the point where distributed optimization is requested, it will also request some number of workers (see parametersDistributedMIPJobs
,ConcurrentJobs
, orTuneJobs
). The first choice will beWORKER
nodes. If not enough are available, it will useCOMPUTE
nodes. The workload associated with managing the distributed algorithm is quite light, so the initial job will act as both the manager and the first worker.The manager can be the client program itself. The manager does not participate in the distributed optimization. It simply coordinates the efforts of the distributed workers. The manager will request distributed workers (using the
WorkerPool
parameter), and the cluster will first select theWORKER
nodes. If not enough are available, it will useCOMPUTE
nodes as well.
In both cases, the machine where the manager runs must be licensed to
run distributed algorithms (you should see a DISTRIBUTED=
line in
your license file).
It is typically better to use the Compute Server itself as the distributed manager, rather than the client machine. This is particularly true if the Compute Server and the workers are physically close to each other, but physically distant from the client machine. In a typical environment, the client machine will offload the Gurobi computations onto the Compute Server, and the Compute Server will then act as the manager for the distributed computation.
Grouping¶
With the Remote Services grouping feature, you can define a subset of the nodes in your cluster as a group, and then submit jobs specifically to that group. This can be quite useful when some nodes in the cluster are different from others. For example, some nodes may have more memory or faster CPUs. Using this feature, you can force jobs to only run on the appropriate type of machines. If all nodes of the requested group are at capacity, jobs will be queued until a member of that group is available.
In order to define a group, you will need to add the GROUP
property
to the grb_rs.cnf
configuration file and give a name to the group:
GROUP=group1
The groups are static and can only be changed in the node configuration file. If you wish to change the group of a node, you will need to stop the node, edit the configuration file, and restart the node. A node can only be a member of one group.
The grbcluster nodes
command displays the assigned group for each
node (in the GRP
column):
> grbcluster nodes
ID ADDRESS STATUS TYPE GRP LICENSE PROCESSING #Q #R JL IDLE %MEM %CPU
b7d037db server1:61000 ALIVE COMPUTE group1 VALID ACCEPTING 0 0 10 19m 15.30 5.64
735c595f server2:61000 ALIVE COMPUTE group1 VALID ACCEPTING 0 0 10 19m 10.45 8.01
eb07fe16 server3:61000 ALIVE WORKER group2 VALID ACCEPTING 0 0 1 <1s 11.44 2.33
4f14a532 server4:61000 ALIVE WORKER group2 VALID ACCEPTING 0 0 1 <1s 12.20 5.60
You can submit an optimization job to a given group by using the
GROUP
property of the client license file (see
set up a client license). You can also set the
CSGROUP
parameter in the programming interface.
The value of this parameter can be a single group to target a subset of nodes as explained. It can also be a list of groups, and you can also specify a priority for each group. Here is an example to submit a job to the group1 nodes with priority 10, and to group2 with priority 50.
group1:10,group2:50
Note that if a group is not specified for a submitted job, the job can run on any nodes of any group.
Processing State and Scaling¶
Each node of the cluster can be in one of three processing states:
- ACCEPTING
The node is accepting new jobs.
- DRAINING
The node is not accepting new jobs, but it is still processing existing jobs.
- STOPPED
The node is not accepting new jobs and no jobs are running.
A node always starts in the ACCEPTING
state. If you need to perform
maintenance on a node, or if you want the node to leave the cluster in a
clean state for other reasons, the system administrator can issue the
node stop
command:
> grbcluster node stop --server=server1:61000
If jobs are currently running, the state will change to DRAINING
until the jobs finish, at which point it will change to STOPPED
. If
no jobs are running, the state will change to STOPPED
immediately.
In the DRAINING
or STOPPED
states, new jobs are rejected on that
node. However, the node is still a member of the cluster and all of the
other features, commands, and APIs are still active.
Once a node is in the STOPPED
state, you can safely remove it from
the cluster (to perform maintenance, shut it down, etc.). To return it
to the cluster and resume job processing, run the node start
command:
> grbcluster node start --server=server1:61000
The flag —-server
is used to target a specific node in the cluster.
Adding the —-all
flag requests that the command (e.g., start
or
stop
) be applied to all nodes in the cluster.
By using the start
and stop
with a cluster of Compute Servers,
you can effectively scale your cluster up or down, depending on the
current cluster workload:
You can scale down the cluster by stopping the processing on some nodes.
You can scale up the cluster by starting new nodes or resuming processing on some nodes. As soon as a node starts or resumes processing, it will pick up jobs from the current queue or wait for new jobs to be submitted.