Friday, 20 June 2014

Submitting a topology to Remote Storm Cluster

It is very easy to write a topology and submit to the same Storm Cluster.
But problem arises when we need to submit a topology remotely to remote Storm cluster from a local machine.
What should we do in that case?

Here is the approach to submit a topology to remote cluster.

I have a local windows machine and one Storm Cluster(1 nimbus Linux machine and 2 supervisor Linux machine)
Let's say following are the machines in cluster :

Nimbus Machine : 192.168.1.5
Supervisor Machine 1: 192.168.1.6
Supervisor Machine 2: 192.168.1.7

Storm cluster should be up and running on above machine.

Now from local machine, use NimbusClient to submit Jar to cluster.

NimbusClient nimbus = new NimbusClient(storm_conf,"<nimbus machine ip>",<nimbus port>);
nimbus.getClient().submitTopology(topologyName,uploadedJarLocation,jsonConf, builder.createTopology());

Here is a running example:

import java.util.Map;
import org.json.simple.JSONValue;
import backtype.storm.Config;
import backtype.storm.StormSubmitter;
import backtype.storm.generated.AlreadyAliveException;
import backtype.storm.generated.Nimbus.Client;
import backtype.storm.topology.TopologyBuilder;
import backtype.storm.utils.NimbusClient;
import backtype.storm.utils.Utils;

public class RunningClusterTopology {
    public static void main(String[] args) throws Exception {
  TopologyBuilder builder = new TopologyBuilder();
                Config conf = new Config();
                conf.put(Config.NIMBUS_HOST, "192.168.1.5");
                conf.setDebug(true);
                Map storm_conf = Utils.readStormConfig();
                storm_conf.put("nimbus.host", "192.168.1.5");
                Client client = NimbusClient.getConfiguredClient(storm_conf)
                                .getClient();
                String inputJar = "C:\\workspace\\TestStormRunner\\target\\TestStormRunner-0.0.1-SNAPSHOT-jar-with-dependencies.jar";
                NimbusClient nimbus = new NimbusClient(storm_conf, "192.168.1.5",
                                6627);
      // upload topology jar to Cluster using StormSubmitter
               String uploadedJarLocation = StormSubmitter.submitJar(storm_conf,
                                inputJar);
                try {
                        String jsonConf = JSONValue.toJSONString(storm_conf);
                        nimbus.getClient().submitTopology("testtopology",
                                        uploadedJarLocation, jsonConf, builder.createTopology());
                } catch (AlreadyAliveException ae) {
                        ae.printStackTrace();
                }
                Thread.sleep(60000);
        }
}

It will submit a topology on Nimbus Machine where it’ll run on 2 supervisor machines(192.168.1.6 and 192.168.1.7)
To test it, open storm UI in browser : http://<nimbus-ip>:<port>

6 comments:

  1. What if we need multiple parameters while submitting the topology ?

    ReplyDelete
    Replies
    1. I mean pass multiple args along with topology name.

      Delete
  2. Hi Ritesh,
    Did you figured out how to submit topology with multiple arguments ??

    ReplyDelete
  3. It works, in storm UI we can see the topology deployed also. Three is one problem whic i faced, bolts and spout details were not visible. Any possible solutions??

    ReplyDelete
  4. Thanks for sharing this informative information..
    Installing a Storm Cluster Following are the prerequisites for setting the cluster http://www.s4techno.com/blog/2016/08/13/installing-a-storm-cluster/

    ReplyDelete
  5. Hi All,

    I successfully running storm cluster with two supervisors. In storm ui also getting two supervisors id's.

    My question is how to know our code executing in cluster mode as we can find in local mode in eclipse console.
    In local mode cluster, we are able to execute entire code and we can see code execution in debugger mode.

    Coming to multi cluster mode after storm.jar submitted to nimbus node we are not able to find code execution.

    In my case am inserting table rows from one db to another db.It working fine for local mode. where as in multi node cluster after submitting nothing is happening and can not find table rows in new db while cluster is running.

    Please help me in this if anyone have solution.

    ReplyDelete