• Feeds

  • 利用Gearman来实现远程监控与管理

    Gearman是一个分发任务的程序框架,可以用在各种场合,与Hadoop相比,Gearman更偏向于任务分发功能。它的任务分布非常简单,简单得可以只需要用脚本即可完成。Gearman最初用于LiveJournal的图片resize功能,由于图片resize需要消耗大量计算资源,因此需要调度到后端多台服务器执行,完成任务之后返回前端再呈现到界面。

    gearman

    Gearman分布式任务实现原理上只用到2个字段,function name和data。function name即任务名称,由client传给job server, job server根据function name选择合适的worker节点来执行。data通常为执行任务所需的自定义的内容,比如简单的做法可以把需要执行的脚本当成data即可(当然要注意其中的安全防范)。如果有多个worker可以处理同一个function name, 则job server会自动分配一个。当用于远程监控场景时,我们可以让每个worker注册成不同的业务名称,以达到方便控制每台worker节点的目的。

    下面介绍Gearman实践中可用于多服务器管理的一些小功能,如果你的服务器有多台,重复执行命令过于繁琐,那可以借助下面一些脚本来完成一些有趣的功能。比如要实时看到所有服务器上的 netstat -nat | grep 80, 按下面步骤搭好环境之后,只需要几行脚本代码即可实现。

    1. 安装gearman
    最早的gearman是perl版的,由于国内开发人员对perl熟悉的不多,所以推荐安装C版本
    http://gearman.org/index.php?id=download
    解开后 ./configure; make; make install 即可

    2. 启动 job server
    cd sbin; ./gearmand -d

    3. 到 worker 节点启动 worker

    先按步骤1在worker机上安装 gearman, 然后创建一个执行worker的python脚本。

    #!/usr/bin/env python
    import os
    
    def main():
        cmd = raw_input()
        print os.popen(cmd).read()
    
    if __name__ == "__main__":
        main()

    cd bin; ./gearman -w -h <job_server_ip> -f server1 — ./monitor.py

    4. 通过远程 web 管理
    JSP示例:

    <%@ page language="java" import="java.util.*,java.util.concurrent.*,
    org.gearman.client.*,org.gearman.common.*"%><%!
            private static String host = "192.168.1.1";
            private static int port = 4730;
            public String testGearman(String func, String data) {
                    byte[] input = data.getBytes();
                    String uniqueId = null;
    
                    GearmanJobServerConnection conn =
                        new GearmanNIOJobServerConnection(host, port);
                    GearmanClient client = new GearmanClientImpl();
                    client.addJobServer(conn);
                    GearmanJob job = GearmanJobImpl.createJob(func, input, uniqueId);
                    Future<GearmanJobResult> f = client.submit(job);
                    GearmanJobResult jr = null;
                    String result = null;
                    try {
                            jr = f.get(3, TimeUnit.SECONDS);
                            result = new String(jr.getResults());
                    } catch (Exception e) {
                            result = e.getMessage();
                    }
    
                    client.shutdown();
                    return result;
            }
    %><%
    out.println(testGearman("test", "netstat -nat | grep 80");
    %>

    PHP示例:

    # Create our client object.
    $client= new GearmanClient();
    
    # Add default server (localhost).
    $client->addServer();
    
    echo "Sending job\n";
    
    # Send reverse job
    $result = $client->do("server1", "netstat -nat | grep 80");
    if ($result)
      echo "Success: $result\n";

    如想及时阅读Tim Yang的文章,可通过页面右上方扫码订阅最新更新。

    « | »

    25 Comments  »

    1. 我觉得这种东西适合那种不需要返回结果的计算,如果要等等结果返回的话,通过网络传输启不速度更慢,响应能力反而更低了~

    2. 我觉得返回并不是问题,很多场合的情况下,我们处理任务,肯定放在一台任务服务器上跑的。那肯定会新启一个线程来异步等待。并不会用一个线程在那个地方等着。如果一个线程等着,那我们就不需要任务服务器了。

    3. Tim

      对于gearman应用等待返回肯定是要的,否则还不如用MQ了。
      1. 通常gearman部署是不跨IDC的,排除网络延迟因素,如果有延迟也是worker程序造成。
      2. 如果worker程序存在卡的问题,client可以加超时,如例子所示。
      jr = f.get(3, TimeUnit.SECONDS);在实践中很少出现超时的情况。

    4. xLight

      如果服务器都开着ssh,
      简单写个脚本循环执行就好了啊

    5. Tim

      @xLight ssh通常需要交互输入密码,同时直接用ssh shell交互也容易造成安全隐患。
      使用gearman, worker script可以只接收指定的协议,比如用自定义的A/B/C/D来代替具体的执行命令。
      另外易用性上gearman不管是web client还是worker都比配置ssh更简单。

    6. xLight

      @Tim
      看来我是ssh权限用惯了,呵呵
      没考虑你说的这种应用场景

    7. balancer 不知道好不好用

    8. Robin

      Hi Tim,我正在尝试这种方案,不过发现向Job Server发送的指令只会在某一台Worker上去执行?并不会到所有的Worker上执行?

    9. Robin

      呃。看明白你的意思了,将每个worker注册成不同的名称。

    10. leiyang

      gearman不是用来解决者问题的……

      让这么多不会coding的SA情何以堪……

      SSH-AGENT + SSH-ADD + 有passphrase的private key就简单了

      连perl都不用装

    11. wss8848

      自学得不错,你不会是扫描进来的吧~~?

    12. wss8848

      字写的不错,呵呵,打错了

    13. wellhome

      如果执行的东西超过2分钟。
      web client会一直keep connected知道等到返回结果?

    14. billy

      对于各种任务的返回结果需要自定义,有些较复杂,如数据量在某一时段较大时,另对WORK执行范围也需要配置好,以免发生错误

    15. 晓梦

      这个和func比有什么优势略势呢?

    16. peter

      个人觉得:
      1.返回结果,是一个复杂任务需要利用多台机子的处理能力,或者均衡负载,或者利用并行完成复杂的操作步骤,而这些步骤可以均衡负载处理,map到多个机子并行处理,之后reduce返回结果,比如一个sphinx搜索引擎,一个完整的搜索过程可能对这个引擎来说是几步,得到ids collection,查询多个data source,最后得到结果,这几个过程可以并行处理。
      2不返回结果,是某些任务直接扔给后端处理,并且保证大部分成功,或者失败也能够跟踪处理,比如推送服务或者邮件服务。

    17. 基本找不到生产环境的示例,都说得太简单了。

    18. 如果没有地方的话,我就帮忙想想办法

    19. 感谢您分享这一信息。 我非常喜欢您的博客文章。 您确实与人们分享了一篇内容丰富且有趣的博客文章

    20. Your feedback helps me a lot, A very meaningful event, I hope everything will go well

    21. Great events are being shared the best quality of 2020

    22. Online casinos consume become increasingly popular in Holocene years, and the Combined Realm is no elision. With a ontogeny list of online casinos in the UK, players immediately experience More options than e’er in front.

      Ace of the primal benefits of online casinos is toilet facility. Players tin can delight their darling gambling casino games from the solace of their possess home, without having to jaunt to a strong-arm gambling casino. This makes online casinos an apotheosis option for players World Health Organization survive ALIR away from a gambling casino or WHO don’t hold the clip to bring down unity.

      Another gain of online casinos is the across-the-board sort of games on put up. Online casinos offering everything from classic mesa games similar Quercus marilandica and toothed wheel to the in vogue picture slots and continuous tense jackpots. This way that players keister easy discovery games that wooing their tastes and preferences.

      In addition, many online casinos proffer bonuses and promotions to appeal novel players and suppress existing ones approaching backbone. These hind end include welcome bonuses, discharge spins, and commitment rewards. Players should always learn the terms and conditions of these offers to ready for sure they read the wagering requirements and any early restrictions.

      When choosing an online casino, players should e’er take in for sure that the land site is licenced by the UK Play Direction. This ensures that the cassino is operating de jure and that players are saved. Players should besides feel for casinos that employ reputable software program providers to secure that the games are clean and random.

      Overall, online casinos cause become a democratic pick for players in the UK. With their convenience, wide run of games, and magnetic bonuses, they extend a majuscule alternative to traditional brick-and-trench mortar casinos. As tenacious as players pick out a accredited and reputable casino, they send away bask a secure and pleasurable gambling experience from the ease of their have plate.

    Leave a Comment