1. Test environment

1.1 Hardware/OS

2 Linux boxes in a gigabit ethernet LAN, 1 server and 1 test client
Linux Centos 5.2 64bit
Intel(R) Xeon(R) CPU E5410  @ 2.33GHz (L2 cache: 6M), Quad-Core * 2
8G memory
SCSI disk (standalone disk, no other access)

1.2 Software version

nginx, nginx-0.7.63.tar.gz
Erlang, otp_src_R13B02-1.tar.gz
Java, jdk-6u17-linux-x64.bin, mina-2.0.0-RC1.tar.gz, netty-3.2.0.ALPHA1-dist.tar.bz2
Go, hg clone -r release https://go.googlecode.com/hg/ $GOROOT (Nov 12, 2009)

1.3 Source code and configuration

Linux, run sysctl -p

net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 68719476736
kernel.shmall = 4294967296
kernel.panic = 1
net.ipv4.tcp_rmem = 8192	873800	8738000
net.ipv4.tcp_wmem = 4096	655360	6553600
net.ipv4.ip_local_port_range = 1024	65000
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216

# ulimit -n
150000

C: ngnix hello world module, copy the code ngx_http_hello_module.c from http://timyang.net/web/nginx-module/

in nginx.conf, set “worker_processes  1; worker_connections 10240″ for 1 cpu test, set “worker_processes  4; worker_connections 2048″ for multi-core cpu test. Turn off all access or debug log in nginx.conf, as follows

worker_processes  1;
worker_rlimit_nofile 10240;
events {
    worker_connections  10240;
}
http {
    include       mime.types;
    default_type  application/octet-stream;
    sendfile        on;
    keepalive_timeout  0;
    server {
        listen       8080;
        server_name  localhost;
        location / {
            root   html;
            index  index.html index.htm;
        }
          location /hello {
            ngx_hello_module;
            hello 1234;
        }

        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }
    }
}

$ taskset -c 1 ./nginx or $ taskset -c 1-7 ./nginx

Erlang hello world server
The source code is available at yufeng’s blog, see http://blog.yufeng.info/archives/105
Just copy the code after “cat ehttpd.erl”, and compile it.

$ erlc ehttpd.erl
$ taskset -c 1 erl +K true +h 99999 +P 99999 -smp enable +S 2:1 -s ehttpd
$ taskset -c 1-7 erl +K true -s ehttpd
We use taskset to limit erlang vm to use only 1 CPU/core or use all CPU cores. The 2nd line is run in single CPU mode, and the 3rd line is run in multi-core CPU mode.

Java source code, save the 2 class as HttpServer.java and HttpProtocolHandler.java, and do necessary import.

public class HttpServer {
    public static void main(String[] args) throws Exception {
        SocketAcceptor acceptor = new NioSocketAcceptor(4);
        acceptor.setReuseAddress( true );

		int port = 8080;
		String hostname = null;
		if (args.length > 1) {
			hostname = args[0];
			port = Integer.parseInt(args[1]);
		}

        // Bind
        acceptor.setHandler(new HttpProtocolHandler());
        if (hostname != null)
        	acceptor.bind(new InetSocketAddress(hostname, port));
        else
        	acceptor.bind(new InetSocketAddress(port));

        System.out.println("Listening on port " + port);
        Thread.currentThread().join();
    }
}

public class HttpProtocolHandler extends IoHandlerAdapter {
    public void sessionCreated(IoSession session) {
        session.getConfig().setIdleTime(IdleStatus.BOTH_IDLE, 10);
        session.setAttribute(SslFilter.USE_NOTIFICATION);
    }

    public void sessionClosed(IoSession session) throws Exception {}
    public void sessionOpened(IoSession session) throws Exception {}
    public void sessionIdle(IoSession session, IdleStatus status) {}
    public void exceptionCaught(IoSession session, Throwable cause) {
        session.close(true);
    }

    static IoBuffer RESULT = null;
	public static String HTTP_200 = "HTTP/1.1 200 OK\r\nContent-Length: 13\r\n\r\n" +
			"hello world\r\n";
	static {
    	RESULT = IoBuffer.allocate(32).setAutoExpand(true);
    	RESULT.put(HTTP_200.getBytes());
    	RESULT.flip();
    }
    public void messageReceived(IoSession session, Object message)
            throws Exception {
        if (message instanceof IoBuffer) {
        	IoBuffer buf = (IoBuffer) message;
        	int c = buf.get();
        	if (c == 'G' || c == 'g') {
        		session.write(RESULT.duplicate());
        	}
        	session.close(false);
        }
    }
}

Nov 24 update Because the above Mina code doesn’t parse HTTP request and handle the necessary HTTP protocol, replaced with org.jboss.netty.example.http.snoop.HttpServer from Netty example, but removed all the string builder code from HttpRequestHandler.messageReceived() and just return a “hello world” result in HttpRequestHandler.writeResponse(). Please read the source code and the Netty documentation for more information.

$ taskset -c 1-7 \
java -server -Xmx1024m -Xms1024m -XX:+UseConcMarkSweepGC -classpath . test.HttpServer 192.168.10.1 8080

We use taskset to limit java only use cpu1-7, and not use cpu0, because we want cpu0 dedicate for system call(Linux use CPU0 for network interruptions).

Go language, source code

package main
import (
   "http";
    "io";
)
func HelloServer(c *http.Conn, req *http.Request) {
    io.WriteString(c, "hello, world!\n");
}
func main() {
     runtime.GOMAXPROCS(8); // 8 cores
     http.Handle("/", http.HandlerFunc(HelloServer));
     err := http.ListenAndServe(":8080", nil);
    if err != nil {
        panic("ListenAndServe: ", err.String())
    }
}

$ 6g httpd2.go
$ 6l httpd2.6
$ taskset -c 1-7 ./6.out

1.4 Performance test client

ApacheBench client, for 30, 100, 1,000, 5,000 concurrent threads
ab -c 30 -n 1000000 http://192.168.10.1:8080/
ab -c 100 -n 1000000 http://192.168.10.1:8080/
1000 thread, 334 from 3 different machine
ab -c 334 -n 334000 http://192.168.10.1:8080/
5000 thread, 1667 from 3 different machine
ab -c 1667 -n 334000 http://192.168.10.1:8080/

2. Test results

2.1 request per second

2.2 latency, 99% requests within(ms)

3. Notes

* On large concurrent connections, C, Erlang, Java no big difference on their performance, results are very close.
* Java runs better on small connections, but the code in this test doesn’t parse the HTTP request header (the MINA code).
* Although Mr. Yu Feng (the Erlang guru in China) mentioned that Erlang performance better on single CPU(prevent context switch), but the result tells that Erlang has big latency(> 1S) under 1,000 or 5,000 connections.
* Go language is very close to Erlang, but still not good under heavy load (5,000 threads)
After redo 1,000 and 5,000 tests on Nov 18
* Nginx module is the winner on 5,000 concurrent requests.
* Although there is improvement space for Go, Go has the same performance from 30-5,000 threads.
* Erlang process is impressive on large concurrent request, still as good as nginx (5,000 threads).

4. Update Log

Nov 12, change nginx.conf work_connections from 1024 to 10240
Nov 13, add runtime.GOMAXPROCS(8); to go’s code, add sysctl -p env
Nov 18, realized that ApacheBench itself is a bottleneck under 1,000 or 5,000 threads, so use 3 clients from 3 different machines to redo all tests of 1,000 and 5,000 concurrent tests.
Nov 24, use Netty with full HTTP implementation to replace Mina 2 for the Java web server. Still very fast and low latency after added HTTP handle code.

• 如何写nginx module
• MemcacheDB, Tokyo Tyrant, Redis performance test
• Thrift, Protocol Buffers installation and Java code howto
• Thrift and Protocol Buffers performance in Java
• Thrift and Protocol Buffers performance in Java Round 2

1. 动态改变程序运行时参数的能力。如Config.limitValue = 50
2. 动态查看运行时候变量状态的能力，如 print(userMap.size())
3. 执行代码的能力，如 userMap.clear()

对于需求1,非Web的程序没法像PHP/JSP那样直接改就生效，往往改了某个值，即使是一个配置参数都需要服务器重启一下。这对于很多线上服务来说成本太高。

对于上述2的需求，通常是编写庞大的后台管理程序来满足。或者是通过增加日志输出，但问题是某些变量只需要偶尔才看一下，因此很多系统中大部分日志开销基本上是无用的。

对于需求3,只有通过在IDE的debug模式下才能达到。

我觉得所有的后台程序都应该具备这样的能力，为了满足以上需求，于是考虑在所有程序里面嵌入一个小型的解释脚本引擎来实现。就像很多游戏程序嵌入Lua/Python来动态执行代码目的一样。

今天先说下Java中的实现方法，几年前用过BeanShell, 可以在Java VM里面解释执行多种类型脚本语言。但是发现已经多年不更新，原来是Java 6已经内建Scripting的支持，所以这些第三方的工具也都结束了历史使命。

Java 6自带的实际上是 Mozilla Rhino 的Java Script引擎，它使用JavaScript的语法，可以调用任意Java代码。更多介绍见Java Scripting Programmer’s Guide. Rhino除了import写法有点特殊外，基本语法和Java代码基本一致。如：

engine.eval("importClass(java.lang.System)");
engine.eval("println(System.currentTimeMillis())");
engine.eval("println('Source: http://timyang.net/java/java-scriptingjava-scripting/')");

于是做了一个简单的socket服务器，把发过来的文本直接执行，然后再把脚本执行产生的内容返回给发送方。目标达成。共40余行代码(包括注释和花括号)，无需任何第三方library。代码如下：

// create a script engine manager
ScriptEngineManager factory = new ScriptEngineManager();
// create a JavaScript engine
ScriptEngine engine = factory.getEngineByName("JavaScript");

while (true) {
    ServerSocket serverSocket = null;
    try {
        serverSocket = new ServerSocket(4444, 50, InetAddress.getByName("localhost"));
    } catch (IOException e) {
        System.err.println("Could not listen on port: 4444.");
        System.exit(1);
    }

    Socket clientSocket = null;
    try {
        clientSocket = serverSocket.accept();
    } catch (IOException e) {
        System.err.println("Accept failed.");
        System.exit(1);
    }

    PrintWriter out = new PrintWriter(clientSocket.getOutputStream(),
            true);
    BufferedReader in = new BufferedReader(new InputStreamReader(
            clientSocket.getInputStream()));
    String inputLine, outputLine;

    while ((inputLine = in.readLine()) != null) {
        if ("quit".equals(inputLine))
                break;
        try {
            // evaluate JavaScript code from String
            out.println(engine.eval(inputLine));
        } catch (Exception e) {
            out.println("error:" + e.getMessage());
        }
    }
    out.close();
    in.close();
    clientSocket.close();
    serverSocket.close();
}

启动程序之后只要 telnet localhost 4444 就可以执行任意代码并实时查看返回结果。

注意上面为了安全起见，只绑定了localhost地址，防止此功能被外部用户恶意使用。

• C, Erlang, Java and Go Web Server performance test
• Ideas for creating a friendfeed like feed aggregator system
• 利用Gearman来实现远程监控与管理
• Lua coroutine vs Java wait/notify
• 用Python实现CRUD功能REST服务

源代码

Lua code

co = coroutine.create(function(loops)
    for i = 1, loops do
        coroutine.yield()
    end
end)

local x = os.clock()
local loops = 100 * 1000 * 1000
coroutine.resume(co, loops)
for i = 1, loops do
    coroutine.resume(co)
end
print(string.format("elapsed time: %.2f\n", os.clock() - x))

Java code

public class TestWait {
    public static void main(String[] args) {
        WaitClass wc = new WaitClass();
        wc.start();
        int loops = 100 * 1000 * 1000;
        long t1 = System.currentTimeMillis();
        for (int i = 0; i < loops; i++) {
            synchronized (wc) {
                wc.notify();
            }
        }
        long t2 = System.currentTimeMillis();
        System.out.println("elapsed time: " + (t2 - t1) / 1000l);
    }
}

class WaitClass extends Thread {
    public void run() {
        while (true) {
            synchronized (this) {
                try {
                    this.wait();
                } catch (InterruptedException e) {
                    e.printStackTrace();
                }
            }
        }
    }
}

运行结果

Lua elapsed time: 53.36
Java elapsed time: 51

CPU占用

运行环境：4core XEON

Lua 1CPU 100%, 其他CPU0%, total 25% (其中CPU sys 0%)

Java 2个CPU 40%-50%, 其他CPU 0%, total 25% (其中CPU sys 5%-10%)

从结果看，coroutine只利用了一个CPU, 和原理所说完全一致。

Java利用了2个CPU, 各占用了50%的CPU时间运行和50%的时间等待，和设计也一致。另外Java用了5-10%的sys CPU时间用于线程context switch

结论

虽然这两种程序没有直接可比性，但仍然可以看出一些有趣的地方：

• Lua虽然在各种性能评比中performance比Java低一个数量级，但在这个场景中也跑平了Java
• Java为了调用notify/wait, 用了同步锁，因此测试场景对Java不利。

再谈coroutine应用场景

今天又看到qiezi的文章并发编程模型：Thread, Coroutine, Callback … 分析得很深入，对这方面感兴趣的可以进一步去深入了解。

另外qiezi在Coroutine在并发程序中的应用中提到四种场景，可以理解是对我上篇文章对coroutine应用场景的一种答案。

1. 状态机。
2. 异步IO操作：异步IO操作通常是发起一个IO请求，由操作系统完成以后回调指定的方法或者使用其它方式通知。
3. 高并发网络服务器，高并发服务器除了要处理场景一的情况外，可能还要结合场景二，多线程方案有时候完全不能接受，更多的是基于事件、异步IO或者是混合事件和多线程的模型。
4. 客户端并发应用

但是我还是觉得存在疑虑，后面几种我觉得用多线程/线程池模式也可以很好解决。其实select/epoll异步IO方式跟多线程并不矛盾。多线程并不代表每个线程需要recv阻塞在那里。目前网络服务器的多线程通常是指业务逻辑处理部分使用多线程。比如Java中用mina来处理连接(相当于epoll)，mina在收到数据包之后再分发给负责业务逻辑的thread pool处理。如果是CPU密集型任务，简单把线程池的线程数设成CPU数即可达到性能最佳。这时如果把线程数设成1，就很类似coroutine模式了。而Java模式所增加的消耗，主要是new runnable class以及线程池调度的开销。

• Lua coroutine 不一样的多线程编程思路
• Java垃圾回收调优
• Twitter架构图(cache篇)
• Python thread socket server
• 关于阙宏宇lighttpd与mod_cache在twitter上的一些讨论

Thrift Loop    : 10,000,000
Get object     : 14,394msec
Serdes thrift  : 37,671msec
Objs per second: 265,456
Total bytes    : 1,130,000,000

ProtoBuf Loop  : 10,000,000
Get object     : 8,170msec
Serdes protobuf: 33,054msec
Objs per second: 302,535
Total bytes    : 829,997,866

From the result, Protocol Buffers is 1.1 times faster than Thrift!

And from the Google Protocol Buffers group, why the optimize for speed was not turn on by default.

When using C++ or Java protocol buffers, for best performance you need to add a line to your .proto files:

option optimize_for = SPEED;

Otherwise, by default, the compiler optimizes for code size.  Optimizing for code size results in generated code that around a half to a third of the size, but runs an order of magnitude slower…

Here is the original post

• Thrift and Protocol Buffers performance in Java
• Thrift, Protocol Buffers installation and Java code howto
• MemcacheDB, Tokyo Tyrant, Redis performance test
• C, Erlang, Java and Go Web Server performance test
• 利用Gearman来实现远程监控与管理

一，要解决long pause time首先要了解JVM中heap的结构

java gc heap

• Java Heap为什么要分成几个不同的代(generation)? 由于80%-98%的对象的生存周期很短，大部分新对象存放在young generation可以很高效的回收，避免遍历所有对象。
• young与old中内存分配的算法完全不同。young generation中由于存活的很少，要mark, sweep 然后再 compact 剩余的对象比较耗时，干脆把 live object copy 到另外一个空间更高效。old generation完全相反，里面的 live object 变化较少。因此采用 mark-sweep-compact更合适。

二，Java中四种垃圾回收算法

Java中有四种不同的回收算法，对应的启动参数为
–XX:+UseSerialGC
–XX:+UseParallelGC
–XX:+UseParallelOldGC
–XX:+UseConcMarkSweepGC

1. Serial Collector
大部分平台或者强制 java -client 默认会使用这种。
young generation算法 = serial
old generation算法 = serial (mark-sweep-compact)
这种方法的缺点很明显，stop-the-world, 速度慢。服务器应用不推荐使用。

2. Parallel Collector
在linux x64上默认是这种，其他平台要加 java -server 参数才会默认选用这种。
young = parallel，多个thread同时copy
old = mark-sweep-compact = 1
优点：新生代回收更快。因为系统大部分时间做的gc都是新生代的，这样提高了throughput(cpu用于非gc时间)
缺点：当运行在8G/16G server上old generation live object太多时候pause time过长

3. Parallel Compact Collector (ParallelOld)
young = parallel = 2
old = parallel，分成多个独立的单元，如果单元中live object少则回收，多则跳过
优点：old old generation上性能较 parallel 方式有提高
缺点：大部分server系统old generation内存占用会达到60%-80%, 没有那么多理想的单元live object很少方便迅速回收，同时compact方面开销比起parallel并没明显减少。

4. Concurent Mark-Sweep(CMS) Collector
young generation = parallel collector = 2
old = cms
同时不做 compact 操作。
优点：pause time会降低, pause敏感但CPU有空闲的场景需要建议使用策略4.
缺点：cpu占用过多，cpu密集型服务器不适合。另外碎片太多，每个object的存储都要通过链表连续跳n个地方，空间浪费问题也会增大。

几条经验：
1. java -server
2. 设置Xms=Xmx=3/4物理内存
3. 如果是CPU密集型服务器，使用–XX:+UseParallelOldGC, 否则–XX:+UseConcMarkSweepGC
4. 新生代,Parallel/ParallelOld可设大于Xmx1/4，CMS可设小，小于Xmx1/4
5. 优化程序，特别是每个用户的session中的集合类等。我们的一个模块中session中曾经为每个用户使用了一个ConcurrentHashMap, 里面通常只有几条记录，后来改成数组之后，每台机大概节约了1~2G内存。

不过总的说来，Java的GC算法感觉是业界最成熟的，目前很多其他语言或者框架也都支持GC了，但大多数都是只达到Java Serial gc这种层面，甚至分generation都未考虑。JDK7里面针对CMS又进行了一种改进，会采用一种G1(Garbage-First Garbage Collection)的算法。实际上Garbage-First paper(PDF) 2004年已经出来了，相信到JDK7已经可以用于严格生产环境，有时间也会进一步介绍一下G1。
另外在今年的Sun Tech Days上Joey Shen讲的Improving Java Performance(PDF)也是一个很好的Java GC调优的入门教程。

• Thrift and Protocol Buffers performance in Java
• Thrift and Protocol Buffers performance in Java Round 2
• Thrift, Protocol Buffers installation and Java code howto
• Lua coroutine vs Java wait/notify
• 某分布式应用实践一致性哈希的一些问题