1. Test environment

1.1 Hardware/OS

2 Linux boxes in a gigabit ethernet LAN, 1 server and 1 test client
Linux Centos 5.2 64bit
Intel(R) Xeon(R) CPU E5410  @ 2.33GHz (L2 cache: 6M), Quad-Core * 2
8G memory
SCSI disk (standalone disk, no other access)

1.2 Software version

nginx, nginx-0.7.63.tar.gz
Erlang, otp_src_R13B02-1.tar.gz
Java, jdk-6u17-linux-x64.bin, mina-2.0.0-RC1.tar.gz, netty-3.2.0.ALPHA1-dist.tar.bz2
Go, hg clone -r release https://go.googlecode.com/hg/ $GOROOT (Nov 12, 2009)

1.3 Source code and configuration

Linux, run sysctl -p

net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 68719476736
kernel.shmall = 4294967296
kernel.panic = 1
net.ipv4.tcp_rmem = 8192	873800	8738000
net.ipv4.tcp_wmem = 4096	655360	6553600
net.ipv4.ip_local_port_range = 1024	65000
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216

# ulimit -n
150000

C: ngnix hello world module, copy the code ngx_http_hello_module.c from http://timyang.net/web/nginx-module/

in nginx.conf, set “worker_processes  1; worker_connections 10240″ for 1 cpu test, set “worker_processes  4; worker_connections 2048″ for multi-core cpu test. Turn off all access or debug log in nginx.conf, as follows

worker_processes  1;
worker_rlimit_nofile 10240;
events {
    worker_connections  10240;
}
http {
    include       mime.types;
    default_type  application/octet-stream;
    sendfile        on;
    keepalive_timeout  0;
    server {
        listen       8080;
        server_name  localhost;
        location / {
            root   html;
            index  index.html index.htm;
        }
          location /hello {
            ngx_hello_module;
            hello 1234;
        }

        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }
    }
}

$ taskset -c 1 ./nginx or $ taskset -c 1-7 ./nginx

Erlang hello world server
The source code is available at yufeng’s blog, see http://blog.yufeng.info/archives/105
Just copy the code after “cat ehttpd.erl”, and compile it.

$ erlc ehttpd.erl
$ taskset -c 1 erl +K true +h 99999 +P 99999 -smp enable +S 2:1 -s ehttpd
$ taskset -c 1-7 erl +K true -s ehttpd
We use taskset to limit erlang vm to use only 1 CPU/core or use all CPU cores. The 2nd line is run in single CPU mode, and the 3rd line is run in multi-core CPU mode.

Java source code, save the 2 class as HttpServer.java and HttpProtocolHandler.java, and do necessary import.

public class HttpServer {
    public static void main(String[] args) throws Exception {
        SocketAcceptor acceptor = new NioSocketAcceptor(4);
        acceptor.setReuseAddress( true );

		int port = 8080;
		String hostname = null;
		if (args.length > 1) {
			hostname = args[0];
			port = Integer.parseInt(args[1]);
		}

        // Bind
        acceptor.setHandler(new HttpProtocolHandler());
        if (hostname != null)
        	acceptor.bind(new InetSocketAddress(hostname, port));
        else
        	acceptor.bind(new InetSocketAddress(port));

        System.out.println("Listening on port " + port);
        Thread.currentThread().join();
    }
}

public class HttpProtocolHandler extends IoHandlerAdapter {
    public void sessionCreated(IoSession session) {
        session.getConfig().setIdleTime(IdleStatus.BOTH_IDLE, 10);
        session.setAttribute(SslFilter.USE_NOTIFICATION);
    }

    public void sessionClosed(IoSession session) throws Exception {}
    public void sessionOpened(IoSession session) throws Exception {}
    public void sessionIdle(IoSession session, IdleStatus status) {}
    public void exceptionCaught(IoSession session, Throwable cause) {
        session.close(true);
    }

    static IoBuffer RESULT = null;
	public static String HTTP_200 = "HTTP/1.1 200 OK\r\nContent-Length: 13\r\n\r\n" +
			"hello world\r\n";
	static {
    	RESULT = IoBuffer.allocate(32).setAutoExpand(true);
    	RESULT.put(HTTP_200.getBytes());
    	RESULT.flip();
    }
    public void messageReceived(IoSession session, Object message)
            throws Exception {
        if (message instanceof IoBuffer) {
        	IoBuffer buf = (IoBuffer) message;
        	int c = buf.get();
        	if (c == 'G' || c == 'g') {
        		session.write(RESULT.duplicate());
        	}
        	session.close(false);
        }
    }
}

Nov 24 update Because the above Mina code doesn’t parse HTTP request and handle the necessary HTTP protocol, replaced with org.jboss.netty.example.http.snoop.HttpServer from Netty example, but removed all the string builder code from HttpRequestHandler.messageReceived() and just return a “hello world” result in HttpRequestHandler.writeResponse(). Please read the source code and the Netty documentation for more information.

$ taskset -c 1-7 \
java -server -Xmx1024m -Xms1024m -XX:+UseConcMarkSweepGC -classpath . test.HttpServer 192.168.10.1 8080

We use taskset to limit java only use cpu1-7, and not use cpu0, because we want cpu0 dedicate for system call(Linux use CPU0 for network interruptions).

Go language, source code

package main
import (
   "http";
    "io";
)
func HelloServer(c *http.Conn, req *http.Request) {
    io.WriteString(c, "hello, world!\n");
}
func main() {
     runtime.GOMAXPROCS(8); // 8 cores
     http.Handle("/", http.HandlerFunc(HelloServer));
     err := http.ListenAndServe(":8080", nil);
    if err != nil {
        panic("ListenAndServe: ", err.String())
    }
}

$ 6g httpd2.go
$ 6l httpd2.6
$ taskset -c 1-7 ./6.out

1.4 Performance test client

ApacheBench client, for 30, 100, 1,000, 5,000 concurrent threads
ab -c 30 -n 1000000 http://192.168.10.1:8080/
ab -c 100 -n 1000000 http://192.168.10.1:8080/
1000 thread, 334 from 3 different machine
ab -c 334 -n 334000 http://192.168.10.1:8080/
5000 thread, 1667 from 3 different machine
ab -c 1667 -n 334000 http://192.168.10.1:8080/

2. Test results

2.1 request per second

2.2 latency, 99% requests within(ms)

3. Notes

* On large concurrent connections, C, Erlang, Java no big difference on their performance, results are very close.
* Java runs better on small connections, but the code in this test doesn’t parse the HTTP request header (the MINA code).
* Although Mr. Yu Feng (the Erlang guru in China) mentioned that Erlang performance better on single CPU(prevent context switch), but the result tells that Erlang has big latency(> 1S) under 1,000 or 5,000 connections.
* Go language is very close to Erlang, but still not good under heavy load (5,000 threads)
After redo 1,000 and 5,000 tests on Nov 18
* Nginx module is the winner on 5,000 concurrent requests.
* Although there is improvement space for Go, Go has the same performance from 30-5,000 threads.
* Erlang process is impressive on large concurrent request, still as good as nginx (5,000 threads).

4. Update Log

Nov 12, change nginx.conf work_connections from 1024 to 10240
Nov 13, add runtime.GOMAXPROCS(8); to go’s code, add sysctl -p env
Nov 18, realized that ApacheBench itself is a bottleneck under 1,000 or 5,000 threads, so use 3 clients from 3 different machines to redo all tests of 1,000 and 5,000 concurrent tests.
Nov 24, use Netty with full HTTP implementation to replace Mina 2 for the Java web server. Still very fast and low latency after added HTTP handle code.

• 如何写nginx module
• MemcacheDB, Tokyo Tyrant, Redis performance test
• Thrift, Protocol Buffers installation and Java code howto
• Thrift and Protocol Buffers performance in Java
• Thrift and Protocol Buffers performance in Java Round 2

• MemcacheDB, use memcached client protocol.
• Tokyo Tyrant (Tokyo Cabinet), use memcached client protocol
• Redis, use JRedis Java client

1. Test environment

1.1 Hardware/OS

2 Linux boxes in a LAN, 1 server and 1 test client
Linux Centos 5.2 64bit
Intel(R) Xeon(R) CPU E5410  @ 2.33GHz (L2 cache: 6M), Quad-Core * 2
8G memory
SCSI disk (standalone disk, no other access)

1.2 Software version

db-4.7.25.tar.gz
libevent-1.4.11-stable.tar.gz
memcached-1.2.8.tar.gz
memcachedb-1.2.1-beta.tar.gz
redis-0.900_2.tar.gz
tokyocabinet-1.4.9.tar.gz
tokyotyrant-1.1.9.tar.gz

1.3 Configuration

Memcachedb startup parameter
Test 100 bytes
./memcachedb -H /data5/kvtest/bdb/data -d -p 11212 -m 2048 -N -L 8192
(Update: As mentioned by Steve, the 100-byte-test missed the -N paramter, so I added it and updated the data)
Test 20k bytes
./memcachedb -H /data5/kvtest/mcdb/data -d -p 11212 -b 21000 -N -m 2048

Tokyo Tyrant (Tokyo Cabinet) configuration
Use default Tokyo Tyrant sbin/ttservctl
use .tch database, hashtable database

ulimsiz=”256m”
sid=1
dbname=”$basedir/casket.tch#bnum=50000000″ # default 1M is not enough!
maxcon=”65536″
retval=0

Redis configuration
timeout 300
save 900 1
save 300 10
save 60 10000
# no maxmemory settings

1.4 Test client

Client in Java, JDK1.6.0, 16 threads
Use Memcached client java_memcached-release_2.0.1.jar
JRedis client for Redis test, another JDBC-Redis has poor performance.

2. Small data size test result

Test 1, 1-5,000,000 as key, 100 bytes string value, do set, then get test, all get test has result.
Request per second(mean)

Server Load Average

3. Larger data size test result

Test 2, 1-500,000 as key, 20k bytes string value, do set, then get test, all get test has result.
Request per second(mean)
(Aug 13 Update: fixed a bug on get() that read non-exist key)

4. Some notes about the test

When test Redis server, the memory goes up steadily, consumed all 8G and then use swap(and write speed slow down), after all memory and swap space is used, the client will get exceptions. So use Redis in a productive environment should limit to a small data size. It is another cache solution rather than a persistent storage. So compare Redis together with MemcacheDB/TC may not fair because Redis actually does not save data to disk during the test.

Tokyo cabinet and memcachedb are very stable during heavy load, use very little memory in set test and less than physical memory in get test.

MemcacheDB peformance is poor for write large data size(20k).

The call response time was not monitored in this test.

• Thrift and Protocol Buffers performance in Java Round 2
• Thrift and Protocol Buffers performance in Java
• C, Erlang, Java and Go Web Server performance test
• Ideas for creating a friendfeed like feed aggregator system
• Python thread socket server

Thrift Loop    : 10,000,000
Get object     : 14,394msec
Serdes thrift  : 37,671msec
Objs per second: 265,456
Total bytes    : 1,130,000,000

ProtoBuf Loop  : 10,000,000
Get object     : 8,170msec
Serdes protobuf: 33,054msec
Objs per second: 302,535
Total bytes    : 829,997,866

From the result, Protocol Buffers is 1.1 times faster than Thrift!

And from the Google Protocol Buffers group, why the optimize for speed was not turn on by default.

When using C++ or Java protocol buffers, for best performance you need to add a line to your .proto files:

option optimize_for = SPEED;

Otherwise, by default, the compiler optimizes for code size.  Optimizing for code size results in generated code that around a half to a third of the size, but runs an order of magnitude slower…

Here is the original post

• Thrift and Protocol Buffers performance in Java
• Thrift, Protocol Buffers installation and Java code howto
• MemcacheDB, Tokyo Tyrant, Redis performance test
• C, Erlang, Java and Go Web Server performance test
• 利用Gearman来实现远程监控与管理

1. Like the friendfeed, The product can import external RSS, so we separate the system into two parts. The rss crawl system and the feed pubsub system. The pubsub system has no responsbility to grab the feed from source. The feed itself only save feed summary and url.
2. We decided to use the INBOX approach, which will push the published feed to be saved in all subscriber’s data table. More information of how this work can refer to Scaling a Microblogging Service – Part I.
3. User’s homepage is an aggregation result. We choose to return a limited recent real-time date, no infinity pagination. But the user’s own feed(user’s profile page) may have a bigger date range.
4. The unsubscibe logic have two choice, delete or keep the history data from one’s inbox. We decide to keep them.
5. If the feed source had been deleted, do we need to delete all references in all subscriber’s inbox?  If need delete, each feed push to the pubsub system need to have a unique resource id. Another problem is after the source updated whether to publish a new feed or update the current feed?
6. How to manage the group(QUN in Chinese) feed, deliver to all member’s inbox? Or share a group inbox?
7. How to impl the feed comment logic, publish the comment to feed system or design a standalone comment system. We prefer to use a standalone comment system which doesn’t publish the comment back to the feed system.
8. Every feed has a media type, such as text, video, image so the subscribe API can only retrieve a limited media type (text for mobile device). And a feed may have tags.
9. The read/unread count is easy to implement. But the load is heavy. (QQ / QQzone may has such logic.)
10. Need open API for 3rd party client(like twitter client), and RSS feed, may have OAuth integration.
11. The storage may like friendfeed’s mysql schema (see How FriendFeed uses MySQL to store schema-less data) or use Amazon simpledb.
12. May add support for XMPP Publish-Subscribe, or PEP(Personal Eventing Protocol) for pushing realtime time to users in the future.

• 用Python实现CRUD功能REST服务
• Friendfeed的MySQL key/value存储
• Python thread socket server
• Facebook的feed格式设计
• 如何写nginx module

The test is very similiar as the Python test. the .proto and .thrift file are copied from the above python test.

The .thrift content:

struct dns_record {
1: string key,
2: string value,
3: string type = 'A',
4: i32 ttl = 86400,
5: string first,
6: string last
}

typedef list<dns_record> biglist

struct dns_response {
1: biglist records
}

service PassiveDns {
biglist search_question(1:string q);
biglist search_answer(1:string q);
}

The .proto content

package passive_dns;

message DnsRecord {
required string key = 1;
required string value = 2;
required string first = 3;
required string last = 4;
optional string type = 5 [default = "A"];
optional int32  ttl = 6 [default = 86400];
}

message DnsResponse {
repeated DnsRecord records = 1;
}

From the document, I learn that the optional and default values are one of the benefits of both serialization libraries. A record that matches the default value does not need to be included in the serialized output.

I wrote up a simple test program to compare thrift, protocol buffers. I tested the serialize and deserialize together, because this is the most called part in most scenarioes.

Test 1: 10,000,000 times

ProtoBuf Loop  : 10,000,000
Get object     : 15,130msec
Serdes protobuf: 68,600msec
Objs per second: 145,772
Total bytes    : 829,996,683

Thrift Loop    : 10,000,000
Get object     : 12,651msec
Serdes thrift  : 36,904msec
Objs per second: 270,973
Total bytes    : 1,130,000,000

Test 2: 1,000,000 times

ProtoBuf Loop  : 1,000,000
Get object     : 1,094msec
Serdes protobuf: 7,467msec
Objs per second: 133,922
Total bytes    : 83,000,419

Thrift Loop    : 1,000,000
Get object     : 524msec
Serdes thrift  : 5,969msec
Objs per second: 167,532
Total bytes    : 113,000,000

The serde_* functions are the times needed to serialize, and de-serialize the java object to and from a byte[].

The result in Java was that Protocol Buffers 1.2-2 times slower than Thrift. (in the python test was 4~10 times). And PB binary size is smaller than Thrift. I think this is acceptable, and Google may improve the Protocol Buffers performance in the future version.

Download my test code in Java: thrift-protocol-buffers-java.tgz,

More information about thrift and protocol buffers: Thrift, Protocol Buffers installation and Java code howto

Update: There is another Thrift vs. Protocol Buffers compare non-performance factors.

UPDATE 2 (Apr 17): There is a performance tuning parameter optimize_for = SPEED (thanks Steve Chu) for Protocol Buffers, please see my next performance tests Thrift and Protocol Buffers performance in Java Round 2

• Thrift and Protocol Buffers performance in Java Round 2
• Thrift, Protocol Buffers installation and Java code howto
• MemcacheDB, Tokyo Tyrant, Redis performance test
• C, Erlang, Java and Go Web Server performance test
• 利用Gearman来实现远程监控与管理

1. build and install thrift

install boost
cd <boost_root>/tools/jam
./build_dist.sh
# linux* will depends the platform
cp stage/bin.linux*/bjam <boost_root>
# build boost, use bjam will faster
cd <boost_root>
./configure –without-icu –prefix=/usr/local/boost
./bjam -toolset=gcc –build-type=release install –prefix=/usr/local/boost

# build thrift
./bootstrap.sh
./configure –with-boost=/usr/local
make
make install

2. Build Thrift java library

install apache ant if necessary

cd lib/java/
ant

get libthrift.jar

3. Create .thrift file and gen Java code

(See http://wiki.apache.org/thrift/Tutorial for more .thrift tutorial info)
tim.thrift

struct dns_record {
1: string key,
2: string value,
3: string type = 'A',
4: i32 ttl = 86400,
5: string first,
6: string last
}

service TestDns {
dns_record test(1:string q);
}

<thrift_root>/bin/thrift –gen java tim.thrift
code will be generated in gen-java/*.java

4. Write java code

// new object
dns_record dr = new dns_record(key, value, type, ttl, first, last)
// serialize
TSerializer serializer = new TSerializer(new TBinaryProtocol.Factory());
TDeserializer deserializer = new TDeserializer(new TBinaryProtocol.Factory());
byte[] bytes = serializer.serialize(dr);

see also: http://wiki.apache.org/thrift/ThriftUsageJava

II. Protocol Buffers install and Java code

1. Build and install Protocol buffers

./configure
make
make install

2. Build protobuf Java library

install maven if not necessary

cd protobuf/java
mvn test
mvn package

get jar from target/protobuf-java-x.x.x.jar

3. Create .proto file and gen Java code

tim.proto

package dns;

message DnsRecord {
required string key = 1;
required string value = 2;
required string first = 3;
required string last = 4;
optional string type = 5 [default = "A"];
optional int32  ttl = 6 [default = 86400];
}

message DnsResponse {
repeated DnsRecord records = 1;
}

bin/protoc –java_out . tim.proto

4. Write Java code

// protocol buffer need a builder to create object
Dns.DnsRecord.Builder b = Dns.DnsRecord.newBuilder();
b.setKey("key");
b.setValue("value...");
...
b.builder();

byte[] bytes = dr.toByteArray();
Dns.DnsRecord dr2 = Dns.DnsRecord.parseFrom(bytes);

III. Resources

Thrift: http://incubator.apache.org/thrift/

Protocol Buffers: http://code.google.com/apis/protocolbuffers/

Tim’s Blog: http://timyang.net/

• Thrift and Protocol Buffers performance in Java
• Thrift and Protocol Buffers performance in Java Round 2
• MemcacheDB, Tokyo Tyrant, Redis performance test
• C, Erlang, Java and Go Web Server performance test
• 利用Gearman来实现远程监控与管理