• Feeds

  • Posts Tagged ‘English’


    C, Erlang, Java and Go Web Server performance test

    I had tested a hello world web server in C, Erlang, Java and the Go programming language.
    * C, use the well-known high performance web server nginx, with a hello world nginx module
    * Erlang/OTP
    * Java, using the MINA 2.0 framework, now the JBoss Netty framework.
    * Go, http://golang.org/

    1. Test environment

    1.1 Hardware/OS

    2 Linux boxes in a gigabit ethernet LAN, 1 server and 1 test client
    Linux Centos 5.2 64bit
    Intel(R) Xeon(R) CPU E5410  @ 2.33GHz (L2 cache: 6M), Quad-Core * 2
    8G memory
    SCSI disk (standalone disk, no other access)

    1.2 Software version

    nginx, nginx-0.7.63.tar.gz
    Erlang, otp_src_R13B02-1.tar.gz
    Java, jdk-6u17-linux-x64.bin, mina-2.0.0-RC1.tar.gz, netty-3.2.0.ALPHA1-dist.tar.bz2
    Go, hg clone -r release https://go.googlecode.com/hg/ $GOROOT (Nov 12, 2009)

    1.3 Source code and configuration

    Linux, run sysctl -p

    net.ipv4.ip_forward = 0
    net.ipv4.conf.default.rp_filter = 1
    net.ipv4.conf.default.accept_source_route = 0
    kernel.sysrq = 0
    kernel.core_uses_pid = 1
    net.ipv4.tcp_syncookies = 1
    kernel.msgmnb = 65536
    kernel.msgmax = 65536
    kernel.shmmax = 68719476736
    kernel.shmall = 4294967296
    kernel.panic = 1
    net.ipv4.tcp_rmem = 8192	873800	8738000
    net.ipv4.tcp_wmem = 4096	655360	6553600
    net.ipv4.ip_local_port_range = 1024	65000
    net.core.rmem_max = 16777216
    net.core.wmem_max = 16777216

    # ulimit -n
    150000

    C: ngnix hello world module, copy the code ngx_http_hello_module.c from http://timyang.net/web/nginx-module/

    in nginx.conf, set “worker_processes  1; worker_connections 10240″ for 1 cpu test, set “worker_processes  4; worker_connections 2048″ for multi-core cpu test. Turn off all access or debug log in nginx.conf, as follows

    worker_processes  1;
    worker_rlimit_nofile 10240;
    events {
        worker_connections  10240;
    }
    http {
        include       mime.types;
        default_type  application/octet-stream;
        sendfile        on;
        keepalive_timeout  0;
        server {
            listen       8080;
            server_name  localhost;
            location / {
                root   html;
                index  index.html index.htm;
            }
              location /hello {
                ngx_hello_module;
                hello 1234;
            }
    
            error_page   500 502 503 504  /50x.html;
            location = /50x.html {
                root   html;
            }
        }
    }

    $ taskset -c 1 ./nginx or $ taskset -c 1-7 ./nginx

    Erlang hello world server
    The source code is available at yufeng’s blog, see http://blog.yufeng.info/archives/105
    Just copy the code after “cat ehttpd.erl”, and compile it.

    $ erlc ehttpd.erl
    $ taskset -c 1 erl +K true +h 99999 +P 99999 -smp enable +S 2:1 -s ehttpd
    $ taskset -c 1-7 erl +K true -s ehttpd
    We use taskset to limit erlang vm to use only 1 CPU/core or use all CPU cores. The 2nd line is run in single CPU mode, and the 3rd line is run in multi-core CPU mode.

    Java source code, save the 2 class as HttpServer.java and HttpProtocolHandler.java, and do necessary import.

    public class HttpServer {
        public static void main(String[] args) throws Exception {
            SocketAcceptor acceptor = new NioSocketAcceptor(4);
            acceptor.setReuseAddress( true );
    
    		int port = 8080;
    		String hostname = null;
    		if (args.length > 1) {
    			hostname = args[0];
    			port = Integer.parseInt(args[1]);
    		}
    
            // Bind
            acceptor.setHandler(new HttpProtocolHandler());
            if (hostname != null)
            	acceptor.bind(new InetSocketAddress(hostname, port));
            else
            	acceptor.bind(new InetSocketAddress(port));
    
            System.out.println("Listening on port " + port);
            Thread.currentThread().join();
        }
    }
    
    public class HttpProtocolHandler extends IoHandlerAdapter {
        public void sessionCreated(IoSession session) {
            session.getConfig().setIdleTime(IdleStatus.BOTH_IDLE, 10);
            session.setAttribute(SslFilter.USE_NOTIFICATION);
        }
    
        public void sessionClosed(IoSession session) throws Exception {}
        public void sessionOpened(IoSession session) throws Exception {}
        public void sessionIdle(IoSession session, IdleStatus status) {}
        public void exceptionCaught(IoSession session, Throwable cause) {
            session.close(true);
        }
    
        static IoBuffer RESULT = null;
    	public static String HTTP_200 = "HTTP/1.1 200 OK\r\nContent-Length: 13\r\n\r\n" +
    			"hello world\r\n";
    	static {
        	RESULT = IoBuffer.allocate(32).setAutoExpand(true);
        	RESULT.put(HTTP_200.getBytes());
        	RESULT.flip();
        }
        public void messageReceived(IoSession session, Object message)
                throws Exception {
            if (message instanceof IoBuffer) {
            	IoBuffer buf = (IoBuffer) message;
            	int c = buf.get();
            	if (c == 'G' || c == 'g') {
            		session.write(RESULT.duplicate());
            	}
            	session.close(false);
            }
        }
    }

    Nov 24 update Because the above Mina code doesn’t parse HTTP request and handle the necessary HTTP protocol, replaced with org.jboss.netty.example.http.snoop.HttpServer from Netty example, but removed all the string builder code from HttpRequestHandler.messageReceived() and just return a “hello world” result in HttpRequestHandler.writeResponse(). Please read the source code and the Netty documentation for more information.

    $ taskset -c 1-7 \
    java -server -Xmx1024m -Xms1024m -XX:+UseConcMarkSweepGC -classpath . test.HttpServer 192.168.10.1 8080

    We use taskset to limit java only use cpu1-7, and not use cpu0, because we want cpu0 dedicate for system call(Linux use CPU0 for network interruptions).

    Go language, source code

    package main
    import (
       "http";
        "io";
    )
    func HelloServer(c *http.Conn, req *http.Request) {
        io.WriteString(c, "hello, world!\n");
    }
    func main() {
         runtime.GOMAXPROCS(8); // 8 cores
         http.Handle("/", http.HandlerFunc(HelloServer));
         err := http.ListenAndServe(":8080", nil);
        if err != nil {
            panic("ListenAndServe: ", err.String())
        }
    }

    $ 6g httpd2.go
    $ 6l httpd2.6
    $ taskset -c 1-7 ./6.out

    1.4 Performance test client

    ApacheBench client, for 30, 100, 1,000, 5,000 concurrent threads
    ab -c 30 -n 1000000 http://192.168.10.1:8080/
    ab -c 100 -n 1000000 http://192.168.10.1:8080/
    1000 thread, 334 from 3 different machine
    ab -c 334 -n 334000 http://192.168.10.1:8080/
    5000 thread, 1667 from 3 different machine
    ab -c 1667 -n 334000 http://192.168.10.1:8080/

    2. Test results

    2.1 request per second

    30 (threads) 100 1,000 5,000
    Nginx html(1C) 21,301 21,331 23,746 23,502
    Nginx module(1C) 25,809 25,735 30,380 29,667
    Nginx module(Multi-core) 25,057 24,507 31,544 33,274
    Erlang(1C) 11,585 12,367 12,852 12,815
    Erlang(Multi-Core) 15,101 20,255 26,468 25,865
    Java, Mina2(without HTTP parse)
    30,631 26,846 31,911 31,653
    Java, Netty 24,152 24,423 25,487 25,521
    Go 14,080 14,748 15,799 16,110

    c_erlang_java_go
    2.2 latency, 99% requests within(ms)

    30 100 1,000 5,000
    Nginx html(1C) 1 4 42 3,079
    Nginx module(1C) 1 4 32 3,047
    Nginx module(Multi-core) 1 6 205 3,036
    Erlang(1C) 3 8 629 6,337
    Erlang(Multi-Core) 2 7 223 3,084
    Java, Netty 1 3 3 3,084
    Go 26 33 47 9,005

    3. Notes

    * On large concurrent connections, C, Erlang, Java no big difference on their performance, results are very close.
    * Java runs better on small connections, but the code in this test doesn’t parse the HTTP request header (the MINA code).
    * Although Mr. Yu Feng (the Erlang guru in China) mentioned that Erlang performance better on single CPU(prevent context switch), but the result tells that Erlang has big latency(> 1S) under 1,000 or 5,000 connections.
    * Go language is very close to Erlang, but still not good under heavy load (5,000 threads)
    After redo 1,000 and 5,000 tests on Nov 18
    * Nginx module is the winner on 5,000 concurrent requests.
    * Although there is improvement space for Go, Go has the same performance from 30-5,000 threads.
    * Erlang process is impressive on large concurrent request, still as good as nginx (5,000 threads).

    4. Update Log

    Nov 12, change nginx.conf work_connections from 1024 to 10240
    Nov 13, add runtime.GOMAXPROCS(8); to go’s code, add sysctl -p env
    Nov 18, realized that ApacheBench itself is a bottleneck under 1,000 or 5,000 threads, so use 3 clients from 3 different machines to redo all tests of 1,000 and 5,000 concurrent tests.
    Nov 24, use Netty with full HTTP implementation to replace Mina 2 for the Java web server. Still very fast and low latency after added HTTP handle code.

    MemcacheDB, Tokyo Tyrant, Redis performance test

    I had tested the following key-value store for set() and get()

    1. Test environment

    1.1 Hardware/OS

    2 Linux boxes in a LAN, 1 server and 1 test client
    Linux Centos 5.2 64bit
    Intel(R) Xeon(R) CPU E5410  @ 2.33GHz (L2 cache: 6M), Quad-Core * 2
    8G memory
    SCSI disk (standalone disk, no other access)

    1.2 Software version

    db-4.7.25.tar.gz
    libevent-1.4.11-stable.tar.gz
    memcached-1.2.8.tar.gz
    memcachedb-1.2.1-beta.tar.gz
    redis-0.900_2.tar.gz
    tokyocabinet-1.4.9.tar.gz
    tokyotyrant-1.1.9.tar.gz

    1.3 Configuration

    Memcachedb startup parameter
    Test 100 bytes
    ./memcachedb -H /data5/kvtest/bdb/data -d -p 11212 -m 2048 -N -L 8192
    (Update: As mentioned by Steve, the 100-byte-test missed the -N paramter, so I added it and updated the data)
    Test 20k bytes
    ./memcachedb -H /data5/kvtest/mcdb/data -d -p 11212 -b 21000 -N -m 2048

    Tokyo Tyrant (Tokyo Cabinet) configuration
    Use default Tokyo Tyrant sbin/ttservctl
    use .tch database, hashtable database

    ulimsiz=”256m”
    sid=1
    dbname=”$basedir/casket.tch#bnum=50000000″ # default 1M is not enough!
    maxcon=”65536″
    retval=0

    Redis configuration
    timeout 300
    save 900 1
    save 300 10
    save 60 10000
    # no maxmemory settings

    1.4 Test client

    Client in Java, JDK1.6.0, 16 threads
    Use Memcached client java_memcached-release_2.0.1.jar
    JRedis client for Redis test, another JDBC-Redis has poor performance.

    2. Small data size test result

    Test 1, 1-5,000,000 as key, 100 bytes string value, do set, then get test, all get test has result.
    Request per second(mean)key-value-performance-1(Update)

    Store Write Read
    Memcached 55,989 50,974
    Memcachedb 25,583 35,260
    Tokyo Tyrant 42,988 46,238
    Redis 85,765 71,708

    Server Load Average

    Store Write Read
    Memcached 1.80, 1.53, 0.87 1.17, 1.16, 0.83
    MemcacheDB 1.44, 0.93, 0.64 4.35, 1.94, 1.05
    Tokyo Tyrant 3.70, 1.71, 1.14 2.98, 1.81, 1.26
    Redis 1.06, 0.32, 0.18 1.56, 1.00, 0.54

    3. Larger data size test result

    Test 2, 1-500,000 as key, 20k bytes string value, do set, then get test, all get test has result.
    Request per second(mean)
    (Aug 13 Update: fixed a bug on get() that read non-exist key)
    key-value-performance-2(update)

    Store Write Read
    Memcachedb 357 327
    Tokyo Tyrant 3,501 257
    Redis 1,542 957

    4. Some notes about the test

    When test Redis server, the memory goes up steadily, consumed all 8G and then use swap(and write speed slow down), after all memory and swap space is used, the client will get exceptions. So use Redis in a productive environment should limit to a small data size. It is another cache solution rather than a persistent storage. So compare Redis together with MemcacheDB/TC may not fair because Redis actually does not save data to disk during the test.

    Tokyo cabinet and memcachedb are very stable during heavy load, use very little memory in set test and less than physical memory in get test.

    MemcacheDB peformance is poor for write large data size(20k).

    The call response time was not monitored in this test.

    Thrift and Protocol Buffers performance in Java Round 2

    In my last test Thrift and Protocol Buffers performance in Java, Some comments told me that there are some tuning parameter for Protocol Buffer which can improve performance magically. The parameter was not turn on by default. I added
    option optimize_for = SPEED
    to the proto file, and re-generated the Java class, and the result:

    Thrift Loop    : 10,000,000
    Get object     : 14,394msec
    Serdes thrift  : 37,671msec
    Objs per second: 265,456
    Total bytes    : 1,130,000,000
    
    ProtoBuf Loop  : 10,000,000
    Get object     : 8,170msec
    Serdes protobuf: 33,054msec
    Objs per second: 302,535
    Total bytes    : 829,997,866
    

    From the result, Protocol Buffers is 1.1 times faster than Thrift!

    And from the Google Protocol Buffers group, why the optimize for speed was not turn on by default.

    When using C++ or Java protocol buffers, for best performance you need to add a line to your .proto files:

    option optimize_for = SPEED;

    Otherwise, by default, the compiler optimizes for code size.  Optimizing for code size results in generated code that around a half to a third of the size, but runs an order of magnitude slower…

    Here is the original post

    12