Ruby or JRuby? What’s Faster on OpenShift PaaS - Archived

Several people recently asked me whether they should use JRuby or classic MRI Ruby on OpenShift. My general answer is that it depends on your use-case. And then the question comes – is JRuby faster? Well, it basically depends on your use case (again), but to get at least some view, I will run some tests to get the idea about how fast the two implementations are.

Test case

I will run apache benchmark against different web servers with the same application in two shots. First batch has 10000 requests to warm up the virtual machine and after that a shot of 1000 requests to test the speed. Both of the two batches are run with concurrency of 5.

I have not done any performance adjustments. That means that one any of the test can significantly gain from changing some command line switches, let me know and I will run the test again.

Ruby MRI 1.9

OpenShift comes with support for Ruby 1.9. The deployment is done using Phusion Passenger that manages the cluster of processes to provide concurrency for your application. The application is deployed this way for the test.

The Ruby version is

[bench-mjelen.rhcloud.com data]\> ruby -v
ruby 1.9.3p194 (2012-04-20 revision 35410) [x86_64-linux]

The warm up round

[bench-mjelen.rhcloud.com data]\> ab -n 10000 -c 5 http://${OPENSHIFT_INTERNAL_IP}:8080/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
 
Benchmarking 127.9.234.129 (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests
 
 
Server Software:        Apache/2.2.17
Server Hostname:        127.9.234.129
Server Port:            8080
 
Document Path:          /
Document Length:        16 bytes
 
Concurrency Level:      5
Time taken for tests:   84.321 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Total transferred:      3410000 bytes
HTML transferred:       160000 bytes
Requests per second:    118.59 [#/sec] (mean)
Time per request:       42.160 [ms] (mean)
Time per request:       8.432 [ms] (mean, across all concurrent requests)
Transfer rate:          39.49 [Kbytes/sec] received
 
Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    3  13.9      0     195
Processing:     2   38  58.1      7    1134
Waiting:        0   33  54.0      5    1098
Total:          2   41  58.8      9    1164
 
Percentage of the requests served within a certain time (ms)
  50%      9
  66%     70
  75%     82
  80%     87
  90%     95
  95%    101
  98%    193
  99%    214
 100%   1164 (longest request)

and now the benchmark

[bench-mjelen.rhcloud.com data]\> ab -n 1000 -c 5 http://${OPENSHIFT_INTERNAL_IP}:8080/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
 
Benchmarking 127.9.234.129 (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests
 
 
Server Software:        Apache/2.2.17
Server Hostname:        127.9.234.129
Server Port:            8080
 
Document Path:          /
Document Length:        16 bytes
 
Concurrency Level:      5
Time taken for tests:   8.858 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      341000 bytes
HTML transferred:       16000 bytes
Requests per second:    112.89 [#/sec] (mean)
Time per request:       44.290 [ms] (mean)
Time per request:       8.858 [ms] (mean, across all concurrent requests)
Transfer rate:          37.59 [Kbytes/sec] received
 
Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    4  15.3      0     104
Processing:     2   39  50.1     13     699
Waiting:        0   34  48.2      9     698
Total:          2   43  50.7     15     699
 
Percentage of the requests served within a certain time (ms)
  50%     15
  66%     76
  75%     83
  80%     88
  90%     97
  95%    100
  98%    180
  99%    199
 100%    699 (longest request)

JRuby

OpenShift has no native support for JRuby, however to deploy JRuby applications is not difficult, better to say it’s easy, using the DIY cartridge.

The JRuby version used is

[bench-mjelen.rhcloud.com data]\> jruby -v
jruby 1.7.0 (1.9.3p203) 2012-10-22 ff1ebbe on OpenJDK Server VM 1.7.0_09-icedtea-mockbuild_2012_10_15_11_02-b00 [linux-i386]

Mizuno (Jetty)

Mizuno is lightweight web server based on Jetty.

Version used is

mizuno (0.6.4)

The command to start the server

mizuno -p 15632 -o $OPENSHIFT_INTERNAL_IP -E production

The warm up of the JVM

[bench-mjelen.rhcloud.com data]\> ab -n 10000 -c 5 http://127.9.234.129:15632/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
 
Benchmarking 127.9.234.129 (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests
 
 
Server Software:        
Server Hostname:        127.9.234.129
Server Port:            15632
 
Document Path:          /
Document Length:        16 bytes
 
Concurrency Level:      5
Time taken for tests:   155.794 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Total transferred:      1560312 bytes
HTML transferred:       160032 bytes
Requests per second:    64.19 [#/sec] (mean)
Time per request:       77.897 [ms] (mean)
Time per request:       15.579 [ms] (mean, across all concurrent requests)
Transfer rate:          9.78 [Kbytes/sec] received
 
Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    7  22.8      1     301
Processing:     1   70 132.3     22    3100
Waiting:        0   62 128.5     11    3096
Total:          1   77 132.8     79    3101
 
Percentage of the requests served within a certain time (ms)
  50%     79
  66%     93
  75%     97
  80%    100
  90%    184
  95%    208
  98%    391
  99%    497
 100%   3101 (longest request)

and the benchmark

[bench-mjelen.rhcloud.com data]\> ab -n 1000 -c 5 http://127.9.234.129:15632/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
 
Benchmarking 127.9.234.129 (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests
 
 
Server Software:        
Server Hostname:        127.9.234.129
Server Port:            15632
 
Document Path:          /
Document Length:        16 bytes
 
Concurrency Level:      5
Time taken for tests:   10.897 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      156468 bytes
HTML transferred:       16048 bytes
Requests per second:    91.77 [#/sec] (mean)
Time per request:       54.483 [ms] (mean)
Time per request:       10.897 [ms] (mean, across all concurrent requests)
Transfer rate:          14.02 [Kbytes/sec] received
 
Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0   12  30.9      1     209
Processing:     1   42  70.6      7    1187
Waiting:        0   34  66.9      4    1107
Total:          1   54  72.7     14    1189
 
Percentage of the requests served within a certain time (ms)
  50%     14
  66%     88
  75%     94
  80%     96
  90%     99
  95%    183
  98%    206
  99%    299
 100%   1189 (longest request)

Trinidad (Tomcat)

Trinidad is quite similar to Mizuno with the difference it’s not based on Jetty but on Tomcat.

Trinidad’s version

trinidad (1.4.4)
trinidad_jars (1.1.1)

The command to start the server

trinidad --address $OPENSHIFT_INTERNAL_IP -p 15632 -e production

The warm up

ab -n 10000 -c 5 http://127.9.234.129:15632/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
 
Benchmarking 127.9.234.129 (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests
 
 
Server Software:        Apache-Coyote/1.1
Server Hostname:        127.9.234.129
Server Port:            15632
 
Document Path:          /
Document Length:        16 bytes
 
Concurrency Level:      5
Time taken for tests:   217.572 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Total transferred:      2390000 bytes
HTML transferred:       160000 bytes
Requests per second:    45.96 [#/sec] (mean)
Time per request:       108.786 [ms] (mean)
Time per request:       21.757 [ms] (mean, across all concurrent requests)
Transfer rate:          10.73 [Kbytes/sec] received
 
Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0   13  30.3      2     381
Processing:     1   95 2001.0      9  147576
Waiting:        0   83 1999.5      5  147576
Total:          2  108 2000.8     32  147576
 
Percentage of the requests served within a certain time (ms)
  50%     32
  66%     86
  75%     93
  80%     97
  90%    115
  95%    207
  98%    392
  99%    517
 100%  147576 (longest request)

and the benchmark

[bench-mjelen.rhcloud.com repo]\> ab -n 1000 -c 5 http://127.9.234.129:15632/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
 
Benchmarking 127.9.234.129 (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests
 
 
Server Software:        Apache-Coyote/1.1
Server Hostname:        127.9.234.129
Server Port:            15632
 
Document Path:          /
Document Length:        16 bytes
 
Concurrency Level:      5
Time taken for tests:   8.178 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      239000 bytes
HTML transferred:       16000 bytes
Requests per second:    122.28 [#/sec] (mean)
Time per request:       40.891 [ms] (mean)
Time per request:       8.178 [ms] (mean, across all concurrent requests)
Transfer rate:          28.54 [Kbytes/sec] received
 
Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0   14  27.7      3     197
Processing:     1   26  37.8      5     282
Waiting:        0   18  32.5      3     204
Total:          2   40  43.4     11     291
 
Percentage of the requests served within a certain time (ms)
  50%     11
  66%     73
  75%     83
  80%     87
  90%     94
  95%     99
  98%    123
  99%    195
 100%    291 (longest request)

Puma

Puma is relatively new web server based on solid Mongrel. Most of the code has been rewritten, so from Mongrel there is just the awesome HTTP parser.

Puma’s version

puma (1.6.3 java)

The command to start the server

puma -b tcp://${OPENSHIFT_INTERNAL_IP}:15632 -e production > temp

The warm up

[bench-mjelen.rhcloud.com repo]\> ab -n 10000 -c 5 http://127.9.234.129:15632/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
 
Benchmarking 127.9.234.129 (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests
 
 
Server Software:        
Server Hostname:        127.9.234.129
Server Port:            15632
 
Document Path:          /
Document Length:        16 bytes
 
Concurrency Level:      5
Time taken for tests:   188.601 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Total transferred:      1750000 bytes
HTML transferred:       160000 bytes
Requests per second:    53.02 [#/sec] (mean)
Time per request:       94.300 [ms] (mean)
Time per request:       18.860 [ms] (mean, across all concurrent requests)
Transfer rate:          9.06 [Kbytes/sec] received
 
Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    7  25.8      0     489
Processing:     1   87 110.1     89    1697
Waiting:        0   68  93.2     77    1697
Total:          2   94 112.1     92    1698
 
Percentage of the requests served within a certain time (ms)
  50%     92
  66%     98
  75%    102
  80%    107
  90%    197
  95%    292
  98%    400
  99%    510
 100%   1698 (longest request)

and the benchmark

[bench-mjelen.rhcloud.com repo]\> ab -n 1000 -c 5 http://127.9.234.129:15632/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
 
Benchmarking 127.9.234.129 (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests
 
 
Server Software:        
Server Hostname:        127.9.234.129
Server Port:            15632
 
Document Path:          /
Document Length:        16 bytes
 
Concurrency Level:      5
Time taken for tests:   13.347 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      175175 bytes
HTML transferred:       16016 bytes
Requests per second:    74.92 [#/sec] (mean)
Time per request:       66.736 [ms] (mean)
Time per request:       13.347 [ms] (mean, across all concurrent requests)
Transfer rate:          12.82 [Kbytes/sec] received
 
Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    6  21.5      0     193
Processing:     1   61  62.0     73     304
Waiting:        0   50  55.5     14     298
Total:          2   67  63.2     81     305
 
Percentage of the requests served within a certain time (ms)
  50%     81
  66%     94
  75%     97
  80%     99
  90%    115
  95%    195
  98%    218
  99%    293
 100%    305 (longest request)

Thick

Thick is my own prototypish web server for JRuby written using the excellent Netty library.

Thick’s version

thick (0.0.4 java)

The command to start the server

thick -o ${OPENSHIFT_INTERNAL_IP} -p 15632 -E production

The warm up

[bench-mjelen.rhcloud.com ~]\> ab -n 10000 -c 5 http://127.9.234.129:15632/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
 
Benchmarking 127.9.234.129 (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests
 
 
Server Software:        
Server Hostname:        127.9.234.129
Server Port:            15632
 
Document Path:          /
Document Length:        16 bytes
 
Concurrency Level:      5
Time taken for tests:   172.893 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Total transferred:      1560312 bytes
HTML transferred:       160032 bytes
Requests per second:    57.84 [#/sec] (mean)
Time per request:       86.447 [ms] (mean)
Time per request:       17.289 [ms] (mean, across all concurrent requests)
Transfer rate:          8.81 [Kbytes/sec] received
 
Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    9  26.4      0     291
Processing:     1   77 116.2     81    2090
Waiting:        0   69 109.9     16    2087
Total:          1   86 116.6     89    2095
 
Percentage of the requests served within a certain time (ms)
  50%     89
  66%     96
  75%     99
  80%    102
  90%    193
  95%    284
  98%    403
  99%    566
 100%   2095 (longest request)

and the benchmark

[bench-mjelen.rhcloud.com ~]\> ab -n 1000 -c 5 http://127.9.234.129:15632/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
 
Benchmarking 127.9.234.129 (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests
 
 
Server Software:        
Server Hostname:        127.9.234.129
Server Port:            15632
 
Document Path:          /
Document Length:        16 bytes
 
Concurrency Level:      5
Time taken for tests:   9.893 seconds
Complete requests:      1000
Failed requests:        0
Write errors:           0
Total transferred:      156000 bytes
HTML transferred:       16000 bytes
Requests per second:    101.08 [#/sec] (mean)
Time per request:       49.467 [ms] (mean)
Time per request:       9.893 [ms] (mean, across all concurrent requests)
Transfer rate:          15.40 [Kbytes/sec] received
 
Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    8  23.0      1     100
Processing:     1   42  53.8      7     392
Waiting:        0   35  50.8      5     321
Total:          1   49  56.4     11     395
 
Percentage of the requests served within a certain time (ms)
  50%     11
  66%     87
  75%     91
  80%     94
  90%    101
  95%    110
  98%    195
  99%    291
 100%    395 (longest request)

Conclusion

Do these numbers mean anything? I do not think so. A different server can make JRuby faster then MRI or significantly slower. Can JVM option changes help? I guess so. Can the bottleneck then be the application? Definitely!

In conclusion, you can use both of these implementations in for your application and run happily on OpenShift.

My opinion still stands

The choice of technology should not be based on some numbers from artificial benchmarks, but the problem you are trying to solve.

What’s Next?

Categories
Java, OpenShift Online, Ruby
Tags
, ,
Comments are closed.