benchmark – Shining Ray

不使用lock：

Concurrency Level:      100
Time taken for tests:   191.103 seconds
Complete requests:      50000
Failed requests:        0
Write errors:           0
Total transferred:      40517615 bytes
HTML transferred:       23050000 bytes
Requests per second:    261.64 [#/sec] (mean)
Time per request:       382.205 [ms] (mean)
Time per request:       3.822 [ms] (mean, across all concurrent requests)
Transfer rate:          207.05 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    3   7.1      0      70
Processing:    16  379 1594.6    118   10288
Waiting:       16  379 1594.6    117   10288
Total:         46  382 1594.8    119   10318

Percentage of the requests served within a certain time (ms)
  50%    119
  66%    125
  75%    130
  80%    133
  90%    146
  95%    163
  98%  10125
  99%  10136
 100%  10318 (longest request)

Concurrency Level: 100

Time taken for tests: 191.103 seconds

Complete requests: 50000

Failed requests: 0

Write errors: 0

Total transferred: 40517615 bytes

HTML transferred: 23050000 bytes

Requests per second: 261.64 [#/sec] (mean)

Time per request: 382.205 [ms] (mean)

Time per request: 3.822 [ms] (mean, across all concurrent requests)

Transfer rate: 207.05 [Kbytes/sec] received

Connection Times (ms)

min mean[+/-sd] median max

Connect: 0 3 7.1 0 70

Processing: 16 379 1594.6 118 10288

Waiting: 16 379 1594.6 117 10288

Total: 46 382 1594.8 119 10318

Percentage of the requests served within a certain time (ms)

50% 119

66% 125

75% 130

80% 133

90% 146

95% 163

98% 10125

99% 10136

100% 10318 (longest request)

使用lock

Concurrency Level:      100
Time taken for tests:   131.197 seconds
Complete requests:      50000
Failed requests:        0
Write errors:           0
Total transferred:      40509485 bytes
HTML transferred:       23050000 bytes
Requests per second:    381.11 [#/sec] (mean)
Time per request:       262.393 [ms] (mean)
Time per request:       2.624 [ms] (mean, across all concurrent requests)
Transfer rate:          301.53 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    2   5.0      0      71
Processing:    38  260 990.4    127   10326
Waiting:       38  260 990.4    127   10326
Total:         53  262 990.4    129   10333

Percentage of the requests served within a certain time (ms)
  50%    129
  66%    136
  75%    142
  80%    146
  90%    164
  95%    460
  98%    493
  99%   5378
 100%  10333 (longest request)

Concurrency Level: 100

Time taken for tests: 131.197 seconds

Complete requests: 50000

Failed requests: 0

Write errors: 0

Total transferred: 40509485 bytes

HTML transferred: 23050000 bytes

Requests per second: 381.11 [#/sec] (mean)

Time per request: 262.393 [ms] (mean)

Time per request: 2.624 [ms] (mean, across all concurrent requests)

Transfer rate: 301.53 [Kbytes/sec] received

Connection Times (ms)

min mean[+/-sd] median max

Connect: 0 2 5.0 0 71

Processing: 38 260 990.4 127 10326

Waiting: 38 260 990.4 127 10326

Total: 53 262 990.4 129 10333

Percentage of the requests served within a certain time (ms)

50% 129

66% 136

75% 142

80% 146

90% 164

95% 460

98% 493

99% 5378

100% 10333 (longest request)

结果很明显，启用了lock之后，SuperCache的长达10s的请求数明显更少，使得平均响应速度有明显的提升。

在实际的生产环境中，一般长达10s的请求很少，只缓存5s的请求也很少，但是通常长时间的请求伴随着都是较高的数据库负载，一旦大量出现，容易拖垮整个服务。通过这种机制，就可以明显地让缓存失效时负载曲线平滑。

暴走漫画的服务器使用了这个机制之后，使得平均响应速度从300ms左右降低至了200ms左右。

前面我写了SuperCache的介绍之后，两位朋友都表示他们在生产环境中使用了NFS，我直觉是NFS性能并不如memcached好，因为NFS牵扯了一个文件系统，但是NFS可以很好利用nginx的gzip_static。于是我做了这个评测。

我准备了三台服务器，一台服务器A用于提供memcached和NFS服务，B服务器上有nginx，挂载了A服务器上的NFS，最后通过C服务器对B服务器进行压力测试。三台服务器都是某云服务上的三台VPS，4核8G内存。

为了模拟真实环境，我首先给NFS和memcached准备了200,000条4000字节的html文件，路径格式为/articles/.html，内容是一些随机字符。

首先单纯就写入的速度来看：

      user     system      total        real
initializing NFS datas
 23.450000  83.800000 107.250000 (1020.760133)
initializing Memcached datas
 60.390000  22.500000  82.890000 (226.429582)

user system total real

initializing NFS datas

23.450000 83.800000 107.250000 (1020.760133)

initializing Memcached datas

60.390000 22.500000 82.890000 (226.429582)

写入速度明显是memcached占优，这是很容易理解的。因为NFS走了文件系统，而memcached减少了文件系统所需的诸如磁盘操作以及索引等等额外的东西。

然后我使用了siege对其进行测试

siege -c 500 -i -b -t 3M -f urls.txt

1	siege -c 500 -i -b -t 3M -f urls.txt

NFS

NFS服务器端/etc/exports配置

/data/tmp  *(rw,sync,no_root_squash)

1	/data/tmp *(rw,sync,no_root_squash)

使用NFS的nginx配置：

server {
  listen   80;
       
  root /data/tmp;
  index index.html index.htm;
  location / {
    root /data/tmp;
    #gzip_static on;
  }
}

server {

listen 80;

root /data/tmp;

index index.html index.htm;

location / {

root /data/tmp;

#gzip_static on;

}

NFS后端的结果如下：

Transactions:                  356418 hits
Availability:                 100.00 %
Elapsed time:                 179.91 secs
Data transferred:            1368.80 MB
Response time:                  0.25 secs
Transaction rate:            1981.09 trans/sec
Throughput:                     7.61 MB/sec
Concurrency:                  491.26
Successful transactions:      356418
Failed transactions:               1
Longest transaction:           15.20
Shortest transaction:           0.00

Transactions: 356418 hits

Availability: 100.00 %

Elapsed time: 179.91 secs

Data transferred: 1368.80 MB

Response time: 0.25 secs

Transaction rate: 1981.09 trans/sec

Throughput: 7.61 MB/sec

Concurrency: 491.26

Successful transactions: 356418

Failed transactions: 1

Longest transaction: 15.20

Shortest transaction: 0.00

我还额外对所有文件进行了gzip压缩（过程非常缓慢），然后启用了gzip_static：

Transactions:                  324620 hits
Availability:                 100.00 %
Elapsed time:                 179.53 secs
Data transferred:            1246.69 MB
Response time:                  0.27 secs
Transaction rate:            1808.17 trans/sec
Throughput:                     6.94 MB/sec
Concurrency:                  484.45
Successful transactions:      324620
Failed transactions:               0
Longest transaction:           15.04
Shortest transaction:           0.00

Transactions: 324620 hits

Availability: 100.00 %

Elapsed time: 179.53 secs

Data transferred: 1246.69 MB

Response time: 0.27 secs

Transaction rate: 1808.17 trans/sec

Throughput: 6.94 MB/sec

Concurrency: 484.45

Successful transactions: 324620

Failed transactions: 0

Longest transaction: 15.04

Shortest transaction: 0.00

结果效率并没有提升，原因不明。

memcached

使用memcached的nginx配置：

upstream memcache {
  server 192.168.12.26:11211;
  keepalive 512;
}
server {
  listen 80;
  root /data/tmp;
  location ~ "^/(.*)" {
    set $memcached_key $1;
    memcached_pass memcache;
  }
}

upstream memcache {

server 192.168.12.26:11211;

keepalive 512;

}

server {

listen 80;

root /data/tmp;

location ~ "^/(.*)" {

set $memcached_key $1;

memcached_pass memcache;

}

memcached后结果如下：

Transactions:                  365348 hits
Availability:                 100.00 %
Elapsed time:                 179.56 secs
Data transferred:            1403.10 MB
Response time:                  0.24 secs
Transaction rate:            2034.68 trans/sec
Throughput:                     7.81 MB/sec
Concurrency:                  490.65
Successful transactions:      365348
Failed transactions:               0
Longest transaction:            3.35
Shortest transaction:           0.00

Transactions: 365348 hits

Availability: 100.00 %

Elapsed time: 179.56 secs

Data transferred: 1403.10 MB

Response time: 0.24 secs

Transaction rate: 2034.68 trans/sec

Throughput: 7.81 MB/sec

Concurrency: 490.65

Successful transactions: 365348

Failed transactions: 0

Longest transaction: 3.35

Shortest transaction: 0.00

访问速度上基本没有太大差异，memcached略胜一筹

结论

NFS基于文件系统，Memcached完全存与内存。

所以，在需要频繁更新缓存的场景中，NFS受限于IO和网络条件，更新的效率较低。
由于NFS不具备memcached这种自动过期的功能，必须额外写脚本进行Sweep，可能不够方便
同时局限于文件系统的结构，清理过程中需要遍历文件也是十分消耗时间和磁盘IO的。

但如果不需频繁更新，NFS基于文件系统的特性更能节省内存。

另外我对NFS配置不够了解，对于NFS如何做高可用性不太清楚，但如果是memcached的话，则相对比较方便，可以在upstream中添加多个节点，并配置memcached_next_upstream等选项。

还有本文没有测试的：

如果AB机器是同一台服务器，那么结果如何
NFS有一些配置选项，比如async选项，我在这里配置的是sync选项，是否对性能有所影响

标签：benchmark

SuperCache的lock选项性能测试

不使用lock：

nginx+NFS vs nginx+memcached

NFS

memcached

结论