socket.io 高并发实战

socket.io实战

nodejs mult process + nginx + redis + socket.io 实现c50K,目前正在优化app,希望能实现c100K. 目前 c50K 非常稳定。c100K 下会有写 400 和 502 错误，以及个别进程 cpu 100%.

通过 v8 自带的 prof 监控的日志：

ticks  total  nonlib   name
11453687   94.9%    0.0%  /lib64/libc-2.12.so
327343    2.7%    0.0%  /usr/bin/node
32393    0.3%    0.0%  /lib64/libpthread-2.12.so
1189    0.0%    0.0%  7fff269c3000-7fff269c4000
996    0.0%    0.0%  /lib64/libm-2.12.so
947    0.0%    0.0%  /usr/lib64/libstdc++.so.6.0.13
475    0.0%    0.0%  /lib64/librt-2.12.so
391    0.0%    0.0%  ffffffffff600000-ffffffffff601000
  1    0.0%    0.0%  /lib64/ld-2.12.so

JavaScript

ticks  total  nonlib   name
34551    0.3%   13.9%  LazyCompile: *exports._unrefActive timers.js:425
 9061    0.1%    3.6%  Stub: CompareStub_EQ
6462    0.1%    2.6%  LazyCompile: EventEmitter.emit events.js:53
4519    0.0%    1.8%  LazyCompile: EventEmitter.addListener events.js:126
3929    0.0%    1.6%  KeyedLoadIC: A keyed load IC from the snapshot
3818    0.0%    1.5%  LazyCompile: ~onread net.js:496
3817    0.0%    1.5%  LazyCompile: *remove _linklist.js:47
3117    0.0%    1.3%  LazyCompile: *append _linklist.js:63
2857    0.0%    1.1%  LazyCompile: *writeOrBuffer _stream_writable.js:200
2653    0.0%    1.1%  Builtin: A builtin from the snapshot {5}
2590    0.0%    1.0%  Stub: CEntryStub
2242    0.0%    0.9%  LazyCompile: *EventEmitter events.js:26
2171    0.0%    0.9%  LazyCompile: EventEmitter.removeListener events.js:191
2047    0.0%    0.8%  Builtin: A builtin from the snapshot
1976    0.0%    0.8%  Stub: FastNewClosureStub
1971    0.0%    0.8%  LazyCompile: *Url.parse url.js:105
1824    0.0%    0.7%  LazyCompile: *Buffer buffer.js:156
1763    0.0%    0.7%  Stub: CompareICStub {2}
1710    0.0%    0.7%  CallMegamorphic: args_count: 2
1633    0.0%    0.7%  LazyCompile: *Decode native uri.js:208
1509    0.0%    0.6%  LazyCompile: *Readable.read _stream_readable.js:252
1491    0.0%    0.6%  CallMegamorphic: args_count: 1
1457    0.0%    0.6%  LazyCompile: *_nextDomainTick node.js:493
1444    0.0%    0.6%  Stub: ToBooleanStub_UndefinedSpecObject
1411    0.0%    0.6%  Stub: InstanceofStub
1410    0.0%    0.6%  LazyCompile: ~OutgoingMessage.end http.js:915
1339    0.0%    0.5%  LazyCompile: _tickDomainCallback node.js:426
。。。。。。。。。

C++

ticks  total  nonlib   name

ticks  total  nonlib   name
29804    0.2%

Bottom up (heavy) profile

Note: percentage shows a share of a particular caller in the total
amount of its parent calls.
Callers occupying less than 2.0% are not shown.

ticks parent name

11453687   94.9%  /lib64/libc-2.12.so

327343    2.7%  /usr/bin/node
18923    5.8%    LazyCompile: EventEmitter.addListener events.js:126
14701   77.7%      LazyCompile: *Readable.on _stream_readable.js:688
 4420   30.1%        LazyCompile: *connectionListener http.js:1903
4121   93.2%          LazyCompile: EventEmitter.emit events.js:53
4121  100.0%            LazyCompile: *onconnection net.js:1163
200    4.5%          LazyCompile: *onconnection net.js:1163
 89    2.0%          LazyCompile: ~EventEmitter.emit events.js:53
 89  100.0%            LazyCompile: *onconnection net.js:1163
3964   27.0%        LazyCompile: *ServerResponse.assignSocket http.js:1108
3964  100.0%          LazyCompile: ~parser.onIncoming http.js:2038
3812   96.2%            LazyCompile: *parserOnHeadersComplete http.js:69
136    3.4%            LazyCompile: ~parserOnHeadersComplete http.js:69
3287   22.4%        LazyCompile: *EventEmitter.once events.js:169
1673   50.9%          LazyCompile: *Duplex _stream_duplex.js:39
1656   99.0%            LazyCompile: *Socket net.js:135
688   20.9%          LazyCompile: ~onread net.js:496
542   16.5%          LazyCompile: *onSocketEnd net.js:243
495   91.3%            LazyCompile: EventEmitter.emit events.js:53
 29    5.4%            LazyCompile: *onread net.js:496
 17    3.1%            LazyCompile: ~EventEmitter.emit events.js:53
241    7.3%          LazyCompile: *afterShutdown net.js:222
131    4.0%          LazyCompile: *onread net.js:496
1220    8.3%        LazyCompile: *Socket net.js:135
1217   99.8%          LazyCompile: *onconnection net.js:1163

高并发Nodejs参数调整

关闭v8 空时通知机制

--nouse-idle-notification

修改http.Agent

官网说明：
agent.maxSockets
By default set to 5. Determines how many concurrent sockets the agent can have open per host.
(为了http请求能复用connection连接，Nodejs在http.Agent创建了一个默认大小为5的连接池)
修改后如下:
require("http").globalAgent.maxSockets = Infinity;

修改–max-old-space-size

--max-old-space-size=2048（根据自己情况，可以调大,单位是M）
说明：v8 在64位操作系统默认使用的max-old-space-size是1.7G,大家可以通过：node --v8-options 查看V8参数

使用PM2管理

例如：
{
"apps" : [
    {
        "name": "comet-server-4000",
        "script": "server.js",
        "port": 4000,
        "args": "['-p4000','-t','plan']",
        "run-as-group" : "comet",
        "exec_mode": "cluster_mode",
        "node-args": "--nouse-idle-notification --gc_global --max-old-space-size=2048 --max-new-space-size=1024"
    },
    {
        "name": "comet-server-4001",
        "script": "server.js",
        "port": 4001,
        "run-as-group": "comet",
        "args": "['-p4001','-t','plan']",
        "exec_mode": "cluster_mode",
        "node-args": "--nouse-idle-notification --gc_global --max-old-space-size=2048 --max-new-space-size=10240"
    }
]
}

避免在socket.io实时推送项目中使用同步代码，推送项目应该是以中间件的身份出现的,只传输数据