Erlang, delta replica system

erlang节点间传输eterm前，erlang会对eterm进行编码，如果eterm本身比较大，编解码会对系统性能造成较大的影响。对于某些特定的应用，例如备份系统，是可以通过某些方法进行优化的。
我们有一套N+1备份系统，enode里，一个worker流程结束时，会向备份enode发送状态数据，我们简单的使用{replica_handler, rep_enode} ! Msg来实现enode间消息的传递。随着系统日益庞大以及用户量的增长，平均每个worker的状态数据达到了4M左右，状态数据编解码对系统性能造成了不小的影响。
通过系统一些调试工具我们发现再我们的系统内, 单个worker相邻的两次状态数据，大部分是相同的。因此我们引入了差量备份(delta replica)机制。核心思想比较简单（类似于视频编码的关键帧和参考帧)

1. 状态备份时，首次必须发送完整的状态信息给备份节点
2. 随后i次状态备份，仅发送与上一次状态信息的差量部分给备份节点
3. 当i%10 == 0时，强制发送完整的状态信息

详见
https://github.com/dead911/erlang/wiki/erlang-delta_replica

差量备份核心代码开源如下，仅供娱乐~
https://github.com/dead911/erlang/tree/master/delta_replica

%%% -------------------------------------------------------------------------%%% Delta spec util %%% %%% delta_spec_util:get_delta_spec(Old, New) -> DeltaSpec%%%   to cacluate the delta spec of two given tuples%%%      Old - {user_info, {timestamp,86996746}, {tid, 1}, {loc, {10,11}, na}}%%%      New - {user_info, {timestamp,86996817}, {tid, 1}, {loc, {10,12}, na}}%%%%%%   When delta_spec_level = 3, the output should be%%%      DeltaSpec -[{'$1',%%%                     [{'$2',[{'$2',86996817}]},%%%                      {'$4',[{'$2',[{'$2',12}]}]}]%%%                  }]%%%   DeltaSpec indicates the difference between New and Old, each difference %%%   is presented in format [{<pos>, <value>}].%%%      {'$2',[{'$2',86996817}]} the first '$2' implies the the 2nd element %%%      of New is different from Old. In this case, it is the 'timestamp'.%%%      The following [{'$2',86996817}] means the second element of timestamp%%%      of New is differnt from of Old and the latest value is 86996817%%%%%% merge_delta_spec(OldSpec, NewSpec) -> NewSpec%%%   to merge two specs%%%   Considering the following scneario, when value changes from%%%   {hello, 1,2,3} -> {hello, 1,2,4} -> {hello, 1,1,4}%%%   Two specs will be generated,%%%   OldSpec = [{'$4',4}], {hello, 1,2,3} -> {hello, 1,2,4}, %%%   NewSpec = [{'$3',1}], {hello, 1,2,4} -> {hello, 1,1,4} %%%   After merge, NewSpec = [{'$3',1}, {'$4',4}]%%%%%% apply_spec_to_data(Data, Spec) -> NewData%%%   to apply spec to data%%%   Data    - {1, [ 2,3], 4}%%%   Spec    - [{'$1, {'$2',[{'$1',12}]},{'$3',5}]}]%%%   NewData - {1, [12,3], 5}%%% -------------------------------------------------------------------------