内存优化之Redis数据结构的设计优化实践[原创分享]


参考资料:
http://www.mysqlops.com/2011/09/06/redis-kv-design.html
http://blog.nosqlfan.com/html/3379.html

通过对文章《节约内存:Instagram的Redis实践》的阅读之后,感觉受益不少。
在文章中,Instagram 通过对数据结构的设计优化,使内存从之前的21GB逐步降低到15GB,5GB最后到达了3GB,效果非常显著。

因此自己打算在测试环境中模拟其思路,通过实践加深理解并得出一些真实的数据。

首先,需要生成一些数据,为了方便理解,我从本地CloudStack中的vm_instance表中取了一些数据。
下面我们来看一个关系型数据库的设计:

mysql> select id,instance_name,private_ip_address,uuid,created from vm_instance;
+----+---------------+--------------------+--------------------------------------+---------------------+
| id | instance_name | private_ip_address | uuid                                 | created             |
+----+---------------+--------------------+--------------------------------------+---------------------+
|  1 | s-1-VM        | 10.6.59.6          | 8c252255-82b8-4934-830e-0573cc9e0a1c | 2012-05-27 04:06:54 |
|  2 | v-2-VM        | 10.6.88.209        | 1aae6ab9-73cb-46e3-aafb-985f6a143a08 | 2012-05-27 04:06:54 |
|  4 | r-4-VM        | 169.254.1.42       | 5520f0e9-4c5a-4599-be5c-0ea74b59d6dd | 2012-05-27 10:45:42 |
|  5 | i-2-5-VM      | 10.6.8.55          | 2191b464-58be-423d-9863-ce9c0397fc67 | 2012-05-27 11:10:06 |
|  6 | i-2-6-VM      | 10.6.8.56          | c5be506a-aaae-475a-beb7-e6af2a33c8d3 | 2012-05-28 02:07:55 |

下面我们采用Redis作为数据库,首先需要将关系型数据转化为Key/Value数据。
可采用如下的方式来实现:
Key --> 表名:主键值:列名
Value --> 列值

使用冒号作为分隔符,目前算是一个不成文的规矩。例如工具php-admin for redis就是默认以冒号分割的。
下面我以前五行数据为例,数据转化的命令如下:

SET vm_instance:1:instance_name s-1-VM
SET vm_instance:2:instance_name v-2-VM
SET vm_instance:4:instance_name r-4-VM
SET vm_instance:5:instance_name i-2-5-VM
SET vm_instance:6:instance_name i-2-6-VM

SET vm_instance:1:uuid 8c252255-82b8-4934-830e-0573cc9e0a1c
SET vm_instance:2:uuid 1aae6ab9-73cb-46e3-aafb-985f6a143a08
SET vm_instance:4:uuid 5520f0e9-4c5a-4599-be5c-0ea74b59d6dd
SET vm_instance:5:uuid 2191b464-58be-423d-9863-ce9c0397fc67
SET vm_instance:6:uuid c5be506a-aaae-475a-beb7-e6af2a33c8d3

SET vm_instance:1:private_ip_address 10.6.59.6 
SET vm_instance:2:private_ip_address 10.6.88.209
SET vm_instance:4:private_ip_address 169.254.1.42
SET vm_instance:5:private_ip_address 10.6.8.55
SET vm_instance:6:private_ip_address 10.6.8.56 

SET vm_instance:1:created "2012-05-27 04:06:54" 
SET vm_instance:2:created "2012-05-27 04:06:54" 
SET vm_instance:4:created "2012-05-27 10:45:42" 
SET vm_instance:5:created "2012-05-27 11:10:06" 
SET vm_instance:6:created "2012-05-28 02:07:55"

后面在大数据量生成时我将通过脚本来实现。
这样在已知主键的情况下,通过GET,SET就可以获得或修改instance_name,private_ip_address等属性了。

一般的用户是无法知道自己的id的,只知道自己的instance_name,所以增加一个从instance_name到id的映射是个不错的注意。

SET vm_instance:s-1-VM:id 1
SET vm_instance:v-2-VM:id 2
SET vm_instance:r-4-VM:id 4
SET vm_instance:i-2-5-VM:id 5
SET vm_instance:i-2-6-VM:id 6

这样,就可以通过instance_name来方便的查找所需的值了,如下所示:

redis 127.0.0.1:6379> GET vm_instance:r-4-VM:id
"4"
redis 127.0.0.1:6379> GET vm_instance:4:private_ip_address
"169.254.1.42"

redis 127.0.0.1:6379> GET vm_instance:i-2-5-VM:id
"5"
redis 127.0.0.1:6379> GET vm_instance:5:created
"2012-05-27 11:10:06"

浏览一下当前所有的KEY数据:

redis 127.0.0.1:6379> KEYS *
 1) "vm_instance:r-4-VM:id"
 2) "vm_instance:v-2-VM:id"
 3) "vm_instance:1:instance_name"
 4) "vm_instance:i-2-5-VM:id"
 5) "vm_instance:2:instance_name"
 6) "vm_instance:i-2-6-VM:id"
 7) "vm_instance:1:uuid"
 8) "vm_instance:1:created"
 9) "vm_instance:4:instance_name"
10) "vm_instance:2:uuid"
11) "vm_instance:1:private_ip_address"
12) "vm_instance:2:created"
13) "vm_instance:5:instance_name"
14) "vm_instance:2:private_ip_address"
15) "vm_instance:6:instance_name"
16) "vm_instance:4:uuid"
17) "vm_instance:4:created"
18) "vm_instance:5:uuid"
19) "vm_instance:4:private_ip_address"
20) "vm_instance:5:created"
21) "vm_instance:5:private_ip_address"
22) "vm_instance:6:uuid"
23) "vm_instance:s-1-VM:id"
24) "vm_instance:6:created"
25) "vm_instance:6:private_ip_address"

下面,我将通过脚本来生成大量的数据(100万条)。
首先,清除所有的数据:

redis 127.0.0.1:6379> FLUSHALL

查看当前的内存耗用:

redis 127.0.0.1:6379> INFO
redis_version:2.4.17
redis_git_sha1:00000000
redis_git_dirty:0
arch_bits:64
multiplexing_api:epoll
gcc_version:4.4.5
process_id:1326
run_id:362c470ccf38b87aa955d1e1e447f58522a271c6
uptime_in_seconds:54193
uptime_in_days:0
lru_clock:643288
used_cpu_sys:571.23
used_cpu_user:72.46
used_cpu_sys_children:46.74
used_cpu_user_children:162.62
connected_clients:1
connected_slaves:1
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:0
used_memory:735864
used_memory_human:718.62K
used_memory_rss:6701056
used_memory_peak:236219680
used_memory_peak_human:225.28M
mem_fragmentation_ratio:9.11
mem_allocator:jemalloc-3.0.0
loading:0
aof_enabled:0
changes_since_last_save:30
bgsave_in_progress:0
last_save_time:1348610141
bgrewriteaof_in_progress:0
total_connections_received:1812645
total_commands_processed:5430976
expired_keys:0
evicted_keys:0
keyspace_hits:18
keyspace_misses:9
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:1144
vm_enabled:0
role:master
slave0:10.6.1.144,6379,online

内存的耗用非常少,仅为718.62K (735864)。

下面的Shell脚本将生成100万条数据(20万*5):
dongguo@redis:~/shell$ vim redis-cli-generate.sh

#!/bin/bash

REDISCLI="redis-cli -a slavepass -n 2 SET"
ID=1

while(($ID<200000))
do
  INSTANCE_NAME="i-2-$ID-VM"
  UUID=`cat /proc/sys/kernel/random/uuid`
  PRIVATE_IP_ADDRESS=10.`echo "$RANDOM % 255 + 1" | bc`.`echo "$RANDOM % 255 + 1" | bc`.`echo "$RANDOM % 255 + 1" | bc`\
  CREATED=`date "+%Y-%m-%d %H:%M:%S"`

  $REDISCLI vm_instance:$ID:instance_name $INSTANCE_NAME
  $REDISCLI vm_instance:$ID:uuid $UUID
  $REDISCLI vm_instance:$ID:private_ip_address $PRIVATE_IP_ADDRESS
  $REDISCLI vm_instance:$ID:created $CREATED

  $REDISCLI vm_instance:$INSTANCE_NAME:id $ID

  ID=$(($ID+1))
done

创建一个screen终端,将脚本放到终端中后台执行是个不错的注意。
dongguo@redis:~/shell$ screen -dmS redis
dongguo@redis:~/shell$ screen -r redis
dongguo@redis:~/shell$ ./redis-cli-generate.sh
同时按下Ctrl+AD三个按钮退出终端。

等待大约2个小时以后,数据终于写入完成(因为是虚拟机环境,所以才等这么久)。

查看一下当前的内存开销:

redis 127.0.0.1:6379> info
redis_version:2.4.17
redis_git_sha1:00000000
redis_git_dirty:0
arch_bits:64
multiplexing_api:epoll
gcc_version:4.4.5
process_id:1326
run_id:362c470ccf38b87aa955d1e1e447f58522a271c6
uptime_in_seconds:60658
uptime_in_days:0
lru_clock:643935
used_cpu_sys:858.31
used_cpu_user:105.16
used_cpu_sys_children:58.20
used_cpu_user_children:190.09
connected_clients:1
connected_slaves:1
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:0
used_memory:130548280
used_memory_human:124.50M
used_memory_rss:134524928
used_memory_peak:236219680
used_memory_peak_human:225.28M
mem_fragmentation_ratio:1.03
mem_allocator:jemalloc-3.0.0
loading:0
aof_enabled:0
changes_since_last_save:1
bgsave_in_progress:0
last_save_time:1348616616
bgrewriteaof_in_progress:0
total_connections_received:2863881
total_commands_processed:8584847
expired_keys:0
evicted_keys:0
keyspace_hits:31
keyspace_misses:10
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:19620
vm_enabled:0
role:master
slave0:10.6.1.144,6379,online
db2:keys=999995,expires=0

目前的内存耗用为124.50M (130548280)。

在数据生成之后,接下来才是本文的重点,即参考Instagram的例子做一些优化的实践。

首先,让我们确认现在的内存开销:
124.50M (130548280)

第一个优化点很明显也很简单,可以把所有key值前面相同的vm_instance:去掉,也就是之前定义的表名,将其放置在一个独立的数据库(这里选择2号)中,避免其他的数据混进来就可以了。
这里就立刻节省了12个字节的开销,然后剩下的继续设法减少开销,可以将instance_name优化为name,private_ip_address优化为ip,这样就累积节省了12+9+16=37个字节的开销。

初步优化过后的数据如下:

SET 1:name i-2-1-VM
SET 1:uuid 8c252255-82b8-4934-830e-0573cc9e0a1c
SET 1:ip 10.6.59.6 
SET 1:created "2012-05-27 04:06:54" 
SET i-2-1-VM:id 1

通过脚本导入优化过后的数据,并做内存开销上的对比。
dongguo@redis:~/shell$ cat redis-cli-generate_2.sh

#!/bin/bash

REDISCLI="redis-cli -a slavepass -n 2 SET"
ID=1

while(($ID<200000))
do
  INSTANCE_NAME="i-2-$ID-VM"
  UUID=`cat /proc/sys/kernel/random/uuid`
  PRIVATE_IP_ADDRESS=10.`echo "$RANDOM % 255 + 1" | bc`.`echo "$RANDOM % 255 + 1" | bc`.`echo "$RANDOM % 255 + 1" | bc`\
  CREATED=`date "+%Y-%m-%d %H:%M:%S"`

  $REDISCLI $ID:name "$INSTANCE_NAME"
  $REDISCLI $ID:uuid "$UUID"
  $REDISCLI $ID:ip "$PRIVATE_IP_ADDRESS"
  $REDISCLI $ID:created "$CREATED"

  $REDISCLI $INSTANCE_NAME:id "$ID"

  ID=$(($ID+1))
done

清除数据,用脚本导入新的数据。

redis 127.0.0.1:6379> FLUSHALL

dongguo@redis:~/shell$ screen -r redis
dongguo@redis:~/shell$ ./redis-cli-generate_2.sh
同时按下Ctrl+AD三个按钮退出终端。

等待大约2个小时以后,数据再次写入完成。

查看内存开销:

redis 127.0.0.1:6379> info
redis_version:2.4.17
redis_git_sha1:00000000
redis_git_dirty:0
arch_bits:64
multiplexing_api:epoll
gcc_version:4.4.5
process_id:1326
run_id:362c470ccf38b87aa955d1e1e447f58522a271c6
uptime_in_seconds:65449
uptime_in_days:0
lru_clock:644414
used_cpu_sys:1140.08
used_cpu_user:139.45
used_cpu_sys_children:66.33
used_cpu_user_children:211.75
connected_clients:1
connected_slaves:1
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:0
used_memory:117601616
used_memory_human:112.15M
used_memory_rss:121319424
used_memory_peak:236219680
used_memory_peak_human:225.28M
mem_fragmentation_ratio:1.03
mem_allocator:jemalloc-3.0.0
loading:0
aof_enabled:0
changes_since_last_save:2795
bgsave_in_progress:0
last_save_time:1348621199
bgrewriteaof_in_progress:0
total_connections_received:3863886
total_commands_processed:11584940
expired_keys:0
evicted_keys:0
keyspace_hits:41
keyspace_misses:12
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:2101
vm_enabled:0
role:master
slave0:10.6.1.144,6379,online
db2:keys=999995,expires=0

所占内存大小为112.15M (117601616)。

结论:
通过对字节数的优化,内存从124.50M (130548280) 减少到了 112.15M (117601616)。
比例为 1 - (117601616/130548280) = 1 - 0.9008285363851596 = 0.0991714636148404,即节省了9%的内存,感觉效果并不是很明显。

这个结果倒也不出乎以外,因为Instagram将内存得到了显著提升,是在使用了Hash结构对数据进行存储之后。

具体的做法呢就是将数据分段,每一段使用一个Hash结构来存储,这一点在String结构里是不存在的。
据称经过一些开发者们的实验,将hash-zipmap-max-entries设置为1000时,性能比较好,超过1000后HSET命令就会导致CPU消耗变得非常大。

于是我们可以考虑将数据做成如下结构:

redis 127.0.0.1:6379> GET 63233:name
i-2-63233-VM

redis 127.0.0.1:6379> HSET 63:name 233 i-2-63233-VM
redis 127.0.0.1:6379> HGET 63:name 233
i-2-63233-VM

redis 127.0.0.1:6379> get 63233:uuid
"556caf0f-3e6a-4b4f-a2d2-165144edaa5f"

redis 127.0.0.1:6379> HGET 63:uuid 233
"556caf0f-3e6a-4b4f-a2d2-165144edaa5f"

将4位数以上的ID值转换为Hash结构的Key值,保证每个Hash内部只包含3位的Key,也就是1000个。
对4位数以下的处理呢就很简单了,全部把他们放到ID为0的key值中。

对应的脚本如下,重新设计数据,采用Hash结构来存储:
dongguo@redis:~/shell$ cat redis-cli-generate_3.sh

#!/bin/bash

REDISCLI="redis-cli -a slavepass -n 2 HSET"
ID=1

while(($ID<1000))
do
  INSTANCE_NAME="i-2-$ID-VM"
  UUID=`cat /proc/sys/kernel/random/uuid`
  PRIVATE_IP_ADDRESS=10.`echo "$RANDOM % 255 + 1" | bc`.`echo "$RANDOM % 255 + 1" | bc`.`echo "$RANDOM % 255 + 1" | bc`
  CREATED=`date "+%Y-%m-%d %H:%M:%S"`

  $REDISCLI 0:name $ID "$INSTANCE_NAME"
  $REDISCLI 0:uuid $ID "$UUID"
  $REDISCLI 0:ip $ID "$PRIVATE_IP_ADDRESS"
  $REDISCLI 0:created $ID "$CREATED"

  $REDISCLI i-2-0:id $ID-VM $ID

  ID=$(($ID+1))
done

while(($ID<200000))
do
  INSTANCE_NAME="i-2-$ID-VM"
  UUID=`cat /proc/sys/kernel/random/uuid`
  PRIVATE_IP_ADDRESS=10.`echo "$RANDOM % 255 + 1" | bc`.`echo "$RANDOM % 255 + 1" | bc`.`echo "$RANDOM % 255 + 1" | bc`
  CREATED=`date "+%Y-%m-%d %H:%M:%S"`

  LENGTH=`expr length $ID`
  LENGTHCUT=`expr $LENGTH - 3`
  LENGTHEND=`expr $LENGTHCUT + 1`

  VALUE1=`echo $ID | awk '{print substr($1,1,"'$LENGTHCUT'")}'`
  VALUE2=`echo $ID | awk '{print substr($1,"'$LENGTHEND'",3)}'`

  $REDISCLI $VALUE1:name $VALUE2 "$INSTANCE_NAME"
  $REDISCLI $VALUE1:uuid $VALUE2 "$UUID"
  $REDISCLI $VALUE1:ip $VALUE2 "$PRIVATE_IP_ADDRESS"
  $REDISCLI $VALUE1:created $VALUE2 "$CREATED"

  $REDISCLI i-2-$VALUE1:id $VALUE2-VM $ID

  ID=$(($ID+1))
done

清除数据:

redis 127.0.0.1:6379> FLUSHALL

停止Redis服务器,以便修改配置文件参数:
dongguo@redis:~/shell$ sudo /etc/init.d/redis stop
Stopping ...
Redis stopped.

修改配置文件参数:
dongguo@redis:~/shell$ sudo vim /opt/redis/etc/redis_6379.conf

hash-max-zipmap-entries 1000

用脚本导入新的数据
dongguo@redis:~/shell$ screen -r redis
dongguo@redis:~/shell$ ./redis-cli-generate_2.sh
同时按下Ctrl+AD三个按钮退出终端。

等待大约2个小时以后,数据再次写入完成。

激动人心的时刻就要到来了。
查看内存开销:

redis 127.0.0.1:6379> info
redis_version:2.4.17
redis_git_sha1:00000000
redis_git_dirty:0
arch_bits:64
multiplexing_api:epoll
gcc_version:4.4.5
process_id:31354
run_id:35f282a72a80f2a82c13c89ba78b1b1d1281ae47
uptime_in_seconds:6064
uptime_in_days:0
lru_clock:645106
used_cpu_sys:300.16
used_cpu_user:47.59
used_cpu_sys_children:3.14
used_cpu_user_children:4.49
connected_clients:1
connected_slaves:1
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:0
used_memory:27022992
used_memory_human:25.77M
used_memory_rss:29540352
used_memory_peak:27022968
used_memory_peak_human:25.77M
mem_fragmentation_ratio:1.09
mem_allocator:jemalloc-3.0.0
loading:0
aof_enabled:0
changes_since_last_save:0
bgsave_in_progress:0
last_save_time:1348628119
bgrewriteaof_in_progress:0
total_connections_received:1000026
total_commands_processed:3000338
expired_keys:0
evicted_keys:0
keyspace_hits:14
keyspace_misses:14
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:1679
vm_enabled:0
role:master
slave0:10.6.1.144,6379,online
db2:keys=1000,expires=0

所占内存大小为25.77M (27022992)。

结论:
使用HASH结构25.77M (27022992)和使用String结构112.15M (117601616) 相比,节省内存为 1 - (27022992/117601616) = 1 - 0.229784189360119 = 0.770215810639881 。
节省了 77% 的内存

优化结果果然十分显著,由此看来,我们在Redis中,通过采用HASH结构来存储数据,和直接使用String结构相比,可以十分有效的优化内存的占用。

目前公司的线上数据大部分都采用了String结构,且String中的内容是经过加密过后的JSON数据。

我的想法是,可以尝试通过对现有的key进行修改或再次设计,将数据存储到HASH结构中,来实现对内存占用的优化。

  1. #1 by Don on 2014/08/29 - 17:17

    大受启发,谢谢!

(will not be published)
*