redis持久化之aof

标签: redis

一、简介

  • aof,全称叫"append only file", aof是以日志的形式来记录每个写操作,将Redis执行过的所有写指令记录下来(读操作不记录),只许追加文件但不可以改写文件,redis启动之初会读取该文件重新构建数据,redis重启的话根据日志文件的内容将写指令从头到尾执行一次以完成数据的恢复。AOF方式持久化默认保存的是appendonly.aof文件。
  • 配置文件位置:
# APPEND ONLY MODE redis持久化之aof方式配置
############################## APPEND ONLY MODE ###############################

# By default Redis asynchronously dumps the dataset on disk. This mode is
# good enough in many applications, but an issue with the Redis process or
# a power outage may result into a few minutes of writes lost (depending on
# the configured save points).
#
# The Append Only File is an alternative persistence mode that provides
# much better durability. For instance using the default data fsync policy
# (see later in the config file) Redis can lose just one second of writes in a
# dramatic event like a server power outage, or a single write if something
# wrong with the Redis process itself happens, but the operating system is
# still running correctly.

# aof和rdb文件可以同时存在,redis启动时会优先加载aof文件。 
# AOF and RDB persistence can be enabled at the same time without problems.
# If the AOF is enabled on startup Redis will load the AOF, that is the file
# with the better durability guarantees.
#
# Please check http://redis.io/topics/persistence for more information.

# aof持久化开关,默认是关闭的
appendonly no

# The name of the append only file (default: "appendonly.aof")

# aof备份文件的名称
appendfilename "appendonly.aof"

# The fsync() call tells the Operating System to actually write data on disk
# instead of waiting for more data in the output buffer. Some OS will really flush
# data on disk, some other OS will just try to do it ASAP.
#
# Redis supports three different modes:
#
# no: don't fsync, just let the OS flush the data when it wants. Faster.
# always: fsync after every write to the append only log. Slow, Safest.
# everysec: fsync only one time every second. Compromise.

# 默认是每秒记录,如果一秒内宕机,有数据丢失,最多只会丢失一秒内的数据
# The default is "everysec", as that's usually the right compromise between
# speed and data safety. It's up to you to understand if you can relax this to
# "no" that will let the operating system flush the output buffer when
# it wants, for better performances (but if you can live with the idea of
# some data loss consider the default persistence mode that's snapshotting),
# or on the contrary, use "always" that's very slow but a bit safer than
# everysec.
#
# More details please check the following article:
# http://antirez.com/post/redis-persistence-demystified.html
#
# If unsure, use "everysec".

# aof持久化策略
# Always:同步持久化,每次发生数据变更会立即记录到磁盘,性能较差但数据完整性比较好
# appendfsync always
# Everysec:默认配置,异步操作,每秒记录,如果一秒内宕机,有数据丢失
appendfsync everysec
# 
# appendfsync no

# When the AOF fsync policy is set to always or everysec, and a background
# saving process (a background save or AOF log background rewriting) is
# performing a lot of I/O against the disk, in some Linux configurations
# Redis may block too long on the fsync() call. Note that there is no fix for
# this currently, as even performing fsync in a different thread will block
# our synchronous write(2) call.
#
# In order to mitigate this problem it's possible to use the following option
# that will prevent fsync() from being called in the main process while a
# BGSAVE or BGREWRITEAOF is in progress.
#
# This means that while another child is saving, the durability of Redis is
# the same as "appendfsync none". In practical terms, this means that it is
# possible to lose up to 30 seconds of log in the worst scenario (with the
# default Linux settings).
#
# If you have latency problems turn this to "yes". Otherwise leave it as
# "no" that is the safest pick from the point of view of durability.

# 重写时是否可以运行Appendfsync,用默认no即可,保证数据安全性
no-appendfsync-on-rewrite no

# Automatic rewrite of the append only file.
# Redis is able to automatically rewrite the log file implicitly calling
# BGREWRITEAOF when the AOF log size grows by the specified percentage.
#
# This is how it works: Redis remembers the size of the AOF file after the
# latest rewrite (if no rewrite has happened since the restart, the size of
# the AOF at startup is used).
#
# This base size is compared to the current size. If the current size is
# bigger than the specified percentage, the rewrite is triggered. Also
# you need to specify a minimal size for the AOF file to be rewritten, this
# is useful to avoid rewriting the AOF file even if the percentage increase
# is reached but it is still pretty small.
#
# Specify a percentage of zero in order to disable the automatic AOF
# rewrite feature.

# 默认配置是当AOF文件大小是上次rewrite后大小的一倍且文件大于64M时触发重写。
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb

# An AOF file may be found to be truncated at the end during the Redis
# startup process, when the AOF data gets loaded back into memory.
# This may happen when the system where Redis is running
# crashes, especially when an ext4 filesystem is mounted without the
# data=ordered option (however this can't happen when Redis itself
# crashes or aborts but the operating system still works correctly).
#
# Redis can either exit with an error when this happens, or load as much
# data as possible (the default now) and start if the AOF file is found
# to be truncated at the end. The following option controls this behavior.
#
# If aof-load-truncated is set to yes, a truncated AOF file is loaded and
# the Redis server starts emitting a log to inform the user of the event.
# Otherwise if the option is set to no, the server aborts with an error
# and refuses to start. When the option is set to no, the user requires
# to fix the AOF file using the "redis-check-aof" utility before to restart
# the server.
#
# Note that if the AOF file will be found to be corrupted in the middle
# the server will still exit with an error. This option only applies when
# Redis will try to read more data from the AOF file but not enough bytes
# will be found.
aof-load-truncated yes

# When rewriting the AOF file, Redis is able to use an RDB preamble in the
# AOF file for faster rewrites and recoveries. When this option is turned
# on the rewritten AOF file is composed of two different stanzas:
#
#   [RDB file][AOF tail]
#
# When loading Redis recognizes that the AOF file starts with the "REDIS"
# string and loads the prefixed RDB file, and continues loading the AOF
# tail.
aof-use-rdb-preamble yes
  • 实现原理:aof方式持久化通过保存redis执行时候的所有写操作命令,注意,读操作命令不追加,将写命令追加到appendonly.aof文件中,理论上只要保存了所有写操作命令,那么恢复时将这些命令从头到尾再执行一次就可以从正常恢复数据。

二、aof示例

下面通过正常恢复和异常恢复两种方式详细说明如何在redis中使用aof持久化,具体操作如下:

(一)正常恢复

首先编辑redis.conf配置文件,打开aof开关:

接着我们先进行一些写操作:

通过右边窗口可以看到,执行完写操作后就生成了appendonly.aof文件;下面我们去看看文件里面是怎么记录命令的:

对照着左边窗口的操作命令看,相信小伙伴们很容易就看出是怎么存储的。下面我们重启redis:

可以发现,并没有恢复数据,原因如下: 因为刚刚执行了flushall,redis也将该命令记录到appendonly.aof文件中,这样当恢复完数据之后又被清空了。

接着,我们不执行flushall,直接shutdown,然后再次重启之后,可以发现redis帮我们恢复了原来的数据。

 (二)异常恢复

 下面我们来搞搞破坏,之所以破坏就是为了模拟生产环境一些突发情况损坏aof文件,

vim appendonly.aof 

我们随便在最后面追加点内容。

然后我们尝试重启redis:

发现redis重启失败,这其实说明了redis启动的时候,如果存在dump.rdb文件和aof文件的话,会先加载aof文件,因为aof文件被破坏,所以导致重启失败。

下面我们来修复aof文件,使用下面的命令修复:

redis-check-aof --fix appendonly.aof

 

修复完我们重启redis:

可以发现,redis服务可以重启成功了,也帮我们恢复数据了,修复后的aof文件如下所示:

四、aof持久化策略

 redis的aof默认提供了三种持久化的策略:

# 每执行一条命令持久化一次
# appendfsync always
# 每秒持久化
appendfsync everysec
# 从不同步
# appendfsync no

下面是各个策略的说明:

  • appendfsync always:Always:同步持久化,每次发生数据变更会立即记录到磁盘,性能较差但数据完整性比较好,always会严重影响服务器的性能;
  • appendfsync everysec:Everysec:默认就是此方式,异步操作,每秒记录,如果一秒内宕机,有数据丢失;
  • appendfsync no:从不同步  

五、重写机制rewrite

aof采用文件追加方式,文件会越来越大,为避免出现此种情况,redis新增了重写机制,当aof文件的大小超过所设定的阈值时,Redis就会启动aof文件的内容压缩,只保留可以恢复数据的最小指令集。当然我们也可以使用命令bgrewriteaof手动告诉redis进行aof文件的重写,这样可以有效减小aof文件的体积。

重写原理:

redis会fork出一个子进程来对aof文件进行重写,原理跟快照持久化命令 bgsave 的工作原理类似,值得注意的是,进行aof文件重写时,如果原来的 aof文件体积已经非常大,那么重写aof并删除旧aof 文件的过程将会对 Redis 的性能造成较大的影响。

Redis提供了两个配置选项来对 aof 重写进行配置:

auto-aof-rewrite-percentage: 配置当aof文件需要比旧 aof文件增大多少时才进行重写
auto-aof-rewrite-min-size:配置当aof文件需要达到多大体积时才进行重写

只有这两个配置的条件都达到时,才会进行重写。

六、总结

  •  优点:
  • 使用默认的持久化策略:每秒同步一次,Redis 最多只会丢失 1 秒的数据;
  • aof文件的格式可读性较强,同时还可以检查aof文件中的错误并进行修复;
  •  
  • 缺点:
  • 对相同的数据集来说,aof文件的体积通常要大于rdb文件的体积,恢复速度慢于rdb;
  • aof运行效率要慢于rdb,每秒同步策略效率较好,不同步的话效率和rdb相同;
  • 总结
  • redis允许同时存在rdb和aof两种持久化方式,但是重启时会优先加载aof文件;
  • 如果在项目中只是将redis作为缓存服务器的话,我们其实可以不用进行持久化,因为缓存对数据敏感性没那么强;
  • 因为rdb只是作为后备使用,建议只在slave从服务器上备份rdb文件,而且只需要15分钟备份一次就够了,只保留save 900 1这条规则即可;
  • aof持久化方式最多只会丢失一秒的数据,但是这带来了很多额外的IO操作,其次就是重写造成的阻塞不可避免,所以应该尽量减少aof重写的频率,建议将参数auto-aof-rewrite-min-size修改为5G或者更大;
  • 如果不使用aof只使用主从同步的话,可以减少很多的IO操作,但是当主机master和slave从机都宕机时丢失的数据会比较多,安全性和可靠性不够强;
原文链接:加载失败,请重新获取