Deduplication 去重
Going by the definition in the context of filesystems, it’s a process of
looking up identical data blocks tracked separately and creating a shared
logical link while removing one of the copies of the data blocks. This leads to
data space savings while it increases metadata consumption.
根据文件系统上下文中的定义，去重是指查找单独跟踪的相同数据块并创建共享逻辑链接的过程，同时删除其中一个数据块的副本。这会节省数据空间，但会增加元数据消耗。
There are two main deduplication types:
有两种主要的去重类型：
in-band (sometimes also called on-line) -- all newly written data are considered for deduplication before writing
带内（有时也称为在线）--所有新写入的数据在写入之前都会被视为重复数据out-of-band (sometimes also called offline) -- data for deduplication have to be actively looked for and deduplicated by the user application
带外（有时也称为离线）--需要用户应用程序积极查找和重复数据
Both have their pros and cons. BTRFS implements only out-of-band type.
两者各有优缺点。BTRFS 仅实现带外类型。
BTRFS provides the basic building blocks for deduplication allowing other tools
to choose the strategy and scope of the deduplication. There are multiple
tools that take different approaches to deduplication, offer additional
features or make trade-offs. The following table lists tools that are known to
be up-to-date, maintained and widely used.
BTRFS 为去重提供了基本的构建模块，允许其他工具选择去重的策略和范围。有多种工具采用不同的去重方法，提供额外功能或进行权衡。以下表格列出了已知的最新、维护良好且广泛使用的工具。
Name |
File based 基于文件 |
Block based 基于块 |
Incremental |
|---|---|---|---|
No |
Yes |
Yes |
|
Yes |
No |
Yes |
File based deduplication
基于文件的去重
The tool takes a list of files and tries to find duplicates among data only
from these files. This is suitable e.g. for files that originated from the same
base image, source of a reflinked file. Optionally the tool could track a
database of hashes and allow to deduplicate blocks from more files, or use that
for repeated runs and update the database incrementally.
该工具接受文件列表，并尝试仅从这些文件中查找重复数据。例如，适用于源自相同基础图像、重定向文件源的文件。可选地，该工具可以跟踪哈希数据库，并允许从更多文件中去重块，或者用于重复运行并增量更新数据库。
Block based deduplication
基于块的去重
The tool typically scans the filesystem and builds a database of file block
hashes, then finds candidate files and deduplicates the ranges. The hash
database is kept as an ordinary file and can be scaled according to the needs.
该工具通常扫描文件系统并构建文件块哈希数据库，然后找到候选文件并去重范围。哈希数据库保存为普通文件，可以根据需要进行扩展。
As the files change, the hash database may get out of sync and the scan has to
be done repeatedly.
随着文件的更改，哈希数据库可能会失步，需要重复进行扫描。
Safety of block comparison
块比较的安全性
The deduplication inside the filesystem is implemented as an ioctl that takes
a source file, destination file and the range. The blocks from both files are
compared for exact match before merging to the same range (i.e. there’s no
hash based comparison). Pages representing the extents in memory are locked
prior to deduplication and prevent concurrent modification by buffered writes
or mmapped writes. Blocks are compared byte by byte and not using any hash-based
approach, i.e. the existing checksums are not used.
文件系统内部的重复数据删除是作为一个 ioctl 实现的，它接受源文件、目标文件和范围作为参数。在合并到相同范围之前，会比较两个文件的块是否完全匹配（即没有基于哈希的比较）。在重复数据删除之前，内存中表示范围的页面会被锁定，防止被缓冲写入或映射写入并发修改。块是逐字节比较的，不使用任何基于哈希的方法，即不使用现有的校验和。
Limitations, compatibility
限制，兼容性
Files that are subject to deduplication must have the same status regarding
COW, i.e. both regular COW files with checksums, or both NOCOW, or files that
are COW but don’t have checksums (NODATASUM attribute is set).
需要进行重复数据删除的文件必须具有相同的 COW 状态，即都是带有校验和的常规 COW 文件，或者都是 NOCOW 文件，或者是 COW 文件但没有校验和（已设置 NODATASUM 属性）。
If the deduplication is in progress on any file in the filesystem, the send
operation cannot be started as it relies on the extent layout being unchanged.
如果文件系统中的任何文件正在进行去重操作，则无法启动发送操作，因为它依赖于范围布局保持不变。