Every critical business asset has the potential to hold duplicate data. An incremental backup will back up the entire file, even though you may have changed only one byte. Imagine how many times you make a tiny change to a document. If it has never been seen before, the new shard is written to storage and the hash is added to the hash table/database if not, it is discarded and an additional reference added to the hash table/database. The value of that hash is then checked against a hash table or hash database to see if it’s ever been seen before.

The deduplication system runs each shard through a hashing algorithm, such as SHA-1, SHA-2, or SHA-256, which creates a cryptographic alpha-numeric (referred to as a hash) for the shard. Once the dataset has been split into a series of small pieces of data, referred to as chunks or shards, the rest of the process usually remains the same. Most block-level deduplication occurs at fixed block boundaries, but there is also variable-length deduplication or variable block deduplication, where data is split up at non-fixed block boundaries. If they are referring to file-level deduplication, they will use that modifier. When most people say deduplication, they are referring to block-level deduplication. This is called block-level deduplication or sub-file deduplication, and it frees up storage space. This is also called single instance storage (SIS) or file-level deduplication.Īt the next level, deduplication identifies and eliminates redundant segments of data that are the same, even when the files they’re in are not entirely identical. In its most basic form, the process happens at the level of single files, eliminating identical files. There is more than one kind of data deduplication.

Best deduplication software