Finding and removing duplicate files with fdupes

Published on 2009-08-01.

If you have a lot of duplicate files "fdupes" can help you remove them very easily.

On Debian derived systems you can install fdupes using the command:

# apt-get install fdupes

The program is run from the terminal and the options are very basic. To search a directory for duplicate files, simply run the following command:

$ fdupes -r my_dir/

The command will recursively search the directory my_dir for any duplicate files, and display the results.

You can also create a text file containing the names of you duplicate files running this command:

$ fdupes -r my_dir/ > my_duplicate_files.txt

What I needed was to remove all duplicate files except for one of each file. I also needed a list of the files that was kept and the ones that was removed. To do that I used this command:

$ fdupes -rdN my_dir/ > dubs.txt

The r option makes fdupes search for files recursively. The d option makes fdupes delete duplicates. The N option, when used together with d, preserve the first file in each set of duplicates and delete the others without prompting the user.

See man fdupes for further instructions.