The proper way to compare two files and print lines that match:
awk 'NR==FNR{arr[$0];next} $0 in arr' file1.txt file2.txt
Here is a shell script you can run:
#!/bin/bash
# Filename: showdupes.sh
# source: http://brakertech.com/compare-two-files-and-print-lines-that-match/
# this file takes two text files as input
# sorts them and outputs lines from
# file 2 that match file 1
if [ -f "$1" ] && [ -f "$2" ]
then
awk 'NR==FNR{arr[$0];next} $0 in arr' $1.tmp $2.tmp;
else
echo "Usage: dedupe.sh file1 file2";
echo "where file1 is the 'master' file";
echo "and file2 is the file possibly containing duplicates";
fi