Answer in brief
Duplicate detection sounds like an obvious feature for a SharePoint cleanup tool. Find the files that exist more than once, keep one, delete the rest and watch your storage drop. In practice, the space you get back is usually small. Where duplicates really cost you is search.
- Exact copies are easy to find by comparing content hashes, but they rarely add up to much storage.
- Most “duplicates” are edited copies, which aren’t exact matches and are far harder to identify.
- Even when you find them, similarity doesn’t tell you which file is safe to delete.
- The real payoff from cleaning up duplicates is better search results, for people and for Copilot.
That’s why SProbot doesn’t include duplicate detection. We believe the storage benefit is too small to justify the effort, and that your time is better spent on what actually consumes space: version history, large files and content nobody has touched in years.
If storage is your priority, start there instead: SharePoint cleanup checklist for IT admins.
Why duplicate detection seems like an easy win
In most tenants like yours, you see people taking these actions:
- Download a file from a SharePoint site and re-upload it to a Teams channel.
- Reuse last year’s proposal as a template. A revised version gets saved with “FINAL” on the end, then “FINAL v2” a week later.
- Export to PDF several times after editing the same Word file to get it to a ready state (the Word doc stays a single file because of versioning).
It’s natural to assume these kinds of activities add up to a lot of wasted storage, but it's worth testing that assumption before you spend money chasing it.
What counts as an exact duplicate?
This is the easy one. Two files have identical content, even if they’re named differently or stored in different sites. You don’t need AI to spot that, just a dependable way to compare what’s inside them.
Microsoft Graph gives you one. It exposes several hash properties for files, and Microsoft provides quickXorHash as a proprietary hash you can use to tell whether a file’s contents have changed.
If hashes are new to you, a hash is a short value calculated from a file’s contents. Identical content gives you an identical hash, so instead of comparing files byte by byte, you compare their hashes. Take these two:
Different names, different locations. If the contents match, so will the hashes, and you have a solid exact-copy candidate. That’s useful to know. Whether it’s worth much in storage terms is another question.
Then somebody edits the copy
Copying is rarely the end of it. People copy a document because they want to do something with it: rewrite a paragraph, update a price, strip out a worksheet, swap in a different customer name or simply carry on from an older version.
Customer proposal.docx
Customer proposal v2.docx
Customer proposal FINAL.docx
Customer proposal FINAL comments.docx
Customer proposal - sent.docx
Look at that list and you’d assume there’s duplicated content in there. A hash comparison won’t find it, because once the contents change, the files are no longer exact copies. Finding those means looking inside the documents.
Finding similar files is expensive
AI and other content-analysis techniques can extract the text from documents and measure how alike they are. That works, but the question has changed. You’re no longer checking whether two files are identical. You’re estimating how similar two different documents are, which takes far more processing than comparing hashes.
On one library, that can be reasonable. Across a whole tenant, it’s a big job for an uncertain return.
Why the storage savings are usually small
Even if you found every duplicate in your tenant, the storage you’d recover is usually a small slice of the total. Every version of a file counts toward your storage, so one heavily edited 50 MB presentation with a few hundred versions can outweigh a handful of copies of it. Large media files and project sites nobody has opened in years tend to dwarf duplicated documents too.
And most duplicates are edited copies, which are the hardest to find and the riskiest to delete. That trade-off is why we have so far left duplicate detection out of SProbot.
Similar doesn’t mean redundant
Even a perfect similarity score leaves the important question open. A contract before and after signature can be almost word for word the same, yet the difference may be the most important thing in its history. AI can show that two documents look alike. It's much harder (and expensive) for it to figure out:
- which one is the authoritative version
- why the second copy exists
- whether the differences matter
- whether either can safely be removed
Similarity is a reason to review, but deletion is a separate decision.
Where duplicate detection does help: search
Duplicates have a much bigger effect on findability than on storage. When five versions of the same proposal sit across three sites, a search returns all five, and the person searching has to work out which one is current. Sometimes they pick the wrong one.
That’s where duplicate detection earns its keep. Finding clusters of similar documents helps site owners decide which version is authoritative, archive or label the rest, and point people to the right file. The win is accuracy and time saved, not gigabytes.
A practical way to handle duplicates
Treat duplicates as a findability problem. Pick the sites and libraries people search most, or the ones Copilot is likely to draw on, and deal with exact copies there first. Use similarity analysis only on the smaller set that’s left. Then someone who knows the documents has to answer the one question the technology can’t:
Do we still need both files?
For storage, focus on what actually takes up space
If reclaiming storage is the goal, go after version history, very large files, inactive content and abandoned sites. That’s where the space tends to go, and it’s relatively cheap to identify because nothing has to understand what’s inside each document.
Start here: SharePoint storage limit exceeded: 6 proven ways to reclaim space
The bottom line
Duplicate detection is worth doing when your search results are a mess and people keep working from the wrong version. It’s not a strong storage play. Exact copies are easy to find but rarely add up to much space, and edited copies take real effort to find and judge.
So if a cleanup tool promises to cut storage by finding duplicates, ask how much space that’s realistically going to recover in your tenant, and whether there’s a bigger opportunity it isn’t showing you.
FAQ
Can SharePoint automatically find duplicate files?
No, but the building blocks are there. Microsoft Graph exposes file hashes, including QuickXorHash, which can be used to tell whether a file’s contents have changed. That lets you compare exact content, but it won’t tell you whether two different documents are redundant.
Is duplicate detection worth looking for in a SharePoint cleanup tool?
It depends on what you’re trying to fix. For search quality and Copilot readiness, it can help, provided someone reviews the results before anything is deleted. For storage, it’s rarely where the big wins are. Inactivity, file size and version history are better places to look.
Does deleting duplicate files free up much SharePoint storage?
Usually not much. Exact copies are easy to remove, but most SharePoint storage tends to sit in version history, large files and inactive content. Edited copies are harder to find and riskier to delete, so the effort rarely matches the space you get back.
