How to deal with large document libraries in SharePoint Online

If you are reading this post, then you most likely committed the most famous sin in SharePoint – exceeded the item threshold in a document library. Don’t worry, we all make mistakes in life. So I am here to forgive you :-)  and explain what this issue is all about, how to mitigate it, and how to deal with large document libraries in SharePoint in general.

What is this item limit threshold on a document library?

It all started with SharePoint 2010. To maintain the performance of a document library (and the servers behind the scenes), Microsoft introduced the 5,000 view item limit on a document library. What that meant was that while you could put more than 5,000 items in a single library, you could not surface up (display) more than 5,000 items on a view. I actually wrote a separate post on the 5,000 view item limit issue back in 2015.

What happens if I put more than 5,000 items in a document library

Classic SharePoint/Classic Document Library

In the past, with older versions of SharePoint, the classic library pretty much became unusable. Yes, you could put 20,000 documents in a library, but if you tried to do anything in that library (search, set unique permissions, create views, sort, filter, group metadata, etc.) – you would get an error message every little step of the way.

Modern Document Library

A breakthrough occurred in 2017 with the introduction of a Modern Document Library. All of a sudden, it became more “tolerant” towards the larger volumes of data. Over the years, Microsoft made additional improvements to the modern document library, like auto-indexing libraries that contain less than 20,000 documents.

With the modern document library, you can now upload 20K, 30K, 50K documents. However, the limits are still there, and chances are – you will encounter the error messages described below.

What happens if I put a large number of documents in a document library

The structure of a library matters

So while 5,000 is no longer a hard limit, there is still a limit to how many docs a document library can handle. There is no exact number I can give you where you will get to see the errors. It depends on how you structure your library, whether or not files are organized in folders or metadata, and whether or not you sync your library to the computer.

OneDrive sync makes the issues even worse

Another factor to consider is that often users Sync the libraries to their computer via OneDrive Sync. This exacerbates the issue as OneDrive sync has limitations of its own. For example, when you have lots of files and folders in a single folder, you will get an error message below

Microsoft also recommends that users do not sync more than 300,000 files across all SharePoint document libraries/OneDrive. So the larger the library, the sooner your users will encounter the issues.

Microsoft recommendation taken from their SharePoint limits article

Common Error messages with large document library threshold

Below I would like to explain the most common error messages you will see (and respective operations that cause them).

1. The user tries to move/reorganize folders with a large number of documents
2. The user tries to create unique permissions for the folder or a library that contains more than 100,000 items. The second image (captured from Mircosoft SharePoint limits post) summarizes this hard limit
3. The user tries to rename a folder with a large library or folder that contains a large number of items
4. The user tries to move/reorganize files and folders within a document library (takes a lot of time)
5. The user tries to rename a folder (takes a lot of time)
6. The user tries to set unique security on a folder (takes a lot of time)
7. The user tries to bulk edit metadata for multiple files
8. The user tries to sort and group on some metadata columns (which reminds me, I need to buy myself a bicycle)
9. The user tries to create a custom view and hide folders
10. The user tries to create a Calculated Column
11. The user gets an error message about too many files in a folder while syncing using OneDrive sync client
12. The user gets an error message about sync issues in OneDrive
13. The user gets a warning/error message about sync delays due to high service activity
14. The user unable to access SharePoint sites and gets Throttle message when trying to access SharePoint (while also syncing a large document library)
15. The user’s OneDrive sync wheel is always spinning, freezes sometimes

How to properly mitigate large document library issue

If you read my old post on 5,000 view item limit, as well as posts written by others on the same topic, there are some techniques available to improve the usability of a large library. Things like indexing columns, creating custom views, limiting the number of items in a view. That said, don’t waste your time on any of them. They will only provide marginal improvements (if any), and your users will still encounter most of the issues/error messages above.

There is only one fool-proof way to avoid and mitigate the issue:

Do not put more than 20-30K files and folders into a single library. Create many sites and libraries.

Just like with driving, if you do not want to be caught speeding, well, then maybe you should not be speeding! Same thing with SharePoint document libraries. Don’t do it! Instead, concentrate on the following:

1. Think in terms of sites and not folders.
2. Create many sites and many libraries, spread the content among them
3. Don’t use SharePoint like Dropbox or file share
4. Spend some time building proper Information Architecture
5. Train your users on dos and don’ts of SharePoint