Tags, being one of the most basic implementations of boolean retrieval, tend to suffer from feast-or-famine a lot, at least in my experience. Once you introduce hierarchical tagging, people will just use them like folders with each item having 1.0x tags on average.

The Didget system was designed to allow both a hierarchical folder tree as well as tags attached to individual files and folders. The tags do not replace the hierarchy unless you want them to. If you have never ever seen a system that actually works, then maybe you should put Didgets to the test. I created 20 million files in it and attached an average of 100 tags to each one. Each tag had a value randomly picked among 1000 choices. Queries to find all files with a certain tag (e.g. Tag_134 = Value_875) each completed in less than a second.

> I created 20 million files in it and attached an average of 100 tags to each one. Each tag had a value randomly picked among 1000 choices.

That is really impressive, but when the parent commented that they have never seen a tag-based system work with a large number of files and tags, I don't think they were making a statement of technical capability but of human fallibility.

My experience has largely been identical in both personal usage and in enterprise settings. Every time I've used a system that used human-defined tags as the primary organizing mechanism it has always ended in an unusable mess and in every case it is eventually replaced by some kind of hierarchy which usually ends up being a slightly more usable mess.

Perhaps combining them will yield the best of both worlds and perhaps with enough organizational discipline one can make a tag-based organizational system work. And I'm all for better search. But at the end of the day I am skeptical that giving normal people even more flexibility with how they organize their files will make their lives easier.

Most tagging systems that I have seen are free form. Anyone can just tag something with tags like 'James', '2002', or 'Bank Statement'. This makes it difficult to distinguish between them and easily find things like misspellings. All tags must be of the same data type (string). A generic term like 'Tank' might refer to a water storage device, a military vehicle, or someone's nickname.

With Didgets, I decided to go with a contextual approach to tagging. Just like columns in a relational table, a tag must be defined before you can use it and all like tags are managed together. A tag can have a data type so 'Year' can be an Integer, for example. The system comes with a set of pre-defined tags, but users can easily add whatever tags they might need. That way a tag has the form 'Author = James' or 'Device = Camera'. I went further and decided each tag definition would have two levels. '.person.FirstName = James' might be a tag on a picture of someone named James. This makes it easier to search for tags by group (e.g. find all documents that have '.person.*' tags attached). By managing the tag values together, the UI can quickly show a list of values that have been used previously (and order them by use count). When attaching names to photos, it can show you a list of the most used names and let you pick one or ignore the list and add a new one.

This system is far from perfect. Users can still misspell tags or categorize them incorrectly. But this can happen with folder names in existing file systems as well.

Just to be clear, tags in Didgets do not have to be the primary organizing mechanism. It has 'Set Didgets' that contain the IDs of all members and can be arranged in a hierarchy just like folders. When importing files, the UI creates these sets (unless the user specifically turns it off) and preserves the hierarchy of the source file system.

Those are improvements over free-form tags and I especially like the namespaces for attribute/field names, that's clever. The enterprise software my company makes has decent support for defining and applying structured tags like business terms and attribute values to the data objects our customers manage, and our users do make use of those when filtering and searching.

But (in my experience) many people still seem to gravitate towards storing and navigating objects hierarchically. I can think of a few possible reasons:

First, some people intuitively think of a data element as having a location in an information space. That is, they seem to intuitively remember "where" something is by piggy-backing on spatial memory in a way that tags don't seem to trigger.

Second, navigating a hierarchy involves a sequence of constrained choices, like a wizard. Having a sequence of decisions can be especially helpful for novices. It also generally takes a predictable number of steps to locate an item which can be preferable to something that is faster on average but has slow edge cases.

Third, at each level of the hierarchy you can often display all of the options meaning we can rely on recognition over recall[1].

(You could constrain yourself to hierarchical tags and use hierarchy-like positional language such as "object is in baz" at which point I'd consider it a hierarchy.)

Of course relying on tags has plenty of upsides -- typically faster, better mental model for overlapping sets -- and large-scale data storage systems need both. But at the end of the day they don't seem to be a replacement for hierarchical systems for most people.

[1] https://www.nngroup.com/articles/recognition-and-recall/

>I have never (never ever ever) seen a tag based system actually work once you have large amounts of files and tags

If you've ever done online shopping, you probably have. For example, try going on Amazon or Newegg and searching for a GPU. You're shown a sidebar where you can easily filter results be certain tags such as: brand, price range, memory size, core count, in stock, energy star certified, free shipping, etc.

I don't think these systems work nearly as well as you do.

Simple example right now:

Go on amazon, and search for "intel CPU" - I see the following:

1-16 of 942 results for "intel cpu"

Now go back and search for "cpu", then filter by brand "intel" - I see the following:

1-24 of 835 results for "cpu"

It turns out the tags are exactly what I said they would be - a hodgepodge of things not correctly applied. For example - searching "intel cpu" actually returns items that include intel CPUs (such as motherboard + cpu bundles) that are missing in just the tagged search. But it's still absolutely a valid result if I was interested in buying a cpu.

---

as mostly an aside - I don't really trust Amazon or Newegg to be neutral in their results either, a tagged view is convenient to them as a seller where they can control results.

... and on the top of the list, the first thing you click as a part of the filter, is a hierarchical locator:

Home Components -> Video Cards & Video Devices -> Desktop Graphics -> Cards Search Results: "GPU"

The tags system works for specific areas. For example, tags in photo management apps are great. But they don't really work across separate domains, so what you want is top-level hierarchy, and, where needed, tags for the subtrees. That's how existing tag systems work.