So we have hand-curated directories, and we have social networks. Put them together and you could have a scalable, crawlable, searchable, customizable and non-gamable decentralized indexing system. Just don't know why nobody has come with this yet. Maybe lack of (monetary) incentives? Maybe something for IETF or WWWC to initiate?
Just for fun, imagine an alternate reality where nobody cracked the problem of an effective automated indexing algorithm. Hand-curated directories emerge, some structured by taxonomic categorization, others using keyword tagging, still others reliant on user-based rankings. The most successful grow in popularity and consolidate, becoming highly lucrative properties with recognizable brand names. As they grow to gargantuan size, research in the field explodes and innovators race to come up with improved methods to connect people to content they seek. Sergey Dewey and Larry Linnaeus make a breakthrough they call SageRank. Instead of computer code, it leverages "social algorithms" and game theory to incentivize participant behavior. Vladmir Bezos sinks billions into a clandestine effort to game the system. Once the story breaks, public backlash rallies into a worldwide, anti-"fake links" campaign. In one little corner of the internet, some schmuck says, "Imagine an alternate reality where two nerds came up with an impartial computer program to crawl the whole web..."
Since Twitter is their preferred platform, go put the activity of journalist Twitter accounts into a relational DB and start searching for who always boosts who. You'll find patterns. Of course there's nothing inherently wrong with this, but at the end of the day I don't need to know what a dozen NY Times journos think of a NY Times oped which is clearly written in bad faith, pushing a false narrative about a particular news event.
Non-gameable ultimately means people who influence the results can have no monetary interest in the results.
That search engine is like a gold mine! I searched for "black people love us" in DDG and it was the first result, followed by an article written this year explaining how it came about and .. the web felt like such a smaller place back in 2002 and I just don't remember this at all.
Which makes me wonder about how newspapers and free to air tv kept the culture pretty shallow before exploding with the internet and now, possibly contracting again as our filter bubbles shrink? Just an errant though.
But so much novelty and interesting stuff at wiby.me - search for 'trump' and the first result is just surreal.
Social networks are becoming walled garden so it's kind of tough to crawl them. I mean look at Facebook and Instagram, the biggest social networks. They have robots.txt configured to disallow any crawlers.
Facebook also used have RSS feed for public pages and posts. Now they've not only removed that feature but also have heavy restrictions for third apps.
Facebook and the like made the decision to optimise for distraction instead of connection.
Services connected to the Fediverse, an alternative framework that focusses on connectivity, is slowing growing. It's only a matter of time before they are more successful than the walled gardens.
