Personalizing news on the long tail

Online news is a crowded field, and personalized news is becoming the Holy Grail for news publishers facing decreased revenues and outdated business models.

The challenge in personalizing the news: matching what people want with what they get. I believe that effectively personalizing the news on the long tail requires three approaches with their own unique sources of “power”: curation, search, and social participation. Relying solely on one of these engages its power and delivers its benefits, but also limits the effectiveness.

The benefits of advertising on the long tail

Editor’s note: For context about The Long Tail concept and its applications, you can watch the first part of Chris Anderson’s 2008 PARC Forum talk (delivered independently  from this post), visit his blog, and/or read his original article.

To news consumers, the appeal of personalized news is that they can keep up on the news that they care about, better manage their reading  time, and address their information overload. For online news producers, the appeal is increased consumer satisfaction and potentially greater revenues.

Advertising revenues for online mainstream news are limited, because general ads delivered to general audiences yield weak results to advertisers. For example, in one mainstream media outlet I was presented with ads for a play opening on Broadway, medical treatments for back pain, and real estate — none of which are relevant to me. Pricing for such ads is dropping as the number of sites competing to display generic ads is rapidly expanding.

Now, consider an online news site for amateur and professional woodworkers. Such sites can run highly targeted advertisements for woodworking tools, supplies, outlets, and training. Producers and distributors of these products do not generally advertise on general news pages, though such ads are run profitably by search companies in conjunction with user-supplied queries. In general, the more specific the interest, the more precisely the ads can be targeted.

To get the revenue advantage, a personal news system must not only target the news but also target the ads. The promise of personalized news: when the news is focused by topic, the personal news system gains a precise model of the reader’s immediate interest. Similar to web search engines, the personal news system can use this model to guide ad targeting without employing behavioral targeting.

Classic approaches to personalizing news on the long tail

News alert systems

News alert  systems ask users to provide a query that indicates what kinds of news they want. This approach treats personalization as search.

Advantages. Search is powered by computers collecting and identifying relevant information. These systems search for news no matter how specialized the topic is down the long tail.

Disadvantages. The very precision of queries inherently limits their potential for surprise and discovery. In struggling to get just the right query, news consumers potentially miss articles that express things with different words. Furthermore, news consumers want to find out about what’s happening — without anticipating and specifying what the breaking news will be. Finally, in pursuit of complete coverage, search systems consider a wide range of sources, which vary in their quality and authority.

Publisher curation

Traditional news systems serve topics at the head of the long tail well because they have good sources of articles and good curators choosing them. Their editors judge which articles are important. Major papers enable a degree of personalization by offering specialized feeds in particular areas — for example, technology, sports, business, China, and so on.

Advantages. The power of curation removes the requirement of users specifying exactly what they want. In effect, the news consumer says “give me the news that the editor says is important”. This approach addresses both source quality and news discovery.

Disadvantages. Publisher curation does not scale to the long tail. Individual publishers lack both the topical coverage and expertise needed to curate the long tail. And even if they have it, Chris Anderson notes:

…the gatekeepers “got it wrong every time.” Every month, Anderson…picks which story will be on the cover of Wired, and every single month some other story ends up being the most read.

Some traditional publishers also treat “personalization” as merely a selection from their feeds. Although this approach is straightforward, it fails to provide consumers with topical coverage deep in the tail — and falls short in mining potential advertising revenues from it.

Social approaches to personalizing news

The very idea of using social approaches to personalize news may sound like a contradiction in terms, since “personal” refers to one person and “social” refers to many people.

However, for my specialized or tail topics (“biofuels” for example), I have networks of professional colleagues and friends with a shared interest and desire to discuss them. Although general or head articles (“earthquake in Indonesia”) are of general interest, they don’t usually represent areas where I have enduring engagement or special expertise. Although head articles occasionally trigger social discussion when they are very important, articles on my specialized topics much more frequently lead to social interactions.

Social news

Social news sites such as Reddit or Digg enable groups of people to submit articles. The articles are assigned priorities according to reader votes.  (Social bookmarking sites such as delicious are near cousins to social news sites. Their primary purpose is to organize a personal set of browser bookmarks to webpages, and their secondary purpose is to share the bookmarks.)

Advantages. This approach relies on social participation both for collecting and ranking articles, and with enough participants can address topics on the long tail.

Disadvantages. However, by relying strictly on social participation — especially when that participation is in the form of voting — there is a risk that the loudest or biggest voice will drive curation, leading to heightening or suppressing controversial positions. Furthermore, curation helps when it imposes a coherent point of view and way of organizing that information — but large committees do not usually excel at such nuanced decisions. Consequently, the presentation of news on social news sites often appears rather haphazard as articles are listed by popularity but without topical coherence. Finally, social news by itself misses the opportunity for automatic and systematic collection of news. There is often a challenge getting an adequate stream of articles in narrow topics, especially when the participating groups are small and getting established.

Collaborative filtering

The idea that “birds of a feather” are useful for recommending particular news (and movies, books, music) is reflected in an approach called collaborative filtering first proposed by my colleagues at PARC in 1992. Collaborative filtering collects data about user preferences, matches users to established groups of people with similar interests, and makes recommendations based on articles preferred by those groups. Findory and DailyMe are examples of early and current news systems, respectively, that use collaborative filtering to deliver personalized news.

Advantages. Collaborative filtering addresses the problem of scale to the extent that like-minded groups of people can be found for topics on the long tail. If a person is matched to a group, new articles and topics of interest to the group can (in principle) be delivered to the members.

Disadvantages. In practice, affinity groups need to be explicitly identified. In most cases, collaborative filtering systems require that people specify their interests so they can be matched with others. Since people typically have several news interests, each interest has to be separately described. By itself, collaborative filtering provides no powerful means for organizing the information for a group.

Social indexing

Social indexing delivers personalized news by enabling people to subscribe to news organized by curators with whom they share an interest. In an online setting, when a person cannot find an index that matches their interests, they can start one and share it with their friends.

Social indexing engages all three sources of power: the power of search (“the tireless work of machines”), curation (“the hard work of the few”), and social participation (“the light work of the many”). News consumers can personalize their information diets by selecting the indexes and curators they follow.

Social indexing could be a game-changer for scaling curated personal news to the long tail. There are some near cousins to social indexing, such as socialmedian, which combines automatic collection with voting in social networks.  Other sites such as Twine employ semantic models of concepts developed by experts to relate and help organize articles. Although semantic models are like curators in that they recognize conceptual relationships, they typically fail to recognize how different topical organizations serve different human purposes.

Social indexing is in its early stages. We are currently in invitational beta for the early-stage Kiffets system being developed and incubated at PARC. New indexes are built as fast as our beta-users build them. The system itself also changes every few days. Some of the questions we ask ourselves as we evolve the system include: How do we help people with limited time find the news they want to follow and give them a sense that they are not missing anything important? How can the system engage the power of both personal choice and curation in deciding what news to emphasize? And many more.

Acknowledgments. Special thanks to Lance Good, Sanjay Mittal, Barbara Stefik, Prateek Sarkar, and Lawrence Lee for engaging conversations about these themes on social indexing while I was preparing this post.