• Feeds

  • Ideas for creating a friendfeed like feed aggregator system

    I was invited to join a meeting about implement a friendfeed like feed system. Here are some ideas about requirement and architecture, which I typed on my BlackBerry during the meeting.

    1. Like the friendfeed, The product can import external RSS, so we separate the system into two parts. The rss crawl system and the feed pubsub system. The pubsub system has no responsbility to grab the feed from source. The feed itself only save feed summary and url.
    2. We decided to use the INBOX approach, which will push the published feed to be saved in all subscriber’s data table. More information of how this work can refer to Scaling a Microblogging Service – Part I.
    3. User’s homepage is an aggregation result. We choose to return a limited recent real-time date, no infinity pagination. But the user’s own feed(user’s profile page) may have a bigger date range.
    4. The unsubscibe logic have two choice, delete or keep the history data from one’s inbox. We decide to keep them.
    5. If the feed source had been deleted, do we need to delete all references in all subscriber’s inbox?  If need delete, each feed push to the pubsub system need to have a unique resource id. Another problem is after the source updated whether to publish a new feed or update the current feed?
    6. How to manage the group(QUN in Chinese) feed, deliver to all member’s inbox? Or share a group inbox?
    7. How to impl the feed comment logic, publish the comment to feed system or design a standalone comment system. We prefer to use a standalone comment system which doesn’t publish the comment back to the feed system.
    8. Every feed has a media type, such as text, video, image so the subscribe API can only retrieve a limited media type (text for mobile device). And a feed may have tags.
    9. The read/unread count is easy to implement. But the load is heavy. (QQ / QQzone may has such logic.)
    10. Need open API for 3rd party client(like twitter client), and RSS feed, may have OAuth integration.
    11. The storage may like friendfeed’s mysql schema (see How FriendFeed uses MySQL to store schema-less data) or use Amazon simpledb.
    12. May add support for XMPP Publish-Subscribe, or PEP(Personal Eventing Protocol) for pushing realtime time to users in the future.

    如想及时阅读Tim Yang的文章,可通过页面右上方扫码订阅最新更新。

    « | »

    8 Comments  »

    1. 不知道这个有没有做出成品?我自己写了个类似的东西,不过只是自己在用,单用户的,把google reader,推特,围脖,豆瓣的消息都放一起方便看这样

    2. Pretty section of contᥱnt.I just stumbled
      upon your website and in accession capіtal
      to assert that I get actually еnjoyed account yߋur blog posts.
      Anyway I will bee subscribing to your augment and even I
      achievement you access consistently quickly.

    3. Nishant Gaurav

      get the access to live Psn codes, and also avail free discounts from psn platforms. I love to play psn games so playstation plus codes I just thought to share my psn website.

    4. Nishant Gaurav

      nice forum to post, like wise I wanted to inform you all that paypal generator I got a website which offer free paypal money and many more related offers from Paypal. Paypal rocks.

    Leave a Comment