Parse.ly Blog

Should be a good time.  See http://www.meetup.com/Hoboken-Tech-Meetup/ to join the meetup!

Tomorrow Parse.ly will be presenting at the NY Tech Meetup.  We’re part of the “university demo” segment, though we’re not actually university students anymore (if only!).  This is a particularly good time to for us to talk to the New York Tech community.  We have a few upcoming product offerings for developers, publishers, and individuals that we’re super excited about.

Primarily, our presentation will be about our API launch and what developers can do with Parse.ly’s personalized recommendation technology.  Developers of news/blog content mashups or online content sites can use our technology to offer Amazon/Netflix-style recommendations to their users.  Here is what the Parse.ly API does for you:

1) parses and cleans RSS/Atom feeds and other content sources in near-real-time, via an integration with PubSubHubbub (PuSH) technology

2) builds a full-text index of your content, as well as personalized “resonance profiles” for different users that can be trained and queried

3) delivers personalized recommendations (Amazon/Netflix-style) of content to users, that can be listed, searched, and filtered

Our whole value proposition is that, yes, you could build algorithms to do personalized recommendations yourself and in-house, but it’s hard. There’s a lot of infrastructure that goes along with it. You or your engineering team will spend months — not days — getting it right. So, why not just plug into our nice API instead?

Our API is a standard HTTP/JSON RESTful API, and we already have a Python binding, with more bindings on the way.  We also have an overview of our algorithms online in our developer docs.

We know there are lots of awesome ways our developer community can leverage this technology.  However, we want to break the ice, so here are a few ideas to get the creative juices flowing:

• an iPad application that creates an elegant, full-screen experience for browsing news content from across the web.  Think Pandora.com, but for news stories and other content that can be read online.
• an iPhone / Blackberry application for getting personalized content recommendations on the go.
• an Adobe Air or other desktop technology application that delivers content recommendations as desktop notifications integrated into the user’s operating system.
• Blogging platform plugins (e.g. Wordpress) that assist with content writing and editing based on current blog posts, future blog post drafts, and user interaction with existing content.
• Custom publishing applications that can produce beautiful, printable flowed layouts of online content, powered by personalized recommendations.
• Personalized versions of any of your favorite content sites on the web; for example, personalized versions of TechMeme, TweetMeme, TechCrunch, or other aggregators.

We’ll be collecting e-mail addresses and info for developers that want an API key to play with our tech.  We’ll automatically add interested people to the Parsely API Developer Google Group.  Interesting ideas and discussions should emerge there. Then, within the next couple weeks, we’ll send out API keys to those who signed up.

At the meetup we’ll also discuss our re-vamped Consumer Beta that will launch within the coming months.  When we launched our private beta last August, we wanted to release a minimal, productive reading interface that allows users to interact with as much content as they wished.  Since then, we’ve been curating feature requests and usage to plan for the next release.  We have excellent ideas about how to make the our web application the best reading interface on the web.  Expect to hear more about it soon.

Finally, Parse.ly is partnering with a number of high-traffic, original content sites on the web through our Parse.ly Publisher Platform, aka P3.  Within the next couple months, you’ll see Parse.ly powering content personalization features ranging from personalized e-mail solutions, to widgets, to full-on Netflix-style experiences.  We’ll be rolling this out with some top online publishers, and will let you know once these are live!

Mike Singleton of FourSquare recently wrote a blog post entitled, “Algorithms as a Service”:

I think there’s a market opportunity to crease an AAS (algorithms as a service) company which provides simple APIs to implementations of common algorithms… Algorithms as a service would give you development efficiency, problem scalability (access to CPU farms), and confidence in the results.

Andrew chimed in with this:

I think what you’ve identified is that some APIs are about getting data into and out of an existing system that sort of lives on its own — e.g., Twitter’s, FourSquare’s, Flickr’s.

Then, other APIs are about abstracting certain problems and simplifying them to a simple API call. These are “algorithms as a service”.

So, in this category I put things like OpenCalais.com (entity extraction algorithms) and SimpleGeo.com (geolocation algorithms). I also put my own startup, Parse.ly, in this category; see http://parse.ly/p3 and http://parse.ly/api. For Parse.ly, what we’re doing is simplifying the following painful steps:

1) parsing and cleaning RSS/Atom feeds and other content sources in near-real-time
2) building personalized “resonance profiles” for different users that can be trained and queried
3) delivering personalized recommendations (Amazon/Netflix-style) of content to users, that can be listed, searched, and filtered

Our whole value proposition is that, yes, you could build algorithms to do personalized recommendations yourself and in-house, but it’s hard. There’s a lot of infrastructure that goes along with it. Your engineering team will spend months — not days — getting it right. So, why not just plug into our nice API instead?

I don’t think it needs a new name — it’s just an evolution of APIs and SaaS given the growing needs of developers to build more complex, dynamic applications and their increasing willingness to license best-of-breed 3rd-party platforms to do so.

January was an exciting month for Parse.ly. At the end of 2009, we were heads-down, polishing our own “algorithms-as-a-service” offering. We aligned our development around a public launch of it at the SIIA Information Industry Summit in NYC, where we were invited to present. Sachin gave a great presentation; here’s what one blogger had to say about it:

Parse.ly, a semantic tool that recommends content, steers users towards content towards personalization and recommendation through their licensed content. When and how [do] personalization really happen? […] Parse.ly collects a little personal interest information from users, “listens” to their content habits and provides recommendations that can be embedded in any number of content applications. Market segmentation data and other demographics fall out of this information naturally. Parse.ly is available to publishers now for integration via their new P3 platform.

At the same time as launching the Parse.ly Publisher Platform (P3), we also put online our API docs and made it possible for you get an API key. Then, we started conversations with some great brands in online / digital publishing (household names, even) about using our platform. These conversations have been going really well — almost too well! These companies know how much more valuable their online properties would be if they were built around engaging, personalized recommendations in the Amazon/Netflix style. And they have a lot of ideas about how to use the data and recommendations P3 will give them. We’ve already started to mock up new user interfaces for our API to make the integration with publishers as smooth as possible.

We’re excited for this new direction for Parse.ly. We agree with Mike that there are opportunities all around us to simplify algorithmically-tough problems to simple and highly-usable APIs. This will not only make web developers more productive, but it will also make the websites we use daily more useful and powerful!

Our good friends at HiiDef just launched a new app that has been in beta for awhile, Flavors.me. This is an excellent tool that has a great, simple, and usable design.

What’s the value preposition of Flavors.me? It’s to unify your various “online identities” into a single, dynamic, automatically-updated, and elegant website.

From the article:

Flavors.me lets you take all that information and put it together in a single website to serve as your “online identity”. All your publicly shared information, aggregated in one place, and displayed beautifully. […] It’s this kind of simplicity, design sense, and user-centric approach that makes me love the web as a place to develop, deploy, and use software.

Check out Andrew’s full review over at his blog.

Sorry for the lack of posts recently, but we’ve been busy changing and improving Parse.ly for the better!

We did, though, get picked up by a couple popular blogs in the past few weeks. Here a few snippets from both ReadWriteWeb and ZDNet.

Bloggers, muckrakers and news fanatics, lend me your ears. It’s entirely possible that we’ve discovered one of the best approaches to media monitoring since RSS itself. My mother always said, “You’ll never get what you want unless you ask.” But with adaptive feed application Parse.ly, that simply isn’t true. Rather than forcing us to abandon our overflowing feed readers, Parse.ly records our preferences and learns to work with us.

ReadWriteWeb 11/11/09

I haven’t figured out a way to manage Google Reader. I tried using Fever, but it doesn’t find news that matters to me… and it cost $30.  Techmeme is my home page, but I think it needs an upgrade.  I would like a feed reader that saves favorite feeds for me, and finds other content that is similar and interesting.

A new product called Parse.ly caught my eye that makes content discovery a painless process.

ZDNet - The Web Life 11/24/09

Check out our press page for more articles written about Parse.ly.  We’ll update you soon about what we have in store for the future!

Hi Parse.ly fans.  Andrew here.  I just wanted to let you know that I presented Parse.ly at the NYC Search & Discovery Meetup on Thurs, Oct. 29.  The meetup is organized by Otis Gospodnetic (blog), who is one of the authors of Lucene in Action and the author of the forthcoming Solr in Action book.  It was graciously hosted at kgbweb (thanks for making that happen, Joe West!).

We make heavy use of Lucene and Solr on Parse.ly, so it was exciting to get an opportunity to present to a community of fellow technologists building systems with these excellent technologies.

Here is the abstract from the talk:

Parse.ly: Inside a modern RIA built with Solr
Andrew Montalenti
—-
Parse.ly is a rich, adaptive web application that discovers your unique interests to filter and prioritize content from countless news and blog sources on the web. This talk will introduce Parse.ly with a quick demo and then delve right into how the Parse.ly engineering team makes use of the Solr open source search engine. This will include discussion of initial design mistakes that were later revised and “real world issues” that were overcome in scaling a system that currently processes millions of articles per week. Finally, we will discuss the existing Solr and Python landscape, and how we at Parse.ly aim to help the Solr community with the open source release of high-quality, Pythonic components for doing common Solr tasks.

Otis has written about the talk, and the slides are online, as well.  Special thanks to my kickass Parse.ly colleague Didier for setting up our BitBucket repository and starting to tease the code out that is ready for the community.

Thanks also to everyone who attended, and if you have any questions about it, feel free to contact us.

Hi Parse.ly users,

Around 12:16am this morning, Parse.ly’s main database server that powers the Parse.ly news reading interface went down.  Unfortunately, our system administrator is on vacation in Greece at the moment — or, should we say, fortunately for him! :-)

I’ve successfully rolled our backups to the failover Parse.ly database server, however since our last complete backup happened before the start of the weekend, approximately 5% of our total users may be affected by some strange behavior until we can recover the data off the original server that turned off today.

• If you signed up for Parse.ly over the weekend or activated your account over the weekend, you may no longer be able to sign in with your credentials.  If this is the case, feel free to contact us, and we can look into getting you access to your account more quickly.


• If you were using Parse.ly over the weekend, you may notice that some of your star / read states may be out of date.

ALL USERS will notice notice a “gap” in your articles for the weekend days, since any articles our system processed over the weekend haven’t yet been reflected in this restored database.  Tomorrow, we are going to work hard to recover that last bit of data so that these 5% of directly affected users no longer have the strange experience, and so that the rest have no other issues.  But if you notice anything strange that isn’t covered in this blog post, please do contact us to let us know.

Finally, the engineering team here at Parse.ly apologizes for any interruption of service you may have experienced.  We are a young service and though we are careful to ensure our systems are backed up and that we have failover servers available for each of our production servers, we do not yet have automated failovers, aka high availability built into our free web application.  Tonight’s experience is one of our first lessons about how useful it would be to build out this system.  We were fortunate in that today’s outage happened at a relatively off-peak hour, but there is no guarantee this will be the case in the future.  Once our sysadmin is back in the states, we’re going to work on making Parse.ly’s disaster recovery more robust, so that users don’t have to suffer the downtime they did tonight.  Thanks for baring with us!  And we’ll update you on this blog once we’re totally done with the remaining data recovery tasks.

Sorry Parse.ly users, but one of our servers unexpectedly failed on us. As a result, we took Parse.ly down.  We’re working hard now roll our backups over to our failsafe and get Parse.ly back up and running.  We should back to normal in a few hours.  We’ll update this post when we’re all good to go.

Andrew’s a lifetime New Yorker. He grew up in the city and Long Island (and now lives in Astoria, Queens), and has always been a fan of the New York. I’ve been in NYC for the past eight years bouncing between Mnhattan and Brooklyn (I just moved to Ft. Greene this past weekend), and have grown to love the city

Well, this past week we had the opportunity — the privilege — to present Parse.ly (our baby) at NYC Demo Day, an event focused exclusively on New York City, early-stage startups.  You can check out some photos of Andrew and I presenting here.

The event was a blast, as evidenced by Andrew’s inexplicably happy face after it was over (see the photos to see what I mean).  Good luck to all the companies: Renthop, Trendsta, SECWatch, LegalRiver, Sensobi, Localytics, Seatgeek, Postling … and, of course, Parse.ly!

If you were trying to log into Parse.ly between 11pm-1am this Sunday, you may have noticed that it was intermittently down for maintenance.  Over the last several weeks, we’ve been working hard to roll out some new features, polish some rough edges, and improve our infrastructure after our launch last month.  Our first beta users have been amazing in providing us with detailed and specific feedback on what works and doesn’t work well within Parse.ly.  We’ve diligently addressed many of the issues raised by these users and rolled out a new version of Parse.ly this weekend.

So, what’s new in Parse.ly?

• Faster archiving and deleting: many users have complained that archiving and deleting items in Parse.ly was extremely slow.  This was a shortcoming of the beta release we pushed out in August, and we have now rectified this issue.  Archiving and deleting should be very speedy, and bulk archives/deletes are just as fast as single ones now.


• Keyboard shortcuts: we bill Parse.ly as the most productive way to read the news, but our initial beta users noticed that one productivity feature we are sorely lacking is keyboard shortcuts.  We have implemented an experimental cut of keyboard shortcuts in the latest release.  Simply click inside the article grid (under “Current Items”) and press the question mark key (?).  This will bring up a message box with a list of our keyboard shortcuts.  They are very similar to GMail’s; for example, e archives an item, and shift+i marks an item as read.


• Username casing problems: we had a silly bug where your username was case-sensitive, even though we use e-mail addresses for usernames.  This has been addressed and so now you can use any case you like for your username.


• Better De-duplication: we did include a deduplicator in our initial beta release, and we thought it was working pretty well: nearly 50% of the articles processed by our crawler were being deduped.  However, many users reported duplicates in their accounts, and when we investigated, we saw that there were many scenarios that fooled our deduper and resulted in an unpleasant reading experience.  In particular, resyndicated articles (e.g., those republished after a press release) often had similar titles/summaries but were not exactly the same.  So we have included a more aggressive deduper that makes use of fuzzy string matching and seems to work better in the testing we have done so far.


• Better sources: we are continuing to improve the breadth and depth of Parse.ly’s sources.  We have restructured our crawler so that it gets access to sources that weren’t available before, and also better delivers content based upon your interests.


• More frequent crawls: many users were reporting that crawls wouldn’t happen often enough for them, so they wouldn’t see articles published e.g. in their twitter feed right away in Parse.ly.  We have re-engineered our crawl infrastructure to be much more performant, which will mean fresher content inside Parse.ly for you.


• You can kill the Parse.ly share bar: when you click a link inside Parse.ly, we top-frame it to allow you to share it easily with your friends.  However, users rightfully complained that they wanted the ability to easily remove this bar from any page.  You spoke, we listened! :-)


• Password recovery: some users forgot their passwords and we simply forgot to implement a password recovery feature in our initial release.  It’s there now.


• User profile editing: users can now change their password from inside the UI.  This was an obvious feature just plain missing from the initial release.

We also made some improvements to our Interest Setup Wizard, but this will mostly affect users who first sign up for the system.  (You did remember to invite your friends to join Parse.ly, right? :-))  These include:

• Better performance when adding new interests: before, there was a bit of a lag when you entered in interests.  This should be fixed now.


• E-mail is sent to user upon initial setup: some of our users were rightfully confused when they entered in their interests and … nothing happened.  We aren’t activating accounts right away, but at the very least, we should send you an e-mail to let you know that your interest entry actually worked!  This is fixed now.

We are also working on some big changes within the Parse.ly engineering team to take our product to the next level.  We have partnered with our excellent hosting company, The New York NOC (NYNoc), to scale out our infrastructure and process more content than ever before.  We are also planning our future iterations where we hope to innovate and deliver more features to save you time and let you discover the content that best matches your interests.

A Personal Note from Andrew

As the lead developer for Parse.ly, I just want to say “Thanks!” to all our awesome users.  The thing that has impressed me most about Parse.ly’s users so far is how detailed, intelligent and thoughtful their feedback has been.  We want our company to be driven by your feedback, so please, do not hesitate to provide more of it on our Cog Tree Get Satisfaction Page, on Twitter, or directly via e-mail.

This last few weeks reminds me of something Jim Young, the founder of HotOrNot.com, said to us at DreamIt this summer: “when the site became successful, we were left having to figure out how to change the engine while the car was still running.”  It’s going to be a lot of work evolving Parse.ly even while providing the service to our existing users, but I’m looking forward to the challenge.

Thanks again to all our users for their feedback so far, and I hope you enjoy the new version of Parse.ly!  (Also, don’t forget to report any bugs or tell us what is working well for you.)  And if you haven’t signed up for Parse.ly yet, what are you waiting for?  Do it now at http://parse.ly