System Status: click here.

Gnip pushed a new platform release this week

December 12, 2008, 12:28pm by Shane Pearson

We just pushed out a new release this week that includes new publishers and capabilities. Here is a summary of the release highlights. Enjoy!

  • New YouTube publisher: Do you need an easy way to access, filter and integrate YouTube content to your web application or website? Gnip now provides a YouTube publisher so go create some new filters and start integrating YouTube based content.
  • New Flickr publisher: Our first Flickr publisher had some issues with data consistency and could almost be described as broken. We built a brand new Flickr publisher to provide better access to content from Flickr. Creating filters is a snap so go grab some Flickr content.
  • Now publisher information can be shared across accounts: When multiple developers are using Gnip to integrate web APIs and feeds it sometimes is useful to see other filters as examples. Sharing allows a user to see publisher activity and statistics, but does grant the ability to edit or delete.
  • New Data Producer Analytics Dashboard: If your company is pushing content through Gnip we understand it is important to see how, where and who is accessing the content using our platform and with this release we have added a web-based data producer analytics dashboard. This is a beta feature, not where we want it yet, and we have some incomplete data issues. However, we wanted to get something available and then iterate based on feedback. If you are a data producer let us know how to take this forward. The current version provides access to the complete list of filters created against a publisher and the information can be downloaded in XML or CSV format

Also, we have a few things we are working on for upcoming releases:

  • Gnip Polling: Our new Flickr and YouTube publishers both leverage our new Gnip Polling service, which we have started using internally for access to content that is not available via our push infrastructure. We plan to make this feature available externally to customers in the future, so stay tuned or contact us if you want to learn more.
  • User generated publishers from RSS Feeds: We are going to open up the system so anyone can create new publishers from RSS Feeds. This new feature makes it easy to access, filter and integrate tons of web based content.
  • Field level mapping on RSS feeds: A lot of times the field naming of RSS feeds across different endpoints does not map to the way the field is named in your company. This new feature will allow the editing and mapping at the individual field level to support normalization across multiple feeds.
  • Filter rule batch updates: When your filters start to get big adding lots of new rules can be a challenge. Based on direct customer feedback it will soon be possible to batch upload filter rules.

Tags: ,

   Read More »



Preview: Gnip Publisher Analytics

November 17, 2008, 10:09am by Shane Pearson

With everything going on here at Gnip we want to try and regularly preview some of the new features we are working on so people can send us feedback and plan ahead. One feature that we know a lot of people are interested in us delivering is usage and operational reporting and analytics. The reasons for adding an analytics dashboard are many and the primary reason is that we believe it will help companies and developers better understand the richness and variability of the data streams they care about.

Below is one example of the analytics features that we are planning to provide in the near future. This image shows the Digg Data Stream summary view with individual diggs, comments and submissions per second being streamed by the Gnip platform.

Figure: Gnip — Digg Data Stream Activity View

Obviously we could pivot on the summary view to show different types of details depending on any number of variables that Gnip partners and customers find interesting. If your company has specific requests for analytics and reporting please let us know.

Tags: , ,

   Read More »



Solution Spotlight: Strands now using Gnip

November 7, 2008, 11:54am by Shane Pearson

Strands is the newest company using the Gnip messaging platform for their web API data integration needs. Welcome Strands and thank you to Aaron for sharing what the team is doing!

Who is Strands?
Strands develops technologies to better understand people’s taste and help them discover things they like and didn’t know about. Strands has created a social recommendation engine that is able to provide real-time recommendations of products and services through computers, mobile phones and other Internet-connected devices. This enables users to discover new things, based on their online, offline and mobile activities. The Strands.com website helps people discover new things from other people. Visit http://www.strands.com to learn more.

Real-world results Strands says they are realizing from using Gnip
Strands.com is now able to give people updates faster and more reliably. In addition, Strands has seen reduced load on their system by not having to poll for updates on sites like Twitter, Flickr, Delicious, and Digg. Gnip allows Strands to receive push data from several of these sites, and at a minimum receive notifications when a user on these sites has made an update.

Tags: , ,

   Read More »



More examples of how companies are using Gnip

November 5, 2008, 12:36pm by Shane Pearson

We have noticed that we are interacting with two distinct groups of companies; those who instantly understand what Gnip does and those that struggle with what we do, so we decided to provide a few detailed real-world examples of the companies we are actively working with to provide data integration and messaging services today.

First, we are not an end-user facing social aggregation application. (We repeat this often.) We see a lot of people wanting to put Gnip in that bucket along with social content aggregators like FriendFeed, Plaxo and many others. These content aggregators are destination web sites that provide utility to end users by giving them flexibility to bring their social graph or part of their graph together in one place. Also, many of these services are now providing web APIs that allow people to use an alternative client to interact with their core services around status updates and conversations as well other features specific to the service.

Gnip is an infrastructure service and specifically we provide an extensible messaging system that allows companies to more easily access, filter and integrate data from web based APIs. While someone could use Gnip as a way to bring content into a personal social media client they want to write for a specific social aggregator it is not something we are focused. Below are the company use cases we are focused:

  1. Social content aggregators: One of the main reasons we started Gnip was to solve the problems being caused by the point-to-point integration issues that were springing up with the increase of user generated content and corresponding open web APIs. We believe that any developer who has written a poller once, twice, or to their nth API will tell you how unproductive it is to write and maintain this code. However, writing one-off pollers has become a necessary evil for many companies since the content aggregators need to provide access to as many external services as possible for their end users. Plaxo, who recently integrated to Gnip as a way to support their Plaxo Pulse feature is a perfect example, as are several other companies.
  2. Business specific applications: Another main reason we started Gnip was that we believe more and more companies are seeing the value of integrating business and social data as a way to add additional compelling value to their own applications. There are a very wide set of examples, such as how Eventvue uses Gnip as a way to integrate Twitter streams into their online conference community solution, and the companies we have talked to about how they can use Gnip to integrate web-based data to power everything from sales dashboards to customer service portals.
  3. Content producers: Today, Gnip offers value to content producers by providing developers an alternative tool that can be used to integrate to their web APIs. We are working with many producers, such as Digg, Delicious, Identi.ca, and Twitter, and plan to continue to grow the producers available aggressively. The benefits that producers see from working with Gnip include off-loading direct traffic to their web apis as well as providing another channel to make their content available. We are also working very hard to add new capabilities for producers, which includes plans to provide more detailed analytics on how their data is consumed and evaluating publishing features that could allow producers to define their own filters and target service endpoints and web sites where they want to push relevant data for their own business needs.
  4. Market and brand research companies: We are working with several companies that provide market research and brand analysis. These companies see Gnip as an easy way to aggregate social media data to be included in their brand and market analysis client services.

Hopefully this set of company profiles helps provide more context on the areas we are focused and the typical companies we are working with everyday. If your company does something that does not fit in these four areas and is using our services please send me a note.

Tags: , ,

   Read More »



Update on Gnip Platform and Twitter Publisher

October 25, 2008, 10:22am by Shane Pearson

We wanted to provide an update on how people are able to interact with Twitter via the Gnip platform since people creating new filters will notice a change in behavior as of today.

Gnip has been working with Twitter for several months. This has given developers another option for how they access events and data from the Twitter APIs. During this time Twitter was making their meta-data available to a limited set of partners like Gnip, as they state in their blog post from earlier this week, this meta-data access was on an experimental basis. In addition, Twitter announced in their blog post that they are working on a solution that will better allow people to use a service like Gnip for data-driven applications, but the new solution is not ready. At Gnip we plan to continue to work with Twitter as they bring this new solution to market.

In the interim people who create new filters on the Gnip platform will notice that these filters only provide access to the Gnip Notifications features, and not the Gnip Data Streams features. This change was put in place for new filter creation until the updated solution for data becomes available from Twitter. Existing filters that use the Gnip Data Streams features will continue to function normally.

At Gnip we see this as yet another example of how we are focusing on delivering a standard way for developers to access all the web’s data. We will continue to work on with our partners and the industry to provide reliable tools and services that allow you to get real-time access to the data you need when and where you need it. However, in some cases we in the industry will experience growing pains as the social Internet continues to expand and innovate.

Tags: , ,

   Read More »



Web APIs of all shapes and sizes

October 13, 2008, 1:58pm by Shane Pearson

Not all APIs have the same capabilities and therefore they provide different levels of access to events, procedures and data. Seems obvious, but you would not think that based on the normal questions we see from people. In fact we have found that APIs can be like a lot like apples and oranges. So, with the number of available APIs growing, at a rate that can be more than 60 per month we thought people would benefit from some simple way to think of API categorization based on how they expose events and data.

We work with a large variety of APIs from a variety of service providers and have noticed that most APIs fall into a few descriptive types based on how they expose events and data. The following are the main ways we are starting to look at APIs.

  • Fire hose or “full stream”. Identi.ca and Twitter are two examples, but Flickr also has a fire hose
  • User-based stream: These services do not directly expose a full stream, but instead give people a way to assemble an aggregate stream based on a list of users. Flickr again is a good example and there are many others.
  • Activity-based Tag-based and “other”: The main way to work with these services is usually some defined activity (tag, bookmark, etc) access to information or pre-defined streams based on feeds. An example would be Delicious, which allows multiple methods to access information by APIs and feeds.

This bi-frication in API types is something people should keep in mind when they want to access a service for some specific need. If you need to get events and data for a specific need then obviously the behavior of the API is going to impact your approach. And of course here at Gnip we are hard at work trying to provide consistent approaches across all types of APIs, so back to work!

Tags: , , ,

   Read More »



What we are up to at Gnip

October 2, 2008, 10:32am by Shane Pearson

As the newest member of the Gnip team I have noticed that people are asking a lot of the same questions about what we are doing at Gnip and what are the ways people can use our services in their business.

What we do

Gnip provides an extensible messaging platform that allows for the publishing or subscribing of events and data from across the Internet, which makes data portability exponentially less painful and more automatic once it is set up. Because Gnip is being built as a platform of capabilities and not a web application the core services are instantly useful for multiple scenarios, including data producers, data consumers and any custom web applications. Gnip already is being used with many of the most popular Internet data sources, including Twitter, Delicious, Flickr, Digg, and Plaxo.

How to use Gnip

So, who is the target user of Gnip? It is a developer, as the platform is not a consumer-oriented web application, but a set of services meant to be used by a developer or an IT department for a set of core use cases.

  • Data Consumers: You’ve built your pollers, let us tell you when and where to fire them. Avoid throttling and decrease latency from hours to seconds.
  • Data Producers: Push your data to us and reduce API traffic by an order of magnitude while increasing distribution through aggregators.
  • Custom web applications: You want to embed or publish content to be used in your own application or for a third-party application. Decide who, or what, you care about for any Publisher, give us an end-point, and we push the data to you so you can solve your business use cases, such as customer service websites, corporate websites, blogs, or any web application.

Get started now

By leveraging the Gnip APIs, developers can easily design reusable services, such as, push-based notifications, smart filters and data streams that can be used for all your web applications to make them better. Are you a developer? Give the new 2.0 version a try!

Tags: ,

   Read More »