- And also testing gnip. Wondering how I can make it do... something. Not sure what yet exactly. thepatrick (thepatrick)
- high probability of Gnip completing the build-out of its core developer team this coming week. fingers crossed! jvaleski (Jud)
- Visited @SantaClaus site, post on your old favorite toys. Brought bck gr8t memories, Chrissy Dolls, Gnip-Gnop, Easy Bake Ovens. Whats yours? Maggie5565 (Maggie)
- is gnip a bus-as-a-service? pcalcado (Phillip Calcado)
- @howardlindzon Life is gooood, amigo. Gnip is on the precipice of some major feature releases and Zentact is about to make a big splash. bpm140 (Eric Marcoullier)
Lijit Search
Previous Entries
November 2008
October 2008
September 2008
August 2008
July 2008
Blogroll
What we are up to at Gnip
October 2, 2008, 10:32am by Shane Pearson
As the newest member of the Gnip team I have noticed that people are asking a lot of the same questions about what we are doing at Gnip and what are the ways people can use our services in their business.
What we do
Gnip provides an extensible messaging platform that allows for the publishing or subscribing of events and data from across the Internet, which makes data portability exponentially less painful and more automatic once it is set up. Because Gnip is being built as a platform of capabilities and not a web application the core services are instantly useful for multiple scenarios, including data producers, data consumers and any custom web applications. Gnip already is being used with many of the most popular Internet data sources, including Twitter, Delicious, Flickr, Digg, and Plaxo.
How to use Gnip
So, who is the target user of Gnip? It is a developer, as the platform is not a consumer-oriented web application, but a set of services meant to be used by a developer or an IT department for a set of core use cases.
- Data Consumers: You’ve built your pollers, let us tell you when and where to fire them. Avoid throttling and decrease latency from hours to seconds.
- Data Producers: Push your data to us and reduce API traffic by an order of magnitude while increasing distribution through aggregators.
- Custom web applications: You want to embed or publish content to be used in your own application or for a third-party application. Decide who, or what, you care about for any Publisher, give us an end-point, and we push the data to you so you can solve your business use cases, such as customer service websites, corporate websites, blogs, or any web application.
Get started now
By leveraging the Gnip APIs, developers can easily design reusable services, such as, push-based notifications, smart filters and data streams that can be used for all your web applications to make them better. Are you a developer? Give the new 2.0 version a try!
Gnip 2.0 is here…
September 30, 2008, 9:23pm by jud
Gnip 2.0 is in production! Gnip 1.0 is no longer accessible.
Open Wide: Full Data Is Here
We’re proud to introduce full data and expanded meta-data. You now have access to all the data associated with activities, as well as continued support for activity notifications. Full data provides you with additional meta-data about an activity, as well as the actual activity “body” (e.g. the comment text on a blog, or the text of a tweet). Gnip has been contributing to Diso in order to distill activity streams into a common format, and this release is our first attempt at getting this right.
Collections Are Dead; Long Live Filters!
Collections have been migrated to more feature rich “Filters.” You can now do more complex rule based filtering on data that comes through Gnip. For example, Publishers that support “tags” on their activities (e.g. delicious), can be filtered on specific tags/keywords). We currently only support the OR conditional, and not AND. AND introduces not only some interesting technical challenges, but also drives Gnip into some specific product directions we’re still evaluating.
Introducing XMPP
Data Producers can “publish” activities to Gnip via ATOM, REST, or XMPP. Data Consumers can get activities from Gnip via REST, and as of this release, XMPP. We’re just starting to dabble with XMPP for outbound activities, so please help test. Just give us your JID, and add our JID to your roster, and watch it flow. Note, this is not an implementation of XMPP PubSub; baby steps.
Migration Notes
- Existing implementations are now broken if you haven’t updated to the new API.
- Data Producers: please start pushing activities into Gnip using the new schema.
- Data Consumers
- Your accounts, and data, have been migrated to the new service.
- REST endpoints have changed, so please update (API doc here).
- Your collections have been migrated to the new Filter format as “actor” based Filters.
- The schema has changed, so please update (new schema).
- We’ve adopted the microblogging verb/noun of “notice” for microblogging service activity actions. e.g. a “tweet” is no longer a “tweet”, it’s a “notice.”
We’re taking part in the Boulder Job Fair — would you like a free trip to check out Boulder (and Gnip)?
September 23, 2008, 11:23am by eric
Boulder has the highest per capita programmers in the country. It also has the healthiest people on the planet (I can’t back this one up with stats, just anecdotal evidence of 60-year-old grandmothers zooming up and down the mountains). Basically, Boulder is the land of the mathlete, and it’s awesome!
A ton of local startups are competing for the best developers in Boulder and we’ve come to a common conclusion — we need to expand the pool of applicants. It’s time to give developers living in the Bay Area, Boston and Bentonville a taste of the Boulder lifestyle and simultaneously introduce them to some of the coolest companies Boulder has to offer.
Are you a badass developer? Do you code PHP or Jave or C++ in you sleep? Can you denormalize a database with your eyes closed or create elegant streams of CSS? We’d like to meet you. In fact, we’d like to fly you out all expense paid to Boulder for a couple of days to meet some awesome companies, including Gnip, to see if there’s a love connection. You’ll fly out on day one and spend time checking out the town, spend day two meeting with 20 killer tech companies and then have a third day to follow up with companies you like the best and then fly home. Not a bad way to spend the last week of October.
If you’d like to know more, check out the additional details at Boulder.Me and then click the button to apply.
We’re looking forward to meeting you in Boulder next month. We think you’ll dig the town as much as we do, and the companies are pretty rad, too.
Gnip 2.0 is coming…
September 23, 2008, 10:50am by eric
Friends,
Gnip’s first major update is right around the corner, and we want to ensure everyone has enough time to prepare for the upcoming changes. Gnip 1.0’s launch a few months ago provided a long overdue optimization to the way Data Consumers access information. We’ve saved the network billions of API calls/polls since our launch, providing Data Producers operational relief, and allowing Data Consumers to access multiple data sources in a “one stop shopping” manner.
As previously promised, Gnip 2.0 takes a giant step forward by adding full data to the activities that flow through the system. With the upcoming release, you can simply tell us who (or what) you care about, provide us an endpoint, and we’ll push those full activities to you in real time. Data aggregators are no longer required to build one-off integrations with each data source they are interested in. When inspiration strikes, you can have Gnip push relevant data to you and instead focus your efforts on the vital front-end work that makes-or-breaks a new service.
We’re currently testing the following improvements. If you are pushing data into the system or pulling it out, please make sure you check out the new API and get ready for the roll out Oct 1. Improvements include:
- XML schema format change to provide extended meta-data and full data payload. New schema is at http://groups.google.com/group/gnip-community/web/gnip2.xsd
- Collections are now called Filters and have expanded rule based functionality.
- In addition to having your Filter activities HTTP POSTed to you, you can provide a JID for delivery via XMPP.
We’re also pleased to announce that complete Public Timelines for several services, including Twitter, Delicious and Digg, will be available for testing at the same time. You’ll still need to abide by each service’s Terms of Service, but now the data will flow to an endpoint of your choosing. If you’d like to take part in the test, please email us with your company and contact info and we’ll reach out to you. We’re finalizing the billing structure for delivering full data to you and we’ll post more about that this week and next. Notifications, as Gnip provides today, will always be free.
We’ve posted a more technically in-depth announcement on the Community Boards; please head over there if you’d like to get the complete listing of changes and the new API document.
Cheers!
Eric
G[oogle]Data API & Standards
September 9, 2008, 1:47pm by jud
We’re in the throws of re-visioning gnip.xsd and that’s led to pondering Google’s Data API. If you haven’t noticed, at the interface level (not at a service level), there is a high degree of overlap between Gnip’s API, and Google’s Data API. We both chose REST as the primary interface, and data moves through as XML. Google decided to support both RSS and ATOM, while Gnip has constructed it’s own XML. From a system efficiency standpoint, our own boiled down schema makes sense. We’re a message aggregator, and message transmission and processing have to be done at scale (RSS & ATOM are heavy). That said, we’ll be offering ATOM and RSS based formats in the future, as our internal view of data doesn’t always match how folks want to consume it.
As for adopting the Google Data API, we have other priorities at the moment. A GData interface to Gnip as a service definitely has its appeal. I could see Gnip using it as the stepping stone to accessing Gnip activities as RSS/ATOM. Selfishly, Gnip could leverage GData’s convenience libs, and any time you can aggregate use of convenience libraries, everyone wins.
Software Evolution
September 5, 2008, 7:19pm by jud
Those of us who have been around for awhile constantly joke about how “I remember building that 10 years ago” everytime some big “new” trend emerges. It’s always a lesson in market readiness and timing for a given idea. The flurry around Google Chrome has rekindled the conversation around distributed apps. Most folks are tied up in the concept of a “new browser,” but Chrome is actually another crack at the age old “distrbuted/server-side application” problem; albeit an apparent good one. The real news in Chrome (I’ll avoid the V8 vs. TraceMonkey conversation for now) is native Google Gears support.
My favorite kind of technology is the kind that quietly gets built, then one day you wake up and it’s changed everything. Google Gears has that potential and if Chrome winds up with meaningful distribution (or Firefox adopts Gears) web apps as we know them will finally have mark-up-level access to local resources (read “offline functionality”). This kind of evolution is long overdue.
Another lacking component on the network is the age-old, CS101, notion of event-driven architectures. HTTP GET dominates web traffic, and poor ‘ol HTTP POST is rarely used. Publish and subscribe models are all but unused on the network today, and Gnip aims to change that. We see a world that is PUSH driven rather than PULL. The web has come a looooong way on GET, but apps are desperate for traditional flow paradigms such as local processor event loops. Our goal is to do this in a protocol agnostic manner (e.g. REST/HTTP POST, XMPP, perhaps some distributed queuing model)
Watching today’s web apps poll eachother to death is hard. With each new product that integrates serviceX, the latency of serviceX’s events propegating through the ecosystem degrades, and everyone loses. This is a broken model that if left unresolved, will drive our web apps back into the dark ages once all the web service endpoints are overburdened to the point of being uninteresting.
We’ve seen fabulous adoption of our API since launching a couple of months ago. We hope that more Data Producers and Data Consumers leverage it going forward.
Building A Platform: one feature at a time.
September 4, 2008, 10:56am by jud
As Gnip evolves, we are amazed, almost daily, at what folks build using our API. My favorite thing each morning is reading the search.twitter.com feed for Gnip. Seeing how we’re being used is so cool, and much of the time we’re used in ways we didn’t anticipate. When that starts happening, you realize you’re building a Platform with a capital ‘P’.
A recent product that caught our eye was “tweetrush” (out of Cork, Ireland); http://tweetrush.com/. They’re using Gnip to build statistical information/visualization around Twitter. Look for more cool stuff from these guys; the current product is just the beginning!
The vast majority of products leveraging Gnip, do so to make existing aspects of their product better (e.g. lower the latency of activity notification), but it’s really cool when we see entire products built up around our offering.
As the wheel turns.
Incremental Collection Updates
August 22, 2008, 7:02pm by jud
The Gnip API supports incremental collection updates. We’ve supported this for awhile, but we didn’t do a good job communicating when it came out. Several folks are taking advantage of it, but over the past few days it’s become clear not everyone knows the functionality exists. Please see “Collection Updates” in the API doc for details.
Gnip, Mahalo, and L.A.
August 16, 2008, 3:11pm by jud
Eric and I just spent the past few days in Los Angeles. We were lucky enough to be invited as one of two presenters to Mahalo’s first hosted tech meetup; Google (Chris Schalk and Kevin Marks) represented the other slot with Open Social. Attendance of 175 left standing room only, and Jason Calacanis, Mark Jeffrey, and crew were great hosts; thanks for having us guys! You can view the event here. Lots of great business ideas in the LA area, but focus seems to be around media (no surprise) rather than technology. 75% of the attendees I talked to after the meetup were heavy technologists however, so clearly folks want more tech representation; hopefully Mahalo’s regular tech meetups can help facilitate.
While we were out there, we spent time with a dozen or so companies/people about Gnip. The discussions ranged from revenue opportunities, and integration details, to our product roadmap, and how folks want it to look. We left with renewed focus on our coming feature set, and the need to hire more great people.
Everyone wants full data (aka activity/message payload) in activities, extended meta-data (beyond the current “type”, “guid”, “uid”, and “at” fields), and meta-data normalization. On the activity normalization front, checkout the work going on at DiSo, and contribute where you can. Data Consumers obviously want a broader range of Data Producers as well, and that’s where Gnip’s polling infrastructure will come into play. We’re cranking to get as much of this done by end of this calendar quarter as we can. If you think you can help us, please send us a note!
Delicious 2 is yummy.
August 8, 2008, 11:00am by jud
Gnip now has Delicious v2 data flowing through it. The delicous bookmarking data flowing through the system now includes bookmarking/tagging done via delicious plugins/API tools (e.g. toolbar buttons). Nice, clean and pure stream of data from delicous now. Enjoy!