- And also testing gnip. Wondering how I can make it do... something. Not sure what yet exactly. thepatrick (thepatrick)
- high probability of Gnip completing the build-out of its core developer team this coming week. fingers crossed! jvaleski (Jud)
- Visited @SantaClaus site, post on your old favorite toys. Brought bck gr8t memories, Chrissy Dolls, Gnip-Gnop, Easy Bake Ovens. Whats yours? Maggie5565 (Maggie)
- is gnip a bus-as-a-service? pcalcado (Phillip Calcado)
- @howardlindzon Life is gooood, amigo. Gnip is on the precipice of some major feature releases and Zentact is about to make a big splash. bpm140 (Eric Marcoullier)
Lijit Search
Previous Entries
November 2008
October 2008
September 2008
August 2008
July 2008
Blogroll
Gnip Is Hiring Engineers
July 30, 2008, 9:31am by jud
Checkout our job posting here. If you think you fit the bill, let us know; we want to talk to you.
Please no recruiter or 3rd party inquiries.
Three (Six?) Week Software Retrospective
July 24, 2008, 7:28am by jud
I had to go back into older blog posts to remind myself when we launched; July 1st. It feels like we’ve been live since June 1st.
Looking Back
Things have gone incredibly well from an infrastructure standpoint. We’ve had to add/adjust some system monitoring parameters to accommodate the variety of Data Producers publishing into the system; different frequencies/volumes call for for specialized treatment. We weren’t expecting the rate, or volume, of Collection creation we wound up with. Within three hours of going live, we had enough Collections in the system to adversely impact node startup/sync times. We patiently tuned our data model, and tuned TerraCotta locks to get things back to normal. It’s looking like we’ll be in bed with TerraCotta for the long haul.
Amazon
I’m not sure I could be any more pleased with AWS. Our core service is heavily dependent on EC2, and that’s been running sans issues. We’re working on non-Amazon failover solutions that assure un-interrupted service even if all of EC2 dies. Our backups are S3 dependent so we had some behind the scenes issues last weekend when S3 was flaky; see my previous post on this issue. We haven’t had our day in the sun with outages, and I obviously hope we never do, but so far I’m walking around with a big “I <3 AWS” t-shirt on.
Other
On the convenience library front, we (Gnip + community) have made all of our code available on github. We’ve had tremendous community support and contribution on this front; so cool to see; thanks everyone!
Collections are by far the primary data access pattern (as opposed to raw public activity stream polling); not really a surprise.
Summize/Twitter has been a totally cool way to track ether discussion around Gnip. When we notice folks talking about Gnip, positive or negative, we can reach out in “real-time” and strike up a conversation.
That’s all for now.
Thanks to all the Data Producers and Consumers that have integrated with Gnip thus far!
New Gnip data consumer: Retaggr
July 22, 2008, 12:55pm by eric
Just got word that the very cool London-based service retagger has started working with Gnip to reduce the number of calls they make to data providers. Here’s what co-founder Nicholas Smit has to say:
Retaggr aggregates your online identity into an interactive embeddable business card. Contained within it is actual content from services like flickr, twitter and so on. We’re using Gnip to receive notifications about when our users publish data on these services, so we don’t have to poll them unnecessarily. This is great for efficiency, alleviates problems with API quotas, and helps us provide a consistent level of service to our users.
Welcome Nicholas and the Retaggr team!
New publishing partner: Muti
July 21, 2008, 12:57pm by eric
Muti is a social bookmarking site inspired by reddit and Digg but dedicated to content of interest to Africans or those interested in Africa.
Many thanks to Neville Newey and the Muti team for pushing notifications into Gnip.
Make sure you’re on the official publisher list
July 21, 2008, 12:17pm by eric
Once you start pushing notifications to Gnip, you should reach out to info@gnipcentral.com and let us know. In addition to adding your company to the official publisher list, we’ll blog about it here, add your logo to the home page and include you in a weekly announcement to the developers list.
In short, make us aware that you have integrated with us and we’ll ask everyone to start integrating with you.
a note about Gnip and S3’s weekend party
July 20, 2008, 9:24pm by jud
Amazon’s S3 outage over the weekend did not affect Gnip’s live service. Gnip uses S3 for system state archival/backup purposes, but the live data flow through Gnip was not affected as we keep it in local instances (memory/local disk). We weren’t able to backup data while S3 was down, but its outage was intermittent, so during online windows, we did our backups. Eventually the S3 outage was “over” and balance between local-storage and S3/remote storage was restored. At some magical point if S3 simply wasn’t coming back online, we’d move our backups to another service.
Building scalable, redundant, highly-available, systems is the next big game. It actually has been for decades, but now a larger web application audience is becoming accutely aware of its importance, and subsequently, how to accomplish it. At the end of the day, everything fails. The game becomes isolating the weak points, butressing the critical points of your service to ensure “instant” recovery from all the failures you can anticipate, and minimizing complete system setup/restart time in case everything craters and you have to scramble to come back online.
I hope Gnip never has it’s day in the searing outage sun, but we’re not naive.
Brush your teeth before bed, eat right, exercise, and eliminate your Single Points of Failure.
We got Twitter, yes we do…
July 18, 2008, 10:49am by eric
Hell. Yes.
They’re pushing to us. Now we’re pushing to you. Twitter notifications can be found here.
Coverage:
Why is this working?
July 17, 2008, 4:43pm by jud
Jay Ridgeway (switchabit, bit.ly) and I go back a ways. A handful of us spent a couple years of our life transitioning AOL away from it’s “old way” to the “new way” of doing (proprietary content hosting/infrastructure to modern web stuff). It was fun, and hard. Anyway… we were recently speculating as to why our stuff (gnip, bit.ly, switchabit) is getting so much traction (obviously we’re bias). Without going into some thesis-like blog post, the following paraphrased exchange took place:
Jud: the system is a mess right now
Jay: "we r making band aids and aspirin"
Truer words were never spoken. We’re in the midst of the larger system evolving through its “API phase.” APIs have sprouted up like weeds, and now folks are waking up to their lawns dying. Everyone has spent the last ~8 months talking about how to fix things, and our products are the first crack at tangible tools that are getting traction. To be fair, some big players have tossed “specs”, “frameworks”, and even more APIs into the mix in order to impact some change; helpful, but far from enough. We’re incrementally injecting framework fundamentals into the broader system; join us. Age old concepts applied to a growing ecosystem in need.
“Do, or do not. There is no ‘try.’” - Yoda
TheSocialWeb.tv gets a cookie, a big gooey chocolate chip cookie made with love
July 16, 2008, 12:47pm by eric
John McCrea, David Recordon and Joseph Smarr have knocked it out of the park with a brand new weekly video podcast where they discuss the complex world of data distribution. Since they say it better than I ever could, I’ll quote them directly:
With a revolving cast of characters, we’ll have some of the key technologists working on building the Social Web to explain what is going on; but this isn’t a show about technology. It’s about explaining what’s going on in the fight to make sure you have control of your data, your content, and your privacy — and the freedom to access your stuff from all over the Web.
In this first episode, they spend some time discussing Gnip, how it helps developers and how it will hopefully accelerate the move to true data portability standards. Plus, they break out a copy of Gnip Gnop. I mean, how cool is that!
That Twitter thing
July 10, 2008, 6:39pm by eric
Oh, crap, Eric’s gone and written another long post…
Since we publicly launched Gnip last week, we’ve been asked numerous times if we can integrate with Twitter or somehow help Twitter with the scaling issues they are facing. We can, but we depend on Twitter giving us access to their XMPP feed.
We are huge fans of Twitter so we’re patiently waiting for that access. In the mean time, the questions we’ve received have prompted us to explain two things: (1) How we would benefit Twitter and anyone who wants access to Twitter data and (2) Why - if you are a web service - it’s worth integrating now with Gnip rather than waiting either for (a) Gnip to integrate with Twitter or (b) you to get as popular as Twitter and have scale issues.
Let’s address the first issue: How we would benefit Twitter and anyone that wants to integrate with Twitter data.
Twitter has found that XMPP doesn’t scale for them and as a result, people are forced to poll their API *a lot* to get updates for their users. MyBlogLog has over 25,000 Twitter users that they throw against the Twitter API every 15 minutes. This results in nearly 2.5 million queries against the API every day, for maybe 250K updates. Now add millions of pings from Plaxo and SocialThing and Lijit and heaven forbid Yahoo starts beating up their API…
If Twitter starts pushing updates to us, via our dead simple API or Atom or their XMPP server, we can immediately reduce by an order of magnitude the number of requests that some very large sites are making against their API. At the same time, we reduce the latency between when someone Tweets and when it shows up on consuming sites like Plaxo. From 15 minutes or more to 60 seconds or less.
We expect that Twitter has their collective heads down and are working around the clock to buttress their infrastructure, and it’s unlikely that they’re going to do anything optional until that’s sorted out. Unfortunately, “integrate with Gnip” probably falls into the optional category. We expect, however, that at some point Twitter will start opening up their data to more partners once they feel like they have their arms around their infrastructure.
If you run a web service and integrate with Gnip today, you’ll automatically be able to integrate with Twitter data once they give us access. Presumably you won’t have to wait in line to get direct Twitter integration. In addition, you’ll have immediate access to all of the other data providers that we integrate with. Such as Delicious, Flickr, Magnolia, Get Satisfaction, Intense Debate and Six Apart. For example, only took Brightkite 15 minutes to integrate our API and start pushing data to our partners via us.
Now for the second topic. Why - if you are a web service - it’s worth integrating with Gnip now rather than waiting either for (a) Gnip to integrate with Twitter or (b) you to get as popular as Twitter and have scale issues.
All things considered, it’s best not to end up in Twitter’s position. They have a ton of passionate users (I’m one of them) who want reliable service and don’t have infinite patience. The old startup cliche of “these are problems we’d like to have” is bullshit.
You don’t want to be in the position where your business suddenly takes off and your infrastructure falls over because people are banging your APIs to death. You don’t want your most passionate users calling for mass exodus. It’s better to take a few minutes to start pushing notifications to Gnip now than when you’re doing 20-hour days rebooting servers.
You also don’t want to be in the position that your company takes off and you suddenly get throttled by an API provider. Nothing is worse than have to pull data sources because you’ve over-polled and the host decides to turn off the spigot. Start pulling notifications from Gnip and feel secure that you’re only asking for data when there’s something new.
I still use Twitter every day. Don’t try to kid me; I know you still do too. Let them get on with their work and rest assured that we’ll integrate with them the instant we get the okay from them.