- Gnip Is Hiring Software Developers: ... http://bit.ly/17byNZ July 3, 2009 salessushi (Sean Black)
- Blog - Gnip Is Hiring Software Developers http://is.gd/1mFN6 July 3, 2009 danquincy (Dan Quincy)
- RT @bfeld: boulder software developers looking for a job - gnip is hiring - http://fndry.gr/1R July 3, 2009 ruchitgarg (ruchitgarg)
- Feld Thoughts Feed: Gnip Is Hiring Software Developers http://tinyurl.com/r7u3tw Full http://tinyurl.com/muafvz July 3, 2009 StartupGuruNews (Brian Tsuchiya)
- Feld: Gnip Is Hiring Software Developers: ... http://tinyurl.com/muafvz July 3, 2009 prblogs (prblogs)
Lijit Search
Previous Entries
June 2009
May 2009
April 2009
March 2009
February 2009
January 2009
December 2008
November 2008
October 2008
September 2008
August 2008
July 2008
Blogroll
New Flickr Gnip Publisher available on http://api.gnip.com
March 27, 2009, 8:25am by Shane Pearson
This is one people have asked about a lot. We just pushed out a new publisher today for Flickr.
The new Flickr Publisher supports the Gnip TAG rule-type and allows people to easily integrate data from the Flickr API using the Gnip platform. In the near future we plan to add support for the Gnip ACTOR rule-type, so stay tuned. In the mean time it is very easy to define the tags that match your interests. Not sure what tags to use, just check out some of the most popular tags being used on Flickr.
Check it out on http://api.gnip.com and go use Gnip to integrate some data from Flickr!
Beta 2 Technical Update
March 26, 2009, 6:02pm by Jud Valeski
api.gnip.com has been up for a couple of weeks now and I thought I’d take a moment to update folks on how things are going.
One thing I’ll note early is that our original production environment remains our on-call monitored environment. While we’re honing api.gnip.com it’s monitoring does not “page” us when there are issues. Plenty of internal alarms sound when things crater, but the “on-call” person doesn’t get paged.
The three new major components in this Beta:
- New Schema
- Polling Infrastructure
- Data Normalization
New Schema
We’ve vastly improved our schema. Now Gnip offers normalized meta-data across Publishers/services. We’re striving to minimize the guesswork you have to go through in order to display/process activities in your application. We digested our community feedback, and out popped the new schema. So far so good. A few minor issues have been brought to our attention, and we’ll address those in the next rev of the schema.
Polling Infrastructure
This one’s been fun. Along the road to “delivering the web’s data,” we realized we were going to need to acquire some of the data ourselves. Some Publishers aren’t interested in PUSHing to Gnip yet, however demand for their data remains strong. As a result, we’ve embarked on our own polling infrastructure to ensure our Consumer’s needs are met (visit gnip.uservoice.com and tell us which services/data sources/Publishers you’d like to see Gnip support next). We’ve built a traditional job scheduling/queuing model to poll variabled URL endpoints for data. Replete with various back-off algorithms and rate-limiting, the system grabs activities out on the network, normalizes the results, and publishes them back into Gnip’s PubSub API.
We’re plowing through bugs and the challenges that come with job scheduling (starvation, ordering, prioritization, fairness) at scale. We long for some of the linear, relatively simple, polling architectures some of our partners have built. Building polling infrastructure for a few hundred thousand endpoints is one thing; building polling infrastructure for a few hundred million endpoints is quite another.
Until we resolve these issues you may experience intermittent results from Polled Publishers in Gnip. Things can go from working smoothly, to sporadic gaps in data. Bottom line is we have starvation issues we’re working to address. Bare with us.
Data Normalization
We’re vastly expanding the breadth of our Publisher offering by leveraging our Polling Infrastructure. For Polled Publishers we’ve built a layer that translates from arbitrary feed/API formats, into Gnip XML. While our core is written in Java, we leverage Python’s Universal Feed Parser to ingest XML, and map fields to Gnip XML. When UFP can’t handle things (even it has its limitations), we punt out to text processing with our own simple mapping language (with regex support). Our investment in this layer has highlighted two things really well: one, sadly even with the RSS/ATOM/XML’ization of data, severe challenges remain with data handling on the web (rampant inconsistencies remain). two, Gnip’s value proposition continues to grow. We handle the headache of this kind of mapping/parsing once, and many of you get the reap the rewards.
AWS/Ec2
We remain exclusively in “the Cloud.” Some thoughts on our 12 month experience with it…
- Cross instance latency remains high (say 5-10x higher than average non-Zen local interconnects). While we’ve been able to build a “real-time” system regardless, it’s certainly gotten in our way at times.
- Inbound packet transmission “into the Cloud” is slow. It’s not uncommon for sustained, large, uploads (say moving builds onto Ec2 instances) to average 70kB/s. This is lame and needs to be fixed by Amazon. Happy to pay more for throughput, just give me the option.
- Dedicated instances. One way to look at Ec2 performance is that the cheaper/smaller the instance, the more issues you’re going to have. Basically, Amazon heavily vslices and dices the cheaper boxes; the cheaper the box, the more loaded it can be with other apps. The larger instances (e.g. XL) equate to dedicated hardware just for you; your own CPUs, your own mem, your own NIC.
- I get asked “how many times have you lost instances?” all the time. The answer is “rarely enough that it’s never on anyone’s mind and I can’t remember the last time it happened.” Maybe 3 instances out of 50 continuously running over the course of 12 months.
- We continue to use the free version of RightScale to manage our deployments.
Please let us know what features you’d like to see going forward, or if you have questions. info@gnip.com
We made updates to the Gnip Developer Site today
March 25, 2009, 2:38pm by Shane Pearson
Next time you login to the Gnip Developer Site at http://api.gnip.com you will notice that we made some tweaks to the layout and design. The prior site was first pulled together right before the v1.0 went live last summer and it had stayed the same up until this update. (yeah, we should have updated it sooner….)
In the updated design we added more explanations of where to start for newer users, where to go for information and cleanup up the interface. We are planning a more complete update in the future and at that time will add more functionality so if there are wish lists out there please do let us know.
New Gnip.com is live
March 19, 2009, 5:39pm by Shane Pearson
We pushed out our new corporate website just now along with a whole new brand for the company. The new site is located at http://www.gnip.com.
We are putting the www.gnipcentral.com domain to rest and all of these URLs will begin to redirect to the correct gnip.com address as the DNS entries propogate.
While a new website and domain is a small thing in our overall plans we do hope that people find the new site more informational when it comes to the business value the Gnip platform provides and the great companies and customers we are working with every day.
Thanks to everyone in the community for helping us reach this milestone, which coincides with our 1 year anniversary this week. The last year was great, and we still have a lot of $h*t to pop!
Gnip Beta 2 launching today, new www.gnip.com coming
March 18, 2009, 2:59pm by Shane Pearson
Several new updates to Gnip websites are being released this week that will impact companies and people using the Gnip developer site and the Gnip corporate website.
1) We just started a maintenance on the demo system with a post on the forum and to the @gnipsupport Twitter account.
- The reason for this maintenance period is that we are moving to the next stage of this release and entering Beta 2. Once we come back online the Beta developer site that is currently hosted at demo.gnip.com will be available at a new location: http://api.gnip.com.
- We are migrating all the account information to the new site so if you have an existing account on the prior demo site it will be waiting for you at http://api.gnip.com.
- If your integration to the demo Gnip API is being done using our convenience libraries then you will need to manually update your hostname to https://api-v21.gnip.com .
- Documentation is available for the new Gnip version 2.1 API and schema at http://gnip.com/docs
- The new URL, http://api.gnip.com, will be the final destination and future home for the version 2.1 version of the Gnip platform after we conclude Beta 2. We are making this update now so that the integrations completed to api.gnip.com will contiue to work once we flip the switch to making this generally available sometime in the next few weeks.
- For people using the current version of the Gnip platform (v2.0) we are continuing to provide support and will do so through the Beta 2 period and 30 days after the general release of version 2.1.
2) We are also doing an update to our corporate website. Today the website is located at www.gnipcentral.com, and because we were able to acquire the gnip.com domain we are moving! Everyone going to any gnipcentral.com based URL will be redirected to the apporpriate gnip.com URL. So, look for the new website before the end of the week.
Here is a glimpse of the new home page for the curious…..
Gnip adds three new publishers Clipmarks, Dailymotion and deviantART
March 12, 2009, 1:22pm by Shane Pearson
Today we added three new publishers.
When you go to http://demo.gnip.com you will find Clipmarks (support for ACTOR rules), Dailymotion (support for ACTOR and TAG rules), and deviantART (support for ACTOR and TAG rules).
Remember to visit our crowd-sourcing website and vote for the services that you would like to see us integrate next.
New publishers in demo.gnip.com
March 4, 2009, 9:11am by Shane Pearson
With our schema now finalized and in beta at http://demo.gnip.com and the crowd-sourcing application launched to help us prioritize our publisher integration schedule the team is now heads down building out more publishers on the Gnip platform.
Today we put nine ten new publishers into demo.gnip.com. All of these are using the updated schema and provide support for notifications and activities with full-data. Have fun integrating some data!
- Delicious
- Fotolog
- Plurk
- Slideshare (added after original blog post)
- Stumbleupon
- Tumblr
- Twitter-search
- Vimeo
- Webshots
Thanks for those that voted at gnip.uservoice.com - week 1 results
February 14, 2009, 1:50pm by Shane Pearson
Thanks to everyone who has taken the time to visit the new community voting web application at http://gnip.uservoice.com to help us prioritize what services to integrate.
Here is a look at the top of the list:
Tell Gnip What Services To Integrate Next
February 11, 2009, 6:13pm by Shane Pearson
Now that we have the beta of the new Gnip schema up in the new demo system we are ready to roll out something we hope everyone in our partner and user community will be excited to participate.
Today we have turned on a new web application that we are hosting at UserVoice, who specializes in hosting customer feedback forums. The new Gnip Forum provides anyone and everyone the chance to tell us the social media and business services that we should integrate to and the priority of integration.
You heard us, we want you to tell us what to do! Just go to http://gnip.uservoice.com. What you will find is a list that as of right now includes 352 different services in priority order based on votes for those services. Just create an account and decide how to allocate your votes. Also, if there is a service we did not include feel free to add the service and tell us the URL and it will be added automatically.
The list also includes status labels that will allow people to track our progress. There are a few new services in progress right now and after we complete the current beta on the new schema we expect to be adding publishers at a good clip.
New Gnip schema live on demo.gnip.com
February 10, 2009, 8:53am by Shane Pearson
We have flipped the switch to allow people to start working with our new schema at http://demo.gnip.com. In addition to standing up the site with the updated schema we have moved over all the existing accounts from the current system, so your existing gnipcentral.com user and password also get you access to the demo system.
The following publishers are in the demo system and we plan to add more over the course of the next month during the beta period.
- Digg
- Identi.ca
- Seesmic
- Six Apart
We will be posting additional examples of how we mapped these social media services to the updated Gnip Schema in the Gnip Community and will link to those examples from the blog as well as point to them in our standard release newsletter that will go out later today.
Based on the feedback we have received there is a lot of interest in the enhanced metadata in the new schema that can be used to support additional types of URLs, multiple tags, geo data, and rich media. Now, go grab some data and do something cool with it.