Software etc.

I transitioned the social analysis application to http://socialgraphanalysis.appspot.com/ last week. It turned out to be quite easy. What I came across was this (among others): When the application is being developed, database structures change often. In MySQL I could select my old data and convert it without much of a hassle. The kind of queries that you can run on App Engine are so limited that the old data and the new data cannot be properly distinguished. In the end, I added a version attribute to all my entities, so that I could later do a "SELECT  * WHERE version = '3'" to perform a transition.

For my Twitter analysis program I need access to quite a lot of data from the twitter api (about 200 requests per visitor). Unfortunately the twitter api has an awful rate limit of 150 requests per hour per IP, so it is impossible to route this all through my server. Fortunately the api accepts jsonp requests, so I decided to offload fetching the data to the client. The problem is that for most of my users 150 requests just won't cut it. After looking up a single person with 150 followers, the user has to wait for an hour. A caching solution is needed. I decided to create a workflow like this for every request that would normally go through the twitter api: Ask my cache at waleson.com for the info. If successful, return. Try the twitter api via json p. If successful, send to waleson.com cache before returning. Ask my server to act as a proxy to the twitter api (Actually I've hooked up some others webservers that I happen to manage that my server uses as proxies). Howe...

Over the last couple of months I've been spending some of my spare time on this project. A web-based interface to twitter analysis. It works thus: Generate a network. (consisting of all followers of @xxxx, @xxxx himself and all followers of @yyyy but not @yyyy himself) Download the data from twitter. (And cache the results to my server, twitter has a horrible 150 requests per hour policy) Let a grouping algorithm put similar people close together (in a web worker, hurray!) Let the user analyse the created network. In the image above I have selected the followers of some of my friends and I am currently checking my influence within their connections. Looks promising, when do you release? Within a week or two. I have some cool stuff that I want to throw in and I need to fix some caching/proxy issues. How did you make this? Javascript, jQuery, web-workers, intelligence (in my brain), and simple html/css