Thursday, 15 December 2011

Yahoo Pipes uses multiple IP's

For my Twitter analysis program I need access to quite a lot of data from the twitter api (about 200 requests per visitor). Unfortunately the twitter api has an awful rate limit of 150 requests per hour per IP, so it is impossible to route this all through my server. Fortunately the api accepts jsonp requests, so I decided to offload fetching the data to the client.

The problem is that for most of my users 150 requests just won't cut it. After looking up a single person with 150 followers, the user has to wait for an hour. A caching solution is needed.

I decided to create a workflow like this for every request that would normally go through the twitter api:

  1. Ask my cache at waleson.com for the info. If successful, return.
  2. Try the twitter api via json p. If successful, send to waleson.com cache before returning.
  3. Ask my server to act as a proxy to the twitter api (Actually I've hooked up some others webservers that I happen to manage that my server uses as proxies).
However, I had played with Yahoo Pipes some months ago. (And I _really_ liked it.) Pipes accepts jsonp requests too. Perhaps it could be used to bypass the 150/hour/ip limit? Maybe not, maybe my Pipe is put on a single server and all requests will be made from that server's IP addresses?

Well, I set up a pipe, made it do a request to a logger on my server and did about 60 results. As it turns out, Pipes uses multiple IPs (this contains 7 unique IP's, there might be many more, but this should do for now):

83.87.80.102
87.248.125.49
87.248.125.49
217.12.1.125
87.248.125.48
217.146.191.18
217.12.1.124
217.146.191.19
217.146.191.19
87.248.125.49
217.12.1.124
217.12.1.124
217.12.1.125
87.248.125.49
87.248.125.48
87.248.125.48
87.248.125.48
217.12.1.125
217.146.191.19
217.146.191.19
217.12.1.124
87.248.125.48
217.12.1.124
217.12.1.124
217.12.1.125
87.248.125.48
217.12.1.124
217.12.1.124
87.248.125.48
217.146.191.18
217.12.1.125
217.12.1.124
217.146.191.19
87.248.125.49
217.12.1.125
217.12.1.124
217.12.1.125
217.146.191.18

217.12.1.125
217.12.1.125
87.248.125.49
87.248.125.48
217.146.191.19
217.12.1.125
217.146.191.18
87.248.125.49
87.248.125.49
217.12.1.125
87.248.125.49
217.12.1.125
217.12.1.125
217.12.1.125
217.12.1.124
217.146.191.19
217.12.1.124
217.146.191.19
217.146.191.19
217.146.191.19
217.12.1.124
217.146.191.19
217.12.1.124


User agent: Mozilla/5.0 (compatible; Yahoo Pipes 2.0; +http://developer.yahoo.com/yql/provider) Gecko/20090729 Firefox/3.5.2

So, I am currently putting Pipes somewhere in my workflow to act as another set of proxies to support even more users. Awesome!