Yahoo Pipes uses multiple IP's

First published on
For my Twitter analysis program I need access to quite a lot of data from the twitter api (about 200 requests per visitor). Unfortunately the twitter api has an awful rate limit of 150 requests per hour per IP, so it is impossible to route this all through my server. Fortunately the api accepts jsonp requests, so I decided to offload fetching the data to the client.

The problem is that for most of my users 150 requests just won't cut it. After looking up a single person with 150 followers, the user has to wait for an hour. A caching solution is needed.

I decided to create a workflow like this for every request that would normally go through the twitter api:

  1. Ask my cache at waleson.com for the info. If successful, return.
  2. Try the twitter api via json p. If successful, send to waleson.com cache before returning.
  3. Ask my server to act as a proxy to the twitter api (Actually I've hooked up some others webservers that I happen to manage that my server uses as proxies).
However, I had played with Yahoo Pipes some months ago. (And I _really_ liked it.) Pipes accepts jsonp requests too. Perhaps it could be used to bypass the 150/hour/ip limit? Maybe not, maybe my Pipe is put on a single server and all requests will be made from that server's IP addresses?

Well, I set up a pipe, made it do a request to a logger on my server and did about 60 results. As it turns out, Pipes uses multiple IPs (this contains 7 unique IP's, there might be many more, but this should do for now):

83.87.80.102
87.248.125.49
87.248.125.49
217.12.1.125
87.248.125.48
217.146.191.18
217.12.1.124
217.146.191.19
217.146.191.19
87.248.125.49
217.12.1.124
217.12.1.124
217.12.1.125
87.248.125.49
87.248.125.48
87.248.125.48
87.248.125.48
217.12.1.125
217.146.191.19
217.146.191.19
217.12.1.124
87.248.125.48
217.12.1.124
217.12.1.124
217.12.1.125
87.248.125.48
217.12.1.124
217.12.1.124
87.248.125.48
217.146.191.18
217.12.1.125
217.12.1.124
217.146.191.19
87.248.125.49
217.12.1.125
217.12.1.124
217.12.1.125
217.146.191.18
217.12.1.125
217.12.1.125
87.248.125.49
87.248.125.48
217.146.191.19
217.12.1.125
217.146.191.18
87.248.125.49
87.248.125.49
217.12.1.125
87.248.125.49
217.12.1.125
217.12.1.125
217.12.1.125
217.12.1.124
217.146.191.19
217.12.1.124
217.146.191.19
217.146.191.19
217.146.191.19
217.12.1.124
217.146.191.19
217.12.1.124


User agent: Mozilla/5.0 (compatible; Yahoo Pipes 2.0; +http://developer.yahoo.com/yql/provider) Gecko/20090729 Firefox/3.5.2

So, I am currently putting Pipes somewhere in my workflow to act as another set of proxies to support even more users. Awesome!

Comments

Post a Comment

Popular posts from this blog

"Security Is Our Top Priority" is BS

First published on
A couple of years ago I was asked to give a conference talk about software security. Well, actually I wasn't really asked, my company bought a sponsorship package with a speaker slot and I replied to the internal email asking for volunteers  🤣  Anyway, while preparing my talk, I realized a couple of important points about security that have not left my mind since: Security is limitless . You can always spend more effort to make things more secure. The same goes for quality, safety, employee happiness, etc. The needs of security are opposed to the needs of a convenient user experience . Improving one typically hurts the other. Now some organizations say "Security is our #1 priority". Really? You want to make something that has no limits your number one priority? I mean security is a good thing, but this seems a bit too simple? In fact, hollow marketing claims like that can make me a bit angry. In this post I'll help you understand what to make of statements like that...
Read more

AI programming tools should be added to the Joel Test

First published on
Here's a wake-up call to all CTOs: AI programming tools are getting freaking amazing and if you don't allow your teams to use them somehow, it will bite you in the ass in a couple of years. You will be slower and you will lose your best people. The infamous Joel Test is a list from the year 2000 of 12 things all great software companies do. Since then most companies have implemented Git and CI/CD, checking of three items, so we have some space left in the 2024 update ;) I believe "Do you allow your developers to use AI assisted development environments?" is a necessary addition. I get that you don't want your source code to end up on some OpenAI / Microsoft / Github server somewhere, sure, but find a way to use your own models or learn to live with it. Note that this is often not the same as "#9 - Do you use the best tools money can buy?" as blocking AI tools is about data security, not money. So why do I think developers need AI programming tools? I...
Read more

The long long tail of AI applications

First published on
Image
Last year multiple companies asked me for advice. " We are evaluating this AI powered product, but do you think it makes sense at all? It seems a bit niche and we think ChatGPT might make this entire thing obsolete soon. " Long tail - credit to Wikipedia My answer so far has always been: "No, bare LLMs are not going to compete with this product." I think that people are failing to understand the distinction between different classes of AI companies. I see it like this: Foundational AI - creating models like GPT-4 (text), Sora (video) etc. Applied AI - using existing models to create smart applications. Note that there are also companies not focusing on AI - they will start lagging behind, let's forget about them ;) There are orders of magnitude  (!)   more companies that will deal with Applied AI than Foundational AI, and they will be very busy for decades to come. Here's why: To get the most out of LLMs, you have to ask the right questions. LLMs have acc...
Read more