The Wayback Machine - https://web.archive.org/web/20150117060505/https://github.com/atmos/camo/pull/36
Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

1.4.0 Upgrade #36

Merged
merged 72 commits into from

4 participants

Corey Donohoe Joshua Peek Mat Schaffer Greg Ose
Corey Donohoe
Owner

This is a backport of @josh's work on our camo install for GitHub. The StreamLimiter stuff that was introduced in #34 failed to run reliably in production so we've reverted it. We've also removed the host blacklisting feature since it's not a concern anymore since we now run on heroku.

The two big improvements over the old versions are:

  • No longer forward client ip headers to third party systems. This avoids leaking information in things like README's to 3rd parties building advertising platforms.
  • Proxy SSL images instead of returning an http 301. Hosts that fail to verify ssl will 404 which is the same behavior that chrome and firefox adopt currently.

/cc @josh am I forgetting anything?

and others added some commits
Corey Donohoe handle chunked response, fixes #6 c0c83b3
Mat Schaffer matschaffer Use new octal syntax for CoffeeScript 1.3
Fixes #11.
d1c9a07
Corey Donohoe update server.js in production 45fa92a
Corey Donohoe bump version 68aa942
Corey Donohoe fix a busted merge c5d32b8
Corey Donohoe Merge branch 'public'
Conflicts:
	package.json
536a9ef
Corey Donohoe Merge remote-tracking branch 'atmos/master' 29f624d
Corey Donohoe Merge pull request #2 from github/accept_header
Empty Accept Header Fixup & Internal Redirect 404ing
7daf511
Corey Donohoe markdown 46a5a67
Corey Donohoe fuck those files 6cc17c4
Corey Donohoe try latest nodes 9fe95e6
Corey Donohoe Merge pull request #6 from github/almost-to-11
Try latest nodes, fixed #5
0ac3d49
Corey Donohoe Merge remote-tracking branch 'os/master' 093b380
Greg Ose gregose Ensure Location header on redirects before parsing
Fixes crash when HTTP redirect received without a Location header set.
6a992de
Greg Ose gregose 404 assert will detect the server crashing af468c4
Greg Ose gregose Revert "404 assert will detect the server crashing"
This reverts commit af468c4.
7834497
Corey Donohoe Merge pull request #7 from github/survive_redirect_with_no_location
Ensure Location header on redirects before parsing
dd7e832
Corey Donohoe Merge remote-tracking branch 'open/master'
Conflicts:
	Rakefile
2a9ddbd
Corey Donohoe Merge remote-tracking branch 'open/master' 03555d1
Corey Donohoe heroku supports the latest node now 24ab36f
Corey Donohoe Merge pull request #8 from github/bump-node-version
Heroku supports the latest node now
c1f2eb7
Corey Donohoe version bump for 1.1.3 5a9b26a
Corey Donohoe version bump for 1.1.3 75020a7
Corey Donohoe Merge remote-tracking branch 'public/master' 18e0896
Corey Donohoe Merge remote-tracking branch 'public/master'
Conflicts:
	package.json
	server.coffee
	server.js
35aa8a9
Corey Donohoe Merge pull request #9 from github/bump-version
Bump version 1.2.0
f567f33
Corey Donohoe follow https redirects to images 2194f9f
Corey Donohoe Merge remote-tracking branch 'kevinmehall/fixes-for-upstream' into fi…
…xes-for-upstream
f538197
Corey Donohoe let's call this 1.2.1 fdbb4a2
Corey Donohoe ignore node-version c633535
Corey Donohoe Merge remote-tracking branch 'origin/master' into fixes-for-upstream
Conflicts:
	server.js
149e9df
Corey Donohoe Merge branch 'updates' into fixes-for-upstream
Conflicts:
	server.js
5583805
Corey Donohoe Merge pull request #10 from github/fixes-for-upstream
Fixes for upstream
7b0e316
Corey Donohoe switch to 301 on https redirects d1de5cf
Corey Donohoe Merge branch 'patches-from-kevin'
Conflicts:
	server.coffee
	server.js
ce636e3
Corey Donohoe Revert "Merge branch 'patches-from-kevin'"
This reverts commit ce636e3, reversing
changes made to d1de5cf.
4123daa
Corey Donohoe don't forward headers along anymore d527ec7
Corey Donohoe Merge pull request #14 from github/no-forwarded-for
Don't forward headers along anymore
2f70b1e
Joshua Peek josh Add https image test cases b3aedf3
Joshua Peek josh Cleanup duplicate kill_server rake task 8a2a30e
Joshua Peek josh Spawn test server just for redirect test faa6d92
Joshua Peek josh More specific name for redirect server test file 92ceab8
Joshua Peek josh Use standard rackup 09d63bb
Joshua Peek josh yield server host to block 8690d49
Joshua Peek josh Test is a task 9f3e34f
Joshua Peek josh Merge branch 'server-tests-cleanup' into proxy-ssl 1c9050a
Joshua Peek josh Merge pull request #16 from github/server-tests-cleanup
Server tests cleanup
b355835
Joshua Peek josh Use http.request rather than http.createClient
node docs say createClient is deprecated
ab3af70
Joshua Peek josh Style, no empty parens 53e65d3
Joshua Peek josh Return 404 on client request errors b974618
Joshua Peek josh Add crash test server d461ea2
Joshua Peek josh Camel case b9555af
Joshua Peek josh Fix camelcase in js 8624bea
Joshua Peek josh Add positive proxy test case 63531f9
Joshua Peek josh Work around spawn issues 3555221
Joshua Peek josh Add camo test env b4855d9
Joshua Peek josh Enable test mode when camo key is unconfigured 0088fdb
Joshua Peek josh Remove special env test from readme 54108f7
Joshua Peek josh Merge pull request #17 from github/http-request
Use http.request rather than http.createClient
a5371d8
Joshua Peek josh Add https image test cases b5cefb4
Joshua Peek josh Merge branch 'master' into proxy-ssl 0ea8cc9
Joshua Peek josh Proxy https protocol requests 9b6f506
Joshua Peek josh Merge remote-tracking branch 'origin/rollback' into rollback-merge
Conflicts:
	server.coffee
	server.js
f15b4fe
Corey Donohoe bump the version so we know what we're serving 7f7c5a4
Corey Donohoe Merge pull request #19 from github/rollback-merge
Rollback
df93560
Joshua Peek josh Merge branch 'master' into proxy-ssl
Conflicts:
	server.coffee
	server.js
0f8891d
Joshua Peek josh Add missing protocol colon 796412c
Joshua Peek josh Fix protocol conditional 1f5c6a9
Joshua Peek josh Merge pull request #15 from github/proxy-ssl
HTTPS Proxying
59f3171
Corey Donohoe update license shit 7611f07
Corey Donohoe update version in package.json 6bb5a08
Corey Donohoe Merge remote-tracking branch 'origin/master' into 1.4.0-upgrade 4dd39b9
Joshua Peek

That sounds good @atmos.

Thanks again for the help.

Corey Donohoe atmos closed this
Corey Donohoe atmos merged commit ddb0965 into from
Corey Donohoe atmos deleted the branch
Jim Cummins jimthedev referenced this pull request in toopay/depending
Closed

How to get image on repo? #7

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Jun 28, 2012
  1. handle chunked response, fixes #6

    authored
  2. Mat Schaffer

    Use new octal syntax for CoffeeScript 1.3

    matschaffer authored committed
    Fixes #11.
  3. update server.js in production

    authored
  4. bump version

    authored
  5. fix a busted merge

    authored
Commits on Jul 27, 2012
  1. Merge branch 'public'

    authored
    Conflicts:
    	package.json
Commits on Jan 20, 2013
Commits on Jan 22, 2013
  1. Merge pull request #2 from github/accept_header

    authored
    Empty Accept Header Fixup & Internal Redirect 404ing
Commits on Jul 4, 2013
  1. markdown

    authored
  2. fuck those files

    authored
  3. try latest nodes

    authored
  4. Merge pull request #6 from github/almost-to-11

    authored
    Try latest nodes, fixed #5
Commits on Aug 8, 2013
  1. Greg Ose

    Ensure Location header on redirects before parsing

    gregose authored
    Fixes crash when HTTP redirect received without a Location header set.
  2. Greg Ose
  3. Greg Ose

    Revert "404 assert will detect the server crashing"

    gregose authored
    This reverts commit af468c4.
  4. Merge pull request #7 from github/survive_redirect_with_no_location

    authored
    Ensure Location header on redirects before parsing
Commits on Aug 9, 2013
  1. Merge remote-tracking branch 'open/master'

    authored
    Conflicts:
    	Rakefile
Commits on Sep 6, 2013
Commits on Oct 19, 2013
  1. Merge pull request #8 from github/bump-node-version

    authored
    Heroku supports the latest node now
  2. version bump for 1.1.3

    authored
  3. version bump for 1.1.3

    authored
Commits on Nov 16, 2013
  1. Merge remote-tracking branch 'public/master'

    authored
    Conflicts:
    	package.json
    	server.coffee
    	server.js
  2. Merge pull request #9 from github/bump-version

    authored
    Bump version 1.2.0
Commits on Nov 19, 2013
  1. follow https redirects to images

    authored
  2. let's call this 1.2.1

    authored
  3. ignore node-version

    authored
  4. Merge branch 'updates' into fixes-for-upstream

    authored
    Conflicts:
    	server.js
Commits on Nov 20, 2013
  1. Merge pull request #10 from github/fixes-for-upstream

    authored
    Fixes for upstream
Commits on Nov 21, 2013
  1. switch to 301 on https redirects

    authored
Commits on Nov 22, 2013
  1. Merge branch 'patches-from-kevin'

    authored
    Conflicts:
    	server.coffee
    	server.js
Commits on Nov 24, 2013
  1. Revert "Merge branch 'patches-from-kevin'"

    authored
    This reverts commit ce636e3, reversing
    changes made to d1de5cf.
Commits on Jan 8, 2014
  1. Merge pull request #14 from github/no-forwarded-for

    authored
    Don't forward headers along anymore
Commits on Jan 14, 2014
  1. Joshua Peek

    Add https image test cases

    josh authored
Commits on Jan 15, 2014
  1. Joshua Peek
  2. Joshua Peek
  3. Joshua Peek
  4. Joshua Peek

    Use standard rackup

    josh authored
  5. Joshua Peek

    yield server host to block

    josh authored
  6. Joshua Peek

    Test is a task

    josh authored
  7. Joshua Peek
Commits on Jan 16, 2014
  1. Joshua Peek

    Merge pull request #16 from github/server-tests-cleanup

    josh authored
    Server tests cleanup
  2. Joshua Peek

    Use http.request rather than http.createClient

    josh authored
    node docs say createClient is deprecated
  3. Joshua Peek

    Style, no empty parens

    josh authored
  4. Joshua Peek
  5. Joshua Peek

    Add crash test server

    josh authored
  6. Joshua Peek

    Camel case

    josh authored
  7. Joshua Peek

    Fix camelcase in js

    josh authored
  8. Joshua Peek

    Add positive proxy test case

    josh authored
  9. Joshua Peek

    Work around spawn issues

    josh authored
  10. Joshua Peek

    Add camo test env

    josh authored
  11. Joshua Peek
  12. Joshua Peek
  13. Joshua Peek

    Merge pull request #17 from github/http-request

    josh authored
    Use http.request rather than http.createClient
  14. Joshua Peek

    Add https image test cases

    josh authored
  15. Joshua Peek
  16. Joshua Peek

    Proxy https protocol requests

    josh authored
Commits on Jan 17, 2014
  1. Joshua Peek

    Merge remote-tracking branch 'origin/rollback' into rollback-merge

    josh authored
    Conflicts:
    	server.coffee
    	server.js
  2. Merge pull request #19 from github/rollback-merge

    authored
    Rollback
  3. Joshua Peek

    Merge branch 'master' into proxy-ssl

    josh authored
    Conflicts:
    	server.coffee
    	server.js
  4. Joshua Peek

    Add missing protocol colon

    josh authored
  5. Joshua Peek

    Fix protocol conditional

    josh authored
Commits on Jan 21, 2014
  1. Joshua Peek

    Merge pull request #15 from github/proxy-ssl

    josh authored
    HTTPS Proxying
  2. update license shit

    authored
  3. update version in package.json

    authored
This page is out of date. Refresh to see the latest.
1  .gitignore
View
@@ -1,3 +1,4 @@
node_modules
tmp/camouflage.pid
tmp/camo.pid
+.node-version
2  LICENSE → LICENSE.md
View
@@ -1,4 +1,4 @@
-Copyright (c) 2010 Corey Donohoe, Rick Olson
+Copyright (c) 2010-2014 Corey Donohoe, Rick Olson
Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the
1  README.md
View
@@ -18,7 +18,6 @@ Features
* Follow redirects to a configurable depth
* Proxy remote images with a content-type of `image/*`
* 404s for anything other than a 200, 301, 302, 303, 304 or 307 HTTP response
-* Disallows proxying to private IP ranges
At GitHub we render markdown and replace all of the `src` attributes on the `img` tags with the appropriate URL to hit the proxies. There's example code for creating URLs in [the tests](https://github.com/atmos/camo/blob/master/test/proxy_test.rb).
20 Rakefile
View
@@ -7,24 +7,12 @@ task :bundle do
system("bundle install --gemfile test.gemfile")
end
-namespace :test do
- desc "Start test server"
- task :server do |t|
- $SERVER_PID = Process.spawn("ruby test/proxy_test_server.rb")
- end
-
- desc "Run the tests against localhost"
- task :check do |t|
- system("BUNDLE_GEMFILE=test.gemfile bundle exec ruby test/proxy_test.rb")
- end
-
- desc "Kill test server"
- task :kill_server do |t|
- Process.kill(:QUIT, $SERVER_PID) && Process.wait
- end
+desc "Run the tests against localhost"
+task :test do
+ system("BUNDLE_GEMFILE=test.gemfile bundle exec ruby test/proxy_test.rb")
end
-task :default => [:build, :bundle, "test:server", "test:check", "test:kill_server"]
+task :default => [:build, :bundle, :test]
Dir["tasks/*.rake"].each do |f|
load f
2  package.json
View
@@ -1,6 +1,6 @@
{
"name": "camo",
- "version": "1.3.0",
+ "version": "1.4.0",
"dependencies": {
},
"engines": {
111 server.coffee
View
@@ -1,12 +1,12 @@
Fs = require 'fs'
-Dns = require 'dns'
Url = require 'url'
Http = require 'http'
+Https = require 'https'
Crypto = require 'crypto'
QueryString = require 'querystring'
port = parseInt process.env.PORT || 8081
-version = "1.3.0"
+version = "1.4.0"
shared_key = process.env.CAMO_KEY || '0x24FEEDFACEDEADBEEFCAFE'
max_redirects = process.env.CAMO_MAX_REDIRECTS || 4
camo_hostname = process.env.CAMO_HOSTNAME || "unknown"
@@ -24,8 +24,6 @@ error_log = (msg) ->
unless logging_enabled == "disabled"
console.error("[#{new Date().toISOString()}] #{msg}")
-RESTRICTED_IPS = /^((10\.)|(127\.)|(169\.254)|(192\.168)|(172\.((1[6-9])|(2[0-9])|(3[0-1]))))/
-
total_connections = 0
current_connections = 0
started_at = new Date
@@ -40,73 +38,30 @@ finish = (resp, str) ->
current_connections = 0 if current_connections < 1
resp.connection && resp.end str
-# A Transform Stream that limits the piped data to the specified length
-Stream = require('stream')
-class LimitStream extends Stream.Transform
- constructor: (length) ->
- super()
- @remaining = length
-
- _transform: (chunk, encoding, cb) ->
- if @remaining > 0
- if @remaining < chunk.length
- chunk = chunk.slice(0, @remaining)
- @push(chunk)
- @remaining -= chunk.length
- if @remaining <= 0
- @emit('length_limited')
- @end()
- cb()
-
- write: (chunk, encoding, cb) ->
- if @remaining > 0
- super
+process_url = (url, transferredHeaders, resp, remaining_redirects) ->
+ if url.host?
+ if url.protocol is 'https:'
+ Protocol = Https
+ else if url.protocol is 'http:'
+ Protocol = Http
else
- false
-
-process_url = (url, transferred_headers, resp, remaining_redirects) ->
- if !url.host?
- return four_oh_four(resp, "Invalid host", url)
-
- if url.protocol == 'https:'
- error_log("Redirecting https URL to origin: #{url.format()}")
- resp.writeHead 301, {'Location': url.format()}
- finish resp
- return
- else if url.protocol != 'http:'
- four_oh_four(resp, "Unknown protocol", url)
- return
-
- Dns.lookup url.hostname, (err, address, family) ->
- if err
- return four_oh_four(resp, "No host found: #{err}", url)
-
- if address.match(RESTRICTED_IPS)
- return four_oh_four(resp, "Hitting excluded IP", url)
-
- fetch_url address, url, transferred_headers, resp, remaining_redirects
+ four_oh_four(resp, "Unknown protocol", url)
+ return
- fetch_url = (ip_address, url, transferred_headers, resp, remaining_redirects) ->
- src = Http.createClient url.port || 80, url.hostname
-
- src.on 'error', (error) ->
- four_oh_four(resp, "Client Request error #{error.stack}", url)
-
- query_path = url.pathname
+ queryPath = url.pathname
if url.query?
- query_path += "?#{url.query}"
+ queryPath += "?#{url.query}"
- transferred_headers.host = url.host
+ transferredHeaders.host = url.host
+ debug_log transferredHeaders
- debug_log transferred_headers
+ requestOptions =
+ hostname: url.hostname
+ port: url.port
+ path: queryPath
+ headers: transferredHeaders
- srcReq = src.request 'GET', query_path, transferred_headers
-
- srcReq.setTimeout (socket_timeout * 1000), ()->
- srcReq.abort()
- four_oh_four resp, "Socket timeout", url
-
- srcReq.on 'response', (srcResp) ->
+ srcReq = Protocol.get requestOptions, (srcResp) ->
is_finished = true
debug_log srcResp.headers
@@ -148,15 +103,7 @@ process_url = (url, transferred_headers, resp, remaining_redirects) ->
debug_log newHeaders
resp.writeHead srcResp.statusCode, newHeaders
-
- limit = new LimitStream(content_length_limit)
- srcResp.pipe(limit)
- limit.pipe(resp)
-
- limit.on 'length_limited', ->
- srcResp.destroy()
- error_log("Killed connection at content_length_limit: #{url.format()}")
-
+ srcResp.pipe resp
when 301, 302, 303, 307
srcResp.destroy()
if remaining_redirects <= 0
@@ -171,7 +118,7 @@ process_url = (url, transferred_headers, resp, remaining_redirects) ->
newUrl.protocol = url.protocol
debug_log "Redirected to #{newUrl.format()}"
- process_url newUrl, transferred_headers, resp, remaining_redirects - 1
+ process_url newUrl, transferredHeaders, resp, remaining_redirects - 1
when 304
srcResp.destroy()
resp.writeHead srcResp.statusCode, newHeaders
@@ -179,10 +126,12 @@ process_url = (url, transferred_headers, resp, remaining_redirects) ->
srcResp.destroy()
four_oh_four(resp, "Origin responded with #{srcResp.statusCode}", url)
- srcReq.on 'error', ->
- finish resp
+ srcReq.setTimeout (socket_timeout * 1000), ->
+ srcReq.abort()
+ four_oh_four resp, "Socket timeout", url
- srcReq.end()
+ srcReq.on 'error', (error) ->
+ four_oh_four(resp, "Client Request error #{error.stack}", url)
resp.on 'close', ->
error_log("Request aborted")
@@ -191,6 +140,8 @@ process_url = (url, transferred_headers, resp, remaining_redirects) ->
resp.on 'error', (e) ->
error_log("Request error: #{e}")
srcReq.abort()
+ else
+ four_oh_four(resp, "No host found " + url.host, url)
# decode a string of two char hex digits
hexdec = (str) ->
@@ -216,7 +167,7 @@ server = Http.createServer (req, resp) ->
url = Url.parse req.url
user_agent = process.env.CAMO_HEADER_VIA or= "Camo Asset Proxy #{version}"
- transferred_headers =
+ transferredHeaders =
'Via' : user_agent
'User-Agent' : user_agent
'Accept' : req.headers.accept ? 'image/*'
@@ -253,7 +204,7 @@ server = Http.createServer (req, resp) ->
if hmac_digest == query_digest
url = Url.parse dest_url
- process_url url, transferred_headers, resp, max_redirects
+ process_url url, transferredHeaders, resp, max_redirects
else
four_oh_four(resp, "checksum mismatch #{hmac_digest}:#{query_digest}")
else
140 server.js
View
@@ -1,24 +1,22 @@
// Generated by CoffeeScript 1.6.3
(function() {
- var Crypto, Dns, Fs, Http, LimitStream, QueryString, RESTRICTED_IPS, Stream, Url, camo_hostname, content_length_limit, current_connections, debug_log, error_log, finish, four_oh_four, hexdec, logging_enabled, max_redirects, port, process_url, server, shared_key, socket_timeout, started_at, total_connections, version,
- __hasProp = {}.hasOwnProperty,
- __extends = function(child, parent) { for (var key in parent) { if (__hasProp.call(parent, key)) child[key] = parent[key]; } function ctor() { this.constructor = child; } ctor.prototype = parent.prototype; child.prototype = new ctor(); child.__super__ = parent.prototype; return child; };
+ var Crypto, Fs, Http, Https, QueryString, Url, camo_hostname, content_length_limit, current_connections, debug_log, error_log, finish, four_oh_four, hexdec, logging_enabled, max_redirects, port, process_url, server, shared_key, socket_timeout, started_at, total_connections, version;
Fs = require('fs');
- Dns = require('dns');
-
Url = require('url');
Http = require('http');
+ Https = require('https');
+
Crypto = require('crypto');
QueryString = require('querystring');
port = parseInt(process.env.PORT || 8081);
- version = "1.3.0";
+ version = "1.4.0";
shared_key = process.env.CAMO_KEY || '0x24FEEDFACEDEADBEEFCAFE';
@@ -46,8 +44,6 @@
}
};
- RESTRICTED_IPS = /^((10\.)|(127\.)|(169\.254)|(192\.168)|(172\.((1[6-9])|(2[0-9])|(3[0-1]))))/;
-
total_connections = 0;
current_connections = 0;
@@ -68,87 +64,31 @@
return resp.connection && resp.end(str);
};
- Stream = require('stream');
-
- LimitStream = (function(_super) {
- __extends(LimitStream, _super);
-
- function LimitStream(length) {
- LimitStream.__super__.constructor.call(this);
- this.remaining = length;
- }
-
- LimitStream.prototype._transform = function(chunk, encoding, cb) {
- if (this.remaining > 0) {
- if (this.remaining < chunk.length) {
- chunk = chunk.slice(0, this.remaining);
- }
- this.push(chunk);
- this.remaining -= chunk.length;
- if (this.remaining <= 0) {
- this.emit('length_limited');
- this.end();
- }
- }
- return cb();
- };
-
- LimitStream.prototype.write = function(chunk, encoding, cb) {
- if (this.remaining > 0) {
- return LimitStream.__super__.write.apply(this, arguments);
+ process_url = function(url, transferredHeaders, resp, remaining_redirects) {
+ var Protocol, queryPath, requestOptions, srcReq;
+ if (url.host != null) {
+ if (url.protocol === 'https:') {
+ Protocol = Https;
+ } else if (url.protocol === 'http:') {
+ Protocol = Http;
} else {
- return false;
- }
- };
-
- return LimitStream;
-
- })(Stream.Transform);
-
- process_url = function(url, transferred_headers, resp, remaining_redirects) {
- var fetch_url;
- if (url.host == null) {
- return four_oh_four(resp, "Invalid host", url);
- }
- if (url.protocol === 'https:') {
- error_log("Redirecting https URL to origin: " + (url.format()));
- resp.writeHead(301, {
- 'Location': url.format()
- });
- finish(resp);
- return;
- } else if (url.protocol !== 'http:') {
- four_oh_four(resp, "Unknown protocol", url);
- return;
- }
- Dns.lookup(url.hostname, function(err, address, family) {
- if (err) {
- return four_oh_four(resp, "No host found: " + err, url);
- }
- if (address.match(RESTRICTED_IPS)) {
- return four_oh_four(resp, "Hitting excluded IP", url);
+ four_oh_four(resp, "Unknown protocol", url);
+ return;
}
- return fetch_url(address, url, transferred_headers, resp, remaining_redirects);
- });
- return fetch_url = function(ip_address, url, transferred_headers, resp, remaining_redirects) {
- var query_path, src, srcReq;
- src = Http.createClient(url.port || 80, url.hostname);
- src.on('error', function(error) {
- return four_oh_four(resp, "Client Request error " + error.stack, url);
- });
- query_path = url.pathname;
+ queryPath = url.pathname;
if (url.query != null) {
- query_path += "?" + url.query;
+ queryPath += "?" + url.query;
}
- transferred_headers.host = url.host;
- debug_log(transferred_headers);
- srcReq = src.request('GET', query_path, transferred_headers);
- srcReq.setTimeout(socket_timeout * 1000, function() {
- srcReq.abort();
- return four_oh_four(resp, "Socket timeout", url);
- });
- srcReq.on('response', function(srcResp) {
- var content_length, is_finished, limit, newHeaders, newUrl;
+ transferredHeaders.host = url.host;
+ debug_log(transferredHeaders);
+ requestOptions = {
+ hostname: url.hostname,
+ port: url.port,
+ path: queryPath,
+ headers: transferredHeaders
+ };
+ srcReq = Protocol.get(requestOptions, function(srcResp) {
+ var content_length, is_finished, newHeaders, newUrl;
is_finished = true;
debug_log(srcResp.headers);
content_length = srcResp.headers['content-length'];
@@ -190,13 +130,7 @@
}
debug_log(newHeaders);
resp.writeHead(srcResp.statusCode, newHeaders);
- limit = new LimitStream(content_length_limit);
- srcResp.pipe(limit);
- limit.pipe(resp);
- return limit.on('length_limited', function() {
- srcResp.destroy();
- return error_log("Killed connection at content_length_limit: " + (url.format()));
- });
+ return srcResp.pipe(resp);
case 301:
case 302:
case 303:
@@ -214,7 +148,7 @@
newUrl.protocol = url.protocol;
}
debug_log("Redirected to " + (newUrl.format()));
- return process_url(newUrl, transferred_headers, resp, remaining_redirects - 1);
+ return process_url(newUrl, transferredHeaders, resp, remaining_redirects - 1);
}
break;
case 304:
@@ -226,10 +160,13 @@
}
}
});
- srcReq.on('error', function() {
- return finish(resp);
+ srcReq.setTimeout(socket_timeout * 1000, function() {
+ srcReq.abort();
+ return four_oh_four(resp, "Socket timeout", url);
+ });
+ srcReq.on('error', function(error) {
+ return four_oh_four(resp, "Client Request error " + error.stack, url);
});
- srcReq.end();
resp.on('close', function() {
error_log("Request aborted");
return srcReq.abort();
@@ -238,7 +175,9 @@
error_log("Request error: " + e);
return srcReq.abort();
});
- };
+ } else {
+ return four_oh_four(resp, "No host found " + url.host, url);
+ }
};
hexdec = function(str) {
@@ -253,7 +192,7 @@
};
server = Http.createServer(function(req, resp) {
- var dest_url, encoded_url, hmac, hmac_digest, query_digest, transferred_headers, url, url_type, user_agent, _base, _ref, _ref1;
+ var dest_url, encoded_url, hmac, hmac_digest, query_digest, transferredHeaders, url, url_type, user_agent, _base, _ref, _ref1;
if (req.method !== 'GET' || req.url === '/') {
resp.writeHead(200);
return resp.end('hwhat');
@@ -268,12 +207,11 @@
current_connections += 1;
url = Url.parse(req.url);
user_agent = (_base = process.env).CAMO_HEADER_VIA || (_base.CAMO_HEADER_VIA = "Camo Asset Proxy " + version);
- transferred_headers = {
+ transferredHeaders = {
'Via': user_agent,
'User-Agent': user_agent,
'Accept': (_ref = req.headers.accept) != null ? _ref : 'image/*',
'Accept-Encoding': req.headers['accept-encoding'],
- 'x-forwarded-for': req.headers['x-forwarded-for'],
'x-content-type-options': 'nosniff'
};
delete req.headers.cookie;
@@ -301,7 +239,7 @@
hmac_digest = hmac.digest('hex');
if (hmac_digest === query_digest) {
url = Url.parse(dest_url);
- return process_url(url, transferred_headers, resp, max_redirects);
+ return process_url(url, transferredHeaders, resp, max_redirects);
} else {
return four_oh_four(resp, "checksum mismatch " + hmac_digest + ":" + query_digest);
}
2  test.gemfile
View
@@ -1,4 +1,4 @@
source 'https://rubygems.org'
+gem 'rack'
gem 'rest-client', '~>1.3'
gem 'addressable', '~>2.3'
-gem 'thin'
8 test.gemfile.lock
View
@@ -2,21 +2,15 @@ GEM
remote: https://rubygems.org/
specs:
addressable (2.3.4)
- daemons (1.1.9)
- eventmachine (1.0.3)
mime-types (1.23)
rack (1.5.2)
rest-client (1.6.7)
mime-types (>= 1.16)
- thin (1.5.1)
- daemons (>= 1.0.9)
- eventmachine (>= 0.12.6)
- rack (>= 1.0.0)
PLATFORMS
ruby
DEPENDENCIES
addressable (~> 2.3)
+ rack
rest-client (~> 1.3)
- thin
76 test/proxy_test.rb
View
@@ -4,7 +4,6 @@
require 'openssl'
require 'rest_client'
require 'addressable/uri'
-require 'thin'
require 'test/unit'
@@ -14,10 +13,41 @@ def config
'host' => ENV['CAMO_HOST'] || "http://localhost:8081" }
end
+ def spawn_server(path)
+ port = 9292
+ config = "test/servers/#{path}.ru"
+ host = "localhost:#{port}"
+ pid = fork do
+ STDOUT.reopen "/dev/null"
+ STDERR.reopen "/dev/null"
+ exec "rackup", "--port", port.to_s, config
+ end
+ sleep 2
+ begin
+ yield host
+ ensure
+ Process.kill(:TERM, pid)
+ Process.wait(pid)
+ end
+ end
+
+ def test_proxy_localhost_test_server
+ spawn_server(:ok) do |host|
+ response = RestClient.get("http://#{host}/octocat.jpg")
+ assert_equal(200, response.code)
+
+ response = request("http://#{host}/octocat.jpg")
+ assert_equal(200, response.code)
+ end
+ end
+
def test_proxy_survives_redirect_without_location
- assert_raise RestClient::ResourceNotFound do
- request('http://localhost:9292')
+ spawn_server(:redirect_without_location) do |host|
+ assert_raise RestClient::ResourceNotFound do
+ request("http://#{host}")
+ end
end
+
response = request('http://media.ebaumsworld.com/picture/Mincemeat/Pimp.jpg')
assert_equal(200, response.code)
end
@@ -48,6 +78,16 @@ def test_proxy_valid_chunked_image_file
assert_nil(response.headers[:content_length])
end
+ def test_proxy_https_octocat
+ response = request('https://octodex.github.com/images/original.png')
+ assert_equal(200, response.code)
+ end
+
+ def test_proxy_https_gravatar
+ response = request('https://1.gravatar.com/avatar/a86224d72ce21cd9f5bee6784d4b06c7')
+ assert_equal(200, response.code)
+ end
+
def test_follows_redirects
response = request('http://cl.ly/1K0X2Y2F1P0o3z140p0d/boom-headshot.gif')
assert_equal(200, response.code)
@@ -64,6 +104,14 @@ def test_follows_redirects_with_path_only_location_headers
end
end
+ def test_404s_on_request_error
+ spawn_server(:crash_request) do |host|
+ assert_raise RestClient::ResourceNotFound do
+ request("http://#{host}/cats.png")
+ end
+ end
+ end
+
def test_404s_on_infinidirect
assert_raise RestClient::ResourceNotFound do
request('http://modeselektor.herokuapp.com/')
@@ -94,32 +142,12 @@ def test_404s_on_non_image_content_type
end
end
- def test_404s_on_10_0_ip_range
+ def test_404s_on_connect_timeout
assert_raise RestClient::ResourceNotFound do
request('http://10.0.0.1/foo.cgi')
end
end
- 16.upto(31) do |i|
- define_method :"test_404s_on_172_#{i}_ip_range" do
- assert_raise RestClient::ResourceNotFound do
- request("http://172.#{i}.0.1/foo.cgi")
- end
- end
- end
-
- def test_404s_on_169_254_ip_range
- assert_raise RestClient::ResourceNotFound do
- request('http://169.254.0.1/foo.cgi')
- end
- end
-
- def test_404s_on_192_168_ip_range
- assert_raise RestClient::ResourceNotFound do
- request('http://192.168.0.1/foo.cgi')
- end
- end
-
def test_404s_on_environmental_excludes
assert_raise RestClient::ResourceNotFound do
request('http://iphone.internal.example.org/foo.cgi')
3  test/servers/crash_request.ru
View
@@ -0,0 +1,3 @@
+run lambda { |env|
+ raise "b00m"
+}
BIN  test/servers/octocat.jpg
View
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
5 test/servers/ok.ru
View
@@ -0,0 +1,5 @@
+run lambda { |env|
+ path = File.expand_path('../octocat.jpg', __FILE__)
+ data = File.read(path)
+ [200, {'Content-Type' => 'image/jpg'}, [data]]
+}
6 test/proxy_test_server.rb → test/servers/redirect_without_location.ru
View
@@ -1,11 +1,7 @@
-require 'thin'
-
class ProxyTestServer
def call(env)
[302, {"Content-Type" => "image/foo"}, "test"]
end
end
-Thin::Server.start('127.0.0.1', 9292) do
- run ProxyTestServer.new
-end
+run ProxyTestServer.new
Something went wrong with that request. Please try again.