How to mirror and serve ajax.googleapis.com from your own box with nginx

Last libraries.txt update: 2015-10-26

If you're looking for an easy solution you may want to try "Decentraleyes" instead of what's documented on this page. It is a Firefox add-on that does local mirroring and replacement for not only google's CDN like this page but, "Google Hosted Libraries, Microsoft Ajax CDN, CDNJS (Cloudflare), jQuery CDN (MaxCDN), jsDelivr (MaxCDN), Yandex CDN, Baidu CDN, Sina Public Resources and UpYun Libraries."

This is how I set up a fairly simple local googleapis mirror using nginx and /etc/hosts. It works for most types of requests. I snagged the first list of files and their directories from Ryan McGeary's awesome ruby based googleapis-mirror which is at, https://github.com/rmm5t/googleapis-mirror .

I didn't want to run an extra webserver since I already had nginx running or I would've just used it. Unfortunately he stopped updating his list in 2014 so you probably want my hand updated libraries.txt which is also here in my dir structure too.

[comment on this post] Append "/@say/your message here" to the URL in the location bar and hit enter.

[webmention/pingback] Did you respond to this URL? What's your URL?

Mirroring and directory name tweaks.

In 2013 and 2014 I would just use a simple wget command like below. Unfortunately google has begun blocking wget requests so it no longer works. 

$ wget -x -i libraries.txt

I tried lots of spoofing before settling finally on the below workaround. Make sure to remove newlines and feel free to increase the speed by removing --wait=5.

wget -x -i libraries.txt --wait=5 --header='Host: ajax.googleapis.com' 
--header='User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:41.0) Gecko/20100101 Firefox/41.0' 
--header='Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' 
--header='Accept-Language: en-US,en;q=0.5' --header='Content-Type: application/x-www-form-urlencoded' -c

After it finishes move the created ./ajax subdirectory to someplace you want to keep it. Go into ./ajax/libs/. There's a list of softlinks you need to make in order to fullfill stupid GET requests like,

GET /ajax/libs/jquery/1.4/jquery.min.js?ver=3.4.2

See how it calls for 1.4, but in the dir stucture there is only,

ajax/libs/jquery/1.4.0/

So correct it by hand using the list below. You can throw it into a bash script or whatever if you trust me.

cd ajax/
cd libs/
cd jquery
ln -s 1.3.0/ 1.3
ln -s 1.4.0/ 1.4
ln -s 1.5.0/ 1.5
ln -s 1.6.0/ 1.6
ln -s 1.7.0/ 1.7
ln -s 1.8.0/ 1.8
ln -s 1.9.0/ 1.9
ln -s 1.10.0/ 1.10
ln -s 2.0.0/ 2.0
ln -s 2.0.0/ 2
ln -s 2.1.0/ 2.1
cd ..
cd jquerymobile/
ln -s 1.4.0/ 1.4
cd ..
cd webfont/
ln -s 1.0.0/ 1.0
cd ..
cd mootools/
ln -s 1.3.0/ 1.3
ln -s 1.4.0/ 1.4
ln -s 1.5.0/ 1.5
cd ..
cd jqueryui/
ln -s 1.6.0/ 1.6
ln -s 1.7.0/ 1.7
ln -s 1.8.0/ 1.8
ln -s 1.9.0/ 1.9
ln -s 1.10.0/ 1.10
ln -s 1.11.0/ 1.11
cd ..
cd ext-core/
ln -s 3.0.0/ 3.0
ln -s 3.0.0/ 3
cd ..
cd dojo/
ln -s 1.2.0/ 1.2
ln -s 1.3.0/ 1.3
ln -s 1.4.0/ 1.4
ln -s 1.5.0/ 1.5
ln -s 1.6.0/ 1.6
ln -s 1.7.0/ 1.7
ln -s 1.8.0/ 1.8
ln -s 1.9.0/ 1.9
ln -s 1.10.0/ 1.10
cd ..
cd chrome-frame/
ln -s 1.0.0/ 1.0
ln -s 1.0.0/ 1
Don't want to do all of that? Download the rar.

Here's ~ 37 MB rar file with all the folders, js/css, and fixes already done. It expands into two folders: ./ajax/libs/. I made this on October 27, 2015 and won't keep it up to date.

Webserver and system configuration.

If all the mirroring and folder tweaking is done then it's time to setup the nginx webserver config/location. I did it like,

location /ajax/ {
	# make sure to put the trailing "/" on the alias path.
	alias /home/superkuh/app_installs/googleapis-mirror/ajax/;
}

Now all that's left is to set up the /etc/hosts file so that ajax.googleapis.com resolves to 127.0.0.1

# Google
127.0.0.1	ajax.googleapis.com

Reload your nginx config if you haven't (sudo /etc/init.d/nginx reloads for me).

If all went well you have regular pages making requests to localhost for the javascript libraries hosted on ajax.googleapis.com To see if it's working and diagnose potential problems I watch the tail of the nginx logs.

$ watch -n 2 'tail -n 12 /var/log/nginx/access.log'
$ watch -n 5 'grep "/ajax/" < /var/log/nginx/access.log | tail -n 8'

To be honest that section of my hosts file actually looks like the below. Most of those are just opt-ed out with no replacement. It'll be nice to slowly replace them with safe, non-tracking versions. 

# Google
127.0.0.1	ajax.googleapis.com
127.0.0.1       www.google-analytics.com 
127.0.0.1	plusone.google.com
127.0.0.1 	ssl.google-analytics.com
127.0.0.1 	pagead.googlesyndication.com
127.0.0.1 	pagead2.googlesyndication.com

Interests

Other

Good Books

Type, "/@say/Your message here." after the end of any URL on my site and hit enter to leave a comment. You can view them here. An example would be, http://superkuh.com/rtlsdr.html/@say/Your message here.

Member of The Internet Defense League

Legal Bullshit

DMCA Requests

Terms of Use:

You may not access or use the site superkuh.com if you are under 90 years of age. If you do not agree then you must leave now.

The US Dept. of Justice has determined that violating a website's terms of service is a felony under CFAA 1030(a)2(c). Under this same law I can declare that you may only use one IP address to access this site; circumvention is a felony. Absurd, isn't it?

It is my policy to regularly delete server logs. I don't log at all for the tor onion service.

bellcop.

search. (via google)

door.