Below is part of "dig" output.
The columns are <Machine-Name> <Lifetime of cached information> <IN (for query-type)> <A (for query-class)> <IP Address.
(There is no need to worry about the query type and query class here.)
ZA.akamaitech.NET. 3h30m6s IN A 18.104.22.168
ZB.akamaitech.NET. 3h30m6s IN A 22.214.171.124
ZC.akamaitech.NET. 3h30m6s IN A 126.96.36.199
ZD.akamaitech.NET. 3h30m2s IN A 188.8.131.52
ZE.akamaitech.NET. 3h30m2s IN A 184.108.40.206
ZF.akamaitech.NET. 1d44m5s IN A 220.127.116.11
ZG.akamaitech.NET. 23h24m25s IN A 18.104.22.168
ZH.akamaitech.NET. 23h24m25s IN A 22.214.171.124
zA.akamaitech.NET. 1d16m48s IN A 126.96.36.199
ZB.akamaitech.NET. 1d16m48s IN A 188.8.131.52
ZC.akamaitech.NET. 1d16m48s IN A 184.108.40.206
ZD.akamaitech.NET. 1d16m48s IN A 220.127.116.11
ZE.akamaitech.NET. 1d16m48s IN A 18.104.22.168
ZF.akamaitech.NET. 1d16m48s IN A 22.214.171.124
ZG.akamaitech.NET. 1d16m48s IN A 126.96.36.199
ZH.akamaitech.NET. 1d16m48s IN A 188.8.131.52
A couple of points are of interest here.
n1g.akamaitech.NET. 27m48s IN A 184.108.40.206
n2g.akamaitech.NET. 16m45s IN A 220.127.116.11
n7g.akamaitech.NET. 31m44s IN A 18.104.22.168
n3g.akamaitech.NET. 18m14s IN A 22.214.171.124
n4g.akamaitech.NET. 18m14s IN A 126.96.36.199
n8g.akamaitech.NET. 18m14s IN A 188.8.131.52
n5g.akamaitech.NET. 18m14s IN A 184.108.40.206
n0g.akamaitech.NET. 18m14s IN A 220.127.116.11
n6g.akamaitech.NET. 18m14s IN A 18.104.22.168
n2g.akamaitech.NET. 46m13s IN A 22.214.171.124
n3g.akamaitech.NET. 16m13s IN A 126.96.36.199
n8g.akamaitech.NET. 16m13s IN A 188.8.131.52
n4g.akamaitech.NET. 16m13s IN A 184.108.40.206
n5g.akamaitech.NET. 16m13s IN A 220.127.116.11
n6g.akamaitech.NET. 16m13s IN A 18.104.22.168
n0g.akamaitech.NET. 16m13s IN A 22.214.171.124
n1g.akamaitech.NET. 31m13s IN A 126.96.36.199
n7g.akamaitech.NET. 31m13s IN A 188.8.131.52
Note that all the IP addresses are not unique in a set. My take is that this makes future expansions easier with changed restricted to lesser places.
My belief is that this is THE step where all the Akamai DNS magic is. It will hand out different set of name server IPs to client contacting from different IP addresses. How it determines the nearest set of servers from IP addresses is anybody's guess (proprietary, but allegedly, they use BGP peering with ISPs that host the Akamai cluster, thus giving them a rough estimate of the distance of requesting user from that site - courtesy Tommy Larsen). Apart from wire latency other factors they claim to consider are load on their servers and Internet congestion. They also claim to be able to monitor their servers in real time (once per second). This means that gives out different sets of name server IPs at different times, which explains the short lifetime (30mins - 1hour) of this information. For instance following is part of dig output from the same UW machine at different times (or even different machines at same time - see below).
n2g.akamaitech.NET. 46m11s IN A 184.108.40.206
n3g.akamaitech.NET. 16m11s IN A 220.127.116.11
n8g.akamaitech.NET. 16m11s IN A 18.104.22.168
n4g.akamaitech.NET. 16m11s IN A 22.214.171.124
n5g.akamaitech.NET. 16m11s IN A 126.96.36.199
n6g.akamaitech.NET. 16m11s IN A 188.8.131.52
n0g.akamaitech.NET. 16m11s IN A 184.108.40.206
n1g.akamaitech.NET. 31m11s IN A 220.127.116.11
n7g.akamaitech.NET. 31m11s IN A 18.104.22.168
@UW2 (same machine at a different time)
n3g.akamaitech.NET. 21m28s IN A 22.214.171.124
n4g.akamaitech.NET. 21m28s IN A 126.96.36.199
n6g.akamaitech.NET. 21m28s IN A 188.8.131.52
n5g.akamaitech.NET. 21m28s IN A 184.108.40.206
n0g.akamaitech.NET. 21m28s IN A 220.127.116.11
n7g.akamaitech.NET. 36m28s IN A 18.104.22.168
n1g.akamaitech.NET. 36m28s IN A 22.214.171.124
n2g.akamaitech.NET. 21m28s IN A 126.96.36.199
n8g.akamaitech.NET. 21m28s IN A 188.8.131.52
This has another nice (for Akamai load balancing), or not so nice (for things like organization wide caches) side effect. Different machines could be downloading the same object from different Akamai servers at the same time, if those machines connect to different primary name servers within in the department (like our setup at UW-CSE).
a1388.g.akamaitech.net. 20S IN A 184.108.40.206
a1388.g.akamaitech.net. 20S IN A 220.127.116.11
Observe that the lifetime of this information is a mere 20 seconds (which corresponds to one or two web pages viewed) . So after this time period you will go back to the Akamai name server to get the IP addresses. What this means is that even if both the machines go down, it is highly unlikely that this will be seen by the client. (assuming that the Akamai name servers find this out and return a different IP in a failure scenario).
It is likely that all the machines don't host the content of all the
customers. Suppose that there are three servers (three different machines)
- A,B and C at a particular site (a site will typically have multiple machines).
And the customers are X, Y and Z. So A will host X,Y, B will host Y,Z and
C will host Z,X. This kind of an arrangement has a two-fold advantage
1) No server has to host all the customers' content. Easing the load on it and also making the content serving faster.
2) If any one server goes down, no customer is fully disconnected as there is another server (potentially more) with its tree.
A customer's content could be present at more than two servers at a site, but it makes sense to return the same two machines (till they are up) because of file caching, the object does not have to retrieved from the disk most of the times.
Another observed feature is that all the name servers in the set (n[0-9]g.akamaitech.net) returned in above step, give the same two IP addresses for a queried server (like a1388.g.akamaitech.net). This could mean that the configuration of all the servers in a set is identical and multiple servers are there just for sharing the load.
Name server differences
An artifact of the above exploration is the observation that different versions of named might be running in the department (@UW-CSE).
While 18.104.22.168 (bs4) always returns the two IP addresses in different order (to get some sort of load balancing), 22.214.171.124 (bs1) does no such thing.
And the one at MIT (126.96.36.199) seems to be returning the IP addresses in a random order.