Oracle Dyn outage

A number of large internet services use Oracle's Dyn service to host their domain's DNS.  This service has had an outage overnight (27 May 2022).  This problem was caused by a DNS misconfiguration and, although the underlying problem has now been resolved, it will take some time for the broken DNS records to be cleared from DNS caches around the world.

Oracle have said that "On-going impact has been improving as DNS cache clears via normal TTL propagation or DNS flushing. As of 04:30 UTC, DNS resolution response was largely recovered globally and will continue to improve as remaining TTL's expire and refresh cache."  However, it is not clear to us why they believe that DNS resolution had recovered by 04:30 UTC - the affected DNS records had a fairly long time-to-live (TTL) (we think they probably had a TTL of 1 day) and we expect that they will therefore be gradually cleared over the following 24 hours.  If you are struggling to access an important site through an Opendium system, please get in touch so that we can arrange to manually flush your system's DNS cache.


Oracle's report does not give a technical overview of the problem, so we thought we'd provide a little analysis for those interested.

In order to connect to a service on the internet, the domain name (we'll use in this example) needs to be converted into an IP address.  The process for doing this is known as recursive DNS resolution.  This is a slightly involved process, so we'll skip a few steps, but first of all to find the IP address of, we look to see which DNS servers are responsible for that domain, and in this case we find 6 of them:    IN    NS    IN    NS    IN    NS    IN    NS    IN    NS    IN    NS

The * ones are part of Oracle's Dyn service and in Amazon's case they also use UltraDNS to provide an independent backup.  While the Dyn service is broken, users should still be able to get to using UltraDNS, although connectivity may be a bit slow and flaky.  Not all internet services have an independent backup though.

Now we know the names of the servers responsible for the domain, we need to get their IP addresses, and this is done in the same way:    IN      NS    IN      NS    IN      NS    IN      NS    IN      NS    IN      NS    IN      NS    IN      NS    IN      NS    IN      NS    IN      NS

We can see that's DNS servers are largely hosted by Oracle, so we look for their IP addresses and get:    IN      NS    IN      NS    IN      NS    IN      NS    IN      NS    IN      NS

And here you may spot a problem - most of the DNS servers responsible for are under the * domain, and we can't find their IP addresses because we haven't yet found the IP addresses for the ( DNS servers which are responsible for the domain.  At this point the whole thing fails - our attempt to lookup ends in failure.  The problem was caused by Oracle setting up this loop.

The DNS servers responsible for have now been changed to remove the loop:    IN    NS    IN    NS    IN    NS    IN    NS    IN    NS    IN    NS

However, recursive DNS lookups are obviously quite a lot of work, so DNS servers cache the records for some time.  The amount of time the cached records remain valid for is known as the time-to-live (TTL).  Unfortunately, the broken records had quite a long TTL (probably 1 day), so the fixed records won't be used until the cached records in all of the DNS servers that you are using have expired.

Although it might appear that the recursive lookups would go on forever, needing to look up a new domain each time, but in reality you eventually find a "glue record" which gives you the IP address of a DNS server.  In fact, it is good practice for DNS providers to ensure that they register suitable glue records registered in order to reduce the number of lookups needed.  All of the DNS servers have glue records, so the full process, using the fixed records, would be:

Look to see which DNS servers are responsible for    IN    NS    IN    NS    IN    NS    IN    NS    IN    NS    IN    NS

Look to see which DNS server are responsible for    IN    NS    IN    NS    IN    NS    IN    NS    IN    NS    IN    NS

Look to see which DNS servers are responsible for    IN    NS    IN    NS    IN    NS    IN    NS    IN    NS    IN    NS    IN AAAA    2600:2000:2230::136    IN AAAA    2600:2000:2230::136    IN AAAA    2600:2000:2210::136    IN AAAA    2600:2000:2220::136    IN AAAA    2600:2000:2210::136    IN AAAA    2600:2000:2240::136    IN A    IN A    IN A    IN A    IN A    IN A

Notice that the response to this lookup (above) includes the glue records, which contain the IP addresses of the DNS servers.  We can now use one of those DNS servers to look up the * DNS servers:    IN    AAAA    2600:2000:2210::31    IN    AAAA    2600:2000:2220::31    IN    AAAA    2600:2000:2230::31    IN    AAAA    2600:2000:2240::31    IN    A    IN    A    IN    A    IN    A

And we can use those DNS servers to look up the domain itself:    IN    A    IN    A    IN    A