A lesson in DNS (Quizlet debugging)

DNS (Domain Name System) is how the internet converts host names, such as example.com, into numeric IP addresses, such as 192.0.2.123 or 2001:db8::c0:ffee.  You can think of it like a phone book, which lets you find someone's phone number if you know their name.

An interesting bit of debugging came up recently, and a good example of how DNS works (or doesn't if you get it wrong!): It was reported that Quizlet was intermittently being "blocked".  It is pretty common that anything that doesn't work is reported as being "blocked" or "filtered", and we quickly determined that it wasn't actually being blocked, it was simply not working:

21/May/2026:13:22:30 +0100.329     32 xx:xx:xx:xx:xx:xx 10.x.x.x NONE_NONE_ABORTED/200 0 CONNECT jg7y.quizlet.com:443 jg7y.quizlet.com - HIER_NONE/- - "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/148.0.0.0 Safari/537.36 Edg/148.0.0.0" 3128 - ERR_DNS_FAIL WITH_SERVER

This is a line from the web proxy log, and the ERR_DNS_FAIL error shows that the DNS lookup failed.  But DNS lookups that we made seemed to be working, and it was clear that Quizlet was working some of the time - very odd!

We were initially concerned that this could be a bug in our product and we spent some time looking at the network traffic, before realising that DNS requests were being made to two servers that did not respond — 104.16.133.27 and 104.16.134.27.

Screenshot of Wireshark showing a network traffic capture, which shows DNS queries with no responses.
Network traffic, showing DNS queries to 104.16.133.27 and 104.16.134.27, with no responses from those servers.

It wasn't immediately obvious why the lookups were being made to those servers though — the authoritative servers listed for the quizlet.com domain appeared to be 173.245.59.32 and 173.245.58.37.  Eventually it became apparent that this was not our bug and that Quizlet's DNS records are completely broken.

Glue records

DNS (domain name system) is used to convert a host name (such as www.example.com, or jq7y.quizlet.com) into an IP address.  In order to do that, you first have to find the DNS server, or name server, which is authoritative for that host name.  There are a set of "root" name servers on the internet, and DNS lookups are made recursively, starting with those servers.  So to look up jg7y.quizlet.com:

  1. Lookup jg7y.quizlet.com on one of the root name servers.
  2. The root name server responds with "I'm not authoritative for jg7y.quizlet.com, but try these servers", together with a list of servers that are authoritative for the .com generic top level domain.
  3. Lookup jg7y.quizlet.com on one of the .com servers.
  4. The .com name server responds with "I'm not authoritative for jg7y.quizlet.com, but try these servers", together with a list of servers that are authoritative for the quizlet.com domain.
  5. Lookup jg7y.quizlet.com on one of the quizlet.com servers.
  6. The quizlet.com server responds with "I'm authoritative for jg7y.quizlet.com, and its IP address is aaa.bbb.ccc.ddd."  Note, this server is authoritative for the whole of quizlet.com, but there could have been a different server authoritative for jg7y.quizlet.com specifically.  If that were the case, we would have been referred again.  However, in this case the quizlet.com server is an authoritative server, so the process stops here and we've successfully translated jg7y.quizlet.com into a numeric IP address.

Quizlet have told the .com servers that the authoritative name servers are cns1.quizlet.com and cns2.quizlet.com, and that those servers are 173.245.58.37 and 173.245.59.32 respectively.  These records are a bit "special", in that even though they are under the quizlet.com domain, they are being returned by the servers that are authoritative for .com, rather than those authoritative for quizlet.com - they are known as "glue" records and look like this:

quizlet.com. 172800 IN NS cns1.quizlet.com.
quizlet.com. 172800 IN NS cns2.quizlet.com.
cns1.quizlet.com. 172800 IN A 173.245.58.37
cns2.quizlet.com. 172800 IN A 173.245.59.32

As you can see, this tells us that the authoritative name servers are cns1.quizlet.com and cns2.quizlet.com (which matches what is listed in Quizlet's whois record), and gives us the IP addresses of those name servers.

The authoritative name servers

The servers that the .com servers say are authoritative for the quizlet.com domain (173.245.58.37 and 173.245.59.32) do respond, and do indeed claim to be authoritative.  If we do a lookup against them, we get:

quizlet.com. 86400 IN NS ian.ns.cloudflare.com.
quizlet.com. 86400 IN NS lily.ns.cloudflare.com.
cns1.quizlet.com. 300 IN A 104.16.133.27
cns1.quizlet.com. 300 IN A 104.16.134.27
cns2.quizlet.com. 300 IN A 104.16.133.27
cns2.quizlet.com. 300 IN A 104.16.134.27

Err... oh dear.  These should all match the records that we got from the .com servers, and they... don't.  And those IP addresses for cns1 and cns2 - those are the ones that don't respond to DNS requests.  The IP addresses which ian.ns.cloudflare.com and lily.ns.cloudflare.com resolve to also don't match the glue records held by the .com name servers:

ian.ns.cloudflare.com. 86353 IN A 172.64.33.118
ian.ns.cloudflare.com. 86353 IN A 108.162.193.118
ian.ns.cloudflare.com. 86353 IN A 173.245.59.118
lily.ns.cloudflare.com. 86353 IN A 172.64.32.130
lily.ns.cloudflare.com. 86353 IN A 108.162.192.130
lily.ns.cloudflare.com. 86353 IN A 173.245.58.130

So:

  • The .com servers say that cns1.quizlet.com and cns2.quizlet.com are authoritative, but the authoritative name servers say that ian.ns.cloudflare.com and lily.ns.cloudflare.com are authoritative.  The records held by the .com servers and the authoritative servers should match, so Quizlet need to either change NS records held by the .com servers to ian.ns.cloudflare.com and lily.ns.cloudflare.com (this is done through the domain registrar), or change the NS records held by the authoritative servers to cns1.quizlet.com and cns2.quizlet.com.
  • The .com servers have different IP addresses for cns1.quizlet.com and cns2.quizlet.com than the authoritative servers.  Again, these records should match.
  • The IP addresses of the authoritative name servers listed in the glue records held by the .com servers don't match the IP addresses of the servers that the authoritative servers say are authoritative.
  • 104.16.133.27 and 104.16.134.27 are not responding to DNS requests, so cns1.quizlet.com and cns2.quizlet.com should not point to them at all, ever.
  • cns1.quizlet.com and cns2.quizlet.com should each point to the IP address of a separate name server (or separate set of name servers), but they have both been set to the same pair of servers.  The purpose of having multiple DNS servers is to add redundancy so that things keep working if one of them is down.  Although Quizlet's configuration doesn't reduce redundancy, it does potentially increase the impact of an outage.  This is because clients may try to contact both copies of the "down" server instead of falling back to only the working one.

The other authoritative name servers

The servers that 173.245.58.37 and 173.245.59.32 say are authoritative — ian.ns.cloudflare.com (172.64.33.118, 108.162.193.118 and 173.245.59.118), and lily.ns.cloudflare.com (172.64.32.130, 108.162.192.130, 173.245.58.130) — do also respond, also claim to be authoritative for the quizlet.com domain, and appear to return the same records as 173.245.58.37 and 173.245.59.32.  That's probably not too surprising since all of these name servers are Cloudflare's, so they probably all work off the same source data.  None the less, it would be wise to ensure that all of the records across all name servers (including the .com servers) are consistent!

Why this broke

As we saw, when looking up jg7y.quizlet.com afresh, this works as expected:

  1. Lookup jg7y.quizlet.com on one of the root name servers.
  2. The root name server responds with "I'm not authoritative for jg7y.quizlet.com, but try these servers", together with a list of servers that are authoritative for the .com generic top level domain.
  3. Lookup jg7y.quizlet.com on one of the .com servers.
  4. The .com name server responds with "I'm not authoritative for jg7y.quizlet.com, but try cns1.quizlet.com (173.245.58.37) and cns2.quizlet.com (173.245.59.32)."
  5. Lookup jg7y.quizlet.com on 173.245.58.37 or 173.245.59.32.
  6. The server responds with "I'm authoritative for jg7y.quizlet.com, and its IP address is aaa.bbb.ccc.ddd."

The problem comes when you've already got some of these records cached.  For example:

  1. Lookup jg7y.quizlet.com on one of the root name servers.
  2. The root name server responds with "I'm not authoritative for jg7y.quizlet.com, but try these servers", together with a list of servers that are authoritative for the .com generic top level domain.
  3. Lookup jg7y.quizlet.com on one of the .com servers.
  4. The .com name server responds with "I'm not authoritative for jg7y.quizlet.com, but try cns1.quizlet.com (173.245.58.37) and cns2.quizlet.com (173.245.59.32)."
  5. Although the .com name server provided us with the glue IP addresses of cns1 / cns2 (173.245.58.37 and 173.245.59.32), if we have previously received IP addresses for cns1 / cns2 from the authoritative servers, those are used in preference.  Authoritative records trump glue records, and they say the IP addresses for cns1 / cns2 are 104.16.133.27, 104.16.134.27, 104.16.133.27 and 104.16.134.27 (and yes, this list contains duplicates, because the authoritative records specified duplicates!).
  6. Lookup jg7y.quizlet.com on 104.16.133.27 or 104.16.134.27.
  7. The server does not respond and the DNS request fails.

This results in intermittent problems:

  • Some times we have no records cached, so everything proceeds from scratch and we end up contacting cns1.quizlet.com on 173.245.58.37, or cns2.quizlet.com on 173.245.59.32, which works.
  • Or we already have NS records for the quizlet.com domain cached and so try to contact ian.ns.cloudflare.com on 172.64.33.118, 108.162.193.118 or 173.245.59.118, or lily.ns.cloudflare.com on 172.64.32.130, 108.162.192.130, 173.245.58.130, which works.
  • Or we already have A records for cns1 / cns2 cached and we try to contact cns1.quizlet.com on 104.16.133.27, or cns2.quizlet.com on 104.16.134.27.  These servers do not respond to DNS queries, so the lookups fail.

Reporting the problem

We are very happy to investigate problems that our customers are having, often in quite significant depth.  When our investigations show that the problems are caused by bugs in our software, or by problems within the customer's network, we can usually engineer a fix quickly.  Unfortunately where the problem is with a third party's systems or software, we can't fix it ourselves and often end up with the perennial problem of trying to convince the vendor that there is a problem.  This is especially true when dealing with:

  • Big companies — Google, Microsoft and Apple basically don't care unless the problem is affecting tens of thousands of their customers.  At the end of the day, if a handful of people can't access their services, it is a drop in the ocean for the big vendors and really not worth their time fixing things.
  • Intermittent problems — if a vendor's first line support engineers can say "works for me", then that's often where things end.  Convincing them that there really is a problem which needs to be escalated is very hard, often impossible.

We sent a problem report to Quizlet, containing:

  • a description of the effects of this problem (intermittent DNS lookup failures);
  • a list of the numerous configuration errors described above; and
  • an explanation of how to fix them.

Unfortunately, although we included technical details and an explanation of why they were a problem, and the effects on the end users, we got back a reply just asking for "a bit more information".  They didn't ask for any specific pieces of information, and we had already said that users were intermittently unable to access the Quizlet website due to intermittent DNS lookup failures, so we're not really sure what additional information they need.

Hi there, 

Sorry to hear this is giving you trouble!
 
Thank you for reaching out and bringing these technical details to our attention. To help our network and engineering teams investigate this efficiently, could you please provide a bit more information on what they're experiencing? 
 
Once we have a clearer picture of the specific operational errors occurring on the user's end, we can pass these details straight to our engineering team to cross-reference with our current DNS infrastructure settings.
 
We look forward to your reply so we can get this thoroughly reviewed!

Best,
Quizlet Support Team

We again reiterated what the users were experiencing, the cause, provided a link to an independent web tool which would examine the domain and highlight the configuration errors, and we offered to help validate any changes they made to their DNS configuration.

Thank you for the detailed information and for the extensive troubleshooting your team has already completed.
 
To help us investigate this further, could you please have the affected users reach out to us directly with additional details about the issue they are experiencing? Specifically, it would be helpful if they could include:
 

  • The approximate date and time the issue occurred
  • Their location and ISP/network environment
  • Screenshots or exact error messages received
  • Any troubleshooting steps already attempted
 
This information will help our team review the reports more thoroughly and coordinate with the appropriate teams as needed.
 
We also appreciate you sharing the DNS analysis and recommendations. We’ll ensure the information is documented internally for further review.

Kind Regards,
Quizlet Support Team

Why do they need the affected users to "reach out" directly?  The customer has asked us to investigate, which we have done and provided a technical report to Quizlet with enough information for them to examine their own configuration and fix it.  There is basically nothing a user can do to help them at this point.  Despite us demonstrating that the problem is with Quizlet's configuration, it looks like they are assuming that they need to diagnose a client-side problem.  They would likely ask the user to turn their computer off and on again, clear cookies, all the usual stuff that will not fix the problem with Quizlet's DNS configuration.

We have responded to Quizlet's latest email, but ultimately it is not in the customer's interest to have their users' time wasted, and at this point we don't have much confidence in Quizlet fixing their DNS configuration.

After publishing this blog article, we received another reply from Quizlet, which seems to make clear that they aren't going to fix their broken config, so unfortunately there's not much more we can do:

Thank you for your patience here!
 
To further investigate, can you please share additional details and a short screen recording of the issue occurring live so we can better understand your experience and compare patterns internally? If you could provide the necessary details below, that would really help!
 

  • Their browser and browser version
  • Operating system/version
  • Whether the issue occurs across multiple browsers/devices/networks
  • A short screen recording showing the issue occurring, ideally including the URL bar and exact error message if possible

 
Once I have the necessary details, I should be able to help you with this.
 
Best,
Quizlet Support Team