Archive for October, 2007

Leopard DNS Issues (and work-around)

Shortly after Leopard was released we began receiving a few reports of issues, typically a spinning color wheel on startup, that affected a small number of users. We were unable to replicate the problem in-house and eventually determined it was something specific to the user’s network connection (since they could often get it to work on the same machine from work but not home or vise-versa). With some additional help and debugging from affected users we were able to track the problem down to changes in DNS resolution in Leopard. We’re documenting the problem here to help Leopard users who may be having problems with other applications, as well as application developers or ISPs who may start getting complaints.

Note: the rest of this post is technical info. For users on Leopard having trouble with Jungle Disk, you can download the latest Beta version for a fix. This problem only affects some users with old or misconfigured DNS servers - most Leopard users should be fine already. We’ll be releasing a non-beta version shortly incorporating the change.

Summary
The DNS resolver in Leopard has been changed to first attempt SRV requests for lookups initiated by the getaddrinfo() function. If the user’s DNS server drops these requests the DNS lookup may take an extended period of time to complete (30 seconds to several minutes) as Leopard tries different domain requests and eventually falls back to making an A record request. This can result in application freezes or timeouts, as was occurring with Jungle Disk.

Details
There are two primary BSD APIs for domain name resolution - gethostbyname() and getaddrinfo(). getaddrinfo() is the more robust API with better support for things like IPV6. Jungle Disk uses libcurl for HTTP access (as do many other apps), which by default uses getaddrinfo() when compiled with IPV6 support.

On OSX 10.4 (Tiger) both APIs perform similarly - they do an A record lookup of the provided domain. On 10.5 (Leopard) the getaddrinfo() API attempts an SRV lookup first when provided with a well-known port. For example, when performing a lookup for “s3.amazonaws.com” on port 80, you will first see requests for:

  • SRV _http._tcp.s3.amazonaws.com
  • SRV _http._tcp.s3.amazonaws.com.yourlocaldomain.com

Depending on your DNS server, you will get one of three possible responses:

  • The correct SRV response (in this case a CNAME record for s3-directional-w.amazonaws.com)
  • NXDOMAIN (if your server does not support or understand SRV requests)
  • No response at all

In the second case, Leopard will fall back to doing a A record query after the SRV requests fail and although there is a slight delay it is not generally visible to the user. In the last case Leopard will make several retries of the query over a period of several minutes, finally falling back to an A record query. During this retry period the application will appear frozen or unresponsive (depending on what thread the lookup occurs on).

It’s important to note that this problem does not appear to be a bug in Leopard - it’s caused by old, buggy, or misconfigured DNS servers. The change in Leopard to use the latest IETF recommendations for DNS lookups is simply bringing the DNS server problem to the surface. It’s unclear how many users or applications will be affected by this change, since it only appears with some DNS servers and only for applications using getaddrinfo (most applications still use gethostbyname). For many users these days, their DNS server is actually their home router, which then proxies the request (possibly with a local cache) - so updating router firmware may address the issue.

To work around the issue in Jungle Disk we’ve switched off IPV6 support in libcurl, which changes it to use the gethostbyname() function. It’s not clear if there is a way to disable SRV lookups system-wide on Leopard to fix other applications using getaddrinfo(). Anyone with further information on this issue is encouraged to post in the comments.

If you’d like to see the behavior for yourself, try the following test on Tiger and Leopard:

  1. Open up two terminal windows
  2. In one window, run “sudo tcpdump port 53″
  3. In the second window, run “curl http://s3.amazonaws.com” (or another domain)

On Tiger you will see an initial request for “A? s3.amazonaws.com” and reply. On Leopard you will see a request for “SRV? _http._tcp.s3.amazonaws.com”. Depending on your DNS server you will then see either the correct response, a NXDomain response, or a series of timeouts or retries.

Comments (20)

Jungle Disk and Leopard

We’ve done some initial testing of Jungle Disk on the release version of OSX 10.5 (Leopard) and so far haven’t seen any issues.

One item to be aware of - Leopard no longer shows Network Servers (like Jungle Disk) in Finder by default (who knows why?). The good news it that enabling them is easy. Just open Finder and go to the Finder -> Preferences menu. Check the “Connected Servers” item under “Show these items on the Desktop”.

Your Jungle Disk volume will now appear on the desktop when running. If you want it on the Finder sidebar (as it was in 10.4) you’ll just need to drag the volume from your desktop to the “Devices” list in the sidebar.

We’re going to see about automating this in the future, to reduce confusion for new Leopard users.

Update: A few users with outdated or buggy DNS servers have had issues with Jungle Disk on Leopard (typically a spinning color wheel on startup). If you’re affected by this you can download the latest beta version of Jungle Disk which includes a work-around. We’ll be putting out a new official release in the near future that includes the fix as well.

Comments (6)

Jungle Disk 1.45 Beta Released

We’re preparing the next release of Jungle Disk and are ready to make a beta version available for those interested in trying it early. This version includes some major under-the-hood changes to our automatic backup engine and some highly requested new features as well.

Here are the highlights:

  • Support for archiving previous versions and deleted files
    This can be enabled for both automatic backup and the mapped drive (if desired) and lets you keep a full history of previous files for a configurable period of time. Any files “deleted” or overwritten are moved into a special archive directory where they can be restored if needed. Having multi-version backup is essential to protect against file corruption or accidental deletion.
  • New restore dialog for easily restoring large numbers of files or previous versions
    Restoring large amounts of data via the mapped drive can sometimes be a cumbersome process. The new restore dialog can be used to easily restore files to the original (or different) locations. It can even be used when automatic backup is not enabled to restore data from your mapped drive bucket. The restore dialog also allows you to easily restore previous versions and deleted files (if you have that feature enabled) - either to your local drive, or to their original Jungle Disk location.
  • Option to password protect the software - either the configuration only or the entire application
    This is a request we’ve gotten many times from users who want an additional layer of security for the application, or in office situations where the software should not be reconfigured by other users. Paid users can now protect the application or just the configuration using the Jungle Tools account password.
  • Option to run backup cleanup on each backup run
    You can now choose to run the backup cleanup operation each time a backup runs (the manual menu option is available as well). Backup Cleanup removes locally deleted files and directories from your backup. Previously we recommend users only do this manually, after reviewing the list of files that would be deleted (in case they had deleted local files accidentally). The new previous version archiving feature means that even if you “clean up” accidentally deleted files they will still be available for restore in your previous versions. This makes it safer to run backup cleanup automatically.

For a full list of changes, please see the Release Notes.

Remember that this is beta software - it has not yet received the same amount of testing and feedback as our release versions. As such we recommend that most users wait until the release version is available. For those wanting to help test the new features, you can download the beta version from the Beta Download Page.

Comments (3)