Give Rails Autoloading a Boot to the Head

In my last post on doing modularity like you mean it, I discussed the fact that Rails makes using modular applications challenging because you don’t get automatic reloading. Rails engines have the necessary hooks to get things working relatively smoothly, but if you are making truly Rails-independant code, you are on your own. Now that I’ve confirmed my suspicions by reading the source, I’m going to talk about how to give Rails autoloading a boot the head and keep gem based development from driving you crazy.

The code itself is somewhat long and convoluted – just what you’d expect from a library that hacks core Ruby semantics, and has been battle tested by being embedded in one of the most popular Ruby projects. The tests cover a lot of corner cases, and I imagine that many of them had their origin in bugs filed by frustrated users. What I’m really going to describe is my own understanding of the process, which has been confirmed by reading the code.

Note that my code check, and almost all of my experience, is with Rails 3.0 and 3.1.

Rails, during development, operates like this:

  • An unknown constant is detected
  • The constant name is converted to a relative path
  • The autoload paths are consulted, to see if any of them contains the desired file. The file is loaded.
  • Constants defined by loading the file are tracked.
  • If the file doesn’t create the constant that triggered the load, an error is raised.
  • At the end of the request, all the autoloaded constants (which were tracked) are removed. This allows them to be reloaded on the next request and receive updated code.

At almost every step in this process, things can go wrong.

An Unknown Constant is Detected

Ruby is somewhat famous for method_missing, which allows you to intercept an undefined method, and provide some behavior instead of (or in addition to) an exception. Less well known is it’s cousin, const_missing.

This is far from the most common stumbling block, but it’s still helpful to know how it works. If you define your own const_missing, Rails autoloading won’t work. Perhaps more common, if your Rail-independant code requires your files, the constants will be defined, and won’t trip const_missing. While this may seem obvious, we’ll be returning to this fact shortly.

The other important detail is that if you have nested namespaces, the outer namespace gets fully resolved before the inner namespaces have a chance to trip const_missing. To fully understand the implications of this, we need to work through the next step.

The Constant Name is Converted to a Relative Path

The assumption is made that there is a one to one correspondence between the fully qualified module name and the relative file path. You violate this assumption at your own peril.

Fully qualified means the complete name needed to reach the constant, starting from the base level. Gem::Submodule::MyClass is fully qualified. MyClass or Submodule::MyClass are not. You might be able to use the shorter forms inside the appropriate module or class declaration, but I honestly haven’t investigated how smart the autoloader is in this case.

It’s the fully qualified name that gets converted to a path. So in the above example, gem/submodule/my_class.rb. Notice that namespace levels get converted to directories, and CamelCase gets converted to under_score. If classes and modules don’t exactly match this convention, they aren’t going to be found, with one exception.

Now that I’ve established how filenames are created, I can revisit the point about the order constants are looked up. Recall that const_missing trips “in order”. In our running example, the first thing that will happen is that Gem will generate a call to const_missing – not Gem::Submodule::MyClass. If gem.rb defined Gem::Submodule::MyClass, then things can stop there. The later constants are defined, so const_missing won’t fire, and no more autoloading will occur. If you define the parts in the higher level namespace, you aren’t foreced to define files and directories for the parts.

The opposite, however is not true. If you define Gem::Submodule::MyClass in gem/submodule/my_class.rb, but don’t have a gem/submodule.rb, the autoloading process will fail when it hits Gem::Submodule and can’t find gem/submodule.rb. Work arounds include having a stub file, or declaring the module in gem.rb.

The Autoload Paths are Consulted

Just as Ruby has a LOAD_PATH where it looks for gems, Rails has an autoload_path where it tries to append the relative path for a missing constant to find someplace to load it. Specifically, it is config.autoload_paths, or ActiveSupport::Dependencies.autoload_paths. A recent experience indicates you use config in application.rb (right where the comment suggests it),

config.autoload_paths += Dir["#{config.root}/vendor/modules/**/lib/"]

and the fully qualified form if you need to add some later, such as in development.rb or an initializer.

ActiveSupport::Dependencies.autoload_paths += %w[ .... ]

Also of note is autoload_once_paths, which enjoy the same searching benefits, but don’t have their constants tracked (and eventually unloaded).

Constants Defined by Loading the File are Tracked.

The autoload system keeps track the constants defined during an autoload operation. It records these as autoloaded constants, for later use.

It’s important to note that ‘autoloaded constants’ are a key feature in Rails ability to reload files during development without restarting the server. If you caused a constant to be defined by an explicit require, (or even a straight up definition) it won’t be marked as autoloaded. But I’m getting two steps ahead of myself.

If the File Doesn’t Define the Constant that Triggered the Load, an Error is Raised.

This is the infamous LoadError: "Expected #{file_path} to define #{qualified_name}", which means pretty much was it says. When const_missing gets called, it by necessity knows which constant was missing, and to keep running the program it has to provide a value for that constant. It’s going to get that value by attempting to read the constant again after loading it’s best guess for the appropriate file. If the constant isn’t defined, not only would there be no value to provide, but const_missing would trigger and you’d get an infinite loop (at least until the stack ran out).

At the End of the Request, All the Autoloaded Constants are Removed.

In order to get automatic reloading, the system will have to trigger autoloading again next time around. In order for that to happen, the constants need to be missing for const_missing. During request cleanup, all the autoloaded constants are purged, so that they will be able to trigger again, bringing in fresh code.

If you are developing your application as gems, the first challenge is to get listed in autoload_paths. After that, you may still have the issue that files were loaded by require, so their constants weren’t marked as autoloaded. Fortunately, there is also configuration parameter for this (I keep it in development.rb)

  ActiveSupport::Dependencies.explicitly_unloadable_constants += %w[ Gem AnotherGem ]

This is actually a separate list than the autoloaded constants. The autoload list is cleared after each run, (with the expectation that it will be repopulated) whereas the system attempts to remove the explicit constants every time.

And that’s Rails autoloading in a nutshell – at least until the Rails 3.2 announcement where “we now only reload classes from files you’ve actually changed”.

Posted Friday, December 23rd, 2011 under Essay.

Comments are closed.