	Configurable Fail Over

Before configurable failover, we had this:

authorize {
  preprocess
  files
}

which instructed module_authorize to first pass the request through
rlm_preprocess, and if that returned success, pass it through rlm_files,
and if that returned success, module_authorize itself would then return
success. Processing was strictly linear and if one module failed, the whole
function would fail immediately.

Configurable failover provides more flexibility. It takes advantage of the
tree structure of radiusd.conf to support a config language that allows you
to specify groups of modules that should work together in ways other than
execute-in-order-return-on-fail. Basically you can redesign the flow of
module_authorize to fit your needs, without touching C code, just by altering
radiusd.conf.

I will soon explain this new language in detail, but first, if you just want
to know how to make failover happen without understanding why it works or how
to tweak it, here's your quick fix: just put a redundant{} block around those
module instances which refer to redundant databases. Example:

  modules {
    sql sql1 {
      server="myfirstserver.example"
      # Insert other necessary parameters here
    }
    sql sql2 {
      server="mysecondserver.example"
      # Insert other necessary parameters here
    }
    always handled {
      rcode = handled
    }
  }
  accounting {
      redundant {
        sql1			# try module sql1
        sql2			# if that's down, try module sql2
	handled			# otherwise drop the request as
				# it's been "handled" by the "always"
				# module (see doc/rlm_always)
      }
  }

OK, now for the exhaustive documentation. The things unexpectedly CAPITALIZED
are the key terms...

The fundamental object is called a MODCALLABLE, because it is something that
can be passed a specific radius request and returns one of the RLM_MODULE_*
results. It is a function - if you can accept the fact that pieces of
radiusd.conf are functions. There are two kinds of MODCALLABLEs: GROUPs and
SINGLEs.

A SINGLE is a reference to a module instance that was set up in the modules{}
section of radiusd.conf, like "preprocess" or "sql1". When a SINGLE is
called, the corresponding function in the rlm is invoked, and whichever
RLM_MODULE_* it returns becomes the RESULT of the SINGLE.

A GROUP is a section of radiusd.conf that includes some MODCALLABLEs.
Examples of GROUPs above include "authorize{...}", which implements the C
function module_authorize, and "redundant{...}", which contains two SINGLEs
that refer to a couple of redundant databases. Note that a GROUP can contain
other GROUPs - "Auth-Type SQL{...}" is also a GROUP, which implements the C
function module_authenticate when Auth-Type is set to SQL.

Now here's the fun part - what happens when a GROUP is called? It simply runs
through all of its children in order, and calls each one, whether it is
another GROUP or a SINGLE. It then looks at the RESULT of that child, and
takes some ACTION, which is basically either "return that RESULT immediately"
or "Keep going". In the first example, any "bad" RESULT from the preprocess
module causes an immediate return, and any "good" RESULT causes the
authorize{...} GROUP to proceed to the files module.

We can see the exact rules by writing them out the long way:

authorize {
  preprocess {
    notfound = 1
    noop     = 2
    ok       = 3
    updated  = 4
    fail     = return
    reject   = return
    userlock = return
    invalid  = return
    handled  = return
  }
  files {
    notfound = 1
    noop     = 2
    ok       = 3
    updated  = 4
    fail     = return
    reject   = return
    userlock = return
    invalid  = return
    handled  = return
  }
}

This is the same as the first example, with the default behavior explicitly
spelled out. Each SINGLE becomes its own section, containing a list of
RESULTs that it may return and what ACTION should follow from them. So
preprocess is called, and if it returns for example RLM_MODULE_REJECT, then
the reject=return rule is applied, and the authorize{...} GROUP itself
immediately returns RLM_MODULE_REJECT.

If preprocess returns RLM_MODULE_NOOP, the corresponding ACTION is "2". An
integer ACTION serves two purposes - first, it tells the parent GROUP to go
on to the next module. Second, it is a hint as to how desirable this RESULT
is as a candidate for the GROUP's own RESULT. So files is called... suppose
it returns RLM_MODULE_NOTFOUND. The ACTION for notfound inside the files{...}
block is "1". We have now reached the end of the authorize{...} GROUP and we
look at the RESULTs we accumulated along the way - there is a noop with
preference level 2, and a notfound with preference level 1, so the
authorize{...} GROUP as a whole returns RLM_MODULE_NOOP, which makes sense
because to say the user was not found at all would be a lie, since preprocess
apparently found him, or else it would have returned RLM_MODULE_NOTFOUND too.

[Take a deep breath - the worst is over]

That RESULT preference/desirability stuff is pretty complex, but my hope is
that it will be complex enough to handle the needs of everyone's real-world
imperfect systems, while staying out of sight most of the time since the
defaults will be right for the most common configurations.

So where does redundant{...} fit in with all that? Well, redundant{...} is
simply a group that changes the default ACTIONs to something like

  fail = 1
  everythingelse = return

so that when one module fails, we keep trying until we find one that doesn't
fail, then return whatever it returned. And at the end, if they all failed,
the redundant GROUP as a whole returns RLM_MODULE_FAIL, just as you'd want it
to (I hope).

There are two other kinds of grouping: group{...} which does not have any
specialized default ACTIONs, and append{...}, which should be used when you
have separate but similarly structured databases that are guaranteed not to
overlap.

That's all that really needs to be said. But now a few random notes:

1. GROUPs may have RESULT=ACTION specifiers too! It would look like this:

  authorize {
    preprocess
    redundant {
      sql1
      sql2
      notfound = return
    }
    files
  }

which would prevent rlm_files from being called if neither of the SQL
instances could find the user.

2. redundant{...} and append{...} are just shortcuts. You could write
    group {
      sql1 {
        fail     = 1
        notfound = 2
        noop     = return
        ok       = return
        updated  = return
        reject   = return
        userlock = return
        invalid  = return
        handled  = return
      }
      sql2 {
        fail     = 1
        notfound = 2
        noop     = return
        ok       = return
        updated  = return
        reject   = return
        userlock = return
        invalid  = return
        handled  = return
      }
    }
  instead of
    redundant {
      sql1
      sql2
    }
  but the latter is just a whole lot easier to read.

3. "authenticate{...}" itself is not a GROUP, even though it contains a list
of Auth-Type GROUPs, because its semantics are totally different - it uses
Auth-Type to decide which of its members to call, and their order is
irrelevant.

4. The default rules are context-sensitive - for authorize, the defaults are
what you saw above - notfound, noop, ok, and updated are considered
success, and anything else has an ACTION of "return". For authenticate, the
default is to return on success *or* reject, and only try the second and
following items if the first one fails. You can read all the default ACTIONs
in modcall.c (int defaultactions[][][]), or just trust me. They do the right
thing.

5. There are some rules that can't be implemented in this language - things
like "if the absolutelypositivelymandatory module returns notfound, the group
should immediately return reject". It would be possible to extend the
language to include that, as "notfound = return-reject", and the obvious
followup feature would be "notfound = 1-reject", "noop = 2-ok", "ok = 3-ok",
etc. But I don't feel justified adding that complexity in the first draft.
There are already enough things here that may never see real-world usage.
Like append{...}

-- Pac. 9/18/2000
