% Software Maintenance in a Campus Environment: The Xhier Approach
% John Sellens, University of Waterloo
% jmsellens@watdragon.waterloo.edu

%%%\documentstyle[12pt,multicol,fancyheadings,twoside]{article}
\documentstyle[12pt,fancyheadings,twoside]{article}

%%%% Stolen from UW's fullpage.sty
%%%% SIDE MARGINS:
\oddsidemargin  15pt    % left margin on odd pages
\evensidemargin -35pt   % left margin on even pages
%%%\marginparwidth 50pt 
%%%\marginparsep 5pt       % Horizontal space between outer margin and 
%%%% VERTICAL SPACING:        
\topmargin 0pt           %    Nominal distance from top of page to top of
%%%% DIMENSION OF TEXT:
\textheight 595pt        % Height of text (including footnotes and figures, 
\textwidth 6.5in         % Width of text line.
\headheight 15pt         % space for running head


%%%\newenvironment{mysection}[1]{\typeout{Section: #1}%
%%%\begin{multicols}{2}[\section{#1}]}%
%%%{\end{multicols}}
\newenvironment{mysection}[1]{\typeout{Section: #1}%
\section{#1}}%
{}

\def\myitemize{\list{}{\settowidth{\labelwidth}{$\bullet$\ }%
\leftmargin\labelwidth
\addtolength{\leftmargin}{\labelsep}%
\topsep 0pt\itemsep 0pt\parsep 0pt\parskip 0pt\partopsep 0pt
\def\makelabel{$\bullet$\hfill}}}
\let\endmyitemize\endlist

\newenvironment{myfigure}[2]{\begin{figure}
\def\mylabel{#1}
\def\mycaption{#2}
\hrule\begingroup\begin{center}}{\end{center}\endgroup
\hrule\caption{\mycaption}\label{\mylabel}\end{figure}}

\def\Xhier{{\bf Xhier}}
\def\xhier{{\bf xhier}}
\def\Depot{{\bf Depot}}
\def\depot{{\bf depot}}
% used for program and command names in the text
\newcommand{\prog}[1]{\mbox{\it #1}}
% used for man page names
\newcommand{\manpage}[2]{\mbox{{\it #1\/}(#2)}}
% used for pathnames
\newcommand{\path}[1]{\mbox{\tt #1}}
% used for filetypes
\newcommand{\type}[1]{\mbox{\bf #1}}
% used for package names
\newcommand{\package}[1]{\mbox{\bf #1}}
% used for environment variable names
\newcommand{\env}[1]{\mbox{\tt #1}}

\title{Software Maintenance in a Campus Environment: The Xhier Approach}
\author{Math Faculty Computing Facility\thanks{The
	work described in this paper is
	due to the efforts of all members of the Math Faculty Computing
	Facility software group,
	although the paper itself was compiled by John Sellens,
	currently with the Department of Computing Services at the
	University of Waterloo. Any and all errors, confusion and
	inaccuracies should be attributed to him.  Contact
	{\tt jmsellens@watserv1.uwaterloo.ca}.} \\
	University of Waterloo \\
	Waterloo, Ontario \\
	N2L 3G1 Canada}
%%\date{Draft of \today}
\date{July 10, 1991}

% fancy schmancy headings and footings
%\pagestyle{myheadings}
%\markright{The Xhier Approach -- Draft of \today}
\pagestyle{fancyplain}
\renewcommand{\sectionmark}[1]{\markright{\thesection\ #1}}
\lhead[\fancyplain{}{\bf\thepage}]{\fancyplain{}{\bf\rightmark}}
\rhead[\fancyplain{}{\bf The Xhier Approach}]{\fancyplain{}{\bf\thepage}}
%\cfoot{DRAFT of \today}
\cfoot{July 10, 1991}


\begin{document}
\maketitle

\begin{abstract}
\Xhier\ is a system for software maintenance and distribution,
currently
in use at the University of Waterloo.  It allows easy, automatic software
installation on a variety of UNIX systems.   This paper describes some of
the design goals of \xhier, its structure and operation, as well as some
of the problems we have encountered, and some future goals.
\end{abstract}

\begin{mysection}{The Motivation for Xhier}

The Math Faculty Computing Facility (MFCF) is a major UNIX support
organization at the University of Waterloo, and, in addition to other
responsibilities specific to the Math Faculty,
provides software support
for UNIX machines all over campus using the \xhier\footnote{The name
\xhier\ is derived from the BSD \manpage{hier}{7} ``file system hierarchy''
man page, where the ``x'' prefix indicates eXperimental, though eXtended,
eXtravagant, and eXhausting have all been offered as alternatives.}
software maintenance
and distribution system.

MFCF started doing UNIX support before networks and workstations were
so widespread.  It was not too many years ago when the MFCF UNIX
environment was a single DEC VAX 11/780 running 4.2BSD UNIX.  In that
environment, adding and updating software was relatively easy, since
operating system releases were infrequent and the amount of third party
software was small.  It was easy to just add some source to \path{/usr/src}
and install some commands in \path{/usr/bin}.
If there were bugs in the OS that
needed fixing, or enhancements that were desired, it was a (relatively)
simple matter to modify the BSD source, and install the change.  A few
more BSD VAXes were added without too much pain, and careful use of
tools like \manpage{rdist}{1} and \manpage{rcp}{1}
made it possible to keep things up
to date.

Once UNIX machines started being more common on campus, and different
models and brands of machines started appearing, it quickly become
obvious that a different approach was required.  With operating systems
being updated as often as every few months, different operating
systems using different file system organizations,
and with useful third party software proliferating,
a more structured and automated approach was
needed.

Development of the \xhier\ system was started in early 1989
and has been ongoing ever since, with many enhancements and
fixes still planned.
MFCF currently provides software support for 11 different machine
``architectures'', using the automated tools provided by \xhier.
\Xhier\ is currently used by over 250 machines,
with over 1200 commands in more than 200 software ``packages'' available,
with over 14,000 package
installations on the various machines.

\end{mysection}

\begin{mysection}{The Problem to be Solved}

MFCF was faced with the task of providing software support for multiple
architectures, serving multiple machine administrations, and different
user groups and purposes.  A research machine in the Computer Science
department might have different needs than a student workstation cluster
in Electrical Engineering, though both would want some form of software
support, and some selection of locally installed commands.
Additionally, most users want a consistent environment, regardless of the
architecture and operating system of the machine they currently happen to
be using.

Accordingly, it was necessary to find some way to distribute software to,
and make it available on, large numbers of different machines.  This had
to be done with a large amount of configuration flexibility (so it could
be tailored to the needs of different machines and administrations), and
it had to be as automatic as possible, in order to make the most
efficient use of the software support staff.

Faced with this environment, MFCF's work on \xhier\ has proceeded with
these primary goals:
\begin{myitemize}
\item to simplify the distribution of updates
\item to simplify the installation (on multiple architectures)
\item to facilitate sharing software via remote file systems, and
\item to minimize changes and additions to the file structure
distributed with operating systems.
\end{myitemize}
An important underlying goal is that of automation of as many aspects of
the software maintenance and distribution process as possible.

\Xhier\ takes a different and more automated approach than systems like the
\depot\cite{depot}.  Appendix \ref{comparison} discusses some of the
differences between \xhier\ and the \depot.

\end{mysection}

\begin{mysection}{The Overall Structure}

In order to minimize the changes to stock file organizations, it was
decided to locate \xhier'd software in a separate (hopefully) unique
location -- under \path{/software}\footnote{\path{/software} is typically a
symbolic link to some appropriate location, but references to \xhier'd
software use the standardized name \path{/software}, rather than the
actual location of the directory on a particular machine.}.

Under \path{/software} are the package directories.  In \xhier,
software is organized into ``packages'' of related software.  For
example, there is a \package{tex} package,
containing \TeX\ related software, and
a \package{gnu} package containing GNU software.  Separating software into
separate packages has two main advantages: it allows a system
administrator to pick and choose what software to make available on a
machine (which may depend on licensing or size considerations),
and it allows the grouping of related software for maintenance
and similar purposes.  Additionally, it allows multiple versions of
the same software to be installed and available at the same time.

The other goals are approached using the details of the organization
of software packages, and by the use of support programs.

\Xhier\ recoginizes that there are three main steps in making software
available to users:
\begin{myitemize}
\item The files must be placed in an appropriate location, and be
compiled if necessary.
\item The software must be configured and initialized appropriately.
\item The software must be made available to the users (typically
through search rules).
\end{myitemize}
The structure and tools of \xhier\ are intended to make these steps as
easy as possible.

\end{mysection}


\begin{mysection}{Software Package Organization}

Each software package contains all the files associated with that
software, along with other files required for the distribution and
installation of the package.  A package typically consists of a number of
sub-directories, with the common ones being:
\begin{myitemize}
\item \path{bin}, \path{maintenance} and \path{servers}
contain commands for users, system and software maintainers, and other
programs (respectively) to use
\item \path{lib} for libraries
\item \path{man} for man pages
\item \path{data} for data and configuration files
\end{myitemize}
The structure and contents of a particular package are determined by the
needs of the package, but directories like \path{bin} and \path{man} are
used by other programs to make package components available to users
(more on this below).
\begin{myfigure}{pkgdir}{A Typical Package Directory}
\begin{verbatim}
% lc /software/x11
Directories:
.admin         bin            data           doc            export
include        lib            logs           man
\end{verbatim}
\end{myfigure}
Figure \ref{pkgdir} shows a typical package directory.

Two sub-directories are used by \xhier\ for distribution and installation.
\begin{myfigure}{export}{A Typical Package \path{export} Directory}
\begin{verbatim}
% lc /software/x11/export
Files:
boottime       crontab        inetd.conf     services
\end{verbatim}
\end{myfigure}
The optional \path{export} directory
(figure \ref{export}) contains information to be used
outside the package.  It contains such things as \path{services}
and \path{inetd.conf} file entries required by the package (e.g. if the
package contains a network daemon), commands to be run by \manpage{cron}{8},
commands to be run on system startup,
as well as information on userids and
groups required by the package, and patches to standard system files like
\path{/etc/rc}.
These file fragments and patches are applied automatically to the system
configuration files when a package is installed.

\begin{myfigure}{admin}{A Typical Package \path{.admin} Directory}
\begin{verbatim}
% lc /software/x11/.admin
Files:
Dependencies   Install        Maintainer     Options        Targets
file-types
\end{verbatim}
\end{myfigure}
Each package is required to have a \path{.admin} directory
(figure \ref{admin}), which
contains information used in the installation and distribution of the
package.  It typically contains 4 files:
\begin{myitemize}
\item \path{Maintainer}, which gives the mail address of the person
responsible for installing and maintaining the software
\item \path{Dependencies}, which lists other packages required for
proper operation of the package (e.g. a package might make use of
commands provided by other packages)
\item \path{Options}, which sets various options that determine how the
package is distributed and installed
\item \path{Install}, a program (typically a shell script)
that performs any package specific
initialization to prepare the package for use on a particular machine
(e.g. initialize configuration files, check for errors, replace
stock files if necessary, etc.)
\end{myitemize}
A couple of other files (\path{Targets} and \path{file-types})
are sometimes used to influence the distribution
of a package\footnote{These files are somewhat of a kludge, and may
disappear in the future.}.

The package structure makes each package a separate entity, containing
all the
knowledge required to install and use the
software it contains.  It also provides a consistent structure so that
packages tend to look very similar, which makes maintenance easier.

See appendix \ref{packagemaking} for an example of how packages are
created and installed.

\end{mysection}

\begin{mysection}{Complicating the Structure}

Now for the complication.  Recent versions of UNIX have
reorganized parts of the file system to make file sharing easier.  For
example, SunOS has \path{/usr/share}, which contains those files that can
be shared among all machines, regardless of CPU type.  \Xhier\ takes this
idea a step (or several steps) further.

Separating files based on their type has two main advantages, both of
which help to satisfy the goals of \xhier.  It makes it easier to share
software via remote file systems (e.g. NFS), because you can share as
much of a package as possible, and minimize duplication across machines
of the same or different hardware architectures.  And it makes software
distribution easier because it makes it easier to
use a tool like \manpage{rdist}{1} to
distribute the appropriate file types to the appropriate machines e.g.
shared files like man pages can be distributed everywhere, while
executable binaries can be distributed to just those machines with the
same architecture.

\Xhier\ currently recognizes six file types:
\begin{myitemize}
\item \type{share}: files like man pages, shell scripts, and (typically)
the \path{.admin} and \path{export} directories
\item \type{arch}: files that are different on different
hardware/operating system combinations, most commonly executable binaries
\item \type{spool}: files of a transient nature, like print requests
\item \type{local}: files specific to a particular machine, such as local
configuration or log files
\item \type{regional}: files shared between clients and their password
and home directory server, such as nntp server names
\item \type{admin}: files shared among all machines in a particular
administration, such as printer permission files
\end{myitemize}
The \type{regional} and \type{admin} types are variations on the
\type{local} type that seem to be useful.  In practice, it seems that
there are actually more file types (e.g. \type{man} might be used to
describe man pages and documentation, because you might not want those
locally on {\em every} machine, in order to save space),
but these six types are
recognized explicitly in the \xhier\ file structure.

This is accomplished through the use of the \path{/.software} directory.
\path{/.software} contains one sub-directory (or, more typically, a
symbolic link to, or an NFS mount of, an appropriately located directory)
for each of the six basic file types, with each sub-directory containing
package directories.  For example, the man pages for the \package{tex}
package are in \path{/.software/share/tex/man}, while the
%\package{tex}
command
binaries are in \path{/.software/arch/tex/bin}.  Note that
\path{/software} and \path{/.software/share} are the same
place\footnote{This means that one of
\path{/software} and \path{/.software/share} is redundant, but
\path{/software} was the original name, and provides a nice name for people
to use, while \path{/.software/share} was included primarily for
completeness.}.

Every package has a shared component; in particular, the \path{.admin}
directory must exist, and is always a shared directory.  And since
\path{/software} and \path{/.software/share} are the same place,
this ensures that, for a package called \package{pkg}, \path{/software/pkg}
will always exist.

It is desirable to hide the file type structure of a
package from the outside world, so that a user or another program does
not need to be concerned with the internal structure of a package.
So, even though every file in a package can be referenced through
\path{/.software} type directories,
references from outside the package itself
are always made through
\path{/software/pkg}.
(A package is allowed to know its own structure, and refer through
\path{/.software} as appropriate.)
This is accomplished by the (liberal) use of symbolic links in the
\path{/software/package} directory.  For example,
\path{/software/tex/bin} is actually a symbolic link to
\path{/.software/arch/tex/bin}.
It is also possible to have a shared directory contain real files along
with symbolic links to files in directories of other types.
For example, a package might have commands that
are primarily shareable shell scripts, with the few compiled commands
being referenced through symbolic links in the shared \path{bin}
directory\footnote{This latter structure, while within the rules,
can sometimes be confusing, so it is usually better to make the directory
the symbolic link, and its contents be real files, rather than a mixture
of the two.}.
Note that these links, while pointing to different absolute locations on
different machines, are shareable, since they link through
\path{/.software}.

The result is that the file type directories under \path{/.software}
provide separate hierarchies for different types, which makes file sharing
and distribution much easier.  For example, diskless workstations will
usually share and mount all file types from their server,
with the exception of the \path{local} type, which each client will have
its own version of.  The \path{/software} link provides a naming
convention for access to the components of a package.
Note that a given machine will have the correct type hierarchies for
itself only, e.g. \path{/.software/arch} on a VAX contains only VAX
binaries, and on a Sun will contain only Sun binaries.

\end{mysection}


\begin{mysection}{Automating Everything}

\Xhier\ is built on the idea that automation is good.  Accordingly, there
is an \xhier\ program to do just about anything to do with software
installation and distribution.

There are currently about 80 commands in the \xhier\ package, most of
them shell scripts.  Commands are used for such things as package
creation, maintenance and installation, and distribution, along with some
utilities that are useful for day-to-day operation (log file rollers,
etc.).  The following is a summary of some of the more prominent commands.

The \prog{xh-mkpkg} command creates a package skeleton in the
\path{/software} directory, with command and support directories, and
some template files, that can be filled in by the package creator.  This
makes the initial creation of a package simpler, because a
package creator doesn't have to remember all the details.
\prog{xh-make} is used as a cover for \manpage{make}{1} and
\manpage{imake}{1} and uses a standard \prog{imake} template to make
it easy to compile and install a program
\begin{myfigure}{xhimakefile}{Typical \path{xh-imakefile} file for
use with \prog{xh-make}}
\begin{verbatim}
#include "../PackageName"
Program(callpat,c,user)
\end{verbatim}
\end{myfigure}
(see figure \ref{xhimakefile}).
\prog{xh-make}, when used with an \path{xh-imakefile}, ensures that any
architecture-specific libraries and commands are used.

Once a package has been compiled and installed into the package
directory, the \prog{xh-install} command makes it available for use.
It does this by running the package's \path{Install} script, to
do any package-specific tasks (such as package configuration, or
replacement of any stock files), and then, if that is successful,
invoking other programs
to patch system files (like \path{/etc/services}) from the fragments in
the package's \path{export} directory.

Once a package has been installed, and is working on one
machine\footnote{Currently, this must be a machine higher up the tree,
usually the one central machine, but this will change in the future, so
that any machine can be the initial ``maintenance'' machine for a
package.}, it can be distributed to, and built on other machines.
This is a two-step process.  The \prog{xh-distribute} command is used to
distribute the package contents and structure to other machines.  It
creates a \path{Distfile} for use with \prog{rdist} that sends the
appropriate files to the appropriate machines lower on the tree.  For
example, it will send the shared files everywhere, but
architecture specific files will only be sent
to machines of the same architecture (though
it will complain if files on a machine of a different architecture are
out of date, or missing).  \prog{xh-distribute} then invokes \prog{rdist}
and summarizes the output for easier human consumption.

The second step is \prog{xh-sdist}, which distributes
the source for the
package to machines of other hardware/software architectures
(the ``architecture master'' 
machines)\footnote{\prog{xh-sdist}
sends the source to a temporary directory on the remote machines, and it
is automatically removed after a few days.},
and runs
\prog{xh-make} on those machines in the background, collecting the output
and mailing it to the invoker.
\prog{xh-sdist} has a number of options and configuration files to
control exactly what gets distributed to the remote machines
(e.g.~\path{.o} files are not distributed).
\prog{xh-sdist} has turned out to be an
incredibly useful tool -- it makes updating software and installing fixes
almost trivial\footnote{It should be pointed out that \prog{xh-sdist} is
somewhat of a kludge, because it introduces an arbitrary distinction
to some machines, that of
being an architecture master, when it should be possible to distribute
source to and build a package on just about any machine.}.

To help ensure that software is kept up to date, working, and consistent
on all machines, \prog{xh-maintenance} is run weekly from the
\prog{cron} (on every \xhier'd machine).
\prog{xh-maintenance} does some housekeeping and
consistency inspection, and runs \prog{xh-install} and
\prog{xh-distribute} on all packages, arranging to send the appropriate
output to the appropriate package and system maintainers.

Since \prog{xh-maintenance} is run weekly, starting at the same time on
every machine, it can take several weeks for package updates to reach the
bottom of the distribution tree\footnote{This propagation delay can be
a blessing, because it can keep broken packages from infecting the
entire distribution tree.}.
And so the other important distribution-related tool is \prog{xh-dist-recurse},
which attempts to distribute a package recursively down the distribution
tree.  This can be very useful in distributing fixes quickly, in case
you've inadvertently broken something and don't want to wait for the
usual propagation delays to distribute the fix.

\end{mysection}



\begin{mysection}{Making it all Available to the Users}

In a conventional software installation, software is installed in one
location only (commonly \path{/usr/local/bin}), or in other OS-specific
directories (such as Sun's \path{/usr/openwin/bin}), which the user is
expected to add to \env{PATH}.
Expecting a user to know where software is installed is not always
reasonable.  For many years, MFCF has had a command,
\prog{showpath}, to help users set
their \env{PATH} without having to know the specifics of a particular
machine.

With \xhier,
once the packages are installed, there are many different
package directories, each containing command directories
(e.g. \path{bin}), \path{man} directories, \path{include} directories,
and/or \path{lib} directories.
The \prog{showpath} command could simply produce a list of all the
\path{bin} directories in all the installed packages.  This, however,
quickly becomes too cumbersome to handle -- a 3,000 character \env{PATH}
containing 200 directories is a little large.  It also becomes
inefficient since some shells hash the \env{PATH} contents, but often
only the first few elements
(csh apparently hashes the first 8 elements of \env{PATH}).

\Xhier\ approaches this problem by creating a directory that contains a
link to each of the installed, packaged commands, using the \prog{xh-make-links}
command.  These links have to be symbolic
links, since the commands are not always going to be on the same
filesystem (or even the same machine).
There are actually three of these directories, one for user commands,
one for maintenance commands, and one for commands used by other programs
(servers).  These directories are system-wide and system-specific in that they
contain links to the commands for only those packages that the system
administrator wishes to be ``default'' packages\footnote{In practice,
the set of ``default'' packages is virtually the same everywhere.}.
A user's \env{PATH} consists of the directory of links,
the appropriate stock directories (for that architecture),
and any other directories the user
wishes to include.
These directories of symbolic links might be expected to cause a
performance penalty on command invocation, but, in practice, it seems not
to be a problem.

This ``searchrules'' approach to commands works well, but it is harder to
apply it to the other parts of a package.  Many \manpage{man}{1} commands
understand a \env{MANPATH} variable, but most compilers and loaders don't
have an easy way (i.e. an environment variable) to extend the searchrules
for include files and libraries.  Include files and libraries are linked
(again using \prog{xh-make-links}) into the standard locations honoured by
the OS, usually \path{/usr/include} and \path{/usr/local/lib}.

One more minor complication.  MFCF had been using the \prog{rman} remote
man command\footnote{By Jonathan C.~Broome, posted to {\tt net.sources} in
1985.} to provide man pages to smaller machines, where there is often not
enough disk space to keep man pages locally.  Local modifications to
\prog{rman} allow the remote man page server to deal with packaged man
pages by special interpretation of the \env{MANPATH} variable.

\begin{myfigure}{cshrc}{Typical extract from a user's \path{.cshrc} file,
and resultant variables}
\begin{verbatim}
setenv PATH    `showpath usedby=user $HOME/bin standard`
setenv MANPATH `showpath class=man standard`
setenv EDITOR  `showpath findfirst=vi`

% echo $PATH
/u/jmsellens/bin:/.software/local/.admin/bins/bin:/usr/ucb:/bin:/usr/bin
% echo $MANPATH
/software/.admin/man/:/usr/man/
% echo $EDITOR
/.software/local/.admin/bins/bin/vi
\end{verbatim}
\end{myfigure}
Figure \ref{cshrc} shows typical commands that would be included
in a \path{.cshrc} file, and common values of key environment variables.

\end{mysection}

\begin{mysection}{Some Lessons Learned from Xhier}

\Xhier\ tends to point out some of the more obscure points of software
installation.  One prime lesson is that doing software distribution can
turn out to be a complicated process, with a certain amount of overhead.

Automation is a necessity, once more than a few machines are involved,
but it turns out that some things can be hard to automate.  For example,
it's not too hard to add an entry to the \path{/etc/services} file, but
it's a little harder to notice conflicts or replacements, and some
packages might want a particular userid to exist (e.g. ``games''), but
it's hard to add an appropriate account to a \path{passwd} file
automatically.
The current \xhier\ utilities handle \prog{inetd}, \prog{cron}, and
\path{/etc/rc} entries fairly well (by automatically editing the
appropriate system files, and disabling conflicting stock entries), but
\path{passwd} and \path{group} file entries still have to be applied by hand
(local administrators are prompted to do it when a package is
installed).
And while automation allows easy installation and
distribution, it also makes it easy to distribute broken software and
automatically destroy dozens of machines.

Some software comes with particular pathnames built-in, often hard-wired
pathnames are sprinkled throughout the code (especially files destined
for \path{/etc}), and it's often hard to change these pathnames to refer
to locations under \path{/software} instead.  In other cases, they are
easy to change, but the same symbolic name is used for different kinds of
files.  For example, the \prog{make} variable ``USRLIB'' might be set to
\path{/usr/lib}, and be the location for both object libraries and
data files, or ``ETCDIR'' might be used to contain local
configuration files and system maintenance commands.  Files are often
named for their location, rather than for their function.
\Xhier-ing a package forces an installer to interpret the given
pathnames, and choose appropriate \xhier\ pathnames instead.

In addition, certain pathnames,
like \path{/usr/lib/sendmail} and \path{/usr/ucb/lpr},
are known to many programs, and so any replacement versions of these
programs must also replace the well-known stock pathnames (with links to
\xhier\ file locations).

It's hard to organize NFS (or, presumably, other remote file system)
mounts to make software sharing foolproof.  This problem is especially
evident with symbolic links.  A symlink to an absolute pathname will
start looking for the file at \path{/} on the local machine, even though
the link is on the remote machine.  And conversely, a relative symlink
will tend to stay on the remote machine, unless it passes up through a
mount point.  Some uses seem to call for absolute symlinks, and some seem
to call for relative symlinks, but so far we have been unable to come up
with a rule (or even a guideline) that works in all cases.

Choosing what goes in what package is hard.  It's not always easy to
group software into appropriate packages, and different administrators
might want different parts of a package, instead of the whole thing.
Large packages are easier to deal with, but small packages give greater
flexibility.

It's easy to generate lots of junk mail, so efficient ways of
summarizing and distributing it are required.

\end{mysection}

\begin{mysection}{Future Plans and Dreams}

The current distribution mechanism uses a tree structure, with each
member in the tree ``pushing'' packages to lower nodes in the tree.
This is less than optimal,
but was necessary since the only distribution
tool available at the time of implementation was \prog{rdist}.
Some of the problems with this approach are
\begin{myitemize}
\item Distribution is scheduled by the upstream host(s).  A
subsidiary node has (essentially) no control over when its software
gets updated.  This is annoying when a guaranteed stable environment is
desired, such as when one is trying to finish a thesis, or a machine is
used for student classwork.  In practice, this hasn't turned out to be a
problem, since a request to an upstream administrator is usually enough
to modify the distribution list.
\item It is inconvenient to have two (or more) distinct machines be the
``masters'' of two different packages and exchange packages with each
other, because the use of \prog{rdist} forces essentially ``wide-open''
access for the upstream machine on the downstream machine (via
\path{/.rhosts}).
\item This style of distribution seems to require wide open access to a
machine by its distribution machine.
\item Its hard for a given machine to dictate exactly what software it
gets, because that is decided by the upstream host(s).
\end{myitemize}
For these reasons, distribution control will pass to the receiving
machines, and the tree structure will be replaced with a more flexible
one, where any two cooperating machines can exchange whatever software
they wish to.

This, however, requires a program like \manpage{track}{1} that can do
most of the same things that (the locally enhanced) \manpage{rdist}{1}
can do. This work is currently underway.  It also requires fairly major
changes to the mechanisms that determine which parts of which packages get
sent to which machines. These changes are almost in place, to be used
with the current distribution mechanism until something ``track-like'' is
ready.

Most machines don't have enough local disk space to contain all packages,
or even just the ones they want to use.
Current space usage on
the primary maintenance machine\footnote{``watmath'' a MIPS RC6280}
is
\begin{verbatim}
    % du -s /.software/*
    105     /.software/admin
    272164  /.software/arch
    11823   /.software/local
    727     /.software/regional
    231340  /.software/share
\end{verbatim}
or a total of over 500 megabytes, in addition to the
space required by the OS\footnote{This excludes the spool files (especially
news articles) under \path{/.software/spool}.}.
As a result, remote file
system mounts (currently only via NFS) are required to make the necessary
software available on many machines.  This is usually done in
bulk, where a machine might mount all of \path{/.software/share} and
\path{/.software/arch} from a willing server.  Things get complicated
when you wish to have some packages locally and some remote mounted,
since you either have to mount each package individually, or arrange that
\path{/.software/*/package}  are either real directories or symbolic
links through the appropriate mount points.

At present, {\em all} remote mounting is done manually, which is getting
less and less practical.  Some method needs to be devised to
automatically make the appropriate mounts and links, which will allow a
local administrator to easily add a software package to a machine.  It's
not at all clear what the best approach to this problem is.

\Xhier{}ing a software package forces a software maintainer to look
beyond the local machine, and consider how a package would be installed
and used on different machines, with different administrators, perhaps
wishing different defaults or environments.  However, the current outlook
tends to stop at the edges of the campus.  A ``shared'' file tends to
mean one that applies to the campus as a whole, rather than to the
``world''.  In fact, there is currently no way in \xhier\ to indicate a
file with type ``campus'' i.e. there is no \path{/.software/campus}.  It
would be nice if it was possible to distribute \xhier\ and \xhier'd
packages to other organizations, but that brings other problems to light,
such as the necessity to trust a remote software provider.

\end{mysection}

\begin{mysection}{Conclusions and Observations}

With \Xhier, it is now almost trivial to make a new software package
available to hundreds of machines.  It's easy to make and distribute
fixes or enhancements.

\Xhier\ allows MFCF to provide software support, and a consistent
user environment,
on many more machines than would be otherwise possible.

\Xhier\ does have a number of faults
\begin{myitemize}
\item It is
large, and still changing,
and it is not always easy to understand how all the parts fit
together.
However, most software maintainers do not need to know how it works,
just how to create a package and make it available.
\item It tends to create a maze of symbolic links.
\item Requires a fair amount of machine overhead to keep it shuffling software
around.
\item It relies on having a fairly BSD-ish set of system services.
\end{myitemize}
but overall it tends to meet the needs.  For MFCF and the University of
Waterloo at least, \xhier\ is a success.

\end{mysection}

\appendix

\begin{mysection}{Creating and Installing a Package}\label{packagemaking}

Most software packages are very easy to create, though large pieces of
software, like \prog{gcc} or the X11 software, are more complicated.
Typical package creation and installation goes something like the
following example installation of package \package{foo}.

First, the source is unpacked into \path{/usr/source/foo}, and the
\path{Makefile}s are modified as appropriate, to set the right options,
and installation locations.  Many times this is done using an
\path{xh-imakefile} (see figure \ref{xhimakefile} for an example)
that is processed with \prog{xh-imake} to allow
easier per-architecture customizing.

The command
\begin{verbatim}
    % xh-mkpkg foo
\end{verbatim}
is used to create a skeleton structure in \path{/software/foo}.  This
structure is then adjusted by hand as required.  Usually this just means
removing the parts of the skeleton not required by this package.

Then the files in the \path{.admin} directory are edited as appropriate.
The appropriate options are set from the defaults
in the \path{Options} file, and the
\path{Install} script is modified to do the appropriate things on
installation and uninstallation.  Often this means just ending up with
the script
\begin{verbatim}
    #!/bin/csh -f
    exit 0
\end{verbatim}
The \package{foo} package name is then ``registered'' by adding it to the
\manpage{software}{i} man page\footnote{This step needs a little
refinement.}.

The software is installed via
\begin{verbatim}
    % xh-make install
\end{verbatim}
in the source directory.  Once that completes, the commands
\begin{verbatim}
    % xh-install foo
    % xh-distribute foo
    % xh-sdist -m install
\end{verbatim}
install the package locally, and then distribute the package structure
and source to the other architecture master machines, and compile and
install the package there.

Once this has been done, the \package{foo} package is available to any
\xhier'd machine on campus, with very little effort.

\end{mysection}

\begin{mysection}{Comparison to the Depot}\label{comparison}

The \depot, as described in \cite{depot}, has some similarities to
\xhier, but is less automated, and is based on a client-server
distributed file system environment, rather than \xhier's greater
emphasis on the use of local file space to hold packages.

Both systems separate software into
packages, and define a structure for the individual packages.  The
\depot\ requires human intervention for each installation, in order to
execute an optional installation script, and to modify system files (such
as \path{inetd.conf}) if necessary; \xhier\ uses the \prog{xh-install}
program and the package's \path{Install} script.  The \depot\ uses
redundant servers for the packages, but distributing a package to a
redundant server is done manually; \xhier\ has automatic distribution of
the packages.  The \depot\ is geared primarily to the use of server
machines that contain all the files for all architectures.  This makes it
less useful for standalone machines (such as a home workstation), since
such a machine would have to have the binaries for every architecture,
requiring more disk space (unless manual pruning was used).

Maintenance activities are less automated in the \depot\ as well.  In
order to recompile or update a package, a maintainer has to go to the
\depot\ administration machine of the appropriate architecture, and
compile and install the package there.  \Xhier, on the other hand, uses
the \prog{xh-sdist} command to rebuild and install a package on all
subsidiary architectures, which helps ensure that packages are kept up to
date on all architectures.

Since the \depot\ uses \prog{automount} mount points, as listed in an NIS
database, to make packages available to client machines, it is harder for
a system administrator to tailor the set of packages available on a
particular machine or set of machines.  \Xhier\ makes it easy for an
administrator to pick and choose what packages are available on his or
her machine(s).  Admittedly, in a primarily client-server, workstation
environment, this is less of a problem.

With \xhier, a system administrator can control when packages get
updated on his or her machine, which is useful in a student lab
environment, or on a personal workstation when trying to complete a
project.  Since the \depot\ is based on the client-server model, it is
much more difficult for a client to ``refuse'' an update.

\Xhier\ provides a set of tools for modifying, or replacing
vendor provided files, which makes it possible to correct flaws in stock
systems, or customize them to local requirements.  \Xhier\ provides
simple interfaces to system services such as \path{/etc/rc} and \prog{cron}.
This means that a package maintainer does not usually have to worry about
system-specific details.

\Xhier\ makes extensive use of search paths in order to hide the details
of an implementation from the users.  With the \depot, a user must know
where programs and man pages are located in the hierarchy in order to
make use of them.

\end{mysection}

\begin{mysection}{Xhier Availability}

\Xhier\ is still under development, and still contains many transitional
provisions to correct past mistakes,  and it is currently difficult to
bootstrap \xhier\ into a new environment (such as another campus).  It is
also a large collection of source, which relies in part on
modifications to licensed software.

As such, \xhier\ is not currently available for distribution, though we
hope to be able to make all or part of it and its tools available at some
point in the future.

\end{mysection}

\begin{thebibliography}{9}

\bibitem{depot} Manheimer, K., B. A. Warsaw, S. N. Clark, and W. Rowe,
``The Depot: A Framework for Sharing Software Installation Across
Organizational and UNIX Platform Boundaries'', {\em LISA IV Proceedings},
October 1990, pp 37-46.

\end{thebibliography}

\end{document}
