IP Address-Based AFS Access Control for Fun and Profit

You may have heard that the Andrew File System -- AFS -- allows IP address-based entries for directory ACLs. The idea is that processes running on a machine with a given IP address can access protected directories without needing further authentication. You may have even tried it. If you're like me, you were frustrated. Now I've discovered the secrets, and I'm going to share them with you. You don't even have to bribe me.

Why would you want IP-based access? One scenario is that you have files whose access should be limited to certain machines. Maybe you have software that should only be run on certain machines because of legal or hardware limitations. Maybe authentication for certain processes is a drag to work out. Whatever your reason, this is pretty easy to do after you know the tricks, and seems to provide a moderate level of security.

Some introductory information and relevant links:

These examples were created under AFS 3.4a and Sun's Solaris 2.6.
Documentation links were to Transarc's documentation, until IBM changed the links and made the version 3.6. Now my documentation links point to the open source version's site, OpenAFS.
Transarc, the original commercial developer of AFS, is now IBM's Pittsburgh Lab.
AFS was first developed at Carnegie Mellon University as part of the Andrew Project.
For more information about AFS, try reading the FAQ for the USENET newsgroup alt.filesystems.afs.

All links in this web page should pop up in a single side window, so you won't have to flip back and forth between pages. I've tried to link every command to a relevant page, but other references may only be linked at their first instance.

Ok, you'll be dealing with just an IP number for the machine that needs access. No DNS names: AFS wants just an IP address and none of your carefully thought-out, clever naming scheme. Yeah, I know -- no fun. You'll have to get over it. Sorry.

The IP number must be in your AFS PTS database, just like a user. I'll use your-machine.your-domain.com (1.2.3.4) as an example.

Using klog, get an admin token for your AFS cell.

Check to see if 1.2.3.4 is in your PTS database:

     bash-2.02$ pts examine 1.2.3.4
     pts: User or group doesn't exist so couldn't look up id for 1.2.3.4

It's not there. So create it:

     bash-2.02$ pts createuser 1.2.3.4
     User 1.2.3.4 has id 2147418256

The AFS protection database server (PTS) assigns an AFS UID (the number after "User 1.2.3.4 has id ") to the machine. Accumulated wisdom recommends allowing that instead of trying to pick one, so don't use the -id flag unless you're sure know what you're doing and have consulted a ?SysAdmin quorum at your site.

You can wildcard/subnet these IP-based PTS entries. So you can get PTS ids for IP ranges denoted by, e.g.:

     1.2.3.0
     1.2.0.0
     1.0.0.0

This is supposed to do what you'd think it would: grant privileges to machines within the IP range denoted by the leading non-zero elements. Now you can easily give thousands of people and machines access when you didn't mean to. If you get the idea of wildcarding everything, as in 0.0.0.0, use system:anyuser instead. God knows why you'd want that, and if you're reading this you really ought to know about system:anyuser anyway, but that's how to do it. I don't know what happens if you need to give access to a machine that has a zero as the last tuple of the IP address. Maybe you should get another IP address for that machine.

OK, now your-machine has an AFS PTS id. But don't put the IP number directly on the ACL. Instead, create an appropriate PTS group and add the machine to it. Don't mix IP and principal entries in the same group; create another group (more on this later).

I'll create a PTS group:

     bash-2.02$ pts creategroup your-stuff system:administrators
     group your-stuff has id -1723

Again, the number at the end is an ID assigned by AFS. I've chosen here to have group your-stuff owned by system:administrators, but ownership by another group or principal might be appropriate. Joe User -- a cousin of mine -- with login joeuser could only create groups starting "joeuser:". Administrators can create groups without this restriction.

What does the group your-stuff look like?

     bash-2.02$ pts examine your-stuff
     Name: your-stuff, id: -1723, owner: system:administrators, creator: admin,
        membership: 0, flags: S-M--, group quota: 0.

I'll ignore everything right now except the flags, which are an access control list for the PTS entry. What's above is my local default for groups. The "S" means anyone can run pts examine on the group. The "M" means anyone can run pts membership on the group. The flags are

     SOMAR

and affect the operation of the pts commands examine, listowned, membership, adduser, and removeuser, respectively. The flags can generally have values of uppercase, lowercase, or hyphen. I won't go into the flags much here, but read the man page, and think carefully about what you want to be seen by whom. For instance, if the group is going to be for machines where users will be allowed to run certain software, you might want to make it easy for folks to discover what those machines are. If you're trying to keep users out, you might want to hide the membership from prying eyes. Pay as much attention to this as you would to any other ACL or permission.

By the way, pts examine works for any entity in the PTS database: groups, principals, and machines. E.g.:

     bash-2.02$ pts examine 1.2.3.4
     Name: 1.2.3.4, id: 2147418256, owner: system:administrators, creator: admin,
       membership: 0, flags: S----, group quota: 20.

The above flags are my current local default for machine entries.

Since pts examine works for three different types of entries -- machines, principals, and groups -- the man pages get confusing. The flags mean different things depending on what you're examining.

* When using pts examine, keep clearly in mind what type entry you're examining, and pay attention just to the parts of the manual page that speak to your type entry. If you do this and keep your mind clear of distraction, all will be well. *

I assume you'll want to add 1.2.3.4 to your-stuff:

      bash-2.02$ pts adduser 1.2.3.4 your-stuff

You'll get back just a prompt. So group your-stuff should now contain

      bash-2.02$ pts membership your-stuff
      Members of your-stuff (id: -1723) are:
        1.2.3.4

By the way, it's really easy to confuse pts adduser and pts createuser. Just like the rest of the AFS command suite, there are too many commands that sound too much alike or don't do exactly what you'd think from the name. Read the man page. Query a local ?SysAdmin quorum.

You did have an AFS directory you wanted to protect, right? Let's say that directory's permissions currently look like

      bash-2.02$ fs listacl /afs/your-domain/your-dir
      Access list for /afs/your-domain/your-dir is
      Normal rights:
        system:administrators rlidwka
        system:anyuser rl

Set directory permissions with fs setacl:

      bash-2.02$ fs setacl /afs/your-domain/your-dir your-stuff write

"write" is short for all perms except administer. Now fs listacl should return something like

      bash-2.02$ fs listacl /afs/your-domain/your-dir
      Access list for /afs/your-domain/your-dir is
      Normal rights:
        your-stuff rlidwk
        system:administrators rlidwka
        system:anyuser rl

So, you're set, right?

Wrong.

You put a file in your-dir. You get on other-machine.your-domain.com, and try to look at the file. You can't. That's good.

Then you get on your-machine.your-domain.com and try. You still can't look at the file. That's bad.

This has always been where I scratched my head and said "I don't get it." But I figured it out. I will now attempt to dispel confusion.

An AFS file server treats ACL IP entries differently from principal entries:
Changes to ACLs for principals are reflected almost right away.

Changes to ACLs for IP entries are NOT.

There's a flag to the fileserver instance called -hr:

-hr

Specifies how often the File Server refreshes its knowledge of the machines that belong to protection groups (its "host CPS"). The File Server must update this information to enable users from machines recently added to protection groups to access data for which those Machines now have the necessary ACL permissions.

Farther down in both the IBM and ?OpenAFS documentation mentioned above for the fileserver instance, the default setting for this "-hr" parameter is given as two hours:

The File Server maintains a host current protection subgroup (host CPS) for each client machine from which it has received a data access request. Like the CPS for a user, a host CPS lists all of the Protection Database groups to which the machine belongs, and the File Server compares the host CPS to a directory's ACL to determine in what manner users on the machine are authorized to access the directory's contents. When the pts adduser or pts removeuser command is used to change the groups to which a machine belongs, the File Server must recompute the machine's host CPS in order to notice the change. By default, the File Server contacts the Protection Server every two hours to recompute host CPSs, implying that it can take that long for changed group memberships to become effective. To change this frequency, use the -hr argument.

Maybe there's some overhead involved in these checks, so they're not done very often. Who knows? Maybe you'll want to diddle this flag for your site's fileserver instances, but probably not, or you'll never get to it, or you don't want to declare a downtime.

When I ran into this problem, one suggestion was

Wait a while and it'll start working. I imagine those folks didn't know about the flag and didn't know how long to wait. I guess most folks would have to wait between 0 and 2 hours, depending on luck. I don't know about you, but I don't like waiting for a bus if I don't know when it's going to show up. Another suggestion was
Restart the appropriate fileserver process. I guess those folks don't have any users they have to worry about pissing off, or only have two or three volumes on their file server. You can probably tell I didn't like either of the above suggestions. At that point I thought, "Gee, I bet if I move a volume onto a different file server with the ACL already in place, the new file server will need to grok all the volume's ACLs right then." And I was right. And the angels started to sing. So my hurry-up suggestion:
vos move the affected volume to another file server.

You should now be working. That was easy, wasn't it?

A few caveats:

Local users: Are there users on your-machine who should be kept away from the file? If so, this whole scheme may not work for you.
IP spoofing: If someone spoofed an IP address, ran an AFS client (probably in their own cell), and put your cell in their ?CellServDB, maybe they could read your file. Bad.
Fileserver/client encryption: If a client's cache manager is configured to encrypt data transfer with fileservers (i.e., if fs setcrypt is on), the server and client use key exchange to set up the encryption. However, if access to the data is allowed with an IP-based ACL, then there's no key exchange, and the data is sent in the clear. If you're not encrypting traffic or don't care, then don't worry about it.
Correct regular Unix groups and permissions: I ran into someone recently who had followed my instructions, and was able to read his file correctly on some machines but not others. The problem was the regular Unix groups. Ideally this wouldn't happen, but some applications will stat a file and assume the user can't read it because the permission bits are set wrong. Forestall this by setting the regular Unix modes and groups appropriately. Yes, you shouldn't have to do it, but do it anyway.
Mixed groups: AFS documentation warns against administering groups containing both principals and IP addresses. They say use just principals or just IP addresses in a group. You can always make another group, but often you'll be adding principals directly to the ACL anyway with no intervening group. Maybe this restriction has to do with the posited overhead and only occasional checking by the fileserver instance on IP addresses in PTS groups. So remember: No miscegenation!
No group rights inheritance: Groups are owned by users or by other groups, but child groups do not inherit parent group rights, nor do parent groups get the rights their children have. For example: Group a:a owns group a:b. Group a:a is placed on an ACL. Members of group a:a have rights in that directory. Members of group a:b do not inherit those rights. If group a:b is on a directory's ACL, group a:a doesn't assume any rights in that directory either.

No hierarchical group membership: Groups are not in other groups, even if they're owned by another group. Only principals and PTS-registered IP addresses can be in a group. But please note "Update" below. -- Hypothetical portion --

This no-nested-groups arrangement is kind of unfortunate: it might be nice to have structures like, e.g., a group called

        machines

consisting of

        machines:servers
        machines:clients
        machines:other

with, say, machines:servers containing

        machines:fileservers
        machines:loginservers
        machines:mailmachines

Somewhere at the bottom you'd have actual machines populating groups. This would be nice, but it doesn't work this way: you can't have a group as a member of a group. Groups can own other groups, but don't reside in groups. I've made this mistake over and over in my thinking -- enough for all of us, so please don't make it yourself.

I'm told someone at CMU has AFS/Solaris 2.6 code that lets groups be in other groups, which would allow this sort of hierarchical group structure. Alas, I can't find any information about it. I've also heard there's work (maybe at University of Michigan?) on similar code for <span class="createlink"><a href="/ikiwiki.cgi?page=OpenAFS&amp;from=devel%2FIPAccessControl&amp;do=create" rel="nofollow">?</a>OpenAFS</span>. If anyone has any specific information about either, would you let me know? A URL referring to the work would be great.

Update -- it turns out that recent verions of <span class="createlink"><a href="/ikiwiki.cgi?page=OpenAFS&amp;from=devel%2FIPAccessControl&amp;do=create" rel="nofollow">?</a>OpenAFS</span> now have a compile-time option for supergroups (groups that contain group). If I ever get any experience using this, I'll let you know.

-- End hypothetical portion --

There's a newly exposed problem on some platforms with occasional loss of IP-based access. The problem is discussed on the ?OpenAFS-Devel mailing list in this thread. Please take a look if you're having problems. Derrick Brashear is currently (30 June 2006) testing a patch, which you can find at /afs/andrew.cmu.edu/usr/shadow/ubik-all.diff-2

Was this page useful, or even interesting? Please link to it: it's difficult to find.

Send comments, questions, bad links, &c. to me if you like, or fix them -- this is a Wiki, after all. And to the many people who've sent comments: Thank you! Your feedback has helped!