I've said in earlier chapters that CVS is not "black box" software. Black boxes don't let you peek inside; they don't give you internal access so that you can fix (or break) things. The premise is that the black box usually doesn't need to be fixed. Most of the time, the software should work perfectly, so users don't need internal access. But when black boxes do fail, they tend to fail completely. Any problem at all is a showstopper, because there aren't many options for repair.
CVS is more like a perfectly transparent box -- except without the box. Its moving parts are exposed directly to the environment, not hermetically sealed off, and bits of that environment (unexpected file permissions, interrupted commands, competing processes, whatever) can sometimes get inside the mechanism and gum up the gears. But even though CVS does not always work perfectly, it rarely fails completely, either. It has the advantage of graceful degradation; the degree to which it doesn't work is usually proportional to the number and severity of problems in its environment. If you know enough about what CVS is trying to do -- and how it's trying to do it -- you'll know what to do when things go wrong.
Although I can't list all of the problems that you might encounter, I've included some of the more common ones here. This chapter is divided into two sections: The first describes those parts of the environment to which CVS is most sensitive (mainly repository permissions and the working copy administrative area), and the second describes some of the most frequently encountered problems and their solutions. By seeing how to handle these common situations, you will get a feeling for how to approach any unexpected problem in CVS.
As a CVS administrator (read "field doctor"), you will find that 90 percent of your users' problems are caused by inconsistent working copies, and the other 90 percent by incorrect repository permissions. Therefore, before looking at any specific situations, I'll give a quick overview of the working copy administrative area and review a few important things about repository permissions.
You've already seen the basics of working copy structure in section An Overview of CVS; in this section, we'll go into a bit more detail. Most of the details concern the files in the CVS/ administrative subdirectories. You already know about Entries, Root, and Repository, but the CVS/ subdirectory can also contain other files, depending on the circumstances. I'll describe those other files here, partly so they don't surprise you when you encounter them, and partly so you can fix them if they ever cause trouble.
Sometimes, a file named `CVS/Entries.Log' will mysteriously appear. The sole purpose of this file is to temporarily cache minor changes to CVS/Entries, until some operation significant enough to be worth rewriting the entire Entries file comes along. CVS has no ability to edit the Entries file in place; it must read the entire file in and write it back out to make any change. To avoid this effort, CVS sometimes records small changes in Entries.Log, until the next time it needs to rewrite Entries.
The format of Entries.Log is like Entries, except for an extra letter at
the beginning of each line.
A means that the line is to be added
to the main Entries file, and
R means it is to be removed.
For the most part, you can ignore Entries.Log; it's rare that a human has to understand the information it contains. However, if you're reading over an Entries file to debug some problem in a working copy, you should also examine Entries.Log.
The CVS/Entries.Backup file is where CVS actually writes out a new Entries file, before renaming it to `Entries' (similar to the way it writes to temporary RCS files in the repository and then moves them to their proper name when they're complete). Because it becomes Entries when it's complete, you'll rarely see an Entries.Backup file; if you do see one, it probably means CVS got interrupted in the middle of some operation.
If the CVS/Entries.Static file exists, it means that the entire directory has not been fetched from the repository. (When CVS knows a working directory is in an incomplete state, it will not bring additional files into that directory.)
The Entries.Static file is present during checkouts and updates and
removed immediately when the operation is complete. If you see
Entries.Static, it means that CVS was interrupted, and its presence
prevents CVS from creating any new files in the working copy. (Often,
cvs update -d solves the problem and removes
The absence of Entries.Static does not necessarily imply that the working copy contains all of the project's files. Whenever a new directory is created in the project's repository, and someone updates their working copy without passing the -d flag to update, the new directory will not be created in the working copy. Locally, CVS is unaware that there is a new directory in the repository, so it goes ahead and removes the Entries.Static file when the update is complete, even though the new directory is not present in the working copy.
If the CVS/Tag file is present, it names a tag associated, in some sense, with the directory. I say "in some sense" because, as you know, CVS does not actually keep any revision history for directories and, strictly speaking, cannot attach tags to them. Tags are attached to regular files only or, more accurately, to particular revisions in regular files.
However, if every file in a directory is on a particular tag, CVS likes to think of the entire directory as being on the tag, too. For example, if you were to check out a working copy on a particular branch:
floss$ cvs co -r Bugfix_Branch_1
and then add a file inside it, you'd want the new file's initial revision to be on that branch, too. For similar reasons, CVS also needs to know if the directory has a nonbranch sticky tag or date set on it.
Tag files contain one line. The first character on the line is a single-letter code telling what kind of tag it is, and the rest of the line is the tag's name. Currently, CVS uses only these three single-letter codes:
floss$ cvs checkout -D 1999-05-15 myprojor
floss$ cvs update -D 1999-05-15 myprojis run.
(If you see some other single-letter code, it just means that CVS has added a new tag type since this chapter was written.)
You should not remove the Tag file manually; instead, use
There are a few other files you may occasionally find in a CVS/ subdirectory:
These files are usually not the cause of problems, so I'm just listing them (see section CVS Reference for their full descriptions).
As features are added to CVS, new files (not listed here) may appear in working copy administrative areas. As new files are added, they'll probably be documented in the Cederqvist manual, in the node Working Directory Storage. You can also start looking in src/cvs.h in the source distribution, if you prefer to learn from code.
Finally, note that all CVS/* files -- present and future -- use whatever line-ending convention is appropriate for the working copy's local system (for example, LF for Unix or CRLF for Windows). This means that if you transport a working copy from one kind of machine to the other, CVS won't be able to handle it (but then, you'd have other problems, because the revision-controlled files themselves would have the wrong line-end conventions for their new location).
CVS does not require any particular repository permission scheme -- it can handle a wide variety of permission arrangements. However, to avoid getting confusing behaviors, you should make sure your repository setup meets at least the following criteria:
LockDir=/usr/local/cvslocksOf course, then you would need to make sure the directory /usr/local/cvslocks is writeable by all CVS users. Either way, most CVS operations, including read-only ones, are going to require a writeable directory somewhere. By default, that directory is the project's repository; if you're very security conscious, you can change it to be somewhere else.
The bulk of this chapter is organized into a series of questions and answers, similar to an Internet FAQ (Frequently Asked Questions) document. These are all based on actual CVS experiences. But before we look at individual cases, let's take a moment to consider CVS troubleshooting from a more general point of view.
The first step in solving a CVS problem is usually to determine whether it's a working copy or repository problem. The best technique for doing that, not surprisingly, is to see if the problem occurs in working copies other than the one where it was first noticed. If it does, it's likely a repository issue; otherwise, it's probably just a local issue.
Working copy problems tend to be encountered more frequently, not because working copies are somehow less reliable than repositories, but because each repository usually has many working copies. Although most working copy knots can be untied with enough patience, you may occasionally find it more time-efficient simply to delete the working copy and check it out again.
Of course, if checking out again takes too long, or there is considerable uncommitted state in the working copy that you don't want to lose, or if you just want to know what's wrong, it's worth digging around to find the cause of the problem. When you start digging around, one of the first places to look is in the CVS/ subdirectories. Check the file contents and the file permissions. Very occasionally, the permissions can mysteriously become read-only or even unreadable. (I suspect this is caused by users accidentally mistyping Unix commands rather than any mistake on CVS's part.)
Repository problems are almost always caused by incorrect file and directory permissions. If you suspect a problem may be due to bad repository permissions, first find out the effective repository user ID of the person who's having the trouble. For all local and most remote users, this is either their regular username or the username they specified when they checked out their working copy. If they're using the pserver method with user-aliasing (see the section section Anonymous Access in section Repository Administration), the effective user ID is the one on the right in the CVSROOT/passwd file. Failure to discover this early on can cause you to waste a lot of time debugging the wrong thing.
And now, without further ado...
All of these situations are ones I've encountered in my real-life adventures as a CVS troubleshooter (plus a few items that are not really problems, just questions that I've heard asked so often that they may as well be answered here). The list is meant to be fairly comprehensive, and it may repeat material you've seen in earlier chapters.
The situations are listed according to how frequently they seem to arise, with the most common ones first.
If you see a message like this
cvs update: [22:58:26] waiting for qsmith's lock in /usr/local/newrepos/myproj
it means you're trying to access a subdirectory of the repository that is locked by some other CVS process at the moment. A process is being run in that directory so it may not be in a consistent state for other CVS processes to use.
However, if the wait message persists for a long time, it probably means that a CVS process failed to clean up after itself, for whatever reason. It can happen when CVS dies suddenly and unexpectedly, say, due to a power failure on the repository machine.
The solution is to remove the lock files by hand from the repository subdirectory in question. Go into that part of the repository and look for files named `#cvs.lock' or that begin with `#cvs.wfl' or `#cvs.rfl'. Compare the file's timestamps with the start times of any currently running CVS processes. If the files could not possibly have been created by any of those processes, it's safe to delete them. The waiting CVS processes eventually notice when the lock files are gone -- this should take about 30 seconds -- and allow the requested operation to proceed.
See the node Locks in the Cederqvist manual for more details.
Don't panic -- it just means that the file has changed in the repository since the last time you checked it out or updated it.
cvs update on the file to merge in the changes from the
repository. If the received changes conflict with your local changes,
edit the file to resolve the conflict. Then try your commit again -- it
will succeed, barring the possibility that someone committed yet another
revision while you were busy merging the last changes.
The most common, less obvious cause of this problem is that you forgot
to list the repository using an
--allow-root option in your inetd
Recall this example /etc/inetd.conf line from section Repository Administration:
cvspserver stream tcp nowait root /usr/local/bin/cvs cvs \ --allow-root=/usr/local/newrepos pserver
(In the actual file, this is all one long line, with no backslash.)
--allow-root=/usr/local/newrepos portion is a security
measure, to make sure that people can't use CVS to get pserver access to
repositories that are not supposed to be served remotely. Any
repository intended to be accessible via pserver must be mentioned in an
--allow-root. You can have as many different
options as you need for all of your system's repositories (or anyway, as
many as you want until you bump up against your inetd's argument limit).
See section Repository Administration for more details on setting up the password-authenticating server.
Okay, if the problem is not a missing
--allow-root, here are a
few other possibilities:
cvspserver 2401/tcpso inetd is not even listening on that port to pass connections off to CVS.
That's because CVS commits happen in pieces, not atomically. :-)
More specifically, CVS operations happen directory by directory. When you do a commit (or an update, or anything else, for that matter) spanning multiple directories, CVS locks each corresponding repository directory in turn while it performs the operation for that directory.
For small- to medium-sized projects, this is rarely a problem -- CVS manages to do its thing in each directory so quickly that you never notice the nonatomicity. Unfortunately, in large projects, scenarios like the following can occur (imagine this taking place in a project with at least two deep, many-filed subdirectories, A and B):
Clearly, when this is all over, jrandom's working copy reflects qsmith's changes to B but not A. Even though qsmith intended the changes to be committed as a single unit, it didn't happen that way. Now jrandom's working copy is in a state that qsmith never anticipated.
The solution, of course, is for jrandom to do another cvs update to fetch the uncaught changes from qsmith's commit. However, that assumes that jrandom has some way of finding out in the first place that he only got part of qsmith's changes.
There's no easy answer to this quandary. You simply have to hope that the inconsistent state of the working copy will somehow become apparent (maybe the software won't build, or jrandom and qsmith will have a conversation that's confusing until they realize what must have happened).
CVS's failure to provide atomic transaction guarantees is widely considered a bug. The only reason that locks are not made at the top level of the repository is that this would result in intolerably frequent lock contentions for large projects with many developers. Therefore, CVS has chosen the lesser of two evils, reducing the contention frequency but allowing the possibility of interleaved reads and writes. Someday, someone may modify CVS (say, speeding up repository operations) so that it doesn't have to choose between two evils; until then, we're stuck with nonatomic actions.
For more information, see the node Concurrency in the Cederqvist manual.
In general, CVS doesn't do a very good job of preserving permissions on files. When you import a project and then check it out, there is no guarantee that the file permissions in the new working copy will be the same as when the project was imported. More likely, the working copy files will be created with the same standard permissions that you normally get on newly created files.
However, there is at least one exception. If you want to store executable shell scripts in the project, you can keep them executable in all working copies by making the corresponding repository file executable:
floss$ ls -l /usr/local/newrepos/someproj total 6 -r--r--r-- 1 jrandom users 630 Aug 17 01:10 README.txt,v -r-xr-xr-x 1 jrandom users 1041 Aug 17 01:10 scrub.pl,v* -r--r--r-- 1 jrandom users 750 Aug 17 01:10 hello.c,v
Notice that although the file is executable, it is still read-only, as all repository files should be (remember that CVS works by making a temporary copy of the RCS file, doing everything in the copy, and then replacing the original with the copy when ready).
When you import or add an executable file, CVS preserves the executable bits, so if the permissions were correct from the start, you have nothing to worry about. However, if you accidentally add the file before making it executable, you must go into the repository and manually set the RCS file to be executable.
The repository permissions always dominate. If the file is nonexecutable in the repository, but executable in the working copy, the working copy file will also be nonexecutable after you do an update. Having your files' permissions silently change can be extremely frustrating. If this happens, first check the repository and see if you can solve it by setting the appropriate permissions on the corresponding RCS files.
A feature called
PreservePermissions has recently been added to
CVS that may alleviate some of these problems. However, using this
feature can cause other unexpected results (which is why I'm not
recommending it unconditionally here). Make sure you read the nodes
config and Special Files in the Cederqvist before putting
PreservePermissions=yes in CVSROOT/config.
For pserver connections, CVS on the client side tries to find the
.cvspass file in your home directory. Windows machines don't have a
natural "home" directory, so CVS consults the environment variable
%HOME%. However, you have to be very careful about how you set
HOME. This will work:
This will not:
That final backslash is enough to confuse CVS, and it will be unable to open `C:\.cvspass'.
So, the quick and permanent solution is to put
into your autoexec.bat and reboot. CVS pserver should work fine after that.
You mean different subdirectories of your working copy somehow got on different branches? You probably ran updates with the -r flag, but from places other than the top level of the working copy.
No big deal. If you want to return to the trunk, just run this
cvs update -r HEAD
cvs update -A
from the top directory. Or, if you want to put the whole working copy on one of the branches, do this:
cvs update -r Branch_Name
There's nothing necessarily wrong with having one or two subdirectories of your working copy on a different branch than the rest of it, if you need to do some temporary work on that branch just in those locations. However, it's usually a good idea to switch them back when you're done -- life is much less confusing when your whole working copy is on the same line of development.
This is due to a clock difference between the repository and local machines. You can solve it by resetting one or both of the clocks, or specifying a different date as the argument to -D. It's perfectly acceptable to specify a date in the future (such as -D tomorrow), if that's what it takes to compensate for the time difference.
If you see an error like this:
cvs [export aborted]: cannot write /usr/local/myproj/CVSROOT/val-tags: \ Operation not permitted
it means the user CVS is running as does not have permission to write to the CVSROOT/val-tags file. This file stores valid tag names, to give CVS a fast way to determine what tags are valid. Unfortunately, CVS sometimes modifies this file even for operations that are read-only with respect to the repository, such as checking out a project.
This is a bug in CVS and may be fixed by the time you read this. Until then, the solution is either to make val-tags world-writeable or, failing that, to remove it or change its ownership to the user running the CVS operation. (You'd think just changing the permissions would be enough, but on several occasions I've had to change the ownership, too.)
Various CVS operations cause the working copy to have a sticky tag, meaning a single tag that corresponds to each revision for each file (in the case of a branch, the sticky tag is applied to any new files added in the working copy). You get a sticky tagged working area whenever you check out or update by tag or date, for example:
floss$ cvs update -r Tag_Name
floss$ cvs checkout -D '1999-08-16'
If a date or a nonbranch tag name is used, the working copy will be a frozen snapshot of that moment in the project's history -- so naturally you will not be able to commit any changes from it.
To remove a sticky tag, run update with the -A flag
floss$ cvs update -A
which clears all the sticky tags and updates each file to its most recent trunk revision.
This is just a case of a bad error message in CVS; probably someone will get around to fixing it sooner or later, but meanwhile it may bite you. The error message looks something like this:
floss$ cvs co -d bwf-misc user-space/bwf/writings/misc cvs server: cannot find module `user-space/bwf/writings/misc' - ignored cvs [checkout aborted]: cannot expand modules
CVS appears to be saying that there's something wrong with the CVSROOT/modules file. However, what's really going on is a permission problem in the repository. The directory I'm trying to check out isn't readable, or one of its parents isn't readable. In this case, it was a parent:
floss$ ls -ld /usr/local/cvs/user-space/bwf drwx------ 19 bwf users 1024 Aug 17 01:24 bwf/
Don't let that egregiously wrong error message fool you -- this is a repository permission problem.
You probably did
floss$ cvs watch remove
on all the files, but forgot to also do:
floss$ cvs watch off
A hint for diagnosing watch problems: Sometimes it can be immensely clarifying to just go into the repository and examine the CVS/fileattr files directly. See section Repository Administration for more information about them.
Did you remember to use -kb when you added them? If not, CVS may have performed line-end conversion or RCS keyword substitution on them. The easiest solution is usually to mark them as binary
floss$ cvs admin -kb foo.gif
and then commit a fixed version of the file. CVS will not corrupt the new commit or any of the commits thereafter, because it now knows the file is binary.
If you're running the CVS client on a non-Unix platform and are not getting the line-end conventions that you want in some working copy files, it's usually because they were accidentally added with -kb when they shouldn't have been. This can be fixed in the repository with, believe it or not, the command:
floss$ cvs admin -kkv FILE
The -kkv means to do normal keyword substitution and implies normal line-end conversions as well. (Internally, CVS is a bit confused about the difference between keyword substitution and line-end conversion. This confusion is reflected in the way the -k options can control both parameters.)
Unfortunately, that admin command only fixes the file in the repository -- your working copy still thinks the file is binary. You can hand edit the CVS/Entries line for that file, removing the -kb, but that won't solve the problem for any other working copies out there.
Well, you can't exactly remove the subdirectory, but you can remove all of the files in it (first remove them, then cvs remove them, and then commit). Once the directory is empty, people can have it automatically pruned out of their working copies by passing the -P flag to update.
Yes, you can. You can copy .cvspass files from machine to machine, and you can even copy individual lines from one .cvspass file to another. For high-latency servers, this may be faster than running cvs login from each working copy machine.
Remember that if you transport a .cvspass file between two machines with different line-ending conventions, it probably won't work (of course, you can probably do the line-end conversion manually without too much trouble).
You don't need to hand-edit anything in the repository to solve this. Just run admin with the -m flag. Remember to have no space between -m and its argument, and to quote the replacement log message as you would a normal one:
floss$ cvs admin -m1.17:'I take back what I said about the customer.' hello.c
In the repository, copy (don't move) the RCS files to the desired new location in the project. They must remain in their old locations as well. Then, in a working copy, do:
floss$ rm oldfile1 oldfile2 ... floss$ cvs remove oldfile1 oldfile2 ... floss$ cvs commit -m Ā“removed from hereĀ” oldfile1 oldfile2 ...
When people do updates after that, CVS correctly removes the old files and brings the new files into the working copies just as though they had been added to the repository in the usual way (except that they'll be at unusually high revision numbers for supposedly new files).
Currently, there is no convenient way to do this in CVS. The lack is
sorely felt by all users, and I believe work is under way to make this
feature available. By the time you read this, a
command or something similar may be available.
Until then, there are workarounds. You can run cvs log -h and read the
sections of the output following the header
symbolic names:. Or,
if you happen to be on the repository machine, you can just look at the
beginnings of some of the RCS files directly in the repository. All of
the tags (branches and nonbranches) are listed in the
floss$ head /usr/local/newrepos/hello.c,v head 2.0; access; symbols Release_1_0:1.22 Exotic_Greetings-2:1.21 merged-Exotic_Greetings-1:1.21 Exotic_Greetings-1:1.21 merged-Exotic_Greetings:1.21 Exotic_Greetings-branch:188.8.131.52 Root-of-Exotic_Greetings:1.21 start:184.108.40.206 jrandom:1.1.1; locks; strict; comment @ * @;
As with getting a list of tags, this is not implemented in the most current version of CVS, but it's highly likely that it will be implemented soon. I imagine the command will be called cvs list with a short form of cvs ls, and it probably will both parse the modules file and list the repository subdirectories.
In the meantime, examining the CVSROOT/modules file (either directly or by running cvs checkout -c) is probably your best bet. However, if no one has explicitly made a module for a particular project, it won't show up there.
Sometimes there's a problem in the communication between the client and the server. If so, it's a bug in CVS, but how would you go about tracking down such a thing?
CVS gives you a way to watch the protocol between the client and server.
Before you run the command on the local (working copy) machine, set the
CVS_CLIENT_LOG. Here's how in Bourne shell
floss$ CVS_CLIENT_LOG=clog; export CVS_CLIENT_LOG
Once that variable is set, CVS will record all communications between client and server in two files whose names are based on the variable's value:
floss$ ls CVS/ README.txt a-subdir/ b-subdir/ foo.gif hello.c floss$ cvs update ? clog.in ? clog.out cvs server: Updating . cvs server: Updating a-subdir cvs server: Updating a-subdir/subsubdir cvs server: Updating b-subdir floss$ ls CVS/ a-subdir/ clog.in foo.gif README.txt b-subdir/ clog.out hello.c floss$
The `clog.in' file contains everything that the client sent into the server, and `clog.out' contains everything the server sent back out to the client. Here are the contents of clog.out, to give you a sense of what the protocol looks like:
Valid-requests Root Valid-responses valid-requests Repository \ Directory Max-dotdot Static-directory Sticky Checkin-prog Update-prog \ Entry Kopt Checkin-time Modified Is-modified UseUnchanged Unchanged \ Notify Questionable Case Argument Argumentx Global_option Gzip-stream \ wrapper-sendme-rcsOptions Set expand-modules ci co update diff log add \ remove update-patches gzip-file-contents status rdiff tag rtag import \ admin export history release watch-on watch-off watch-add watch-remove \ watchers editors init annotate noop ok M ? clog.in M ? clog.out E cvs server: Updating . E cvs server: Updating a-subdir E cvs server: Updating a-subdir/subsubdir E cvs server: Updating b-subdir ok
The clog.in file is even more complex, because it has to send revision numbers and other per-file information to the server.
There isn't space here to document the client/server protocol, but you
can read the
cvsclient Info pages that were distributed with CVS
for a complete description. You may be able to figure out a good deal
of it just from reading the raw protocol itself. Although you probably
won't find yourself using client logging until you've eliminated all of
the other possible causes of a problem, it is an invaluable tool for
finding out what's really going on between the client and server.
Email an accurate and complete description of your problem to firstname.lastname@example.org, the CVS discussion list. Its members are located in many different time zones, and I've usually gotten a response within an hour or two of sending a question. Please join the list by sending email to email@example.com, so you can help answer questions, too.
CVS is far from perfect -- if you've already tried reading the manual and posting a question on the mailing list, and you still think you're looking at a bug, then you probably are.
Send as complete a description of the bug as you can to firstname.lastname@example.org (you can also subscribe to that list; just use email@example.com instead). Be sure to include the version number of CVS (both client and server versions, if applicable), and a recipe for reproducing the bug.
If you have written a patch to fix the bug, include it and mention on the subject line of your message that you have a patch. The maintainers will be very grateful.
(Further details about these procedures are outlined in the node BUGS in the Cederqvist manual and the file HACKING in the source distribution.)
Same as with a bug: Send the patch to firstname.lastname@example.org. Make sure you've read over the HACKING file first, though.
The troubleshooting techniques and known bugs described in this chapter are accurate as of (approximately) CVS Version 1.10.7. Things move fast in the CVS world, however. While I was writing the last few chapters, the unofficial mantle of CVS maintainership passed from Cyclic Software to SourceGear, Inc (http://www.sourcegear.com), which purchased Cyclic. SourceGear has publicly announced its intention to take an active role in CVS maintainer-ship and has received Cyclic's approval, which is more or less enough to make it the "lead maintainer" of CVS as of right now. (The http://www.cyclic.com address will continue to work, however, so all of the URLs given previously in this book should remain valid.)
SourceGear is, at this very moment, busy organizing and cleaning up various patches that have been floating around, with the intention of incorporating many of them into CVS. Some of these patches will probably fix bugs listed previously, and others may afford new troubleshooting tools to CVS users.
The best way to stay up to date with what's going on is to read the NEWS file in your CVS distribution, watch the mailing lists, and look for changes to the Cederqvist manual and the online version of this book (http://cvsbook.red-bean.com).
Go to the first, previous, next, last section, table of contents.