Sysadmin’s plan to manage system config changes backfires spectacularly

Who, Me? Welcome once more to Who, Me?, the column for Reg readers to get their worst deeds off their chest.

This week, "Ryan" tells us about a time many years ago when he got a little bit cocky with root-level commands.

At the time, he was the senior systems and network administrator for a major research lab.

"I administered a number of Sun servers and many Sun client workstations that were used in the lab," he told us.

In an effort to manage system configuration changes on the servers, Ryan started to maintain various text-based configuration files using a software revision control program.

Crucially, this system meant he had to check out a configuration file in write-mode, make a change, check it back in, and then check out a read-only copy of the file.

"I progressively added more operationally sensitive configuration files to the list that I used with revision control," Ryan said.

"At some point I added the file system mount configuration file /etc/fstab to the scheme.”

Ryan said that all went very well for months, until one day, when he was in a hurry. "I made a change to the file system config file, checked it in, and… forgot to check it out in read-only mode again."

But it wasn't until the next time he rebooted the system for maintenance that the stuff hit the proverbial fan. "As one might expect," Ryan said, "Unix does not boot well when it does not have an idea where its filesystems are located on the disks."

After what Ryan described as 15 minutes of sheer panic, he had calmed down enough to begin recovering everything.

"I manually mounted the root filesystem on the bare-bones Unix kernel, and using only the sh shell built-in command echo re-created a bare bones /etc/fstab file on the root filesystem," he said.

"That was sufficient to allow the kernel to actually mount the root file system and /usr filesystem during boot."

After that, Ryan had made it over the worst. "The system was intelligent enough to allow me to check out a read-only full fstab file – and it all worked when I rebooted the system."

But this isn't the first time Ryan has 'fessed up – he told his director and some colleagues at the time.

Luckily the fix won their praise, and he was told to chalk it up as a learning experience.

"From that point on, I was much more diligent about double checking my use of dangerous root-level commands!"

When was the last time your best-laid plans went very awry? Tell Who, Me? and your story might feature in next week's column. ®