The Command Line I: Operating systems and Unix


Although this blog will generally be presenting hands-on, technical tutorials on how to perform various bioinformatic tasks, this first post turned out to be a bit more philosophical than I originally thought it would be. The starting point for a look at Unix is necessarily an explanation of operating systems; what they are, what they do and what they can do for us. In the course of trying to answer these questions I got a little carried away and the result is a first post with almost no technical details. Nonetheless, what I hope this post achieves is to give some context for latter posts and to introduce, at a very basic level, some of the issues that dictate the way we interact with computers the way we do. The other aim is to show that computing doesn’t begin and end with Windows, that there are other ways of doing things, and for a lot of tasks these ways are much more efficient. This blog is aimed at people with limited exposure to Unix, programming and bioinformatics and one of the hardest things to do when learning something like Unix is `unlearning’ the way things are done on your PC and wrapping your brain around a different concept of computing. This can take a while, but once the penny drops things all start making sense. This post is about helping that penny drop.

What’s an operating system?

When beginning to learn a new operating system (hereafter referred to as an OS), especially one as different as Unix is from Window or OS X, a good starting point is to consider exactly what an OS does. For most people OS’s aren’t really something we think about that much. We fire up a computer and Windows or OS X is sitting there waiting for us to open a file, start a game or do whatever else we want to get done. Thats about it for OS’s for most people. Most of us understand that the OS makes things work, but usually it sits in the background while we Tweet, post on Facebook, or, occasionally, work. People have their favourite OS, usually the one they are used to, but apart from reasonably cosmetic preferences, usually, for Windows over OS X, or vice versa, OS’s are not a hot topic for most people. That we are able to use computers in this modern age without even thinking about them is a testament to just how well modern OS’s do their thing and for most intents and purposes this is exactly the way it should be. OS’s exist so we do not have to worry about them, like waiting staff at a very expensive restaurant the best OS’s are conspicuous by not being conspicuous (I’m looking at you Windows Vista when I say this).

This changes though when you start programming. By its nature programming involves doing something that no one has done before. If it has been done before you wouldn’t be writing a program, you would be installing one. No one writes a word processor every time they want to write a memo. If you are trying to achieve something new, or have a novel problem you are trying to solve, and you are writing a program to do this then it is inevitable that you will come to a point where the grittier details of your OS will become important. How do you write something to disk? Can you parallelise your program? What are the security features of the OS? How does the OS handle file permissions? There are many things happening under the hood of your OS and when you start programming these things will inevitably impact on what you are trying to do. This is when some rudimentary idea of what is going on will serve you well, at the very least it will give you a starting point for learning more.

So what is an OS? The simplest way to think of an OS is as a bunch of software that sits between us and the computer hardware. Hardware is pretty stupid and it requires things explained in excruciating detail. The OS represents us to the hardware and translates our vague desires into specific instructions understandable by the hardware. In addition to this the OS plays the traffic cop, coordinating activities so that the hardware isn’t confused by simultaneous incoming requests and manages the hardware resources so individual programs can’t, for instance, consume all available memory. The figure below presents a highly conceptualised view of a computer and lists some of the tasks that the OS is concerned with. In computer speak these activities are described as low-level, the `low’ basically representing closeness to hardware. In mediating between us and the hardware the OS presents us a higher-level abstraction of the hardware. This abstraction frees us from the nitty-gritty of low level issues and lets us get on with all the stuff we normally associate with computing — watching YouTube, Tweeting, assembling genomes and writing papers for publication in high-ranking journals.

Schematic of a computer system.

Overview of a computer system.

This way of organising things is very efficient. One way of understanding just how efficient is to consider what would happen if it things were not done this way. Imagine, for instance, if OS’s did not handle writing something to disk. That is something like saving a word document or a high score in Angry Birds on to the harddrive. Without an OS a programmer would need to incorporate in their program the low level details for writing something to disk. In computer parlance this would mean incorporating a device driver into every program (a device driver being a program that allows communication with a piece of hardware). The consequences of this are more severe than you may be thinking. If we all had to write a device driver every time we wrote a new program the progress of computing would proceed very much slower than what we are used to. We would need to write a different driver for every type of hardrive on the market and we would need to update the driver in every program written anytime a hardrive changed slightly — this would be the end of cheap games for your mobile phone, the cost of maintaining all that code would mean most of us would need a mortgage just to afford Angry Birds. Another important consequence would be drivers of varying quality. Not all device driver programmers are created equal and some programs would have efficient harddrive interfaces and others very inefficient ones. Not only would inefficient drivers slow down the program in which they were embedded, but they would also slow down the computer as a whole, making everything run badly. In a worst case scenario, a badly written driver could wipe the hardrive, so every time you installed a program you would be risking all your data to the driver writing skills of that particular programmer. Now this is a very artificial example, for example one important function of an OS is handling the timing of events and at the least some type of external control would be needed to make sure programs weren’t trying to write to the disk at the same time, but what it does do is highlight the importance of the OS — without the OS computing would be risky (for your data anyway) and so complicated to be available only to the experts.

Another consequence of all these different device drivers would be a collective increase in anger amongst computer users. We are all familiar with the complaints of those unlucky enough to be forced onto a Mac if they are used to Windows, or vice versa. Things are done a little differently and it is annoying and time consuming working out how to do something you know you can do in the blink of an eye on your usual OS. Now imagine if every program on your computer did basic things like saving a file differently; there would not be enough hours in the day to express the outrage. OS’s lower blood pressures by adhering to a philosophy that we will run into time and time again, particularly so in Unix, and one way of expressing this philosophy goes something like “Do something once, properly, and re-use it”. So in our harddrive example if the OS presents one way of achieving this goal, then users need only learn one way of saving their file, instead of the thousand different ways a thousand different programmers could come up to do the same thing. This is so both at the level of the `Save file’ dialogue we are familiar with and also at the less familiar, to non-programmers at least, programmatic calls that are made when writing programs (functions we will become more familiar with in later posts). The OS presents these functions to users and programmers in a consistent manner and, almost as importantly, in a way that doesn’t change, even if the details underlying these methods do change.

This last point is worth considering as from this the other great service of OS’s to mankind springs. As mentioned above if everyone writes their own way of writing to disk then whenever something changes, say a new disk drive comes on the market, then every single program needs to be changed. If the OS is the only way of writing to the harddrive then when a program wants to write something to disk the program merely needs to ask the OS to write something to disk for it. The advantages of this is are obvious; if the hardrive changes, if a bug is discovered or a more efficient way of writing things to disk is discovered only the OS needs to be changed — not every program on the computer. Moreover, as long as the OS is written correctly and efficiently, then a program installed on your computer will not accidentally wipe all your data when trying to save something. There are many, many details that I have left out of this but the basic principle of parsimony, that is only writing new code when there is no other alternative, is one that perfuses computing at all levels. As we will find out in later posts this principle of writing code and reusing it is a very important part of programming.

From this discussion it would appear that OS’s are a panacea, simplifying computing and making sure everything is done the same way. Whats the point of even thinking about the OS if its abstracted all the details into irrelevance? To the a user cruising the internet or checking email this is exactly the case, but when you start programming or scripting, you quickly realise that a modern computer is a system of incredible complexity and although the many of the details are effectively hidden there are still many ways to get different tasks done. Which of the ways you chose to get things done can have profound implications on the efficiency of your program. If your writing a trivial program to automate your Twitter posts the consequences wont be that dire, but if you are writing a program to deal with 100 GB of Illumina reads the methods you use can mean the difference between a program that runs for three hours and one that runs for three weeks. To have an understanding of what is the best way to do something often realies on an understanding of how the underlying hardware operates. The way an OS achieves these wonders is way beyond the scope of this little blog post, but as you become more expert in programming, and start trying to achieve more complex programming goals more efficiently, the details of these processes can become of great importance. As beginners we aren’t going to leap into the details of writing device drivers and for the most part we are going to ignore the underlying details until we need to know them, but it is important to understand that they are there and that we may have to deal with them eventually. A passing familiarity with Figure 1 is a good start for the beginner and it is thanks to OS’s that we can do this, without the humble OS our first steps in learning computing would be learning assembly language and writing device drivers. If computer course started with fascinating topics like these most beginners would stay beginners.

The vision thing

In the previous section we had a look at how the OS provides a uniform interface to the underlying hardware and we touched on the idea of this interfacing representing an abstraction of the hardware. Implicit in this concept of abstraction are decisions made on the way things are to be represented and how the abstraction will provide the means to perform common tasks. The interface the OS represents to the user is the result of this abstraction and it incorporates all the decisions made by the authors of the OS on how users are expected to interact with the OS. Although a particular hardrive is a physical object that will not change, the way that it is interacted with changes based on whether it is attached to a PC or a Mac. One could think that a single abstraction would be enough, then we would all only need to learn one OS, but this is not the case. It is hard getting people to agree on a choice of restaurant let alone a `Save as’ dialogue and so throughout the history of many different people have created many different OS’s.

This is not a bad thing, as we shall see, one size doesn’t fit all and often these different OS’s were, and are, excellent at what they were designed to do. Of course many were also dogs and apart from a brief existence on the hard drives of enthusiasts they passed away without any lasting effect on the history of computing. But whether it is Windows or RISC_OS, all of these OS’s represented someones `vision’ of computing and the way that we should interact with the computer hardware. For example, LISP is a venerable programming language and as a LISP enthusiast I have always been fascinated by LISP machines. These computers used an operating system that was written in LISP and offered total immersion in LISPy goodness. Unfortunately, they didn’t last long, only 7,000 were made during the 1980’s, but they did commercially pioneer concepts that are now common place, including effective garbage collection, laser printing, windowing systems, computer mice, high-resolution bit-mapped graphics, computer graphic rendering, and networking. I have never used a LISP machine, or even seen one in person, but the `vision’ of a LISP machine is very attractive to me. Maybe one day it will make a comeback but I am not holding my breath.

There are many OS’s with a similar history to the LISP machine. A short list of OS would include BeOS, created by Jean-Louis Gassée, an Apple engineer who worked on the original Macintosh, and the OS which almost ended up on Macs instead of OS X, he asked Apple for $200 million so, instead, they brought back Steve Jobs; OS/2, a joint venture between Microsoft and IBM that many considered far superior to Windows but which, some say, died as a result of a betrayal by Windows in the form of Windows 3. Today OS/2 can only be found in some ATMs; CP/M, a very early OS for intel chips that could have been what DOS became if Gary Kildall, its developer, hadn’t turned down IBM who needed an OS for their new PCs. Of course, his loss ended up as DOS’, and Bill Gates, good fortune; NeXT, Steve Jobs’ OS, created for NeXT computers and which, despite NeXT never really taking off, became the basis of OS X; AmigaOS, the OS for Commodores Amiga computers, generally considered far ahead of its time, it incorporated a color screen, four-channel sampled stereo sound, preemptive multitasking GUI, and custom chips to accelerate both sound and graphics. Although putting the Apple Macintosh to shame it never really took off. All of these OS’s basically did the same thing — they provided a bridge between the user space and the hardware, they provided for things like writing to disk or displaying something on a screen or terminal. What made them different was the way in which they presented these functions to the user and in how they allowed the user to interact with the computer. The way in which these operating systems exposed the computer to the user space represented the `vision’ of their creators about the way we should interact with our computers.

The most obvious example of this is the massively different `visions’ represented by Microsoft DOS and the Apple Macintosh during the 1980’s. DOS offered a reasonably limited command line interface that allowed the user to launch programs, perform file system operations and other common computing tasks. The original Macintosh was, and still is, hailed as a breakthrough in computing because it introduced, to the mass market the desktop metaphor of computing that we are all now so familiar with. There was no command line at all in the Macintosh, everything was managed through a graphical interface by a mouse. There could not be two more different visions of how a user was to interact with the computer hardware. Of course the Macintosh way of doing things was the winner but, unfortunately for Apple, it was Microsofts implementation of this vision, Windows, that most people are now familiar with. In recent years mobile computing, i.e the iPad and Android, has started to erode the dominance of the desktop metaphor. In mobile computing the application, not the file, is at the center of the computing experience, on something like the iPad files are associated and opened from within apps and the system gives almost no access to the files themselves. The point of all this is that the OS embodies a way of interacting with a computer. It abstracts out low level functions and in the course of doing so it imposes on a user a predefined set of assumptions on how to achieve certain goals. One metaphor for computing that we haven’t really discussed is the command line. The command line is potentially more foreign to the casual computer user than any of the other OS’s we have discussed. It was the one of the earliest ways of interacting with a computer but it is still an integral part of many scientists and programmers every day life. This interface, and metaphor for computing, will be the major subject of this series of posts, but before getting stuck into the command line we will take a look at the OS that in some sense exemplifies the command line, Unix and its many derivatives.

Learn More

I find the history of computing fascinating and although it wont help you much with programming having some knowledge of the way computers have developed over the years gives you some perspective and also helps you understand why things are done in a certain way. If you want to read some more, as always, Wikipedia is a great resource, see here and here for the entries on Unix and Windows. But also have a look at some of these resources for an interesting read:

Unix and the command line

Like Windows and OS X, Unix is an OS. It was developed in 1969 by a group of AT & T employees working in the famous Bell Laboratories. The original Unix was written in Assembly language for a computer known as the PDP-7, but very early on it was rewritten in the C language. This was very important for Unix as it made it reasonably easy to modify to run on different types of computers (in computer parlance `to be ported’). This portability mean’t that during the 1970’s and 80’s Unix spread through academic and government departments and became established as the de facto OS for computing on main frames and what we often call `super-computers’. The term `Unix’ actually refers to a family of OS’s based on the original AT & T Unix. The Unix History website has charted the, at least, 200 or so variants of Unix that exist and even has a downloadable chart plotting the evolution of all these Unix and Unix-like OS’s. The history of Unix, and the reason why there are so many different flavours, of Unix is beyond this post but it is an interesting story of legal wrangling as various companies tried to claim the intellectual property of Unix. This was a hard thing to do; the earliest forms of Unix had been freely distributed by AT & T due to a restriction on entering the computer industry imposed by a 1958 antitrust settlement and this distribution of the source code encouraged others to tinker and improve the OS. The ability to tinker probably increased the uptake of Unix but it also made it very hard to put the intellectual property cat back in the bag. AT & T learnt this in the 1980’s when it attempted to regain control of Unix after restrictions were lifted and Unix litigation has continued unabated ever since. The SCO group, for instance, was still involved in litigation against companies using Linux as late as 2011.

As most of us in the PC world are familiar with Windows or, to a lesser extent, Apple’s OS X, networking, multi-tasking and multi-user systems are comparatively new concepts. It wasn’t until the mid-90’s (with Bill Gates famous “Internet Tidal Wave” Microsoft memo) that PC’s really started to become networked, the original Mac’s and Windows computers could only run a single program at a time and true multi-user computing, with memory protection and access privileges, was really only introduced in Window XP (although it was in NT but not many people ran this on their home system). Unix, however, was designed from the beginning to be a networking, multi-tasking and multi-user OS. This was a consequence of the fact that Unix was developed specifically for ‘super-computers’, computers shared by large numbers of people in University departments or large companies. Although light-weight by todays standards, these computers were at the time powerful enough to support these tasks and as they were used by multiple people they needed to handle multi-tasking, multiple users and security procedures to stop everyone writing over each others files or hogging system resources. Linux, BSD and other Unices have since jumped over to the personal computer, the most notable example being OS X which is a custom graphical interface over a BSD core that Apple were forced to borrow when their own attempts to write an OS to replace the old Mac OS failed. However, almost all high performance computing facilities use some flavour of Unix, generally Linux, so some familiarity with Unix is good thing for anyone wanting to do bioinformatics.

Unlike Windows and Os X, Unix incorporates a way of interacting with the computer that predates graphical interfaces — the command line. Modern Unices now come with sophisticated graphical user interfaces but they have retained the command line as a primary way of interacting with the computer. Back in the day, at the dawn of computing, there was no monitor and one would communicate with the computer via a teletype with the ‘conversation’ printed to paper. As things progressed this arcane and clunky method gave way to monitors, or terminals as they came to be called, that could display lines of text scrolling up the screen and ultimately to the situation we have today where any position on the screen is addressable by the computer. Directly addressable screen positions, of course, mean graphics and almost all computer interactions are now through a graphical interface. I am not writing this early history from personal remembrances — I’m old, but I am not that old, and to write it I needed to rely on first hand accounts on wikipedia or other descriptions of the teletype interface. For me, apart from the briefest of times in the late seventies and early eighties, computers have always meant graphics, my earliest computers the Apple II and the Sinclair ZX80 being the exceptions. However, unlike the teletype which now exist almost exclusively in computer museums, the terminal has never gone away. In computers running Unix-like OS’s, even OS X, there lurks, often hidden away in a dark corner, a program called a terminal emulator. This program, when started, will present the user with a window containing a terminal just like the ones used back in the days when this was the only way of interacting with your computer. In Windows you can pull up a DOS prompt, which is similar but different from the terminals found in Linux and OS X, but similar principle I suppose. With this terminal you can perform any task that you can with the graphical interface (within reason — its hard to play World of Warcraft only using the command line).

Windows and the PowerShell

It is worth pointing out the Microsoft has incorporated the PowerShell into Windows 7. This shell is a command line shell and associated scripting language built on top of the .Net framework. Although Windows has always had a DOS prompt it has generally possessed a fraction of the power of the various Unix shells. If Microsoft is incorporating a new powerful command line shell in the latest release of its operating system it is clear the command line is still relevant, useful and in demand. Although I am not going to discuss the PowerShell in this guide (because I know nothing about it) I understand that a lot of the commands discussed here have an analogue in the PowerShell, so a useful exercise for those wanting to learn how to use this tool would be to translate this guide into a PowerShell guide.

To many people the command line seems a weird, uncomfortable and backward way of doing things. Have we not developed these elaborate and pretty graphical interfaces so we do not have to type commands at the computer anymore? Why learn how to use a command line when I can easily drag files around with a mouse? Why bother learning how a computer works when I can rely on the abstraction provided by the graphical interface? After all it even has little pictures of files and cool animations. Well the answer to these questions, and the reason that the terminal has never gone away, is that once you learn how to use the terminal you can get more done, more quickly and more elegantly than you can with current graphical interfaces. Once you have climbed, the admittedly steep, learning curve you have an amazing tool at your disposal that will enable you to do some things more efficiently than you can at the moment. Ultimately computer hardware depends on a stream of 0’s and 1’s and unless we want to sit at the computer and manually input a stream of 1’s and 0’s every time we want to watch a video or send an email we need to use an abstraction that takes care of some of this for us. The graphical interface is an abstraction and it is very good at doing some things but pretty poor at doing some other things. This is a consequence of the decisions made in abstracting out the low level chores handled by the OS. The command line is also an abstraction but it represents a set of decisions aimed at making other things very easy, of course other things are poorly at the command line. The typical user of a computer system who wants to send an email, view a video and look up movie times in a browser have no need for the command line, the graphical user interface provides a standard interface for doing well defined tasks such as these. But, if you want more control or need to perform non-standard tasks, particularly repetitive tasks or tasks that depend on filtering input in a sophisticated manner than the command line will make your life much, much easier. Then you go and watch You Tube in the graphical interface with all the extra time that you have.

Learn More

Any Google search will bring up a multitude of pages discussing the command line. Many of the commands and activities you perform on the command line, in the beginning, seem arcane and pointless but there is an ultimate reason for pressing on and becoming proficient in the command line. Hopefully these blog posts will help you but others have done this much better than I have and it is worth having a read of some of these theses.

  • In the beginning was the command line: Neil Stephenson (the famous author of Cryptonomicon, Snow Crash and many other novels) wrote a long essay on the command line and its usefulness. In it he drags in Ronald Reagan, world politics, and the demise of Microsoft. Its a little dated now but still well worth a read.
  • The Command Line – The Best Newbie Interface?: An article on OSnews considering whether the command line is the best thing to teach to people who are first learning computers.
  • Basics of the Unix Philosophy: Believe it or not there is a Unix philosophy. Summarised it states that a program should do only one thing, do it well and provide output that is easily read by another program. This allows the construction of complex programs using a combination of simpler programs. Of course much more has been written on this philosophy and this is one example.



Powered by Facebook Comments

Comments are closed.