Lecture Identity

File/lecture name: The shell and bash (Rob Hackman, Winter 2025, University of Alberta) Main theme: Introduction to Operating System interfaces, Shell capabilities, and I/O redirection mechanisms. Prereqs assumed by slides: Basic familiarity with file systems, command line usage, and hardware resources. What this lecture enables you to do:

• Understand the fundamental purposes of an operating system (resource management, security, file access).
• Define and utilize a Shell as a textual interface for system tasks.
• Chain programs together using pipes to process data streams.
• Redirect standard error and input streams to manage program output and file sources.
• Read and modify file execute/read/write permissions with ls -l and chmod.
• Connect to remote systems using ssh username@serverAddress.
• Write and execute basic Bash scripts using shebang, conditionals, and loops.
• Locate and execute system utilities using the PATH environment.

Big Picture Map (High-Level)

1. Operating System Fundamentals: Review of OS purposes (hardware management, security, file access).
2. Shell Definition: Introduction to the Shell as a textual interface for tasks (run programs, manage files, terminate programs).
3. Program Execution and PATH: How command lookup works and when full file paths are needed.
4. File Permissions: Read/write/execute bits, ls -l, and permission changes with chmod.
5. Interpreters and Executable Scripts: Shebang basics and why execute permission is required.
6. Working Over a Network: SSH fundamentals and the ssh username@serverAddress command pattern.
7. Bash Scripts: Script structure, simple conditionals, and for loops for automation.
8. Variable Handling: Best practices for temporary files (mktemp) and variable expansion ($() vs $).
9. Pipes: Mechanism for passing output of one program as input to another (Unix philosophy of small single-purpose programs).
10. Standard Streams: Overview of stdin, stdout, and stderr default connections.
11. Redirection: Overriding default connections using > and 2> to write to files.
12. Input Methods: Comparing command line arguments vs. input redirection (<).

Key Concepts & Definitions

Shell

• Definition: A textual interface allowing users to ask the operating system to perform tasks.
• Why it matters: It acts as the bridge between the user and the OS kernel, enabling program execution and file management.
• Common confusion: Confusing the Shell (interface) with the Operating System (kernel/hardware manager).
• Tiny example: ls (Run a program via shell).

Pipe (|)

• Definition: A mechanism to provide the output of one program as the input to another.
• Why it matters: Enables chaining small, single-purpose programs to solve complex tasks (Unix philosophy).
• Common confusion: Thinking pipes connect files directly; they connect program streams.
• Tiny example: head -20 alice.txt | wc.

Standard Error (stderr)

• Definition: A default stream for error messages, distinct from standard output (stdout).
• Why it matters: Allows error messages to be captured separately from normal program output.
• Common confusion: Assuming all output goes to the screen; errors can be redirected independently.
• Tiny example: wc 2> errorFile.txt.

PATH

• Definition: A variable indicating where the system looks for executable programs.
• Why it matters: Determines where commands like wc or ls are located on the machine.
• Common confusion: Thinking commands are magic; they are utilities provided with the OS (e.g., Ubuntu).
• Tiny example: wc alice.txt.

Input Redirection (<)

• Definition: Feeding a file as input to a command instead of typing arguments.
• Why it matters: Allows processing of large files without passing them as arguments.
• Common confusion: Output difference between argument and redirection (slides note output is "slightly different").
• Tiny example: wc < /usr/share/dict/words.

Core Mechanics / How-To

How to use a Pipe

1. Identify the command that produces the data (left side).
2. Identify the command that consumes the data (right side).
3. Combine them with the pipe character |.
4. Syntax: cmd1 | cmd2 | cmd3.
5. Decision Point: Ensure the left command writes to stdout and the right reads from stdin.

How to Redirect Standard Error

1. Identify the command producing errors (e.g., wc).
2. Use 2> followed by the target filename.
3. Syntax: command 2> filename.
4. Warning: The operating system creates the file; it will overwrite (delete) any existing file with that name.

How to Read from a File as Input

1. Identify the command expecting input.
2. Use < followed by the filename.
3. Syntax: command < filename.
4. Note: This differs from passing the filename as an argument (e.g., wc file.txt vs wc < file.txt).

How to Handle Temporary Files Safely

1. Use mktemp to create a temporary file.
2. Store the result in a variable (e.g., myTemp).
3. Use $() when running the command to capture output into the variable.
4. Use $ when reading the variable later.
5. Crucial Step: Ensure you delete the temp file with rm to avoid clutter.

Code Patterns & Idioms

Pattern: Command Chaining

• When to use it: When the output of one tool is needed as input for another.
• Correct minimal example: head -20 alice.txt | wc.
• Common bug: Forgetting the pipe character, resulting in two separate commands running sequentially.

Pattern: Error Redirection

• When to use it: When you want to log errors to a file instead of the terminal.
• Correct minimal example: wc 2> errorFile.txt.
• Common bug: Using > instead of 2>, which redirects standard output (data) instead of errors.

Pattern: Input Redirection

• When to use it: When a command expects input from a file rather than arguments.
• Correct minimal example: wc < /usr/share/dict/words.
• Common bug: Confusing this with wc /usr/share/dict/words (argument passing), which may yield different output behavior.

Pitfalls, Edge Cases, and Debugging

1. File Overwrite: 2> filename creates the file if it doesn't exist, but overwrites it if it does.
   • Symptom: Data loss in filename.
   • Cause: Existing file with same name.
   • Fix: Check if file exists before redirecting or use >> (append) if appropriate (not covered in slides, but implied risk).
2. Variable Expansion: Using $ vs $() changes how output is captured.
   • Symptom: Command fails or variable is empty.
   • Cause: Incorrect expansion syntax for command substitution.
   • Fix: Use $() for command output, $ for variable content.
3. Temp File Cleanup: Failing to delete mktemp files.
   • Symptom: Disk space usage increases over time.
   • Cause: rm command omitted.
   • Fix: Always run rm on the temp variable.
4. Input Method Difference: wc file vs wc < file.
   • Symptom: Output is "slightly different" (as per slides).
   • Cause: Argument parsing vs stream redirection differences.
   • Fix: Understand which method the specific utility expects.
5. Standard Stream Confusion: Mixing up stdout and stderr.
   • Symptom: Errors appear on screen instead of file.
   • Cause: Using > instead of 2>.
   • Fix: Use 2> for errors specifically.
6. PATH Issues: Commands not found.
   • Symptom: command not found.
   • Cause: Program not in PATH or not installed.
   • Fix: Check PATH or install utility (e.g., Ubuntu utilities).
7. Pipe Chain Breakage: One command in a chain fails.
   • Symptom: Downstream commands receive no input.
   • Cause: Upstream command error or empty output.
   • Fix: Check exit status of upstream commands.
8. File Permissions: OS manages access to resources.
   • Symptom: Cannot read/write files.
   • Cause: OS security environment.
   • Fix: Check file permissions (covered in "Executing Programs and File Permissions" section).
9. Variable Scope: Shell variables vs environment variables.
   • Symptom: Variable not accessible in subshells.
   • Cause: Local vs global scope.
   • Fix: Export variables if needed (not explicitly covered, but implied by "environment").
10. Redirection Overwrite Risk: Overwriting critical system files.
    • Symptom: System error.
    • Cause: Redirecting to a system file path.
    • Fix: Use unique filenames for redirection.
11. Input Stream Closure: Closing stdin prematurely.
    • Symptom: Program waits for input.
    • Cause: Input redirection not set up.
    • Fix: Use < to provide input source.
12. Command Substitution Syntax: Mixing $ and $().
    • Symptom: Syntax error or empty variable.
    • Cause: Using $ for command output.
    • Fix: Use $() for command output.
13. Temp File Persistence: Leaving temp files behind.
    • Symptom: Disk full.
    • Cause: mktemp result not deleted.
    • Fix: rm the variable content.
14. Standard Error Capture: Capturing errors to wrong stream.
    • Symptom: Errors mixed with output.
    • Cause: Redirecting 1> instead of 2>.
    • Fix: Use 2> for errors.
15. File Creation Behavior: OS creates file on redirect.
    • Symptom: Unexpected file creation.
    • Cause: 2> creates file if missing.
    • Fix: Be aware of side effects of redirection.

Exam-Style Questions (with answers)

1. Concept Check: What is the primary purpose of a Shell?
   • Answer: To provide a textual interface for users to ask the OS to perform tasks (run programs, manage files).
2. Predict-the-output: What happens if you run wc 2> errorFile.txt?
   • Answer: Standard error from wc is sent to errorFile.txt instead of the screen. The file is created/overwritten.
3. Trace Reasoning: Why did we store the result of mktemp in a variable?
   • Answer: To reference the temporary file path later (e.g., for deletion or processing).
4. Bug Finding: Why is wc < file different from wc file?
   • Answer: The slides note the output is "slightly different" between input redirection and command line arguments.
5. Design Question: How do you chain three commands to process data?
   • Answer: Use cmd1 | cmd2 | cmd3.
6. Concept Check: What does the pipe character | do?
   • Answer: It redirects the standard output of the left command to the standard input of the right command.
7. Predict-the-output: What happens to an existing file named errorFile.txt when using 2> errorFile.txt?
   • Answer: It is overwritten (deleted) by the operating system.
8. What goes wrong here? What is the issue with wc $myTemp vs wc $()myTemp?
   • Answer: $() is for command substitution (running a command), $ is for variable expansion. Using $() without a command is invalid.
9. Design Question: Why does Unix advocate for small single-purpose programs?
   • Answer: To combine them with other programs to solve many tasks using pipes.
10. Concept Check: Where do Ubuntu utilities typically exist?
    • Answer: Somewhere on the machine, located via the PATH.
11. Trace Reasoning: If head -20 alice.txt | wc is run, what is the input to wc?
    • Answer: The first 20 lines of alice.txt (stdout of head).
12. Bug Finding: Why might a temp file persist on disk?
    • Answer: The rm command was not executed to delete the file created by mktemp.

Quick Reference / Cheat Sheet

• Pipe: cmd1 | cmd2 (Output of 1 -> Input of 2).
• Stdout Redirection: cmd > file (Write output to file).
• Stderr Redirection: cmd 2> file (Write errors to file).
• Input Redirection: cmd < file (Read input from file).
• Command Substitution: $() (Capture command output into variable).
• Variable Expansion: $var (Use stored value).
• Temp File: mktemp (Create), rm (Delete).
• Unix Philosophy: Write small programs, combine with pipes.
• File Overwrite: 2> file creates/overwrites file.
• PATH: Location of system utilities.

Mini Glossary

1. Shell: Textual interface to the operating system.
2. Pipe (|): Connects stdout of one program to stdin of another.
3. Stdin: Standard input stream.
4. Stdout: Standard output stream.
5. Stderr: Standard error stream.
6. Redirection: Overriding default stream connections.
7. PATH: Environment variable for program locations.
8. mktemp: Command to create temporary files.
9. Variable: Storage for data values in the shell.
10. Utility: Small single-purpose program (Unix philosophy).
11. Overwrite: Deleting existing file content when creating a new one.
12. Command Substitution: Capturing output into a variable.
13. Input Redirection: Feeding a file as input to a command.
14. Operating System: Manages hardware, security, and files.
15. File Permissions: Controls access to resources.
16. Ubuntu: The operating system used in this course context.
17. CMPUT 274: Course context mentioned in slides (ignored for content).
18. Terminal: Interface for shell commands.
19. Stream: Data flow channel (stdin/stdout/stderr).
20. Argument: Data passed to a command.

What This Lecture Does NOT Cover (Boundary)

• C++ Programming: Not covered in these slides (Course description mentions C++, but slides focus on Shell).
• Object-Oriented Programming: Not covered in these slides.
• Graph Algorithms: Not covered in these slides.
• Dynamic Programming: Not covered in these slides.
• Client-Server Computing: Not covered in these slides.
• Recursion: Not covered in these slides.
• Advanced Bash Scripting: Only basic commands and redirection are covered.
• Bash Robustness Features: Strict mode, traps, and advanced script debugging are not covered.
• Secure File Transfer Tools: Tools like scp and rsync are not covered.

Slide Anchors (Traceability)

• OS Purposes: (Citation 1, Section 2) "Manage and provide interfaces for access to your resources (physical hardware) ... Provide environment to run programs safely and securely".
• Shell Definition: (Citation 1, Section 3) "A shell is a textual interface to allow users to ask their operating system to perform tasks".
• Shell Tasks: (Citation 1, Section 3) "Run a program, Create, read, or delete a file, Terminate a program".
• mktemp Variable Usage: (Citation 2, Section 32) "Why did we have to store the result of mktemp in a variable? ... Note that we made sure to delete the temp file we created with rm!".
• Variable Expansion: (Citation 2, Section 32) "Why did we use $() when running the mktemp command but $ when reading myTemp?".
• Pipes Concept: (Citation 2, Section 33) "Needing to provide the output of one program as the input to another is a common problem".
• Unix Philosophy: (Citation 2, Section 33) "advocate for writing small single-purpose programs which can then be combined with other programs".
• Pipe Syntax: (Citation 2, Section 33) "The general form of this is $ cmd1 | cmd2".
• Pipe Behavior: (Citation 2, Section 33) "The standard output of a command on the left-hand side of a pipe will be redirected as the standard input to the command on the right-hand side".
• Standard Error Redirection: (Citation 3, Section 16) "Redirect stderr with a two followed by the greater than sign (2>) followed by the filename".
• Stderr File Creation: (Citation 3, Section 16) "The operating system also creates the file errorFile.txt, which will overwrite (delete) any existing file".
• Input vs Arguments: (Citation 3, Section 16) "Consider the following two commands $ wc /usr/share/dict/words $ wc < /usr/share/dict/words ... output is slightly different".
• PATH: (Citation 2, Section 35) "The programs we've used so far have been utilities provided with Ubuntu and are located somewhere on our machine".