Learning Journal - Week 33

Overview

We are now on Week 7 of my studies at CST-334, which looks to be my final module barring the Final Exam. In addition to recapping the final week, I also want to take a moment and begin by reflecting back on the course overall.

Going into the course, back when I first saw the name of the course was 'Operating Systems', I must be honest and say that I was expecting a course focused on navigating Linux as an operating system, learning shell commands, and how to interface with the Operating System. What I learned instead, however, ended being way more intriguing than what I was originally anticipating. While we didn't necessarily tackle the user-oriented side of operating systems that I was expecting, I ended up with a new-found appreciation and deep understanding of the kernel, memory management, process scheduling, and I/O interfacing that operates under the hood. 

Something else that I wasn't expecting was that learning about how kernels power our machines explained a lot of missing pieces in my computer science journey so far. I learned a lot of the why's behind certain design choices in programming, as well as a lot of connections between Operating Systems and networked systems. For example, learning about process scheduling taught me the why behind C/C++ giving us the power to multi-thread. Learning about race conditions taught me the why behind locks, and also shows that we often share solutions to problems in computer programming. Locks aren't only found in Operating Systems but also in database design; same with caching. 

Especially as we begin to wrap up our team project, I begin to appreciate more and more the research and commitment behind making hardware work with our software. The operating system truly is the driving force of all software, and without a kernel to handle a process then a program is just as good as a text file sitting on a hard drive. I'm glad that this course exposed me to such a fascinating world I never knew existed, and I look forward to actually continuing to learn more about how our Operating Systems operate.

I owe it to our professor, Dr. Sam Ogden, for helping provide this understanding and delivering easy-to-follow lectures, as well as being readily available and open to explain and elaborate on concepts. Had it not been for him and his shown enthusiasm and energy in the lecture videos, I don't think the material would have stuck as well. My biggest regret in this class is not making the time to join office hours and learning as much as I could. My greatest appreciation goes to our professor.

 Weekly Recap

This past week of learning went over how the operating system manages and handles I/O, as well as the layout of file systems. While most of the course had gone over how Operating Systems work with short-term information, such as memory and process management, we went over interface with external-devices and working with long-term storage solutions.

 The module began with an overview of I/O devices and separated all I/O devices as either block devices or character/stream devices. The name describes the method on which the I/O actually sends information, so a block device (like a hard drive) transmits data in fixed-size blocks of data, usually with a block size between 512 bytes to 32KB. Stream devices on the other hand, like keyboards and printers, transmit a stream of bytes and don't have addressable information, unlike block devices. More interestingly, however, is the fact that every single I/O device acts like its very own computer, consisting of a small central processor, memory, and special purpose chips. It has registers to see its status, issue commands, or pass data to/from the device. The computer can either communicate with this I/O by polling -- a method we've seen before that sits in a while loop until necessary information comes in -- by using hardware interrupts, or by using another piece of hardware for Direct Memory Access to bypass the CPU entirely.

 After this we began our dive into one of the biggest missing pieces from our system thus far: the file system. First learning about Hard Disk Drives (HDDs). We started by understanding the anatomy of the drive itself -- consisting of multiple layers of spinning two-sided platters, each side with an actuator arm that navigates the tracks and sectors of the spinning plate. This is a very familiar design to anyone that's seen a record player in action before, and shows the philosophy of repeating proven engineering. With the physical aspect understood, we could either interpret the information using a coordinate-style system of Cylinder, Head, Sector, or the more modern approach of using logical block addressing to map all the information in a linear fashion, similar to what we've seen with memory addresses. The specifications of a hard drive reveals a lot of information as well. The speed in which we can access information is limited by the mechanical movement speed of the drive, as well as the processing time. We can use the RPM rotation speed to figure out how long it takes on average for the platter to spin to our desired sector, and we can use the seek time and transfer rate as well to see the total amount of time it takes to write data.

As far as the software interface is concerned, learning about files and directories was a natural next step. We needed a system that offered high performance, reliability, protection, and a meaningful naming scheme. In the end, we ended with the modern file system design philosophy. Of course, there are many types of file system formats, but there are some similarities. In our example, there are two key abstractions: files and directories (folders). Each carries an identification number, called the inode number, but represent different information. A file is simply a linear array of bytes, but a directory keeps a list of pairs of (user-readable name, inode). We can interact with files and directories using their related APIs: open(), read(), write(), unlink(), etc. for files, and rmdir(), mkdir(), opendir(), etc. for directories. We also get introduced to the importance of these hard links, as well as symbolic links, which tell the OS how to map the file system, and how a hard drive is split into partitions with file structures, as well as how volumes could be used to organize collections of same-file-system partitions/drives.

 Finally this week we discussed how to implement a File System, which outlines the overall structure of how our operating system interacts with the hard drive to learn about the layout of its contents to better navigate through files and directories. We studied the Very Simple File System which splits up the disk into blocks of data, and provides metadata to tell us which blocks data is stored as well as the allowed privileges of said data. The Superblock is the very first block the OS encounters and lets it know which file format it is, how many inodes and data blocks there are, and where the inode list begins. Following this information, we have a bitmap to tell us which i-nodes are active, as well as a bitmap to let us know which data blocks are in use. Overall, this information is very quickly accessible very early into the drive so that the drive won't have to traverse all of its data.

 

 Reflection

The content from this module was admittedly one of the hardest for me to digest. I understand how drives operate on a conceptual level, and I feel confident translating disk address and looking at bitmaps and inodes. I found it incredibly natural to explain how the hard drive is broken down, There are some aspects that I still don't entirely understand, and the quiz from this week really felt like a challenge. I always learn more from the video lectures than I do from my independent reading as I get to see the information put into practice and elaborated on, and for some reason I struggled when it came time to walk through resolving a path given a path hierarchy as well as remembering all what functions like open and dup do.

Something that also feels natural up to this point is working with bitmaps as it's something I've been working with since my previous database course. When we were designing database schemes from scratch in Python, there was a point where we used bitmaps to represent valid and invalid data. Because of this, each time I see a bitmap used again in this class I feel like it's solidified as an easy and low-storage cost solution for saving this kind of information.