Course 3 - Graduate Operating Systems

Brief Overview

This course focuses on understanding how operating systems are able to abstract and arbitrate the use of a computer system. In general it sits between managing user applications, hardware, and managing systems despite the complexity and diversity of hardware components.

In addition the course dives into three major projects that teach concepts on the following ideas:

Develop a multi-threaded client server communication protocol from scratch using sockets, TCP protocol, and the ability to send and receive any file type.
Create two processes which are able to pass information across shared memory space using inter process communication
Using gRPC develop a multi-threaded client server able to pass files using HTTP2 and a .proto file.

Detailed Overview

Project Overview

Class was broken into two parts: Lectures and Projects.

Lectures begin by focusing on a large overview of the purpose of an operating system and how it abstracts and adheres to policies as well as utilizes mechanisms to manage processes.
Projects focus on building a multi-threaded client server protocol to transfer files, an inter process communication client server which is able to pass data across two separate processes, and a gRPC client server able to allow multiple clients to make changes to read and make changes to a file in real time.

Project 1: Multi-Threaded GETFILE Protocol

This project involved implementing a custom file transfer protocol and building a multi-threaded client-server system in C. The main technologies used were C programming, pthreads, TCP sockets, mutex/condition variables, file I/O, protocol design.

Part 1: Echo/Transfer Server focuses on building a client-server application using TCP sockets with the following features.

Echo server/client for basic request/response communication.
File transfer server/client with chunked data transmission.
Socket setup using getaddrinfo, IPv4/IPv6 support, and proper error handling.

Part 2: GETFILE protocol in which a custom multi-threaded protocol for file transfer is developed to handle multiple file transfers at once.

Multi-threaded Server

Boss thread: Continues listening and accepting new clients.
Worker threads: Handles the actual serving operations.
Connections are handed off to workers for processing.

Multi-threaded Client

Boss thread: Enqueues file download requests into work queue.
Worker threads: Process download requests concurrently.
Worker poool initialized at startup based on command line argument.
Boss waits for all requests to complete, then terminates workers.

Key concepts in this project were developing a thread-safe work queue using mutexes. Condition variables for worker synchronization, and how to avoid race conditions on shared data while still allowing concurrent file serving and downloading.

Difficulty: 5/5

Summary: If you're new to C Programming this will be quite difficult. It's a great project and will teach a lot through trial and error. Probably one of the best projects I&aposve completed so far in this program.

Project 3: Inter Process Communication (No Project 2)

This project builds on Project 1 by converting the getfile server into a proxy server with caching capabilities using IPC mechanisms.

Proxy Server with Sockets

Goal of part 1 is to convert the getfile server into a proxy server that translates GETFILE requests into HTTP request for remote servers.

Replaces the disk retrieval with web-based retrieval using libcurl's "easy" C interface.
Implement handle_with_curl() callback function.
Server uses the boss-worker multi-threaded framework created in project 1.

Goal of part 2 is implement a cache daemon that communicates with the proxy via shared memory IPC, avoiding network requests for cached files.

Cache creates and manages this channel.
Used for request/response coordination.
Communicates shared memory segment IDs

Difficulty: 4/5

Summary: Great project in understanding how to setup a cache server that can handle reducing large bandwidth transfers and caching files on a server. My solution didn't work for a large quantity of requests, but I believe it was related to the max a message queue requests could handle.

Project 4: gRPC and Distributed Systems

In part 1 of this project the goal was to implement a basic client server distirbuted file transfer system using gRPC by providing fundamental file operations (store, fetch delete, list, stat) without synchronization, locks, or mult-client coordination. Part 1 is able to handle large file transfers, manage proper error handling, and is not performing any client caching.

Develop the functions using a domain specific language through proto buffers.
Develop a client with the core methods fetch, set, delete, and status
Develop a server with the core methods fetch, delete, list, and status

Part 1 was mostly a matter of following the tutorials on gRPC's web site to understand how gRPC works and then implementing the solutions with it.

Would recommend spending time drawing out the general idea before starting.

Difficulty: 5/5

Summary: Took probably an hour or two to solve. The lectures help in identifying how to create a classification model.