IO

Interactive Software is Good Software

Interactivity

• Software does computation on data and returns data
• The more data we provide to our application, the more relevant its result becomes

What We Want

• Means for our application to communicate
• Compatibility with various devices
• Responsiveness (Non-blocking I/O)
• Performance (fast transfer, I/O multiplexing)

• Creates a new session to communicate with the file
• Returns a file handle - used to reference the communication channel

File Descriptor Table

Open File Structure

• Contains:
  • Permissions
  • The offset inside the file
  • Number of handles referencing it
  • inode (pointer to data and metadata)
• The OS keeps track of all Open File Structures and stores them in the Open File Table

File Interface - read()

• Uses the file handle to read bytes
• Return number of read bytes
  • -1 - read failed
  • 0 - EOF reached
  • < num - partial read

File Interface - write()

• Uses the file handle to write bytes
• Return number of written bytes
  • -1 - write failed
  • < num - partial write

• Decrements the reference count in the Open File Structure
  • Deletes the Open File Structure if the reference count is 0
• Flushes OS buffers

• Uses the file handle to update the offset in file
• Returns the new offset in file
  • -1 on error

Sparse file

What happens if the offset is bigger than the file size?

• demo/file-interface/sparse-file.c

• Truncates the file indicated by the file handle to the indicated length
• If the file size is smaller than length, the file is extended and "filled" with binary zeros to the indicated length

ftruncate() demo

• demo/file-interface/truncate.c

File Interface in C

• demo/file-interface/c-file-ops.c

• Results in two file handles that refer the same Open File Structure
  • Duplicates a file descriptor into the smallest unused file descriptor

open() vs dup() demo

• demo/file-interface/open-vs-dup.c
• demo/file-interface/close-stdout.c
  • What could go wrong between closing stdout and calling dup?
  • "Everything."

File Interface - dup2()

• Duplicates a file descriptor into a designated file descriptor
• If the new file descriptor is open, it will be closed before being reused
  • This action will be performed atomically

How does the OS operate them?

Representing I/O Devices

Linux abstracts an I/O device as a special file

Device Special Files

• Device files are in /dev
• The letter before permissions describes the file type

student@OS:~$ ls -l /dev/
crw-rw-rw-   1 root root      1,   3 nov  6 20:31 null
crw-rw-rw-   1 root root      1,   8 nov  6 20:31 random
brw-rw----   1 root disk      8,   1 nov  6 20:31 sda1
brw-rw----   1 root disk      8,   2 nov  6 20:31 sda2
brw-rw----   1 root disk      8,   5 nov  6 20:31 sda5
brw-rw----   1 root disk      8,   6 nov  6 20:31 sda6
crw-rw-rw-   1 root tty       5,   0 nov  6 20:31 tty
crw-rw-rw-   1 root root      1,   9 nov  6 20:31 urandom
crw-rw----+  1 root video    81,   0 nov  6 20:31 video0
crw-rw----+  1 root video    81,   1 nov  6 20:31 video1
crw-rw-rw-   1 root root      1,   5 nov  6 20:31 zero
...

Device Types demo

• demo/devices/read-from-device.sh

If not friend, why friend shaped?

Software Stacks All Over Again

• The file abstraction and file interface are only the middle part
• The overlay is a communication protocol describing how to encode/decode data
• The underlay is the driver interface communicating with the device through control codes

ioctl() - I/O Control

• General purpose interface that uses control codes to communicate directly with the device driver
• Might be used for every IO operation
• Not intuitive: requires knowledge on how the device communicates
• Not portable: arguments depend on the OS version and the device

ioctl() demo

• demo/devices/hwaddr-ioctl.c
• demo/devices/filesystem-size.c

read() from /dev/null

static ssize_t read_null(struct file *file, char __user *buf,
            size_t count, loff_t *ppos) {
    return 0;
}

write() to /dev/null

static ssize_t write_null(struct file *file,
      const char __user *buf, size_t count, loff_t *ppos) {
    return count;
}

read() from /dev/zero (simplified)

static ssize_t read_zero(struct file *file, char __user *buf,
             size_t count, loff_t *ppos) {
    size_t cleared = 0;

    while (count) {
        size_t chunk = min_t(size_t, count, PAGE_SIZE);
        size_t left = clear_user(buf + cleared, chunk);

        cleared += chunk;
        count -= chunk;
    }

    return cleared;
}

write() to /dev/zero

static ssize_t write_zero(struct file *file,
      const char __user *buf, size_t count, loff_t *ppos) {
    return count;
}

IPC - File Interface

• Writing to disk and reading from disk is tedious

IPC - Pipe

• Same idea as above

• Proc1 writes at one end
• Proc2 reads from the other end
  • Why is it better?
  • The pipe is stored in Kernel Space, we no longer use the disk
• Only works for related processes (fork)

Pipe - Walkthrough

Pipe - Walkthrough

Pipe - Walkthrough

Pipe - Walkthrough

Pipe - Walkthrough

pipe() demo

• demo/IPC/pipe.c

mkfifo() demo

• demo/IPC/fifo.c

Socket

• The endpoint in the inter-process communication
• Uniquely identifies the process in the communication
• Supports multiple transmission types
  • e.g.: stream, datagram

Socket Interface - socket()

• Creates a socket with given domain and type
• Returns a file descriptor
  • Compatible with read()/write() operations
  • Does not support fseek

Socket Attributes

• Domain
  • AF_UNIX for communication on the same host
  • AF_INET for communication over the internet
• Type
  • Stream establishes a reliable connection and ensures all data is transmitted
  • Datagram sends data faster without checking its integrity

Stream and Datagram

Server - socket()

• Create socket

Server - bind()

• Assign address to the socket

Server - listen()

• Mark the socket passive - will be used to accept connections

Server - accept()

• Wait for a connection
• Accept the connection and create a socket for it

Client - socket()

• Create a socket

Client - connect()

• Establish connection with server

send()

• Use socket to send bytes
• Return number of sent bytes
  • -1 - send() failed
  • < num - partial send()

recv()

• Use socket to receive bytes
• Return number of received bytes
  • -1 - recv() failed
  • 0 - connection end (similar to EOF)
  • < num - partial recv()

Unix Socket demo

• demo/IPC/unix_socket_server.c
• demo/IPC/unix_socket_client.c

Stream Socket demo

• demo/IPC/stream_socket_server.c
• demo/IPC/stream_socket_client.c

Datagram Socket demo

• demo/IPC/datagram_socket_server.c
• demo/IPC/datagram_socket_client.c

stdout vs stderr

• demo/optimizations/stdout-vs-stderr.c

Investigate

student@os:~/.../demo/optimizations$ ./stdout-vs-stderr
Join the dark side!
Hello from the other side!

• Not what we expected

strace --trace=write ./stdout-vs-stderr

Observations

• Each print to stderr results in a write()
• Prints to stdout result in one single write()

Behind The Scenes

• Printing to stdout is buffered to avoid multiple context switches
• The same is not true for stderr as we want to see errors as they occur

Does It Work?

• Attempt to write 10000 bytes to a file, 1 byte at a time
  • demo/optimizations/fwrite-10000.c
  • demo/optimizations/write-10000.c

Can We Do More?

Context

• libc buffering reduces the number of context switches
• But synchronizing the disk after every write() is a major bottleneck

Kernel Buffer

• Same as idea as libc
• Each write() fills a buffer in kernel space
• Transfer data to disk when
  • Buffer is full
  • fsynk() is called
  • Enough time has passed since the last write()

student@os:~$ cat /proc/sys/vm/dirty_expire_centisecs
3000

Overview

Does It Work?

• Write 10000 bytes to disk, 1 byte at a time - with buffering
  • demo/optimizations/write-10000.c
• Write 1000 bytes to disk, 1 byte at a time - without buffering
  • demo/optimizations/write-1000-unbuf.c

Obvious Problems

• We have an intermediary buffer
  • Data from read buffer is copied to application buffer
  • Data from application buffer is copied to socket buffer
• We perform two context switches

zero-copy

• sendfile() instructs kernel to copy data from one of its buffers to another
• We perform a single context switch
• Data is copied a single time

Non-Blocking IO

• O_NONBLOCK makes operations return rather than block
  • SOCK_NONBLOCK for sockets
• Allows handling of input on multiple file descriptors
  • Does not scale with number of file descriptors
  • The thread is busy waiting instead of entering WAITING state
  • Ugly

epoll_create()

epoll_ctl()

epoll_ctl()

epoll_ctl()

epoll_ctl()

epoll_ctl()

Trivial Server

• Run server in a loop
• demo/optimizations/server.py

Good Old Processes

• Create a process to handle each new client
• demo/optimizations/mp_server.py

Good Old Lightweight Processes

• Create a process to handle each new client
• demo/optimizations/mt_server.py

• The python asynchronous interface abstracts a lot of the functionality
• The idea behind:
  • The server socket is added to an epoll instance
  • The specified handle is called for each event (new connection)