Process Memory

Memory Regions

To better manage a program's memory, the operating systems creates an address space for each process. The address space is compartmentalized in multiple areas, each with its own role. Memory addresses use different permissions to decide what actions are allowed.

Let's investigate the memory areas of a given process. We use pmap to see the memory layout of a running process. The command below shows the memory layout of the current shell process:

student@os:~$ pmap -p $$
1127:   /bin/bash
000055fb4d77d000   1040K r-x-- /bin/bash
000055fb4da80000     16K r---- /bin/bash
000055fb4da84000     36K rw--- /bin/bash
000055fb4da8d000     40K rw---   [ anon ]
000055fb4e9bb000   1604K rw---   [ anon ]
00007f8fcf670000   4480K r---- /usr/lib/locale/locale-archive
00007f8fcfad0000     44K r-x-- /lib/x86_64-linux-gnu/libnss_files-2.27.so
00007f8fcfadb000   2044K ----- /lib/x86_64-linux-gnu/libnss_files-2.27.so
00007f8fcfcda000      4K r---- /lib/x86_64-linux-gnu/libnss_files-2.27.so
00007f8fcfcdb000      4K rw--- /lib/x86_64-linux-gnu/libnss_files-2.27.so
00007f8fcfcdc000     24K rw---   [ anon ]
00007f8fcfce2000     92K r-x-- /lib/x86_64-linux-gnu/libnsl-2.27.so
00007f8fcfcf9000   2044K ----- /lib/x86_64-linux-gnu/libnsl-2.27.so
00007f8fcfef8000      4K r---- /lib/x86_64-linux-gnu/libnsl-2.27.so
00007f8fcfef9000      4K rw--- /lib/x86_64-linux-gnu/libnsl-2.27.so
00007f8fcfefa000      8K rw---   [ anon ]
00007f8fcfefc000     44K r-x-- /lib/x86_64-linux-gnu/libnss_nis-2.27.so
00007f8fcff07000   2044K ----- /lib/x86_64-linux-gnu/libnss_nis-2.27.so
00007f8fd0106000      4K r---- /lib/x86_64-linux-gnu/libnss_nis-2.27.so
00007f8fd0107000      4K rw--- /lib/x86_64-linux-gnu/libnss_nis-2.27.so
00007f8fd0108000     32K r-x-- /lib/x86_64-linux-gnu/libnss_compat-2.27.so
00007f8fd0110000   2048K ----- /lib/x86_64-linux-gnu/libnss_compat-2.27.so
00007f8fd0310000      4K r---- /lib/x86_64-linux-gnu/libnss_compat-2.27.so
00007f8fd0311000      4K rw--- /lib/x86_64-linux-gnu/libnss_compat-2.27.so
00007f8fd0312000   1948K r-x-- /lib/x86_64-linux-gnu/libc-2.27.so
00007f8fd04f9000   2048K ----- /lib/x86_64-linux-gnu/libc-2.27.so
00007f8fd06f9000     16K r---- /lib/x86_64-linux-gnu/libc-2.27.so
00007f8fd06fd000      8K rw--- /lib/x86_64-linux-gnu/libc-2.27.so
00007f8fd06ff000     16K rw---   [ anon ]
00007f8fd0703000     12K r-x-- /lib/x86_64-linux-gnu/libdl-2.27.so
00007f8fd0706000   2044K ----- /lib/x86_64-linux-gnu/libdl-2.27.so
00007f8fd0905000      4K r---- /lib/x86_64-linux-gnu/libdl-2.27.so
00007f8fd0906000      4K rw--- /lib/x86_64-linux-gnu/libdl-2.27.so
00007f8fd0907000    148K r-x-- /lib/x86_64-linux-gnu/libtinfo.so.5.9
00007f8fd092c000   2048K ----- /lib/x86_64-linux-gnu/libtinfo.so.5.9
00007f8fd0b2c000     16K r---- /lib/x86_64-linux-gnu/libtinfo.so.5.9
00007f8fd0b30000      4K rw--- /lib/x86_64-linux-gnu/libtinfo.so.5.9
00007f8fd0b31000    164K r-x-- /lib/x86_64-linux-gnu/ld-2.27.so
00007f8fd0d24000     20K rw---   [ anon ]
00007f8fd0d53000     28K r--s- /usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache
00007f8fd0d5a000      4K r---- /lib/x86_64-linux-gnu/ld-2.27.so
00007f8fd0d5b000      4K rw--- /lib/x86_64-linux-gnu/ld-2.27.so
00007f8fd0d5c000      4K rw---   [ anon ]
00007ffff002f000    132K rw---   [ stack ]
00007ffff00c5000     12K r----   [ anon ]
00007ffff00c8000      4K r-x--   [ anon ]
ffffffffff600000      4K --x--   [ anon ]
 total            24364K

Information will differ among different systems.

See the different regions:

• the first region, with r-x permissions is the .text (code) area
• the second region, with r-- premissions is the .rodata area
• the third region, with rw- permissions is the .data area, for initialized global variables
• the fourth region, with rw- permissions is the .bss area
• the fifth region, with the rw- permissions is the dynamic data memory area, also known as heap
• there are multiple dynamic libraries mapped in the virtual address space of the process, each library with their own regions
• there is a [stack] memory region, with rw- permissions

pmap also shows the total amount of virtual memory available to the process (24364K), as a total of the sizes of the regions. Note that this is virtual memory, not actual physical memory used by the process. For the process investigated above (with the 1127 pid) we could use the command below to show the total virtual size and physical size (also called resident set size):

student@os:~$ ps -o pid,rss,vsz -p $$
  PID   RSS    VSZ
 1127  1968  24364

The resident size is 1968K, much smaller than the virtual size.

Note how each region has a size multiple of 4K, this has to do with the memory granularity. The operating system allocates memory in chunks of a predefined size (in our case 4K) called pages.

Quiz

Practice

Enter the support/memory-areas/ directory. We investigate other programs.

1. The hello.c program prints out a message and then sleeps. Build it:

   student@os:~/.../lab/support/memory-areas$ make

   then run it (it will block):

   student@os:~/.../lab/support/memory-areas$ ./hello
   Hello, world!

   In another terminal, use the command below to show the memory areas of the process:

   student@os:~/.../lab/support/memory-areas$ pmap $(pidof hello)
   8220:   ./hello
   000055c0bef4b000      8K r-x-- hello
   000055c0bf14c000      4K r---- hello
   000055c0bf14d000      4K rw--- hello
   000055c0bf454000    132K rw---   [ anon ]
   00007f2a9e4a5000   1948K r-x-- libc-2.27.so
   00007f2a9e68c000   2048K ----- libc-2.27.so
   00007f2a9e88c000     16K r---- libc-2.27.so
   00007f2a9e890000      8K rw--- libc-2.27.so
   00007f2a9e892000     16K rw---   [ anon ]
   00007f2a9e896000    164K r-x-- ld-2.27.so
   00007f2a9ea8c000      8K rw---   [ anon ]
   00007f2a9eabf000      4K r---- ld-2.27.so
   00007f2a9eac0000      4K rw--- ld-2.27.so
   00007f2a9eac1000      4K rw---   [ anon ]
   00007ffee6471000    132K rw---   [ stack ]
   00007ffee6596000     12K r----   [ anon ]
   00007ffee6599000      4K r-x--   [ anon ]
   ffffffffff600000      4K --x--   [ anon ]
    total             4520K

   The output is similar, but with fewer dynamic libraries than bash, since they are not used by the program.

2. Make a program in another language of your choice that prints Hello, world! and sleeps and investigate it with pmap. Note that in the case of interpreted languages (Python, Lua, Perl, Ruby, PHP, JavaScript etc.) you have to investigate the interpreter process.

Memory Layout of Statically-Linked and Dynamically-Linked Executables

We want to see the difference in memory layout between the statically-linked and dynamically-linked executables.

Enter the support/static-dynamic/ directory and build the statically-linked and dynamically-linked executables hello-static and hello-dynamic:

student@os:~/.../lab/support/static-dynamic$ make

Now, by running the two programs and inspecting them with pmap on another terminal, we get the output:

student@os:~/.../lab/support/static-dynamic$ pmap $(pidof hello-static)
9714:   ./hello-static
0000000000400000    876K r-x-- hello-static
00000000006db000     24K rw--- hello-static
00000000006e1000      4K rw---   [ anon ]
00000000017b5000    140K rw---   [ anon ]
00007ffc6f1d6000    132K rw---   [ stack ]
00007ffc6f1f9000     12K r----   [ anon ]
00007ffc6f1fc000      4K r-x--   [ anon ]
ffffffffff600000      4K --x--   [ anon ]
 total             1196K

student@os:~/.../lab/support/static-dynamic$ pmap $(pidof hello-dynamic)
9753:   ./hello-dynamic
00005566e757f000      8K r-x-- hello-dynamic
00005566e7780000      4K r---- hello-dynamic
00005566e7781000      4K rw--- hello-dynamic
00005566e8894000    132K rw---   [ anon ]
00007fd434eb8000   1948K r-x-- libc-2.27.so
00007fd43509f000   2048K ----- libc-2.27.so
00007fd43529f000     16K r---- libc-2.27.so
00007fd4352a3000      8K rw--- libc-2.27.so
00007fd4352a5000     16K rw---   [ anon ]
00007fd4352a9000    164K r-x-- ld-2.27.so
00007fd43549f000      8K rw---   [ anon ]
00007fd4354d2000      4K r---- ld-2.27.so
00007fd4354d3000      4K rw--- ld-2.27.so
00007fd4354d4000      4K rw---   [ anon ]
00007ffe497ba000    132K rw---   [ stack ]
00007ffe497e3000     12K r----   [ anon ]
00007ffe497e6000      4K r-x--   [ anon ]
ffffffffff600000      4K --x--   [ anon ]
 total             4520K

For the static executable, we can see there are no areas for dynamic libraries. And the .rodata section has been coalesced in the .text area.

We can see the size of each section in the two executables by using the size command:

student@os:~/.../lab/support/static-dynamic$ size hello-static
text    data     bss     dec     hex filename
893333   20996    7128  921457   e0f71 hello-static

student@os:~/.../lab/support/static-dynamic$ size hello-dynamic
text    data     bss     dec     hex filename
4598     736     824    6158    180e hello-dynamic

Quiz

Based on the information above, answer this quiz.

Practice

1. Let's investigate another static executable / process.

   If not already installed, install the busybox-static package on your system. On Debian/Ubuntu systems, use:

   student@os:~$ sudo apt install busybox-static

   Start a process using:

   student@os:~$ busybox sleep 1000

   Investigate the process using pmap and the executable using size.

Modifying Memory Region Size

We want to observe the update in size of memory regions for different instructions used in a program.

Enter the support/modify-areas/ directory. Browse the contents of the hello.c file; it is an update to the hello.c file in the memory-areas/ directory. Build the executable:

student@os:~/.../lab/support/modify-areas$ make

Use size to view the difference between the new executable and the one in the memory-areas/ directory:

student@os:~/.../lab/support/modify-areas$ size hello
   text    data     bss     dec     hex filename
  13131   17128   33592   63851    f96b hello

student@os:~/.../lab/support/modify-areas$ size ../memory-areas/hello
   text    data     bss     dec     hex filename
   4598     736     824    6158    180e ../memory-areas/hello

Explain the differences.

Then use the pmap to watch the memory areas of the resulting processes from the two different executables. We will see something like this for the new executable:

student@os:~/.../lab/support/modify-areas$ pmap $(pidof hello)
18254:   ./hello
000055beff4d0000     16K r-x-- hello
000055beff6d3000      4K r---- hello
000055beff6d4000     20K rw--- hello
000055beff6d9000     32K rw---   [ anon ]
000055beffb99000    324K rw---   [ anon ]
00007f7b6c2e6000   1948K r-x-- libc-2.27.so
00007f7b6c4cd000   2048K ----- libc-2.27.so
00007f7b6c6cd000     16K r---- libc-2.27.so
00007f7b6c6d1000      8K rw--- libc-2.27.so
00007f7b6c6d3000     16K rw---   [ anon ]
00007f7b6c6d7000    164K r-x-- ld-2.27.so
00007f7b6c8cd000      8K rw---   [ anon ]
00007f7b6c900000      4K r---- ld-2.27.so
00007f7b6c901000      4K rw--- ld-2.27.so
00007f7b6c902000      4K rw---   [ anon ]
00007ffe2b196000    204K rw---   [ stack ]
00007ffe2b1d8000     12K r----   [ anon ]
00007ffe2b1db000      4K r-x--   [ anon ]
ffffffffff600000      4K --x--   [ anon ]
 total             4840K

We notice the size increase of text, data, bss, heap and stack sections.

Practice

1. Comment out different parts of the hello.c program to notice differences in only specific areas (text, data, bss, heap, stack).

2. Use a different argument (order) for the call to the alloc_stack() function. See how it affects the stack size during runtime (investigate with pmap).

3. Do a static build of hello.c and check the size of the memory areas, both statically and dynamically.

4. The extend_mem_area.py Python script allocates a new string at each step by merging the two previous versions. Start the program and investigate the resulting process at each allocation step. Notice which memory area is updated and explain why.

Quiz

Allocating and Deallocating Memory

Memory areas in a process address space are static or dynamic. Static memory areas are known at the beginning of process lifetime (i.e. at load-time), while dynamic memory areas are managed at runtime.

.text, .rodata, .data, .bss are allocated at load-time and have a predefined size. The stack and the heap and memory mappings are allocated at runtime and have a variable size. For those, we say we use runtime allocation and deallocation.

Memory allocation is implicit for the stack and explicit for the heap. That is, we don't make a particular call to allocate data on the stack; the compiler generates the code that the operating system uses to increase the stack when required. For the heap, we use the malloc() and free() calls to explicitly allocate and deallocate memory.

Omitting to deallocate memory results in memory leaks that hurt the resource use in the system. Because of this, some language runtimes employ a garbage collector that automatically frees unused memory areas. More than that, some languages (think of Python) provide no explicit means to allocate memory: you just define and use data.

Let's enter the support/alloc_size/ directory. Browse the alloc_size.c file. Build it:

student@os:~/.../lab/support/alloc_size$ make

Now see the update in the process layout, by running the program in one console:

student@os:~/.../lab/support/alloc_size$ ./alloc_size
Press key to allocate ...
[...]

And investigating it with pmap on another console:

student@os:~/.../lab/support/alloc_size$ pmap $(pidof alloc_size)
21107:   ./alloc_size
000055de9d173000      8K r-x-- alloc_size
000055de9d374000      4K r---- alloc_size
000055de9d375000      4K rw--- alloc_size
000055de9deea000    132K rw---   [ anon ]
00007f1ea4fd4000   1948K r-x-- libc-2.27.so
00007f1ea51bb000   2048K ----- libc-2.27.so
00007f1ea53bb000     16K r---- libc-2.27.so
00007f1ea53bf000      8K rw--- libc-2.27.so
00007f1ea53c1000     16K rw---   [ anon ]
00007f1ea53c5000    164K r-x-- ld-2.27.so
00007f1ea55bb000      8K rw---   [ anon ]
00007f1ea55ee000      4K r---- ld-2.27.so
00007f1ea55ef000      4K rw--- ld-2.27.so
00007f1ea55f0000      4K rw---   [ anon ]
00007ffcf28e9000    132K rw---   [ stack ]
00007ffcf29be000     12K r----   [ anon ]
00007ffcf29c1000      4K r-x--   [ anon ]
ffffffffff600000      4K --x--   [ anon ]
 total             4520K

student@os:~/.../lab/support/alloc_size$ pmap $(pidof alloc_size)
21107:   ./alloc_size
000055de9d173000      8K r-x-- alloc_size
000055de9d374000      4K r---- alloc_size
000055de9d375000      4K rw--- alloc_size
000055de9deea000    452K rw---   [ anon ]
00007f1ea4fd4000   1948K r-x-- libc-2.27.so
00007f1ea51bb000   2048K ----- libc-2.27.so
00007f1ea53bb000     16K r---- libc-2.27.so
00007f1ea53bf000      8K rw--- libc-2.27.so
00007f1ea53c1000     16K rw---   [ anon ]
00007f1ea53c5000    164K r-x-- ld-2.27.so
00007f1ea55bb000      8K rw---   [ anon ]
00007f1ea55ee000      4K r---- ld-2.27.so
00007f1ea55ef000      4K rw--- ld-2.27.so
00007f1ea55f0000      4K rw---   [ anon ]
00007ffcf28e9000    132K rw---   [ stack ]
00007ffcf29be000     12K r----   [ anon ]
00007ffcf29c1000      4K r-x--   [ anon ]
ffffffffff600000      4K --x--   [ anon ]
 total             4840K

student@os:~/.../lab/support/alloc_size$ pmap $(pidof alloc_size)
21107:   ./alloc_size
000055de9d173000      8K r-x-- alloc_size
000055de9d374000      4K r---- alloc_size
000055de9d375000      4K rw--- alloc_size
000055de9deea000    420K rw---   [ anon ]
00007f1ea4fd4000   1948K r-x-- libc-2.27.so
00007f1ea51bb000   2048K ----- libc-2.27.so
00007f1ea53bb000     16K r---- libc-2.27.so
00007f1ea53bf000      8K rw--- libc-2.27.so
00007f1ea53c1000     16K rw---   [ anon ]
00007f1ea53c5000    164K r-x-- ld-2.27.so
00007f1ea55bb000      8K rw---   [ anon ]
00007f1ea55ee000      4K r---- ld-2.27.so
00007f1ea55ef000      4K rw--- ld-2.27.so
00007f1ea55f0000      4K rw---   [ anon ]
00007ffcf28e9000    132K rw---   [ stack ]
00007ffcf29be000     12K r----   [ anon ]
00007ffcf29c1000      4K r-x--   [ anon ]
ffffffffff600000      4K --x--   [ anon ]
 total             4808K

The three runs above of the pmap command occur before the allocation, after allocation and before deallocation and after deallocation. Notice the update toe the 4th section, the heap.

Now, let's see what happens behind the scenes. Run the executable under ltrace and strace:

student@os:~/.../lab/support/alloc_size$ ltrace ./alloc_size
malloc(32768)                                                                                                    = 0x55e33f490b10
printf("New allocation at %p\n", 0x55e33f490b10New allocation at 0x55e33f490b10
)                                                                 = 33
[...]
free(0x55e33f490b10)                                                                                             = <void>
[...]

student@os:~/.../lab/support/alloc_size$ strace ./alloc_size
[...]
write(1, "New allocation at 0x55ab98acfaf0"..., 33New allocation at 0x55ab98acfaf0
) = 33
write(1, "New allocation at 0x55ab98ad7b00"..., 33New allocation at 0x55ab98ad7b00
) = 33
brk(0x55ab98b08000)                     = 0x55ab98b08000
write(1, "New allocation at 0x55ab98adfb10"..., 33New allocation at 0x55ab98adfb10
) = 33
write(1, "Press key to deallocate ...", 27Press key to deallocate ...) = 27
read(0,
"\n", 1024)                     = 1
brk(0x55ab98b00000)                     = 0x55ab98b00000
[...]

The resulting output above shows us the following:

• malloc() and free() library calls both map to the brk syscall, a syscall that updates the end of the heap (called program break).
• Multiple malloc() calls map to a single brk syscall for efficiency. brk is called to preallocate a larger chunk of memory that malloc will then use.

Update the ALLOC_SIZE_KB macro in the alloc_size.c file to 256. Rebuild the program and rerun it under ltrace and strace:

student@os:~/.../lab/support/alloc_size$ ltrace ./alloc_size
[...]
malloc(262144)                                                                                                   = 0x7f4c016a9010
[...]
free(0x7f4c016a9010)                                                                                             = <void>
[...]

student@os:~/.../lab/support/alloc_size$ strace ./alloc_size
[...]
mmap(NULL, 266240, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7feee19f2000
write(1, "New allocation at 0x7feee19f2010"..., 33New allocation at 0x7feee19f2010
) = 33
mmap(NULL, 266240, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7feee19b1000
write(1, "New allocation at 0x7feee19b1010"..., 33New allocation at 0x7feee19b1010
) = 33
write(1, "Press key to deallocate ...", 27Press key to deallocate ...) = 27
read(0,
"\n", 1024)                     = 1
munmap(0x7feee19b1000, 266240)          = 0
[...]

For the new allocation size, notice that the remarks above don't hold:

• malloc() now invokes the mmap syscall, while free() invokes the munmap syscall.
• Each malloc() calls results in a separate mmap syscall.

This is a behavior of the malloc() in libc, documented in the manual page. A variable MALLOC_THRESHOLD holds the size after which mmap is used, instead of brk. This is based on a heuristic of using the heap or some other area in the process address space.

Practice

1. Use pmap to analyze the process address space for ALLOC_SIZE_KB initialized to 256. Notice the new memory areas and the difference between the use of mmap syscall and brk syscall.

2. Use valgrind on the resulting executable, and notice there are memory leaks. They are quite obvious due to the lack of proper freeing. Solve the leaks.

3. Use valgrind on different executables in the system (in /bin/, /usr/bin/) and see if they have memory leaks.

Quiz

Quiz

Memory Mapping

The mmap syscall is used to allocate memory as anonymous mapping, that is reserving memory in the process address space. An alternate use is for mapping files in the memory address space. Mapping of files is done by the loader for executables and libraries. That is why, in the output of pmap, there is a column with a filename.

Mapping of a file results in getting a pointer to its contents and then using that pointer. This way, reading and writing to a file is an exercise of pointer copying, instead of the use of read / write-like system calls.

In the support/copy/ folder, there are two source code files and two scripts:

• read_write_copy.c implements copying with read / write syscalls
• mmap_copy.c implements copying using mmap
• generate.sh script generates the input file in.dat
• benchmark_cp.sh script runs the two executables mmap_copy and read_write_copy

Open the two source code files and investigate them. You will notice that the open() system call has the following prototype int open(const char *pathname, int flags). The argument flags must include one of the following access modes: O_RDONLY, O_WRONLY, or O_RDWR - indicating that the file is opened in read-only, write-only, or read/write mode. You can add an additional flag - O_CREAT - that will create a new file with pathname if the file does not already exist. This is only the case when opening the file for writing (O_WRONLY or O_RDWR). If O_CREAT is set, a third argument mode_t mode is required for the open() syscall. The mode argument specifies the permissions of the newly created file. For example:

// If DST_FILENAME exists it will be open in read/write mode and truncated to length 0
// If DST_FILENAME does not exist, a file at the path DST_FILENAME will be create with 644 permissions
dst_fd = open(DST_FILENAME, O_RDWR | O_CREAT | O_TRUNC, 0644);

Let's generate the input file:

student@os:~/.../lab/support/copy$ ./generate.sh

and let's build the two executable files:

student@os:~/.../lab/support/copy$ make

Run the benchmark_cp.sh script:

Run the script in your local environment, not in the Docker container. Docker does not have permission to write to /proc/sys/vm/drop_caches file.

student@os:~/.../lab/support/copy$ ./benchmark_cp.sh
Benchmarking mmap_copy on in.dat
time passed 54015 microseconds

Benchmarking read_write_copy on in.dat
time passed 42011 microseconds

Run the script a few more times. As you can see, there isn't much of a difference between the two approaches. Although we would have expected the use of multiple system calls to cause overhead, it's too little compared to the memory copying overhead.

If you inspect benchmark_cp.sh, you will notice a weird-looking command sh -c "sync; echo 3 > /proc/sys/vm/drop_caches". This is used to disable a memory optimization that the kernel does. It's called "buffer cache" and it's a mechanism by which the kernel caches data blocks from recently accessed files in memory. You will get more detailed information about this in the I/O chapter.

Browse the two source code files (mmap_copy.c and read_write_copy.c) for a glimpse on how the two types of copies are implemented.

Quiz

Practice

1. Use a different value for BUFSIZE and see if that affects the comparison between the two executables.

2. Add a sleep() call to the mmap_copy.c file after the files were mapped. Rebuild the program and run it. On a different console, use pmap to view the two new memory regions that were added to the process, by mapping the in.dat and out.dat files.