This post explains the fundamental principles enabling high-speed satellite internet possible. We will build an intuition for the constraints without hand-waving the physics.

Disclaimer: I am no expert and have no formal electrical engineering background. This post is backed by my readings of nonauthoritative online public resources. I will not be updating this post, so please leave corrections in the comments section.

Satellite internet crosses multiple different links to provide connectivity. Users connect devices to a satellite terminal, usually a small dish, that talks directly to a satellite. The satellite may relay data across other satellites before reaching a ground station and the internet. This post focuses on satellite-ground link, though these calculations apply all to wireless connections.

There are two orbits we will look at: low Earth orbit (LEO) and geostationary orbit (GEO). In summary, satellites closer to Earth have stronger signal, lower latency, and require smaller antennas. However, further satellites get more coverage, so a handful of GEO satellites can mostly achieve global coverage, while a LEO constellation requires many more satellites.

The Fundamental Challenge

Imagine shouting across a room–you can only shout so loud and it gets quieter with distance. Satellites have the same issue but from hundreds of kilometers away.

When transmitting a signal, you can think of the energy spreading uniformly in all directions. The power density (watts per square meter) follows the inverse square law:

This represents the primary constraint in satellite communications: doubling distance quarters signal strength.

At 1,000km away, the signal spreads over 12.6 million \(\mathrm{km}^2\). At geostationary orbit (GEO) altitude of 36,000km, that same energy covers 16.3 billion \(\mathrm{km}^2\). The antenna in your device receives a tiny fraction of that radiated power.

The inverse square law explains low Earth orbit (LEO) advantages: moving satellites closer from 36,000km (GEO) to 550km (LEO) above the Earth’s surface improves received signal strength by over 4,000 times. The satellite needs to “shout” across 35,450 fewer kilometers. Check out this Wikipedia diagram for a true sense of scale and this post if you need a refresher on orbital mechanics.

Link Budget

Measuring connection performance requires accounting for all signal gains and losses with the Friis Transmission Formula¹:

Where:

• \(P_r\) = Power received at your antenna (watts)
• \(P_t\) = Power transmitted by satellite (watts)
• \(G_t\) = Gain of satellite’s transmitting antenna (dimensionless)
• \(G_r\) = Gain of your receiving antenna (dimensionless)
• \(\lambda\) = Wavelength of the radio signal (meters)
• \(R\) = Distance between satellite and your antenna (meters)

Decibels (dB) are logarithmic, so they turn multiplication into addition:

The Free Space Path Loss (FSPL) term captures the inverse square law:

Let’s go through each component and its bottlenecks.

Transmit Power (Pt) faces different bottlenecks at each scale. Large GEO satellites can generate 100+ watts per beam but hit thermal limits as it is hard to dissipate waste heat in vacuum. Smaller LEO satellites are constrained by their total power budget from solar panels, limiting power to about 10 watts per beam. Phones trasmit 0.2 watts, limited by battery life and SAR regulations that prevent excessive radio frequency exposure to human tissue.

Antenna Gain (Gt, Gr) measures how well the antenna focuses the signal’s energy. Larger apertures provide higher gain. Satellite phased arrays achieve approximately 35dBi (3,000× amplification) within size and weight constraints. A dish reaches ~38dBi (6,000× amplification) but is bigger and require precise pointing. Phones get ~2dBi (1.6× amplification).

Distance (R): As we know from the inverse square law, this quadratic term becomes the dominant factor in link performance. LEO satellites at 550km altitude experience 173dB of free space path loss, while GEO satellites at 35,786km suffer 209dB loss. This 36dB difference means GEO signals arrive 4,000 times weaker than LEO signals. We will go into the tradeoffs of different orbits near the end of this post.

Shannon Limit

Once you receive a signal, you need to decipher the encoded data. Maximum data rate follows the Shannon–Hartley theorem:

Where:

• \(C\) = Channel capacity (bits per second)
• \(B\) = Bandwidth (Hz)
• \(S/N\) = Signal-to-Noise Ratio (dimensionless)

This equation reveals you either need more bandwidth or better signal-to-noise ratio.

More bandwidth (B) linearly increases the data rate, which explains why access to frequency spectrum can go for billions of dollars. Higher frequencies are more susceptible to the weather. LTE is a relatively low frequency (~2GHz), which allows the signal to go through buildings.

Improving signal (S) quality offers diminishing returns due to the logarithmic relationship, but the effect remains significant.

Noise (N) originates from thermal agitation of electrons and is unavoidable above absolute zero:

Where:

• \(k\) = Boltzmann constant \((1.38 \times 10^{-23} \ \mathrm{J/K})\)
• \(T\) = System noise temperature (~150K for good receivers)
• \(B\) = Bandwidth (Hz)

Since noise is fixed by physics, improving the Signal-to-Noise Ratio requires increasing signal power–which brings us back to the link budget.

GEO vs LEO Path Loss Calculation

Now we can quantify why lower orbit satellites are transforming satellite communications.

Given a 20 GHz signal (λ = 0.015m):

GEO Path Loss (35,786km):

LEO Path Loss (550km):

Difference: 36.27dB The LEO signal is \(10^{(36.27/10)} =\) 4,233 times stronger than the GEO signal.

This 36dB difference drastically changes system design. LEO’s stronger signal enables higher-order modulation schemes. While GEO systems might use 16-QAM (4 bits per symbol), LEO can use 1024-QAM (10 bits per symbol) at the same error rate. Also, a 19-inch flat panel can achieve what previously required a 4-foot parabolic dish.

Worlds Away

The proximity advantage comes with complexity costs. GEO satellites match the Earth’s rotation, so they appear stationary in the sky and require no tracking. LEO satellites complete an orbit every 90 minutes, demanding rapid beam steering, satellite handoffs, and constellation management.

LEO signals round trip in ~4ms while GEO signals require ~240ms, impacting real-time applications over GEO networks.

Dish vs Phone

How much bandwidth can you actually get from a LEO satellite based on your antenna?

Assumptions:

• LEO satellite at 550km altitude
• 20GHz downlink frequency
• 250MHz channel bandwidth
• 10W satellite transmit power
• 35dBi satellite antenna gain
• 150K system noise temperature

Dish Link Budget:

Smartphone Antenna Link Budget:

A gigabit to my phone from space!? These calculations assume perfect conditions. In reality, you need at least 10-15dB margin to account for atmospheric losses from rain and clouds, pointing errors as satellites move across the sky, receiver implementation losses from non-ideal components, and Doppler effects from the satellite’s 27,000km/h orbital velocity.

The dish has plenty of margin (52.56dB), but the phone is borderline. This is why a cellular service needs to use much narrower bandwidths.

Real Smartphone Antenna:

Keeping the same frequency with a 5MHz channel instead of 250MHz:

A phone-scale antenna could expect closer to 60Mbps.

Engineering Constraints

Engineers need to navigate the physics to balance trade-offs to make a viable system.

Power Budget

A satellite’s power budget determines beam transmit power.

The solar panels need to generate enough energy to power the onboard electronics and are mainly used to power the beams. LEO satellites orbit many times per day, so they need to use battery power for the third of time that the Earth blocks sunlight. Ultimately, it’s the power that determines the size of the satellite and the number of beams and their strength.

Frequency Licensing

Radio spectrum is a finite resource managed by country governments. Spectrum allocation creates fundamental trade-offs between bandwidth and propagation characteristics. Higher frequencies offer more bandwidth but suffer worse propagation losses and weather effects.

L-band (1-2GHz) offers excellent propagation through atmosphere and modest power requirements, but provides limited bandwidth. Ka-band (26-40GHz) enables high-bandwidth broadband services but suffers significant rain attenuation that can shut down links during storms. V-band (40-75GHz) promises future high-capacity links but faces extreme weather sensitivity.

Orbital Mechanics

LEO satellites orbit Earth every 90 minutes, creating dynamic coverage patterns. Lower altitudes have less path loss but need magnitudes more satellites to maintain coverage.

Orbital inclination determines coverage zones: polar orbits provide global reach including Arctic regions, while equatorial orbits focus capacity on populated areas but leave polar regions uncovered. Plane spacing controls satellite visibility. Too few planes create coverage gaps, while too many increase constellation complexity and inter-satellite interference.

Antenna Pointing

User terminals must track satellites moving at 7.5km/s. Mechanical tracking systems using motors and gears are expensive, slow to respond, and prone to mechanical failure. Phased arrays enable fast electronic steering without moving parts but require complex beamforming algorithms and hundreds of antenna elements, increasing cost and power consumption. Multi-satellite handoffs demand seamless switching between satellites as they pass overhead, requiring precise timing coordination and careful interference management.

Cost

The cost to deliver one Viasat-3 satellite (6,400kg) to geostationary orbit was $150M aboard an expendable Falcon Heavy. The same rocket type launched to LEO with 56 Starlink satellites (310kg) and can be reused, so the launch costs a reduced $97M. Furthermore, the Viasat-3 satellite costs $420M, whereas each Starlink satellite is below $1M.

Viasat-3 satellites get global coverage with three satellites, while a LEO constellation needs tens of thousands.

Why Now?

LEO constellations are not new–Iridium had a network in 1998. If the physics have not changed in the past quarter century, what has?

Launch Economics

Historically, launch costs have limited access to space. Since the 1990s, the cost to launch payload to LEO has dropped from $20,000/kg to below $2,000/kg. Rapidly reusable Starships promise to reduce costs another order of magnitude. This trend has enabled large LEO constellations to be economically viable.

Satellite Miniaturization

Traditional satellites were custom-built, multi-ton systems designed for >15 year lifespans in a harsh radiation environment. LEO satellites operate within the Van Allen radiation belts where Earth’s magnetic field provides partial radiation shielding. Relaxing radiation tolerance allows satellites to use modern electronics that are decades ahead of radiation-hardened components. This modernization means smaller satellites can deliver more bandwidth and use standard components, enabling satellite mass production.

Phased Array

Originally developed for military radar, phased arrays electronically steer beams and can track and hop between fast-moving LEO satellites. Semiconductor advances have reduced phased array costs such that they are now ubiquitous in new wireless standards.

Configurable Hardware

Software-defined radios can adapt modulation, coding, and protocols through software updates. Digital signal processing no longer requires specialized hardware. Field-programmable gate arrays (FPGAs) and digital signal processors enable efficient implementation of advanced techniques like 1024-QAM modulation and adaptive coding.

Global Impact

Satellites already provide essential internet to ships and planes and can be used in emergencies after disasters wipe out terrestrial infrastructure. However, there remain over two billion people without internet access. Widespread, low cost, and high bandwidth satellite connectivity offers gigabit internet for the first time to many rural communities, remote islands, and places with undeveloped terrestrial infrastructure.

This post does not begin to get into designing a satellite constellation. There is a tradeoff between wanting global satellite coverage, but demand is not evenly distributed. Viasat has a cool visualization of flight paths and satellite data usage.

The application of electromagnetic theory, information theory, and orbital mechanics bring the world’s knowledge to you no matter where you hike, fly, or sail. This post explored a first principles dive into the terabit cosmic beams that are closing the digital divide.

Further Reading

• Three great satellite constellation visualizers: Viasat, Starlink, and general
• Great intuitive RF Electromagnetics Visualization
• Good slides overview of satcomm
• Taste for how complicated satellite capacity modeling is
• Alright, though outdated series on LEO basics
• Overview of design space of a LEO constellation
• Antenna Theory
• Some background on why LEO internet is succeeding now: Timeline of Spaceflight, Space Launch Market Competition
• MATLAB has some great resources for doing calculations much better than this post

Fun Quiz

Read the Report PDF

Abstract

Binary Room is a static binary translator from RISC-V to ARM binaries. It targets userspace applications and supports a subset of the RISC- V instruction set. We can translate applications that perform memory operations, control flow and system calls. This report outlines the details of the implementation along with challenges we ran into.

1. Introduction

Binary Room statically translates binaries from a subset of the riscv64 to aarch64 instruction set architectures (ISA). We support arbitrary 64-bit userspace binaries, which can include function calls and syscalls. We support a subset of RISC-V instructions which perform arithmetic, memory operations, control flow, and system calls. First, Section 3 provides an overground of our design decisions. Next, Section 4 provides an overview of the aspects of our implementation. We outline the high-level flow of binary translation, along with the difficulties in writing each module of the code (Section 5). Lastly, we evaluate Binary Room's input and output binaries (Section 6).

2. Background

RISC-V is the future of open computing. Binary translation is a form of emulation which enables running binaries from other architectures. To smooth the transition to the RISC-V ISA, we present Binary Room, a tool to run unmodified RISC-V binaries on ARM. Binary Room statically translates binaries without the need for any hardware virtualization extensions.

Our translator is inspired by Apple’s x86 to ARM binary translator, Rosetta 2. Similarly, our tool transparently converts userspace binaries from riscv64 to aarch64 instructions. Binary translation has proved to be a flexible and efficient approach for ensuring binary compatibility (A Comparison of Software and Hardware Techniques for X86 Virtualization, 2006).

3. Design

At a high level, Binary Room operates by consuming a list of RISC-V instructions, iteratively translating each instruction from RISC-V to ARM, converting the ARM instructions to an ARM assembly file, and forming an ARM executable.

Binary room operates at the assembly level, so there are a few steps before producing an executable binary. We assemble the assembly code into an object. Then, we take the object code and link it into the final binary file. On Linux, where we performed the bulk of our testing, this was an ELF file (Tool Interface Standard (TIS) Executable and Linking Format (ELF) Specification Version 1.2, 1995).

4. Implementation

4.1. Representing Instructions

We exploited the Rust type system to encode information about the RISC-V and ARM instructions (RISC-V Instruction Set Specifications, 2019). Specifically, we extensively used Rust's algebraic data types, enums, throughout our codebase.

Parsing the assembly into the Rust type system provides guarantees of the arguments to instructions at compile time. There are many steps that can go wrong, which makes debugging challenging. Parsing, translation, and runtime errors are examples of issues during development. One difficulty is that to support general binaries, you must implement the entire source and destination instruction set and must also encode all the alignment, calling convention, and syscall constraints. To deal with this complexity, we introduce Panic Driven Development (PDD). PDD ensures that binaries we cannot translate fail at the parsing stage. The code will be unable to encode instructions it cannot translate by the design of the enum. This strategy prevents debugging malformed assembly output.

Each enum variant encodes the exact instruction and the types of its arguments.

pub enum RiscVInstruction {
  /// add register
  /// either add or addw
  /// (addw is 32 bits on 64 bit riscv)
  ///
  /// `x[rd] = sext((x[rs1] + x[rs2])[31:0])`
  #[strum(serialize = "add")]
    Add {
    // dest = arg1 + arg2
    width: RiscVWidth,
    dest: RiscVRegister,
    arg1: RiscVRegister,
    arg2: RiscVRegister,
  },
  // Copy register
  // `mv rd, rs1` expands to `addi rd, rs, 0`
  #[strum(serialize = "mv")]
  Mv {
    dest: RiscVRegister,
    src: RiscVRegister,
  },
  …
}

For the ADD instruction case, the width field specifies whether the instruction is invoked with double or word register width. The arguments to the instruction are defined in another enum which represents CPU registers.

pub enum RiscVRegister {
  #[default]
  #[strum(serialize = "x0")]
  /// Hard-wired zero
  X0,
  #[strum(serialize = "ra")]
  /// Return address
  RA,
  …
}

Our instruction type supports ARM instructions that are not present in the RISC-V ISA. Notably, conditional branching in ARM requires a CMP instruction to set condition flags based on registers operands. For example, to branch to the label foo if x1 != x2, we must first cmp x1, x2 and then conditionally branch bne foo. RISC-V handles conditional branches by taking additional operands in the branch instruction, e.g. bne x1, x2 foo. During translation, Binary Room may output multiple ARM instructions for a given RISC-V instruction.

One limitation of our enum is that we must manually parse from the disassembled binary into the Rust enum. We prioritized implementing the binary translation and did not finish writing a complete parser. Encoding the enum into assembly is straightforward as we use Rust's formatter to convert the list of instructions into validly formatted assembly.

4.2. Variable Width ARM Registers

One feature we encode in the type system is register width. Both ARM And RISC-V have 32 general-purpose registers. However, ARM registers can be used as either 32-bit or 64-bit registers. ARM registers are indicated by the first letter of the register name in the assembly. An “x” indicates 64 bits of a register and a “w” indicates the least significant 32 bits of a register (Registers in Aarch64 State, 2024). This variation means that translation must account for the register width that RISC-V instructions specify and output an ARM instruction with the correct register widths.

We ensured correctness of the binary translation by unit testing simple cross-compiled RISC-V programs. Our tests automatically run in GitHub actions on every commit, and can be run locally using cargo, the Rust build system.

4.3. Register Mapping

During instruction translation, we had to be cautious in how we translate the registers in use. Some registers are not general purpose and have semantic meanings, such as the stack pointer (ARM Register Names, 2024). Other registers are expected to have some meaning after function calls, such as the return register. We needed to be mindful of these invariants and avoid overwriting any important state.

To keep our mapping of registers consistent, we created a look-up table to convert each RISC-V register into its corresponding ARM register.

fn map_register(riscv_reg: RiscVRegister, riscv_width: &RiscVWidth) -> ArmRegister {
  
    ArmRegister {
        width: map_width(riscv_width),
        name: map_register_name(riscv_reg),
    }
    
}

Critically, ARM registers can vary in width and RISC-V cannot. For this reason, we added a width argument to this function and decompose the conversion of width and register name into separate functions, map_width and map_register_name. We define widths to be the size of a register and define register names to be the width-agnostic name of the register in assembly (i.e. sp or general purpose register 1).

4.4. Calling Convention

The calling convention of an ISA ensures compatibility with binaries from other compilers (Volume I: RISC-V User-Level ISA V2.1draft, Chapter 18, Calling Convention, 2014). If a program does not interact with foreign functions, then it is not required to abide by the calling convention. This flexibility makes the register translation easier due to less constraints on how Binary Room handles register mappings. To ensure that local register values are preserved between function calls, Binary Room efficiently saves registers onto the stack before a call instruction and restores them from the stack when returning from a function call.

The calling convention constraint still applies to syscalls and dynamic libraries.

4.4.1. Syscalls

We chose to directly translate syscall instructions. Alternatively, we could have taken advantage of the syscall wrappers built into libc (Syscalls(2) - Linux Manual Page, n.d.). However, we chose not to deal with complexities of dynamic libraries, instead directly modifying the instructions which set up syscall arguments and invoke the syscall.

Originally, we thought syscall translation would be equivalent to normal instruction translation. However, there were some nuances that made this more challenging. One such nuance was that syscall numbers change across architectures.

# write(stdout, buf, len)

# syscall number
li a7,64

# arg 2: len
li a2,13

# arg 1: buf
lui a0,%hi(buf)
addi a1,a0,%lo(buf)

# arg 0: stdout fd number
li a0,1

# trigger syscall
ecall

When you want to perform a syscall, you need to indicate which syscall you want to execute by loading the syscall number into a register. In Listing 3, the syscall number, 64, is loaded into register a7. However, this syscall number is defined by the operating system. Linux's syscall numbers vary across architectures and ISA variants (i.e. 32-bit and 64-bit) (Arm.syscall.sh, 2024).

Our initial approach inserted instructions to use a lookup table to convert the syscall numbers. At runtime, this code would look at the requested syscall and rewrite the correct value for aarch64 (Linux Kernel System Calls for All Architectures, 2025). Luckily, Linux shares identical syscall numbers between ARM 64-bit and RISC-V 64-bit. It was sufficient for our architecture pair to directly translate the same syscall numbers without modification.

4.4.2. Branching

Binary Room supports relative branching to labels. We extract labels from the disassembled input binary and map them verbatim to the output ARM binary. This insertion guarantees that the label ends up at the same point in the execution of the translated binary. The assembler handles the complexity of turning labels into addresses. Binary Room does not support absolute branches to arbitrary addresses.

5. Challenges

5.1. ISA Quirks

Instruction sets handle the same primitives differently. For example, RISC-V uses a zero register to produce zero arguments to instructions. In ARM, you must use a normal register or dummy instruction such as xoring a register with itself to get a zero (RISC-V Instruction Set Specifications, 2019).

The instruction behaviors also differ between the ISAs. We needed to handle the cases where arguments can change which instruction we need to translate to. For instance, RISC-V's add immediate (addi) opcode can take positive or negative constants, whereas ARM's immediate argument to add can only be positive. Translating this requires handling the different cases of argument values. Sometimes, these translations cannot be determined statically, as seen with syscall numbers.

Figure 3 shows the information provided by the ISA reference. Instructions have high level implementations which authoritatively define their behavior (RISC-V Reference Card, n.d.). This pseudocode enables us to ensure instruction maps have the same behavior.

5.1.1. Pseudo-instructions

Binary disassemblies contain the list of instructions of the program. However, there are also RISC-V pseudo-instructions that can get generated (RISC-V Assembler: Arithmetic, 2024). These are higher level operations that correspond to multiple individual operations. These abstractions are not documented in the architecture manual, so we had to use unofficial online posts to decipher the meaning of these pseudo-instructions.

5.2. ISA Documentation

The largest obstacle we encountered during this project was tracking down relevant documentation and discovering what we needed to know. The general ISAs for RISC-V and ARM are open, but exact implementations of the specifications can differ greatly. For example, when looking for documentation, we would often find incorrect or misleading information. Initially we faced confusion with conflicting information online. This mismatch is because the 32-bit and 64-bit versions of the ISAs change the behavior of registers and syscalls. Instructions work differently on different ISA versions, for instance the many versions and parts of ARM (armv7, aarch64, etc.).

Another difficulty was fully understanding how each ISA works. We needed to understand the semantics of each ISA in order to create a mapping. We found unofficial documentation, such as Figure 3, to be more complete and intelligible than official documentation. Mapping instructions and registers required carefully diffing the register specifications and calling conventions for each ISA. The issues with syscalls led us to decide not to support 32-bit binaries.

As we produced the output binaries, we ended up having to debug invalid assembly. Assembly is itself high level, so we had to deal with linking errors with illegal arguments, unexplainable segfaults, and dealing with OS specific assembly problems such as alignment constraints and extern symbols.

Testing our implementation proved difficult as you need to get everything working in order to test. Each part of the pipeline is dependent on previous parts. Unit testing did not help find bugs as much as running test translations on real programs. Every program you compile with gcc requires dynamic libraries, as there is an implicit libc call to exit the program. We were unable to get static nolibc binaries to work as it has dependencies on the libc provided by the host OS, which needs to be built for RISC-V or ARM.

Another complication is that even simple binaries do not follow the conventions you would expect. For a "hello world" program, we implemented conversion of the text section. Even at minimum optimizations, gcc puts the literal string behind a call to the IO_stdin_used symbol. We decided to not handle all the complications with process and library loading, so a "hello world" program only works with a direct offset to the constant.

5.3. Development environment

Developing Binary Room requires running a Linux machine with QEMU and cross-compile toolchains for riscv64 and aarch64. Binary Room is built with Rust's cargo toolchain and tests run with cargo and some platform-specific build scripting.

To handle the complexity of the build environment, we use the Nix package manager. Nix shells allow us to create a fully reproducible build environment and easily manage parallel compiler toolchains for each architecture we need. (Cross Compiling - Nixos Wiki, 2025)

5.4. Debugging

Debugging the output assembly proved difficult. There are no layers of abstraction below the binary, so there is no debug information when a program goes wrong. Additionally, the issues are typically related to hardware or illegal instructions, so debuggers such as gdb cannot provide any help. We ran into such issues when supporting macOS on our output binaries. macOS has different alignment and linking policies than Linux hosts. Debugging crashes can be difficult as there is not much documentation online explaining these failures. When Binary Room introduced unexpected segmentation faults, we debugged the ARM binary using gdb on an AWS EC2 instance with an ARM-based Graviton CPU.

6. Evaluation

Binary Room provides several programs which it is able to translate. We run translation against all of them by using cargo test. We currently support programs that compute Fibonacci, run arithmetic in a loop, print hello world, and echo stdin. You can view them on GitHub.

6.1. Benchmarking

We evaluate Binary Room in comparison with QEMU. We run the original RISC-V and translated ARM assembly via QEMU on a x86 machine. We use 100 warmup trials and then repeatedly run the test program 1,000 times to get the average and standard deviation of runtime.

We find that the RISC-V binaries run about twice as fast, as compared to our translation. We predict most of the runtime is due to overhead from loading the process and QEMU. We do not have access to a riscv64 machine, so we are unable to do a fair comparison of bare-metal results. We find both ARM translated binaries run 88-103% slower in QEMU. We are unsure of the cause of this slowdown as the instruction mappings are near one-to-one, so each program should run a similar number of instructions.

7. Discussion

Binary Room introduces a static binary translator from riscv64 to aarch64. Unlike other emulation options, Binary Room produces a fully compatible binary with no dependencies. QEMU and Rosetta2 require just-in-time runtime support to support their translations. Statically translated binaries are more portable, though we forego optimizations that could happen based on runtime values. Furthermore, Binary Room is written in Rust and natively supports both macOS and Linux targets.

8. Conclusion

Binary Room is open source and freely licensed. You may access the source code on GitHub:

https://github.com/samir-rashid/binary-room

References

Today we journey into Linux to discover how programs are loaded. Building new systems requires understanding how the current system actually works—not how the documentation claims it works. I wrote this post for people who want to learn more about kernel internals but have been hesitating to dive in for themselves; it’s the post I wish I had when starting out. This article aims to convince you that kernel development is not intimidating. We will explore how to deal with ambiguity when researching something new. You will come out with an understanding of the exact actions Linux takes to load a realistic program. Moreover, you will learn the tools to deconstruct any large program.

Goal: How does Linux load a program?

The first thing you may note is that this question is a bit odd. There are several interpretations for what it means to “load a program”. This is a good point. However, a priori, I cannot know what the correct question to ask is. Our expedition begins assuming basic C knowledge and becomes more complex. We will learn about the Linux kernel and its internal mechanisms from scratch.

We must step in any direction to start. The results of each experiment will allow us to iterate, refine our questions, and learn new concepts.

On first blush, the question seems obvious. Running a program simply means going to the first instruction and running in order until the end. Let’s test this theory.

Load a program

Let’s try running a simple program and inspect what it is doing. nautilus is the file explorer application that ships with GNOME. We want to know what Linux is doing when nautilus starts. Luckily, there is a tool called strace which records every call into Linux. Before we look at the output, let’s stop to recursively explain what is going on.

:information_desk_person:: Why Nautilus?

No reason. I wanted a simple program that doesn’t do much.

:raising_hand:: How do you know to look at calls into the kernel?

Think about how the way programs run. Starting any process requires notifying the kernel of the program you are starting. These interactions are called syscalls. User programs use these functions to ask the operating system to do something privileged. For example, a process cannot open a file without asking for permission first. You can think of system calls as normal function calls for the purposes of this exercise.

$ strace nautilus
execve("/run/current-system/sw/bin/nautilus", ["nautilus"], 0x7ffe54d428a0 /* 75 vars */) = 0
brk(NULL)                               = 0x30e3a000
--------- 23,744 lines snipped ---------

What was that output??? My terminal gets flooded with nautilus’ tens of thousands of syscalls. I have made a grave mistake. That’s okay, it’s too early to get demoralized. Evidently, my mental model that a program which does not appear to be doing anything is doing nothing is wrong. How can we adjust course to make better progress?

Debugging Process:

load “simple” program ⟶ ❌

Load “Hello World”

Instead, we can write a minimal hello_world.c program to test with¹. What happens when the kernel loads this program and prints some text?

#include <stdio.h>

int main(void) {
        printf("hello world!\n");
        return 0;
}

$ gcc hello_world.c
$ ./a.out
hello world!

Aha, we have solved the puzzle! The kernel starts running the instructions of main() and printf is just a function call. Spoiler alert: this is wrong. We still haven’t explained what this printf reference is.

:angry:: You said I wouldn’t need any prerequisite knowledge to understand this. I don’t understand these keywords. What is this weird void thing?

If you haven’t seen this before, you can ignore it for now. It means that the function has no input.

:confounded:: What is that \n after hello world?

Wonderful question, though you may not realize it :smile:. Simply, the \n asks printf to print a newline. Check out my fork post and the resources on printf at the end of this post to get closer to what \n actually does.

In fact, if we remove all unfamiliar syntax, the program behaves the same as before.

main() {
        printf("hello world!\n");
}

$ gcc hello_world_skeptic.c
$ ./a.out
hello world!

This time our compiler, gcc, complains with a bunch of warnings:

hello_world.c:1:2: warning: return type defaults to ‘int’ [-Wimplicit-int]
    1 |  main() {
      |  ^~~~
hello_world.c: In function ‘main’:
hello_world.c:2:9: warning: implicit declaration of function ‘printf’ [-Wimplicit-function-declaration]
    2 |         printf("hello world!\n");
      |         ^~~~~~
hello_world.c:1:1: note: include ‘<stdio.h>’ or provide a declaration of ‘printf’
  +++ |+#include <stdio.h>
    1 |  main() {
hello_world.c:2:9: warning: incompatible implicit declaration of built-in function ‘printf’ [-Wbuiltin-declaration-mismatch]
    2 |         printf("hello world!\n");
      |         ^~~~~~
hello_world.c:2:9: note: include ‘<stdio.h>’ or provide a declaration of ‘printf’

Weird. The program runs the same, so why are we getting all these warnings? One line jumps out to me. gcc warns us four times about something to do with the “declaration of ‘printf’”. Before we get ahead of ourselves, allow me to fix these compiler warnings. Fixing these warnings will prevent any weird unrelated issues that interfere with our later debugging results.

Following the error message include ‘<stdio.h>’ or provide a declaration of ‘printf’ tells us precisely how to fix the error. C becomes aware of functions it can link with by using header files. Including stdio.h tells gcc about the definition of printf.

> #include <stdio.h>
 
> int main() {
        printf("hello world!\n");
>       return 0;
  }

It’s a function call

I claim that printf is not a function call. Let’s either verify or disprove this conclusion. We can run strace on hello_world and save the output to a file “hello_world.strace”.

$ gcc hello_world.c
$ strace -o hello_world.strace ./a.out

strace on this binary returns an overwhelming 45 lines of output, which confirms much more is happening.

Abridged output:

$ strace ./a.out
execve("./a.out", ["./a.out"], 0x7ffc3c8056d0 /* 158 vars */) = 0
brk(NULL)                               = 0x33d6f000
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f5e57473000
---------- snipped for clarity ----------
write(1, "hello world!\n", 13)          = 13
exit_group(13)                          = ?
+++ exited with 13 +++

Full output (expand)

$ strace ./a.out
execve("./a.out", ["./a.out"], 0x7ffc3c8056d0 /* 158 vars */) = 0
brk(NULL)                               = 0x33d6f000
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f5e57473000
access("/etc/ld-nix.so.preload", R_OK)  = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/sga46w4h0l00adh433634s7kp724czvn-shell/lib/glibc-hwcaps/x86-64-v4/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/nix/store/sga46w4h0l00adh433634s7kp724czvn-shell/lib/glibc-hwcaps/x86-64-v4/", 0x7ffc1f6fbf30, 0) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/sga46w4h0l00adh433634s7kp724czvn-shell/lib/glibc-hwcaps/x86-64-v3/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/nix/store/sga46w4h0l00adh433634s7kp724czvn-shell/lib/glibc-hwcaps/x86-64-v3/", 0x7ffc1f6fbf30, 0) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/sga46w4h0l00adh433634s7kp724czvn-shell/lib/glibc-hwcaps/x86-64-v2/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/nix/store/sga46w4h0l00adh433634s7kp724czvn-shell/lib/glibc-hwcaps/x86-64-v2/", 0x7ffc1f6fbf30, 0) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/sga46w4h0l00adh433634s7kp724czvn-shell/lib/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/nix/store/sga46w4h0l00adh433634s7kp724czvn-shell/lib/", 0x7ffc1f6fbf30, 0) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/nqb2ns2d1lahnd5ncwmn6k84qfd7vx2k-glibc-2.40-36/lib/glibc-hwcaps/x86-64-v4/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/nix/store/nqb2ns2d1lahnd5ncwmn6k84qfd7vx2k-glibc-2.40-36/lib/glibc-hwcaps/x86-64-v4/", 0x7ffc1f6fbf30, 0) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/nqb2ns2d1lahnd5ncwmn6k84qfd7vx2k-glibc-2.40-36/lib/glibc-hwcaps/x86-64-v3/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/nix/store/nqb2ns2d1lahnd5ncwmn6k84qfd7vx2k-glibc-2.40-36/lib/glibc-hwcaps/x86-64-v3/", 0x7ffc1f6fbf30, 0) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/nqb2ns2d1lahnd5ncwmn6k84qfd7vx2k-glibc-2.40-36/lib/glibc-hwcaps/x86-64-v2/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/nix/store/nqb2ns2d1lahnd5ncwmn6k84qfd7vx2k-glibc-2.40-36/lib/glibc-hwcaps/x86-64-v2/", 0x7ffc1f6fbf30, 0) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/nqb2ns2d1lahnd5ncwmn6k84qfd7vx2k-glibc-2.40-36/lib/libc.so.6", O_RDONLY|O_CLOEXEC) = 4
read(4, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0@\244\2\0\0\0\0\0"..., 832) = 832
pread64(4, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"..., 784, 64) = 784
fstat(4, {st_mode=S_IFREG|0555, st_size=2335712, ...}) = 0
pread64(4, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"..., 784, 64) = 784
mmap(NULL, 2067928, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 4, 0) = 0x7f5e5727a000
mmap(0x7f5e572a2000, 1474560, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 4, 0x28000) = 0x7f5e572a2000
mmap(0x7f5e5740a000, 352256, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 4, 0x190000) = 0x7f5e5740a000
mmap(0x7f5e57460000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 4, 0x1e5000) = 0x7f5e57460000
mmap(0x7f5e57466000, 52696, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f5e57466000
close(4)                                = 0
mmap(NULL, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f5e57277000
arch_prctl(ARCH_SET_FS, 0x7f5e57277740) = 0
set_tid_address(0x7f5e57277a10)         = 467906
set_robust_list(0x7f5e57277a20, 24)     = 0
rseq(0x7f5e57278060, 0x20, 0, 0x53053053) = 0
mprotect(0x7f5e57460000, 16384, PROT_READ) = 0
mprotect(0x403000, 4096, PROT_READ)     = 0
mprotect(0x7f5e574aa000, 8192, PROT_READ) = 0
prlimit64(0, RLIMIT_STACK, NULL, {rlim_cur=16384*1024, rlim_max=RLIM64_INFINITY}) = 0
fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(0x88, 0x2), ...}) = 0
getrandom("\xc3\xaa\x18\x11\xb5\x45\x97\x82", 8, GRND_NONBLOCK) = 8
brk(NULL)                               = 0x33d6f000
brk(0x33d90000)                         = 0x33d90000
write(1, "hello world!\n", 13)          = 13
exit_group(13)                          = ?
+++ exited with 13 +++

Oh my. Have we even gotten closer to the solution? Of the following output, the only recognizable line is write(1, "hello world!\n", 13) = 13, which tells us that printing is using a write syscall to send the output. The output has a lot of information we do not understand yet, and I do not know what data we want to look for. On the positive side, at least the output fits on a screen now. Tracing the output hasn’t gotten us anywhere and these syscalls still look daunting.

We need to break out our debugging hats :tophat:. If we have faith that we will be able to solve this problem, then we will be able to figure out what is going on. Let us try repeating the strategy of looking at the output of our tools. This time gcc and the standard output on the command line haven’t suggested anything to us.

This will be quite tedious. Can we offload the busy work an LLM and see what it says? We live in the 21st century, so although I wouldn’t trust an LLM to give the correct answer, it can helpfully suggest terms to look up.

Instructor: You are going down the wrong track. Let’s back up.

We need to take a moment to think about this approach. strace is too hard to interpret because it requires us to first understand what all these syscalls are.

Debugging Process:

↛ Load and trace a “simple” program ⟶ ❌

↛ Load a minimal program

⇝ Idea: printf is a function call

↛ check compiler output ⟶ ❌

↛ check strace output ⟶ use LLM ⟶ ❌

⇝ dead end…?

Maybe we should take a look at a different level of abstraction? There is nothing more we can learn from the source C file. It seems like we need to go deeper. Is there another file we have?

Binary spelunking

What is inside this a.out thing? I know it is a compiled binary, so this must contain all the machine code instructions that are running, right?

$ vim a.out

Uh, that’s not right. I don’t think there is anything useful here to see. I do see some characters rendering correctly, but they look like nonsense. I see “ELF” and then some file paths at the start and end of this file.

Let’s put a pin in this stuff we have seen in case it comes up later. Writing a log of what you have tried while debugging is essential to be able to keep all the relevant context in your head. What works, as well as what didn’t, helps narrow the scope of debugging.

So far, strace did not help. We have some hints on how to proceed. We can either look at the filepaths in the binary file, or we can look further into what gcc was telling us about the implicit definition of printf.

How to open a binary?

:monocle_face:: Dear search engine, how can I open a.out C binary file?

The the tool we need is objdump. objdump comes installed with the gcc package. This bundling indicates that objdump is a default tool important to be familiar with.

But first, I want to slightly modify the program we are looking at. Our hello_world.c is deterministic*, which means it does the same thing every time. Instead, let’s make something more realistic, such as a program that asks for the user’s name and responds.

“Hello {name}”

> void main(int argc, char** argv) {
>   if (argc < 2) {
>     printf("hello: error with argc\n");
>     return -1;
>   }
<   printf("hello world!\n");
>   printf("hello, %s!\n", argv[1]);
  return 0;
  }

Now we print out a name given to us from the command line, as so ./a.out Samir. Take a moment to think about whether you understand the change I made.

Back to the objdump

$ objdump # first look at the help output
<omitted>
$ objdump -D ./a.out
<abbreviated for readability>
0000000000401050 <main>:
  401050:       48 83 ec 08             sub    $0x8,%rsp
  401054:       83 ff 01                cmp    $0x1,%edi
  401057:       7e 1e                   jle    401077 <main+0x27>
  401059:       48 8b 56 08             mov    0x8(%rsi),%rdx
  40105d:       bf 02 00 00 00          mov    $0x2,%edi
  401062:       31 c0                   xor    %eax,%eax
  401064:       48 8d 35 b0 0f 00 00    lea    0xfb0(%rip),%rsi        # 40201b <_IO_stdin_used+0x1b>
  40106b:       e8 d0 ff ff ff          call   401040 <__printf_chk@plt>
  401070:       31 c0                   xor    %eax,%eax
  401072:       48 83 c4 08             add    $0x8,%rsp
  401076:       c3                      ret
  401077:       48 8d 3d 86 0f 00 00    lea    0xf86(%rip),%rdi        # 402004 <_IO_stdin_used+0x4>
  40107e:       e8 ad ff ff ff          call   401030 <puts@plt>
  401083:       83 c8 ff                or     $0xffffffff,%eax
  401086:       eb ea                   jmp    401072 <main+0x22>
  401088:       0f 1f 84 00 00 00 00    nopl   0x0(%rax,%rax,1)
  40108f:       00

I see some nonsense in the objdump. I cannot recognize this binary is the same as my original hello_name.c.

One tip is to try doing the simplest possible test when trying a new strategy. Minimizing the experiment ensures we do not get overloaded with unnecessary context.

First, we must recompile with debug flags to make this easier. I think we have explored our way to the right level of abstraction on what printf is actually doing. Binary spelunking is the perfect way to see what the CPU runs to make “hello world” show up.

$ gcc -g hello_world.c
$ objdump -D ./a.out

We must be persistent. Making new systems wouldn’t be fun if it was easy. By the end, I promise your appreciation for the immense scope of real systems will grow.

There are many flags to objdump, which you can find on the man page. We might need to try some random flags before finding something useful. Again, the goal is to both induct new hypotheses on what is happening and deduct possibilities that are definitely not valid explanations for the behavior we are seeing ².

Debugging tools let us slice the problem space into smaller pieces until we’ve dissected every behavior in the system.

$ objdump -p ./a.out
<omitted>
$ objdump -h ./a.out
<omitted>

Ooh, I like the output of -h. It’s concise and readable. We are getting somewhere, so I want to be more thoughtful about what step we take next. Trying a few more commands…

$ objdump -t ./a.out
<omitted>
$ objdump -d ./a.out
<omitted>
$ objdump -D ./a.out
a.out:     file format elf64-x86-64

Disassembly of section .interp:
0000000000400318 <.interp>:
---------- snipped for clarity ----------

… this also looks helpful. Let’s recap. What can we check out next? There was the thing called “ELF”. I dismissed it before when looking at the binary, but it keeps coming up. I also want to look deeper into the disassembly.

A spoiler: we are going to need to understand both of these things, so the order doesn’t matter.

dumpster fire

There’s a lot of output from this disassembly, 1257 lines. Scrolling through the assembly, it does not seem productive to jump into tracing the entire program; we need to start with something smaller.

where art thou, printf?

Find printf with grep.

$ objdump -D ./a.out | grep "printf"
<output is empty...>

We turned on debug symbols, what gives?

:fearful:: wtf? :skull:

:cursing_face:: You know, I have a great idea how to finish debugging this. Let’s give up.

The challenge is what makes debugging fun! Anticipate how much satisfaction you’ll feel when we figure this out together! Is there something simple we missed, similar to how we skipped over the seemingly irrelevant references to “ELF”?

Another low hanging fruit to investigate is the objdump section titles. Analyzing the sections will give us a foothold on the huge assembly output.

$ objdump -D ./a.out | grep "section"
Disassembly of section .interp:
Disassembly of section .note.gnu.property:
Disassembly of section .note.ABI-tag:
Disassembly of section .hash:
Disassembly of section .gnu.hash:
Disassembly of section .dynsym:
Disassembly of section .dynstr:
Disassembly of section .gnu.version:
Disassembly of section .gnu.version_r:
Disassembly of section .rela.dyn:
Disassembly of section .rela.plt:
Disassembly of section .init:
Disassembly of section .plt:
Disassembly of section .text:
<snipped>

Nevermind, these sections are more information that we do not know what to do with. Since we’re stuck, we need to backtrack and try another path forward. Instead of using my contrived program, let’s go back to a simple real program like nautilus.

objdump -D $(which nautilus) | grep "printf"
0000000000401080 <__asprintf_chk@plt>:
  401080:       ff 25 4a 3f 00 00       jmp    *0x3f4a(%rip)        # 404fd0 <__asprintf_chk@GLIBC_2.8>
  40132f:       e8 4c fd ff ff          call   401080 <__asprintf_chk@plt>

What? I was not expecting this at all. I was staring at this wall we got stuck at and thought I would have to try a bunch of things to get through. But this is something we didn’t expect. Why did changing the binary we use make a difference?

We have found something that doesn’t fit in our mental model. Great! Now we get an opportunity to learn something cool and update our knowledge.

Does this weird thing also happen for hello_name.c? Indeed, we see printf.

$ objdump -D ./a.out | grep "printf"
0000000000401040 <__printf_chk@plt>:
  401040:       ff 25 a2 2f 00 00       jmp    *0x2fa2(%rip)        # 403fe8 <__printf_chk@GLIBC_2.3.4>
  401067:       e9 d4 ff ff ff          jmp    401040 <__printf_chk@plt>

Comparing the objdump of hello_name and hello_world reveals two different kinds of library calls.

  40106b:       e8 d0 ff ff ff          call   401040 <__printf_chk@plt>

  40107e:       e8 ad ff ff ff          call   401030 <puts@plt>

The hello_world binary doesn’t have any printf, but hello_name has both. What is the difference between puts and printf? The binary appears to call one reference to puts@plt and one to __printf_chk@plt. puts is simpler than printf because it does not do string formatting (ex: %d). If you are printing a fixed literal string, then the compiler will probably make a call to the more efficient puts function instead.

We can get back to debugging the main issue of what printf is doing. The code is a few lines, so we can check this manually. Let us go into the objdump where this is found, no expert tooling needed:

$ objdump -D ./a.out | less
# then type "/printf" <enter>

Disassembly of section .plt:

0000000000401020 <puts@plt-0x10>:
  401020:       ff 35 aa 2f 00 00       push   0x2faa(%rip)        # 403fd0 <_GLOBAL_OFFSET_TABLE_+0x8>
  401026:       ff 25 ac 2f 00 00       jmp    *0x2fac(%rip)        # 403fd8 <_GLOBAL_OFFSET_TABLE_+0x10>
  40102c:       0f 1f 40 00             nopl   0x0(%rax)

0000000000401030 <puts@plt>:
  401030:       ff 25 aa 2f 00 00       jmp    *0x2faa(%rip)        # 403fe0 <puts@GLIBC_2.2.5>
  401036:       68 00 00 00 00          push   $0x0
  40103b:       e9 e0 ff ff ff          jmp    401020 <_init+0x20>

0000000000401040 <__printf_chk@plt>:
  401040:       ff 25 a2 2f 00 00       jmp    *0x2fa2(%rip)        # 403fe8 <__printf_chk@GLIBC_2.3.4>
  401046:       68 01 00 00 00          push   $0x1
  40104b:       e9 d0 ff ff ff          jmp    401020 <_init+0x20>

Disassembly of section .text:

0000000000401050 <main>:
  401050:       83 ff 01                cmp    $0x1,%edi
  401053:       7e 17                   jle    40106c <main+0x1c>
  401055:       48 8b 56 08             mov    0x8(%rsi),%rdx
  401059:       bf 02 00 00 00          mov    $0x2,%edi
  40105e:       31 c0                   xor    %eax,%eax
  401060:       48 8d 35 b4 0f 00 00    lea    0xfb4(%rip),%rsi        # 40201b <_IO_stdin_used+0x1b>
  401067:       e9 d4 ff ff ff          jmp    401040 <__printf_chk@plt>

Disassembly of section .plt:
[...]
0000000000401040 <__printf_chk@plt>:
  401040:       ff 25 a2 2f 00 00       jmp    *0x2fa2(%rip)        # 403fe8 <__printf_chk@GLIBC_2.3.4>
  401046:       68 01 00 00 00          push   $0x1
  40104b:       e9 d0 ff ff ff          jmp    401020 <_init+0x20>

Disassembly of section .text:
[...]
0000000000401050 <main>:
  [...]
  401067:       e9 d4 ff ff ff          jmp    401040 <__printf_chk@plt>

Pretty Little Thing

We see all these references to plt. The Internet informs us that the PLT is some sort of table with pointers to the real locations of functions.

Why would we not know where printf is? We must remember this function is from libc. In fact, the dump notes printf is defined from GLIBC version 2.3.4.

We have debunked the theory that printf is a normal function call. Then what is it? You will be unable to find a definition of printf in the binary, regardless of how hard you look.

:hear_no_evil:: how can I be sure printf is actually a part of libc? You’re some rando on the internet, not my teacher. :dizzy_face:

Instructor: Spoken like a true skeptic.

:hear_no_evil:: When I run a binary, all the OS does is go to main() and then start executing instructions. My entire binary has been printed, so something is definitely wrong with objdump.

Alas, another twist in the road. First one nitpick–main() is not the first thing that runs. The dump has a _start symbol which is the entrypoint that libc uses to call main³. Yes, it is the case that we have the whole binary. Think back to when we opened a.out in vim. Did we see something in addition to some uninterpretable binary? Hint: at the start and end of the file.

Return of the Elves

Yes! There are ELF headers. The Internet reveals that ELF is a file format. The readelf tool is used to inspect ELF headers. Let’s look at this data for hints about our elusive printf.

The man page suggests to use -a to print all headers. We can use our newfound knowledge of PLTs to grep for plt. This leads us to the “Relocation section” of the readelf output. For readability, we can print this one section and inspect it.

$ readelf -r ./a.out
Relocation section '.rela.dyn' at offset 0x5f0 contains 2 entries:
  Offset          Info           Type           Sym. Value    Sym. Name + Addend
000000403ff0  000100000006 R_X86_64_GLOB_DAT 0000000000000000 __libc_start_main@GLIBC_2.34 + 0
000000403ff8  000300000006 R_X86_64_GLOB_DAT 0000000000000000 __gmon_start__ + 0

Relocation section '.rela.plt' at offset 0x620 contains 2 entries:
  Offset          Info           Type           Sym. Value    Sym. Name + Addend
000000403fe0  000200000007 R_X86_64_JUMP_SLO 0000000000000000 puts@GLIBC_2.2.5 + 0
000000403fe8  000400000007 R_X86_64_JUMP_SLO 0000000000000000 __printf_chk@GLIBC_2.3.4 + 0

Lucky! printf has conveniently appeared here. Indeed, these addresses match up with the offsets we saw in the binary. A call to printf will dereference the value at the offset and use that implementation # 403fe8 <__printf_chk@GLIBC_2.3.4>.

We can see that the compiler has told us in the “Sym[bol] Value” column that printf is located at 0x0. But wait, isn’t that a suspicious address? 0x0 is a NULL pointer.

We have all the information we need. Let’s continue backtracking and synthesizing this new intel we gathered. Let’s go back to strace. This will capture everything happening at runtime and the order in which it happens.

Ignoring the noisy failures of looking for libc, the program takes a few steps.

openat(AT_FDCWD, "/nix/store/nqb2ns2d1lahnd5ncwmn6k84qfd7vx2k-glibc-2.40-36/lib/libc.so.6", O_RDONLY|O_CLOEXEC) = 4
read(4, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0@\244\2\0\0\0\0\0"..., 832) = 832
pread64(4, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"..., 784, 64) = 784
fstat(4, {st_mode=S_IFREG|0555, st_size=2335712, ...}) = 0
pread64(4, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"..., 784, 64) = 784
mmap(NULL, 2067928, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 4, 0) = 0x7f5c576cd000
mmap(0x7f5c576f5000, 1474560, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 4, 0x28000) = 0x7f5c576f5000
mmap(0x7f5c5785d000, 352256, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 4, 0x190000) = 0x7f5c5785d000
mmap(0x7f5c578b3000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 4, 0x1e5000) = 0x7f5c578b3000
mmap(0x7f5c578b9000, 52696, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f5c578b9000
close(4)                                = 0

If you roll up your sleeves and do some good, old-fashioned documentation sleuthing, you can understand what this previously overwhelming strace output means. We also know that our LLM above was speaking confidently about things it doesn’t know. Although it looked believable, it got many facts completely wrong. Instead of LLMs, let’s apply our willpower to see what these straced lines mean. man 2 has all the info we need. You can see more detail in now it’s your turn.

1. openat(libc.so.6): open the libc executable file on disk
2. read: read libc.so.6 from disk into memory
3. pread64: continue reading from where it left off
4. mmap: map the library into the process address space as readable and executable, but not writable. Allocates some writable memory.
5. close: close the file descriptor of the library

Student: I’m still not satisfied. We never saw any reference to any logic of looking for libc in our objdump.

Instructor: That’s the spirit. Do you have any ideas?

Student: The first thing that pops into my mind is let’s open this file .../libc.so.6 in vim

Instructor: Good idea. I can’t discern anything from this file, but I do see this “ELF” mention again. What do we do when we see this?

Student: readelf!

$ readelf -a ...path/to/libc.so.6

There is a lot of output, but we can take a few seconds to skim the output. I piped this multi-thousand line output into less and held the spacebar to skim if anything catches my eye. I see a huge section “Symbol table ‘.dynsym’ contains 3158 entries” which has a lot of “FUNC” entries. This table appears to contain a reference to every function in libc. But why are these all in the dynamic symbol section? It appears this file doesn’t know where these functions are located. The rabbit hole goes deeper; let’s not lose hope.

Go Deeper, Buddy

We first need to search up some background information to understand what is going on. The Procedure Linkage Table (PLT) contains stub code which looks up the location of a function in the Global Offset Table (GOT) and jumps there. The GOT is an array of pointers which holds the actual addresses of these functions.

Instructor: That’s great you know that. But what is it actually doing?

We can use gdb to gain confidence in (or to invalidate) our new theory. We know the address of our plt, so let’s print out that memory and confirm it gets loaded even before our code runs.

Tracing PLT and GOT

#include <stdio.h>

int main() {
        printf("hello world!\n");
        printf("hello world!\n");
        return 0;
}

$ gcc -fno-pie -no-pie -g explore_plt.c
$ gdb a.out
(gdb) b explore_plt.c:4
Breakpoint 1 at 0x401040: file explore_plt.c, line 4.
(gdb) b explore_plt.c:5
Breakpoint 2 at 0x40104e: file explore_plt.c, line 5.
(gdb) r

(gdb) disass main
Dump of assembler code for function main:
   0x0000000000401050 <+0>:     sub    $0x8,%rsp
   0x0000000000401054 <+4>:     mov    $0x402000,%edi
   0x0000000000401059 <+9>:     call   0x401020 <puts@plt>
   0x000000000040105e <+14>:    mov    $0x402000,%edi
   0x0000000000401063 <+19>:    add    $0x8,%rsp
   0x0000000000401067 <+23>:    jmp    0x401020 <puts@plt>
End of assembler dump.
(gdb) disass 'puts@plt'
Dump of assembler code for function puts@plt:
   0x0000000000401030 <+0>:     jmp    *0x2fb2(%rip)        # 0x403fe8 <puts@got.plt>
   0x0000000000401036 <+6>:     push   $0x0
   0x000000000040103b <+11>:    jmp    0x401020
End of assembler dump.
(gdb) p/x *(void**)0x403fe8
$1 = ...
# Exercise: What does this line print at different points in the function?

The jmp instruction in the plt is a jump which follows a pointer to the jump location. gdb left a comment telling us that this value points to the entry for the “puts” symbol in the GOT. Cool! We can confirm this pointer lives in the area of virtual memory which strace told us that libc got read to. This discovery confirms printf is not a function call inside our code. Your OS’s libc provides the implementation.

If we look back at the strace and search for this address 0x403fe8, we will see this line get matched.

mprotect(0x403000, 4096, PROT_READ)     = 0

This line makes the table entries read only. The pwning blog post in further reading explains more. In context of our program, this means that the entire GOT got resolved and then marked as read only for security⁴.

The execution flows from the assembly which points to the PLT that contains a shim to do a GOT lookup. The happy path leads directly to the library code. On the first use of a library function, the execution falls through using the red path, which asks some code to locate the library function.

In the process’ memory layout, the GOT entries initially point to loader executable code, then to the location of the instructions inside the library.

Knowledge Check

Armed with this knowledge, we can both understand the world and bend it to our will. Let’s use gdb to change where the function points. If I write a second function and recompile, I can modify execution from printf to my own function! (Left as an exercise for you in this section.) How does it feel to be able to fully understand the system and control its behavior?

With what you know, you should be able to complete this fun and difficult exercise using normal control flow. Can you find an input to this function which causes the printf line to run?

#include <stdlib.h>
#include <unistd.h>
#include <stdio.h>
#include <string.h>

int target;

void hello()
{
  printf("code execution redirected! you win\n");
  _exit(1);
}

void vuln()
{
  char buffer[512];

  fgets(buffer, sizeof(buffer), stdin);

  printf(buffer);

  exit(1);  
}

int main(int argc, char **argv)
{
  vuln();
}

Source: protostar, updated version

Background knowledge: LiveOverflow video

Solution: LiveOverflow video

Even if you do not go through the exercise, this video is well worth watching. It is far easier to understand the execution flow through the GOT and PLT in video form.

We learned how Linux loads processes and dynamically loads libraries. We even know how to manipulate a running program. The “GOT and PLT for pwning” in further reading explains how to pwn the PLT/GOT.

Now it’s your turn

We have covered everything now. Can you explain it? Do it, I’m serious. I’m listening from the other side of your screen. Draw out all the steps.

To make sure we haven’t missed anything, we can map these steps to the each syscall strace lists when running a program.

1. Pressing enter with ./a.out in bash
2. bash will fork and then execve ./a.out
3. Linux reads the ELF and loads the headers
4. ld-loader
5. libc start
6. run app
7. printf
8. exit

For a detailed explanation of every line, expand this dropdown:

syscall tracing a dynamic-linked 'Hello World' program with 1 format argument
maizure.org

$strace ./printf1
1  - execve("./printf1", ["./printf1"], [/* 47 vars */]) = 0
2  - brk(NULL)                               = 0x1dde000
3  - mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f59bce82000
4  - access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
5  - open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
6  - fstat(3, {st_mode=S_IFREG|0644, st_size=83694, ...}) = 0
7  - mmap(NULL, 83694, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f59bce6d000
8  - close(3)                                = 0
9  - open("/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
10 - read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20\35\2\0\0\0\0\0"..., 832) = 832
11 - fstat(3, {st_mode=S_IFREG|0755, st_size=2127336, ...}) = 0
12 - mmap(NULL, 3940800, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f59bc89f000
13 - mprotect(0x7f59bca57000, 2097152, PROT_NONE) = 0
14 - mmap(0x7f59bcc57000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1b8000) = 0x7f59bcc57000
15 - mmap(0x7f59bcc5d000, 16832, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f59bcc5d000
16 - close(3)                                = 0
17 - mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f59bce6c000
18 - mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f59bce6a000
19 - arch_prctl(ARCH_SET_FS, 0x7f59bce6a740) = 0
20 - mprotect(0x7f59bcc57000, 16384, PROT_READ) = 0
21 - mprotect(0x600000, 4096, PROT_READ)     = 0
22 - mprotect(0x7f59bce83000, 4096, PROT_READ) = 0
23 - munmap(0x7f59bce6d000, 83694)           = 0
24 - fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 0), ...}) = 0
25 - mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f59bce81000
26 - write(1, "Hello World 1\n", 14Hello World 1)         = 14
27 - exit_group(0) = ?
+++ exited with 0 +++

Line 1  - Program is forked from bash and calls execve() to load ./printf1 with the default argument of program name.
Line 2  - Program calls brk() to find the current end of the data segment, which is 0x1dde000. If this program wanted
          space on the heap, there would be subsequent calls to brk() using higher values. That's not the case this time.
Line 3  - Memory maps a page of read-write memory that is not backed by a file (all 0s) and not sharable. At 0x7f59bce82000
Line 4  - Checks permissions of user-specified shared libraries. There are none, so this access attempt fails.
Line 5  - Opens the file containing the list of directors to search for shared libraries. Returned as file descriptor 3.
Line 6  - Checks the status of the new file descriptor. File is 21 pages of memory size (83694 bytes, 4096 bytes per page).
Line 7  - Memory maps the directory list as private and read-only. It now lives between 0x7f59bce6d000 - 0x7f59bce81000.
Line 8  - Closes the shared-library directory list, file descriptor 3 is available again
Line 9  - Opens the symbolic link to the shared standard library (libc.so.6) as file descriptor 3.
Line 10 - Reads the first 832 bytes of the library, header + extras for 64-bit. Success, with 832 bytes read.
Line 11 - Gets the file data for the shared library. Size is 2127336 (520 pages of memory)
Line 12 - Memory map the standard library as read-able and executable, but not writable. It will live at 0x7f59bc89f000.
Line 13 - Removes all access to ~2MB of memory after libc code (likely guard pages)
Line 14 - Memory maps 6 more pages from libc (pages 440-445) as read-write. Scratch space?
Line 15 - Maps four more pages of generic read-write memory between 0x7f59bcc5d000 and 0x7f59bcc62000
Line 16 - Closes the shared standard library file. File descriptor 3 is now available again.
Line 17 - Memory maps a page of read-write memory, probably to suppose thread usage.
Line 18 - Memory maps another 2 pages of memory for read and write. This is probably used for thread local storage
Line 19 - Sets FS to the newly reserved thread local storage area
Line 20 - Sets 4 pages of the standard library to read only
Line 21 - Sets the read-only data application segment (subsequent page is .BSS)
Line 22 - Sets a page of the dynamic linker library to read-only
Line 23 - Unmaps the shared library directory memory space, since the library has been loaded
Line 24 - Gets the status of the stdout file descriptor (/dev/pts/0)
Line 25 - Memory maps a page used in printf's temporary buffer for format resolution
Line 26 - Printf's write syscall for Hello World 1
Line 27 - Exit with status 0

We can be confident that these are the steps that get taken. We found a path that definitely explains the chain of events. However, the question remains: who is doing all this work? Nothing happens by magic. There is code somewhere that is populating our PLT table, and it sure is not in a.out. Or is it?

There are two cases. Either this logic is in a.out or it is done by the kernel during the execve syscall to set up the process. It’s easy to blame behavior we don’t understand on things we don’t understand. In either case, we should be able to trace what is going on. What if I told you dynamic loading does not use any special kernel features? We saw every syscall that gets called. If strace is observing a syscall happening, then it must be getting triggered by something in userspace. All the syscalls we see getting called are completely general. I don’t see syscalls called anything like “dynamic load” to suggest a special mechanism is happening. Unix is all about simplicity. Given this hint, let’s take it from the top and trace to see where the code is coming from.

Look at the bytes of a.out in vim again. I can see there is a bunch of text at the start and end of the binary. There is compile-time info at the end. This suggests that the text at the end is debug info and the text at the start is the ELF header.

If we carefully scan the binary and readelf this time, there is this file path in both places, which we have been skipping.

[Requesting program interpreter: /nix/store/nqb2ns2d1lahnd5ncwmn6k84qfd7vx2k-glibc-2.40-36/lib/ld-linux-x86-64.so.2]

If you have seen bash or python scripts, this is like the #! shebang placed at the start of a file⁵. This file path leads to a loader program which does the work of loading our libraries.

:exploding_head:: What is the loader doing to reach the start of the program?

The loader copies code and data from an executable file on disk into memory. The loader starts execution of the program at the entrypoint specified in the ELF header. The address of the first instruction is usually the start of .text, which contains a _start symbol that runs some CPU specific assembly. This architecture-specific assembly sets up memory and any bookkeeping that need to be done before starting main() by calling __libc_start_main ⁶.

Isn’t that cool? We loaded libraries with standard kernel mechanisms. The kernel knows how to read ELF headers and called the binary at the provided absolute file path. The kernel does not implement any logic to load libraries.

Now this will blow your mind.

Programs are files. Code is just binary wrapped with some ELF headers sitting on your disk drive. We can see dynamic linking is done using normal file operations. All it takes to dynamically load is to map the library into the memory space at runtime. Using the mmap syscall does just this! Programs can modify their own address space and load libraries however they wish. No special kernel support required! Linux provides us with these general syscall mechanisms which we can flexibly use.

If we zoom out, what was the point of dynamically loading in the first place? We have observed most programs dynamically load libc. But it is all code in the end, why not include it in the same file? Indeed, this would have less runtime overhead since you don’t have to do any loading.

Dynamic linking saves a lot of disk space by deduplicating the code for libc functions like printf. Libraries save disk space and make updating libraries as easy as swapping a single file. Note the size differences between the source code and static vs. dynamically linked binaries.

$ ls -lah hello_world*
-rw-r--r-- 1 user group   64 Apr 11 00:00 hello_world.c
-rwxr-xr-x 1 user group  16K Apr 11 00:00 hello_world.dynamic
-rwxr-xr-x 1 user group  22K Apr 11 00:00 hello_world.static

But, if we load many programs, then we still have the issue that these libc functions are now duplicated in the memory of each program.

Instructor: How would you do that if you were designing it? How would we need to modify library loading to support sharing the same physical memory for every process?

Consider that loading libraries simply puts bytes into memory. All applications run in their own virtual memory spaces. What if we could load libc once and then share this same physical memory.

The diagram⁷ shows how we could facilitate this sharing. We can keep exactly one copy of a library in physical memory and each process’ virtual addresses map into it.

Let’s dive into what mmap is doing to hypothesize how to add support for sharing physical memory.

:disguised_face:: Hoooold on. This doesn’t make any sense. The libraries may need to be loaded into a different place in memory for each process depending on the libraries they specify and the process’ memory layout. It is impossible for a program to be at different points in memory because the addresses in the instructions will be wrong.

Very thoughtful! The libraries are instructions, so any pointers you create will be in a different stack for that process. Pointers should be correct within each individual process. And remember, even if the library is loaded in a different place, our program can locate the libraries by looking up the symbol in the PLT/GOT. This solves the relocation problem from the perspective of our user program. However, the libraries have the challenge that they cannot rely on using fixed addresses in their code because they do not know where they will get loaded in memory. For symbols that call into other libraries it is easy as we can reuse the same idea of symbol to address lookup tables.

Recall in assembly you need to specify a destination address to jump to. Luckily, compiler engineers invented Position Independent Code (PIC/PIE) to support the shared library use case. Using gcc with -pie will make the compiler emit instructions that only use relative addressing, allowing it to run from any start address. Here is a concrete example exercise that shows the conversion of a normal assembly to position independent.

Gimme a man

We need to get a better understanding of what the kernel is doing when we load a library to figure out how many processes changes the way libraries get loaded. We can start by reverse engineering the actions the kernel takes in the strace. Tracing how the library gets loaded by the mmap call should give us some ideas how to support memory sharing.

mmap(NULL, 3940800, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f59bc89f000

The mmap man documentation explains the behavior of mmap and its flags. man mmap notes to check the other manual, so I check man 2 mmap.

Most of the arguments to mmap are self explanatory from the function prototype: void *mmap(void addr[.length], size_t length, int prot, int flags, int fd, off_t offset);. However, the flag MAP_PRIVATE appears to significant change the way mmap operates.

$ man mmap

MMAP(3P)                                      POSIX Programmer's Manual                                      MMAP(3P)

PROLOG
       This  manual  page  is  part of the POSIX Programmer's Manual.  The Linux implementation of this interface may
       differ (consult the corresponding Linux manual page for details of Linux behavior), or the interface  may  not
       be implemented on Linux.

[...]

       The  parameter  flags provides other information about the handling of the mapped data.  The value of flags is
       the bitwise-inclusive OR of these options, defined in <sys/mman.h>:
                                      ┌───────────────────┬─────────────────────────┐
                                      │ Symbolic Constant │       Description       │
                                      ├───────────────────┼─────────────────────────┤
                                      │ MAP_SHARED        │ Changes are shared.     │
                                      │ MAP_PRIVATE       │ Changes are private.    │
                                      │ MAP_FIXED         │ Interpret addr exactly. │
                                      └───────────────────┴─────────────────────────┘

       It is implementation-defined whether MAP_FIXED shall be supported.  MAP_FIXED shall be supported  on  XSI-con‐
       formant systems.

       MAP_SHARED and MAP_PRIVATE describe the disposition of write references to the memory object. If MAP_SHARED is
       specified,  write references shall change the underlying object. If MAP_PRIVATE is specified, modifications to
       the mapped data by the calling process shall be visible only to the calling process and shall not  change  the
       underlying  object.   It is unspecified whether modifications to the underlying object done after the MAP_PRI‐
       VATE mapping is established are visible through the MAP_PRIVATE mapping. Either MAP_SHARED or MAP_PRIVATE  can
       be specified, but not both. The mapping type is retained across fork().

[...]

$ man 2 mmap
mmap(2)                                          System Calls Manual                                          mmap(2)

NAME
       mmap, munmap - map or unmap files or devices into memory

LIBRARY
       Standard C library (libc, -lc)

SYNOPSIS
       #include <sys/mman.h>

       void *mmap(void addr[.length], size_t length, int prot, int flags,
                  int fd, off_t offset);
       int munmap(void addr[.length], size_t length);

       See NOTES for information on feature test macro requirements.

DESCRIPTION
       mmap()  creates  a  new mapping in the virtual address space of the calling process.  The starting address for
       the new mapping is specified in addr.  The length argument specifies the length of the mapping (which must  be
       greater than 0).

[...]

       MAP_PRIVATE
              Create a private copy-on-write mapping.  Updates to the mapping are not visible to other processes map‐
              ping the same file, and are not carried through to the underlying  file.   It  is  unspecified  whether
              changes made to the file after the mmap() call are visible in the mapped region.

The MAP_PRIVATE flag tells mmap to “Create a private copy-on-write mapping”. This is a key phrase. Without getting into the weeds of what copy-on-write (COW) entails, this point means that each process’ call to map libc will share the same physical memory. There is no coordination needed between processes, nor any special kernel mechanism to support shared libraries.

:thinking:: Wait, so it is magic?

Operating systems sure are magic.

By exercising your curiosity, we have quickly reached the frontier of how libraries get loaded. The fun part of diving into everyday things is you find out that everything is created by real people. I find it immensely rewarding to learn about the history behind how these abstractions have developed. The mmap docs give us a glimpse into kernel lore.

RATIONALE
       After considering several other alternatives, it was decided to adopt the mmap() definition found in SVR4  for
       mapping  memory objects into process address spaces. The SVR4 definition is minimal, in that it describes only
       what has been built, and what appears to be necessary for a general and portable mapping facility.

       Note that while mmap() was first designed for mapping files, it is actually a general-purpose  mapping  facil‐
       ity. It can be used to map any appropriate object, such as memory, files, devices, and so on, into the address
       space of a process.

Instructor: Cool. You know what the docs claim about mmap, but what sequence of instructions actually run to support sharing?

Kernel Sourcerers

Ooh let’s look at source code. We are still in the realm of theory and haven’t proved our ideas are what gets run.

Instead of pulling the latest kernel, I like using a web search such as livegrep or Elixir bootlin. We want to find the entrypoint of calling the mmap syscall. Searching “mmap” does not get anywhere. Stackoverflow tells us to look for sys_mmap or the macro which defines syscalls ⁸.

arch/arm64/kernel/sys.c

SYSCALL_DEFINE6(mmap, unsigned long, addr, unsigned long, len,
		unsigned long, prot, unsigned long, flags,
		unsigned long, fd, unsigned long, off)
{
	if (offset_in_page(off) != 0)
		return -EINVAL;

	return ksys_mmap_pgoff(addr, len, prot, flags, fd, off >> PAGE_SHIFT);
}

The GitHub symbols panel makes it easy to find definitions and references of symbols. GitHub reveals the definition of ksys_mmap_pgoff is in mmap.c.

The filename indicates we are in the right place. A manual inspection of this file shows us that the core logic is in do_mmap. Here we have a source code comment telling us the exact mechanics of what is happening. It fills in some of the implementation details.

/**
 * do_mmap() - Perform a userland memory mapping into the current process
 * address space of length @len with protection bits @prot, mmap flags @flags
 * (from which VMA flags will be inferred), and any additional VMA flags to
 * apply @vm_flags. If this is a file-backed mapping then the file is specified
 * in @file and page offset into the file via @pgoff.
 * [...]
 */
unsigned long do_mmap(struct file *file, unsigned long addr,
			unsigned long len, unsigned long prot,
			unsigned long flags, vm_flags_t vm_flags,
			unsigned long pgoff, unsigned long *populate,
			struct list_head *uf)
{ [...] }

do_mmap()

if ((prot & PROT_READ) && (current->personality & READ_IMPLIES_EXEC))
		if (!(file && path_noexec(&file->f_path)))
			prot |= PROT_EXEC;

	/* force arch specific MAP_FIXED handling in get_unmapped_area */
	if (flags & MAP_FIXED_NOREPLACE)
		flags |= MAP_FIXED;

	if (!(flags & MAP_FIXED))
		addr = round_hint_to_min(addr);

	/* Careful about overflows.. */
	len = PAGE_ALIGN(len);
	if (!len)
		return -ENOMEM;

	/* offset overflow? */
	if ((pgoff + (len >> PAGE_SHIFT)) < pgoff)
		return -EOVERFLOW;

	/* Too many mappings? */
	if (mm->map_count > sysctl_max_map_count)
		return -ENOMEM;

You can see all this function does is a lots of permission checking. Here we find the source of truth for the MAP_PRIVATE flag⁹.

MAP_PRIVATE case

	case MAP_PRIVATE:
			if (!(file->f_mode & FMODE_READ))
				return -EACCES;
			if (path_noexec(&file->f_path)) {
				if (vm_flags & VM_EXEC)
					return -EPERM;
				vm_flags &= ~VM_MAYEXEC;
			}

			if (!file->f_op->mmap)
				return -ENODEV;
			if (vm_flags & (VM_GROWSDOWN|VM_GROWSUP))
				return -EINVAL;
			break;

code after permissions checking

	addr = mmap_region(file, addr, len, vm_flags, pgoff, uf);
	if (!IS_ERR_VALUE(addr) &&
	    ((vm_flags & VM_LOCKED) ||
	     (flags & (MAP_POPULATE | MAP_NONBLOCK)) == MAP_POPULATE))
		*populate = len;
	return addr;
}

I follow this technique to trace functions. I sequentially find the definition of referenced symbols, recursively searching within as needed. Once you find a function, tracing backwards from the return code helps to reduce the irrelevant code to sift through.

We repeat tracing the function calls mmap_region ⟶ __mmap_region

/**
 * mmap_region() - Actually perform the userland mapping of a VMA into
 * current->mm with known, aligned and overflow-checked @addr and @len, and
 * correctly determined VMA flags @vm_flags and page offset @pgoff.
 *
 * This is an internal memory management function, and should not be used
 * directly.
 *
 * The caller must write-lock current->mm->mmap_lock.
 *
 * @file: If a file-backed mapping, a pointer to the struct file describing the
 * file to be mapped, otherwise NULL.

We can ignore the details about virtual memory; the full details of Linux Virtual Memory Areas are in this document. We are focusing on tracing the path to loading the backing library file. If you continue tracing you will get stuck. How is the file never actually used in any of this code? This dead end demonstrates why looking at real code gets complicated. We need to first understand what we are looking at.

Linux tries to be as fast as possible. It will not load data from disk unless necessary. Thus, this mmap call is a lie. It is not loading any files. The kernel simply sets up page table entries for the virtual memory space.

Instructor: Can you think about how the OS would know when it has to load a page in?

The OS can rely on the hardware generating a page fault when the user tries to access this unloaded memory. The OS marks the page table entries as invalid, which causes the hardware to generate a fault. This execution transfer allows the OS to inspect what it should do in response, such as loading the backing memory or killing the process with a segmentation fault.

You might guess that we need to look at the logic on a page fault of one of these addresses. Inside of arm64’s do_page_fault we see where the memory mapping code gets called

arch/arm64/mm/fault.c

static int __kprobes do_page_fault(unsigned long far, unsigned long esr,
				   struct pt_regs *regs)
{
  [...]
	fault = handle_mm_fault(vma, addr, mm_flags | FAULT_FLAG_VMA_LOCK, regs);

⟶ handle_mm_fault

vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address,
			   unsigned int flags, struct pt_regs *regs)
{
  [...]
		ret = __handle_mm_fault(vma, address, flags);

⟶ __handle_mm_fault

static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma,
		unsigned long address, unsigned int flags)
{

Are you getting a hang of it? The high quality kernel code makes it easy to explore and pick up their naming conventions. For example, do_ functions do the work, sys_ and SYSCALL_DEFINE help you find where syscalls enter, and __function_name functions are internal functions that implement core logic and get called after all permission checks. Reading unfamiliar code is difficult, but you can follow the same process we have been using: recursively explore function definitions until you understand what the acronyms are, what functions do, and what the jargon means in context.

Going back to __handle_mm_fault, you will realize these function calls are not relevant. They have to do with unrelated unhappy paths that can get hit on a page fault. At a high level, this code walks and allocates the multi-level page table.

⟶ handle_pte_fault

static vm_fault_t handle_pte_fault(struct vm_fault *vmf)
{
 [...]

	if (!vmf->pte)
		return do_pte_missing(vmf);

	if (!pte_present(vmf->orig_pte))
		return do_swap_page(vmf);

	if (pte_protnone(vmf->orig_pte) && vma_is_accessible(vmf->vma))
		return do_numa_page(vmf);

Now the fault must get handled. The backing memory must become valid before the OS can return to the faulted program. In our case, the address that faulted is inside libc, which needs to be placed into our process’ address space from disk.

Overview of Page Fault Execution Flow

• Check if the Page Middle Directory (PMD) is present.
  • If the Page Middle Directory (PMD) entry is missing, it means there is no page table yet. If the PTE is empty (pte_none()), then the page is not allocated yet, so we will handle it as a missing page.
• Get or allocate a Page Table Entry (PTE). Determine the type of fault and handle it accordingly:
  • Page is missing → Allocate a new page (do_pte_missing()).
  • Page is swapped out → Swap it back in (do_swap_page()).
  • Page migration/NUMA fault → Handle it (do_numa_page()).
  • Write protection fault → Handle copy-on-write (COW) (do_wp_page()).
  • Otherwise, mark the page as accessed and dirty if needed.

When the faulted page is missing or swapped out, the execution path continues as follows:

• Page is missing → do_pte_missing()
  • If the mapping is anonymous, a new page is allocated.
  • If the mapping is file-backed, the function calls filemap_fault() to fetch data from disk.
• File-backed page → filemap_fault()
  • If the page is not in memory, it requests disk access via the block layer.
• Disk access happens via the filesystem & block layer
  • The requested page is fetched from disk into the page cache. Once loaded, the page is mapped and execution continues.

⟶ We fall into do_pte_missing because the page for this mmap has not been created nor has the file been loaded.

This page is not a normal stack or heap page, so we need to do work to load the file.

⟶ do_fault

The code branches on the types of faults ⟶ This is a do_read_fault ⟶ __do_fault

This leads to a function pointer for the exact fault handler we want to use for this filesystem

This layer of indirection (VFS) allows the implementation to be general for the exact filesystem in use.

ret = vma->vm_ops->fault(vmf);

The types help us figure this out struct vm_fault ⟶ vm_area_struct ⟶ vm_operations_struct

Bingo. We now know what the type of fault is: vm_fault_t (*fault)(struct vm_fault *vmf);.

Although we have hit a dead end, applying the same exploration tricks gets us to filemap_fault. grep finds us many definitions, but we can choose the ext4 filesystem as an example. We can use the types to find this does nothing other than call the generic implementation of filemap_fault. Yet again, the comments on the function are extremely helpful and let us confirm we are in the right place.

Let’s update our snapshot of our current understanding.

Summary of Execution Path

1. The kernel walks the page table in __handle_mm_fault(). This generates a page fault.
2. A missing file-backed page triggers handle_pte_fault().
3. It calls do_pte_missing(), which routes to do_fault().
4. do_fault() identifies the fault as a read fault and calls do_read_fault().
5. This invokes vma->vm_ops->fault(), which for files is filemap_fault().
6. filemap_fault() checks if the page is in the page cache.
   • If present → Map it into the process and return.
   • If missing → Request disk I/O via the block layer[^2].
7. The page is read from disk into the page cache.
8. The process blocks while the page is mapped, and execution resumes.

This looks much more precise than before! We can continue recursively unfolding function defintions. There is a call to filemap_get_folio which returns a folio type. The type’s comment indicates something interesting. Both the filesystem function and this function mention a “page cache”

folio

 * A folio is a physically, virtually and logically contiguous set
 * of bytes.  It is a power-of-two in size, and it is aligned to that
 * same power-of-two.  It is at least as large as %PAGE_SIZE.  If it is
 * in the page cache, it is at a file offset which is a multiple of that
 * power-of-two.

A folio is a reference counted page cache entry. We are loading a new file for the first time, so this page will not be in the page cache.

Instructor: Or will it?

What file are we loading again? We are loading libc. Effectively, every program is using libc. This file will surely be in memory already. Alas, this file is in use by other processes, so we cannot reuse the file.

Instructor: Or can we?

Let’s look back at the syscall we are executing in the first place. Take a look at the flags. We only want to execute libc. Thus, every program can share the same read-only copy of this physical memory mapped into their virtual address spaces.

This is magic. There was no coordination between the programs reading this same file. No special mechanisms are used in the kernel. The same ideas that support virtual memory are working in harmony to make dynamic library loading invisibly work.

Now, let’s follow the path when a file-backed page is missing and needs to be read from disk.

Hardware tries to access an uninitialized page

⟶ handle_pte_fault(): If the page is not present, it calls do_fault().

⟶ do_fault(): If the mapping is from a file, it calls filemap_fault().

⟶ filemap_fault(): This function tries to load the missing page from disk. It calls page_cache_read() if the page is not already cached.

⟶ page_cache_read() Reads the page from the filesystem’s page cache. If the page is not cached, it requests disk access via the block layer.

There you have it! The library has been mapped into the process from the cache. Our process can continue running with shared libraries, oblivous to all this work the kernel has done in the background. Yet, the kernel did not do anything special to support this one library sharing use case! Every kernel action was on behalf of a user syscall.

Recap

The solution is elegant: there is no special “dynamic linking” support in the kernel.

These general purpose mechanisms are used:

• mmap() with MAP_PRIVATE creates copy-on-write mappings
• Page faults trigger on-demand loading
• The page cache automatically shares identical file content between processes
• Virtual memory lets each process see libraries at different addresses while sharing the same physical memory

At a high level:

More precisely:

It’s for the pedagogy ¹⁰

:lying_face:: Holupaminnit, how did we get to printf and page faults? I thought we were talking about loading binaries?

Alas, I have tricked you into learning something new. One way to ask this question is to ask what happens when there is a printf in your code, but I think the framing of our initial question at the start led us along a more interesting route.

Undoubtedly, there are many other aspects of loading a program I have skipped. The point to take home is that exploration is nonlinear. All the irrelevant things we tried while inspecting system behavior turned out to be completely relevant! There is a lot going on, but I hope you will believe me and believe yourself that you could have figured this out. The approach we took for figuring this puzzle out is great because it works in general. For kernel code, the better approach is probably to go directly to kernel documentation and to start reading the source code. However, documentation is not enough to see what precisely happens beneath the hood and make changes.

What we didn’t cover

Think about these for yourself.

• Can you write a C program to test your mental model? Can you spelunk your kernel to find out? Remember, just because LLMs act authoritative, it does not mean they know anything.
• How do we know when a program has finished executing? What does it mean to be the last instruction of the file?
• How does printf actually work. This post is supremely interesting. I love the presentation of the post and it goes into much more detail than I get into here. If you think you can explain how printf works, the author will humble you.
• We stopped at the assembly level, but how is each of these instructions actually executing? What hardware is involved? What is the hardware doing?
• Can you have multiple versions of the same library installed?
• If you remove the return statement in hello_world.c, can you explain why the program returns 13? Hint: what does the void keyword mean? Where are return values stored?
• Why does a simple program with no library calls, such as addition, still load libc? Hint: try looking at the output of strace. What is the last system call called by the program?
• Can you figure out what dynamic library an address belongs to in the virtual address space of a program on your computer? $ cat /proc/$(pidof binary_name)/maps. What is the program being run? You should be able to tell from the sections.

You now have all the tools to learn anything you want. To keep this knowledge, you must exercise it. I leave you with a quest: embark on your personal journey of Linux spelunking and discover what happens when you start running a program with the exec syscall. You can find a sample solution in the “The exec Functions” section of Understanding the Linux Kernel. The “How programs get run” series by LWN also has a great explanation (part 1, part 2).

Wrapping up

In this post, we demystified how Linux loads processes and discovered how Linux works on a journey that reminds me of the joys of computers and discovery.

The great part about learning things this way is that you feel empowered to learn anything. The power of open source allows us to figure absolutely anything out with enough effort. We also do not need to trust anything. We looked at the source ourselves and can be quite confident that we have created a mental model accurate to the real world.

This post was inspired by this video on Linux signals. It is one of the best videos I have ever seen. I rewatch it every few months to remind myself how beautiful it is to be able to understand these precise and complex beasts which are the machines we use. I hope you can appreciate the beauty of the ridiculous world we live in. All it takes is a sense of wonder and you can build anything.

Thank you for reading! This article took me 49 hours over 8 weeks. I have surely gotten details wrong in the post. I am sorry. Point them out! Please email me any feedback at process-loading <at> this domain or leave it in the comments below.

The objective is curiosity maximization. Happy exploring.

Further Reading:

• Best resource I have found on shared libraries: How To Write Shared Libraries by Ulrich Drepper
• Dynamic linker wiki
• mmap wiki
• How statically linked programs run on Linux (and how you get from libc to the start symbol)
• Understanding Linux ELF RTLD internals
• GOT and PLT for pwning
• man ld.so: Describes in detail all the places dynamic libraries come from and how you indicate what gets loaded.
  • ldd trivia: it has arbitrary code execution as it may run the target binary
• Try compiling some files and disassmbling them. Here are some flags to make it more readable. objdump --disassemble=main --no-show-raw-insn --visualize-jumps --disassembler-color=on a.out | less
• “Libraries” part of Understanding the Linux Kernel, 3rd Edition
• a similar blog post for tracing ls
• Anatomy of a system call part 1, part 2
• You can be a wizard

Resources for filesystem details I glossed over:

• For more details on the exact actions to read from disk, blk layer slides
• An Introduction to the Linux Kernel Block I/O Stack
• How the Linux VFS, block layer, and device drivers fit together
• Linux VFS and Block Layers
• The Generic Block Layer from Understanding the Linux Kernel

• kernel VFS docs

• If you want extreme detail on Linux Virtual Memory, there’s an entire book. The “Page Faulting” section is a good supplement to this post.

I tried to make every logical jump explicit as an educational method. When you are debugging for real, you should reach to the best tool for the job instead of the winding path I took. For example, I used vim to open files to show that binary is not scary. You would never look at a binary in a normal text editor like vim–reach directly for objdump or readelf. If high level tools don’t work, then you can use a hex viewer.

Another much faster path to figuring out what a program does is to use a disassembler. We were manually going through the process of a small part of what disassemblers can automatically do for us. This would have shown what hello_world.c is actually doing as something like this

function main {
  puts("Hello world!");
  exit(0x0);
  return;
}

Of course, there is a lot more code running. You can use a full-fat disassembler such as Ghidra. This dogbolt web disassembly of a hello world program is enough to get an idea of what the output looks like.

You can read the full paper here, or read on for some relevant excerpts.

Embedded and connected devices have become a ubiquitous and integral part of our world. From medical devices and smart home products to industrial controllers, these devices perform an increasing set of complex, critical tasks. To abstract these complexities, embedded operating systems (OSes) evolved to provide rich development environments and high-level abstractions, such as platform-independent hardware models and separable tasks. Despite the availability of these abstractions and isolation mechanisms for tasks, we find that in practice many embedded applications lack strong isolation and do not enable security reinforcements. This paper explores projects using embedded OSes to quantify how configurable security features are used and how they factor into the given project’s design.

Historically, embedded devices did not offer connectivity. For instance, industrial controllers or automotive Electronic Control Units (ECUs) required a technician physically access and “plug-into” the device. As embedded devices have evolved into the Internet of Things (IoT), many now offer internet connectivity to gather data, offer remote control, and provide firmware updates. Developers now expect embedded OSes to provide library features such as over-the-air updates and network stacks. Internet connectivity and the global scale of many billions of deployed devices implies that embedded devices now face an attack surface more akin to that of a traditional operating system. As such, the security embedded OSes provide is paramount.

With this connectivity, embedded devices now face a more complex attack surface, underscoring the importance of device security. Embedded operating systems are able to span diverse application and hardware domains because they are highly configurable. This flexibility, however, implies that downstream embedded applications may be flexible in how they use security features. This paper investigates how downstream applications use configurable security features in practice. We find that a majority of applications do not alter the default configuration provided by their chosen runtime, and as a result, do not utilize available security options. Early evidence suggests that this under-utilization is due to both runtime and development overhead.

Result 1: Zephyr & FreeRTOS MPU Feature Usage.

Zephyr and FreeRTOS offer similar MPU-based security configurations, and both disable process isolation and stack overflow protection by default. Of surveyed Zephyr and FreeRTOS applications, 17% of Zephyr and 8% of FreeRTOS projects enable both process isolation and hardware stack guards. We subsequently observe that 89% of surveyed FreeRTOS/Zephyr projects do not enable the full suite of opt-in, configurable MPU security features. We do not include RIOT in this result as RIOT does not support MPU process isolation.

Result 2: SW/HW Stackguard Usage.

Surveyed FreeRTOS repositories exhibit a more than sixfold increase of software-based stackguards usage in comparison to hardware-based stack guard usage.

Result 3: Opt-out Configuration Usage.

For RIOT platforms possessing an MPU, RIOT enables mpu_stack_guard by default. We find that no surveyed project disables this feature.

Result 4: Runtime Overhead.

Our preliminary benchmarking of Zephyr in Result 4 finds that hardware security features impose a mere 224 cycle overhead per context switch. This result agrees with the documentation excerpts and shows an overhead that is in fact noticeable, but likely only problematic for high performance applications. We argue that the vast majority of IoT embedded applications do not require “every last drop of performance,” but instead that the primary overhead impeding the usage of configurable security features is not performance but developer overhead.

This paper shows the disparity between the configurable security features the studied embedded OSes offer and the features that are actually used in the wild. Our preliminary results and analysis suggest the primary overhead limiting the use of these features is developer overhead. Nonetheless, further investigation is required to gain a more complete understanding of why these features are left disabled by developers. So long as downstream applications see limited use in enabling such optional features, the potential security benefits embedded OSes can provide is left unrealized. This is particularly problematic given the increased attack surface of modern IoT devices.

Read the full paper

Results

Counts by pronoun

The count for “person” is tainted by all of its usages in license verbiage. Similarly, “man” is polluted with all the references to “man pages”.

I pruned non-pronoun usages. I did not include uses of “they” or “them” because I cannot grep to distinguish between uses of the gender-neutral pronoun “they” or as the plural for neuter terms (such as referring to function calls or variable values).

Pronoun usage over time

Analysis

The kernel overwhelmingly uses male pronouns. This result is expected due to the >90% male contributor population. Bitergia looked at the contributions of men and women to the Linux kernel in 2016. Their study found that women produced 6.8% of git activity as 9.9% of the developer population. This analysis was based on identifying gender from names.

I expected there to only be uses of male or gender-neutral phrases. This hypothesis led me to inspect every usage of female pronouns.

Excluding some gender-neutral uses of “he/she”, “his/her”, and “[s]he”, the results for grepping “her” are peculiar.

“her” usage

“she” usage

“woman”/”women”

The only reference to a “woman” in the kernel is in a eulogy

I would ask that anyone benefiting from this work, especially those
using it in commercial products, consider making a donation to my local
non-profit hospice organization in the name of the woman I loved, who
passed away Feb. 12, 2003.

And the only usage of “women” is in contribution guidelines documentation.

Swearing in Linux

This paper shows there is a correlation between swear word usage and code quality. In unrelated news, there is a large, negative trend line for the amount of swearing in Linux over the past ten years.

Age

The kernel continues to evolve as new developers come in. The amount of contributors joining and staying on the Linux project keeps dropping.

For example, efforts to push Rust into Linux have been causing controversy. Eventually, Linus himself will have to step down. Being a maintainer is stressful, and even maintainers are human.

Further, contributions are done less than ever by open source hackers. Linux contributions are becoming more commercial, which means that contributors’ incentives may not align with that of the Linux project as a whole. Over 90% of contributors are being paid by their employer to submit patches. Nobody wants to pay to build the infrastructure to make Linux development better (ex: Coccinelle) or to write high quality documentation. These create barriers to the natural growth of the project among younger developers and other demographics.

Part 1

By inspecting the code, how many processes are created and how many times is ls run?

#include <stdio.h>
#include <unistd.h>

int main (int argc, char *arg[])
{
    fork ();

    if (fork ()) {
      fork ();
    } else {
        char *argv[2] = {"/bin/ls", NULL};
        execv (argv[0], argv);
        fork ();
    }
}

Part 2

What is the output of this program? How many times do you think “start” and “else” are printed?

#include <stdio.h>
#include <unistd.h>

int main (int argc, char *arg[])
{
    printf("start");
    fork ();

    if (fork ()) {
      fork ();
    } else {
        char *argv[2] = {"/bin/ls", NULL};
        execv (argv[0], argv);
        printf("else");
        fork ();
    }
}

$ ./a.out 
startstartstartstarta.out  main.c
a.out  main.c

Part 3

With access to edit and run the code, explain this behavior.

int main (int argc, char *arg[])
{
    printf("start");
    raise(SIGABRT);
}

Thanks to Anon. Calc and Professor Voelker.

Tock OS secures millions of devices and uses language, architectural, and formal techniques to statically enforce system guarantees. This talk will go over how the Tock operating system eliminates many bugs that plague other OSes at compile time. I will demonstrate how we are mitigating logic bugs in Tock OS by using lightweight formal verification (Flux).

Watch now

Watch on YouTube Watch on Vimeo

Slides

View slides in fullscreen

Find the pdf of the slides and the source code for the slides and demo in this repo.

Connect with us!

Tock: safe embedded operating system

• Tock is open source!
• tockos.org
• Join Tock Slack, Matrix, and verification mailing list

Flux: scalable Rust verification

Proving should be as easy as programming

• github.com/flux-rs/flux/
• Live demo

Share the presentation permalink: godsped.com/safe-firmware

I want to credit the amazing support and work on this project done by Pat Pannuto, Nico Lehmann, Evan Johnson, and Ranjit Jhala.

I must to give an enormous thanks to OSFC for sponsoring me to present. If you found this presentation interesting, you should consider going to the next OSFC!

This post is a reprint of the original post I wrote on the Triton UAS blog. This post has been edited to add COMMENTARY blocks with my own commentary.

Triton UAS is a club at UC San Diego which makes Unmanned Aerial Systems, aka autonomous planes.

LA JOLLA, CA - UC San Diego’s massive campus offers hundreds of clubs to join and thousands of classes to take. When I came to UCSD, I spent a lot of time shopping around for what I wanted to do with my free time here. I read through the list of all clubs multiple times to find a place to belong. I want to share why Triton UAS members chose to join, and why sticking with Triton UAS for four years turned out to be the best decision I could ever make.

1. The project is exciting

Triton UAS participates in the Student Unmanned Aerial Systems competition. Each year, we design and manufacture a plane from scratch to compete. Then our plane autonomously flies an obstacle course, searches for lost targets on the ground, and safely delivers aid to their locations.

The project is so cool! We build a real, double-than-human sized plane that carries out a mission autonomously. It is inspiring to take part in the process of going from nothing to plane. At the end of the year, I can squint up into the sky and see my work flying.

Flying to the moon

2. Making something cool

I love that we build a real system. It is no comparison that an autonomous flying plane is cooler than any class project. Some of the things we do have never been done, by students or professionals. Autonomous vehicles are still a largely unsolved problem — you cannot search for existing solutions to the problems we face. Building planes is tough, and moreso given our time and monetary constraints.

However, we rise to the challenge. Triton UAS is a team effort. It is amazing to create something that I could never do on my own.

3. Mentorship

Triton UAS taught me that you can learn anything. Each team lead is an expert at what they do. The leads will teach you everything you need to know. We have an onboarding plane project which allows new members to hit the ground running. Rookie members learn our manufacturing processes and the onboarding projects help all the teams work together towards a unified goal. Experienced members use this time to work on the design of the competition plane. New recruits get to deploy something they built within their first quarter.

Even outside of TUAS, making friends with upperclasspeople can help you navigate school and choosing your future.

4. Collaboration

There are several subteams which are critical to our mission: airframe, business, embedded, and software. We actively encourage collaboration between teams and make sure that we are foremost learning and having fun. Our airdrop task requires us to design a mechanism to deliver a package to the target. For the dropping mechanism, airframe designed a door in the bottom of the plane and iterated on the internal machinery. Members collaborated with embedded to make the electronics for the guided payload and enlisted software to handle the logic for dropping and making sure that the communications fail safely.

TUAS is in the sweet spot of size. You can get to know everyone in the club. Also, you are not a cog working on a random minor aspect of the plane. Another great part of the community is that you get to interact with the local domain experts. Within TUAS, I have met students who have spent hundreds of hours working with carbon composites, the state of the art of machine learning research, and experts in PCB manufacturing.

Check out why members of our different subteams think:

And remember that these distinctions are fluid. We have people in multiple subteams, or people who jump over for a single project. For example, a software member could hop into the embedded meetings for a quarter to work on our camera gimbal.

5. Competition

Our work culminates in flying out to the competition in Maryland each year. We fly our plane autonomously all the way from California to Maryland! Unfortunately, it only sits autonomously in the cargo hold of a Boeing airliner.

I find the deadline very motivating. The bonding we experience at deadlines is stronger than I have ever experienced elsewhere. Everyone bands together to meet the integration test deadlines. I know that if I ever need help, I can call out and multiple people will come to help me at any time of day or night.

Competition also means that a lot of work has to be done each year; we design and build a new plane every year to meet the ever-changing competition objectives. We are not creating a project that gets thrown away after demoing 24 hours of work or presenting a quarter of work; our plane is not a one-off launch that then turns into an art exhibit. We must work to create reliable systems which will be maintainable by other people. We work to improve our systems year-over-year. In addition to short term goals, we have multi-year projects to improve the plane, such as turning electronics wiring into a custom PCB, developing new Computer Vision models for dynamic detection, and experimenting with new carbon fiber manufacturing techniques for the plane.

TUAS is an established club. TUAS has been around for nearly two decades, whereas many of the other robotics and aviation clubs were founded a few years ago. This means that we have to work with some systems that have been made entirely by graduated students. We have the realistic situation of working with hardware and decisions made long ago.

6. TUAS is approachable

There is no mandatory attendance policy. There is no required previous experience. There are no interviews. There is no member fee. I love that TUAS is open to everyone from every background, experience, and history. In Fall, our number one goal is to mentor new members. We are not screening new hires for their experience to be able to work on their minor part of the system and learn nothing new.

The more you put into the club, the more you get back. We understand people have other obligations, especially with the gruelling quarter system. We just ask that members communicate with us. The great part is that you get to “own” your task and get to make decisions yourself.

We do not just say we are open to everyone, we actually are. You can check out all our meeting times on our website, you can join our Discord server, or meet our members at campus outreach events (library walk tabling, EOTG). We want to help the greater engineering community outside of UCSD. We visit San Diego fairs and schools, publish documentation and tutorials publicly on our wiki, publish a whitepaper of our work, and license our code as Free Software.

7. Culture

The leads are so kind. I have seen that when people see notice someone struggling, they band together to raise the struggling person up. I saw this camaraderie when the software team switched the core flight controller to C++ from Python. To do this, we switched to the CMake build system. We were all unfamiliar with setting up C++ project infrastructure and had to figure out many things. Together, we succeeded against The Great CMake War (c. 2024).

We have open documentation, actively share, and encourage people to attend other subteams’ meetings. I have so much secondhand knowledge about carbon fiber molding from watching and listening to explanations from airframe members. People who have spent thousands of hours on ANSYS modelling entertain my dumb questions. The team is so open minded to new ideas, even from inexperienced members.

This year we were weighing whether to add VTOL (vertical takeoff and landing) to our plane. We needed to make the decision judiciously as has huge implications on both software and hardware. The team came together and openly presented the pro and con considerations at our town halls. We did not devolve into a war between the two sides and did not do this through democracy. All the debate was done in public for anyone to add to, and then the team project manager made the ultimate decision. I like that we trust others’ decisions in TUAS. Instead of silos for each subteam, we have cohesive all hands “townhall” meetings to make sure everyone is on the same page. We have necessary hierarchy because leads need to make choices for cross-subteam projects quickly.

I have not seen any other club that solicits member feedback so much. TUAS sends member CAPES, an evaluation form to give feedback on your experience in the club. Feedback is constantly being asked for and being shared at our townhall meetings.

TUAS is not all serious decision making. There is so much fun club lore and our business team does a fantastic job organization social events and fundraisers. These are some schenanigans that we have definitely not done:

• shorted the electrical board on carbon fiber, setting the plane on fire
• worked in the lab overnight
• set the world record for any% fastest Tapex order
• made a Tapex time leaderboard
• run regression algorithms to predict Tapex order numbers
• carved the Tapex logo into a pumpkin
• flown bread
• there are NO easter eggs at all in our ground control station software
• gotten a plane prop stuck on the motor
• eaten 1000 oreos
• gone to Tuas
• played Minecraft in a hotel until 4am
• snacked on packing peanuts
• canary in the Jacob’s mine
• 1 inch thick exploding laptops
• amogus world record
• yummy orange fumes tier list

Being around other TUAS people is so fun. Going to a school with 40,000 students, I’m accustomed to encountering people randomly and in classes and never seeing them again. But with TUAS, you are growing with the other people. It’s so awesome to be around other curious people who are interested in hearing about what you are working on and meeting people who are passionate about learning. In a huge campus, the TUAS lab feels more like home than my apartment does.

8. Career Opportunities

The culture in TUAS has shown me that these people are in this club because of their passion and not for the resume line item. Nevertheless, the experience is a boon for our future careers. Interviewers often ask me about my work in TUAS, and TUAS has led to me scoring an internship. I know many of our members have gotten jobs at aviation and engineering companies because this club experience sets them apart from people who only know theoretical course material. The effect of TUAS on career outcomes is clear if you look at out alumni.

9. Alumni

You get opportunities to interact with alumni of the club, who have gone onto great graduate programs, founded successful startups, and worked their dream jobs at JPL. While a student, it’s great to have the mentorship of elder students who have gone through the same path as you. Beyond graduation, you can get advice from people who have gone down different paths.

10. And Takeoff

I’ll mention this again. But launching at the end of the year is so fun. The first successful full-system integration test is pure euphoria.

I went from knowing nothing about planes to making one of the coolest things I have ever seen.

11. Mentoring

A last bonus benefit: After making it through your first competition year, you are no longer a rookie. It is an amazing journey slowly becoming the expert that you looked up to as a rookie. We are here to help each other out.

TUAS can be stressful. Dealing with deadline and scrambling as things break during testing can start to consume all your time. But we have bonded and become better engineers, better teammates, and better people having come through the other side.

For all these reasons, the TUAS lab holds a special place in my heart. All the work we have done together and all the people I have met will always stay with me.

You can choose to spend your time anywhere, but you’ll be glad to have spent it with us.

Join us near the start of any quarter. See you in the skies.

This post is inspired by a post from the MIT rocketry team. Check out this post with my unofficial, personal commentary.

Why is Nix a good choice for a home server? Learn how Nix enables maintenance free, secure, and reproducible home servers. The talk will cover why Nix is a powerful choice compared to other technologies like Docker or Ansible. It will also showcase how Nix makes it easy to get up and running with server applications such as Nginx, Wireguard, Jellyfin, Samba, and more.

Watch on YouTube

If you want to set up your own home server, check out our repo with a config to follow along after the talk.

View the slides

View slides in fullscreen

You can download the rendered pdf file here or check out the Markdown source code here.