Linux Kernel Programming: A comprehensive and practical guide to kernel internals, writing modules, and kernel synchronization
()
About this ebook
The 2nd Edition of Linux Kernel Programming is an updated, comprehensive guide for new programmers to the Linux kernel. This book uses the recent 6.1 Long-Term Support (LTS) Linux kernel series, which will be maintained until Dec 2026, and also delves into its many new features. Further, the Civil Infrastructure Project has pledged to maintain and support this 6.1 Super LTS (SLTS) kernel right until August 2033, keeping this book valid for years to come!
You’ll begin this exciting journey by learning how to build the kernel from source. In a step by step manner, you will then learn how to write your first kernel module by leveraging the kernel’s powerful Loadable Kernel Module (LKM) framework. With this foundation, you will delve into key kernel internals topics including Linux kernel architecture, memory management, and CPU (task) scheduling. You’ll finish with understanding the deep issues of concurrency, and gain insight into how they can be addressed with various synchronization/locking technologies (e.g., mutexes, spinlocks, atomic/refcount operators, rw-spinlocks and even lock-free technologies such as per-CPU and RCU).
By the end of this book, you’ll have a much better understanding of the fundamentals of writing the Linux kernel and kernel module code that can straight away be used in real-world projects and products.
Read more from Kaiwan N Billimoria
Linux Kernel Debugging: Leverage proven tools and advanced techniques to effectively debug Linux kernels and kernel modules Rating: 0 out of 5 stars0 ratingsHands-On System Programming with Linux: Explore Linux system programming interfaces, theory, and practice Rating: 0 out of 5 stars0 ratingsLinux Kernel Programming: A comprehensive guide to kernel internals, writing kernel modules, and kernel synchronization Rating: 0 out of 5 stars0 ratings
Related to Linux Kernel Programming
Related ebooks
Mastering Embedded Linux Programming: Create fast and reliable embedded solutions with Linux 5.4 and the Yocto Project 3.1 (Dunfell) Rating: 0 out of 5 stars0 ratingsLinux Device Driver Development: Everything you need to start with device driver development for Linux kernel and embedded Linux Rating: 0 out of 5 stars0 ratingsFedora Linux System Administration: Install, manage, and secure your Fedora Linux environments Rating: 0 out of 5 stars0 ratingsMastering Ubuntu Server: Explore the versatile, powerful Linux Server distribution Ubuntu 22.04 with this comprehensive guide Rating: 0 out of 5 stars0 ratingsFundamentals of Linux: Explore the essentials of the Linux command line Rating: 0 out of 5 stars0 ratingsMastering Embedded Linux Programming Rating: 5 out of 5 stars5/5Learning Docker Networking Rating: 0 out of 5 stars0 ratingsLinux Mint Essentials Rating: 3 out of 5 stars3/5Linux for Networking Professionals: Securely configure and operate Linux network services for the enterprise Rating: 0 out of 5 stars0 ratingsLinux - a Secure Personal Computer for Beginners Rating: 0 out of 5 stars0 ratingsMastering Embedded Linux Programming - Second Edition Rating: 5 out of 5 stars5/5BeagleBone Media Center Rating: 0 out of 5 stars0 ratingsMastering Linux Shell Scripting,: A practical guide to Linux command-line, Bash scripting, and Shell programming Rating: 0 out of 5 stars0 ratingsUsing Yocto Project with BeagleBone Black Rating: 0 out of 5 stars0 ratingsLinux for Beginners: Linux Command Line, Linux Programming and Linux Operating System Rating: 4 out of 5 stars4/5LPI Linux Certification Questions: LPI Linux Interview Questions, Answers, and Explanations Rating: 4 out of 5 stars4/5Practical System Programming with C: Pragmatic Example Applications in Linux and Unix-Based Operating Systems Rating: 0 out of 5 stars0 ratingsExtending Docker Rating: 0 out of 5 stars0 ratingsLearning CoreOS Rating: 0 out of 5 stars0 ratingsLinux Administration Cookbook: Insightful recipes to work with system administration tasks on Linux Rating: 0 out of 5 stars0 ratingsMastering Ubuntu Server.: Master the art of deploying, configuring, managing, and troubleshooting Ubuntu Server 18.04 Rating: 0 out of 5 stars0 ratingsFreePBX 2.5 Powerful Telephony Solutions: Configure, deploy, and maintain an enterprise-class VoIP PBX Rating: 0 out of 5 stars0 ratingsUbuntu Linux Toolbox: 1000+ Commands for Power Users Rating: 4 out of 5 stars4/5All My IT Tech Posts Rating: 0 out of 5 stars0 ratingsContainerization with LXC Rating: 0 out of 5 stars0 ratings
Operating Systems For You
Excel : The Ultimate Comprehensive Step-By-Step Guide to the Basics of Excel Programming: 1 Rating: 5 out of 5 stars5/5Hacking : The Ultimate Comprehensive Step-By-Step Guide to the Basics of Ethical Hacking Rating: 5 out of 5 stars5/5OneNote: The Ultimate Guide on How to Use Microsoft OneNote for Getting Things Done Rating: 1 out of 5 stars1/5PowerShell: A Comprehensive Guide to Windows PowerShell Rating: 4 out of 5 stars4/5Make Your PC Stable and Fast: What Microsoft Forgot to Tell You Rating: 4 out of 5 stars4/5Mastering Windows PowerShell Scripting Rating: 4 out of 5 stars4/5Linux All-In-One For Dummies Rating: 2 out of 5 stars2/5Linux Command-Line Tips & Tricks Rating: 0 out of 5 stars0 ratingsLinux: Learn in 24 Hours Rating: 5 out of 5 stars5/5Raspberry Pi Cookbook for Python Programmers Rating: 0 out of 5 stars0 ratingsThe Mac Terminal Reference and Scripting Primer Rating: 4 out of 5 stars4/5Bash Command Line Pro Tips Rating: 5 out of 5 stars5/5Linux Command Line and Shell Scripting Bible Rating: 3 out of 5 stars3/5RHCSA Red Hat Enterprise Linux 8 (UPDATED): Training and Exam Preparation Guide (EX200), Second Edition Rating: 5 out of 5 stars5/5Networking for System Administrators: IT Mastery, #5 Rating: 5 out of 5 stars5/5The Linux Command Line Beginner's Guide Rating: 4 out of 5 stars4/5CompTIA Linux+ Study Guide: Exam XK0-004 Rating: 0 out of 5 stars0 ratingsMacBook Pro User Manual: 2022 MacBook Pro User Guide for beginners and seniors to Master Macbook Pro like a Pro Rating: 0 out of 5 stars0 ratingsmacOS Sonoma For Dummies Rating: 0 out of 5 stars0 ratingsThe Windows Command Line Beginner's Guide: Second Edition Rating: 4 out of 5 stars4/5Help! iOS 17 - iPhone: How to Use iOS17 Rating: 0 out of 5 stars0 ratingsWindows 11 For Dummies Rating: 0 out of 5 stars0 ratingsLinux Bible Rating: 0 out of 5 stars0 ratingsThe Darknet Superpack Rating: 0 out of 5 stars0 ratingsExploring Windows 11: The Illustrated, Practical Guide to Using Microsoft Windows Rating: 0 out of 5 stars0 ratingsLinux All-in-One For Dummies Rating: 3 out of 5 stars3/5
Reviews for Linux Kernel Programming
0 ratings0 reviews
Book preview
Linux Kernel Programming - Kaiwan N Billimoria
Linux Kernel Programming
Second Edition
A comprehensive and practical guide to kernel internals, writing modules, and kernel synchronization
Kaiwan N. Billimoria
BIRMINGHAM—MUMBAI
Linux Kernel Programming
Second Edition
Copyright © 2024 Packt Publishing
All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.
Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.
Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.
Senior Publishing Product Manager: Aaron Tanna
Acquisition Editor – Peer Reviews: Gaurav Gavas
Project Editor: Rianna Rodrigues
Content Development Editor: Shikha Parashar
Copy Editor: Safis Editing
Technical Editor: Aniket Shetty
Proofreader: Safis Editing
Indexer: Manju Arasan
Presentation Designer: Rajesh Shirsath
Developer Relations Marketing Executive: Meghal Patel
First published: March 2021
Second edition: February 2024
Production reference: 1270224
Published by Packt Publishing Ltd.
Grosvenor House
11 St Paul’s Square
Birmingham
B3 1RB, UK.
ISBN 978-1-80323-222-5
www.packt.com
Contributors
About the author
Kaiwan N. Billimoria learned to code on his Dad’s IBM PC (1983). He was programming in C and Assembly on DOS until he discovered Unix, and soon after, Linux! Kaiwan has worked extensively on the Linux system programming stack, including drivers and embedded Linux. He has actively worked on several commercial/FOSS projects. His contributions include drivers on Linux and many smaller projects hosted on GitHub. His passion for Linux helps when teaching these topics to engineers, which he has done for well over two decades now. He considers his major contribution to be his books: Hands-On System Programming with Linux, Linux Kernel Programming (and its Part 2 book), and Linux Kernel Debugging. He is a recreational runner too.
First, to my wonderful family: my parents, Nads and Diana, my wife, Dilshad, my kids, Sheroy and Danesh, my bro, Darius, and the rest of the family. Thanks for being there! The Packt team has shepherded me through this work with patience and excellence, as usual. A special call out to Rianna Rodrigues, Aaron Tanna, and Aniket Shetty – thanks for all your timely support throughout!
About the reviewer
Chi-Thanh Hoang is currently working as a Senior Lead Radio SW Architect at Mavenir Systems, currently developing O-RAN 5G radios. He has over 30 years of software development experience, specializing mostly in embedded networking systems (switches, routers, Wi-Fi, and mobile networks) from chipsets up to communication protocols and of course kernel/RTOS. His first experience with the Linux kernel was in 1993. He still does hands-on debugging inside the kernel. He has a bachelor’s degree in electrical engineering from Sherbrooke University, Canada. He is also an avid tennis player and invariably tinkers with electronics and software.
Learn more on Discord
To join the Discord community for this book – where you can share feedback, ask questions to the author, and learn about new releases – follow the QR code below:
https://packt.link/SecNet
Contents
Preface
Who this book is for
What’s been added in the second edition?
What this book covers
Get in touch
Linux Kernel Programming – A Quick Introduction
Kernel workspace setup
Technical requirements
Cloning this book’s code repository
Building the 6.x Linux Kernel from Source – Part 1
Technical requirements
Preliminaries for the kernel build
Understanding the Linux kernel release nomenclature
Fingers-and-toes releases
Kernel development workflow – understanding the basics
Viewing the kernel’s Git log via the command line
Viewing the kernel’s Git log via its GitHub page
The kernel dev workflow in a nutshell
Exercise
Exploring the types of kernel source trees
LTS kernels – the new mandate
Which kernel should I run?
Steps to build the kernel from source
Step 1 – Obtaining a Linux kernel source tree
Downloading a specific kernel tree
Cloning a Git tree
Step 2 – Extracting the kernel source tree
A brief tour of the kernel source tree
Step 3 – Configuring the Linux kernel
Minimally understanding the Kconfig/Kbuild build system
How the Kconfig+Kbuild system works – a minimal take
Arriving at a default configuration
Obtaining a good starting point for kernel configuration
Kernel config using distribution config as a starting point
Tuned kernel config via the localmodconfig approach
Kernel config for typical embedded Linux systems
Seeing all available config options
Getting going with the localmodconfig approach
Tuning our kernel configuration via the make menuconfig UI
Sample usage of the make menuconfig UI
Kernel config – exploring a bit more
Searching within the menuconfig UI
Looking up the differences in configuration
Using the kernel’s config script to view/edit the kernel config
Configuring the kernel for security
Miscellaneous tips – kernel config
Customizing the kernel menu, Kconfig, and adding our own menu item
Understanding the Kconfig* files
Creating a new menu item within the General Setup menu
A few details on the Kconfig language
Summary
Exercise
Questions
Further reading
Building the 6.x Linux Kernel from Source – Part 2
Technical requirements
Step 4 – building the kernel image and modules
Getting over a cert config issue on Ubuntu
Step 5 – installing the kernel modules
Locating the kernel modules within the kernel source
Getting the kernel modules installed
Overriding the default module installation location
Step 6 – generating the initramfs image and bootloader setup
Generating the initramfs image – under the hood
Understanding the initramfs framework
Why the initramfs framework?
Understanding the basics of the boot process on the x86
More on the initramfs framework
Peeking into the initramfs image
Step 7 – customizing the GRUB bootloader
Customizing GRUB – the basics
Selecting the default kernel to boot into
Booting our VM via the GNU GRUB bootloader
Experimenting with the GRUB prompt
Verifying our new kernel’s configuration
Kernel build for the Raspberry Pi
Step 1 – cloning the Raspberry Pi kernel source tree
Step 2 – installing an x86_64-to-AArch64 cross-toolchain
Step 3 – configuring and building the Raspberry Pi AArch64 kernel
Miscellaneous tips on the kernel build
Minimum version requirements
Building a kernel for another site
Watching the kernel build run
A shortcut shell syntax to the build procedure
Dealing with missing OpenSSL development headers
How can I check which distro kernels are installed?
Summary
Questions
Further reading
Writing Your First Kernel Module – Part 1
Technical requirements
Understanding the kernel architecture – part 1
User space and kernel space
Library and system call APIs
Kernel space components
Exploring LKMs
The LKM framework
Kernel modules within the kernel source tree
Writing our very first kernel module
Introducing our Hello, world LKM C code
Breaking it down
Kernel headers
Module macros
Entry and exit points
Return values
Common operations on kernel modules
Building the kernel module
Running the kernel module
A quick first look at the kernel printk()
Listing the live kernel modules
Unloading the module from kernel memory
Our lkm convenience script
Understanding kernel logging and printk
Using the kernel memory ring buffer
Kernel logging and systemd’s journalctl
Using printk log levels
The pr_
Writing to the console
Writing output to the Raspberry Pi console
Turning on debug-level kernel messages
Rate limiting the printk instances
Rate-limiting macros to use
Generating kernel messages from user space
Standardizing printk output via the pr_fmt macro
Portability and the printk format specifiers
Understanding the new printk indexing feature
Understanding the basics of a kernel module Makefile
Summary
Questions
Further reading
Writing Your First Kernel Module – Part 2
Technical requirements
A better
Makefile template for your kernel modules
Configuring a debug
kernel
Cross-compiling a kernel module
Setting up the system for cross-compilation
Attempt 1 – setting the ARCH and CROSS_COMPILE environment variables
Attempt 2 – pointing the Makefile to the correct kernel source tree for the target
Attempt 3 – cross-compiling our kernel module
Examining Linux kernel ABI compatibility issues
Attempt 4 – cross-compiling our kernel module
Summarizing what went wrong with the module cross-buildd/load and how it was fixed
Gathering minimal system information
Being a bit more security-aware
Licensing kernel modules
Licensing of inline kernel code
Licensing of out-of-tree kernel modules
Emulating library-like
features for kernel modules
Performing library emulation via linking multiple source files
Understanding function and variable scope in a kernel module
Understanding module stacking
Trying out module stacking
Emulating ‘library-like’ features – summary and conclusions
Passing parameters to a kernel module
Declaring and using module parameters
Getting/setting module parameters after insertion
Learning module parameter data types and validation
Validating kernel module parameters
Overriding the module parameter’s name
Hardware-related kernel parameters
Floating point not allowed in the kernel
Forcing FP in the kernel
Auto-loading modules on system boot
Module auto-loading – additional details
Kernel modules and security – an overview
Proc filesystem tunables affecting the system log
A quick word on the dmesg_restrict sysctl
A quick word on the kptr_restrict sysctl
Understanding the cryptographic signing of kernel modules
The two module-signing modes
Disabling kernel modules altogether
The kernel lockdown LSM – an introduction
Coding style guidelines for kernel developers
Contributing to the mainline kernel
Getting started with contributing to the kernel
Summary
Questions
Further reading
Kernel Internals Essentials – Processes and Threads
Technical requirements
Understanding process and interrupt contexts
Understanding the basics of the process Virtual Address Space (VAS)
Organizing processes, threads, and their stacks – user and kernel space
Running a small script to see the number of processes and threads alive
User space organization
Kernel space organization
Summarizing the kernel with respect to threads, task structures, and stacks
Viewing the user and kernel stacks
Traditional approach to viewing the stacks
eBPF – the modern approach to viewing both stacks
The 10,000-foot view of the process VAS
Understanding and accessing the kernel task structure
Looking into the task structure
Accessing the task structure with current
Determining the context
Working with the task structure via ‘current’
Built-in kernel helper methods and optimizations
Trying out the kernel module to print process context info
Seeing that the Linux OS is monolithic
Coding for security with printk
Iterating over the kernel’s task lists
Iterating over the task list I – displaying all processes
Iterating over the task list II – displaying all threads
Differentiating between the process and thread – the TGID and the PID
Iterating over the task list III – the code
Summary
Questions
Further reading
Memory Management Internals – Essentials
Technical requirements
Understanding the VM split
Looking under the hood – the Hello, world C program
Going beyond the printf() API
Virtual addressing and address translation
VM split on 64-bit Linux systems
Common VM splits
Understanding the process VAS – the full view
Examining the process VAS
Examining the user VAS in detail
Directly viewing the process memory map using procfs
Frontends to view the process memory map
Understanding VMA basics
Examining the kernel VAS
High memory on 32-bit systems
Writing a kernel module to show information about the kernel VAS
Macros and variables describing the kernel VAS layout
Trying it out – viewing kernel VAS details
The kernel VAS via procmap
Trying it out – the user segment
Viewing kernel documentation on the memory layout
Randomizing the memory layout – KASLR
User memory randomization with ASLR
Kernel memory layout randomization with KASLR
Querying/setting KASLR status with a script
Understanding physical memory organization
Physical RAM organization
Nodes and NUMA
Zones within a node
Direct-mapped RAM and address translation
An introduction to physical memory models
Understanding the sparsemem[-vmemmap] memory model in brief
Summary
Questions
Further reading
Kernel Memory Allocation for Module Authors – Part 1
Technical requirements
Introducing kernel memory allocators
Understanding and using the kernel page allocator (or BSA)
The fundamental workings of the page allocator
Understanding the organization of the page allocator freelist
The workings of the page allocator
Working through a few scenarios
Page allocator internals – a few more details
Learning how to use the page allocator APIs
Dealing with the GFP flags
Freeing pages with the page allocator
A few guidelines to observe when (de)allocating kernel memory
Writing a kernel module to demo using the page allocator APIs
Deploying our lowlevel_mem_lkm kernel module
The page allocator and internal fragmentation (wastage)
The GFP flags – digging deeper
Never sleep in interrupt or atomic contexts
Page allocator – pros and cons
Understanding and using the kernel slab allocator
The object caching idea
A few FAQs regarding (slab) memory usage and their answers
Learning how to use the slab allocator APIs
Allocating slab memory
Freeing slab memory
Data structures – a few design tips
The actual slab caches in use for kmalloc
Writing a kernel module to use the basic slab APIs
Size limitations of the kmalloc API
Testing the limits – memory allocation with a single call
Checking via the /proc/buddyinfo pseudofile
Slab allocator – a few additional details
Using the kernel’s resource-managed memory allocation APIs
Additional slab helper APIs
Control groups and memory
Caveats when using the slab allocator
Background details and conclusions
Testing slab allocation with ksize() – case 1
Testing slab allocation with ksize() – case 2
Interpreting the output from case 2
Graphing it
Finding internal fragmentation (wastage) within the kernel
The easy way with slabinfo
More details with alloc_traces and a custom script
Slab layer - pros and cons
Slab layer – a word on its implementations within the kernel
Summary
Questions
Further reading
Kernel Memory Allocation for Module Authors – Part 2
Technical requirements
Creating a custom slab cache
Creating and using a custom slab cache within a kernel module
Step 1 – creating a custom slab cache
Step 2 – using our custom slab cache’s memory
Step 3 – destroying our custom cache
Custom slab – a demo kernel module
Extracting useful information regarding slab caches
Understanding slab shrinkers
Summarizing the slab allocator pros and cons
Debugging kernel memory issues – a quick mention
Understanding and using the kernel vmalloc() API
Learning to use the vmalloc family of APIs
Trying out vmalloc()
A brief note on user-mode memory allocations and demand paging
Friends of vmalloc()
Is this vmalloc-ed (or module region) memory?
Unsure which API to use? Try kvmalloc()
Miscellaneous helpers – vmalloc_exec() and vmalloc_user()
Specifying memory protections
The kmalloc() and vmalloc() APIs – a quick comparison
Memory allocation in the kernel – which APIs to use when
Visualizing the kernel memory allocation API set
Selecting an appropriate API for kernel memory allocation
A word on DMA and CMA
Memory reclaim – a key kernel housekeeping task
Zone watermarks and kswapd
The new multi-generational LRU (MGLRU) lists feature
Trying it out – seeing histogram data from MGLRU
A quick introduction to DAMON – the Data Access Monitoring feature
Running a memory workload and visualizing it with DAMON’s damo front-end
Stayin’ alive – the OOM killer
Deliberately invoking the OOM killer
Invoking the OOM killer via Magic SysRq
Invoking the OOM killer with a crazy allocator program
Understanding the three VM overcommit_memory policies
VM overcommit from the viewpoint of the __vm_enough_memory() code
Case 1: vm.overcommit_memory == 0 (the default, OVERCOMMIT_GUESS)
Case 2: vm.overcommit_memory == 2 (VM overcommit turned off, OVERCOMMIT_NEVER) and vm.overcommit_ratio == 50
Demand paging and OOM
The optimized (unmapped) read
Understanding the OOM score
Closing thoughts on the OOM killer and cgroups
Cgroups and memory bandwidth – a note
Summary
Questions
Further reading
The CPU Scheduler – Part 1
Technical requirements
Learning about the CPU scheduling internals – part 1 – essential background
What is the KSE on Linux?
The Linux process state machine
The POSIX scheduling policies
Thread priorities
Visualizing the flow
Using the gnome-system-monitor GUI to visualize the flow
Using perf to visualize the flow
Trying it out – the command-line approach
Trying it out – the graphical approach
Visualizing the flow via alternate approaches
Learning about the CPU scheduling internals – part 2
Understanding modular scheduling classes
A conceptual example to help understand scheduling classes
Asking the scheduling class
The workings of the Completely Fair Scheduling (CFS) class in brief
Scheduling statistics
Querying a given thread’s scheduling policy and priority
Learning about the CPU scheduling internals – part 3
Preemptible kernel
The dynamic preemptible kernel feature
Who runs the scheduler code?
When does schedule() run?
Minimally understanding the thread_info structure
The timer interrupt housekeeping – setting TIF_NEED_RESCHED
The process context part – checking TIF_NEED_RESCHED
CPU scheduler entry points – a summary
The core scheduler code in brief
Summary
Questions
Further reading
The CPU Scheduler – Part 2
Technical requirements
Understanding, querying, and setting the CPU affinity mask
Querying and setting a thread’s CPU affinity mask
Using taskset to perform CPU affinity
Setting the CPU affinity mask on a kernel thread
Querying and setting a thread’s scheduling policy and priority
Setting the policy and priority within the kernel – on a kernel thread
A real-world example – threaded interrupt handlers
An introduction to cgroups
Cgroup controllers
Exploring the cgroups v2 hierarchy
Enabling or disabling controllers
The cgroups within the hierarchy
Systemd and cgroups
Our cgroups v2 explorer script
Trying it out – constraining the CPU resource via cgroups v2
Leveraging systemd to set up CPU resource constraints on a service
The manual way – a cgroups v2 CPU controller
Running Linux as an RTOS – an introduction
Pointers to building RTL for the mainline 6.x kernel (on x86_64)
Miscellaneous scheduling related topics
A few small (kernel space) routines to check out
The ghOSt OS
Summary
Questions
Further reading
Kernel Synchronization – Part 1
Technical requirements
Critical sections, exclusive execution, and atomicity
What is a critical section?
A classic case – the global i ++
Concepts – the lock
Critical sections – a summary of key points
Data races – a more formal definition
Concurrency concerns within the Linux kernel
Multicore SMP systems and data races
Preemptible kernels, blocking I/O, and data races
Hardware interrupts and data races
Locking guidelines and deadlock
Mutex or spinlock? Which to use when
Determining which lock to use – in theory
Determining which lock to use – in practice
Using the mutex lock
Initializing the mutex lock
Correctly using the mutex lock
Mutex lock and unlock APIs and their usage
Mutex lock – via [un]interruptible sleep?
Mutex locking – an example driver
The mutex lock – a few remaining points
Mutex lock API variants
The semaphore and the mutex
Priority inversion and the RT-mutex
Internal design
Using the spinlock
Spinlock – simple usage
Spinlock – an example driver
Test – sleep in an atomic context
Testing the buggy module on a 6.1 debug kernel
Locking and interrupts
Scenario 1 – driver method and hardware interrupt handler run serialized, sequentially
Scenario 2 – driver method and hardware interrupt handler run interleaved
Scenario 2 on a single-core (UP) system
Scenario 2 on a multicore (SMP) system
Solving the issue on UP and SMP with the spin_[un]lock_irq() API variant
Scenario 3 – some interrupts masked, driver method and hardware interrupt handler run interleaved
Interrupt handling, bottom halves, and locking
Interrupt handling on Linux – a summary of key points
Bottom halves and locking
Using spinlocks – a quick summary
Locking – common mistakes and guidelines
Common mistakes
Locking guidelines
Solutions
Summary
Questions
Further reading
Kernel Synchronization – Part 2
Technical requirements
Using the atomic_t and refcount_t interfaces
The newer refcount_t versus older atomic_t interfaces
The simpler atomic_t and refcount_t interfaces
Examples of using refcount_t within the kernel codebase
64-bit atomic integer operators
A note on internal implementation
Using the RMW atomic operators
RMW atomic operations – operating on device registers
Using the RMW bitwise operators
Using bitwise atomic operators – an example
Efficiently searching a bitmask
Using the reader-writer spinlock
Reader-writer spinlock interfaces
Trying out the reader-writer spinlock
Performance issues with reader-writer spinlocks
The reader-writer semaphore
Understanding CPU caching basics, cache effects, and false sharing
An introduction to CPU caches
The risks – cache coherency, performance issues, and false sharing
What is the cache coherency problem?
The false sharing issue
Lock-free programming with per-CPU and RCU
Per-CPU variables
Working with per-CPU variables
Per-CPU – an example kernel module
Per-CPU usage within the kernel
Understanding and using the RCU (Read-Copy-Update) lock-free technology – a primer
How does RCU work?
Trying out RCU
RCU: detailed documentation
RCU usage within the kernel
Lock debugging within the kernel
Configuring a debug kernel for lock debugging
The lock validator lockdep – catching locking issues early
Catching deadlock bugs with lockdep – a few examples
Example 1 – catching a self deadlock bug with lockdep
Example 2 – catching an AB-BA deadlock with lockdep
Brief notes on lockdep – annotations and issues
A note on lockdep annotations
A note on lockdep – known issues
Kernel lock statistics
Viewing and interpreting the kernel lock statistics
Introducing memory barriers
An example of using memory barriers in a device driver
A note on marked accesses
Summary
Questions
Further reading
Other Books You May Enjoy
Index
Landmarks
Cover
Index
Preface
This book, in its second edition now, has been explicitly written with a view to helping you learn Linux kernel development in a practical, hands-on fashion, along with the necessary theoretical background to give you a well-rounded view of this vast and interesting topic area. It deliberately focuses on kernel development via the powerful Loadable Kernel Module (LKM) framework; this is because the vast majority of real-world/industry kernel projects and products, which includes device driver development, are done in this manner.
The focus is kept on both working hands-on with, and understanding at a sufficiently deep level, the internals of the Linux OS. In this regard, we cover everything from building the Linux kernel from source to understanding and working with complex topics such as synchronization within the kernel.
To guide you on this exciting journey, we divide this book into three sections. The first section covers the basics – setting up an appropriate workspace for kernel development, building the modern kernel from source, and writing your first kernel module.
The next section, a key one, will help you understand essential kernel internals details; its coverage includes the Linux kernel architecture, the task structure, user - and kernel-mode stacks, and memory management. Memory management is a key and interesting topic – we devote three whole chapters to it (covering the internals to a sufficient extent, and importantly, how exactly to efficiently allocate and free kernel memory). The internal working and deeper details of CPU (task) scheduling on the Linux OS round off this section.
The last section of the book deals with the more advanced topic of kernel synchronization – a necessity for professional design and code on the Linux kernel. We devote two whole chapters to covering key topics here.
The book uses the kernel community’s 6.1 Long Term Support (LTS) Linux kernel. It’s a kernel that will be maintained (both bug and security fixes) from December 2022 right through to December 2026. Moreover, the CIP (Civil Infrastructure Project) has adopted 6.1 as an SLTS (Super LTS) release and plans to maintain it for 10 years, until August 2033! This is a key point, ensuring that this book’s content remains current and valid for years to come!
We very much believe in a hands-on approach: some 40 kernel modules (besides several user apps and shell scripts, double that of the first edition!) in this book’s GitHub repository make the learning come alive, making it fun, practical, interesting, and useful.
We highly recommend you also make use of this book’s companion guide, Linux Kernel Programming Part 2 – Char Device Drivers and Kernel Synchronization: Create user-kernel interfaces, work with peripheral I/O, and handle hardware interrupts. It’s an excellent industry-aligned beginner’s guide to writing
misc
character drivers, performing I/O on peripheral chip memory, and handling hardware interrupts. You can get this book for free along with your print copy; alternately, you can also find this eBook in the GitHub repository at https://github.com/PacktPublishing/Linux-Kernel-Programming/tree/master/Linux-Kernel-Programming-(Part-2).
We really hope you learn from and enjoy this book. Happy reading!
Who this book is for
This book is primarily for those of you beginning your journey in the vast arena of learning and understanding modern Linux kernel architecture and internals, Linux kernel module development and, to some extent, Linux device driver development. It’s also very much targeted at those of you who have already been working on Linux modules and/or drivers, who wish to gain a much deeper, well-structured understanding of Linux kernel architecture, memory management, task scheduling, cgroups, and synchronization. This level of knowledge about the underlying OS, covered in a properly structured manner, will help you no end when you face difficult-to-debug real-world situations.
What’s been added in the second edition?
A pretty huge amount of new material has been added into this, the Second Edition of the Linux Kernel Programming book. As well, being based on the very recent (as of this writing) 6.1 LTS release, its information and even code will remain industry-relevant for many, many years to come.
Here’s a quick chapter-wise summarization of what’s new in this second edition:
Materials updated for the 6.1 LTS kernel, maintained until December 2026, and until August 2033 via the CLP (6.1 SLTS)!
Updated, new, and working code for the 6.1 LTS kernel
Several new info-rich sections added to most chapters, many new diagrams, and new code examples to help explain concepts better
Chapter 1, Linux Kernel Programming – A Quick Introduction
Introduction to the book
Chapter 2, Building the 6.x Linux Kernel from Source – Part 1
The new LTS kernel lifetime mandate
More details on the kernel’s Kconfig+Kbuild system
Updated approaches on configuring the kernel
Chapter 3, Building the 6.x Linux Kernel from Source – Part 2
More details on the
initramfs
(
initrd
) image
Cross-compiling the kernel on an x86_64 host to an AArch64 target
Chapter 4, Writing Your First Kernel Module – Part 1
new-ish
printk
indexing feature covered
Powerful kernel dynamic debug feature introduced
Rate-limiting macros updated (deprecated ones not used)
Chapter 5, Writing Your First Kernel Module – Part 2
A better, ‘better’ Makefile (v0.2)
Chapter 6, Kernel Internals Essentials – Processes and Threads
New linked list demo module
Chapter 7, Memory Management Internals – Essentials
New coverage on how address translation works (including diagrams)
Chapter 8, Kernel Memory Allocation for Module Authors – Part 1
Coverage on using the exact
page allocator API pair
FAQs regarding (slab) memory usage and their answers
The graphing demo (via
gnuplot
) is now automated and even saved to an image file, via a helper script
Finding internal fragmentation (wastage) within the kernel
Chapter 9, Kernel Memory Allocation for Module Authors – Part 2
Extracting useful information regarding slab caches
A word on slab shrinkers
Better coverage on the OOM killer (and
systemd-oomd
) and how it’s triggered; includes a flowchart depicting demand-paging and possible OOM killer invocation
Better coverage on kernel page reclaim, as well as the new MGLRU and DAMON technologies
Chapter 10, The CPU Scheduler – Part 1
New coverage on CFS scheduling period and timeslice. Coverage on the thread_info structure as well
New: the preempt dynamic feature
Enhanced coverage on exactly how and when
schedule()
is invoked
Chapter 11, The CPU Scheduler – Part 2
Much more depth in the powerful cgroups (v2) coverage plus an interesting script to let you explore its content
Leveraging the cgroups v2 CPU controller via both
systemd
and manually to perform CPU bandwidth allocation
A note on Google’s ghOSt OS
Chapter 12, Kernel Synchronization – Part 1
A new intro to the LKMM (Linux Kernel Memory Model)
More on locking plus deadlock avoidance guidelines
Chapter 13, Kernel Synchronization – Part 2
Expanded coverage on CPU caching and cache effects
New coverage on the powerful lock-free RCU synchronization technology
Online Chapter, Kernel Workspace Setup
Fixed errors in package names and versions
Ubuntu-based helper script that auto-installs all required packages
Most (if not all) earlier code errors, typos, and URLs are now fixed, based on prompt feedback, raising Issues/PRs on the book’s GitHub repo, from you, our wonderful readers!
What this book covers
Chapter 1, Linux Kernel Programming – A Quick Introduction, briefs you about the exciting journey in the sections of the book, which cover everything from building the Linux kernel from source to understanding and working with complex topics such as synchronization within the kernel.
Chapter 2, Building the 6.x Linux Kernel from Source – Part 1, is the first part of explaining how to build the modern Linux kernel from scratch with its source code. In this part, you will first be given necessary background information – the kernel version nomenclature, the different source trees, and the layout of the kernel source. Next, you will be shown in detail how exactly to download a stable vanilla Linux kernel source tree onto your Linux VM (Virtual Machine). We shall then learn a little regarding the layout of the kernel source code, getting, in effect, a 10,000-foot view
of the kernel code base. The actual work of extracting and configuring the Linux kernel then follows. Creating and using a custom menu entry for kernel configuration is also explained in detail.
Chapter 3, Building the 6.x Linux Kernel from Source – Part 2, is the second part on performing kernel builds from source code. In this part, you will continue from the previous chapter, now actually building the kernel, installing kernel modules, understanding what exactly the
initramfs
(
initrd
) image is and how to generate it, and setting up the bootloader (for the x86_64). Also, as a valuable add-on, this chapter then explains how to cross-compile the kernel for a typical embedded ARM target (using the popular Raspberry Pi 4 64-bit as a target device). Several tips and tricks on kernel builds, and even kernel security (hardening) tips, are detailed.
Chapter 4, Writing Your First Kernel Module – Part 1, is the first of two parts that cover a fundamental aspect of Linux kernel development – the LKM framework and how it is to be understood and used by you, the module user,
the kernel module or device driver programmer. It covers the basics of the Linux kernel architecture and then, in great detail, every step involved in writing a simple Hello, world
kernel module, compiling, inserting, checking, and removing it from kernel space.
We also cover kernel logging via the ubiquitous
printk
API in detail. This edition also covers
printk
indexing and introduces the powerful dynamic debug feature.
Chapter 5, Writing Your First Kernel Module – Part 2, is the second part that covers the LKM framework. Here, we begin with something critical – learning how to use a better
Makefile, which will help you generate more robust code (this so-called ‘better’ Makefile helps by having several targets for code-checking, code-style correction, static analysis, and so on. This edition has a superior version of it). We then show in detail the steps to successfully cross-compile a kernel module for an alternate architecture, how to emulate library-like
code in the kernel (via both the linking and module-stacking approaches), and how to pass parameters to your kernel module. Additional topics include how to perform auto-loading of modules at boot, important security guidelines, and some information on the kernel coding style and upstream contribution. Several example kernel modules make the learning more interesting.
Chapter 6, Kernel Internals Essentials – Processes and Threads, delves into some essential kernel internals topics. We begin with what is meant by the execution of kernel code in process and interrupt contexts, and minimal but required coverage of the process user virtual address space (VAS) layout. This sets the stage for you; you’ll then learn about Linux kernel architecture in more depth, focusing on the organization of process/thread task structures and their corresponding stacks – user- and kernel-mode. We then show you more on the kernel task structure (a root
data structure), how to practically glean information from it (via the powerful ‘current’ macro), and even how to iterate over various (task) lists (there’s sample code too!). Several kernel modules make the topic come alive.
Chapter 7, Memory Management Internals – Essentials, a key chapter, delves into essential internals of the Linux memory management subsystem, to the level of detail required for the typical module author or driver developer. This coverage is thus necessarily more theoretical in nature; nevertheless, the knowledge gained here is crucial to you, the kernel developer, both for deep understanding and usage of appropriate kernel memory APIs, as well as for performing meaningful debugging at the level of the kernel. We cover the VM split (and how it’s defined on various actual architectures), gaining deep insight into the user VAS as well as the kernel VAS (our
procmap
utility will prove to be an eye-opener here!). We also cover more on how address translation works. We then briefly delve into the security technique of memory layout randomization ([K]ASLR), and end this chapter with a discussion on physical memory organization within Linux.
Chapter 8, Kernel Memory Allocation for Module Authors –Part 1, gets our hands dirty with the kernel memory allocation (and, obviously, deallocation) APIs. You will first learn about the two allocation layers
within Linux – the slab allocator that’s layered above the kernel memory allocation engine,
the page allocator (or BSA). We shall briefly learn about the underpinnings of the page allocator algorithm and its freelist
data structure; this information is valuable when deciding which layer to use. Next, we dive straight into the hands-on work of learning about the usage of these key APIs. The ideas behind the slab allocator (or slab cache) and the primary kernel allocator APIs – the
kzalloc()
/
kfree()
pair (and friends) – are covered. Importantly, the size limitations, downsides, and caveats when using these common APIs are covered in a lot of detail as well.
Also, especially useful for driver authors, we cover the kernel’s modern resource-managed memory allocation APIs (the
devm_*()
routines). Finding where internal fragmentation (wastage) occurs is another interesting area we delve into.
Chapter 9, Kernel Memory Allocation for Module Authors– Part 2, goes further, in a logical fashion, from the previous chapter. Here, you will learn how to create custom slab caches (useful for high-frequency (de)allocations for, say, your custom driver). Next, you’ll learn how to extract useful information regarding slab caches as well as understanding slab shrinkers (new in this edition). We then move onto understanding and using the
vmalloc()
API (and friends). Very importantly, having covered many APIs for kernel memory (de)allocation, you will now learn how to pick and choose an appropriate API given the real-world situation you find yourself in. This chapter is rounded off with important coverage of the kernel’s memory reclamation technologies and the dreaded Out Of Memory (OOM) killer
framework. Understanding OOM and related areas will also lead to a much deeper understanding of how user space memory allocation really works, via the demand paging technique. This edition has more and better coverage of kernel page reclaim, as well as the new MGLRU and DAMON technologies.
Chapter 10, The CPU Scheduler – Part 1, the first of two chapters on this topic, covers a useful mix of theory and practice regarding CPU (task) scheduling on the Linux OS. The minimal necessary theoretical background on what the KSE (kernel schedulable entity) is – it’s the thread! – and available kernel scheduling policies, are some of the initially covered topics. Next, we cover how to visualize the flow of a thread via tools like
perf
. Sufficient kernel internal details on CPU scheduling are then covered to have you understand how task scheduling on the modern Linux OS works. Along the way, you will learn about thread scheduling attributes (policy and real-time priority) as well. This edition includes new coverage on CFS scheduling periods/timeslices, enhanced coverage on exactly how and when the core scheduler code is invoked, and coverage of the new preempt dynamic feature.
Chapter 11, The CPU Scheduler – Part 2, the second part on CPU (task) scheduling, continues to cover the topic in more depth. Here, you learn about the CPU affinity mask and how to query/set it, controlling scheduling policy and priority on a per-thread basis – such powerful features! We then come to a key and very powerful Linux OS feature – control groups (cgroups). We understand this feature along with learning how to practically explore it (a custom script is also built). We further learn the role the modern systemd framework plays with cgroups. An interesting example on controlling CPU bandwidth allocation via cgroups v2 is then seen, from different angles. Can you run Linux as an RTOS? Indeed you can! We cover an introduction to this other interesting area…
Chapter 12, Kernel Synchronization – Part 1, first covers the really key concepts regarding critical sections, atomicity, data races (from the LKMM point of view), what a lock conceptually achieves, and, very importantly, the ‘why’ of all this. We then cover concurrency concerns when working within the Linux kernel; this moves us naturally on to important locking guidelines, what deadlock means, and key approaches to preventing deadlock. Two of the most popular kernel locking technologies – the mutex lock and the spinlock – are then discussed in depth along with several (simple device driver based) code examples. We also point out how to work with spinlocks in interrupt contexts and close the chapter with common locking mistakes (to avoid) and deadlock-avoidance guildelines.
Chapter 13, Kernel Synchronization – Part 2, continues the journey on kernel synchronization. Here, you’ll learn about key locking optimizations – using lightweight atomic and (the more recent) refcount operators to safely operate on integers, RMW bit operators to safely perform bit ops, and the usage of the reader-writer spinlock over the regular one. What exactly the CPU caches are and the inherent risks involved when using them, such as cache false sharing,
are then discussed. We then get into another key topic – lock-free programming techniques with an emphasis on per-CPU data and Read Copy Update (RCU) lock-free technologies. Several module examples illustrate the concepts! A critical topic – lock debugging techniques, including the usage of the kernel’s powerful lockdep
lock validator – is then covered. The chapter is rounded off with a brief look at locking statistics and memory barriers.
Online Chapter – Kernel Workspace Setup
The online chapter on Kernel Workspace Setup, published online, guides you on setting up a full-fledged Linux kernel development workspace (typically, as a fully virtualized guest system). You will learn how to install all required software packages on it. (In this edition, we even provide an Ubuntu-based helper script that auto-installs all required packages.) You will also learn about several other open-source projects that will be useful on your journey to becoming a professional kernel/driver developer. Once this chapter is done, you will be ready to build a Linux kernel as well as to start writing and testing kernel code (via the loadable kernel module framework). In our view, it’s very important for you to actually use this book in a hands-on fashion, trying out and experimenting with code. The best way to learn something is to do so empirically – not taking anyone’s word on anything at all, but by trying it out and experiencing it for yourself. This chapter has been published online; do read it, here:
You can read more about the chapter online using the following link: http://www.packtpub.com/sites/default/files/downloads/9781803232225_Online_Chapter.pdf.
To get the most out of this book
To get the most out of this book, we expect the following:
You need to know your way around a Linux system, on the command line (the shell).
You need to know the C programming language.
It’s not mandatory, but experience with Linux system programming concepts and technologies will greatly help.
The details on hardware and software requirements, as well as their installation, are covered completely and in depth in Online Chapter, Kernel Workspace Setup. It’s critical that you read it in detail and follow the instructions therein.
Also, we have tested all the code in this book (it has its own GitHub repository) on these platforms:
x86_64 Ubuntu 22.04 LTS and guest OS (running on Oracle VirtualBox 7.0)
x86_64 Ubuntu 23.04 LTS guest OS (running on Oracle VirtualBox 7.0)
x86_64 Fedora 38 (and 39) on a native (laptop) system
ARM Raspberry Pi 4 Model B (64-bit, running both its distro
kernel as well as our custom 6.1 kernel); lightly tested
We assume that, when running Linux as a guest (VM), the host system is either Windows 10 or later (of course, even Windows 7 will work), a recent Linux distribution (for example, Ubuntu or Fedora), or even macOS.
If you are using the digital version of this book, we advise you to type the code yourself or, much better, access the code via the GitHub repository (link available in the next section). Doing so will help you avoid any potential errors related to the copying and pasting of code.
I strongly recommend that you follow the empirical approach: not taking anyone’s word on anything at all, but trying it out and experiencing it for yourself. Hence, this book gives you many hands-on experiments and kernel code examples that you can and must try out yourself; this will greatly aid you in making real progress, deepening your understanding of the various aspects of Linux kernel development.
Download the example code files
You can download the example code files for this book from GitHub at https://github.com/PacktPublishing/Linux-Kernel-Programming_2E. If there’s an update to the code, it will be updated on the existing GitHub repository. (So be sure to regularly do a "
git pull
" as well to stay up to date.)
We also have other code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing/. Check them out!
Download the color images
We also provide a PDF file that has color images of the screenshots/diagrams used in this book. You can download it here: https://packt.link/gbp/9781803232225.
Conventions used
There are a number of text conventions used throughout this book.
CodeInText
: Indicates code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles. Here is an example: "The
ioremap()
API returns a KVA of the
void *
type (since it’s an address location)."
A block of code is set as follows:
static
int
__init
miscdrv_init
(
void
)
{
int
ret;
struct
device
*dev;
When we wish to draw your attention to a particular part of a code block, the relevant lines or items are set in bold:
#
if
LINUX_VERSION_CODE < KERNEL_VERSION(5, 8, 0)
vrx = __vmalloc(
42
* PAGE_SIZE, GFP_KERNEL, PAGE_KERNEL_RO);
if
(!vrx) {
pr_warn
(
__vmalloc failed\n
);
goto
err_out5; } [ … ]
Any command-line input or output is written as follows:
pi@raspberrypi:~ $ sudo cat /proc/iomem
Bold: Indicates a new term, an important word, or words that you see onscreen. For instance, words in menus or dialog boxes appear in the text like this. Here is an example: "Select System info from the Administration panel."
Warnings or important notes appear like this.
Tips and tricks appear like this.
Get in touch
Feedback from our readers is always welcome.
General feedback: Email
and mention the book’s title in the subject of your message. If you have questions about any aspect of this book, please email us at
.
Errata: Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you have found a mistake in this book, we would be grateful if you reported this to us. Please visit http://www.packtpub.com/submit-errata, click Submit Errata, and fill in the form. We shall strive to update all reported errors/omissions in a Known Errata section on this book’s GitHub repo as well, allowing you to quickly spot them.
Piracy: If you come across any illegal copies of our works in any form on the internet, we would be grateful if you would provide us with the location address or website name. Please contact us at
with a link to the material.
If you are interested in becoming an author: If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, please visit http://authors.packtpub.com.
Share your thoughts
Once you’ve read Linux Kernel Programming, Second Edition, we’d love to hear your thoughts! Please click here to go straight to the Amazon review page for this book and share your feedback.
Your review is important to us and the tech community and will help us make sure we’re delivering excellent quality content.
Download a free PDF copy of this book
Thanks for purchasing this book!
Do you like to read on the go but are unable to carry your print books everywhere?
Is your eBook purchase not compatible with the device of your choice?
Don’t worry, now with every Packt book you get a DRM-free PDF version of that book at no cost.
Read anywhere, any place, on any device. Search, copy, and paste code from your favorite technical books directly into your application (using the book’s GitHub repo is even better!).
The perks don’t stop there, you can get exclusive access to discounts, newsletters, and great free content in your inbox daily
Follow these simple steps to get the benefits:
Scan the QR code or visit the link below
https://packt.link/free-ebook/9781803232225
Submit your proof of purchase
That’s it! We’ll send your free PDF and other benefits to your email directly
1
Linux Kernel Programming – A Quick Introduction
Hello, and welcome to this book on learning Linux kernel programming! This book, now in its second edition, has been explicitly written with a view to helping you learn Linux kernel development in a practical, hands-on fashion, along with the necessary theoretical background to give you a well-rounded view of this vast and interesting topic. It deliberately focuses on kernel development via the powerful Loadable Kernel Module (LKM) framework. This is because many, if not most, real-world/industry projects and products that involve kernel feature and/or device driver development, are done in this manner.
The focus is kept on both working hands-on with, and understanding at a sufficiently deep level, the internals of the Linux OS. In this regard, we cover everything from building the Linux kernel from source through to understanding and working with complex topics such as synchronization within the kernel.
To guide you on this exciting journey, we have divided this book into three sections. The first section covers the basics of setting up an appropriate workspace for kernel development, building the modern kernel from source, and writing your first kernel module.
The next section, a key one, will help you understand the essential details of kernel internals. Its coverage includes the Linux kernel architecture, the task structure, user and kernel-mode stacks, and memory management. Memory management is a key and interesting topic to which we have devoted three whole chapters (covering the internals to a sufficient extent, and importantly, how exactly to efficiently allocate and free kernel memory). The internal workings and deeper details of CPU (task) scheduling on the Linux OS round off this section.
The last section of the book deals with the more advanced topic of kernel synchronization – a necessity for professional design and code on the Linux kernel. We devote two whole chapters to covering key topics within this.
The book uses the kernel community’s 6.1 Long Term Support (LTS) Linux kernel. It’s a kernel that will be maintained (with both bug and security fixes) from December 2022 right through December 2026. Moreover, the Civil Infrastructure Project (CIP) has adopted 6.1 as an SLTS (Super LTS) release and plans to maintain it for 10 years, until August 2033. This is a key point, ensuring that this book’s content will remain current and valid for years to come.
We very much believe in a hands-on approach, with some 40 kernel modules (besides several user apps and shell scripts, double that of the first edition!) on this book’s GitHub repository to bring the learning to life, making it fun, practical, interesting, and useful. (Link: https://github.com/PacktPublishing/Linux-Kernel-Programming_2E)
We highly recommend you also make use of this book’s companion guide, Linux Kernel Programming Part 2 – Char Device Drivers and Kernel Synchronization: Create user-kernel interfaces, work with peripheral I/O, and handle hardware interrupts. It’s an excellent industry-aligned beginner’s guide to writing
misc
character drivers, performing I/O on peripheral chip memory, and handling hardware interrupts. You can get the eBook version for free along with your print copy, or alternately you can also find this eBook in the GitHub repository at: https://github.com/PacktPublishing/Linux-Kernel-Programming/tree/master/Linux-Kernel-Programming-(Part-2).
We really hope you learn from and enjoy this book. Right, let’s get on to the topic at hand now and study how to set up an appropriate kernel workspace, so that you can get the most out of this book with some practical hands-on sessions.
Kernel workspace setup
To get the most out of this book, it is very important that you first set up the workspace environment that we will be using throughout the book. This chapter will teach you exactly how to do this and get started.
Keeping size constraints in mind, the full and enhanced content of this chapter can be found within this book’s GitHub repository, here: http://www.packtpub.com/sites/default/files/downloads/9781803232225_Online_Chapter.pdf. I request that you download and read it in full. What follows here is an introduction to this first chapter of ours. Thanks!
In this chapter, we will install a recent Linux distribution (or distro
) as a Virtual Machine (VM), and set it up to include all the required software packages. We will also clone this book’s code via its GitHub repository. Furthermore, this chapter also introduces a few useful projects that will help you along this exciting journey into the Linux kernel.
Right at the outset, something I would like to emphasize is that the best way to learn something is to do so empirically, hands-on, not taking anyone’s word on anything at all but trying it out and experiencing it for yourself. Hence, this book gives you many hands-on experiments and kernel code examples that you can and indeed must try out yourself. This will greatly aid you in making real progress, learning deeply, and understanding the various aspects of Linux kernel and driver development. So, remember:
Be empirical! Also, be bold, be daring, and try things out!
We will cover the following topics in this chapter in detail, which will help you to set up your working environment:
Running Linux as a guest VM
Installing an x86_64 Linux guest
Additional useful projects
Technical requirements
You will need a modern and preferably powerful desktop PC or laptop. Ubuntu Desktop specifies some recommended minimum system requirements for the installation and usage of the distribution here: https://help.ubuntu.com/community/Installation/SystemRequirements. I would definitely suggest you go with a system well beyond the minimum recommendations – as powerful a system as you can afford to use. This is because performing tasks such as building a Linux kernel from source is a very memory- and CPU-intensive process. It should be pretty obvious that the more RAM, CPU power, and disk space the host system has, the better!
Like any seasoned kernel developer, I would say that working on a native Linux system is best. However, for this book, we cannot assume that you will always have a dedicated native Linux box available to you. So, we shall assume that you are working on a Linux guest. Working within a guest VM also adds an additional layer of isolation and thus safety. Of course, the downside is performance; working on a high-spec native Linux box can be up to twice as fast as compared to working on a VM!
Cloning this book’s code repository
The complete source code for this book is freely available on GitHub at https://github.com/PacktPublishing/Linux-Kernel-Programming_2E. You can clone and work on it by cloning the
git
tree, like so:
git clone https://github.com/PacktPublishing/Linux-Kernel-Programming_2E
The source code is organized chapter-wise. Each chapter is represented as a directory – for example,
ch1/
has the source code for this chapter. The root of the source tree has some code that is common to all chapters, such as the
convenient.h
and
klib.c
source files, among others.
For efficient code browsing, I would strongly recommend that you always index the code base(s) with
ctags
and/or
cscope
. For example, to set up the
ctags
index on a source tree, just
cd
to the root of the source tree and type
ctags -R
. (If you haven’t already, please invest the time in learning code browsing tools like
cscope
and
ctags
.)
Unless noted otherwise, the code output we show in the book is the output as seen on an x86_64 Ubuntu 22.04 LTS guest VM (running under Oracle VirtualBox 7.0). You should realize that due to (usually minor) distribution differences – and even within the same distributions but differing versions – the output shown in the book may not perfectly match what you see on your Linux system.
With that, I now leave it to you to download (from its GitHub repo) and read the full Chapter 1 of this book, then move forward with the kernel workspace setup described therein.
Once you’re done, I’ll assume you have the work environment for the coming chapters ready. Awesome. So, next, let’s move on and explore the brave world of Linux kernel development; your kernel journey is about to begin! The next two chapters will teach you how to download, extract, configure, and build a Linux kernel from source.
Learn more on Discord
To join the Discord community for this book – where you can share feedback, ask questions to the author, and learn about new releases – follow the QR code below:
https://packt.link/SecNet
2
Building the 6.x Linux Kernel from Source – Part 1
Building the Linux kernel from source code is an interesting way to begin your kernel development journey! Be assured, the journey’s a long and arduous one, but that’s the fun of it, right? The topic of kernel building itself is large enough to merit being divided into two chapters, this one and the next.
Recall what we have learned till now in the extended first chapter that has been published online (http://www.packtpub.com/sites/default/files/downloads/9781803232225_Online_Chapter.pdf): primarily, how to set the workspace for Linux kernel programming. You have also been introduced to user and kernel documentation sources and several useful projects that go hand-in-hand with kernel/driver development. By now, I assume you’ve completed the chapter that has been published online, and thus the setup of the workspace environment; if not, please do so before proceeding forward.
The primary purpose of this chapter and the next is to describe in detail how exactly you can build a modern Linux kernel from scratch using source code. In this chapter, you will first learn about the required basics: the kernel version nomenclature, development workflow, and the different types of source trees. Then, we’ll get hands-on: you’ll learn how to download a stable vanilla Linux kernel source tree onto a guest Linux Virtual Machine (VM). By vanilla kernel,
we mean the plain and regular default kernel source code released by the Linux kernel community on its repository, https://www.kernel.org. After that, you will learn a little bit about the layout of the kernel source code – getting, in effect, a 10,000-foot view of the kernel code base. The actual kernel build recipe then follows.
Before proceeding, a key piece of information: any modern Linux system, be it a supercomputer or a tiny, embedded device, has three required components:
A bootloader
An Operating System (OS) kernel
A root filesystem
It additionally has two optional components:
If the processor family is ARM or PPC (32- or 64-bit), a Device Tree Blob (DTB) image file
An
initramfs
(or
initrd
) image file
In these two chapters, we concern ourselves only with the building of the OS (Linux) kernel from source code. We do not delve into the root filesystem details. In the next chapter, we will learn how to minimally configure the x86-specific GNU GRUB bootloader.
The complete kernel build process – for x86[_64] at least – requires a total of six or seven steps. Besides the required preliminaries, we cover the first three here and the remaining in the next chapter.
In this chapter, we will cover the following topics:
Preliminaries for the kernel build
Steps to build the kernel from source
Step 1 – Obtaining a Linux kernel source tree
Step 2 – Extracting the kernel source tree
Step 3 – Configuring the Linux kernel
Customizing the kernel menu, Kconfig, and adding our own menu item
You may wonder: what about building the Linux kernel for another CPU architecture (like ARM 32 or 64 bit)? We do precisely this as well in the following chapter!
Technical requirements
I assume that you have gone through Online Chapter, Kernel Workspace Setup, and have appropriately prepared an x86_64 guest VM running Ubuntu 22.04 LTS (or equivalent) and installed all the required packages. If not, I highly recommend you do this first.
To get the most out of this book, I also strongly recommend you clone this book’s GitHub repository (https://github.com/PacktPublishing/Linux-Kernel-Programming_2E) for the code and work on it in a hands-on fashion.
Preliminaries for the kernel build
It’s important to understand a few things right from the outset that will help you as we proceed on our journey of building and working with a Linux kernel. Firstly, the Linux kernel and its sister projects are completely decentralized – it’s a virtual, online open-source community! The closest we come to an office
for Linux is this: stewardship of the Linux kernel (as well as several dozen related projects) is in the capable hands of the Linux Foundation (https://linuxfoundation.org/); further, it manages the Linux Kernel Organization, a private foundation that distributes the Linux kernel to the public free of charge (https://www.kernel.org/nonprofit.html).
Did you know? The terms of the GNU GPLv2 license – under which the Linux kernel’s released and will continue to be held for the foreseeable future – do not in any manner prevent original developers from charging for their work! It’s just that Linus Torvalds, and now the Linux Foundation, makes the Linux kernel software available to everybody for free. This doesn’t prevent commercial organizations from adding value to the kernel (and their products bundled with it) and charging for it, whether via an initial upfront charge or today’s typical SaaS/IaaS subscription model.
Thus, open source is definitely viable for business, as has been, and continues to be, proved daily. Customers whose core business lies elsewhere simply want value for money; businesses built around Linux can provide that by providing the customer with expert-level support, detailed SLAs, and upgrades.
Some of the key points we’ll discuss in this section include the following:
The kernel release, or version number nomenclature
The typical kernel development workflow
The existence of different types of kernel source trees within the repository
With this information in place, you will be better armed to move through the kernel build procedure. All right let’s go over each of the preceding points.
Understanding the Linux kernel release nomenclature
To see the kernel version number, simply run
uname -r
on your shell. How do you precisely interpret the output of
uname -r
? On our x86_64 Ubuntu distribution version 22.04 LTS guest VM, we run
uname
, passing the
-r
option switch to display just the current kernel release or version:
$
uname
-r 5.19.0-40-generic
Of course, by the time you read this, the Ubuntu 22.04 LTS kernel will very likely have been upgraded to