The benefits of Mainline Linux and the mindset of upstream development

BY KEVIN CHALMERS
Linux Core Product Development Manager
Texas Instruments, www.ti.com

Understanding the benefits of Mainline Linux design methodology requires an aligned definition of mainline and an understanding of the mindset of upstream development. The benefits of this method are shared by both microcontroller/microprocessor manufacturers and design engineers alike and include quality, scalability, and reduced time to market.

Defining mainline

The definition of Mainline Linux varies among developers. Some believe that by pulling the mainline kernel from kernel.org, carrying several thousand patches on top, and productizing it for release, that they are “mainline.” But this is not the case if you want to get all of the advantages of mainline development.

The better definition of mainline is a system where anyone, a customer, a contractor, or a developer to pull the mainline (or stable) kernel release from kernel.org and boot a supported platform directly without any additional patches being required. If this is achieved, then Mainline Linux can be claimed. This definition holds true for all the components of a Linux distribution: U-Boot, the Linux kernel, the Filesystem and the tool chain. Each of these components has its own “mainline” community consisting of well documented development flows, coding styles, source control and maintainers.

Figure 1:  Contributions to kernel.org with comprehensive code reviews increase quality and robustness

In this article we will refer to Mainline Linux as encompassing all of the common components of a typical Linux distribution – including the development process, workflow, and repositories shared by communities of developers interested in helping to develop and improve Linux and Linux-based solutions.

Mindset of upstream development

Support of Mainline Linux does not happen overnight. It requires both a time and resource commitment. But the benefits are lasting – likely as long as the lifetime of the devices supported. Mainline Linux support is a strategic decision. Some of its benefits are appreciated in the short term, such as the quality impact. Others are longer term and may take many months or years after dedicated support and upstreaming of your work to see the full benefit.

Upstream development is a process where engineers working on a common code base (such as Linux) use a shared repository to contain all of their changes. In this model maintainers and the rest of the development community review all changes being applied to the shared repository to ensure that any changes do not break existing or newly enabled use cases and that the code base maintains high quality and robustness. The process of submitting changes to this community is referred to as “upstreaming” because the changes are applied to the source code base from which all users derive their work.

A new kernel is released, on average, every 2.5 months. There is no set schedule to this development flow, so developers must be engaged with the community and understand through conversation and participation what the community and maintainers expect in terms of the code and timeliness as merge windows approach.

A merge window is a period of time during which the maintainers are actively accepting new features to be merged into the common code base. This period is generally about two weeks but can vary. Changes that are submitted after the merge window are deferred to the next one, with the exception of bug fixes. 

Because the community will not keep merge windows open when waiting for new submissions to be accepted, resources must be dedicated well ahead of the projected merge window openings to work with the community to address feedback and refactor the code. Splitting time or sharing an upstream developer’s bandwidth will significantly delay the process.

Benefits of mainline

There are numerous benefits of Mainline Linux for both semiconductor manufacturers and customers.

Ensuring quality

The largest area of concern when considering Mainline Linux is often the perceived instability. Those expressing these concerns often do not understand the development process that is designed to ensure quality or the obligation to thoroughly test Mainline Linux to ensure the support required is functioning properly.

The process of upstreaming code can be an arduous one, especially when starting out. There are few code reviews more technically critical than those received from the Linux community when submitting a patch. The benefit to this is that developers will likely receive perspectives and feedback they have not considered when developing their original work. The hierarchy of maintainers in the Linux community helps ensure quality by forcing developers to consider the impact of their code base on others. No developer wants to break someone else’s work. The intent of the Linux community is to move Linux and everyone’s work forward and provide scalability across future kernel versions while ensuring quality.

Scalability

Too often Linux developers place themselves in the role of Sisyphus and continue to refactor and redevelop their code for a particular version of Linux and then they must redo that work for the next version. Scalability and reuse can only be achieved through support of Mainline Linux. Once support of a common IP is pushed upstream, it will move forward as Mainline Linux moves forward. Mainline Linux is the only solution that ensures scalability of devices derived from one another with common IP or from one kernel to another for a given device with minimum required resources.

The silicon IP for UART, SPI and I² C typically does not change from one device to another. Neither does SATA, PCIe, USB or even core architectures of new devices. Therefore the Linux code required to support this IP should be common, upstreamed, and reused for any new device that contains this same IP. The development work should be supported in Mainline Linux and moved forward by the release cadence of Mainline Linux, requiring only maintenance and regression testing.

As new devices are released that share common IP supported in Mainline Linux, the only development required is typically centered on the new features and IP. Development of support for new devices in Linux should be incremental, leveraging work previously pushed into Mainline Linux.

Reducing time to market

As future work becomes more incremental due to the common IP across devices being pushed upstream and supported in Mainline Linux, there will be significant time saved in the required Linux development and debug stages of a product. This time saved may allow developers to create additional features and frameworks, optimize the performance of existing drivers, or focus more time on applications and solutions that provide product differentiation.

By embracing an upstream model to maximize re-use and minimize the amount of time spent re-fixing bugs or porting existing code companies give themselves a competitive advantage. All the time developers would spend working on (re)developing “commodity” support is time added to the product release schedule that delays time to market without adding new value.

How to get started

Key tips for getting started with Mainline Linux include:

• Make the commitment in the required time and resources.
• Realize that some of the benefits of Mainline Linux will only be appreciated in the long term while some will show value immediately
• Read the kernel documentation:  https://www.kernel.org/doc/Documentation/HOWTO
• Understand the kernel development process and flow.
• Track the kernel merge windows. Visit the kernel crystal ball:  http://phb-crystal-ball.org/
• Push your work upstream and engage the community and maintainers.
• Ensure there is a test infrastructure in place for nightly builds and testing. Managing regressions is critical to ensuring (ensure?) issues do not propagate.