How Chrome OS works upstream
Google has a long and interesting history contributing to the upstream Linux kernel. With Chrome OS, Google has tried to learn from some of the mistakes of its past and is now working with the upstream Linux kernel as much as it can. In a session at the 2019 Open Source Summit North America, Google software engineer Doug Anderson detailed how and why Chrome OS developers work upstream. It is an effort intended to help the Linux community as well as Google.
The Chrome OS kernel is at the core of Google's Chromebook devices, and is based on a Linux long-term support (LTS) kernel. Anderson explained that Google picks an LTS kernel every year and all devices produced in that year will use the selected kernel. At least once during a device's lifetime, Google expects to be able to "uprev" (switch to a newer kernel version). Anderson emphasized that if Google didn't upstream its own patches from the Chrome OS kernel, it would make the uprev process substantially more difficult.
Simply saying that you'll work upstream and actually working upstream can be two different things. The process by which Chrome OS developers get their patches upstream is similar to how any other patches land in the mainline Linux kernel. What is a bit interesting is the organizational structure and process of how Google has tasked Chrome OS developers to work with upstream. Anderson explained that developers need to submit patches to the kernel mailing list and then be a little patient, giving some time for upstream to respond. A key challenge, however, is when there is no response from upstream. "When developing an upstream-first culture, the biggest problem anyone can face is silence," Anderson said.
Anderson emphasized that when submitting a patch to the mailing list, what a developer is looking for is some kind of feedback; whether it's good or bad doesn't matter, but it does matter that someone cares enough to review it. What the Chrome OS team does in the event that there is no community review is it will have other Chrome OS engineers publicly review the patch. The risk and worry of having Chrome OS engineers comment on Chrome OS patches is that the whole process might look a little scripted and there could be the perception of some bias as well. Anderson noted that it is important that only honest feedback and review is given for a patch.
Yak shaving
While feedback from the mailing list is what the developers want, there can be situations where that feedback gets into what Anderson referred to as "yak shaving". It means that a small patch is submitted and the feedback that comes back covers more than just the patch itself, asking the developer to, for example, clean up a related subsystem or redo an API. "Basically, your little patch ends up being someone requesting you to do a month's worth of work," Anderson said.
Sometimes if a yak shaving request is not a lot of work; he suggested that the best thing is to just do the work in that case. Other times, the work is far outside the scope of a Chrome OS developer's efforts and it's not something that the developer is able to do. Anderson said that there are times that upstream will allow a developer to land a patch without completing all of the yak shaving, while there are other times when that doesn't happen.
The Chrome OS kernel tree
Even with a stated policy of trying to be upstream first, the Chrome OS kernel tree isn't entirely pulled from the mainline kernel. Anderson explained that developers generally try to tag all the commits in the Chrome OS tree with some information about where the commits came from. He emphasized that whenever Google has something in its Chrome OS kernel tree, developers are pushing hard for it to go upstream into Linus Torvalds's tree or into the right maintainer's tree.
The UPSTREAM tag is used to identify a commit that was pulled directly from Torvalds's tree, while the FROMGIT tag refers to a commit that was picked from a maintainer's tree that will eventually feed the mainline. The FROMLIST tag is used by Chrome OS developers for commits that are picked from a mailing list. The CHROMIUM tag is a bit different than the others, in that it refers to a commit that is not going to go upstream and, ideally, is just for Chrome-OS-specific configuration items.
Fundamentally, Anderson argued that it is a win-win for both the overall kernel development community and Chrome OS to work upstream. He noted that working upstream provides extra testing and review bandwidth, as well as pushing hardware vendors that work with Chrome OS devices to work upstream as well. For example, Anderson said that the Rockchip rk3288 and rk3399 Arm systems-on-chip (SoCs) are fairly well supported upstream because of Chrome OS.
Learning from the mistakes of Android
Google hasn't always had an upstream-first approach for its operating systems, which is something that Chrome OS developers are aiming not to repeat. "We're trying not to do what Android did 10 years ago," Anderson said. Android phones tended to have large stacks of patches that had not been upstreamed, he said. In addition, Android devices had come up with all kinds of different ways of doing things that were not aligned with the upstream kernel. Rather than follow the path of Android, Chrome OS developers want to get as much as possible into the mainline, giving kernel developers the chance to comment and review along the way.
During the question-and-answer segment that followed the session, a member of the audience pointed out that even today Android development has lots of code that isn't upstream. They asked if in fact the Chrome OS kernel is basically the same as the upstream kernel. Anderson explained that Google has a full fork of the mainline kernel. So for example, there is a Chrome OS 4.19 tree, where patches are picked and tagged so it's clear where the patches come from. "So it's totally a whole fork of the upstream kernel, but the idea is that we try to keep posting things upstream and landing them there," he said. "And in a perfect world, our Chrome OS 4.19 would be just pure 4.19 with a whole bunch of upstream bits from newer kernels, but it doesn't always work out that way."
Keeping everything upstream, as Anderson explained more than once,
during his session, isn't easy. But it seems clear that Google's
Chrome OS development team is committed to that path, and has a
well-defined process that will hopefully keep them from straying too far.
Index entries for this article | |
---|---|
GuestArticles | Kerner, Sean |
Conference | Open Source Summit North America/2019 |
Posted Sep 6, 2019 16:50 UTC (Fri)
by marcH (subscriber, #57642)
[Link] (3 responses)
One thing that makes ChromeOS (or any other project) an actual open-source project is good documentation. Random example:
> They asked if in fact the Chrome OS kernel is basically the same as the upstream kernel.
Is there any real-world product that ships with absolutely zero backport and a SHA1 from Linus tree? The real question is not "if" but "how much" technical debt. That's what the tags above help measure and manage.
Posted Sep 9, 2019 11:06 UTC (Mon)
by peda (subscriber, #5285)
[Link] (2 responses)
Which means that bisecting problems is actually possible and simple, something which I found very difficult before the SoC was sufficiently supported upstream.
It is highly recommended to seek this position!
That said, we pick a kernel from the stable tree (i.e. it has backports) when we do update, so it's not actually a SHA1 from Linus' tree. But close enough, and we could do that if we wanted to...
Posted Sep 9, 2019 12:48 UTC (Mon)
by marcH (subscriber, #57642)
[Link] (1 responses)
Posted Sep 9, 2019 15:27 UTC (Mon)
by peda (subscriber, #5285)
[Link]
arch/arm/boot/dts/at91-tse850-3.dts
Posted Sep 11, 2019 13:04 UTC (Wed)
by pauly (subscriber, #8132)
[Link] (1 responses)
The last half dozen Chromebooks that stopped by in our helpdesk really gave us a hard time to get eduroam to work.
The last Chromebook I came across was an HP one. It would neither import a correct ONC file nor could I get the
BTW: This is the file we tried to use -- anything wrong with it (it's from cat.eduroam.org)?
Posted Sep 12, 2019 15:36 UTC (Thu)
by pauly (subscriber, #8132)
[Link]
Wifi is strange: For a certificate check with 802.1X networks like eduroam,
Martin
Posted Sep 18, 2019 3:33 UTC (Wed)
by ndesaulniers (subscriber, #110768)
[Link]
How Chrome OS works upstream
https://www.google.com/search?q=UPSTREAM+CHROMIUM+BACKPORT
Most of it is being transferred to Gerrit to enable contributions from outside Google
How Chrome OS works upstream
How Chrome OS works upstream
How Chrome OS works upstream
Network config ONC?
Why don't recent Chromebooks import ONC files reliably any more?
(Technically, it's all about 802.1X or 802.11i, of course.) Some years ago the interface used to import ONC files silently,
not giving any feedback. While this is not exactly perfect, it did work. I was even able to hook up devices to our wired
eduroam installation, provided we got a supported USB adapter.
certificate settings right manually. In the end, we put the device into developer mode, and, as root, created a proper
config file and ran wpa_supplicant manually. This way, both WiFi and wired eduroam worked flawlessly.
But CLI hacking is not what users buy Chromebooks for, right?
{
"Type": "UnencryptedConfiguration",
"Certificates": [
{
"GUID": "{d8bcdd62-b725-8a9c-9805-55915b7a142e}",
"Remove": false,
"Type": "Authority",
"X509": "MIIDwzCCAqugAwIBAgIBATANBgkqhkiG9w0BAQsFADCBgjELMAkGA1UEBhMCREUxKzApBgNVBAoMIlQtU3lzdGVtcyBFbnRlcnByaXNlIFNlcnZpY2VzIEdtYkgxHzAdBgNVBAsMFlQtU3lzdGVtcyBUcnVzdCBDZW50ZXIx
JTAjBgNVBAMMHFQtVGVsZVNlYyBHbG9iYWxSb290IENsYXNzIDIwHhcNMDgxMDAxMTA0MDE0WhcNMzMxMDAxMjM1OTU5WjCBgjELMAkGA1UEBhMCREUxKzApBgNVBAoMIlQtU3lzdGVtcyBFbnRlcnByaXNlIFNlcnZpY2VzIEdtYkgxHzAdBgNVBAsMF
lQtU3lzdGVtcyBUcnVzdCBDZW50ZXIxJTAjBgNVBAMMHFQtVGVsZVNlYyBHbG9iYWxSb290IENsYXNzIDIwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCqX9obX+hzkeXaXPSi5kfl82hVYAUdAqSzm1nzHoqvNK38DcLZSBnuaY\/JIPwhq
gcZ7bBcrGXHX+0CfHt8LRvWurmAwhiCFoT6ZrAIxlQjgeTNuUk\/9k9uN0goOA\/FvudocP05l03Sx5iRUKrERLMjfTlH6VJi1hKTXrcxlkIF+3anHqP1wvzpesVsqXFP6st4vGCvx9702cu+fjOlbpSD8DT6IavqjnKgP6TeMFvvhk1qlVtDRKgQFRzl
AVfFmPHmBiiRqiDFt1MmUUOyCxGVWOHAD3bZwI18gfNycJ5v\/hqO2V81xrJvNHy+SE\/iWjnX2J14np+GPgNeGYtEotXHAgMBAAGjQjBAMA8GA1UdEwEB\/wQFMAMBAf8wDgYDVR0PAQH\/BAQDAgEGMB0GA1UdDgQWBBS\/WSA2AHmgoCJrjNXyYdK4
LMuCSjANBgkqhkiG9w0BAQsFAAOCAQEAMQOiYQsfdOhyNsZt+U2e+iKo4YFWz827n+qrkRk4r6p8FU3ztqONpfSO9kSpp+ghla0+AGIWiPACuvxhI+YzmzB6azZie60EI4RYZeLbK4rnJVM3YlNfvNoBYimipidx5joifsFvHZVwIEoHNN\/q\/xWA5br
XethbdXwFeilHfkCoMRN3zUA7tFFHei4R40cR3p1m0IvVVGb6g1XqfMIpiRvpb7PO4gWEyS8+eIVibslfwXhjdFjASBgMmTnrpMwatXlajRWc2BQN9noHV8cigwUtPJslJj0Ys6lDfMjIq2SPDqO\/nBudMNva0Bkuqjzx+zOAduTNrRlPBSeOE6Fuwg=
="
}
],
"NetworkConfigurations": [
{
"GUID": "07bd992d-0d9f-f537-cbd0-d9af02a3c86a",
"Name": "eduroam",
"Remove": false,
"Type": "WiFi",
"WiFi": {
"AutoConnect": true,
"EAP": {
"Outer": "PEAP",
"Inner": "MSCHAPv2",
"SaveCredentials": true,
"ServerCARefs": [
"{d8bcdd62-b725-8a9c-9805-55915b7a142e}"
],
"UseSystemCAs": false,
"AnonymousIdentity": "[email protected]"
},
"HiddenSSID": false,
"SSID": "eduroam",
"Security": "WPA-EAP"
},
"ProxySettings": {
"Type": "WPAD"
}
},
{
"GUID": "aa4575a3-caf4-1fc7-3fe6-8ca0d71c5d4e}",
"Name": "eduroam configuration (wired network)",
"Remove": false,
"Type": "Ethernet",
"Ethernet": {
"Authentication": "8021X",
"EAP": {
"Outer": "PEAP",
"Inner": "MSCHAPv2",
"SaveCredentials": true,
"ServerCARefs": [
"{d8bcdd62-b725-8a9c-9805-55915b7a142e}"
],
"UseSystemCAs": false,
"AnonymousIdentity": "[email protected]"
}
},
"ProxySettings": {
"Type": "WPAD"
}
}
]
}
Network config ONC?
The next Chromebook was here today (Acer, last time it was HP).
The picture was exactly the same. Wired 802.1X not working at all,
we did everything directly as root.
there are only two settings: "Do not validate" and "Default".
No way to pin the cert. When you choose "Default", the dialogue
keeps switching back to "Do not validate".
So the device might well be vulnerable to Evil Twin attack.
How Chrome OS works upstream