Hírolvasó

The 5.16.16, 5.15.30, 5.10.107, and 5.4.186 stable kernel updates have been released; each contains another set of important fixes.

Hrvoje Popovski writes in with some result from his performance tests, like he did a few years ago:

I've tested Alexander Bluhm's (bluhm@) parallel ip forwarding diff and i've got some nice results. Readers should be aware that bluhm@'s diff sets NET_TASKQ=4 which means that forwarding will use 4 CPU threads and that this diff will affect only network cards that have multiqueue support (at the time of writing those cards are ix(4), ixl(4), and mcx(4). In my tests I was sending 14Mpps UDP packet over ix(4) interfaces which have 16 queues:

ix0 at pci10 dev 0 function 0 "Intel 82599" rev 0x01, msix, 16 queues ix1 at pci10 dev 0 function 1 "Intel 82599" rev 0x01, msix, 16 queues

OpenBSD box is Supermicro AS-1114S-WTRT with 24 x AMD EPYC 7413 24-Core Processor, 2650.37 MHz CPUs so this box is nice to test those 16 queues.

Read more…

Working for a famous tech company, I get asked a lot "Why don't you use technology X?" X may be an application, programming language, operating system, hypervisor, processor, or tool. It may be because: - It performs poorly. - It is too expensive. - It is not open source. - It lacks features. - It lacks a community. - It lacks debug tools. - It has serious bugs. - It is poorly documented. - It lacks timely security fixes. - It lacks subject matter expertise. - It's developed for the wrong audience. - Our custom internal solution is good enough. - Its longevity is uncertain: Its startup may be dead or sold soon. - We know under NDA of a better solution. - We know other bad things under NDA. - Key contributors told us it was doomed. - It made sense a decade ago but doesn't today. - It made false claims in articles/blogs/talks and has lost credibility. - It tolerates brilliant jerks and has no effective CoC. - Our lawyers won't accept its T&Cs or license. It's rarely because we don't know about it or don't understand it. Sometimes we do know about it, but have yet to find time to check it out. For big technical choices, it is often the result of an internal evaluation that involved various teams, and for some combination of the above reasons. Companies typically do not share these internal product evaluations publicly. It's easier to say what you do use and why, than what you don't use and why not. I used to be the outsider asking the big companies, and their silence would drive me nuts. Do they not know about this technology? Why wouldn't they use it?...I finally get it now having seen it from the other side. They are usually well aware of various technologies but have reasons not to use them which they usually won't share. ## Why companies won't say why **Private reasons**: - It is too expensive. - We know under NDA of a better solution. - We know other bad things under NDA. - Key contributors told us it was doomed. These have all happened to me. Technology X may be too expensive because we're using another technology with a special discount that's confidential. If the reasons are under NDA, then I also cannot share it. In one case I was interested in a technology, only to have the CEO of the company that developed it, under NDA, tell me that he was abandoning it. They had not announced this publicly. I've also privately chatted with key technology contributors at conferences who are looking for something different to work on, because they believe their own technology is doomed. At some companies, whatever reason may just be considered competitive knowledge and thus confidential. **Complicated reasons**: - It performs poorly. - Our lawyers won't accept its T&Cs or license ([example]). - Our custom internal solution is good enough. - It made sense a decade ago but doesn't today. - It tolerates brilliant jerks and has no effective CoC. - It lacks subject matter expertise. - It's developed for the wrong audience. These are complicated and time consuming to explain properly, and it may not be a good use of engineering time. If you rushed a quick answer instead, you can put the company in a worse position than by saying nothing: that of providing a _weak_ explanation. Those vested in the technology will find it easy to attack your response – and have more time, energy, and interest in doing so – which can make your company look worse. I've found one of these reasons is common: Discovering that a product was built without subject matter expertise. I've seen startups that do nothing new and nothing well in a space that's crying for new or better solutions. I usually find out later that the development team has no prior domain experience. It's complicated to explain, even to the developer team, as they lack the background to best understand why they missed the mark. A common reason I've seen at smaller companies is when an internal solution is good enough. Adopting new technologies isn't "free": It takes (their limited) engineering time away from other projects, it adds technical debt, and it may add security risk (agents running as root). In this case, the other technology just doesn't have enough features or improved performance to justify a switch. **Safer reasons**: - It is poorly documented. - It lacks features. - It lacks debug tools. - It has serious bugs. - etc. Those are objective and easy to discuss, right? Sort of. There are usually multiple reasons not to use a product, which might include all three: "safe", "complicated", and "private." It can be misleading to only mention the safe reasons. Also, if you do, and they get fixed, people will feel let down that you don't switch. **Bad reasons**: - Not invented here syndrome. - Pride/ego. - Corporate politics. - etc. I didn't include them in the above list, but there can be some poor reasons behind technology choices. At large tech companies (with many staff and their collective expertise) I'd expect that most technical choices will have valid reasons, even if the company won't share them, and poor reasons are the exception and not the rule. People think it's poor technical choices ten times more than it actually is. Just as an example, I've been asked many times why my company uses FreeBSD for its CDN, with the assumption that there must be some dumb reason we didn't choose Linux. No, we do regular Linux vs FreeBSD production tests (which I've helped analyze) and FreeBSD is faster for that workload. For a long time we just avoided this question. I finally got approval to mention it in a footnote in my last book (SysPerf2 page 124). The only sure way to find out why a company doesn't use a given technology is to join that company and then ask the right team. Otherwise, try asking questions that are safer to answer. For example: "Are you aware of technology X?," and "Are you aware that technology X claims to be better at Y?". This can at least rule out ignorance. Update: This was discussed on [Hacker News], which included some other reasons, including: - Engineers don't know X and can't hire X engineers (certainly true at smaller companies). - The cost of switching outweighs the benefits. - Legacy. And I liked the comment about "...random drive-by comments where the person is really just advertising x, rather than being genuinely curious." [example]: /blog/2014-05-17/free-as-in-we-own-your-ip.html [Hacker News]: https://news.ycombinator.com/item?id=30755861

Over on the Software Freedom Conservancy blog, Bradley M. Kuhn considers the question of the interaction between copyleft and the "ethical source" effort that seeks to use copyleft-like licensing to bring about additional changes, beyond just software freedom; the Hippocratic License is an example of such a license. In his view, copyleft and ethical software are not really compatible, even though many in free-software world (including Kuhn) are highly sympathetic to the goals, especially in light of the recent invasion of Ukraine by Russia. I suspect activists will continue to disagree about whether we have a moral imperative to change FOSS licenses themselves to contractually forbid Putin to copy, modify, redistribute and reinstall the FOSS he already has (or surreptitiously downloaded by circumventing sanctions). However, these horrendous events in Ukraine offer real world examples to consider the viability of expanding copyleft term expansion beyond software, and consider how it might work. My analysis is that such changes would only give us the false sense of having "done something". Ultimately enforcement of such licensing changes would either be impossible or pointless. The very entities (such as the varied international courts and treaty organizations) that could enforce such terms will also have plenty of other war crimes and sanctions violations to bring against Putin and his cronies anyway. The penalties for the actions of war that Putin took will be much stronger than Putin's contractual breach or copyright infringement claim that could be brought under a modified copyleft license and/or the Hippocratic License.

Jason Donenfeld has published a lengthy look at the changes to the Linux random-number generator (RNG) for Linux 5.17 and the upcoming 5.18 kernel. It covers his efforts "to modernize both the code and the cryptography used" and also peers into the future for changes that may be coming. random.c was introduced back in 1.3.30, steadily grew features, and was a pretty impressive driver for its time, but after some decades of tweaks, the general organization of the file, as well as some coding style aspects were showing some age. The documentation comments were also out of date in several places. That’s only natural for a driver this old, no matter how innovative it was. So a significant amount of work has gone into general code readability and maintainability, as well as updating the documentation. I consider these types of very unsexy improvements to be as important if not more than the various fancy modern cryptographic improvements. My hope is that this will encourage more people to read the code, find bugs, and help improve it. And it should make the task of academics studying and modeling the code a little bit easier.

The kernel community has a number of excuses for the relative paucity of regression-test coverage in the project, some of which hold more water than others. One of the more convincing reasons is that a great deal of kernel code is hardware-specific, and nobody can ever hope to put together a testing system with even a small fraction of all the hardware that the kernel supports. A new driver-testing framework called roadtest, posted by Vincent Whitchurch, may make that excuse harder to sustain, though, at least for certain kinds of hardware.

Security updates have been issued by Debian (python-treq), Fedora (openvpn, pesign, rust-regex, and thunderbird), Oracle (expat), Red Hat (kpatch-patch-4_18_0-147_58_1), Slackware (bind and openssl), SUSE (python-lxml), and Ubuntu (apache2).

For undeadly readers, our Errata column on the right side of the web site automatically updates and as of March 15th, 2022 some of you may have already noticed that there is a new security fix related to LibreSSL. Salient excerpt from the release notes as follows:

"* A malicious certificate can cause an infinite loop. Reported by and fix from Tavis Ormandy and David Benjamin, Google."

Subsequently, LibreSSL 3.5.1 (the development branch for those tracking -current/7.1-beta), 3.4.3 (the stable branch for those tracking 7.0-release) and 3.3.6 (the last supported branch for those stragglers still on OpenBSD 6.9) have been released!

Please see https://www.libressl.org/releases.html for more details and release notes specific to each version. It appears that the same bug was present in OpenSSL and has been fixed there too.

The Open Source Initiative reports on a ruling in the US Court of Appeals reaffirming the meaning of "open source" in a software license.

The court only confirmed what we already know – that “open source” is a term of art for software that has been licensed under a specific type of license, and whether a license is an OSI-approved license is a critically important factor in user adoption of the software. Had the defendants’ desire to license its software as AGPLv3-only been permissible, its claims of “100% open source” wouldn’t have been false and there would have been no false advertising. But adding the non-free Commons Clause created a different license such that the software could not be characterized as “open source” and doing so in these circumstances was unlawful false advertising.

CPU scheduling can be a challenging task; the scheduler must ensure that every process gets a fair share of the available CPU time while, at the same time, respecting CPU affinities, avoiding the migration of processes away from their cached memory contents, and keeping all CPUs in the system busy. Even then, users can become grumpy if specific processes do not get their CPU share quickly; from that comes years of debates over desktop responsiveness, for example. The latency-nice priority proposal recently resurrected by Vincent Guittot aims to provide a new tool to help latency-sensitive applications get their CPU time more quickly.

Security updates have been issued by Debian (flac, openssl, and openssl1.0), Fedora (nbd, pesign, and rust-regex), openSUSE (ansible, java-1_8_0-openjdk, libreoffice, and stunnel), Oracle (expat, glibc, and virt:ol and virt-devel:rhel), Red Hat (expat, redhat-ds:11.3, and virt:av and virt-devel:av), SUSE (atftp, java-1_8_0-openjdk, libreoffice, python3, and stunnel), and Ubuntu (apache2, bind9, firefox, fuse, and man-db).

The LWN.net Weekly Edition for March 17, 2022 is available.

Python has often been touted as a "batteries included" language because of its rich standard library that provides access to numerous utility modules and is distributed with the language itself. But those libraries need maintenance, of course, and that is provided by the Python core development team. Over the years, it has become clear that some of the modules are not really being maintained any longer and they probably are not really needed by most Python users—either because better alternatives exist or because they address extremely niche use cases. A long-running project to start the removal of those modules has recently been approved.

Debian's annual ritual of electing a project leader is underway. There are three candidates this time: Felix Lechner, Hideki Yamane, and incumbent Jonathan Carter. Platforms for the candidates are being placed on this page as they become available.

The 5.16.15, 5.15.29, 5.10.106, 5.4.185, 4.19.235, 4.14.272, and 4.9.307 stable updates have all been released; each contains another set of important fixes.

Security updates have been issued by Debian (openssl and python-scrapy), openSUSE (chrony, expat, java-1_8_0-openj9, libqt5-qtbase, openssl-1_0_0, php7, and rust, rust1.58, rust1.59), Oracle (389-ds:1.4, httpd:2.4, libarchive, libxml2, and vim), Red Hat (389-ds:1.4, glibc, httpd:2.4, kpatch-patch, libarchive, libxml2, vim, and virt:rhel and virt-devel:rhel), SUSE (chrony, compat-openssl098, expat, libqt5-qtbase, openssl, openssl-1_0_0, openssl-1_1, openssl1, php7, rust, rust1.58, rust1.59, and squid3), and Ubuntu (libreoffice, netkit-rsh, openssl, openssl, openssl1.0, tar, and tcpdump).

Disruptive changes are not much fun for anyone involved, though they may be necessary at times. Moving away from the SHA-1 hash function, at least for cryptographic purposes, is probably one of those necessary disruptive changes. There are better alternatives to SHA-1, which has been "broken" from a cryptographic perspective for quite some time now, and most of the software components that make up a distribution can be convinced to use other hash functions. But there are still numerous hurdles to overcome in making that kind of a switch as a recent discussion on the Fedora devel mailing list shows.

The OpenSSL project has disclosed a vulnerability wherein an attacker presenting a malicious certificate can cause the execution of an infinite loop. It is thus a denial-of-service vulnerability for any application — server or client — that handles certificates from untrusted sources. The OpenSSL 3.0.2 and 1.1.1n releases contain fixes for the problem. This advisory makes it clear that LibreSSL, too, suffers from this vulnerability; updated releases are available there too.

While debugging, it’s invaluable to be able to trace functions in the Linux kernel. There are several tools for that task: perf, ftrace (with helper tool like trace-cmd), bpftrace, etc.

For my own debug of kernel internals and in the last few years to debug the i915 module for Intel graphics, I usually use tracepoints or dynamic tracepoints (with perf probe) as the trigger for something I want to trace.

One question I get some times is “how do I trace the initial probe of a kernel module?”. At that time you (usually) don’t yet have the module loaded and as a consequence, the symbols of that module are still not available for using tracepoints. For example:

# trace-cmd list -e '^i915.*' #

One thing to realize is that a lot of time we don’t want to debug the module loading, but rather the module binding to a device. These are 2 separate things, that happen in a cascade of events. It probably also adds to the confusion that the userspace tools to load the modules are not named very consistently: modprobe and insmod, and they end up calling the [f]init_module syscall.

The various buses available in the kernel have mechanisms for stopping the autoprobe (and hence binding to a device), when a module is loaded. With i915, all we have to do is to set a few things in the /sys/bus/pci/:

# echo 0 > /sys/bus/pci/drivers_autoprobe # modprobe i915

With the i915 module, but not attached to any device, we are ready to attach to any tracepoint or create new dynamic probes. Let’s suppose I want to debug the i915_driver_probe()() function, and any function called by it during the initialization. This is one of the functions to initialize the GPU in i915 called when we are binding to a device.

F=i915_driver_probe echo 0 > /sys/bus/pci/drivers_autoprobe modprobe i915 cd /sys/kernel/debug/tracing cat /dev/null > trace echo $F > set_graph_function echo _printk > set_graph_notrace echo 10 > max_graph_depth echo function_graph > current_tracer echo 1 > tracing_on echo 0000:03:00.0 | tee /sys/bus/pci/drivers/i915/bind cp trace /tmp/trace.txt echo 0 > tracing_on cat /dev/null > trace

With the snippet above we will start tracing whenever function i915_driver_probe() is executed. We also set a few additional parameters: set the maximum call depth to graph, disable graphing on printk() since it’s usually not very interesting. Depending on the size of your trace you may also need to increase the buffer_size_kb to avoid the ring buffer to rotate, or have something to pump data out off the ringbuffer to a file.

Even after we enable tracing by echo’ing 1 to the tracing_on file, nothing happens as we are not automatically binding to the device. In the snippet above we bind it manually, by writting the pci slot to the /sys/bus/pci/drivers/i915/bind file. We then should start seeing the function to be traced:

# head /tmp/trace.txt # tracer: function_graph # # CPU DURATION FUNCTION CALLS # | | | | | | | 3) | i915_driver_probe [i915]() { 3) | __devm_drm_dev_alloc() { 3) | __kmalloc() { 3) | kmalloc_order_trace() { 3) | kmalloc_order() { 3) | __alloc_pages() {

Another thing very useful to do is to attach to a specific tracepoint. For example, to trace the MMIO read/write we can do the commands below. Now I’m using trace-cmd directly and omitting the device bind, that we should do after start recording:

# trace-cmd record -e i915:i915_reg_rw Hit Ctrl^C to stop recording ^C # trace-cmd report ... tee-2239 [009] 778.693172: i915_reg_rw: read reg=0x51000, len=4, val=(0xc1900, 0x0) tee-2239 [009] 778.693174: i915_reg_rw: write reg=0xc6204, len=4, val=(0x10251000, 0x0) tee-2239 [009] 778.693796: i915_reg_rw: read reg=0xa26c, len=4, val=(0x4, 0x0) tee-2239 [009] 778.693805: i915_reg_rw: read reg=0xd00, len=4, val=(0x14, 0x0) tee-2239 [009] 778.693808: bprint: intel_gt_init_clock_frequency: 0000:03:00.0 Using clock frequency: 19200kHz, period: 53ns, wrap: 113816ms tee-2239 [009] 778.693817: i915_reg_rw: read reg=0x913c, len=4, val=(0xff00, 0x0) tee-2239 [009] 778.693825: i915_reg_rw: read reg=0x9144, len=4, val=(0xff00, 0x0) tee-2239 [009] 778.693908: i915_reg_rw: read reg=0x9134, len=4, val=(0xff, 0x0) tee-2239 [009] 778.693918: i915_reg_rw: read reg=0x9118, len=4, val=(0xd02, 0x0) tee-2239 [009] 778.693933: i915_reg_rw: read reg=0x9140, len=4, val=(0x30005, 0x0) tee-2239 [009] 778.693941: i915_reg_rw: read reg=0x911c, len=4, val=(0x3000000, 0x0)

Red Hat recently filed a request to have the domain name WeMakeFedora.org transferred from its current owner, Daniel Pocock, alleging trademark violations, bad faith, and more. The judgment that came back will not have been to the company's liking:

The Panel finds that Respondent is operating a genuine, noncommercial website from a domain name that contains an appendage ("we make") that, as noted in the Response, is clearly an identifier of contributors to Complainant’s website. In registering the domain name using an appendage that identifies Complainant’s contributors, Respondent is not attempting to impersonate Complainant nor misleadingly to divert Internet users. Rather, Respondent is using the FEDORA mark in the domain name to identify Complainant for the purpose of operating a website that contains some criticism of Complainant. Such use is generally described as "fair use" of a trademark.

The judgment concludes with a statement that this action was an abuse of the process.