Daniel Duan's Articles About Rust

Daniel Duan's Articles About Rust https://duan.ca/tag/rust/ StreamLogger After turning off stream on <a href="https://twitch.tv/daniel_duan">Twitch</a>, the first thing I normally do is exporting the stream video to YouTube, so that the stream has an archive that survives Twitch's short-ish retain policy. These videos, perhaps surprisingly, get a few views! It's not a lot by any measures, but I'm conscientious of the fact that a typical stream archive is a multi-hour long video with no content curation, no clear schedule, and it sometimes contains breaks. Needless to say, watching them after-the-fact requires some (or a lot of) fast-forwarding. So, last week <a href="https://twitch.tv/daniel_duan">on stream</a>, I set out to improve the fast-forwarding experience. YouTube has this feature that lets you jump to specific timestamp in the video through a URL parameter in the video' link. Further, they generate this parameter for text in video's description, if the text is in the right format. This is handy for generating a "table of content" for the video so that viewers can click the timestamp in the description to jump to the section they are most interested in. <a href="https://github.com/dduan/StreamLogger">StreamLogger</a> is a little utility that lets me note down what happened while I'm streaming. It's kind of like writing a commit message, except the message describes what happened since the last "commit". These messages, along with their associated timestamps, will be used by StreamLogger to generate the "table of content". Using it in command line looks like this: <pre><code class="language-bash"># Turn on stream, maybe check signs of being live, etc. Then slog start # start a new log # Do stuff, when it comes to a natural conclusion point... slog -- 'I did stuff' # Do more stuff... slog -- 'some other stuff' # Some time later... end stream # No action is required to end the stream as far as StreamLogger is concerned. # Now, to generate the table-of-content slog stamp -s 1:32 </code></pre> That last command outputs <pre><code>0:01:32 I did stuff 1:41:59 some other stuff </code></pre> ... which goes to the video's description. There's a few subtleties in this overall simple tool. Whenever a message is added, it gets associated with the time at which the previous message was add. So the act of logging marks both the end of a chapter and the beginning of the next. In reality, there's always going to be a gap between the start of the stream and the time the log is initialized. That's what the <code>-s 1:32</code> in the last command is trying to correct. It tells StreamLogger the length of the gap. Now the absolute time associated with each event has a relationship with the archive video. <hr /> I chose to write this in Rust because I'm going to need the final product on Linux, macOS, and Windows since I stream on all 3 platforms. (fun fact: I never built it directly on my PC running Linux. Instead, I simply downloaded the musl-based build from the GitHub release, which was built by GitHub Actions. It works beautifully.) Maybe one day I'll add a GUI for it that works across these platforms, too, so that it's more friendly to wider group of users. <hr /> The more I use and think about StreamLogger, the more I like it. You can see me working its entirety in the following archives ;) Part 1: https://youtu.be/xWRcdaEjir4 Part 2: https://youtu.be/RS-ZMBzu9Dg Sat, 13 Jun 2020 12:36:55 -0700 https://duan.ca/2020/06/13/stream-logger/ https://duan.ca/2020/06/13/stream-logger/ Naive NixOS Rust Development tl;dr: To work on Rust project with nix-shell, rls and extensions such as <code>rust-analysis</code>, <code>rust-src</code>, without caring too much about specific Rust toolchain version (except for it being <code>stable</code>), use the following <code>shell.nix</code>: <pre><code class="language-nix">let moz_overlay = import (builtins.fetchTarball https://github.com/mozilla/nixpkgs-mozilla/archive/master.tar.gz); nixpkgs = import <nixpkgs> { overlays = [ moz_overlay ]; }; ruststable = (nixpkgs.latest.rustChannels.stable.rust.override { extensions = [ "rust-src" "rust-analysis" ];} ); in with nixpkgs; stdenv.mkDerivation { name = "rust"; buildInputs = [ rustup ruststable ]; } </code></pre> When you have a Nix hammer, everything looks like a Nix expression. Having used NixOS on a real PC for a number of days, this is the impression I get from the world of Nix. Unfortunately, so far, it's been a negative for me. One of the most exciting thing I want to use Nix for is to bootstrap development environment with <code>nix-shell</code>. I imagined it to be similar to using <a href="https://pipenv-fork.readthedocs.io/en/latest/">pipenv</a> with Python, except for everything. Well, I've since learned that it's not true (yet?) for many reasons. Modern programming languages come with their own attempt at reproducibility. Some does it better than others. To make it concrete, I'm talking about things like <a href="https://haskellstack.org">Stack</a> for Haskell or <a href="https://rustup.rs/">rustup</a> for Rust: given the source code, how do I make it build in the way the project intended? What's the correct version of the compiler, runtime, and tools that works best with this revision of the source code? The common solution usually follows this pattern: as author of a project, specify as much as you can, the environment best suited for the current state of the project. As a "builder", use a single program that's capable of updating itself, as well as ensuring that the project builds exactly as specified, including managing the compiler/runtime/tooling versions, etc. This single program's role is very much the same as the Nix system, except the latter is independent of programming languages: <code>rustup</code> installs Rust, so does Nix. That's a bad thing. As a package manager, Nix either have to tightly integrate with each of these other package managers, leveraging their evolving behaviors to give its user the build environment; or, it must replace them. The former is impractical; the latter, well, sucks. Back to reality. This is the experience I want to have with NixOS: Some programs I use daily such as Alacritty, NeoVim, Firfox, etc, are installed globally and readily available. They are part of my <code>/etc/nixos/configuration.nix</code>. So far so good. Now, I regularly program in a few languages. For each of the project, I'd like to have a <code>shell.nix</code> that brings in its compilers, libraries, LSP servers, etc. This is what <code>nix-shell</code> is supposed to give me! This is known as the "per project" setup. Let's see: with Rust, that means <code>rustc</code> (compiler), <code>cargo</code> (package manager), <code>rls</code>, <code>rust-src</code> and <code>rust-analysis</code> (LSP). In macOS, I'd install all of these globally with <code>rustup</code>. In NixOS...well, I can ask for <code>rustup</code> for my project: <pre><code class="language-nix">with import <nixpkgs> {}; stdenv.mkDerivation { name = "rust"; nativeBuildInputs = [ rustup ]; } </code></pre> ...which gives me <code>rustup</code> and nothing else. That's right, you don't even get a <code>rustc</code> after running <code>nix-shell</code>. But <code>rustup</code> can get you everything else, all I need to do is ask. Hmm, do I need to run a series of set-up commands with <code>rustup</code> every time I enter the environment? No? I just need to run it the first time? Until the cached tools get deleted by some garbage collection mechanism? That seems unsatisfying, doesn't it? Instead of <code>rustup</code>, I could also ask Nix to use the <code>rustc</code>/<code>cargo</code>/<code>rls</code> it knows about directly. This is marginally better. Except I still need <code>rust-src</code> and <code>rust-analysis</code> for my needs. As far as I can tell, these <a href="https://github.com/rust-lang/rls">RLS</a> components are out of Nix's control (as of today). Everywhere on the internet I looked, for every problem that Nix-the-package-manager doesn't work out-of-the-box, there's someone responding along the line of "you can write some Nix expression yourself". In other words, Nix-the-language is powerful enough to solve it, probably. In the case of Rust, luckily, Mozilla wrote enough Nix expressions for us and provides them via an <a href="https://github.com/mozilla/nixpkgs-mozilla">overlay</a>. These expressions are rich enough to meet my needs. As you can see in the tl;dr at the top, when entering the development environment, nix-shell would: download the overlay's source code (from the internet, or local cache), load the expression it includes, mix in my customization, and execute it. That marks the end of my search. I like the final solution because it's mostly "vanilla" Nix and doesn't require me to mess with a bunch of other tools. For solutions that do, read <a href="https://xeiaso.net/blog/how-i-start-nix-2020-03-08/">this</a>. <hr /> At end of the day, my needs are pretty basic: consistency from rustup and convenience from nix-shell. I didn't need to pin the compiler to a specific Rust release, or checksum the final build output. I'm very new to both technologies so there may be a follow-up post sometime in the future. Thu, 07 May 2020 11:04:58 -0700 https://duan.ca/2020/05/07/nix-rust-development/ https://duan.ca/2020/05/07/nix-rust-development/ Site Improvements 2020 I took back my website. I took it back from the claws of Jekyll and Ruby. I took it back from some random template among a few that were immediately available. I took it back from my own ignorance of modern web technology. This time, I rewrote the hole effing thing from scratch. <h3>The Why</h3> The <a href="/2017/01/16/site-changes/">last time</a> I ended with <blockquote> I wrote the most Ruby in my life today. Yay? </blockquote> That question mark turned out to be prescient. Ruby is not my thing. No judgement against the language per ce. But the Ruby ecosystem is not friendly to a casual user who needs it once every few months. No really, when I write an article, it's a toss-up whether I can deploy it without being forced to mess with Ruby/Gems/Jekyll/Homebrew etc. I'm almost certain there's a set of best practices I could learn to improve this. But it'd be a skill that I barely need and probably forget a few times. Meanwhile I just want to translate a new Markdown file to HTML and put it on Github. As a teen, one of my favorite things about the web is how accessible it is. I could sit in front of anyone's computer, open Notepad.exe and type in some tags, and open the file with a browser to see the result. That simple bootstrap process, however repetitive, never got old for me. In the last decade, my professional work is focused on native, mobile applications. This experience biased me in a few ways. "Native" made me appreciate the closeness to "the metal": you have an OS; you get the executable; you launch strace, and boom, everything the OS thinks what your code does is revealed to you. "Mobile" forced me to reckon with the reality: desktop experience has become a niche. It's nowhere near as important as it was prior to the iPhone. Not making your software run well on mobile devices is a particular kind of choice that come with some severe trade-offs. And, finally, I subscribe to the idea that plain text is supreme. Yes, even more supreme than the web. This website is a derivative of the articles I write in Markdown. Plain text as a format will out-last the web in the long run (not necessarily these texts). So it really bothers me when I have to put "front matter" in YAML/TOML/whatever in front of the real Markdown. Yes, if you want reasonable HTMLs, these metadata is necessary. But the text is Supreme. The text is to be readable directly. Succumbing the supreme to the derivative is wrong. Okay, so where does all that leave me? In this iteration of this website, I wrote every single line of CSS and HTML (there's barely any JavaScript) which look decent on mobile. It's generated by portable Linux and Darwin executables that are part of the website. As long as the OSes stay relatively stable, the site will build without any dependencies, in a matter of milliseconds. The best part? It'll stay like this unless I wish it otherwise. <h3>The Frontend</h3> A few words on the design of the site. This site gained the concept of "links" the <a href="/2017/01/16/site-changes/">last time</a>. It has since become clear that I don't use this feature (blame Twitter). It's gone, for now. The old "about" page is replaced by the home page. When I decided the site needed a rewrite, I fantasized a place with only HTML. Perhaps users who want a better reading experience can simply put it in Reader mode. Alas, if browsers (with the exception of Safari) implemented automatic dark mode with a <a href="https://drafts.csswg.org/css-color-adjust-1/#color-scheme-meta">color-scheme meta</a> tag, it'd almost be a working idea. So, that's my starting point. The site is a list of articles organized by tags and dates, and a few web pages. I want you to read the site, not navigate it. So text is the point. It's the only design element. The site uses 2 fonts and 2 text-color (not counting highlighted code). Links are always underscored (because you can't hover in mobile browsers to find something clickable). The site is responsive to mobile layout, and dark mode. It's aware that it could be added to the home screen of a mobile device, or linked to some external site that wants to generate a preview. I did end up using <a href="https://ethanschoonover.com/solarized/">solarized</a> theme for code highlighting, which lead to 2 static CSS files. Other than that, all CSS and HTML are hand-written. There's no build step for them, farm to table, Vim to your browser. It turned out standardized CSS variable is game-changing. Combined with media queries, I barely needed any class to support dark mode/mobile layout. More importantly, it makes my programmer brain happy. Oh, yes I'm talking about them here because this is the first time I truly attempted to catch up since they were introduced to the world. It's freeing to let go of constrains of an read-to-use theme, or some CSS frameworks. <h3>The Backend</h3> This site is a collection of static files. The so-called backend is a program that assembles these files from some HTML templates and Markdown files. In the past, this program had been <a href="https://jekyllrb.com/">Jekyll</a>. This time around, I replaced it with some Rust code. Boy, this thing is cool, if I say so myself. I'm going to refer to it as "the generator". The biggest "feature" is the fact that it doesn't pretend to be re-usable. The generator is as part of this site as the articles. Thanks to Rust, the programs are built into executable binaries. On macOS, it requires libSystem to run. On Linux, I can (and prefer) build with <a href="https://www.musl-libc.org/">musl</a>. The binaries for these two OSes are checked in with the content of the site. So it requires zero installation to "build" the site. (I may need to include a 3rd executable for the upcoming ARM-based Macs soon). The generator spits out the final content of the website. It's deployed without further modification. The build process is pretty fast. As of this writing, it averages around 250 ms. I could probably make it faster by avoiding some repeated reads when it comes to article inputs. The generator handles HTML/XML templating with a library named <a href="https://github.com/djc/askama">Askama</a>. Learning it had been an eye opening experience. Askama is built atop Rust's macro system. For each template (e.g. web page), it requires users to write a Rust data structure that fulfills its variable requirements. Here's the kicker: in this data structure, you cannot miss any variable the corresponding template requires. When you do, the Rust project won't compile! Rust's tooling is so good that these errors were surfaced in my editor as I wrote this part in real time. This level of type-safety for template language felt like magic. Syntax highlighting is powered by <a href="https://github.com/trishume/syntect">syntect</a>, the library behind <a href="https://github.com/sharkdp/bat">bat</a>. But the interesting bit here is how the syntax definitions are embedded within the final executable. To support a particular syntax, syntect takes the syntax's SublimeText definition. So this is a configurable, extendible system. The generator includes 118 syntax definitions. Uncompressed, their files take up 5MB of disk space. As one can imagine, loading 5MB from 118 files each time the program runs is quite slow. Turns out, Rust has a standard library macro <a href="https://doc.rust-lang.org/std/macro.include_bytes.html">include_bytes!</a> that solves this problem. <a href="https://doc.rust-lang.org/std/macro.include_bytes.html">include_bytes!</a> embeds contents of a file as literals in source code, as if it's hand-written. syntect takes advantage of this feature by supporting serialization of its in-memory representation of syntaxes into bytes, and accepting in-line byte literals in reverse in order to create these representations. This system solves two problems for me: <ol> <li>I no longer need to ship the SublimeText syntax files along with the generator.</li> <li>The generator doesn't need to perform all that disk I/O, so it runs significantly faster.</li> </ol> Overall, the Rust ecosystem delivered. Although a lot of library are still in rough shape/early stage, there usually are multiple alternatives for roughly the same purpose. Gluing a few of them together for this generator project had been fun. Finally, let's talk metadata, I can't realistically manage to generate the site without them. Articles and static pages each define their URL by their file locations relative to the root. For example, <code>/articles/2020/04/23/hello.md</code> means the URL is <code>/2020/04/23/hello/</code>. Each article still has a front matter. But I made the text version look as "natural" as possible. There's no markers for beginning and end of the metadata since I know exactly what's needed. The title is marked as an H1. Date is in RFC3339 format. Tags are comma-separated values. So an example of an article's beginning looks like: <pre><code class="language-markdown"># Site Improvements 2020 2020-04-22T21:58:03-07:00 tags: Rust, HTML, CSS, Ruby, Jekyll, Markdown I took back my website... </code></pre> Text supremacy! Glory! <h3>Onward</h3> Over the course of roughly 15 years, I've had blogs of quite a few variates. I wish I'd done a better job archiving them as I moved on from one to the next. This is not the first time I attempted to gain full control over the theming, generating, hosting, and deployment of a site. But I'm hopeful that it'll last longer. You could say that's what I optimized for. And who knew, maybe my experience from past failures counts for something. Here's to less stressful deploys and more writing! Wed, 22 Apr 2020 21:58:03 -0700 https://duan.ca/2020/04/22/site-improvements-2020/ https://duan.ca/2020/04/22/site-improvements-2020/