Daniel Duan's Articles About Windows

Daniel Duan's Articles About Windows https://duan.ca/tag/windows/ Fantastic Beasts in C and Where To Find Them in Swift Swift has a pretty decent C-interoperability story. But C has many features! Today, I'll tell you a story involving a few not-so-well supported C features and Swift. It all started when I decided to re-write <a href="https://github.com/dduan/Pathos">Pathos</a> with Windows support. One of the library's offering is reading the literal target of a symbolic link: if <code>b</code> is a link to <code>a</code>, then <code>Path("b").readSymlink()</code> should return a another path that's equivalent to <code>Path("a")</code>. The Windows API that returns this information is <a href="https://docs.microsoft.com/en-us/windows/win32/api/ioapiset/nf-ioapiset-deviceiocontrol">DeviceIoControl</a>: <pre><code class="language-c">BOOL DeviceIoControl( HANDLE hDevice, DWORD dwIoControlCode, LPVOID lpInBuffer, DWORD nInBufferSize, LPVOID lpOutBuffer, DWORD nOutBufferSize, LPDWORD lpBytesReturned, LPOVERLAPPED lpOverlapped ); </code></pre> Notice anything weird? Hint: <code>LPVOID</code> is <code>void *</code> in standard C. This function is, for the lack of better words, polymorphic: depending on your input, it can intake and output different types. As a caller, it is your responsibility to look up what type is needed and cast them to and from those <code>void *</code>s. The size of the data structure is returned as well. We'll have a lot to talk about that later. Perhaps, surprisingly, this is not too hard to deal with in Swift. In my <a href="/2020/09/09/free-c-strings">last article</a>, I detailed how we can use an Swift API to work with C buffers: <pre><code class="language-swift">/// get the file `handle`... /// now call `DeviceIoControl` var data = ContiguousArray<CChar>( unsafeUninitializedCapacity: kMax ) { buffer, count in var size: DWORD = 0 DeviceIoControl( handle, FSCTL_GET_REPARSE_POINT, nil, 0, buffer.baseAddress, DWORD(buffer.count), &size, nil ) count = Int(size) } </code></pre> So this fills the array of <code>CChar</code>s with the necessary bytes for out result. I named the variable <code>data</code> because it is exactly the same concept as <code>Foundation</code>'s Data, as most Swift programmers know. As promised, we'll cast this data to the actual type so that we can retrieve information from its bytes. Side note: casting in this context is a documented usage, So it really is more awkward rather than bad. And there's a safe way to do it: <pre><code class="language-swift">withUnsafePointer(to: data) { $0.withMemoryRebound( to: [ReparseDataBuffer].self, capacity: 1 ) { buffer in // first element in `buffer` is /// a `ReparseDataBuffer`! Yay } } </code></pre> It's very important to note that <code>ReparseDataBuffer</code> is a struct with fixed, predictable layout, that the API <code>DeviceIoControl</code> promises to return. In practice, this means it is defined in C. Swift does not currently guarantee struct layout. So unless you really know what you are doing and don't care about forward compatibility, you should not do this with Swift structs. So far this story has been boring for avid Swift programmers. Fear not, things will get spicy now. Let's talk about this <code>ReparseDataBuffer</code>. It's an imported C type with a few notable features. <pre><code class="language-c">typedef struct { unsigned long ReparseTag; unsigned short ReparseDataLength; unsigned short Reserved; union { struct { unsigned short SubstituteNameOffset; unsigned short SubstituteNameLength; unsigned short PrintNameOffset; unsigned short PrintNameLength; unsigned long Flags; wchar_t PathBuffer[1]; } SymbolicLinkReparseBuffer; struct { unsigned short SubstituteNameOffset; unsigned short SubstituteNameLength; unsigned short PrintNameOffset; unsigned short PrintNameLength; wchar_t PathBuffer[1]; } MountPointReparseBuffer; struct { unsigned char DataBuffer[1]; } GenericReparseBuffer; }; } ReparseDataBuffer; </code></pre> Feature #1: it has a union member. A <code>union</code> in C is an area in memory that could be any of the types specified in the union: <pre><code class="language-c">// X.a is a `char` and X.b is a `uint64_t`. // And they occupy the same memory because // only 1 of them exists at a time. typedef union { char a; uint64_t b; } X; </code></pre> Swift does not own a direct analog for this. So if we import this <code>ReparseDataBuffer</code> definition, there wouldn't be a good way to access the data inside the union. As I pointed out in the comment, members of a union occupy the same space in memory. The largest member defines the size of that space, so everyone can fit inside of it. Each union member interprets the same bytes according to their own definition. Given this knowledge, we can derive a solution that works around Swift's limitations: break up the union (sorry, this whole paragraph reads super suggestive of the real world union. It's probably why this word is picked for this data structure in the first place. But I do not intend to say anything about the real world here)! <pre><code class="language-c">typedef struct { unsigned long reparseTag; unsigned short reparseDataLength; unsigned short reserved; unsigned short substituteNameOffset; unsigned short substituteNameLength; unsigned short printNameOffset; unsigned short printNameLength; unsigned long flags; wchar_t pathBuffer[1]; } SymbolicLinkReparseBuffer; typedef struct { unsigned long reparseTag; unsigned short reparseDataLength; unsigned short reserved; unsigned short substituteNameOffset; unsigned short substituteNameLength; unsigned short printNameOffset; unsigned short printNameLength; wchar_t pathBuffer[1]; } MountPointReparseBuffer; // we don't care about the 3rd union // member in this use case </code></pre> Conveniently for us, the union member in <code>ReparseDataBuffer</code> is at the end. So we don't need to worry about padding the unused space for smaller alternatives. Back in Swift, instead of dealing with <code>ReparseDataBuffer</code> directly, we can work with <code>SymbolicLinkReparseBuffer</code> or <code>MountPointReparseBuffer</code>, depending on our expectation of which union member to read. Yeah, this is a good time to mention that, <a href="https://github.com/dduan/Pathos">Pathos</a> has to include copies of these definition in a separate C module. Not only because we need to "break up the union", the original definition is also only accessible after importing some headers in the NT kernel. So the standard <code>import WinSDK</code> won't suffice. Moving on to notable feature #2. The last member of both <code>SymbolicLinkReparseBuffer</code> and <code>MountPointReparseBuffer</code> <code>pathBuffer</code> is a 1-character long array...why? In C, this is a flexible array member. Such member must always appear at the end of a struct. The word "flexible" in this context refers to the amount of memory allocated for this type of structs : it can vary according to the length of the array as needed. The member such as <code>pathBuffer</code> is here to provide access to the beginning of the buffer. To Swift, <code>pathBuffer</code> looks like a single <code>UInt16</code>. The language obviously don't have a good idea of this C feature. So how to we get the rest of the data from this array? Once again, we have to lean on our understanding of memory layout in C structs. As is common in APIs for flexible array members, the length of the array trailing the struct is provide to us. Let's call it <code>flexibleLength</code>. <img src="/assets/2020/09/flexible_array_member.png" alt="Illustration of memory layout for a C struct with flexible array member" /> We already have the memory for these structs in bytes (remember <code>data</code>?). And we can get the size for the fixed potion of the structs with <pre><code class="language-swift">let fixedStructSize = MemoryLayout< SymbolicLinkReparseBuffer >.stride </code></pre> Putting it all together, we can get the full content of the array by <ol> <li>chopping off the content for struct itself,</li> <li>casting the rest of the raw buffer to the expected element type, and</li> <li>include the last member in this struct as the first element in the array</li> </ol> <pre><code class="language-swift">// Include the first element, which is at // the end of the fixed struct potion. let arrayStart = fixedStructSize - 1 // Cast the data buffer so it's composed // of `wchar_t` aka `UInt16`s. let array = withUnsafePointer(to: data) { $0.withMemoryRebound( to: [UInt16].self, capacity: data.count / 2 ) { sixteenBitData in // chop off the non-array potion sixteenBitData.pointee[ arrayStart ..< (arrayStart + flexibleLength) ] } } // now, go nuts on the array! You earned it! </code></pre> Considerations such as error handling are intentionally left out in this article. You can checkout the source code of <a href="https://github.com/dduan/Pathos">Pathos</a> (on the <code>next</code> branch) for the full glory. Anyways, the flexible array member turns out to be the literal target of the symbolic link. So here is the end of our story. I'm interested to hear about alternative approaches for dealing with union members and flexible array members in Swift. Let me know on <a href="https://twitter.com/daniel_duan">Twitter</a>, or <a href="">Twitch</a> when I'm streaming! Sat, 12 Sep 2020 23:11:48 -0700 https://duan.ca/2020/09/12/fantastic-c-beasts-and-where-to-find-them-in-swift/ https://duan.ca/2020/09/12/fantastic-c-beasts-and-where-to-find-them-in-swift/ Introducing Dye Recently, I got a PC. And I started writing some code on Windows for the giggles. Naturally, I gravitate towards stuff I use on macOS and Linux when it comes to tooling. To my delight, NeoVim, ripgrep and fzf all work out of the box in terminal simulators, which brings us to todays main topic... I made a terminal coloring library for Swift that works on Windows, <a href="https://github.com/dduan/Dye/releases/tag/0.0.1">Dye 0.0.1 is available now</a>! <h2>So, why?</h2> When I started working on <a href="https://github.com/dduan/tre">tre</a>, I search in the Rust ecosystem for a CLI interface library that supports as many platforms as possible. Eventually I found <a href="https://github.com/BurntSushi/termcolor">termcolor</a> among an ocean of options. As a result, tre, like a lot of other CLI tools (like rg) written in Rust, has a consistent UI on Windows and Unix. This experience has brought lots of joy, as a user of both the library, and the app. I want to pay it forward to my fellow Swift CLI makers, and their users. Zooming out slightly, success of Swift on Windows makes Swift as a skill more valuable. And <a href="https://duan.ca/2019/01/20/kick-ass-cli-tools-in-swift/">I want more CLI tools written in Swift</a>. So it's a double-win, really. Lastly, it's a small library, all things considered. Being able to get it to a shippable state on a weekend is a key reason I decided to work on it. <h2>Technical tidbits</h2> I love Max Howell's <a href="https://github.com/mxcl/Chalk">Chalk</a> library. It's a 100-line Swift file that implements <a href="https://en.wikipedia.org/wiki/ANSI_escape_code">ANSI escape code</a> with Swift's custom string interpolation API. It demonstrates well how simple it is to customize your terminal output. Enter Windows, where ANSI sequences are ignored by built-in terminal simulators from the past. The console is customized via a entirely separate, stateful, imperative APIs (Newer simulators such as the freshly released <a href="">Terminal</a> actually supports ANSI codes pretty well). This is our lowest common API denominator, which ultimately dictated the design of Dye. Dye's API is centered around Swift's built-in protocol <code>TextOutputStream</code>. You create a stream object and mutate the style need for upcoming output: <pre><code class="language-swift">let output = OutputStream.standardOutput() output.color.foreground = .blue print("blue text", to &stream) // blue text is blue </code></pre> If the stream is redirected to something other than the terminal, styling gets automatically disabled. There are various options to customize this behavior. Take a look at this <a href="https://github.com/dduan/Dye/blob/master/Examples/main.swift">sample app</a> to get a more concrete picture of how things work. <hr /> I'll end with a screenshot of the sample app running in Command Prompt: <img src="/assets/2020/06/01/windows-example-screenshot.jpg" alt="Dye sample app running in Windows Command Prompt" /> Let's build more. Mon, 01 Jun 2020 15:08:01 -0700 https://duan.ca/2020/06/01/dye/ https://duan.ca/2020/06/01/dye/