Daniel Duan's Articles About Pathos

Daniel Duan's Articles About Pathos https://duan.ca/tag/pathos/ Fantastic Beasts in C and Where To Find Them in Swift Swift has a pretty decent C-interoperability story. But C has many features! Today, I'll tell you a story involving a few not-so-well supported C features and Swift. It all started when I decided to re-write <a href="https://github.com/dduan/Pathos">Pathos</a> with Windows support. One of the library's offering is reading the literal target of a symbolic link: if <code>b</code> is a link to <code>a</code>, then <code>Path("b").readSymlink()</code> should return a another path that's equivalent to <code>Path("a")</code>. The Windows API that returns this information is <a href="https://docs.microsoft.com/en-us/windows/win32/api/ioapiset/nf-ioapiset-deviceiocontrol">DeviceIoControl</a>: <pre><code class="language-c">BOOL DeviceIoControl( HANDLE hDevice, DWORD dwIoControlCode, LPVOID lpInBuffer, DWORD nInBufferSize, LPVOID lpOutBuffer, DWORD nOutBufferSize, LPDWORD lpBytesReturned, LPOVERLAPPED lpOverlapped ); </code></pre> Notice anything weird? Hint: <code>LPVOID</code> is <code>void *</code> in standard C. This function is, for the lack of better words, polymorphic: depending on your input, it can intake and output different types. As a caller, it is your responsibility to look up what type is needed and cast them to and from those <code>void *</code>s. The size of the data structure is returned as well. We'll have a lot to talk about that later. Perhaps, surprisingly, this is not too hard to deal with in Swift. In my <a href="/2020/09/09/free-c-strings">last article</a>, I detailed how we can use an Swift API to work with C buffers: <pre><code class="language-swift">/// get the file `handle`... /// now call `DeviceIoControl` var data = ContiguousArray<CChar>( unsafeUninitializedCapacity: kMax ) { buffer, count in var size: DWORD = 0 DeviceIoControl( handle, FSCTL_GET_REPARSE_POINT, nil, 0, buffer.baseAddress, DWORD(buffer.count), &size, nil ) count = Int(size) } </code></pre> So this fills the array of <code>CChar</code>s with the necessary bytes for out result. I named the variable <code>data</code> because it is exactly the same concept as <code>Foundation</code>'s Data, as most Swift programmers know. As promised, we'll cast this data to the actual type so that we can retrieve information from its bytes. Side note: casting in this context is a documented usage, So it really is more awkward rather than bad. And there's a safe way to do it: <pre><code class="language-swift">withUnsafePointer(to: data) { $0.withMemoryRebound( to: [ReparseDataBuffer].self, capacity: 1 ) { buffer in // first element in `buffer` is /// a `ReparseDataBuffer`! Yay } } </code></pre> It's very important to note that <code>ReparseDataBuffer</code> is a struct with fixed, predictable layout, that the API <code>DeviceIoControl</code> promises to return. In practice, this means it is defined in C. Swift does not currently guarantee struct layout. So unless you really know what you are doing and don't care about forward compatibility, you should not do this with Swift structs. So far this story has been boring for avid Swift programmers. Fear not, things will get spicy now. Let's talk about this <code>ReparseDataBuffer</code>. It's an imported C type with a few notable features. <pre><code class="language-c">typedef struct { unsigned long ReparseTag; unsigned short ReparseDataLength; unsigned short Reserved; union { struct { unsigned short SubstituteNameOffset; unsigned short SubstituteNameLength; unsigned short PrintNameOffset; unsigned short PrintNameLength; unsigned long Flags; wchar_t PathBuffer[1]; } SymbolicLinkReparseBuffer; struct { unsigned short SubstituteNameOffset; unsigned short SubstituteNameLength; unsigned short PrintNameOffset; unsigned short PrintNameLength; wchar_t PathBuffer[1]; } MountPointReparseBuffer; struct { unsigned char DataBuffer[1]; } GenericReparseBuffer; }; } ReparseDataBuffer; </code></pre> Feature #1: it has a union member. A <code>union</code> in C is an area in memory that could be any of the types specified in the union: <pre><code class="language-c">// X.a is a `char` and X.b is a `uint64_t`. // And they occupy the same memory because // only 1 of them exists at a time. typedef union { char a; uint64_t b; } X; </code></pre> Swift does not own a direct analog for this. So if we import this <code>ReparseDataBuffer</code> definition, there wouldn't be a good way to access the data inside the union. As I pointed out in the comment, members of a union occupy the same space in memory. The largest member defines the size of that space, so everyone can fit inside of it. Each union member interprets the same bytes according to their own definition. Given this knowledge, we can derive a solution that works around Swift's limitations: break up the union (sorry, this whole paragraph reads super suggestive of the real world union. It's probably why this word is picked for this data structure in the first place. But I do not intend to say anything about the real world here)! <pre><code class="language-c">typedef struct { unsigned long reparseTag; unsigned short reparseDataLength; unsigned short reserved; unsigned short substituteNameOffset; unsigned short substituteNameLength; unsigned short printNameOffset; unsigned short printNameLength; unsigned long flags; wchar_t pathBuffer[1]; } SymbolicLinkReparseBuffer; typedef struct { unsigned long reparseTag; unsigned short reparseDataLength; unsigned short reserved; unsigned short substituteNameOffset; unsigned short substituteNameLength; unsigned short printNameOffset; unsigned short printNameLength; wchar_t pathBuffer[1]; } MountPointReparseBuffer; // we don't care about the 3rd union // member in this use case </code></pre> Conveniently for us, the union member in <code>ReparseDataBuffer</code> is at the end. So we don't need to worry about padding the unused space for smaller alternatives. Back in Swift, instead of dealing with <code>ReparseDataBuffer</code> directly, we can work with <code>SymbolicLinkReparseBuffer</code> or <code>MountPointReparseBuffer</code>, depending on our expectation of which union member to read. Yeah, this is a good time to mention that, <a href="https://github.com/dduan/Pathos">Pathos</a> has to include copies of these definition in a separate C module. Not only because we need to "break up the union", the original definition is also only accessible after importing some headers in the NT kernel. So the standard <code>import WinSDK</code> won't suffice. Moving on to notable feature #2. The last member of both <code>SymbolicLinkReparseBuffer</code> and <code>MountPointReparseBuffer</code> <code>pathBuffer</code> is a 1-character long array...why? In C, this is a flexible array member. Such member must always appear at the end of a struct. The word "flexible" in this context refers to the amount of memory allocated for this type of structs : it can vary according to the length of the array as needed. The member such as <code>pathBuffer</code> is here to provide access to the beginning of the buffer. To Swift, <code>pathBuffer</code> looks like a single <code>UInt16</code>. The language obviously don't have a good idea of this C feature. So how to we get the rest of the data from this array? Once again, we have to lean on our understanding of memory layout in C structs. As is common in APIs for flexible array members, the length of the array trailing the struct is provide to us. Let's call it <code>flexibleLength</code>. <img src="/assets/2020/09/flexible_array_member.png" alt="Illustration of memory layout for a C struct with flexible array member" /> We already have the memory for these structs in bytes (remember <code>data</code>?). And we can get the size for the fixed potion of the structs with <pre><code class="language-swift">let fixedStructSize = MemoryLayout< SymbolicLinkReparseBuffer >.stride </code></pre> Putting it all together, we can get the full content of the array by <ol> <li>chopping off the content for struct itself,</li> <li>casting the rest of the raw buffer to the expected element type, and</li> <li>include the last member in this struct as the first element in the array</li> </ol> <pre><code class="language-swift">// Include the first element, which is at // the end of the fixed struct potion. let arrayStart = fixedStructSize - 1 // Cast the data buffer so it's composed // of `wchar_t` aka `UInt16`s. let array = withUnsafePointer(to: data) { $0.withMemoryRebound( to: [UInt16].self, capacity: data.count / 2 ) { sixteenBitData in // chop off the non-array potion sixteenBitData.pointee[ arrayStart ..< (arrayStart + flexibleLength) ] } } // now, go nuts on the array! You earned it! </code></pre> Considerations such as error handling are intentionally left out in this article. You can checkout the source code of <a href="https://github.com/dduan/Pathos">Pathos</a> (on the <code>next</code> branch) for the full glory. Anyways, the flexible array member turns out to be the literal target of the symbolic link. So here is the end of our story. I'm interested to hear about alternative approaches for dealing with union members and flexible array members in Swift. Let me know on <a href="https://twitter.com/daniel_duan">Twitter</a>, or <a href="">Twitch</a> when I'm streaming! Sat, 12 Sep 2020 23:11:48 -0700 https://duan.ca/2020/09/12/fantastic-c-beasts-and-where-to-find-them-in-swift/ https://duan.ca/2020/09/12/fantastic-c-beasts-and-where-to-find-them-in-swift/ Faster C Strings in Swift One of the goals in the re-write of my VFS library <a href="https://github.com/dduan/Pathos">Pathos</a> is to make it go fast. What does that mean when most of the time users are hitting the hard drive when running your code? Well, let's not dwell on that for now. A library like this passes file paths back and forth with C APIs from the OS a lot. So one way to go fast is to keep the original representation of the paths as they are provided to us. On macOS and Linux (and other OSes that Swift doesn't officially support yet), paths are bought and sold in the so called "C strings": <code>NUL</code>-terminated bytes (8-bit integers) with POSIX APIs and 16-bit values on Windows with <code>UNICODE</code>. Let's walk over a couple of examples. Here's how to get the current working directory: <pre><code class="language-c">// POSIX char *getcwd(char *buf, size_t size); // Windows // LPTSTR (with right environment) is `wchar_t *` DWORD GetCurrentDirectory( DWORD nBufferLength, LPTSTR lpBuffer ); </code></pre> The expected steps for using APIs like this are the following: <ol> <li>you allocate a buffer large enough to store any result you expect.</li> <li>you give the buffer to the C function.</li> <li>the C function fills the buffer with some characters, and a <code>NUL</code> (or <code>0</code>) right after the last character in the result.</li> <li>the function use a separate variable to tell you the size of the actual result, not counting the <code>NUL</code>.</li> </ol> It's very generous of these functions to give us the size of the result. Because the point of using <code>NUL</code> to terminate "strings" is to avoid having to use a separate size. Let's use setting the current working directory as the next example: <pre><code class="language-c">// POSIX int chdir(const char *path); // Windows BOOL SetCurrentDirectory(LPCTSTR lpPathName); </code></pre> Yup, these APIs don't need you to tell them the content size of your buffer. But if your content doesn't end with a <code>NUL</code>, they'll happily read beyond your intended stopping point until it finds a random <code>0</code> in memory. Anyways, this is pretty classic C stuff. Now let's talk about Swift! The default option to store a file path for most should be a <code>Swift.String</code>, which is a encoding-agnostic, Unicode glyph based list of characters. But as I mentioned earlier, I want to avoid the cost of decoding and encoding in the case where the path is only passing through the Swift code from C to C, without being analyzed or modified. (If you need a refresher, <a href="https://youtu.be/Vy2r21kli0Q">I recently made a video on Unicode and encoding</a>.) So I chose to use an <a href="https://developer.apple.com/documentation/swift/contiguousarray">ContiguousArray</a> to store these C values: <pre><code class="language-swift">// Simplified for POSIX struct Path { let storage: ContiguousArray<CChar> // ... } </code></pre> <code>ContiguousArray</code> (and <code>Array</code>) provides an excellent way to interact with C APIs we encountered earlier: <pre><code class="language-swift">init( unsafeUninitializedCapacity: Int, initializingWith initializer: ( inout UnsafeMutableBufferPointer<Element>, inout Int ) throws -> Void ) rethrows </code></pre> Don't let the complex-looking signature intimidate you. Essentially, it gives you write access to the array's memory right after its allocation, but before Swift does standard initialization to it. It works really well with the C APIs we looked at earlier: <pre><code class="language-swift">// Store the current directory in a ContiguousArray // Using the Windows API let storage = ContiguousArray( unsafeUninitializedCapacity: Int(MAX_PATH) ) { buffer, count let length = GetCurrentWorkingDirectoryW( DWORD(MAX_PATH), buffer.baseAddress // C API writes in the allocated memory ) count = length // you are responsible for setting size of the array } </code></pre> Read the steps 1-4 again from earlier, it's easy to see how this initializer is designed to fit that pattern. The resulting array will have all the characters as its content, and carries the correct size. When it's time to pass the array back to C, we can provide a pointer easily: <pre><code class="language-swift">storage.withUnsafeBufferPointer { SetCurrentDirectory($0.baseAddress!) } </code></pre> This is not great, because we don't have a <code>NUL</code> at the end of our array. The C function that read our array will sometimes read over the contents memory until it finds a 0! Yikes. So here's an easy fix: <pre><code class="language-swift">(storage + [0]).withUnsafeBufferPointer { SetCurrentDirectory($0.baseAddress!) } </code></pre> Instead of using the memory of <code>storage</code>, we construct a new array with an 0 as its last value. This lets C APIs pick the right place to stop reading. (Incidentally, Swift includes a built-in version of this <a href="https://developer.apple.com/documentation/swift/string/2430818-utf8cstring">for converting String to UTF-8 (8-bit) C strings</a>, which includes the <code>NUL</code> and it's possible to further encode with different encodings.) Although we've fixed the correctness bug, doing this defeats the purpose of storing the C string directly somewhat: constructing a new array each time we want to call a C API is kind of expensive. It involves allocating new memories and copying over the content, etc. Alright. How about we carry around the <code>NUL</code> in our array? Let's update the construction code: <pre><code class="language-swift">let storage = ContiguousArray( unsafeUninitializedCapacity: Int(MAX_PATH) + 1 ) { buffer, count let length = GetCurrentWorkingDirectoryW( DWORD(MAX_PATH), buffer.baseAddress ) buffer[length] = 0 count = length + 1 } </code></pre> We add 1 every time we have a say in size. Then we manually set a 0 at the end of the stuff from C. Having done this, we've solved both the correctness problem and performance concern from earlier! The last bit of of this journey is ergonomics. Carrying an extra <code>NUL</code> is fine if you never look at the array's content. But when you do, it's important to remember that the content we care about is almost all of the array, except for the <code>NUL</code> at the end. In other words, simply don't make off-by-1 mistakes and everything will be fine. Alright, that's easier said than done. To alleviate this off-by-1 painfulness, I ended up exposing a "view" into the array storage that excludes the last element. Here's the actual definition in <a href="https://github.com/dduan/Pathos">Pathos</a>: <pre><code class="language-swift">struct CString<Unit: BinaryInteger>: Equatable, Hashable { private var storage: ContiguousArray<Unit> var content: ContiguousArray<Unit>.SubSequence { storage[0 ..< storage.count - 1] } public func c<T>( action: (UnsafePointer<Unit>) throws -> T) throws -> T { try content.withUnsafeBufferPointer { try action($0.baseAddress!) } } init(cString: UnsafePointer<Unit>) { var length = 0 while cString.advanced(by: length).pointee != 0 { length += 1 } storage = ContiguousArray( unsafeUninitializedCapacity: length + 1 ) { buffer, count in for offset in 0 ..< length { buffer[offset] = cString.advanced(by: offset).pointee } buffer[length] = 0 count = length + 1 } } // ... more stuff } </code></pre> <code>storage</code> in this solution is an private implementation detail. <code>content</code> is the primary access to the content of the string. And finally, this type interops with C APIs correctly and efficiently because of the extra <code>NUL</code> we put at the end of <code>storage</code>. Wed, 09 Sep 2020 01:21:26 -0700 https://duan.ca/2020/09/09/free-c-strings/ https://duan.ca/2020/09/09/free-c-strings/