Daniel Duan's Articles

Daniel Duan's Articles https://duan.ca/ Idle coding I've discovered a new hobby. It involves creating non-trivial libraries from scratch. (Not UI slop!) Except I don't write any code in editors. An agent does all of it. It goes like this: Set up an infinite loop so the coding agent that... <ol> <li>has a clear guard for correctness (unit tests)</li> <li>has an objectively measurable, pass/fail outcome (performance benchmarks)</li> <li>has tools for styling (dead-code tool, formatter)</li> <li>leaves artifacts for future instances (commit changes on a branch, writes failure_analysis_SHA.md)</li> </ol> ... and watch number go up. You'll need a robust prompt, then just feed it to a single agent, over and over. No need to convince the agent instance it needs to repeat (doesn't hurt to try, but they usually give up after a while). It's more fun if you ask for a success artifact (benchmark_success_SHA.tsv) and have a script that monitors for new ones. I run this script with @steipete's VibeTunnel so I can watch the number go up on my phone. <img src="/assets/2026/01/idle_monitoring.jpeg" alt="monitoring benchmark progress in terminal" /> It may take some setup to get going. For example, at the beginning you want to import a large number of unit tests. You'll want to source some input for benchmarking. You'll want some infrastructure (SwiftPM project, Scripts directory, git). But quickly, it becomes an <a href="https://en.wikipedia.org/wiki/Incremental_game">idle game</a>. At first, you'll watch the number of unit tests go from 100% fail to 100% pass. Then, the performance benchmarks. Bonus points if you benchmark against existing libraries, and beat them in performance. This has happened on every project I've done. Codex btw. I should clarify: this is not serious software engineering. The goal is having fun. Although the end result usually is really useful, and because of robust unit testing + benchmarking, it might be good enough to use in production. We'll find out. My favorite idle game in 2026 so far. Sat, 24 Jan 2026 11:52:04 -0800 https://duan.ca/2026/01/24/idle-coding/ https://duan.ca/2026/01/24/idle-coding/ TOMLDecoder Is Now Faster Than C (Thanks to AI) Recently, I gave my TOML library written in Swift <a href="/2025/12/10/TOMLDecoder-0.4.1/">an 800% speed boost</a>. The natural question after that is: how much faster can I push it? <details> <summary> I'm happy to report that TOMLDecoder now parses the <a href="https://github.com/dduan/TOMLDecoder/blob/cea8f0bee33f37e0fcc33b566a742485c71196e7/Sources/Resources/fixtures/twitter.toml">Twitter payload example</a> 1.8x faster than the C library <a href="https://github.com/cktan/tomlc99">tomlc99</a>, and 5x faster than <a href="https://github.com/marzer/tomlplusplus">TOML++</a>. </summary> I tried to be as charitable as possible for the non-Swift libraries while keeping the call sites in Swift. For example, it takes time to create or copy the UTF-8 bytes of a <code>Swift.String</code> into a contiguous region. And that's not counted towards the other libraries' parsing time. TOML++ runs faster with exceptions enabled. So that's the path I chose to benchmark. When bridging the C++ code, I made sure there's no allocation, no checking for input/output, etc, so that the bridging overhead is trivial. Here's the benchmark code run repeatedly to collect an average, with warmups ahead of time: <pre><code class="language-swift">func benchmarkTOMLDecoder(source: String) throws -> Double { let start = CFAbsoluteTimeGetCurrent() let table = try TOMLTable(source: source) let end = CFAbsoluteTimeGetCurrent() blackhole(table) return end - start } func benchmarkCTOML99(source: String) -> Double { var source = source var duration: Double = 0 source.withUTF8 { $0.withMemoryRebound(to: CChar.self) { buffer in let baseAddress = UnsafeMutableRawPointer(mutating: buffer.baseAddress!) let start = CFAbsoluteTimeGetCurrent() let table = toml_parse(baseAddress, nil, 0) duration = CFAbsoluteTimeGetCurrent() - start blackhole(table) } } return duration } func benchmarkCTOMLPlusPlus(source: String) -> Double { var source = source var duration: Double = 0 source.withUTF8 { $0.withMemoryRebound(to: CChar.self) { buffer in let start = CFAbsoluteTimeGetCurrent() let table = tomlpp_parse(buffer.baseAddress, buffer.count) duration = CFAbsoluteTimeGetCurrent() - start blackhole(table) } } return duration } </code></pre> where <code>tomlpp_parse</code> is a minimal wrapper for the TOML++ library: <pre><code class="language-cpp">void *tomlpp_parse(const char *conf, size_t conf_len) { try { static toml::table table{}; table = toml::parse(std::string_view{conf, conf_len}); return static_cast<void *>(&table); } catch (...) { return nullptr; } } </code></pre> If any of these measures are unfair to the C/C++ libraries, I'd love your feedback! </details> Here's the output of the benchmark program I wrote: <pre><code>Benchmarking TOML parsers... File size: 443461 bytes Warming up... Running 100 iterations... Results: ═══════════════════════════════════════════════════════════ TOMLDecoder: Average: 1.232 ms Min: 1.203 ms Max: 1.332 ms cTOML99: Average: 2.226 ms Min: 2.190 ms Max: 2.341 ms cTOMLPlusPlus: Average: 6.107 ms Min: 6.038 ms Max: 6.377 ms TOMLDecoder is 1.81x faster than cTOML99 TOMLDecoder is 4.96x faster than cTOMLPlusPlus ═══════════════════════════════════════════════════════════ </code></pre> I charted the wall clock time and instruction counts over the commit history. You can see that the latest release is a lot faster than 0.4.1: <iframe id="benchmark-iframe" src="/assets/2026/01/tomldecoder-0.4.3-improvements.html" width="100%" height="1200" frameborder="0" style="border: none; display: block; margin: 20px 0;"></iframe> <script> window.addEventListener('message', function(event) { if (event.data.type === 'resize') { const iframe = document.getElementById('benchmark-iframe'); if (iframe) { iframe.style.height = event.data.height + 'px'; iframe.style.transition = 'none'; } } }); </script> ... and, the majority of these commits are authored by AI! How did that happen? <h2>It's old-fashioned engineering, baby!</h2> I ended the <a href="/2025/12/10/TOMLDecoder-0.4.1/">last post</a> with the following (emphasize in bold): <blockquote> ...the project also gained a bunch of infra improvements. <ul> <li>It has a DocC-based documentation site.</li> <li>The entirety of the official test suite is now programmatically imported as unit tests.</li> <li>The source code style is now enforced by swiftformat</li> <li>Platform checks are more comprehensive and modern on CI.</li> <li>Benchmarks are now modernized with ordo-one/package-benchmark.</li> </ul> </blockquote> If you set out to optimize the runtime performance of a software project, infra improvements like these will ensure that your engineer can explore options for optimization with confidence that they won't break the expected behavior, and their efforts can be measured objectively. Most importantly, as detailed in the last post, the architecture of the TOML parser has received some significant upgrades. This type of change is rare in a small project, and I don't expect it to happen again in the next phase of optimization. I set up a separate project that calls into TOMLDecoder so that I can profile it with Instruments. It was during the holidays, and although the idea of trying my hand at micro-optimizing the code, and gradually squeezing out performance juice sounded really fun, I also had a bunch of travel planned. So what else is there to do? I booted up codex. <h2>gpt-5.2-codex, my performance engineer</h2> For the most part, I simply fed this prompt to codex over and over again: <blockquote> Objective: Try to make the p50 of "parse twitter.toml" benchmark improve by > 1.1% on instructions or retains compared to the <code>main</code> branch. Improvements on either is acceptable as a success, but regression in either should be considered a failure. Other metrics in the benchmark does not matter. Verify iteratively: <ol> <li>Make code changes</li> <li>Format with <code>make format</code>.</li> <li>Make sure all tests passes by running <code>swift test</code>.</li> <li>Create a branch prefixed with <code>cc/</code> in name</li> <li>Commit all changes. Include description of the optimization as body of the commit message.</li> <li>Use Scripts/benchmark.sh to run the benchmark, recording its output in a text file</li> <li>If the benchmark result meets the improvement threshold, cherry-pick the change onto main. Otherwise, Commit the benchmark results file to the branch you created, switch back to main and start over.</li> </ol> When you run the benchmark script, NEVER use <code>HEAD</code> as its argument. Use explicit SHAs. Only use Scripts/benchmark.sh SHA_OF_BASE SHA_OF_TARGET to run the benchmarks. Do not try to run the underlying commands directly. You must NOT look at the content of Benchmarks/, or the content of Sources/Resources. To give you some direction, I've profiled the parsing the twitter example, and included the inverted time profile call tree in /tmp/trace-tree.txt. </blockquote> The prompt changed gradually in these ways: <ol> <li>Wording became more streamlined as I figured out how gpt-5.2-codex interprets specific things.</li> <li>The optimization threshold decreased as the lower-hanging fruits got picked</li> <li>The benchmark to optimize changed a bunch of times because they have different data profiles</li> </ol> Each time the optimization threshold is met, I collect another time profile from instruments with the latest change, and restart the session with the same prompt. I actually started the journey with gpt-5.1-codex-max. It would find 5-10% improvements consecutively at the beginning. Then it would start to struggle, then I'd switch to gpt-5.2-codex with the default "Medium" setting, then "High", and eventually "Extra high". Towards the end, the LLM could barely find any speed improvements without regressing other benchmarks in some way. That's when I decided it's time to cut a release. <h2>My observations of the model</h2> Despite occasional struggles with conventional Swift coding style, I find that gpt-5.2-codex is good at analyzing the flow of the parser, and finds ways to short circuit certain logic. These types of discoveries made the parser quite a bit faster. It replaced key comparisons in a hot loop with hash value comparisons, which brought in a significant speedup. In retrospect, this idea seemed fairly obvious, but, in my imagination, I wouldn't have been bold enough to try it. The LLM has a few favorite things to try at the start of each session. <ul> <li>It would see a linear search and try to replace it with a dictionary lookup</li> <li>It would try unrolling loops (in a few cases, this actually helped),</li> <li>It would reserve array capacities ahead of time,</li> <li>It would eliminate copies by converting things into classes</li> </ul> But then the benchmark would regress, which forces it to explore other paths. Although my prompt tells the model to not look at the benchmark itself, it sometimes goes and does it anyways. I suppose benchmark-maxing is a temptation too great to resist for it? As reported by <a href="https://steipete.me/posts/2025/shipping-at-inference-speed">others</a>, I also observed that gpt-5.2-codex would spend a lot of time just analyzing, before attempting any changes. The code change it produces is almost always one-shot. It rarely goes back and revises the idea it's attempting to implement. <h2>Conclusions</h2> Good engineering practices continue to pay dividends with LLMs. TOMLDecoder reached a point where its runtime performance is itself a feature worth talking about. The setup of this project can serve as a benchmark for LLMs, I think? Here's a prompt, a concrete, measurable outcome represented by numbers, a huge test suite. How far can you push those numbers? Thu, 01 Jan 2026 11:07:05 -0800 https://duan.ca/2026/01/01/TOMLDecoder-Is-Faster-Than-C/ https://duan.ca/2026/01/01/TOMLDecoder-Is-Faster-Than-C/ TOMLDecoder 0.4 is 800% Faster I just released version 0.4.1 of <a href="https://github.com/dduan/TOMLDecoder">TOMLDecoder</a>, a TOML 1.0 parser, and <a href="https://developer.apple.com/documentation/swift/codable">decoder</a> implemented in pure Swift. When decoding a TOMLDocument such as <a href="https://github.com/dduan/TOMLDecoder/blob/cea8f0bee33f37e0fcc33b566a742485c71196e7/Sources/Resources/fixtures/twitter.toml">this twitter payload</a>, TOMLDecoder 0.4.1 is roughly 800% faster by wall clock time than 0.3.x. In this post, I’ll discuss how this was achieved. tl;dr: among other things, the gains comes from making the parsing algorithm lazier, and eliminating overheads from bound checking when accessing substrings. Update: An earlier version of this post claimed that adopting Span eliminates cost of all bound checking when accessing the underlying bytes of the TOML content, that turns out to be wrong. The reality is more interesting. The post has been revised to discuss what really brought the performance gains after adopting Span. <h2>The Benchmark</h2> TOMLDecoder now includes benchmarks implemented with <a href="https://github.com/ordo-one/package-benchmark">ordo-one/package-benchmark</a>. I plotted the median from the aforementioned benchmark results below. Each chart includes data points for deserializing the TOML document, and decoding it on top. (Unsurprisingly, decoding takes a bit longer.) The results show wall clock time, CPU instructions, as well as retain count all trending down significantly. In addition to the before and after, there's an extra data point measured specifically prior to adopting Swift's <code>Span</code>. More on that later. <iframe id="benchmark-iframe" src="/assets/2025/12/tomldecoder-0.4.0-benchmark-charts.html" width="100%" height="1200" frameborder="0" style="border: none; display: block; margin: 20px 0; min-height: 1200px;"></iframe> <script> window.addEventListener('message', function(event) { if (event.data.type === 'resize') { const iframe = document.getElementById('benchmark-iframe'); if (iframe) { iframe.style.height = event.data.height + 'px'; iframe.style.transition = 'none'; } } }); </script> <h2>How to make a parser go fast</h2> <h3>Improving data structure and algorithms</h3> ... also known as cheating. Yes, really. In 0.3.x, <code>TOMLDecoder</code> behaves like <a href="https://developer.apple.com/documentation/foundation/jsonserialization">JSONSerialization</a>. When you ask it to decode TOML data, with <code>TOMLDecoder.tomlTable(from:)</code> it goes through the entire document, creates matching container structures within it. For each TOML table, it creates a <code>[String: Any]</code>, for each TOML array, it creates a <code>[Any]</code>. When a table contains an array, for example, a corresponding <code>["key": [...]]</code> entry is created to match. Along the way, the parser also validates the leaf types, so things like a ill-formed date causes an error to be thrown. The end result is a <code>[String: Any]</code> in which everything is known to be valid. A number of things are slow in this process: <ul> <li>The frequent creation and subsequent usage of intermediary Swift arrays and dictionaries require heap allocations.</li> <li>Validating every leaf value takes time.</li> <li>Retrieved values are <code>Any</code>s, so you have to cast it to the expected type to consume them.</li> </ul> TOMLDecoder 0.4 does away with all of that. To represent the containers, and leaf values, 0.4 introduces some light-weight structs, These structs don't manage the actual memory used to store their contents. As the parser work through the bytes of a TOML document, it creates these light weight data types to record the shape of the document, as well as the byte-offsets of the leaf values. These intermediary data are stored in a centralized location to avoid unnecessary heap allocations. Here's what I mean by "cheating": during this phase, the parser doesn't do much validation of the leaf values. What it does is more akin to "lexing", it finds the tokens that could represent a leaf value, and remembers where they are. No work is done to actually validate and create the leaf values. To retrieve any values from the result, you must state what type is expected: <pre><code class="language-swift">// a valid TOML document is always a table at the root level let serverIP = try TOMLTable(source: tomlString) .string(forKey: "ip") // validate this token as a `String` </code></pre> This is an API change. It delays the validation work, and helps avoid conversions from <code>Any</code>. If you only need one field, no validation is necessary on the rest of the leaf values in the entire document. Swift's decoding APIs ask for typed access: if your <code>Codable</code> type has a <code>Date</code> field, you ask the container for a <code>Date</code>, if the matching value at the spot is of a different type, an error is thrown. So the more efficient access pattern benefits the decoding process as well. <h3>Eliminating bound checks</h3> A major source of slowness in TOMLDecoder 0.3.x comes from inefficient patterns when the underyling bytes of a TOML document. The parser holds a reference to the original string, and hands <code>String.UTF8View.SubSequence</code>s to small functions to descend on. A typical piece of the parser might look like this: <pre><code class="language-swift">func skipWhitespaces(_ text: inout String.UTF8View.SubSequence) { let bytes = text.utf8 var i = bytes.startIndex while i < bytes.endIndex { if !isWhitespace(bytes[i]) { // very slow! break } bytes.formIndex(after: &i) } text = bytes[i...] } </code></pre> Using UTF8View makes sure that we aren't dealing with <code>Character</code>s, which could have variable lengths. However, accessing the bytes in this way introduces multiple rounds of bound checks that ends up being super expensive in the hot path of the parser: <ol> <li>The standard library needs to check that a index is valid for the <code>SubSequence</code> aka <code>Substring</code> by comparing it against the start and end indices.</li> <li>Then, the index is used to access the underlying <code>UTF8View</code>, at this point, the standard library checks whether the index is out of bound again.</li> <li>A the end, the library goes into the buffer pointer of the string to retrieve the actual byte.</li> </ol> (All of that assumes that the string's buffer is contiguously stored in memory. There's a even slower path that I could eliminate by ensuring the string is native.) A parser does a whole lot of such accesses. The cost of these bound checks seriously adds up. Since the release of TOMLDecoder 0.3.0, Swift has gained a whole set of features that led to the introduction of <a href="https://github.com/swiftlang/swift-evolution/blob/main/proposals/0447-span-access-shared-contiguous-storage.md">Span</a>. <code>Span</code> is built on compile-time lifetime checks. These checks guarantee the safety when accessing its content. The same function updated for <code>Span</code> looks extremely similar to the original: <pre><code class="language-swift">func skipWhitespace( bytes: Span<UTF8.CodeUnit>, // aka Span<UInt8> remainingBytes: inout Range<Int>, ) { var i = remainingBytes.lowerBound while i < bytes.count { if !isWhitespace(bytes[i]) { break } i += 1 } remainingBytes = i ..< remainingBytes.upperBound } </code></pre> Here, the subscript access of <code>bytes</code> does not incur multiple rounds of bound checks! Rather, it skips step 1, which eliminates 2 integer comparisons per access, a 2/3 reduction in bound check overhead. Further, the compiler can see the access pattern more clearly, it can heuristically eliminate even the final remaning bound checks in some casse. Not having to perform all the bound checks in the tight loop of the parser results significant performance gains as shown in the benchmark results. Here's the kicker. With <code>Span</code>, the bound checks are eliminated because the compiler is confident that the access is safe by construction. If you make a mistake that would lead to unsafe access, Swift will refuse to compile your code. But <code>Span</code> is a language feature that requires new language runtime. You cannot use it on older operating systems. There's other, older ways to avoid bound checks, using <code>UnsafeBufferPointer</code>s. The problem of doing so is that you are responsible for ensuring that the access is safe. In particular, the point of access must occur in a valid scope for the pointer. A piece of parser using such API may look like this: <pre><code class="language-swift">func skipWhitespace( bytes: UnsafeBufferPointer<UTF8.CodeUnit>, remainingBytes: inout Range<Int>, ) { var i = remainingBytes.lowerBound while i < bytes.count { if !isWhitespace(bytes[i]) { break } i += 1 } remainingBytes = i ..< remainingBytes.upperBound } </code></pre> But WAIT! This code using the buffer pointer look extremely similar to the <code>Span</code> version! And if you think carefully, the requirement for maintaining valid scope for the <code>UnsafeBufferPointer</code> is already enforced for any <code>Span</code>s, syntactically! Enter <a href="https://nshipster.com/swift-gyb/">gyb</a>. A script that Swift uses to generate repetitive code in the complier. In TOMLDecoder 0.4, the parser implementation uses it to generate 2 version of the same set of parsing logic: <pre><code class="language-swift">configs = [ ("Span<UInt8>", "@available(iOS 26, macOS 26, watchOS 26, tvOS 26, visionOS 26, *)"), ("UnsafeBufferPointer<UInt8>", "@available(iOS 13, macOS 10.15, watchOS 6, tvOS 13, visionOS 1, *)"), ] }% % for byte_type, availability in configs: ${availability} func parse(bytes: ${byte_type}) throws -> TOMLTable { // same code } % end </code></pre> ... and there's a single place that checks for the OS at runtime: <pre><code class="language-swift">let source: String = ... // TOML string if #available(iOS 26, macOS 26, watchOS 26, tvOS 26, visionOS 26, *) { let bytes = source.utf8Span.span try parse(bytes: bytes) } else { try source.withUTF8 { try parse(bytes: $0) } } </code></pre> The beauty here, is that the compiler does all the work to ensure the access to the <code>Span</code> as well as the buffer pointer are safe, because the logic that does the accessing are identical thanks to <code>gyb</code>. <h2>Conclusion</h2> In reality, there are a ton of other optimizations applied in TOMLDecoder 0.4. For example, instead of doing dictionary look ups, looking up things from a TOMLDocument actually involves a linear search. I know, I know, this goes against what we were taught in CS. But in modern computers, and for typical sizes of TOML documents, a linear search is often faster that computing a hash value, and the subsequent lookups. As part of the release, the project also gained a bunch of infra improvements. <ul> <li>It has a <a href="https://www.swift.org/documentation/docc/">DocC</a>-based <a href="https://dduan.github.io/TOMLDecoder/main/documentation/tomldecoder/">documentation site</a>.</li> <li>The entirety of the <a href="https://github.com/toml-lang/toml-test">official test suite</a> is now programmatically imported as unit tests.</li> <li>The source code style is now enforced by <a href="https://github.com/nicklockwood/SwiftFormat">swiftformat</a>.</li> <li>Platform checks are more comprehensive and modern on CI.</li> <li>Benchmarks are now modernized with <a href="https://github.com/ordo-one/package-benchmark">ordo-one/package-benchmark</a></li> </ul> I think of this release as a preparation for a eventual 1.0 release, which will support the <a href="https://forums.swift.org/t/the-future-of-serialization-deserialization-apis/78585/171">new deserialization APIs from Swift</a>. Even through I went through some optimizations for speed in this post, I still have a bunch of ideas I want to try to squeeze out more performance gains. That's exciting. Wed, 10 Dec 2025 17:44:34 -0800 https://duan.ca/2025/12/10/TOMLDecoder-0.4.1/ https://duan.ca/2025/12/10/TOMLDecoder-0.4.1/ Padding a Squircle in FreeCAD 1.0 I bought a 3D printer, and, as a result, got into 3D modeling recently. I'm still super new to everything. From experience, this means I should write down things I'm learning. The beginner's perspective is valuable to other beginners! A <a href="https://en.wikipedia.org/wiki/Squircle">squircle</a> is a square who's corners are replaced by curves with g3 continuity, which can be thought of as really hecking smooth curves where the derivatives of the curve must also be smooth. In <a href="https://www.freecad.org/">FreeCAD</a> 1.0.x, it's not immediately obvious how to sketch such a shape, in order to create solid shapes out of it. With the help of ChatGPT, and the internet, I figured out a somewhat cohesive way to do this recently. Now I need to write it down lest I forget! <img src="/assets/2025/06/squircle-00-finished.png" alt="Finished 3D shape with a squircle front profile" /> Our goal is to have a sketch of a squircle that we can use to pad or pocket to create 3D shapes. In the sketch, we want to first create the sides of the square that's straight, and connect them with the g3 curve, which can be done via the Curves workbench. But first, let's break down the squircle's geometry a bit. A squircel is both a square and a circle. Each side of the square has a straight line, it ends at a specific point, let's call it A, where the curve of the circle begins. From this point, if you draw a line perpendicluar to the straight part, and repeatly draw it on every such point, the new lines will intercect at point B. The length of the line AB is the radius of the squircle. <img src="/assets/2025/06/squircle-01-radius.png" alt="Image illustrating the radius of a squircle" /> In FreeCAD, we want to include these AB lines as guides. They will define both the straight lines on the side, as well as the correct space for the curves. The guide lines and the sides will form a cross shape. Make sure everything is symmetrical, and the AB lines all have the same size. <img src="/assets/2025/06/squircle-02-sketch-cross-shape.png" alt="Screenshot of the FreeCAD sketch of a cross outline" /> Note that the AB lines are in construction mode represented by dashes. Exit the Sketch. Switch to the Curves workbench, and select 2 of the 4 concrete edges from the sketch. Use the Blend Curve option from the workbench. And you'll get some funky looking curves connecting the straight lines, and it's represented by a "Blend_Curve" in the object tree. <img src="/assets/2025/06/squircle-03-curve-unfixed.png" alt="Screenshot of adding the blend curve" /> Select the "Blend_Curve" in the tree and tweak its Data fields. <ul> <li>Set <code>Auto Scale</code> to <code>true</code>.</li> <li>Set <code>Parameter</code> to <code>0mm</code> for both edges.</li> <li>The new curve should smoothely connect with the ends of the sides that's close to the other straight side. If not, set <code>Reverse</code> to <code>true</code>.</li> <li>Most importantly, set <code>Continuity</code> for both edges to <code>G3</code>.</li> </ul> <img src="/assets/2025/06/squircle-04-curve-fixed.png" alt="Screenshot of fixed blend curve" /> That completes 1 of the 4 corners. Repeat for the other 3. And now you have a 2D squircle! There should be 4 "Blend_Curve"s in the object tree, as well as the original sketch. Now things get a little convoluted. Although things on screen look like a closed wire, we can't use it to create a 3D shape for 2 reasons: <ol> <li>the "Blend_Curve"s aren't part of any body.</li> <li>the curves and the straights don't belong to the same geometry.</li> </ol> There's a easy way to fix both problems, thanks to the fact that multiple shapes in FreeCAD 1.0 can comine into a single sub-objects shape binder. Switch back to the Part Design workbench. Activate the body you want the final 3D shape in (can be an existing or new body). Select all 4 blend curves in the object tree, and the sketch. You should see the entire wire highlighted in a uniform color. Then use the sub-objects shape binder tool, which gets you a single binder in the tree of the active body. You should see the binder on screen. Hide other things if they are distracting you. Now, select the edges in the binder -- all 8 of them! After that, you can use the Pad tool to create a 3D shape. And that's it! <hr /> Obviously, a squircle is the simpliest shape to incorporate a g3 continous curve. But once we have the curve as a wire, anything is possible in FreeCAD. Sun, 15 Jun 2025 16:00:00 -0800 https://duan.ca/2025/06/15/padding-a-squircle/ https://duan.ca/2025/06/15/padding-a-squircle/ GRPC Status With Error Details in Swift <h2>Introduction</h2> In GRPC, one could define an RPC that, in addition to the normal request-response messages, it also defines a custom message to represent errors: <pre><code class="language-proto">message SignUpWithEmailRequest { string email = 1; string password = 2; string referral_code = 3; } message SignUpWithEmailResponse { AccessTokenDTO token = 1; } message SignUpWithEmailErrorResponse { enum Kind { KIND_UNKNOWN = 0; KIND_EMAIL_ALREADY_REGISTERED = 1; KIND_INVALID_PASSWORD = 2; KIND_INVALID_EMAIL = 3; KIND_INVALID_CODE = 4; } Kind kind = 1; repeated string reasons = 2; } service AuthenticationService { rpc SignUpWithEmail(SignUpWithEmailRequest) returns (SignUpWithEmailResponse) {} } </code></pre> ... in this example, <code>SignUpWithEmailErrorResponse</code> is not directly referenced in by <code>AuthenticationService</code>. But a server can use it as GRPC status with details. In Go the code might look like this: <pre><code class="language-go">import ( "google.golang.org/grpc/codes" "google.golang.org/grpc/status" ) // ... _, err = queries.GetUserByEmail(ctx, email) if err == nil { response := &SignUpWithEmailErrorResponse{ Kind: SignUpWithEmailErrorResponse_KIND_EMAIL_ALREADY_REGISTERED, Reasons: []string{}, } st := status.New(codes.AlreadyExists, "Email already registered") stWithDetails, err := st.WithDetails(response) if err != nil { return nil, err } return nil, stWithDetails.Err() } </code></pre> This is all very type-safe, very demure, until you realize that in grpc-swift 1.X there's no API to retrieve this "status with detail". When the information is transimitted over the wire, you will have to dig it out manually. In this post, I'll document how I did this with a client-side interceptor. <h2>The Swift interceptor</h2> In Swift, when you make the RPC request, you'll get a standard error code and error message if the server returns an error with the code shown earlier: <pre><code class="language-swift">// Use the generated client code to make the gRPC request var request = SignUpWithEmailRequest() request.email = // ... request.password = // ... request.referralCode = // ... do { let response = try await client.signUpWithEmail(request) } catch { guard let error = error as? GRPCStatus else { print("Error: \(error)") return } print(error.code) // AlreadyExists print(error.message) // "Email already registered" } </code></pre> But, you, being a responsible client developer/tech lead/CTO, want to use the type-safe enum from the protobuf definition so that you can display the error in the right context, or perhaps localize it properly. Here's the big picture: there may be many such custom RPC error types. Our solution should be universal, and flexible to handle each of them. Enter interceptors! I mean, chances are, you know about them because you are working with gRPC. Let's write one to get our type-safe status details. Starting with a custom receive method, for the ".end" part of the response: <pre><code class="language-swift">final class GRPCErrorDetailsInterceptor<Request, Response>: ClientInterceptor<Request, Response>, @unchecked Sendable { override func receive( _ part: GRPCClientResponsePart<Response>, context: ClientInterceptorContext<Request, Response> ) { switch part { case .end(var status, let headers): // extract the error details, and forward it. default: context.receive(part) } } } </code></pre> ... the "end" part contains the error status, as well as some trailing metadata. The metadata includes our status details under the key <code>grpc-status-details-bin</code>. It's base64 encoded, so we'll need to decode it... <pre><code class="language-swift">switch part { case .end(var status, let headers): guard // grab the status details let statusDetails = headers["grpc-status-details-bin"].first, // decode to data let data = Data(base64Encoded: statusDetails), // ... default: context.receive(part) } </code></pre> At this point, with some experience with GRPC in Swift, you might think it's time to instantiate your custom error type with <code>.init(seralizedData:)</code>. But there'd be 2 problems: <ol> <li>You don't want each custom types from protobuf to make an appearance in an interceptor.</li> <li>This data would not be in the right shape, despite what the metadata key says.</li> </ol> In fact, the data is of the well-known type <code>Google_Rpc_Status</code>. And our stutus details, well, one its <code>.details</code> element. So: <pre><code class="language-swift">switch part { case .end(var status, let headers): guard let statusDetails = headers["grpc-status-details-bin"].first, let data = Data(base64Encoded: statusDetails), // the data, despite being under "grpc-status-details-bin", is // indeed not the status detail, but the statu itself: let googleStatus = try? Google_Rpc_Status(serializedData: data) // and the `details` field contains the actual status detail: let details = googleStatus.details.first, else { context.receive(part) break } // ... default: context.receive(part) } </code></pre> ... <code>details</code> is of type <code>Google_Protobuf_Any</code>. It is indeed a payload with the content for <code>SignUpWithEmailErrorResponse</code> as defined in the Protobuf. One question remains: how do we pass it from our intereceptor to the RPC call site? Look at the call site from earlier: we have 2 code paths. If the call succeeds, we get a <code>SignUpWithEmailResponse</code>. If it fails, we get a <code>GRPCStatus</code> as the thrown error. Lucky for us, <code>GRPCStatus</code> has an unused field, <code>cause</code>. In my version of <code>grpc-swift</code>, the field has the following docstring: <pre><code class="language-swift">/// The cause of an error (not 'ok') status. This value is never transmitted /// over the wire and is **not** included in equality checks. public var cause: Error? { ... } </code></pre> It seems like a perfect vessel for our status details! <pre><code class="language-swift">switch part { case .end(var status, let headers): guard let statusDetails = headers["grpc-status-details-bin"].first, let data = Data(base64Encoded: statusDetails), let googleStatus = try? Google_Rpc_Status(serializedData: data) let details = googleStatus.details.first, else { context.receive(part) break } // isn't it convenient that we declared `status` as a `var` ealier 😉? status.cause = details // forward to the caller, yay! context.receive(.end(status, headers)) default: context.receive(part) } </code></pre> Now our client will get the details of type <code>Google_Protobuf_Any</code> from the <code>.cause</code> field of the thrown error. The client can proceed to decode it using the protobuf-generated specific type with its built-in <code>.init(decodingAny:)</code> initializer: <pre><code class="language-swift">// Use the generated client code to make the gRPC request var request = SignUpWithEmailRequest() request.email = // ... request.password = // ... request.referralCode = // ... do { let response = try await client.signUpWithEmail(request) } catch { guard let error = error as? GRPCStatus else { print("Error: \(error)") return } // let's be type-safe, finally! guard let details = error.cause as? Google_Protobuf_Any, let signUpError = try? SignUpWithEmailErrorResponse(decodingAny: details) else { print("Error: \(error)") return } // 🎉 switch signUpError.kind { // ... } } </code></pre> <h2>Conclusion</h2> I find this to be clean, targeted solution. Knowing the error detail's transmission format is key to making this work. The fact that we also got a clean architecture from exploiting an unused field is also very cool. Sat, 25 Jan 2025 19:53:30 -0800 https://duan.ca/2025/01/25/grpc-status-with-details-in-swift/ https://duan.ca/2025/01/25/grpc-status-with-details-in-swift/ Notes on Using the MLIR C API in Swift For curiosity's sake, I decided I want to play with MLIR's C API with Swift. I spent quite some time to get a skeleton project up and running on my Mac. Here's my notes for future reference. (If you find this useful, I'd be curious to know what you're working on!). Modern LLVM comes shipped with MLIR. At the time of writing, all I had to do to get it is <code>brew install llvm</code>. If you used the default Homebrew installation options, you'll find <code>libMLIR.dylib</code> and friends under <code>/opt/homebrew/opt/llvm/</code>. Building MLIR following the <a href="https://mlir.llvm.org/getting_started/">instructions</a> on the website is also fairly straightforward. You'll want <code>llvm-config</code> from your version of LLVM to be in your path. For the Homebrew-installed version, you want <code>/opt/homebrew/opt/llvm/bin/</code> to be one of the place the shell looks. Now, it's time to make the project. With CMake, of course. Because I couldn't figure out how to tell SwiftPM to link the right dylib :) But worry not, CMake ain't that bad. Like with SwiftPM, we want to make a module for the MLIR C API. I call the module <code>cmlir</code>. Make a directory with that name, and create 2 text files: First, <code>module.modulemap</code>: <pre><code>module cmlir [system] { header "shim.h" export * } </code></pre> Second, <code>shim.h</code>: <pre><code class="language-c">#include <mlir-c/IR.h> </code></pre> Amazing. Let's assume we want to have a Swift library that uses <code>cmlir</code>. And a executable that depends on the library. You can organize the Swift source files for these as you like (yay CMake!). The sample library has one file, <code>lib.swift</code>: <pre><code class="language-swift">import cmlir public func makeAContext() -> MlirContext { mlirContextCreate() } </code></pre> The sample app is just a <code>main.swift</code>: <pre><code class="language-swift">import MLIRSwift print(makeContext()) </code></pre> ... as you can see, through these targets, we are expecting to properly execute some code from MLIR. All that's left is to build all these stuff. AKA, the hard part! But the <code>CMakeLists.txt</code> really isn't that bad. I'll just leave it here with comments: <pre><code class="language-cmake">cmake_minimum_required(VERSION 3.22) # Note we include "C" here, without it there'd be a build error 🤷 project(swift-mlir LANGUAGES C CXX Swift) # This is where llvm-config comes to play find_package(MLIR REQUIRED CONFIG) include_directories(${MLIR_INCLUDE_DIRS}) # Include our modulemap include_directories(cmlir) # I can't believe this is all it takes to make a Swift dylib! add_library(MLIRSwift SHARED lib.swift) # Wasted a lot of time on figuring out the right library to link T-T target_link_libraries(MLIRSwift PRIVATE MLIRCAPIIR) # Nothing special here add_executable(myapp main.swift) target_link_libraries(myapp PRIVATE MLIRSwift) </code></pre> And there you have it. Here's the file structure in the end: <pre><code>. ├── CMakeLists.txt ├── cmlir │ ├── module.modulemap │ └── shim.h ├── lib.swift └── main.swift </code></pre> For completeness, I'll also include commands that produces the final executables. It's just the simplest cmake commands. But it may not be obvious for Swift programmers: <pre><code>make build # make a bulid direcory anywhere, make sure you .gitignore it if necessary cd build cmake -G Ninja .. cmake --build . </code></pre> With this sample project, running <code>build/myapp</code> should get you this output: <pre><code>MlirContext(ptr: Optional(0x0000600001784180)) </code></pre> And that's just exciting, isn't it? Sat, 31 Aug 2024 18:44:48 -0700 https://duan.ca/2024/08/swift-mlir-cmake/ https://duan.ca/2024/08/swift-mlir-cmake/ AI Coding I tried out GitHub Copilot this weekend. I've <a href="/2023/03/03/qualia-that-was-coding/">been thinking</a>, and talking, about AI like your average technology person. So here we are: what's my reaction to LLMs getting better and better at coding? LLMs are great at information retrieval. Copilot or GPT read a lot of code from GitHub, StackOverflow, Reddit, etc. And now, given some context (code in the file I'm editing, its surrounding project info, etc), it can retrieve, by probability, the best snippet of code that could come next. I posted this example on social media recently: <blockquote> Me: imports my own file system library Pathos (first time in this code base). Types name of computed property using it. Copilot: suggests an implementation that uses the library, slightly incorrectly. Me: Accepts suggestion, fixes error. Me: Types in name of second computed property. Copilot: suggests correct implementation using different parts of my library. </blockquote> Fantastic! I'd like to think that the open-source code I put on GitHub made some small contribution to this specfic interarction of programming in Swift. If this is true, then I've contributed to everyone who is going to use Copilot to write Swift. So did everyone who posted their questions, answers, examples, musings, tests, libraries, apps ... about Swift on the internet. Copilot is the child of our collective labor. Before these LLM programming tools, we didn't interact. Now, we have a hive mind. Will LLMs take our jobs? I can't wait for that to happen. But, so far, I just don't see it. Programming is, barely, superfacially, about information retrieval. When it is, it's usually in the context of studying, training, interviewing, or some type of product grind. Sure, that covers a lot of of reason for writing code. BUT, here's the thing: I hate writing code for all of those reasons. If you had some time working in tech, you'd have heard the saying along the lines of "programmer should try their best to replace themselves". This is partially about working with people around you, but it also serves as a lens to evaluate programming tasks: if you are doing simple things over and over again, it means you are stuck at a place of stagnation. You are not growing, and probably not creating as much values as you deserve. LLMs are going to replace some of the things we do as programmers. The parts that are boring, repetitive, stagnating. Will some people lose their jobs because of it? Maybe. At first, these will be, for the sake of simplicity, "low-level" jobs (I'm going to deliberatly leave "low-level" undefined here). For those who get to stay, however, our lives are going to become more fun and creative. Why yes, I included myself in this group because I'm <a href="https://en.wikipedia.org/wiki/Egocentric_bias">egocentrically biased</a>. Let's say I'm wrong, these AI models, very quickly, grew super capable, to such degree that it's no longer commercially viable to hire humans for programming tasks. Well, it would be a kick in the butt for me to have to find a new career. But a kick in the butt is often the right thing to move us along in life, yes? When your comfort zone comes with a biweekly paycheck, it's extra hard to grow outside of it. But I should love to live in the post-progammer world as a software user. Chances are, there'd be less bugs? Anyways, this has been my super unscientific, totally biased, very personal musings about AI this week. Mon, 27 Mar 2023 20:10:14 -0700 https://duan.ca/2023/03/27/ai-coding/ https://duan.ca/2023/03/27/ai-coding/ Don't Say That Alice: Hey, did you know that COVID virus was leaked from a lab? Bob: No no, you don’t know that. Don’t be racist! Alice: Oh, sorry I didn’t know. How so? Bob: Because the lab is in China. Alice: Ok, so where DID the virus come from? Bob: It jumped from bats to pangolin, and then to humans in a wet market. Alice: A what market? Bob: Wet market in China. People buy animals from there. Alice: Who would want to buy a bat. or a … what’s the other one? Bob: They eat bats. It's normal for them. Sat, 11 Mar 2023 22:03:03 -0800 https://duan.ca/2023/03/11/do-not-say-that/ https://duan.ca/2023/03/11/do-not-say-that/ The Qualia that was Coding As of this writing, I'm deeply skeptical of LLM replacing our needs for skilled programmers. Problem solving is a creative process. When a new tool takes away the repetitive part of a job, the worker can focus more on being creative. However, if one believes that LLM, or some alternative models, will bring about the end of programming as a wide-spread human activity, then we'll have a problem (well, probably a few problems) at hand. Programming is a lonely activity. Yes, let's acknowledge the existence of peer programming, code reviews, design meetings, etc. But the deep work -- exploring the problem space, grappling with known/unknown constraints, iterating on the design… these things happen in the programmer's own head. When the satisfying answer emerges in the end, so many paths has been traveled that the traveler can't even retread them in themselves, let alone sharing it with others. This is blatantly obvious to anyone who's accomplished non-trivial programming tasks. The end of programming means the end of such experience. I have no doubt that people will keep writing code even if machines are more capable of doing it wholesale. To what degree would those people share the level of challenges, passion, skill, or, let's face it, pain in writing code, I cannot predict. But it seems abundantly clear that, those people would amass to a small fraction of the total number of programmers today. If you believe you are the last generation of human programmers, the human experience, the qualia of programming is yours to lose. Does that make sense? When AI models can replace us, we will forget what's like to write code without them really soon. Maybe the literal we, who lived through it won't, but future generations may never truly know it. Difficult as it is, it might be worth your time to leave records of what it's like to be in the flow state, what it's like to feel stuck, what it's like to steadily traverse and eliminate alternative theories for a bug, what it's like to perform mastery with primitive tools to generate clues, what it's like to critic a working prototype, what it's like to lay out a suite of tests, what it's like to replace a huge amount of code with better, shorter code... If God is coming, the least we can do is to give future anthropologist a helping hand, and document what it was like to be a human. Fri, 03 Mar 2023 23:42:34 -0800 https://duan.ca/2023/03/03/qualia-that-was-coding/ https://duan.ca/2023/03/03/qualia-that-was-coding/ The Lazy Programmer's Guide to Single-user Mastodon As of today, my identity in the "fediverse" has become <code>@daniel@duan.ca</code>. This post was originally published on <code>duan.ca</code>. Pretty neat, eh? I achieved this by hosting a private instance of the open-source Mastodon project. Here's my experience setting that up, and why it seemed harder than it is. <h2>What exactly did I achieve?</h2> I own the domain <code>duan.ca</code>, which serves as the root of an existing website. I set up an functioning instance of Mastodon with a single user <code>@daniel</code>. This user is discoverable in other Mastodon instances as <code>@daniel@duan.ca</code>. <h2>How did I do it?</h2> As someone who barely touch server-side programming, running a Mastodon instance seemed very daunting to me. What database to use? How? What's the best way to host a Rails app? I need a email server? Where to host the media files I post? There are a lot of step-by-step set-up tutorials on the internet. This post is, definitively, not one of those. But having gone through the process of <a href="https://web.archive.org/web/20221219043954/https://marketplace.digitalocean.com/apps/mastodon">setting up an instance on DigitalOcean</a>, I end up with a few tips and tricks to share! Let's through them: <ol> <li>If you have a DigitalOcean account, setting a server up is as simple as searching up "Mastodon" in their market place as a step in creating your Droplet.</li> <li>You do not need an email server for a single-user instance. Think about it, you have root access to the database, what do you need email for? Verify your own identity? Spam yourself?</li> <li>That said, Sendgrid is free for low-volume emails.</li> <li>Ok, you ssh into your Droplet, the first thing you see is a guided CLI experience for setting things up. Don't go through it just yet! For me, my root domain <code>duan.ca</code> hosts an entirely different site. So I have to pick a subdomain to host Mastodon. Make sure you set up an A record in your website's DNS for your chosen subdomain. If this record isn't read, the guided setup will fail when it tries to get an certificate from Let's Encrypt.</li> <li>Unless you want to buy a new DNS service from DigitalOcean, your A record should be configured with your existing DNS server provider! I needed a few minutes to un-confuse myself about this fact. DigitalOcean's documentation makes it really hard to discover this fact.</li> <li>The only thing you need to get right is your subdomain, and the DNS record for it. In the rest of the guided setup, you can make mistakes, and you can fix them later.</li> <li>I used an s3 bucket for media. s3's UX is kind of terrible, so if you are doing it for the first time, good luck.</li> <li>Instead of "admin", choose your Mastodon username in the setup guide. This user does not have special privilege for the site anyways, despite what the default name suggests. After this step, the guide will provide you with a password, take note of that if you want to log in with this user later :)</li> <li>Again, you could figure out how to get an STMP server to send emails on your Mastodon instance's behalf. But you are the only user, right? So you can just make some value up for all things SMTP/email related. At the end, when the guide asks whether to send a test Email, choose "no".</li> <li>The Mastodon code is located at <code>/home/mastodon/live</code>. Most of the information is stored in <code>.env.production</code> in that directory. This is where you can change your mind/fix your mistake from the guided setup. You can use Vim instead of Nano. (Although if you like Vim, you'd probably find out anyways).</li> <li>Mastodon ships with an admin <a href="https://docs.joinmastodon.org/admin/tootctl/">CLI tool</a> <code>tootctl</code>. It's located in <code>bin/tootctl</code>.</li> <li>Before attempting to run this tool, switch to user <code>mastodon</code> by running <code>su mastodon</code>. Otherwise, you'd be confused as to why you need to install Ruby stuff, and can't connect to the Postgres database.</li> <li>If the guide didn't tell you an error, and your values in <code>.env.production</code> are all right, you will be able to type your subdomain in a browser and see Mastodon! Log in with your username and password from the guide. Upload a profile photo to test whether your media service is set up correctly.</li> <li>(DO NOT DO THIS until you finish the next step!) At this point, you'd be able to go to an existing Mastodon instance, and look yourself up! If your subdomain is <code>social.duan.ca</code>, username <code>daniel</code>, you should be discoverable as <code>@daniel@social.duan.ca</code> on a site such as <a href="https://mastodon.social">mastodon.social</a> now. But if you do this, that site may get confused, and permanently consider <code>@daniel@social.duan.ca</code> as your fediverse identity. So don't do that! Finish the next step.</li> <li>To use your root domain instead of this subdomain, you 301 redirect <code>/.well-known/host-meta*</code> from root to the real location. <a href="https://masto.host/mastodon-usernames-different-from-the-domain-used-for-installation/">this article</a> explains it very well. Make sure to both set up the URL redirect, as well as <code>LOCAL_DOMAIN</code>, and <code>WEB_DOMAIN</code> in <code>.env.production</code>.</li> <li>Make your user the owner of the site (RTFM). You'll see additional, site-wide settings afterwards. Make sure to disable signup, either in the GUI, or with <code>tootctl</code>.</li> </ol> That it! These may seem like a lot of tips, but I think, as a smart programmer, you can eventually figure all of these things out pretty easily. I put them down here because, had I known these, I would've saved a bunch of times in the end. <h2>Why?</h2> Why spend the time, and money, to go through all of this? This is a really good question. I <a href="https://duan.ca/2022/11/20/the-social-graph/">wished</a> less people would leave Twitter. I host my personal site/blog on my own domain, as opposed to using a Medium site or something. So, if I have to "federate" my microblogs, why shouldn't I try to gain more independence? How many of these mastodon instances can do better than Twitter, in terms of site maintenance, and content moderation? Instinct tells me, something between self-hosting, and Twitter, might be the right balance. But I'm here for the long haul, and, if self-hosting proves to be the wrong direction, I'll pivot. Until then, this has been fun. Sat, 10 Dec 2022 18:21:11 -0800 https://duan.ca/2022/12/10/lazy-mastodon/ https://duan.ca/2022/12/10/lazy-mastodon/ The Social Graph From a purely personal perspective, connecting with people via the internet is good. I'm grateful that services such as Twitter exist. Since the year 2008, a significant amount of my life happened on Twitter. The only important component of Twitter is the connections. I'm lucky that some people like what I post. I'm used to a small stream of thoughts from those I follow. These days, those I follow aren't posting much. They are posting on other websites. The connection I so depend on are broken. It saddens me. Therefore, I'm forced to move to maintain the connections. I have many thoughts. A tweet storm? Well, if the cool people are there to read it, sure. Knowing myself, I'll probably cross-post it to the many places where my people are. That's a hustle that I don't have the patience for. That saddens me, too. So, here we are, the good old blog post. Cross-posting a link is doable. <h2>The Twexit</h2> Remember when we make Brexit jokes about everything on Twitter? Good times! Here's what I think about the "Twexit". Thanks to email archive and 1Password, I found my mastodon.social login credentials. According to the site, I joined in 2018 (coincidentally, an exact decade later that me joining Twitter). 4 years ago was when me, and, more importantly, a lot of the people I follow on Twitter, first learned about Mastodon. They had their reasons to give it a try. They were my reason to give it a try. We did. I honestly can't tell if this time is more different than that last time. We were, and are, voting with our feet. Warning: the following is a cynical take. We are leaving this time, partially, because a lot of Twitter employee were forced to leave. They built the site for us, we are grateful for their work. We find them getting laid off / forced to change life style unacceptable, so much so that we... abandon the product they built for lesser ones. Of course, that illogicality was based on some superficial analysis. An easier, logical path to follow, but also a harder truth to swallow, is that the Twexit, both times, were motivated by hate. Hate over a person that possess outsized influence on the platform. To hate is not wrong. Again, I don't have this reason to leave. Simply, I want to maintain my connections. I wish I shared the hate, though. Because that would've made me less sad. (Perhaps "hate" is too strong of a word. Substitute it with the word you associate with the big personalities that drove you or your followees off to other platforms). Will Twexit succeed this time? I've observed (to my relief!) that my timeline on Twitter hasn't completely died. It would've been chilling to see it die (remember those empty streets during early COVID? NOT great, no). I hold out hope that better the user experience will ultimately prevail (more on that in the next section); and negative emotions alone won't sustain this round of Twexit. <h2>A Better Social Graph</h2> That is not to say "I hope everyone return to Twitter". If Mastodon in late 2022 turns out to be a better product, I'm happy to migrate/establish new connections there. Now, I, the master of the Daniel-verse, will tell you what the best user experience for "social media" could/should have been. This text you are reading is originally published on a personal website composed of purely static assets (mainly HTML, minimal amount of CSS, and essentially, no JavasSript). Critically, it also includes this advanced technology called an RSS feed. If I have to psyco-analyze the young Danial from early 2010s (why not?), an important reason he became hopelessly dependent on Twitter was the death of Google Reader. Prior to that event, the web service he habitually check was not predominately Twitter. Some of us remember how the connections were built prior to Facebook and Twitter. There were many sites that served similar purposes, sure. But there's also a community that subscribed to each others' RSS feed. We commented on others' posts, we linked and reposted, and commented on the comments. There were conversations, connections. An event that stood out for me during that era was when <a href="https://en.wikipedia.org/wiki/Larry_Wall">Larry Wall</a> commented on my post about his Perl book. You see, the connections included cool people, too. So, Google Reader. What if a API for posting comments for a web link became standardized? What if the RSS readers includes a text box for that API? What if everyone also include comments (ignore spams for now) in their feeds? Yeah, that would be, arguably, a similar experience to Twitter or Mastodon today. In fact, this isn't even a new idea. The <a href="https://indieweb.org/">IndieWeb</a> folks did their Twexit before Google Reader was killed. In retrospect, they were smart as hell. We didn't follow their footsteps because Twitter turned out to be a better user experience for people that don't own a website. Things could easily have played out differently, though. Blogging and RSS could've become more mainstream sooner. Web 2.0 could've improved interactivity of these blogs and feeds. Facebook and Twitter wouldn't have become so big of a deal. Oh well. <h2>We Are All Indie Web</h2> That's an inspirational statement. In this moment, I'm more convinced of this inspiration than ever. More Mastodon-like open-source software, more protocols such as ActivityPub, smarter economic structures for content ownership. All of these, and more. In late 2022, this opening prose from the ancient Chinese book Romance of the Three Kingdoms keeps echoing in my mind: <blockquote> ...分久必合，合久必分 </blockquote> Literally, it means "Long periods of division precede unity and long periods of unity precede division." From Google Reader to Twitter to Mastodon, this tension between centralization and decentralization is oh-so-familiar. An equilibrium that could stabilize, IMHO, is a centralized user experience running on a decentralized infrastructure. Naively, that's a software thing. Realistically, it all rely on the collective mind of our social graph. Our connections forms the graph, the graph funnels in money, the money makes the good user experience. Sorry, money plays a role in the real world, like it or not. For now, though, I'm happy to work to maintain my connections. I'll use many sites. I'll also try to become "indie". Maybe I'll set up a feed for "microblogging", and write some service to cross-post to those sites. Maybe I'll write my own mastodon-compatible ActivityHub implementation. Maybe these will turn out to be some more ill-fated personal projects -- that sounds awfully fun. Sun, 20 Nov 2022 13:06:48 -0800 https://duan.ca/2022/11/20/the-social-graph/ https://duan.ca/2022/11/20/the-social-graph/ Control For whatever reason, a dialogue from the Matrix trilogy stuck with me. I've never heard anyone else talk about these quotes. Maybe they aren't that deep and I have been over-projecting meanings. Anyways, Neo (N) and Councillor Hamann (H) are in the last human city Zion, looking at a vast array of machines. They had this exchange about control: <blockquote> H: Down here, sometimes I think about all those people still plugged into the Matrix and when I look at these machines I can't help thinking that in a way we are plugged into them. N: But we control these machines; they don't control us. H: Of course not. How could they? The idea is pure nonsense. But… it does make one wonder… just… what is control? N: If we wanted, we could shut these machines down. H: Of course. That's it. You hit it. That's control, isn't it? If we wanted we could smash them to bits. Although, if we did, we'd have to consider what would happen to our lights, our heat, our air… </blockquote> It is scary how often this definition of (lack of) control applies in life. Figured I'd share. Wed, 22 Jun 2022 13:13:09 -0700 https://duan.ca/2022/06/22/control/ https://duan.ca/2022/06/22/control/ Pro-Science, Anti Research Science discussion in the mainstream always focus on the reductive "pro" vs "anti" dynamic. While there are a lot of people in the world who flat-out deny the existence of established science, making the mainstream conversation a worthy discourse, that's not what I'm trying to participate here. Within the "for-science faction", there are two seemingly compatible points of view: <ol> <li>We should save humanity from distinction.</li> <li>We should improve the human condition.</li> </ol> See the difference? Some do, others blind themselves to it. Surely, distinction is a worse human condition? Correct! ... therefore, we shouldn't get distracted by things like making rockets, digitizing our minds, creating suns on earth. Oh, we still need basic science research but... if we could spend money on saving ourselves, those far-out ideas should take a chill pill. Survival is the most important thing! The argument is around austerity: <a href="https://en.wikipedia.org/wiki/Roulette">why put the chips on 21, when we can put it on black</a>? It's strange, isn't it? The crowd that wants dirty energy sources gone, also dislike space exploration? The "progressives" wants nothing but to conserve? Scientific research should be expensive, because, in order to improve our lives in unimaginable ways, "the science" must actuate. Knowing an asteroid's existence is not the same as harvesting the resource from it; knowing how the sun works is not the same as replicating it here; knowing a monkey can play pong with brain probes is not the same as humans doing it safely. (Some may categorize these things as "engineering" or "commercialization": yes, all of that.) The austerity argument is anti-scientific research. Saying we should save money on fusion to spend more money on solar panels is the same as "but children are starving in Africa!". The world sucks in multiple ways. You may feel overwhelmed that, as a whole, humanity is under siege for one reason or another. But others may feel differently. They want to put their limited chips on a riskier bet for bigger returns. They <a href="https://en.wikipedia.org/wiki/Expected_value">are no less rational</a> than you. And (thank god) we don't have a world dictator that forces us to take their bet, or yours. I'm at peace with that. Fri, 20 May 2022 15:33:25 -0700 https://duan.ca/2022/05/20/pro-science-yet-anti-research/ https://duan.ca/2022/05/20/pro-science-yet-anti-research/ Visceral Growing up in China, I remember using google.cn daily. At some point in my teens, it stopped working, and I had to use google.com.hk . Then ... I left. The first time I got off the plane in the US, and got on a computer, google.com was the browser homepage. Now, this is not a story about how important Google is to me (to the dismay of the fully adult version of me, it is). Around the same time, there were other, perhaps less relatable things happening to the internet that was once niche, and open to Chinese users (small website hosting services got shut down one by one, blog posts got deleted a few days later, all websites forced to have government approval to exist). As a teen, I didn't like that feeling of spaces closing around me. I didn't have money, the freedom to explore virtually was precious. Most people on Twitter never had visceral experience of censorship like mine. It's not something that happens to some asshole spewing hate on a website; not a sinister plot that aims at your group with some shared some beliefs. True censorship feels like environment decay, like oxygen leaving the air. It's quite, gradual, steady. It happens to everyone, and no one. It happens to you, whether you know it or not. The thing about "visceral" tho, is that it's personal by definition. These words won't duplicate my experience in you. But maybe you will pause and smell the roses, and the subtle decrease of oxygen in your air. Tue, 26 Apr 2022 10:42:58 -0700 https://duan.ca/2022/04/26/visceral/ https://duan.ca/2022/04/26/visceral/ Using Instruments on Non-macOS, Non-iOS Executables How do you use use Instruments, the instrumentation tool distributed alongside Xcode? It's easy if you are profiling a macOS or iOS app. No so with a regular old command-line tool built, say, with SwiftPM. To do so, you'll need to codesign the target binary. Otherwise Instruments will give you a <code>"Failed to gain authorization"</code> error message. Here's the steps: <ol> <li>Create a entitlement file with <code>com.apple.security.get-task-allow</code> set to true:</li> </ol> <pre><code class="language-xml"><?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> <plist version="1.0"> <dict> <key>com.apple.security.get-task-allow</key> <true/> </dict> </plist> </code></pre> Let's call it <code>Entitlements.plist</code> <ol start="2"> <li> Build the executable. </li> <li> Codesign your executable with the entitlement file </li> </ol> <pre><code>codesign -s - -f --entitlements path/to/Entitlements.plist path/to/executable </code></pre> Now proceed with your profiling in Instruments. Not sure where this is officially documented. So I'm officially documenting it here. Thu, 31 Mar 2022 20:39:05 -0700 https://duan.ca/2022/03/31/instrumenting-commandline-executables/ https://duan.ca/2022/03/31/instrumenting-commandline-executables/ Flake For Non-Nix Projects The ideal outcome of using Nix or home-manager is to have as many things as possible under their control. Among these things are developer tools for coding. You may need <code>cargo</code>, and <code>rust-analyzer</code> for Rust projects; <code>ipython</code>, and <code>python-lsp-server</code> for Python projects, etc. Of course, the simplest solution is to have them all installed and configured in home-manager or the OS level. But, over time, you'll end up with a ton of stuff. Maybe that won't bother you. Fine, but this approach is at odds with some convention of the Nix community. You see, there's this command called <code>nix-shell</code>, or <code>nix develop</code>, which sets up a environment with certain packages, with the expectation that you only need them sometimes, usually within the context of developing a certain software project. So, arguably, a better alternative to installing everything you possibly need is to keep only the essential tools you always need, regardless of what you are working on. Things like <code>neovim</code>, <code>git</code>, <code>ripgrep</code>, etc. When a project demands things such as <code>rust-analyzer</code>, running <code>nix develop</code> should set it up for you. How do we implement that? In a imaginary world where every single software project is built with a flake.nix, the <code>devShell</code> property should provide everything the software project owner expect you to need. And you just <code>cd</code> into the root of the project, run <code>nix develop</code>, and you are off to the races. Several problems: <ol> <li>running <code>nix develop</code> is repetitive, and therefore, annoying.</li> <li>the owner of a flake.nix may have drastically different taste for the best dev tools for their project (bet!).</li> <li>we don't live in a world built with Nix flakes.</li> </ol> Yikes. To solve <code>1</code>, enter <a href="https://github.com/nix-community/nix-direnv">direnv</a>. It's a piece of software that let you create a environment for a directory, and automatically switch to it when you enter said directory. The "environment" could contain envvars, and, you guessed it: a Nix developer shell. If you have a flake.nix, add <code>use flake</code> in your project directory's <code>.envrc</code> file, direnv will automatically call <code>nix develop</code> the next time you <code>cd</code> in. Neat! To make this work in practice, you'll want to add <code>.envrc</code> and <code>.direnv/</code> to your global git ignore list, as they are personal preferences that probably shouldn't end up in git history. Ok, we are so close to solve problem <code>2</code>, and <code>3</code> now. In short, <code>nix develop</code> may set up a environment, but it may not be the environment that suits you the best. <a href="https://github.com/nix-community/nix-direnv">nix-direnv</a> extends direnv to save us: <pre><code>use flake path/to/flake#target </code></pre> ... with this in <code>.envrc</code>, direnv will set up the Nix environment according to <code>path/to/flake#target</code>. That could point to any location on your hard drive! You can have a flake.nix whose <code>devShell</code> has <code>cargo</code>, and <code>rust-analyzer</code>. You can have another with <code>ipython</code>, and <code>python-lsp-server</code>. Mix and match to your liking to the infinite degree... For now, I've decided to give nix-direnv a try. Alongside my home-manager configuration, I've also included extra flakes for generic Python/Rust projects, and specific projects that may require a mix of tools. A lot of the project I work on don't use flake as their package manager. With this approach, I get to customize my setup for them each, and stay in the comfort of Nix and home-manager. direnv and nix-direnv can be configured together with home-manager. To achieve everything mentioned in this article, including direnv's shell integration (Fish, for me), it's as simple as <pre><code class="language-Nix"># In home manager config... programs.direnv = { enable = true; nix-direnv = { enable = true; enableFlakes = true; }; } </code></pre> To recap, having the snippet above in my home-manager setup, I now can enter any project's root directory and add a <code>.envrc</code> file with the content <code>use flake ~/src/dotfiles/direnvs/python</code>. <code>~/src/dotfiles/direnvs/python</code> contains a <code>flake.nix</code> (and a <code>flake.lock</code>) that has the <code>devShell</code> value I like for all Python projects. When I <code>cd</code> into this project, <code>(nix-)direnv</code> will read from that <code>devShell</code> and set every tool listed under there. The tools are cached in a <code>.direnv</code> directory so when I return here, the setup is basically instantaneous. Since I make <code>git</code> to ignore <code>.envrc</code>, and <code>.direnv</code> no matter where they are, this project I'm working on is unaffected by all this. Sat, 19 Mar 2022 00:37:01 -0700 https://duan.ca/2022/03/19/nix-dirnev/ https://duan.ca/2022/03/19/nix-dirnev/ Flake, Home Manager, and Extra Packages So, you use a standalone home-manager, it's set up with flake, tracking a particular nixpkgs channel. How do you use a package from another channel? This seemingly simple task took me, a Nix noob, quite a bit of research to solve. Here's how I did it. A simple flake.nix for home-manager might look like this: <pre><code class="language-nix">{ inputs = { nixpkgs.url = "github:nixos/nixpkgs/nixos-21.11"; home-manager.url = "github:nix-community/home-manager/release-21.11"; home-manager.inputs.nixpkgs.follows = "nixpkgs"; }; outputs = { self, nixpkgs, home-manager }: { "dan@some-mac" = home-manager.lib.homeManagerConfiguration { username = "dan"; system = "x86_64-darwin"; homeDirectory = "/home/dan"; configuration = { config, pkgs ... }: { home.packages = [ pkgs.hello ]; }; }; }; } </code></pre> A few things of note: <ol> <li>home manager is set to follow <code>nixpkgs</code>, which tracks <code>nixos-21.11</code>.</li> <li><code>pkgs.hello</code> refers to the package in <code>nixos-21.11</code> as well.</li> </ol> To put things in concrete terms, our goal is to put a package from a channel other than <code>nixos-21.11</code> alongside <code>pkgs.hello</code>. The key to my solution is the <code>extraModules</code> in <code>home-manager.lib.homeManagerConfiguration</code>'s argument set. We'll leverage it to modify the environment made available to its sibling, <code>configuration</code>. First, add the new channel as input <code>nixpkgs-unstable</code>: <pre><code class="language-nix">{ inputs = { # ... nixpkgs-unstable.url = "github:nixos/nixpkgs"; }; ... } </code></pre> Then, add a small module to <code>extraModules</code>. In it we make <code>nixpkgs-unstable</code> an argument to <code>_module</code>. <pre><code class="language-nix">{ input = { ... }; outputs = { self, nixpkgs, nixpkgs-unstable, home-manager }: { # Note we also pass in nixpkgs-unstable here "dan@some-mac" = home-manager.lib.homeManagerConfiguration { extraModules = [ ({ pkgs, ... }: rec { _module.args.nixpkgs-unstable = import nixpkgs-unstable { inherit system; }; }) ]; # ... }; }; } </code></pre> ... now that we added <code>args.nixpkgs-unstable</code>, it becomes available in the configuration: <pre><code class="language-nix">{ input = { ... }; outputs = { self, nixpkgs, nixpkgs-unstable, home-manager }: { # Note we also pass in nixpkgs-unstable here "dan@some-mac" = home-manager.lib.homeManagerConfiguration { # ... configuration = { config, pkgs, nixpkgs-unstable, ... }: home.packages = [ pkgs.hello nixpkgs-unstable.python39Packages.python-lsp-server ]; }; }; }; } </code></pre> There, we made <code>python39Packages.python-lsp-server</code> from nixpkgs's master branch appear alongside our standard, default channel. And that's how you add packages from a different channel in a flake setup for standalone home-manager. Tue, 15 Mar 2022 18:53:59 -0700 https://duan.ca/2022/03/15/standalone-home-manager-flake-channels/ https://duan.ca/2022/03/15/standalone-home-manager-flake-channels/ An Invisible Chasm In the journey of growing software systems, there comes a point where certain engineering decision will create a chasm so large that it's practically impossible to go back. The initial crossing of this chasm is so easy that it often happens without anyone noticing. Nonetheless, it is a costly mistake. On one side of this chasm, any member of the engineering team can answer the question "how, exactly, is my code being consumed in our engineering org?". Here lies clarity. Your team fearlessly delete dead code. Legacy logic has their days marked. Lacking documentation? Not ideal, but you can always find how an API gets used in all scenarios easily. On the other side, no one has the answer. The corpses pile up. Deprecating things comes with uncertainty. You warn your team "please carefully consider adding things here, because we'll keep it around forever". Need to respond to an incident? Better hope your monitoring system is good. Trying to trace the code path won't get you answers in time. We'll leave that for the post-mortem. Tooling, yes, exactly what decision leads to the chasm depends on the state of tooling for your technology stack. The programming languages, build system, repository setup, coding style guides, IDEs, linters, search engine ... all factor into the final answer. A statically typed language compiler won't help you look up users in a different micro-service. State-of-art symbol lookup may exist for a dynamically typed system because some companies have thrown enormous amount of money to solve it. Maybe your kick-ass style guides enforced by linters across multiple languages makes it possible to locate anything by text. You see, it is a quantitative matter: it is always possible to revert to the good old days. But it will be expensive in terms of dollars, and/or will power. When did you cross the chasm? Beware of the chasm. Thu, 17 Feb 2022 13:40:27 -0800 https://duan.ca/2022/02/17/invisible-chasm/ https://duan.ca/2022/02/17/invisible-chasm/ Building IndexStoreDB on Linux: The Portable Technique <h2>The problem</h2> As of writing of this article, when you attempt to use Apple's Swift library <a href="https://github.com/apple/indexstore-db">IndexStoreDB</a> on Linux as a normal SwiftPM dependency, it won't build successfully: <pre><code class="language-fish">❯ cat Package.resolved { "object": { "pins": [ { "package": "IndexStoreDB", "repositoryURL": "https://github.com/apple/indexstore-db", "state": { "branch": "swift-5.5.2-RELEASE", "revision": "e771994778265c2efe8d33a7ca30adf5f3d2065a", "version": null } } ] }, "version": 1 } ❯ swift build > /dev/null ❯ echo $status 1 </code></pre> The issue is documented in the build instruction for Linux: <blockquote> The C++ code in the index requires <code>libdispatch</code>, but unlike Swift code, it cannot find it automatically on Linux. You can work around this by adding a search path manually. </blockquote> <pre><code>$ swift build -Xcxx -I<path_to_swift_toolchain>/usr/lib/swift -Xcxx -I<path_to_swift_toolchain>/usr/lib/swift/Block </code></pre> Okay, so, how would my package build in Linux environments where the Swift toolchain's setup is unknown? How do we avoid building with one toolchain while mixing in <code>libdispatch</code> from another toolchain somewhere? Here's what I did for my command-line tool <a href="https://github.com/dduan/Clue">Clue</a>. <h2>The solution</h2> The essence of the problem is about the installation location of the Swift toolchain. Our solution makes the following assumptions: <ol> <li>A Swift toolchain is installed on the file system on Linux (duh!).</li> <li>The toolchain is at least similar to the one distributed on Swift.org. So <code>libdispatch</code> is distributed alongside the other binaries, in a stable relative directory.</li> </ol> Having the <code>swift</code> command available (assumption #1), we can just let it tell us about itself with the <code>-print-target-info</code> flag: <pre><code class="language-fish">❯ swift -print-target-info { "compilerVersion": "Swift version 5.5.2 (swift-5.5.2-RELEASE)", "target": { "triple": "x86_64-unknown-linux-gnu", "unversionedTriple": "x86_64-unknown-linux-gnu", "moduleTriple": "x86_64-unknown-linux-gnu", "compatibilityLibraries": [ ], "librariesRequireRPath": false }, "paths": { "runtimeLibraryPaths": [ "/home/dan/.swiftenv/versions/5.5.2/usr/lib/swift/linux" ], "runtimeLibraryImportPaths": [ "/home/dan/.swiftenv/versions/5.5.2/usr/lib/swift/linux", "/home/dan/.swiftenv/versions/5.5.2/usr/lib/swift/linux/x86_64" ], "runtimeResourcePath": "/home/dan/.swiftenv/versions/5.5.2/usr/lib/swift" } } </code></pre> Great! We see where the runtime is installed. Now we can invoke assumption #2, that <code>libdispatch</code> is at a relative location to the rest of the runtime. In the output from above, the value for <code>runtimeResourcePath</code> happens to be the parent directory for <code>libdispatch</code>'s headers. The <code><path_to_swift_toolchain></code> value in <a href="https://github.com/apple/indexstore-db">IndexStoreDB</a>'s official instruction in this particular setup would be <code>/home/dan/.swiftenv/versions/5.5.2</code>. So the following command would have worked: <pre><code>swift build -Xcxx -I/home/dan/.swiftenv/versions/5.5.2/usr/lib/swift -Xcxx -I/home/dan/.swiftenv/versions/5.5.2/usr/lib/swift/Block </code></pre> All we need to do is parse this information at build time, and it should work on every Linux setup! Choose whatever parsing method you like. Here's (more or less) the <code>Makefile</code> for <a href="https://github.com/dduan/Clue">Clue</a>: <pre><code class="language-make">SHELL = /bin/bash ifeq ($(shell uname),Darwin) EXTRA_SWIFT_FLAGS = "--disable-sandbox" else SWIFT_TOOLCHAIN = "$(shell swift -print-target-info | grep runtimeResourcePath | cut -f 2 -d ':' | cut -f 2 -d '"')" EXTRA_SWIFT_FLAGS = -Xcxx -I${SWIFT_TOOLCHAIN} -Xcxx -I${SWIFT_TOOLCHAIN}/Block endif define build @swift build --configuration $(1) -Xswiftc -warnings-as-errors ${EXTRA_SWIFT_FLAGS} endef .PHONY: build build: $(call build,release) .PHONY: test test: @swift test ${EXTRA_SWIFT_FLAGS} .PHONY: debug debug: $(call build,debug) </code></pre> <ol> <li><code>make build</code> / <code>make test</code> / <code>make debug</code> all work as expected, building IndexStoreDB successfully.</li> <li>As-is, this snippet is project-agnostic. So you can throw it in your SwiftPM project and it should "just work".</li> </ol> Alright! Wed, 09 Feb 2022 15:58:41 -0800 https://duan.ca/2022/02/09/building-indexstoredb-on-linux/ https://duan.ca/2022/02/09/building-indexstoredb-on-linux/ Making of a TOML parser During the holidays, I spent some time on the parser that underlies <a href="https://github.com/dduan/TOMLDecoder">TOMLDecoder</a>. The original implementation targeted TOML's spec version 0.5. A few release-candidates for version 1.0 has come out since then. So it's a good time to give this project attention. Back in the 0.5 days, the TOML spec was (well) written in English with a few examples for each element in the language. The spec document both: <ol> <li>how a TOML document should be spelled out (table header should have a <code>[</code>, followed by 0 or more whitespace, then, a key, 0 or more whitespace, then <code>]</code>).</li> <li>rules for detecting conflicts in a syntactically correct document (if <code>a.b</code> is a table, then it's invalid to set its value to be a date in the same document).</li> </ol> My memory of hand-rolling a recursive-descent scanner that conforms to both of the above requirements was unpleasant. My lesser understanding of the markup language, the vagueness for the validation rules in the spec, as well as (I think?) poorer skill for writing parsers all contributed to the unpleasantness. TOML has gained an ABNF for its syntax since then, making it less necessary to reply on the English descriptions. It's the holidays, maintaining existing code is hard. So, I decided to have some fun, and re-write the parser from scratch. The rest of this post contains notable things from the rewrite. I translated the ABNF with parser combinators. The recent <a href="https://pointfree.co">Pointfree</a> video series made me curious about the performance of this parsing style in Swift. At the end of the rewrite, I added benchmarks to find out. Here's benchmark for parsing an example TOML document <pre><code>name time std iterations ----------------------------------------------------------- example-toml.decoder 454840.000 ns ± 3.63 % 3045 example-toml.combinator 422721.500 ns ± 4.28 % 3266 example-toml.scanner 82232.000 ns ± 9.39 % 16793 example-toml.c 5901.000 ns ± 24.72 % 216140 </code></pre> <code>example-toml.decoder</code> is time spent on parsing + decoding. <code>example-toml.scanner</code> is the time spent by the old parser. <code>example-toml.c</code> is the performance of a C library. So, the new parser is much much slower! However, I know from debugging through 200+ unit tests that, the new parser fixed a few serious bugs compared to the old. It also conforms to diffs between version 0.5 and 1.0 of the TOML spec. Speed alone is not good enough of a reason to throw this work away! There's another reason the new parser is superior, but I'll talk about it later. To be fair to parser combinators, I made several decisions that preferred development speed over runtime speed during development. These were conscious decisions. There were no benchmarks at time so I didn't want to worry about it pre-maturely. Point is, there are rooms for significant runtime speed improvements. Perhaps I'll write a follow-up post detailing my journey to make the parser go fast later; the C library is part of the benchmark for a reason! I was also inspired by Joe Groff's blog post <a href="http://duriansoftware.com/joe/Constructing-human-grade-parsers.html">Constructing human-grade parsers</a>. Instead of stopping at the first syntax error, a parser (any parser!) should treat the error as part of its successful output, deal with the erroneous part of the input, and recover from it. This approach means the portion of the input after the first error gets parsed, and any error it may contain can be found and reported, too! Joe's post explains this well. To talk about this further, let's dive into some details. As mentioned earlier, to validate a TOML document, there are rules for both syntax and semantics to consider. TOML has a few top-level constructs: table header, array-table header, and key-value pairs. Each of these alone can be validated purely based on syntax. On a high level, a TOML parser can do the following: <ol> <li>parse a list of top-level constructs</li> <li>iterate over this list, gradually assembly the complete TOML object</li> </ol> Errors could exist from each of these two steps To make the parser human-grade, the errors must not propagate by disrupting the parsing logic. In Swift, this means the code don't throw the (conceptual) errors, instead, the top-level constructs include error as a possible value: <pre><code class="language-swift">enum TopLevel: Equatable { case table(Key) case arrayTable(Key) case keyValue(KeyValuePair) case error(Reason) } </code></pre> Upon evaluating a <code>TopLevel</code> value, the parser must not stop if it generates a conflict according to TOML's requirement. We take note of this error, and move on to consume the next <code>TopLevel</code>. Errors from this step will join the error from <code>TopLevel.error</code> at the end for users to see. Therefore, the second step's code roughly does this: <pre><code class="language-swift">var errors = [Error]() for value in topLevelValues { switch value { case .error(let reason): errors.append(reason) default: do { evaluate(value) } catch { errors.append(error) } } } if !errors.isEmpty { throw UserFacingError(details: errors) } // no error! parsing succeeded </code></pre> This code "synchronizes" the errors from both the syntactical, and semantical level. With some additional effort to make the errors <code>CustomStringConvertible</code>, a erroneous TOML document such as <pre><code class="language-TOML">a = "Hello b = 43 [b] </code></pre> makes the parser generate the following error message: <pre><code>Deserialization failure: * Value |1, 5| Missing closing character `"` in string * Conflict |3, 2| Conflicting value at [b] Existing value is 43 </code></pre> The explanation I presented above is simplified by a lot. In reality, the "human-grade" upgrade can go a lot further. For example, if the definition of <code>a</code> has a syntax error, instead of rejecting it, as long as we can figure out what the intent is, we can pretend it's a good value. That way, any semantic issues related to <code>a</code> can be discovered. Improvements like this can be added to the parser in many places still. Anyways, here's where I abruptly stop the story of my new TOML parser. There are a lot of exciting space for improvment. So this article might be a "part 1". No promises, though. Sun, 03 Jan 2021 14:56:10 -0800 https://duan.ca/2021/01/03/human-grade-toml-parser-in-swift/ https://duan.ca/2021/01/03/human-grade-toml-parser-in-swift/ Installing Nix on macOS 11 Big Sur Note: as of Feburary, 2022, the steps written below has been incorporated by the official Nix install script for macOS. This article is, therefore, obsolete. Here are the steps for installing nix on macOS 11 Big Sur. <h3>Preparation</h3> Decide on which disk the nix store is to be installed. By end of this process, you'll have a disk name like <code>disk4</code>. Personally, I like looking it up in the Disk Utility app: <img src="/assets/2020/12/disk-utility.png" alt="Disk Utility" /> You can also use other methods, such us the <code>diskutil list</code> command. Have this disk name ready. <h3>Installation</h3> <h4>1. Create the path <code>/nix</code> the macOS way.</h4> Edit or create the file <code>/etc/synthetic.conf</code>, adding this line: <pre><code>nix </code></pre> Reboot the Mac. <h4>2. Create a APFS volume for the nix store.</h4> <pre><code>sudo diskutil apfs addVolume diskX APFS 'Nix Store' -mountpoint /nix </code></pre> Make sure to replace <code>diskX</code> with the disk you identified in the preparation step. <h4>3. Mount the volume.</h4> Edit or create the file <code>/etc/fstab</code>, adding this line: <pre><code>LABEL=Nix\040Store /nix apfs rw,nobrowse </code></pre> Reboot the Mac. <h4>4. Install Nix the "normal" way</h4> <pre><code>sh <(curl -L https://nixos.org/nix/install) </code></pre> <h4>5. You are done! Try it out. I ran this little <a href="https://web.archive.org/web/20201130222947/https://nix.dev/tutorials/dev-environment.html">example</a> as a test and things appears to work as expected.</h4> <h3>Bonus: using Nix in Fish shell</h3> As a fish user, I'm used to fixing things up myself since software from non-fish users usually pretends fish doesn't exist. Anyways, in the final steps of its installation, Nix sets up a few environment variables in a bash script. I've adapted these specifically for fish on macOS. Add these to your config.fish: <pre><code>set -x NIX_PROFILES "/nix/var/nix/profiles/default $HOME/.nix-profile" set -x NIX_SSL_CERT_FILE "$HOME/.nix-profile/etc/ssl/certs/ca-bundle.crt" set -x NIX_PATH /nix $HOME/.nix-defexpr/channels set -x PATH $HOME/.nix-profile/bin $PATH </code></pre> It shouldn't be to hard to do this on other OSes. The relevant information is in <code>~/.nix-profile/etc/profile.d/nix.sh</code>. Sun, 13 Dec 2020 12:00:52 -0800 https://duan.ca/2020/12/13/nix-on-macos-11-big-sur/ https://duan.ca/2020/12/13/nix-on-macos-11-big-sur/ Markra 1.0 I've never been patient enough to learn the JIRA plain-text syntax, depite being a frequent user throughout the years. So a while ago I wrote this little utility that lets me compose JIRA ticket in Markdown. Some thoughts were put into whether this should be an OSS and free (as in beer) app, or something I distribute via the Mac App Store. Ultimately, the first option prevailed. So, without further adiue, here's the repo for <a href="https://github.com/dduan/Markra">Markra</a> with <a href="https://github.com/dduan/Markra/releases/tag/1.0">version 1.0 macOS app download in the release page</a>. <img src="/assets/2020/11/markra-screenshot.png" alt="A Marka macOS app screenshot" /> As far as I'm concerned, the app is pretty much done. And its feature set is small and self-explanatory. So I don't expect more releases other than minor bug fix in the future. I hope some find it as useful as I do. Sun, 22 Nov 2020 12:53:52 -0800 https://duan.ca/2020/11/22/marka/ https://duan.ca/2020/11/22/marka/ Fantastic Beasts in C and Where To Find Them in Swift Swift has a pretty decent C-interoperability story. But C has many features! Today, I'll tell you a story involving a few not-so-well supported C features and Swift. It all started when I decided to re-write <a href="https://github.com/dduan/Pathos">Pathos</a> with Windows support. One of the library's offering is reading the literal target of a symbolic link: if <code>b</code> is a link to <code>a</code>, then <code>Path("b").readSymlink()</code> should return a another path that's equivalent to <code>Path("a")</code>. The Windows API that returns this information is <a href="https://docs.microsoft.com/en-us/windows/win32/api/ioapiset/nf-ioapiset-deviceiocontrol">DeviceIoControl</a>: <pre><code class="language-c">BOOL DeviceIoControl( HANDLE hDevice, DWORD dwIoControlCode, LPVOID lpInBuffer, DWORD nInBufferSize, LPVOID lpOutBuffer, DWORD nOutBufferSize, LPDWORD lpBytesReturned, LPOVERLAPPED lpOverlapped ); </code></pre> Notice anything weird? Hint: <code>LPVOID</code> is <code>void *</code> in standard C. This function is, for the lack of better words, polymorphic: depending on your input, it can intake and output different types. As a caller, it is your responsibility to look up what type is needed and cast them to and from those <code>void *</code>s. The size of the data structure is returned as well. We'll have a lot to talk about that later. Perhaps, surprisingly, this is not too hard to deal with in Swift. In my <a href="/2020/09/09/free-c-strings">last article</a>, I detailed how we can use an Swift API to work with C buffers: <pre><code class="language-swift">/// get the file `handle`... /// now call `DeviceIoControl` var data = ContiguousArray<CChar>( unsafeUninitializedCapacity: kMax ) { buffer, count in var size: DWORD = 0 DeviceIoControl( handle, FSCTL_GET_REPARSE_POINT, nil, 0, buffer.baseAddress, DWORD(buffer.count), &size, nil ) count = Int(size) } </code></pre> So this fills the array of <code>CChar</code>s with the necessary bytes for out result. I named the variable <code>data</code> because it is exactly the same concept as <code>Foundation</code>'s Data, as most Swift programmers know. As promised, we'll cast this data to the actual type so that we can retrieve information from its bytes. Side note: casting in this context is a documented usage, So it really is more awkward rather than bad. And there's a safe way to do it: <pre><code class="language-swift">withUnsafePointer(to: data) { $0.withMemoryRebound( to: [ReparseDataBuffer].self, capacity: 1 ) { buffer in // first element in `buffer` is /// a `ReparseDataBuffer`! Yay } } </code></pre> It's very important to note that <code>ReparseDataBuffer</code> is a struct with fixed, predictable layout, that the API <code>DeviceIoControl</code> promises to return. In practice, this means it is defined in C. Swift does not currently guarantee struct layout. So unless you really know what you are doing and don't care about forward compatibility, you should not do this with Swift structs. So far this story has been boring for avid Swift programmers. Fear not, things will get spicy now. Let's talk about this <code>ReparseDataBuffer</code>. It's an imported C type with a few notable features. <pre><code class="language-c">typedef struct { unsigned long ReparseTag; unsigned short ReparseDataLength; unsigned short Reserved; union { struct { unsigned short SubstituteNameOffset; unsigned short SubstituteNameLength; unsigned short PrintNameOffset; unsigned short PrintNameLength; unsigned long Flags; wchar_t PathBuffer[1]; } SymbolicLinkReparseBuffer; struct { unsigned short SubstituteNameOffset; unsigned short SubstituteNameLength; unsigned short PrintNameOffset; unsigned short PrintNameLength; wchar_t PathBuffer[1]; } MountPointReparseBuffer; struct { unsigned char DataBuffer[1]; } GenericReparseBuffer; }; } ReparseDataBuffer; </code></pre> Feature #1: it has a union member. A <code>union</code> in C is an area in memory that could be any of the types specified in the union: <pre><code class="language-c">// X.a is a `char` and X.b is a `uint64_t`. // And they occupy the same memory because // only 1 of them exists at a time. typedef union { char a; uint64_t b; } X; </code></pre> Swift does not own a direct analog for this. So if we import this <code>ReparseDataBuffer</code> definition, there wouldn't be a good way to access the data inside the union. As I pointed out in the comment, members of a union occupy the same space in memory. The largest member defines the size of that space, so everyone can fit inside of it. Each union member interprets the same bytes according to their own definition. Given this knowledge, we can derive a solution that works around Swift's limitations: break up the union (sorry, this whole paragraph reads super suggestive of the real world union. It's probably why this word is picked for this data structure in the first place. But I do not intend to say anything about the real world here)! <pre><code class="language-c">typedef struct { unsigned long reparseTag; unsigned short reparseDataLength; unsigned short reserved; unsigned short substituteNameOffset; unsigned short substituteNameLength; unsigned short printNameOffset; unsigned short printNameLength; unsigned long flags; wchar_t pathBuffer[1]; } SymbolicLinkReparseBuffer; typedef struct { unsigned long reparseTag; unsigned short reparseDataLength; unsigned short reserved; unsigned short substituteNameOffset; unsigned short substituteNameLength; unsigned short printNameOffset; unsigned short printNameLength; wchar_t pathBuffer[1]; } MountPointReparseBuffer; // we don't care about the 3rd union // member in this use case </code></pre> Conveniently for us, the union member in <code>ReparseDataBuffer</code> is at the end. So we don't need to worry about padding the unused space for smaller alternatives. Back in Swift, instead of dealing with <code>ReparseDataBuffer</code> directly, we can work with <code>SymbolicLinkReparseBuffer</code> or <code>MountPointReparseBuffer</code>, depending on our expectation of which union member to read. Yeah, this is a good time to mention that, <a href="https://github.com/dduan/Pathos">Pathos</a> has to include copies of these definition in a separate C module. Not only because we need to "break up the union", the original definition is also only accessible after importing some headers in the NT kernel. So the standard <code>import WinSDK</code> won't suffice. Moving on to notable feature #2. The last member of both <code>SymbolicLinkReparseBuffer</code> and <code>MountPointReparseBuffer</code> <code>pathBuffer</code> is a 1-character long array...why? In C, this is a flexible array member. Such member must always appear at the end of a struct. The word "flexible" in this context refers to the amount of memory allocated for this type of structs : it can vary according to the length of the array as needed. The member such as <code>pathBuffer</code> is here to provide access to the beginning of the buffer. To Swift, <code>pathBuffer</code> looks like a single <code>UInt16</code>. The language obviously don't have a good idea of this C feature. So how to we get the rest of the data from this array? Once again, we have to lean on our understanding of memory layout in C structs. As is common in APIs for flexible array members, the length of the array trailing the struct is provide to us. Let's call it <code>flexibleLength</code>. <img src="/assets/2020/09/flexible_array_member.png" alt="Illustration of memory layout for a C struct with flexible array member" /> We already have the memory for these structs in bytes (remember <code>data</code>?). And we can get the size for the fixed potion of the structs with <pre><code class="language-swift">let fixedStructSize = MemoryLayout< SymbolicLinkReparseBuffer >.stride </code></pre> Putting it all together, we can get the full content of the array by <ol> <li>chopping off the content for struct itself,</li> <li>casting the rest of the raw buffer to the expected element type, and</li> <li>include the last member in this struct as the first element in the array</li> </ol> <pre><code class="language-swift">// Include the first element, which is at // the end of the fixed struct potion. let arrayStart = fixedStructSize - 1 // Cast the data buffer so it's composed // of `wchar_t` aka `UInt16`s. let array = withUnsafePointer(to: data) { $0.withMemoryRebound( to: [UInt16].self, capacity: data.count / 2 ) { sixteenBitData in // chop off the non-array potion sixteenBitData.pointee[ arrayStart ..< (arrayStart + flexibleLength) ] } } // now, go nuts on the array! You earned it! </code></pre> Considerations such as error handling are intentionally left out in this article. You can checkout the source code of <a href="https://github.com/dduan/Pathos">Pathos</a> (on the <code>next</code> branch) for the full glory. Anyways, the flexible array member turns out to be the literal target of the symbolic link. So here is the end of our story. I'm interested to hear about alternative approaches for dealing with union members and flexible array members in Swift. Let me know on <a href="https://twitter.com/daniel_duan">Twitter</a>, or <a href="">Twitch</a> when I'm streaming! Sat, 12 Sep 2020 23:11:48 -0700 https://duan.ca/2020/09/12/fantastic-c-beasts-and-where-to-find-them-in-swift/ https://duan.ca/2020/09/12/fantastic-c-beasts-and-where-to-find-them-in-swift/ Faster C Strings in Swift One of the goals in the re-write of my VFS library <a href="https://github.com/dduan/Pathos">Pathos</a> is to make it go fast. What does that mean when most of the time users are hitting the hard drive when running your code? Well, let's not dwell on that for now. A library like this passes file paths back and forth with C APIs from the OS a lot. So one way to go fast is to keep the original representation of the paths as they are provided to us. On macOS and Linux (and other OSes that Swift doesn't officially support yet), paths are bought and sold in the so called "C strings": <code>NUL</code>-terminated bytes (8-bit integers) with POSIX APIs and 16-bit values on Windows with <code>UNICODE</code>. Let's walk over a couple of examples. Here's how to get the current working directory: <pre><code class="language-c">// POSIX char *getcwd(char *buf, size_t size); // Windows // LPTSTR (with right environment) is `wchar_t *` DWORD GetCurrentDirectory( DWORD nBufferLength, LPTSTR lpBuffer ); </code></pre> The expected steps for using APIs like this are the following: <ol> <li>you allocate a buffer large enough to store any result you expect.</li> <li>you give the buffer to the C function.</li> <li>the C function fills the buffer with some characters, and a <code>NUL</code> (or <code>0</code>) right after the last character in the result.</li> <li>the function use a separate variable to tell you the size of the actual result, not counting the <code>NUL</code>.</li> </ol> It's very generous of these functions to give us the size of the result. Because the point of using <code>NUL</code> to terminate "strings" is to avoid having to use a separate size. Let's use setting the current working directory as the next example: <pre><code class="language-c">// POSIX int chdir(const char *path); // Windows BOOL SetCurrentDirectory(LPCTSTR lpPathName); </code></pre> Yup, these APIs don't need you to tell them the content size of your buffer. But if your content doesn't end with a <code>NUL</code>, they'll happily read beyond your intended stopping point until it finds a random <code>0</code> in memory. Anyways, this is pretty classic C stuff. Now let's talk about Swift! The default option to store a file path for most should be a <code>Swift.String</code>, which is a encoding-agnostic, Unicode glyph based list of characters. But as I mentioned earlier, I want to avoid the cost of decoding and encoding in the case where the path is only passing through the Swift code from C to C, without being analyzed or modified. (If you need a refresher, <a href="https://youtu.be/Vy2r21kli0Q">I recently made a video on Unicode and encoding</a>.) So I chose to use an <a href="https://developer.apple.com/documentation/swift/contiguousarray">ContiguousArray</a> to store these C values: <pre><code class="language-swift">// Simplified for POSIX struct Path { let storage: ContiguousArray<CChar> // ... } </code></pre> <code>ContiguousArray</code> (and <code>Array</code>) provides an excellent way to interact with C APIs we encountered earlier: <pre><code class="language-swift">init( unsafeUninitializedCapacity: Int, initializingWith initializer: ( inout UnsafeMutableBufferPointer<Element>, inout Int ) throws -> Void ) rethrows </code></pre> Don't let the complex-looking signature intimidate you. Essentially, it gives you write access to the array's memory right after its allocation, but before Swift does standard initialization to it. It works really well with the C APIs we looked at earlier: <pre><code class="language-swift">// Store the current directory in a ContiguousArray // Using the Windows API let storage = ContiguousArray( unsafeUninitializedCapacity: Int(MAX_PATH) ) { buffer, count let length = GetCurrentWorkingDirectoryW( DWORD(MAX_PATH), buffer.baseAddress // C API writes in the allocated memory ) count = length // you are responsible for setting size of the array } </code></pre> Read the steps 1-4 again from earlier, it's easy to see how this initializer is designed to fit that pattern. The resulting array will have all the characters as its content, and carries the correct size. When it's time to pass the array back to C, we can provide a pointer easily: <pre><code class="language-swift">storage.withUnsafeBufferPointer { SetCurrentDirectory($0.baseAddress!) } </code></pre> This is not great, because we don't have a <code>NUL</code> at the end of our array. The C function that read our array will sometimes read over the contents memory until it finds a 0! Yikes. So here's an easy fix: <pre><code class="language-swift">(storage + [0]).withUnsafeBufferPointer { SetCurrentDirectory($0.baseAddress!) } </code></pre> Instead of using the memory of <code>storage</code>, we construct a new array with an 0 as its last value. This lets C APIs pick the right place to stop reading. (Incidentally, Swift includes a built-in version of this <a href="https://developer.apple.com/documentation/swift/string/2430818-utf8cstring">for converting String to UTF-8 (8-bit) C strings</a>, which includes the <code>NUL</code> and it's possible to further encode with different encodings.) Although we've fixed the correctness bug, doing this defeats the purpose of storing the C string directly somewhat: constructing a new array each time we want to call a C API is kind of expensive. It involves allocating new memories and copying over the content, etc. Alright. How about we carry around the <code>NUL</code> in our array? Let's update the construction code: <pre><code class="language-swift">let storage = ContiguousArray( unsafeUninitializedCapacity: Int(MAX_PATH) + 1 ) { buffer, count let length = GetCurrentWorkingDirectoryW( DWORD(MAX_PATH), buffer.baseAddress ) buffer[length] = 0 count = length + 1 } </code></pre> We add 1 every time we have a say in size. Then we manually set a 0 at the end of the stuff from C. Having done this, we've solved both the correctness problem and performance concern from earlier! The last bit of of this journey is ergonomics. Carrying an extra <code>NUL</code> is fine if you never look at the array's content. But when you do, it's important to remember that the content we care about is almost all of the array, except for the <code>NUL</code> at the end. In other words, simply don't make off-by-1 mistakes and everything will be fine. Alright, that's easier said than done. To alleviate this off-by-1 painfulness, I ended up exposing a "view" into the array storage that excludes the last element. Here's the actual definition in <a href="https://github.com/dduan/Pathos">Pathos</a>: <pre><code class="language-swift">struct CString<Unit: BinaryInteger>: Equatable, Hashable { private var storage: ContiguousArray<Unit> var content: ContiguousArray<Unit>.SubSequence { storage[0 ..< storage.count - 1] } public func c<T>( action: (UnsafePointer<Unit>) throws -> T) throws -> T { try content.withUnsafeBufferPointer { try action($0.baseAddress!) } } init(cString: UnsafePointer<Unit>) { var length = 0 while cString.advanced(by: length).pointee != 0 { length += 1 } storage = ContiguousArray( unsafeUninitializedCapacity: length + 1 ) { buffer, count in for offset in 0 ..< length { buffer[offset] = cString.advanced(by: offset).pointee } buffer[length] = 0 count = length + 1 } } // ... more stuff } </code></pre> <code>storage</code> in this solution is an private implementation detail. <code>content</code> is the primary access to the content of the string. And finally, this type interops with C APIs correctly and efficiently because of the extra <code>NUL</code> we put at the end of <code>storage</code>. Wed, 09 Sep 2020 01:21:26 -0700 https://duan.ca/2020/09/09/free-c-strings/ https://duan.ca/2020/09/09/free-c-strings/ Fall Guys are Among Us I noticed a lot of people are playing these two games on Twitch recently: Fall Guys and Among Us. Personally, I think Fall Guys is a pretty fun game to play and Among Us is an entertaining game to watch. I'm glad these games are out and popular in this strange year, even though I'm not playing either frequently (perhaps that's a mistake). Anyways, I want to write about this observation that seems unique from me: that the in-game characters in these two games are both blob-y, short and colorful. This similarity stuck in my head for a while, and I'm getting increasingly anxious because those who frequently play these games aren't making similar remarks at all. So, in this Sunday evening, I just had to get it out of my head: <video id="video-tag" width="100%" autoplay muted loop playsinline> <source src="/assets/2020/08/FallGuysAmongUs.mov" type="video/quicktime" /> </video> <img id="img-tag" width="100%" src="/assets/2020/08/FallGuysAmongUs.mov"> <script type="text/javascript"> function BrowserDetection() { //Check if browser is Safari, if it is, hide the <video> tag, otherwise hide the <img> tag if (navigator.userAgent.search("Safari") >= 0 && navigator.userAgent.search("Chrome") < 0) { document.getElementById('video-tag').style.display= "none"; } else { document.getElementById('img-tag').style.display= "none"; } } //And run the script. Note that the script tag needs to be run after HTML so where you place it is important. BrowserDetection(); </script> This animation is made from Pixelmator and Keynote. Those two along with Sketch and Final Cut Pro make me feel invincible these days. Sun, 16 Aug 2020 23:42:44 -0700 https://duan.ca/2020/08/16/fall-guys-and-among-us/ https://duan.ca/2020/08/16/fall-guys-and-among-us/ Taking Screenshots On Linux Having used macOS for a number of years, I've formed some habit that I'm never going to shake. Among them is taking screenshots with <code>Alt-Shift-3</code> and <code>Alt-Shift-4</code>, follow up with area or window selection. So, when I bought a PC and put Linux on it, it's time to mess with it for these habits! And, it didn't take too long to solve. I've found <a href="https://github.com/naelstrof/maim">maim</a>. Writing this down for mine and maybe, yours, future reference. To capture the entire screen: <pre><code>bash -c "maim ~/Screenshots/fullscreen_$(date +%s).png" </code></pre> To active area/windows selection for the capture: <pre><code>bash -c "maim -s ~/Screenshots/area_$(date +%s).png" </code></pre> Here's how the latter in action (with keybinds via Ubuntu 20.20 desktop's Settings app). <img src="/assets/2020/06/18/area-screenshot-on-ubuntu.gif" alt="How selecting an area with maim works on Ubuntu" /> Thu, 18 Jun 2020 13:22:01 -0700 https://duan.ca/2020/06/18/taking-linux-screenshots/ https://duan.ca/2020/06/18/taking-linux-screenshots/ StreamLogger After turning off stream on <a href="https://twitch.tv/daniel_duan">Twitch</a>, the first thing I normally do is exporting the stream video to YouTube, so that the stream has an archive that survives Twitch's short-ish retain policy. These videos, perhaps surprisingly, get a few views! It's not a lot by any measures, but I'm conscientious of the fact that a typical stream archive is a multi-hour long video with no content curation, no clear schedule, and it sometimes contains breaks. Needless to say, watching them after-the-fact requires some (or a lot of) fast-forwarding. So, last week <a href="https://twitch.tv/daniel_duan">on stream</a>, I set out to improve the fast-forwarding experience. YouTube has this feature that lets you jump to specific timestamp in the video through a URL parameter in the video' link. Further, they generate this parameter for text in video's description, if the text is in the right format. This is handy for generating a "table of content" for the video so that viewers can click the timestamp in the description to jump to the section they are most interested in. <a href="https://github.com/dduan/StreamLogger">StreamLogger</a> is a little utility that lets me note down what happened while I'm streaming. It's kind of like writing a commit message, except the message describes what happened since the last "commit". These messages, along with their associated timestamps, will be used by StreamLogger to generate the "table of content". Using it in command line looks like this: <pre><code class="language-bash"># Turn on stream, maybe check signs of being live, etc. Then slog start # start a new log # Do stuff, when it comes to a natural conclusion point... slog -- 'I did stuff' # Do more stuff... slog -- 'some other stuff' # Some time later... end stream # No action is required to end the stream as far as StreamLogger is concerned. # Now, to generate the table-of-content slog stamp -s 1:32 </code></pre> That last command outputs <pre><code>0:01:32 I did stuff 1:41:59 some other stuff </code></pre> ... which goes to the video's description. There's a few subtleties in this overall simple tool. Whenever a message is added, it gets associated with the time at which the previous message was add. So the act of logging marks both the end of a chapter and the beginning of the next. In reality, there's always going to be a gap between the start of the stream and the time the log is initialized. That's what the <code>-s 1:32</code> in the last command is trying to correct. It tells StreamLogger the length of the gap. Now the absolute time associated with each event has a relationship with the archive video. <hr /> I chose to write this in Rust because I'm going to need the final product on Linux, macOS, and Windows since I stream on all 3 platforms. (fun fact: I never built it directly on my PC running Linux. Instead, I simply downloaded the musl-based build from the GitHub release, which was built by GitHub Actions. It works beautifully.) Maybe one day I'll add a GUI for it that works across these platforms, too, so that it's more friendly to wider group of users. <hr /> The more I use and think about StreamLogger, the more I like it. You can see me working its entirety in the following archives ;) Part 1: https://youtu.be/xWRcdaEjir4 Part 2: https://youtu.be/RS-ZMBzu9Dg Sat, 13 Jun 2020 12:36:55 -0700 https://duan.ca/2020/06/13/stream-logger/ https://duan.ca/2020/06/13/stream-logger/ Introducing Dye Recently, I got a PC. And I started writing some code on Windows for the giggles. Naturally, I gravitate towards stuff I use on macOS and Linux when it comes to tooling. To my delight, NeoVim, ripgrep and fzf all work out of the box in terminal simulators, which brings us to todays main topic... I made a terminal coloring library for Swift that works on Windows, <a href="https://github.com/dduan/Dye/releases/tag/0.0.1">Dye 0.0.1 is available now</a>! <h2>So, why?</h2> When I started working on <a href="https://github.com/dduan/tre">tre</a>, I search in the Rust ecosystem for a CLI interface library that supports as many platforms as possible. Eventually I found <a href="https://github.com/BurntSushi/termcolor">termcolor</a> among an ocean of options. As a result, tre, like a lot of other CLI tools (like rg) written in Rust, has a consistent UI on Windows and Unix. This experience has brought lots of joy, as a user of both the library, and the app. I want to pay it forward to my fellow Swift CLI makers, and their users. Zooming out slightly, success of Swift on Windows makes Swift as a skill more valuable. And <a href="https://duan.ca/2019/01/20/kick-ass-cli-tools-in-swift/">I want more CLI tools written in Swift</a>. So it's a double-win, really. Lastly, it's a small library, all things considered. Being able to get it to a shippable state on a weekend is a key reason I decided to work on it. <h2>Technical tidbits</h2> I love Max Howell's <a href="https://github.com/mxcl/Chalk">Chalk</a> library. It's a 100-line Swift file that implements <a href="https://en.wikipedia.org/wiki/ANSI_escape_code">ANSI escape code</a> with Swift's custom string interpolation API. It demonstrates well how simple it is to customize your terminal output. Enter Windows, where ANSI sequences are ignored by built-in terminal simulators from the past. The console is customized via a entirely separate, stateful, imperative APIs (Newer simulators such as the freshly released <a href="">Terminal</a> actually supports ANSI codes pretty well). This is our lowest common API denominator, which ultimately dictated the design of Dye. Dye's API is centered around Swift's built-in protocol <code>TextOutputStream</code>. You create a stream object and mutate the style need for upcoming output: <pre><code class="language-swift">let output = OutputStream.standardOutput() output.color.foreground = .blue print("blue text", to &stream) // blue text is blue </code></pre> If the stream is redirected to something other than the terminal, styling gets automatically disabled. There are various options to customize this behavior. Take a look at this <a href="https://github.com/dduan/Dye/blob/master/Examples/main.swift">sample app</a> to get a more concrete picture of how things work. <hr /> I'll end with a screenshot of the sample app running in Command Prompt: <img src="/assets/2020/06/01/windows-example-screenshot.jpg" alt="Dye sample app running in Windows Command Prompt" /> Let's build more. Mon, 01 Jun 2020 15:08:01 -0700 https://duan.ca/2020/06/01/dye/ https://duan.ca/2020/06/01/dye/ Naive NixOS Rust Development tl;dr: To work on Rust project with nix-shell, rls and extensions such as <code>rust-analysis</code>, <code>rust-src</code>, without caring too much about specific Rust toolchain version (except for it being <code>stable</code>), use the following <code>shell.nix</code>: <pre><code class="language-nix">let moz_overlay = import (builtins.fetchTarball https://github.com/mozilla/nixpkgs-mozilla/archive/master.tar.gz); nixpkgs = import <nixpkgs> { overlays = [ moz_overlay ]; }; ruststable = (nixpkgs.latest.rustChannels.stable.rust.override { extensions = [ "rust-src" "rust-analysis" ];} ); in with nixpkgs; stdenv.mkDerivation { name = "rust"; buildInputs = [ rustup ruststable ]; } </code></pre> When you have a Nix hammer, everything looks like a Nix expression. Having used NixOS on a real PC for a number of days, this is the impression I get from the world of Nix. Unfortunately, so far, it's been a negative for me. One of the most exciting thing I want to use Nix for is to bootstrap development environment with <code>nix-shell</code>. I imagined it to be similar to using <a href="https://pipenv-fork.readthedocs.io/en/latest/">pipenv</a> with Python, except for everything. Well, I've since learned that it's not true (yet?) for many reasons. Modern programming languages come with their own attempt at reproducibility. Some does it better than others. To make it concrete, I'm talking about things like <a href="https://haskellstack.org">Stack</a> for Haskell or <a href="https://rustup.rs/">rustup</a> for Rust: given the source code, how do I make it build in the way the project intended? What's the correct version of the compiler, runtime, and tools that works best with this revision of the source code? The common solution usually follows this pattern: as author of a project, specify as much as you can, the environment best suited for the current state of the project. As a "builder", use a single program that's capable of updating itself, as well as ensuring that the project builds exactly as specified, including managing the compiler/runtime/tooling versions, etc. This single program's role is very much the same as the Nix system, except the latter is independent of programming languages: <code>rustup</code> installs Rust, so does Nix. That's a bad thing. As a package manager, Nix either have to tightly integrate with each of these other package managers, leveraging their evolving behaviors to give its user the build environment; or, it must replace them. The former is impractical; the latter, well, sucks. Back to reality. This is the experience I want to have with NixOS: Some programs I use daily such as Alacritty, NeoVim, Firfox, etc, are installed globally and readily available. They are part of my <code>/etc/nixos/configuration.nix</code>. So far so good. Now, I regularly program in a few languages. For each of the project, I'd like to have a <code>shell.nix</code> that brings in its compilers, libraries, LSP servers, etc. This is what <code>nix-shell</code> is supposed to give me! This is known as the "per project" setup. Let's see: with Rust, that means <code>rustc</code> (compiler), <code>cargo</code> (package manager), <code>rls</code>, <code>rust-src</code> and <code>rust-analysis</code> (LSP). In macOS, I'd install all of these globally with <code>rustup</code>. In NixOS...well, I can ask for <code>rustup</code> for my project: <pre><code class="language-nix">with import <nixpkgs> {}; stdenv.mkDerivation { name = "rust"; nativeBuildInputs = [ rustup ]; } </code></pre> ...which gives me <code>rustup</code> and nothing else. That's right, you don't even get a <code>rustc</code> after running <code>nix-shell</code>. But <code>rustup</code> can get you everything else, all I need to do is ask. Hmm, do I need to run a series of set-up commands with <code>rustup</code> every time I enter the environment? No? I just need to run it the first time? Until the cached tools get deleted by some garbage collection mechanism? That seems unsatisfying, doesn't it? Instead of <code>rustup</code>, I could also ask Nix to use the <code>rustc</code>/<code>cargo</code>/<code>rls</code> it knows about directly. This is marginally better. Except I still need <code>rust-src</code> and <code>rust-analysis</code> for my needs. As far as I can tell, these <a href="https://github.com/rust-lang/rls">RLS</a> components are out of Nix's control (as of today). Everywhere on the internet I looked, for every problem that Nix-the-package-manager doesn't work out-of-the-box, there's someone responding along the line of "you can write some Nix expression yourself". In other words, Nix-the-language is powerful enough to solve it, probably. In the case of Rust, luckily, Mozilla wrote enough Nix expressions for us and provides them via an <a href="https://github.com/mozilla/nixpkgs-mozilla">overlay</a>. These expressions are rich enough to meet my needs. As you can see in the tl;dr at the top, when entering the development environment, nix-shell would: download the overlay's source code (from the internet, or local cache), load the expression it includes, mix in my customization, and execute it. That marks the end of my search. I like the final solution because it's mostly "vanilla" Nix and doesn't require me to mess with a bunch of other tools. For solutions that do, read <a href="https://xeiaso.net/blog/how-i-start-nix-2020-03-08/">this</a>. <hr /> At end of the day, my needs are pretty basic: consistency from rustup and convenience from nix-shell. I didn't need to pin the compiler to a specific Rust release, or checksum the final build output. I'm very new to both technologies so there may be a follow-up post sometime in the future. Thu, 07 May 2020 11:04:58 -0700 https://duan.ca/2020/05/07/nix-rust-development/ https://duan.ca/2020/05/07/nix-rust-development/ Site Improvements 2020 I took back my website. I took it back from the claws of Jekyll and Ruby. I took it back from some random template among a few that were immediately available. I took it back from my own ignorance of modern web technology. This time, I rewrote the hole effing thing from scratch. <h3>The Why</h3> The <a href="/2017/01/16/site-changes/">last time</a> I ended with <blockquote> I wrote the most Ruby in my life today. Yay? </blockquote> That question mark turned out to be prescient. Ruby is not my thing. No judgement against the language per ce. But the Ruby ecosystem is not friendly to a casual user who needs it once every few months. No really, when I write an article, it's a toss-up whether I can deploy it without being forced to mess with Ruby/Gems/Jekyll/Homebrew etc. I'm almost certain there's a set of best practices I could learn to improve this. But it'd be a skill that I barely need and probably forget a few times. Meanwhile I just want to translate a new Markdown file to HTML and put it on Github. As a teen, one of my favorite things about the web is how accessible it is. I could sit in front of anyone's computer, open Notepad.exe and type in some tags, and open the file with a browser to see the result. That simple bootstrap process, however repetitive, never got old for me. In the last decade, my professional work is focused on native, mobile applications. This experience biased me in a few ways. "Native" made me appreciate the closeness to "the metal": you have an OS; you get the executable; you launch strace, and boom, everything the OS thinks what your code does is revealed to you. "Mobile" forced me to reckon with the reality: desktop experience has become a niche. It's nowhere near as important as it was prior to the iPhone. Not making your software run well on mobile devices is a particular kind of choice that come with some severe trade-offs. And, finally, I subscribe to the idea that plain text is supreme. Yes, even more supreme than the web. This website is a derivative of the articles I write in Markdown. Plain text as a format will out-last the web in the long run (not necessarily these texts). So it really bothers me when I have to put "front matter" in YAML/TOML/whatever in front of the real Markdown. Yes, if you want reasonable HTMLs, these metadata is necessary. But the text is Supreme. The text is to be readable directly. Succumbing the supreme to the derivative is wrong. Okay, so where does all that leave me? In this iteration of this website, I wrote every single line of CSS and HTML (there's barely any JavaScript) which look decent on mobile. It's generated by portable Linux and Darwin executables that are part of the website. As long as the OSes stay relatively stable, the site will build without any dependencies, in a matter of milliseconds. The best part? It'll stay like this unless I wish it otherwise. <h3>The Frontend</h3> A few words on the design of the site. This site gained the concept of "links" the <a href="/2017/01/16/site-changes/">last time</a>. It has since become clear that I don't use this feature (blame Twitter). It's gone, for now. The old "about" page is replaced by the home page. When I decided the site needed a rewrite, I fantasized a place with only HTML. Perhaps users who want a better reading experience can simply put it in Reader mode. Alas, if browsers (with the exception of Safari) implemented automatic dark mode with a <a href="https://drafts.csswg.org/css-color-adjust-1/#color-scheme-meta">color-scheme meta</a> tag, it'd almost be a working idea. So, that's my starting point. The site is a list of articles organized by tags and dates, and a few web pages. I want you to read the site, not navigate it. So text is the point. It's the only design element. The site uses 2 fonts and 2 text-color (not counting highlighted code). Links are always underscored (because you can't hover in mobile browsers to find something clickable). The site is responsive to mobile layout, and dark mode. It's aware that it could be added to the home screen of a mobile device, or linked to some external site that wants to generate a preview. I did end up using <a href="https://ethanschoonover.com/solarized/">solarized</a> theme for code highlighting, which lead to 2 static CSS files. Other than that, all CSS and HTML are hand-written. There's no build step for them, farm to table, Vim to your browser. It turned out standardized CSS variable is game-changing. Combined with media queries, I barely needed any class to support dark mode/mobile layout. More importantly, it makes my programmer brain happy. Oh, yes I'm talking about them here because this is the first time I truly attempted to catch up since they were introduced to the world. It's freeing to let go of constrains of an read-to-use theme, or some CSS frameworks. <h3>The Backend</h3> This site is a collection of static files. The so-called backend is a program that assembles these files from some HTML templates and Markdown files. In the past, this program had been <a href="https://jekyllrb.com/">Jekyll</a>. This time around, I replaced it with some Rust code. Boy, this thing is cool, if I say so myself. I'm going to refer to it as "the generator". The biggest "feature" is the fact that it doesn't pretend to be re-usable. The generator is as part of this site as the articles. Thanks to Rust, the programs are built into executable binaries. On macOS, it requires libSystem to run. On Linux, I can (and prefer) build with <a href="https://www.musl-libc.org/">musl</a>. The binaries for these two OSes are checked in with the content of the site. So it requires zero installation to "build" the site. (I may need to include a 3rd executable for the upcoming ARM-based Macs soon). The generator spits out the final content of the website. It's deployed without further modification. The build process is pretty fast. As of this writing, it averages around 250 ms. I could probably make it faster by avoiding some repeated reads when it comes to article inputs. The generator handles HTML/XML templating with a library named <a href="https://github.com/djc/askama">Askama</a>. Learning it had been an eye opening experience. Askama is built atop Rust's macro system. For each template (e.g. web page), it requires users to write a Rust data structure that fulfills its variable requirements. Here's the kicker: in this data structure, you cannot miss any variable the corresponding template requires. When you do, the Rust project won't compile! Rust's tooling is so good that these errors were surfaced in my editor as I wrote this part in real time. This level of type-safety for template language felt like magic. Syntax highlighting is powered by <a href="https://github.com/trishume/syntect">syntect</a>, the library behind <a href="https://github.com/sharkdp/bat">bat</a>. But the interesting bit here is how the syntax definitions are embedded within the final executable. To support a particular syntax, syntect takes the syntax's SublimeText definition. So this is a configurable, extendible system. The generator includes 118 syntax definitions. Uncompressed, their files take up 5MB of disk space. As one can imagine, loading 5MB from 118 files each time the program runs is quite slow. Turns out, Rust has a standard library macro <a href="https://doc.rust-lang.org/std/macro.include_bytes.html">include_bytes!</a> that solves this problem. <a href="https://doc.rust-lang.org/std/macro.include_bytes.html">include_bytes!</a> embeds contents of a file as literals in source code, as if it's hand-written. syntect takes advantage of this feature by supporting serialization of its in-memory representation of syntaxes into bytes, and accepting in-line byte literals in reverse in order to create these representations. This system solves two problems for me: <ol> <li>I no longer need to ship the SublimeText syntax files along with the generator.</li> <li>The generator doesn't need to perform all that disk I/O, so it runs significantly faster.</li> </ol> Overall, the Rust ecosystem delivered. Although a lot of library are still in rough shape/early stage, there usually are multiple alternatives for roughly the same purpose. Gluing a few of them together for this generator project had been fun. Finally, let's talk metadata, I can't realistically manage to generate the site without them. Articles and static pages each define their URL by their file locations relative to the root. For example, <code>/articles/2020/04/23/hello.md</code> means the URL is <code>/2020/04/23/hello/</code>. Each article still has a front matter. But I made the text version look as "natural" as possible. There's no markers for beginning and end of the metadata since I know exactly what's needed. The title is marked as an H1. Date is in RFC3339 format. Tags are comma-separated values. So an example of an article's beginning looks like: <pre><code class="language-markdown"># Site Improvements 2020 2020-04-22T21:58:03-07:00 tags: Rust, HTML, CSS, Ruby, Jekyll, Markdown I took back my website... </code></pre> Text supremacy! Glory! <h3>Onward</h3> Over the course of roughly 15 years, I've had blogs of quite a few variates. I wish I'd done a better job archiving them as I moved on from one to the next. This is not the first time I attempted to gain full control over the theming, generating, hosting, and deployment of a site. But I'm hopeful that it'll last longer. You could say that's what I optimized for. And who knew, maybe my experience from past failures counts for something. Here's to less stressful deploys and more writing! Wed, 22 Apr 2020 21:58:03 -0700 https://duan.ca/2020/04/22/site-improvements-2020/ https://duan.ca/2020/04/22/site-improvements-2020/ Git Commit Message, Vim, and Markdown It's been bothering me for years. That is, <code>#</code> is both the start of a comment in a git commit message, but also the syntax for headings in Markdown. Personally, I prefer using commit messages to populate pull request descriptions whenever possible. On GitHub, This happens automatically when the pull request contains a single commit. But I can't type, say an H3 as <code>### Heading</code> in the commit message (in Vim, most of the time) because it gets treated as a comment! But thanks to this twitter interaction with <a href="https://twitter.com/dmartincy">@dmartincy</a>, I finally solved this problem: <blockquote class="twitter-tweet">I haven't found myself in that situation, but maybe you could do 'git config core.commentChar ";"' before creating the commits? That will change the default git commit marker (#) to something that won't conflict with Markdown titles.— Daniel Martín (@dmartincy) <a href="https://twitter.com/dmartincy/status/1247271508420026368?ref_src=twsrc%5Etfw">April 6, 2020</a></blockquote> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script> As mentioned by Daniel, Git has introduced a setting called <code>core.commentChar</code>, documented <a href="https://git-scm.com/docs/git-config#Documentation/git-config.txt-corecommentChar">here</a>, which lets us control which character becomes the start of a comment line. Let's say we want to replace the default <code>#</code> with <code>;</code>, we can edit <code>~/.gitconfig</code> to include this preference: <pre><code>[core] commentChar = ";" </code></pre> … and this will affect every Git repository on this computer. For me, though, this broke git commit syntax highlighting in Vim: the comments, beginning with <code>;</code>, are no longer recognized as comments. To fix this, I updated Vim's syntax for file type <code>gitcommit</code>. In your Vim setting directory (<code>~/.config/nvim/</code> for me), create a file <code>syntax/gitcommit.vim</code> file (unless you already have one), add the following line: <pre><code>syn match gitcommitComment "^;.*" </code></pre> Note <code>;</code> matches my preferred <code>core.commentChar</code> for Git. Et voilà! Git commit message looks tippy-top in Vim again! <img src="/assets/2020/04/gitcommit.png" alt="editing git commit in vim, with alternative character being the beginning of a comment" /> Mon, 13 Apr 2020 13:41:02 -0700 https://duan.ca/2020/04/13/git-commit-comment/ https://duan.ca/2020/04/13/git-commit-comment/ On the Subject of Interface Hygiene In a purly reactive world, your entire program merge into a single stream. Now, close your eyes, and envision: your project as one, beautiful, stream. Now open your eyes. Yeah, it's not. Your project is a Mac or iOS app. It's full of your memories, sweat, and blood. And you are ready to sweat and bleed some more by putting some Combine and SwiftUI into it. You watched the WWDC19 sessions and learned that "Subjcets are super powerful". You looked into your code and realized you can't really do anything with Combine without <code>Subject</code>s at the current state of the project. Well… Here are a few habits that help keeping your project that prevasively uses <code>Combine.Subject</code> sane. They should seem obvious to anyone who understands Murphy's law and the value of minialism in interfaces. If you already are using some reactive stream implementation, substitute the types with their counterparts in your framework and these rules should seem down right basic. <h2>Vend <code>Subject</code>s as <code>Publisher</code>s</h2> <code>Subject</code>s help bridge from the imperitive to the reactive world. Somewhat paradoxically, sharing them is not very "RX-y". This is akin to prefering <code>let</code>s over <code>var</code>. Most of the time, what you want to share is the values pumped into the stream, not the privilage to mutate it. Because Subjects conform to <code>Publisher</code>, it's easy to hide from the users the fact that your stream is backed by them. <pre><code class="language-swift">// Bad: now anyone who get a hold of it can mess with your stream! public enum GreatInts { public var updates = CurrentValueSubject<Int, Never>(0) } </code></pre> With Combine this conversion happens via type-erasure: <pre><code class="language-swift">// Better: all your users care is the stream (publisher), so give them that! public enum GreatInts { // Internally, it's backed by a subject. var subject = CurrentValueSubject<Int, Never>(0) // Externally, it's just a Publisher. public var subject: AnyPublisher<Int, Never> { subject.eraseToAnyPublisher() } } </code></pre> <h2>CurrentValueSubject natually caches the latest value</h2> RX theorists will hate this: sometimes it's just practical to expose a synchronous interface to the latest vaule in the stream! Two things. <ol> <li>It might be tempting to expose the subject and let your user use its <code>.value</code>. Well, you shouldn't (as explained in the previous section). A separate interface dedicated to the latest value prevents people from polluting your stream.</li> </ol> <pre><code class="language-swift">// (Still) bad public final class GreatInts { public var updates = CurrentValueSubject<Int, Never>(0) } </code></pre> <ol start="2"> <li>Remember <code>CurrentValueSubject</code> has that <code>.value</code> property! It may seem surprising, but I've seen folks transitioning to RX clinging to the old ways:</li> </ol> <pre><code class="language-swift">public final class GreatInts { // well, at least it's not a public subject... var subject = CurrentValueSubject<Int, Never>(0) // <- initial value 0 public var updates: AnyPublisher<Int, Never> { subject.eraseToAnyPublisher() } // Wait, there's that 0 again public var latest: Int = 0 { didSet { subject.send(latest) // ? } } } </code></pre> First, you'll notice that <code>0</code>, the initial value, is duplicated as both the subject's initial value, as well as the value of a stored property. And these duplicated sources of truth persist throughout the parent's life time. Weird, right? Here's a slightly better version: <pre><code class="language-swift">public final class GreatInts { var subject = PassthroughSubject<Int, Never>() public var updates: AnyPublisher<Int, Never> { subject.eraseToAnyPublisher() } public var latest: Int = 0 { didSet { subject.send(latest) // ? } } } </code></pre> Now there's no two copy of the latest value in memory anymore. But in my opinion it does not embrace the full power of Combine. Here's the most natual way to do this: <pre><code class="language-swift">public final class GreatInts { /// This is a CurrentValueSubject again. var subject = CurrentValueSubject<Int, Never>(0) public var updates: AnyPublisher<Int, Never> { subject.eraseToAnyPublisher() } public var latest: Int { subject.value } } </code></pre> Essentially, you create separate public interface, each vends a little piece of <code>CurrentValueSubject</code>'s power. <h2>No really, don't use <code>Subject</code>s</h2> Even a well-scoped <code>Subject</code> (properly being private or internal, depending on your tool of choice for access control) still has a mutable state that you probably don't want: its stream can go from "alive" to "complete". And, again, anyone with access can make this state transition happen, leaving you in the undefensible position of … hoping everyone on your team to not misuse your stuff? Lucky for you (and me), a "incompletable" subject is a established concept -- a "Relay". I've put together a repo for you to look and/or use: <a href="https://github.com/dduan/relay">https://github.com/dduan/relay</a> Yeah, ban all Subjects in your project with a linter. Seriously. <h2>Fin</h2> That's all for now. I'm not an expert with RX myself. Hopefully these perspective can help you avoid some nasties. Mon, 01 Jul 2019 20:22:48 -0700 https://duan.ca/2019/07/01/combine-hygiene/ https://duan.ca/2019/07/01/combine-hygiene/ TOML Decoder Playlist I enjoyed making <a href="https://github.com/dduan/TOMLDeserializer">TOMLDeserializer</a> and <a href="https://github.com/dduan/TOMLDecoder">TOMLDecoder</a> recently. Let's hope some projects will pick them up and TOML as a configuration format will start to spead in Swift communities. What's outstanding about these projects is that I started working on them while <a href="https://twitch.tv/daniel_duan">streaming</a>. Personally, I consume a lot of content like this. So now people can watch me writing Swift, too. I've been archiving recordings of these streams as much as I can. Here are the links to these videos, each is around 1-1.5 hours long: <ul> <li><a href="https://youtu.be/XLLlCEfuFLw">Turn TOML Test Fixtures Into XCTests With Vim Macros</a></li> <li><a href="https://youtu.be/Ztq3K3cq8f0">Passing TOML Deserialization Tests</a></li> <li><a href="https://youtu.be/GTCYSIc6n3U">Finishing TOML Invalidation Tests</a></li> <li><a href="https://youtu.be/HP2Wzh8MVvg">TOML Decoder: The Beginning</a></li> <li><a href="https://youtu.be/EZ7VpbrLkH0">TOML Decoder: Filling In Missing Implementation Pieces</a></li> <li><a href="https://youtu.be/bISMVVWOHKg">TOML Decoder: Adding The First TOML Decoding Strategy!</a></li> <li><a href="https://youtu.be/oOOS4bnhwx4">TOML Decoder: Project Overview And Data Decoding Support</a></li> <li><a href="https://youtu.be/9yDtXsrMtbU">TOML Decoder: Key Decoding Strategies</a></li> <li><a href="https://youtu.be/g1V872HL4Dk">TOML Decoder: Project Updates</a></li> </ul> The TOMLDecoder projects would've been capture on camera entirely if I weren't such a streaming n00b and messed up a few streams. Hilarious. Sun, 10 Mar 2019 20:32:37 -0700 https://duan.ca/2019/03/10/toml-decoder/ https://duan.ca/2019/03/10/toml-decoder/ Kick-ass CLI Tools In Swift As someone who lives in a terminal simulator, I'm pleasantly surprised by the new toys we get in recent years such as <a href="https://github.com/junegunn/fzf">fzf</a>, <a href="https://github.com/BurntSushi/ripgrep">ripgrep</a>, <a href="https://github.com/sharkdp/fd">fd</a>, etc. A great number of these are written in relatively young programming languages such as Go and Rust. But, noticibly, none of them are written in Swift. In this post, I'll try to explain why that is. <h2>POSIX Ergonomics</h2> Unix-like virtual file systems has been around for decades. API that manupulates such systems has standardized a long time ago and exists in most computers running Linux/BSD/macOS today (and, to a large extend, smart phones). To Swift users, Using these APIs is straight-forward (<code>rmdir("path/to/dir")</code>). So Swift programmers are all happy campers (re-)inventeing all sorts of file system utilities, right? Well, not quite. Okay, I lied about POSIX APIs being "straight-forward" in Swift. Or rather, this is very subjective. Continuing with the <code>rmdir</code> example, we must first import it from either <code>Glibc</code> or <code>Darwin</code>, depending on your OS. To know whether the operation is successful, we need to see whether it returned integer 0. To learn why 0 was not returned, we need to read the "magical" variable <code>errno</code>. <code>errno</code> could be written to by other APIs so we'd better capture it in time… And that's one of the simpler APIs in POSIX calls! Programmers whine about ergonomics partially because we are previlidged and spoiled. But mostly because our attention is a limited resources. Mixing API conventions distracts us from solving the problem at hand. Bad ergonomics, therefore, drives away a good potion of users who cares about quality of their tools. <h2>Culture and History</h2> As of this writing, the release of Swift 5 is imminent. The vast majority of existing Swift code is written to run on iOS. The concept of a file, or the traditional virtal file system, is hidden to iOS users, and sandboxed for developers. I bet most Swift users rarely think about the fact that there's a entire set of POSIX API at their disposal. <code>Foundation</code> alleviates the need to deal with files and directories: <code>Bundle</code> locates the files; <code>CoreData</code>, <code>UserDefaults</code> or the keychain is your primary way to persist data; <code>Data</code>, <code>String</code> or <code>NSCoding</code> has methods to read and write to files. And finally, if you really need to deal with files, <code>NSFileManager</code> has everything you'll ever need. Why would a productive Swift programmer think about POSIX in this environment? Why would a tutor teach POSIX over the useful/practical/"native" alternatives? We can trace "riding on the Apple platform" mentality back to the pre-iPhone days, where a very small Mac developer community labors on on a niche platform (compared to iOS today) and they loved it. However, I'm sure they used more POSIX stuff back then than the average iOS developers today. Having a great library such as Foundation on the most popular developer platform where the language thrives means it'll take longer for "subcultures" to emerge, if they do at all. <h2>The Standard Library And Its Influence on New Users</h2> File system APIs being in <code>Foundation</code> as opposed to the standard library is probably a temporary condition. Nevertheless, it has at least the following implications: <ol> <li> Its quality of implementation is not held on the same standard that those APIs in the standard library. This is especially true for the separate, open-source <code>Foundation</code> implementation. Getting consistent and correct behaviors across macOS and Linux is hard. </li> <li> A person learning Swift won't explore the language with a file system API. This I suspect, is the most important reason many of these great CLI utilites are written in other programming languages. Programmers seek instant gratification when they learn. And they usually stay in a limited domain (like iOS) at first. This is where the built-in library is special: no matter which domain is chosen, it's always available. Languages such as Go and Rust include things like paths and files in their built-in library. Playing with these APIs while learning the lanugage plants a seed for future, serious, projects. There are less users of these languages compared to Swift, but there are more people thinking about projects that involves file systems in thoes communities. (Note I don't have statistics here, just a guess.) </li> </ol> <h2>Conclusion</h2> The next killer CLI tool is still more likely to be written in Go or Rust, than in Swift. Hopefully, somewhere in these speculations is a true cause of this phenomena. Maybe someone reading this will be inspired to accelerate change that will eventually revert the condition. (I'm <a href="https://github.com/dduan/Pathos">trying</a>). Sun, 20 Jan 2019 16:09:38 -0800 https://duan.ca/2019/01/20/kick-ass-cli-tools-in-swift/ https://duan.ca/2019/01/20/kick-ass-cli-tools-in-swift/ Magical Interactions I want to talk about this little function: <pre><code class="language-swift">func interact(_ process: (String) -> String) { var input = "" while let line = readLine() { input += line } print(process(input)) } </code></pre> Brief explanation: it reads all the input from stdin as a <code>String</code>, feeds it into a closure <code>process</code>, which it takes in as its only argument, and prints <code>process</code>'s output. Here's how one might use it: <pre><code class="language-swift">// counts characters from stdin and prints result to stdout interact { String($0.count) } </code></pre> Got that? Well, now I'm going to rewrite it in a slightly less Swift-y way: <pre><code class="language-swift">let count: (String) -> String = { return String($0.count) } interact(count) </code></pre> The argument for <code>interact</code> got defined with a name and an explicit type signature. So, what's so special about this <code>interact</code> function? Two words: side effects. More precisely, it took away the concern of side-effects from the user. <code>count</code> belongs in the realm of pure functions. It has no worries about file handles or operating systems. It's <code>(String) -> String</code>. I wanted to emphasize this with the rewrite. Look at that empty line. Now you see a boundary between 2 worlds. This may all seem contrived. But when I learned about <a href="http://hackage.haskell.org/package/base-4.11.1.0/docs/Prelude.html#v:interact">this function in Haskell</a>, I was blown away. It's like a great magic trick: you are presented a scenario, say, writing a little script. Maybe you need to process some CLI output and print out a CSV or JSON string (literally 90% of the script I write). A Haskell programmer would jump into the bottom level of the problem and start writing these little pure functions: one to split the string, one to convert some numbers, one to manipulate a list, one to match some patterns...gradually the broken-down absractions get built back up via function composition. You can see the light at the end of the tunnel, yes, yes! If you feed this list into that function that returns a string you'll have the right value to print out! Okay, now the problem is solved in the pure functional world! The only thing left to do is... Now, the setup of the magic is complete. Now, you are onboard with the solution, you thought the problem through with the magician...you are distracted. The ending came so...quickly, but unexpected. What? You just feed your solution to this <code>interact</code> function and...that's it? I was expecting some <code>readLine</code>s or <code>print</code>s (okay, at least 1 <code>print</code> statement)! That's the thing: <code>interact</code> deals with two side effects, the input and the output. But its user deals with zero. It's as if the two effects "cancel" each other out! It's a neat trick, really. Small, low-key, easy to miss. But I'm glad I noticed it and come to appreciate its power and simplicity. Sat, 01 Sep 2018 11:08:57 -0700 https://duan.ca/2018/09/01/magical-interaction/ https://duan.ca/2018/09/01/magical-interaction/ A Tale of Two Dates Recently, I discovered a curious thing about <code>Date</code>s in two large projects I work on. Simply put, both projects receives, from various HTTP endpoints, the same object component: a timestamp, and a duration. Combining these two pieces, both projects derives two <code>Foundation.Date</code>s to represent a time range. So far, so good. However, project <code>A</code> uses <code>Fonudation.DateInterval</code> to represent this concept, while project <code>B</code> uses <code>Range<Date></code>. But why? Why represent the same component differently? What a gigantic waste of brain power for everyone on both projects! So I set out to unify this thing. Wherever a <code>Range</code> literal is used, I swap in <code>DateInterval.init(start:end:)</code>; <code>Range.lowerBound</code> becomes <code>DateInterval.start</code>; <code>Range.upperBound</code> becomes <code>DateInterval.end</code>, etc. It didn't take long to complete the conversion to <code>DateInterval</code> in project <code>B</code>, now it builds and runs! Oh, wait, why are some tests failing in project <code>B</code>? Shouldn't this just be an mechanical change? I spent time investigating. The failing tests are for some very specific business logic that I'm not familiar with. So things took a while to become clear. What felt like a long time later, I realized my mistake. (I'm sorry if this has been obvious to you. You are a better Swift programmer!) Somewhere in project <code>B</code> is the following: <pre><code class="language-swift">struct Item { let range: Range<Date> } struct Container { let items: [Item] } func containerFactory(range: Range<Date>, items: [Item]) -> Container { /// pretend there's more code here return Conatiner(items: items.filter { $0.range.contains(range.lowerBound) }) } </code></pre> And of course, after my "refactor", it became <pre><code class="language-swift">struct Item { let range: DateInterval } struct Container { let items: [Item] } func containerFactory(range: DateInterval, items: [Item]) -> Container { /// pretend there's more code here return Conatiner(items: items.filter { $0.range.contains(range.start) }) } </code></pre> Tests for <code>containerFactory</code> failed. And here's why: <code>DateInterval.contains</code> is inclusive for its upper bound (<code>.end</code>), whereas <code>Range.contains</code> isn't! You can see it plainly by running the following <pre><code class="language-swift">import Foundation let sooner = Date(timeIntervalSince1970: 0) let later = sooner.addingTimeInterval(1000) DateInterval(start: sooner, end: later).contains(later) // true (sooner..<later).contains(later) // false </code></pre> So here's what stumped me: <ol> <li> The 2 projects chose to interpret the same component differently, which I didn't not expect. </li> <li> I didn't know how <code>Foundation.DateInterval</code> works. </li> </ol> Well, today I learned. Sat, 25 Aug 2018 17:11:43 -0700 https://duan.ca/2018/08/25/a-tale-of-two-dates/ https://duan.ca/2018/08/25/a-tale-of-two-dates/ Supporting Foundation.Data Without Depending On It While implementing some file I/O APIs in <a href="https://github.com/dduan/Pathos">Pathos</a>, I decided reading/writing file content as <code>Foundation.Data</code> is kind of important (can you blame me?). But Pathos, by accident, does not depend on Swift <code>Foundation</code>. Now what? After browsing the <a href="https://developer.apple.com/documentation/foundation/data">documentation</a>, a pretty good solution emerged: <code>Data</code> is a sequence of bytes! Lets say we hand our users some bytes, they can easily construct a <code>Data</code> from it: <pre><code class="language-swift">let content: [UInt8] = try readBytes(fromPath "/tmp/test") Data(bytes: content) </code></pre> Okay, so this built-in initializer makes <code>[UInt8]</code> an acceptable substitute for returning <code>Data</code>. What can we do about about <code>Data</code> as input? Well, turns out, <code>Data</code> is a <code>Collection</code> of <code>UInt8</code>s! So we can accept <code>Data</code> indirectly like so: <pre><code class="language-swift">func write<Bytes>(_ bytes: Bytes) where Bytes: Collection, Bytes.Element == UInt8 { // blah } </code></pre> User can pass in a <code>Data</code> as argument and it just works™. The only disadvantage of supporting <code>Data</code> in these ways is that it requires your user to discover it either via your excellent documentation, or through their super good knowledge of <code>Foundation</code>. Update: this could also <a href="https://mastodon.social/@helge/100573358160444340">be</a> <a href="https://twitter.com/daniel_dunbar/status/1030938273047179264">slower</a> than using Data directly. Luckily I'm only doing file I/O here. But this is pretty nice, regardless. Sat, 18 Aug 2018 11:00:41 -0700 https://duan.ca/2018/08/18/supporting-data-without-depending-on-it/ https://duan.ca/2018/08/18/supporting-data-without-depending-on-it/ Unix File Permissions In Swift Wrap file permissions from <code>stat</code> and <code>chmod</code> in an <code>OptionSet</code> in Swift. <div class="video-container"> <iframe src="https://www.youtube.com/embed/1ZrouR_F-AU" frameborder="0" gesture="media" allow="encrypted-media" allowfullscreen></iframe> </div> Sun, 06 May 2018 16:43:20 -0700 https://duan.ca/2018/05/06/unix-file-permissions-in-swift/ https://duan.ca/2018/05/06/unix-file-permissions-in-swift/ An Interpreter Without A Language From the <a href="/2018/01/13/coding-without-a-language/">AST</a> and <a href="/2018/01/20/scopes/">scopes</a>, come a simple interpreter. This intepreter can "execute" the fibonacci function AST and produce a correct results. <div class="video-container"> <iframe src="https://www.youtube.com/embed/C16PD44qyOY" frameborder="0" gesture="media" allow="encrypted-media" allowfullscreen=""></iframe> </div> Tue, 30 Jan 2018 12:53:38 -0800 https://duan.ca/2018/01/30/an-interpreter-without-a-language/ https://duan.ca/2018/01/30/an-interpreter-without-a-language/ Scopes Without A Language A short video on the concept of static scope. <div class="video-container"> <iframe src="https://www.youtube.com/embed/o0jwyd8nUK4" frameborder="0" gesture="media" allow="encrypted-media" allowfullscreen></iframe> </div> Sat, 20 Jan 2018 19:50:24 -0800 https://duan.ca/2018/01/20/scopes/ https://duan.ca/2018/01/20/scopes/ Coding Without A Language <div class="video-container"> <iframe src="https://www.youtube.com/embed/OKjQtn_Gvug" frameborder="0" gesture="media" allow="encrypted-media" allowfullscreen=""></iframe> </div> Sorry for the click-baity title. I actually really like this one. Sat, 13 Jan 2018 12:07:19 -0800 https://duan.ca/2018/01/13/coding-without-a-language/ https://duan.ca/2018/01/13/coding-without-a-language/ Scripting A 'cat' Under 1 Minute This video demonstrates a few things: the <code>readLine</code> standard library function, scripting with Swift, and pattern matching as loop conditions. Enjoy! <div class="video-container"> <iframe src="https://www.youtube.com/embed/8rAqzvChbak" frameborder="0" gesture="media" allow="encrypted-media" allowfullscreen></iframe> </div> Sat, 06 Jan 2018 18:34:39 -0800 https://duan.ca/2018/01/06/cat-under-1-minute/ https://duan.ca/2018/01/06/cat-under-1-minute/ Raw Coding Sessions Planning, recording and editing a video is something I find very enjoyable to do. However, I enjoy just working on stuff as well. So I'll occassionally post videos of me doing just that: writing code, mumbling, with all my verbal ticks and bad habits left unedited. Not sure who these videos are for but why the heck not, amiright? The first one was scheduled to publish today, so I guess I should post something about it. <iframe width="560" height="315" src="https://www.youtube.com/embed/videoseries?list=PLuQ4I4VwboBw58RmXDbPGOUh4FQxyZ4UA" frameborder="0" gesture="media" allow="encrypted-media" allowfullscreen></iframe> Sat, 06 Jan 2018 14:44:13 -0800 https://duan.ca/2018/01/06/raw-coding-session-videos/ https://duan.ca/2018/01/06/raw-coding-session-videos/ Hello World In WebAssembly Every now and then, I check on the progress of Web Assembly. I did it again around the time of this post and finally found enough tutorials, examples, and working software to get myself started in this area. In doing so, I made a video to demo some progress. (this article includes all the same information and more, so just read on if you don't have 15 minutes for YouTube). <div class="video-container"> <iframe src="https://www.youtube.com/embed/yEYtwmI7bDg" frameborder="0" gesture="media" allow="encrypted-media" allowfullscreen></iframe> </div> <h2>Our goal:</h2> <ol> <li>Use as much built-in tools on a Mac as possible. The web development toolchain scares me.</li> <li>Target the browser. That's where the value of WebAssembly is. (Node supports it as well. BUT, WHY THO?)</li> <li>Build from scratch. In the video I started from <code>mkdir</code> a folder. We should strive to understand details on every level whenever possible. Boilerplates and dependencies should come later.</li> </ol> <h2>Things you'll need:</h2> <ol> <li>Safari 11+</li> <li>Xcode. More specifically, you should be able to run <code>clang</code> in a shell.</li> </ol> <h2>The Workflow</h2> Having these things installed, get a copy of <a href="https://github.com/WebAssembly/wabt">The WebAssembly Binary Toolkit</a> (wabt). Build it. The README has detailed instructions. I just went into the folder and ran <pre><code>make clang-release </code></pre> This will generate a bunch of binary files in <code>out/clang/Release</code> and you need to make sure you can run them from wherever you want to work on WebAssembly project (so either copy them into a folder included in your <code>PATH</code> environment variable or add the absolute path to <code>out/clang/Release</code> to <code>PATH</code>). Among the binaries "wabt" builds, <code>wat2wasm</code> takes a <code>.wat</code> file and compiles it to a WebAssembly binary. A <code>.wat</code> is a source file in the <a href="https://developer.mozilla.org/en-US/docs/WebAssembly/Understanding_the_text_format">text format</a> for WebAssembly, which is in the form of S-expressions. So <pre><code>wat2wasm main.wat -o main.wasm </code></pre> …will compile your WebAssembly module in <code>main.wat</code> to generate <code>main.wasm</code>, the binary file. For now, <code>main.wat</code> can be the simplest WebAssembly program: <pre><code class="language-lisp">(module) </code></pre> Running the binary in a browser demands the bulk of the work. First, we'll need a web page. It doesn't need any content other than invoking some JavaScript code. <pre><code class="language-html"><!DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8"> <title></title> </head> <body>  <script src="play.js"></script> </body> </html> </code></pre> First, the Javascript logic needs to fetch and instantiate the compiled WebAssembly module. Since this is not a JS or WebAssembly tutorial, I'll point you to the docmuntation for <a href="https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API">Fetch API</a>, <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise">Promises</a>, and <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/WebAssembly">the WebAssembly object</a> for details: <pre><code class="language-javascript">fetch("main.wasm").then(reponse => reponse.arrayBuffer() ).then(bytes => WebAssembly.instantiate(bytes, {}) ).then(result => result.instance ).then(main); </code></pre> This snippet fetches <code>main.wasm</code> (adjust this URL according to your choosing), instantiate it, then pass it into a function named <code>main</code>, we can put a placeholder logic for it for now: <pre><code class="language-javascript">function main(wasm) { console.log(wasm); } </code></pre> Before we move on, you'll find that simply opending your HTML file in browser and looking at developer console won't work. Safari would complain about cross-domain request error for <code>fetch</code>. So we need to serve these resources locally. I usually use the built in server module from Python standard library for this kind of things: <pre><code># In your source folder, run python -m SimpleHTTPServer </code></pre> Now go to <a href="http://localhost:8000">http://localhost:8000</a> and click on your HTML file. If everything went well, you should see a WebAssembly instance logged in the developer console. Congratulations! You can start writing WebAssembly locally. Just remember to re-compile <code>main.wat</code> with <code>wat2wasm</code> whenever you want to test things out in browser. <h2>An Actual "Hello, World!" Implementation</h2> This is my implementation: <pre><code class="language-lisp">(module ;; Allocate a page of linear memory (64kb). Export it as "memory" (memory (export "memory") 1) ;; Write the string at the start of the linear memory. (data (i32.const 0) "Hello, world!") ;; write string at location 0 ;; Export the position and length of the string. (global (export "length") i32 (i32.const 12)) (global (export "position") i32 (i32.const 0))) </code></pre> In other words, we expose information of the linear memory we manipulated to the JavaScript environment. Things that has been <code>export</code>ed will show up as properties of <code>exports</code> of the <code>WebAssembly</code> instance. We can access them in the <code>main</code> JavaScript functions: <pre><code class="language-javascript">function main(wasm) { const memory = wasm.exports.memory; const length = wasm.exports.length; const position = wasm.exports.position; ... } </code></pre> Then it's just plain-old Javascript (tho I had to steal it from tutorials). <code>memory.buffer</code> is of type <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/ArrayBuffer">ArrayBuffer</a>. We need to convert it into a string and log it to the console: <pre><code class="language-javascript">function main(wasm) { const memory = wasm.exports.memory; const length = wasm.exports.length; const position = wasm.exports.position; const bytes = new Uint8Array(memory.buffer, position, length); const s = new TextDecoder('utf8').decode(bytes); console.log(s); } </code></pre> Et, voilà! <code>Hello, World!</code> hot off a Web Assembly module in your developer console. To conclude, I personally like to use a <code>Makefile</code> to streamline some of the typing. Here's what I used for this demo: <pre><code class="language-makefile">compile: wat2wasm main.wat -o main.wasm serve: python -m SimpleHTTPServer </code></pre> <h2>Conclusion</h2> No fancy schmancy Javascript build stack, no 3rd-party code dependency. Write code, compile, run on your (virtual, in browser) machine, repeat. That sounds like "assembly" to me! Sat, 30 Dec 2017 18:43:49 -0800 https://duan.ca/2017/12/30/hello-world-in-webassembly/ https://duan.ca/2017/12/30/hello-world-in-webassembly/ Building 'pwd' Under 1 Minute In Swift I guess I'm starting a series of minute-long videos. In this first video, I built <code>pwd</code> out of <code>getcwd</code>. The code turned out shorter than I imagined (1 line). Which is a good demonstration of Swift's great C-interop capability. <div class="video-container"> <iframe width="560" height="315" src="https://www.youtube.com/embed/YR4MCcMnHrI" frameborder="0" gesture="media" allow="encrypted-media" allowfullscreen></iframe> </div> Some details are lost in this video, however. For example, the POSIX standard actually says <code>getcwd</code>'s 2nd argument being <code>0</code> is undefined. Luckily, both macOS and Linux define it the way we want (allocate as much memory as needed). Swift's handling of <code>char *</code> aka <code>UnsafeMutablePointer<CChar></code> in context of a <code>var</code> array is also very cool. Wed, 27 Dec 2017 01:13:07 -0800 https://duan.ca/2017/12/27/building-pwd-under-1-minute-in-swift/ https://duan.ca/2017/12/27/building-pwd-under-1-minute-in-swift/ Contributing To Open-Source Swift I discover, report, fix and merge a bug in the open-source Foundation project. And why, yes, on YouTube. <div class="video-container"> <iframe src="https://www.youtube.com/embed/rwuj80W8TtI" frameborder="0" gesture="media" allow="encrypted-media" allowfullscreen></iframe> </div> <ul> <li>Pull request: <a href="https://github.com/apple/swift-corelibs-foundation/pull/1376">https://github.com/apple/swift-corelibs-foundation/pull/1376</a></li> <li>SR-6647: <a href="https://bugs.swift.org/browse/SR-6647">https://bugs.swift.org/browse/SR-6647</a></li> </ul> Sat, 23 Dec 2017 13:21:10 -0800 https://duan.ca/2017/12/23/contributing-to-open-source-foundation/ https://duan.ca/2017/12/23/contributing-to-open-source-foundation/ Smart Color Invert And Your iOS Apps <ol> <li><a href="#introduction">Introduction</a></li> <li><a href="#the-basics">The Basics</a></li> <li><a href="#compatibility-with-older-ios-versions">Compatibility with older iOS versions</a></li> <li><a href="#how-to-screenshot-with-inverted-color-settings">How to screenshot with inverted color settings</a></li> <li><a href="#conclusion">Conclusion</a></li> </ol> <h2>Introduction</h2> Apple sneakily shipped a "dark mode" in iOS 11, native apps looks gorgeous in this mode (especially on iPhone X with it's OLED display): <img src="/assets/2017/12/native-ios-apps-with-smart-invert.png" srcset="/assets/2017/12/native-ios-apps-with-smart-invert.png 2x" alt="iOS 11 apps in smart invert mode"/> To see it for yourself, go to Settings -> General -> Accessibility Display Accommodations -> Invert Colors and toggle the switch to on (I personally set triple tap of the power buttton to toggle this on/off): <img src="/assets/2017/12/invert-color-settings.png" alt="turn on smart invert mode in Settings" /> Note that in Apple's first-party apps, not every view's color is inverted (hence, "smart"). For apps that we built, that would not be the case automatically. Instead, we get the full color inversion under this setting: <img src="/assets/2017/12/third-party-app-no-change.png" srcset="/assets/2017/12/third-party-app-no-change.png 2x" alt="default third party app before and after smart color"/> In this article, we'll see how to best adapt our apps so they look just as good as first-party iOS apps under the smart invert setting. <h2>The Basics</h2> TL;DR: use <a href="https://developer.apple.com/documentation/uikit/uiview/2865843-accessibilityignoresinvertcolors">accessibilityIgnoresInvertColors</a> on your <code>UIView</code>s to prevent their color to be inverted. iOS 11 introduced <code>accessibilityIgnoresInvertColors</code>, a property on <code>UIView</code>. From the <a href="https://developer.apple.com/documentation/uikit/uiview/2865843-accessibilityignoresinvertcolors">offical documentation</a>: <blockquote> If inverting the colors would have a negative impact on your view's content, set this property to true to prevent it from inverting its colors. Setting the property to true prevents the system from inverting the colors of the view and all of its subviews. </blockquote> So, for views containing photos or videos with real-world content, we probably want to set it to <code>true</code>. In our sample app's source code: <img src="/assets/2017/12/setting-accessibilityIgnoresInvertColors-to-true-in-code.png" alt="setting accessibilityIgnoresInvertColors to true in code" /> This gives us the desired effect for that view 🎉: <img src="/assets/2017/12/smart-invert-in-code-result.png" srcset="/assets/2017/12/smart-invert-in-code-result.png 2x" alt="result of supporting invert color in code"/> The same can be done in Interface Builder. Let's fix the other view by setting a runtime attribute: <img src="/assets/2017/12/setting-accessibilityIgnoresInvertColors-to-true-in-interface-builder.png" alt="setting accessibilityIgnoresInvertColors to true in Interface Builder" /> Result: <img src="/assets/2017/12/smart-invert-in-interface-builder-result.png" srcset="/assets/2017/12/smart-invert-in-interface-builder-result.png 2x" alt="result of supporting invert color in interface builder"/> Tada! <h2>Compatibility with older iOS versions</h2> The story gets slightly complicated when your app need to run on older iOS versions. For one, our code won't compile: <img src="/assets/2017/12/compile-error-on-older-oses.png" alt="compile error on older iOS versions" /> But that's nothing new to iOS developers, really. We can fix that with a <code>#available</code>: <img src="/assets/2017/12/fix-compile-error.png" alt="fix compile error on older iOS versions" /> When you run the app in older iOS devices or simulators, Xcode will complain about our runtime attribute as well: <img src="/assets/2017/12/xcode-complain-about-runtime-attirbute.png" alt="Xcode complains about runtime attribute on older iOS" /> If that bothers you, I recommend writing your own wrapper for this property as an extension to <code>UIView</code>. To put a cherry on top, make it <code>@IBInspectable</code>! <pre><code class="language-swift">extension UIView { /// Whether or not to set accessibilityIgnoresInvertColors to true for iOS 11. /// For older OS, this value is false and setting it has no effect. @IBInspectable public var ignoreColorInversion: Bool { get { if #available(iOS 11, *) { return self.accessibilityIgnoresInvertColors } return false } set { if #available(iOS 11, *) { self.accessibilityIgnoresInvertColors = newValue } } } } </code></pre> Having this snippet in the project, we set it in interface builder to any view with a couple of clicks: <img src="/assets/2017/12/set-custom-invert-property-in-interface-builder.png" alt="set custom invert color properties in Interface Builder" /> <h2>How to screenshot with inverted color settings</h2> Here's a bonus hint: screenshots taken on device won't have their color inverted even if you have the setting set to "on"! Go ahead, try it :). To show off our handy work for supporting smart invert color, we'll need some other means to take screenshots. All screenshots in this article is taken using QuickTime on a Mac. So plug in your iOS device, open QuickTime Player, in menu select File -> New Movie Recording and select your device from the drop-down menu by clicking the arrow next to the record button: <img src="/assets/2017/12/select-ios-device-in-quicktime-player.png" alt="Select iOS device for screenshots in QuickTime Player" /> Now you can screen shot the QuickTime Player window the normal way. <h2>Conclusion</h2> You can <a href="/assets/2017/12/SmartInvert.zip">download the sample project</a> and play with it. Smart invert is awesome and I personally wish all 3rd-party apps will update their apps for it as time goes on. If your favorite app (including your own!) doesn't support it properly, please consider informing the developer about <a href="https://developer.apple.com/documentation/uikit/uiview/2865843-accessibilityignoresinvertcolors">accessibilityIgnoresInvertColors</a>, or just send this article their way! Wed, 20 Dec 2017 13:17:44 -0800 https://duan.ca/2017/12/20/smart-invert-support-for-you-app/ https://duan.ca/2017/12/20/smart-invert-support-for-you-app/ BitArray The Final Act Alas, there's the 3rd and last video about <code>BitArray</code>, originated on my spark of inspiration on a Caltrain ride home. This makes my first YouTube playlist as well, even tho the first 2 videos were planned to be completely separate. We added more standard library functionalities to BitArray (ExpressibleByArrayLiteral, CustomStringConvertible, Equatable, etc). Then, we added documentation and other open-source things left undone from "episode 2". <div class="video-container"> <iframe src="https://www.youtube.com/embed/ckrd_FoFMl0" frameborder="0" gesture="media" allow="encrypted-media" allowfullscreen></iframe> </div> Tue, 19 Dec 2017 12:17:49 -0800 https://duan.ca/2017/12/19/bitarray-final-act/ https://duan.ca/2017/12/19/bitarray-final-act/ Making An Open-Source Swift Project This video shows what it takes to open source a Swift project. Starting from some code in a playground, I created a framework that supports distribution via Swift Package Manager, CocoaPods, Xcode project and Carthage. The framework can be used across iOS, macOS, watchOS, tvOS and Linux. In addition, we also added SwiftLint to the project, added a Makefile, put everything on GitHub and set up continuous integration via Travis. README, COC, LICENSE and CHANGELOG are stubbed in as well. <div class="video-container"> <iframe src="https://www.youtube.com/embed/pA0T1CdqMt8" frameborder="0" gesture="media" allow="encrypted-media" allowfullscreen></iframe> </div> This is not at all a comprehensive list of things a good Open Source project should have. We are still missing the version tag, documentation generation, automatic Carthage binary generation, test coverage... Anyways, here's some stuff mentioned in the video: <ul> <li>Sample project from the video: <a href="https://github.com/dduan/BitArray">https://github.com/dduan/BitArray</a></li> <li>Video of me writing the code in this project: <a href="https://www.youtube.com/watch?v=-k_jrIoD56k">https://www.youtube.com/watch?v=-k_jrIoD56k</a></li> <li>Swift Package Manager: <a href="https://swift.org/package-manager/">https://swift.org/package-manager/</a></li> <li>CocoaPods: <a href="https://cocoapods.org">https://cocoapods.org</a></li> <li>Carthage: <a href="https://github.com/Carthage/Carthage">https://github.com/Carthage/Carthage</a></li> <li>Xcconfigs for universal framework: <a href="https://github.com/mrackwitz/xcconfigs">https://github.com/mrackwitz/xcconfigs</a></li> <li>SwiftLint: <a href="https://github.com/realm/SwiftLint">https://github.com/realm/SwiftLint</a></li> <li>Travis CI: <a href="https://travis-ci.org">https://travis-ci.org</a></li> </ul> Sun, 17 Dec 2017 21:24:43 -0800 https://duan.ca/2017/12/17/making-an-open-source-swift-project/ https://duan.ca/2017/12/17/making-an-open-source-swift-project/ Let's Build A BitArray In Swift In this video, I build a space-efficient BitArray that confirms to <code>RandomAccessCollection</code> in Swift in a playground. It involves some basic (but neat!) bitwise manipulation tricks. <div class="video-container"> <iframe src="https://www.youtube.com/embed/-k_jrIoD56k" frameborder="0" gesture="media" allow="encrypted-media" allowfullscreen></iframe> </div> I guess I'm a YouTuber now. You can subscribe to my <a href="https://www.youtube.com/channel/UCkBVC0dMgyUnxzettP7qE-A">channel</a> 😛. Tue, 12 Dec 2017 22:51:04 -0800 https://duan.ca/2017/12/12/lets-build-a-bitarray-in-swift/ https://duan.ca/2017/12/12/lets-build-a-bitarray-in-swift/ List Comprehension In Swift Let's explore ways to add list comprehension to Swift. <h2>Motivation</h2> <a href="https://en.wikipedia.org/wiki/List_comprehension">List comprehension</a> should be no stranger to a Python or (and?) Haskell user. It's a really compact syntax that deals with <a href="https://en.wikipedia.org/wiki/Cartesian_product">Cartesian product</a> of lists. In the case of Python, it's probably responsible for the lack of evolution of lambda expressions, since it's much nicer to write one-liners with it in place of <code>map</code>s and <code>filter</code>s. Here's an example of an list comprehension in Haskell from Wikipedia: <pre><code class="language-haskell">a = [(x,y) | x <- [1..5], y <- [3..5]] -- [(1,3),(1,4),(1,5),(2,3),(2,4) ... </code></pre> In this example, a list of pair of integers is constructed from 2 lists of integers. Here is what that example would be in Python: <pre><code class="language-python">a = [(x, y) for x in range(1,6) for y in range(3, 6)] # [(1, 3), (1, 4), (1, 5), (2, 3), (2, 4) ... </code></pre> Here's what it would be in mathematics (except we are dealing with sets, not lists, but I'll only refer to lists from here on.): <pre><code>Let (a, b) be an ordered list of elements {(x, y)|x ∈ {1,2,3,4,5}, y ∈ {3,4,5}} </code></pre> One can filter out unwanted elements with predicates, and apply arbitrary functions to elements of the result. Let's say we only want even numbers from the first list, and we want the sum of x and y, continuing on our examples: <pre><code class="language-haskell">a = [x+y | x <- [1..5], y <- [3..5], x `mod` 2 == 0] </code></pre> <pre><code class="language-python">a = [x + y for x in range(1,6) for y in range(3, 6) if x % 2 == 0] </code></pre> <pre><code>{x+y|x ∈ {1,2,3,4,5}, y ∈ {3,4,5}, x is even} </code></pre> In theory, this syntax can be applied to an arbitrary number of lists. Putting aside how often this need comes up in day-to-day programming in your domain, it should be obvious that it's alternative, be it nested loops or <code>map</code>s and <code>filter</code>s, is pretty clumsy in comparison. <h2>Adding List Comprehension in Swift</h2> A comprehension can be considered in 3 parts: <ol> <li>some lists, each may contain a different type of elements.</li> <li>a predicate (or a series of them joined logically) to filter out elements.</li> <li>a function to process the combination of elements into results.</li> </ol> In Swift, if our input is only one list, there's a pretty sweet way to achieve that: <pre><code class="language-swift">list.filter(predicate).map(processor) </code></pre> To make comprehension work with more lists, we have some syntax options. <h3>Option One</h3> The "brute force" option would be a function that parameterize all 3 parts of the comprehension. Such as <pre><code class="language-swift">// going with order of appearance in Python/Haskell syntax func comprehension<Element, List, Result>( predicate: (Element) -> Bool, list: List, processor: (Element) -> Result ) where List: Sequence, List.Element == Element { // implementation } </code></pre> To supporting more than one list, just add more parameters to both types and the function itself. (Can't wait until we can have <a href="https://github.com/apple/swift/blob/master/docs/GenericsManifesto.md#variadic-generics">variadic generic parameters</a>!) <h3>Option Two</h3> Deploy more syntax tricks. Somehow make it visually similar to the math/Haskell/Python notation. If we can accept some temporary data structure and introduce/implement some operators, there'd be many possibilities. <pre><code class="language-swift">/// Just an example of the infinite possibilities. processor | list0 &&& list1 | predicate </code></pre> I'll leave the implementation of this example as an exercise to the reader. <h3>Option That I Like</h3> I spent quite some time exploring the realm of possibilities in "option two". However, introducing data structures and custom operators just to do what "option one" offers seems really unappealing. It's not entirely clear that doing so would be "Swift-y" anyways! Eventually, I did find an arrangement that fits in Swift, and requires no fancy syntax trickery. The result of list comprehension is a list. The goal of this operation is to construct a list. Yep, thinking along this line, it became obvious that using a "list"'s initializer is just natural: <pre><code class="language-swift">let a = Array(1..<5, 3..<5, where: { n, _ in n % 2 == 0 }) { ($0, $1) } // [(2,3),(2,4),(2,5) ... </code></pre> The processing function is at the end to take advantage of the trailing closure syntax. It's nicer when there's not predicate: <pre><code class="language-swift">let a = Array(1..<5, 3..<5) { ($0, $1) } // [(1,3),(1,4),(1,5),(2,3),(2,4) ... </code></pre> This syntax seems both succinct and Swift-y. I put an implementation on <a href="https://github.com/dduan/Comprehension">github</a>, in case you find it useful. <h2>Parting Thoughts</h2> There's no doubt that the conclusion in this post is imperfect. Though it feels more Swift-y, it deviates from the mathematical syntax by a lot. We can only implement it for finite number of lists. When many lists are involved, using a embedded closure as the predicate would make the compiler complain that the expression is too complex. We suffer from the normal woes with Swift closures where anonymous arguments (<code>$0</code>, <code>$1</code>, etc) won't work unless the last one is mentioned in the closure's body. Overloading <code>Array</code> initializer may negatively affect compilation speed in large projects. Not all of these issues are temporary. Does list comprehension warrant a language change in Swift? Can you think of better ways to implement it with the current compiler? Sat, 09 Dec 2017 12:26:30 -0800 https://duan.ca/2017/12/09/list-comprehension-in-swift/ https://duan.ca/2017/12/09/list-comprehension-in-swift/ Building Swift Projects In Source Compatibility Mode It's a few weeks before WWDC, so naturally¹ it's time to try build your Swift projects in the compiler's compatibility mode! <ol> <li> Download and install a snapshot for the next major version <a href="https://swift.org/download/">on Swift.org</a>. <img src="/assets/2017/05/download-swift-snapshot.png" alt="download swift toolchain" /> <img src="/assets/2017/05/install-swift-snapshot.png" alt="install swift toolchain" /> </li> <li> Choose the newly installed toolchain in Xcode. <img src="/assets/2017/05/choose-toolchain.png" alt="choose swift toolchain in Xcode" /> </li> <li> Ask the compiler to use compatibility mode. This means using the complier flag <code>-swift-version X</code>, where "X" is the current major Swift version. In project's "Build Settings", value for "Other Swift Flags" should contain <code>-swift-version X</code>. This could mean setting it in Xcode, in <code>.xcconfig</code> files you are using and/or in your dependency managers such as Cocoapods. <img src="/assets/2017/05/compat-mode-flag.png" alt="compatibility flag in xcode" /> For example, with Cocoapods, you'll need to add the following in your <code>Podfile</code> to compile 3rd party libraries in compatibility mode: <pre><code> post_install do |installer| installer.pods_projects.targets.each do |target| target.build_configurations.each do |config| config.build_settings["OTHER_SWIFT_FLAGS"] = "$(inherited) -swift-version 3" end end end </code></pre> </li> <li> Build your project! This is where things start to get exciting. You should expect some warnings. Hopefully they are self-explanatory enough. Most of them should correspond to <a href="https://apple.github.io/swift-evolution/">a swift-evolution proposal</a>. Improvement to the language or compiler usually means some of these warnings tell you problems in your code that has been ignored by the compiler previously. Fix them today! The project should compile successfully in compatibility mode (despite warnings). This where you can stop reading. Go celebrate with your coworkers, friends, and family! Things could go wrong for compiler snapshots, of course. Read on if see errors or crashes (whaaaaat 😸). </li> <li> It's time to tell the compiler team about the error or crash you encountered. Reduce the error or crash to a state that your are comfortable reporting in public. Then go to <a href="https://bugs.swift.org">bugs.swift.org</a> and file a JIRA ticket describing the error or compiler crash. During the process of code reduction you may find ways to work around the compile error or crash. Make the changes for the workaround and repeat steps 4-5. Maybe your project will compile this time. </li> <li> The issue you discovered will be fixed in the official Swift release come fall. You've ensured a smooth Swift upgrade for your project and contributed to the Swift community 🎉! </li> </ol> <hr /> ¹ source compatibility mode is a thing starting with Swift 4. As new major version of Swift is released, code written in the previous version should compile without change in compatibility mode. Mon, 22 May 2017 09:04:01 -0700 https://duan.ca/2017/05/22/build-in-swift-compatibility-mode/ https://duan.ca/2017/05/22/build-in-swift-compatibility-mode/ Replying To Old Mailing List Threads One common complains on the Swift Evolution mailing list is about its inscrutable interface. If you see an inactive thread on the web archive but haven't subscribed, there seems to be no way to participate or "revive" it since you never received any of its emails. With a <a href="https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20170206/031537.html">hint</a> and some experiments, I've discovered that there is a way. This post reveals the magic step by step (without commenting on the merits of email or this solution). Like HTTP, Email is a text-based <a href="https://tools.ietf.org/html/rfc2822">protocol</a>. Unlike HTTP, we directly consume payloads of this protocol and, as a consequence, don't think much about that very fact. Like HTTP, each email includes headers hidden by normal email clients. Each header is essentially two strings, a key and a value, separated by a colon. Among these headers are <code>Message-Id</code> and <code>In-Reply-To</code>. When we tell our email client applications to start a reply, they take the value from <code>Message-Id</code> and use it as value for <code>In-Reply-To</code> automatically. To observe all this in action, we can open some emails with our favorite text editors and look for these headers. Of course, this require us to know where the emails exist as files. On macOS, an easy thing to do is to drag from Mail.app to Finder/Desktop and open the resulting .eml file: <img src="/assets/2017/02/open-email.gif" alt="Open an email in text editors" /> Among the (perhaps overwhelming amount of) headers, we'll find the two fields we are looking for: <img src="/assets/2017/02/email-headers.png" alt="Email Headers Message-Id and In-Reply-To" /> … I'll leave the clients' reply behavior regarding these fields for the reader to verify. Mailing list software such as GNU Mailman, which <a href="https://swift.org">swift.org</a> and <a href="http://llvm.org">llvm.org</a> use to host various mailing lists, associate emails in threads by chaining them with the headers explained above, among other things. As long as we have a message's <code>Message-Id</code>, we can reply to it "officially" by adding its value to the <code>In-Reply-To</code> header in our email, regardless of whether we have received that email ourselves previously. So here are the steps to reply retroactively to a mailing list thread. <ol> <li> find <code>Message-Id</code> of the mailing list message we want to reply to. This value is contained in the mailing list web archive's <code>mailto</code> link. Unfortunately Mail.app doesn't recognize it. It's easy enough to find it ourselves though: <img src="/assets/2017/02/find-message-id.gif" alt="Finding Message-Id on mailman archive" /> Note the <code><</code>, <code>></code>, <code>@</code> characters are percent-quoted. We have to recover the id values to the format <code><alphanumerics-potentially-separated-by-dash@address.com></code>. Another way to acquire this value is from the gzip'd archive. There they just exist as plain text. The downside is you have to dig out the message itself first. </li> <li> Add <code>In-Reply-To</code> header to our email. Draft a new email, make its title the same as our email chain's title (this is a good idea because lots of email clients do use the title to thread messages). Set the appropriate recipients, and CCs, including the mailing list's address. Now save this email as draft and open it in a text editor as we did in our investigations. Add in the line <code>In-Reply-To: <id-we-found-in-step-one.address.com></code>, save it. Then send this email (for example, open it with Mail.app and use Message->Send Again in the menu). Of course, some email client <a href="http://www.pixelbeat.org/docs/thunderbird-threading.html">supports</a> adding it from their GUI. </li> </ol> You'll find the in addition to functioning as an normal message to the recipients, the mailing list will properly put your message to the original thread (in the case of swift-evolution, only if your reply is within the same week). Tue, 07 Feb 2017 10:21:19 -0800 https://duan.ca/2017/02/07/replying-to-old-mailing-list-threads/ https://duan.ca/2017/02/07/replying-to-old-mailing-list-threads/ Site Improvements I spent a whole day making changes to this site. You may notice a new section is now added: <a href="/links">links</a>. Yes, I want a link list format for this site. Often I find stuff that's insightful and not always visible. I want to collect those and bookmark them here. Modelling after <a href="http://daringfireball.net">Daring Fireball</a>, I've added a new <a href="http://duan.ca/articles/feed.xml">feed</a> which only contains long-form, original articles. I'll be tagging the links so that they show up along side my normal posts in tag pages. Each tag page now gets its own feed! You can read about it in the <a href="/">about page</a>. A few small issues has been fixed. I wrote the most Ruby in my life today. Yay? Mon, 16 Jan 2017 23:24:13 -0800 https://duan.ca/2017/01/16/site-changes/ https://duan.ca/2017/01/16/site-changes/ Making TUI Applications In Swift One of my fondest memory in programming is writing the game Snake on one of these: <img src="/assets/2016/12/e-dictionary.jpg" alt="An Electronic/Digital Dictionary" /> At the time, I was in a very restrictive school where a real computer wasn't permited in class. The actual thing (that's an "electronic dictionary" by the way) was probably older, slower and of a different brand than the one in the photo. In fact, I didn't know how to make a copy of the code, and the device wasn't even mine. So the game just stayed there when the device was returned to the owener and its fate unknown. But I love it nontheless. Design the game, write it, playing it, showing it off <ul> <li>everything.</li> </ul> Fastforward to today. I use and write software applications for a living. Programs I use directly range from terminal commands to GUI apps and web apps in a browser. Some programs, however, don't belong to those categories: vim, tig, irssi, etc. They run in a terminal like CLI commands. But unlike commands like <code>ls</code>, they take over the entire buffer to offer different modes of interaction. You can use hotkeys to navigate menus, trigger functions and so on. Sometimes you can use mouse, too. Their layout often resemble that of a GUI application running in it's own "window": <img src="/assets/2016/12/tig.png" alt="tig" /> Just to make things painfully clear, these apps's interface is composed exclusively of text. We can vaguely categorize them as "Text-based User Interface(TUI)" application. There are many reasons to like TUI applications. Compared to a CLI program, they can display information in 2-dimentional, more organized layout (aka more like a GUI app). Unlike a GUI application, they almost only require a terminal to run. If you ssh onto a Linux server on your iPad, they'd work the same as running locally on a Mac. For people who live in terminal simulators, it's nice to run apps that don't open a new Window. Oh, they often tend to look the same no matter how your OS changes. Most importantly, they remind me of the programs I built on that electronic dictionary. I want to make them today, with my favorite programming language Swift! The go-to library for writing TUI apps is <a href="https://www.gnu.org/software/ncurses/">ncurses</a>. Classics such as vim, emacs and mutt were written with it to some degree. But I find it hard to dive into: it has a large number of strange legacy function names and offers pre-built UI elements like panels, menus, forms etc. The terminal in my mind is a spread of characters that can be updated by me. That sounds fun. Ncurses seems more like UIKit :P. Luckily, I found the fun library: <a href="https://github.com/nsf/termbox">termbox</a> by <a href="https://github.com/nsf">nsf</a>. Termbox offers around a dozen functions that gives us size of the terminal, a way to update text at a row/column coordinates, and user inputs in forms of key-presses and mouse-clicks. And that's it! After making <a href="https://github.com/dduan/Termbox">a Swift wrapper</a>, I was able to build a painter app in <a href="https://github.com/dduan/TerminalPaint">a handleful of lines</a>: <img src="/assets/2016/12/terminal-paint.png" alt="terminal paint" /> So that's an TUI app drawing according to mouse clicks built with only a few primitives from termbox. Awww yeah! Here's the code with the comment stripped, to give you a taste: <pre><code class="language-swift">import Termbox func printAt(x: Int32, y: Int32, text: String, foreground: Attributes = .default, background: Attributes = .default) { let border = Termbox.width for (c, xi) in zip(text.unicodeScalars, x ..< border) { Termbox.put(x: xi, y: y, character: c, foreground: foreground, background: background) } } func updateHelp(drawing: UnicodeScalar) { let lastY = Termbox.height - 1 let content = [ "Drawing [\(drawing)]", "Press other character to change", "Use mouse to draw", "Press 'q' to quit" ].joined(separator: " | ") let filler = String(repeating: " ", count: Int(Termbox.width) - content.unicodeScalars.count) printAt(x: 0, y: lastY, text: content + filler, foreground: .white, background: .blue) } func paint() { do { try Termbox.initialize() } catch let error { print(error) return } Termbox.inputModes = [.esc, .mouse] var drawingCharacter: UnicodeScalar = "." { didSet { updateHelp(drawing: drawingCharacter) } } updateHelp(drawing: drawingCharacter) Termbox.present() outer: while true { guard let event = Termbox.pollEvent() else { continue } switch event { case let .character(_, value): if value == "q" { break outer } drawingCharacter = value case let .key(_, value): if value == .space { drawingCharacter = " " } case let .mouse(x, y): Termbox.put(x: x, y: y, character: drawingCharacter, foreground: .red) default: continue } Termbox.present() } Termbox.shutdown() } paint() </code></pre> <a href="https://github.com/dduan/Termbox">Library</a> and this <a href="https://github.com/dduan/TerminalPaint">app</a> are both on Github. Go and have fun :) Mon, 26 Dec 2016 21:39:05 -0800 https://duan.ca/2016/12/26/making-tui-programs-in-swift/ https://duan.ca/2016/12/26/making-tui-programs-in-swift/ Rebase And Sign Multiple Git Commits I never bothered <a href="https://git-scm.com/book/en/v2/Git-Tools-Signing-Your-Work">signing</a> my git commits until Github started putting those shiny badges on them. Since then I've encountered a problem: rebased commits lose their hash and their signed-ness. That's not a big deal for a single commit. Just run <code>git commit -S --amend --no-edit</code> after rebase. And that commit, which is at <code>HEAD</code> will have a new hash and be signed. What if we have more than one commit to rebase? My instinctive answer was sign the one on <code>HEAD</code>, then use interactive rebase to rotate the next one on top and sign it, repeat until each one is signed and reorder them. But that doesn't work at all because a commit's position in history is part of its content. That means after a commit is rebased, it and every commits following it will have a new hash and git doesn't give us an oppurtunity to say "include my signature, please". You cannot rebase and keep the commits signed! Well, that kind of sucks. But you are reading about it here, so I'd better give you something to help, right? There is, at least, one way to achieve rebase and sign multiple commits: <ol> <li>Run interactive rebase: <code>git rebase -i branch</code>.</li> <li>Mark all commits and "edit" or "e".</li> <li>Git now stops at every commit for you to make changes. This is an oppurtunity to sign it! <code>git commit -S --amend --no-edit</code>, just like how you would sign <code>HEAD</code>.</li> <li><code>git rebase --continue</code>, move on the next commit until all every one is edited.</li> </ol> There's virtually nothing you cannot do with git. Hence the reputation for it's glorious UI, ya know? Sat, 24 Dec 2016 01:31:14 -0800 https://duan.ca/2016/12/24/rebase-multiple-git-commits-with-signing/ https://duan.ca/2016/12/24/rebase-multiple-git-commits-with-signing/ How To Omit Needless Words A typical experience in adopting Swift 3 may look like this: <ul> <li>Run the migrator.</li> <li>Fix up what the migrator left undone so your project complies.</li> <li>Remove artifacts that migrator added to ensure successful compilation (you know what this means if you finished previous step in a mildly sized code base).</li> <li>Fix bugs caused by mechanical changes.</li> <li>Try to adapt your code base to the new style: lowercased static members, etc.</li> </ul> The last step is particularly challenging in a team environment because it involves naming things. Luckily we have (IMHO) the best "features" in Swift 3 for help: the <a href="https://swift.org/documentation/api-design-guidelines/">API Design Guidelines</a>. <hr /> <a href="https://swift.org/documentation/api-design-guidelines/#omit-needless-words">Omit needless words</a> is one of the most differentiating guideline for function names between Swift 2 and 3. In Swift 2 we used to name things like so… <pre><code>func doStuffWithThing(thing: Thing) { … } </code></pre> … and use it like … <pre><code>doStuffWithThing(someThing) </code></pre> That's a lot of "thing"s, have you noticed? Since the word "Thing" is merely repeating the type information, we can omit it in Swift 3: <pre><code>// declare func doStuff(with thing: Thing) { … } // use doStuff(with: someThing) </code></pre> It's clear at both declaration and use site what we are doing stuff to. So … you should go through you code base and make this change! In the next section, we'll discuss one way to make it happen. <hr /> <h2>Step 1: Automate</h2> A good indicator of "merely repeating type information" is repetition of words. If you have been disciplined about naming your functions in Swift 2, finding such repetition should take no more than one or two good regular expressions. After the migrator, your code may contain a mixture of these: <pre><code class="language-swift">func doStuffWithX(_: X) { … } func doStuff(forY y: Y) { … } </code></pre> In other words, the repeating word (<code>X</code> or <code>Y</code>) is preceded by a <a href="https://web.archive.org/web/20161006012135/https://www.englishclub.com/grammar/prepositions-list.htm">preposition</a>. Finding these in a large code base is quite fun. If you don't feel like writing the scripts yourself, I've made a little tool for it <a href="https://github.com/dduan/needless">here</a>. (Sometimes the repeating words aren't a exact match because of prefixes such as "CG" in <code>CGSize</code>. That's covered by the tool as well.) <h2>Step 2: Update</h2> You can further rearrange the pieces in function name mechanically to: <pre><code class="language-swift">func doStuff(preposition originalArgName: Type) { … } </code></pre> The <a href="https://github.com/dduan/needless">tool</a> I wrote can suggest alternatives like this. Applying them is technically automatable. But I find manually doing tasks such as fixing up indentation for multi-line functions or updating their use site to be easier. Your conclusion may depend the size of your code base and your patience. <h2>Step 3: Audit</h2> The main goal for the API Design Guidelines is "clarity at the point of use". Our automated process will yield some undesirable results. I'll list a few scenarios here. Poorly named function name becomes poorer. Example: <code>alertForScreenSize(_ size CGRize)</code> becomes <code>alert(forScreen size: CGSize)</code>. Obviously, "for screen" is misleading since a <code>CGSize</code> is not a screen. The problem here is the "alert" is not for a "size" to begin with. We as human intuitively pick up that it's for a "screen with a certain size". So the correct renaming here should be <code>alertForScreen(with size: CGSize)</code>. Awkward literal values results in ungrammatical phrases. Imagine our argument is a enum: <pre><code class="language-swift">enum Mode { case add case edit } </code></pre> … and we renamed a function <code>updateUIForMode(mode: Mode)</code> to <code>update(for mode: Mode)</code>. The call site, therefore, becomes <code>update(for: .add)</code>. What makes it feels wrong? Well, a preposition followed by a verb ("for add") is not very grammatical. Normally we would say "update UI for adding". So here we need to update the literals in that enum to "adding" and "editing". Type information becomes too weak. This can happen very often with enum literals. We often name enum values with an adjective to qualify its name: <pre><code class="language-swift">enum UIControlState { case automatic // which state? the *automatic* state! … } </code></pre> After renaming, we'll have functions that simply becomes too generic and mysterious at call site: <pre><code class="language-swift">update(for: .automatic) // automatic what? 😂 </code></pre> This is where we need to be flexible and use our own good judgement. Perhaps it's simply better to keep the qualifying postfix: <pre><code class="language-swift">update(forControlState: .automatic) // better </code></pre> In the guideline's parlance, this is <a href="https://swift.org/documentation/api-design-guidelines/#weak-type-information">compensate for weak type information</a>. Other argument labels needs updates too. A function at call site should read like a sentence as much as possible. We removed words at beginning of the "sentence", it's important to take a look at labels for the rest of the sentence and ensure the whole thing fits together. <hr /> There are many, many other fallouts from mechanically omitting needless words that I didn't cover in this post. There are many many other things in the API Design Guidelines that are worth conforming to. There will be much much bike shedding among your team members. Just remember, the API guidelines don't provide an answer to every naming decision. Following it means your code base will fit in with the rest of the community and APIs from Foundation/Swift standard library. But find what feels right for your team is the most important thing. In the end, you should be glad that everyone cares so much about your code base's "Swiftness" and it'll be all worth it! Sat, 22 Oct 2016 17:08:41 -0700 https://duan.ca/2016/10/22/how-to-omit-needless-words/ https://duan.ca/2016/10/22/how-to-omit-needless-words/ Distribution Tests There are several ways to build dependencies for a Swift 3 project: Swift Package Manager, Cocoapods, Carthage, etc. Many have an opinion on which is the best choice for their projects. As a library author who want to help as many people as possible, however, they can't ignore any of them. I sometimes question the sanity of that conclusion: instead of becoming an export in each package manager, I find myself a novice of all. Did I break support for any of them with this project change? Is it still working on a particular platform, say, tvOS? Can I really know? The only way to really know is to verify yourself: both Cocoapods and Carthage support four platforms -- iOS, macOS, watchOS, tvOS; Swift Package Manager only works on Mac or Linux. So that's 2*4+1=9 targets. All these targets need is have the library in question fetched and imported. Every Swift library author should have such verification before publishing updates of their project. The steps to verify can be triggered with commands: fetch and build dependencies, build Xcode/Swift project. To automate the process, put these commands in a script or a Makefile. But wait, there's more! One shouldn't have to create these dummy projects every time they create a new library. If all these projects do is importing a library and attempt to build, they should work for any library. The config in <code>Package.swift</code>/<code>Cocoapods</code>/<code>Cartfile</code> and the <code>import</code> statements just needs some strings replaced: name of the next library, URL for its git repository, etc. And that's a scriptable process as well! To recap, one could, in theory, copy in some dummy projects, run a command to inject information about a new library, run another command to build all these project, verifying that support for those package managers remain functional. In reality, I have created <a href="https://github.com/dduan/DistributionTests">DistributionTests</a> and put it on <a href="https://github.com/dduan/DistributionTests">Github</a> 😉. The script <code>customize</code> requires 3 pieces of information of the library: its import name, its git repository URL and a major version number. The assumption here is the library generates uniformly named artifacts: the file name for <code>.framework</code> and the name users use to import it are the same. Testing distribution of a library is as simple as: <ol> <li>clone the project.</li> <li>customize the project with <code>customize</code>.</li> <li>run <code>make</code>.</li> </ol> If you do step 1 and 2, include the projects in library's repository, then only step 3 is necessary! This makes testing distribution methods trivial on a continuous integration server. Go forth and create fearlessly! Mon, 10 Oct 2016 10:34:49 -0700 https://duan.ca/2016/10/10/distribution-tests/ https://duan.ca/2016/10/10/distribution-tests/ A Case For OOP? Python's standard library includes a very handy <code>defaultdict</code>. It behaves almost exactly like the standard dictionary except it'll supply a pre-defined value for any non-existence keys. It is, unsurpringly, a subclass of <code>dict</code>. I find my self missing this handy container in Swift. Especially when I use a normal <code>Dictionary</code> to accumulate/coalesce values under distinct keys. So I wrote my own: <script src="https://gist.github.com/dduan/31ed39c4c98ecb88290f0743cb394c20.js"></script> There are a few noticable things about this implementation: <ul> <li>It does not conform to the <code>DictionaryLiteralConvertible</code> protocol, for no good reasons, really. The initializer in this protocol takes a varadic argument. There's no conevient way to forward this array to a normal dictionary's initializer (incidentally, this is a Swift feature I really want). Plus, I don't need <code>DefaultDictionary</code> to be a literal convertible.</li> <li>Most of the code, including the imaginary <code>init</code> mentioned in previous point, simply reuses stuff from <code>Dictionary</code>: asscociated type, indexes, generator, subscript, etc.</li> </ul> In comparison, Python implements <code>defaultdict</code> in a more intuitive way -- via inheritance. But do we want inheritance for <code>struct</code>s and <code>enum</code>s in Swift? What does that even mean? Is it simply a case that will go away when protocols become more powerful? Thu, 19 May 2016 12:03:02 -0700 https://duan.ca/2016/05/19/a-case-for-oop/ https://duan.ca/2016/05/19/a-case-for-oop/ Cheap Orders How to create order? If the second law of thermal dynamics tells us anything, we'd better get to work, right? {: .lead} Before going full existential, let's limit "order" to Swift's set and dictionaries -- there is none. Of course, you can take values/keys out and sort them. But what if all you care about is some order? <hr /> Recently, I wrote <pre><code class="language-swift">enum Token { static let all = [ "=": .Equal, "-": .Minus, // more token mappings … ] case Equal case Minus // more tokens … } </code></pre> … in the hope that I can take each value from <code>Token.all.keys</code> and see if a prefix of a string is a matching token. It started to fail as the tokens expands to multiple characters: <pre><code class="language-swift">enum Token { static let all = [ "=": .Equal, "-": .Minus, // more token mappings … "==": .Equality, "->": .Arrow, // more token mappings … "===": .Identity, // more token mappings … ] case Equal case Minus case Equality case Arrow case Identity // … more tokens … } </code></pre> <code>-></code> could get a match with <code>-</code> and <code>===</code> would match to either <code>==</code> or <code>=</code>, etc. Since the tokens in this exercise have at most 3 characters, I decided to group them by length and match from the longer group first. The groups became: <pre><code class="language-swift">enum Token { static private let group1 = [ "=": .Equal, "-": .Minus, // more token mappings … ] static private let group2 = [ "==": .Equality, "->": .Arrow, // more token mappings … ] static private let group3 = [ "===": .Identity, // more token mappings … ] } </code></pre> Now I can choose which group to take values first. There's a way to do it without adding some control flow logic: <pre><code class="language-swift">[group3.keys, group2.keys, group3.keys].flatten() </code></pre> Even better, I'll make it a lazy property… <pre><code class="language-swift">enum Token { static var all = { [group3.keys, group2.keys, group3.keys].flatten() }() } </code></pre> …except an important piece of information is missing from the property: what's <code>all</code>'s type? Turns out, it's become <pre><code class="language-swift">FlattenCollection<Array<LazyMapCollection<Dictionary<String, Token>, String>>> </code></pre> Ahh, it seems that in the pursue of cheap, lazy creation of these structures, we are forced to deal with a bunch of type wrappers, each having a good reason to be here! But I really just need something like <code>Array<String></code> for the consumer. If only there's a way to make all this stuff go away from my type signature, as if they are <a href="http://robnapier.net/erasure">erased</a> :). Okay, I'm talking about <code>AnySequence</code> now. Rob Napier has an excellent post on this topic <a href="http://robnapier.net/erasure">here</a> if you need to catch up. Our code eventually end up like this: <pre><code class="language-swift">enum Token { static var all: AnySequence<String> = { AnySequence( [group3.keys, group2.keys, group3.keys] .flatten()) }() } </code></pre> Instead of <code>Array<String></code>, we have an <code>AnySequence<String></code>. Our tokens now gets checked with the correct order. We didn't need to sort the entire set of tokens, nor did we do any heavy data massage upfront, making a bunch of copies along the way. <hr /> Looking back, this post really failed to capture the eureka moment as I came up with the erasure method. I discovered a series of small challenges and got help from Swift's designers in each step. Everything fell together in the end. Sun, 15 May 2016 17:42:27 -0700 https://duan.ca/2016/05/15/cheap-orders/ https://duan.ca/2016/05/15/cheap-orders/ Deploy Jekyll Site To Github With Make At risks of being too meta, I want to talk about how this post is going to go from my computer to a repository on Github on both the <code>master</code> and <code>gh-pages</code> branch. The former contains the file I'm typing, and the latter an updated HTML site generated by Jekyll. Here's how: I type <code>make deploy</code> once I commit this file in my local repository. I spent a day to set up a Jekyll site. The last part involves deploying it to Github, my preferred way to host a static site. There are a lot of posts and scripts on how to achieve it. And Github seems to have an "afficial" support for Jekyll site that disables plugin scripts (understandably so). But having worked with git and make for all these years made me immune to all the fanciness the Internet tries to sell. Here's my script for deploying a Jekyll project to Github pages: <pre><code>DEPLOY_PATH=/tmp/jekyll_deploy build: jekyll build deploy: git checkout -f gh-pages git clean -d -x -f git pull git checkout master jekyll build rm -rf ${DEPLOY_PATH} mkdir ${DEPLOY_PATH} cp -R .git ${DEPLOY_PATH} cd ${DEPLOY_PATH}; git checkout gh-pages; git clean -d -x -f cp -R _site/* ${DEPLOY_PATH} cd ${DEPLOY_PATH}; git add .; git commit -m "`curl whatthecommit.com/index.txt`" cd ${DEPLOY_PATH}; git push -f origin gh-pages </code></pre> Drop it into root of your Jekyll project, commit all changes (including the <code>Makefile</code>). And type <code>make deploy</code>. BOOM, done. There are a couple of assumptions: <ul> <li>You build the site in the <code>_site</code> folder</li> <li>You store Jekyll source on <code>master</code> and have the <code>gh-pages</code> branch up.</li> <li>You can build the site by running <code>jekyll build</code>.</li> <li>You don't care about commit message on the deploy branch.</li> </ul> I like it better than other methods because it does not require installing additional software/scripts and there's no restriction on plugins. Sun, 15 May 2016 01:00:00 -0700 https://duan.ca/2016/05/15/deploy-jekyll-to-github/ https://duan.ca/2016/05/15/deploy-jekyll-to-github/ char ** in Swift A "string" in C is just a continuous chunk of <code>char</code> values in memory with <code>\0</code> at the end. To reference it, a variable of type <code>char *</code> is used to store the address of the first <code>char</code> (commonly known as a pointer 😉). It's common to have string manipulating functions take this form: <pre><code class="language-swift">void foo(char **errorMessage) { // ... } </code></pre> To mutate the arugment <code>errorMessage</code> of type <code>char *</code>, <code>foo</code> takes a pointer to it, <code>(char *)*</code>. <hr /> How do we call <code>foo</code> in Swift? Here's the tl;dr. We can wrap it in a Swift function that have the same interface: <pre><code class="language-swift">func fooSwift(inout errorMessage: String?) { var message: UnsafeMutablePointer<CChar> = nil foo(&message) errorMessage = String.fromCString(message) } </code></pre> <code>errorMessage</code> will contain whatever our C function <code>foo</code> assigns to it. <hr /> So, what's really going on here? Inspecting <code>foo</code>'s signature in Swift, we see <pre><code class="language-swift">func foo(errorMessage: UnsafeMutablePointer<UnsafeMutablePointer<CChar>>) </code></pre> … okey, <code>errorMessage</code>'s type is little intimidating to someone who doesn't use C functions in Swift everyday (like me)! Let's break it down. Swift does a ton of work for us to interoperate with C: <ol> <li> <code>CChar</code> is Swift's name for <code>char</code> in C (shocking, amiright?) </li> <li> <code>UnsafeMutablePointer<Type></code> roughly translates to <code>Type *</code>, so syntactically, we can see why <code>UnsafeMutablePointer<UnsafeMutablePointer<CChar>></code> is used to bridge the C type <code>(char *)*</code>. </li> <li> A function that takes <code>UnsafeMutablePointer<Type></code> argument accepts <code>inout Type</code> values. Therefore, we can look at <code>foo</code> as <pre><code>foo(inout errorMessage: UnsafeMutablePointer<CChar>) </code></pre> </li> <li> Swift acknowledge C's string representation and provides convenient methods for converting <code>char *</code> / <code>UnsafeMutablePointer<CChar></code> to its own <code>String</code> type (<code>String.fromCString()</code>). </li> </ol> Hopefully you can see how <code>fooSwift</code> works now. <hr /> Taking a step back, to deal with <code>char **</code> in Swift, we overcame 2 obstacles: <ol> <li> The difference in string representation between C and Swift. </li> <li> Compared to C pointers, Swift's <code>inout</code> expresses mutability of function arguments in a more restricted way. We can't nest <code>inout</code>s to represent the infinite layers of indirections that pointers achieve. </li> </ol> Sun, 01 Nov 2015 11:18:40 -0800 https://duan.ca/2015/11/01/char-star-star-in-swift/ https://duan.ca/2015/11/01/char-star-star-in-swift/ Let's Play LLVM in Swift: Setup The prospect of Swift being open-source excites me. In preparation for it, I decided to learn a little bit about LLVM. In the offical <a href="http://llvm.org/docs/tutorial/index.html">tutorial</a>, C++ and OCaml are used to create a programming language. So I asked myself: why not Swift? As you'll see in this post, Swift is as good as any language when it comes to leveraging LLVM. LLVM is an "compiler infrastructure". As a user, that translates to "a set of libraries to help us create programming language and tools". Since this is not an introduction to the LLVM project, it suffices to say that LLVM makes creating programming language easy: give your language in <a href="http://llvm.org/docs/LangRef.html">LLVM IR</a> form and you get the rest of a native/JIT language for free, optimization included! LLVM's default API is in C++. In addition, it officially offers a C wrapper. Lucky for us, Swift is C-interoperable – no bridging necessary :) Now we arrive at our goal for this post: create an Xcode project that can make calls into LLVM's C API, aka to compile this single line of code: <pre><code>let module = LLVMModuleCreateWithName("my_module") </code></pre> I assume you have Xcode 7.1 installed on OS X 10.11, but no more. <h2>1. Getting LLVM</h2> There are a lot of materials on the internet dedicated to setting up an LLVM environment. As we are not working on LLVM itself, and we are not on some crazy custom Linux environment (to be fair, it's trivial to set LLVM up on most major Linux distributions), the <a href="http://llvm.org/releases/download.html">pre-built Clang binaries</a> is good enough. Download and unpack the .tar file somewhere handy in on your hard drive. For example, I put it at <code>$(HOME)/usr/local/clang-3.4</code>. Aaaand we're done. We have LLVM. Sidenote: Xcode installs clang the compiler, but a lot of LLVM tools are missing. That's why we need a separate LLVM/Clang setup. <h2>2. Create an Xcode Project</h2> Create a new OS X - Command Line Tool Xcode project, choose Swift as it's language. <img src="/assets/2015/10/llvm-swift-01-create-cmd-project.png" alt="Create A Command Line Xcode Project For LLVM" /> Accessing C stuff in Swift is the same as using your Objective-C classes. So we need to <a href="https://developer.apple.com/library/ios/documentation/Swift/Conceptual/BuildingCocoaApps/MixandMatch.html">create a bridging header</a>. (I usually create a Objective-C class so that Xcode prompts me for creating the header, then I delete the .h and .m files). The project layout is now: <img src="/assets/2015/10/llvm-swift-02-bridging-header.png" alt="Create a bridging header to import LLVM C libraries" /> Import the LLVM headers for its C interface. For our example, this is the entire bridging header: <pre><code>#import <llvm-c/Core.h> </code></pre> It's probably a good time to replace content of <code>main.swift</code> with our awesome LLVM IR-generating code: <pre><code>let module = LLVMModuleCreateWithName("my_module") </code></pre> <h2>3. Teach Xcode About LLVM</h2> Our code would not compile at this point. Xcode complains that the LLVM header can not be found. Before you jump to the target build settings, allow me introduce <code>llvm-config</code>. It turns out that the compliler flags for building a compiler can get complex pretty quickly. So LLVM comes with a command that generates them. For me it lives under <code>$(HOME)/usr/local/clang-3.4/bin</code>. We can ask it for flags that compiles standard C++, links standard and core libraries like so: <pre><code>llvm-config --cxxflags --ldflags --system-libs --libs core </code></pre> (As we use more and more LLVM libraries in the future, the list following <code>--libs</code> will grow. <code>core</code> is all we need to compile our example). To anyone who's used GCC/Clang in command line, the output should be pretty self-explanatory: <pre><code>-I/Users/drchrono/local/clang-3.4/include -DNDEBUG -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -O3 -std=c++11 -fvisibility-inlines-hidden -fno-exceptions -fno-rtti -fno-common -Wcast-qual -L/Users/drchrono/local/clang-3.4/lib -lLLVMCore -lLLVMSupport -lz -lpthread -ledit -lcurses -lm </code></pre> I'll walk through how to ask Xcode to respect these. First, go to build settings and set search paths for header files and libraries according to output from <code>llvm-config</code>. For me that means: <img src="/assets/2015/10/llvm-swift-03-header-search-path.png" alt="Set LLVM header search path in Xcode" /> <img src="/assets/2015/10/llvm-swift-04-library-search-path.png" alt="Set LLVM library search path in Xcode" /> If you try to compile, now Xcode tells you some #define is missing. Again, we can find them in <code>llvm-config</code>'s result. Navigate to "Preprocessing" in build setting and add those values starting with <code>-D</code>, with out the <code>-D</code>: <img src="/assets/2015/10/llvm-swift-05-macros.png" alt="Set preprocessor macros" /> Remember to add these for both "Debug" and "Release". The last step is asking Xcode to link againt the LLVM libraries. Paste in the <code>-l</code> flags from <code>llvm-config</code> at "Other Linker Flags": <img src="/assets/2015/10/llvm-swift-06-link-libraries.png" alt="Ask Xcode to link against LLVM libraries" /> <h2>4. Conclusion</h2> Now our Swift LLVM code compiles! Looking back, setting LLVM up with Xcode is no more special than setting up with any C libraries. Hopefully this post will cut down research time for some. Now go create awesome natively languages in Swift! Sun, 25 Oct 2015 11:53:15 -0700 https://duan.ca/2015/10/25/lets-play-llvm-in-swift-setup/ https://duan.ca/2015/10/25/lets-play-llvm-in-swift-setup/ Swift Algebraic Data Types <h2>The Basics</h2> <a href="https://en.wikipedia.org/wiki/Algebraic_data_type">Algebraic Data Type</a> is a fancy name for "a type creaded by combining other types" in programming languages. One aspect of the "algebraic-ness" is how many potential new values there are for the new type, given a set of type as its building block. To better illustrate this, consider the following example in Swift. <pre><code class="language-swift">enum StarkChild { case Rickon, Bran, Arya, Sansa, Robb, Jon } enum Direwolf { case Shaggydog, Summer, Nymeria, Lady, Greywind, Ghost } enum Actor { case Wolf(Direwolf) case Person(StarkChild) } struct Parters { var wolf: Direwolf var person: StarkChild } </code></pre> <code>StarkChild</code> and <code>Direwolf</code> each has 6 potential values. Combining them, we get two new types. Question: how many potentiol values are there for <code>Actor</code>? How many for <code>Parters</code>? <hr> An <code>Actor</code> can be either a <code>StarkChild</code> or a <code>DireWolf</code>, therefore it has `6 <ul> <li>6 = 12<code>potential values – the *sum* of</code>Child<code>'s and </code>DireWolf`'s values.</li> </ul> A <code>Partners</code> requires us to select one value from <code>Child</code> and one from <code>DireWolf</code>, resulting in <code>6 * 6 = 36</code> potential values – the product of <code>Child</code>'s and <code>DireWolf</code>'s values. <code>Actor</code>, an <code>enum</code>, is a sum type. <code>Parters</code>, a <code>struct</code>, is a product type. Here, <code>Parters</code> could easily be defined as a <code>class</code> or a <code>tuple</code> and remain a product type. Because we can create product or sum types in these direct ways, we can say Swift has first class support for Algebraic Data Types. <h2>The Crossovers</h2> However, the story doesn't stop here. In Swift, an <code>enum</code>'s option can have multiple values. If it happens to be the only option, then this <code>enum</code>effectively becomes a product type: <pre><code class="language-swift">// there are 6 * 6 = 36 potential values for Parters enum Parters { case Value(wolf: DireWolf, person: StarkChild) } </code></pre> Incidentally, this makes <code>enum</code> similar to <code>data</code> in Haskell, where product and sum types can be created with a unified construct – <code>data</code>. In C and C++, <code>union</code>s are the closest thing to sum types. However, <code>union</code> is hardly used to combine arbitrary types due to its lack of associated values. What do people do in need of sum types? They make do with product types. Here's one way to achive that in Swift: <pre><code class="language-swift">// Actor.value can have only 6 + 6 = 12 potential values thanks to // manual enforcement class Actor { var child: StarkChild? var wolf: Direwolf? var value: Any { get { return child == nil ? wolf! : child! } set(newValue) { if newValue is StarkChild { child = (newValue as! StarkChild) wolf = nil } if newValue is Direwolf { wolf = (newValue as! Direwolf) child = nil } } } init(wolf: Direwolf) { self.wolf = wolf } init(child: StarkChild) { self.child = child } init() { fatalError("must initialize with a child or a wolf") } } </code></pre> It's… ugly. <h2>Recursion Types</h2> Besides sum and product, another common class of algebraic type is recursion types. The interesting bit here is that Swift struggles to support it. In WWDC 2015, it was announced that <code>enum</code>s can be defined recursively in Swift 2: <pre><code class="language-swift">enum Tree { case Empty indirect case Node(Tree, Tree) } </code></pre> As of this writing, Xcode 7 beta 3 has not delivered this feature yet. Also, it's a good bet that <code>indirect</code> is not going to be available in tuple aliases, such as: <pre><code class="language-swift">typealias Node = (indirect Node, indirect Node) </code></pre> I hope this is on the Swift team's (understandably) gigantic todo list 😉. Sun, 12 Jul 2015 15:51:27 -0700 https://duan.ca/2015/07/12/swift-algebraic-data-types/ https://duan.ca/2015/07/12/swift-algebraic-data-types/ Dynamic Swift Framework Without Xcode I came up with this question recently: <blockquote class="twitter-tweet" lang="en">Can we even use Frameworks with <a href="https://twitter.com/hashtag/Swiftlang?src=hash">#Swiftlang</a> on Linux?— Daniel Duan (@daniel_duan) <a href="https://twitter.com/daniel_duan/status/617470929241706496">July 4, 2015</a></blockquote> <script async src="//platform.twitter.com/widgets.js" charset="utf-8"></script> And I'm going to give the answer in this post (spoiler alert: yes, sort of). Here's the content of a framework created by Xcode: <img src="/assets/2015/07/just-framework-structure.png" alt="Just Framework Structure" /> Some of the files, such as <code>Info.plist</code>, are obviously construct of Xcode. Play with <code>swiftc</code> long enough, one would find that the <code>.swiftdoc</code>s, the <code>.swiftmodule</code>s and the one binary file came from the Swift compiler. Instead of listing the relevant <code>swiftc</code> command options, I've created a <a href="https://github.com/dduan/Swift-Framework-Without-Xcode">sample project</a> to demonstrate how one can complie and link to frameworks so that they can be <code>import</code>ed in the application code. The key ingredient for achieving it lies in Makefile. In summary, <code>swiftc</code> can do these for us: <ol> <li>generate a binary as a library/framework</li> <li>emit a <code>.swiftmodule</code> file, which Swift needs to understand that binary.</li> <li>assign a path the Swift runtime needs to locate this framework.</li> <li>compile source code that imports the framework, given that they exist in the paths relative to the app binary in step 3.</li> </ol> Based on these observations, it's not hard to imagine more sophisticated build systems, such as IDEs and package/dependency management systems. Sun, 05 Jul 2015 22:36:09 -0700 https://duan.ca/2015/07/05/dynamic-swift-framework-without-xcode/ https://duan.ca/2015/07/05/dynamic-swift-framework-without-xcode/ Let's Build A 'cat' In Swift 2 As a homework in one of the early college classes, I was asked to write unix commands such as <code>cat</code> in C. Let's do that in Swift today! To make things interesting, let's pretend we are on Linux. That means no Xcode nor Foundation can be used. It's hard to find a simpler unix program than <code>cat</code>: It takes a list of file names from the shell and write the content of each file to <code>stdout</code>. When no argument is given, it uses <code>stdin</code> as the source of its output. Writing it in C is trivial. Swift has exellent support for leveraging C. But to call even the standard C functions, we need to import them first. The <code>swiftc</code> command can compile a pure Swift source file like this: <pre><code>swiftc cat.swift -o cat </code></pre> We can add Objective-C bridging headers with the argument <code>-import-objc-header</code>. But to import the standard C functions, we also need to specify path to an SDK: <pre><code>swiftc -sdk $(xcrun --show-sdk-path --sdk macosx)\ -import-objc-header bridge.h\ cat.swift\ -o cat </code></pre> Instead of typing/copying that command, save this <code>Makefile</code> to the same directory as <code>cat.swift</code>: <pre><code>SDKPATH = $(shell xcrun --show-sdk-path --sdk macosx) CBRIDGEHEADER = bridge.h TARGETS := cat .PHONY : all $(TARGETS) all: $(TARGETS) $(TARGETS): swiftc -sdk $(SDKPATH) $@.swift -import-objc-header $(CBRIDGEHEADER) -o $@ </code></pre> Now <code>make cat</code> should take care of the compilation. Since file I/O is the only concern, we'll need C APIs from <code>stdio.h</code>, so <code>bridge.h</code> is a one liner: <pre><code class="language-c">#import <stdio.h> </code></pre> The standard C function for opening a file is <code>fopen</code>: <pre><code class="language-c">FILE * fopen ( const char *filename, const char *mode ); </code></pre> Hmmmm, how do we deal with all those pesky '*'s? To reference a certain C <code>Type</code> in Swift, we can use <code>UnsafePointer<Type></code> or <code>UnsafeMutablePointer<Type></code>. To make our lives easier, Swift <code>String</code>s automatically bridge to <code>const char *</code>. In other words, we can treat the signature of <code>fopen</code> as if it's the following: <pre><code>func fopen( filename: String, mode: String ) -> UnsafeMutablePointer<FILE> </code></pre> A character in C is represented by a byte in memory. Therefore Swift sees a <code>char</code> as of type <code>Int8</code> (8-bit integer). So a <code>char *</code> would be referenced as <code>UnsafeMutablePointer<Int8></code> in Swift. So <code>getline</code>, a function from POSIX <pre><code class="language-c">ssize_t getline( char **lineptr, size_t *n, FILE *stream ); </code></pre> would look like this in Swift: <pre><code class="language-c">func getline( inout lineptr: UnsafeMutablePointer<Int8>, inout n: UInt, stream: UnsafeMutablePointer<FILE> ) -> Int </code></pre> It returns the number if characters it finds. We now can open a file, read and print its content line by line, and close it with: <pre><code class="language-c">func fclose(stream: UnsafeMutablePointer<FILE>) </code></pre> Repeat this on each file specified in <code>Process.arguments</code>, or simply read from <code>stdin</code>, and we have a <code>cat</code>! Here's a screenshot of it displaying its own code: <img src="/assets/2015/07/swift-cat.png" alt="Swift cat" /> The code is also available in this <a href="https://gist.github.com/dduan/f6d359019db8b0b55962">gist</a>. Sat, 04 Jul 2015 20:56:19 -0700 https://duan.ca/2015/07/04/lets-build-a-cat-in-swift-2/ https://duan.ca/2015/07/04/lets-build-a-cat-in-swift-2/ NeoVim, Swift and Make When it comes to Swift source code editing, nothing beats Xcode 6.3! That includes Xcode 6.2, which drove me to good'O Vim for a while. Except it's not the old Vim, I'm trying out <a href="http://neovim.org">NeoVim</a>. The most noticable difference in NeoVim compared to Vim is its recent addition of a built-in terminal. With the help of syntax highlighting and a Makefile, working with Swift this way turns out to be a fine alternative. (As a side benefit, I'm forced to use only local-context autocompletion. Now I can actually spell out full UIKit APIs 😉.) Here's a gif: <img src="/assets/2015/04/demo.gif" alt="Make Swift NeoVim Demo" /> Some details are easy to miss. I pressed <code>,m</code>, <code>,</code> being my binding for <code><leader></code>. A terminal session launched with the <code>make</code> command running. When that's done, I could press any key and the terminal pane was dismissed. Since <code>test</code> is the default target in my <code>Makefile</code>, the test suit for my Swift codes actually ran. Here's how the shortcut is set in <code>.nvimrc</code>: <pre><code>if has('nvim') nnoremap <leader>m :rightbelow vertical split <bar> :term make<cr> endif </code></pre> The makefile is pretty straightforword if you've worked with <code>xcodebuild</code>. <pre><code>test : @xcodebuild test -project ProjectName.xcodeproj -scheme SchemeName -destination 'platform=iOS Simulator,name=iPhone 6' | xcpretty clean : @xcodebuild clean </code></pre> <a href="https://github.com/supermarin/xcpretty">xcpretty</a> is a nifty script that makes <code>xcodebuild</code>s output much more readable. Happy vimming, Swifties :) Wed, 22 Apr 2015 11:46:17 -0700 https://duan.ca/2015/04/22/neovim-swift-and-make/ https://duan.ca/2015/04/22/neovim-swift-and-make/ Swift Function Fun Facts You love Swift. You want to write a class that does HTTP, it might have methods like the following: <pre><code class="language-swift">func get(URLString:String, params:[String:AnyObject], headers:[String:String]) func post(URLString:String, params:[String:AnyObject], headers:[String:String]) func put(URLString:String, params:[String:AnyObject], headers:[String:String]) // and more for HEAD, OPTIONS … </code></pre> But you don't want to force your user to supply all arguments each time. You know that Swift supports default arguments, so you added some. Take GET as an example: <pre><code class="language-swift">func get(URLString:String, params:[String:AnyObject]=[:], headers:[String:String]=[:]) </code></pre> Now users can do things like {% highlight swift %} get("http://github.com") get("http://httpbin.org/get", headers:["Answer":42]) {% endhighlight %} That's flexible! Woohoo! After you thought about implementing these, though, you realize that <code>HTTPMethod</code> is merely a property on <code>NSURLRequest</code>. In other words, all of the previous methods can share the same implementation. In honor of the DRY principle, you write a function that accepts the method as an arguments and the previous functions each forwards the arguments to this function: <pre><code class="language-swift">func impl(method:String, URLString:String, params:[String:AnyObject], headers:[String:String]) { // … } func get(URLString:String, params:[String:AnyObject]=[:], headers:[String:String]=[:]) { impl("GET", URLString:URLString, params:params, headers:headers) } func post(URLString:String, params:[String:AnyObject]=[:], headers:[String:String]=[:]) { impl("POST", URLString:URLString, params:params, headers:headers) } </code></pre> This seems like a sensible solution. Except that later you realize that there needs to be more parameters for each function, so in the end, each function looks like this: <pre><code class="language-swift">func post( URLString : String, params : [String:AnyObject] = [:], json : [String:AnyObject]? = nil, headers : [String:AnyObject] = [:], auth : (String,String)? = nil, allowRedirects : Bool = true, requestBody : NSData? = nil, URLQuery : String? = nil, asyncCompletionHandler: ((HTTPResult!) -> Void)? = nil ) -> HTTPResult { return impl( "POST", URLString : URLString, params : params, json : json, headers : headers, auth : auth, data : requestBody, URLQuery : URLQuery, redirects : allowRedirects, asyncCompletionHandler: asyncCompletionHandler ) } </code></pre> Remembering that your goal is to respect DRY, and there are now giant blocks of code that all look the same except that first argument to <code>impl()</code>, you became determined to find a better alternative. Well, why not give currying a try? This example of currying with Swift comes to your mind: {% highlight swift %} func add(a:Int)(b:Int) -> Int { return a + b } let add3 = add(3) add3(b:2) // 5 {% endhighlight %} If we apply this technique and treat <code>method</code> in <code>impl()</code> as <code>a</code> in the example, we would get: <pre><code class="language-swift">func impl(method:String)( URLString:String, params:[String:AnyObject], headers:[String:String], …) { // … } let get = impl("GET") let post = impl("POST") </code></pre> right? However, you are forcing users to supply each argument again. To make things worse, the number of arguments is a lot larger. Hmm, but that's a solved problem, just add default values to <code>impl()</code>'s parameters: <pre><code class="language-swift">func impl(method:String)( URLString:String, params:[String:AnyObject] = [:], headers:[String:String] = [:], …) { // … } </code></pre> Ta-da! Wait a minute, Xcode now refuse to compile you code! <code>Default argument is only permitted for a non-curried function parameter</code>, it saids. Stubborn as you are, you decide that perhaps the Swift team hasn't got around to implementing this feature for curry syntax yet. Functions are first-class citizens! Surely if you return a function with default arguments…? <pre><code class="language-swift">func methodFactory(method:String) -> (params:[String:AnyObject] = [:], headers:[String:String] = [:], …) -> Void { return {(params, headers, …) in impl(method, params:params, headers:headers, …) } } let get = methodFactory("GET") let post = methodFactory("POST") </code></pre> Turns out, this manual form of currying only works when default arguments aren't involved! Now, you hate Swift. (Just to be clear, I don't really hate Swift as in, uh, hate Swift. Judgning from some comments, I might have failed to convey the lightheartedness that I felt writing this up. It's really like saying to a friend "I hate you" after he/she pulls a prank on you.) Sat, 18 Apr 2015 16:20:16 -0700 https://duan.ca/2015/04/18/swift-function-fun-facts/ https://duan.ca/2015/04/18/swift-function-fun-facts/ When Default Parameter Values Fall Short In Swift Swift supports default value for function parameters: <pre><code> func f(x=1, y=2, z=3) { return x * y + z } </code></pre> Functions defined in this way is extremely flexible as they allows the user to arbitrarily omit any parameters: <pre><code> f(y:6) // 9 </code></pre> Today, however, I ran into a scenario where default values stopped working. imagine a group of functions who share the same implementation function by supplying different values as its parameters. Furthermore, each of these functions also expose a same set of parameters. <pre><code> func impl(x:String, a:Int, b:Int, c:Int, d:Int, f:Int, e:Int) </code></pre> Sat, 18 Apr 2015 00:00:00 -0800 https://duan.ca/2015/04/18/when-default-parameter-values-fall-short-in-swift/ https://duan.ca/2015/04/18/when-default-parameter-values-fall-short-in-swift/ The Two Swifts onpatient 2.0 went live in App Store on September 27th 2014, 10 days after iOS 8 was officially released. For the next few months, it was featured for HealthKit intergration on the App Store front page. In conversations during that period, people would ask me "what was the experience like?" What they referred to, of course, was the fact that I wrote that version in 100% Swift. I have a very different answer to that question now compared to the standard one I gave back then. I hope to explain to you, my dear readers, how my view on Swift has evolved since the beginning. <h2>Swift The Puller</h2> <blockquote> After you get used to a few nuances, it's Cocoa Touch after all. </blockquote> People kind of went "huh", and didn't know how to respond to this answer, at the time when iOS 8 was just out. I think they were expecting horror stories. Well, they would have been right. One day <code>UIStoryboardSegue</code> would have an <code>identifier</code> of type <code>String!</code>, the next day it would become a <code>String?</code>. And the incomprehensible error message from Xcode made sure you woundn't figure it out with ease. This kind of changes would happen to your entire codebase each time a beta Xcode was released. Compunding with these implementation bugs, API changes and poor tooling, was the fact that you had to learn a new programming lanauge. If you and your family's livelihood depends direclty on the software you put out, avoiding that whole situation was (and is) the correct business choice. To me, however, all that was just plain fun. Within about a week, the strangeness of having <code>optional</code>s went away. It's kind of nice to have the compiler bark at you when something was unintialized. My old friends never left: ARC, auto-synthesized properties, blocks, protocols… As for the beta changes, they were only as often as the Xcode releases. Reading the release notes and a few careful inspections of the compile errors usually was enough to help you "fix" everything. A couple weeks in, Swift disappeared. I was just writing a brand new Cocoa Touch app. My attention was on architecture, user experience and patterns that works well with varies iOS APIs, business logic … We submitted the app on the first day it became possible. Turning back the clock to the ancient days when the world was <a href="http://arstechnica.com/apple/2010/06/copland-2010-revisited/">blissfully unaware</a> of Swift's existence. If one wanted to become a Mac or iOS developer, he/she would have gotten the advice along the lines of "learning the Objective-C is easy, knowing the system API is harder". Well, that hasn't changed post-Swift. My experience building software with UIKit/Cocoa Touch applied 100% in building the 100% Swift App. Things did change when I went back to Objective-C after a couple months of writing Swift exclusively, however. The lack of type inference made typing cumbersome. The worst part is the missing concept of optionality on data types. Every object suddenly were <code>optional</code> in my eyes. I exprienced a sense of insecurity, as if I was driving in a foregn country with simular but significantly less respected traffic rules. <h2>Swift The Pusher</h2> <blockquote> I'm humbled and excited by the future because of what it's teaching me. </blockquote> By the time I shipped a few major feature updates in Swift, something had been lurking on the back of my mind: I had left a whole bunch of its features untouched, yet I seemed to have worked with it to some degree of success. What gives? The emerging Swift community has shown me the way since. Observe: Exhibit A, the following Swift function is frequently found in <a href="https://github.com/typelift/Swiftz/blob/master/Sources/Swiftz/Curry.swift">libraries</a> and <a href="http://www.objc.io/books/">books</a>, if not shipping code: <pre><code>func curry<A, B, C>(f: (A, B) -> C) -> A -> B -> C { return { x in { y in f(x,y) } } } </code></pre> Currying is a simple concept. The fact that you can do it in Swift is nothing to write home about. What's striking to me, is how succinct, or, dare I say, elegant this can be expressed in Swift. In this short piece of code, you see generics, higher order functions, closures, type inference expressed on top of a strong type system…to form this basic construct found in functional programming. It's as if the language has been designed to enable it… which leads us to exhibit B: In a job <a href="http://clang-developers.42468.n3.nabble.com/ADVERTISEMENT-open-positions-in-Apple-s-Swift-compiler-team-td4039949.html">posting</a> for Apple's Swift compiler team, the following qualities are listed for Swift core library engineers (emphasis mine): <blockquote> <ul> <li>A passion for making a difference in the lives of other programmers</li> <li>A deep understanding of generic programming principles</li> <li>Experience with functional programming languages and/or idioms</li> <li>Experience bridging languages, especially bridging dynamic languages with static ones</li> <li>Exposure to innovative and/or comprehensive library designs</li> </ul> </blockquote> So in the Swift team's mind, an ideal "customer" of the language, the standard library authors being the most intimate ones, should be "exploiting" the language features to practice functional idioms. This detail stuck and ultimately inspired me to pick up <a href="http://www.objc.io/books/">Functional Programming in Swift</a>. The evidence piled on: <a href="http://www.objc.io/issue-16/">articles</a>, <a href="http://2014.funswiftconf.com">talks</a> and <a href="https://medium.com/swift-programming/2-functional-swift-c98be9533183">Blog</a> <a href="http://ijoshsmith.com/2014/11/30/getting-into-functional-programming-with-swift/">posts</a>. I realized that the language I've been using for several month has introduced the possiblity to do what the Haskell/Ocaml/Scala/F# folks do all day long (I won't discuss much functional programming in this post). The introduction has been gentle and could be safely ignored, but things like <a href="https://github.com/jeffh/Fox">automatically generated tests</a> and parser combinators got me hooked. I believe anyone who write code can benefit from knowing and appreciating the funtional paradigm. What's unique about Swift is that it's presented on Apple's huge developer eco-system. That means a huge part of our industry will benefit as Swift spread futher and futher. In that sense, it really stands out among all the other languages that enables functional programming. Tue, 10 Feb 2015 10:50:16 -0800 https://duan.ca/2015/02/10/the-two-swifts/ https://duan.ca/2015/02/10/the-two-swifts/ My Xcode 6.2 And 6.3 Prediction We have a situation here: three versions of Xcode are available to devleopers as of this writing. <ol> <li>Xcode 6.1.1</li> <li>Xcode 6.2 beta</li> <li>Xcode 6.3 beta</li> </ol> 6.2 is released with the purpose of making WatchKit availale. 6.3, so far, has been focused on Swift 1.2 and new Objective-C features, and a few non-language related features. The fact that there are two betas indicates to me each will be followed up by an official release. So how? My guess: there will be an official Xcode 6.2 release after Watch release. It doesn't make much sense to skip a minor version. Then things get interesting. Watch is coming out in April. Shortly after that, we'll have WWDC. Usually, WWDC comes with major Xcode beta release. So ... what happens to version 6.3? My guess: either we'll never get Xcode 6.3, or we won't get Xcode 7 this year. The former make sense in the context that <a href="http://9to5mac.com/2015/02/09/apples-ios-9-to-have-huge-stability-and-optimization-focus-after-years-of-feature-additions/">Apple Is Focused on Stablity</a>. Xcode needs some love on that front, let's face it! On the other hand, no harm would be done if Apple market 6.3 as version 7 come WWDC. Afterall, Swift 1.2 is significant enough to justify a major version. Tue, 10 Feb 2015 02:09:50 -0800 https://duan.ca/2015/02/10/my-xcode-62-and-63-prediction/ https://duan.ca/2015/02/10/my-xcode-62-and-63-prediction/ Build And Run iOS Apps In Commmand Line Xcode is slow. Enough said. What's worse, sometimes I find myself relying too much on auto-completion with Cocoa Touch, a blessing and a curse! So I searched for an alternative workflow in command line. The result was rather confusing: there are posts about using <code>xctool</code> or <code>xcodebuild</code> to build Xcode targets, using <code>ios-sim</code>, <code>simctl</code> or <code>instruments</code> to manage and manage or launch simulators. Most of the information is out of date. Eventually though, I was able to piece together an answer for my needs. That is, given an iOS project set up with Xcode 6, I want to <ol> <li>build a target.</li> <li>launch a iOS simulator.</li> <li>install the built .app bundle to the launched simulator.</li> <li>run the installed app.</li> <li>uninstall the app from the simulator.</li> </ol> All in command line, with Xcode closed. Before we proceed to the steps, you need to gather a few pieces of information: <ol> <li>the Xcode build scheme of your choice (e.g. "AwesomeApp").</li> <li>your app bundle ID (e.g. "com.awesome.app").</li> <li>name of an existing simulator (e.g. "iPhone 6 Plus"). If you don't want to look it up in Xcode GUI, look for it in output of command <code>xcrun simctl list</code> .</li> </ol> Ready? Here we go. (These commands should be run in the project folder). Build the target: <pre><code>xcodebuild -scheme AwesomeApp -destination 'platform=iphonesimulator,name=iPhone 6 Plus' -derivedDataPath build </code></pre> Launch the simulator: <pre><code>xcrun instruments -w 'iPhone 6 Plus' </code></pre> Install the bundle (after simulator is launched and target is built with previous commands): <pre><code>xcrun simctl install booted build/Build/Products/Debug-iphonesimulator/AwesomeApp.app </code></pre> Launch the app in simulator (after it's installed with the previous command): <pre><code>xcrun simctl launch booted com.awesome.app </code></pre> Uninstall the bundle: <pre><code>xcrun simctl uninstall booted com.awesome.app </code></pre> Quite a few parameters needs to be added for the build step if you have a comlex project. Please RTFMs. Write some script to automate the steps, if are a lazy typiest like me. Sat, 07 Feb 2015 20:35:02 -0800 https://duan.ca/2015/02/07/build-and-run-ios-apps-in-commmand-line/ https://duan.ca/2015/02/07/build-and-run-ios-apps-in-commmand-line/ tableView:didSelectRowAtIndexPath: In Two Lines You have a <code>UITableViewController</code> with a couple of static cells, you want to invoke some code for each cell in the delegate. Here's a quick way to do it: <pre><code> - (void)tableView:(UITableView *)tableView didSelectRowAtIndexPath:(NSIndexPath *)indexPath { SEL action = (SEL[]){@selector(method1), @selector(method2)}[indexPath.row]; ((void (*)(id, SEL))[self methodForSelector: action])(self, action); } </code></pre> It's got everything you love about C and Objective-C: array literals, function pointers, casting, selectors and something called <a href="http://www.cocoawithlove.com/2008/02/imp-of-current-method.html">IMPs</a>. This piece of code maps selected cells to methods by putting methods by indexing selectors in a C array. Why all the fuss on the second line? wouldn't a simple <code>performSelector:</code> work? The short answer is: to show the compiler that we are responsible adults. You can read more about it [here](SO Answer); Sat, 03 May 2014 13:42:00 -0600 https://duan.ca/2014/05/03/tableview58didselectrowatindexpath58-in-two-lines/ https://duan.ca/2014/05/03/tableview58didselectrowatindexpath58-in-two-lines/ One Weird Trick To Make Vim Go Faster On Your Mac I noticed something strange today. While playing with Ubuntu on a VirtualBox hosted by OS X Mavericks, Vim seems much faster than it being in iTerms2. How could that be? So I took the following steps to test things out: <ul> <li>Installed exact configurations with Vundle on the VM.</li> <li>Vim in Terminal.app</li> <li>MacVim with GUI.</li> </ul> Nope, Vim is still more responsive on Ubuntu. In fact, text input seem more responsive on this VM than the host OS in general! I thought I came to the light switching to OS X after using Linux as desktop for years, and now this? <a href="http://developer.android.com/reference/android/util/Log.html#wtf(java.lang.String,%20java.lang.Throwable)">WTF</a>? Defeat and confused, I went to the Internet, and found <a href="http://stackoverflow.com/questions/4489885/how-can-i-increase-cursor-speed-in-terminal">something close to an answer</a>. Under System Preferences-Keyboard, drag the two slide widget ("Key Repeat" and "Deley Until Repeat") to the right most. Suddenly, Vim become faster! Turns out, key repeating is very important for Vim users. Fri, 02 May 2014 10:25:00 -0600 https://duan.ca/2014/05/02/one-weird-trick-to-make-vim-go-faster-on-your-mac/ https://duan.ca/2014/05/02/one-weird-trick-to-make-vim-go-faster-on-your-mac/ Installing PyQt5 with Python 3 On OS X Today I installed PyQt5 on OS X 10.9. Turns out, it's not as straight-forward as one would think. Using <code>homebrew</code> will not work: <pre><code> brew install PyQt5 --with-python3 </code></pre> This is because of <a href="https://github.com/Homebrew/homebrew/issues/25735">an unfortunate choice</a> made by the homebrew maintainer regarding Python 3. So installing from <a href="http://pyqt.sourceforge.net/Docs/PyQt5/installation.html">source</a> is the way to go. Following the installing instruction, you would download and install <a href="https://web.archive.org/web/20140410074945/http://pyqt.sourceforge.net/Docs/sip4/installation.html">sip</a> first, the install <a href="http://pyqt.sourceforge.net/Docs/PyQt5/installation.html">PyQt5</a> itself. Except that's not enough. When you run <code>python configure.py</code>, you see this error: <pre><code> error: Use the --qmake argument to explicitly specify a working Qt qmake. </code></pre> It's pretty self-explanatory. <code>qmake</code>, the build tool for qt is needed here. Install qt5 with homebrew and proceed: <pre><code> brew install qt5 </code></pre> Afterwards, you should be able to import <code>PyQt5</code> in a Python 3 REPL. Wed, 23 Apr 2014 10:36:00 -0600 https://duan.ca/2014/04/23/installing-pyqt5-with-python-3-on-os-x/ https://duan.ca/2014/04/23/installing-pyqt5-with-python-3-on-os-x/ Generating Xcode Build Number From Git The build version number in an Xcode project (CFBundleVersion in Info.plist) must be a monotonically increasing string for each archive submitted to the App Store. Neglection of this requirement will result in an error during the binary upload. Some automation will help avoiding this issue. First, we want to generate this version number from our version (duh) control system (VCS) each time our target gets built. My VCS of choice is Git, users of other systems just need to get a increasing number from their code history. On a *nix system, this command will count the number of commits on 'develop' branch up until HEAD: <pre><code> git rev-list develop | wc -l </code></pre> This is a very good candidate for our build number, for longer history generally correlates to later builds. Next, we make Xcode automatically run this command and use its result as the build number. Go to project navigator, select your build target under the project icon, click Build Phases, select Editor→Add Build Phase→Add Run Script Build Phase in the menu. Remove the content in editor of the new Run Script phase and replace it with: <pre><code>revnum=`git rev-list develop | wc -l` echo "#define BUILD_NUMBER $revnum" > InfoPlist.h touch InfoPlist.h </code></pre> Go to build settings, change value of Preprocess Info.plist File to "YES". Add "InfoPlist.h" to Info.plist Preprocessor Prefix File→release. We ask Xcode to run our command and save its result as a "#define" in a header file when it builds the project. This is done so that we can replace the "hardcoded" build number with the name of the constant: Open Info.plist, Double click the value of Bundle Version and replace it with <pre><code>BUILD_NUMBER </code></pre> There's only one issue left: <code>BUILD_NUMBER</code> is now saved in InfoPlist.h. Its value comes from our commit history. So we want to exclude this piece of information as part of our commit history (I'll leave the reason as an exercise for the reader). Ignore this file by adding "InfoPlist.h" to .gitignore (or that of your other VCS). To recap, when you build the project now, Xcode will find out how many commits are in the history, define it in a header file as <code>BUILD_NUMBER</code>, which gets used as the build number. As long as you keep with with version control, the build number problem goes away. Sun, 29 Sep 2013 15:50:50 -0600 https://duan.ca/2013/09/29/generating-xcode-build-number-from-git/ https://duan.ca/2013/09/29/generating-xcode-build-number-from-git/ Sencha Touch 2 and PhoneGap integration As one of my pet Sencha Touch project gets close to finish, I started looking into distribute it as native apps with Phonegap/Cordova. One of the concerns in do so is the 'deviceready' event provided by Phonegap, according to the documentation: <blockquote> This is a very important event that every Cordova application should use. ... </blockquote> <blockquote> The Cordova deviceready event fires once Cordova has fully loaded. After the device has fired, you can safely make calls to Cordova function. Typically, you will want to attach an event listener with document.addEventListener once the HTML document's DOM has loaded. </blockquote> In particular, the nice Sencha Touch microloader complicate the matter by being the sole Javascript file that's supposed to be included in <code>index.html</code> and is in charge of starting the actual code of our apps. Yet we need the starting point of the code be a response to the <code>deviceready</code> event. After some googling, I found that most information on this matter is either inaccurate, incomplete or outdated, that is until I found <a href="http://stackoverflow.com/a/10457158/243798">this answer</a> by <a href="http://dougan.me">Robert Dougan</a> on StackOverflow: <blockquote> Sencha Touch 2 will listen to that event and call your onReady/launch methods <ul> <li>therefore if you try listening to them in the launch method, it has already been fired.</li> </ul> Just put your logic inside the launch method in your application. </blockquote> To verify this claim, I dug into <code>sencha-touch-debug.js</code> distributed with Sencha Touch 2.2 and found the following code: <pre><code>if (Ext.browser.is.PhoneGap && !Ext.os.is.Desktop) { if (!Ext.readyListenerAttached) { Ext.readyListenerAttached = true; document.addEventListener('deviceready', triggerFn, false); } } </code></pre> It appears that the <code>deviceready</code> event is taken into account here as long as <code>Ext.browser.is.PhoneGap</code> is true in a mobile browser envronment, which, in the same source code, means: <pre><code>if (typeof window.PhoneGap != 'undefined' || typeof window.Cordova != 'undefined' || typeof window.cordova != 'undefined') { isWebView = true; this.setFlag('PhoneGap'); } </code></pre> Here, the global variable PhoneGap, cordova or Cordova needs to be defined to satisfy Sencha Touch 2's expectation of PhoneGap environment. Those globals are defined in the <code>cordova-x.y.x.js</code> file included in the PhoneGap/Cordova project files. So what needs to be done for the integration is simple (if not clear): include <code>cordova-x.y.x.js</code> in the js section of <code>app.json</code> project file so that the microloader knows to load it up early: <pre><code>"js": [ { "path": "path/to/cordova-x.y.z.js", }, { "path": "touch/sencha-touch.js", "x-bootstrap": true }, { "path": "app.js", "bundle": true, "update": "delta" } ], </code></pre> Run <code>sencha app build package</code> and drop the files it produces to the <code>www</code> folder in the PhoneGap project. Compile, ship. Tue, 28 May 2013 15:24:30 -0600 https://duan.ca/2013/05/28/sencha-touch-2-and-phonegap-integration/ https://duan.ca/2013/05/28/sencha-touch-2-and-phonegap-integration/ Sencha Touch Workflow with GNU Make and Tmux I throw this Makefile to the root directory of my Sencha Touch 2 projects for workflow automation. Assuming you write in CoffeeScript and run Tmux in a terminal: <code>make develop</code> will put <code>compass</code> and <code>coffee</code> to watch mode, in addition to spawning a local web server with Python 3. The three commands will run in three separate Tmux panes. <code>make watch</code> does the same thing sans the server spawning. <code>make</code> simply compile coffee script files and sass files. You can figure out the granular commands with some minimal knowledge of GNU Make. <script src="https://gist.github.com/DaNmarner/5659003.js"></script> Mon, 27 May 2013 15:00:00 -0600 https://duan.ca/2013/05/27/sencha-touch-workflow-with-gnu-make-and-tmux/ https://duan.ca/2013/05/27/sencha-touch-workflow-with-gnu-make-and-tmux/ Dynamic Height for List Item in Sencha Touch 2 tl;dr: Set the <code>itemHeight</code> value to <code>auto</code> and you'll get list items with dynamic height in Sencha Touch 2. In the Ext.List component provided by Sencha Touch 2, all SimpleListItem (or ListItem) has the same height. This means if your items each has content of different height, the list would look awkward. Fear not! Here's a solution (and its discovery). Load up a Ext.List and inspect one of the item element with Chrome/Safari developer tool, you'll find its <code>element.style</code> has <code>height: 47px !important;</code>: <img src="/assets/2013/05/height.png" alt="Default Height on List Item in Sencha Touch 2" /> Here's the key: CSS properties under <code>element.style</code> are set by Javascript. In other words, any attempt to override this property in stylesheet will fail. (Try it by specifying a height value on <code>div.x-list-item</code>, or any other class you suspect, if you need some convincing). So, how do we fix this with Javascript? If you examine the documentaion, Ext.List has a config option <code>itemHeight</code> with a default value. You can set it to a value that works best with all potential heights of your item content, resulting in items with identical heights. Setting <code>itemHeight</code> to <code>auto</code>, however, will make each item container flow with its inner element, thus achive dynamic height. Sun, 19 May 2013 16:03:30 -0600 https://duan.ca/2013/05/19/dynamic-height-for-list-item-in-sencha-touch-2/ https://duan.ca/2013/05/19/dynamic-height-for-list-item-in-sencha-touch-2/ Sencha Touch 2.2 Alpha Sass Bug Workaround Sencha <a href="http://cdn.sencha.com/touch/alpha/touch-2.2.0-alpha.zip">released</a> a new version Sencha Touch with Windows Phone 8 support. But since it's an alpha, there are a few more things to do than what the <a href="http://cdn.sencha.com/touch/alpha/2.2.0.52/release-notes.html">release note</a> says to get it working. One thing I've noticed is that when you generate a new app with <pre><code>sencha generate app Foo path/to/foo </code></pre> and make the changes to <code>resources/sass/app.scss</code> according to the release note, <code>compass compile path/to/foo/resources/sass</code> fails complaining a font file is missing: <pre><code>File not found or cannot be read: path/to/foo/resources/sass/stylesheets/fonts/pictos/pictos-web.woff </code></pre> To fix this, copy the needed fonts to where it supposed to be: <pre><code>cd path/to/foo mkdir -p resources/sass/stylesheets/fonts/pictos cp touch/resources/themes/fonts/pictos/pictos-web.* resources/sass/stylesheets/fonts/pictos/ </code></pre> The next error you'll see from <code>compass compile</code> is caused by the name changes to a few sass files in the framework. Long-story short, you need to change <code>resources/sass/app.scss</code> to the following: <script src="https://gist.github.com/4533833.js"></script> and <code>compass</code> should be happy from there. While you are at it, why not checkout the magical Windows Phone 8 theme included in Sencha Touch 2.2 alpha. Mon, 14 Jan 2013 13:47:00 -0600 https://duan.ca/2013/01/14/sencha-touch-22-alpha-sass-bug-workaround/ https://duan.ca/2013/01/14/sencha-touch-22-alpha-sass-bug-workaround/ Windows Phone 8 Theme in Sencha Touch 2.2 Sencha Touch 2.2 Alpha shipped with Windows Phone 8/IE 10 support. And it's pretty impressive! Just take a look at the starter app under the original theme and the Windows Phone 8 theme: The Default Theme: <img src="/assets/2013/01/original-theme.png" alt="Sencha Touch 2.2 default theme" /> WP8 Theme: <img src="/assets/2013/01/wp8-theme.png" alt="Sencha Touch 2.2 windows phone 8 theme" /> And it only takes a minute or two to get to the second screen. (if you have Sencha Cmd and Sencha Touch SDK 2.2 alpha ready). Here's how. <ol> <li>In SDK folder, generate the app with <code>sencha generate app Foo path/to/foo</code></li> <li><a href="/2013/01/14/Sencha-Touch-22-Alpha-Sass-Bug-Workaround.html">Workaround</a> the bug shipped in this alpha version. This won't be necessary once the bug has been fixed in the next release</li> <li>Open <code>resources/sass/app.scss</code> in the generated project folder )it should look like <a href="https://gist.github.com/4533833">this</a> after step 2). Replace every appearance of the word default to "windows". Then run <code>compass compile resources/sass</code> in the project root.</li> <li>Serve the project in a web server (I usually do <code>python -m SimpleHTTPServer</code>), fire up its url (localhost:800) and...</li> </ol> Boom! Mon, 14 Jan 2013 13:45:00 -0600 https://duan.ca/2013/01/14/windows-phone-8-theme-in-sencha-touch-22/ https://duan.ca/2013/01/14/windows-phone-8-theme-in-sencha-touch-22/ "Hello, World!" The Hard Way with Sencha Touch When I first got into the amazing Sencha Touch HTML5 framwork, it came across as a compilation of visual components that looks mobile, <a href="http://compass-style.org">easily themable</a>, leverage the latest HTML5 technology to be efficient and, best of all, are created within the Javascript code as oppose to being shoved in to an HTML file and demand DOM tinkering later. But a closer look would reveal a lot more goodies beyond those handy components in Sencha Touch. It offers <a href="https://web.archive.org/web/20130117172701/http://docs.sencha.com/touch/2-1/#!/guide/class_system">a class system</a>, a <a href="https://web.archive.org/web/20130120111258/http://www.sencha.com/learn/architecting-your-app-in-ext-js-4-part-1/">M</a><a href="https://web.archive.org/web/20130106185309/http://www.sencha.com/learn/architecting-your-app-in-ext-js-4-part-2">V</a><a href="https://web.archive.org/web/20130124092547/http://www.sencha.com/learn/architecting-your-app-in-ext-js-4-part-3/">C</a><a href="https://web.archive.org/web/20130109001131/http://www.sencha.com/blog/architecting-your-app-with-sencha-touch-2-mvc/">pattern</a>, tools that handles code dependency, compression and native packaging, etc. Albeit daunting, learning and embracing all of those offerings makes a quite enjoyable coding experience and rewards me with development effieciency overall. Sometimes though, I need complete control of a visual component that doesn’t exist in the framework. How to make this work with everything mentioned above is implied in various tutorials posted by the Sencha team, but I couldn’t find a clear illustration of that, which is why I decided to write one here. Here’s our goal: make a component that displays data from a model with our own custom HTML; manage it along with some provided components from the framework and follow the MVC pattern. I assume you have the basics set up. I’m using Sencha Cmd 3.0.0.250, Sencha Touch SDK 2.1.0 and OS X Mountain Lion as of this writing. Let’s start by generating a MVC-ready skeleton project called HelloWorld. Go to the SDK’s folder and type: <pre><code> sencha generate app HelloWorld ~/helloworld </code></pre> replace the last parameter with the path you would like for the project files to stay. We’ll be working in this folder from now on. In the skeleton project, a main view was created under <code>app/view/Main.js</code> and declared as dependency for <code>app.js</code>. An instance of it is created when the app finished loading. We’ll keep this setup as our main view. Let’s reduce<code>app/view/Main.js</code> to a simplest possble form for our purposes: <pre><code> Ext.define('HelloWorld.view.Main', { extend: 'Ext.Container', config: { items: [ { xtype: 'helloview' } ] } }); </code></pre> Following the convention, we name the class in corrispondance with its filepath within the <code>app</code> folder (‘view/Main.js’ => ‘view.Main’). All classes created with <code>Ext.define</code> should follow this convention so that Sencha tools can relate code dependencies to file structure and do its magic for us. We’ll circle back to this. Our main view will be a plain container and has a <code>helloview</code> in it. An<code>Ext.Container</code> has the ability to … contain stuff. Specifically, it can organize <code>Ext.Component</code>s visually. <code>helloview</code> will be that component. Let’s define it next. Create the file <code>app/view/HelloView.js</code> as the following: <pre><code> Ext.define('HelloWorld.view.HelloView', { extend: 'Ext.Component', xtype: 'helloview', config: { tpl: '<div class="greeting">Hello, {name}!</div>' } }); </code></pre> As promised, <code>HelloWorld.view.HelloView</code> is a <code>Ext.Component</code>. We declare its<code>xtype</code> to be the one we used in the main view. The really interesting part is its <code>tpl</code> configuration. This is where our customization integrates with the rest of the Sencha Touch framework, so it’s worth dive into a bit deeper. (Caution: <code>tpl</code> can not be mixed with <code>items</code>, if you want standard components mixed in with yours, make them share a container and arrange a proper layout there.) Take a look at the official <a href="https://web.archive.org/web/20130117172701/http://docs.sencha.com/touch/2-1/#!/api/Ext.Component-cfg-tpl">documentation for <code>tpl</code></a>. It accepts an<a href="https://web.archive.org/web/20130117172701/http://docs.sencha.com/touch/2-1/#!/api/Ext.XTemplate"><code>Ext.XTemplate</code></a>, which is basically free-range HTML plus some syntax to insert data provided to the component. Apply all your HTML skills here! For illustration purposes, we only throw in a basic <code><div></code>. The <code>{name}</code> part will be replaced by the actual data, which every<code>Ext.Component</code> has as a configuration option by default. Being such option means two things: <pre><code>1. You can specify its value in the class definition, as we did for `tpl`. 2. It gets a "getter/setter" that let you query/change its value at anytime through the instance's existence. </code></pre> We’ll use the setter for <code>data</code> – <code>setData()</code> to populate this field in the template later. <code>setData()</code> accept one raw Javascript object and make its properties accessible to the template. Next, let’s make a simplistic model in <code>app/model/Greetee.js</code>: <pre><code> Ext.define('HelloWorld.model.Greetee', { extend: 'Ext.data.Model', config: { fields: ['name'] } }); </code></pre> A model, with a field ‘name’. Hey, that’s <code>as simple as possible, but no simpler</code>, Einstein would endorse it! We won’t use any proxy or store in conjunction with the model because we only need to show how a customized view works within the Sencha Touch MVC pattern. Speaking of which, a controller does just that. So to tie everything togeter, here’s <code>app/controller/Main.js</code>: <pre><code> Ext.define('HelloWorld.controller.Main', { extend: 'Ext.app.Controller', config: { models: ['Greetee'], views: ['HelloView'], refs: { helloView: '.helloview' } }, launch: function() { var m = Ext.create('HelloWorld.model.Greetee', { name: 'World' }); return this.getHelloView().setData(m.getData()); } }); </code></pre> To get reference to the view components, Sencha Touch provide <code>refs</code> in controllers, through which we map <code>hello</code> to our customized component’s xtype. There are more details about this in the <a href="https://web.archive.org/web/20130117172701/http://docs.sencha.com/touch/2-1/#!/guide/controllers">offical documentation</a>. We now can use <code>getHelloView()</code> in other methods. <code>launch()</code> gets invoked after everything gets loaded, and we tie the model and the view together here. Again, a little imagination might help. By that I mean the data source of the model could be from a RESTful network API, a picture from a phone camera via Phonegap, a record from browser’s localstorage, etc. We simply created one in memory for illustration. We use <code>getHelloView()</code> to get reference to our <code>helloview</code> instance, then use its <code>setData()</code> to populate its template field. But we can’t pass in the model object directly (as mentioned above, a raw Javascript object is needed), so we convert it with its <code>getData()</code> method. Finally, we specify the <code>models</code> and <code>views</code> involved with this controller in<code>config</code> so that the files of these classes gets loaded properly. For the same purpose, we need to open <code>app.js</code> and add the following line into the object passed to <code>Ext.application()</code> (which is a controller too): <pre><code> controllers: ['Main'], </code></pre> This makes the framework aware of our <code>HelloWorld.controller.Main</code>. It’s unnecessary to use the full class name becaue we followed the naming convention. At this point, our code is complete. Go to the project folder in terminal and fire up a web server: <pre><code> python -m SimpleHTTPServer </code></pre> Open <a href="http://localhost:8000">http://localhost:8000</a> in your browser and take a gander! A screenshot of this app would be an overkill, if you follow along correctly, the phrase “Hello, World!” will show, and that’s all. So this long-winded excercise results in not much. But I hope you won’t find it pointless. When I try to construct my app UI with Sencha Touch, I first try my best to make the Senche component work with the design. When breaking out and customize is inevitable, I try to stay within the framwork as much as possible to make the most out of it. What’s described in this article is a common way to do that. Fri, 11 Jan 2013 00:00:00 -0700 https://duan.ca/2013/01/11/hello-world-the-hard-way-with-sencha-touch/ https://duan.ca/2013/01/11/hello-world-the-hard-way-with-sencha-touch/ MongoDB, PHP And Mountain Lion Here are the necessary steps to make PHP work with MongoDB on vanilla Mac OS X 10.8 (Mountain Lion): <ol> <li> Install <a href="https://developer.apple.com/downloads/index.action">Command Lion Tools for Xcode</a> </li> <li> Install Homebrew: <pre><code>ruby -e "$(curl -fsSkL raw.github.com/mxcl/homebrew/go)" </code></pre> </li> <li> Prioritize Homebrew's binaries in $PATH: <pre><code>echo "export /usr/local/bin:$PATH" >> ~/.bash_profile </code></pre> </li> <li> Install PHP 5.4 with Homebrew: <pre><code># Setup the tap for dependencies brew tap homebrew/dupes # This is the best brew tap josegonzalez/homebrew-php # And install brew install php54 </code></pre> </li> <li> Install MongoDB with Homebrew <pre><code>brew install mongodb </code></pre> </li> <li> Install "PHP Driver" for MongoDB: <pre><code>sudo pecl install mongo </code></pre> </li> <li> Done. </li> </ol> Now go run some PHP-MongoDB scripts with the build-in web server from PHP 5.4 like you normally would (don't forget to start <code>mongod</code>) and everything should work. Thu, 29 Nov 2012 09:37:30 -0600 https://duan.ca/2012/11/29/mongodb-php-and-mountain-lion/ https://duan.ca/2012/11/29/mongodb-php-and-mountain-lion/ CoffeeScript And Mountain Lion Here are the necessary steps to install CoffeeScript on Mac OS X 10.8 (Mountain Lion): <ol> <li> Install <a href="https://developer.apple.com/downloads/index.action">Command Lion Tools for Xcode</a> </li> <li> Install Homebrew: <pre><code>ruby -e "$(curl -fsSkL raw.github.com/mxcl/homebrew/go)" </code></pre> </li> <li> Install Node.js with Homebrew: <pre><code>brew install node </code></pre> </li> <li> Install npm (the version comes with node doesn't install CoffeeScript properly for some reason): <pre><code>curl https://npmjs.org/install.sh | sh </code></pre> </li> <li> Install CoffeeScript with npm: <pre><code>npm install -g coffee-script # add environment variable to ~/.bashrc echo "export NODE_PATH=/usr/local/lib/node_modules" . ~/.bashrc </code></pre> </li> <li> Done! </li> </ol> I had to google around to figure out the environment variable and the issue with npm. Hopefully this will help some folks in the similar need. Wed, 28 Nov 2012 15:11:00 -0600 https://duan.ca/2012/11/28/coffeescript-and-mountain-lion/ https://duan.ca/2012/11/28/coffeescript-and-mountain-lion/ Keep Git Social A project I'm working on uses <a href="http://about.gitlab.com">GitlabHQ</a>, I think it is a cool open source project and the developers are doing an excellent job of making a sweet web interface for Git projects. However, as I found out today, <a href="https://github.com/gitlabhq/gitlabhq/issues/1298" title="Gitlab Issue: Closed">my one issue</a> with it was closed without fixing. And that's what finally propelled me to writing this airticle. Do you know, the Linux kernel is broken by hundreds of commited patches everyday? That, of course, is a trick question. Yes, technically, hundreds (if not thousands) of patches were commited to a git repository of the Linux kernel that isn't different in anyway from the repository. What makes the repository "conanical" is the fact that Linus Torvalds owns it. If the world lost access to his hard drive because Linus was hit by a bus (I would never wish this to happen) today, all the Linux community need to do is figure out who is next BDFL. Then his/hers would become the Linux repository. In other words, what makes the difference is purely social. Not trying to fit the social structure of a software project into a repository is the biggest strength of git. This is why I frown everytime I see a discussion about locking down branches of a Git repository. Making developers "own" branches on a shared repository is a pratice inheritated from the days of centralized version control. Git doesn't provide built-in support for this, for good reasons. With Subversion, the ultimate goal of branch-wise access control is to keep out bad changes made by developers while give them the benefits of version control, namely: <ol> <li>provide a backup of the change history</li> <li>let others to follow his/her progress so that they can contribute via patches or collaborate if they have write access.</li> </ol> With Git, these are easily achieved by a personal repository read-accessible for the team. Meanwhile, write access is granted at the repository level, but only to very few or, more preferably, one person. This is possible because each team member can have and only push to his/her own repositories. No change made by others will make their way to the "conanical" repository unless its owner(s) activaly pulls them in. Bad changes therefore is kept out. Additionally, Git has <a href="http://git-scm.com/book/en/Git-Branching">features</a> flexible enough to support all kinds of <a href="http://nvie.com/posts/a-successful-git-branching-model">development process</a>es, partially because its branches, by design, are not responsible for enforcing access permissions. Consider developer John working on a shared git repository with locked branches: where would he push his temporary branch to for backup, if he can only write to <code>feature42</code> and <code>johns_branch</code>? How does he get <code>emergency_bugfix_14159</code> pair reviewed at 4am? If he uses his own repo for those, does it mean he has to force the whole team to know about it? How does it affect work when a <code>git branch --all</code> produces a phone book? Break Git's design gets you no where nice. And no, Git's social model doesn't add work to the owner of the "official" repository. Afterall, Git was designed with he linux kernel in mind! Linus himself explains it the best (summary provided below): <div class="video-container"> <iframe src="http://www.youtube.com/embed/4XpnKHJAok8" frameborder="0" allowfullscreen></iframe> </div> In short, Linus only review pull requests from a few "lieutenants" he trusts, and they each follow the same process with their trusted few. And the pyramid trickles all the way down. Here agian, Git solves a problem by getting out of the way of the project's social stucture, instead of trying to encapsulate it. Git was the first version control system I've ever used starting in 2008. I've since gradually realized that not everyone is lucky like me, in the sense that svn has been working just fine for a lot of people. When their project switches to Git, emulating the svn/old workflow with functionality provided by projects like <code>gitolite</code> is only natural. But if you are one of them, and want more creativity from your team, perhaps embracing the social aspect of Git by breaking the shackles on their hand is a good thing to try. Wed, 31 Oct 2012 10:02:30 -0600 https://duan.ca/2012/10/31/keep-git-social/ https://duan.ca/2012/10/31/keep-git-social/ Windows 8 First Impression I use KDE on Arch Linux for 80% of the time, OS X the other 20%. But I got a copy of Windows 8 Pro for free as student so I decided to install it along with Visual Studio 2012 Ultimate so that if I ever need to develop on any Microsoft platforms, I'll have the environment ready. I jogged down these impression after about a 30 minutes trial. What I didn't like: <ul> <li>I have to use a .iso file the hardway (in my case, burning it to an usb drive)even if I only wanted to upgrade from Windows 7.</li> <li>When upgrading from Windows 7, only the free space is counted as usable for the new system, which means upgrading is not an option for I have a mere 26G partition reserved for Windows.</li> <li>The old Mobile ATI graphic card driver stopped working after installation.</li> <li>Apps don't share same session between the tile and deskop interface.</li> <li>Contacts from different accounts (in my case, facebook and twitter) are mixed together and there doesn't seem to be an easy way to filter them.</li> <li>I signed in with a Mircrosoft account for the operating system. But when I started Visual Studio, it asked me to sign in with the same account.</li> </ul> What I did like: <ul> <li>The tile interface looked nice.</li> <li>IE10 is fast. Looked particularly smooth under the tile interface.</li> <li>For an old machine I installed it on, the whole system is very responsive.</li> <li>The tile interface is not unintuitive. When in doubt, one of these actions always gets me where I wanted: <ol> <li>Mouse to upper-right corner.</li> <li>Mouse to upper-left corner and then move down.</li> <li>Press the Win key on keyboard.</li> </ol> </li> </ul> I'm not going to speculate on whether the general public will love or hate this version of Windows. But one thing is for sure: there is going to be an reaction. I can see it go both ways. Sat, 27 Oct 2012 23:59:59 -0600 https://duan.ca/2012/10/27/windows-8-first-impression/ https://duan.ca/2012/10/27/windows-8-first-impression/ Testing Method Within Constructor With Jasmine.js How do you test whether a method is called witin the constructor of an object with Jasmine.js? Turns out, you need to <code>spyOn()</code> the "raw" reference of the "method", which really just a function on the <code>prototype</code> of your "class" object. An example will make it clear: <pre><code>// is 'load()' getting called during construction? describe("MyObject", function() { it("should call load() during construction", function() { spyOn(MyObject.prototype, 'load'); new MyObject(); expect(MyObject.prototype.load).toHaveBeenCalled(); }); } </code></pre> Sun, 30 Sep 2012 00:00:00 -0600 https://duan.ca/2012/09/30/testing-method-within-constructor-with-jasminejs/ https://duan.ca/2012/09/30/testing-method-within-constructor-with-jasminejs/ Integrating Sencha Touch 2 and Cordova (pre-2.0) <h2>"The Goals</h2> <ol> <li> Use Sencha Command to create and package an MVC-structured Sencha Touch 2 project. </li> <li> Use Cordova/PhoneGap (1.9) to wrap the project to deliver it in the App Store. </li> <li> Leverage the APIs provided by Cordova/PhoneGap to give Sencha Touch 2 project more native capabilities. </li> </ol> <h2>The Process</h2> Goal #1 and #2 can be achieved by <a href="http://robertdougan.com/posts/packaging-sencha-touch-2-with-phonegap-cordova" title="Packaging Sencha Touch 2 with PhoneGap (Cordova)">these steps well described by Robert Dougan </a>. In essence, you need to create a normal Sencha Touch 2 project using Sencha Command; a Cordova (1.7 in Roberts post, but works fine with 1.9) project. Then tell Sencha Command to build in the <code>www</code> folder, where Cordova looks for the HTML5 assets to package. Finally, build and deploy the Cordova project the usual way. That, of course, is not the end of the story, otherwise you wouldn't be reading this. When you try to achive goal #3, that is, when you use the Cordova API in the Sencha Touch project, such as: <pre><code> navigator.notification.alert( 'Winter is coming!', noted, // callback function 'be warned', ""Yes M'lord"" ); </code></pre> Sencha Command will report error and refuse to ""compile"" if you run <pre><code> sencha app build package </code></pre> because of the unknown namespace introduced by Cordova. One way to workaround this problem is manually replacing the command, which involves compiling the SASS files, consolidate all Javascript dependencies into a single file, minify everything and move the result to the <code>www</code> folder. Doesn't sound fun, does it? So, here's a trick to put Sencha Command back on track, use <pre><code> if (!Ext.os.is.Desktop) { } </code></pre> to wrap around the Cordova API calls, so the previous example becomes <pre><code> if (!Ext.os.is.Desktop) { navigator.notification.alert( // ... ); } </code></pre> This way, Sencha Command will ignore the conditioned code block since its running on the desktop environment, therefore compiles like a charm. And the code will work as intended in emulator/device environment. You are welcome. <h2>The Bonus</h2> Now that you are using Sencha Touch 2 and Cordova API (like a boss!), here's another tip to improve your workflow. When I'm developing hybrid apps, I spend the majority of time editing Java(Coffee)script/(S)CSS files. But surely enough, there will come a period where I have to debug the packaged app in the emulator. Switching from my text editor (Vim) to Xcode and hitting ⌘r REALLY gets old. Luckily, Cordova provides some command line alternatives. If you generate a Cordova project as described in <a href="https://web.archive.org/web/20161025042756/http://robertdougan.com/posts/packaging-sencha-touch-2-with-phonegap-cordova" title="Cordova Command-Line Documentaion">Cordova's documentaion</a> like so: <pre><code>./path/to/cordova-ios/bin/create /path/to/project com.example.name Project </code></pre> There'll be a folder called <code>cordova</code> in the newly created project, which won't be in your project if you create it with Xcode with <a href="http://robertdougan.com/posts/packaging-sencha-touch-2-with-phonegap-cordova" title="Packaging Sencha Touch 2 with PhoneGap (Cordova)">Robert's method</a>. In <code>cordova</code> folder are three lovely Bash scripts that let's you compile the Xcode project (<code>debug</code>, which also does the next thing), run the result in an ios emulator (<code>emulate</code>) and watch the logging information (<code>log</code>). Copy the folder to the path of <code>YourProject.xcodeproj</code>, and boom! You can now quit Xcode and run the scripts in there directly from your terminal. (What I like to do is to use GNU Make to combine Sencha commands with those scripts so that I could just hit ⌘b in MacVim to make my latest code run in the simulator.) Happy coding! Mon, 20 Aug 2012 19:09:34 -0600 https://duan.ca/2012/08/20/integrating-sencha-touch-2-and-cordova-pre-20/ https://duan.ca/2012/08/20/integrating-sencha-touch-2-and-cordova-pre-20/