# rss_item

guid: https://strongly-typed-thoughts.net/blog/al_0.1.1.2Tue, 03 Mar 2015 00:00:00 GMT
title: al 0.1.1.2 was shipped
author: Dimitri Sabadie <dimitri.sabadie@gmail.com>
url: https://strongly-typed-thoughts.net/blog/al_0.1.1.2
feedurl: https://phaazon.net/blog/feed
content:
<h1>al 0.1.1.2</h1>
<h2>Changes</h2>
<p><a href="http://hackage.haskell.org/package/al-0.1.1.2">al 0.1.1.2</a> was shipped. It
includes several improvements, among them:</p>
<ul>
<li>fix CPU architecture issues ;</li>
<li>since version 0.1.1, the <code>stdcall</code> flag is available ;</li>
<li><code>alcIsExtensionSupported</code> was renamed to <code>alcIsExtensionPresent</code> as the
former just doesn’t exist – sorry for the typo.</li>
</ul>
<p>The <code>stdcall</code> flag might be a great ally for people compiling on a 32-bit
Windows. For people on 64-bit Windows, the default is sufficient – and for
UNIX systems, you don’t have anything special to do.</p>
<p>I’m also looking for hackers to test the library on the most OS as possible. I
have issues with it (see <a href="http://stackoverflow.com/questions/28829976/ffi-and-static-libraries-used-in-application">this</a>)
and I’d like to hear from people. Even though I have that issue, al compiles
well and runs great in <strong>ghci</strong>, which is weird regarding the fact running an
application compiled with al silently crashes at startup.</p>
<h2>About paths…</h2>
<p>That’s a nasty issue I don’t really know how to correctly fix. Up to now,
default OpenAL 1.1 SDK installation are detected for Windows. For people with
custom installation and other systems, <strong>you have to pass the path of the SDK by
hand</strong>. I know, it’s a pain in the ass, but I don’t want to depend on tools like
<code>pkg-config</code> as that tool is not available everywhere – getting it on Windows is
not that simple.</p>


guid: https://strongly-typed-thoughts.net/blog/aoc-21-day-16Fri, 24 Dec 2021 14:54:00 GMT
title: Optimizing Advent of Code 2021 Day 16 part 1 & 2
author: Dimitri Sabadie <dimitri.sabadie@gmail.com>
url: https://strongly-typed-thoughts.net/blog/aoc-21-day-16
feedurl: https://phaazon.net/blog/feed
content:
<p>I have been doing <a href="https://adventofcode.com">Advent of Code 2021</a>, like every year since 2018, and I would like to share
some reflections about it. This year, I would like to talk about the implementation I made for <a href="https://adventofcode.com/2021/day/16">day
16</a>.</p>
<!-- vim-markdown-toc GFM -->
<ul>
<li><a href="#----day-16-packet-decoder------reflections">--- Day 16: Packet Decoder --- // reflections</a>
<ul>
<li><a href="#header-parsing">Header parsing</a></li>
<li><a href="#the-encoding-problem">The encoding problem</a></li>
<li><a href="#the-other-encoding">The other encoding</a></li>
<li><a href="#bit-packing-solution">Bit packing solution</a></li>
<li><a href="#packed-next_bits">Packed next_bits</a></li>
<li><a href="#decoding-a-literal-packet">Decoding a literal packet</a></li>
<li><a href="#parsing-operators">Parsing operators</a></li>
<li><a href="#part-1-summing-packet-headers-versions">Part 1: summing packet headers’ versions</a></li>
<li><a href="#part-2-evaluating-the-expression">Part 2: evaluating the expression</a></li>
</ul>
</li>
</ul>
<!-- vim-markdown-toc -->
<h1>--- Day 16: Packet Decoder --- // reflections</h1>
<p>This problem doesn’t look too hard at the first glance, but has some hidden complexities than I thought pretty
interesting. Just to sum up if you don’t want to open the link, the first part requires you to parse a hexadecimal
number and interpret its binary representation as a <em>packet</em>, like what you would find in a network packet format,
basically. You need to be able to parse correctly a single packet, which can contain nested packets. In this puzzle,
packets represent expressions, that can either be:</p>
<ul>
<li>A literal number.</li>
<li>An operation between packets.</li>
</ul>
<p>Each packet have a header, comprising a header and its type (literal or operator), and then the actual data of the
packet.</p>
<p>From this context, it seems pretty obvious to assume this is going to be a parsing problem. The first part asks you to
parse the packet and its nested packets, and add all the version numbers of the headers of all packets. The format goes
as this (quoting):</p>
<blockquote>
<p>Every packet begins with a standard header: the first three bits encode the packet <code>version</code>, and the next three bits
encode the packet <code>type ID</code>. These two values are numbers; all numbers encoded in any packet are represented as binary
with the most significant bit first. For example, a version encoded as the binary sequence 100 represents the number
4.</p>
</blockquote>
<p>Then, you have the description of a packet representing a literal number:</p>
<blockquote>
<p>Packets with <code>type ID 4</code> represent a literal value. Literal value packets encode a single binary number. To do this,
the binary number is padded with leading zeroes until its length is a multiple of four bits, and then it is broken
into groups of four bits. Each group is prefixed by a 1 bit except the last group, which is prefixed by a 0 bit.
These groups of five bits immediately follow the packet header. For example, the hexadecimal string <code>D2FE28</code> becomes:</p>
</blockquote>
<pre><code>110100101111111000101000
VVVTTTAAAAABBBBBCCCCC
</code></pre>
<blockquote>
<p>Below each bit is a label indicating its purpose:</p>
<ul>
<li>The three bits labeled <code>V</code> (<code>110</code>) are the packet <code>version</code>, <code>6</code>.</li>
<li>The three bits labeled <code>T</code> (<code>100</code>) are the packet <code>type ID</code>, <code>4</code>, which means the packet is a literal value.</li>
<li>The five bits labeled <code>A</code> (<code>10111</code>) start with a <code>1</code> (not the last group, keep reading) and contain the first four bits of the number, <code>0111</code>.</li>
<li>The five bits labeled <code>B</code> (<code>11110</code>) start with a <code>1</code> (not the last group, keep reading) and contain four more bits of the number, <code>1110</code>.</li>
<li>The five bits labeled <code>C</code> (<code>00101</code>) start with a <code>0</code> (last group, end of packet) and contain the last four bits of the number, <code>0101</code>.</li>
<li>The three unlabeled <code>0</code> bits at the end are extra due to the hexadecimal representation and should be ignored.</li>
</ul>
<p>So, this packet represents a literal value with binary representation <code>011111100101</code>, which is <code>2021</code> in decimal.</p>
</blockquote>
<p>I’m just going to focus on this first kind of packet because it’s already interesting.</p>
<h2>Header parsing</h2>
<p>From this description, if you have ever done bit parsing, you should already know that there is going to be a problem.
You can parse a literal packet using this kind of pseudo code (ignoring errors for simplicity, it’s AoC!):</p>
<pre><code class="language-rust">fn parse_packet(&amp;mut self) -&gt; Packet {
  let version = self.next_bits(3);
  let type_id = self.next_bits(3);

  match type_id {
    4 =&gt; todo!(&quot;literal parser&quot;),
    _ =&gt; todo!(&quot;operator parser&quot;),
  }
}
</code></pre>
<p>The big question is: how do we implement <code>next_bits</code>? Getting 3 bits of information? As you might know, the smallest
amount of information a computer can manipulate is a <em>byte</em>, which is 8 bits, so… how do we manipulate less than that?</p>
<h2>The encoding problem</h2>
<p>I would like to say one thing before going on. The input is a hexadecimal string. The first step of your solution is to
extract the binary representation. For instance, <code>A</code> (10 in decimal) is encoded as <code>1010</code> and <code>5A</code> is encoded as
<code>01011010</code>. As you can see (might already know), hexadecimal is easy to parse into bytes: group hexadecimal digits by
two, convert each digit to 4 bits, and glue them together. <code>5A</code> is <code>5 -&gt; 0101</code> and <code>A -&gt; 1010</code>, which gives <code>01011010</code>
as seen above.</p>
<p>But as we have seen with <code>next_bits</code> above, we will want to manipulate bits, not bytes. So you have several choices and
design decisions to make here:</p>
<p>You can go the « easy » way and store each bit as a byte. This is very suboptimal, because you are going to waste 8
times the amount of memory you actually need to store all this. For instance, <code>5A</code> (<code>01011010</code>), which is a single byte
on your computer, will require 8 bytes to be stored: <code>00000000, 00000001, 00000000, 00000001, 00000001, 00000000, 00000001, 00000000</code>. That is very inefficient, but it would work, if you put them in a <code>Vec&lt;u8&gt;</code>. To do that, you would
basically need to read one hexadecimal digit at a time (which has 4 bits), and then perform bitwise right shifts with
masking (<code>&amp; 0x1</code>) to extract the bit. Something like:</p>
<pre><code class="language-rust">fn inefficient_bit_extract(&amp;mut self, four_bits: u8) {
  for i in 0..4 {
    self.bits.push((four_bits &gt;&gt; i) &amp; 0x1);
  }
}
</code></pre>
<p>With this solution, implementing <code>next_bits</code> is easier, because you only need to <code>pop</code> the <code>Vec&lt;u8&gt;</code>. Let’s write
<code>next_bits</code> with this kind of implementation. Because we do not need more than 15 bits in this puzzle (explained later in
the puzzle text, not really important for now), we will return the read bits on <code>u16</code>:</p>
<pre><code class="language-rust">fn next_bits(&amp;mut self, n: usize) -&gt; u16 {
  assert!(n &lt; 16);

  let mut out = 0;

  for i in 0..n {
    out = (out &lt;&lt; i) | self.bits.pop() as u16;
  }

  out
}
</code></pre>
<p>Let’s explain quickly.</p>
<pre><code class="language-rust">    out = (out &lt;&lt; i) | self.bits.pop() as u16;
</code></pre>
<p>This will extract a single bit, like <code>00000001</code>, will convert it to 16-bit (so <code>0000000000000001</code>) and will <code>OR</code> it to
the number we are building. That number is left shifted at each iteration so that we only <code>OR</code> the least significant bit
at each iteration. For instance, if we call <code>next_bits(3)</code> and we get <code>101</code>, this implementation will basically store
this in <code>out</code> at each iteration:</p>
<ul>
<li>End of 1st iteration: <code>0000000000000001</code>.</li>
<li>End of 2nd iteration: <code>0000000000000010</code>.</li>
<li>End of 3nd iteration: <code>0000000000000101</code>.</li>
</ul>
<p>Easy, and works.</p>
<h2>The other encoding</h2>
<p>However, I am a perfectionist. This is not satisfying to me, and it has been years since I have been wanting to
implement something like what I am about to describe. I have been thinking about this problem with a different context
but it is similar: a <code>bool</code> value, in a computer, is stored on the minimal amount of data a computer can manipulate:
a byte. It means than storing a boolean value wastes 7 bits. If you have a single boolean value, there is nothing you
can do. But if you need several boolean values, like, six booleans… using 6 bytes still wastes 42 bits for 6 bits of
useful data! And 6 bits can hold in a single byte. So we should be able to store booleans in a packed array of bytes.
The idea is the following: imagine that you want to store six boolean value in a byte:</p>
<pre><code>00100101
ABCDEFGH
</code></pre>
<p>I have labelled each bit with a letter to make it easier to reference them. If you want to, for instance, the boolean
value stored at index 2 (<code>F</code> in our representation), all you have to do is to right shift that number using the index of
the element and 1-bit mask:</p>
<pre><code class="language-rust">fn read_boolean(byte: u8, index: usize) -&gt; u8 {
  assert!(index &lt; 8);

  (byte &gt;&gt; index) &amp; 0x1
}
</code></pre>
<p>If you want to set it, you need to do the opposite operation using a logical <code>OR</code>:</p>
<pre><code class="language-rust">fn set_boolean(byte: &amp;mut u8, index: usize) {
  *byte = *byte | (0x1 &lt;&lt; index);
}
</code></pre>
<p>I wanted to use this AoC 18 day to encode my solution as a packed array of bits, and still have the <code>next_bits</code>
interface that would allow me to get between 0 and 15 bits of data, encoded as <code>u16</code>. The catch is: because I bit pack
the hexadecimal number, it is going to lie in a <code>Vec&lt;u8&gt;</code>. And the whole complexity relies in the fact asking for <code>N</code>
bits can span across several bytes in the packed array…</p>
<h2>Bit packing solution</h2>
<p>The first thing we need to do is to think about the interface. I want that mutable <code>next_bits</code> interface. What it means
is that I want to be able to read as many bits as I want (given less than 16 bits), I should not have to worry about the
byte implications behind the scenes. The puzzle text clearly shows that when we are going to parse a literal value, we
will have to read groups of 5 bits. Reading two literal numbers already implies two bytes (10 bits), and even the first
group implies reading two bytes, because before that, you have the header on 6 bits (3 bits for the version, 3 bits for
the type ID), living 2 bits to read for the beginning of the group, and 3 bits in the next byte to read. That’s a lot of
complexity that should be hidden from the interface.</p>
<p>Let’s start with the <code>Decoder</code> type:</p>
<pre><code class="language-rust">#[derive(Clone, Debug)]
struct Decoder {
  bytes: Vec&lt;u8&gt;,
  bitpos: usize, // the position in bit in the byte array
}
</code></pre>
<p>This is fairly simple: <code>bytes</code> is the bit-packed array I mentioned, and <code>bitpos</code> is the current position in that array.
But it’s not the position in the <em>array elements</em>, it’s the position in <em>bits</em>. If <code>bytes</code> has 3 bytes (24 bits), then
<code>bitpos</code> can be between <code>0</code> and <code>23</code>. To know in which actual array element the bit is, we simply need to divide by 8.
For instance, if <code>bitpos = 17</code>, then <code>17 / 8 = 2</code> tells us that the bit is in <code>bytes[2]</code>. To know the actual bit
position in that byte, you will have guessed it, you need to take the modulo: <code>17 % 8 = 1</code>, which is the second most
significant bit. You can convert to least significant bit by subtracting that number to 8: <code>8 - 1 = 7</code>.</p>
<p>With all that information, let’s fill the bytes:</p>
<pre><code class="language-rust">impl Decoder {
  fn new(input: &amp;str) -&gt; Self {
    let digits: Vec&lt;_&gt; = input.chars().flat_map(|hexa| hexa.to_digit(16)).collect();
    let bytes = digits.chunks(2).map(|d| (d[0] &lt;&lt; 4 | d[1]) as u8).collect();

    Self { bytes, bitpos: 0 }
  }
</code></pre>
<p>Here, because this is AoC, I don’t care about errors, and hence use <code>flat_map</code> to remove <code>Option</code> or <code>Result</code> in
iterators. The important thing here is that I group hexadecimal digits by 2 (the <code>.chunks(2)</code> part), and then simply do
this:</p>
<pre><code class="language-rust">|d| (d[0] &lt;&lt; 4 | d[1]) as u8
</code></pre>
<p><code>d[0]</code> are the most significant 4 bits, so I left shift them by 4 bits, and <code>d[1]</code> is already correctly positioned, so
nothing to do with it. I convert to <code>u8</code> parse <code>to_digit(16)</code> yields <code>u32</code>.</p>
<p>Then, let’s implement <code>next_bits</code>.</p>
<h2>Packed next_bits</h2>
<pre><code class="language-rust">  fn next_bits(&amp;mut self, n: usize) -&gt; u16 {
    // how many bits are still available from the current byte
    //
    // 0000000011111111
    //      ^ position is 5, so we have 3 bits still available
    let bits_avail = 8 - self.bitpos % 8;

    if bits_avail &gt;= n {
      // we have enough in the current byte; compute the new position and right shift to align correctly
      let shift = bits_avail - n;
      let byte = (self.bytes[self.bitpos / 8] as u16 &gt;&gt; shift) &amp; ((1 &lt;&lt; n) - 1);
      self.bitpos += n;
      byte
    } else {
      // …
    }
</code></pre>
<p>The first step to do is to check whether the amount of bits requested would make us overlap with another byte given the
current <code>bitpos</code>. If it’s not the case, then the problem is trivial. As shown in the example, we can just simply right
shift by the number of available bits minus the requested amount (because we might still have room afterwards), and mask
the N least significant bits. By the way, this is a funny one: setting to 1 the N significant bits. If you have <code>N = 3</code>,
you want (on 8-bit here to simplify) <code>00000111</code>. How do you make that quickly? It’s actually quite simple: what is this
number? This <code>111</code>? It’s an odd number (because its least significant bit is <code>1</code>) and it looks like it’s almost 2³. It’s
actually 2³ - 1. And this makes sense: remember in school / when you were learning binary that the biggest number you
can represent on N bits is <code>2^n - 1</code>. Like, on 8-bit, 2⁸ is 256, and the biggest number you can represent is 255, as you
might already know, which is <code>11111111</code>. So, we can simply deduce <code>00111…1</code> is the biggest number for 2^N where N is the
number of <code>1</code>. And that number is <code>2^N - 1</code>. A power of 2, in binary, is a simple shift. So, <code>2^N</code> is <code>1 &lt;&lt; N</code>. And `2^N</p>
<ul>
<li>1<code>is…</code>(1 &lt;&lt; N) - 1`. You have it.</li>
</ul>
<p>We then increment <code>bitpos</code> by the number of read bits, and return the actual byte, for which the read bits are right
shifted.</p>
<p>Now, what happens when <code>bits_avail &lt; n</code>? We have to read <code>bits_avail</code> and shift them in the 16-bit output. Then we will
have to switch to the next byte, and read <code>n - bits_avail</code> additional bits, because <code>n</code> is the requested number of bits
to read and since we read every bits available from the current bytes, well, <code>n - bits_avail</code>. The « byte switching »
logic can be a bit overwhelming at first but is actually not that hard: once we have read <code>bits_avail</code>, we know that the
next byte will have 8 bits available. We can then divide the problem in two sub-problems, easier:</p>
<ol>
<li>Read and shift 8 bits of data as long as <code>n &gt;= 8</code>.</li>
<li>Read the remaining bits in the last byte.</li>
</ol>
<p>This logic is implemented in the <code>} else {</code> branch, so I will just copy the <code>else</code> and put the code, and explain it:</p>
<pre><code class="language-rust">    } else {
      // this is the annoying part; first, get all the bits from the current byte
      let byte0 = (self.bytes[self.bitpos / 8] as u16) &amp; ((1 &lt;&lt; bits_avail) - 1);
      self.bitpos += bits_avail;

      // then, left shift those bits as most significant bits for the final number
      let mut final_byte = byte0;

      // then, we need to get the remaining bits; compute the number of bytes we need
      let n = n - bits_avail;
      let bytes_needed = n / 8;

      // while we require more than 1 byte, the problem is trivial; just copy and left shift the bytes
      for _ in 1..=bytes_needed {
        let byte = self.bytes[self.bitpos / 8] as u16;
        self.bitpos += 8;
        final_byte = (final_byte &lt;&lt; 8) | byte;
      }

      // the remaining bits can be extracted from the last byte
      let n = n % 8;
      let shift = 8 - n;
      let byte = (self.bytes[self.bitpos / 8] as u16 &gt;&gt; shift) &amp; ((1 &lt;&lt; n) - 1);
      self.bitpos += n;

      final_byte = (final_byte &lt;&lt; n) | byte;

      final_byte
    }
  }
</code></pre>
<p>Let’s dive in.</p>
<pre><code class="language-rust">      let byte0 = (self.bytes[self.bitpos / 8] as u16) &amp; ((1 &lt;&lt; bits_avail) - 1);
      self.bitpos += bits_avail;
</code></pre>
<p>This is the current byte, from which, as said earlier, we need to read all available bits. Because we are going to read
everything from the current position to the least significant bit, we can just read the whole byte and apply a mask for
the <code>bits_avail</code> least significant bits (remember the explanation earlier). So <code>&amp; ((1 &lt;&lt; bits_avail) - 1)</code> does the job.
Because we have read <code>bits_avail</code> bits, we increment <code>bitpos</code> as well.</p>
<pre><code class="language-rust">      // then, left shift those bits as most significant bits for the final number
      let mut final_byte = byte0;
</code></pre>
<p>This part is not strictly needed as I could have called <code>byte0</code> <code>final_byte</code> in the first place, but I wanted a clear
mind about what was doing what. The idea for the following steps is that we are going to accumulate bits in <code>final_byte</code>
by left shifting it before applying the logical <code>OR</code>. The amount of shift to apply depends on the amount of bits read.
For the case where we read a whole byte, we will just right shift by 8 bits.</p>
<pre><code class="language-rust">      // then, we need to get the remaining bits; compute the number of bytes we need
      let n = n - bits_avail;
      let bytes_needed = n / 8;
</code></pre>
<p>This is a pretty straight-forward code given what I said above. <code>n - bits_avail</code> is the remaining amount of bits to
read. <code>bytes_needed</code> is the number of full bytes we will have to go through (in <code>self.bytes</code>), and from which we will
extract 8 bits each. The rest then makes more sense:</p>
<pre><code class="language-rust">      // while we require more than 1 byte, the problem is trivial; just copy and left shift the bytes
      for _ in 1..=bytes_needed {
        let byte = self.bytes[self.bitpos / 8] as u16;
        self.bitpos += 8;
        final_byte = (final_byte &lt;&lt; 8) | byte;
      }
</code></pre>
<p>Then, the remaining bits are the last part of the parsing problem. The idea is that we know we will have to read between
1 and 7 bits from that byte. That information is stored as <code>n</code> and <code>shift</code>:</p>
<pre><code class="language-rust">      let n = n % 8;
      let shift = 8 - n;
</code></pre>
<p>We then extract the information, and return the N-bit number:</p>
<pre><code class="language-rust">      let byte = (self.bytes[self.bitpos / 8] as u16 &gt;&gt; shift) &amp; ((1 &lt;&lt; n) - 1);
      self.bitpos += n;

      final_byte = (final_byte &lt;&lt; n) | byte;

      final_byte
</code></pre>
<h2>Decoding a literal packet</h2>
<p>Armed with <code>next_bits</code>, the problem really becomes trivial. Let’s decode a literal packet and introduce the <code>Packet</code>
type:</p>
<pre><code class="language-rust">#[derive(Clone, Debug, Eq, PartialEq)]
struct Header {
  version: u8,
  type_id: u8,
}

#[derive(Clone, Debug, Eq, PartialEq)]
enum Packet {
  Literal {
    header: Header,
    value: u128,
  },

  Operator {
    header: Header,
    length_type_id: u8,
    packets: Vec&lt;Packet&gt;,
  },
}
</code></pre>
<p>And <code>Decoder::decode</code>:</p>
<pre><code class="language-rust">impl Decoder {
  // …

  fn decode(&amp;mut self) -&gt; Packet {
    let version = self.next_bits(3) as u8;
    let type_id = self.next_bits(3) as u8;
    let header = Header { version, type_id };

    match type_id {
      // literal value
      4 =&gt; {
        let mut value = 0u128;
        loop {
          let bits = self.next_bits(5) as u128;
          value = (value &lt;&lt; 4) | (bits &amp; 0xF);

          if (bits &gt;&gt; 4) &amp; 0x1 == 0 {
            break Packet::Literal { header, value };
          }
        }
      }

      // operator
      _ =&gt; {
        todo!()
      }
    }
  }
}
</code></pre>
<p>Some explanation. Given the format, we know that a literal number is made of packets of 5 bits, where the most
significant bit does not belong to the number, but instead is a flag. If set to <code>0</code>, it means that the literal is over.</p>
<pre><code class="language-rust">          value = (value &lt;&lt; 4) | (bits &amp; 0xF);
</code></pre>
<p>This accumulates the 4 least significant bits of the 5 bits into the actual literal number.</p>
<pre><code class="language-rust">          if (bits &gt;&gt; 4) &amp; 0x1 == 0 {
</code></pre>
<p>And this performs the test on the most significant bit to check whether the literal number is fully built.</p>
<h2>Parsing operators</h2>
<p>Operators represent the other kind of packets, and are introduced, quoting:</p>
<blockquote>
<p>Every other type of packet (any packet with a type ID other than 4) represent an operator that performs some
calculation on one or more sub-packets contained within. Right now, the specific operations aren't important;
focus on parsing the hierarchy of sub-packets.</p>
<p>An operator packet contains one or more packets. To indicate which subsequent binary data represents its
sub-packets, an operator packet can use one of two modes indicated by the bit immediately after the packet
header; this is called the length type ID:</p>
<ul>
<li>If the length <code>type ID</code> is <code>0</code>, then the next <code>15</code> bits are a number that represents the total length in bits of the
sub-packets contained by this packet.</li>
<li>If the length <code>type ID</code> is <code>1</code>, then the next <code>11</code> bits are a number that represents the number of sub-packets
immediately contained by this packet.</li>
</ul>
<p>Finally, after the <code>length type ID</code> bit and the 15-bit or 11-bit field, the sub-packets appear.</p>
</blockquote>
<p>There is nothing else to do than just applying those rules:</p>
<pre><code class="language-rust">      // in Decoder::decode’s match on type ID

      // operator
      _ =&gt; {
        let length_type_id = self.next_bits(1) as u8;
        let mut packets = Vec::new();

        if length_type_id == 0 {
          let total_length_bits = self.next_bits(15) as usize;
          let mut bits_read = 0;

          while bits_read &lt; total_length_bits {
            let bitpos = self.bitpos;
            packets.push(self.decode());
            bits_read += self.bitpos - bitpos;
          }
        } else {
          let sub_packets_nb = self.next_bits(11);

          for _ in 0..sub_packets_nb {
            packets.push(self.decode());
          }
        }

        Packet::Operator {
          header,
          length_type_id,
          packets,
        }
      }
</code></pre>
<p>Because of the <code>next_bits</code> interface, for when <code>length_type_id == 0</code>, we have no way to know deterministically when to
stop, because we don’t really know how long each packet is. Because of this, the <code>bits_read</code> variable is introduced so
that we know how many bits we read. That inforation is easy to get by comparing the value of <code>bitpos</code> before and after
reading a packet. For the other branch, there is nothing interesting to comment here.</p>
<h2>Part 1: summing packet headers’ versions</h2>
<p>The first question was:</p>
<blockquote>
<p>Decode the structure of your hexadecimal-encoded BITS transmission; what do you get if you add up the version numbers
in all packets?</p>
</blockquote>
<p>And my input was (it was a single string but I break it on several lines, because it’s pretty long):</p>
<pre><code>60552F100693298A9EF0039D24B129BA56D67282E600A4B5857002439CE580E5E5AEF67803600D2E294B2FCE8AC489BAEF37FEACB31A678548034EA0
086253B183F4F6BDDE864B13CBCFBC4C10066508E3F4B4B9965300470026E92DC2960691F7F3AB32CBE834C01A9B7A933E9D241003A520DF31664700
2E57C1331DFCE16A249802DA009CAD2117993CD2A253B33C8BA00277180390F60E45D30062354598AA4008641A8710FCC01492FB75004850EE5210AC
EF68DE2A327B12500327D848028ED0046661A209986896041802DA0098002131621842300043E3C4168B12BCB6835C00B6033F480C493003C4008002
9F1400B70039808AC30024C009500208064C601674804E870025003AA400BED8024900066272D7A7F56A8FB0044B272B7C0E6F2392E3460094FAA500
2512957B98717004A4779DAECC7E9188AB008B93B7B86CB5E47B2B48D7CAD3328FB76B40465243C8018F49CA561C979C182723D76964220041275627
1FC80460A00CC0401D8211A2270803D10A1645B947B3004A4BA55801494BC330A5BB6E28CCE60BE6012CB2A4A854A13CD34880572523898C7EDE1A9F
A7EED53F1F38CD418080461B00440010A845152360803F0FA38C7798413005E4FB102D004E6492649CC017F004A448A44826AB9BFAB5E0AA8053306B
0CE4D324BB2149ADDA2904028600021909E0AC7F0004221FC36826200FC3C8EB10940109DED1960CCE9A1008C731CB4FD0B8BD004872BC8C3A432BC8
C3A4240231CF1C78028200F41485F100001098EB1F234900505224328612AF33A97367EA00CC4585F315073004E4C2B003530004363847889E200C45
985F140C010A005565FD3F06C249F9E3BC8280804B234CA3C962E1F1C64ADED77D10C3002669A0C0109FB47D9EC58BC01391873141197DCBCEA401E2
CE80D0052331E95F373798F4AF9B998802D3B64C9AB6617080
</code></pre>
<p>So:</p>
<ol>
<li>Parse the input and decode it into a <code>Packet</code>.</li>
<li>Recurse on the <code>version</code> field of all sub-packets, summing them.</li>
</ol>
<pre><code class="language-rust">fn solve1(input: &amp;str) -&gt; u32 {
  let mut decoder = Decoder::new(input);
  checksum(&amp;decoder.decode())
}

fn checksum(packet: &amp;Packet) -&gt; u32 {
  match packet {
    Packet::Literal { header, .. } =&gt; header.version as _,
    Packet::Operator {
      header, packets, ..
    } =&gt; header.version as u32 + packets.iter().map(checksum).sum::&lt;u32&gt;(),
  }
}
</code></pre>
<p>Yes, it’s that simple once you have made the right framework. :) Note that this function is not tail-recursive. A
tail-recursive version would probably perform faster, but it was so convenient to write and took me 15s.</p>
<h2>Part 2: evaluating the expression</h2>
<p>Part 2 explains that the operator packets are actually expression operators, operating on literal values. Basically,
it’s a whole <a href="https://en.wikipedia.org/wiki/Abstract_syntax_tree">AST</a>. The rules are:</p>
<ul>
<li>Packets with <code>type ID</code> <code>0</code> are sum packets - their value is the sum of the values of their sub-packets. If they only
have a single sub-packet, their value is the value of the sub-packet.</li>
<li>Packets with <code>type ID</code> <code>1</code> are product packets - their value is the result of multiplying together the values of their
sub-packets. If they only have a single sub-packet, their value is the value of the sub-packet.</li>
<li>Packets with <code>type ID</code> <code>2</code> are minimum packets - their value is the minimum of the values of their sub-packets.</li>
<li>Packets with <code>type ID</code> <code>3</code> are maximum packets - their value is the maximum of the values of their sub-packets.</li>
<li>Packets with <code>type ID</code> <code>5</code> are greater than packets - their value is 1 if the value of the first sub-packet is greater
than the value of the second sub-packet; otherwise, their value is 0. These packets always have exactly two
sub-packets.</li>
<li>Packets with <code>type ID</code> <code>6</code> are less than packets - their value is 1 if the value of the first sub-packet is less than
the value of the second sub-packet; otherwise, their value is 0. These packets always have exactly two sub-packets.</li>
<li>Packets with <code>type ID</code> <code>7</code> are equal to packets - their value is 1 if the value of the first sub-packet is equal to the
value of the second sub-packet; otherwise, their value is 0. These packets always have exactly two sub-packets.</li>
</ul>
<p>This is my solution:</p>
<pre><code class="language-rust">fn solve2(input: &amp;str) -&gt; u64 {
  let mut decoder = Decoder::new(input);
  eval(&amp;decoder.decode())
}

fn eval(packet: &amp;Packet) -&gt; u64 {
  match packet {
    Packet::Literal { value, .. } =&gt; *value as _,
    Packet::Operator {
      header, packets, ..
    } =&gt; match header.type_id {
      0 =&gt; packets.iter().map(eval).sum::&lt;u64&gt;(),
      1 =&gt; packets.iter().map(eval).product::&lt;u64&gt;(),
      2 =&gt; packets.iter().map(eval).min().unwrap(),
      3 =&gt; packets.iter().map(eval).max().unwrap(),
      5 =&gt; {
        if eval(&amp;packets[0]) &gt; eval(&amp;packets[1]) {
          1
        } else {
          0
        }
      }
      6 =&gt; {
        if eval(&amp;packets[0]) &lt; eval(&amp;packets[1]) {
          1
        } else {
          0
        }
      }
      7 =&gt; {
        if eval(&amp;packets[0]) == eval(&amp;packets[1]) {
          1
        } else {
          0
        }
      }

      id =&gt; panic!(&quot;unknown type id: {}&quot;, id),
    },
  }
}
</code></pre>
<p>And here you have it all. The
<a href="https://github.com/phaazon/advent-of-code-2021/blob/master/day16/src/main.rs">complete solution is available on my GitHub repository for AoC 21</a>.
I don’t think it was the hardest problem, nor the more interesting, but the decision (and challenge) I decided to go
with (bit-packing with a bit-driven interface) made the whole thing much more fun and interesting to me.</p>
<p>I hoped you liked it and that you learned something, especially regarding bitwise operations. That was pretty heavy! I
think that you should also now have an idea about how you could write a specialized version of <code>Vec&lt;bool&gt;</code> that
bit-packs booleans value. Obviously, we do not want <code>Vec&lt;bool&gt;</code> to actually do that kind of specialization behind the
scenes (think FFI: when getting a <code>*const bool</code> or <code>*mut bool</code> from the <code>Vec&lt;bool&gt;</code>, if those pointers point to
bit-packed data… it’s going to be a nightmare if the FFI function you use expect an actual array of unpacked boolean
values).</p>
<p>For the rest of AoC, I’m probably going to continue working on it but I’m currently pausing at day 18 (Christmas,
fatigue, a bit of OSS burnout too). I will probably write some more about AoC once I feel rested.</p>
<p>Have fun and keep the vibes!</p>


guid: https://strongly-typed-thoughts.net/blog/splines-1.0.0-rc.1Mon, 22 Apr 2019 01:10:00 GMT
title: splines-1.0.0 and first release candidate
author: Dimitri Sabadie <dimitri.sabadie@gmail.com>
url: https://strongly-typed-thoughts.net/blog/splines-1.0.0-rc.1
feedurl: https://phaazon.net/blog/feed
content:
<h1>splines-1.0.0: the first Release Candidate</h1>
<p>Oh hello there! It’s been a while since I wrote on here. My last article was posted on February
25th and then I decided it was time to blog more about what I do. Especially, I think I will try to
blog more often, even if it’s about trivial things, as long as I share my thoughts.</p>
<p>So today, I’m going to talk about the <a href="https://crates.io/crates/splines">splines</a> crate. And more specifically, the <a href="https://crates.io/crates/splines/1.0.0-rc.1">splines-1.0.0-rc.1</a>
release candidate I uploaded today on crates.io.</p>
<h2>Foreword: you said splines?</h2>
<p>Maybe you’re wondering what a spline is, in the first place. A spline is a mathematic curve that is
defined by several polynomials. You can picture them mentally by several small and simple curves
combined to each others, giving the curve an interesting shape and properties. Now why we want
splines is easy to understand: imagine a curve, something smooth and a bit complex (maybe even with
loops). Now, imagine you want to make an object move along that curve. How do you represent that
curve and how to you “make something advance along it?”</p>
<p>Splines are defined by several “control points”, also known “keys”, “knots” or whatever makes sense
to you. Those are just points of interest that will have a direct impact on how the curve will bend
and behave. Most of control points have the interesting property that the curve <strong>will</strong> pass
through them, allowing you to define points you want your object to pass by, for instance. Not all
control points have that property, though.</p>
<p>Now all the magic happens with those control points: when you want to sample a value at a given
control point, you just get back the value carried by this control point… but when you want to
sample in between two control points, for instance, you get back an <em>interpolated value</em>. That
interpolation is the key to why we want splines. Several interpolation mechanisms exist and I’m sure
you know at least two:</p>
<ul>
<li><strong>Constant interpolation</strong>: it means that given two control points, sampling at the first
control point or any part between the control points will give you the carried value of the
first control point. You will get the carried value of the second control point if you sample at
its sampling value. That might be like a useless feature but it is very handy when designing and
implementing “stop-by” movement with camera.</li>
<li><strong>Linear interpolation</strong>: this one you might know it, especially if you attended math courses at
high-school. Linear interpolation is often referred to as “drawing a straight line”. And
typically, “everything linear” can often graphed with a straight line! Thank about linear
complexity (<em>O(n)</em>) and what it looks like on a graph. Linear interpolation that gives you a
strict blending of arguments depending at which distance you sample from them. If sample at
<code>t = 0</code>, you get 100% of the first value and 0% of the second value. If you sample at <code>t = 0,5</code>,
you get an equal amount of both value (so you get the mean between both value). At <code>t = 1</code>, you
get 0% of the first value and 100% of the second value. Simple but really useful.</li>
<li><strong>Cosine interpolation</strong>: this one is known by people who are used to animation, graphics,
motion design, etc. Cosine interpolation gives you a smoother interpolation than the linear one
and provides a first derivative that is more interesting (i.e. it translates to <em>acceleration</em>).</li>
<li>Others: <a href="https://crates.io/crates/splines">splines</a> provide you with some other interpolation modes and some (that I really want!)
are missing. For instance, cubic <a href="https://en.wikipedia.org/wiki/B%C3%A9zier_curve">Bézier interpolation</a> is something I want and that I already
have coded in the Haskell version of the <a href="https://crates.io/crates/splines">splines</a> crate. It should be easy to add but so far,
no one has asked for them and I don’t need them, so… ;)</li>
</ul>
<blockquote>
<p>Keep in mind that a spline is a just an abstractio mathematical object and that you can use it for
lots of concepts and situations. For instance, in demoscene productions of mine, I use splines for
camera movement, animation transitions, color-blending and more!</p>
</blockquote>
<h2>The splines crate</h2>
<p>The <a href="https://crates.io/crates/splines">splines</a> crate has been around for some time now and I received some feedback from people on
the <a href="https://github.com/phaazon/splines">GitHub page</a> asking for more features. Basically, the crate
can now:</p>
<ul>
<li>Support failible sampling more sanely (instead of panicking on some very trick situations, you
get a <code>None</code>).</li>
<li>Support for polymorphic sampling types (you have <code>f32</code> but also <code>f64</code> or whatever).</li>
</ul>
<p>I think something interesting to dissect in this blog article is the last feature about polymorphic
sampling types and what I had to do in order to make it happen. Also, this is highly correlated to
a design choice I have made months ago: supporting foreign crates to interpolate specific types is
done inside the <a href="https://crates.io/crates/splines">splines</a> crate, not in other crates. I will discuss why just bellow.</p>
<h2>Polymorphic sampling</h2>
<p>Basically, in <a href="https://crates.io/crates/splines">splines</a>, prior to <code>1.0.0</code>, a spline was polymorphic type mapping a key of type <code>f32</code>
to a control point of type <code>Key&lt;V&gt;</code>, <code>V</code> being the carried value. You then manipulate a <code>Spline&lt;V&gt;</code>.
The problem with that is whenever someone wants to use a different time, like, <code>f64</code> or something of
their own.</p>
<p>My first solution was to turn the spline type to <code>Spline&lt;T, V&gt;</code>. That would allow people to use both
<code>f32</code> and <code>f64</code> as sampling type. The problem kicks in with the code handling those sampling values.
For instance, for the <code>Interpolation::Cosine</code> interpolation mode, I need to take the cosine of a
value which type is either <code>f32</code> or <code>f64</code>. How can we do that in a generic and polymorphic maneer?</p>
<blockquote>
<p><a href="https://crates.io/crates/num-traits">num-traits</a> to the rescue!</p>
</blockquote>
<p>Yeah, so was my first thoughts. I then re-wrote most of the interpolating code using <a href="https://crates.io/crates/num-traits">num-traits</a>…
and then decided to tackle all the feature gates. Because, I haven’t told you yet, but <a href="https://crates.io/crates/splines">splines</a> has
several feature gates allowing you to use, e.g., implement the <code>Interpolate</code> trait for some famous
crates’ types (<a href="https://crates.io/crates/nalgebra">nalgebra</a>, <a href="https://crates.io/crates/cgmath">cgmath</a>), enable <a href="https://crates.io/crates/serde">serde</a> serialization, etc. One feature gate that is
important to me as a demoscener is the <code>&quot;std&quot;</code> feature gate, that, if not defined, makes the
<a href="https://crates.io/crates/splines">splines</a> crate compile with the <code>no_std</code> feature.</p>
<p>And here comes the first problems. The <a href="https://crates.io/crates/num-traits">num-traits</a> has a trait that isn’t compatible with <code>no_std</code>,
the <a href="https://docs.rs/num-traits/0.2.6/num_traits/float/trait.Float.html"><code>Float</code></a> trait. I then sat
in front of my computer and <a href="https://phaazon.net/media/uploads/trying_to_think.jpg">thought</a>. I came
to the realization that whatever library I would use for that trait, I would always get my hands
tied by the <em>contracts of the public interface of that crate</em>, for a feature that is almost central
to the whole <a href="https://crates.io/crates/splines">splines</a> crate. It quickly became obvious that I couln’t use <a href="https://crates.io/crates/num-traits">num-traits</a>.</p>
<blockquote>
<p>Yes, I know about the <a href="https://docs.rs/num-traits/0.2.6/num_traits/float/trait.FloatCore.html"><code>FloatCore</code></a>
trait. However, I was stuck with <code>FloatConst</code> right after that and I just wanted to clean the
interface.</p>
</blockquote>
<p>The monomorphic version of <a href="https://crates.io/crates/splines">splines</a> is easy to implement for <code>no_std</code>, but now I’m stuck because of
some simple traits? Naaah.</p>
<p>So I decided to write my own traits. Enter: <code>Additive</code>, a simple trait with no content but <code>Add</code> and
<code>Sub</code> as supertraits (and some non-important ones for comprehension here), <code>One</code>, a trait that
provides the neutral element of the multiplication for a given type (i.e. the multiplication monoid)
and the <code>Linear&lt;T&gt;</code> trait, providing linear combinations, mandatory for all kind of interpolations
mentioned earlier.</p>
<p>Those traits have universal implementors so that you don’t have to implement them. And enabling a
feature gate for, e.g., <a href="https://crates.io/crates/cgmath">cgmath</a> will also implement those traits accordingly in an enough
polymorphic way that you shouldn’t have to ever worry about them.</p>
<p>Just a little remark of mine: implementing the support for <a href="https://crates.io/crates/cgmath">cgmath</a> was pretty simple and
straight-forward. As feared (because I used this crate), <a href="https://crates.io/crates/nalgebra">nalgebra</a> was way harder to get right —
and I even decided not to implement the <code>Point&lt;..&gt;</code> type because I’ve been struggling all the
afternoon with error types that reminded me old and dark times with C++ Boost library. I’ve already
discussed that with people from the <a href="https://crates.io/crates/nalgebra">nalgebra</a> world and none seemed to agree with me that this
crate is <em>a little bit too overengineered</em>. For instance, adding the support for <a href="https://crates.io/crates/nalgebra">nalgebra</a> in
splines also adds several mandatory dependencies — i.e. <a href="https://crates.io/crates/num-traits">num-traits</a> and <a href="https://crates.io/crates/alga">alga</a>. It’s not that bad,
but if you want to play with simple vector spaces, I really do not recommend <a href="https://crates.io/crates/nalgebra">nalgebra</a> as it’s a
really convoluted library with lots of <code>type</code> aliases and types with infinite counts of type
variables. Maybe it’s just a confirmation bias of mine, because I’ve always used (and written, back
then in C++) linear libraries that were both simple and fast. <a href="https://crates.io/crates/nalgebra">nalgebra</a> makes me <a href="https://www.boost.org/doc/libs/1_55_0/libs/geometry/doc/html/geometry/design.html">think of this and
this is driving me crazy</a>.
I’m not saying it’s bad. I’m saying it’s not for me and that I wouldn’t recommend it for people
wanting to do simple things — and yes, video game still falls in that <em>simple things</em> bag.</p>
<h2>Let’s talk about feature gates</h2>
<p>So, the design of feature gating in <a href="https://crates.io/crates/splines">splines</a>. I know it might feel a bit weird to have gates like
<code>&quot;impl-nalgebra&quot;</code> or <code>&quot;impl-cgmath&quot;</code> but to me it makes lots of sense. For a simple reason: a crate
exposes several types and functions via its public API. Everything that is not public is not for a
very good reason. Mostly, <em>invariants</em>. An invariant is something that must remain true <em>before</em> a
function / morphism / whatever is called and <em>after</em>. What happens in between can actually break
that invariant. The idea is that <em>“If a function breaks internally an invariant, it must restore it
before returning”</em>, so that the API remains safe to use.</p>
<p>Invariants are central in my way of thinking. Everytime I write some code, everytime I design an
interaction between several piece of algorithms, crates or even cross-language and cross-systems, I
always asks myself <em>“Am I breaking an invariant here? Is it possible to fuck up a state or a hidden
property somewhere?”</em></p>
<p>In Rust, invariants are not a first-class citizen of the language (have a look at how the <a href="https://dlang.org/spec/contracts.html#Invariants">D</a>
language handles <code>invariant</code>: it’s interesting!). However, they still exist. Imagine this piece of
code:</p>
<pre><code class="language-rust">pub struct Even(pub u64);

impl Even {
  pub fn new(nth: u64) -&gt; Self {
    Even(nth * 2)
  }
}

impl Add&lt;u64, Output = Even&gt; for Even {
  fn add(self, rhs: u64) -&gt; Output {
    Even(self.0 + rhs * 2)
  }
}
</code></pre>
<p>Imagine this is wrapped in a crate <code>num-even</code>. Adding a <code>u64</code> to an <code>Even</code> gives you the nth next
even number. All of this is cool. But there is an invariant. The carried <code>u64</code> by <code>Even</code> must…
remain even. If at some time a user handle a <code>Even</code> with an odd <code>u64</code> inside of it, something has
gone wrong terribly. And with this current implementation, it’s very easy to break such an
invariant. Consider:</p>
<pre><code class="language-rust">use num_even::Even;

fn main() {
  let mut a = Even::new(1); // the 1st even number
  a.0 += 1; // oooooooh, fuuuuuuu-
}
</code></pre>
<p>I had long heated debates with people on IRC about why this kind of library should be patched. The
main argument of people who would think it shouldn’t be patched is “Yeah but we want users to have
access to the underlying objects in any way they want.” I think invariants matter most. The patch
version is actually a negative diff:</p>
<pre><code class="language-rust">pub struct Even(u64);
</code></pre>
<p>Done. Here, the invariant cannot be broken anymore because people cannot create or modify <code>Even</code>
by hand. Constraining is powerful. You should try it. :)</p>
<p><a href="https://crates.io/crates/splines">splines</a> holds keys in a <code>Spline&lt;..&gt;</code> that must remain sorted. Hence, you don’t have a direct
access to the keys nor you can mutate them. Mutating keys would imply resorting the keys in a
smart way, which is something I haven’t needed yet.</p>
<p>So, that’s all for me for today. If you have any question about <a href="https://crates.io/crates/splines">splines</a>, please feel free to open
an issue on GitHub. Also, if you’re interested in trying it, please have a look at the
<a href="https://crates.io/crates/splines/1.0.0-rc.1">splines-1.0.0-rc.1</a> release candidate. It contains everything you need to get started!</p>
<blockquote>
<p>The documentation is not up to date and some is missing, but it should be enough to start.</p>
</blockquote>
<p>As always, keep the vibes!</p>


guid: https://strongly-typed-thoughts.net/blog/luminance_framebuffer_textureSat, 01 Aug 2015 00:00:00 GMT
title: Luminance – framebuffers and textures
author: Dimitri Sabadie <dimitri.sabadie@gmail.com>
url: https://strongly-typed-thoughts.net/blog/luminance_framebuffer_texture
feedurl: https://phaazon.net/blog/feed
content:
<p>I’m happily surprised that so many <strong>Haskell</strong> people follow
<a href="https://github.com/phaazon/luminance">luminance</a>! First thing first, let’s tell
you about how it grows.</p>
<p>Well, pretty quickly! There’s – yet – no method to make actual renders, because
I’m still working on how to implement some stuff (I’ll detail that below), but
it’s going toward the right direction!</p>
<h1>Framebuffers</h1>
<p>Something that is almost done is the
<a href="https://www.opengl.org/wiki/Framebuffer_Object">framebuffer</a> part. The main
idea of <em>framebuffers</em> – in <strong>OpenGL</strong> – is supporting <em>offscreen renders</em>, so
that we can render to several framebuffers and combine them in several fancy
ways. Framebuffers are often bound <em>textures</em>, used to pass the rendered
information around, especially to <em>shaders</em>, or to get the pixels through
texture reads CPU-side.</p>
<p>The thing is… <strong>OpenGL</strong>’s <em>framebuffers</em> are tedious. You can have incomplete
framebuffers if you don’t attach textures with the right format, or to the wrong
attachment point. That’s why the <em>framebuffer</em> layer of <strong>luminance</strong> is there
to solve that.</p>
<p>In <strong>luminance</strong>, a <code>Framebuffer rw c d</code> is a framebuffer with two formats. A
<em>color</em> format, <code>c</code>, and a <em>depth</em> format, <code>d</code>. If <code>c = ()</code>, then no color will
be recorded. If <code>d = ()</code>, then no depth will be recorded. That enables the use
of <em>color-only</em> or <em>depth-only</em> renders, which are often optimized by GPU. It
also includes a <code>rw</code> type variable, which has the same role as for <code>Buffer</code>.
That is, you can have <em>read-only</em>, <em>write-only</em> or <em>read-write</em> framebuffers.</p>
<p>And of course, all those features – having a <em>write-only</em> <em>depth-only</em>
framebuffer for instance – are set through… <strong>types</strong>! And that’s what is so
cool about how things are handled in <strong>luminance</strong>. You just tell it what you
want, and it’ll create the required state and manage it for you GPU-side.</p>
<h2>Textures</h2>
<p>The format types are used to know which textures to create and how to attach
them internally. The textures are hidden from the interface so that you can’t
mess with them. I still need to find a way to provide some kind of access to the
information they hold, in order to use them in shaders for instance. I’d love to
provide some kind of <em>monoidal</em> properties between framebuffers – to mimick
<a href="https://hackage.haskell.org/package/gloss">gloss</a> <code>Monoid</code> instance for its
<a href="https://hackage.haskell.org/package/gloss-1.9.2.1/docs/Graphics-Gloss-Data-Picture.html#t:Picture">Picture</a>
type, basically.</p>
<p>You can create textures, of course, by using the <code>createTexture w h mipmaps</code>
function.  <code>w</code> is the <em>width</em>, <code>h</code> the <em>height</em> of the texture. <code>mipmaps</code> is the
number of <em>mipmaps</em> you want for the texture.</p>
<p>You can then upload <em>texels</em> to the texture through several functions. The
basic form is <code>uploadWhole tex autolvl texels</code>. It takes a <em>texture</em> <code>tex</code> and
the <code>texels</code> to upload to the whole texture region. It’s your responsibility to
ensure that you pass the correct number of texels. The <code>texels</code> are represented
with a polymorphic type. You’re not bound to any kind of textures. You can pass
a list of texels, a <code>Vector</code> of texels, or whatever you want, as long as it’s
<code>Foldable</code>.</p>
<p>It’s also possible to fill the whole texture with a single value. In <strong>OpenGL</strong>
slang, such an operation is often called <em>clearing</em> – clearing a <em>buffer</em>,
clearing a <em>texture</em>, clearing the <em>back buffer</em>, and so on. You can do that
with <code>fillWhole</code>.</p>
<p>There’re two over functions to work with subparts of textures, but it’s not
interesting for the purpose of that blog entry.</p>
<h2>Pixel format</h2>
<p>The cool thing is the fact I’ve unified pixel formats. <em>Textures</em> and
<em>framebuffers</em> share the same pixel format type (<code>Format t c</code>). Currently,
they’re all phantom types, but I might unify them further and use <code>DataKinds</code> to
promote them to the type-level. A format has two type variables, <code>t</code> and <code>c</code>.</p>
<p><code>t</code> is the underlying type. Currently, it can be either <code>Int32</code>, <code>Word32</code> or
<code>Float</code>. I might add support for <code>Double</code> as well later on.</p>
<p><code>c</code> is the channel type. There’re basically five channel types:</p>
<ul>
<li><code>CR r</code>, a red channel ;</li>
<li><code>CRG r g</code>, red and green channels ;</li>
<li><code>CRGB r g b</code>, red, green and blue channels ;</li>
<li><code>CRGBA r g b a</code>, red, green, blue and alpha channels ;</li>
<li><code>CDepth d</code>, a depth channel (special case of <code>CR</code>; for depths only).</li>
</ul>
<p>The type variables <code>r</code>, <code>g</code>, <code>b</code>, <code>a</code> and <code>d</code> represent <em>channel sizes</em>.
There’re currently three kind of <em>channel sizes</em>:</p>
<ul>
<li><code>C8</code>, for 8-bit ;</li>
<li><code>C16</code>, for 16-bit ;</li>
<li><code>C32</code>, for 32-bit.</li>
</ul>
<p>Then, <code>Format Float (CR C32)</code> is a red channel, 32-bit float – the <strong>OpenGL</strong>
equivalent is <code>R32F</code>. <code>Format Word32 (CRGB C8 C8 C16)</code> is a <em>RGB</em> channel with
red and green 8-bit unsigned integer channels and the blue one is a 16-bit
unsigned integer channel.</p>
<p>Of course, if a pixel format doesn’t exist on the <strong>OpenGL</strong> part, you won’t be
able to use it. Typeclasses are there to enforce the fact pixel format can be
represented on the <strong>OpenGL</strong> side.</p>
<h1>Next steps</h1>
<p>Currently, I’m working hard on how to represent vertex formats. That’s not a
trivial task, because we can send vertices to <strong>OpenGL</strong> as
interleaved – or not – arrays. I’m trying to design something elegant and safe,
and I’ll keep you informed when I finally get something. I’ll need to find an
interface for the actual render command, and I should be able to release
something we can actually use!</p>
<p>By the way, some people already tried it (Git HEAD), and that’s amazing! I’ve
created the <code>unstable</code> branch so that I can push unstable things, and keep the
master branch as clean as possible.</p>
<p>Keep the vibe, and have fun hacking around!</p>


guid: https://strongly-typed-thoughts.net/blog/what_went_revision_2017Thu, 20 Apr 2017 00:00:00 GMT
title: Postmortem #1 – Revision 2017
author: Dimitri Sabadie <dimitri.sabadie@gmail.com>
url: https://strongly-typed-thoughts.net/blog/what_went_revision_2017
feedurl: https://phaazon.net/blog/feed
content:
<p>On the weekend of 14th – 17th of April 2017, I was attending for the forth time the
<a href="https://2017.revision-party.net">easter demoparty Revision 2017</a>. This demoparty is the biggest so
far in the world and gathers around a thousand people coming from around the world. If you’re a
demomaker, a demoscene passionated or curious about it, that’s the party to go. It hosts plenty of
competitions, among <em>photos</em>, <em>modern graphics</em>, <em>oldschool graphics</em>, <em>games</em>, <em>size-limited demos</em>
(what we call <em>intros</em>), <em>demos</em>, <em>tracked and streamed music</em>, etc. It’s massive.</p>
<p>So, as always, once a year, I attend Revision. But this year, it was a bit different for me. Revision
is <em>very</em> impressive and most of the <em>“big demogroups”</em> release their productions they’ve been
working on for months or even years. I tend to think <em>“If I release something here, I’ll just be
kind of muted by all those massive productions.”</em> Well, less than two weeks before Revision 2017, I
was contacted by another demogroup. They asked me to write an <em>invitro</em> – a kind of <em>intro</em> or
<em>demo</em> acting as a communication production to invite people to go to another party. In my case, I
was proposed to make the <a href="http://outlinedemoparty.nl">Outline 2017</a> invitro. Ouline was the first
party I attended years back then, so I immediately accepted and started to work on something. That
was something like 12 days before the Revision deadline.</p>
<p>I have to tell. It was a challenge. All productions I wrote before was made in about a month and a
half and featured less content than the Outline Invitation. I had to write a lot of code from
scratch. <em>A lot</em>. But it was also a very nice project to test my demoscene framework, written in
Rust – you can find <a href="https://github.com/phaazon/spectra">spectra here</a> for now; it’s unreleased by
the time I’m writing this but I plan to push it onto crates.io very soon.</p>
<p>An hour before hitting the deadline, the beam team told me their Ubuntu compo machine died and that
it would be neat if I could port the demo to Windows. I rushed like a fool to make a port – I even
forked and modified my OpenAL dependency! – and I did it in 35 minutes. I’m still a bit surprised
yet proud!</p>
<p>Anyways, this post is not about bragging. It’s about hindsight. I did that for
<a href="http://www.pouet.net/prod.php?which=67966">Céleri Rémoulade</a> as I was the only one working on it –
music, gfx, direction and obviously the Rust code. I want to draw a list of <em>what went wrong</em> and
<em>what went right</em>. In the first time, for me. So that I have enhancement axis for the next set of
demos I’ll do. And for sharing those thoughts so that people can have a sneak peek into the
internals of what I do mostly – I do a lot of things! :D – as a hobby on my spare time.</p>
<h1>What went wrong</h1>
<p>Sooooooooo… What went wrong. Well, a lot of things! <strong>spectra</strong> was designed to build demo
productions in the first place, and it’s pretty good at it. But something that I have to enhance is
the <em>user interaction</em>. Here’s a list of what went wrong in a concrete way.</p>
<h2>Hot-reloading went wiiiiiiiiiiild²</h2>
<p>With that version of <strong>spectra</strong>, I added the possibility to <em>hot-reload</em> almost everything I use as
a resource: shaders, textures, meshes, objects, cameras, animation splines, etc. I edit the file,
and as soon as I save it, it gets hot-reloaded in realtime, without having to interact with the
program (for curious ones, I use the straight-forward <a href="https://crates.io/crates/notify">notify crate</a>
for registering callbacks to handle file system changes). This is very great and I save a <strong>lot</strong> of
time – Rust compilation is slow, and that’s a lesson I had learned from Céleri Rémoulade: keeping
closing the program, make a change, compiling, running is a wast of time.</p>
<p>So what’s the issue with that? Well, the main problem is the fact that in order to implement
hot-reloading, I wanted performance and something very simple. So I decided to use <em>shared mutable
smart states</em>. As a <strong>Haskeller</strong>, I kind of offended myself there – laughter! Yeah, in the
Haskell world, we try hard to avoid using shared states – <code>IORef</code> – because it’s not referentially
transparent and reasoning about it is difficult. However, I tend to strongly think that in some very
specific cases, you need such side-effects. I’m balanced but I think it’s the way to go.</p>
<p>Anyway, in Rust, shared mutable state is implemented via two types: <code>Rc/Arc</code> and <code>Cell/RefCell</code>.</p>
<p>The former is a runtime implementation of the <em>borrowing rules</em> and enables you to share a pointer.
The borrowing rules are not enforced at compile-time anymore but dynamically checked. It’s great
because in some time, you can’t know how long your values will be borrowed or live. It’s also
dangerous because you have to pay extra attention to how you borrow your data – since it’s checked
at runtime, you can crash your program if you’re not extra careful.</p>
<blockquote>
<p><code>Rc</code> means <em>ref counted</em> and <code>Arc</code> means <em>atomic-ref counted</em>. The former is for values that stay
on the same and single thread; the latter is for sharing the pointer between threads.</p>
</blockquote>
<p><code>Cell/RefCell</code> are very interesting types that provide <em>internal mutation</em>. By default, Rust gives
you <em>external mutation</em>. That is, if you have a value and its address, can mutate what you have at
that address. On the other hand, <em>internal mutation</em> is introduced by the <code>Cell</code> and <code>RefCell</code>
types. Those types enable you to mutate the content of an object stored at a given address without
having the exterior mutatation property. It’s a bit technical and related to Rust, but it’s often
used to mutate the content of a value via a function taking an immutable value.</p>
<blockquote>
<p><code>Cell</code> only accepts values that can be copied bit-wise and <code>RefCell</code> works with references.</p>
</blockquote>
<p>Now, if you combine both – <code>Rc&lt;RefCell&lt;_&gt;&gt;</code>, you end up with a shareable – <code>Rc&lt;_&gt;</code> – mutable –
<code>RefCell&lt;_&gt;</code> – value. If you have a value of type <code>Rc&lt;RefCell&lt;u32&gt;&gt;</code> for instance, that means you
can clone that value and store it everywhere in the same thread, and at any time, borrow it and 
inspect and/or mutate its content. All copies of the values will observe the change. It’s a bit like
C++’s <code>shared_ptr</code>, but it’s safer – thank you Rust!</p>
<p>So what went wrong with that? Well, the borrow part. Because Rust is about safety, you still need to
tell it how you want to borrow at runtime. This is done with the [<code>RefCell::borrow()</code>](https://doc.rust-lang.org/std/cell/struct.RefCell.html#method.borrow]
and <a href="https://doc.rust-lang.org/std/cell/struct.RefCell.html#method.borrow_mut"><code>RefCell::borrow_mut()</code></a>
functions. Those functions return a special object that borrows the pointed object as long as it
lives. Then, when it goes out of scope, it releases the borrow.</p>
<p>So any time you want to use an object that is hot-reloadable with my framework, you have to call
one of the borrow functions presented above. You end up with a lot of borrows, and you have to keep
in mind that you can litterally crash your program if you violate the borrowing rules. This is a
nasty issue. So far, I haven’t really spent time trying to fix that, but that something I have to
figure out.</p>
<h2>Resources declaration in code</h2>
<p>This is a bit tricky. As a programmer, I’m used to write algorithms and smart architectures to
transform data and resolve problems. I’m given inputs and I provide the outputs – the solutions.
However, a demoscene production is special: you don’t have inputs. You create artsy audiovisual
outputs from nothing but time. So you don’t really write code to solve a problem. You write code
to create something that will be shown on screen or in a headphone. This aspect of demo coding has
an impact on the style and the way you code. Especially in crunchtime. I have to say, I was pretty
bad on that part with that demo. To me, code should only be about transformations – that’s why I
love Haskell so much. But my code is clearly not.</p>
<p>If you know the <code>let</code> keyword in Rust, well, imagine hundreds and hundreds of lines starting with
<code>let</code> in a single function. That’s most of my demo. In rush time, I had to declare a <em>lot</em> of things
so that I can use them and transform them. I’m not really happy with that, because those were data
only. Something like:</p>
<pre><code class="language-rust">let outline_emblem = cache.get::&lt;Model&gt;(&quot;Outline-Logo-final.obj&quot;, ()).unwrap();
let hedra_01 = cache.get::&lt;Model&gt;(&quot;DSR_OTLINV_Hedra01_Hi.obj&quot;, ()).unwrap();
let hedra_02 = cache.get::&lt;Model&gt;(&quot;DSR_OTLINV_Hedra02_Hi.obj&quot;, ()).unwrap();
let hedra_04 = cache.get::&lt;Model&gt;(&quot;DSR_OTLINV_Hedra04_Hi.obj&quot;, ()).unwrap();
let hedra_04b = cache.get::&lt;Model&gt;(&quot;DSR_OTLINV_Hedra04b_Hi.obj&quot;, ()).unwrap();
let twist_01 = cache.get::&lt;Object&gt;(&quot;DSR_OTLINV_Twist01_Hi.json&quot;, ()).unwrap();
</code></pre>
<p>It’s not that bad. As you can see, <strong>spectra</strong> features a <em>resource cache</em> that provides several
candies – hot-reloading, resource dependency resolving and resource caching. However, having to
declare those resources directly in the code is a nasty boilerplate to me. If you want to add a new
object in the demo, you have to turn it off, add the Rust line, re-compile the whole thing, then run
it once again. It breaks the advantage of having hot-reloading and it pollutes the rest of the code,
making it harder to spot the actual transformations going on.</p>
<p>This is even worse with the way I handle texts. It’s all <code>&amp;'static str</code> declared in a specific file
called <code>script.rs</code> with the same insane load of <code>let</code>. Then I rasterize them in a function and use
them in a very specific way regarding the time they appear. Not fun.</p>
<h2>Animation edition</h2>
<p>Most of the varying things you can see in my demos are driven by animation curves – splines. The
bare concept is very interesting: an animation contains control points that you know have a specific
value at a given time. Values in between are interpolated using an interpolation mode that can
change at each control points if needed. So, I use splines to animate pretty much everything: camera
movements, objects rotations, color masking, flickering, fade in / fade out effects, etc.</p>
<p>Because I wanted to be able to edit the animation in a comfy way – lesson learned from Céleri
Rémoulade, splines can be edited in realtime because they’re hot-reloadable. They live in JSON files
so that you just have to edit the JSON objects in each file and as soon as you save it, the
animation changes. I have to say, this was very ace to do. I’m so happy having coded such a feature.</p>
<p>However, it’s JSON. It’s already a thing. Though, I hit a serious problem when editing the
orientations data. In <strong>spectra</strong>, an orientation is encoded with a
<a href="https://en.wikipedia.org/wiki/Quaternion#Unit_quaternion">unit quaternion</a>. This is a 4-floating
number – hypercomplex. Editing those numbers in a plain JSON file is… challenging! I think I really
need some kind of animation editor to edit the spline.</p>


guid: https://strongly-typed-thoughts.net/blog/editors-in-2020Mon, 07 Sep 2020 00:03:00 GMT
title: My thoughts about editors in 2020
author: Dimitri Sabadie <dimitri.sabadie@gmail.com>
url: https://strongly-typed-thoughts.net/blog/editors-in-2020
feedurl: https://phaazon.net/blog/feed
content:
<p>I have been trying a lot of editors lately. When I say “trying”, I really meant I spent some time configuring and
using those editors. The ones I spent time using are:</p>
<ul>
<li><a href="https://neovim.io">neovim</a>, my main, daily editor I use for pretty much almost all projects I work on.</li>
<li><a href="https://www.jetbrains.com/idea">IntelliJ IDEA</a>, that I currently use at work to hack on Java codebases.</li>
<li><a href="https://code.visualstudio.com">VS Code</a>, that I have been trying mainly in Rust, TOML and Markdown.</li>
<li><a href="https://www.gnu.org/software/emacs">emacs</a>, that I highly hacked around to play on my Haskell and Rust codebases (and YAML / Markdown / TOML too).</li>
<li><a href="https://github.com/hlissner/doom-emacs">DOOM Emacs</a>, that I tried when I saw a colleague in a previous company I worked in (I was impressed by
the “united” feeling of the UI and by how everything looked so slick). So I gave it a try.</li>
<li><a href="https://atom.io">atom</a>, <a href="https://github.com">GitHub</a>’s take on editors. Same thing, mostly Rust, Haskell, etc.</li>
<li>A bunch of others that I won’t talk about because I quickly stopped using.</li>
</ul>
<p>The goal of this article is to create a temporal “snapshot” of my views on editors and what I think about the
current situation. I have been using vim and especially neovim for more than a decade. I need to explain about
my current workflow and what I cherish in editing before talking about each editors. Expect a point of view
from a neovimer which is looking around at lots of editors.</p>
<blockquote>
<p>This is only a personal workflow / point of view that works well <em>for me right now</em>. It doesn’t mean it will
for you and it doesn’t mean a different workflow would be worse.</p>
</blockquote>
<!-- vim-markdown-toc GFM -->
<ul>
<li><a href="#what-i-think-powerful-editing-should-be">What I think powerful editing should be</a>
<ul>
<li><a href="#keyboard-layout">Keyboard layout</a></li>
<li><a href="#modal-editors">Modal editors</a></li>
<li><a href="#how-i-like-to-move-around">How I like to move around</a></li>
<li><a href="#all-the-other-modal-candies">All the other modal candies</a></li>
</ul>
</li>
<li><a href="#the-editors">The editors</a>
<ul>
<li><a href="#neovim">neovim</a>
<ul>
<li><a href="#my-neovim-setup">My neovim setup</a></li>
<li><a href="#what-i-like-about-neovim">What I like about neovim</a></li>
<li><a href="#what-i-dislike-about-neovim">What I dislike about neovim</a></li>
</ul>
</li>
<li><a href="#intellij-idea">IntelliJ IDEA</a>
<ul>
<li><a href="#what-i-like-about-intellij-idea">What I like about IntelliJ IDEA</a></li>
<li><a href="#what-i-dislike-about-intellij-idea">What I dislike about IntelliJ IDEA</a></li>
</ul>
</li>
<li><a href="#vs-code">VS Code</a>
<ul>
<li><a href="#what-i-like-about-vs-code">What I like about VS Code</a></li>
<li><a href="#what-i-dislike-about-vs-code">What I dislike about VS Code</a></li>
</ul>
</li>
<li><a href="#emacs-and-doom-emacs">emacs and DOOM emacs</a>
<ul>
<li><a href="#what-i-like-about-emacs--doom-emacs">What I like about emacs / DOOM emacs</a></li>
<li><a href="#what-i-dislike-about-emacs--doom-emacs">What I dislike about emacs / DOOM Emacs</a></li>
</ul>
</li>
<li><a href="#atom">atom</a>
<ul>
<li><a href="#what-i-like-about-atom">What I like about atom</a></li>
<li><a href="#what-i-dislike-about-atom">What I dislike about atom</a></li>
</ul>
</li>
</ul>
</li>
<li><a href="#wrap-it-up">Wrap it up</a></li>
</ul>
<!-- vim-markdown-toc -->
<h1>What I think powerful editing should be</h1>
<h2>Keyboard layout</h2>
<p>I am French and I’m using a keyboard layout that is made to type very quickly in French and to code. With hindsight,
since I type more often in English than in French, maybe I should have picked another keyboard layout, but the coding
experience in my keyboard layout is really great, so I stick around.</p>
<p>The keyboard layout is <strong>bépo</strong>. I learned bépo the “recommended” way — i.e. you have to practice <em>typing</em> (« dactylographie » in French). It means that I use all my fingers to type on a keyboard, and that each key on the
keyboard is assigned <em>a single finger</em> that will press it. That helps a lot with muscle memory and to reduce wrist
pain (my wrists barely move when I type on a keyboard), among other things. The typing speed is a side-effect of
being accurate and having good comfort (if you are curious, I’m pretty fast but there are faster people — I type at
around 120 to 130 words per minute). Because I think the speed doesn’t matter when programming, I think the most
important part to remember here is the comfort: the wrists don’t move and my fingers fly around the keyboard, whatever
the speed.</p>
<h2>Modal editors</h2>
<p>I think a modal editor is superior, for various reasons. The first one is that I truly <strong>hate</strong> having to use a
<em>mouse</em> for something that can be done without having to move around hundred of pixels with your cursor and clicking
on buttons. For instance, running an application, on my current machines, is simply typing <code>alt d</code>, the name of the
program (I typically use completion, so I never type fully the name of the program) and hit enter. All this without
moving my hands from the keyboard. And I do that for programs like <code>firefox</code>, <code>kdenlive</code>, etc. but for terminal
applications, I simply type and complete them in my terminal, which I open simply with <code>alt return</code>.</p>
<p>So, using a mouse to move around a cursor in an editor feels like completely suboptimal to me, especially because we
write code (i.e. we type on a keyboard) most of the time, so moving around with the mouse implies several back and
forth movements between the keyboard and the mouse. Maybe you don’t care and it’s cool to you, but to me, this is
truly <em>horror land</em>. I feel <em>very</em> uncomfortable when doing this.</p>
<p>Also, <em>modern editors</em> that are not modal typically make people move by using the arrows keys, which are either far
on your keyboard, or — like my keyboard, a 60% home made one — not in direct access and then require a function key
to enable them.</p>
<p>So that’s the first reason why I like modal editors. They make a smarter use of your keyboard for simple yet
recurrent features, like moving around — e.g. <code>h j k l</code>. The second reason why I like them is because of the facts
they have a completely new mode for non-modal editor (i.e. the <em>normal</em> mode), you have a whole keyboard / lots of
keys to bind actions to and do lots of things people usually do with the mouse. Being able to split an editor into
several buffers, move around the buffers, go at the beginning of the paragraph, search and replace, register actions
into macros and replay them, etc. All this without even moving the wrists. The learning curve is steep if you’re used
to the mouse, but once you’ve passed the mental barrier, really, and this is a personal opinion, but I truly think
that using the mouse again after that feels like handicap to me.</p>
<h2>How I like to move around</h2>
<p>When I look at people coding, I see several kind of programmers:</p>
<ul>
<li>The ones who move with the arrows or with <code>h j k l</code> in modal editors. You can spot them very easily at how the
cursor moves in a document. It typically implies keeping a key pressed until the cursor reach a given row, then
pressing another key until the cursor reach a given column and then adjust if they went passed the location they
had in mind.</li>
<li>People using the mouse, by clicking at the place they want to put the cursor at.</li>
<li>People using <em>relative numbers</em>. That’s an enhancement of the first group: they typically use relative numbers to
know which lines they want to jump to very quickly, so that they don’t have to press the up / down keys for ages
until reaching the line they want to. Instead, they look at the number on the lines, press that number, and the
direction, and bam, they are on the lines. They then use typical motions from vim like <code>$</code> (go to end of line),
<code>f</code> (go to the first occurrence of the next character typed after <code>f</code>, like <code>f(</code> will make your cursor go to the
next <code>(</code>), <code>%</code> (go to the matching delimiter) or <code>w</code> (go to the beginning of the next word) / <code>b</code> (go to the
beginning of the previous word), etc. to move faster on the same line (it works across lines too).</li>
</ul>
<p>I think that represents 99,9% of what I see people do. Obviously, you will get that I don’t belong to the second set
of people… but I don’t really belong to any, actually. How I move is, I guess, convoluted for most people and
I know some people won’t understand how it can feel. I use <code>h j k l</code> and all the motions from vim described in the
third group (and even more; I full lecture of four hours would be required to explain all of them :D), but it all
depends on the <em>distance</em> I need to travel. If my cursor is on a word and I want to move to the beginning of a word
located very closely to my cursor on the same line, I’ll simply type <code>www</code> if it’s three words apart (or <code>3w</code> if I’m
feeling funny). If the distance is higher, I use a tool called [EasyMotion].</p>
<p>Easymotion really is a wonderful tool. The idea is that it has several modes, depending on the kind of move you want
to perform:</p>
<ul>
<li><em>By lines</em>: this mode allows you to jump to any line in the current (or all open) buffers.</li>
<li><em>By words</em>: tihs mode allows you to jump to any “word” in the current (or all open) buffers.</li>
<li><em>By characters</em>: when the <em>word</em> mode doesn’t help with jumping to a special operator or character (because it’s not
recognized as a word), this mode allows you to jump to any character in the current or (or all open) buffers.</li>
<li>It has other modes but I have never really found useful cases for them.</li>
</ul>
<p>The way I typically do it is by mapping the three modes to <code>&lt;leader&gt;l</code>, <code>&lt;leader&gt;w&gt;</code> and <code>&lt;leader&gt;c</code> (in my case,
<code>&lt;leader&gt;</code> is the <em>space</em> key).</p>
<p>Typing <code>SPC l</code> in my current buffer results in this:</p>
<p><img src="https://phaazon.net/media/uploads/easymotion_lines.png" alt="EasyMotion lines" /></p>
<p>Typing any highlighted character will make my cursor jump to it. The same applies for words, with <code>SPC w</code>:</p>
<p><img src="https://phaazon.net/media/uploads/easymotion_words.png" alt="EasyMotion words" /></p>
<p>For the <em>character</em> mode, after pressing <code>SPC c</code>, I have to press another character (the one I want to jump to). Let’s
say we want to jump to a <code>#</code> (which is not part of words): <code>SPC c #</code>:</p>
<p><img src="https://phaazon.net/media/uploads/easymotion_chars.png" alt="EasyMotion characters" /></p>
<p>This way of moving is not intuitive at first, but once you get used to it… it’s a <em>must have</em>.</p>
<h2>All the other modal candies</h2>
<p>Among all the things that I like about modal editing, here is a non-exhaustive list of features I expect to have around
my fingers:</p>
<ul>
<li><code>C-i</code> and <code>C-o</code>: those allows me to jump to something / a file / a place in a buffer and then go back to where I was
right before with <code>C-o</code> (read it like <em>out</em>) or go back again with <code>C-i</code> (read it like <em>in</em>).</li>
<li>Macros and registers: those allow me to yank content into different registers (like clipboards) by assigning a single
key to paste their content. For instance, I can put a few lines in my <code>t</code> register with <code>&quot;tyi(</code> (“put in the <code>t</code>
register the <code>y</code>ank <code>i</code>nside matching <code>(</code>), and paste that content later with <code>&quot;tp</code>. Macros allow more powerful
editing control by assigning a key to set of actions with the <code>q</code> keyword (<code>qa</code> will register all the next keystrokes
and actions into the <code>a</code> macro). Then simply replay the macro with <code>@a</code>, for instance.</li>
<li>Obviously, all the basic vim motions, like <code>d</code> to delete, <code>y</code> to yank, <code>c</code> to change, <code>t</code> to go to the character right
before the one you search, <code>%</code> to go to the other delimiter, etc. And more complex text manipulation, such as “Let’s
change what’s inside this function parameter list, delimited by <code>(</code>”: <code>ci(</code>.</li>
</ul>
<p>It would take too much time to list everything, but the main idea is: I need the modal features when editing code.</p>
<h1>The editors</h1>
<p>So let’s talk about the list of editors I mentioned in the preface. The idea is to give <em>my own opinion</em> on those
editors. What I like about them regarding the way I like to edit code and what I think is missing.</p>
<h2>neovim</h2>
<p>I currently use <a href="https://neovim.io">neovim</a> in its <a href="https://en.wikipedia.org/wiki/Text-based_user_interface">TUI</a> version, because it’s the most stable, fast and easy neovim experience out
there so far to me. I have tried several GUI versions but never found what I was looking for — the main reason being
that they pretty much all of them use Web™ technologies, and it’s a hard no to me. I think I should elaborate a bit
more about that last point.</p>
<p>Why not using Web tech:</p>
<ul>
<li>Editing something on my machine has nothing to do with Web technology. This applies to lot of other things, but
truly, running a complex JavaScript VM / CSS engine in an editor seems so wrong on so many levels (performance being
the first one).</li>
<li>Most of the time, editors based on Web tech take ages to load. Yes, even VS Code — remember that my daily editor is
neovim, <strong>which loads in around 23ms</strong> with almost 50 packages installed (you can profile with <code>:profile</code> to get that
value). Among those 23ms, <a href="https://github.com/neoclide/coc.nvim">coc.nvim</a> take around 12ms.</li>
<li>Scripting. Scripting in JavaScript or CoffeeScript is a hard no to me.</li>
<li><code>npm</code> is one of the worst piece of software ever written. Please don’t make me use it again.</li>
</ul>
<h3>My neovim setup</h3>
<p>So I use several plugins that I’m going to describe. I think it’s important that people know about all those
because, in the collective mind, people still think vim / neovim are editors from the past. Well, not really.</p>
<ul>
<li><a href="https://github.com/ryanoasis/vim-devicons">ryanoasis/vim-devicons</a>
<ul>
<li>Add lots of unicode icons that other packages will use to provide a fancier and sexier interface experience.</li>
</ul>
</li>
<li><a href="https://github.com/sainnhe/sonokai">sainnhe/sonokai</a>
<ul>
<li>The current colorscheme I am using. It is very similar to what you get with <a href="https://github.com/hlissner/doom-emacs">DOOM Emacs</a> — even though it’s a
bit less contrasted.</li>
</ul>
</li>
<li><a href="https://github.com/neovimhaskell/haskell-vim">neovimhaskell/haskell-vim</a>
<ul>
<li>Haskell syntax support for <a href="https://neovim.io">neovim</a>.</li>
</ul>
</li>
<li><a href="https://github.com/rust-lang/rust.vim">rust-lang/rust.vim</a>
<ul>
<li>Rust syntax support for <a href="https://neovim.io">neovim</a>.</li>
</ul>
</li>
<li><a href="https://github.com/plasticboy/vim-markdown">plasticboy/vim-markdown</a>
<ul>
<li>Markdown support for <a href="https://neovim.io">neovim</a>. By default, <a href="https://neovim.io">neovim</a> already provides a good support for Markdown, but this package
has several cool features, like folds. I might get rid of it as I’m using folds that much.</li>
</ul>
</li>
<li><a href="https://github.com/mzlogin/vim-markdown-toc">mzlogin/vim-markdown-toc</a>
<ul>
<li>A very cool package that gives a way to create table of content in Markdown buffers and let neovim automatically
update the sections when you edit headers.</li>
</ul>
</li>
<li><a href="https://github.com/tikhomirov/vim-glsl">tikhomirov/vim-glsl</a>
<ul>
<li>GLSL syntax support for <a href="https://neovim.io">neovim</a>.</li>
</ul>
</li>
<li><a href="https://github.com/cespare/vim-toml">cespare/vim-toml</a>
<ul>
<li>TOML syntax support for <a href="https://neovim.io">neovim</a>.</li>
</ul>
</li>
<li><a href="https://github.com/ElmCast/elm-vim">ElmCast/elm-vim</a>
<ul>
<li>Elm syntax support for <a href="https://neovim.io">neovim</a>.</li>
</ul>
</li>
<li><a href="https://github.com/idris-hackers/idris-vim">idris-hackers/idris-vim</a>
<ul>
<li>Idris syntax support for <a href="https://neovim.io">neovim</a>.</li>
</ul>
</li>
<li><a href="https://github.com/posva/vim-vue">posva/vim-vue</a>
<ul>
<li>Vue.js syntax support for <a href="https://neovim.io">neovim</a>.</li>
</ul>
</li>
<li><a href="https://github.com/baskerville/vim-sxhkdrc">baskerville/vim-sxhkdrc</a>
<ul>
<li><a href="https://wiki.archlinux.org/index.php/Sxhkd">sxhkd</a> support for <a href="https://neovim.io">neovim</a>.</li>
</ul>
</li>
<li><a href="https://github.com/norcalli/nvim-colorizer.lua">norcalli/nvim-colorizer.lua</a>
<ul>
<li>A super cool extension that will automatically change the background color of texts containing an hexadecimal
value, such as #f8324F or #42cf69:</li>
<li><img src="https://phaazon.net/media/uploads/nvim-colorizer.png" alt="nvim-colorizer.lua" /></li>
</ul>
</li>
<li><a href="https://github.com/airblade/vim-gitgutter">airblade/vim-gitgutter</a>
<ul>
<li>One of the best plugin I have installed. It provides the <em>gutters</em> you see on the top of your buffer when editing a
file versioned in git (<em>added</em>, <em>modified</em>, <em>deleted</em>, etc.):</li>
<li><img src="https://phaazon.net/media/uploads/vim-gitgutter-editions.png" alt="vim-gitgutter gutters" /></li>
<li>But it can do more, and among all the things it can do, it can help you preview <em>hunks</em>, stage or discard them
<strong>inside</strong> your editor. That’s just a blast to me:</li>
<li><img src="https://phaazon.net/media/uploads/vim-gitgutter-hunk-preview.png" alt="vim-gitgutter hunk preview" /></li>
</ul>
</li>
<li><a href="https://github.com/tpope/vim-fugitive">tpope/vim-fugitive</a>
<ul>
<li>All the features you expect to find from git inside <a href="https://neovim.io">neovim</a>. You can see diff, resolve merge conflicts, write
commit messages, etc. etc. However, I tend to still keep around the command line as I’m not completely solved on
how this plugin deals with <em>fixups</em> and <em>squashing</em> in general.</li>
</ul>
</li>
<li><a href="https://github.com/rhysd/git-messenger.vim">rhysd/git-messenger.vim</a>
<ul>
<li>A blame-at-cursor tool. Not really that useful actually, as fugitive already has a git blame window that will
annotate each line with the commits. I sometimes use this one, but I might get rid of it.</li>
</ul>
</li>
<li><a href="https://github.com/tveskag/nvim-blame-line">tveskag/nvim-blame-line</a>
<ul>
<li>A git blam inlined, at the right part of your lines. Very similar to the default git plugin from VS Code, for
instance.</li>
</ul>
</li>
<li><a href="https://github.com/junegunn/fzf.vim">junegunn/fzf.vim</a>
<ul>
<li>If there is <strong>one</strong> plugin you should install, it’s this one. It has <em>so many</em> features: opening files, git files,
buffers, rip-grepping, searching through history, commands, colorschemes, etc. etc. And as the name implies, it
uses <code>fzf</code> as a backend, so you get a very cool fuzzy search experience (with <em>refined</em> search, which I struggle
to find in <em>any other editor</em> — i.e. you can type something, then put a space and type again to match things
occurring before the actual match).</li>
</ul>
</li>
<li><a href="https://github.com/machakann/vim-highlightedyank">machakann/vim-highlightedyank</a>
<ul>
<li>A funny one: it will highlight the lines / objects you yank for a better visual feedback. Surprising <a href="https://neovim.io">neovim</a>
doesn’t have that by default.</li>
</ul>
</li>
<li><a href="https://github.com/liuchengxu/vista.vim">liuchengxu/vista.vim</a>
<ul>
<li>I use this one from time to time to get a <em>symbol tree</em>, but the output is not really appealing to me right now. I
might ditch it too.</li>
</ul>
</li>
<li><a href="https://github.com/neoclide/coc.nvim">neoclide/coc.nvim</a>
<ul>
<li>The best completion engine for <a href="https://neovim.io">neovim</a>, by far. I used to use others, such as <code>ale</code>, but this one is ace. It
basically provides you with LSP completion for lots of languages. It has an integrated marketplace you can use to
install new LSP servers and integrations, and it even has support for things completely unrelated (a bit
surprising, I think these should be standalone plugins), such as <code>coc-explorer</code> (which is a replacement of
<code>NERDTree</code> to me now), <code>coc-snippets</code>, etc.</li>
</ul>
</li>
<li><a href="https://github.com/tpope/vim-commentary">tpope/vim-commentary</a>
<ul>
<li>Easily comment / uncomment lines without having to insert the comment symbols yourself.</li>
</ul>
</li>
<li><a href="https://github.com/liuchengxu/vim-which-key">liuchengxu/vim-which-key</a>
<ul>
<li><code>which-key</code> from <a href="https://www.gnu.org/software/emacs">emacs</a>, but for <a href="https://neovim.io">neovim</a>. Basically, when setup properly, it will provide you with a visual list
of possible keybindings for sequences. I think it’s not that useful (or it is for people trying to use your
configuration or if you install a plugin that ships a lot of keybindings — which I truly dislike, btw), but it looks
pretty cool.</li>
</ul>
</li>
<li><a href="https://github.com/itchyny/lightline.vim">itchyny/lightline.vim</a>
<ul>
<li>A status line that looks pretty cool.</li>
</ul>
</li>
<li><a href="https://github.com/SirVer/ultisnips">SirVer/ultisnips</a>
<ul>
<li>Snippets support. The snippet engine of <code>ultisnips</code> allows for a lot of room, like interpolation via shell, viml,
python, etc.</li>
</ul>
</li>
<li><a href="https://github.com/honza/vim-snippets">honza/vim-snippets</a>
<ul>
<li>A collection of snippets for commonly used languages and file formats.</li>
</ul>
</li>
<li><a href="https://github.com/junegunn/vim-easy-align">junegunn/vim-easy-align</a>
<ul>
<li>A pretty neat plugin to easily align texts / tables with a few keystrokes only.</li>
</ul>
</li>
<li><a href="https://github.com/liuchengxu/vim-clap">liuchengxu/vim-clap</a>
<ul>
<li>A take on uniting all possible source of search / fuzzy finders in a modern and fast UI. Unfortunately, right now,
the plugin is pretty unstable to me, so I keep using <code>fzf</code> instead.</li>
</ul>
</li>
<li><a href="https://github.com/easymotion/vim-easymotion">easymotion/vim-easymotion</a>
<ul>
<li>I think I have explained what that is about earlier. ;)</li>
</ul>
</li>
</ul>
<h3>What I like about neovim</h3>
<ul>
<li>It’s fast. It really is. It opens immediately. Moving around, scrolling, etc. is smooth, whatever the terminal
I’m using (even though I’m currently using <a href="https://github.com/alacritty/alacritty">alacritty</a>).</li>
<li>The plugins and effort put into <a href="https://neovim.io">neovim</a> are really great. I love <code>vim-gitgutter</code> so much; I love the colorizer
plugin a lot too. <a href="https://github.com/neoclide/coc.nvim">coc.nvim</a> has been a blast so far (for most part). EasyMotion is typingporn to me. <code>fzf</code> is so
fast it should be illegal.</li>
<li><a href="https://neovim.io">neovim</a> has a community that is truly passionated about what they are doing, and new versions add lots of very cool
features that we quickly adopt into new plugins, such as the popup / floating windows / virtual texts for linter
annotations, etc. etc.</li>
<li>It’s lightweight: your RAM will thank you.</li>
<li>Something I haven’t talked about but is a killer-feature of vim / neovim: the <code>:help</code> pages. I think no other
software has such a cool set of help pages. Really, give it a try. You want to know how to configure <a href="https://github.com/neoclide/coc.nvim">coc.nvim</a>?
Simply type <code>:help coc-nvim</code>.</li>
<li>Plugin managers exist (I personally use <a href="https://github.com/junegunn/vim-plug">vim-plug</a> but you will find more), and they make your life so much easier.</li>
</ul>
<h3>What I dislike about neovim</h3>
<ul>
<li>Even though I love how fast the TUI is (I truly haven’t come across anything faster so far), TUIs feel like hacks
to me. For instance, if you split a window into two buffers, the vertical “marks”, “edges”,
whatever-you’d-like-to-call-them, are actual unicode characters. The way terminals work make it possible to
<em>ignore</em> those characters, but still, it feels hacky. If you want something like a minimap or a simply thin frame
around some text, or any kind of visual feedback that is a bit more complicated than simply turning bold mode or
underlying, you’re basically not going to get it.</li>
<li>The GUIs for <a href="https://neovim.io">neovim</a> are… not up to my expectations. Most of them are based on Web tech anyway, so they’re not
good candidates to me. For the rest, like clients based on Qt, they feel a bit abandonned :(. There are people
trying to make new ones, but they are not ready AFAIK.</li>
<li><a href="https://github.com/neoclide/coc.nvim">coc.nvim</a> sometimes feel like is doing the wrong thing. For instance, when editing Java, it timeouts very often
when trying to jump to the definition of a symbol (or simply looking a symbol up). That’s bad.</li>
<li>Changing my colorscheme while the editor is running is a waste of time and generates broken syntax highlighting
everywhere. A pity to me. :(</li>
</ul>
<h2>IntelliJ IDEA</h2>
<p>So I won’t going to be able to talk too much about this one, because I have only started using it at work (community
edition). I’m using the vanilla one, with few modifications to none. I edit only Java with it.</p>
<h3>What I like about IntelliJ IDEA</h3>
<p>The Java support is truly flawless. It does a lot of things for you, and among them, I was really impressed by:</p>
<ul>
<li>The refactor mechanism that would allow me to select a block of code inside a function, ask the editor to “move
that into a dedicated function”. So far, all editors can do that, but what impressed me is that <a href="https://www.jetbrains.com/idea">IntelliJ IDEA</a> was
able to find out which variables needed to be captured and put as arguments to the function, then automatically pass
them at call site, when replacing the block of moved code. Really neat.</li>
<li>The speed at which you can look for symbols, look for implemented functions, inherited classes, super classes, etc.
It’s all instant and it’s all very well presented to you. Me gusta.</li>
<li>The syntax highlighting is okay; I especially like the inlined type ascriptions for both <code>var</code> declarations but also
when passing arguments to functions.</li>
</ul>
<h3>What I dislike about IntelliJ IDEA</h3>
<ul>
<li>I’m using the community edition, which only supports Java and a bunch of other configuration languages. You don’t
get the profiler, for instance — which would be very well appreciated!</li>
<li>Even though the asynchronous part of the editor is impressive, it sometimes index projects at random times and if
you’re working on a laptop, brace yourself for Mars landing as the machine is lifting off! — i.e. it makes a hella
noise and heats up quickly.</li>
<li>I tried the Vim integration and I wasn’t able to use it correctly with my bépo keymap. Unable to remap some motion
and/or mode switches. I disabled it and cried deeply.</li>
<li>Without Vim support, even though the editor has a lot of shortcuts, it feels like you still have to use the mouse
to perform very basic tasks.</li>
</ul>
<h2>VS Code</h2>
<p>So this one is an important one, as it’s Microsoft’s take on editing. Everybody seems to love VS Code, and I get
why. The UI is slick and fast — for a Web-based editor… ;). LSP support is obviously premium and flawless. It has a
lot of community plugins, themes and integrations. For most part, even though it’s Web based, I have been enjoying
it.</p>
<h3>What I like about VS Code</h3>
<ul>
<li>The slickness of the editor / UI.</li>
<li>Language support is premium with LSP and editing with it feels very solid.</li>
<li>Millions of plugins.</li>
<li>Big community and most of people use it nowadays, so I guess that if you get any trouble, you can ask around?</li>
</ul>
<h3>What I dislike about VS Code</h3>
<ul>
<li>The fact it’s written with a Web tech. Among all editors Web-based, it’s the fastest, but still, if you’re used
to vim / <a href="https://neovim.io">neovim</a>, you’ll be dissatisfied with it.</li>
<li>The vim integration is poor / doesn’t work (I tried remapping <code>h</code> to <code>c</code> — remember, <em>bépo</em> keyboard layout). It
simply doesn’t work.</li>
</ul>
<h2>emacs and DOOM emacs</h2>
<p>I have been using <a href="https://www.gnu.org/software/emacs">emacs</a> lately (vanilla) as I saw a colleague using <a href="https://github.com/hlissner/doom-emacs">DOOM Emacs</a>. I’m putting both in the same
section as they are pretty similar. The way I see <a href="https://www.gnu.org/software/emacs">emacs</a> and <a href="https://github.com/hlissner/doom-emacs">DOOM Emacs</a> could be summed up with one word: united.
I don’t know how they do that, but all plugins blend so well with each other. I’m using the <em>ivy</em> interface for
completion and fuzzy search, and everything is so… well done. The UI is gorgeous, themes are awesome (I like the
default dark theme, <em>DOOM One</em>), the editor is pretty fast — still slower than <a href="https://neovim.io">neovim</a> to me, especially when
scrolling, but still much faster than a Web-based editor.</p>
<h3>What I like about emacs / DOOM emacs</h3>
<ul>
<li>Once setup correctly (better defaults, etc.), the editor feels very modern (like what you would find in
<a href="https://code.visualstudio.com">VS Code</a> / <a href="https://atom.io">atom</a>). It feels <em>slick</em> and well designed.</li>
<li>Evil-mode, the Vim mode, is to me the best out there (besides vim and neovim themselves, obviously). They cover
almost everything — they even have support for EasyMotion!</li>
<li>You script / configure it with <a href="https://en.wikipedia.org/wiki/Lisp_(programming_language)">Lisp</a>, which is… super great! <a href="https://en.wikipedia.org/wiki/Lisp_(programming_language)">Lisp</a> is that kind of old treasure that has been
around forever and still feels like something new. I really love it.</li>
<li>If you are using <a href="https://github.com/hlissner/doom-emacs">DOOM Emacs</a>, you are going to get a lot of candies for free. Its modules approach works pretty
well and provides a very innovative way to enable / disable features. There are tons of resources out there to
get into <a href="https://github.com/hlissner/doom-emacs">DOOM Emacs</a> and I highly suggest having a look, even if you don’t plan on using <a href="https://www.gnu.org/software/emacs">emacs</a> or <a href="https://github.com/hlissner/doom-emacs">DOOM Emacs</a>.
That’s how I discovered <code>which-key</code>, for instance — that I now use in <a href="https://neovim.io">neovim</a>.</li>
<li>The LSP integration is pretty well done. It will download for you the servers and if you open a file that you have
never gotten the server for before, it will gently ask you whether you’d like to.</li>
<li>On a more general note, both <a href="https://www.gnu.org/software/emacs">emacs</a> and <a href="https://github.com/hlissner/doom-emacs">DOOM emacs</a> feel more <em>interactive</em> than editors like vim or <a href="https://neovim.io">neovim</a>,
which I think is a definitive better approach.</li>
<li>It uses <a href="https://www.gtk.org">gtk</a> as main backend on Linux. I think it’s important to note it, because it’s not
Web-based! (yay!)</li>
<li>Magit is a wonderful tool.</li>
<li>Org-mode is really great too, even though I think it’s a bit too big to me.</li>
<li>The <em>daemon</em> mode is amazing and I think all editors should have it. It allows you to start an instance of <a href="https://www.gnu.org/software/emacs">emacs</a>
and connect <code>emacsclient</code> to it, reducing the loading time to zero. Very brilliant and very useful!</li>
</ul>
<h3>What I dislike about emacs / DOOM Emacs</h3>
<ul>
<li>Moving across a huge amount of code leads to stuttering and sometimes feels a bit meh, especially if you’re used to
vim / <a href="https://neovim.io">neovim</a>. Most of the time, it should be okay, but just keep in mind that scrolling in <a href="https://www.gnu.org/software/emacs">emacs</a> has always been
a problem.</li>
<li>Even though it could be seen as a positive point, I think that all the great plugins <a href="https://www.gnu.org/software/emacs">emacs</a> has make it very
bloated, and that’s a problem to me. For instance, Org-mode, which is a wonderful piece of software, should have
benefited much more people if it was a standalone application. To me, it feels like starting using <a href="https://www.gnu.org/software/emacs">emacs</a> will lead
to using your computer to run <a href="https://www.gnu.org/software/emacs">emacs</a>, and all your applications inside <a href="https://www.gnu.org/software/emacs">emacs</a>. It even has an IRC plugin and an
email reader plugin!</li>
<li>I’m not sure what’s happening with this one, but LSP servers feel… synchronous?! When opening a file for the
first time, the LSP server starts and you have to wait a few seconds before being able to jump into the file. I don’t
really know whether it’s a configuration thing, but it doesn’t feel very pleasant.</li>
<li>The defaults of <a href="https://www.gnu.org/software/emacs">emacs</a> are really, <em>really</em> bad. And making the whole thing like what you get with <a href="https://github.com/hlissner/doom-emacs">DOOM Emacs</a> is
going to cost you lots of hours reading documentation and experimenting with your configuration. I enjoyed doing it
but in the end… why simply not ship <a href="https://www.gnu.org/software/emacs">emacs</a> with those defaults? Is it due to historical reasons that no one cares
about anymore nowadays? <em>/stares at vim</em></li>
</ul>
<h2>atom</h2>
<p>I’ll finish with <a href="https://atom.io">atom</a>, <a href="https://github.com">GitHub</a>’s editor. I remember my first reaction when running <a href="https://atom.io">atom</a> for the first time
was: “Woah, that’s a pretty editor.” The default colorscheme, <em>One</em>, is a universal colorscheme everybody knows.
There are have been so many forks of it in so many different editors.</p>
<p><a href="https://atom.io">atom</a> looks a lot like <a href="https://code.visualstudio.com">VS Code</a> to me, but is a bit prettier in its UI — I do prefer <a href="https://atom.io">atom</a>’s UI to <a href="https://code.visualstudio.com">VS Code</a>’s.
The UI is slick and very clean. You can find a lot of extensions / plugins / themes, from LSP integrations to
Markdown previews and Vim modes.</p>
<h3>What I like about atom</h3>
<ul>
<li>The killer feature of <a href="https://atom.io">atom</a> to me is its ability to tell you what commands are associated (or erased) for a
keybinding you are pressing, live. It is very practical to debug keybinding issues and I wish more editors
had it. Other editors feature <em>similar</em> stuff, but not quite as good as this <em>echo mode</em> for keybinding.</li>
<li>The themes are pretty cool and the whole typing experience / completion is pretty solid and consistent.</li>
<li>Lots of plugins to play with.</li>
</ul>
<h3>What I dislike about atom</h3>
<ul>
<li>Vim mode. Once again, it’s not complete to me as it doesn’t support well my bépo keyboard layout. Worse, they
have a weird bug with <code>alt-gr</code> (they call it <code>altgraph</code> in the config), that is not correctly recognized. It
sometimes work but I never recall the steps I have to do to fix the problem, and most of the time, I spend a
lot of times finding workarounds for something that just works in terminals.</li>
<li>It’s slow af. You can feel the whole Web with you! :D</li>
<li>Sometimes, after an update, a plugin will break, and you will basically lose a feature. This is a problem that
I have observed with other Web-based softwares (such as the GNOME Desktop Environment), which makes me doubt
more and more that kind of tech choice.</li>
</ul>
<h1>Wrap it up</h1>
<p>When I started coding, I remember of people talking about IDE / editor wars. Today, as I have tried a lot of editors,
I can say that there’s no such thing as an editor war to me. All editors have drawbacks and picking the right editor
is often a matter of taste and personal experience. I’m a keyboard lover (I make my own keyboards) and I truly love
typing — not necessarily code, hence it was pretty obvious to go to <a href="https://www.gnu.org/software/emacs">emacs</a> and vim back then (I actually started
coding in <a href="https://www.gnu.org/software/emacs">emacs</a>). Several years later, I joined the vim side and <a href="https://neovim.io">neovim</a>. It’s only one year ago that I started
looking at <a href="https://www.gnu.org/software/emacs">emacs</a> again to see what has changed. And oh my. So many interesting things out there!</p>
<p>What I like about testing editors is that each and every editor has at least one killer feature no other has:</p>
<ul>
<li>vim and neovim have modal editing and are super fast. They have been the editors of choice for modal editing for
decades and all the motions, macros, commands and mnemonics are implemented the best in those two editors.</li>
<li>IntelliJ IDEA has, to me, the best Java experience out there, with impressive (and so useful!) refactoring features.
It’s not something you’d be absolutely need to be productive, but it will make you feel much comfortable working on
your Java codebase, and I truly wish a plugin existed for that in the editors I use!</li>
<li>VS Code has the best LSP implementation and has — I guess? — on the biggest community. If you like Web-based
editors, don’t seek no more: it’s the right editor for you.</li>
<li>emacs and DOOM emacs have that slick, united interface with lots of amazing plugins and applications. You’ll
completely adore Org-mode, Magit and lots of other plugins!</li>
<li>atom has that echo mode for keybindings, superb defaults for theming and syntax highlighting and is one of — if not
the most? — the friendliest editors out there.</li>
</ul>
<p>Spending some weeks in all those editors made me realize something about vim / neovim: I don’t necessarily think I
should be using them, especially since I have used emacs / DOOM Emacs with its Evil-mode. Today, my thoughts revolve
around the idea that a good neovim client could be a gtk application like emacs: slick, united, with great defaults
and full support of neovim features, with support for gtk floating windows and popups (as it’s native in neovim and
feels a bit hacky in TUI). We already have great plugins for things like git (fugitive / vim-gitgutter), completion
and syntax highlighting with coc.nvim / vim-lsp / vim-treesitter. The only piece that is missing to me is a great
GUI to leverage the “hacks” we have to do in TUI to actually use real popups, “balloons”, etc. Once we get a decent
neovim GUI, I think I will have found my favorite editor of all time.</p>
<p>Until then, I’ll stick around neovim TUI as it’s what is as close at what I would like to get. I hope this article
will have given a bit of hints about what a vim / neovim enthusiast might expect from a modern editor. And I say
<em>a vim enthusiast</em>, not <em>all of them</em>. We all look for different things and I truly thing we live in a wonderful world
with all those editors out there. They won’t fit for everybody, but everybody will find one for them, and damn, that’s
a pretty good conclusion to this article.</p>
<p><a href="https://www.reddit.com/r/programming/comments/inx2ul/my_thoughts_about_editors_in_2020">Feel free to tell me</a>
about what you think about your editor, whether there’re things you dislike about it or what you would like to see
appear in it. Keep the vibes!</p>


guid: https://strongly-typed-thoughts.net/blog/luminance-0.31Thu, 29 Aug 2019 13:12:00 GMT
title: Announcement: luminance-0.31, luminance-derive and luminance-glutin
author: Dimitri Sabadie <dimitri.sabadie@gmail.com>
url: https://strongly-typed-thoughts.net/blog/luminance-0.31
feedurl: https://phaazon.net/blog/feed
content:
<p><a href="https://crates.io/crates/luminance">luminance</a> has been having lots of activity lately. I’ve been working (hard) on making it to the
<code>1.0</code> release for months now and came to a very important realization: lots of changes have been
done and I’m still not ready to release it. I’ve kind of lost myself in the process.</p>
<p>This article is about two main topics:</p>
<ul>
<li>The new release of <a href="https://crates.io/crates/luminance">luminance</a> and its changes.</li>
<li>The introducing of ways to learn <a href="https://crates.io/crates/luminance">luminance</a> (<em>yay</em>!).</li>
</ul>
<blockquote>
<p>If you don’t care about the changes, feel free to go read
<a href="https://github.com/phaazon/luminance-rs#learning">this</a> to know how you can learn about
<a href="https://crates.io/crates/luminance">luminance</a> and graphics programming in general.</p>
</blockquote>
<h1>Motivation</h1>
<p>Lately, <a href="https://crates.io/crates/luminance">luminance</a> has received a wide range of feature additions, bug fixes and design ideas. I’ve
been wanting to release all those changes as part of the <code>1.0</code> release but the thing is: among all
the changes, some have been around on the <code>master</code> branch for months without a proper release on
crates.io… because the <code>1.0</code> milestone was not reached. People have been moving towards <a href="https://crates.io/crates/luminance">luminance</a>
more and more and some provided feedback about using <a href="https://crates.io/crates/luminance">luminance</a>. Happy but indecisive about what to
do, I faced a dilemma:</p>
<ul>
<li>Either wait and release everything as part of <code>1.0</code> but eventually <em>block</em> people from using all
the cool features of <a href="https://crates.io/crates/luminance">luminance</a> because the last release on crates.io is months old.</li>
<li>Break the feature set into two releases: <code>1.0</code> for everything and <code>0.31</code> for all the new
candies.</li>
</ul>
<p>I have decided to go with the second option.</p>
<h1>luminance-0.31</h1>
<p>Just before writing this article, the <a href="https://crates.io/crates/luminance/0.30.1">last luminance version</a> was <code>0.30</code> — just for the record,
that version is eleven months old, ~200 commits behind <code>master</code>. The new and most recent version is
thus <code>0.31</code> and crates got updated:</p>
<ul>
<li><a href="https://crates.io/crates/luminance/0.31.0">luminance-0.31</a></li>
<li><a href="https://crates.io/crates/luminance-windowing/0.3.0">luminance-windowing-0.3</a></li>
<li><a href="https://crates.io/crates/luminance-glfw/0.6.0">luminance-glfw-0.6</a></li>
<li><a href="https://crates.io/crates/luminance-derive/0.1.0">luminance-derive-0.1</a>, <strong>a new crate</strong> that brings procedural macros for the <code>#[derive(..)]</code>
construct.</li>
<li><a href="https://crates.io/crates/luminance-glutin/0.1.0">luminance-glutin-0.1</a>, <strong>a new crate</strong> supporting the <a href="https://crates.io/crates/glutin">glutin</a> windowing crate. Currently, not
all features are implemented but it’s usable.</li>
</ul>
<p>Lot of work has been accomplished and I received several contributions from people all around the
globe, including PRs and issues. I’d like to remind that I appreciate both and you are <strong>really</strong>
encouraged to contribute in <em>any way you want to</em>. Special thanks fly to:</p>
<ul>
<li><a href="https://github.com/ambihelical">@ambihelical</a></li>
<li><a href="https://github.com/linkmauve">@linkmauve</a></li>
<li><a href="https://github.com/bobtwinkles">@bobtwinkles</a></li>
<li><a href="https://github.com/twetzel59">@twetzel59</a></li>
<li>And all others whom I discussed with and who provided feedback, especially on Reddit.</li>
</ul>
<p>About that last point, my Reddit and Twitter interactions about <a href="https://crates.io/crates/luminance">luminance</a> have been <em>very</em>
interesting because I got to test the water about how people feel about <a href="https://crates.io/crates/luminance">luminance</a>, especially
when compared to (not so) similar crates, such as [gfx], <a href="https://crates.io/crates/glium">glium</a> or even <a href="https://crates.io/crates/gl">gl</a>. I came to the
realization, after reading people and what they would love to have as a graphics crate, that
<a href="https://crates.io/crates/luminance">luminance</a> <em>can</em> have the role of the <em>easy</em> crate, that is not necessarily the fastest but fast
enough to be quickly productive. I cannot help it but keep thinking about a simple question: if
some people can make games in Java or C# with a <em>garbage collector</em> or using old tech like
<em>OpenGL 2.1</em> (yes, some people still make pretty good games with that), why would one need a
perfect zero-cost abstraction down-to-the-metal unsafe ultra-optimized crate to write something?
See, I went round and squares about that topic, because I’ve already used <a href="https://crates.io/crates/luminance">luminance</a> in several
projects of mine (mostly demoscene purposes), and <em>it just does the job</em>. So, yes, <a href="https://crates.io/crates/luminance">luminance</a>
doesn’t have that cool, new and modern <em>Vulkan-type</em> API, I must confess. But it has its own API,
which is very functional-based (see the <a href="https://github.com/phaazon/luminance-rs#history">History</a>
section of <a href="https://crates.io/crates/luminance">luminance</a> for further details about that) and, to me, modern enough so that people
don’t get frustrated with the overall design being too clunky.</p>
<p>So, yeah. I gave up on the <a href="https://phaazon.net/blog/pre-luminance-n-random-thoughts">idea of introducing backends</a>
in <a href="https://crates.io/crates/luminance">luminance</a>. I really changed my mind several times about that topic and it’s actually when I
read comments on Reddit about people getting confused about the direction of the crate that I made
up my mind: <a href="https://crates.io/crates/luminance">luminance</a> must remain simple. Having a system of backends that can be hot-switched,
concurrent etc. is just going to make things hard for people to use it — and maintain it! It’s
likely that I will introduce, however, <em>feature-gates</em> to allow to compile <a href="https://crates.io/crates/luminance">luminance</a> on
WebAssembly via WebGL, <em>OpenGL ES</em> or even Vulkan at some point, but the difference is that no backend
will be implemented. That means that those feature-flags aren’t likely to be summable at first. But
all of this will be done in future releases; stay tuned.</p>
<p>The current blog post brings a description of all the changes of <a href="https://crates.io/crates/luminance/0.31.0">luminance-0.31</a>. It should be the
last <code>0.*.*</code> version before hitting the <code>1.0</code> release. To the question:</p>
<blockquote>
<p><em>Why not releasing <code>1.0</code> directly?</em></p>
</blockquote>
<p>I answer that the <code>1.0</code> milestone has a backlog with two major changes that will take time to
implement and I think it would be a pity to postpone a lot of great changes that are already
available because of two features that are yet to be implemented. The concept of versions is to
allow releasing features in a structured way without having to wait too much. So here we are.</p>
<p>This post also show cases a small tutorial about how to get started with <a href="https://crates.io/crates/luminance">luminance</a>. The very first
steps you should have would be to have a look at the <a href="https://github.com/phaazon/luminance-rs/blob/master/luminance/examples/README.md">examples/</a> directory and try to play with all
of the samples.</p>
<blockquote>
<p>Disclaimer: the following section of this article is based on
<a href="https://github.com/phaazon/luminance-rs/blob/master/luminance/CHANGELOG.md#031">luminance’s changelog</a>.</p>
</blockquote>
<h2>Bug fixes</h2>
<ul>
<li>Fix and remove <code>panic!</code> and attributeless renders.</li>
<li>Various internal bug fixes and performance improvements.</li>
<li>Fix pixel code for <code>Format::R</code> and <code>Format::RG</code> when querying a texture’s texels.</li>
</ul>
<h2>Major changes</h2>
<ul>
<li>Remove the concept of <code>GTup</code>. No code was using it and it was not really elegant.</li>
<li>Remove the <code>uniform_interface!</code> macro and replace it with the <code>UniformInterface</code> procedural
derive macro.</li>
<li>Buffer mapping is now always a <code>mut</code> operation. That is required to <em>lock-in</em> the mapped slices
and prevent to generate new ones, which would be an undefined behavior in most graphics backends
such as <em>OpenGL</em>.</li>
<li>Change the framebuffer’s slots types and meanings. Those are now more natural to use (for
instance, you don’t have to repeat the framebuffer’s associated types and dimensions nor even
use the <code>Texture&lt;..&gt;</code> type anymore, as a type family is now used to ease the generation of
color and depth slots).</li>
<li>Change the way the <code>Vertex</code> trait is implemented.
<ul>
<li>The <code>Vertex::vertex_format</code> method has been renamed <code>Vertex::vertex_desc</code>.</li>
<li>Instead of returning a <code>VertexFormat</code>, that method now returns a <code>VertexDesc</code>. Where a
<code>VertexFormat</code> was a set of <code>VertexComponentFormat</code>, a <code>VertexDesc</code> is a set of
<code>VertexBufferDesc</code>.</li>
<li><code>VertexBufferDesc</code> is a new type that didn’t exist back then in <em>0.30</em>. It provides new data
and information about how a vertex attribute will be spread in a GPU buffer. Especially, it has:
<ul>
<li>An <em>index</em>, allowing to map the vertex attribute in a shader.</li>
<li>A <em>name</em>, used by shader programs to perform mapping.</li>
<li>An <em>instancing</em> parameter, used to determine whether we want <strong>vertex instancing</strong>.</li>
<li>A <code>VertexAttribDesc</code>, the new name of <code>VertexComponentFormat</code>.</li>
</ul>
</li>
<li>As said above, <code>VertexComponentFormat</code> was renamed <code>VertexAttribDesc</code>.</li>
<li>Vertex attribute can now be <em>normalized</em> if they are <em>signed integral</em> or <em>unsigned integral</em>.
That is encoded in the <code>VertexAttribType</code>’s integral variants.</li>
<li>A new trait has appeared: <code>VertexAttrib</code>. Such a trait is used to map a type to a
<code>VertexAttribDesc</code>.</li>
<li><code>Vertex</code> has zero implementor instead of several ones in <em>0.30</em>. The reason for that is that
<code>VertexBufferDesc</code> is application-driven and depends on the <em>vertex semantics</em> in place in the
application or library.</li>
<li>Vertex semantics are introduced in this release and are represented via the <code>Semantics</code> trait.
Implementing directly <code>Semantics</code> is possible, even though not recommended. Basically,
<code>Semantics</code> provides information such as the <em>index</em> and <em>name</em> of a given semantics as long
as the list of all possible semantics, encoded by <code>SemanticsDesc</code>.</li>
<li>Users are highly advised to look at the <code>Vertex</code> and <code>Semantics</code> proc-macro derive in the
[luminance-derive] crate.</li>
</ul>
</li>
<li>Revise the <code>Tess</code> type to make it easier to work with.
<ul>
<li>The <code>Tess::new</code> and <code>Tess::attributeless</code> functions were removed.</li>
<li>The <code>TessBuilder</code> type was added and replace both the above function.</li>
<li>That last type has a lot of methods that can be combined in different ways to build powerful
situation of tessellations, among (but not limited to):
<ul>
<li>Normal and indexed tessellations.</li>
<li>Attributeless tessellations.</li>
<li>Tessellations with vertex instancing support.</li>
<li>Deinterleaved tessellations</li>
<li>Tessellations with support for <em>primitive restart indexing</em>.</li>
</ul>
</li>
<li>Slicing was revised too and now has support for two new Rust operators:
<ul>
<li>The <code>a ..= b</code> operator, allowing to slice a <code>Tess</code> with inclusive closed bounds.</li>
<li>The <code>..= b</code> operator, allowing to slice a <code>Tess</code> with inclusive bounds open on the left
side.</li>
</ul>
</li>
<li>Previously, the <code>Tess::new</code> associated function expected indices to be a slice of <code>u32</code>. This
new release allows to use any type that implements the <code>TessIndex</code> trait (mapping a type to a
<code>TessIndexType</code>. Currently, you have <code>u8</code>, <code>u16</code> and <code>u32</code> available.</li>
<li>Add <code>Tess::{as_index_slice,as_index_slice_mut}</code>. Those now enable you to conditionally slice-map
the <em>index buffer</em> of a <code>Tess</code>, if it exists.</li>
</ul>
</li>
<li>Add support for generic texture sampling.
<ul>
<li>This new feature is supported thanks to the <code>SamplerType</code> trait, used as constraint on the
<code>Pixel::SamplerType</code> associated type.</li>
<li>Basically, that feature allows you to bind a <code>Floating</code> texture without caring about the
actual type. That is especially true as you typically use <code>sampler2D</code> in a shader and not
<code>sampler2DRGB32F</code>.</li>
<li>Such a feature reduces the number of combination needed to refactorize code.</li>
</ul>
</li>
<li>Implement <em>vertex attrib explicit binding</em>. This is a huge change that is related to <em>vertex
semantics</em>. Basically, in <em>0.30</em>, you have to ensure that the <code>layout (location = _)</code> is
correctly set to the right value regarding what you have in your <code>Tess</code>’ vertex buffers. That
was both <em>unsafe</em> and terribly misleading (and not very elegant). The new situation, which
relies on <em>vertex semantics</em>, completely gets rid of <em>vertex locations</em> worries, which get
overrided by <a href="https://crates.io/crates/luminance">luminance</a> when a shader program gets linked.</li>
<li>Change boolean-like <code>enum</code>s — such as <code>DepthTest</code> — variants from <code>Enabled</code> / <code>Disabled</code> to
<code>On</code> / <code>Off</code> or <code>Yes</code> / <code>No</code>, depending on the situation.</li>
<li>Move <code>swap_buffers</code> from <code>GraphicsContext</code> to <code>Surface</code> in [luminance-windowing].</li>
<li>Switch to <code>GenMipmaps</code> instead of <code>bool</code> to encode whether mipmaps should be generated in
texture code. That change is a readability enhancement when facing texture creation code.</li>
<li>Make <code>Dimensionable::zero_offset()</code> a constant, <code>Dimensionable::ZERO_OFFSET</code>.</li>
<li>Change the way cursor modes are encoded from <code>bool</code> to <code>CursorMode</code>.</li>
</ul>
<h2>Minor changes</h2>
<ul>
<li>Add the [luminance-glutin] crate, the windowing crate support for <a href="https://crates.io/crates/glutin">glutin</a>.</li>
<li>Add the [luminance-derive] crate.
<ul>
<li>That crate provides several procedural derive macros you can use to easily implement all
required traits to work with <a href="https://crates.io/crates/luminance">luminance</a>. Especially, some traits that are <code>unsafe</code> can be
implemented in a safe way with that crate, so you should definitely try to use it.</li>
<li>Current available proc-macros are:
<ul>
<li><code>#[derive(Vertex)]</code>: derive the <code>Vertex</code> trait for a <code>struct</code>.</li>
<li><code>#[derive(Semantics)]</code>: derive the <code>Semantics</code> trait for an <code>enum</code>.</li>
<li><code>#[derive(UniformInterface)]</code>: derive the <code>UniformInterface</code> trait for a <code>struct</code>.</li>
</ul>
</li>
</ul>
</li>
<li>Support for dynamic uniform queries. Those are used whenever you don’t know which variables will
be used in a shader at compile-time. This might be the case if you’re writing a GUI tool or a
video game that uses a custom scripting language / node-ish representation of shaders. That
feature doesn’t break the already-in-place and great uniform interface but complements it. You
can use a shader <code>Program&lt;_, _, ()&gt;</code> and still set uniform values by querying the uniforms
dynamically. This feature also fully benefits from the strongly typed interface of <code>Uniform&lt;_&gt;</code>,
so you will get <code>TypeMismatch</code> runtime error if you try to trick the type system.</li>
<li>Add the <code>std</code> feature gate, allowing to compile with the standard library – this is enabled by
default. The purpose of this feature is to allow people to use <code>default-features = false</code> to
compile without the standard library. This feature is currently very experimental and shouldn’t
be used in any production releases so far – expect breakage / undefined behaviors as this
feature hasn’t been quite intensively tested yet.</li>
<li>Add support for the <code>R11FG11FB10F</code> pixel format.</li>
<li>Migrate to Rust Edition 2018.</li>
<li>The <code>WindowOpt</code> now has support for multisampling. See the <code>WindowOpt::set_num_samples</code> for
further details.</li>
<li>Implement dynamic edition of windowing types properties. That allows to change data on-the-fly,
such as the cursor mode.</li>
<li>Introduce normalized texturing. That feature is encoded as pixel formats: any pixel format which
symbol’s name starts with <code>Norm</code> is a <em>normalized pixel format</em>. Such formats state that the
texels are encoded as integers but when fetched from a shader, they are turned into
floating-point number by normalizing them. For instance, when fetching pixels from a texture
encoded with <code>R8UI</code>, you get integers ranging in <code>[0; 255]</code> but when fetching pixels from a
texture encoded with <code>NormR8UI</code>, even though texels are still stored as 8-bit unsigned integers,
when fetched, you get floating-point numbers comprised in <code>[0; 1]</code>.</li>
</ul>
<h2>Patch &amp; misc changes</h2>
<ul>
<li>Remove <code>std::mem::uninitialized</code> references, as it is now on deprecation path. Fortunately, the
codes that were using that function got patched with safe Rust (!) and/or simpler constructs.
It’s a win-win.</li>
<li>Add the <code>#[repr(C)]</code> annotation on vertex types in examples. That is a bit unfortunate because
such an annotation is very likely to be mandatory when sending data to the GPU and it should be
done automatically instead of requiring the user to do it. That situation will be fixed in a
next release.</li>
<li>Add more CI testing.</li>
<li>Update examples and made them available to the <code>cargo run --example</code> command. Read more
<a href="./examples/README.md">here</a>.</li>
<li>Massive documentation rewrite (among the use of <code>#![deny(missing_docs)]</code>. The situation is still
not perfect and patch versions will be released to fix and update the documentation. Step by
step.</li>
<li>Add design notes and documents in the repository.</li>
<li>Massive dependencies update. Special thanks to @eijebong for his help!</li>
<li>Add the <code>11-query-texture-texels</code> example, which showcases how to query a texture’s texels and
drop it on the filesystem.</li>
<li>Add and update <em>README</em> files. Especially, the gitter link was removed and an IRC link was
added. If you want to get help:
<ul>
<li>Server: <code>irc.freenode.net</code></li>
<li>Channel: <code>#luminance</code></li>
</ul>
</li>
</ul>
<h1>Learning luminance</h1>
<p>I also spent quite a lot of time and energy working on two ways to learn <a href="https://crates.io/crates/luminance">luminance</a>:</p>
<ul>
<li><a href="https://github.com/phaazon/luminance-rs/blob/master/luminance/examples/README.md">Examples</a>.</li>
<li><a href="https://github.com/phaazon/luminance-rs/wiki">The official, brand new wiki</a>.</li>
</ul>
<p>So far, I’m writing articles and the wiki is going to be updated let’s say, every week, if I find
enough spare time. You can also tell me what kind of stuff you’d like to learn and do.</p>
<p>The <a href="https://docs.rs/luminance/">official documentation</a> is up to date but is not as good as I
expect it to be. I will be adding patches versions to <em>0.31</em> to update it.</p>
<p>I hope you like <a href="https://crates.io/crates/luminance">luminance</a> and will make a good use of it. And of course, keep the vibes!</p>


guid: https://strongly-typed-thoughts.net/blog/glsl-pest-part-2Sat, 17 Nov 2018 20:40:00 GMT
title: Introducing pest into glsl and hindsight about nom vs. pest (part 2)
author: Dimitri Sabadie <dimitri.sabadie@gmail.com>
url: https://strongly-typed-thoughts.net/blog/glsl-pest-part-2
feedurl: https://phaazon.net/blog/feed
content:
<p>This is the second article about my experience at supporting <a href="https://crates.io/crates/pest">pest</a> in my <a href="https://crates.io/crates/glsl">glsl</a> crate – without,
for now, removing the <a href="https://crates.io/crates/nom">nom</a> parser.</p>
<blockquote>
<p>You can read the first article
<a href="https://phaazon.net/blog/glsl-pest-part-1">here</a>.</p>
</blockquote>
<h1>RY: DRY without D</h1>
<p>I feel like I’m either doing something wrong… or that something is wrong with using <a href="https://crates.io/crates/pest">pest</a> to build
typed <a href="https://en.wikipedia.org/wiki/Abstract_syntax_tree">AST</a>s. I spent a whole day writing the GLSL450 <a href="https://en.wikipedia.org/wiki/Parsing_expression_grammar">PEG</a>. After having the compiler stop failing
with left-recursion errors, I was finally able to test my crate!… or was I?</p>
<p>First thing first, I decided to write tests to see whether I can recognize <em>identifiers</em>. I’m a
TDD enthusiast – don’t get the idea twisted: writing tests is boring; but writing tests will save
your life and prevent small and cute kittens joining a project from drowning (true story).</p>
<p>I have a PEG rule for that, <code>identifier</code>. So the idea is the following:</p>
<pre><code>// we set the error type to String for that first version
fn parse_identifier&lt;'a, S&gt;(input: S) -&gt; Result&lt;syntax::Identifier, String&gt;
where S: Into&lt;&amp;'a str&gt; {
  // here we get a set of pairs that match the identifier
  let pairs = Parser::parse(Rule::identifier, input.into()).map_err(|e| format!(&quot;{}&quot;, e))?;

  // since only a full string identifier would match, we can just simply ask to retrieve the string
  // that composes the pair
  Ok(pairs.as_str().to_owned())
}
</code></pre>
<p>As you can see, we have lost the grammar information that states that the <code>identifier</code> rule outputs
a single string – see in the code above how we get a <code>Pairs</code> object, assigned to <code>pairs</code>. This is
my first concern: the <code>Rule</code> type should contain information about what kind of values it was
matched against.</p>
<p>A more drastic example is the <code>primary_expression</code> rule, that defines expressions that can be either
an <code>identifier</code>, a <code>float_constant</code>, <code>int_constant</code>, <code>bool_constant</code> or any general expression with
parenthesis around. Its parser code would look like this:</p>
<pre><code>fn parse_primary_expression&lt;'a, S&gt;(input: S) -&gt; Result&lt;syntax::Expr, String&gt;
where S: Into&lt;&amp;'a str&gt; {
  let pairs = Parser::parse(Rule::primary_expression, input.into()).map_err(|e| format!(&quot;{}&quot;, e))?;

  // we know we only have one expression, so this statement seems a bit out of nowhere here
  let pair = pairs.next().unwrap();

  // now, we know the primary expression is correctly tokenized, so we’re interested in “what’s
  // inside”; here, we HAVE to look at the grammar again to check what are the possible variants;
  // only one possible, let’s take the next pair then as well…
  let inner = pair.into_inner().next().unwrap();

  // here the pair represents the “real” and “useful” sub-rule that was matched; in order to write
  // that match block, we also have to look at the grammar to see all the possible variants
  match inner.as_rule() {
    Rule::identifier =&gt; {
      // the pair represents an identifier… we can just turn it to string and return it
      Ok(syntax::Expr::Variable(inner.as_str().to_owned())) // (1.)
    }

    Rule::float_constant =&gt; {
      // the pair represents a float constant; let’s just parse it
      let f = inner.as_str().parse().unwrap();
      Ok(syntax::Expr::FloatConstant(f))
    }

    Rule::etcetc. =&gt; {
      // etc. etc.
    }

    _ =&gt; unreachable!() // eeew (2.)
  }
}
</code></pre>
<p>As you can see, we have several problems here</p>
<ol>
<li>We need to write our parsers taking as input <code>Pair</code> so that we can compose them… that strangely
resembles how the <a href="https://crates.io/crates/nom">nom</a> parser is actually written… :)</li>
<li>We just repeat the actual grammar! The whole PEG file is just a way to tell our lexer (<a href="https://crates.io/crates/pest">pest</a>)
how to recognize tokens and how to store them in a tree. That’s all. All the parsing analysis
must be done by <em>repeating ourselves</em>.</li>
<li>Whenever we decide to change the grammar by, for instance, modifying the content of a rule
(without changing its name), we’re lost. Done. Terminus. <em>Destination fucked</em>. Destination
sneakily stabbed in the back. The code will still compile but now you will get a runtime error
linked to <code>Result::unwrap</code>, a <code>Parse::parse</code> failure, etc.</li>
</ol>
<p>Maybe I’m plain wrong and that there’s an easier way to do that, but I’ve been writing the parsers
of only a few rules (on more than fifty) and I already cringe.</p>
<h1>Two-stage parsers</h1>
<p>All of this brings a brand new problem: since we’re smart developers and want to write the most
reusable code, we want to be able to write the parser of <code>primary_expression</code> once and reuse it in
other parsers that might need it – like <code>postfix_expression</code>, <code>expression</code>, etc. The current code
that consumes <code>Into&lt;&amp;str&gt;</code> doesn’t allow this as <a href="https://crates.io/crates/pest">pest</a> parses to <code>Pairs</code>. So let’s just write our
functions to take <code>Pair</code> as inputs!</p>
<p>But… now we don’t have a proper public facing interface for our crate. Surely, I don’t want people
to even see that <a href="https://crates.io/crates/pest">pest</a> is used – they’ll just see it as a dependency, but I don’t want any <a href="https://crates.io/crates/pest">pest</a>
symbols in my public interface.</p>
<p>That problem can be solved by introducing a trait, <code>Parse</code>, for instance, that has the right 
parsing signature:</p>
<pre><code>/// Class of types that can be parsed.
pub trait Parse: Sized {
  /// Parse an item from a string.
  ///
  /// Store errors as strings for convenience for now.
  fn parse&lt;'a, S&gt;(input: S) -&gt; Result&lt;Self, String&gt; where S: Into&lt;&amp;'a str&gt;;
}
</code></pre>
<p>Then, to support a new syntax symbol, we must:</p>
<ol>
<li>Write a <code>Pair</code> parser.</li>
<li>Implement our <code>Parse</code> trait for this type, using the <code>Pair</code> parser from just above.</li>
<li>Eventually reuse the <code>Pair</code> parser in the implementations of other <code>Pair</code> parsers if
needed.</li>
</ol>
<h1>Is it the right way to go?</h1>
<p>I must say: I’m astonished by the complexity and verbosity – and dangerosity! – of all of this. This
is “on me” because I’ve constrained myself here:</p>
<ul>
<li>I don’t want <a href="https://crates.io/crates/pest">pest</a> to leak any symbol into my public interface.</li>
<li>I want to be able to reuse parsers for a maximum of coherency between parsers – and more
readable parsing code base.</li>
<li>I want my final output to be folded into an AST type of mine. More specifically, I want users of
the <a href="https://crates.io/crates/glsl">glsl</a> crate to be able to <code>parse</code> any type from the grammar / syntax. This is currently the
case with the <a href="https://crates.io/crates/nom">nom</a> parser, that uses a similar trait. To implement the trait, I just defined a
very small <code>macro_rules</code> that simply invokes the required parser for the given type.
<a href="https://github.com/phaazon/glsl/blob/a31b701850dd7966b14cc59f10c8d455c755c9c6/src/parser.rs#L99">Code here</a>.</li>
</ul>
<p>My main problem here is that there’s no way to have <a href="https://crates.io/crates/pest">pest</a> behave as a <a href="https://en.wikipedia.org/wiki/Scannerless_parsing">scannerless parser</a>: you
will always have this two-phases lexer-parser:</p>
<ol>
<li>Run the lexer to tokenize the input string into a tree of tokens.</li>
<li>Run a (type-blind) parser that somehow <em>rediscovers</em> the token tree.</li>
</ol>
<p>I would be okay with this, however:</p>
<ol>
<li>Run the lexer to tokenize the input string into a <em>typed</em> tree of tokens.</li>
<li>Run a full-type-aware parser that visits the token tree by following typed rules.</li>
<li>It would be even easier if I could write the parser code <em>inline</em>, directly in the PEG rule.</li>
</ol>
<p>Once compiled, a <a href="https://crates.io/crates/pest">pest</a> grammar is basically a very big and flat <code>Rule</code> type. If you have this PEG:</p>
<pre><code>WHITESPACE = _{ &quot; &quot; | NEWLINE }

number = { ASCII_DIGIT+ }
op = { &quot;-&quot; | &quot;+&quot; | &quot;*&quot; }
expr = {
  number ~ (op ~ expr)? |
  &quot;(&quot; ~ expr ~ &quot;)&quot;
}
</code></pre>
<p>A possible AST – one I would definitely write – for that would be:</p>
<pre><code>#[derive(Clone, Copy, Debug, Eq, Hash, PartialEq)]
enum Op {
  Minus,
  Plus,
  Mul
}

#[derive(Clone, Debug, Eq, Hash, PartialEq)]
enum Expr {
  Number(i32),
  Op(Number, Op, Box&lt;Expr&gt;)
}
</code></pre>
<p>However, <a href="https://crates.io/crates/pest">pest</a> outputs something like this:</p>
<pre><code>enum Rule {
  WHITESPACE, // very useful, thank you!
  number,
  op,
  expr
}
</code></pre>
<p>And implements the <code>Parser</code> trait to yield <code>Pairs</code> according to the variant of <code>Rule</code> you choose to
parse with. Instead, a more practical encoding of rules would be:</p>
<pre><code>struct Rule_number;
impl SuperMegaParser&lt;i32&gt; for Rule_number {}

struct Rule_op;
impl SuperMegaParser&lt;Op&gt; for Rule_op {}

struct Rule_expr;
impl SuperMegaParser&lt;Expr&gt; for Rule_expr {}
</code></pre>
<p>That is, <code>Parser::parse(Rule_expr)</code> wouldn’t yield <code>Pairs</code> anymore, but <code>Expr</code>. In order for this
to be possible, we would need to tell our <a href="https://en.wikipedia.org/wiki/Scannerless_parsing">scannerless parser</a> what to do when matching a rule:</p>
<pre><code>WHITESPACE = _{ &quot; &quot; | NEWLINE } // who cares about whitespaces, seriously? they’re MUTED! :D

number: i32 = { ASCII_DIGIT+ =&gt; |s| s.parse().unwrap() }

op: Op = {
  &quot;-&quot; =&gt; Op::Minus |
  &quot;+&quot; =&gt; Op::Plus |
  &quot;*&quot; =&gt; Op::Mul
}

expr: Expr = {
  number ~ (op ~ expr)? =&gt; |(n, opt_op_expr)|{
    match opt_op_expr {
      Some((op, expr)) =&gt; Expr::Op(n, op, Box::new(expr)),
      None =&gt; Expr::Number(Number(n))
    }
  }|

  &quot;(&quot; ~ expr ~ &quot;)&quot;
}
</code></pre>
<p>This would be perfect to me. And I reckon it’s pretty much what <a href="https://github.com/lalrpop/lalrpop">lalrpop</a> uses.</p>
<h1>Conclusion</h1>
<p>I’ve been feeling on and off about <a href="https://crates.io/crates/pest">pest</a> lately, to be very honest. At first I was amazed at the
PEG file, because, yeah, PEG is lovely to work with. However, I think GLSL450 is a really good
candidate to test a lexer / parser and to that matter, the current vanilla <a href="https://crates.io/crates/pest">pest</a> is a nah to me.
It makes the code terribly bloated and harder to maintain – while it should be easier! – than the
reference <a href="https://crates.io/crates/nom">nom</a> implementation. The very reason to that is that even if I had to write thousands
line of macros calls – yiiiik – with <a href="https://crates.io/crates/nom">nom</a>, those macros are correctly typed. The <code>identifier</code> <a href="https://crates.io/crates/nom">nom</a>
parser is a function taking bytes and outputing… <code>syntax::Identifier</code>. Same thing for all other
parsers.</p>
<p>I had to try <a href="https://crates.io/crates/pest">pest</a>. From my – limited – experience of it, I’d say <strong>it’s definitely not a
parser</strong>. <a href="https://crates.io/crates/nom">nom</a> is – a <a href="https://en.wikipedia.org/wiki/Scannerless_parsing">scannerless parser</a>. <a href="https://crates.io/crates/pest">pest</a> <strong>is a lexer</strong> that does a <em>few parsing</em> work
to sort and fold the lexemes (tokens) into a tree. You can see <a href="https://crates.io/crates/pest">pest</a>’s output as a big regular
acyclic tree holding pairs of pointers (representing tokens in the input source). That’s everything
<a href="https://crates.io/crates/pest">pest</a> gives you. In order to turn that representation into a typed AST, a lot of work is awaiting
you. I’ll take a few hours / days to think about what I should do and work on other projects in the
meantime. I doubt I’ll keep going with <a href="https://crates.io/crates/pest">pest</a> because I feel like I’m going to spend entire days
repeating myself by looking the grammar up to, frustrated, shout on IRC that I’m writing untyped
code while I have static assurances (i.e. remember: the grammar is read at compile-time) that I will
never have to look at the <code>Rule::external_declaration</code> while parsing a <code>Rule::postfix_expression</code>.
And as I might be changing the grammar rule a bit when refactoring / fixing bugs, I really really
don’t plan to pay a visit to <em>destination fucked</em> yet.</p>
<blockquote>
<p>Someone on reddit suggested me to have a look at <a href="https://github.com/pest-parser/ast">pest-ast</a>.
Even if this looks promising, it doesn’t seem finished and I think it should be merged into <a href="https://crates.io/crates/pest">pest</a>
directly when done.</p>
</blockquote>
<p>That’s all for me today. I might try a bit further to see if I find a way to make my <a href="https://crates.io/crates/pest">pest</a>
experience less painful but if I don’t, I might try that <a href="https://github.com/lalrpop/lalrpop">lalrpop</a> cutie in a few hours / days! As
a final note, I don’t want to state that <a href="https://crates.io/crates/pest">pest</a> is bad, I think it’s pretty cool and that it does
its job greatly, but my need (typed AST) might be off its scope. That’s all to note, I’d say. :)</p>
<p>Keep the vibes.</p>


guid: https://strongly-typed-thoughts.net/blog/wavefront-0.1Sun, 11 Oct 2015 00:00:00 GMT
title: Load geometries with wavefront-0.1!
author: Dimitri Sabadie <dimitri.sabadie@gmail.com>
url: https://strongly-typed-thoughts.net/blog/wavefront-0.1
feedurl: https://phaazon.net/blog/feed
content:
<p>I’ve been away from <a href="https://hackage.haskell.org/package/luminance">luminance</a> for a few days
because I wanted to enhance the graphics world of Haskell. luminance might be interesting, if you
can’t use the art works of your artists, you won’t go any further for a real-world application. I
decided that I to write a parser/lexer to load 3D geometries from files. The
<a href="https://en.wikipedia.org/wiki/Wavefront_.obj_file">Wavefront OBJ</a> is an old yet simple and
efficient way of encoding such objects. It supports materials, surfaces and a lot of other cool
stuff – I don’t cover them yet, though.</p>
<p>There’s a <a href="http://hackage.haskell.org/package/obj">package</a> out there to do that, but it hasn’t been
updated since 2008 and has a lot of dependencies I don’t like (InfixApplicative, OpenGL,
OpenGLCheck, graphicsFormats, Codec-Image-Devil, and so on…). I like to keep things ultra simple and
lightweight. So here we go. <a href="http://hackage.haskell.org/package/wavefront">wavefront</a>.</p>
<p>Currently, my package only builds up a pure value you can do whatever you want with. Upload it to
the GPU, modify it, pretty print it, perform some physics on it. Whatever you want. The interface is
not frozen yet and I need to perform some benchmarks to see if I have to improve the
performance – the lexer is very simple and naive, I’d be amazed if the performance were that good
yet.</p>
<p>As always, feel free to contribute, and keep in mind that the package will move quickly along the
performance axis.</p>


guid: https://strongly-typed-thoughts.net/blog/luminance_designSun, 28 Aug 2016 00:00:00 GMT
title: luminance designs
author: Dimitri Sabadie <dimitri.sabadie@gmail.com>
url: https://strongly-typed-thoughts.net/blog/luminance_design
feedurl: https://phaazon.net/blog/feed
content:
<p><a href="https://crates.io/crates/luminance/0.7.0">luminance-0.7.0</a> was released a few days ago and I
decided it was time to explain exactly what luminance is and what were the design choices I made.
After a very interesting talk with <a href="https://github.com/nical">nical</a> about other rust graphics
frameworks (e.g. <a href="https://crates.io/crates/gfx">gfx</a>, <a href="https://crates.io/crates/glium">glium</a>,
<a href="https://crates.io/crates/vulkano">vulkano</a>, etc.), I thought it was time to give people some more
information about luminance and how to compare it to other frameworks.</p>
<h1>Origin</h1>
<p>luminance started as <a href="https://hackage.haskell.org/package/luminance">a Haskell package</a>, extracted
from a <em>“3D engine”</em> I had been working on for a while called
<a href="https://github.com/phaazon/quaazar">quaazar</a>. I came to the realization that I wasn’t using the
Haskell garbage collector at all and that I could benefit from using a language without GC. Rust
is a very famous language and well appreciated in the Haskell community, so I decided to jump in and
learn Rust. I migrated luminance in a month or two. The mapping is described in
<a href="http://phaazon.blogspot.fr/2016/04/porting-haskell-graphics-framework-to.html">this blog entry</a>.</p>
<h1>What is luminance for?</h1>
<p>I’ve been writing 3D applications for a while and I always was frustrated by how OpenGL is badly
designed. Let’s sum up the lack of design of OpenGL:</p>
<ul>
<li><em>weakly typed</em>: OpenGL has types, but… it actually does not. <code>GLint</code>, <code>GLuint</code> or <code>GLbitfield</code> are
all defined as <em>aliases</em> to primary and integral types (i.e. something like
<code>typedef float GLfloat</code>). Try it with <code>grep -Rn &quot;typedef [a-zA-Z]* GLfloat&quot; /usr/include/GL</code>. This
leads to the fact that <em>framebuffers</em>, <em>textures</em>, <em>shader stages</em>, <em>shader program</em> or even
<em>uniforms</em>, etc. have the same type (<code>GLuint</code>, i.e. <code>unsigned int</code>). Thus, a function like
<code>glCompileShader</code> expects a <code>GLuint</code> as argument, though you can pass a framebuffer, because it’s
also represented as a <code>GLuint</code> – very bad for us. It’s better to consider that those are just
untyped – :( – handles.</li>
<li><em>runtime overhead</em>: Because of the point above, functions cannot assume you’re passing a value of
a the expected type – e.g. the example just above with <code>glCompileShader</code> and a framebuffer. That
means OpenGL implementations have to check against <em>all</em> the values you’re passing as arguments to
be sure they match the type. That’s basically <strong>several tests for each call</strong> of an OpenGL
function. If the type doesn’t match, you’re screwed and see the next point.</li>
<li><em>error handling</em>: This is catastrophic. Because of the runtime overhead, almost all functions
might set the <em>error flag</em>. You have to check the error flag with the <code>glGetError</code> function,
adding a side-effect, preventing parallelism, etc.</li>
<li><em>global state</em>: OpenGL works on the concept of global mutation. You have a state, wrapped in a
<em>context</em>, and each time you want to do something with the GPU, you have to change something in
the context. Such a context is important; however, some mutations shouldn’t be required. For
instance, when you want to change the value of an object or use a texture, OpenGL requires you to
<em>bind</em> the object. If you forget to <em>bind</em> for the next object, the mutation will occurs on the
first object. Side effects, side effects…</li>
</ul>
<p>The goal of luminance is to fix most of those issues by providing a safe, stateless and elegant
graphics framework. It should be as low-level as possible, but shouldn’t sacrifice
runtime performances – CPU charge as well as memory bandwidth. That is why if you know how to
program with OpenGL, you won’t feel lost when getting your feet wet with luminance.</p>
<p>Because of the many OpenGL versions and other technologies (among them, vulkan), luminance has an
extra aim: abstract over the trending graphics API.</p>
<h1>Types in luminance</h1>
<p>In luminance, all graphics resources – and even concepts – have their own respective type. For
instance, instead of <code>GLuint</code> for both shader programs and textures, luminance has <code>Program</code> and
<code>Texture</code>. That ensures you don’t pass values with the wrong types.</p>
<blockquote>
<p>Because of static warranties provided by compile-time, with such a scheme of strong-typing, the
runtime <strong>shouldn’t have to check for type safety</strong>. Unfortunately, because luminance <em>wraps over</em>
OpenGL in the luminance-gl backend, we can only add static warranties; we cannot remove the
runtime overhead.</p>
</blockquote>
<h1>Error handling</h1>
<p>luminance follows the Rust conventions and uses the famous <code>Option</code> and <code>Result</code> types to specify
errors. You will never have to check against a global error flag, because this is just all wrong.
Keep in mind, you have the <code>try!</code> macro in your Rust prelude; use it as often as possible!</p>
<blockquote>
<p>Even though Rust needs to provide an exception handler – i.e. panics – there’s no such thing as
exceptions in Rust. The <code>try!</code> macro is just syntactic sugar to:</p>
</blockquote>
<blockquote>
<pre><code class="language-rust">match result {
  Ok(x) =&gt; x,
  Err(e) =&gt; return e
}
</code></pre>
</blockquote>
<h1>Stateless</h1>
<p>luminance is stateless. That means you don’t have to bind an object to be able to use it. luminance
takes care of that for you in a very simple way. To achieve this and keep performances running, it’s
required to add a bit of high-level to the OpenGL API by leveraging how binds should happen.</p>
<p>Whatever the task you’re trying to reach, whatever computation or problem, it’s always better to
gather / group the computation by batches. A good example of that is how magnetic hard drive disks
work or your RAM. If you spread your data across the disk region (fragmented data) or across several
non-contiguous addresses in your RAM, it will end up by unnecessary moves. The hard drive’s head
will have to go all over the disk to gather the information, and it’s very likely you’ll destroy the
RAM performance (and your CPU caches) if you don’t put the data in a contiguous area.</p>
<p>If you don’t group your OpenGL resources – for instances, you render 400 objects with shader A, 10
objects with shader B, then 20 objects with shader A, 32 objects with shader C, 349 objects with
shader A and finally 439 objects with shader B, you’ll add more OpenGL calls to the equation – hence
more global state mutations, and those are costly.</p>
<p>Instead of this:</p>
<ol>
<li>400 objects with shader A</li>
<li>10 objects with shader B</li>
<li>20 objects with shader A</li>
<li>32 objects with shader C</li>
<li>348 objects with shader A</li>
<li>439 objects with shader B</li>
</ol>
<p>luminance <strong>forces</strong> you to group your resources like this:</p>
<ol>
<li>400 + 20 + 348 objects with shader A</li>
<li>10 + 439 objects with shader B</li>
<li>32 objects with shader C</li>
</ol>
<p>This is done via types called <code>Pipeline</code>, <code>ShadingCommand</code> and <code>RenderCommand</code>.</p>
<h2>Pipelines</h2>
<p>A <code>Pipeline</code> gathers shading commands under a <code>Framebuffer</code>. That means that all <code>ShadingCommand</code>
embedded in the <code>Pipeline</code> will output to the embedded <code>Framebuffer</code>. Simple, yet powerful, because
we can <em>bind</em> the framebuffer when executing the pipeline and don’t have to worry about framebuffer
until the next execution of another <code>Pipeline</code>.</p>
<h2>ShadingCommand</h2>
<p>A <code>ShadingCommand</code> gathers render commands under a shader <code>Program</code> along with an update function.
The update function is used to customize the <code>Program</code> by providing <em>uniforms</em> – i.e. <code>Uniform</code>.
If you want to change a <code>Program</code>s <code>Uniform</code> once a frame – and only if the <code>Program</code> is only called
once in the frame – it’s the right place to do it.</p>
<p>All <code>RenderCommand</code> embedded in the <code>ShadingCommand</code> will be rendered using the embedded shader
<code>Program</code>. Like with the <code>Pipeline</code>, we don’t have to worry about binding: we just have to use the
embedded shader program when executing the <code>ShadingCommand</code>, and we’ll bind another program the next
time a <code>ShadingCommand</code> is ran!</p>
<h2>RenderCommand</h2>
<p>A <code>RenderCommand</code> gathers all the information required to render a <code>Tessellation</code>, that is:</p>
<ul>
<li>the blending equation, source and destination blending factors</li>
<li>whether the depth test should be performed</li>
<li>an update function to update the <code>Program</code> being in use – so that each object can have different
properties used in the shader program</li>
<li>a reference to the <code>Tessellation</code> to render</li>
<li>the number of instances of the <code>Tessellation</code> to render</li>
<li>the size of the rasterized points (if the <code>Tessellation</code> contains any)</li>
</ul>
<h1>What about shaders?</h1>
<p>Shaders are written in… the backend’s expected format. For OpenGL, you’ll have to write <strong>GLSL</strong>.
The backends automatically inserts the version pragma (<code>#version 330 core</code> for OpenGL 3.3 for
instance). In the first place, I wanted to migrate <strong>cheddar</strong>, my Haskell shader EDSL. But… the sad
part of the story is that Rust is – yet – unable to handle that kind of stuff correctly. I started
to implement an EDSL for luminance with macros. Even though it was usable, the error handling is
seriously terrible – macros shouldn’t be used for such an important purpose. Then some rustaceans
pointed out I could implement a (rustc) compiler plugin. That enables the use of new constructs
directly into Rust by extending its syntax. This is great.</p>
<p>However, with the hindsight, I will not do that. For a very simple reason. luminance is, currently,
simple, stateless and most of all: it works! I released a PC demo in Köln, Germany using luminance
and a demoscene graphics framework I’m working on:</p>
<p><a href="http://www.pouet.net/prod.php?which=67966">pouët.net link</a></p>
<p><a href="https://www.youtube.com/watch?v=pYqZS1C_7PU">youtube capture</a></p>
<p><a href="https://github.com/phaazon/ion">ion demoscene framework</a></p>
<p>While developping Céleri Rémoulade, I decided to bake the shaders directly into Rust – to get used
to what I had wanted to build, i.e., a shader EDSL. So there’re a bunch of constant <code>&amp;'static str</code>
everywhere. Each time I wanted to make a fix to a shader, I had to leave the application, make the
change, recompile, rerun… I’m not sure it’s a good thing. Interactive programming is a very good
thing we can enjoy – yes, even in strongly typed languages ;).</p>
<p>I saw that <a href="https://crates.io/crates/gfx">gfx</a> doesn’t have its own shader EDSL either and requires
you to provide <strong>several shader implementations (one per backend)</strong>. I don’t know; I think it’s not
that bad if you only target a single backend (i.e. OpenGL 3.3 or Vulkan). Transpiling shaders is a
thing, I’ve been told…</p>
<blockquote>
<p><em>sneaking out…</em></p>
</blockquote>
<p>Feel free to dig in the code of Céleri Rémoulade <a href="https://github.com/phaazon/celeri-remoulade">here</a>.
It’s demoscene code, so it had been rushed on before the release – read: it’s not as clean as I
wanted it to be.</p>
<p>I’ll provide you with more information in the next weeks, but I prefer spending my spare time
writing code than explaining what I’m gonna do – and missing the time to actually do it. ;)</p>
<p>Keep the vibe!</p>


guid: https://strongly-typed-thoughts.net/blog/generalized_swapTue, 14 Apr 2015 00:00:00 GMT
title: Generalized swap
author: Dimitri Sabadie <dimitri.sabadie@gmail.com>
url: https://strongly-typed-thoughts.net/blog/generalized_swap
feedurl: https://phaazon.net/blog/feed
content:
<h1>Generalized swap</h1>
<p>I pushed a <em>pull request</em> to <a href="https://hackage.haskell.org/package/either">Edward Kmett’s either package</a>
to implement two functions some guys was complaining not to find: <code>flipEither :: Either e a -&gt; Either a e</code>
and <code>flipEitherT :: EitherT e m a -&gt; EitherT a m e</code>.</p>
<p>When implementing the functions, I wondered: “Hey, flipping stuff is a pretty
common operation. Don’t we have an abstraction for that yet?”. I haven’t found any.</p>
<h1>Meet Swap</h1>
<p>I decided to make a little typeclass to see what it’d be.</p>
<pre><code class="language-haskell">class Swap s where
  swap :: s a b -&gt; s b a

instance Swap (,) where
  swap (a,b) = (b,a)

instance Swap Either where
  swap = flipEither

-- let’s go wild and fooled
instance Swap Map where
  swap = fromList . fmap swap . toList
</code></pre>
<p>If you think that’s handy, I’ll write a little package with default instances
to make it live.</p>
<p>Happy hacking folks!</p>


guid: https://strongly-typed-thoughts.net/blog/glsl-quasiquote-0.2Mon, 22 Oct 2018 02:00:00 GMT
title: Release of glsl-quasiquote-0.2
author: Dimitri Sabadie <dimitri.sabadie@gmail.com>
url: https://strongly-typed-thoughts.net/blog/glsl-quasiquote-0.2
feedurl: https://phaazon.net/blog/feed
content:
<p><a href="https://crates.io/crates/glsl-quasiquote/0.2.0">glsl-quasiquote-0.2</a> was released early this morning. This new version provides a more stable
public API. Two major changes:</p>
<ul>
<li>The <code>glsl_str!</code> proc-macro would have only survived the <code>0.1</code> version. It’s now deprecated and
will be removed soon.</li>
<li>The <code>glsl!</code> proc-macro now supports GLSL pragmas (both <code>#version</code> and <code>#extension</code>).</li>
</ul>
<blockquote>
<p><em>“Wait. GLSL pragmas? You <a href="https://phaazon.net/blog/glsl-quasiquoting#enter-the-nice-solution-quasiquoting">said it wasn’t possible</a>
because of how the Rust tokenizer thinks newlines are meaningless while the GLSL grammar uses them
to separate pragmas from other pragmas and code.”</em></p>
</blockquote>
<p>Yes, at the time of writing <code>glsl-quasiquote</code>, I just thought that since <a href="https://doc.rust-lang.org/proc_macro/struct.TokenStream.html"><code>TokenStream</code></a> is about a
stream of Rust tokens, GLSL’s pragmas would be impossible to encode with this mechanism. In fact,
it’s actually impossible per-se: newlines are not Rust tokens and are just completely ignored by the
tokenizer… or are they?</p>
<h1>Enter the motivation</h1>
<p>At the announcement of the release of <code>glsl-quasiquote</code> on Reddit, someone – actually,
<a href="https://www.reddit.com/r/rust/comments/9lo9s4/glsl_quasiquoting_in_rust/e78nf85">somebodddy</a> just
pointed out that I could use Rust attributes to encode GLSL pragmas. That would enable me to have
them as regular Rust tokens at the cost of modifying very, very briefly the GLSL grammar for the
pragmas – nothing bad, really. Their idea, which I liked, was this:</p>
<p>Instead of:</p>
<pre><code>#version 330 core
</code></pre>
<p>We could write:</p>
<pre><code>#![version 330 core]
</code></pre>
<p>Such a small change yet effective, right? I was appealed by the idea, so I implemented it. The code
was a bit messy because I did all the token pattern-matching by hand but I sketched something up
that worked in a few minutes. After having a discussion with a friend on IRC (antoyo!), I realized
that I was plain wrong from the beginning assertion that newlines are ignored by the Rust tokenizer:
they’re not.</p>
<h1>Enter the hack</h1>
<p>The <a href="https://doc.rust-lang.org/proc_macro/index.html"><code>proc_macro</code></a> crate, which is used to manipulate Rust tokens, has several items that I used
to implement the quasiquoter:</p>
<ul>
<li><code>TokenStream</code>, a stream of tokens. This is a very opaque type, you can just create an empty one,
display it as a <code>String</code> and marshall to and from an iterator interface, which element type is
<code>TokenTree</code>.</li>
<li><code>TokenTree</code>, a single token or a delimited sequence of token trees, like a group of tokens
logically united (with brackets, parens, etc.).</li>
<li>All the types used in <code>TokenTree</code> variants, such as <code>Literal</code>, <code>Punct</code>, <code>Group</code>, etc.</li>
</ul>
<p>There’s something I just had completely forgotten when claiming that newlines were ignored: pretty
much all of the tokens contained in <code>TokenTree</code> have an associated <a href="https://doc.rust-lang.org/proc_macro/struct.Span.html"><code>Span</code></a>. I’m not sure what the
initial intent was about with this type, but in order to use it, I had to add the <code>proc_macro_span</code>
feature. That type gives positional information along with macro expansion on a token. Specifically,
it has a method that gives the line and column at which the token starts and ends.</p>
<p>This information gives us the line on which a token lays. That’s it. We have the newlines! A newline
is just whenever a token following another token has a different line in its span. That’s as easy as
it gets.</p>
<h2>The real fix</h2>
<p>So instead of implementing GLSL pragmas via Rust attributes, I decided to implement them the way
they are defined in the spec and the <a href="https://crates.io/crates/glsl">glsl</a> crate: plain GLSL. The idea was just to match a <code>#</code> and
accumulate tokens in a collection as long as tokens lay on the same line. As soon as the line
changes, we have the full pragma and we can call <code>TokenStream::from_iter</code>.</p>
<p>The current implementation expects the pragmas to be at the top part of the quasiquoted GLSL code.
You shouldn’t be trying to put the <code>#version</code> at the bottom of your file, but I wanted you to be
noticed: don’t do that!</p>
<p>Since <a href="https://crates.io/crates/glsl-quasiquote/0.2.0">glsl-quasiquote-0.2</a>, this Rust code now completely compiles and generates, at compile-time,
a GLSL AST:</p>
<pre><code>let ast = glsl!{
  #version 330 core
  #extension GL_ARB_separate_shader_objects : require

  void main() {
  }
};
</code></pre>
<h1>Future announcements about glsl and glsl-quasiquote</h1>
<p>Some future announcements to come (things that I’ve already been working on and that I might release
soon):</p>
<ul>
<li>The quasiquoter will have interpolation at some time.</li>
<li><a href="https://crates.io/crates/glsl">glsl</a> is about to be released with an experimental SPIR-V transpiler. The current
implementation uses <a href="https://crates.io/crates/shaderc">shaderc</a> as backend in order to test how everything goes.</li>
</ul>


guid: https://strongly-typed-thoughts.net/blog/you_are_being_spied_onTue, 21 Apr 2015 00:00:00 GMT
title: Losing our privacy
author: Dimitri Sabadie <dimitri.sabadie@gmail.com>
url: https://strongly-typed-thoughts.net/blog/you_are_being_spied_on
feedurl: https://phaazon.net/blog/feed
content:
<h1>Big brother of NSA</h1>
<p>On the 19th of March 2015, a law was introduced in France. That law was
entitled <em>“loi du renseignement”</em> and was presented in regard of what happened
in the <em>Charlie Hebdo</em> awful week. The main idea of the law is to put peole on
wire so that terrorist activities could be spotted and dismantled.</p>
<p>That law will be voted and apply on the the 5th of May 2015.</p>
<p>Although such a law sounds great for counter-terrorism, it has several major
issues people should be aware of.</p>
<h2>You’re on wire</h2>
<p>Every people – in France and abroad as well of course – could be on wire. The French
government might have access to whatever you do with your internet connection.
Crypto-analysts working for the goverment will read dozens of thousands of
messages from individuals. They’ll know where you go to and where you go from –
think of your smartphone’s GPS capabilities. They’ll know which pictures you
take, how much time you spend in your bathroom. They’ll even be able to read
your e-mails if they want to.</p>
<p>Of course, most people “just don’t care”. <em>“Why should I care that the
government knows the litter brand I bought to my cat? I don’t give a damned
heck as long as they catch bad people</em>”. That’s a point, but that’s also being
blind. Digital communication is not only about yourself. It’s about people.
It’s about you and your friends. You, and your contacts. You can’t choose for
them whether they’d like people to watch over their shoulders every now and
then. If the government has access to your private data, it has access to your
friends’ and contacts’ data as well. Not giving a damned heck is being selfish.</p>
<h2>You’ll be violated</h2>
<p>Then comes the issue with what the law is. It gives the government the right
to spy on masses. They could even sell the data they collect. As a counter
argument, the “Commission de contrôle des techniques de renseignement” – which
French stands for “Control commission of intelligence techniques” – was
created. That committe is there to watch the government and ensure it doesn’t
go out of control and doesn’t violate people’s rights. The issue with that is
that our prime minister has the right to ignore the committee’s decision. If
the committee says <em>“Wait a minute. You don’t have the right to gather those
information without asking for M. Dupont’s approval”</em>, the prime minister may
answer back <em>“Well, fuck off. I will and you can’t stop me”</em>. And the sad truth
is that… yeah, with that law, the prime minister and their team has the right
to ignore the committee’s decision.</p>
<p>The committee then has no point. It just gives an <strong>opinion</strong>. Not a veto. What
would happen if a terrorist hacked into your computer. Would you go to jail
because the prime minister would have stated you were a terrorist? Damn me if I
know.</p>
<h1>We’re going to lose a right</h1>
<p>French people will lose a very important right: the right to privacy. It’ll be
sacrificied without your opinion for counter-terrorism, giving the government
a power it shouldn’t have. You thought the NSA was/is wrong? Sure it is. But
when the NSA watches over American and worldwide people, it is illegal whereas
when the French government watches over French and worldwide people, it is
legal. That makes the French government way worse than the NSA to me.</p>
<p>I think the first thing I’ll do will be to revoke my French VPS subscription
to move it out of France, in a country in which people still have privacy. Keep
your communication encrypted as much as possible (ssh and so on). And as a bad
joke, don’t leave your camera in your bedroom. You might be spied on by Manuel
Valls while having sex with your girlfriend.</p>


guid: https://strongly-typed-thoughts.net/blog/kakoune-philosophySat, 23 Dec 2023 14:15:00 GMT
title: The Kakoune philosophy
author: Dimitri Sabadie <dimitri.sabadie@gmail.com>
url: https://strongly-typed-thoughts.net/blog/kakoune-philosophy
feedurl: https://phaazon.net/blog/feed
content:
<p>I have been using <a href="https://kakoune.org">Kakoune</a> for a long time, and I want to talk about the User Experience (UX). When it comes down to
editors / productivity platforms, people tend to confuse UX with User Interface (UI). Both are related, of course, but
having a great UX doesn’t mean the UI is “great” and having a great UI doesn’t mean the UX is “great”: it’s all a matter
of perspective.</p>
<p>In this blog entry, I want to talk about UX more specifically, because I do think it’s more important than UI. I will
try to show that the default UX of Kakoune is incredible and that you can <strong>very quickly</strong> create a super expressive
and pragmatic programmer environment. It will cover:</p>
<ul>
<li>Surrounding pairs.</li>
<li>Grepping around</li>
<li>Pickers.</li>
<li>System clipboard.</li>
<li>Bonus: some tools I’ve been making.</li>
</ul>
<blockquote>
<p>Once again, I’d like to greet and thank <a href="https://github.com/alexherbo2">@Taupiqueur</a> for sharing his thoughts and joy about Kakoune.</p>
</blockquote>
<h1>What I mean with UI and UX</h1>
<p><strong>UI</strong> means “anything interfacing the user to the system.” It’s both the visual depiction of the service (the menu, the
colors, the fonts, etc.) and the way you interact with the system (with the keyboard, by clicking on buttons, when a
system event happens, etc.).</p>
<p><strong>UX</strong> means “how the user experiences the system.” For instance, something that is not UI at all but enhances the UX
is having a way to filter data in a system with high volume of events with a tag. An even better UX is having fuzzy
filtering with any tags. Etc.</p>
<p>There are many possible UI implementations for a given UX item: implementing filtering can be done with a select box in
a GUI, but the UX is not great because the user is presented with a set of finite choices, and if there are many, it’s
pretty annoying to scroll down the list to find the one we want. Even with this bad UX design, most of select box
implementations (e.g. web) allow to press keys to jump to the entry in the select options — most of the time, you don’t
see what you type -&gt; bad UX for the user. Another possible UI implementation would be to use a free text box that would
filter based on its content — it could even be live for an even better UX. Etc. etc.</p>
<p>Now, would you prefer a nice looking GUI with the select box, or a blander UI but with the fuzzy search box? Clearly,
in terms of UX, the second option is much better. But now, imagine a GUI with the fuzzy search box. It would be pretty
similar to the blander UI in terms of UX, but it would look (much) better… which is likely to enhance the UX as well!</p>
<h1>Small disgression: TUI vs. GUI</h1>
<blockquote>
<p>TUI: Text User Interface, which is a program mimicking traditional GUIs in the terminal.</p>
</blockquote>
<blockquote>
<p>GUI: Graphics User Interface, the typical window-based applications you run on your machine.</p>
</blockquote>
<p>Before jumping to the Kakoune content, I just want to disgress slightly to talk about something I often see something
that itches me a bit: many individuals seem to say that a TUI is often written in a way that optimizes UX and GUI the
UI, and hence, oftentimes, using a TUI feels much better. I agree with this (this is the reason why I’m using editors
and tools inside my terminal instead of a dedicated GUI, even though I think the UI is worse, for many reasons: no
pixel-perfect alignment of things; no direct integration of the application at the OS level, it has to go through the
terminal and shell; etc. etc.).</p>
<p>However, is there anything forcing a GUI not to provide the same kind of interactions as a TUI? I don’t think so. If
you make your TUI keyboard-oriented, everything you do is just listening to keyboard events provided by whatever
terminal protocol / library you use… which could be done exactly the same way in a GUI.</p>
<p>I do think that we should be able to make a GUI as good as a TUI in terms of UX, and some programs did it. For instance,
<a href="https://www.gnu.org/software/emacs/">emacs</a> can run both as a TUI and a GUI (and today, people recommend actually using the GUI). I think that could be the
topic of another blog article. Let’s go back to the original matter.</p>
<h1>Kakoune UX</h1>
<p>Kakoune, upon installation, already provides a lot of good things in terms of UX. But as you get more productive with
it, you will face some problems. For instance, the first one I came across (pretty quickly being honest, coming from
<a href="https://helix-editor.com">Helix</a>) was surrounding: adding, deleting and replacing surrounding delimiters. Where you need a plugin to do that in
the Vim world, in Kakoune, it’s another topic. Some plugins exist to do exactly that, but honestly, read along.</p>
<p>Selections in Kakoune — which are so much more than <em>“iT’s JUsT LikE muLtIcURsOrs oR a nOOb WaY Of DoINg RegEX in
vIM”</em> — change the Vim interpretation of <em>appending</em> and <em>preppending</em>. In <a href="https://www.vim.org">Vim</a>, <code>i</code> will start inserting on the cursor,
whereas <code>a</code> will start inserting after the cursor. In Kakoune, <code>i</code> inserts <strong>at the beginning of each selection</strong> and
<code>a</code> inserts <strong>at the end of each selection</strong>. That’s already a better UX, and it allows to do many things out of the
box.</p>
<p>For instance, since we have the power to insert at the beginning and at the end of each selection… then pressing <code>i'</code>
should insert a quote at the beginning of the selection… and <code>a'</code> will do the same at the end! So you can already have
a somewhat working surrounding add operation by typing <code>i'&lt;esc&gt;a'</code>!</p>
<blockquote>
<p>If you are adventurous, you can read the documentation of <code>&lt;a-;&gt;</code>, which allows to leave insert mode to normal mode
for a single command, and come back to insert mode. You can use this to replicate what we did above without the
<code>&lt;esc&gt;</code> key: <code>i'&lt;a-;&gt;a'</code>. Magic.</p>
</blockquote>
<p>It’s a bit annoying to have to type all that, though, right? So instead, we could make a command! Commands in Kakoune
are really simple, but they require reading a bit about them. I recommend the following reads if you want to dig in a
bit:</p>
<ul>
<li><code>:doc commands</code> to know about all the commands. Search for <code>^define-command</code>.</li>
<li><code>:doc execeval</code>, which explains what <code>evaluate-commands</code> and <code>execute-keys</code> do. Especially, you will want to read the
part of <code>-draft</code>.</li>
</ul>
<p>So let’s wrap that sequence of keys in a command. Instead of using <code>i</code> and <code>a</code>, we are going to use <code>P</code> and <code>p</code>, which,
as the name implies, <em>paste</em> from the default register. <code>p</code> <strong>pastes at the end of the each selection</strong> and <code>P</code> <strong>pastes
at the beginning of each selection</strong>.</p>
<h2>Surrounding pairs</h2>
<p>What is great is that the default register, <code>&quot;</code>, is selection-aware: its content will be different from one selection to
another. Said otherwise, there is one <code>&quot;</code> register for each selection. Hence, we can write this:</p>
<pre><code>define-command -override my-surround-add -params 2 %{
  evaluate-commands -draft -save-regs '&quot;' %{
    set-register '&quot;' %arg{1}
    execute-keys -draft P
    set-register '&quot;' %arg{2}
    execute-keys -draft p
  }
}
</code></pre>
<p>And here you go. You can now invoke <code>:my-surround-add ( )&lt;ret&gt;</code> to add parenthesis around your selections, for instance.</p>
<p>Kakoune has the concept of user modes, which is a nice feature allowing to declare a keyset (that will be displayed
by the help in the bottom right of your screen) when entered.</p>
<pre><code>declare-user-mode my-surround-add
</code></pre>
<p>We can make one with a bunch of mappings in that user mode:</p>
<pre><code>map global my-surround-add (   ':my-surround-add ( )&lt;ret&gt;'         -docstring 'surround with parenthesis'
map global my-surround-add )   ':my-surround-add ( )&lt;ret&gt;'         -docstring 'surround with parenthesis'
map global my-surround-add [   ':my-surround-add [ ]&lt;ret&gt;'         -docstring 'surround with brackets'
map global my-surround-add ]   ':my-surround-add [ ]&lt;ret&gt;'         -docstring 'surround with brackets'
map global my-surround-add {   ':my-surround-add { }&lt;ret&gt;'         -docstring 'surround with curly brackets'
map global my-surround-add }   ':my-surround-add { }&lt;ret&gt;'         -docstring 'surround with curly brackets'
map global my-surround-add &lt;   ':my-surround-add &lt; &gt;&lt;ret&gt;'         -docstring 'surround with angle brackets'
map global my-surround-add &gt;   ':my-surround-add &lt; &gt;&lt;ret&gt;'         -docstring 'surround with angle brackets'
map global my-surround-add &quot;'&quot; &quot;:my-surround-add &quot;&quot;'&quot;&quot; &quot;&quot;'&quot;&quot;&lt;ret&gt;&quot; -docstring 'surround with quotes'
map global my-surround-add '&quot;' &quot;:my-surround-add '&quot;&quot;' '&quot;&quot;'&lt;ret&gt;&quot;   -docstring 'surround with double quotes'
map global my-surround-add *   ':my-surround-add * *&lt;ret&gt;'         -docstring 'surround with asteriks'
map global my-surround-add _   ':my-surround-add _ _&lt;ret&gt;'         -docstring 'surround with undescores'
</code></pre>
<p>Obviously, that’s not all; we would need to delete delimiters and to replace them. Deleting is actually even more
straightforward. Kakoune has some native mappings to select everything <em>inside</em> or <em>around</em> a set of delimiters:</p>
<ul>
<li><code>&lt;a-i&gt;</code> to select <em>inside</em>.</li>
<li><code>&lt;a-a&gt;</code> to select <em>outside</em>.</li>
</ul>
<p>So, pressing <code>&lt;a-a&gt;(</code> (or <code>&lt;a-a&gt;)</code>, same thing) will select everything around the cursor up to the next pair of
parenthesis.</p>
<blockquote>
<p>Note: I have personally remapped that to <code>mi</code> and <code>ma</code>, but that collides with the default meaning of the <code>m</code> key in
Kakoune, so I will use the native mapping here instead.</p>
</blockquote>
<p>We can then simply use the previous <code>i</code> and <code>a</code> command mentioned before to start editing at the beginning and end of
the selection. <code>i&lt;del&gt;</code> will start insert mode at the beginning of the selection and will remove a character (left
delimiter) and <code>a&lt;backspace&gt;</code> will insert at the end of the selection and remove the previous character (right
delimiter). Eh, that looks like so simple it’s almost stupid. But that’s what makes Kakoune so damn good: it’s <em>simple</em>
to reason about:</p>
<pre><code>define-command -hidden my-surround-delete-key -params 1 %{
  execute-keys -draft &quot;&lt;a-a&gt;%arg{1}i&lt;del&gt;&lt;esc&gt;a&lt;backspace&gt;&lt;esc&gt;&quot;
}

define-command my-surround-delete %{
  on-key %{
    my-surround-delete-key %val{key}
  }
}
</code></pre>
<p>Because Kakoune composes really well, you can already imagine that you should be able to use the previous commands and
mappings. And indeed:</p>
<pre><code>define-command my-surround-replace %{
  on-key %{
    surround-replace-sub %val{key}
  }
}

define-command -hidden my-surround-replace-sub -params 1 %{
  on-key %{
    evaluate-commands -no-hooks -draft %{
      execute-keys &quot;&lt;a-a&gt;%arg{1}&quot;

      # select the surrounding pair and add the new one around it
      enter-user-mode my-surround-add
      execute-keys %val{key}
    }

    # delete the old one
    my-surround-delete-key %arg{1}
  }
}
</code></pre>
<h2>Revisiting grepping</h2>
<p>Ah… who has never had the problem of trying to locate something in a codebase without <em>really knowing where to start</em>.
That happens a lot to me when working on a front-end project and taking an issue asking to fix a random page, that is
broken. Usually, LSPs don’t help to <em>discover</em> things that are based on the final product. For instance, if you see the
checkout page broken — like a <code>&lt;div&gt;</code> is missing an attribute or a tag is misplaced in the DOM, no LSP will help you
locate the code that needs to be fixed. Instead, you need other tools.</p>
<p>What I like to do is looking at the page and looking for what I call <em>unique tokens</em>. For instance, a short sentence
that might appear only on that page, or modal. Or a header, a title, etc. Then, using that information, I usually grep
the code base. The problem is that, doing that using <code>grep</code> or <code>ripgrep</code> as CLI is pretty boring, and not very
practical. Indeed, if you get many results, you are likely to try to reduce the result set by constraing more your
regex. Once you have some files, you usually look in your terminal scrollback buffer until you find something
interesting, then open that file in your editor.</p>
<p>People using something like <strong>IntelliJ</strong> products, or <strong>VS Code</strong>, or some plugins with <strong>emacs</strong> or <strong>vim</strong> might have
a way to perform the search from within their editors, but again, that’s not composability: it’s extensibility, and I
explained in <a href="https://phaazon.net/blog/more-hindsight-vim-helix-kakoune">a previous article</a> why it’s not a good design <em>to me</em>.</p>
<p>Kakoune, on the other side, went the <em>composition</em> route. <code>grep</code>, <code>ripgrep</code>, etc. are all <ins>amazing</ins> tools.
Kakoune comes with a bunch of what it calls <em>tools</em>, which are basically Kakoune commands shipped with the editor. Among
those, there is one that is of interest here: the <code>:grep</code> command. The <code>:grep</code> command forwards its arguments to the
underlying <code>%opt{grepcmd}</code> (which defaults to something like <code>grep -RHn</code>). Hence, <code>:grep foo</code> will run <code>grep -RHn foo</code>
in a shell behind the scene, then the result will be output in a <code>*grep*</code> buffer. That buffer will get special
highlightings, along with some keybindings, all of that provided by the <code>grep.kak</code> tool. If tried the command, you might
have noticed that it’s basically a list of lines of the form:</p>
<pre><code>&lt;path&gt;:&lt;line&gt;:&lt;column&gt;:&lt;content-of-the-line&gt;
</code></pre>
<p>Then, how do you think <code>grep.kak</code> implements _pressing <code>&lt;ret&gt;</code> jumps to the line, column and file of the line under the
cursor? The cursor can be anywhere on the line. Well, it’s simple: <code>execute-keys</code> again! <code>gh</code> will put your cursor at
the beginning of the line. Then, you can use <code>f:</code> or <code>t:</code> to jump to the next <code>:</code> — there is no need to parse anything,
we can just programmatically interact with the editor!</p>
<p>Here, <code>ghT:&quot;py</code> will go the beginning of the line, select the path and yank it to the <code>p</code> register. We can then do
<code>2l</code> to move the cursor to beginning of the <code>&lt;line&gt;</code> number, and <code>T:&quot;ly</code> will yank the line number to the <code>l</code> register.
<code>2l</code> again to move to the <code>&lt;column&gt;</code> section, ten <code>T:&quot;cy</code> to yank the column number to the <code>c</code> register. And we have
everything we need. We can then just simply run the <code>edit -existing %reg{p} %reg{l} %reg{c}</code> command:</p>
<p>aoc-18.md:1:1:lol</p>
<pre><code>define-command -override my-jump-for-current-line %{
  evaluate-commands -save-regs clp %{
    execute-keys -draft 'ghT:&quot;py2lT:&quot;ly2lT:&quot;cy'
    edit -existing %reg{p} %reg{l} %reg{c}
  }
}
</code></pre>
<p>A couple of comments here:</p>
<ul>
<li><code>-save-regs clp</code> saves the content of the <code>c</code>, <code>l</code> and <code>p</code> register before evaluating the commands, then restores
those registers afterwards. That is required since we copy a bunch of data to those registers, but maybe the user is
already using them. <code>-save-regs</code> allows us to specificy a set of registers we locally need, but we do not want those
registers to bleed outside.</li>
<li><code>-draft</code> makes the command evaluation run in disposable context. That prevents our selections from being changed —
<code>execute-keys</code> runs a couple of move and goto commands here, while we don’t really want the cursor to move.</li>
<li><code>-existing</code> fails if the file doesn’t exist.</li>
</ul>
<p>The <code>:grep</code> tool implements something probably very similar to this, but it uses <code>%opt{grepcmd}</code>, which you can change
to whatever you like. I personally use:</p>
<pre><code>set-option global grepcmd 'rg --column --smart-case --sort path'
</code></pre>
<h2>Kakoune lacks pickers… or does it?</h2>
<p>Something that is very important having around is being able to locate files very quickly. Most editors today ship with
a way to locate files:</p>
<ul>
<li>Vim has <em>netrw</em>. It’s old. It’s ugly. But it’s there. It’s basically a file browser with a (very) minimal set of
features.</li>
<li>VS Code has fuzzy pickers, so you can locate a file by fuzzy searching it.</li>
<li>Etc.</li>
</ul>
<p>Kakoune has a default powerful completion engine. Pressin <code>:edit </code> (or, for short, <code>:e </code>) and then typing something will
auto-complete the file in the current directory with a somewhat fuzzy algorithm. If you select a directory and type the
trailing <code>/</code>, it will list the content of that directory and will auto complete its children. It’s a pretty nice way to
start moving around with the vanilla editor.</p>
<p>However, Kakoune doesn’t ship more than this, because, well, it’s <em>composable</em>. You can use any tool you like to locate
files, and then compose them with Kakoune. I personally really like <code>fd</code> (a Rust rewrite of <code>find</code>). For instance,
<code>fd --type file</code> will locate all the files in the current directory. Piping that to a fuzzy finder would allow to
jump to any file in the current directory. And Kakoune supports that. It has several commands for that, but the one you
will be interested at first is <code>prompt</code>. It prompts the user for some text and pass the entered text to the provided
commands in the <code>%val{text}</code> variable. It supports some switches, and one of interest for us is
<code>-shell-script-candidates</code>. That switch accepts a shell command to execute, reading from its standard output
asynchronously and displaying into the completion items. For instance, try running the following command:</p>
<pre><code>prompt -shell-script-candidates ls file: %{ info &quot;you selected %val{text}&quot; }
</code></pre>
<p>It runs <code>ls</code>, gets the output and allows you to fuzzy complete the result. Now consider this:</p>
<pre><code>prompt -shell-script-candidates 'fd --type file' open: %{ edit -existing %val{text} }
</code></pre>
<p>And bam. Here you have it. Fuzzy picker in Kakoune. You can map that to a command, or wrap it in a command:</p>
<pre><code>define-command my-file-picker %{
  prompt -shell-script-candidates 'fd --type file' open: %{ edit -existing %val{text} }
}
</code></pre>
<p>If — like many people — you want that to be run when you type <code>SPC f</code>:</p>
<pre><code>map global user f ':my-file-picker&lt;ret&gt;'
</code></pre>
<p>You can do the same thing with anything, really. Some plugins like <code>kak-lsp</code> uses that to show document symbols, etc.</p>
<h2>System clipboard</h2>
<p>By default, Kakoune comes up with registers you can yank to. However, oftentimes, you need to yank and paste from the
system cliboard. Kakoune doesn’t have a <em>“system register”</em> as in Vim. Instead, as with the rest, you have to compose
some tools with Kakoune to get that feature. And it’s pretty simple. You need to know how to yank some text in CLI. Two
situations:</p>
<ul>
<li>On <strong>macOS</strong>, we use the <code>pbcopy</code> and <code>pbpaste</code> to respectively copy and paste from/to the system clipboard.</li>
<li>On <strong>Linux</strong>, we use the <code>xclip</code> and <code>xsel -ob</code> commands to do the same.</li>
</ul>
<p>And then, there is nothing much more to do. We can wrap those in two commands agnostic of the platform by using the
<code>uname</code> utility:</p>
<pre><code>declare-option str extra_yank_system_clipboard_cmd %sh{
  test &quot;$(uname)&quot; = &quot;Darwin&quot; &amp;&amp; echo 'pbcopy' || echo 'xclip'
}

declare-option str extra_paste_system_clipboard_cmd %sh{
  test &quot;$(uname)&quot; = &quot;Darwin&quot; &amp;&amp; echo 'pbpaste' || echo 'xsel -ob'
}
</code></pre>
<p>And the actual commands, using the <code>!</code> key (to run a shell command with the current selection piped as standard input
and replace selections with its output) and <code>&lt;a-!&gt;</code> (which runs a shell command and ignore its output):</p>
<pre><code>define-command extra-yank-system -docstring 'yank into the system clipboard' %{
  execute-keys -draft &quot;&lt;a-!&gt;%opt{extra_yank_system_clipboard_cmd}&lt;ret&gt;&quot;
}

define-command extra-paste-system -docstring 'paste from the system clipboard' %{
  execute-keys -draft &quot;!%opt{extra_paste_system_clipboard_cmd}&lt;ret&gt;&quot;
}
</code></pre>
<p>I personally map those as in Helix:</p>
<pre><code class="language-map">map global user y ':extra-yank-system&lt;ret&gt;'  -docstring 'yank to system clipboard'
map global user p ':extra-paste-system&lt;ret&gt;' -docstring 'paste selections from system clipboard'
</code></pre>
<h2>Bonus: some tools I’ve been making</h2>
<p>Because the UX in Kakoune is amazing, and because it composes so well, I have made a couple of binaries and Kakoune
commands to enhance the experience. So far (23rd December of 2023), I have wrote those</p>
<ul>
<li><a href="https://github.com/phaazon/kak-tree-sitter">kak-tree-sitter</a>, which provides <a href="https://tree-sitter.github.io/tree-sitter/">tree-sitter</a> support (so far, only semantic highlighting; semantic selections will
come later).</li>
<li><a href="https://github.com/phaazon/bookmarks.kak">bookmarks.kak</a>, a super small plugin I use to create persistent bookmarked locations (and list them in a dedicated
buffer).</li>
<li><a href="https://github.com/phaazon/git.kak">git.kak</a>, a super set of the (already great) Git integration in Kakoune. I for instance add a cursor-anchored overlay
showing git blame information.</li>
<li><a href="https://github.com/phaazon/hop.kak">hop.kak</a>, Hop brought to Kakoune. My implementation is by far <em>much</em> simpler than what I did in [hop.nvim] because I
read native Kakoune selection descriptions, instead of relying on an internal regex engine run on each line: it
composes so much better.</li>
<li><a href="https://github.com/phaazon/notes.kak">notes.kak</a>, one of my most useful plugin. It enhances Markdown with some features:
<ul>
<li>Lists marked <code>- TODO</code>, <code>- WIP</code>, <code>- DONE</code>, <code>- WONTDO</code>, <code>- IDEA</code>, etc. are highlighted with specific faces.</li>
<li>Todo items from the list above can be gathered in a buffer (similar to a grep buffer) to jump directly to the note
file containing them.</li>
<li>Open the journal of the day.</li>
<li>Open notes.</li>
<li>Search journals and notes.</li>
<li>Archive and list archives.</li>
<li>And much more.</li>
</ul>
</li>
<li><a href="https://github.com/phaazon/swiper.kak">swiper.kak</a>, my interpretation of the famous <a href="https://elpa.gnu.org/packages/swiper.html">swiper</a> emacs package; it allows to reduce a buffer by fuzzy searching
something in it, only showing the lines that matches, but showing them all at once (by removing the lines that don’t
match), and pressing <code>&lt;ret&gt;</code> jump to the line under the cursor (or <code>&lt;esc&gt;</code> goes back to where you were buffer). I
also included a <em>reduce</em> mode that helps with reducing grep buffers or any buffer that can be directly modified.</li>
</ul>
<p>All in all, I’m really happy with my current Kakoune setup, and as time passes, I realize it’s a nice <em>interactive
editing platform</em> so far. For instance, at work, I have started making a <code>k8s.kak</code> to highlight and run commands on
Kubernetes cluster from within Kakoune; and it works pretty well.</p>
<p>Have fun and keep hacking around!</p>


guid: https://strongly-typed-thoughts.net/blog/typesafe-deinterleavingSun, 19 Jul 2020 19:39:00 GMT
title: The compile-time deinterleaving interface of luminance-0.40
author: Dimitri Sabadie <dimitri.sabadie@gmail.com>
url: https://strongly-typed-thoughts.net/blog/typesafe-deinterleaving
feedurl: https://phaazon.net/blog/feed
content:
<p><a href="https://crates.io/crates/luminance/0.40.0">luminance-0.40</a> is finally out, after several months of hardwork. A blog post is available
<a href="https://phaazon.net/blog/luminance-0.40">here</a> if you’ve missed it.</p>
<p>Among all the new features of <a href="https://crates.io/crates/luminance/0.40.0">luminance-0.40</a>, there is one that I think is worth explaining in
details, because it’s… <em>pretty cool</em>. If you like <em>type states</em>, <em>refinement typing</em> and on a
general level, strong typing, you should enjoy this blog article.</p>
<p>While working on <a href="https://crates.io/crates/luminance/0.40.0">luminance-0.40</a>, I spent two weeks working on a feature that is going to have a
huge beneficial impact on how people use luminance. I’ve been wanting to see what people think about
it for a long time, because I think it’s both a very powerful feature and allows to do low-level
memory stuff in a safe way via the type system. Without further ado, let’s dig in.</p>
<!-- vim-markdown-toc GFM -->
<ul>
<li><a href="#the-tess-type">The Tess type</a>
<ul>
<li><a href="#digression-on-interleaved-and-deinterleaved-data-structures">Digression on interleaved and deinterleaved data structures</a></li>
<li><a href="#the-tess-type-revisited">The Tess type… revisited</a></li>
</ul>
</li>
<li><a href="#the-magic-of-type-systems">The magic of type systems</a></li>
<li><a href="#deinterleaving-compile-time-dispatching">Deinterleaving compile-time dispatching</a></li>
<li><a href="#wrap-it-up">Wrap it up</a></li>
</ul>
<!-- vim-markdown-toc -->
<h1>The Tess type</h1>
<p>In pre <strong>luminance-0.40</strong>, it is possible to gather <em>vertices</em> inside a type called <code>Tess</code> (it
stands for <em>tessellation</em>). That type is responsible for holding vertices to render them later.
They might represent point clouds, triangle-based meshes, lines, etc. It contains several properties,
but we are going to focus on three:</p>
<ul>
<li>The actual <em>vertex</em> data (i.e. the held vertices).</li>
<li><em>Indices</em>, that can be used to prevent duplicating vertices and index them to reduce the size
of the mesh.</li>
<li><em>Vertex instances</em>, which are special data that are instance-dependent. In a graphics library or
a graphics engine, an <em>instance</em> often refers to an instance of a template, a pattern, that is to
be shared among instances: the instances simply add some specific values, like positions, colors,
etc. Imagine in a simplistic video game a garden with trees: it’s likely that the trees (all of
them) are rendered via a single draw call, which contains instance data that is going to
customize each instance.</li>
</ul>
<p>In <strong>luminance-0.39</strong>, vertices, indices and instances can be retrieved, both read-only on
read-write, via a mechanism of GPU slicing: you ask luminance to give you a type with which you are
able to use <code>&amp;[T]</code> and <code>&amp;mut [T]</code>.</p>
<p>However… we have several problems. First, the type is <code>Tess</code> and is completely monomorphized. What
it means is that if you create a <code>Tess</code> that contains vertices of type <code>SimpleVertex</code> with indices
of type <code>u8</code>, that information is not encoded in the type system — it is actually encoded as a
value that is read and checked at runtime! When you ask for a slice, for instance to mutate the
content of the <code>Tess</code>, you have to use a function like <a href="https://docs.rs/luminance/0.39.0/luminance/tess/struct.Tess.html#method.as_slice"><code>Tess::as_slice&lt;V&gt; where V: Vertex</code></a>, which
expects you to pass the type of stored vertices — in our case, it would be <code>SimpleVertex</code>. What
happens if someone passed the wrong type? Well, currently, <strong>luminance</strong> checks at runtime that the
stored type is the right one, but this is both wasted checks and not a very elegant API.</p>
<p>The same applies to indices and instance data: you don’t see them in the type. What happens if
you slice the indices with <code>u32</code>? Runtime checks.</p>
<p>Now, there’s also the problem of building. When you build a <code>Tess</code>, you have to pass the vertices
to the <code>TessBuilder</code> type, using functions like <a href="https://docs.rs/luminance/0.39.0/luminance/tess/struct.TessBuilder.html#method.add_vertices"><code>TessBuilder::add_vertices</code></a>. It works, but
it’s not practical. More importantly: if you call several times that function, you will create
something that is called a <em>deinterleaved memory</em> data structure. Let’s digress a bit about that.</p>
<h2>Digression on interleaved and deinterleaved data structures</h2>
<p>Interleaving and deinterleaving makes sense when we talk about several objects / items of a same
type, laid out in memory — typically in arrays. Imagine a type, <code>Vertex</code>, such as:</p>
<pre><code class="language-rust">struct Vertex {
  position: Point3D,
  color: RGBA,
}
</code></pre>
<p>If you take a vector of <code>Vertex</code> (<code>Vec&lt;Vertex&gt;</code>), you get a continuous array of properties in
memory. If you represent that in memory, you have something similar to this for two vertices
(padding omitted):</p>
<pre><code>[x0, y0, z0, r0, g0, b0, a0, x1, y1, z1, r1, g1, b1, a1]
</code></pre>
<p>We say that the memory is <em>interleaved</em>, because you’re going to alternate between fields when
iterating in memory. Everything is interleaved. This kind of memory is what happens when you put a
<code>struct</code> in an array, slice, etc. and is perfectly suited for most situations (even on the GPU).
However, there is a small (yet important) detail: when you iterate (for instance in a <code>for</code> loop)
on that array, you’re going to ask your CPU / GPU to load a bunch of values at once. That is going
to fill your <em>cache lines</em>. If at each iteration you need to use every fields of the vertex, then
that situation is pretty convenient, because you’re going to have a bunch of fields cached ahead
(cache hits).</p>
<p>However… imagine a loop that only needs to access the <code>position</code> field. What’s going to happen is
that your CPU / GPU will still load the same data in cache lines: now you get colors in the cache
that you don’t need and your loop will make more cache misses. What could have been better would
have been to fill the cache lines only with positions. If we had, instead:</p>
<pre><code>[x0, y0, z0, x1, y1, z1]
[r0, g0, b0, a0, r1, g1, b1, a1]
</code></pre>
<p>Those two vectors can then be used independently for each need. Because we only need positions, we
can simply use the first position vector. Now, when the CPU / GPU is going to load something in the
cache, it’s going to cache much more values that we are going to actually use: we get more cache
hits and it’s <em>playa party</em>.</p>
<p>That kind of memory layout is called <em>deinterleaved memory</em>. The way we typically do that is by
simply moving the fields out of the <code>struct</code> and make several arrays of each field.</p>
<p>People tend to use two terms to describe both layouts: <a href="https://en.wikipedia.org/wiki/AoS_and_SoA">AoS and SoA</a>.</p>
<p>So… ~~months~~ <em>years</em> ago, I realized that and decided a needed I better plan. Especially, on
<strong>luminance-0.39</strong>, the way you handle slicing <em>deinterleaved data</em> is… well, inexistent. You
cannot slice such data because it was never supported.</p>
<h2>The Tess type… revisited</h2>
<p>The new type is the following:</p>
<pre><code class="language-rust">pub struct Tess&lt;B, V, I = (), W = (), S = Interleaved&gt;
where ABunchOfThings;
</code></pre>
<p>As you can see, there are a lot of new things there:</p>
<ul>
<li><code>B</code>: the new <em>backend type</em> variable that most types now have; used to forward backend
implementations down the API.</li>
<li><code>V</code>: the <em>vertex type</em>. It’s the type of <code>Vertex</code> you have typically defined yourself using
<a href="https://crates.io/crates/luminance-derive">luminance-derive</a>.</li>
<li><code>I</code>: the <em>index type</em>. Most of the time, you’ll use either <code>u8</code>, <code>u16</code> or <code>u32</code>, depending on
the kind of geometry you index.</li>
<li><code>W</code>: the <em>vertex instance data</em>. Because people might not want that, <code>()</code> is a good default, but
if you really need it, you can basically use any <code>Vertex</code> type here, since vertex instance data
must be part of a <code>Semantics</code>.</li>
<li><code>S</code>: the interleaving mode. Either <code>Interleaved</code> or <code>Deinterleaved</code>.</li>
</ul>
<p>You will find the same type variables with the <code>TessBuilder</code> type.</p>
<h1>The magic of type systems</h1>
<p>The cool thing about that change is how it enabled me to yield much, <em>much</em> better APIs. Consider
the <a href="https://github.com/phaazon/luminance-rs/blob/luminance-0.39/luminance-examples/src/hello-world.rs#L214-L219">previous API to create a deinterleaved <code>Tess</code></a>:</p>
<pre><code class="language-rust">let direct_deinterleaved_triangles = TessBuilder::new(&amp;mut surface)
  .add_vertices(TRI_DEINT_POS_VERTICES)
  .add_vertices(TRI_DEINT_COLOR_VERTICES)
  .set_mode(Mode::Triangle)
  .build()
  .unwrap();
</code></pre>
<p>Notice the two <code>add_vertices</code>. There is no type information checking and ensuring that:</p>
<ul>
<li>You are passing vertex data that are compatible with the <code>Vertex</code> type those fields correspond
to.</li>
<li>You can basically call <code>add_vertices</code> as many times as you want and get a big buffer in GPU
memory that will make no sense.</li>
</ul>
<p>Now, the new API <a href="https://github.com/phaazon/luminance-rs/blob/luminance-0.40/luminance-examples/src/hello-world.rs#L182-L188">looks like this</a>:</p>
<pre><code class="language-rust">let direct_deinterleaved_triangles = surface
  .new_deinterleaved_tess::&lt;Vertex, ()&gt;()
  .set_attributes(&amp;TRI_DEINT_POS_VERTICES[..])
  .set_attributes(&amp;TRI_DEINT_COLOR_VERTICES[..])
  .set_mode(Mode::Triangle)
  .build()
  .unwrap();
</code></pre>
<p>If you try to call <code>set_vertices</code> — the name got changed from <code>add_vertices</code> to <code>set_vertices</code>
— on the builder you get from <code>new_deinterleaved_tess</code>, you will get a compilation error, because you
cannot set vertices on deinterleaved tessellations: you need to set attribute vectors. The
<code>set_attributes</code> has the information that you are doing that for a <code>Vertex</code>, so it can check the
input data you pass and ensure it contains values which type is a field type used in <code>Vertex</code>. If
not, you get a compilation error.</p>
<p>Most importantly: because of how vertices work in luminance, a field type is unique to a vertex: it
doesn’t make sense to use twice the <code>VertexPosition</code> type. If you end up in such a situation, it
means that your <code>Semantics</code> type lacks another variant — remember: vertex fields are basically
semantics-based attributes. That leads to the possibility to automatically find out where exactly
the data you provide needs to go inside the GPU tessellation.</p>
<p>The super cool part is that you can now slice deinterleaved tessellations by simply asking for:</p>
<ul>
<li>The slice of vertices.</li>
<li><em>A</em> slice of attributes.</li>
<li>The slice of indices.</li>
</ul>
<p>In our case, we have a deinterleaved tessellation, which means we cannot slice whole vertices. If
you try to get a slice of <code>Vertex</code>, you will get a compilation error. However, we can retrieve slices
of vertex fields. The way we do this is super simple: we simply call the <code>tess.vertices()</code> or
<code>tess.vertices_mut()</code> methods. It will infer the type of slices you are asking to automatically
slice the right GPU buffer. This is all possible because our types are  unique as the vertex fields.</p>
<pre><code class="language-rust">let positions = tess.vertices().unwrap(); // you have to check for errors normally
let colors = tess.vertices().unwrap(); // you have to check for errors normally
</code></pre>
<blockquote>
<p>Remark: you need to have the types of <code>positions</code> and <code>colors</code> inferred by setting / reading /
passing them around, or you will have to put type ascriptions so that <code>vertices()</code> know what
fields you are referring to.</p>
</blockquote>
<h1>Deinterleaving compile-time dispatching</h1>
<p>So let’s dig a bit into how all this works. The first thing you need to know is that
deinterleaving — the raw concept — is really simple. If you have a type such as:</p>
<pre><code class="language-rust">struct Foo {
  a: A,
  b: B,
  c: C,
}
</code></pre>
<p>We say that <code>Vec&lt;Foo&gt;</code> is the interleaved representation of the collection. The deinterleaved
representation needs to have three vectors of fields:</p>
<pre><code class="language-rust">vec_a: Vec&lt;A&gt;
vec_b: Vec&lt;B&gt;
vec_c: Vec&lt;C&gt;
</code></pre>
<p>In order for a <code>Vertex</code> type to be valid in deinterleaving contexts, we need to have that
tuple of vectors representation. First, we need a mapping between <code>Vec&lt;Foo&gt;</code> to
<code>(Vec&lt;A&gt;, Vec&lt;B&gt;, Vec&lt;C&gt;)</code>. This is the role of two traits: <code>Deinterleave&lt;T&gt;</code> and
<code>TessVertexData&lt;S&gt;</code>.</p>
<p><code>Deinterleave&lt;T&gt;</code> gives, for <code>T</code>, the field rank for <code>T</code>. For instance:</p>
<ul>
<li><code>&lt;Foo as Deinterleave&lt;A&gt;&gt;::RANK == 0</code></li>
<li><code>&lt;Foo as Deinterleave&lt;B&gt;&gt;::RANK == 1</code></li>
<li><code>&lt;Foo as Deinterleave&lt;C&gt;&gt;::RANK == 2</code></li>
</ul>
<p>This is mandatory so that we know exactly which GPU buffers will need to be read / written to
when creating tessellations and slicing them. You don’t have to implement <code>Deinterleave&lt;T&gt;</code> by
yourself: <a href="https://crates.io/crates/luminance-derive">luminance-derive</a> does that automatically for you when you create a <code>Vertex</code> type.
Also, you might be tempted to think that this rank will be used inside the <code>Vertex</code> type to
retrieve data, but since you cannot pass whole vertices… nah. Also, you shouldn’t assume ranks
based on fields declarations in struct (rustc can re-order that).</p>
<p>Next is <code>TessVertexData&lt;S&gt;</code>. It associates an input type — the type of data a tessellation will
receive at creation type — for <code>S</code>, for the implementor type. The easy one
is <code>TessVertexData&lt;Interleaved&gt; for V where V: Vertex</code>. The associated type is simply <code>Vec&lt;V&gt;</code>,
because interleaved geometry simply stores the vertices as a vector of the whole vertex type.
Simple.</p>
<p>It gets more complicated when we talk about deinterleaved geometry.
<code>TessVertexData&lt;Deinterleaved&gt; for V where V: Vertex</code> has its associated type set to…
<code>Vec&lt;DeinterleavedData&gt;</code>. Indeed: there is no simple way with the current stable Rust (and even
nightly) to know the full type of <code>Vertex</code> fields at compile-time here. However, don’t get it wrong:
that <code>Vec</code> is not a vector of vertices. It’s a typically small set of attributes (vectors  too). If
you look at the definition of <code>DeinterleavedData</code>, you get this:</p>
<pre><code class="language-rust">#[derive(Debug, Clone)]
pub struct DeinterleavedData {
  raw: Vec&lt;u8&gt;,
  len: usize,
}
</code></pre>
<p>Yep. Type erasure at its finest. When you pass deinterleaved data, the data is type-erased and
passed as a big blob of bytes, glued with its original size (so that we don’t have to store typing
information – this will be needed when slicing vertex attributes).</p>
<p>Implementing slicing with these traits and data types is now possible: we can add another trait
that we will use to slice vertices, for instance (the <code>backend::VertexSlice</code> trait in luminance)
and based on the type of <code>T</code> in <code>Deinterleave&lt;T&gt;</code>, we can go and grab the GPU buffer we want.
For instance, in both the OpenGL and WebGL luminance backends, buffers are stored in a <code>Vec&lt;_&gt;</code>,
so in order to know which one we need to lookup, we simple use the
<code>&lt;Vertex as Deinterleave&lt;T&gt;&gt;::RANK</code> constant value (a <code>usize</code>) and we’re good to go. You
need two other traits for vertex instance data (<code>InstanceSlice</code>) and vertex indices (<code>IndexSlice</code>),
and you’re good to go.</p>
<h1>Wrap it up</h1>
<p>So, checking at compile-time deinterleaving, in luminance, is done by:</p>
<ul>
<li>Using a trait (<code>Deinterleave&lt;T&gt;</code>) to associate a <em>rank</em> to a field <code>T</code> in a data structure.</li>
<li>Using a trait (<code>TessVertexData&lt;S&gt;</code>) to get the input type of a tessellation: either <code>Vec&lt;T&gt;</code> for
the whole vertices, or <code>Vec&lt;DeinterleavedData&gt;</code> for a set of attributes).</li>
<li>When asking for a typed slice, such as <code>vertices_mut::&lt;T&gt;()</code>, we basically simply require
<code>Deinterleave&lt;T&gt; for V</code> so that we can get a <code>RANK: usize</code>. We can then, in the backend
implementation, find the right GPU buffer and slice it. The reconstruction to a typed slice
(<code>&amp;[T]</code>) is statically assured by the length stored in the <code>DeinterleavedData</code> type and by the
fact <code>DeinterleavedData&lt;T&gt;</code> is implemented for <code>V</code>.</li>
</ul>
<p>A small note though. luminance — actually, I do that with all my code, whatever the language —
considers a lot of operations <code>unsafe</code>. In this case, implementing <code>Deinterleave&lt;T&gt;</code> is <code>unsafe</code>.
The reason is pretty obvious: you can implement <code>Deinterleave&lt;Whatever&gt;</code>, even if your vertex
type doesn’t have a <code>Whatever</code> field. Doing so would yield a hazardous / undefined behavior when
slicing the GPU buffer. <a href="https://crates.io/crates/luminance-derive">luminance-derive</a> takes care of implementing the <code>unsafe</code> interface for
you.</p>
<p>I hope you have learned something new or gotten some ideas. Keep the vibes!</p>


guid: https://strongly-typed-thoughts.net/blog/cheddar-0.1Mon, 09 Jul 2018 01:00:00 GMT
title: The Cheddar shading language
author: Dimitri Sabadie <dimitri.sabadie@gmail.com>
url: https://strongly-typed-thoughts.net/blog/cheddar-0.1
feedurl: https://phaazon.net/blog/feed
content:
<p>It’s been a long time I wanted to announce and reveal this, and this morning is now the moment!
This blog post announces the release of <a href="https://crates.io/crates/cheddar">cheddar</a> in version 0.1!</p>
<h1>Foreword</h1>
<p>Cheddar is a GLSL <em>superset</em> language. What it means is that most of the GLSL constructs and syntax
you’re used to is valid in Cheddar – not <em>all of it</em>; <em>most of it</em>. Cheddar adds a set of features
that I think are lacking to GLSL. Among them:</p>
<ul>
<li>Some non-valid GLSL constructions made valid in Cheddar to ease the writing of certain shader
stages.</li>
<li>A more functional approach to programming shaders on the GPU.</li>
<li>Structures, types and GLSL-specific <em>constructs sharing</em>.</li>
<li>Imports and modules with live reloading and transitive dependencies.</li>
</ul>
<p>However, Cheddar is not:</p>
<ul>
<li>A new shading language that abstracts over any kind of shading language (i.e. unlike <a href="https://en.wikipedia.org/wiki/Cg_(programming_language)">Cg</a> for
instance). However, since it should be possible for GLSL to be transpiled to SPIR-V (hence
Vulkan), it should be possible to transpile Cheddar to SPIR-V.</li>
<li>A dedicated DSL to a specific graphics problem. Cheddar is very similar to GLS (actually, that’s
why it’s called a GLSL <em>superset</em>). If you’re looking for a solution to ease writing shaders in
a way that you don’t have to learn shaders, Cheddar won’t help you much with this.</li>
</ul>
<p>The documentation is still a big <em>work in progress</em> but most of it should give you enough
information to get started.</p>
<h1>A bit of backstory</h1>
<p>Cheddar was imagined and designed while I was working on <a href="https://crates.io/crates/spectra">spectra</a>, a <em>work in progress</em> demoscene
crate I’ve been working for a while now. I released two demos thanks to <a href="https://crates.io/crates/spectra">spectra</a> –
<a href="https://www.youtube.com/watch?v=pYqZS1C_7PU">this one</a> and
<a href="https://www.youtube.com/watch?v=ug7eRowgVw4">this one</a>. Because those demos are very <em>meh</em> to me, I
decided to enhance my tools. Among them was the need to write shaders a better and easier way. Then
Cheddar got born. I used Cheddar while preparing other demos, and I was talking about it on IRC and
Reddit, people seemed to be interested – I even had a friend writing demos in C++ interested!</p>
<p>So here it is. Please, provide feedback if you try it, should you like it or not!</p>
<h1>Dig in</h1>
<blockquote>
<p>Disclaimer: the current state of the language is pretty experimental and unstable. There’s no
semantics checking, for instance.</p>
</blockquote>
<p>You can read the full tutorial and design document on the <a href="https://docs.rs/cheddar/0.1.0/cheddar">official documentation page</a>.</p>
<p>Thanks for having read me, and as always, keep the vibe!</p>


guid: https://strongly-typed-thoughts.net/blog/luminance-0.8Wed, 09 Dec 2015 00:00:00 GMT
title: Existential quantification and GADT in luminance-0.8
author: Dimitri Sabadie <dimitri.sabadie@gmail.com>
url: https://strongly-typed-thoughts.net/blog/luminance-0.8
feedurl: https://phaazon.net/blog/feed
content:
<h1>luminance 0.8 and existential quantification</h1>
<p>It’s been a while I haven’t released anything on my blog. I just wrote a few changes for the latest
version of luminance, <a href="https://hackage.haskell.org/package/luminance-0.8.2">luminance-0.8.2</a> and I
decided to write about it because I think those changes are interesting on a Haskell level.</p>
<h2>The problem</h2>
<p>If you haven’t read the <a href="https://hackage.haskell.org/package/luminance-0.8.2/changelog">changelog</a>
yet, I changed the <code>createProgram</code> function and the way it handles uniform interfaces. In
luminance &lt; 0.8, you were provided with as many functions as there are uniform kinds. Up to now,
luminance supports two uniform kinds:</p>
<ul>
<li>simple uniforms;
<ul>
<li>uniform block (UBO).</li>
</ul>
</li>
</ul>
<p>So you had two rank-2 functions like <code>forall a. (Uniform a) =&gt; Either String Natural -&gt; UniformInterface m (U a)</code> and <code>forall a. (UniformBlock a) =&gt; String -&gt; UniformInterface m (U (Region rw (UB a)))</code> to map whichever uniforms you wanted to.</p>
<p>The issue with that is that it requires to break the interface of <code>createProgram</code> each time we want
to add a new kind of uniform, and it’s also <a href="https://hackage.haskell.org/package/luminance-0.7.2/docs/Graphics-Luminance-Shader-Program.html#v:createProgram">a pretty hard to read function signature</a>!</p>
<p>So… how does luminance-0.8 solve that?</p>
<h2>(Generalized) Algebraic data types, rank-2 and existential quantification</h2>
<p>What is the only way we have to select uniforms? Names. Names can either be a <code>String</code> or a
<code>Natural</code> for explicit semantics. We could encode such a name using an algebraic data type:</p>
<pre><code class="language-haskell">data UniformName
  = UniformName String
	| UniformSemantic Natural
	  deriving (Eq,Show)
</code></pre>
<p>That’s a good start. Though, we still have the problem of choosing the kind of uniform because we
still have several functions – one per kind. We could encode the kind of the uniform directly into
the name. After all, when we ask for a uniform mapping through a name, we require to know the kind.
So that kind of makes sense. Let’s change our <code>UniformName</code> type:</p>
<pre><code class="language-haskell">data UniformName :: * -&gt; * where
  UniformName :: String -&gt; UniformName a
	UniformSemantic :: Natural -&gt; UniformName a
	UniformBlockName :: String -&gt; UniformName (Region rw (UB a))
</code></pre>
<p>That’s neat, but with that definition, we won’t go anywhere, because we’re too polymorphic. Indeed,
<code>UniformName &quot;foo&quot; :: UniformName a</code> can have any <code>a</code>. We need to put constraints on <code>a</code>. And that’s
where <em>GADTs</em> come in so handy! We can hide the constraints in the constructors and bring them into
scope when pattern matching. That’s a very neat feature of <em>GADTs</em>. So now, let’s add some
constraints to our constructors:</p>
<pre><code class="language-haskell">data UniformName :: * -&gt; * where
  UniformName :: (Uniform a) =&gt; String -&gt; UniformName a
	UniformSemantic :: (Uniform a) =&gt; Natural -&gt; UniformName a
	UniformBlockName :: (UniformBlock a) =&gt; String -&gt; UniformName (Region rw (UB a))
</code></pre>
<p>Yes! Now, we can write a function that takes a <code>UniformName a</code>, pattern matches it and call the
appropriate function regarding the infered shape of <code>a</code>!</p>
<p>However, how do we forward the error? In older version of luminance, we were using <code>ProgramError</code>
and more especially two of its constructors: <code>InactiveUniform</code> and <code>InactiveUniformBlock</code>. We
need to shrink that to a single <code>InactiveUniform</code> constructor and find a way to store our
<code>UniformName</code>… But we can’t yet, because of the <code>a</code> parameter! So the idea is to hide it through
existential quantification!</p>
<pre><code class="language-haskell">data SomeUniformName = forall a. SomeUniformName (UniformName a)

instance Eq SomeUniformName where
  -- …

instance Show SomeUniformName where
  -- …
</code></pre>
<p>And now we can store <code>SomeUniformName</code> in <code>InactiveUniform</code>. We won’t need to recover the type, we
just need the constructor and the carried name. By pattern matching, we can recover both those
information!</p>
<h2>Conclusion</h2>
<p>Feel free to have a look at the new <a href="https://hackage.haskell.org/package/luminance-0.8.1/docs/Graphics-Luminance-Shader-Program.html#v:createProgram"><code>createProgram</code> function</a>.
As you will see, the type signature is easier to read and to work with! :)</p>
<p>Have fun, and keep the vibe!</p>


guid: https://strongly-typed-thoughts.net/blog/state_of_luminanceThu, 14 Sep 2017 00:00:00 GMT
title: State of luminance
author: Dimitri Sabadie <dimitri.sabadie@gmail.com>
url: https://strongly-typed-thoughts.net/blog/state_of_luminance
feedurl: https://phaazon.net/blog/feed
content:
<p>I’ve been a bit surprised for a few days now. Some rustaceans posted two links about my <a href="https://crates.io/crates/spectra">spectra</a>
and <a href="https://crates.io/crates/luminance">luminance</a> crates on reddit – links <a href="https://www.reddit.com/r/rust/comments/6z58n3/spectra_a_demoscene_engine_in_rust">here</a>
and <a href="https://www.reddit.com/r/rust_gamedev/comments/6zmtb4/luminance_typesafe_typelevel_and_stateless_rust">here</a>.
I didn’t really expect that: the code is public, on Github, but I don’t communicate about them <em>that
much</em>.</p>
<p>However, I saw interested people, and I think it’s the right time to write a blog post about some
design choices. I’ll start off with luminance and I’ll speak about spectra in another blog post
because I truly think spectra starts to become very interesting (I have my own shading language
bundled up within, which is basically GLSL on steroids).</p>
<h1>luminance and how it copes with fast and as-stateless-as-possible graphics</h1>
<h2>Origins</h2>
<p>luminance is a library that I wrote, historically, in Haskell. You can find the package
<a href="https://hackage.haskell.org/package/luminance">here</a> if you’re interested – careful though, it’s
dying and the Rust version has completely drifted path with it. Nevertheless, when I ported the
library to Rust, I imported the same “way of programming” that I had·have in Haskell – besides the
allocation scheme; remember, I’m a demoscener, I <strong>do care a lot about performance, CPU usage, cache
friendliness and runtime size</strong>. So the Rust luminance crate was made to be hybrid: it has the
cool functional side that I imported from my Haskell codebase, and the runtime performance I wanted
that I had when I wrote my two first 64k in C++11. I had to remove and work around some features
that only Haskell could provide, such as higher-kinded types, type families, functional
dependencies, GADTs and a few other things such as existential quantification (trait objects saved
me here, even though I don’t use them that much in luminance now).</p>
<p>I have to admit, I dithered a lot about the scope of luminance — both in Haskell and Rust. At first,
I thought that it’d be great to have a <em>“base”</em> crate, hosting common and abstracted code, and
<em>“backend”</em> crates, implementing the abstract interface. That would enable me to have several
backends – OpenGL, Vulkan, Metal, a software implementation, something for Amiigaaaaaaa, etc.
Though, time has passed, and now, I think it’s:</p>
<ul>
<li>Overkill.</li>
<li>A waste of framework.</li>
</ul>
<p>The idea is that if you need to be very low-level on the graphics stack of your application, you’re
likely to know what you are doing. And then, your needs will be very precise and well-defined. You
might want very specific piece of code to be available, related to a very specific technology.
That’s the reason why abstracting over very low-level code is not a good path to me: you need to
expose as most as posssible the low-level interface. That’s the goal of luminance: exposing OpenGL’s
interface in a stateless, bindless and typesafe way, with no or as minimal as possible runtime
overhead.</p>
<p>More reading <a href="http://phaazon.blogspot.fr/2016/08/luminance-designs.html">here</a>.</p>
<h2>Today</h2>
<p>Today, luminance is almost stable – it still receives massive architecture redesign from time to
time, but it’ll hit the <code>1.0.0</code> release soon. As discussed with <a href="https://github.com/kvark">kvark</a> lately, luminance is not
about the same scope as <a href="https://crates.io/crates/gfx">gfx</a>’s one. The goals of luminance are:</p>
<ul>
<li>To be a typesafe, stateless and bindless OpenGL framework.</li>
<li>To provide a friendly experience and expose as much as possible all of the OpenGL features.</li>
<li>To be very lightweight (the target is to be able to use it without <code>std</code> nor <code>core</code>).</li>
</ul>
<p>To achieve that, luminance is written with several aspects in mind:</p>
<ul>
<li>Allocation must be explicitely stated by the user: we must avoid as much as possible to allocate
things in luminance since it might become both a bottleneck and an issue to the lightweight aspect.</li>
<li>Performance is a first priority; safety comes second. If you have a feature that can be either
exclusively performant or safe, it must then be performant. Most of the current code is, for our
joy, both performant and safe. However, some invariants are left around the place and you might
shoot your own feet. This is an issue and some reworking must be done (along with tagging
some functions and traits <code>unsafe</code>).</li>
<li>No concept of backends will ever end up in luminance. If it’s decided to switch to <a href="https://www.khronos.org/vulkan">Vulkan</a>, the
whole luminance API will and <strong>must</strong> be impacted, so that people can use <a href="https://www.khronos.org/vulkan">Vulkan</a> the best
possible way.</li>
<li>A bit like the first point, the code must be written in a way that the generated binary is as
small as possible. Generics are not forbidden – they’re actually recommended – but things like
crate dependencies are likely to be forbidden (exception for the <code>gl</code> dependency, of course).</li>
<li>Windowing <strong>must not be addressed by luminance</strong>. This is crucial. As a demoscener, if I want to
write a 64k with luminance, I must be able to use a library over X11 or the Windows API to setup 
the OpenGL context myself, set the OpenGL pointers myself, etc. This is not the typical usecase –
who cares besides demosceners?! – but it’s still a good advantage since you end up with loose
coupling for free.</li>
</ul>
<h2>The new luminance</h2>
<p>luminance has received more attention lately, and I think it’s a good thing to talk about how to use
it. I’ll add examples on github and its <a href="https://docs.rs/luminance">docs.rs</a> online documentation.</p>
<p>I’m going to do that like a tutorial. It’s easier to read and you can test the code in the same
time. Let’s render a triangle!</p>
<blockquote>
<p>Note: keep in mind that you need a nightly compiler to compile luminance.</p>
</blockquote>
<h3>Getting your feet wet</h3>
<p>I’ll do everything from scratch with you. I’ll work in <code>/tmp</code>:</p>
<pre><code>$ cd /tmp
</code></pre>
<p>First thing first, let’s setup a <code>lumitest</code> Rust binary project:</p>
<pre><code>$ cargo init --bin lumitest
  Created binary (application) project
$ cd lumitest
</code></pre>
<p>Let’s edit our <code>Cargo.toml</code> to use luminance. We’ll need two crates:</p>
<ul>
<li>The <a href="https://crates.io/crates/luminance">luminance</a> crate.</li>
<li>A way to open the OpenGL context; we’ll use GLFW, so the <a href="https://crates.io/crates/luminance-glfw">luminance-glfw</a> crate.</li>
</ul>
<p>At the time of writing, corresponding versions are
<a href="https://crates.io/crates/luminance/0.23.0">luminance-0.23.0</a> and
<a href="https://crates.io/crates/luminance-glfw/0.3.2">luminance-glfw-0.3.2</a>.</p>
<p>Have the following <code>[dependencies]</code> section</p>
<pre><code>[dependencies]
luminance = &quot;0.23.0&quot;
luminance-glfw = &quot;0.3.2&quot;
</code></pre>
<pre><code>$ cargo check
</code></pre>
<p>Everything should be fine at this point. Now, let’s step in in writing some code.</p>
<pre><code>
extern crate luminance;
extern crate luminance_glfw;

use luminance_glfw::{Device, WindowDim, WindowOpt};

const SCREEN_WIDTH: u32 = 960;
const SCREEN_HEIGHT: u32 = 540;

fn main() {
  let rdev = Device::new(WindowDim::Windowed(SCREEN_WIDTH, SCREEN_HEIGHT), "lumitest", WindowOpt::default());
}
</code></pre>
<p>The <code>main</code> function creates a <a href="https://docs.rs/luminance-glfw/0.3.2/luminance_glfw/struct.Device.html">Device</a>
that is responsible in holding the windowing stuff for us.</p>
<p>Let’s go on:</p>
<pre><code>
match rdev {
  Err(e) => {
    eprintln!("{:#?}", e);
    ::std::process::exit(1);
  }

  Ok(mut dev) => {
    println!("let’s go!");
  }
}
</code></pre>
<p>This block will catch any <code>Device</code> errors and will print them to <code>stderr</code> if there’s any.</p>
<p>Let’s write the main loop:</p>
<pre><code>
'app: loop {
  for (_, ev) in dev.events() { // the pair is an interface mistake; it’ll be removed
    match ev {
      WindowEvent::Close | WindowEvent::Key(Key::Escape, _, Action::Release, _) => break 'app,
      _ => ()
    }
  }
}
</code></pre>
<p>This loop runs forever and will exit if you hit the escape key or quit the application.</p>
<h3>Setting up the resources</h3>
<p>Now, the most interesting thing: rendering the actual triangle! You will need a few things:</p>
<pre><code>
type Position = [f32; 2];
type RGB = [f32; 3];
type Vertex = (Position, RGB);

const TRIANGLE_VERTS: [Vertex; 3] = [
  ([-0.5, -0.5], [0.8, 0.5, 0.5]), // red bottom leftmost
  ([-0., 0.5], [0.5, 0.8, 0.5]), // green top
  ([0.5, -0.5], [0.5, 0.5, 0.8]) // blue bottom rightmost
];
</code></pre>
<p><code>Position</code>, <code>Color</code> and <code>Vertex</code> define what a vertex is. In our case, we use a 2D position and a
RGB color.</p>
<blockquote>
<p>You have a lot of choices here to define the type of your vertices. In theory, you can choose
any type you want. However, it must implement the <a href="https://docs.rs/luminance/0.23.0/luminance/vertex/trait.Vertex.html"><code>Vertex</code></a>
trait. Have a look at the implementors that already exist for a faster start off!</p>
</blockquote>
<blockquote>
<p><strong>Important</strong>: do not confuse between <code>[f32; 2]</code> and <code>(f32, f32)</code>. The former is a single 2D
vertex component. The latter is two 1D components. It’ll make a huge difference when writing shaders.</p>
</blockquote>
<p>The <code>TRIANGLE_VERTS</code> is a constant array with three vertices defined in it: the three vertices of
our triangle. Let’s pass those vertices to the GPU with the <a href="https://docs.rs/luminance/0.23.0/luminance/tess/struct.Tess.html"><code>Tess</code></a>
type:</p>
<pre><code>
// at the top location
use luminance::tess::{Mode, Tess, TessVertices};

// just above the main loop
let triangle = Tess::new(Mode::Triangle, TessVertices::Fill(&TRIANGLE_VERTS), None);
</code></pre>
<p>This will pass the <code>TRIANGLE_VERTS</code> vertices to the GPU. You’re given back a <code>triangle</code> object. The
<a href="https://docs.rs/luminance/0.23.0/luminance/tess/enum.Mode.html"><code>Mode</code></a> is a hint object that
states how vertices must be connected to each other. <code>TessVertices</code> lets you slice your vertices –
this is typically enjoyable when you use a mapped buffer that contains a dynamic number of vertices.</p>
<p>We’ll need a <em>shader</em> to render that triangle. First, we’ll place its source code in <code>data</code>:</p>
<pre><code>$ mkdir data
</code></pre>
<p>Paste this in <code>data/vs.glsl</code>:</p>
<pre><code>
layout (location = 0) in vec2 co;
layout (location = 1) in vec3 color;

out vec3 v_color;

void main() {
  gl_Position = vec4(co, 0., 1.);
  v_color = color;
}
</code></pre>
<p>Paste this in <code>data/fs.glsl</code>:</p>
<pre><code>
in vec3 v_color;

out vec4 frag;

void main() {
  frag = vec4(v_color, 1.);
}
</code></pre>
<p>And add this to your <code>main.rs</code>:</p>
<pre><code>
const SHADER_VS: &str = include_str!("../data/vs.glsl");
const SHADER_FS: &str = include_str!("../data/fs.glsl");
</code></pre>
<blockquote>
<p>Note: this is not a typical workflow. If you’re interested in shaders, have a look at how I do it
in <a href="https://crates.io/crates/spectra">spectra</a>. That is, hot reloading it via SPSL (Spectra Shading Language), which enables to
write GLSL modules and compose them in a single file but just writing functions. The functional
programming style!</p>
</blockquote>
<p>Same thing as for the tessellation, we need to pass the source to the GPU’s compiler to end up with
a shader object:</p>
<pre><code>
// add this at the top of your main.rs
use luminance::shader::program::Program;

// below declaring triangle
let (shader, warnings) = Program::<Vertex, (), ()>::from_strings(None, SHADER_VS, None, SHADER_FS).unwrap();

for warning in &warnings {
  eprintln!("{:#?}", warning);
}
</code></pre>
<p>Finally, we need to tell luminance in which framebuffer we want to make the render. It’s simple: to
the default framebuffer, which ends up to be… your screen’s back buffer! This is done this way with
luminance:</p>
<pre><code>
use luminance::framebuffer::Framebuffer;

let screen = Framebuffer::default([SCREEN_WIDTH, SCREEN_HEIGHT]);
</code></pre>
<p>And we’re done for the resources. Let’s step in the actual render now.</p>
<h3>The actual render: the pipeline</h3>
<p>luminance’s approach to render is somewhat not very intuitive yet very simple and very efficient:
the render pipeline is explicitly defined by the programmer in Rust, on the fly. That means that
you must express the actual state the GPU must have for the whole pipeline. Because of the nature of
the pipeline, which is an AST (Abstract Syntax Tree), you can batch sub-parts of the pipeline (we
call such parts <em>nodes</em>) and you end up with minimal GPU state switches. The theory is as following:</p>
<ul>
<li>At top most, you have the <a href="https://docs.rs/luminance/0.23.0/luminance/pipeline/fn.pipeline.html"><code>pipeline</code></a>
function that introduces the concept of <em>shading things to a framebuffer</em>.
<ul>
<li>Nested, you find the concept of a <em>shader gate</em>. That is, an object linked to its parent
(pipeline) and that gives you the concept of <em>shading things with a shader</em>.
<ul>
<li>Nested, you find the concept of <em>rendering things</em>. That is, can set on such nodes GPU states,
such as whether you want a depth test, blending, etc.
<ul>
<li>Nested, you find the concept of a <em>tessellation gate</em>, enabling you to render actual <code>Tess</code>
objects.</li>
</ul>
</li>
</ul>
</li>
</ul>
</li>
</ul>
<p>That deep nesting enables you to batch your objects on a very fine granularity. Also, notice that
the functions are not about slices of <code>Tess</code> or hashmaps of <code>Program</code>. The allocation scheme is
completely ignorant about how the data is traversed, which is good: you decide. If you need to
borrow things on the fly in a shading gate, you can.</p>
<p>Let’s get things started:</p>
<pre><code>
use luminance::pipeline::{entry, pipeline};

entry(|_| {
  pipeline(&screen, [0., 0., 0., 1.], |shd_gate| {
    shd_gate.shade(&shader, |rdr_gate, _| {
      rdr_gate.render(None, true, |tess_gate| {
        let t = &triangle;
        tess_gate.render(t.into());
      });
    });
  });
});
</code></pre>
<p>We just need a final thing now: since we render to the back buffer of the screen, if we want to see
anything appear, we need to <em>swap the buffer chain</em> so that the back buffer become the front buffer
and the front buffer become the back buffer. This is done by wrapping our render code in the
<a href="https://docs.rs/luminance-glfw/0.3.2/luminance_glfw/struct.Device.html#method.draw"><code>Device::draw</code></a>
function:</p>
<pre><code>
dev.draw(|| {
  entry(|_| {
    pipeline(&screen, [0., 0., 0., 1.], |shd_gate| {
      shd_gate.shade(&shader, |rdr_gate, _| {
        rdr_gate.render(None, true, |tess_gate| {
          let t = &triangle;
          tess_gate.render(t.into());
        });
      });
    });
  });
});
</code></pre>
<p>You should see this:</p>
<p><img src="https://i.imgur.com/x3jC9qm.png" alt="" /></p>
<p>As you can see, the code is pretty straightforward. Let’s get deeper, and let’s kick some time in!</p>
<pre><code>
use std::time::Instant;

// before the main loop
let t_start = Instant::now();
// in your main loop
let t_dur = t_start.elapsed();
let t = (t_dur.as_secs() as f64 + t_dur.subsec_nanos() as f64 * 1e-9) as f32;
</code></pre>
<p>We have the time. Now, we need to pass it down to the GPU (i.e. the shader). luminance handles that
kind of things with two concepts:</p>
<ul>
<li>Uniforms.</li>
<li>Buffers.</li>
</ul>
<p>Uniforms are a good match when you want to send data to a specific shader, like a value that
customizes the behavior of a shading algorithm.</p>
<p>Because buffers are shared, you can use buffers to share data between shader, leveraging the need to
pass the data to all shader by hand – you only pass the index to the buffer that contains the data.</p>
<p>We won’t conver the buffers for this time.</p>
<p>Because of type safety, luminance requires you to state which types the uniforms the shader contains
are. We only need the time, so let’s get this done:</p>
<pre><code>
// you need to alter this import
use luminance::shader::program::{Program, ProgramError, Uniform, UniformBuilder, UniformInterface, UniformWarning};

struct TimeUniform(Uniform<f32>);

impl UniformInterface for TimeUniform {
  fn uniform_interface(builder: UniformBuilder) -> Result<(Self, Vec<UniformWarning>), ProgramError> {
    // this will fail if the "t" variable is not used in the shader
    //let t = builder.ask("t").map_err(ProgramError::UniformWarning)?;

    // I rather like this one: we just forward up the warning and use the special unbound uniform
    match builder.ask("t") {
      Ok(t) => Ok((TimeUniform(t), Vec::new())),
      Err(e) => Ok((TimeUniform(builder.unbound()), vec![e]))
    }
  }
}
</code></pre>
<p>The <a href="https://docs.rs/luminance/0.23.0/luminance/shader/program/struct.UniformBuilder.html#method.unbound"><code>UniformBuilder::unbound</code></a>
is a simple function that gives you any uniform you want: the resulting uniform object will just do
nothing when you pass values in. It’s a way to say <em>“— Okay, I don’t use that in the shader yet, but
don’t fail, it’s not really an error”.</em> Handy.</p>
<p>And now, all the magic: how do we access that uniform value? It’s simple: via types! Have you
noticed the type of our <code>Program</code>? For the record:</p>
<pre><code>
let (shader, warnings) = Program::<Vertex, (), ()>::from_strings(None, SHADER_VS, None, SHADER_FS).unwrap();
</code></pre>
<p>See the type is parametered with three type variables:</p>
<ul>
<li>The first one – here <code>Vertex</code>, our own type – is for the <em>input</em> of the shader program.</li>
<li>The second one is for the <em>output</em> of the shader program. It’s currently not used at all by
luminance by is reserved, as it will be used later for enforcing even further type safety.</li>
<li>The third and latter is for the <em>uniform interface</em>.</li>
</ul>
<p>You guessed it: we need to change the third parameter from <code>()</code> to <code>TimeUniform</code>:</p>
<pre><code>
let (shader, warnings) = Program::<Vertex, (), TimeUniform>::from_strings(None, SHADER_VS, None, SHADER_FS).unwrap();
</code></pre>
<p>And <em>that’s all</em>. Whenever you <code>shade</code> with a <code>ShaderGate</code>, the type of the shader object is being
inspected, and you’re handed with the uniform interface:</p>
<pre><code>
shd_gate.shade(&shader, |rdr_gate, uniforms| {
  uniforms.0.update(t);

  rdr_gate.render(None, true, |tess_gate| {
    let t = &triangle;
    tess_gate.render(t.into());
  });
});
</code></pre>
<p>Now, change your fragment shader to this:</p>
<pre><code>
in vec3 v_color;

out vec4 frag;

uniform float t;

void main() {
  frag = vec4(v_color * vec3(cos(t * .25), sin(t + 1.), cos(1.25 * t)), 1.);
}
</code></pre>
<p>And enjoy the result! Here’s the <a href="https://gist.github.com/phaazon/268d5c0285c6c7cba90d5cea1b99db76">gist</a>
that contains the whole <code>main.rs</code>.</p>


guid: https://strongly-typed-thoughts.net/blog/where-clause-super-traitSat, 02 May 2020 23:50:00 GMT
title: An interesting subtle property of where clauses with super traits
author: Dimitri Sabadie <dimitri.sabadie@gmail.com>
url: https://strongly-typed-thoughts.net/blog/where-clause-super-trait
feedurl: https://phaazon.net/blog/feed
content:
<p>Lately, I’ve been hacking on the next version of <a href="https://github.com/phaazon/luminance-rs">luminance</a>, luminance-0.40. It should be out
<em>“soon-ish”</em> but in the meantime, I’ve been struggling a bit with some highly and strongly typed
code. I want to share something interesting I discovered with <code>rustc</code>. Especially, I haven’t seen
a mention of that property <a href="https://doc.rust-lang.org/book/ch19-03-advanced-traits.html#using-supertraits-to-require-one-traits-functionality-within-another-trait">in the book</a>, so I’m happily sharing.</p>
<!-- vim-markdown-toc GFM -->
<ul>
<li><a href="#what-is-a-super-trait">What is a super trait?</a></li>
<li><a href="#the-weird-stuff">The weird stuff</a></li>
<li><a href="#a-small-disgression-to-haskell-land">A small disgression to Haskell land</a></li>
</ul>
<!-- vim-markdown-toc -->
<h1>What is a super trait?</h1>
<p>The weird situation is about <em>super traits</em>. A super trait is a trait that must be implemented
for another trait to be usable, because it’s relied on. Traits can be thought of as
<em>constraints</em>, so a super trait is a bit like a <em>dependency</em> when implementing a trait, and an
<em>implication</em> when using a trait on which a super trait is declared. A trait can have zero to
several super traits (added with the <code>+</code> operator). For instance, imagine you have a trait <code>Alive</code>:</p>
<pre><code class="language-rust">trait Alive {
  fn get_health(&amp;self) -&gt; Health;
}
</code></pre>
<p>Now imagine you want a trait to move someone or something around:</p>
<pre><code class="language-rust">trait Move: Alive {
  fn go(&amp;mut self, direction: Direction);
}
</code></pre>
<p>Here, you can see that:</p>
<ul>
<li><code>Move</code> requires to implement <code>Alive</code>, because it’s a super trait. It’s a <em>dependency</em>.</li>
<li>Because <code>Move</code> requires <code>Alive</code>, <code>Alive</code> is implied when you use <code>Move</code>. Indeed, that would be
redundant to annotate a type <code>T: Move + Alive</code>, because an instance (implementor) for <code>Move</code>
cannot exist without <code>Alive</code> to be implemented as well.</li>
</ul>
<p>So now that we understand what super traits are, let’s get to the weird stuff.</p>
<h1>The weird stuff</h1>
<p>When you implement a trait which has a super trait, do you think that:</p>
<ol>
<li>Your implementation is valid <em>when the super trait is implemented</em>? After all, you could simply
assume it is implemented. If it’s not, instances of your trait won’t be pickable.</li>
<li>Or your implementation <em>requires</em> the super trait to be implemented?</li>
</ol>
<p>The distinction is important. (1.) doesn’t require <code>rustc</code> to prove <em>when implementing a trait</em>
that the super trait is implemented. That will be required when using the trait. With (2.), <code>rustc</code>
will have to prove that the super trait is, first, implemented, before even considering the
implementation of the trait you’re making.</p>
<p>Rust currently uses (2.). If you <code>impl Move for Foo</code>, <code>impl Alive for Foo</code> must be in scope for
that implementor to be possible.</p>
<p>But… it would be interesting to actually have (1.). Imagine that you want to implement <code>Move</code> for
something a bit complex, like <code>Creature&lt;T&gt;</code>, but not all <code>Creature&lt;T&gt;</code> are alive. Only a subset
of them, and you can’t tell exactly when — i.e. you just cannot assume anything about <code>T</code>. So
what are you going to write?</p>
<pre><code class="language-rust">impl&lt;T&gt; Move for Creature&lt;T&gt; {
  fn go(&amp;mut self, direction: Direction) {
    // …
  }
}
</code></pre>
<p>This code will not compile, because you haven’t implemented <code>Alive for Creature&lt;T&gt;</code>. Remember that
the trait solver requires to prove super traits. However, and this is where all the interesting /
weird stuff happens:</p>
<pre><code class="language-rust">impl&lt;T&gt; Move for Creature&lt;T&gt; where Self: Alive {
  fn go(&amp;mut self, direction: Direction) {
    // …
  }
}
</code></pre>
<p>This compiles. It compiles because the <code>where</code> clause tells <code>rustc</code> that your implementor <em>will be
valid</em> if used with <code>Creature&lt;T&gt;: Alive</code>. The distinction is really subtle, but is, to my opinion,
very powerful. With the <code>where</code> clause, you state that <code>Move</code> is implemented for any <code>Creature&lt;T&gt;</code>
that is also <code>Alive</code>, but you don’t require them all to be <code>Alive</code>! You could implement <code>Alive</code>
for a bunch of creatures, like <code>Creature&lt;Vampire&gt;</code> and <code>Creature&lt;CloudDog&gt;</code>.</p>
<p>So, I remember having read somewhere (maybe in some Rust book, but I’m not quite sure) that the
<code>where Self: _</code> clause was not really useful, but in our case, you can see that it allows to
express a completely different semantics.</p>
<blockquote>
<p>You could also have used <code>Creature&lt;T&gt;: Alive</code> in place of <code>Self: Alive</code>, as here,
<code>Self = Creature&lt;T&gt;</code>.</p>
</blockquote>
<h1>A small disgression to Haskell land</h1>
<p>In Haskell, that code requires to use <code>UndecidableInstances</code>. I don’t know <em>exactly</em> why, but GHC
states that the constraint (<code>Alive (Creature a)</code>) is no smaller than the instance head
(<code>Move (Creature a)</code>), and this is not permitted, as being undecidable. Enabling the
<code>UndecidableInstances</code> GHC extension will make it possible to compile:</p>
<pre><code class="language-haskell">class Alive a where
  getHealth :: a -&gt; Health

-- The super class is declared on the left side of the =&gt; (the parenthesis are
-- optional, but I’m used to put them all the time, as they are required when you
-- have more constraints).
class (Alive a) =&gt; Move a where
  go :: Direction -&gt; a -&gt; a

-- This instance requires UndecidableInstances to compile.
instance (Alive (Creature a)) =&gt; Move (Creature a) where
  go = -- …

instance Alive (Creature Vampire) where
  getHealth = -- …

instance Alive (Creature CloudDog) where
  getHealth = -- …
</code></pre>
<p>I’m not quite sure why this <em>very exact</em> situation requires <code>UndecidableInstances</code> though. In this
case, it seems fine.</p>
<p>I hope you’ll have learned something with this small yet fun type theory candy. Keep the vibes!</p>


guid: https://strongly-typed-thoughts.net/blog/hid_and_msi_keyboardsSat, 20 Jun 2015 00:00:00 GMT
title: HID and MSI keyboards
author: Dimitri Sabadie <dimitri.sabadie@gmail.com>
url: https://strongly-typed-thoughts.net/blog/hid_and_msi_keyboards
feedurl: https://phaazon.net/blog/feed
content:
<h1>MSI keyboards</h1>
<p>I have a <a href="http://www.mobiletechreview.com/notebooks/MSI-GS60-2QE-Ghost-Pro.htm">MSI GS60 Ghost Pro 2QE</a>
I’m very proud of. It’s sexy and powerful. It comes with a fancy and
configurable backlit keyboard.</p>
<p><img src="http://www.ittipexplorer.info/wp-content/uploads/2014/04/wpid-msigs60ghostpro-8-580-90.jpg" alt="" /></p>
<p>There’s a tool called <em>SteelSeries Engine</em> for Windows we can use to change the
colors of the keyboard. It supports several features:</p>
<ul>
<li>changing colors of three parts of the keyboard (<em>left</em>, <em>middle</em> and <em>right</em>) ;</li>
<li>changing modes (<em>normal</em>, <em>breathe</em>, <em>wave</em>, <em>demo</em>, <em>gaming</em>).</li>
</ul>
<p>Unfortunately, that software doesn’t work on Linux, even with wine. I tried hard
to make it work and never actually found a way to run it. Then, I decided to
look for alternatives and… found nothing <em>working</em>.</p>
<p>Yesterday, I tried a <strong>node.js-powered</strong> tool called
<a href="https://github.com/wearefractal/msi-keyboard">msi-keyboard</a>. And it worked.
However, the interface and user interface was not my cup of tea. I decided to
dig in in order to understand how it works, and I decided to write my own tool
with a decent interface.</p>
<h2>HID access</h2>
<p>The key idea is that such keyboards are just
<a href="https://en.wikipedia.org/wiki/Human_interface_device">HID devices</a>. As a
<strong>Haskell</strong> programmer, I looked for something that would grant me access to
such devices, but nothing was working. There’s a
<a href="https://hackage.haskell.org/package/hidapi">hidapi</a> package, but it doesn’t
work and has bugs.</p>
<p>I didn’t give up though. I wrote my own <strong>Haskell HID API</strong> binding, called
<a href="https://hackage.haskell.org/package/hid">hid</a>. Several good things about it:</p>
<ul>
<li>it does work ;</li>
<li>it’s simple ;</li>
<li>I lately wrote a software using it.</li>
</ul>
<p>Feel free to install and use it!</p>
<h2>msi-kb-backlit</h2>
<p>Then, I wrote another <strong>Haskell</strong> package,
<a href="http://hackage.haskell.org/package/msi-kb-backlit">msi-kb-backlit</a>. It might
require <em>super user rights</em> to work. If you’re not a <strong>Haskeller</strong>, you can
find installation details <a href="https://github.com/phaazon/msi-kb-backlit">here</a>.</p>
<p><strong>Note: if you use Archlinux, you can directly download msi-kb-backlit through
the AUR! Search for msi-kb-backlit with yaourt, or download
<a href="https://aur4.archlinux.org/packages/msi-kb-backlit">the tarball</a>.</strong></p>
<p>The software has an embedded documentation to help you tweak with colors and
modes. ;)</p>
<p>Feel free to use all those pieces of software. I made them with love for you
all!</p>
<p>Enjoy your week end, and keep the vibe!</p>


guid: https://strongly-typed-thoughts.net/blog/editors-in-2021Mon, 22 Mar 2021 21:00:00 GMT
title: My thoughts about editors in (early) 2021
author: Dimitri Sabadie <dimitri.sabadie@gmail.com>
url: https://strongly-typed-thoughts.net/blog/editors-in-2021
feedurl: https://phaazon.net/blog/feed
content:
<p>For the past year, I have been moving back and forth between <a href="https://neovim.io">Neovim</a>, my main, daily editor, and <a href="https://github.com/hlissner/doom-emacs">Doom Emacs</a>, an
<a href="https://www.gnu.org/software/emacs">Emacs</a> configuration distribution focusing on bringing a super consistent, powerful and overall top-notch user
experience for <a href="https://www.gnu.org/software/emacs">Emacs</a>. This article is a direct follow-up to the
<a href="https://phaazon.net/blog/editors-in-2020">previous one</a> from fall 2020.</p>
<p>In this blog entry, I want to share something a bit different: instead of presenting my experiences with all those
editors, I want to take some hindsight and analyze a bit more the technology and what it seriously means in terms of
features and productivity. Obviously, I will reference other editors I have used a lot (I use <a href="https://www.jetbrains.com/idea">IntelliJ</a> a lot at work,
and I am used to <a href="https://code.visualstudio.com">VS Code</a> and <a href="https://atom.io">Atom</a> as well).</p>
<h1>Current status as of March 2021: I am lost</h1>
<p>Yeah, you read that title right. I am lost. I have been a Vim / Neovim user for something like 12 years now. I am used to
the ecosystem, I have installed hundreds of plugins in the past decade, tried so many things, discovered pearls that I
still use today (hello <a href="https://github.com/wellle/context.vim">context.vim</a>), I have been using a workflow based on <a href="https://github.com/tmux/tmux">tmux</a>, Neovim and my shell that has proven
very powerful and yet to be fully replaced. But — and I think it is important to make that note — I did not start coding
with Vim. Today, I am 29. I started coding when I was 11, using a tool called <a href="https://www.codeblocks.org">Code::Blocks</a>. Rapidly, I switched to
Emacs – by the time, Doom Emacs did not even exist, and spent several years using it. Later, when I was between 16
and 17, I switched to Vim — I guess the modal idea got me intrigued and I felt in love with it. In 2014, I was told that
Vim was getting forked for the reasons we all know and I decided to switch to the fork, called Neovim.</p>
<p>Since then, I have been sticking to Neovim but I do not partake in those “editor wars”. “VS Code sucks”, “how do I even
exit Vim?”, “Emacs is bloated!”. I have my personal opinions on the technologies (i.e. I dislike very much the front-end
ecosystem / electron, even though I think the front-end idea — i.e. running code in a browser — is an interesting idea;
I just think the current technology for doing so is really bad), so obviously I will not pick something such as Atom /
VS Code, but I cannot say they suck. I have tried them. I see why people like them and it would be very dishonest to say
those editors suck. They really do not. Same thing applies to the products by JetBrains. In the past year, I did not
understand why some people would pay money for an editor. I have to use IntelliJ at work and even though I would not pay
for an editor, I start to grasp why people would actually want to pay for one. Not all people have the patience and time
to tweak an editor to their taste, such as with Vim / Neovim / Emacs. For those people, editors such as VS Code,
IntelliJ, Atom and others are really just superior.</p>
<p>Just take <a href="https://microsoft.github.io/language-server-protocol">LSP</a>. Configuring LSP in VS Code is a click on an install button, depending on your language. That is all.
You get all the good defaults and behaviors and you can already start enjoying coding with LSP. Doing the same in an
editor like Neovim is a very different experience. I am not saying it is bad; <strong>it is different</strong>. If you are, like me,
someone who enjoys spending time tweaking their tools, then you should not care too much. I have a colleague at work who
gave me an argument that I find very interesting: he basically wants an editor he does not have to configure, so that we
knows how to use it everywhere, on any machine of any colleague. That is not a point I would make, but for someone who
cares about that topic, it is a solid argument in favor of tools such as IntelliJ or VS Code.</p>
<p>In the previous company I was working for, I worked with <a href="https://github.com/gagbo">@gagbo</a>, who is an active Doom Emacs contributor, and I was
surprised by the editor he was using on his laptop. He mentioned Doom and on the night home, I had a look at it, and I
was really blown away. Doom Emacs (Doom for short) was a revelation to me, because it is using emacs as a productivy
platform more than a simple editor. <a href="https://github.com/hlissner">@hlissner</a>, Doom’s author, and the Doom community, have been gathering the best of
emacs and vim into a single configuration distribution. The result is so interesting and so powerful that I have been
doing back and forth between Doom and Neovim. More on that below.</p>
<p>Lately, I have been more present in the Neovim community, contributing mostly via a plugin that I have been making:
<a href="https://github.com/phaazon/hop.nvim">hop.nvim</a>. That plugin is a complete rewrite from scratch of <a href="https://github.com/easymotion/vim-easymotion">EasyMotion</a> (I have not even looked at how they do it on
their side) I made by using the Neovim Lua API directly. Obviously, it provides a much better experience than
EasyMotion <strong>in Neovim</strong>, simply because it uses features such as <em>virtual text</em> and <em>extended marks</em> and does not
touch the buffer, allowing to benefit from Neovim-0.5 crunchy features, such as the native LSP client and treesitter
parsers. I also made a bunch of PRs to Neovim core, reading a bit on the C code, etc.</p>
<p>All that to say that… given the experience that I have with Neovim and Doom… I have no idea what to think about
“editing” anymore. Neovim is growing and lots of interesting ideas are landing / will land in the next release. I do not
really know whether people will stick around Hop, but I think it has found its place — at least, I spotted a missing
piece and provided it. I am not a fan of Lua but I have to admit that it attracts more people doing some awesome things.
Among the things that I think are going into the right direction:</p>
<ul>
<li>The native LSP client.</li>
<li>The native treesitter implementation.</li>
<li><a href="https://github.com/kyazdani42/nvim-tree.lua">Nvim Tree</a>, a file browser written in Lua. Similar to netrw but better and much (much) faster.</li>
<li><a href="https://github.com/nvim-telescope/telescope.nvim">Telescope</a>, a fuzzy finder, a bit similar to <a href="https://github.com/abo-abo/swiper#ivy">Ivy</a> / <a href="https://github.com/emacs-helm/helm#emacs-helm">Helm</a> from the emacs realm. I have issues with it right now
because I am used to <a href="https://github.com/junegunn/fzf.vim">fzf-vim</a> and all the current sorting / picking algorithms are not up to the user experience
of fzf to me. When I compare it to <a href="https://github.com/abo-abo/swiper#ivy">Ivy</a>, it is roughly the same: <a href="https://github.com/abo-abo/swiper#ivy">Ivy</a>’s algorithm is just better to me (it has the
same refinement search mechanism as fzf’s, and it’s much faster than Telescope, especially Swiper). I think it will
change in the near future as algorithms in Telescope are improved and support is added for (I hope!) fzf, but for
now, the bad performances of telescope, the choice of making it pretty before focusing on the algorithms first makes
it not really usable on some code bases. However, I do think it is going into the right direction and I will be
looking forward it in a few weeks / months, giving it time to mature a bit and get better defaults / algorithms.</li>
<li><a href="https://github.com/TimUntersberger/neogit">Neogit</a>, an implementation of the best Git client ever, <a href="https://magit.vc">magit</a>, also from the emacs realm. I think it will take them
months before it reaches the same maturity and power as magit, but I think it is cruising great.</li>
<li><a href="https://github.com/pwntester/octo.nvim">Octo</a>, a GitHub-based workflow plugin allowing to list, create, open, add, remove, etc. lots of GitHub objects
directly from within a Neovim buffer, streaming changes on buffer exit / save. Similar, again, to magit’s <a href="https://magit.vc/manual/forge.html">Forge</a>,
from the emacs world, even if Forge is a bit more powerful as it abstracts away the concept of “remote”. However, the
edit workflow of Octo seems a bit more stable right now, so kudos!</li>
<li>Probably a ton of other plugins I forgot to mention.</li>
</ul>
<h1>So what is wrong?</h1>
<p>My experience with Doom Emacs really changed the way I look at software, and more specifically at “productivity
tools.” See, my current workflow is based on tmux, Neovim and my shell, which is zsh (vanilla, I completely ditched
<a href="https://ohmyz.sh">Oh My Zsh</a> as I find it useless and I uninstalled <a href="https://starship.rs/">starship.rs</a> a few weeks ago because I realized I hated the package
manager version annotations on my prompt, that I actually never needed). For people wondering, we have spent so much
time trying to install cool things that we forgot how powerful raw and vanilla zsh is. This is my current prompt:</p>
<p><img src="https://phaazon.net/media/uploads/current_zsh.png" alt="" /></p>
<p>At work, I need a bit more information (i.e. Kubernetes (k8s) namespace and context / cluster I am currently connected
to and AWS), but as those information are not path-dependent, I put them in my tmux statusline instead.</p>
<p>The thing is… I just realized by using Doom Emacs that I do not need all of these anymore. Doom Emacs provides an
environment that is very similar to my tmux + neovim + shell combo. It provides even more, depending on the packages you
install. And the thing is: for each tool, emacs have more opportunities, because it is not limited by a terminal
implementation (i.e. cell-grid / line based application). So for instance, you can have a k8s / AWS / docker client
inside Emacs that is likely to be much much better than the one you will use in CLI (I honestly dislike <code>kubectl</code> a lot,
it has so many switches and grepping through them by piping with <code>rg</code> or using some switches such as <code>-l</code> feels weird).
You can easily see a buffer containing the list of pods / containers / whatever, that you should be able to work with
the tool you already use (i.e. <a href="https://github.com/abo-abo/swiper#ivy">Ivy</a> for instance), interact dynamically with (instead of having to type one line after
another to make effects). It is even more true when you consider the actual shell: you do not need zsh anymore when
using Emacs, because it has its <a href="https://www.gnu.org/software/emacs/manual/html_mono/eshell.html">eshell</a> implementation that is both super weird and super nice. You want to script
something for your shell? Just write ELisp. You do not have to use that horrible bash script language, and you will
benefit from the whole ELisp ecosystem. Completion for commands in eshell is basically the same as completing a regular
buffer, which is weird at first but actually makes a lot of sense. And you have dozens of other examples. Org-mode is
one of the best plugin of Emacs, allowing to take notes, make wikis, handle tasks and calendars, in a way that feels
very seamless with the rest of the Emacs ecosystem. I know some people will tell me to use another tool (I do, my
current tool for that is <a href="https://github.com/phaazon/toodoux">toodoux</a>, a tool I made, similar to <a href="https://taskwarrior.org/">task-warrior</a>). And of course, <a href="https://magit.vc">magit</a>, which is by far
my favorite Emacs plugin. As others say it, it is hard to explain why magit is so much better than the git CLI or
VS Code’s integration or basically any other editor. It is just made very well, with pretty much anything actionable
from the UI, with amazing workflows and even actions you do not have easily with the CLI version.</p>
<h1>What is an editor in 2021, anymore?</h1>
<p>Today, when I look at Doom, at Neovim, at what people are trying to do in Lua with Neovim, I feel like I need to stop
trying to implement / follow what others are doing and think a bit about whether we are doing the right thing. Neovim,
currently, is going towards a very exciting direction with all the effort regarding Lua, LSP, treesitter and community
plugins. But I think the problem is bigger than Neovim. And I think I know that because of how <strong>I can use emacs with
Doom</strong>. I think I come back to Doom so often because they understand well what it means to create a productivity
platform more than an editor. It does not mean the editor is any less good, <em>au contraire</em>. In Doom, there is a default
package that gets installed if you ask for the project module. That package is <a href="https://github.com/bbatsov/projectile">projectile</a>, a wonderful package that
provides with you project management from within emacs. If you use emacs in daemon mode, it means that you can get the
same workflow as with tmux, Neovim and a shell, but with a native implementation of this project / workspace workflow,
in a much much more powerful way (because it will obviously plug in things like <a href="https://github.com/abo-abo/swiper#ivy">Ivy</a>, allowing you to do fuzzy
searching of projects, etc.). The overall experience by having a consistent platform is something I had completely
ignored for years and now that Doom hit me, it hit me hard. I do not know <em>any other platform</em> like emacs, besides
composable environments such as tmux + Neovim + shell. Yes, you have VS Code / Atom / IntelliJ / etc. that will have
tons of plugins to get you terminals and other kinds of integrations, but for having used them, it is clearly not as
powerful as what Doom does — and it is most of the time based on electron; big no. To sum up, Doom is a bit my
VS Code: it provides this amazing productivity platform, but it is fully native, modal-centric and highly composable /
customizable.</p>
<p>Composability is really important to me, but I have to face reality: there is <em>no way</em> to make a workflow such as tmux +
Neovim as snappy and fast as <a href="https://github.com/bbatsov/projectile">projectile</a>. <a href="https://github.com/tmux/tmux">tmux</a> basically reimplements a terminal inside a terminal, which has a lot
of implications (copy-pasting with the mouse in tmux is a challenge even Chuck Norris does not want to take). And once
you turn the computer off, tmux loses everything. You have plugins, such as <a href="https://github.com/tmux-plugins/tmux-resurrect">tmux-resurrect</a>, that will allow to save
and restore the tmux layout (sessions / windows / panes), but you will obviously lose the contents of each panes. With
<a href="https://github.com/bbatsov/projectile">projectile</a>, because it is a native emacs thing, you will not lose anything.</p>
<p>This whole idea has been obsessing me for months. The concept of a “productivity platform.” People laugh at emacs as
being bloated but… it basically implements their shell, tmux and editor in a more elegant way — and more. Why would one
laugh at that? Especially when it allows more interesting kinds of interaction.</p>
<p>So I have been wondering: what is a productivity platform, today, in 2021? Is it this workflow I have been using since
I am 16, with tmux and Vim / Neovim? A friend on IRC told me lately that if I have been using this workflow (as others
do) for so long… it has proven being good so why would I want to replace it? Even though this is a valid point, I do
think this is missing the point: ask some workers to dig a hole for planting trees, they will start digging the ground
with their bare hands. Later, give them a shovel and they will accept it as a better tool and will enjoy it. But does it
mean it is the best tool for planting trees? What if a better tool existed? This is basically the reason why I never
accept something as an ultimate solution: we can always enhance our tools. And having seen what Doom Emacs do, I do
think that I have found new ideas. Not necessarily a better tool, but at least, I think Neovim needs to evolve.</p>
<p>When I think about Neovim, I try to focus on what makes it this tool I love: modal centric, snappy / performance,
integrates well with tmux, and that is pretty much all. The integration part with tmux is needed because I have to use
the TUI, which is also something that I start not liking anymore. I know since Doom that we can get rid of TUIs and make
great GUIs. So what I think I want to go to with Neovim from now on is this:</p>
<ul>
<li>Contribute to a GUI or even make my own. I have glanced at all the current solutions and none are really exciting to
me. People have basically been making window decorations around TUI buffers, which I truly do not like. Some
exceptions that I might invest time in:
<ul>
<li><a href="https://github.com/akiyosi/goneovim">goneovim</a>, which seems to have the more advanced “GUI” externalized features.</li>
<li><a href="https://github.com/smolck/uivonim">uivonim</a>, but I will not contribute nor use it because, heck, electron.</li>
</ul>
</li>
<li>Contribute to <a href="https://github.com/mjlbach/neovim-ui">neovim-ui</a> and see where I can help with abstractions.</li>
</ul>
<p>Something around one year ago, I chatted with core contributors, telling them that something that puzzles me a bit with
the current Neovim situation is that there is no unified / consistent UI experience. Most plugins will write their own
UI thing, obviously often based on TUI hacks, such as using characters as <code>─</code>, <code>╮</code>, <code>├</code>, etc. The current situation with
<a href="https://github.com/nvim-lua/plenary.nvim">plenary.nvim</a> is another interesting topic, because I think it is not the
answer to my problem. Yes, it will provide an abstraction for people needing to create bordered windows / pop-ups but…
the GUI really does not care about that. Technically, Neovim already makes the distinction between “buffers” and
“pop-ups” and “floats”, so GUIs are free to render them the way they want, but… what if developers start using APIs,
such as plenary, where they pass border arguments? How does that make sense? If you are writing a plugin that needs to
list things and requires a fuzzy interaction with the user, you should not depend on something that expects anything
from the UI implementators. And for this, I think Neovim is lacking abstractions. Obviously, we can discuss that kind of
problems for other topics:</p>
<ul>
<li>Displaying messages.</li>
<li>Displaying notifications (I wish Neovim used the D-Bus protocol or similar on Linux for instance).</li>
<li>Fully customizable sign columns and “franges.”</li>
<li>Etc. etc.</li>
</ul>
<p>I guess some of those items are already addressed, but I have yet to see a Neovim GUI going fully “native without TUI
aspects.” Maybe it already exists and I need to search again. I want to invest time in things like goneovim and some
other implementations, and I really do want to make my own (maybe with Gtk or even using my own 2D/3D graphics Rust
stack).</p>
<h1>Next obsession: CLI, TUI and GUI</h1>
<blockquote>
<p>CLI stands for Command Line Interface; TUI for Terminal User Interface and GUI for Graphics User Interface.</p>
</blockquote>
<p>Among all the things that Doom changed in my mind, there is this concept of TUI. I have been a very big user (power
user?) of the CLI. Pretty much anything that I do on a computer is via the CLI. For more advanced usage, I tend to use
a TUI. I only use GUIs if I have to, such as <a href="https://www.mozilla.org">firefox</a>, <a href="https://www.blender.org">blender</a> or a video game. However, guess what: Doom hit hard
here too, because I am not using the TUI version: I am using the GUI one. And do not get it twisted, I am not using my
mouse, and this is a very important and interesting point about CLI / TUI / GUI.</p>
<p>When talking about this CLI / TUI vs. GUI thing around me, people tend to agree, weirdly, on some misconceptions:</p>
<ul>
<li>CLIs and TUIs, because they are mostly implemented as terminal applications, would not rely (for most) on the mouse and
then implement super good user experiences (UX): good shortcuts, good navigation through the application, easy
completion, great composability, etc. The UI, on the other side, would be rather poorer, because of the limitations of
the terminals: no images, only icons, which most of the time require patched fonts to render icons; weird hacks based
on unicode characters to mimic UIs like borders, splits, buttons, inputs, etc.</li>
<li>GUIs would rely more on the mouse and then focus less on the UX, making them harder to use for people on the keyboard,
with shortcuts most of the time not that good, preventing fluid and seamless navigation on the keyboard; poor
composability with other applications, etc. However, they would have much better UI, because of vector graphics images,
proper UI elements such as real inputs, buttons, border, splits, different font families / sizes, etc.</li>
</ul>
<p>These misconceptions are interesting because, as misconceptions, they are not real. You can find some terrible CLIs
(i.e. <code>npm</code> is such a bad CLI — I had a big issue at work a few years ago when running not on purpose a command such
as <code>npm i --user</code>, thinking it would install it in user space while it just completely ignored the unknown <code>--user</code>
flag and installed the package globally), and you can find terrible GUIs too. However, and this is where I want to
focus, GUIs are not necessarily mouse-focused, and since we can implement nice UX in TUI… we should be able to do the
same in GUIs, right?</p>
<p>Aaaaaaaand… enter Doom, again. I am using the GUI version of emacs — on Archlinux, I am using this
<a href="https://aur.archlinux.org/packages/emacs-native-comp-git">patched version</a> which is a basically <a href="https://www.emacswiki.org/emacs/GccEmacs">gccemacs</a>, an emacs
that recompiles all the Lisp as C! It obviously can accept mouse inputs so that you can click on things if you want to
but… the most important still applies: you can use it exactly as you would use it in a terminal. However, because it is
a GUI, you get access to much more possibilities:</p>
<ul>
<li>Proper “pixel scrolling.”</li>
<li>Use several fonts, of different sizes, for things like Markdowns, note taking, etc.</li>
<li>Proper pop-up windows.</li>
<li>Pixel-perfect annotations, marks, hints and signs in your buffer.</li>
<li>Images, rasterized or vector graphics.</li>
<li>One of the most important part to me: proper UI elements! No more hacks!</li>
<li>Even though it is not implemented yet, since we can have images, and especially vector graphics, we should be able to
render git branches with pretty links and not the <code>/|\*</code> ASCII art mimics we see in all CLIs / TUIs.</li>
<li>Etc. etc.</li>
</ul>
<p>Today, I do think that the right direction is to allow developers to build platforms around the concept of “graphics
toolkits”, and not “terminal applications.” Even for CLIs. Think about <code>git branch</code>. My current git branch workflow, in
CLI, does not use <code>git branch</code> only: I have an alias for each Git commands that require a branch. For instance, if I want
to switch to another branch, this is what my <code>gs</code> alias looks like when invoked:</p>
<p><img src="https://phaazon.net/media/uploads/git_switch.png" alt="" /></p>
<p>This is basically a fuzzy-finder version of <code>git switch</code>, using <a href="https://github.com/junegunn/fzf">fzf</a> – if you want that script, have a look at
<a href="https://github.com/phaazon/config/tree/master/git/scripts">these ones</a>. The same feature is doable with Doom / Neovim
by using the right plugin, and I have to admit that I prefer doing that in magit as it feels more “well integrated.” Of
course, you will always want to have a way to compose applications, and for that, CLIs are great. Something such as
<code>kubectl get pods -l service=my-service | rg whatever</code>. However, I am trying to challenge my own conception of
“program composability” because I do think we can implement this <em>list k8s pods -&gt; filter them</em> inside an UI such as the
ones in Doom Emacs, without having to write a specific tool. And we can actually do it in CLI / TUI as I just showed
with my <code>gs</code> alias, so… why not providing that kind of power natively on the platform?</p>
<h1>Final obsession: configuration ≠ scripting</h1>
<p>And now one of the big issue I have with pretty much any editor today (but more with Neovim as I am actively
contributing to it): if you want to configure your editor, you have to code your configuration <strong>and I highly dislike
that.</strong> It is possible, indeed, to reduce the quantity of code to write, but you will eventually have to get your hands
dirty and write some Lua (<a href="https://phaazon.net/media/uploads/eeeeeeeeeeeeeeeeeeeeeeeew.jpg">eeew</a>). And I have a big issue with this.</p>
<p>See, configuration should be stupid and limiting. It should be <code>key = value</code>, period. If you need a more structured way
to configure things, just add sections / objects like what you can do with TOML or JSON, but that is all. Remember that
a configuration file might be read by a human, and that you <strong>must</strong> do everything possible to make it super clear what
the configuration state is. Configuring should be:</p>
<ul>
<li>Not Turing-complete. You should not have access to logical constructs, such as conditional statements / jump
statements or anything that would turn your configuration language Turing-complete.</li>
<li>Imports / includes should not be possible. Having to import things from a configuration file makes it harder for the
application to read from it and it makes it much harder for people to know what values are currently set.</li>
<li>If you really want to have “scripting” over your configuration files, use some proper languages made exactly for this
purpose, such as <a href="https://dhall-lang.org">Dhall</a>, a template engine or something similar. Leave the choice to the user to decide whether they
want to configure, or code.</li>
</ul>
<p>There is a fundamental difference between <strong>having the power to do something and explicitly refusing that power</strong> and
<strong>lacking that power.</strong> I think that configuration <em>can</em> be done via code, but it does not mean it should. The reason
for that is mainly for cognitive complexity reasons. If you are the author of your application, you will very likely
know how to script it. You will know all the quirks and all the little things to know that you will, obviously, have
well documented (right?!). Your users will not have this kind of knowledge. I think it is wrong to use Lua to configure
a software as it will force people to get into the documentation of the Lua API, while all they should have is a list
of configuration options they can set, what setting does what and what are the possible values, <strong>and that is all.</strong>
Providing more power leads to inconsistencies, broken setup and something that is pretty hard to fix without proper
semantic versioning (and an actual package manager enforcing it): a slowly rotting configuration.</p>
<p>Rotting configuration happens when you start piling up lots of “configuration code”, using different APIs, that will
eventually get deprecated or not even used anymore. If you are lucky, the application will just tell you this and that
API calls are deprecated and you should switch to something else. If you are not, you will get some random crashes and
failures in your applications and you will have to debug code. For configuration. And this happens a lot with Neovim.</p>
<p>See, blurring the line between configuration and scripting is easy, especially if you use the same language for both.
Scripting is enhancing the power of an application, such as an editor or a video game, by providing new <strong>behaviors</strong>.
Those behaviors (should them be native to the applications or new ones provided by plugins) are <strong>configured</strong>. If you
start having to script the configuration side, you lose the distinction between both.</p>
<p>At work, we are using k8s a lot. In order to configure our services, we have to deploy configuration massively
everywhere, and we need to generate that configuration. But see, there is a difference between “the configuration” and
“generating the configuration.” The “generate part” is basically a template language. This is code, because it relies on
the datacenter / cluster / environment / feature flags / etc. we want to deploy in / with. However, we do not configure
the software in the template code. We pass fragments of configuration to the template engine, which generates the final
configuration. And those fragments are basically exactly what I described above: key-value objects, sometimes nested to
implement more complex configuration setup, but nothing else. It is basically YAML. If you do not need the complexity of
the datacenter, cluster or environment, you can just run your application with a configuration file that has no code in
it, just a bunch of key-value pairs.</p>
<p>The way Neovim treats configuration right now is via a convention, well-accepted among the plugin developers (and
since I am now officially a Neovim plugin developer with <a href="https://github.com/phaazon/hop.nvim">hop.nvim</a>, I obviously follow it as well). That convention is
the <code>setup</code> function plugins expose to pass configuration (via an object often called <code>opts</code> for <em>options</em>). So people
have to know whether that convention is used, they have to call that function (i.e. it is a regular Lua function call)
and they have to dig in the Lua API. What I think I want to try (I think I am going to work on a plugin for that) is a
way to have a central “configuration store” so that users do not have to go through this <code>setup</code> thing anymore — which,
for the record, requires you to do something like <code>require'the-plugin'.setup { … }</code>, for every plugin you want to use.
The way I see things would be more like this:</p>
<pre><code class="language-lua">-- In your init.lua, for instance

require'config-store'.setup {
  telescope = {
    -- options used by telescope
  },
  hop = {
    -- options used by hop
  },
  neogit = {
    -- options used by neogit
  },
  -- etc. etc.
}

require'telescope'
require'hop'
require'neogit'
</code></pre>
<p>Then, each plugin, in their <code>init.lua</code>, could do something like this:</p>
<pre><code class="language-lua">-- Assuming this is the plugin for hop
local config = require'config-store'.hop or require'hop.config'.defaults
</code></pre>
<p>This way of doing seems much more natural to me, as it limits the amount of “coding” people have to do to configure
their packages, and it is not really different for package maintainers. Instead of exposing a <code>setup</code> function, they can
just get the configuration via this <code>config-store</code> thing. I think I will try to experiment with this idea. Obviously, it
requires plugins to adopt this <code>config-store</code> thing, and I do think that if people think it is an interesting idea, this
should be a vanilla Lua API and not a random plugin, so that we know this <code>config-store</code> object is there, whatever
plugins the user have installed. As discussed above with how we do that at work, I do not want to prevent people from
coding / scripting their editors. But I think a clear distinction between scripting and configuring is important.</p>
<h1>In the end</h1>
<p>So those were my (lost) thoughts about the current situation. I might do a follow-up later in the year to see how things
have evolved to me. Today, I stick around Neovim in tmux, using various scripts around fzf to make my shell a bit more
interesting to use. I have to admit that I often open emacs to do three / four commits. The experience is just
consistent. Navigating in projectile, Ivy and magit is unbeaten to me. The simple interactions with dired (to navigate
directories a bit like with a file browser) and edit their contents is something I truly seek everywhere else, even in
my shell. I recently did a quick benchmarked of <a href="https://github.com/abo-abo/avy">Avy</a>, the emacs implementation of
EasyMotion, versus <a href="https://github.com/phaazon/hop.nvim">hop.nvim</a>, and Hop is <em>really much faster</em>, so I do not really know what to think (I guess I should
be proud of what I made 🙂). I think that Doom currently has this level of maturity that Neovim will take years to
achieve, not because of talent, but because of the fact Neovim is an editor and has not been designed as being a
productivity platform, while emacs was.</p>
<p>Keep the vibes!</p>


guid: https://strongly-typed-thoughts.net/blog/spectra_pluginsMon, 19 Feb 2018 02:57:00 GMT
title: Spectra gets Rust scripting!
author: Dimitri Sabadie <dimitri.sabadie@gmail.com>
url: https://strongly-typed-thoughts.net/blog/spectra_plugins
feedurl: https://phaazon.net/blog/feed
content:
<h1>Spectra gets Rust scripting!</h1>
<p><a href="https://crates.io/crates/spectra">spectra</a> is a crate I’ve been maintaining for a few months / years now. It’s a crate that I mainly
use for demoscene productions (I released two with it, <a href="https://www.youtube.com/watch?v=pYqZS1C_7PU">Céleri Rémoulade</a> and <a href="https://www.youtube.com/watch?v=OemyLQbDTSk">Outline Invitation</a>)
but I also use it to play around and experiment new rendering, animation and video game techniques.</p>
<p>The crate features a few things that I enjoy daily. Among them:</p>
<ul>
<li>A rendering layer, backed by another crate of mine, <a href="https://crates.io/crates/luminance">luminance</a>.</li>
<li>Hot reloading for scarce resources – the feature was extracted into <a href="https://crates.io/crates/luminance">warmy</a> as folks on IRC asked
me to!</li>
<li>Models (.OBJ) loaders.</li>
<li>Audio simple routines (basically: play a soundtrack and get metadata about it).</li>
<li>A shading language of mine, built upon GLSL (I’ll make a separate blog entry about this one).</li>
<li>Splines, animation primitives, etc.</li>
<li>And more.</li>
</ul>
<p>The last thing I’ve been working on is <em>being productive</em>. That might seem counter-intuitive, but
when you start building <em>“frameworks”</em> and <em>“engines”</em>, you actually end up writing a lot of code
for <em>“the beauty and power of it”</em> and lose focus on what’s important: releasing things. I know
that and looked for what I could enhance to augment my productivity.</p>
<p>Among interesting topics I came up with, I stated:</p>
<ul>
<li>JSON live editing with <a href="https://crates.io/crates/luminance">warmy</a> in <a href="https://crates.io/crates/spectra">spectra</a> is totally awesome (it was an improvement for my
productivity when I was, before, editing the curves in the code – you know, the edit-compile-run
cycle?). However, even if it’s awesome, it’s quickly limiting, especially when you want to edit
quaternions (rotations) or find a specific color mask. So I’d like to implement a way to pick
objects around with the mouse and edit them via a nice UI.</li>
<li>Scripting. Somewhat. I partially done that with hot-reloaded shaders!</li>
<li>Better tooling, especially for live-coding.</li>
</ul>
<p>Clearly, there’s something about live coding.</p>
<h1>Introducing scripting in Rust</h1>
<p>I went through several thought processes. I first had a look at <a href="https://www.lua.org">Lua</a>, because lots of people think
it’s cool. However, I don’t like the syntax nor the overall safety the language
<strike>gives</strike> <em>doesn’t</em> give you.</p>
<p>I remembered that almost a decade ago, when I was 15 or 16 year old, I implemented some kind of a
<em>plugins</em> system in C++ for a side project of mine. The idea was simple:</p>
<ul>
<li>Find functions with <code>dlopen</code>, <code>dlsym</code> to open a relocatable object / archive (.so / .dll).</li>
<li>Use them at runtime.</li>
</ul>
<p>This is not magic. When you link a crate / library, by default, a lot of compilers will perform
<em>static linking</em>. That’s the case of <a href="https://www.haskell.org">ghc</a> or <a href="https://www.rust-lang.org">rustc</a>, in most situations.</p>
<blockquote>
<p>If you don’t know yet, a library / crate / dependency statically-linked in a program means that
all of its symbols (the ones used in the program at least) are placed in a specific section of
the generated executable, so that those symbols have a proper address and “come with the binary”.</p>
</blockquote>
<p>Dynamic linking, on the other side, is a way to express a <em>dependency</em> between your binary and some
code, which is most of the time living in a relocatable archive (<code>.so</code> on Linux or macOSX, <code>.dll</code> on
Windows – macOSX also uses <code>.framework</code> but it’s just for the overall idea). When your binary needs
to call a function defined in a dynamically linked library, it’ll open the shared library with the
<code>dlopen</code> function, try to locate the function by giving its (unmangled) name to <code>dlsym</code> or <code>dladdr</code>
for instance, if the function exists, you’ll be able to run its code.</p>
<p>There exists attacks and interesting things you can do with dynamic libraries. Because the code
doesn’t live in the binary, you can for instance replace a legit and safe <code>.dll</code> on Windows with a
malicious one. Or you can patch a buggy dynamic library by shipping a new <code>.dll</code> / <code>.so</code> without
having its dependent to re-compile their applications (which would be needed in case of static
linking). Another funny thing you can do: <a href="http://www.fraps.com/">fraps</a>, a real-time recorder for your video games,
performs some kind of <code>.dll</code> injection by pushing an OpenGL <code>.dll</code> into your binary (Windows allows
this) and replacing some known functions. For instance, the function responsible in swapping your
renderbuffer chains. It can then intercepts pixels regions, adds overlay, etc. Fascinating! ;)</p>
<p>Anyway, the idea is that whenever you link a dynamic library, your compiler / linker will just
insert the required code in your binary (think of <code>dlopen</code>) so that your binary can load code at
runtime. It’s then pretty easy to implement a plugin system:</p>
<ul>
<li>Define your interface. Define a function type you will need to take from the dynamic library and
give that interface a name so that you can look it up in the library.</li>
<li>Open the library, find the symbol, and just use it!</li>
</ul>
<p>So you could imagine, as a very simple start example, a <code>loop</code> that would simply invokes such a
function. Whenever you change the library, the code getting ran will automatically changes as well!</p>
<h2>Scripting in spectra</h2>
<p>So I decided to reproduce that in Rust using the few crates of mine:</p>
<ul>
<li><code>spectra</code>: obviously, since it’s the crate that will receive the feature first.</li>
<li><code>warmy</code>: it’s indirect since <code>spectra</code> re-exports it and adds a few things over it, like a
<em>resource</em> system (error messages, timing loading and reloading, etc.).</li>
<li><a href="https://crates.io/crates/libloading"><code>libloading</code></a>: the <em>Rust way</em> to go with dynamic libraries (it has an <code>unsafe</code> interface though).</li>
<li>That’s pretty much everything of it.</li>
</ul>
<blockquote>
<p>For unix plateforms, you can see that <code>libloading</code> is just a smart wrapper over the functions I
mentionned. <a href="https://github.com/nagisa/rust_libloading/blob/master/src/os/unix/mod.rs#L281">See for yourself</a>.</p>
</blockquote>
<p>Ok, so, let’s try the experiment!</p>
<h1>Let’s write plugins in Rust!</h1>
<h2>Foreword</h2>
<p>The first thing we must accomplish is very simple: we want to be able to load some (Rust) code at
runtime, inside our application. For achieving that goal, we need to load a dynamic library (<code>.so</code>
on Linux) with <a href="https://docs.rs/libloading/0.5.0/libloading/struct.Library.html#method.new"><code>libloading::Libray::open</code></a>.
Once we have the library, we can just look a symbol up with
<a href="https://docs.rs/libloading/0.5.0/libloading/struct.Library.html#method.get"><code>libloading::Library::get</code></a>.
In case of a successful lookup, that function returns a value of type <code>Symbol</code>, which implements
<code>Deref</code> for the symbol you’re asking.</p>
<blockquote>
<p>Dynamic library typically gathers functions, so we’ll be looking for <code>fn</code> symbols.</p>
</blockquote>
<p>However, we don’t have any dynamic library yet. We only have… a Rust file – <code>.rs</code>. How can we get
a dynamic library out of it?</p>
<h2>Generating a dynamic library</h2>
<p>This is actually pretty simple. We’re gonna start easily by making a <code>.so</code> that will just contain
a function called <code>hello_world</code> that will just print <code>&quot;Hello, world!&quot;</code> on <code>stdout</code>. You all know how
to implement such a function:</p>
<pre><code>/// hello_world.rs
pub fn hello_world() {
  println!(&quot;Hello, world!&quot;);
}
</code></pre>
<blockquote>
<p>We make the function <code>pub</code> so that it gets actually exported when we generate the dynamic library.</p>
</blockquote>
<p>Copy that code and put it in, for instance, <code>/tmp</code>.</p>
<p>Then, let’s generate a dynamic library!</p>
<pre><code>cd /tmp
rustc --crate-type dylib hello_world.rs
</code></pre>
<p>Once <code>rustc</code> returned, you should see a new file in <code>/tmp</code>: <code>/tmp/libhello_world.so</code>! Here we are!
We have a dynamic library! Let’s try to find our function in it with the handy <code>nm</code> program.</p>
<blockquote>
<p>I reckon <code>nm</code> is already installed on your machine. If not, it should come with packages like
<code>base-devel</code>, <code>build-essentials</code> or that kind of meta packages.</p>
</blockquote>
<pre><code>nm /tmp/libhello_world.so | grep hello_world
0000000000000000 N rust_metadata_hello_world_8787f43e282added376259c1adb08b80
0000000000040ba0 T _ZN11hello_world11hello_world17h49fe1e199729658eE
</code></pre>
<p>Urgh. We have a problem. Rust has mangling for its symbol names. It means that it will alter the
symbols so that it can recognize them in a dynamic library. For instance, the mangle version of a
function name <code>foo</code> defined for a type <code>A</code> won’t be the same as the one of <code>foo</code> defined for a type
<code>B</code>. However, in our case, we don’t want mangling, because, well, we won’t be able to lookup the
name up – no one will even try to guess that <code>_ZN11hello_world11hello_world17h49fe1e199729658eE</code>
function name.</p>
<p><code>rustc</code> has a very simple solution to that: just tell it you don’t want a symbol’s name to be
mangled. It’ll be stored in the dynamic library the way you write it. This is done with the
<code>#[no_mangle]</code> attribute.</p>
<pre><code>/// hello_world.rs
#[no_mangle]
pub fn hello_world() {
  println!(&quot;Hello, world!&quot;);
}
</code></pre>
<p>Now recompile with the same <code>rustc</code> line, and run the <code>nm</code> + <code>grep</code> oneliner again.</p>
<pre><code>nm /tmp/libhello_world.so | grep hello_world
0000000000040b70 T hello_world
0000000000000000 N rust_metadata_hello_world_8787f43e282added376259c1adb08b80
</code></pre>
<p>Now you can see there exists a symbol called <code>hello_world</code>: this is our symbol!</p>
<h2>Load the library in Rust and read a symbol</h2>
<p>For our little example here, we’re just gonna load and run the code in a new project.</p>
<pre><code>cd /tmp
cargo new --bin dyn-hello-world
     Created binary (application) `dyn-hello-world` project
cd dyn-hello-world
</code></pre>
<p>Edit the <code>Cargo.toml</code> file to include <code>libloading = &quot;0.5&quot;</code> in the <code>[dependencies]</code> section. Ok,
we’re good to go. Run a second terminal in which you run this command to automatically check
whether your code is okay:</p>
<pre><code>cd /tmp/dyn-hello-world
cargo watch
</code></pre>
<p>Let’s edit our <code>main.rs</code> file.</p>
<pre><code>extern crate libloading;

use libloading::Library;

fn main() {
  let lib = Library::new(&quot;/tmp/libhello_world.so&quot;).unwrap();
  let symbol = unsafe { lib.get::&lt;extern fn ()&gt;(b&quot;hello_world&quot;).unwrap() };

  symbol();
}
</code></pre>
<p>It should check. A bit of explanation:</p>
<ul>
<li>The <code>Library::new</code> accepts as first and only argument the path to the dynamic library. In our
case, we just use our freshly generated <code>libhello_world.so</code>.</li>
<li>The <code>unsafe</code> <code>get</code> call takes the symbol we look for as argument and returns it if it’s found it.</li>
<li>Because <code>Symbol</code> implements <code>Deref</code>, here, we can directly call the function with <code>symbol()</code>.</li>
</ul>
<p>Now compile and run the code:</p>
<pre><code>cargo build
   Compiling cc v1.0.4
   Compiling libloading v0.5.0
   Compiling dyn-hello-world v0.1.0 (file:///tmp/dyn-hello-world)
    Finished dev [unoptimized + debuginfo] target(s) in 2.63 secs
cargo run
    Finished dev [unoptimized + debuginfo] target(s) in 0.0 secs
     Running `target/debug/dyn-hello-world`
Hello, world!
</code></pre>
<p>Hurray! We’ve just built our first dynamic library loader!</p>
<h2>Let’s compile Rust… from Rust!</h2>
<p>Ok, now, let’s iterate: we need to compile the Rust code from our Rust code (haha). For doing that,
we’ll need to generate the dynamic library at some place. Because I don’t like to put artifacts in
random places, I like to use the <code>tempdir</code> crate, which gives you a <code>TempDir</code> that creates a new
temporary directory with a random name when you ask for it, and removes it from your filesystem when
the <code>TempDir</code> goes out of scope. We’ll also be using the <code>std::process</code> module to run <code>rustc</code>.</p>
<p>Add the following to your <code>[dependencies]</code> section in your <code>Cargo.toml</code>:</p>
<pre><code>tempdir = &quot;0.3&quot;
</code></pre>
<p>Ok, let’s compile a Rust source file into a dynamic library from our Rust source code!</p>
<pre><code>extern crate libloading;
extern crate tempdir;

use libloading::Library;
use std::process::Command;
use std::str::from_utf8_unchecked;
use tempdir::TempDir;

fn main() {
  let dir = TempDir::new(&quot;&quot;).unwrap(); // we’ll drop the .so here
  let target_path = dir.path().join(&quot;libhello_world.so&quot;);
  
  let compilation =
    Command::new(&quot;rustc&quot;)
    .arg(&quot;--crate-type&quot;)
    .arg(&quot;dylib&quot;)
    .arg(&quot;/tmp/hello_world.rs&quot;)
    .arg(&quot;-o&quot;)
    .arg(&amp;target_path)
    .output()
    .unwrap();

  if compilation.status.success() {
    let lib = Library::new(&amp;target_path).unwrap();
    let symbol = unsafe { lib.get::&lt;extern fn ()&gt;(b&quot;hello_world&quot;).unwrap() };

    symbol();
  } else {
    let stderr = unsafe { from_utf8_unchecked(compilation.stderr.as_slice()) };
    eprintln!(&quot;cannot compile {}&quot;, stderr);
  }
}
</code></pre>
<p>Compile, run, and…</p>
<pre><code>cargo run
   Compiling dyn-hello-world v0.1.0 (file:///tmp/dyn-hello-world)
    Finished dev [unoptimized + debuginfo] target(s) in 0.58 secs
     Running `target/debug/dyn-hello-world`
Hello, world!
</code></pre>
<p>This piece of code is the first premise of our plugin system. You can see interesting properties:</p>
<ul>
<li>You don’t have to re-compile <code>dyn-hello-world</code> to change its behavior, since it comes from a
dynamic library.</li>
<li>There’s no need to depend on a dynamic library <em>directly</em>, since we can just generate it on the
fly!</li>
<li>You can display error messages (the current code is a bit perfunctory but you could enhance to
have nice error messages!).</li>
</ul>
<p>However, there’s a problem. Try adding an <code>extern crate</code> to <code>hello_world.rs</code>, like, for instance:</p>
<pre><code>extern crate spectra;

#[no_mangle]
pub fn hello_world() {
  println!(&quot;Hello, world!&quot;);
}
</code></pre>
<p>Run <code>dyn-hello-world</code>.</p>
<pre><code>cargo run
    Finished dev [unoptimized + debuginfo] target(s) in 0.0 secs
     Running `target/debug/dyn-hello-world`
cannot compile error[E0463]: can't find crate for `spectra`
 --&gt; /tmp/hello_world.rs:1:1
  |
1 | extern crate spectra;
  | ^^^^^^^^^^^^^^^^^^^^^ can't find crate

error: aborting due to previous error
</code></pre>
<p>Damn, how are we gonna solve this?</p>
<h2>Finding dependencies!</h2>
<p>The key is to understand how <code>cargo</code> deals with the dependencies you declare in your <code>Cargo.toml</code>’s
<code>[dependencies]</code> section. To do this, clean your project, and recompile in <em>verbose</em> mode – we’re
like… <em>reverse engineering</em> <code>cargo build</code>!</p>
<pre><code>cargo clean -p dyn-hello-world
cargo build --verbose
       Fresh libc v0.2.36
       Fresh winapi-build v0.1.1
       Fresh winapi v0.2.8
       Fresh cc v1.0.4
       Fresh rand v0.4.2
       Fresh kernel32-sys v0.2.2
       Fresh libloading v0.5.0
       Fresh remove_dir_all v0.3.0
       Fresh tempdir v0.3.6
   Compiling dyn-hello-world v0.1.0 (file:///tmp/dyn-hello-world)
     Running `rustc --crate-name dyn_hello_world src/main.rs --crate-type bin --emit=dep-info,link -C debuginfo=2 -C metadata=e8b1e5e96f709abe -C extra-filename=-e8b1e5e96f709abe --out-dir /tmp/dyn-hello-world/target/debug/deps -C incremental=/tmp/dyn-hello-world/target/debug/incremental -L dependency=/tmp/dyn-hello-world/target/debug/deps --extern tempdir=/tmp/dyn-hello-world/target/debug/deps/libtempdir-9929bcad6dc8cc47.rlib --extern libloading=/tmp/dyn-hello-world/target/debug/deps/liblibloading-a2854ce154eb4d6f.rlib -L native=/tmp/dyn-hello-world/target/debug/build/libloading-4553b9f132aa813a/out`
    Finished dev [unoptimized + debuginfo] target(s) in 0.50 secs
</code></pre>
<p>We can see a few things going on here. First, there are some <code>Fresh</code> lines we’re not interested
about. Then <code>cargo</code> tries to compile our application. You can see an invokation to <code>rustc</code> with a
long list of arguments. Among them, two interest us:</p>
<ul>
<li>The one specifying the list of dependencies your project needs:
<ul>
<li><code>-L dependency=/tmp/dyn-hello-world/target/debug/deps</code>.</li>
</ul>
</li>
<li>The ones specifying the list of crates you might use in <code>extern crate</code> statements.
<ul>
<li><code>--extern tempdir=/tmp/dyn-hello-world/target/debug/deps/libtempdir-9929bcad6dc8cc47.rlib</code></li>
<li><code>--extern libloading=/tmp/dyn-hello-world/target/debug/deps/liblibloading-a2854ce154eb4d6f.rlib</code></li>
</ul>
</li>
</ul>
<blockquote>
<p>There’s also a <code>-L native=…</code> one. This is explained in the manual of <code>rustc</code> and corresponds to
native library we must link against, like a C library, for instance.</p>
</blockquote>
<p>As you can see, the path used in <code>-L dependencie=…</code> is pretty <em>constant</em>. It seems it has the form:</p>
<pre><code>/path/to/project/target/&lt;build type&gt;/deps
</code></pre>
<p>However, remember that the initial intent was to write a plugin system for <code>spectra</code>, which is a
library. We don’t know the path to the project that will be using <code>spectra</code>, so we must find it in
a way.</p>
<p>Two options:</p>
<ul>
<li>We could use a <em>macro</em> (<code>macro_rules</code>) to automatically insert it at the calling site. However,
I’m not even sure it would work (you need to find a way to have access to <code>cargo</code>’s internal…
maybe via the <code>env!</code> macro?).</li>
<li>We could find the path at runtime.</li>
</ul>
<p>I chose the second option because it was pretty easy and straight-forward to implement. However,
I’m not a huge fan of it – it’s pretty… non-elegant to me. Here’s the code that gives me the root
path of the current project a function is defined in:</p>
<pre><code>/// Try to find the project root path so that we can pick dependencies.
fn find_project_root() -&gt; Result&lt;PathBuf, PluginError&gt; {
  let result =
    Command::new(&quot;cargo&quot;)
    .arg(&quot;locate-project&quot;)
    .output()
    .unwrap();

  if !result.status.success() {
    Err(PluginError::CannotFindMetaData(&quot;cannot locate root project&quot;.to_owned()))
  } else {
    let json = unsafe { from_utf8_unchecked(result.stdout.as_slice()) };
    let root =
      json.split(':').nth(1).and_then(|x| {
        if x.len() &gt;= 3 {
          Path::new(&amp;x[1..x.len()-3]).parent().map(|x| x.to_owned())
        } else {
          None
        }
      });

    root.ok_or_else(|| PluginError::CannotFindMetaData(&quot;cannot extract root project path from metadata&quot;.to_owned()))
  }
}
</code></pre>
<p>As you can see, I use the <code>cargo locate-project</code> feature, that gives you the root path of the
project the <code>cargo</code> invokation is in. It supports calling it from a subdirectory, which is neat (a
bit like <code>git</code>, actually). Most of the code is removing the JSON sugar over it. It’s pretty unsafe
because if the output format of <code>cargo locate-project</code> changes, this code will basically break.</p>
<blockquote>
<p>Pro reminder for myself: write unit tests for that piece of code. :D</p>
</blockquote>
<p>I won’t show the <code>PluginError</code> type, it’s not important and it’s <code>spectra</code> current code for the
feature.</p>
<p>Ok, we lack two things:</p>
<ul>
<li>The build type.</li>
<li>The path to our crate – here, <code>spectra</code>.</li>
</ul>
<p>Finding the build type is pretty easy: you can use the <code>debug_assertions</code> <code>cfg!</code> argument. It’s set
to <code>true</code> on <code>debug</code> target and <code>false</code> on <code>release</code>. You can actually witness it works with the
following oneliner:</p>
<pre><code>println!(&quot;build target is debug: {}&quot;, cfg!(debug_assertions));
</code></pre>
<p>Ok, now, how do we find our <code>spectra</code> crate’s path? You saw it has a metadata glued to its name in
the previous verbose output.</p>
<p>I didn’t find any satisfying solution. I just came up with <em>a solution</em>. I warn you: it’s not
elegant, it’s highly unsafe, but for now, it works. I’ll try to find a better way later.</p>
<p>Here’s the code.</p>
<pre><code>/// Find the spectra crate in the given directory.
fn find_spectra_crate(path: &amp;Path) -&gt; Result&lt;PathBuf, PluginError&gt; {
  // we open the directory we pass in as argument to the function
  if let Ok(dir) = path.read_dir() {
    // we iterate over all of its contained files to find if it has our spectra crate
    for entry in dir {
      let path = entry.unwrap().path();

      match path.file_name() {
        // we try to pattern match its name; this is a bit unsafe because it won’t support two
        // versions of libspectra… meh.
        Some(filename) if filename.to_str().unwrap().starts_with(&quot;libspectra&quot;) =&gt; {
          return Ok(path.to_owned());
        }

        _ =&gt; ()
      }
    }

    Err(PluginError::CannotFindMetaData(&quot;cannot find the spectra crate; try to build your project’s dependencies&quot;.to_owned()))
  } else {
    Err(PluginError::CannotFindMetaData(&quot;cannot find read dependencies&quot;.to_owned()))
  }
}
</code></pre>
<p>If you put those three functions altogether, you can now implement the <code>rustc</code> call without
hardcoding any paths, since they all will be found at runtime and injected in the call!</p>
<blockquote>
<p>A bit of hindsight. I didn’t explain that, but it’s a bit obvious: you won’t be able to use all
the crates you want in your plugins, for a very simple reason: you must have them installed
somewhere. In order for the <code>find_spectra_crate</code> to find anything, you must have <code>spectra = &quot;…&quot;</code>
in the <code>[dependencies]</code> section of your <code>Cargo.toml</code>. If you want to use any crate, you need to
add them in the <code>[dependencies]</code> and write a smarter function that parses the <code>Cargo.toml</code>’s
<code>[dependencies]</code> section and add them to the <code>rustc</code> invokation… which is basically like
re-writting a feature of <code>cargo</code> itself!</p>
</blockquote>
<h2>How do you make auto-reload again?</h2>
<p>I didn’t speak about that, but in <code>spectra</code>, it’s <strong>very easy</strong> to have a resource to auto-reload if
it changes on the disk. This is done via the <code>warmy</code> crate. You just implement the
<a href="https://docs.rs/warmy/0.5.2/warmy/trait.Load.html"><code>Load</code></a> trait and ask for your type at a
<a href="https://docs.rs/warmy/0.5.2/warmy/struct.Store.html"><code>Store</code></a> by providing a typed key. I won’t
talk too much about <code>warmy</code> – if you’re interested, go read the online documentation
<a href="https://docs.rs/warmy">here</a>!</p>
<p>The idea is that I just created a type that wraps both <code>libloading::Library</code> and <code>tempdir::TempDir</code>.
Something like this:</p>
<pre><code>pub struct Plugin {
  lib: Library,
  dir: TempDir
}
</code></pre>
<p>I still don’t know whether I need to implement <code>Drop</code> or not – if the <code>TempDir</code> dies first, I don’t
know whether the <code>Library</code> object is in an unstable state. If not, that means that the whole library
was loaded in RAM at the end of the <code>Library::new</code> call, and that I don’t even need to keep the
temporary directory around!</p>
<h2>It’s just the beginning</h2>
<p>The interface of a <code>Plugin</code> is not yet defined. I’m writing this blog entry on Sunday night / Monday
morning, while I had that plugin experiment over the weekend. A few thoughts, though:</p>
<ul>
<li>My current code enables me to run my demo (graphics-ish) and edit Rust code to script it <em>live</em>,
which is completely amazing to me!</li>
<li>It should be easy to expose a safe interface by asking the plugin to expose a normal function and
inject a header in the Rust code that contains an <code>unsafe fn</code>, calling the safe function or a
trait’s method or whatever.</li>
<li>You have no idea how much I’m excited about all this!</li>
</ul>
<p>I have a <em>lot</em> of things to say, especially lately. I’ll be posting more thoughts and experiments of
mine soon! Thanks for having read through, and as always, keep the vibes!</p>


guid: https://strongly-typed-thoughts.net/blog/hkt-1598Mon, 25 Nov 2019 15:50:00 GMT
title: Emulating RFC 1598, less or… more?!
author: Dimitri Sabadie <dimitri.sabadie@gmail.com>
url: https://strongly-typed-thoughts.net/blog/hkt-1598
feedurl: https://phaazon.net/blog/feed
content:
<p>Lately, I’ve been working on a huge feature that’s coming to <a href="https://crates.io/crates/luminance">luminance</a> soon. While working on
it, I’ve been facing several design problems I think are interesting to talk about.</p>
<!-- vim-markdown-toc GFM -->
<ul>
<li><a href="#the-context">The context</a></li>
<li><a href="#the-problem">The problem</a></li>
<li><a href="#faking-higher-kinded-types">Faking higher-kinded types</a>
<ul>
<li><a href="#a-simple-implementor">A simple implementor</a></li>
<li><a href="#polymorphic-use">Polymorphic use</a></li>
</ul>
</li>
<li><a href="#rationale">Rationale</a></li>
<li><a href="#conclusion">Conclusion</a></li>
</ul>
<!-- vim-markdown-toc -->
<h1>The context</h1>
<p>Imagine you want to expose a trait that defines an interface. People know about the interface and
can use provided types to switch the implementation. That’s a typical use case of traits.</p>
<pre><code class="language-rust">// This trait is pub so that users can see and use it
pub trait System {
  type Err;

  fn do_something(&amp;mut self) -&gt; Result&lt;(), Self::Err&gt;;
}
</code></pre>
<p>You can have a type <code>SimpleSystem</code> that implements <code>System</code> and that will do something special in
<code>do_something</code>, as well as a type <code>ComplexSystem</code> doing something completely different.</p>
<p>Now imagine that your <code>System</code> trait needs to expose a function that returns an object that must be
typed. What it means is that such an object’s type is tagged with another type. Imagine a
type <code>Event</code> that is typed with the type of event it represents:</p>
<pre><code class="language-rust">pub struct Event&lt;T&gt; {
  code: i64,
  msg: String,
  // …
}
</code></pre>
<p>That type must be provided and implemented by systems, not by the actual interface code. How would
we do this?</p>
<p>Furthermore, we might want another trait to restrict what <code>T</code> can be, but it’s off topic for our
current problem here.</p>
<h1>The problem</h1>
<p>Let’s try a naive implementation first:</p>
<pre><code class="language-rust">pub trait System {
  type Event;

  type Err;

  fn do_something(&amp;mut self) -&gt; Result&lt;(), Self::Err&gt;;

  fn emit_event(&amp;mut self, event: Self::Event);
}
</code></pre>
<p>That implementation allows <code>SimpleSystem</code> to implement <code>Event</code> as a single type and then
implement <code>emit_event</code>, taking its <code>Event</code> type. However, that event is not typed as we
wanted to. We want <code>Event&lt;T&gt;</code>, not <code>Event</code>.</p>
<p>However, Rust doesn’t authorize that per-se. The following is currently illegal in Rust:</p>
<pre><code class="language-rust">pub trait System {
  type Event&lt;T&gt;;

  // …

  fn emit_event&lt;T&gt;(&amp;mut self, event: Self::Event&lt;T&gt;);
}
</code></pre>
<blockquote>
<p><a href="https://github.com/rust-lang/rfcs/pull/1598">RFC 1598</a> is ongoing and will allow that, but until then, we need to come up with a solution.</p>
</blockquote>
<p>The problem is that associated types, in Rust, are completely monomorphized when the <code>trait</code> impl
is monomorphized, which is, for instance, not the case for trait’s functions. <code>emit_event</code> will
not have its <code>F</code> type variable substituted when the implementor is monomorphized — it will be sank
to a type when it’s called. So what do we really want to express with our <code>Event&lt;T&gt;</code> type?</p>
<pre><code class="language-rust">pub trait System&lt;EventType&gt; {
  type Event;

  fn emit_event(&amp;mut self, event: Self::Event);
}
</code></pre>
<p>That would work but that’s not <em>exactly</em> the same thing as our previous trait. The rest of the
trait doesn’t really depend on <code>EventType</code>, so we would duplicate code every time we want to support
a new type of event. Meh.</p>
<h1>Faking higher-kinded types</h1>
<p><code>Event&lt;T&gt;</code> is a higher-kinded type (HKT). <code>Event</code>, here, is what we call a <em>type constructor</em> — as
opposed to <em>data constuctor</em>, like associated <code>new</code> functions. <code>Event</code> takes a type and returns a
type, in the type-system world.</p>
<p>As I said earlier, Rust doesn’t have such a construct yet. However, we can emulate it with a nice
hack I came across while trying to simulate kinds.</p>
<p>See, when a HKT has its type(s) variable(s) substituted, it becomes a regular, monomorphized type. A
<code>Vec&lt;T&gt;</code>, when considered as <code>Vec&lt;u32&gt;</code>, is just a simple type the same way <code>u32</code> is one. The key is
to decompose our trait into two distinct yet entangled interfaces:</p>
<ul>
<li>One that will work with the monomorphized version <em>after</em> the trait is monomorphized.</li>
<li>One that will introduce polymorphism via a method, and not an associated type.</li>
</ul>
<p>The first trait is implemented on systems and means “[a type] knows how to handle an event of a
given type.” The second trait is implemented on systems, too, and means “can handle all events.”</p>
<pre><code class="language-rust">pub trait SystemEvent&lt;T&gt;: System {
  type Event;

  fn emit_system_event(&amp;mut self, event: Self::Event) -&gt; Result&lt;(), Self::Err&gt;;
}

pub trait System {
  type Err;

  fn do_something(&amp;mut self) -&gt; Result&lt;(), Self::Err&gt;;

  fn emit_event&lt;T&gt;(
    &amp;mut self,
    event: &lt;Self as SystemEvent&lt;T&gt;&gt;::Event
  ) -&gt; Result&lt;(), Self::Err&gt;
  where Self: SystemEvent&lt;T&gt; {
    &lt;Self as SystemEvent&lt;T&gt;&gt;::emit_system_event(self, event)
  }
}
</code></pre>
<p>The idea is that the <code>SystemEvent&lt;T&gt;</code> trait must be implemented by a system to be able to emit
events of type <code>T</code> inside the system itself. Because the <code>Event</code> type is associated in
<code>SystemEvent&lt;T&gt;</code>, it is the monomorphized version of the polymorphic type in the actual
implementation of the trait, and is provided by the implementation.</p>
<p>Then, <code>System::emit_event</code> can now have a <code>T</code> type variable representing that <code>Event&lt;T&gt;</code> we
wanted. We use a special type-system candy of Rust here: <code>Self: Trait</code>, which allows us to state
that in order to use that <code>emit_event&lt;T&gt;</code> function, the implementor of <code>System</code> must also
satisfy <code>SystemEvent&lt;T&gt;</code>. Even better: because we already have the implementation of
<code>emit_system_event</code>, we can blanket-implement <code>emit_event&lt;T&gt;</code>!</p>
<p>Several important things to notice here:</p>
<ul>
<li>First, a system <em>can</em> now implement several traits:
<ul>
<li><code>System</code>, to work as a system, obviously.</li>
<li><code>SystemEvent&lt;T&gt;</code>, to handle events of type <code>T</code>. The actual type is defined by the system itself.</li>
</ul>
</li>
<li>Second, a system doesn’t have to implement <code>SystemEvent&lt;T&gt;</code> if it doesn’t know how to handle
events of type <code>T</code>. That opt-in property is interesting for us as it allows systems to implement
only what they support without having to make our design bleed to all systems. <em>Neat</em>.</li>
<li>The usage of <code>Self: Trait</code> allows to introduce a form of polymorphism that is not available without
it.</li>
<li>Because we can blanket-implement <code>emit_event</code>, the implementor of <code>System</code> doesn’t even have to
implement <code>emit_event</code>.</li>
<li><code>SystemEvent&lt;T&gt;</code> has <code>System</code> has super-trait to share its <code>Err</code> type.</li>
</ul>
<p>The last point is the <em>hack key</em>. Without <code>Self: Trait</code>, it would be way harder or more convoluted
to achieve the same result.</p>
<h2>A simple implementor</h2>
<p>Let’s see a simple implementor that will just print out the event it gets and does nothing in
<code>do_something</code>.</p>
<pre><code class="language-rust">struct Simple;

#[derive(Debug)]
struct ForwardEvent&lt;T&gt;(pub T);

impl System for Simple {
  type Err = ();

  fn do_something(&amp;mut self) -&gt; Result&lt;(), Self::Err&gt; {
    Ok(())
  }
}

impl&lt;T&gt; SystemEvent&lt;T&gt; for Simple where T: std::fmt::Debug {
  type Event = ForwardEvent&lt;T&gt;;

  fn emit_system_event(&amp;mut self, event: Self::Event) -&gt; Result&lt;(), Self::Err&gt; {
    println!(&quot;emit: {:?}&quot;, event.0);
    Ok(())
  }
}
</code></pre>
<p>As you can see, it’s really simple and straight-forward. We can select which events we want to be
able to handle: in our case, anything that implements <code>Debug</code>.</p>
<p>Let’s use it:</p>
<pre><code class="language-rust">fn main() {
  let mut system = Simple;
  system.emit_event(ForwardEvent(&quot;Hello, world!&quot;));
  system.emit_event(ForwardEvent(123));
}
</code></pre>
<h2>Polymorphic use</h2>
<p>The idea is that, given a type <code>S: System</code>, we might want to emit some events without knowing the
implementation. To make things even more crunchy, let’s say we want the error type to be a
<code>()</code>.</p>
<p>Again, it’s quite simple to do:</p>
<pre><code class="language-rust">// we need this to forward the event from the outer world into the system so that we don’t have to
// know the actual type of event used by the implementation
impl&lt;T&gt; From&lt;T&gt; for ForwardEvent&lt;T&gt; {
  fn from(t: T) -&gt; Self {
    ForwardEvent(t)
  }
}

fn emit_specific_event&lt;S, E&gt;(
  system: &amp;mut S,
  event: E
) -&gt; Result&lt;(), S::Err&gt;
where
  S: System&lt;Err = ()&gt; + SystemEvent&lt;E&gt;,
  S::Event: From&lt;E&gt;,
{
  system.emit_event(event.into())
}

fn main() {
  let mut system = Simple;
  system.emit_event(ForwardEvent(&quot;Hello, world!&quot;));
  emit_specific_event(&amp;mut system, &quot;Hello, world!&quot;);
}
</code></pre>
<p>We could event change the signature of <code>System::emit_event</code> to add that <code>From&lt;E&gt;</code> constraint to
make the first call easier, but I’ll leave you with that. The important aspect of this code snippet
is the fact that the implementor will handle a type of event <code>Event&lt;T&gt;</code> while the interface uses
<code>T</code> directly. We have injected a HKT <code>Event</code>.</p>
<h1>Rationale</h1>
<p>Why do I care about such a design? It might seem complex and hard, but it’s actually a very useful
use of a typeclass / trait type system — especially in <a href="https://crates.io/crates/luminance">luminance</a>, where I use type-system
concepts <strong>a lot</strong>; after all, it’s based on its Haskell version! If you compare my solution to the
next-to-come HKT proposal from <a href="https://github.com/rust-lang/rfcs/pull/1598">RFC 1598</a>, we have:</p>
<ul>
<li><a href="https://github.com/rust-lang/rfcs/pull/1598">RFC 1598</a> will bring the syntax <code>type Event&lt;T&gt;;</code> as associated types, which will allow to remove
our <code>SystemEvent&lt;T&gt;</code> trait.</li>
<li>With that RFC, the type will have to be <em>total</em>. What it means is that the <code>T</code> here is the same
for all implementors: an implementor cannot restrict <code>T</code>, it must be implemented for all of them.
If you want to restrict the type of events your type can receive, if I understand the RFC
correctly, you’re handcuffed. That RFC will authorize the <em>author of the trait</em> to restrict your
<code>T</code>, not the <em>author of the implementor</em>. That might have some very useful advantages, too.</li>
<li>With my solution, the <em>author of the trait</em> can still restrict <code>T</code> by applying constraints on
<code>emit_event</code>.</li>
</ul>
<h1>Conclusion</h1>
<p>In the quest of emulating HKT, I’ve found myself with a <em>type-system toy</em> I like using a lot:
<em>equality constraints</em> and self-constraints (I don’t know how those <code>Self: Trait</code> should be named).
In Haskell, since we don’t implement a typeclass <em>for a type</em> but we provide <em>an instance for a
typeclass</em>, things are slightly different. Haskell doesn’t have a <code>Self</code> type alias, since a
typeclasses can be implemented with several types variables (i.e. with the <code>MultiParamTypeClasses</code>
and <code>FlexibleInstances</code> GHC extensions), only <em>equality constraints</em> are needed.</p>
<p>In the end, Rust continues to prove that even though it’s a systems programming language, I can
express lots of powerful abstractions I miss (<em>a lot</em>) from Haskell with <em>a bit more</em> noise. I think
the tradeoff is worth it.</p>
<p>I still haven’t completely made up my mind about GAT / <a href="https://github.com/rust-lang/rfcs/pull/1598">RFC 1598</a> (for sure I’m among the ones who
want it on stable ASAP but I’m yet to figure out <em>exactly</em> how it’s going to change my codebases).</p>
<p>As always, have fun, don’t drink and drive, use condoms, keep the vibes and don’t use impl Trait in
argument position. Have fun!</p>


guid: https://strongly-typed-thoughts.net/blog/aoc-18-hindsightSat, 12 Jan 2019 00:20:00 GMT
title: Hindsight on Advent of Code 2018
author: Dimitri Sabadie <dimitri.sabadie@gmail.com>
url: https://strongly-typed-thoughts.net/blog/aoc-18-hindsight
feedurl: https://phaazon.net/blog/feed
content:
<p>On the 1st of December 2018, I decided to give it a try to <a href="https://adventofcode.com/about">Advent of Code</a>. AoC is, basically, a
programming challenge website where you get two puzzles unlocked every day of December from 1st to
25th – hence the name. It has a ranking system which scores are based on the absolute time you took
to solve the puzzles – i.e. spent time as soon as the puzzles got unlocked. As a French living in
Paris, I find this a bit unfair (we get puzzles unlocked at around 5:00 AM!) and then I just
realized I could do the puzzles for fun only.</p>
<p>This blog post sums up what I did with AoC#18, my thoughts about the puzzles and even a
meta-discussion about programming challenges. I used Haskell for most challenges and switched to
Rust for no specific reason. Just fun.</p>
<p>Also, because the puzzles’ texts are somewhat long, I will just give a link to the text instead of
copying it and will just make a very short description of the problem each time.</p>
<p>Finally, don’t expect to find all the puzzles: I stopped at day 15. I will explain the reasons why
at the end of this article. My adventure can be found
<a href="https://github.com/phaazon/advent-of-code-2k18">here</a>. Most puzzles have <em>input data</em>. You will
find them in day directories as <code>input.txt</code> files.</p>
<p>Enjoy the reading!</p>
<blockquote>
<p><strong>If you want to try and take the challenges, I advise you not to read any further as this article
would spoil you solutions! You will need a Github account and… a lot of time.</strong></p>
</blockquote>
<!-- vim-markdown-toc GFM -->
<ul>
<li><a href="#day-1-chronal-calibration">Day 1: Chronal Calibration</a>
<ul>
<li><a href="#part-1">Part 1</a></li>
<li><a href="#part-2">Part 2</a></li>
</ul>
</li>
<li><a href="#day-2-inventory-management-system">Day 2: Inventory Management System</a>
<ul>
<li><a href="#part-1-1">Part 1</a></li>
<li><a href="#part-2-1">Part 2</a></li>
</ul>
</li>
<li><a href="#day-3-no-matter-how-you-slice-it">Day 3: No Matter How You Slice It</a>
<ul>
<li><a href="#part-1-2">Part 1</a></li>
<li><a href="#part-2-2">Part 2</a></li>
</ul>
</li>
<li><a href="#day-4-repose-record">Day 4: Repose Record</a>
<ul>
<li><a href="#part-1-3">Part 1</a></li>
<li><a href="#part-2-3">Part 2</a></li>
</ul>
</li>
<li><a href="#day-5-alchemical-reduction">Day 5: Alchemical Reduction</a>
<ul>
<li><a href="#part-1-4">Part 1</a></li>
<li><a href="#part-2-4">Part 2</a></li>
</ul>
</li>
<li><a href="#day-6-chronal-coordinates">Day 6: Chronal Coordinates</a>
<ul>
<li><a href="#part-1-5">Part 1</a></li>
<li><a href="#part-2-5">Part 2</a></li>
</ul>
</li>
<li><a href="#day-7-the-sum-of-its-parts">Day 7: The Sum of Its Parts</a>
<ul>
<li><a href="#part-1-6">Part 1</a></li>
<li><a href="#part-2-6">Part 2</a></li>
</ul>
</li>
<li><a href="#day-8-memory-maneuver">Day 8: Memory Maneuver</a>
<ul>
<li><a href="#part-1-7">Part 1</a></li>
<li><a href="#part-2-7">Part 2</a></li>
</ul>
</li>
<li><a href="#day-9-marble-mania">Day 9: Marble Mania</a>
<ul>
<li><a href="#part-1--2">Part 1 &amp; 2</a></li>
</ul>
</li>
<li><a href="#day-10-the-stars-align">Day 10: The Stars Align</a>
<ul>
<li><a href="#part-1-8">Part 1</a></li>
<li><a href="#part-2-8">Part 2</a></li>
</ul>
</li>
<li><a href="#day-11-chronal-charge">Day 11: Chronal Charge</a>
<ul>
<li><a href="#part-1-9">Part 1</a></li>
<li><a href="#part-2-9">Part 2</a></li>
</ul>
</li>
<li><a href="#day-12-subterranean-sustainability">Day 12: Subterranean Sustainability</a>
<ul>
<li><a href="#part-1-10">Part 1</a></li>
<li><a href="#part-2-10">Part 2</a></li>
</ul>
</li>
<li><a href="#day-13-mine-cart-madness">Day 13: Mine Cart Madness</a>
<ul>
<li><a href="#part-1-11">Part 1</a></li>
<li><a href="#part-2-11">Part 2</a></li>
</ul>
</li>
<li><a href="#day-14-chocolate-charts">Day 14: Chocolate Charts</a>
<ul>
<li><a href="#part-1-12">Part 1</a></li>
<li><a href="#part-2-12">Part 2</a></li>
</ul>
</li>
<li><a href="#day-15-beverage-bandits">Day 15: Beverage Bandits</a>
<ul>
<li><a href="#part-1-13">Part 1</a></li>
</ul>
</li>
<li><a href="#conclusion">Conclusion</a></li>
</ul>
<!-- vim-markdown-toc -->
<h1>Day 1: Chronal Calibration</h1>
<p><a href="https://adventofcode.com/2018/day/1">Text</a></p>
<h2>Part 1</h2>
<blockquote>
<p>This puzzle is about summing arrays and finding duplicates (without counting).</p>
</blockquote>
<p>First puzzle I discovered and first AoC challenge I ever took, I was surprised at the simplicity.
Basically, given lines containing a single number preceded by either <code>+</code> or <code>-</code>, we must compute
the sum of all the numbers.</p>
<p>The first thing to do is to parse the file into a stream of numbers. In Haskell, this was almost
a one liner:</p>
<pre><code>main :: IO ()
main = do
    numbers &lt;- fmap (map parse . lines) getContents
  where
    parse ('+':xs) = read xs -- (1)
    parse n = read n
</code></pre>
<p>If I removed the <code>+</code> from the file, I could even have gone with a one liner:</p>
<pre><code>    numbers &lt;- fmap (map read . lines) getContents
</code></pre>
<blockquote>
<p>The <code>read</code> function is not safe and I don’t recommend using it for real, production projects.
In the context of this challenge, though, it was alright.</p>
</blockquote>
<p>I don’t have much to say about this puzzle as I found it not very interesting.</p>
<h2>Part 2</h2>
<p>We just want to get the first cumulative sum that appears twice. That is, each time a new number
is summed, we get a given “current sum”. We add positive and negative integers, so we might end up,
at some time, with the same number. Because we don’t know when that will happen, we need to turn
the input stream into an infinite stream of numbers that repeat over and over. Haskell has the
<code>cycle</code> function that does exactly that: it takes a list and makes it repeat infinitely.</p>
<p>What is great about <code>cycle</code> in Haskell is that it could be implemented in a very naive way and yet
yield good performance – thanks GHC!:</p>
<pre><code>cycle :: [a] -&gt; [a]
cycle x = let x' = x ++ x' in x'
</code></pre>
<p>Or, prefered (personal opinon):</p>
<pre><code>cycle :: [a] -&gt; [a]
cycle = fix . (++)
</code></pre>
<blockquote>
<p><code>fix</code> is a very powerful function any Haskell should know. Typically, when you want to
corecursively build something, it’s very likely you’ll need <code>fix</code>. I advise you to have a look at
my <a href="https://phaazon.net/blog/getting_into_netwire">netwire tutorial</a> in which I used <code>fix</code> pretty much everywhere.</p>
</blockquote>
<p>There are several ways to fix this problem. I chose to use a <strong>set</strong> to hold all the known
intermediate sums. Because we’re working with integers, the Haskell <code>IntSet</code> type is perfect for
that. The algorithm is then just a special fold over an infinite list that stops when a sum is seen
twice by looking it up in the integer set:</p>
<pre><code>findFreq _ freq [] = freq -- this shouldn’t ever happen since the list is infinite
findFreq knownFreq freq (change:xs)
    | newFreq `member` knownFreq = newFreq
    | otherwise = findFreq (insert newFreq knownFreq) newFreq xs
  where
    newFreq = freq + change
</code></pre>
<p><a href="https://github.com/phaazon/advent-of-code-2k18/blob/master/day-01/src/Main.hs">Haskell solution</a></p>
<h1>Day 2: Inventory Management System</h1>
<p><a href="https://adventofcode.com/2018/day/2">Text</a></p>
<h2>Part 1</h2>
<p>This puzzle requires you to recognize and count characters in strings. For each string, you must
check whether they have any letter that appears exactly once and any letter that appears exactly
three times. Adding those counts provides a checksum for a list of strings.</p>
<p>I did that in Haskell as well. My idea was to use a count-array of size 26 (since the strings are
only in <code>[a-z]</code>. So the first part of my algorithm is to count, for a given string, the number of
time each letter occurs. This is done really easily with a fold that increments the proper value
value in the array at the index representing the character folded on in the string. That index
is obtained by turning a character into a numerical representation and subtracting it the numerical
value of <code>'a'</code>:</p>
<pre><code>treatHash :: (Natural, Natural) -&gt; String -&gt; (Natural, Natural)
treatHash occurs hash =
    let result :: State = foldl' replace initialState hash
    in addPair occurs (foldl' incrOccur (0, 0) result)
  where
    replace st l = accum (+) st [(ord l - ord 'a', 1)]
    incrOccur i@(twos, threes) x
      | x == 2 = (1, threes)
      | x == 3 = (twos, 1)
      | otherwise = i
addPair (a, b) (c, d) = (a + c, b + d)
</code></pre>
<p>The <code>replace</code> function performs the update in an input vector (point-free function). This will
effectively increment the vector’s value at index <code>3</code> if the character is <code>d</code> (because
<code>ord 'd' - ord 'a' = 3</code>. <code>incrOccur</code>, here, is used to fold over the resulting array (<code>Vector</code>) and
compute if a letter appears twice and if a letter appears three times. You will notice that this
function doesn’t sum. For instance, if you have <code>aabb</code>, you get no letter that appears three times
but you have two that appear exactly twice. However, <code>incrOccur</code> will give you <code>(1, 0)</code>, because the
puzzle states it should be done this way. Finally, <code>addPair</code> add the number of twos and threes to a
pre-defined input – that will be handy for an outer fold.</p>
<p>The algorithm is then run on the input and we get the final result:</p>
<pre><code>hashes &lt;- fmap lines getContents
let checksum = uncurry (*) $ foldl' treatHash (0, 0) hashes
</code></pre>
<p>We transform the standard input into lines – i.e. <code>[String]</code> – and then fold them with our
<code>treatHash</code> function. Finally, <code>uncurry (*)</code> is a little oneliner to simply multiply the left part
of a pair with its right part.</p>
<h2>Part 2</h2>
<p>This part is about finding strings that are almost the same and differ and only by one letter. As
doing a programming challenge, the first thing I want to find is <em>a solution</em>, not the best
solution. I also think people should really learn from this:</p>
<ol>
<li>Make it work.</li>
<li>Make it clean.</li>
<li>Make it fast.</li>
</ol>
<p>Especially on programming challenges, you’ll be very, very unlikely to implement anything more than
(1.), because the inputs are often two small to really benefit from a real optimization. Your time
is a real important resource, don’t waste it. Measure whether you really need an optimization. I’m
not saying that you should be doing something bad because it’s easier. I’m saying that if it takes
you N minutes to write a solution that runs in M milliseconds, if you know that you could do it in,
for instance, 10N minutes to write a solution that runs in only 0,7M milliseconds for the foreseen
milliseconds, will, you’re going to waste your time.</p>
<p>So, in my case, I started with a naive approach that runs in <em>O(N²)</em>: comparing all strings to all
others:</p>
<pre><code>searchCommon :: [String] -&gt; String
searchCommon [] = &quot;&quot; -- the empty list has nothing in common
searchCommon (h:hs) = search h hs -- for each string, search in all remaining
  where
    search h [] = searchCommon hs -- exhausted,  search with the next string
    search h (x:xs)
      | almostSame h x = commonLetters h x -- grab the common letters if almost the same
      | otherwise = search h xs -- if not, just try the next string in the sub-list
</code></pre>
<p>The algorithm is pretty straight-forward. The <code>searchCommon</code> and <code>search</code> functions are mutually
recursive functions that go from, respectively, the whole list of strings to test and the local
tail as we advance. <code>almostSame</code> is defined as follows:</p>
<pre><code>almostSame :: String -&gt; String -&gt; Bool
almostSame = go 0
  where
    go diffNb (l:ls) (r:rs)
      | diffNb &gt; 1 = False
      | l /= r = go (succ diffNb) ls rs
      | otherwise = go diffNb ls rs
    go diffNb _ _ = diffNb == 1
</code></pre>
<p>This function is a special <code>zip</code> that short-circuits if it knows there are two many differences.
When both the input strings are exhausted, if <code>diffNb == 1</code>, then only one character has changed, so
the the overwhole function evaluates to <code>True</code>.</p>
<p>The final part we need is <code>commonLetters</code>, that is pretty straight-forward, too:</p>
<pre><code>commonLetters :: String -&gt; String -&gt; String
commonLetters (l:ls) (r:rs)
  | l == r = l : commonLetters ls rs
  | otherwise = ls -- all the rest is identic, smartass
</code></pre>
<p>We construct a list that has the same characters as the input lists as long as they’re equal. As
soon as they differ, we discard the character and just return any of the input lists – they’re
equal. This function only works, obviously, if both the input strings are almost identical (only one
character differs and they have the same length). The <code>otherwise</code> branch is a short-circuit
optimization that prevents us from traversing the whole inputs after the difference.</p>
<p><a href="https://github.com/phaazon/advent-of-code-2k18/blob/master/day-02/src/Main.hs">Haskell solution</a></p>
<h1>Day 3: No Matter How You Slice It</h1>
<p><a href="https://adventofcode.com/2018/day/3">Text</a></p>
<h2>Part 1</h2>
<p>This problem was the first “challenging” as it required more thinking. The idea is that a very large
area – the text gives us the hint it is at least <code>1000</code> inches on each side but truly, we do not
need this information – must be sliced into several rectangular smaller areas (with edges aligned
with X and Y axis). Obviously, some rectangular area might overlap and the aim of the first part of
this problem is to find out <strong>the total overlapping area</strong>.</p>
<p>My idea was simple: we can trick and change the problem by discretizing it. This is something I
have said already but most of AoC problems have <em>hidden properties</em> and <em>hidden hints</em>. By thinking
more about the form of the solution, the area to find is expressed in <em>inches²</em>. The rectangular
areas are all lying on a 2D grid (1000×1000 at least, remember?) and their coordinates are always
natural numbers (they belong to <code>[0;+∞]</code>). Then, my idea was to state that 1 inch² is actually a
single “cell” at any coordinate in that grid – imagine that as a pixel. 4 inches² would be a 2×2
region in that grad (yielding 4 cells).</p>
<p>So instead of using the general idea of an area (M×N for a rectangle which sides are M and N long
wide), we can simply break a rectangular area into its most atomic components (1×1 cells)… and sum
them up! We will effectively end up with the area of this area.</p>
<p>A more interesting property of my way of solving it: we can now have a big array mapping to each
1×1 cell the number of times a rectangle lies on it. When iterating through all the rectangles, we
just break them into a list of 1×1 cells, look ingg up and updating the big count array. Once we’re
done, we just have to filter that big array to remove any element which count is less or equal to 1.
The remaining elements are 1×1 cells that contain at least two overlapping rectangles – we don’t
really care about the number. We don’t actually care about those elements: the length of that
filtered array is the area we are looking for, because it is the sum of 1×1 elements!</p>
<p>Converting a rectangular area – called claim in the text – into 1×1 cells is very easy in Haskell:</p>
<pre><code>type Plan = Map (Int, Int) Natural -- the big array I told you
type Claim = (Id, Pos, Area)
type Id = Text -- the “name” of the fabric the rectangular area is about
type Pos = (Int, Int)
type Area = (Int, Int)

claim1x1 :: Claim -&gt; [Pos]
claim1x1 (_, (line, row), (width, height)) =
  [(x, y) | x &lt;- [line .. line + width - 1], y &lt;- [row .. row + height - 1]]
</code></pre>
<p>As you can see, I chose to use a <code>Map (Int, Int) Natural</code> instead of an array to store the number
of overlaps per 1×1 cell. That enables us not to care at all about the size of the grid (remember
when I told you we don’t need the <code>1000</code> hint?).</p>
<p>The function that updates the <code>Plan</code> to take a claim into account is:</p>
<pre><code>checkClaims :: [Claim] -&gt; Plan
checkClaims = foldl' (\p (x, y) -&gt; checkInch x y p) mempty . concatMap claim1x1

checkInch :: Int
          -&gt; Int
          -&gt; Plan
          -&gt; Plan
checkInch line row = insertWith (+) (line, row) 1
</code></pre>
<p>Given a list of claims (<code>[Claim]</code>), <code>checkClaims</code> maps the <code>claim1x1</code> function and concats the
result, yielding a <code>[Pos]</code> list. That list is then folded over with the <code>checkInch x y p</code> function,
that takes an empty map as initial value. <code>checkInch</code> just increment the value found in the map if
it already exists; otherwise, it sets that value to <code>1</code>.</p>
<p>Finally, we need to compute the area:</p>
<pre><code>overlappingInches :: Plan -&gt; Int
overlappingInches = length . M.filter (&gt; 1)
</code></pre>
<p>As I told you, that is crystal clear: it’s just the length of the filtered map.</p>
<h2>Part 2</h2>
<p>This part is interesting also: you need to find out the <code>Id</code> of the claim that doesn’t overlap with
any other claim. I will not go into too much details about the algorithm as it’s very similar to the
previous one: instead of storing the number of overlaps by 1×1 cell, we store a <code>Set Id</code>, giving all
claims that are overlapping – we can see that as a more general form of the first part. We also need
a <code>Map Id Natural</code> that maps a fabric and the number of times it overlaps another. The fabric that
doesn’t overlap any other is then easily identifiable within that map: it has its associated value
set to <code>0</code>:</p>
<pre><code>searchNonOverlapped :: [Claim] -&gt; Maybe Id
searchNonOverlapped claims =
    case M.toList filtered of
      [(i, _)] -&gt; Just i -- the text supposes there’s only one
      _ -&gt; Nothing
  where
    (_, overlapped) = indexClaims claims
    filtered = M.filter (== 0) overlapped -- hello
</code></pre>
<p><a href="https://github.com/phaazon/advent-of-code-2k18/blob/master/day-03/src/Main.hs">Haskell solution</a></p>
<h1>Day 4: Repose Record</h1>
<p><a href="https://adventofcode.com/2018/day/4">Text</a></p>
<h2>Part 1</h2>
<p>Aaaah, day 4… I really don’t get why I got so annoyed by this one. It is quite simple in theory. But
that made me realize something I don’t really like about AoC: the <em>hidden rules</em>. You will see, if
you read this whole article, that some puzzles require you to make very important assumptions about
the input – and there’re a lot of assumptions you could make, so you have to make the right ones!</p>
<p>This puzzle is about guards that must protect a prototype manufacturing lab. You are given an
unordered list of events that occur about the guards:</p>
<ul>
<li>When they begin their shifts (you are given the ID of the guard here).</li>
<li>When they fall asleep.</li>
<li>When they wake up.</li>
<li>Each action gives you a timestamp so that you can re-create the historical events.</li>
</ul>
<p>The goal of the part 1 is to find the guard that has the most minutes asleep, and especially, find
the minute it is asleep the most – we must multiply the ID of the guard by the minute.</p>
<p>Obviously, the first part of this puzzle is to re-order the input to have it chronologically. Here
is the setup code:</p>
<pre><code>data Entry = Entry {
    timestamp :: LocalTime,
    action :: Text
  } deriving (Show)

type GuardianId = Natural

data Action
  = BeginShift GuardianId
  | WakeUp
  | FallAsleep
    deriving (Eq, Ord, Show)

entryFromString :: Text -&gt; Maybe Entry
entryFromString s = case split (== ']') s of
    [timestamp, action] -&gt; Just $ Entry (parse . unpack $ T.drop 1 timestamp) action
    _ -&gt; Nothing
  where
    parse = parseTimeOrError False defaultTimeLocale &quot;%Y-%-m-%-d %H:%M&quot;
</code></pre>
<p>The parsing part is not really interesting as it’s just challenge code: nasty but working parsing
code. :D</p>
<p>I then re-ordered the input with:</p>
<pre><code>entries &lt;- fmap (fromJust . traverse entryFromString . T.lines) T.getContents
let byTimeSorted = sortBy (comparing timestamp) entries
</code></pre>
<p>By the way, that code made me want to <a href="https://twitter.com/phaazon_/status/1070054255887822848">tweet about how Haskell is actually pretty easy to read and
reason about</a>. Anyway.</p>
<p>The next part of the algorithm is to transform the entries into a list of timed action. I actually
decided to stream it so that I could benefit from Haskell’s stream fusion – and because it’s so
simple and transparent:</p>
<pre><code>readGuardianId :: Text -&gt; Natural
readGuardianId = readT . T.drop 1

treatEntries :: [Entry] -&gt; [(LocalTime, Action)]
treatEntries = map $ \entry -&gt;
  let time = timestamp entry
  in case T.words (action entry) of
    [&quot;Guard&quot;, ident, &quot;begins&quot;, &quot;shift&quot;] -&gt; (time, BeginShift $ readGuardianId ident)
    (&quot;falls&quot;:_) -&gt; (time, FallAsleep)
    (&quot;wakes&quot;:_) -&gt; (time, WakeUp)
    _ -&gt; error &quot;lol&quot;
</code></pre>
<p>That is like mixing streaming and parsing at the same time. Then, the core of my algorithm: dispatch
the actions by guard. That is mandatory if we want to actually accumulate the “state” of a guardian
(when they’re sleeping, waking up, etc.). Otherwise, we get interleaved results.</p>
<pre><code>dispatchActions :: [(LocalTime, Action)] -&gt; Map GuardianId [(LocalTime, Action)]
dispatchActions = go mempty Nothing
  where
    go guardians _ ((t, action@(BeginShift gid)):xs) =
      go (insertAction gid t action guardians) (Just gid) xs

    go guardians jgid@(Just gid) ((t, action):xs) = go (insertAction gid t action guardians) jgid xs

    go guardians _ [] = fmap V.toList guardians

    go _ _ _ = error &quot;dispatchActions: the impossible fucking occurred!&quot;

    insertAction gid t action guardians =
      M.insertWith (flip (&lt;&gt;)) gid (V.singleton (t, action)) guardians
</code></pre>
<p>This is a by-hand fold that just applies the rule of beginning a shift (storing the ID of the
guardian that went napping so that we can correctly dispatch the remaining events).</p>
<p>Then the tricky part:</p>
<pre><code>type Minute = Natural
type Minutes = [Minute]

minutesCounts :: [(LocalTime, Action)] -&gt; Minutes
minutesCounts = go zeroMinutes Nothing
  where
    zeroMinutes = replicate 60 0 -- (1)
    asMinutes = todMin . localTimeOfDay

    -- the guard was sleeping
    go minutes (Just sleepTime) ((t, action):xs) =
      case action of
        BeginShift _ -&gt; go minutes Nothing xs
        FallAsleep -&gt; go minutes (Just t) xs -- not sure if that would even occur in the input
        WakeUp -&gt; go (addSleepCount minutes (asMinutes sleepTime) (asMinutes t)) Nothing xs

    -- the guard was awake, so we’re only interested in when they go to sleep
    go minutes Nothing ((t, action):xs) =
      case action of
        FallAsleep -&gt; go minutes (Just t) xs
        _ -&gt; go minutes Nothing xs

    go minutes _ [] = minutes

    addSleepCount minutes sleepTime t = zipWith (+) minutes range -- (2)
      where
        -- this function is a bit hacky but it generates, for a given range of time, a list of 60
        -- elements where the time period has 1 and all the other has 0 (I leave you to the
        -- exercise of making that a better function)
        range :: Minutes
        range = replicate sleepTime 0 &lt;&gt; replicate (fromIntegral t - sleepTime) 1 &lt;&gt; replicate (60 - t) 0
</code></pre>
<p>This big function generates a list which length is 60 – mapping the number of times a guard has
passed sleeping at a given minute from midnight to 1:00 AM (see <code>(1)</code> and <code>(2)</code>).</p>
<p>Finally, what we need is a way to compute frequencies – or counts. That is, given a list of anyting,
compute that number of time a given anything happens in the list. I wrote a small utility function
for that – I got inspired by [@jle], thanks!:</p>
<pre><code>freqTable :: (Ord a) =&gt; [a] -&gt; Map a Count
freqTable = M.fromListWith (+) . map (,1)
</code></pre>
<p>Then, finding the guard that has slept the more and the minute is easy:</p>
<pre><code>findMostOccurring :: Map a Count -&gt; (a, Count)
findMostOccurring = maximumBy (comparing snd) . M.toList -- and Haskell is hard?! ;)

findSleepiest :: Map GuardianId [(LocalTime, Action)] -&gt; (GuardianId, (Minute, Count))
findSleepiest =
    fmap (findMostOccurring . freqTable . spanIndex) . maximumBy (comparing $ sum . snd) . M.toList . fmap minutesCounts
  where
    spanIndex = concatMap (\(i, x) -&gt; replicate (fromIntegral x) i) . zip [0..]
</code></pre>
<p>We first find the guard that has the most time asleep (<code>maximumBy (comparing $ sum . snd)</code>. Then,
we find the minutes at which they were asleep the most (<code>findMostOccurring</code>). We are given the guard
ID, the given minute and the number of times they were asleep at that minute. Yay!</p>
<h2>Part 2</h2>
<p>For this part, we would like to know which guard is most frequently asleep on the same minute? We
already have written all the code needed for that:</p>
<pre><code>findMostFrequentlySleepy :: Map GuardianId [(LocalTime, Action)] -&gt; (GuardianId, Minute)
findMostFrequentlySleepy =
    fmap findMin . maximumBy (comparing $ maximum . snd) . M.toList . fmap minutesCounts
  where
    findMin = fst . maximumBy (comparing snd) . zip [0..]
</code></pre>
<p>Instead of summing, we find the maximum time a guard was asleep. Pretty easy.</p>
<p><a href="https://github.com/phaazon/advent-of-code-2k18/blob/master/day-04/src/Main.hs">Haskell solution</a></p>
<h1>Day 5: Alchemical Reduction</h1>
<p><a href="https://adventofcode.com/2018/day/5">Text</a></p>
<h2>Part 1</h2>
<p>That puzzle is very natural to solve in Haskell. You are given an ASCII string that contains only
letters (lower case and upper case) that represent polymers. You must compute their final reduction
by following some basic rules:</p>
<ul>
<li>Two adjacent letters will cancel them out if they don’t have the same case (lower vs. upper) but
are the same letter. For instance, <code>aA</code> and <code>Bb</code> cancel each other but not <code>aa</code> nor <code>BB</code>.</li>
<li>This rule happens until no more reduction is possible. For instance, <code>AbBa</code> will first get
reduced to <code>Aa</code> because <code>aB</code> will cancel out, but then <code>Aa</code> will also cancel out and we will be
left with nothing.</li>
</ul>
<p>You must give the number of units left in the final reducted polymer after all reductions have
occurred.</p>
<p>As I said, that is very simple and elegant in Haskell:</p>
<pre><code>reduce :: String -&gt; String
reduce = go []
  where
    go [] (x:xs) = go [x] xs
    go a [] = a
    go (a:xs) (b:bs)
      | not (isLetter b) = go (a:xs) bs
      | (toLower a /= toLower b) || (isLower a &amp;&amp; isLower b) || (isUpper a &amp;&amp; isUpper b) = go (b:a:xs) bs
      | otherwise = go xs bs
</code></pre>
<p>I decided to use a <a href="https://wiki.haskell.org/Zipper">zipper</a>-like traversal. My idea is the following:</p>
<ul>
<li>Read a character. If we have nothing previously seen, just place it in the previous list.</li>
<li>If we have something previously seen, check whether it reacts with the last previously seen
character. If so, just drop it and drop the previously seen character, and go on with the next
character.</li>
<li>When we have exhausted the input list, the previously seen list of characters is the final form
of the reduction.</li>
</ul>
<p>This algorithm allows me to reduce by doing a forwards-and-backwards kind of sweeping, yielding
nice performance. Also, notice that the resulting list is reversed because of how we accumulate the
seen characters. Because we don’t care about the order, we will not reverse it back to its original
order.</p>
<p>The result is just the length of the output list.</p>
<h2>Part 2</h2>
<p>This part asks us to find the polymer that is the smaller if we remove one kind of unit (a single
letter type). So if we remove <code>a</code> for instance, we must remove all <code>a</code> and <code>A</code>.</p>
<p>As there’re only 26 possible solutions (from <code>a</code> to <code>z</code>), and because my solution to part 1 was
already fast, I decided to go brute-force with this one: reducing the input string without a’s,
reducing the input string without b’s, reducing without c’s, etc. And then simply take the shorter
one.</p>
<pre><code>bruteForce :: String -&gt; Int
bruteForce polymers = minimum (map length allReduced)
  where
    types = ['a' .. 'z']
    allReduced = map (\c -&gt; reduce $ filter (\x -&gt; toLower x /= c) polymers) types
</code></pre>
<p>Winner winner chicken dinner.</p>
<p><a href="https://github.com/phaazon/advent-of-code-2k18/blob/master/day-05/src/Main.hs">Haskell solution</a></p>
<h1>Day 6: Chronal Coordinates</h1>
<p><a href="https://adventofcode.com/2018/day/6">Text</a></p>
<h2>Part 1</h2>
<p>That puzzle was one of the funniest I did. The idea is that, given an infinite 2D map, you are given
a list of several <em>points of interest</em> (POIs) in the form of <code>(x, y)</code> coordinates. The goal, for
this first part, is to find the largest zone in which all points have the same POI. What it means
is that, given several POIs, every positions on the map has a nearest POI (it can have several if
it’s at equal distance to several POIs – those must be discarded by the algorithm so that they do
not count into any zone). Several positions with the same nearest POI and adjacent to each others
form a zone, so that anyone in that zone knows that the nearest POI is the same accross all spanning
positions of the zone – you can picture the zone easily as discs centered on the POIs, but deformed
by other POIs.</p>
<p>The tricky part is that the map is infinite and POIs are scattered around it.</p>
<p>My idea was based on something I do <strong>a lot</strong> on my spare time with my 3D projects: compute AABBs.
An <a href="https://en.wikipedia.org/wiki/Bounding_volume">AABB</a> is an enclosing box in which all points lie. The idea is that its size must be as minimal
as possible. An <a href="https://en.wikipedia.org/wiki/Bounding_volume">AABB</a>, which stands for Axis-Aligned Bounding Box, is the minimal bounding volume
that enclose a set of points and which has its edged aligned with the axis (in our case, the X and
Y axis). By the way, an <a href="https://en.wikipedia.org/wiki/Bounding_volume">AABB</a> in 2D is also called <strong>MBR</strong>, which stands for Minimum Bounding
Rectangle. I chose AABB instead of MBR because I’m more used to work in 3D, but they’re essentially
the same thing.</p>
<p>So, the first thing I wanted to do is to compute the AABB of all the POIs for a two reasons:</p>
<ul>
<li>The most important one: it helped me reduce the problem space from an infinite space to a
finite one: instead of having an infinite 2D map, I now have a finite rectangular 2D region to
explore. Which is, you’ll have to admit, much more comfortable. ;)</li>
<li>If you read the puzzle’s text, you’ll have notice this:
<ul>
<li><em>“Your goal is to find the size of the largest area that isn't infinite.”</em></li>
<li>What it means is that all the points <strong>outside</strong> of the AABB are in an area that is always
infinite, so we will never be interested in those zones.</li>
</ul>
</li>
</ul>
<p>Ok, let’s see my code:</p>
<pre><code>type Point = (Int, Int)

-- Note to readers: this the way I like to encode AABB because it eases some operations on points.
-- The lower point is a point that belongs to the AABB that satisfies the rule that no other point
-- with different coordinate are lower than it. Same thing for upper. I also like that encoding
-- because generating an AABB from a set of points is trivial.
data AABB = AABB {
    aabbLower :: Point,
    aabbUpper :: Point
  } deriving (Eq, Show)

findAABB :: [Point] -&gt; AABB
findAABB [] = error &quot;nein&quot; -- this will never be called, so we don’t care about type safety here
findAABB (a:ls) = foldl' updateAABB (AABB a a) ls
  where
    updateAABB (AABB (lx, ly) (ux, uy)) (x, y) = AABB {
        aabbLower = (min (min lx x) lx, min (min ly y) ly),
        aabbUpper = (max (max ux x) ux, max (max uy y) uy)
      }

-- This function gives me a list of points that are in the AABB. It actually gives me all the points
-- the AABB wraps.
aabbToStream :: AABB -&gt; [Point]
aabbToStream (AABB (lx, ly) (ux, uy)) = [(x, y) | x &lt;- [lx .. ux], y &lt;- [ly .. uy]]

-- Test whether a point lies on any edges of the AABB. You’ll get why this function is important
-- later.
liesOnAABB :: Point -&gt; AABB -&gt; Bool
liesOnAABB (x, y) (AABB (lx, ly) (ux, uy)) = x == lx || x == ux || y == ly || y == uy
</code></pre>
<p>I annotated the code with comments so that you can get what it’s for.</p>
<p>So, I create the <code>AABB</code> and then I call <code>aabbToStream</code> in order to get all points. You might already
have guessed the next step: we are going to find the nearest POI to all the points. For this, I
just went naive and just computed the <a href="https://en.wikipedia.org/wiki/Taxicab_geometry">Manhattan distance</a> to all POI and kept the smallest. If we
map that function to all coordinates generated by the AABB, we get the first part of the solution.</p>
<pre><code>manhDist :: Point -&gt; Point -&gt; Int
manhDist (a, b) (c, d) = abs (a - c) + abs (b - d)

nearest :: Point -&gt; [(Int, Point)] -&gt; Maybe Int
nearest p points =
  case sortBy (comparing snd) $ map (\(i, x) -&gt; (i, manhDist p x)) points of
    [a] -&gt; Just (fst a)
    a:b:_ -&gt; if snd a == snd b then Nothing else Just (fst a) -- (1)
    _ -&gt; error &quot;nearest&quot;
</code></pre>
<p>Here, <code>(1)</code> applies the rule I described earlier about at least two POIs at the same distance: we
just discard the point and it doesn’t participate in creating any zone.</p>
<p>Then, how do we find the biggest area? Easy: we re-use our <code>freqTable</code> function from Day 4 to
compute a frequency table! In my case, I just renamed that function <code>freqs</code>:</p>
<pre><code>freqs :: (Ord a) =&gt; [a] -&gt; Map a Natural
freqs = fromListWith (+) . map (,1)
</code></pre>
<p>If we call that function on a list of <code>[Int]</code>, we end up with <code>Map (Maybe Int) Natural</code> that
gives us the number of positions a given POI is the nearest. It’s perfect, because it’s exactly
what we are looking for!</p>
<pre><code>biggestArea :: [Maybe Int] -&gt; Natural
biggestArea = snd . maximumBy (comparing snd) . M.toList . freqs . catMaybes
</code></pre>
<p>Here, <code>catMaybes</code> just remove the <code>Nothing</code> case so that we go from <code>[Maybe Int]</code> to <code>[Int]</code>. We
then find out which POI has the biggest number of nearest positions and we simply return it. Because
those are positions, their sum is then the area of the zone: we’re done. Or almost. Remember that
some zones have an infinite areas. Thanks to the AABB, it’s actually easy to find which ones: those
have at least one point that lies on the AABB’s edges. We just have to iterate through all the
points and black list some points:</p>
<pre><code>blackListPoints :: [(Point, Maybe Int)] -&gt; AABB -&gt; Set Int
blackListPoints points aabb = foldl' blacklist mempty points
  where
    blacklist blist (p, Just i) = if liesOnAABB p aabb then S.insert i blist else blist
blacklist blist _ = blist
</code></pre>
<h2>Part 2</h2>
<p>The part 2 asks something different: now we want to find the area of the region containing all
locations for which the total distance to all POI is less than a given constant (<code>10000</code>). My
solution was actually way easier than expected, surprisingly:</p>
<pre><code>safeArea = filter (\p -&gt; sum (map (manhDist p) coords) &lt;= 10000) points
</code></pre>
<p>Done. :)</p>
<p><a href="https://github.com/phaazon/advent-of-code-2k18/blob/master/day-06/src/Main.hs">Haskell solution</a></p>
<h1>Day 7: The Sum of Its Parts</h1>
<p><a href="https://adventofcode.com/2018/day/7">Text</a></p>
<h2>Part 1</h2>
<p>Here we go again: graph theory. Fortunately for us, it’s not a hard graph puzzle. That first part is
to simply display a string that shows the order in which a graph must be traversed. If two nodes can
be traversed at the same time, the node which letter comes first alphabetically is traversed first.</p>
<p>I’ll just show the traversal because the rest is not really interesting for that puzzle:</p>
<pre><code>-- The graph encodes the relationship in reverse: it maps each node its list of dependencies.
-- So if we have something like A -&gt; [], it means that the A node doesn’t have any dependency.
type Graph = Map Step (Set Step)
type Step = Char

-- Get the list of all available steps; i.e. they don’t have any dependency.
getAvailable :: Graph -&gt; [Step]
getAvailable gr = [step | (step, set) &lt;- M.toList gr, S.null set]

-- Traverse the graph and get the ordered steps to go through.
stepAvailable :: Graph -&gt; [Step]
stepAvailable gr = case sort (getAvailable gr) of
  [] -&gt; []
  (s:sx) -&gt; s : stepAvailable (removeStep s gr)

removeStep :: Step -&gt; Graph -&gt; Graph
removeStep s = purgeDep s . M.delete s
  where
    purgeDep = fmap . S.delete
</code></pre>
<p>It’s a typical functional problem that gets solved very easily in Haskell.</p>
<h2>Part 2</h2>
<p>The second part is pretty interesting. Instead of stepping through all the steps sequentially, you
ar given a <em>pool of workers</em>. It will take a given amount of time for a given worker to complete a
task and able us to visit a given node in the graph. We have to guess how many time it will take to
complete all of the steps.</p>
<p>I won’t post the code (it’s on GitHub if you want to have a look at it) as it’s a bit boring and the
idea of my solution is enough. The concept is to have a stepped simulation (i.e. you perform a set
of action in a given <em>“round”</em>, then repeat). In my case, each round is composed of several steps:</p>
<ol>
<li>First, partition the current work load into a set of <em>done tasks</em> and <em>running tasks</em>. This is
quite easy to do by just checking at the remaining time of each task. If the remaining time is
<code>0</code>, then it’s done, otherwise it’s still running.</li>
<li>Generate the <em>time increment</em>. This is the minimal duration until a next interesting action
occurs (i.e. a task gets done). Nothing can happen below that duration. That value can easily
be found by looking up the remaining duration of the running tasks and taking the minimum.</li>
<li>If we still have running tasks, step forward (i.e. recursively call the same function) by
advancing the current time by the time increment and removing the done tasks.</li>
<li>The <em>backlog</em>, that is implicit, can be created by monoidal operations and is detailed in the
code on GitHub.</li>
<li>When the backlog gets empty, we have the final time and the answer to the initial question.</li>
</ol>
<p>This part was interesting because it made me write a parallel graph traversal (a graph task
scheduler) that could be used as base for a real and parallelized (I/O) task scheduler. Interesting
stuff.</p>
<p><a href="https://github.com/phaazon/advent-of-code-2k18/blob/master/day-07/src/Main.hs">Haskell solution</a></p>
<h1>Day 8: Memory Maneuver</h1>
<p><a href="https://adventofcode.com/2018/day/8">Text</a></p>
<h2>Part 1</h2>
<p>In its simple form, this puzzle is not really interesting and I could definitely present you the
solution in a single paragraph. However, I found it pretty fun to do so I’ll go a bit further.</p>
<p>The problem is the following: we are given a list of numbers that represent a data structure. That
data structure is basically a tree with tagged metadata. The goal is to parse the list of numbers
to generate a memory representation of that tree and compute checksums on it. The structure is:</p>
<ul>
<li>A header, that has the number of children of a given node and the number of metadata entries.</li>
<li>Zero or many children.</li>
<li>At least one or many metadata entries.</li>
</ul>
<p>The file format is made so that the data are nested. I’ll copy and explain an example:</p>
<pre><code>2 3 0 3 10 11 12 1 1 0 1 99 2 1 1 2
A----------------------------------
    B----------- C-----------
                     D-----
</code></pre>
<p>Here, only the first line is present in the input file. The first <code>2</code> means that the first (<code>A</code>)
node has two children (we don’t know anything about them yet) and the <code>3</code> means it has three
metadata. Those are the header. Then, since it has two children, the next <code>0</code> is the start of the
header of its first children (<code>B</code>), which has no child and three metadata (<code>3</code>). The next <code>10</code>, <code>11</code>
and <code>12</code> are then those metadata (since it doesn’t have any child). This node is then complete. If
you go back up in the tree, you know that <code>A</code> still has another child. So the next number, <code>1</code>, is
the number of child of <code>C</code> and <code>1</code> its number of metadata. The next number <code>0</code> is the number of
child of <code>D</code> and it has <code>1</code> metadata, which is <code>99</code>. <code>C</code>, as seen above, has one metadata, so <code>2</code> is
<code>C</code>’s metadata. Then, since <code>A</code> has three metadata, <code>1</code>, <code>1</code> and <code>2</code> are its.</p>
<p>Pfiou. Seems hard to read for a human, right? However, if you’re used a bit to recursive data
structure and more specifically recursive parsers, this kind of encoding is actually pretty neat!</p>
<p>Let’s go and implement the parser of that tree. First, the structure. We will not need the header in
the output (it’s only used for parsing), so we will not encode that directly (it’s still available
as the length of the list of children and length of the list of metadata entries):</p>
<pre><code>data Node = Node {
    nodeChildren :: [Node],
    nodeMetadata :: NonEmpty Natural
  } deriving (Eq, Show)
</code></pre>
<p>Pretty simple, right? This is a self-recursing data structure that is pretty simple and basic to any
functional programmer.</p>
<blockquote>
<p>The <code>NonEmpty a</code> data type, in Haskell, is a list that cannot have zero element. That is enforced
at compilation as it’s impossible to create such a list without explicitly giving at least one
element. All the operations defined on <code>NonEmpty a</code> respect that rule (for instance, removing an
element from it might not remove anything if it has only one element – otherwise you’d break its
invariant and Haskell is not Javascript. <a href="https://phaazon.net/media/uploads/inner_troll.png">Lol</a>.</p>
</blockquote>
<p>So, how do we parse that? I’ve already spoiled you the solution: we need to implement a recursive
parser. I know that because I’ve been using <a href="http://hackage.haskell.org/package/parsec">parsec</a> for like 7 years now, so I’m pretty used to
that kind of parsing and as you use it, you will quickly recognize when you can use such an idiom.</p>
<p>However, instead of using <a href="http://hackage.haskell.org/package/parsec">parsec</a> directly, I will implement it myself with some very basic types
every Haskellers know – if you don’t: go learn them! I’ll be using the <code>State</code> type only, which is
basically <em>just</em> a recursive function used in a monadic fancy way:</p>
<pre><code>-- A possible representation of the State monad is just a function that takes a value 's' and
-- returns a new, altered 's' (we call that a state, but as you can see, it’s completely pure code,
-- no mutation happens at all) and an output value 'a'.
data State s a = State { runState :: s -&gt; (s, a) }
</code></pre>
<blockquote>
<p>The real <code>State</code> type is defined in the <a href="http://hackage.haskell.org/package/mtl-2.2.2/docs/Control-Monad-State-Lazy.html#t:State">mtl</a>
Haskell library.</p>
</blockquote>
<p>So here, you have the <em>impression</em> or <em>illusion</em> of <code>s</code> being a state, but what it truly is is just
a value that is passed around to a recursive function – and plot twist: recursion is the way to
implement locally-defined states, but I will not explain that (read my <a href="https://phaazon.net/blog/getting_into_netwire">netwire tutorial</a> and the
<code>Auto</code> type, if you are interested).</p>
<p>The functor / monadic part:</p>
<pre><code>instance Functor (State s) where
  fmap f = State . fmap (fmap f) . runState

instance Applicative (State s) where
  pure x = State (, x)
  p &lt;*&gt; q = State $ \s -&gt;
    let (s', f) = runState p s
        (s'', a) = runState q s'
    in (s'', f a)

instance Monad (State s) where
  return = pure
  q &gt;&gt;= f = State $ \s -&gt; let (s', a) = runState q s in runState (f a) s'
</code></pre>
<blockquote>
<p>All of this can be generated automatically by GHC with <code>deriving</code> data annotation.</p>
</blockquote>
<p>And some combinators we’ll need:</p>
<pre><code>-- Get the current value of the “state”.
get :: State s s
get = State $ \s -&gt; (s, s)

-- Get the current value of the “state” with a function pre-applied to it.
gets :: (s -&gt; a) -&gt; State s a
gets = flip fmap get

-- Change the value of the “state” by applying a function to the state.
modify :: (s -&gt; s) -&gt; State s ()
modify f = State $ \s -&gt; (f s, ())

-- Just a convenient method to just get the output value and discard the final state. You need the
-- initial value to use as state.
evalState :: State s a -&gt; s -&gt; a
evalState st = snd . runState st
</code></pre>
<p>So basically, since this is a very basic and simple code (I think all Haskellers should write that
in their first month using Haskell, it’s a good exercise), I just included the <code>mtl</code> library and
used its <code>State</code> type to write my recursive parser.</p>
<p>This is my parser:</p>
<pre><code>newtype Parser a = Parser { runParser :: State [Natural] a } deriving (Applicative, Functor, Monad)
</code></pre>
<p>So basically, a <code>Parser a</code> generates value of type <code>a</code> and maintains a list of <code>Natural</code> around.
Those <code>Natural</code> are the numbers from the input we are going to parse. Let’s write the actual parser
now.</p>
<pre><code>-- Turns the (string-encoded) list of numbers and generates the root node, that contains all of
-- the children.
parse :: String -&gt; Node
parse = evalState (runParser parseNode) . map read . words

-- Read a single number from the input and consume it from the state.
readInput :: Parser Natural
readInput = Parser $ gets head &lt;* modify tail

parseNode :: Parser Node
parseNode = do
  -- We read the two first numbers (header)
  childrenNb &lt;- readInput
  metadataNb &lt;- readInput

  -- Recursive parser! The NE.fromList is an unsafe function that is used for convenience for this
  -- puzzle part.
  children &lt;- replicateM (fromIntegral childrenNb) parseNode
  metadata &lt;- fmap NE.fromList (replicateM (fromIntegral metadataNb) readInput)

  pure $ Node children metadata
</code></pre>
<p>As you can see, the parser code is extremely simple with a recursive combinator parser! And we’re
actually done for the first part. The checksum is simple and is:</p>
<pre><code>checksum :: Node -&gt; Natural
checksum node = metadataChecksum (nodeMetadata node) + sum (map checksum $ nodeChildren node)

metadataChecksum :: NonEmpty Natural -&gt; Natural
metadataChecksum = sum . NE.toList
</code></pre>
<h2>Part 2</h2>
<p>The second part is not interesting as it just requires a new method to compute the “value” of a
given node:</p>
<pre><code>nodeValue :: Node -&gt; Natural
nodeValue (Node [] metadata) = metadataChecksum metadata
nodeValue (Node children metadata) = sum [nodeValue n | Just n &lt;- map index (NE.toList metadata)]
  where
    index i =
      let i' = fromIntegral i - 1
      in if i' &lt; 0 || i' &gt;= length children then Nothing else Just (children !! i')
</code></pre>
<p><a href="https://github.com/phaazon/advent-of-code-2k18/blob/master/day-08/src/Main.hs">Haskell solution</a></p>
<h1>Day 9: Marble Mania</h1>
<p><a href="https://adventofcode.com/2018/day/9">Text</a></p>
<h2>Part 1 &amp; 2</h2>
<p>This puzzle was the first one when I decided to go full Rust! All the remaining puzzles were solved
in Rust – if you were reading only for Haskell, sorry for your loss. :(</p>
<p>This puzzle is not really interesting as it’s just a fancy algorithm that adds element to a
collection and sometimes removes from it. There was a trick, though: the second part requires to run
our algorithm on an input that was a hundred times larger.</p>
<p>The typical trap is that when you add value in the middle of a collection, the complexity in terms
of memory and CPU can largely vary. Everything depends on what you do. For very rare additions /
deletions, it’s possible that you can accept <em>O(n)</em> complexities. However, if you often insert
stuff, you might want something else. In the same spirit, some data structure can efficiently add
in <em>O(1)</em> at the beginning or end of the collection or might require a complete copy.</p>
<p>Even though the puzzle is not interesting in itself, it reminds us how crucial and critical it is
that a programmer must know what structure to use depending on the inputs <strong>and operations that will
be performed on the data</strong>. In our case, we are going to add and remove <em>a lot</em> at arbitrary places
in the collection. Vectors are really bad candidates at that kind of operations, because they will
require a complete scan of the right part of the collection, which is <em>O(n)</em>, every time you add
or delete something (to shift right / left, respectively). This is bad. Vectors are also bad when
you want to add at its beginning (it requires the same right shift as the random case).</p>
<p>Double-ended queue (<a href="https://doc.rust-lang.org/std/collections/struct.VecDeque.html"><code>VecDeque</code></a> in
Rust) are a solution to the problem to insert at the beginning. That insertion is <em>O(1)</em> amortized.</p>
<p>My solution to this was to use a <a href="https://wiki.haskell.org/Zipper">zipper</a> to stay focus on a given number in the collection but also
“move around” in <em>O(1)</em>. The idea is the following:</p>
<pre><code>struct Zipper {
  left: VecDeque&lt;isize&gt;,
  middle: isize,
  right: VecDeque&lt;isize&gt;,
}
</code></pre>
<p>When you want to move to the left, you just take the <code>middle</code> number and <code>push_front</code> it to the
<code>right</code> double-ended queue and you <code>pop_back</code> the <code>left</code> one and that value becomes the new
<code>middle</code>. You do the opposite thing to go to the right.</p>
<p>To insert an element at the current location, you just <code>push_front</code> <code>middle</code> to the <code>right</code> and then
<code>middle</code> is assigned the new value. To remove, you just <code>pop_front</code> <code>right</code> into <code>middle</code>.</p>
<p>Then, the all puzzle is just adding and removing according to a predicate. Since all the operations
I mentioned above run in <em>O(1)</em> amortized (they might allocate if the buffer is too small), we will
not suffer from the typical <em>O(n²)</em> complexity a <code>Vec</code> implementation has.</p>
<p><a href="https://github.com/phaazon/advent-of-code-2k18/blob/master/day-09/src/main.rs">Rust solution</a></p>
<h1>Day 10: The Stars Align</h1>
<p><a href="https://adventofcode.com/2018/day/10">Text</a></p>
<h2>Part 1</h2>
<p>This is the kind of problem I suck the most at. Not because they’re hard. Because they’re easier
than expected. As an engineer, I tend to overthink about the context, the input’s hidden properties,
the possible errors, the heuristics, what could go wrong, etc. On a regular job basis, that is
actually a good thing – it’s better to foresee things than to have to repair them. However, at a
given extreme, that way of thinking will make you go nuts and will make you think of an easy and
straightforward problem as a complex and convoluted one. I know that and the main thing my mind does
when solving a problem is to think about how <strong>really hard</strong> a problem is. I just completely failed
on this one, haha. :D</p>
<p>So, you are given a short list (~360) of 2D points. You know nothing about how they’re spread nor
the limits they lie in. You <strong>only</strong> have ~360 points on a 2D plane. Those points, however, come
with two attributes:</p>
<ul>
<li>Their positions, as a pair of relative numbers ([<code>-∞; +∞]</code>).</li>
<li>Their (constant) velocities, as a pair of relative numbers as well ([<code>-∞; +∞]</code>).</li>
</ul>
<p>So basically, each point starts at a given position and goes into a straight line forever. The unit
of the velocity is not known and is not really needed – even though it might be <code>unit/s</code>.
Nevertheless, the text gives the hinting that, at a given (unknown) time, the points form a message
that can be visually witness. The goal is to give the message.</p>
<p>In the first 15 minutes, I went through several thoughts. “Whoa, they want us to write an <a href="https://en.wikipedia.org/wiki/Optical_character_recognition">OCR</a>?!
Really?!”. Nah, you dumbass. Since we all have a unique input (and hence, a unique expected output),
we don’t have to write an algorithm that can recognize any text. We just have to get to visualize
our input.</p>
<p>However, when does the text appear? At <code>t = 0</code>, we have a large cloud of spread points. We don’t
know when the text will form. Also, the challenge explicitly states that the text forms only once:
the points will never gather into text afterwards. We must not miss it then.</p>
<p>My idea was that to find hidden properties of the overall text first. By being able to extract a
useful information telling me whether or not I’m far or close from having a visualizable text, I was
able to run a loop-like simulation, moving each points by its respectiv velocities, until that
hidden information reaches a local minimum. As an engineer, I was annoyed by that, because I had no
idea whether the first local minimum was the right one – the puzzle’s text doesn’t state anything
about that and I had not found any information to help with that in the input. I could also use the
wrong criteria (maybe we’re looking for a local maximum?). I got stuck with those ideas for long
minutes.</p>
<p>Finally, I decided to implement a specific criteria:</p>
<ul>
<li>At each simulation step, we compute the <a href="https://en.wikipedia.org/wiki/Bounding_volume">AABB</a> that encloses all the input points that have
moved.</li>
<li>If that <a href="https://en.wikipedia.org/wiki/Bounding_volume">AABB</a>’s <a href="https://en.wikipedia.org/wiki/Area">area</a> is less then the previous one, we store the <a href="https://en.wikipedia.org/wiki/Area">area</a>, the width and height
of the <a href="https://en.wikipedia.org/wiki/Bounding_volume">AABB</a> and the current time of the simulation. We also print those values on the standard
output.</li>
<li>If not, we continue.</li>
</ul>
<p>When I ran that loop, I got the first local minimum in <strong>10011</strong> seconds. Clearly, if you tried to
actually run that simulation with the real time, you’d be waiting for a long time – <strong>10011</strong>
seconds is 2 hours, 46 minutes and 51 seconds.</p>
<p>The size of the <a href="https://en.wikipedia.org/wiki/Bounding_volume">AABB</a> at <code>t = 10011</code> was also pretty small (around <strong>60×60</strong>). I then decided to
display the message directly in the console. In order to do that, I had to transform my 2D points
(expressed in the natural ℝ² basis we use in <em>world space</em> coordinates) into a space that I could
easily use to display (basically, <code>[0; w]</code> and <code>[0; h]</code>). That transformation is done with the
following code:</p>
<pre><code>// The rendered “map”
let mut lol = vec!['.'; w as usize * h as usize];

for p in points {
  let x = (p.position.0 - aabb.lower.0) * (w - 1) / w;
  let y = (p.position.1 - aabb.lower.1) * (h - 1) / h;
  let o = x + y * w;

  lol[o as usize] = '#';
}
</code></pre>
<p>Then, we just need to iterate on all the points and render them to the terminal to finish the
challenge:</p>
<pre><code>for row in 0 .. h {
  for col in 0 .. w {
    print!(&quot;{}&quot;, lol[(col + row * w) as usize]);
  }

  println!(&quot;&quot;);
}
</code></pre>
<h2>Part 2</h2>
<p>Part 2 was almost a joke: we were asked to give the time at which the text appeared. As this was a
<em>hidden property</em> to find in the first place, completing part 2 took a few seconds: <code>10011</code>.</p>
<p><a href="https://github.com/phaazon/advent-of-code-2k18/tree/master/day-10/src/main.rs">Rust solution</a></p>
<h1>Day 11: Chronal Charge</h1>
<p><a href="https://adventofcode.com/2018/day/11">Text</a></p>
<h2>Part 1</h2>
<p>A pretty common algorithm to implement: sliding window. Basically, you are given a matrix of numbers
and you have to compute several sums using a sliding <em>kernel</em> which size is <em>3×3</em>. The size of the
matrix is <em>300×300</em> and you just want to compute the biggest <em>3×3</em> square (and give its index in the
matrix as row / column).</p>
<p>This was my solution:</p>
<pre><code>let mut largest = (0, i8::min_value()); // (index, power)

// 298 is 300 - 2: we want to stop there so that the 3×3 square won’t overflow
for row in 0 .. 298 {
  for col in 0 .. 298 {
    let mut power = 0;

    // sum the square
    for i in 0 .. 3 {
      for k in 0 .. 3 {
        power += grid[index(col + i, row + k)];
      }
    }

    let i = index(col, row);

    // if its power is any larger, store it along with its index
    if (power == largest.1 &amp;&amp; i &lt; largest.0) || power &gt; largest.1 {
      largest = (i, power);
    }
  }
}

println!(&quot;Largest fuel cell: ({}, {})&quot;, 1 + largest.0 % 300, 1 + largest.0 / 300);
</code></pre>
<p>That’s pretty much it. Second part is more interesting.</p>
<h2>Part 2</h2>
<p>For this part, the problem changes a bit: we still want to sum squares, but we want to get the find
the square that has the largest total power of any size comprised between <em>1×1</em> and <em>300×300</em> – we
want its index and its size.</p>
<p>That problem can be solved in several ways, with different complexities. It’s easy to see that you
can quickly go with a bad complexity if you decide to refactor the previous algorithm to take a
dimension (that will be squared) and call it 300 times. Maybe that would be enough.</p>
<p>However, I wanted to implement something smarter on this one. It’s easy to see that a lot of
spanning squares will overlap. For instance:</p>
<pre><code>+-+-+-+-+-+
|0|1|2|3|4|
+-+-+-+-+-+
|6|7|8|9|A|
+-+-+-+-+-+
|B|C|D|E|F|
+-+-+-+-+-+
</code></pre>
<p>If you consider the first, top-leftmost <em>2×2</em> square:</p>
<pre><code>+-+-+
|0|1|
+-+-+
|6|7|
+-+-+
</code></pre>
<p>And the top-left-mostmost <em>3×3</em> square:</p>
<pre><code>+-+-+-+
|0|1|2|
+-+-+-+
|6|7|8|
+-+-+-+
|B|C|D|
+-+-+-+
</code></pre>
<p>You can see that a the smaller one is included in the bigger one. What it means is that each
spanning square is a partial sum to spanning square of a higher dimension. My algorithm benefits
from that in order to reduce the number of elements to sum at each given dimension.</p>
<p>Also, another thing I did that suprised people on IRC: I reversed the way the algorithm works in
terms of traversal. Instead of traversing dimensions and then traversing the grid (for all
dimensions then for all squares), I traverse the grid and then I traverse the dimensions (for all
square in the grid, for all the dimensions). This gives me a more natural way to write the partial
sums in my code.</p>
<p>Finally, I also work out some formalæ to know “what’s the biggest dimension we can go up to given
the current grid cell.” Yeah, think twice: when you want to go through all dimensions from the
top-leftmost cell, you will be able to sum squares from <em>1×1</em> up to <em>300×300</em>. But which dimension
can you go to when the starting (<em>1×1</em>) cell is in the middle of the grid? This is actually pretty
easy. The formalæ can be found very quickly by thinking in terms of size of the grid (<em>300×300</em>) and
the index of a grid cell. The biggest dimension is just the minimum of the maximal allowed row
iterations and the maximal allowed column iterations. You can picture that mentally by “which edge
I am the closest to?”. For rows, it’s simply <code>300 - current_row</code> and for columns,
<code>300 - current_column</code>. The minimum value gives you the maximal spanning dimension you can go up to.</p>
<p>Finally, a word on how the partial sums are created: when you start computing the sum of the <code>N</code>
dimension, you already have the partial sum of dimension <code>N-1</code> (the algorithm starts with the first
dimension set to a given value). Then, instead of summing <code>N²</code> element, since you already have the
sum of <code>(N-1)²</code>, you just need to sum <em>2 × (N - 1) + 1</em> values. If you’re not convinced, at
dimension <code>278</code>, <code>278² = 77284</code> sums while my algorithm is <code>2 × (278 - 1) + 1 = 555</code> sums. It’s
around <strong>139 times less</strong>.</p>
<p>In my code, I do that by adding – relative to the previous spanning square – the right column
(which size is <em>N - 1</em>), the bottom line (<em>N - 1</em> as well) and the single element in the diagonal.
Hence <code>2 × (N - 1) + 1</code>. And that completes a new partial sum, that will be used for higher
dimensions!</p>
<p>Here’s just a very quick schema to show you how to compute the sum at dimension <code>5</code> by using the
sum of the spanning square of dimension <code>4</code> – <code>·</code> is already computed and <code>R</code> are the right column,
<code>B</code> the bottom line and <code>D</code> the element in the diagonal:</p>
<pre><code>+-+-+-+-+-+
|·|·|·|·|R|
+-+-+-+-+-+
|·|·|·|·|R|
+-+-+-+-+-+
|·|·|·|·|R|
+-+-+-+-+-+
|·|·|·|·|R|
+-+-+-+-+-+
|B|B|B|B|D|
+-+-+-+-+-+
</code></pre>
<p>So, here’s the code:</p>
<pre><code>let mut largest2 = (0, i64::min_value(), 0); // (index, power, dimension)

// for all rows…
for row in 0 .. 300 {
  let max_iter_row = 300 - row; // 300 -&gt; 1

  // for all columns…
  for col in 0 .. 300 {
    let max_iter_col = 300 - col; // 300 -&gt; 1
    let max_dim_squared = max_iter_row.min(max_iter_col); // 300x300 -&gt; 1x1

    // power used for nested dimensions
    let mut nested_power = grid[index(col, row)] as i64;

    // note: we don’t have to compute the first dimension because it’s set right away to the given
    // nested power

    // for all dimensions up to the max
    for d in 1 .. max_dim_squared {
      let mut power = nested_power;

      // compute the 2 × (N - 1) elements
      for k in 0 .. d {
        power += grid[index(col + d, row + k)] as i64;
        power += grid[index(col + k, row + d)] as i64;
      }

      // add the diagonal
      power += grid[index(col + d, row + d)] as i64;

      let i = index(col, row);

      if (power == largest2.1 &amp;&amp; i &lt; largest2.0) || power &gt; largest2.1 {
        largest2 = (index(col, row), power, d + 1);
      }

      nested_power = power;
    }
  }
}

println!(&quot;Largest fuel cell of all: ({}, {}, {}, of power {})&quot;, 1 + largest2.0 % 300, 1 + largest2.0 / 300, largest2.2, largest2.1);
</code></pre>
<p><a href="https://github.com/phaazon/advent-of-code-2k18/tree/master/day-11/src/main.rs">Rust solution</a></p>
<h1>Day 12: Subterranean Sustainability</h1>
<p><a href="https://adventofcode.com/2018/day/12">Text</a></p>
<h2>Part 1</h2>
<p>This puzzle looked a bit like the double-ended queue one from day 9. The extra bit of information
is that you now have to apply a pattern on several values to know how they should be mutated. Given
a list of flower pots and some rules that give you how a pot should grow flowers (or not) according
to the state of itself and its neighbors, the goal is to predict the sum of the pots (their index in
the list) for all pots that contain flowers after <strong>20 generations</strong>.</p>
<p>In the first place, I had to recognize that I needed a double-ended queue. As always, the puzzle’s
text doesn’t explicitly tell you that the pots at leftmost and rightmost positions can “spawn” new
pots by applying the rules on empty pots (infinite). I was confused at that for a while.</p>
<p>My encoding of rules is pretty simple and wasteful: since a rule gives you a pattern (which pots)
and an output (should have flowers / shouldn’t), a single byte should be enough for that (the length
of a rule is five: it gives you the state of the two left neighbors, the state of the current pot
and the state of the two right neighbors). However, I encoded those with a simple array of five
binary states (<code>Rule::Empty</code> and <code>Rule::Pot</code>).</p>
<p>In order to apply a pattern, we must retreive the current pot and two at its left position and two
at its right position (if pots are missing because we’re at edges, we must spawn empty pots with
negative / positive indices). Then we can just apply the pattern by looking it up in a hashmap: we
get the next generation value.</p>
<p>Nothing really interesting code-wise to show here.</p>
<h2>Part 2</h2>
<p>Part 2 is funny. We’re told that instead of finding the sums after <strong>20 generations</strong>, we need to
find it <strong>after fifty billion (50000000000) generations.</strong> Obviously, trying to run the above
algorithm for 50000000000 generations will take ages, so we need to find a better way.</p>
<p>My first inital idea was that if I took a look at the actual sum value at each generation, I could
– perhaps – see some kind of patterns. At first I was looking for cycles and hence cycling sums. I
then run my algorithm and had a look at the output data. I was suprised to find that, very quickly,
the flowers grow linearily. What it means is that, after a given number of generations, you can
guess how many flowers there will be at a given future generation by applying a linear formula
(typically, a simple multiplication and addition).</p>
<p>In my case, I noticed that at generation <code>100</code>, the sum was <code>6346</code>. At <code>101</code>, it was <code>6397</code>. At
<code>102</code>, it was <code>6448</code>. At <code>200</code>, it was <code>16546</code>. You can see the pattern – if you don’t, compute the
difference between the sum at <code>101</code> and the sum at <code>100</code>… and the difference of sum at <code>102</code> and
<code>101</code>.</p>
<p>Hence, I came up with the following linear formula:</p>
<pre><code>// O(1) get the score at a given generation – works only for gen ≥ 100.
fn score_at(gen: u64) -&gt; u64 {
  6346 + (gen - 100) * 51
}
</code></pre>
<p>The actual implementation uses <code>101</code> instead of <code>100</code> because we want to get the sum <strong>after</strong> a
given number of generations, not <strong>at</strong>.</p>
<p>That kind of linear optimization was really fun to write yet a bit tricky to find. :)</p>
<p><a href="https://github.com/phaazon/advent-of-code-2k18/blob/master/day-12/src/main.rs">Rust solution</a></p>
<h1>Day 13: Mine Cart Madness</h1>
<p><a href="https://adventofcode.com/2018/day/13">Text</a></p>
<h2>Part 1</h2>
<p>I think this was the puzzle I enjoyed the most – among the ones I did. The goal is to parse is rails
map on which wagons go and make wagons move around by respecting some specific rules: we must find
wagon collision is report their positions 2D position.</p>
<p>Parsing is actually pretty simple: the input data is the 2D map that contains the rail system along
with the initial position of wagons. About the map, I decided to store only crossings, not the
actual rails, because I didn’t need them! So in order to do so, I changed a bit the regular way I
encode 2D maps in memory and decided to use a hashmap!</p>
<pre><code>#[derive(Clone, Copy, Debug, Eq, Hash, PartialEq)]
enum Rail {
  Cross, // +
  RampRight, // /
  RampLeft // \
}

struct Map(HashMap&lt;(u32, u32), Rail&gt;);
</code></pre>
<p>The rest of the code is actually pretty simple: there are several functions, one for moving carts,
one for changing directions at cross, one for detecting collision. A main <code>loop</code> is responsible in
moving carts and checking if there’s any collision. If no collision is detected, we just loop back
up. If a detection is detected, we break the loop and display the position of the crash. The
collision algorith returns the IDs of the carts that collided into each other.</p>
<pre><code>let collision = loop {
  // sort the cart by y component
  carts.sort_by(|a, b| a.pos.cmp(&amp;b.pos));

  let collisions = move_carts(&amp;map, &amp;mut carts);
  if !collisions.is_empty() {
    break collisions[0];
  }
};

println!(&quot;First collision: {:?}&quot;, carts[collision.0].pos);
</code></pre>
<p>The <code>sort_by</code> is needed because of priority rules in the puzzle’s text.</p>
<p>Moving carts and detecting collision is pretty straightforward:</p>
<pre><code>fn move_carts(
  map: &amp;Map,
  carts: &amp;mut Vec&lt;Cart&gt;,
) -&gt; Vec&lt;(usize, usize)&gt; {
  let mut collisions: Vec&lt;(usize, usize)&gt; = Vec::new(); // no collision to begin with

  // move and check collision for all carts
  'outer: for i in 0 .. carts.len() {
    // check that this cart hasn’t been collided into yet
    for &amp;collision in &amp;collisions {
      if i == collision.0 || i == collision.1 {
        // already collided, don’t move that
        continue 'outer;
      }
    }

    // move the cart and check if it’s collided into another
    move_cart(map, &amp;mut carts[i]);
    let collision = find_collision(&amp;carts, i);

    if let Some(collider) = collision {
      collisions.push((collider, i));
    }
  }

  collisions
}
</code></pre>
<p>This code is not really optimized – we redo the same thing very often – but it’s way than enough to
solve that puzzle’s part. Finding collision is very simple: we just try to find a cart with the
same position.</p>
<p>The tricky part is for moving at cross. The rules state that if you arrive at a cross, you have to
turn in a given direction and change the future direction you will take at the future cross, if any.
This was encoded inside each cart, so that they have a <em>“memory”</em> of turns to take.</p>
<pre><code>#[derive(Debug)]
struct Cart {
  pos: (u32, u32),
  dir: Direction,
  next_turn: Turn
}
</code></pre>
<p>A cart starts by going on its (relative!) <code>Turn::Left</code>, then at the next turn it will go
<code>Turn::Straight</code>, then <code>Turn::Right</code> and finaly will loop back to <code>Turn::Left</code>. Note how different
it is to <code>Direction</code>: a <code>Turn</code> is relative to the current movement of a cart while a <code>Direction</code> is
absolute (at first, I wanted to have <code>North</code>, <code>East</code> etc. for <code>Direction</code> so that <a href="https://kaamelott-soundboard.2ec0b4.fr/#son/de_tout_facon_on_dit_le_nord_selon_comment_on_est_tourne_ca_change_tout">confusion is not
possible</a>).</p>
<h2>Part 2</h2>
<p>In that part, we want to find the last standing cart, assuming that crashing carts are immediately
removed from the map. The code is actually very similar: instead of breaking the loop at the first
collision, we break it when there’s only one cart left on the map – we don’t forget te remove the
crashed carts!</p>
<pre><code>loop {
  // sort the cart by y component
  carts.sort_by(|a, b| a.pos.cmp(&amp;b.pos));

  for (ci, ck) in move_carts(&amp;map, &amp;mut carts) {
    carts = carts.into_iter().enumerate().filter(|&amp;(i, _)| i != ci &amp;&amp; i != ck).map(|(_, c)| c).collect();
  }

  if carts.len() == 1 {
    break;
  }
};

println!(&quot;Last standing cart: {:?} &quot;, carts); // this contains only one cart
</code></pre>
<p><a href="https://github.com/phaazon/advent-of-code-2k18/blob/master/day-13/src/main.rs">Rust solution</a></p>
<h1>Day 14: Chocolate Charts</h1>
<p><a href="https://adventofcode.com/2018/day/14">Text</a></p>
<h2>Part 1</h2>
<p>Very similar to the double-ended queue puzzle as well, this one doesn’t actually require any
deletion, just indexing correctly into a growing buffer. Nothing really interesting to show about
this problem, except maybe the way recipes are created.</p>
<blockquote>
<p>To create new recipes, the two Elves combine their current recipes. This creates new recipes from
the digits of the sum of the current recipes' scores. With the current recipes' scores of 3 and 7,
their sum is 10, and so two new recipes would be created: the first with score 1 and the second
with score 0. If the current recipes' scores were 2 and 3, the sum, 5, would only create one
recipe (with a score of 5) with its single digit.</p>
</blockquote>
<p>In order to implement that, I recognized that I will always create at least one recipe: <code>0 + 0 = 0</code>,
and as soon as I create two recipes, the other one will always be <code>1</code>, because the maximum value is
<code>9 + 9 = 18 (1 and 8)</code>. Here’s my code that gets those two recipe numbers:</p>
<pre><code>fn create_new_recipe(a: usize, b: usize) -&gt; (Option&lt;usize&gt;, usize) {
  let s = a + b;
  let x = s / 10;
  let y = s % 10;

  (if x == 1 { Some(x) } else { None }, y)
}
</code></pre>
<h2>Part 2</h2>
<p>Part 2 is really not interesting as it’s just using <code>ends_with</code> to find a suffix. I’ll let you read
the code if you’re interested.</p>
<p><a href="https://github.com/phaazon/advent-of-code-2k18/blob/master/day-14/src/main.rs">Rust solution</a></p>
<h1>Day 15: Beverage Bandits</h1>
<p><a href="https://adventofcode.com/2018/day/15">Text</a></p>
<h2>Part 1</h2>
<p>Aaaaaaaaaand this is the last puzzle I attempted. I actually decided not to finish it because it was
taking me time. I will tell you more about that in the conclusion.</p>
<p>The goal is to write a simulation of elves fighting goblins (or goblins fighting elves) and finding
paths in a map that has holes / mountains in it. So most of the code to write is about <a href="https://en.wikipedia.org/wiki/Dijkstra%27s_algorithm">Dijkstra</a> or
<a href="https://en.wikipedia.org/wiki/A*_search_algorithm">A*</a>. The puzzle seemed interesting but it was way too much for my spare time to spend on. I advise
you to have a look at the insanely long puzzle’s text – that will give you an idea of everything you
need to implement in order to get the your solution working.</p>
<h1>Conclusion</h1>
<p>Ah, my first Advent of Code. It was both interesting, exciting, frustrating and time-consuming. I
found several pros and drawbacks:</p>
<p>Pros, first:</p>
<ul>
<li>It’s all fun. Since I live in France, I couldn’t compete with people who get the puzzles
unlocked in the morning around 10:00 AM while they were unlocked around 5:00 AM in France – I
have a job, I have to, you know… SLEEP! So I just did it for fun.</li>
<li>Writing algorithms to solve math and operational problems is always interesting and makes it a
great training.</li>
<li>Comparing solutions on IRC was also pretty fun!</li>
</ul>
<p>And drawbacks:</p>
<ul>
<li>The fact west-europeans cannot compete (the timezone issue).</li>
<li>The difficulty of the puzzles is not correctly balanced. Some puzzle are insanely simple to
solve and some others take you hours (if not days if you don’t have a lot of spare time).
Sometimes I felt frustrated at this, because I’ve been lacking sleep time for weeks and had to
go to bed in order not to destroy my physical health. I’m curious to hear about other
west-europeans: how did you manage your time, social life and work life with AoC?</li>
<li>Some problems had a very long first part 2 and the second part was really stupid (read
<a href="#part-2-8">The Stars Align – Part 2</a>). I would have preferred to have more consistency or more
parts, for instance.</li>
</ul>
<p>My general feeling is that it was fun, but I think that I won’t do it next year, because I had to
put all my spare projects on hold for that. I didn’t learn anything new – all the algorithms had me
write algorithms I already knew, except maybe the partial dimension squared algorithm I “invented”:
someone told me that it’s very similar to a real and well-known algorithm! How funny is that! The
algorithm is <a href="https://en.wikipedia.org/wiki/Summed-area_table">Summed-area table</a> and my solution
is, indeed, very similar to it. But the thing is: I came up with the idea, and this is priceless for
training brains!</p>
<p>Now I’ll return to my graphics, Rust and over experiment projects of mine! I hope you liked that
article, it was a bit long (it took me almost two weeks to write!) but I felt I needed to make it.
To… well… scrap and forget about Advent of Code and all my spare time I didn’t use for my own
projects. :)</p>
<p>Keep the vibes – especially you, <a href="https://github.com/lqd">@lqd</a>.</p>


