Jekyll2024-02-09T04:50:37+00:00https://model-checking.github.io//kani-verifier-blog/feed.xmlKani Rust Verifier BlogThe Kani Rust Verifier (Kani) is an open-source automated reasoning tool that proves properties about Rust code. This is our blog for the project.Function Contracts for Kani2024-01-29T00:00:00+00:002024-01-29T00:00:00+00:00https://model-checking.github.io//kani-verifier-blog/2024/01/29/function-contracts<p>In this blogpost we discuss function contracts which are now available as an unstable feature, enabled with the <code class="language-plaintext highlighter-rouge">-Zfunction-contracts</code> flag. If you would like to learn more about the development and implementation details of this feature please refer to <a href="https://model-checking.github.io/kani/rfc/rfcs/0009-function-contracts.html">the RFC</a>. If you try out this new feature and want to leave feedback join the discussion in <a href="https://github.com/model-checking/kani/issues/2652">the feature tracking issue</a>.</p>
<h2 id="introduction">Introduction</h2>
<p>Today we want to introduce you to a new feature in Kani that lets us verify larger programs: function contracts <sup id="fnref:eiffel-1" role="doc-noteref"><a href="#fn:eiffel-1" class="footnote" rel="footnote">1</a></sup><sup id="fnref:eiffel-2" role="doc-noteref"><a href="#fn:eiffel-2" class="footnote" rel="footnote">2</a></sup>. Contracts let us safely break down the verification of a complex piece of code individually by function and efficiently compose the results, driving down the cost of verifying code with long call chains and repeated calls to the same function. This technique is called <em>modular verification</em> as the verification task is broken down into modules (in this case by function), verified independently and then recombined.</p>
<p>The best example for how function contracts improve verification time are recursive functions. With a contract a recursive function can be verified in a single step, a technique called <em>inductive verification</em>. In this post we will explore how function contracts can be used to modularize the verification of a harness in the <a href="https://firecracker-microvm.github.io/">Firecracker</a> Virtual Machine Monitor by modularly verifying an implementation of Euclid’s greatest common divisor algorithm inductively and then use that result to verify the harness.</p>
<p>Aside: we’ve actually blogged twice previously about the exciting verification work we’ve been doing on the Firecracker project. You can find those posts <a href="/2023/08/31/using-kani-to-validate-security-boundaries-in-aws-firecracker.html">here</a> and <a href="/2022/07/13/using-the-kani-rust-verifier-on-a-firecracker-example.html">here</a>.</p>
<h2 id="the-example-firecrackers-tokenbucket">The Example: Firecracker’s <code class="language-plaintext highlighter-rouge">TokenBucket</code></h2>
<p>We will explore employing a function contract when verifying the methods of <code class="language-plaintext highlighter-rouge">TokenBucket</code> in the <code class="language-plaintext highlighter-rouge">vmm</code> module of the Firecracker Virtual Machine Manager. To keep the example concise we will use the relatively simple <code class="language-plaintext highlighter-rouge">TokenBucket::new</code> function and its verification harness. The code is slightly simplified and we also use a recursive implementation of <code class="language-plaintext highlighter-rouge">gcd</code>. Firecracker actually uses a <code class="language-plaintext highlighter-rouge">gcd</code> with a loop which would need loop contracts to verify inductively. These are not supported yet by Kani, but conceptually they function the same way as function contracts. You can find the original version of all harnesses and verified code <a href="https://github.com/firecracker-microvm/firecracker/blob/a774e26d63981fc031b974741a39519b08a61d3b/src/vmm/src/rate_limiter/mod.rs">here</a>.</p>
<p>Let’s first look at the harness that tests <code class="language-plaintext highlighter-rouge">TokenBucket::new</code>. It is straightforward, setting up a series of non-deterministic inputs, calling the function under verification and then asserting a validity condition.</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">#[kani::proof]</span>
<span class="k">fn</span> <span class="nf">verify_token_bucket_new</span><span class="p">()</span> <span class="p">{</span>
<span class="k">let</span> <span class="n">size</span> <span class="o">=</span> <span class="nn">kani</span><span class="p">::</span><span class="nf">any_where</span><span class="p">(|</span><span class="n">s</span><span class="p">|</span> <span class="o">*</span><span class="n">s</span> <span class="o">!=</span> <span class="mi">0</span><span class="p">);</span>
<span class="k">let</span> <span class="n">one_time_burst</span> <span class="o">=</span> <span class="nn">kani</span><span class="p">::</span><span class="nf">any</span><span class="p">();</span>
<span class="k">let</span> <span class="n">complete_refill_time_ms</span> <span class="o">=</span> <span class="nn">kani</span><span class="p">::</span><span class="nf">any_where</span><span class="p">(|</span><span class="n">t</span><span class="p">|</span>
<span class="o">*</span><span class="n">t</span> <span class="o">!=</span> <span class="mi">0</span> <span class="o">&&</span> <span class="o">*</span><span class="n">t</span> <span class="o"><</span> <span class="nn">u64</span><span class="p">::</span><span class="n">MAX</span> <span class="o">/</span> <span class="n">NANOSEC_IN_ONE_MILLISEC</span>
<span class="p">);</span>
<span class="k">let</span> <span class="n">bucket</span> <span class="o">=</span> <span class="nn">TokenBucket</span><span class="p">::</span><span class="nf">new</span><span class="p">(</span><span class="n">size</span><span class="p">,</span> <span class="n">one_time_burst</span><span class="p">,</span> <span class="n">complete_refill_time_ms</span><span class="p">)</span><span class="nf">.unwrap</span><span class="p">()</span>
<span class="k">assert</span><span class="o">!</span><span class="p">(</span><span class="n">bucket</span><span class="nf">.is_valid</span><span class="p">());</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Since the most important call here is to <code class="language-plaintext highlighter-rouge">TokenBucket::new</code> let’s take a closer look at its implementation.</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">pub</span> <span class="k">fn</span> <span class="nf">new</span><span class="p">(</span><span class="n">size</span><span class="p">:</span> <span class="nb">u64</span><span class="p">,</span> <span class="n">one_time_burst</span><span class="p">:</span> <span class="nb">u64</span><span class="p">,</span> <span class="n">complete_refill_time_ms</span><span class="p">:</span> <span class="nb">u64</span><span class="p">)</span> <span class="k">-></span> <span class="nb">Option</span><span class="o"><</span><span class="n">Self</span><span class="o">></span> <span class="p">{</span>
<span class="k">if</span> <span class="n">size</span> <span class="o">==</span> <span class="mi">0</span> <span class="p">||</span> <span class="n">complete_refill_time_ms</span> <span class="o">==</span> <span class="mi">0</span> <span class="p">{</span>
<span class="k">return</span> <span class="nb">None</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">let</span> <span class="n">complete_refill_time_ns</span> <span class="o">=</span>
<span class="n">complete_refill_time_ms</span><span class="nf">.checked_mul</span><span class="p">(</span><span class="n">NANOSEC_IN_ONE_MILLISEC</span><span class="p">)</span><span class="o">?</span><span class="p">;</span>
<span class="k">let</span> <span class="n">common_factor</span> <span class="o">=</span> <span class="nf">gcd</span><span class="p">(</span><span class="n">size</span><span class="p">,</span> <span class="n">complete_refill_time_ns</span><span class="p">);</span>
<span class="k">let</span> <span class="n">processed_capacity</span><span class="p">:</span> <span class="nb">u64</span> <span class="o">=</span> <span class="n">size</span> <span class="o">/</span> <span class="n">common_factor</span><span class="p">;</span>
<span class="k">let</span> <span class="n">processed_refill_time</span><span class="p">:</span> <span class="nb">u64</span> <span class="o">=</span> <span class="n">complete_refill_time_ns</span> <span class="o">/</span> <span class="n">common_factor</span><span class="p">;</span>
<span class="nf">Some</span><span class="p">(</span><span class="n">TokenBucket</span> <span class="p">{</span>
<span class="n">size</span><span class="p">,</span>
<span class="n">one_time_burst</span><span class="p">,</span>
<span class="n">initial_one_time_burst</span><span class="p">:</span> <span class="n">one_time_burst</span><span class="p">,</span>
<span class="n">refill_time</span><span class="p">:</span> <span class="n">complete_refill_time_ms</span><span class="p">,</span>
<span class="n">budget</span><span class="p">:</span> <span class="n">size</span><span class="p">,</span>
<span class="n">last_update</span><span class="p">:</span> <span class="nn">Instant</span><span class="p">::</span><span class="nf">now</span><span class="p">(),</span>
<span class="n">processed_capacity</span><span class="p">,</span>
<span class="n">processed_refill_time</span><span class="p">,</span>
<span class="p">})</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Most of what happens here is just a series of divisions, with the exception of this call: <code class="language-plaintext highlighter-rouge">gcd(size, complete_refill_time_ns)</code> so let us look at that function next:</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">fn</span> <span class="nf">gcd</span><span class="p">(</span><span class="k">mut</span> <span class="n">max</span><span class="p">:</span> <span class="nb">u64</span><span class="p">,</span> <span class="k">mut</span> <span class="n">min</span><span class="p">:</span> <span class="nb">u64</span><span class="p">)</span> <span class="k">-></span> <span class="nb">u64</span> <span class="p">{</span>
<span class="k">if</span> <span class="n">min</span> <span class="o">></span> <span class="n">max</span> <span class="p">{</span>
<span class="nn">std</span><span class="p">::</span><span class="nn">mem</span><span class="p">::</span><span class="nf">swap</span><span class="p">(</span><span class="o">&</span><span class="k">mut</span> <span class="n">max</span><span class="p">,</span> <span class="o">&</span><span class="k">mut</span> <span class="n">min</span><span class="p">);</span>
<span class="p">}</span>
<span class="k">let</span> <span class="n">rest</span> <span class="o">=</span> <span class="n">max</span> <span class="o">%</span> <span class="n">min</span><span class="p">;</span>
<span class="k">if</span> <span class="n">rest</span> <span class="o">==</span> <span class="mi">0</span> <span class="p">{</span> <span class="n">min</span> <span class="p">}</span> <span class="k">else</span> <span class="p">{</span> <span class="nf">gcd</span><span class="p">(</span><span class="n">min</span><span class="p">,</span> <span class="n">rest</span><span class="p">)</span> <span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>This is by far the costliest part of the code of <code class="language-plaintext highlighter-rouge">TokenBucket::new</code>. The rest of the function was a fixed set of divisions, but here we have a division and a recursive call back to <code class="language-plaintext highlighter-rouge">gcd</code>. It is not immediately obvious how busy this computation is, but in the <a href="https://en.wikipedia.org/wiki/Euclidean_algorithm#Worst-case">worst case</a> the number of steps (recursions) approaches 1.5 times the number of bits needed to represent the input numbers. Meaning that for two large 64-bit numbers it can take almost 96 iterations for a single call to <code class="language-plaintext highlighter-rouge">gcd</code>. Recall that our smallest input to <code class="language-plaintext highlighter-rouge">TokenBucket::new</code> is a non-deterministic value in the range $0< x < 18446744073709$ (up to 45 non-zero bits). If a harness that uses <code class="language-plaintext highlighter-rouge">gcd</code> does not heavily constrain the input, Kani would have to unroll the recursion at least 68 times and then execute it symbolically; an expensive operation.</p>
<p>Since <code class="language-plaintext highlighter-rouge">gcd</code> is the most expensive part of verifying this harness it is an ideal target for using a contract to make it more efficient. In fact by replacing it with a contract we not only make this harness more efficient but also any harness that uses <code class="language-plaintext highlighter-rouge">gcd</code> directly or indirectly (e.g., by calling <code class="language-plaintext highlighter-rouge">TokenBucket::new</code>). In Firecracker, for instance, there are three more harnesses that all call <code class="language-plaintext highlighter-rouge">gcd</code> and thus benefit from modular verification.</p>
<h2 id="introducing-a-function-contract-with-postconditions">Introducing a Function Contract with Postconditions</h2>
<p>Function contracts are conceptually rather simple actually. They comprise a set of conditions which characterize the behavior of the function, similar to the conditions you would use in a test case. There are different types of conditions which are also called the <em>clauses</em> of the contract. The first type of clause we will introduce here is the <code class="language-plaintext highlighter-rouge">ensures</code> clause.</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">#[kani::ensures(max</span> <span class="err">%</span> <span class="nd">result</span> <span class="nd">==</span> <span class="mi">0</span> <span class="err">&&</span> <span class="nd">min</span> <span class="err">%</span> <span class="nd">result</span> <span class="nd">==</span> <span class="mi">0</span> <span class="err">&&</span> <span class="nd">result</span> <span class="err">!</span><span class="nd">=</span> <span class="mi">0</span><span class="nd">)]</span>
<span class="k">fn</span> <span class="nf">gcd</span><span class="p">(</span><span class="k">mut</span> <span class="n">max</span><span class="p">:</span> <span class="nb">u64</span><span class="p">,</span> <span class="k">mut</span> <span class="n">min</span><span class="p">:</span> <span class="nb">u64</span><span class="p">)</span> <span class="k">-></span> <span class="nb">u64</span> <span class="p">{</span>
<span class="k">if</span> <span class="n">min</span> <span class="o">></span> <span class="n">max</span> <span class="p">{</span>
<span class="nn">std</span><span class="p">::</span><span class="nn">mem</span><span class="p">::</span><span class="nf">swap</span><span class="p">(</span><span class="o">&</span><span class="k">mut</span> <span class="n">max</span><span class="p">,</span> <span class="o">&</span><span class="k">mut</span> <span class="n">min</span><span class="p">);</span>
<span class="p">}</span>
<span class="k">let</span> <span class="n">rest</span> <span class="o">=</span> <span class="n">max</span> <span class="o">%</span> <span class="n">min</span><span class="p">;</span>
<span class="k">if</span> <span class="n">rest</span> <span class="o">==</span> <span class="mi">0</span> <span class="p">{</span> <span class="n">min</span> <span class="p">}</span> <span class="k">else</span> <span class="p">{</span> <span class="nf">gcd</span><span class="p">(</span><span class="n">min</span><span class="p">,</span> <span class="n">rest</span><span class="p">)</span> <span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>The <code class="language-plaintext highlighter-rouge">ensures</code> clause describes the relationship of the return of the function (e.g. <code class="language-plaintext highlighter-rouge">result</code>) with the arguments that the function was called with. It is often also called a <em>postcondition</em>, because it is a <em>condition</em> that must hold after (<em>post</em>) the execution of the function. With Kani, the contents of the <code class="language-plaintext highlighter-rouge">ensures</code> clause can be any Rust expression that returns <code class="language-plaintext highlighter-rouge">bool</code>. However the expression may not perform any <em>side effects</em>, that is: allocate, deallocate or modify heap memory or perform I/O. A single function may have multiple <code class="language-plaintext highlighter-rouge">ensures</code> clauses which functions as though they had been joined with <code class="language-plaintext highlighter-rouge">&&</code>. Our example could thus also have been written as</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">#[kani::ensures(max</span> <span class="err">%</span> <span class="nd">result</span> <span class="nd">==</span> <span class="mi">0</span><span class="nd">)]</span>
<span class="nd">#[kani::ensures(min</span> <span class="err">%</span> <span class="nd">result</span> <span class="nd">==</span> <span class="mi">0</span><span class="nd">)]</span>
<span class="nd">#[kani::ensures(result</span> <span class="err">!</span><span class="nd">=</span> <span class="mi">0</span><span class="nd">)]</span>
<span class="k">fn</span> <span class="nf">gcd</span><span class="p">(</span><span class="k">mut</span> <span class="n">max</span><span class="p">:</span> <span class="nb">u64</span><span class="p">,</span> <span class="k">mut</span> <span class="n">min</span><span class="p">:</span> <span class="nb">u64</span><span class="p">)</span> <span class="k">-></span> <span class="nb">u64</span> <span class="p">{</span> <span class="o">...</span> <span class="p">}</span>
</code></pre></div></div>
<p>You may wonder at this point what this is actually useful for. It turns out that for a verifier the <em>abstraction</em> of the function as described by the conditions in our <code class="language-plaintext highlighter-rouge">ensures</code> clause, is much easier to reason about than running through the many recursive calls in the actual implementation of <code class="language-plaintext highlighter-rouge">gcd</code>. You may also notice that the abstraction only approximates <code class="language-plaintext highlighter-rouge">gcd</code>. <code class="language-plaintext highlighter-rouge">max % result == 0</code> and <code class="language-plaintext highlighter-rouge">min % result == 0</code> describe <em>a</em> common divisor of <code class="language-plaintext highlighter-rouge">max</code> and <code class="language-plaintext highlighter-rouge">min</code> but not necessarily the <em>greatest</em>. However in this case this is acceptable to us when it comes to verifying the callers of <code class="language-plaintext highlighter-rouge">gcd</code>. Later we will add an additional check to ensure we generate the largest divisor.</p>
<p>Our goal will be to eventually use the efficient abstraction everywhere where <code class="language-plaintext highlighter-rouge">gcd</code> is called but first we must ensure that <code class="language-plaintext highlighter-rouge">gcd</code> respects these conditions. We need to <em>verify the contract</em>.</p>
<p>Doing so is the same as verifying that the postcondition(s) hold for any possible input to the function. If we were to manually create a harness that does this verification it would look like this</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">#[kani::proof]</span>
<span class="k">fn</span> <span class="nf">gcd_stub_check</span><span class="p">()</span> <span class="p">{</span>
<span class="c">// First we create any possible input using a</span>
<span class="c">// non-deterministic value</span>
<span class="k">let</span> <span class="n">max</span> <span class="o">=</span> <span class="nn">kani</span><span class="p">::</span><span class="nf">any</span><span class="p">();</span>
<span class="k">let</span> <span class="n">min</span> <span class="o">=</span> <span class="nn">kani</span><span class="p">::</span><span class="nf">any</span><span class="p">();</span>
<span class="c">// Sadly necessary for performance right now.</span>
<span class="nn">kani</span><span class="p">::</span><span class="nf">assume</span><span class="p">(</span><span class="n">max</span> <span class="o"><=</span> <span class="mi">255</span> <span class="o">&&</span> <span class="n">min</span> <span class="o"><=</span> <span class="mi">255</span><span class="p">);</span>
<span class="c">// We create any possible result</span>
<span class="k">let</span> <span class="n">result</span> <span class="o">=</span> <span class="nf">gcd</span><span class="p">(</span><span class="n">max</span><span class="p">,</span> <span class="n">min</span><span class="p">);</span>
<span class="k">assert</span><span class="o">!</span><span class="p">(</span>
<span class="c">// And here is our postcondition</span>
<span class="n">max</span> <span class="o">%</span> <span class="n">result</span> <span class="o">==</span> <span class="mi">0</span> <span class="o">&&</span> <span class="n">min</span> <span class="o">%</span> <span class="n">result</span> <span class="o">==</span> <span class="mi">0</span> <span class="o">&&</span> <span class="n">result</span> <span class="o">!=</span> <span class="mi">0</span>
<span class="p">);</span>
<span class="p">}</span>
</code></pre></div></div>
<p><strong>Caveat:</strong> In this harness we’ve had to constrain the size of <code class="language-plaintext highlighter-rouge">min</code> and <code class="language-plaintext highlighter-rouge">max</code>. This is not because of contracts but because there are currently performance problems with verifying unconstrained <code class="language-plaintext highlighter-rouge">u64</code> in CBMC. Doing this makes the harness unsound, which we explain in more detail in <a href="#soundness-and-comparison-with-stubbing">this section</a>.</p>
<p>If we run this harness however we will discover a verification failure, because of a division by <code class="language-plaintext highlighter-rouge">0</code> in <code class="language-plaintext highlighter-rouge">let rest = max % min;</code> which brings us to the introduction of the second type of clause: <code class="language-plaintext highlighter-rouge">requires</code>.</p>
<h2 id="preconditions">Preconditions</h2>
<p>Sometimes functions, such as our <code class="language-plaintext highlighter-rouge">gcd</code> are not defined for all of their possible input (e.g. they panic on some). Function contracts let us express this using a condition that that the function arguments must satisfy at the beginning (<em>pre</em>) of a function’s execution: a <em>precondition</em>. This condition limits the inputs for which a contract will be checked during verification, and it will also be used to ensure that if we use the contract conditions for modular verification, we don’t do it with any values that the function would not be defined for. We can add a precondition buy using the <code class="language-plaintext highlighter-rouge">requires</code> clause in <code class="language-plaintext highlighter-rouge">gcd</code> like so</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">#[kani::requires(max</span> <span class="err">!</span><span class="nd">=</span> <span class="mi">0</span> <span class="err">&&</span> <span class="nd">min</span> <span class="err">!</span><span class="nd">=</span> <span class="mi">0</span><span class="nd">)]</span>
<span class="nd">#[kani::ensures(max</span> <span class="err">%</span> <span class="nd">result</span> <span class="nd">==</span> <span class="mi">0</span> <span class="err">&&</span> <span class="nd">min</span> <span class="err">%</span> <span class="nd">result</span> <span class="nd">==</span> <span class="mi">0</span> <span class="err">&&</span> <span class="nd">result</span> <span class="err">!</span><span class="nd">=</span> <span class="mi">0</span><span class="nd">)]</span>
<span class="k">fn</span> <span class="nf">gcd</span><span class="p">(</span><span class="k">mut</span> <span class="n">max</span><span class="p">:</span> <span class="nb">u64</span><span class="p">,</span> <span class="k">mut</span> <span class="n">min</span><span class="p">:</span> <span class="nb">u64</span><span class="p">)</span> <span class="k">-></span> <span class="nb">u64</span> <span class="p">{</span> <span class="o">...</span> <span class="p">}</span>
</code></pre></div></div>
<p>As with postconditions, the precondition allows any side-effect free rust expressions of type <code class="language-plaintext highlighter-rouge">bool</code>, may mention the function arguments and multiple <code class="language-plaintext highlighter-rouge">requires</code> clauses act as though they were joined with <code class="language-plaintext highlighter-rouge">&&</code>.</p>
<p>To understand how this makes our contract verification succeed, let’s integrate it into the manual harness we had written before.</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">#[kani::proof]</span>
<span class="k">fn</span> <span class="nf">gcd_stub_check</span><span class="p">()</span> <span class="p">{</span>
<span class="c">// First we create any possible input using a</span>
<span class="c">// non-deterministic value</span>
<span class="k">let</span> <span class="n">max</span> <span class="o">=</span> <span class="nn">kani</span><span class="p">::</span><span class="nf">any</span><span class="p">();</span>
<span class="k">let</span> <span class="n">min</span> <span class="o">=</span> <span class="nn">kani</span><span class="p">::</span><span class="nf">any</span><span class="p">();</span>
<span class="c">// Sadly necessary for performance right now.</span>
<span class="nn">kani</span><span class="p">::</span><span class="nf">assume</span><span class="p">(</span><span class="n">max</span> <span class="o"><=</span> <span class="mi">255</span> <span class="o">&&</span> <span class="n">min</span> <span class="o"><=</span> <span class="mi">255</span><span class="p">);</span>
<span class="c">// Limit the domain of inputs with precondition</span>
<span class="nn">kani</span><span class="p">::</span><span class="nf">assume</span><span class="p">(</span><span class="n">max</span> <span class="o">!=</span> <span class="mi">0</span> <span class="o">&&</span> <span class="n">min</span> <span class="o">!=</span> <span class="mi">0</span><span class="p">);</span>
<span class="c">// We create any possible result</span>
<span class="k">let</span> <span class="n">result</span> <span class="o">=</span> <span class="nf">gcd</span><span class="p">(</span><span class="n">max</span><span class="p">,</span> <span class="n">min</span><span class="p">);</span>
<span class="k">assert</span><span class="o">!</span><span class="p">(</span>
<span class="c">// And here is our postcondition</span>
<span class="n">max</span> <span class="o">%</span> <span class="n">result</span> <span class="o">==</span> <span class="mi">0</span> <span class="o">&&</span> <span class="n">min</span> <span class="o">%</span> <span class="n">result</span> <span class="o">==</span> <span class="mi">0</span> <span class="o">&&</span> <span class="n">result</span> <span class="o">!=</span> <span class="mi">0</span>
<span class="p">);</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Running this in Kani succeeds, giving us confidence that now our contract properly approximates the effects of <code class="language-plaintext highlighter-rouge">gcd</code>.</p>
<p>You may be wondering now why we’ve even written the <code class="language-plaintext highlighter-rouge">requires</code> and <code class="language-plaintext highlighter-rouge">ensures</code> clause, given that we verified it using a harness we wrote ourselves. Well in actuality Kani will do most of it for you. We will see <a href="#a-bit-of-cleanup">later</a> exactly how little is needed to verify the contract, for now you may assume that Kani does it for you automatically.</p>
<h2 id="using-contracts">Using Contracts</h2>
<p>We have proved that our function upholds the contract we specified so now we can use the abstraction at the call sites.</p>
<p>If we cast our mind back to the implementation of <code class="language-plaintext highlighter-rouge">TokenBucket::new</code> we can replace the function as follows:</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">pub</span> <span class="k">fn</span> <span class="nf">new</span><span class="p">(</span><span class="n">size</span><span class="p">:</span> <span class="nb">u64</span><span class="p">,</span> <span class="n">one_time_burst</span><span class="p">:</span> <span class="nb">u64</span><span class="p">,</span> <span class="n">complete_refill_time_ms</span><span class="p">:</span> <span class="nb">u64</span><span class="p">)</span> <span class="k">-></span> <span class="nb">Option</span><span class="o"><</span><span class="n">Self</span><span class="o">></span> <span class="p">{</span>
<span class="k">if</span> <span class="n">size</span> <span class="o">==</span> <span class="mi">0</span> <span class="p">||</span> <span class="n">complete_refill_time_ms</span> <span class="o">==</span> <span class="mi">0</span> <span class="p">{</span>
<span class="k">return</span> <span class="nb">None</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">let</span> <span class="n">complete_refill_time_ns</span> <span class="o">=</span>
<span class="n">complete_refill_time_ms</span><span class="nf">.checked_mul</span><span class="p">(</span><span class="n">NANOSEC_IN_ONE_MILLISEC</span><span class="p">)</span><span class="o">?</span><span class="p">;</span>
<span class="c">// Make sure the precondtions are respected</span>
<span class="k">assert</span><span class="o">!</span><span class="p">(</span><span class="n">size</span> <span class="o">!=</span> <span class="mi">0</span> <span class="o">&&</span> <span class="n">complete_refill_time_ns</span> <span class="o">!=</span> <span class="mi">0</span><span class="p">);</span>
<span class="c">// Create a non-deterministic value for the result</span>
<span class="k">let</span> <span class="n">common_factor</span> <span class="o">=</span> <span class="nn">kani</span><span class="p">::</span><span class="nf">any</span><span class="p">();</span>
<span class="c">// Assume that the postconditions hold (as we know they would)</span>
<span class="nn">kani</span><span class="p">::</span><span class="nf">assume</span><span class="p">(</span>
<span class="n">size</span> <span class="o">%</span> <span class="n">common_factor</span> <span class="o">==</span> <span class="mi">0</span>
<span class="o">&&</span> <span class="n">complete_refill_time_ns</span> <span class="o">%</span> <span class="n">common_factor</span> <span class="o">==</span> <span class="mi">0</span>
<span class="o">&&</span> <span class="n">common_factor</span> <span class="o">!=</span> <span class="mi">0</span>
<span class="p">);</span>
<span class="k">let</span> <span class="n">processed_capacity</span><span class="p">:</span> <span class="nb">u64</span> <span class="o">=</span> <span class="n">size</span> <span class="o">/</span> <span class="n">common_factor</span><span class="p">;</span>
<span class="k">let</span> <span class="n">processed_refill_time</span><span class="p">:</span> <span class="nb">u64</span> <span class="o">=</span> <span class="n">complete_refill_time_ns</span> <span class="o">/</span> <span class="n">common_factor</span><span class="p">;</span>
<span class="nf">Some</span><span class="p">(</span><span class="n">TokenBucket</span> <span class="p">{</span>
<span class="n">size</span><span class="p">,</span>
<span class="n">one_time_burst</span><span class="p">,</span>
<span class="n">initial_one_time_burst</span><span class="p">:</span> <span class="n">one_time_burst</span><span class="p">,</span>
<span class="n">refill_time</span><span class="p">:</span> <span class="n">complete_refill_time_ms</span><span class="p">,</span>
<span class="n">budget</span><span class="p">:</span> <span class="n">size</span><span class="p">,</span>
<span class="n">last_update</span><span class="p">:</span> <span class="nn">Instant</span><span class="p">::</span><span class="nf">now</span><span class="p">(),</span>
<span class="n">processed_capacity</span><span class="p">,</span>
<span class="n">processed_refill_time</span><span class="p">,</span>
<span class="p">})</span>
<span class="p">}</span>
</code></pre></div></div>
<p>We have now replaced the potential 68 unrollings and multiplications of <code class="language-plaintext highlighter-rouge">gcd</code> with a single check and one assumption, all of which the verifier can reason about efficiently.</p>
<p>Of course we wouldn’t replace the calls to <code class="language-plaintext highlighter-rouge">gcd</code> by hand. In fact we are not going to make any changes to the code under verification. Instead we will instruct Kani to perform this replacement for us using a new attribute for the harness: <code class="language-plaintext highlighter-rouge">stub_verified</code>.</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">#[kani::proof]</span>
<span class="nd">#[kani::stub_verified(gcd)]</span>
<span class="k">fn</span> <span class="nf">verify_token_bucket_new</span><span class="p">()</span> <span class="p">{</span>
<span class="k">let</span> <span class="n">size</span> <span class="o">=</span> <span class="nn">kani</span><span class="p">::</span><span class="nf">any_where</span><span class="p">(|</span><span class="n">s</span><span class="p">|</span> <span class="o">*</span><span class="n">s</span> <span class="o">!=</span> <span class="mi">0</span><span class="p">);</span>
<span class="k">let</span> <span class="n">one_time_burst</span> <span class="o">=</span> <span class="nn">kani</span><span class="p">::</span><span class="nf">any</span><span class="p">();</span>
<span class="k">let</span> <span class="n">complete_refill_time_ms</span> <span class="o">=</span> <span class="nn">kani</span><span class="p">::</span><span class="nf">any_where</span><span class="p">(|</span><span class="n">t</span><span class="p">|</span>
<span class="o">*</span><span class="n">t</span> <span class="o">!=</span> <span class="mi">0</span> <span class="o">&&</span> <span class="o">*</span><span class="n">t</span> <span class="o"><</span> <span class="nn">u64</span><span class="p">::</span><span class="n">MAX</span> <span class="o">/</span> <span class="n">NANOSEC_IN_ONE_MILLISEC</span>
<span class="p">);</span>
<span class="k">let</span> <span class="n">bucket</span> <span class="o">=</span> <span class="nn">TokenBucket</span><span class="p">::</span><span class="nf">new</span><span class="p">(</span><span class="n">size</span><span class="p">,</span> <span class="n">one_time_burst</span><span class="p">,</span> <span class="n">complete_refill_time_ms</span><span class="p">)</span><span class="nf">.unwrap</span><span class="p">()</span>
<span class="k">assert</span><span class="o">!</span><span class="p">(</span><span class="n">bucket</span><span class="nf">.is_valid</span><span class="p">());</span>
<span class="p">}</span>
</code></pre></div></div>
<p>This attribute works similarly to <code class="language-plaintext highlighter-rouge">kani::stub</code> which we’ve explored in a <a href="https://model-checking.github.io/kani-verifier-blog/2023/02/28/kani-internship-projects-2022-stubbing.html">previous post</a>. In short, the attribute replaces each call to <code class="language-plaintext highlighter-rouge">gcd</code> that is reachable from the harness with new code. In fact, under the hood, <code class="language-plaintext highlighter-rouge">stub_verified</code> uses <code class="language-plaintext highlighter-rouge">stub</code> but instead of replacing <code class="language-plaintext highlighter-rouge">gcd</code> by some arbitrary function, it is replaced with the abstraction from the contract, the same way we did before manually.</p>
<h2 id="inductive-verification">Inductive Verification</h2>
<p>The discerning reader might already notice that, if we can replace every call to <code class="language-plaintext highlighter-rouge">gcd</code> with the contract, then would we be able to do this also <em>within</em> <code class="language-plaintext highlighter-rouge">gcd</code>? The answer is <strong>yes</strong> we can, and it will allow us to completely skip unrolling <code class="language-plaintext highlighter-rouge">gcd</code>, even when verifying the implementation against its contract. This technique is called <em>inductive verification</em> and it substantially improves the performance of verifying recursive functions. Side note: there are also loop contracts which enable inductive verification for loops. They are not yet supported in Kani.</p>
<p>You may be skeptical as to how this can work. As mentioned before, replacing the call to <code class="language-plaintext highlighter-rouge">gcd</code> is only safe because we previously verified the function against its contract so how would it be safe to already use the replacement <em>while</em> we’re still verifying the contract? Let us first look at what a manual harness for inductive verification of <code class="language-plaintext highlighter-rouge">gcd</code> would look like and step through it to convince ourselves that this is valid.</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">#[kani::proof]</span>
<span class="k">fn</span> <span class="nf">gcd_expanded_inductive_check</span><span class="p">()</span> <span class="k">-></span> <span class="nb">i32</span> <span class="p">{</span>
<span class="c">// Unconstrainted, non-deterministic inputs</span>
<span class="k">let</span> <span class="n">max</span> <span class="o">=</span> <span class="nn">kani</span><span class="p">::</span><span class="nf">any</span><span class="p">();</span>
<span class="k">let</span> <span class="n">min</span> <span class="o">=</span> <span class="nn">kani</span><span class="p">::</span><span class="nf">any</span><span class="p">();</span>
<span class="c">// Sadly necessary for performance right now.</span>
<span class="nn">kani</span><span class="p">::</span><span class="nf">assume</span><span class="p">(</span><span class="n">max</span> <span class="o"><=</span> <span class="mi">255</span> <span class="o">&&</span> <span class="n">min</span> <span class="o"><=</span> <span class="mi">255</span><span class="p">);</span>
<span class="k">let</span> <span class="n">result</span> <span class="o">=</span> <span class="p">{</span>
<span class="c">// Preconditions</span>
<span class="nn">kani</span><span class="p">::</span><span class="nf">assume</span><span class="p">(</span><span class="n">max</span> <span class="o">!=</span> <span class="mi">0</span> <span class="o">&&</span> <span class="n">min</span> <span class="o">!=</span> <span class="mi">0</span><span class="p">);</span>
<span class="c">// Inlined first execution of `gcd`</span>
<span class="k">if</span> <span class="n">min</span> <span class="o">></span> <span class="n">max</span> <span class="p">{</span>
<span class="nn">std</span><span class="p">::</span><span class="nn">mem</span><span class="p">::</span><span class="nf">swap</span><span class="p">(</span><span class="o">&</span><span class="k">mut</span> <span class="n">max</span><span class="p">,</span> <span class="o">&</span><span class="k">mut</span> <span class="n">min</span><span class="p">);</span>
<span class="p">}</span>
<span class="k">let</span> <span class="n">rest</span> <span class="o">=</span> <span class="n">max</span> <span class="o">%</span> <span class="n">min</span><span class="p">;</span>
<span class="k">if</span> <span class="n">rest</span> <span class="o">==</span> <span class="mi">0</span> <span class="p">{</span> <span class="n">min</span> <span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
<span class="c">// Inlined recursion</span>
<span class="k">let</span> <span class="n">max</span> <span class="o">=</span> <span class="n">min</span><span class="p">;</span>
<span class="k">let</span> <span class="n">min</span> <span class="o">=</span> <span class="n">rest</span><span class="p">;</span>
<span class="c">// Make sure preconditions are respected for recursion</span>
<span class="k">assert</span><span class="o">!</span><span class="p">(</span><span class="n">max</span> <span class="o">!=</span> <span class="mi">0</span> <span class="o">&&</span> <span class="n">min</span> <span class="o">!=</span> <span class="mi">0</span><span class="p">);</span>
<span class="k">let</span> <span class="n">result</span> <span class="o">=</span> <span class="nn">kani</span><span class="p">::</span><span class="nf">any</span><span class="p">();</span>
<span class="nn">kani</span><span class="p">::</span><span class="nf">assume</span><span class="p">(</span><span class="n">max</span> <span class="o">%</span> <span class="n">result</span> <span class="o">==</span> <span class="mi">0</span> <span class="o">&&</span> <span class="n">min</span> <span class="o">%</span> <span class="n">result</span> <span class="o">==</span> <span class="mi">0</span> <span class="o">&&</span> <span class="n">result</span> <span class="o">!=</span> <span class="mi">0</span><span class="p">);</span>
<span class="n">result</span>
<span class="p">}</span>
<span class="p">};</span>
<span class="c">// Make sure postconditions hold</span>
<span class="k">assert</span><span class="o">!</span><span class="p">(</span><span class="n">max</span> <span class="o">%</span> <span class="n">result</span> <span class="o">==</span> <span class="mi">0</span> <span class="o">&&</span> <span class="n">min</span> <span class="o">%</span> <span class="n">result</span> <span class="o">==</span> <span class="mi">0</span> <span class="o">&&</span> <span class="n">result</span> <span class="o">!=</span> <span class="mi">0</span><span class="p">);</span>
<span class="p">}</span>
</code></pre></div></div>
<p>First it is helpful to think about the recursion as a sequence of individual steps that build on top of one another, like a pyramid. Each step performs the same computation, but with different inputs, because the previous step calls into it with <code class="language-plaintext highlighter-rouge">min</code> and <code class="language-plaintext highlighter-rouge">rest</code>, which differ from <code class="language-plaintext highlighter-rouge">max</code> and <code class="language-plaintext highlighter-rouge">min</code>. Now consider the first step. It is called with non-deterministic inputs that are only constrained by the precondition. Then a computation is performed to calculate <code class="language-plaintext highlighter-rouge">min</code> and <code class="language-plaintext highlighter-rouge">rest</code> that is used for the recursive call and at the end the postconditions are enforced with <code class="language-plaintext highlighter-rouge">assert!(max % result == 0 && ...)</code>. What does that mean? It means if this step passes verification we can be sure that the postconditions hold for the inputs we verified with, e.g. any combination of <code class="language-plaintext highlighter-rouge">max</code> and <code class="language-plaintext highlighter-rouge">min</code> that satisfies <code class="language-plaintext highlighter-rouge">max != 0 && min != 0</code>. Now let’s consider again the recursive call. We call with <code class="language-plaintext highlighter-rouge">min</code> and <code class="language-plaintext highlighter-rouge">rest</code> which are not the same as <code class="language-plaintext highlighter-rouge">max</code> and <code class="language-plaintext highlighter-rouge">min</code>. However remember that actually what our first step verification proves not just that the postconditions hold for <em>a</em> <code class="language-plaintext highlighter-rouge">max</code> and <code class="language-plaintext highlighter-rouge">min</code>, but in fact for <em>any</em> <code class="language-plaintext highlighter-rouge">max</code> and <code class="language-plaintext highlighter-rouge">min</code>, so long as the precondition is satisfied. Therefore we can conclude that it will also hold for <code class="language-plaintext highlighter-rouge">gcd(min, rest)</code>, if <code class="language-plaintext highlighter-rouge">min</code> and <code class="language-plaintext highlighter-rouge">rest</code> satisfy the preconditions, e.g. <code class="language-plaintext highlighter-rouge">min != 0 && rest != 0</code>. And that is precisely what is enforced where the abstraction is employed with <code class="language-plaintext highlighter-rouge">assert!(max != 0 && min != 0)</code>.</p>
<p>As a small technicality: induction is usually split into the so called “base case” and the “induction step”. In our example a single run of the function actually combines both, which is the <code class="language-plaintext highlighter-rouge">if rest == 0</code> split. When <code class="language-plaintext highlighter-rouge">rest == 0</code> then we have reached the base case, otherwise we are in the induction step. Both the base case and the induction step must uphold the ponstconditions. The way that our proof is set up here already ensures that because the postcondition <code class="language-plaintext highlighter-rouge">assert!</code> gets enforced regardless of whether we took the base case or the induction step.<sup id="fnref:return" role="doc-noteref"><a href="#fn:return" class="footnote" rel="footnote">3</a></sup></p>
<p>To put it another way, when we recurse, we are allowed to assume that the verification passed, because the step that we are currently verifying is being checked for any possible input, including the one we are recursing with. If a problem were to occur anywhere in the recursive calls, we would see the same problem as a verification failure of the first step.</p>
<p>Notice that using this trick, we reduced the up to 68 unrollings to just one, a substantial win. What is even better that doing this requires no input from you. Kani does this automatically. Any function with a contract is always verified inductively. Side note: this works even if the recursive call is not direct but buried somewhere in the call chain.</p>
<h2 id="a-bit-of-cleanup">A Bit of Cleanup</h2>
<p>Before we close with the example, a few more details about the contract verification process. We mentioned earlier that Kani does most of the contract verification work for you. It injects the pre- and postconditions and sets up the inductive verification. However a small amount of manual but important labor is required by the user. Currently, Kani is not able to generate the non-deterministic inputs and so it requires the user to write a simple harness. In this case it would look like this:</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">#[kani::proof_for_contract(gcd)]</span>
<span class="k">fn</span> <span class="nf">gcd_stub_check</span><span class="p">()</span> <span class="p">{</span>
<span class="c">// Create non-determinstic values</span>
<span class="k">let</span> <span class="n">max</span> <span class="o">=</span> <span class="nn">kani</span><span class="p">::</span><span class="nf">any</span><span class="p">();</span>
<span class="k">let</span> <span class="n">min</span> <span class="o">=</span> <span class="nn">kani</span><span class="p">::</span><span class="nf">any</span><span class="p">();</span>
<span class="nf">gcd</span><span class="p">(</span><span class="n">max</span><span class="p">,</span> <span class="n">min</span><span class="p">)</span>
<span class="p">}</span>
</code></pre></div></div>
<p>This may be surprising to you since, in our example, we use a straightforward <code class="language-plaintext highlighter-rouge">kani::any()</code>. Indeed for finite types like <code class="language-plaintext highlighter-rouge">u64</code> that implement <a href="https://model-checking.github.io/kani/tutorial-nondeterministic-variables.html"><code class="language-plaintext highlighter-rouge">Arbitrary</code></a> Kani <em>could</em> generate this harness automatically. However for any type involving references or pointers (like <code class="language-plaintext highlighter-rouge">Vec</code>) Kani is unable to generate the harness. Because it would be confusing if sometimes the harness is generated automatically and sometimes not, we decided that for the time being a manual harness is always required. We are working on adding auto generation in the future.</p>
<p>A few words about writing contract harnesses: the harness should generate completely unconstrained values, otherwise the verification will be unsound. Usually this means calling <code class="language-plaintext highlighter-rouge">kani::any()</code>. Sometimes it is not possible to create completely unconstrained values, as is the case with recursive types and array-backed types. In these cases you must use careful judgement and ensure that the value you create is large enough to exercise all possible behaviors of the function. You can think of this as covering all branches.</p>
<p>Finally we mentioned we would be adding a check to ensure that our <code class="language-plaintext highlighter-rouge">gcd</code> actually produces the greatest divisor. For this we will need a separate harness and in this case we sadly cannot use any contract substitution. The reason is that, if we substitute, we get values that satisfy the postcondition, but nothing more. Since the postcondition does not ensure the result is the largest divisor possible, it won’t be.</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">#[kani::proof]</span>
<span class="k">fn</span> <span class="nf">gcd_greatest_check</span><span class="p">()</span> <span class="p">{</span>
<span class="c">// Create non-determinstic values</span>
<span class="k">let</span> <span class="n">max</span> <span class="o">=</span> <span class="nn">kani</span><span class="p">::</span><span class="nf">any</span><span class="p">();</span>
<span class="k">let</span> <span class="n">min</span> <span class="o">=</span> <span class="nn">kani</span><span class="p">::</span><span class="nf">any</span><span class="p">();</span>
<span class="k">let</span> <span class="n">result</span> <span class="o">=</span> <span class="nf">gcd</span><span class="p">(</span><span class="n">max</span><span class="p">,</span> <span class="n">min</span><span class="p">);</span>
<span class="k">let</span> <span class="n">greatest</span> <span class="o">=</span> <span class="nn">kani</span><span class="p">::</span><span class="nf">any_where</span><span class="p">(|</span><span class="n">a</span><span class="p">|</span>
<span class="n">max</span> <span class="o">%</span> <span class="o">*</span><span class="n">a</span> <span class="o">==</span> <span class="mi">0</span> <span class="o">&&</span> <span class="n">min</span> <span class="o">%</span> <span class="o">*</span><span class="n">a</span> <span class="o">==</span> <span class="mi">0</span>
<span class="p">);</span>
<span class="k">assert</span><span class="o">!</span><span class="p">(</span><span class="o">!</span><span class="p">(</span><span class="n">greatest</span> <span class="o">></span> <span class="n">result</span><span class="p">));</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Currently our function contracts can’t express that the result is the largest divisor, which necessitates this additional harness. In future Kani’s function contracts will be extended with <em>quantifiers</em>, which will allow the postcondition to express this property. For instance such a postcondition may look like this: <code class="language-plaintext highlighter-rouge">forall(|i| : max % i == 0 && min % i == 0 => i <= result)</code> which states that if any integer <code class="language-plaintext highlighter-rouge">i</code> exists that is also a divisor of <code class="language-plaintext highlighter-rouge">min</code> and <code class="language-plaintext highlighter-rouge">max</code>, then it must either be the result or smaller than it. Clearly such a condition can only be satisfied by a <code class="language-plaintext highlighter-rouge">result</code> that is the largest common divisor.</p>
<h2 id="concluding-the-example">Concluding the Example</h2>
<p>This concludes our walkthrough of function contracts and inductive verification for the Firecracker example. We have seen how functions can be abstracted using the <code class="language-plaintext highlighter-rouge">requires</code> and <code class="language-plaintext highlighter-rouge">ensures</code> clause. We have seen how Kani would verify the contract holds efficiently, using inductive verification. We then saw how after verification we can use the cheap contract in other proofs that call <code class="language-plaintext highlighter-rouge">gcd</code>.</p>
<p>You can use function contracts yourself with Kani since version 0.33.0. To enable the feature use <code class="language-plaintext highlighter-rouge">-Zfunction-contracts</code>. For an overview of the API, including all supported types of clauses, see our <a href="https://model-checking.github.io/kani/crates/doc/kani/contracts/index.html">rustdocs</a>. If you decide to try out this new feature we would very much like to hear your feedback, so join the discussion in the <a href="https://github.com/model-checking/kani/issues/2652">feature tracking issue</a>.</p>
<p>There are more features in the pipeline for contracts. In the near future we are focusing on better support for mutable pointers. For more information on implementation details and features to come take a look at the <a href="https://model-checking.github.io/kani/rfc/rfcs/0009-function-contracts.html">RFC</a>.</p>
<p>What follows hereafter are a few more sections with additional information, such as the <code class="language-plaintext highlighter-rouge">modifies</code> clause used to reason about mutable memory, history expressions and a comparison with the stubbing feature we explored in an <a href="https://model-checking.github.io/kani-verifier-blog/2023/02/28/kani-internship-projects-2022-stubbing.html">earlier blog post</a>.</p>
<hr />
<h2 id="a-neat-trick-contracts-for-non-local-functions">A Neat Trick: Contracts for Non-local Functions</h2>
<p>Attributes for attaching function contracts only work on crate-local items but you may wish to stub an out of crate function. There is currently no builtin way to do so but you can achieve the same effect using a technique known as the “double stub”. The idea is to first stub an external function to a local function that immediately calls the external function. Then the desired contract is attached to this local function which is then then stubbed as a <code class="language-plaintext highlighter-rouge">stub_verified</code> using the contract.</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">use</span> <span class="nn">external_crate</span><span class="p">::</span><span class="n">gcd</span><span class="p">;</span>
<span class="nd">#[kani::ensures(result</span> <span class="err"><</span> <span class="nd">max</span> <span class="err">&&</span> <span class="nd">result</span> <span class="err"><</span> <span class="nd">min</span> <span class="err">&&</span> <span class="nd">result</span> <span class="err">!</span><span class="nd">=</span> <span class="mi">0</span><span class="nd">)]</span>
<span class="k">fn</span> <span class="nf">local_gcd</span><span class="p">(</span><span class="n">max</span><span class="p">:</span> <span class="nb">i32</span><span class="p">,</span> <span class="n">min</span><span class="p">:</span> <span class="nb">i32</span><span class="p">)</span> <span class="k">-></span> <span class="nb">i32</span> <span class="p">{</span>
<span class="c">// immediate call with same arguments to the external function</span>
<span class="nf">gcd</span><span class="p">(</span><span class="n">max</span><span class="p">,</span> <span class="n">min</span><span class="p">)</span>
<span class="p">}</span>
<span class="k">fn</span> <span class="nf">function_under_verification</span><span class="p">(</span><span class="o">...</span><span class="p">)</span> <span class="p">{</span>
<span class="o">...</span> <span class="nf">gcd</span><span class="p">(</span><span class="o">...</span><span class="p">)</span> <span class="o">...</span>
<span class="p">}</span>
<span class="nd">#[kani::proof]</span>
<span class="nd">#[kani::stub(gcd,</span> <span class="nd">local_gcd)]</span>
<span class="nd">#[kani::stub_verified(local_gcd)]</span>
<span class="k">fn</span> <span class="nf">harness</span><span class="p">()</span> <span class="p">{</span>
<span class="o">...</span> <span class="nf">function_under_verification</span><span class="p">(</span><span class="o">...</span><span class="p">)</span> <span class="o">...</span>
<span class="p">}</span>
</code></pre></div></div>
<p>The order of <code class="language-plaintext highlighter-rouge">stub</code> and <code class="language-plaintext highlighter-rouge">stub_verified</code> does not matter. With this technique we can attach a contract to the external <code class="language-plaintext highlighter-rouge">gcd</code> without having to change the code of <code class="language-plaintext highlighter-rouge">function_under_verification</code>. Unlike other uses of stubbing this does not pose a threat to soundness because the potentially unsound <code class="language-plaintext highlighter-rouge">stub</code> effectively replaces <code class="language-plaintext highlighter-rouge">gcd</code> with itself.</p>
<h2 id="soundness-and-comparison-with-stubbing">Soundness and Comparison with Stubbing</h2>
<p>Function contracts and stubbing are closely related as both allow the replacement of a costly computation with a cheap one via harness attributes. The crucial difference is that contracts preserve soundness. We will get into more detail later, but put simply: function contracts being sound means you can trust a successful verification.</p>
<p>Conceptually stub <code class="language-plaintext highlighter-rouge">#[kani::stub(target, source)]</code> replaces all calls to <code class="language-plaintext highlighter-rouge">target</code> with <code class="language-plaintext highlighter-rouge">source</code> with weak constraints on what <code class="language-plaintext highlighter-rouge">source</code> is. <code class="language-plaintext highlighter-rouge">source</code> needs to be a function definition with a signature such that each call after the replacement typechecks. The details of what that means are unimportant, what matters is that no behavioral equivalence is checked beyond type compatibility.</p>
<p>To illustrate this, lets consider an <a href="https://xkcd.com/221/">example from XKCD</a> which stubs a <code class="language-plaintext highlighter-rouge">random</code> function by one that just returns <code class="language-plaintext highlighter-rouge">4</code>. This type of use of stubbing is completely legal and would not lead to a type error or verification failure.</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">// extern</span>
<span class="n">crate</span> <span class="n">rand</span> <span class="p">{</span>
<span class="k">pub</span> <span class="k">fn</span> <span class="n">random</span><span class="o"><</span><span class="n">T</span><span class="o">></span><span class="p">()</span> <span class="k">-></span> <span class="n">T</span>
<span class="k">where</span>
<span class="n">Standard</span><span class="p">:</span> <span class="n">Distribution</span><span class="o"><</span><span class="n">T</span><span class="o">></span>
<span class="p">{</span> <span class="cm">/* low level system calls */</span> <span class="p">}</span>
<span class="p">}</span>
<span class="k">fn</span> <span class="nf">get_random_number</span><span class="p">()</span> <span class="k">-></span> <span class="nb">u32</span> <span class="p">{</span>
<span class="mi">4</span> <span class="c">// chosen by fair dice roll</span>
<span class="c">// guaranteed to be random</span>
<span class="p">}</span>
<span class="nd">#[kani::proof]</span>
<span class="nd">#[kani::stub(rand::random,</span> <span class="nd">get_random_number)]</span>
<span class="k">fn</span> <span class="nf">my_program</span><span class="p">()</span> <span class="p">{</span>
<span class="k">let</span> <span class="n">v</span> <span class="o">=</span> <span class="nd">vec!</span><span class="p">[</span><span class="mi">1</span><span class="p">;</span><span class="mi">5</span><span class="p">];</span>
<span class="k">let</span> <span class="n">i</span> <span class="p">:</span> <span class="nb">u32</span> <span class="o">=</span> <span class="nn">rand</span><span class="p">::</span><span class="nf">random</span><span class="p">();</span>
<span class="nd">println!</span><span class="p">(</span><span class="s">"{}"</span><span class="p">,</span> <span class="n">v</span><span class="p">[</span><span class="n">i</span> <span class="k">as</span> <span class="nb">usize</span><span class="p">]);</span>
<span class="p">}</span>
</code></pre></div></div>
<p>It is rather obvious that <code class="language-plaintext highlighter-rouge">get_random_number</code> is not the same as <code class="language-plaintext highlighter-rouge">rand::random</code>. In the harness included in the example we can see how this leads to problems. After the stubbing is applied the random number is always going to be 4, which means the verification of the harness will succeed. However at runtime, where <code class="language-plaintext highlighter-rouge">rand::random</code> is no longer replaced a number greater than 4 can be produced and cause a panic. This type of situation where a runtime panic is not caught by verification is referred to as a <em>violation of soundness</em> or <em>unsoundness</em>.</p>
<p>When <em>soundness</em> is preserved that means that any runtime problem (such as a panic) cause a verification failure. Dually if soundness is violated then verification succeeds, but a runtime error is still possible. As you may imagine unsoundness significantly weakens the usefulness of a verifier. After all why bother verifying if we can still get issues at runtime? As a result verifiers like Kani take great care to preserve soundness. Stubbing is one of the features that <strong><em>can</em></strong> introduce unsoundness. As a result it is important to ensure that every stub over-approixmates the function it replaces, at least for how it is used in the code under verification. For instance it is completely sound to replace <code class="language-plaintext highlighter-rouge">rand::random</code> with a non-deterministic value with the following implementation.</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">fn</span> <span class="nf">get_random_number</span><span class="p">()</span> <span class="k">-></span> <span class="nb">u32</span> <span class="p">{</span>
<span class="nn">kani</span><span class="p">::</span><span class="nf">any</span><span class="p">()</span> <span class="c">// actually all possible values</span>
<span class="c">// guaranteed to be random</span>
<span class="p">}</span>
</code></pre></div></div>
<p>The verifier cannot execute the actual <code class="language-plaintext highlighter-rouge">rand::random</code>, because that relies on a syscall that returns a source of randomness from the machine, and so we need to replace it with something for the purposes of verification. Now if you imagine the verifier modeling all possible outputs from a random number generator simultaneously, well that is the same as using a non-deterministic value. As a result this use of stubbing is useful, safe and necessary.</p>
<p>You may wonder how we can determine whether a given situation is sound. We do this by inspecting the relationship between a function and its replacement. The replacement, by design, always approximates the behavior of the code it replaces and the question is how it does this. Conceptualize the original function as simply a black box that produces a set of possible outputs. If its replacement produces a subset of the outputs, then that is called an <em>under-approximation</em>, which causes unsoundness. We see this in our example where <code class="language-plaintext highlighter-rouge">get_random_number</code> produced fewer (e.g. only the integer <code class="language-plaintext highlighter-rouge">4</code>) instead of all possible outputs (all <code class="language-plaintext highlighter-rouge">u32</code> integers). Conversely if the replacement produces a superset of outputs this is called an <em>over-approximation</em> and crucially always sound!</p>
<p>Side note: the model of the black box function also extends to functions that take arguments, though in this case we have to consider separately the output sets for every combination of argument values.</p>
<p>Function contracts are designed such that they are always over-approximations and thus sound. This is guaranteed by the following: any value the implementation produces must pass the postcondition, but not every value that passes the postcondition must be produced by the implementation. During replacement with the contract all values that pass the postcondition are created, thus a superset of the actual outputs.</p>
<p>In our contract design there however a remaining source of unsoundness, which is the manual harnesses for verifying contracts. This was mentioned before but as a small recap: if a manual harness restricts the input such that certain behaviors of the function and contract under verification are not covered, unsoundness arises.</p>
<p>This happens in our example too where we sadly cannot verify <code class="language-plaintext highlighter-rouge">gcd</code> for an unconstrained <code class="language-plaintext highlighter-rouge">u64</code> input. The unsoundness arises when we stub using the contract. We may supply the stub with argument values that the function was never verified against and thus we have no assurance that the behavior of the function and the contract is going to be the same. A sound way to deal with this issue would be to use a precondition instead of <code class="language-plaintext highlighter-rouge">kani::assume</code>.</p>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:eiffel-1" role="doc-endnote">
<p>Meyer, Bertrand: Design by Contract, in <em>Advances in Object-Oriented Software Engineering</em>, eds. D. Mandrioli and B. Meyer, Prentice Hall, 1991, pp. 1–50 <a href="#fnref:eiffel-1" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:eiffel-2" role="doc-endnote">
<p>Meyer, Bertrand: Applying “Design by Contract”, in Computer (IEEE), 25, 10, October 1992, pp. 40–51 (<a href="http://se.ethz.ch/~meyer/publications/computer/contract.pdf">online article</a>) <a href="#fnref:eiffel-2" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:return" role="doc-endnote">
<p>The astute reader might note that this would not be the case if the body of the function we are verifying contained a <code class="language-plaintext highlighter-rouge">return</code> statement. Rest assured that the code you see here is just an illustration and the actual code generated for contract verification is not so easily fooled. If you want to see actual examples of how this is set up there is one in the implementation documentation of the <a href="https://github.com/model-checking/kani/blob/main/library/kani_macros/src/sysroot/contracts.rs">contracts macros</a> or you can dump the generated code with the rustc flag <code class="language-plaintext highlighter-rouge">-Zunpretty=expanded</code>. <a href="#fnref:return" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>In this blogpost we discuss function contracts which are now available as an unstable feature, enabled with the -Zfunction-contracts flag. If you would like to learn more about the development and implementation details of this feature please refer to the RFC. If you try out this new feature and want to leave feedback join the discussion in the feature tracking issue.Using Kani to Validate Security Boundaries in AWS Firecracker2023-08-31T00:00:00+00:002023-08-31T00:00:00+00:00https://model-checking.github.io//kani-verifier-blog/2023/08/31/using-kani-to-validate-security-boundaries-in-aws-firecracker<p>Security assurance is paramount for any system running in the cloud. In order to achieve the highest levels of security, we have applied the <a href="https://github.com/model-checking/kani">Kani model checker</a> to verify safety-critical properties in core components of the <a href="https://firecracker-microvm.github.io/">Firecracker Virtual Machine Monitor</a> using mathematical logic.</p>
<p>Firecracker is an open source project written in Rust which uses the Linux Kernel-based Virtual Machine (KVM) to create and manage microVMs. Firecracker has a minimalist design which allows fast (~150ms) microVM start-up time, secure multi-tenancy of microVMs on the same host and memory/CPU over-subscription. Firecracker is currently used in production by AWS Lambda, AWS Fargate and parts of AWS Analytics to build their service platforms.</p>
<p>For the past 7 months, <a href="https://feliperodri.github.io/">Felipe Monteiro</a>, an Applied Scientist on the Kani team and <a href="https://uk.linkedin.com/in/patrick-roy-31929323a">Patrick Roy</a>, a Software Development Engineer from the AWS Firecracker team, collaborated to develop Kani harnesses for Firecracker. As a result of this collaboration, the Firecracker team is now running 27 Kani harnesses across 3 verification suites in their continuous integration pipelines (taking approximately 15 minutes to complete), ensuring that all checked properties of critical systems are upheld on every code change.</p>
<p>In this blog post, we show how Kani helped Firecracker harden two core components, namely our I/O rate limiter and I/O transport layer (VirtIO), presenting the issues we were able to identify and fix. Particularly, the second part of this post picks up from a <a href="https://model-checking.github.io/kani-verifier-blog/2022/07/13/using-the-kani-rust-verifier-on-a-firecracker-example.html">previous Kani/Firecracker blogpost</a> and shows how improvements to Kani over the last year made verifying conformance with a section of the VirtIO specification feasible.</p>
<h2 id="noisy-neighbor-mitigations-via-rate-limiting">Noisy-Neighbor Mitigations via Rate Limiting</h2>
<p>In multi-tenant systems, microVMs from different customers simultaneously co-exist on the same physical host. We thus need to ensure that access to host resources, such as disk and network, is shared fairly. We should not allow a single “greedy” microVM to transfer excessive amounts of data from disk to the point where other microVMs’ disk access gets starved off (a “noisy neighbor” scenario). Firecracker offers a mitigation for this via <em>I/O rate-limiting</em>. From the <a href="https://github.com/firecracker-microvm/firecracker/blob/4a3e9bd76d9fc57a3538c1aeb7e5687de43a0efa/docs/design.md#io-storage-networking-and-rate-limiting">documentation</a>:</p>
<blockquote>
<p>Firecracker provides VirtIO/block and VirtIO/net emulated devices, along with the application of rate limiters to each volume and network interface to make sure host hardware resources are used fairly by multiple microVMs. These are implemented using a token bucket algorithm […]</p>
</blockquote>
<p>In a token bucket based rate-limiter, each microVM has a budget of “tokens” that can be exchanged for permission to do one byte of I/O. These tokens regenerate at a fixed rate, and if the microVM runs out of tokens, it gets I/O-throttled. This process of draining and replenishing is best visualized by an actual bucket into which water drips at a fixed rate, and from which water can be extracted at some limited rate:</p>
<p><img src="https://model-checking.github.io/kani-verifier-blog/assets/images/token-bucket-diagram.png" alt="Image visualizing the replenishing and draining of a TokenBucket" /></p>
<p>The property we want to verify is that a microVM is not allowed to exceed the configured maximum I/O throughput rate. For a virtual block device rate-limited at 1GB/s, we want to prove that in any one-second interval, at most 1GB of data is allowed to pass through the device.</p>
<p>What sounds simple in theory is actually fairly difficult to implement. For example, due to a <a href="https://github.com/firecracker-microvm/firecracker/pull/3706">rounding error</a> a guest could, in some scenarios, do up to 0.01% more I/O than configured. We discovered this bug thanks to a Kani harness for our throughput property stated above, and this harnesses is the main focus of the rest of this section.</p>
<h3 id="teaching-kani-about-time">Teaching Kani about Time</h3>
<p>The core component of our rate-limiting implementation is a <code class="language-plaintext highlighter-rouge">TokenBucket</code>. In Firecracker, we define it as</p>
<div class="language-rs highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">pub</span> <span class="k">struct</span> <span class="n">TokenBucket</span> <span class="p">{</span>
<span class="c">// Maximal number of tokens this bucket can hold.</span>
<span class="n">size</span><span class="p">:</span> <span class="nb">u64</span><span class="p">,</span>
<span class="c">// Complete refill time in milliseconds.</span>
<span class="n">refill_time</span><span class="p">:</span> <span class="nb">u64</span><span class="p">,</span>
<span class="c">// Current token budget.</span>
<span class="n">budget</span><span class="p">:</span> <span class="nb">u64</span><span class="p">,</span>
<span class="c">// Last time this token bucket was replenished.</span>
<span class="n">last_update</span><span class="p">:</span> <span class="n">Instant</span><span class="p">,</span>
<span class="c">// -- snip --</span>
<span class="p">}</span>
</code></pre></div></div>
<p>It offers an <code class="language-plaintext highlighter-rouge">auto_replenish</code> function which computes how many tokens the leaky bucket algorithm should have generated since <code class="language-plaintext highlighter-rouge">last_update</code> (and then updates <code class="language-plaintext highlighter-rouge">last_update</code> accordingly). This function will be the target of our verification.</p>
<p>A <code class="language-plaintext highlighter-rouge">TokenBucket</code> is inherently tied to time-related APIs such as <code class="language-plaintext highlighter-rouge">std::time::Instant</code>, for which Kani does not have built-in support. This means it is not able to reason about <code class="language-plaintext highlighter-rouge">TokenBucket</code>s. To solve this problem, we use Kani’s <a href="https://model-checking.github.io/kani-verifier-blog/2023/02/28/kani-internship-projects-2022-stubbing.html">stubbing</a> to provide a model for the <code class="language-plaintext highlighter-rouge">Instant::now</code> function. Since Firecracker uses a monotonic clock for its rate-limiting, this stub needs to return non-deterministic monotonically non-decreasing instants.</p>
<p>However, when trying to stub <code class="language-plaintext highlighter-rouge">now</code>, one will quickly notice that <code class="language-plaintext highlighter-rouge">Instant</code> does not offer any constructors for creating an instance from, say, a Unix timestamp. In fact, it is impossible to construct an <code class="language-plaintext highlighter-rouge">Instant</code> outside of the standard library as its fields are private. When in such a situation, the solution is often to go down the call stack of the function that you want to stub, to see if any of the functions further down can be stubbed out instead to achieve the desired effect. In our case, <code class="language-plaintext highlighter-rouge">now</code> calls functions in (private) OS-specific time modules, until it bottoms out at <a href="https://www.gnu.org/software/libc/manual/html_node/Getting-the-Time.html#index-clock_005fgettime"><code class="language-plaintext highlighter-rouge">libc::clock_gettime</code></a>.</p>
<p>The <code class="language-plaintext highlighter-rouge">clock_gettime</code> function is passed a pointer to a <code class="language-plaintext highlighter-rouge">libc::timespec</code> structure, and the <code class="language-plaintext highlighter-rouge">tv_sec</code> and <code class="language-plaintext highlighter-rouge">tv_nsec</code> members of this structure are later used to construct the <code class="language-plaintext highlighter-rouge">Instant</code> returned by <code class="language-plaintext highlighter-rouge">Instant::now</code>. Therefore, we can use the following stub to achieve our goal of getting non-deterministic, monotonically non-decreasing <code class="language-plaintext highlighter-rouge">Instant</code>s:</p>
<div class="language-rs highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">mod</span> <span class="n">stubs</span> <span class="p">{</span>
<span class="k">static</span> <span class="k">mut</span> <span class="n">LAST_SECONDS</span><span class="p">:</span> <span class="nb">i64</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="k">static</span> <span class="k">mut</span> <span class="n">LAST_NANOS</span><span class="p">:</span> <span class="nb">i64</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="k">const</span> <span class="n">NANOS_PER_SECOND</span><span class="p">:</span> <span class="nb">i64</span> <span class="o">=</span> <span class="mi">1_000_000_000</span><span class="p">;</span>
<span class="k">pub</span> <span class="k">unsafe</span> <span class="k">extern</span> <span class="s">"C"</span> <span class="k">fn</span> <span class="nf">clock_gettime</span><span class="p">(</span><span class="mi">_</span><span class="n">clock_id</span><span class="p">:</span> <span class="nn">libc</span><span class="p">::</span><span class="n">clockid_t</span><span class="p">,</span> <span class="n">tp</span><span class="p">:</span> <span class="o">*</span><span class="k">mut</span> <span class="nn">libc</span><span class="p">::</span><span class="n">timespec</span><span class="p">)</span> <span class="k">-></span> <span class="nn">libc</span><span class="p">::</span><span class="nb">c_int</span> <span class="p">{</span>
<span class="k">unsafe</span> <span class="p">{</span>
<span class="c">// kani::any_where provides us with a non-deterministic number of seconds</span>
<span class="c">// that is at least equal to LAST_SECONDS (to ensure that time only</span>
<span class="c">// progresses forward).</span>
<span class="k">let</span> <span class="n">next_seconds</span> <span class="o">=</span> <span class="nn">kani</span><span class="p">::</span><span class="nf">any_where</span><span class="p">(|</span><span class="o">&</span><span class="n">n</span><span class="p">|</span> <span class="n">n</span> <span class="o">>=</span> <span class="k">unsafe</span> <span class="p">{</span> <span class="n">LAST_SECONDS</span> <span class="p">});</span>
<span class="k">let</span> <span class="n">next_nanos</span> <span class="o">=</span> <span class="nn">kani</span><span class="p">::</span><span class="nf">any_where</span><span class="p">(|</span><span class="o">&</span><span class="n">n</span><span class="p">|</span> <span class="n">n</span> <span class="o">>=</span> <span class="mi">0</span> <span class="o">&&</span> <span class="n">n</span> <span class="o"><</span> <span class="n">NANOS_PER_SECOND</span><span class="p">);</span>
<span class="k">if</span> <span class="n">next_seconds</span> <span class="o">==</span> <span class="n">LAST_SECONDS</span> <span class="p">{</span>
<span class="nn">kani</span><span class="p">::</span><span class="nf">assume</span><span class="p">(</span><span class="n">next_nanos</span> <span class="o">>=</span> <span class="n">LAST_NANOS</span> <span class="p">);</span>
<span class="p">}</span>
<span class="p">(</span><span class="o">*</span><span class="n">tp</span><span class="p">)</span><span class="py">.tv_sec</span> <span class="o">=</span> <span class="n">LAST_SECONDS</span><span class="p">;</span>
<span class="p">(</span><span class="o">*</span><span class="n">tp</span><span class="p">)</span><span class="py">.tv_nsec</span> <span class="o">=</span> <span class="n">LAST_NANOS</span><span class="p">;</span>
<span class="n">LAST_SECONDS</span> <span class="o">=</span> <span class="n">next_seconds</span><span class="p">;</span>
<span class="n">LAST_NANOS</span> <span class="o">=</span> <span class="n">next_nanos</span><span class="p">;</span>
<span class="p">}</span>
<span class="mi">0</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Note how the first invocation of this stub will always set <code class="language-plaintext highlighter-rouge">tv_sec = tv_nsec = 0</code>, as this is what the statics are initialized to. This is an optimization we do because the rate-limiter only cares about the delta between two instants, which will be non-deterministic as long as one of the two instants is non-deterministic. <strong>In order to keep Kani performant, it is important to minimize the number of non-deterministic values, especially if multiplication and division are involved</strong>.</p>
<p>Using this stub, we can start writing a harness for <code class="language-plaintext highlighter-rouge">auto_replenish</code> such as</p>
<div class="language-rs highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">#[kani::proof]</span>
<span class="nd">#[kani::unwind(</span><span class="mi">1</span><span class="nd">)]</span> <span class="c">// Enough to unwind the recursion at `Timespec::sub_timespec`.</span>
<span class="nd">#[kani::stub(libc::clock_gettime,</span> <span class="nd">stubs::clock_gettime)]</span>
<span class="k">fn</span> <span class="nf">verify_token_bucket_auto_replenish</span><span class="p">()</span> <span class="p">{</span>
<span class="c">// Initialize a non-determinstic `TokenBucket` object.</span>
<span class="k">let</span> <span class="k">mut</span> <span class="n">bucket</span><span class="p">:</span> <span class="n">TokenBucket</span> <span class="o">=</span> <span class="nn">kani</span><span class="p">::</span><span class="nf">any</span><span class="p">();</span>
<span class="n">bucket</span><span class="nf">.auto_replenish</span><span class="p">();</span>
<span class="c">// is_valid() performs sanity checks such as "budget <= size".</span>
<span class="c">// It is the data structure invariant of `TokenBucket`.</span>
<span class="k">assert</span><span class="o">!</span><span class="p">(</span><span class="n">bucket</span><span class="nf">.is_valid</span><span class="p">());</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Let us now see how we can extend this harness to allow us to verify that our rate limiter is replenishing tokens at exactly the requested rate.</p>
<h3 id="verifying-our-noisy-neighbor-mitigation">Verifying our Noisy-Neighbor Mitigation</h3>
<p>Our noisy neighbor mitigation is correct if we always generate the “correct” number of tokens with each call to <code class="language-plaintext highlighter-rouge">auto_replenish</code>, meaning it is impossible for a guest to do more I/O than configured. Formally, this means</p>
\[0 \leq \left(now - last{\_}update\right) - \left( new{\_}tokens \times \left(\frac{refill{\_}time}{size}\right) \right) < \left(\frac{refill{\_}time}{size}\right)\]
<p>Here, $new\_tokens$ is the number of tokens that <code class="language-plaintext highlighter-rouge">auto_replenish</code> generated. The fraction $\left(\frac{refill\_time}{size}\right)$ is simply the time it takes to generate a single token. Thus, the property states that if we compute the time that it should have taken to generate $new\_tokens$ and subtract it from the time that actually passed, we are left with an amount of time less than what it would take to generate an additional token: we replenished the maximal number of tokens possible.</p>
<p>The difficulty of implementing a correct rate limiter is dealing with “leftover” time: If enough time passed to generate “1.8 tokens”, what does Firecracker do with the “0.8” tokens it cannot (as everything is integer valued) add to the budget? Originally, the rate limiter simply dropped these: if you called <code class="language-plaintext highlighter-rouge">auto_replenish</code> at an inopportune time, then the “0.8” would not be carried forward and the guest essentially “lost” part of its I/O allowance to rounding. Then, with <a href="https://github.com/firecracker-microvm/firecracker/pull/3370">#3370</a>, we decided to fix this by only advancing $last\_update$ by $new\_tokens \times \left(\frac{refill\_time}{size}\right)$ instead of setting it to <code class="language-plaintext highlighter-rouge">now</code>. This way the fractional tokens will be carried forward, and we even hand-wrote a <a href="https://github.com/firecracker-microvm/firecracker/pull/3370#pullrequestreview-1252110534">proof</a> to check that $last\_update$ and the actual system time will not diverge, boldly concluding</p>
<blockquote>
<p>This means that $last\_updated$ indeed does not fall behind more than the execution time of <code class="language-plaintext highlighter-rouge">auto_replenish</code> plus a constant dependent on the bucket configuration.</p>
</blockquote>
<p>Here, the “constant dependent on the bucket configuration” was $\left(\frac{refill\_time}{size}\right)$, rounded down. This is indeed implies our above specified property, so when we revisited <code class="language-plaintext highlighter-rouge">auto_replenish</code> a few months later to add the following two <code class="language-plaintext highlighter-rouge">debug_asserts!</code> derived from our formal property.</p>
<div class="language-rs highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">// time_adjustment = tokens * (refill_time / size)</span>
<span class="nd">debug_assert!</span><span class="p">((</span><span class="n">now</span> <span class="o">-</span> <span class="n">last_update</span><span class="p">)</span> <span class="o">>=</span> <span class="n">time_adjustment</span><span class="p">);</span>
<span class="c">// inequality slightly rewritten to avoid division</span>
<span class="nd">debug_assert!</span><span class="p">((</span><span class="n">now</span> <span class="o">-</span> <span class="n">last_update</span> <span class="o">-</span> <span class="n">time_adjustment</span><span class="p">)</span> <span class="o">*</span> <span class="n">size</span> <span class="o"><</span> <span class="n">refill_time</span><span class="p">);</span>
</code></pre></div></div>
<p>we expected the verification to succeed. However, Kani presented us with The “<strong><span style="color:red">VERIFICATION FAILED</span></strong>” message, which was unexpected to say the least.</p>
<p>So what went wrong? In the hand-written proof, the error was assuming that $-\lfloor -x \rfloor = \lfloor x \rfloor$ (had this step been gotten correctly, the bound would have been $\left(\frac{refill\_time}{size}\right)$ rounded <em>up</em>, which obviously allows for violations). To see how our code actually violates the property, we need to have a look at how the relevant part of <code class="language-plaintext highlighter-rouge">auto_replenish</code> was actually implemented:</p>
<div class="language-rs highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">let</span> <span class="n">time_delta</span> <span class="o">=</span> <span class="k">self</span><span class="py">.last_update</span><span class="nf">.elapsed</span><span class="p">()</span><span class="nf">.as_nanos</span><span class="p">()</span> <span class="k">as</span> <span class="nb">u64</span><span class="p">;</span>
<span class="c">// tokens = time_delta / (refill_time / size) rewritten to not run into</span>
<span class="c">// integer division issues.</span>
<span class="k">let</span> <span class="n">tokens</span> <span class="o">=</span> <span class="p">(</span><span class="n">time_delta</span> <span class="o">*</span> <span class="k">self</span><span class="py">.size</span><span class="p">)</span> <span class="o">/</span> <span class="k">self</span><span class="py">.refill_time</span><span class="p">;</span>
<span class="k">let</span> <span class="n">time_adjustment</span> <span class="o">=</span> <span class="p">(</span><span class="n">tokens</span> <span class="o">*</span> <span class="k">self</span><span class="py">.refill_time</span><span class="p">)</span> <span class="o">/</span> <span class="k">self</span><span class="py">.size</span>
<span class="k">self</span><span class="py">.last_update</span> <span class="o">+=</span> <span class="nn">Duration</span><span class="p">::</span><span class="nf">from_nanos</span><span class="p">(</span><span class="n">time_adjustment</span><span class="p">);</span>
</code></pre></div></div>
<p>The issue lies in the way we compute <code class="language-plaintext highlighter-rouge">time_adjustment</code>: Consider a bucket of size 2 with refill time 3ns and assume a time delta of 11ns. We compute $11 \times 2/3 \approx 7$ tokens, and then a time adjustment of $7 \times 3/2 \approx 10ns$. However, 10ns is only enough to replenish $10 \times 2/3 \approx 6$ tokens! The problem here is that 7 tokens do not take an integer number of nanoseconds to replenish. They take 10.5ns. However the integer division rounds this down, and thus the guest essentially gets to use those 0.5ns twice. Assuming the guest can time when it triggers down to nanosecond precision, and the rate limiter is configured such that $\left(\frac{refill\_time}{size}\right)$ is not an integer, the guest could theoretically cause these fractional nanoseconds to accumulate to get an extra token every $10^{6} \times \left(\frac{refill\_time}{size}\right) \times max\left(1, \left(\frac{refill\_time}{size}\right)\right)$ nanoseconds. <strong>For a rate limiter configured at 1GB/s, this would be an excess of 1KB/s</strong>.</p>
<p>The fix for this was to round up instead of down in our computation of <code class="language-plaintext highlighter-rouge">time_adjustment</code>. For the complete code listing of the rate limiter harnesses, see <a href="https://github.com/firecracker-microvm/firecracker/blob/1a2c6ada116b52df891857d3e82503ad1ef845e5/src/vmm/src/rate_limiter/mod.rs#L525">here</a>.</p>
<h2 id="conformance-to-the-virtio-specification">Conformance to the VirtIO Specification</h2>
<p>Firecracker is a para-virtualization solution, meaning the guest is aware that it is running inside of a virtual machine. This allows host and guests to collaborate when it comes to I/O, as opposed to the host having to do all the heavy lifting of emulating physical devices. Firecracker uses <a href="https://docs.oasis-open.org/virtio/virtio/v1.1/csprd01/virtio-v1.1-csprd01.pdf">VirtIO</a> for the transport-layer protocol of its paravirtualized device stack. It allows the guest and host to exchange messages via pairs of ring buffers called a <em>queue</em>. At a high level, the guest puts requests into a shared array (the “descriptor table”) and puts the index into the descriptor table at which the host can find the new request into the request ring (the “avail ring” in VirtIO lingo). It then notifies the host via interrupt that a new request is available for processing. The host now processes the request, updating the descriptor table entry with its response and, upon finishing, writes the index into the descriptor table into a response ring (the “used ring”). It then notifies the guest that processing of a request has finished.</p>
<p>The Firecracker side of this queue implementation sits right at the intersection between guest and host. According to Firecracker’s <a href="https://github.com/firecracker-microvm/firecracker/blob/main/docs/design.md#threat-containment">threat model</a>:</p>
<blockquote>
<p>From a security perspective, all vCPU threads are considered to be running untrusted code as soon as they have been started; these untrusted threads need to be contained.</p>
</blockquote>
<p>The entirety of the VirtIO queue lives in shared memory and can thus be written to by the vCPU threads. Therefore, Firecracker cannot make any assumptions about its contents. In particular, it needs to operate securely no matter the memory content. For anyone who has worked with Kani before, this yearns for a generous application of <code class="language-plaintext highlighter-rouge">kani::vec::exact_vec</code>, which generates a fixed size vector filled with arbitrary values. We can set up an area of non-deterministic guest memory as follows:</p>
<div class="language-rs highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">fn</span> <span class="nf">arbitrary_guest_memory</span><span class="p">()</span> <span class="k">-></span> <span class="n">GuestMemoryMmap</span> <span class="p">{</span>
<span class="c">// We need ManuallyDrop to "leak" the memory area to ensure it lives for</span>
<span class="c">// the entire duration of the proof.</span>
<span class="k">let</span> <span class="n">memory</span> <span class="o">=</span> <span class="nn">ManuallyDrop</span><span class="p">::</span><span class="nf">new</span><span class="p">(</span><span class="nn">kani</span><span class="p">::</span><span class="nn">vec</span><span class="p">::</span><span class="nn">exact_vec</span><span class="p">::</span><span class="o"><</span><span class="nb">u8</span><span class="p">,</span> <span class="n">GUEST_MEMORY_SIZE</span><span class="o">></span><span class="p">())</span>
<span class="nf">.as_mut_ptr</span><span class="p">();</span>
<span class="k">let</span> <span class="n">region</span> <span class="o">=</span> <span class="k">unsafe</span> <span class="p">{</span>
<span class="nn">MmapRegionBuilder</span><span class="p">::</span><span class="nf">new</span><span class="p">(</span><span class="n">GUEST_MEMORY_SIZE</span><span class="p">)</span>
<span class="nf">.with_raw_mmap_pointer</span><span class="p">(</span><span class="n">memory</span><span class="p">)</span>
<span class="nf">.build</span><span class="p">()</span>
<span class="nf">.unwrap</span><span class="p">()</span>
<span class="p">};</span>
<span class="k">let</span> <span class="n">guest_region</span> <span class="o">=</span> <span class="nn">GuestRegionMmap</span><span class="p">::</span><span class="nf">new</span><span class="p">(</span><span class="n">region</span><span class="p">,</span> <span class="nf">GuestAddress</span><span class="p">(</span><span class="mi">0</span><span class="p">))</span><span class="nf">.unwrap</span><span class="p">();</span>
<span class="c">// Use a single memory region, just as Firecracker does for guests of size < 2GB.</span>
<span class="c">// For largest guests, Firecracker uses two regions (due to the MMIO gap being</span>
<span class="c">// at the top of 32-bit address space).</span>
<span class="nn">GuestMemoryMmap</span><span class="p">::</span><span class="nf">from_regions</span><span class="p">(</span><span class="nd">vec!</span><span class="p">[</span><span class="n">guest_region</span><span class="p">])</span><span class="nf">.unwrap</span><span class="p">()</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Note that this requires a stub for <code class="language-plaintext highlighter-rouge">libc::sysconf</code>, which is used by <code class="language-plaintext highlighter-rouge">.build()</code> to verify that guest memory is correctly aligned. We can use a stub that always returns 1, which causes <code class="language-plaintext highlighter-rouge">vm_memory</code> to consider all pointers to be correctly aligned.</p>
<p>With our non-deterministic guest memory setup, we can start verifying things! On the host side, a queue is just a collection of guest physical addresses. We currently cannot set all of them to non-deterministic values, as the complexity of our mathematical model would explode, but we can get fairly far:</p>
<div class="language-rs highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">impl</span> <span class="nn">kani</span><span class="p">::</span><span class="n">Arbitrary</span> <span class="k">for</span> <span class="n">Queue</span> <span class="p">{</span>
<span class="k">fn</span> <span class="nf">any</span><span class="p">()</span> <span class="k">-></span> <span class="n">Queue</span> <span class="p">{</span>
<span class="c">// Firecracker statically sets the maximal queue size to 256.</span>
<span class="k">let</span> <span class="k">mut</span> <span class="n">queue</span> <span class="o">=</span> <span class="nn">Queue</span><span class="p">::</span><span class="nf">new</span><span class="p">(</span><span class="n">FIRECRACKER_MAX_QUEUE_SIZE</span><span class="p">);</span>
<span class="k">const</span> <span class="n">QUEUE_BASE_ADDRESS</span><span class="p">:</span> <span class="nb">u64</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="c">// Descriptor table has 16 bytes per entry, avail ring starts right after.</span>
<span class="k">const</span> <span class="n">AVAIL_RING_BASE_ADDRESS</span><span class="p">:</span> <span class="nb">u64</span> <span class="o">=</span>
<span class="n">QUEUE_BASE_ADDRESS</span> <span class="o">+</span> <span class="n">FIRECRACKER_MAX_QUEUE_SIZE</span> <span class="k">as</span> <span class="nb">u64</span> <span class="o">*</span> <span class="mi">16</span><span class="p">;</span>
<span class="c">// Used ring starts after avail ring (which has size 6 + 2 * FIRECRACKER_MAX_QUEUE_SIZE),</span>
<span class="c">// and needs 2 bytes of padding.</span>
<span class="k">const</span> <span class="n">USED_RING_BASE_ADDRESS</span><span class="p">:</span> <span class="nb">u64</span> <span class="o">=</span>
<span class="n">AVAIL_RING_BASE_ADDRESS</span> <span class="o">+</span> <span class="mi">6</span> <span class="o">+</span> <span class="mi">2</span> <span class="o">*</span> <span class="n">FIRECRACKER_MAX_QUEUE_SIZE</span> <span class="k">as</span> <span class="nb">u64</span> <span class="o">+</span> <span class="mi">2</span><span class="p">;</span>
<span class="n">queue</span><span class="py">.size</span> <span class="o">=</span> <span class="n">FIRECRACKER_MAX_QUEUE_SIZE</span><span class="p">;</span>
<span class="n">queue</span><span class="py">.ready</span> <span class="o">=</span> <span class="k">true</span><span class="p">;</span>
<span class="n">queue</span><span class="py">.desc_table</span> <span class="o">=</span> <span class="nf">GuestAddress</span><span class="p">(</span><span class="n">QUEUE_BASE_ADDRESS</span><span class="p">);</span>
<span class="n">queue</span><span class="py">.avail_ring</span> <span class="o">=</span> <span class="nf">GuestAddress</span><span class="p">(</span><span class="n">AVAIL_RING_BASE_ADDRESS</span><span class="p">);</span>
<span class="n">queue</span><span class="py">.used_ring</span> <span class="o">=</span> <span class="nf">GuestAddress</span><span class="p">(</span><span class="n">USED_RING_BASE_ADDRESS</span><span class="p">);</span>
<span class="c">// Index at which we expect the guest to place its next request into</span>
<span class="c">// the avail ring.</span>
<span class="n">queue</span><span class="py">.next_avail</span> <span class="o">=</span> <span class="nf">Wrapping</span><span class="p">(</span><span class="nn">kani</span><span class="p">::</span><span class="nf">any</span><span class="p">());</span>
<span class="c">// Index at which we will put the next response into the used ring.</span>
<span class="n">queue</span><span class="py">.next_used</span> <span class="o">=</span> <span class="nf">Wrapping</span><span class="p">(</span><span class="nn">kani</span><span class="p">::</span><span class="nf">any</span><span class="p">());</span>
<span class="c">// Whether notification suppression is enabled for this queue.</span>
<span class="n">queue</span><span class="py">.uses_notif_suppression</span> <span class="o">=</span> <span class="nn">kani</span><span class="p">::</span><span class="nf">any</span><span class="p">();</span>
<span class="c">// How many responses were added to the used ring since the last</span>
<span class="c">// notification was sent to the guest.</span>
<span class="n">queue</span><span class="py">.num_added</span> <span class="o">=</span> <span class="nf">Wrapping</span><span class="p">(</span><span class="nn">kani</span><span class="p">::</span><span class="nf">any</span><span class="p">());</span>
<span class="n">queue</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Here, the final two fields, <code class="language-plaintext highlighter-rouge">uses_notif_suppression</code> and <code class="language-plaintext highlighter-rouge">num_added</code> are relevant for the property we want to verify. Notification suppression is a mechanism described in <a href="https://docs.oasis-open.org/virtio/virtio/v1.1/csprd01/virtio-v1.1-csprd01.pdf">Section 2.6.7 of the VirtIO specification</a> which is designed to reduce the overall number of interrupts exchanged between guest and host. When enabled, it allows the guest to tell the host that it should not send an interrupt for every single processed request, but instead wait until a specific number of requests have been processed. The guest does this by writing a used ring index into a predefined memory location. The host then will not send interrupts until it uses the specified index for a response.</p>
<p>To better understand this mechanism, consider the following queue:</p>
<p><img src="https://model-checking.github.io/kani-verifier-blog/assets/images/virtio-diagram.png" alt="Imagine illustrating used buffer notification suppression" /></p>
<p>The guest just wrote requests 1 through 3 into the avail ring and notified the host. Without notification suppression, the host would now process request 1, write the result into slot 1, and notify the guest about the first request being done. With notification suppression, the host will instead realize that the guest does not want notification until it writes a response to the third slot. This means the host will only notify the request after processing all three requests, and we saved ourselves two interrupts.</p>
<p>This is a much simplified scenario. The exact details of this are written down in <a href="https://docs.oasis-open.org/virtio/virtio/v1.1/csprd01/virtio-v1.1-csprd01.pdf">Section 2.6.7.2 of the VirtIO 1.1 specification</a>. We can turn that specification into the following Kani harness:</p>
<div class="language-rs highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">#[kani::proof]</span>
<span class="nd">#[kani::unwind(</span><span class="mi">2</span><span class="nd">)]</span> <span class="c">// Guest memory regions are stored in a BTreeMap, which</span>
<span class="c">// employs binary search resolving guest addresses to</span>
<span class="c">// regions. We only have a single region, so the search</span>
<span class="c">// terminates in one iteration.</span>
<span class="k">fn</span> <span class="nf">verify_spec_2_6_7_2</span><span class="p">()</span> <span class="p">{</span>
<span class="k">let</span> <span class="n">mem</span> <span class="o">=</span> <span class="nf">arbitrary_guest_memory</span><span class="p">();</span>
<span class="k">let</span> <span class="k">mut</span> <span class="n">queue</span><span class="p">:</span> <span class="n">Queue</span> <span class="o">=</span> <span class="nn">kani</span><span class="p">::</span><span class="nf">any</span><span class="p">();</span>
<span class="c">// Assume various alignment needs are met. Every function operating on a queue</span>
<span class="c">// has a debug_assert! matching this assumption.</span>
<span class="nn">kani</span><span class="p">::</span><span class="nf">assume</span><span class="p">(</span><span class="n">queue</span><span class="nf">.is_layout_valid</span><span class="p">(</span><span class="o">&</span><span class="n">mem</span><span class="p">));</span>
<span class="k">let</span> <span class="n">needs_notification</span> <span class="o">=</span> <span class="n">queue</span><span class="nf">.prepare_kick</span><span class="p">(</span><span class="o">&</span><span class="n">mem</span><span class="p">);</span>
<span class="k">if</span> <span class="o">!</span><span class="n">queue</span><span class="py">.uses_notif_suppression</span> <span class="p">{</span>
<span class="c">// After the device writes a descriptor index into the used ring:</span>
<span class="c">// – If flags is 1, the device SHOULD NOT send a notification.</span>
<span class="c">// – If flags is 0, the device MUST send a notification.</span>
<span class="c">// flags is the first field in the avail_ring, which we completely ignore. We</span>
<span class="c">// always send a notification, and as there only is a SHOULD NOT, that is okay</span>
<span class="k">assert</span><span class="o">!</span><span class="p">(</span><span class="n">needs_notification</span><span class="p">);</span>
<span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
<span class="c">// next_used - 1 is where the previous descriptor was placed.</span>
<span class="c">// queue.used_event(&mem) reads from the memory location at which the guest</span>
<span class="c">// stores the index for which it wants to receive the next notification.</span>
<span class="k">if</span> <span class="n">queue</span><span class="nf">.used_event</span><span class="p">(</span><span class="o">&</span><span class="n">mem</span><span class="p">)</span> <span class="o">==</span> <span class="n">queue</span><span class="py">.next_used</span> <span class="o">-</span> <span class="nf">Wrapping</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span> <span class="o">&&</span> <span class="n">queue</span><span class="py">.num_added</span><span class="na">.0</span> <span class="o">></span> <span class="mi">0</span> <span class="p">{</span>
<span class="c">// If the idx field in the used ring (which determined where that descriptor index</span>
<span class="c">// was placed) was equal to used_event, the device MUST send a notification.</span>
<span class="k">assert</span><span class="o">!</span><span class="p">(</span><span class="n">needs_notification</span><span class="p">)</span>
<span class="p">}</span>
<span class="c">// The other case is handled by a "SHOULD NOT send a notification" in the spec.</span>
<span class="c">// So we do not care.</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Beyond these specification conformance harnesses, we also have standard “absence of panics” harnesses, which led us to discover an issue in our code which validates the in-memory layout of VirtIO queues. A guest could <a href="https://github.com/firecracker-microvm/firecracker/commit/7909c5e6d023cbac98a5b16430d53d13370cf8be">trigger a panic in Firecracker</a> by placing the starting address for a VirtIO queue component into the MMIO gap.</p>
<h2 id="conclusion">Conclusion</h2>
<p>Thanks to Kani, the Firecracker team was able to verify critical areas of code that were intractable to traditional methods. These include our noisy-neighbor mitigation, a rate limiter, where interactions with the system clock resulted in traditional testing being unreliable, as well as our VirtIO stack, where the interaction with guest memory lead to a state space impossible to cover by other means.</p>
<p>We found 5 bugs in our rate limiter implementation, the most significant one a rounding error that allowed guests to exceed their prescribed I/O bandwidth by up to 0.01% in some cases. Additionally, we found one bug in our VirtIO stack, where a untrusted guest could set up a virtio queue that partially overlapped with the MMIO memory region, resulting in Firecracker crashing on boot. Finally, the debug assertions added to the code under verification allowed us to identify a handful of unit tests which were not set up correctly. These have also been fixed.</p>
<p>All in all, Kani proof harnesses has proven a valuable defense-in-depth measure for Firecracker, nicely complementing our existing testing infrastructure. We plan to continue our investment in these harnesses as we develop new Firecracker features, to ensure consistently high security standards. To learn more about Kani, check out the <a href="https://model-checking.github.io/kani/kani-tutorial.html">Kani tutorial</a> and our <a href="https://model-checking.github.io/kani-verifier-blog/">previous blog posts</a>.</p>Security assurance is paramount for any system running in the cloud. In order to achieve the highest levels of security, we have applied the Kani model checker to verify safety-critical properties in core components of the Firecracker Virtual Machine Monitor using mathematical logic.Turbocharging Rust Code Verification2023-08-03T00:00:00+00:002023-08-03T00:00:00+00:00https://model-checking.github.io//kani-verifier-blog/2023/08/03/turbocharging-rust-code-verification<p>Kani is a bit-precise model checker that can verify properties about your Rust code.
To learn more about Kani, check out the <a href="https://model-checking.github.io/kani/kani-tutorial.html">Kani tutorial</a> and our <a href="https://model-checking.github.io/kani-verifier-blog/">previous blog posts</a>.</p>
<p>Over the past 9 months, we have optimized Kani at different levels to either improve general performance or solve some specific performance problems encountered on user verification harnesses.</p>
<p>In this post, we’ll discuss three optimizations enabled through three different components of Kani. In short:</p>
<ul>
<li>By allowing the selection of a specific SAT solver (globally or per harness) we can obtain consistent speedups of 2-8X and up to 200X on specific harnesses, and an 85% reduction in total runtime on a crate like <a href="https://github.com/aws/s2n-quic/tree/main/quic/s2n-quic-core">s2n-quic-core</a>;</li>
<li>By adding a new GOTO program serializer that exports files in CBMC’s own binary format, we obtain a consistent 4X speedup on the GOTO code generation + export step;</li>
<li>By improving constant propagation in CBMC for union types, Kani can now solve in a few seconds verification harnesses that used to explode in time and memory.</li>
</ul>
<p>Compared to the last version of 2022, the current version of Kani (August 2023) is swifter and solves more problems in less time.</p>
<p>But before diving into the details of each new feature and optimization, we’ll start with a high-level overview of Kani’s architecture.</p>
<h2 id="kani-architecture-overview">Kani Architecture Overview</h2>
<p>First, let’s briefly introduce you to Kani’s high level architecture to understand where modifications were performed.</p>
<p>Kani can be invoked either directly on a Rust file using <code class="language-plaintext highlighter-rouge">kani my_file.rs</code> or on a whole package or crate via <code class="language-plaintext highlighter-rouge">cargo kani</code> (see usage <a href="https://model-checking.github.io/kani/usage.html">here</a>). In both cases, the verification process is very similar and Kani will continue to run until it finishes checking all harnesses found in the code, and it will report whether each harness was successfully verified or failed.</p>
<p>Internally, this verification process is a bit more complicated, and can be split into three main stages:</p>
<ol>
<li><strong>Compilation:</strong> The Rust crate under verification and its dependencies are compiled into a program in a format that’s more suitable to verification.</li>
<li><strong>Symbolic execution:</strong> This program is then symbolically executed in order to generate a single logic formula that represents all possible execution paths and the properties to be checked.</li>
<li><strong>Satisfiability solving:</strong> This logic formula is then solved by employing a SAT or SMT solver that either finds a combination of concrete values that can satisfy the formula (a counterexample to a property exists), or prove that no assignment can satisfy the formula (all properties hold).</li>
</ol>
<p>In fact, Kani employs a collection of tools to perform the different stages of the verification.
Kani’s main process is called <code class="language-plaintext highlighter-rouge">kani-driver</code>, and its main purpose is to orchestrate the execution and communication of these other tools:</p>
<ol>
<li>The compilation stage is done mostly<sup id="fnref:build-details" role="doc-noteref"><a href="#fn:build-details" class="footnote" rel="footnote">1</a></sup> by <code class="language-plaintext highlighter-rouge">kani-compiler</code>, which is an extension of the Rust compiler that we have developed. <code class="language-plaintext highlighter-rouge">kani-compiler</code> will generate a <code class="language-plaintext highlighter-rouge">goto-program</code> by combining all the code reachable from a harness.</li>
<li>For the symbolic execution stage, Kani invokes <a href="https://www.cprover.org/cbmc/">CBMC</a>.</li>
<li>The satisfiability checking stage is performed by CBMC itself, by invoking a <a href="https://en.wikipedia.org/wiki/SAT_solver">satisfiability (SAT) solver</a> such as <a href="http://minisat.se/">MiniSat</a>.</li>
</ol>
<p>Finally, Kani ships with a crate (named <code class="language-plaintext highlighter-rouge">kani</code>) that provides a set of APIs that defines attributes, functions, traits and implementations that allow users to create and customize their harnesses. This crate is automatically added as a dependency to every other crate Kani compiles.</p>
<p><img src="/kani-verifier-blog/assets/images/kani-high-level.png" alt="Kani architecture" /></p>
<p>Note that the verification problem is computationally hard. Thus, some optimizations can have a positive effect only on a subset of harnesses, while other optimizations can bring benefits overall. Kani was designed to provide a good out-of-the box experience, but also to allow experimentation and further customization to achieve an optimal performance for each harness.</p>
<h2 id="supporting-multiple-sat-solvers">Supporting Multiple SAT Solvers</h2>
<p>SAT solving is typically the most time-consuming part of a Kani run. There is a large number of SAT solvers available, whose performance can vary widely depending on the specific type of formula, and it can be very helpful to be able to try different SAT solvers on the same problem to determine which one performs best.</p>
<p>By default, CBMC uses the <a href="http://minisat.se/">MiniSat</a> SAT solver.
While CBMC can be configured to use a different SAT solver at build time, having to rebuild it to switch SAT solvers is inconvenient.
Thus, we introduced an enhancement to CBMC’s build system that allows CBMC to be built with multiple SAT solvers, so that the user can select one of them at <em>runtime</em> via an option (<code class="language-plaintext highlighter-rouge">--sat-solver</code>)<sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote" rel="footnote">2</a></sup>.</p>
<p>With this enhancement in CBMC, and to ease the selection of the SAT solver for Kani users, we’ve introduced a Kani attribute, <code class="language-plaintext highlighter-rouge">kani::solver</code>, that can be used to specify the SAT solver to use for each harness.
We’ve also introduced a global command-line switch, <code class="language-plaintext highlighter-rouge">--solver <SOLVER></code>, that overrides the harness <code class="language-plaintext highlighter-rouge">kani::solver</code> attribute.</p>
<p>For instance, one can configure a Kani harness to use <a href="https://github.com/arminbiere/cadical">CaDiCaL</a> as follows:</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">#[kani::proof]</span>
<span class="nd">#[kani::solver(cadical)]</span> <span class="c">// <--- Use CaDiCaL for this harness</span>
<span class="k">fn</span> <span class="nf">my_harness</span><span class="p">()</span> <span class="p">{</span> <span class="o">...</span> <span class="p">}</span>
</code></pre></div></div>
<p>Changing the solver can result in orders of magnitude performance difference. Thus, we encourage users to try different solvers to find the one that performs the best on their harness. At the time of writing, the following three solvers are supported out of the box by Kani: <code class="language-plaintext highlighter-rouge">minisat</code> (the default solver <a href="https://github.com/niklasso/minisat">Minisat</a>), <code class="language-plaintext highlighter-rouge">cadical</code> (<a href="https://github.com/arminbiere/cadical">Cadical</a>), and <code class="language-plaintext highlighter-rouge">kissat</code> (<a href="https://github.com/arminbiere/kissat">Kissat</a>). Kani also allows using other SAT solvers available as standalone binaries in your system <code class="language-plaintext highlighter-rouge">PATH</code>. This can be done using:</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">#[kani::solver(bin=</span><span class="s">"<SAT_SOLVER_BINARY>"</span><span class="nd">)]</span>
</code></pre></div></div>
<p>An example of a SAT solver that we’ve found effective for certain classes of programs (e.g. ones involving cryptographic operations) is <a href="https://github.com/msoos/cryptominisat">CryptoMiniSat</a>. After installing CryptoMiniSat and adding the binary to your path, you can configure a Kani harness to use it via:</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">#[kani::solver(bin=</span><span class="s">"cryptominisat5"</span><span class="nd">)]</span>
</code></pre></div></div>
<p>The graphs below show a comparison of the runtimes in seconds obtained using the global <code class="language-plaintext highlighter-rouge">--solver</code> switch with MiniSat, CaDiCaL and Kissat on the Kani harnesses in the <a href="https://github.com/aws/s2n-quic"><code class="language-plaintext highlighter-rouge">s2n-quic</code></a> repository, with a timeout of 30 minutes.</p>
<p>The comparison was done using Kani 0.33.0 and CBMC 5.88.1.</p>
<p><img src="/kani-verifier-blog/assets/images/cadical.png" alt="Minisat vs Cadical" />
<img src="/kani-verifier-blog/assets/images/kissat.png" alt="Minisat vs Kissat" /></p>
<p>We see that Kissat and CaDiCaL can solve verification harnesses that would timeout with MiniSat, and provide significant speedups on some other verification harnesses.
For example, for <code class="language-plaintext highlighter-rouge">random::tests::gen_range_biased_test</code>, verification time goes down from 1460 seconds with MiniSat to 6.8 and 5.5 seconds with CaDiCaL and Kissat, respectively, thereby providing speedups of more than 200X.
Similarly, for <code class="language-plaintext highlighter-rouge">sync::spsc::tests::alloc_test</code>, verification time goes down from 1004 seconds with MiniSat to 63 seconds with Kissat (a 16X speedup), and 70 seconds with CaDiCaL (a 14X speedup).</p>
<p>We also see that MiniSat remains the fastest solver for harnesses that already run in a matter of seconds.
Of the harnesses that run for more than 10 seconds, Kissat is the fastest on 47% of them, CaDiCaL is the fastest on 24%, and MiniSat on the rest.</p>
<p>By picking the best solver for each harness using the <code class="language-plaintext highlighter-rouge">kani::solver</code> attribute, we can can bring the total cumulative runtime from 2 hours and 20 minutes down to 15 minutes (counting timeouts as 1800s), while solving two more harnesses. Great savings if your harnesses are run in CI!</p>
<h2 id="adding-direct-export-of-goto-binaries">Adding Direct Export of GOTO Binaries</h2>
<p><code class="language-plaintext highlighter-rouge">kani-compiler</code> translates a Rust program into a GOTO program and exports it as a <em>symbol table</em> for CBMC to analyse. A <em>symbol table</em> stores the definitions and source locations of all symbols manipulated by the program (type symbols and their definitions, static symbols and their initial value, function symbols and their bodies, etc.).
The symbol table contents hence mostly consists of serialized abstract syntax trees.
Originally, Kani serialized symbol tables to JSON and then used the CBMC utility <code class="language-plaintext highlighter-rouge">symtab2gb</code> to translate the JSON symbol table into a <em>GOTO binary</em>, which is the format that CBMC actually expects as input.
The JSON to GOTO binary conversion was one of the most time consuming steps in the compilation stage.
We implemented direct GOTO binary serialisation in Kani, which allows us to skip the costly invocation of <code class="language-plaintext highlighter-rouge">symtab2gb</code>.
Kani can now perform the MIR-to-GOTO code generation and GOTO binary export 4X faster than before.</p>
<p>The table below reports total time and memory consumption when running with <code class="language-plaintext highlighter-rouge">kani-0.33.0</code> for code generation only, for three crates of the <code class="language-plaintext highlighter-rouge">s2n-quic</code> project, with JSON symbol table export versus GOTO binary export.</p>
<table>
<thead>
<tr>
<th>Crate</th>
<th style="text-align: right">User time (JSON)</th>
<th style="text-align: right">User time (GOTO bin.)</th>
<th style="text-align: right">Peak Mem. (JSON)</th>
<th style="text-align: right">Peak Mem. (GOTO bin.)</th>
</tr>
</thead>
<tbody>
<tr>
<td><code class="language-plaintext highlighter-rouge">s2n-quic-core</code></td>
<td style="text-align: right">198s</td>
<td style="text-align: right"><strong>111s</strong></td>
<td style="text-align: right"><strong>669MB</strong></td>
<td style="text-align: right">815MB</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">s2n-quic-platform</code></td>
<td style="text-align: right">99s</td>
<td style="text-align: right"><strong>91s</strong></td>
<td style="text-align: right"><strong>449MB</strong></td>
<td style="text-align: right">450MB</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">s2n-quic-xdp</code></td>
<td style="text-align: right">92s</td>
<td style="text-align: right"><strong>95s</strong></td>
<td style="text-align: right"><strong>450MB</strong></td>
<td style="text-align: right">450MB</td>
</tr>
</tbody>
</table>
<p>For <code class="language-plaintext highlighter-rouge">s2n-quic-core</code>, which contains 37 verification harnesses, we observe a reduction of 45% of <em>User time</em> from 199s down to 111s, and a 20% memory consumption increase from ~670Mb to ~815Mb.
For the two other crates which contain respectively 3 and 1 harnesses, the <em>User time</em> is dominated by Rust to MIR compilation and the total gains with GOTO binary export are lower
at roughly 10% and even degrade to negative 4%, albeit for virtually no memory consumption increase.</p>
<p>Looking in more detail at GOTO code generation and GOTO binary export time for individual harnesses, excluding the Rust-to-MIR compilation time,
we see that GOTO binary export is ~9x faster than JSON export, and that the GOTO code generation and export step is now ~4x faster (see the <em>Detailed table</em> below). The savings seem small but accumulate to eventually make a noticeable difference.</p>
<p>GOTO binary export is now the default export mode, because it is faster when there are multiple verification harnesses in a crate, which is more often the case in practice.
The memory consumption can be sometimes higher than with JSON export, but it can also sometimes be lower depending on how much opportunity for sharing of identical subtrees the crate offers (on large crates like <code class="language-plaintext highlighter-rouge">std</code> or <code class="language-plaintext highlighter-rouge">proc_macro</code> we’ve observed up to 2x reduction in memory usage). Should you ever need it, JSON export can still be activated with the command line switch <code class="language-plaintext highlighter-rouge">--write-json-symbtab</code>.</p>
<details>
<summary>
Detailed GOTO codegen and export times table
</summary>
<div>
<table>
<thead>
<tr>
<th>verification harness</th>
<th style="text-align: right">GOTO codegen time (s)</th>
<th style="text-align: right">JSON export (s)</th>
<th style="text-align: right">GOTO-binary export (s)</th>
<th style="text-align: right">export speedup</th>
<th style="text-align: right">codegen + export speedup</th>
</tr>
</thead>
<tbody>
<tr>
<td><code class="language-plaintext highlighter-rouge">ct::tests::ct_ge...</code></td>
<td style="text-align: right">0.32</td>
<td style="text-align: right">1.48</td>
<td style="text-align: right"><strong>0.17</strong></td>
<td style="text-align: right">8.70</td>
<td style="text-align: right">3.67</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">ct::tests::ct_gt...</code></td>
<td style="text-align: right">0.26</td>
<td style="text-align: right">1.32</td>
<td style="text-align: right"><strong>0.14</strong></td>
<td style="text-align: right">9.42</td>
<td style="text-align: right">3.95</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">ct::tests::ct_le...</code></td>
<td style="text-align: right">0.26</td>
<td style="text-align: right">1.32</td>
<td style="text-align: right"><strong>0.14</strong></td>
<td style="text-align: right">9.42</td>
<td style="text-align: right">3.95</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">ct::tests::ct_lt...</code></td>
<td style="text-align: right">0.26</td>
<td style="text-align: right">1.33</td>
<td style="text-align: right"><strong>0.14</strong></td>
<td style="text-align: right">9.50</td>
<td style="text-align: right">3.97</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">ct::tests::rem...</code></td>
<td style="text-align: right">0.26</td>
<td style="text-align: right">1.32</td>
<td style="text-align: right"><strong>0.16</strong></td>
<td style="text-align: right">8.25</td>
<td style="text-align: right">3.76</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">ct::tests::sub...</code></td>
<td style="text-align: right">0.26</td>
<td style="text-align: right">1.33</td>
<td style="text-align: right"><strong>0.14</strong></td>
<td style="text-align: right">9.50</td>
<td style="text-align: right">3.97</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">ct::tests::div...</code></td>
<td style="text-align: right">0.26</td>
<td style="text-align: right">1.33</td>
<td style="text-align: right"><strong>0.14</strong></td>
<td style="text-align: right">9.50</td>
<td style="text-align: right">3.97</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">ct::tests::add...</code></td>
<td style="text-align: right">0.26</td>
<td style="text-align: right">1.33</td>
<td style="text-align: right"><strong>0.14</strong></td>
<td style="text-align: right">9.50</td>
<td style="text-align: right">3.97</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">ct::tests::mul...</code></td>
<td style="text-align: right">0.26</td>
<td style="text-align: right">1.32</td>
<td style="text-align: right"><strong>0.14</strong></td>
<td style="text-align: right">9.42</td>
<td style="text-align: right">3.95</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">frame::crypto::tests::try_fit_test...</code></td>
<td style="text-align: right">0.66</td>
<td style="text-align: right">3.31</td>
<td style="text-align: right"><strong>0.39</strong></td>
<td style="text-align: right">8.48</td>
<td style="text-align: right">3.78</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">frame::stream::tests::try_fit_test...</code></td>
<td style="text-align: right">0.67</td>
<td style="text-align: right">3.44</td>
<td style="text-align: right"><strong>0.38</strong></td>
<td style="text-align: right">9.05</td>
<td style="text-align: right">3.91</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">inet::checksum::tests::differential...</code></td>
<td style="text-align: right">0.51</td>
<td style="text-align: right">2.46</td>
<td style="text-align: right"><strong>0.26</strong></td>
<td style="text-align: right">9.46</td>
<td style="text-align: right">3.85</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">inet::ipv4::tests::header_getter_setter_test...</code></td>
<td style="text-align: right">0.44</td>
<td style="text-align: right">2.11</td>
<td style="text-align: right"><strong>0.22</strong></td>
<td style="text-align: right">9.59</td>
<td style="text-align: right">3.86</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">inet::ipv4::tests::scope_test...</code></td>
<td style="text-align: right">1.01</td>
<td style="text-align: right">5.18</td>
<td style="text-align: right"><strong>0.56</strong></td>
<td style="text-align: right">9.25</td>
<td style="text-align: right">3.94</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">inet::ipv6::tests::header_getter_setter_test...</code></td>
<td style="text-align: right">0.44</td>
<td style="text-align: right">2.17</td>
<td style="text-align: right"><strong>0.23</strong></td>
<td style="text-align: right">9.43</td>
<td style="text-align: right">3.89</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">inet::ipv6::tests::scope_test...</code></td>
<td style="text-align: right">1.08</td>
<td style="text-align: right">5.34</td>
<td style="text-align: right"><strong>0.59</strong></td>
<td style="text-align: right">9.05</td>
<td style="text-align: right">3.84</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">interval_set::tests::interval_set_inset_range_test...</code></td>
<td style="text-align: right">0.83</td>
<td style="text-align: right">4.3</td>
<td style="text-align: right"><strong>0.45</strong></td>
<td style="text-align: right">9.55</td>
<td style="text-align: right">4.00</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">packet::number::map::tests::insert_value...</code></td>
<td style="text-align: right">0.28</td>
<td style="text-align: right">1.49</td>
<td style="text-align: right"><strong>0.16</strong></td>
<td style="text-align: right">9.31</td>
<td style="text-align: right">4.02</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">packet::number::sliding_window::test::insert_test...</code></td>
<td style="text-align: right">0.86</td>
<td style="text-align: right">4.22</td>
<td style="text-align: right"><strong>0.46</strong></td>
<td style="text-align: right">9.17</td>
<td style="text-align: right">3.84</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">packet::number::tests::example_test...</code></td>
<td style="text-align: right">1.09</td>
<td style="text-align: right">5.67</td>
<td style="text-align: right"><strong>0.59</strong></td>
<td style="text-align: right">9.61</td>
<td style="text-align: right">4.02</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">packet::number::tests::rfc_differential_test...</code></td>
<td style="text-align: right">0.30</td>
<td style="text-align: right">1.62</td>
<td style="text-align: right"><strong>0.18</strong></td>
<td style="text-align: right">9.40</td>
<td style="text-align: right">4.0</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">packet::number::tests::truncate_expand_test...</code></td>
<td style="text-align: right">0.63</td>
<td style="text-align: right">3.23</td>
<td style="text-align: right"><strong>0.35</strong></td>
<td style="text-align: right">9.22</td>
<td style="text-align: right">3.9</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">packet::number::tests::round_trip...</code></td>
<td style="text-align: right">0.22</td>
<td style="text-align: right">1.18</td>
<td style="text-align: right"><strong>0.13</strong></td>
<td style="text-align: right">9.00</td>
<td style="text-align: right">4.00</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">random::tests::gen_range_biased_test...</code></td>
<td style="text-align: right">0.31</td>
<td style="text-align: right">1.58</td>
<td style="text-align: right"><strong>0.18</strong></td>
<td style="text-align: right">8.77</td>
<td style="text-align: right">3.85</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">recovery::rtt_estimator::test::weighted_average_test...</code></td>
<td style="text-align: right">0.43</td>
<td style="text-align: right">2.23</td>
<td style="text-align: right"><strong>0.24</strong></td>
<td style="text-align: right">9.29</td>
<td style="text-align: right">3.97</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">slice::tests::vectored_copy_fuzz_test...</code></td>
<td style="text-align: right">0.73</td>
<td style="text-align: right">3.33</td>
<td style="text-align: right"><strong>0.35</strong></td>
<td style="text-align: right">9.51</td>
<td style="text-align: right">3.75</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">stream::iter::fuzz_target::fuzz_builder...</code></td>
<td style="text-align: right">0.26</td>
<td style="text-align: right">1.35</td>
<td style="text-align: right"><strong>0.14</strong></td>
<td style="text-align: right">9.64</td>
<td style="text-align: right">4.02</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">sync::cursor::tests::oracle_test...</code></td>
<td style="text-align: right">0.92</td>
<td style="text-align: right">4.41</td>
<td style="text-align: right"><strong>0.46</strong></td>
<td style="text-align: right">9.58</td>
<td style="text-align: right">3.86</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">sync::spsc::tests::alloc_test...</code></td>
<td style="text-align: right">0.82</td>
<td style="text-align: right">4.15</td>
<td style="text-align: right"><strong>0.45</strong></td>
<td style="text-align: right">9.22</td>
<td style="text-align: right">3.91</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">varint::tests::checked_ops_test...</code></td>
<td style="text-align: right">0.95</td>
<td style="text-align: right">4.94</td>
<td style="text-align: right"><strong>0.55</strong></td>
<td style="text-align: right">8.98</td>
<td style="text-align: right">3.92</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">varint::tests::table_differential_test...</code></td>
<td style="text-align: right">0.95</td>
<td style="text-align: right">4.93</td>
<td style="text-align: right"><strong>0.51</strong></td>
<td style="text-align: right">9.66</td>
<td style="text-align: right">4.02</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">varint::tests::eight_byte_sequence_test...</code></td>
<td style="text-align: right">0.96</td>
<td style="text-align: right">4.97</td>
<td style="text-align: right"><strong>0.52</strong></td>
<td style="text-align: right">9.55</td>
<td style="text-align: right">4.00</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">varint::tests::four_byte_sequence_test...</code></td>
<td style="text-align: right">0.95</td>
<td style="text-align: right">4.95</td>
<td style="text-align: right"><strong>0.52</strong></td>
<td style="text-align: right">9.51</td>
<td style="text-align: right">4.01</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">varint::tests::two_byte_sequence_test...</code></td>
<td style="text-align: right">0.97</td>
<td style="text-align: right">4.98</td>
<td style="text-align: right"><strong>0.52</strong></td>
<td style="text-align: right">9.57</td>
<td style="text-align: right">3.99</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">varint::tests::one_byte_sequence_test...</code></td>
<td style="text-align: right">0.25</td>
<td style="text-align: right">1.32</td>
<td style="text-align: right"><strong>0.14</strong></td>
<td style="text-align: right">9.42</td>
<td style="text-align: right">4.02</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">varint::tests::round_trip_values_test...</code></td>
<td style="text-align: right">0.27</td>
<td style="text-align: right">1.38</td>
<td style="text-align: right"><strong>0.15</strong></td>
<td style="text-align: right">9.20</td>
<td style="text-align: right">3.92</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">xdp::decoder::tests::decode_test...</code></td>
<td style="text-align: right">0.58</td>
<td style="text-align: right">2.93</td>
<td style="text-align: right"><strong>0.31</strong></td>
<td style="text-align: right">9.45</td>
<td style="text-align: right">3.94</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">message::cmsg::tests::round_trip_test...</code></td>
<td style="text-align: right">0.37</td>
<td style="text-align: right">1.59</td>
<td style="text-align: right"><strong>0.19</strong></td>
<td style="text-align: right">8.36</td>
<td style="text-align: right">3.50</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">message::cmsg::tests::iter_test...</code></td>
<td style="text-align: right">0.77</td>
<td style="text-align: right">3.75</td>
<td style="text-align: right"><strong>0.41</strong></td>
<td style="text-align: right">9.14</td>
<td style="text-align: right">3.83</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">message::msg::tests::address_inverse_pair_test...</code></td>
<td style="text-align: right">0.85</td>
<td style="text-align: right">4.07</td>
<td style="text-align: right"><strong>0.43</strong></td>
<td style="text-align: right">9.46</td>
<td style="text-align: right">3.84</td>
</tr>
<tr>
<td><code class="language-plaintext highlighter-rouge">task::completion_to_tx::assign::tests::assignment_test...</code></td>
<td style="text-align: right">0.36</td>
<td style="text-align: right">1.48</td>
<td style="text-align: right"><strong>0.17</strong></td>
<td style="text-align: right">8.70</td>
<td style="text-align: right">3.47</td>
</tr>
<tr>
<td> </td>
<td style="text-align: right"><strong>Total</strong></td>
<td style="text-align: right"><strong>Total</strong></td>
<td style="text-align: right"><strong>Total</strong></td>
<td style="text-align: right"><strong>Avg</strong></td>
<td style="text-align: right"><strong>Avg</strong></td>
</tr>
<tr>
<td> </td>
<td style="text-align: right">22.8</td>
<td style="text-align: right">114.66</td>
<td style="text-align: right"><strong>12.33</strong></td>
<td style="text-align: right">9.29</td>
<td style="text-align: right">3.90</td>
</tr>
</tbody>
</table>
</div>
</details>
<h2 id="enabling-constant-propagation-for-individual-fields-of-union-types">Enabling Constant Propagation for Individual Fields of Union Types</h2>
<p>Union types are very common in goto-programs emitted by Kani, due to the fact that Rust code typically uses <code class="language-plaintext highlighter-rouge">enums</code>, which are themselves modelled as tagged unions at the goto-program level.
Initially the <em>field sensitivity</em> transform in CBMC enabled constant propagation for individual array cells and individual struct fields, but not for union fields.
Since constant propagation helps pruning control flow branches during symbolic execution and can greatly reduce the runtime of an analysis, ensuring that constant propagation also works for union fields is important for Rust programs.
Field-sensitivity was first extended to unions in <code class="language-plaintext highlighter-rouge">cbmc 5.71.0</code>, but did not make it to Kani until <code class="language-plaintext highlighter-rouge">kani v0.17.0</code> built on top of <code class="language-plaintext highlighter-rouge">cbmc v5.72.0</code>.
The feature was then refined and stabilized in several iterations and became stable with <code class="language-plaintext highlighter-rouge">cbmc v5.85.0</code> in early June 2023, and released through <code class="language-plaintext highlighter-rouge">kani v0.31.0</code> built on top of <code class="language-plaintext highlighter-rouge">cbmc v5.86.0</code>.
This new CBMC feature vastly improved performance for Rust programs manipulating <code class="language-plaintext highlighter-rouge">Vec<T></code> and <code class="language-plaintext highlighter-rouge">BTreeSet<T></code> data types, and allowed us to solve a number performance issues reported by our users: <a href="https://github.com/model-checking/kani/issues/705">#705</a>, <a href="https://github.com/model-checking/kani/issues/1226">#1226</a>, <a href="https://github.com/model-checking/kani/issues/1657">#1657</a>, <a href="https://github.com/model-checking/kani/issues/1673">#1673</a>, <a href="https://github.com/model-checking/kani/issues/1676">#1676</a>.</p>
<p>The following tables and plots were obtained by running the kani <code class="language-plaintext highlighter-rouge">perf</code> test suite with <code class="language-plaintext highlighter-rouge">kani 0.33.0</code>, <code class="language-plaintext highlighter-rouge">cbmc 5.88.1</code> with <code class="language-plaintext highlighter-rouge">cadical</code> as the default SAT solver for all tests, a timeout of 900s, with and without applying the union-field sensitivity transform.</p>
<p>Field sensitivity allows 5 new verification harnesses to be solved under the 900s limit: <code class="language-plaintext highlighter-rouge">btreeset/insert_any/main</code>, <code class="language-plaintext highlighter-rouge">btreeset/insert_multi/insert_multi</code>, <code class="language-plaintext highlighter-rouge">btreeset/insert_same/main</code>, <code class="language-plaintext highlighter-rouge">misc/display_trait/slow</code>, <code class="language-plaintext highlighter-rouge">misc/struct_defs/fast_harness</code>.</p>
<p><img src="/kani-verifier-blog/assets/images/field-sens-plots/total-time.png" alt="Total time" /></p>
<p>We observe a 10% to 99% reduction in runtime for roughly a third of verification harnesses (and 5 newly solved harnesses within the 900s time limit), but we also observe a 10% to 55% runtime degradation for 25% of the verification harnesses, so this feature does not bring a consistent benefit. However, the cumulative total time to run the <code class="language-plaintext highlighter-rouge">perf</code> suite without union field sensitivity is roughly 6500s, and it drops to 1784s with union field sensitivity activated. Disabling union field sensitivity for harnesses where it degrades performance would only bring the cumulative total time down to 1676s. So we can say getting rid of timeouts and gains on a subset of harnesses offset the losses on the rest of harnesses.</p>
<details>
<summary>
Detailed total time table
</summary>
<div>
<table>
<thead>
<tr>
<th>Verification harness</th>
<th>no-sens</th>
<th>sens</th>
<th>best</th>
<th>change</th>
</tr>
</thead>
<tbody>
<tr>
<td>misc/struct_defs/fast_harness</td>
<td>900.00</td>
<td>0.89</td>
<td>0.89</td>
<td>-99.9%</td>
</tr>
<tr>
<td>misc/display_trait/slow</td>
<td>900.00</td>
<td>2.67</td>
<td>2.67</td>
<td>-99.7%</td>
</tr>
<tr>
<td>misc/struct_defs/slow_harness2</td>
<td>66.72</td>
<td>0.24</td>
<td>0.24</td>
<td>-99.6%</td>
</tr>
<tr>
<td>btreeset/insert_any/main</td>
<td>900</td>
<td>5.14</td>
<td>5.14</td>
<td>-99.4%</td>
</tr>
<tr>
<td>misc/struct_defs/slow_harness1</td>
<td>20.33</td>
<td>0.21</td>
<td>0.21</td>
<td>-98.9%</td>
</tr>
<tr>
<td>vec/box_dyn/main</td>
<td>97.24</td>
<td>1.79</td>
<td>1.79</td>
<td>-98.1%</td>
</tr>
<tr>
<td>btreeset/insert_same/main</td>
<td>900</td>
<td>16.65</td>
<td>16.65</td>
<td>-98.1%</td>
</tr>
<tr>
<td>btreeset/insert_multi/insert_multi</td>
<td>900</td>
<td>19.87</td>
<td>19.87</td>
<td>-97.7%</td>
</tr>
<tr>
<td>vec/string/main</td>
<td>142.38</td>
<td>4.02</td>
<td>4.02</td>
<td>-97.1%</td>
</tr>
<tr>
<td>s2n-quic/quic/s2n-quic-core/inet::checksum::tests::differential</td>
<td>66.29</td>
<td>26.03</td>
<td>26.03</td>
<td>-60.7%</td>
</tr>
<tr>
<td>s2n-quic/quic/s2n-quic-core/interval_set::tests::interval_set_inset_range_test</td>
<td>4.88</td>
<td>3.87</td>
<td>3.87</td>
<td>-20.6%</td>
</tr>
<tr>
<td>s2n-quic/quic/s2n-quic-core/ct::tests::rem</td>
<td>1.33</td>
<td>1.11</td>
<td>1.11</td>
<td>-16.4%</td>
</tr>
<tr>
<td>s2n-quic/quic/s2n-quic-core/ct::tests::div</td>
<td>1.15</td>
<td>0.98</td>
<td>0.98</td>
<td>-14.4%</td>
</tr>
<tr>
<td>misc/display_trait/fast</td>
<td>2.43</td>
<td>2.13</td>
<td>2.13</td>
<td>-12.3%</td>
</tr>
<tr>
<td>s2n-quic/quic/s2n-quic-core/frame::crypto::tests::try_fit_test</td>
<td>8.9</td>
<td>7.9</td>
<td>7.9</td>
<td>-11.2%</td>
</tr>
<tr>
<td>s2n-quic/quic/s2n-quic-core/frame::stream::tests::try_fit_test</td>
<td>41.29</td>
<td>36.83</td>
<td>36.83</td>
<td>-10.8%</td>
</tr>
<tr>
<td>s2n-quic/quic/s2n-quic-core/packet::number::tests::truncate_expand_test</td>
<td>2.75</td>
<td>2.59</td>
<td>2.59</td>
<td>-5.9%</td>
</tr>
<tr>
<td>s2n-quic/quic/s2n-quic-core/varint::tests::table_differential_test</td>
<td>1.15</td>
<td>1.08</td>
<td>1.08</td>
<td>-5.8%</td>
</tr>
<tr>
<td>s2n-quic/quic/s2n-quic-core/varint::tests::round_trip_values_test</td>
<td>19.27</td>
<td>18.15</td>
<td>18.15</td>
<td>-5.7%</td>
</tr>
<tr>
<td>s2n-quic/tools/xdp/s2n-quic-xdp/task::completion_to_tx::assign::tests::assignment_test</td>
<td>47.06</td>
<td>44.53</td>
<td>44.53</td>
<td>-5.3%</td>
</tr>
<tr>
<td>s2n-quic/quic/s2n-quic-core/slice::tests::vectored_copy_fuzz_test</td>
<td>81.54</td>
<td>77.71</td>
<td>77.71</td>
<td>-4.7%</td>
</tr>
<tr>
<td>s2n-quic/quic/s2n-quic-core/packet::number::tests::example_test</td>
<td>0.38</td>
<td>0.36</td>
<td>0.36</td>
<td>-4.4%</td>
</tr>
<tr>
<td>s2n-quic/quic/s2n-quic-core/sync::cursor::tests::oracle_test</td>
<td>474.06</td>
<td>459.59</td>
<td>459.59</td>
<td>-3.0%</td>
</tr>
<tr>
<td>format/fmt_i8</td>
<td>43.83</td>
<td>42.94</td>
<td>42.94</td>
<td>-2.0%</td>
</tr>
<tr>
<td>misc/array_fold/array_sum_fold_proof</td>
<td>0.93</td>
<td>0.92</td>
<td>0.92</td>
<td>-1.4%</td>
</tr>
<tr>
<td>s2n-quic/quic/s2n-quic-core/packet::number::tests::rfc_differential_test</td>
<td>3.89</td>
<td>3.85</td>
<td>3.85</td>
<td>-1.1%</td>
</tr>
<tr>
<td>s2n-quic/quic/s2n-quic-platform/message::cmsg::tests::iter_test</td>
<td>19.5</td>
<td>19.33</td>
<td>19.33</td>
<td>-0.8%</td>
</tr>
<tr>
<td>s2n-quic/quic/s2n-quic-core/ct::tests::mul</td>
<td>0.75</td>
<td>0.75</td>
<td>0.75</td>
<td>-0.2%</td>
</tr>
<tr>
<td>s2n-quic/quic/s2n-quic-core/packet::number::map::tests::insert_value</td>
<td>2.62</td>
<td>2.65</td>
<td>2.62</td>
<td>1.1%</td>
</tr>
<tr>
<td>vec/vec/main</td>
<td>2.18</td>
<td>2.21</td>
<td>2.18</td>
<td>1.7%</td>
</tr>
<tr>
<td>s2n-quic/quic/s2n-quic-core/ct::tests::add</td>
<td>0.68</td>
<td>0.69</td>
<td>0.68</td>
<td>2.4%</td>
</tr>
<tr>
<td>s2n-quic/quic/s2n-quic-core/ct::tests::ct_gt</td>
<td>0.63</td>
<td>0.65</td>
<td>0.63</td>
<td>2.7%</td>
</tr>
<tr>
<td>s2n-quic/quic/s2n-quic-core/inet::ipv4::tests::scope_test</td>
<td>4.25</td>
<td>4.38</td>
<td>4.25</td>
<td>3.2%</td>
</tr>
<tr>
<td>s2n-quic/quic/s2n-quic-core/ct::tests::sub</td>
<td>0.67</td>
<td>0.69</td>
<td>0.67</td>
<td>3.4%</td>
</tr>
<tr>
<td>s2n-quic/quic/s2n-quic-core/ct::tests::ct_lt</td>
<td>0.63</td>
<td>0.65</td>
<td>0.63</td>
<td>3.9%</td>
</tr>
<tr>
<td>s2n-quic/quic/s2n-quic-core/packet::number::tests::round_trip</td>
<td>19.38</td>
<td>20.2</td>
<td>19.38</td>
<td>4.2%</td>
</tr>
<tr>
<td>s2n-quic/quic/s2n-quic-core/ct::tests::ct_le</td>
<td>0.62</td>
<td>0.65</td>
<td>0.62</td>
<td>4.2%</td>
</tr>
<tr>
<td>s2n-quic/quic/s2n-quic-core/ct::tests::ct_ge</td>
<td>0.62</td>
<td>0.65</td>
<td>0.62</td>
<td>4.9%</td>
</tr>
<tr>
<td>s2n-quic/quic/s2n-quic-core/varint::tests::eight_byte_sequence_test</td>
<td>13.66</td>
<td>14.37</td>
<td>13.66</td>
<td>5.1%</td>
</tr>
<tr>
<td>s2n-quic/quic/s2n-quic-core/stream::iter::fuzz_target::fuzz_builder</td>
<td>0.82</td>
<td>0.87</td>
<td>0.82</td>
<td>6.3%</td>
</tr>
<tr>
<td>s2n-quic/quic/s2n-quic-core/packet::number::sliding_window::test::insert_test</td>
<td>1.46</td>
<td>1.55</td>
<td>1.46</td>
<td>6.5%</td>
</tr>
<tr>
<td>s2n-quic/quic/s2n-quic-core/inet::ipv4::tests::header_getter_setter_test</td>
<td>19.59</td>
<td>21.04</td>
<td>19.59</td>
<td>7.3%</td>
</tr>
<tr>
<td>s2n-quic/quic/s2n-quic-platform/message::cmsg::tests::round_trip_test</td>
<td>463.27</td>
<td>497.46</td>
<td>463.27</td>
<td>7.3%</td>
</tr>
<tr>
<td>misc/array_fold/array_sum_for_proof</td>
<td>0.86</td>
<td>0.95</td>
<td>0.86</td>
<td>9.3%</td>
</tr>
<tr>
<td>s2n-quic/quic/s2n-quic-core/inet::ipv6::tests::scope_test</td>
<td>16.4</td>
<td>17.95</td>
<td>16.4</td>
<td>9.4%</td>
</tr>
<tr>
<td>s2n-quic/quic/s2n-quic-core/inet::ipv6::tests::header_getter_setter_test</td>
<td>48.89</td>
<td>53.53</td>
<td>48.89</td>
<td>9.4%</td>
</tr>
<tr>
<td>s2n-quic/quic/s2n-quic-core/recovery::rtt_estimator::test::weighted_average_test</td>
<td>117.62</td>
<td>129.32</td>
<td>117.62</td>
<td>9.9%</td>
</tr>
<tr>
<td>s2n-quic/quic/s2n-quic-core/varint::tests::checked_ops_test</td>
<td>2.49</td>
<td>2.76</td>
<td>2.49</td>
<td>11.0%</td>
</tr>
<tr>
<td>s2n-quic/quic/s2n-quic-core/varint::tests::two_byte_sequence_test</td>
<td>11.21</td>
<td>12.5</td>
<td>11.21</td>
<td>11.5%</td>
</tr>
<tr>
<td>s2n-quic/quic/s2n-quic-core/varint::tests::four_byte_sequence_test</td>
<td>11.64</td>
<td>14.06</td>
<td>11.64</td>
<td>20.7%</td>
</tr>
<tr>
<td>s2n-quic/quic/s2n-quic-core/varint::tests::one_byte_sequence_test</td>
<td>8.76</td>
<td>10.82</td>
<td>8.76</td>
<td>23.5%</td>
</tr>
<tr>
<td>format/fmt_u8</td>
<td>9.41</td>
<td>12.06</td>
<td>9.41</td>
<td>28.2%</td>
</tr>
<tr>
<td>s2n-quic/quic/s2n-quic-core/sync::spsc::tests::alloc_test</td>
<td>92.84</td>
<td>126.45</td>
<td>92.84</td>
<td>36.2%</td>
</tr>
<tr>
<td>s2n-quic/quic/s2n-quic-core/random::tests::gen_range_biased_test</td>
<td>10.48</td>
<td>14.76</td>
<td>10.48</td>
<td>40.8%</td>
</tr>
<tr>
<td>s2n-quic/quic/s2n-quic-core/xdp::decoder::tests::decode_test</td>
<td>5.17</td>
<td>7.59</td>
<td>5.17</td>
<td>46.7%</td>
</tr>
<tr>
<td>s2n-quic/quic/s2n-quic-platform/message::msg::tests::address_inverse_pair_test</td>
<td>7.09</td>
<td>10.98</td>
<td>7.09</td>
<td>54.8</td>
</tr>
<tr>
<td> </td>
<td><strong>total</strong></td>
<td><strong>total</strong></td>
<td><strong>total</strong></td>
<td> </td>
</tr>
<tr>
<td> </td>
<td>6521.99</td>
<td>1784.57</td>
<td>1676.07</td>
<td> </td>
</tr>
</tbody>
</table>
</div>
</details>
<p><img src="/kani-verifier-blog/assets/images/field-sens-plots/symex-time.png" alt="Symex time" /></p>
<p>We see that symbolic execution time is sometimes improved, sometimes degraded. The degradation can possibly be explained by the fact that symex has to generate more constraints and handle more basic variables with the transform activated, and spend more time applying simplifications. But this extra work has a beneficial impact on the number of symbolic execution steps and number of VCCs generated for the SAT solver, and ultimately SAT solver runtime (as seen below).</p>
<p><img src="/kani-verifier-blog/assets/images/field-sens-plots/symex-steps.png" alt="Symex steps" /></p>
<p>The number of basic symbolic execution steps is sometimes slightly higher but otherwise mostly lower with union field sensitivity activated.</p>
<p><img src="/kani-verifier-blog/assets/images/field-sens-plots/vccs.png" alt="number of VCCs" /></p>
<p>The number of verification conditions generated is sometimes slightly higher but otherwise mostly lower with union field sensitivity activated.</p>
<p><img src="/kani-verifier-blog/assets/images/field-sens-plots/sat-time.png" alt="SAT solving time" /></p>
<p>Overall, the SAT solving time is sometimes slightly degraded but otherwise mostly improved across the board with field-sensitivity.</p>
<details>
<summary>
Details on the field-sensitivity transform
</summary>
<div>
<p>CBMC’s constant propagation algorithm only propagates values for scalar variables with a basic datatype such as <code class="language-plaintext highlighter-rouge">bool</code>, <code class="language-plaintext highlighter-rouge">int</code>, … but not for aggregates. To enable propagation for individual fields of aggregates, CBMC decomposes them into their individual scalar fields, by introducing new variables in the program, and resolving all field access expressions to these new variables.</p>
<p>For unions, just like for structs, the transform introduces a distinct variable for each field of the union. However, contrary to structs, the different fields of a union overlap in the byte-level layout. As a result, every time a union field gets assigned with a new value, all fields of the union are actually impacted, and the impact depends on how the fields overlap in the layout. This means that the variables representing the different fields of the union have to be handled as a group and globally updated after each update to any one of them.</p>
<p>For instance, for a union defined as follows (using C syntax for simplicity):</p>
<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">union</span> <span class="p">{</span>
<span class="kt">unsigned</span> <span class="kt">long</span> <span class="n">a</span><span class="p">;</span>
<span class="kt">unsigned</span> <span class="kt">int</span> <span class="n">b</span><span class="p">;</span>
<span class="p">}</span> <span class="n">u</span><span class="p">;</span>
</code></pre></div> </div>
<p>The byte-level layout of the fields is such that the lowest 4 bytes of <code class="language-plaintext highlighter-rouge">u.a</code> and all bytes of <code class="language-plaintext highlighter-rouge">u.b</code> overlap. As a result, updating <code class="language-plaintext highlighter-rouge">u.a</code> updates all bytes <code class="language-plaintext highlighter-rouge">u.b</code>, and updating <code class="language-plaintext highlighter-rouge">u.b</code> updates the lowest 4 bytes of <code class="language-plaintext highlighter-rouge">u.a</code>:</p>
<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">u</span> <span class="n">uuuuuuuuuuuuuuuu</span>
<span class="n">u</span><span class="p">.</span><span class="n">a</span> <span class="n">aaaaaaaaaaaaaaaa</span>
<span class="n">u</span><span class="p">.</span><span class="n">b</span> <span class="n">bbbbbbbb</span>
<span class="n">idx</span><span class="p">.</span> <span class="mi">16</span> <span class="mi">7</span> <span class="mi">0</span>
<span class="n">MSB</span> <span class="n">LSB</span>
</code></pre></div> </div>
<p>The transform introduces a variable <code class="language-plaintext highlighter-rouge">u_a</code> to represent <code class="language-plaintext highlighter-rouge">u.a</code>, and a variable <code class="language-plaintext highlighter-rouge">u_b</code> to represent <code class="language-plaintext highlighter-rouge">u.b</code>. These two variables are not independent, and every-time one of the fields is updated in the original program, both variables are updated in the transformed program.</p>
<p>Applying the transform to the following program:</p>
<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kt">int</span> <span class="nf">main</span><span class="p">()</span> <span class="p">{</span>
<span class="k">union</span> <span class="p">{</span>
<span class="kt">unsigned</span> <span class="kt">long</span> <span class="n">a</span><span class="p">;</span>
<span class="kt">unsigned</span> <span class="kt">int</span> <span class="n">b</span><span class="p">;</span>
<span class="p">}</span> <span class="n">u</span><span class="p">;</span>
<span class="n">u</span><span class="p">.</span><span class="n">a</span> <span class="o">=</span> <span class="mh">0x0000000000000000</span><span class="p">;</span>
<span class="n">u</span><span class="p">.</span><span class="n">b</span> <span class="o">=</span> <span class="mh">0x87654321</span><span class="p">;</span>
<span class="n">assert</span><span class="p">(</span><span class="n">u</span><span class="p">.</span><span class="n">a</span> <span class="o">==</span> <span class="mh">0x0000000087654321</span><span class="p">);</span>
<span class="n">assert</span><span class="p">(</span><span class="n">u</span><span class="p">.</span><span class="n">b</span> <span class="o">==</span> <span class="mh">0x87654321</span><span class="p">);</span>
<span class="k">return</span> <span class="mi">0</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div> </div>
<p>produces the following transformed program:</p>
<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kt">int</span> <span class="nf">main</span><span class="p">()</span> <span class="p">{</span>
<span class="kt">unsigned</span> <span class="kt">long</span> <span class="n">u_a</span><span class="p">;</span>
<span class="kt">unsigned</span> <span class="kt">int</span> <span class="n">u_b</span><span class="p">;</span>
<span class="c1">// the bytes of u_b are equal the low bytes of u_a</span>
<span class="n">u_b</span> <span class="o">=</span> <span class="p">(</span><span class="kt">unsigned</span> <span class="kt">int</span><span class="p">)</span> <span class="n">u_a</span><span class="p">;</span>
<span class="c1">// u.a = 0x0000000000000000;</span>
<span class="n">u_a</span> <span class="o">=</span> <span class="mh">0x0000000000000000</span><span class="p">;</span>
<span class="n">u_b</span> <span class="o">=</span> <span class="p">(</span><span class="kt">unsigned</span> <span class="kt">int</span><span class="p">)</span> <span class="mh">0x0000000000000000</span><span class="p">;</span>
<span class="c1">// u.b = 0x87654321;</span>
<span class="n">u_a</span> <span class="o">=</span> <span class="p">(</span><span class="n">u_a</span> <span class="o">&</span> <span class="mh">0xFFFFFFFF00000000</span><span class="p">)</span> <span class="o">|</span> <span class="p">((</span><span class="kt">unsigned</span> <span class="kt">long</span><span class="p">)</span> <span class="mh">0x87654321</span><span class="p">);</span>
<span class="n">u_b</span> <span class="o">=</span> <span class="mh">0x87654321</span><span class="p">;</span>
<span class="c1">// assert(u.a == 0x0000000087654321);</span>
<span class="n">assert</span><span class="p">(</span><span class="n">u_a</span> <span class="o">==</span> <span class="mh">0x0000000087654321</span><span class="p">);</span>
<span class="c1">// assert(u.b == 0x87654321);</span>
<span class="n">assert</span><span class="p">(</span><span class="n">u_b</span> <span class="o">==</span> <span class="mh">0x87654321</span><span class="p">);</span>
<span class="k">return</span> <span class="mi">0</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div> </div>
</div>
</details>
<h2 id="conclusion">Conclusion</h2>
<p>In conclusion, we would like to point out the multi-faceted approach to optimizing a tool such as Kani and making verification scalable in general.
Some optimizations are geared towards solving unsolvable cases, whereas others are more general in nature.
A reflection of the difficult problem space we are dealing with.
Specific tools can achieve better performance with niche optimizations, but tools like Kani have to be optimized at various levels to achieve realistic performant verification.
We hope you enjoyed the blog post and got a sense of the need for a multi-faceted approach!</p>
<hr />
<h2 id="footnotes">Footnotes</h2>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:build-details" role="doc-endnote">
<p>To verify <a href="https://doc.rust-lang.org/stable/cargo/">Cargo</a> packages, Kani employs Cargo to correctly build all the dependencies before translating them to GOTO programs. <a href="#fnref:build-details" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:1" role="doc-endnote">
<p>Without our enhancement to CBMC, it was already possible to select a different SAT solver without rebuilding CBMC via the <code class="language-plaintext highlighter-rouge">--external-sat-solver</code> option. However, this option doesn’t use the solver in incremental mode (i.e. through its library API, keeping the solver alive between successive calls), and instead relies on writing DIMACS files to disk, which often results in decreased performance. <a href="#fnref:1" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>Kani is a bit-precise model checker that can verify properties about your Rust code. To learn more about Kani, check out the Kani tutorial and our previous blog posts.Introducing the Kani VS Code Extension2023-06-30T00:00:00+00:002023-06-30T00:00:00+00:00https://model-checking.github.io//kani-verifier-blog/2023/06/30/introducing-the-kani-vscode-extension<p>Kani is a verification tool that can help you systematically test properties about your Rust code.
To learn more about Kani, check out the <a href="https://model-checking.github.io/kani/kani-tutorial.html">Kani tutorial</a> and our <a href="https://model-checking.github.io/kani-verifier-blog/">previous blog posts</a>.</p>
<p>We are delighted to introduce the <a href="https://marketplace.visualstudio.com/items?itemName=model-checking.kani-vscode-extension">Kani VS Code Extension</a>, which is now available on the VS Code marketplace.
The extension integrates the Kani verifier into the VS Code UI.
As you write Kani harnesses, the extension detects them and conveniently displays them within your testing panel.
The extension offers detailed diagnostics, feedback about verification failures, error messages, and stack traces.
This empowers you to find bugs and verify your code quicker.
You can install the extension from the <a href="https://marketplace.visualstudio.com/items?itemName=model-checking.kani-vscode-extension">webpage</a> on VS Code marketplace or by searching for <code class="language-plaintext highlighter-rouge">Kani</code> in the extensions tab of your VS Code window.</p>
<p><img src="https://model-checking.github.io/kani-verifier-blog/assets/images/vs-code-images/kani-demo.gif" style="box-shadow: 0 2px 4px rgba(0, 0, 0, 0.5);" alt="view-kani-demo" /></p>
<h2 id="working-through-a-simple-example">Working through a simple example</h2>
<p>To show some of the features in the extension, we will work through a familiar example, the rectangle example that we introduced in our <a href="https://model-checking.github.io/kani-verifier-blog/2022/05/04/announcing-the-kani-rust-verifier-project.html">first blog post</a>.</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">#[derive(Debug,</span> <span class="nd">Copy,</span> <span class="nd">Clone)]</span>
<span class="k">struct</span> <span class="n">Rectangle</span> <span class="p">{</span>
<span class="n">width</span><span class="p">:</span> <span class="nb">u64</span><span class="p">,</span>
<span class="n">height</span><span class="p">:</span> <span class="nb">u64</span><span class="p">,</span>
<span class="p">}</span>
<span class="k">impl</span> <span class="n">Rectangle</span> <span class="p">{</span>
<span class="k">fn</span> <span class="nf">can_hold</span><span class="p">(</span><span class="o">&</span><span class="k">self</span><span class="p">,</span> <span class="n">other</span><span class="p">:</span> <span class="o">&</span><span class="n">Rectangle</span><span class="p">)</span> <span class="k">-></span> <span class="nb">bool</span> <span class="p">{</span>
<span class="k">self</span><span class="py">.width</span> <span class="o">></span> <span class="n">other</span><span class="py">.width</span> <span class="o">&&</span> <span class="k">self</span><span class="py">.height</span> <span class="o">></span> <span class="n">other</span><span class="py">.height</span>
<span class="p">}</span>
<span class="k">fn</span> <span class="nf">stretch</span><span class="p">(</span><span class="o">&</span><span class="k">self</span><span class="p">,</span> <span class="n">factor</span><span class="p">:</span> <span class="nb">u64</span><span class="p">)</span> <span class="k">-></span> <span class="nb">Option</span><span class="o"><</span><span class="n">Self</span><span class="o">></span> <span class="p">{</span>
<span class="k">let</span> <span class="n">w</span> <span class="o">=</span> <span class="k">self</span><span class="py">.width</span><span class="nf">.checked_mul</span><span class="p">(</span><span class="n">factor</span><span class="p">)</span><span class="o">?</span><span class="p">;</span>
<span class="k">let</span> <span class="n">h</span> <span class="o">=</span> <span class="k">self</span><span class="py">.height</span><span class="nf">.checked_mul</span><span class="p">(</span><span class="n">factor</span><span class="p">)</span><span class="o">?</span><span class="p">;</span>
<span class="nf">Some</span><span class="p">(</span><span class="n">Rectangle</span> <span class="p">{</span> <span class="n">width</span><span class="p">:</span> <span class="n">w</span><span class="p">,</span> <span class="n">height</span><span class="p">:</span> <span class="n">h</span> <span class="p">})</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>In order to verify properties about the rectangle, we wrote a verification harness.
The harness postulates that when the rectangle is stretched, it can hold another rectangle of its original size (dimensions).
If verified succesfully, this means that for any given stretch factor and any height and width of the original rectangle, this property holds true.</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">#[kani::proof]</span>
<span class="k">pub</span> <span class="k">fn</span> <span class="nf">stretched_rectangle_can_hold_original</span><span class="p">()</span> <span class="p">{</span>
<span class="k">let</span> <span class="n">original</span> <span class="o">=</span> <span class="n">Rectangle</span> <span class="p">{</span> <span class="n">width</span><span class="p">:</span> <span class="nn">kani</span><span class="p">::</span><span class="nf">any</span><span class="p">(),</span> <span class="n">height</span><span class="p">:</span> <span class="nn">kani</span><span class="p">::</span><span class="nf">any</span><span class="p">()</span> <span class="p">};</span>
<span class="k">let</span> <span class="n">factor</span> <span class="o">=</span> <span class="nn">kani</span><span class="p">::</span><span class="nf">any</span><span class="p">();</span>
<span class="k">if</span> <span class="k">let</span> <span class="nf">Some</span><span class="p">(</span><span class="n">larger</span><span class="p">)</span> <span class="o">=</span> <span class="n">original</span><span class="nf">.stretch</span><span class="p">(</span><span class="n">factor</span><span class="p">)</span> <span class="p">{</span>
<span class="k">assert</span><span class="o">!</span><span class="p">(</span><span class="n">larger</span><span class="nf">.can_hold</span><span class="p">(</span><span class="o">&</span><span class="n">original</span><span class="p">));</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<h2 id="using-the-vs-code-extension-on-the-example">Using the VS Code extension on the example</h2>
<p>With the extension, running Kani on a verification harness, is as simple as clicking a button.
In the following sections, we’ll walk you through using the extension’s core features to debug and finally verify the rectangle example mentioned above.</p>
<h3 id="view-kani-harnesses">View Kani harnesses</h3>
<p>As soon as your Rust package is opened using the Kani extension in a VS Code instance, you should see the Kani harnesses loaded as regular unit tests in the Testing Panel on the <a href="https://code.visualstudio.com/api/ux-guidelines/sidebars#primary-sidebar">primary side bar</a> of VS Code.
This is how the testing page looks like when you click on the panel.</p>
<p><img src="https://model-checking.github.io/kani-verifier-blog/assets/images/vs-code-images/view-kani-harnesses.png" alt="view-kani-harnesses" /></p>
<h3 id="run-kani-harnesses">Run Kani harnesses</h3>
<p>You can then run your harnesses using the tree view by clicking the play button beside the harness that was automatically picked up by the Kani Extension.
Once you run the harness using the extension, you are shown a green check mark if verification succeeded, or a failure banner if it failed.</p>
<p>In our example, as with the command line, we can see through visual markers such as the failure banner pop-up and the red failure marker, that verification failed.</p>
<p>You are then presented with two options:</p>
<ol>
<li><a href="https://github.com/model-checking/kani-vscode-extension/blob/main/docs/user-guide.md#view-trace-report">Generate the report for the harness</a></li>
<li><a href="https://github.com/model-checking/kani-vscode-extension/blob/main/docs/user-guide.md#use-concrete-playback-to-debug-a-kani-harness">Run concrete playback to generate unit tests</a>.</li>
</ol>
<p><img src="https://model-checking.github.io/kani-verifier-blog/assets/images/vs-code-images/run-kani-harness.gif" style="box-shadow: 0 2px 4px rgba(0, 0, 0, 0.5);" alt="run-kani-harness" /></p>
<p>Kani’s <a href="https://model-checking.github.io/kani-verifier-blog/2022/09/22/internship-projects-2022-concrete-playback.html">concrete playback</a> feature allows you to generate unit tests that call a function with the exact arguments that caused the assertion violation.
The VS Code extension makes using concrete playback easy.
You can read more about concrete playback in our <a href="https://model-checking.github.io/kani/debugging-verification-failures.html">documentation</a>.</p>
<h3 id="generate-a-counterexample-unit-test">Generate a counterexample unit test</h3>
<p>Next, we’ll generate the unit test by clicking on the <code class="language-plaintext highlighter-rouge">Run Concrete Playback for stretched_rectangle_can_hold_original</code> link that appears through a blue link on the error banner.</p>
<p><img src="https://model-checking.github.io/kani-verifier-blog/assets/images/vs-code-images/generate-counter-example.gif" style="box-shadow: 0 2px 4px rgba(0, 0, 0, 0.5);" alt="generate-unit-test" /></p>
<p>By clicking on the link, we have our counterexample unit test pasted directly below the harness.
This is what the unit test looks like:</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">#[test]</span>
<span class="k">fn</span> <span class="nf">kani_concrete_playback_stretched_rectangle_can_hold_original</span><span class="p">()</span> <span class="p">{</span>
<span class="k">let</span> <span class="n">concrete_vals</span><span class="p">:</span> <span class="nb">Vec</span><span class="o"><</span><span class="nb">Vec</span><span class="o"><</span><span class="nb">u8</span><span class="o">>></span> <span class="o">=</span> <span class="nd">vec!</span><span class="p">[</span>
<span class="c">// 4611686018427387904ul</span>
<span class="nd">vec!</span><span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">64</span><span class="p">],</span>
<span class="c">// 0ul</span>
<span class="nd">vec!</span><span class="p">[</span><span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">],</span>
<span class="c">// 2ul</span>
<span class="nd">vec!</span><span class="p">[</span><span class="mi">2</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">0</span><span class="p">],</span>
<span class="p">];</span>
<span class="nn">kani</span><span class="p">::</span><span class="nf">concrete_playback_run</span><span class="p">(</span><span class="n">concrete_vals</span><span class="p">,</span> <span class="n">stretched_rectangle_can_hold_original</span><span class="p">);</span>
<span class="p">}</span>
</code></pre></div></div>
<p>You can see in the GIF above that the source code is now annotated with two options on top of the generated unit test, <code class="language-plaintext highlighter-rouge">Run Test (Kani)</code> and <code class="language-plaintext highlighter-rouge">Debug Test (Kani)</code>, which allow you to run and debug the test just like any other Rust unit test.</p>
<h3 id="run-kani-generated-test">Run Kani-generated test</h3>
<p>Running the unit test using the <code class="language-plaintext highlighter-rouge">Run Test (Kani)</code> button shows us what we were expecting – that the unit test is failing.
This is because the unit test is using the counterexample to invoke the function <code class="language-plaintext highlighter-rouge">stretched_rectangle_can_hold_original</code>.</p>
<p><img src="https://model-checking.github.io/kani-verifier-blog/assets/images/vs-code-images/run-concrete-playback-test.png" alt="run-concrete-playback-test" /></p>
<h3 id="debug-kani-unit-test">Debug Kani unit test</h3>
<p>In order to peek under the hood to find out the missing assumptions that lead to unexpected behavior, it is really important to look at the concrete counterexamples for which our assertions fail.
By setting breakpoints and clicking the <code class="language-plaintext highlighter-rouge">Debug Test (Kani)</code> button, you are taken into the debugger which allows you to inspect the specific values for which the assertion fails.</p>
<p><img src="https://model-checking.github.io/kani-verifier-blog/assets/images/vs-code-images/run-debugger.gif" style="box-shadow: 0 2px 4px rgba(0, 0, 0, 0.5);" alt="run-debugger" /></p>
<p>In our case, we can see that for <code class="language-plaintext highlighter-rouge">original.height = 0</code> , the larger rectangle’s height or <code class="language-plaintext highlighter-rouge">larger.height</code> also stays <code class="language-plaintext highlighter-rouge">0</code>, which shows that for that counterexample, the property <code class="language-plaintext highlighter-rouge">can_hold</code> does not hold.</p>
<h3 id="and-finally-verify-the-harness-with-the-right-assumptions">And finally, verify the harness with the right assumptions</h3>
<p>Now that we know that for <code class="language-plaintext highlighter-rouge">original.width = 0</code>, our assertion fails, we can repeat the experiment with explicit assumptions.
The experiments should reveal that for all parameters, having a <code class="language-plaintext highlighter-rouge">0</code> value will cause the assertion to fail.
Additionally, there is a problem if <code class="language-plaintext highlighter-rouge">factor</code> is <code class="language-plaintext highlighter-rouge">1</code> because in this case <code class="language-plaintext highlighter-rouge">stretch</code> will return <code class="language-plaintext highlighter-rouge">Some(...)</code>, but the stretched rectangle will be the same size as the original.
We missed these cases in our unit and property-based tests.</p>
<p>We will now add these assumptions through <code class="language-plaintext highlighter-rouge">kani::assume</code> and re-run the verification in the extension.</p>
<p><img src="https://model-checking.github.io/kani-verifier-blog/assets/images/vs-code-images/verifying-success.gif" style="box-shadow: 0 2px 4px rgba(0, 0, 0, 0.5);" alt="verifying-success" /></p>
<p>And with that green check mark, you can be assured that the harness has been verified!</p>
<h2 id="wrapping-up">Wrapping up</h2>
<p>You can use Kani through VS Code using the Kani VS Code extension, <a href="https://marketplace.visualstudio.com/items?itemName=model-checking.kani-vscode-extension">available on the marketplace</a> now.
With this new extension you can now iteratively verify your code directly in the VS Code UI.
It lets you verify harnesses with a simple click, highlights property violations, lets you generate unit tests and debug them.</p>
<p>The Kani extension has more features which weren’t mentioned in the blog, which you can read about in our <a href="https://github.com/model-checking/kani-vscode-extension/blob/main/docs/user-guide.md">user guide documentation</a>.</p>
<p>Please go ahead and try the extension yourself!</p>
<p>If you are running into issues with the Kani extension or have feature requests or suggestions, we’d <a href="https://github.com/model-checking/kani-vscode-extension/issues/new/choose">love to hear from you</a>.</p>Kani is a verification tool that can help you systematically test properties about your Rust code. To learn more about Kani, check out the Kani tutorial and our previous blog posts.How s2n-quic uses Kani to inspire confidence2023-05-30T00:00:00+00:002023-05-30T00:00:00+00:00https://model-checking.github.io//kani-verifier-blog/2023/05/30/how-s2n-quic-uses-kani-to-inspire-confidence<p><a href="https://github.com/aws/s2n-quic">s2n-quic</a> is a Rust implementation of the <a href="https://quicwg.org/">QUIC protocol</a>, a transport protocol designed for fast and secure communication between hosts in a network.
QUIC is relatively new, but it builds upon and learns from TCP (Transmission Control Protocol), the transport protocol that has been the standard for communication on the Internet for decades.
TCP is the underlying protocol for HTTP/1.1 and HTTP/2, but for the latest generation HTTP/3, QUIC is used instead.
In that respect, QUIC can be thought of as a next-generation version of TCP with new features such as stream multiplexing and connection migration.
However, the improvements over TCP go beyond just new features and functionality.
Notably, QUIC is secure by default; the specification requires TLS be used within the QUIC handshake to establish encryption and authentication keys, and subsequently every QUIC packet is protected using these keys.
With TCP, confidentiality and integrity were not part of the protocol design, and thus TLS was layered on top of TCP, as is done in HTTPS.</p>
<p>Requiring encryption by default is one of many improvements to security, functionality, and performance that the designers of the QUIC protocol learned from earlier protocols such as TCP.
And similarly, when we implemented s2n-quic, we incorporated learnings from other protocol implementations.
One of these learnings was the importance of using a verification tool like Kani to gain confidence in the properties we assert about critical code that is used for processing terabytes or more of data formatted, transmitted and encrypted as part of the QUIC protocol.
In this blog post, I’ll show a few examples of how we use Kani to easily and automatically verify the correctness of our code, and ultimately inspire confidence in our team to continue improving and optimizing s2n-quic.</p>
<h2 id="optimizing-weighted-round-trip-time">Optimizing weighted round trip time</h2>
<p>Besides the importance of using Kani, another learning s2n-quic took to heart was that if a tool is not easy and minimally disruptive to use, it probably won’t get used much.
For that reason, s2n-quic makes heavy use of the <a href="https://camshaft.github.io/bolero/">Bolero</a> property-testing framework to add Kani’s verification functionality to our existing fuzz testing harnesses.
This unification of multiple testing methods in a single framework has previously been <a href="https://model-checking.github.io/kani-verifier-blog/2022/10/27/using-kani-with-the-bolero-property-testing-framework.html">covered in detail</a> on this blog, but it’s worth revisiting to see how easy it makes incorporating Kani verification into s2n-quic.
To illustrate this, I’ll dive into how we optimized an important component of s2n-quic, and in the next section we’ll see how we ensured the optimized code was correct by adding Kani to a Bolero fuzz test harness.</p>
<p>This particular code involves s2n-quic’s round trip time estimator.
For context, in QUIC and other transport protocols, round trip time (or RTT) is the amount of time it takes a packet to be transmitted to a peer plus the time it takes for the peer’s acknowledgement of that packet to be received by the sender.
RTT is an important measurement used throughout s2n-quic’s loss recovery and congestion control mechanisms.
RTT is susceptible to natural variability, so s2n-quic calculates an exponentially weighted moving average called “smoothed RTT” each time the RTT is updated, which happens when a new packet acknowledgement has been received.
The smoothed RTT is calculated by combining 7/8ths of the existing smoothed RTT measurement with 1/8th of the latest RTT measurement:</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">// smoothed_rtt and rtt are of type std::time::Duration</span>
<span class="k">self</span><span class="py">.smoothed_rtt</span> <span class="o">=</span> <span class="mi">7</span> <span class="o">*</span> <span class="k">self</span><span class="py">.smoothed_rtt</span> <span class="o">/</span> <span class="mi">8</span> <span class="o">+</span> <span class="n">rtt</span> <span class="o">/</span> <span class="mi">8</span><span class="p">;</span>
</code></pre></div></div>
<p>It turns out that this relatively straightforward implementation is less than optimal when compiled, resulting in three separate function calls in the assembly output:</p>
<pre><code class="language-asm">mov edi, 7
**call qword ptr [rip + core::time::<impl core::ops::arith::Mul<core::time::Duration> for u32>::mul@GOTPCREL]**
mov r12, qword ptr [rip + <core::time::Duration as core::ops::arith::Div<u32>>::div@GOTPCREL]
mov rdi, rax
mov esi, edx
mov edx, 8
**call r12**
mov rbp, rax
mov ebx, edx
mov rdi, r15
mov esi, r14d
mov edx, 8
**call r12**
</code></pre>
<p>This smoothed RTT calculation is performed each time an acknowledgement is received from the peer, which could end up being thousands of times per second on a high bandwidth connection.
The contribution of these function calls to CPU utilization starts to add up in such a case.
Since s2n-quic is a lower level library that other applications are built upon, it needs to use as little CPU as possible to improve performance and reduce any impact to customer applications.
Fortunately, this implementation had a straightforward fix: converting from the <code class="language-plaintext highlighter-rouge">Duration</code> type used in the calculation to the primitive type <code class="language-plaintext highlighter-rouge">u64</code>, representing the number of nanoseconds in the duration.</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">let</span> <span class="k">mut</span> <span class="n">smoothed_rtt_nanos</span> <span class="o">=</span> <span class="n">smoothed_rtt</span><span class="nf">.as_nanos</span><span class="p">()</span> <span class="k">as</span> <span class="nb">u64</span><span class="p">;</span>
<span class="n">smoothed_rtt_nanos</span> <span class="o">/=</span> <span class="mi">8</span><span class="p">;</span>
<span class="n">smoothed_rtt_nanos</span> <span class="o">*=</span> <span class="mi">7</span><span class="p">;</span>
<span class="k">let</span> <span class="k">mut</span> <span class="n">rtt_nanos</span> <span class="o">=</span> <span class="n">rtt</span><span class="nf">.as_nanos</span><span class="p">()</span> <span class="k">as</span> <span class="nb">u64</span><span class="p">;</span>
<span class="n">rtt_nanos</span> <span class="o">/=</span> <span class="mi">8</span><span class="p">;</span>
<span class="k">self</span><span class="py">.smoothed_rtt</span> <span class="o">=</span> <span class="nn">Duration</span><span class="p">::</span><span class="nf">from_nanos</span><span class="p">(</span><span class="n">smoothed_rtt_nanos</span> <span class="o">+</span> <span class="n">rtt_nanos</span><span class="p">);</span>
</code></pre></div></div>
<p>This solution sacrifices a bit of unnecessary accuracy at the nanosecond level, but it eliminates the 3 function calls that were impacting CPU utilization:</p>
<pre><code class="language-asm">mov rax, rcx
shr rax, 9
movabs rdx, 19342813113834067
mul rdx
mov rax, rdx
shr rax, 11
imul edx, eax, 1000000000
sub ecx, edx
mov edx, ecx
ret
</code></pre>
<p>Now that we have an optimized solution, we want to test that it has the same results (to the required level of precision) as the unoptimized code.
Next, I’ll show how we do that with both fuzz testing and Kani verification in a single test harness.</p>
<h2 id="from-fuzz-testing-to-kani-verification-to-continuous-integration">From fuzz testing to Kani verification to continuous integration</h2>
<p>The first way we can test this code is by writing a Bolero harness:</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">#[test]</span>
<span class="k">fn</span> <span class="nf">weighted_average_test</span><span class="p">()</span> <span class="p">{</span>
<span class="nn">bolero</span><span class="p">::</span><span class="nd">check!</span><span class="p">()</span>
<span class="py">.with_type</span><span class="p">::</span><span class="o"><</span><span class="p">(</span><span class="nb">u32</span><span class="p">,</span> <span class="nb">u32</span><span class="p">)</span><span class="o">></span><span class="p">()</span>
<span class="nf">.for_each</span><span class="p">(|(</span><span class="n">smoothed_rtt</span><span class="p">,</span> <span class="n">rtt</span><span class="p">)|</span> <span class="p">{</span>
<span class="k">let</span> <span class="n">smoothed_rtt_nanos</span> <span class="o">=</span> <span class="nn">Duration</span><span class="p">::</span><span class="nf">from_nanos</span><span class="p">(</span><span class="o">*</span><span class="n">smoothed_rtt</span> <span class="k">as</span> <span class="mi">_</span><span class="p">);</span>
<span class="k">let</span> <span class="n">rtt_nanos</span> <span class="o">=</span> <span class="nn">Duration</span><span class="p">::</span><span class="nf">from_nanos</span><span class="p">(</span><span class="o">*</span><span class="n">rtt</span> <span class="k">as</span> <span class="mi">_</span><span class="p">);</span>
<span class="k">let</span> <span class="n">weight</span> <span class="o">=</span> <span class="mi">8</span><span class="p">;</span>
<span class="c">// assert that the unoptimized version matches the optimized to the millisecond</span>
<span class="k">let</span> <span class="n">expected</span> <span class="o">=</span> <span class="p">((</span><span class="n">weight</span> <span class="o">-</span> <span class="mi">1</span><span class="p">)</span> <span class="k">as</span> <span class="nb">u32</span> <span class="o">*</span> <span class="n">smoothed_rtt</span><span class="p">)</span> <span class="o">/</span> <span class="n">weight</span> <span class="o">+</span> <span class="n">rtt</span> <span class="o">/</span> <span class="n">weight</span><span class="p">;</span>
<span class="k">let</span> <span class="n">actual</span> <span class="o">=</span> <span class="nn">super</span><span class="p">::</span><span class="nf">weighted_average</span><span class="p">(</span><span class="n">smoothed_rtt</span><span class="p">,</span> <span class="n">rtt</span><span class="p">,</span> <span class="n">weight</span> <span class="k">as</span> <span class="mi">_</span><span class="p">);</span>
<span class="nd">assert_eq!</span><span class="p">(</span><span class="n">expected</span><span class="nf">.as_millis</span><span class="p">(),</span> <span class="n">actual</span><span class="nf">.as_millis</span><span class="p">());</span>
<span class="p">})</span>
<span class="p">}</span>
</code></pre></div></div>
<p>This harness will fuzz test the smoothed RTT calculation code (extracted out into the <code class="language-plaintext highlighter-rouge">weighted_average</code> function) by generating <code class="language-plaintext highlighter-rouge">u32</code> values according to the multiple fuzzing engines that Bolero supports.
These values are then used in both the original, unoptimized version of the code, as well as the optimized <code class="language-plaintext highlighter-rouge">weighted_average</code> function, to assert that both versions match with millisecond precision.
This technique is called “differential testing” and is a powerful method for identifying logical differences between multiple versions of code.
The Amazon Science blog recently <a href="https://www.amazon.science/blog/how-we-built-cedar-with-automated-reasoning-and-differential-testing">highlighted</a> the value of using this technique with the AWS Cedar authorization engine.</p>
<p>Now we have a fuzz test to ensure the assertion that the optimized and unoptimized code result in the same millisecond result holds over millions of different combinations of <code class="language-plaintext highlighter-rouge">smoothed_rtt</code> and <code class="language-plaintext highlighter-rouge">rtt</code> values.
But what if we want to prove that this assertion is true for <strong>all</strong> combinations of <code class="language-plaintext highlighter-rouge">smoothed_rtt</code> and <code class="language-plaintext highlighter-rouge">rtt</code>?
That’s where Kani comes in.</p>
<p>Upgrading this fuzz test to a fuzz + verification test is as easy as adding a few config attributes to the existing test harness:</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">#[test]</span>
<span class="nd">#[cfg_attr(kani,</span> <span class="nd">kani::proof,</span> <span class="nd">kani::solver(kissat))]</span>
<span class="k">fn</span> <span class="nf">weighted_average_test</span><span class="p">()</span> <span class="p">{</span>
</code></pre></div></div>
<p>The <code class="language-plaintext highlighter-rouge">kani::proof</code> attribute lets Bolero know that this test harness can be run as a Kani proof.
<code class="language-plaintext highlighter-rouge">kani::solver(kissat)</code> is an optional attribute we can add to specify the solver used in Kani’s verification engine.
We like using the <a href="https://github.com/arminbiere/kissat">Kissat SAT Solver</a> in s2n-quic as it has good performance for the type of code we are typically verifying.
No one solver is the best at everything, though, so some testing and comparison may be necessary for your use case.</p>
<p>We can run this particular test on the command line and see that the verification is successful:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>cargo kani <span class="nt">--harness</span> recovery::rtt_estimator::test::weighted_average_test <span class="nt">--tests</span>
...
SUMMARY:
<span class="k">**</span> 0 of 344 failed <span class="o">(</span>1 unreachable<span class="o">)</span>
VERIFICATION:- SUCCESSFUL
Verification Time: 44.769085s
</code></pre></div></div>
<p>Running the Kani verification ad-hoc like this is helpful when writing test harnesses or trying different configurations.
But another thing we’ve learned when implementing s2n-quic is that if a test isn’t run automatically and on a regular basis, we can’t consider the code to be tested.
And that is why we run Kani proofs as part of the continuous integration (CI) suite of tests that run every time an s2n-quic code change is proposed in a pull request and whenever code is merged into main.
This is accomplished using the <a href="https://github.com/marketplace/actions/kani-rust-verifier">Kani Rust Verifier Action</a>, a GitHub action that lets us easily execute Kani verification as part of a CI workflow.
The s2n-quic CI runs many different types of tests, including unit tests, integration tests, snapshot tests, performance benchmarking, interoperability testing, and the aforementioned fuzz testing.
All of these tests use different techniques, but collectively they ensure s2n-quic functions as our customers expect even as we continue to add new features and optimize for performance.
Adding Kani to our suite of automated testing provides us with yet another approach to validate the correctness of s2n-quic and increase our confidence in the software, all with a minimum amount of incremental effort.
If you want to learn how to setup the Kani action on your own GitHub repository, see <a href="https://model-checking.github.io/kani-verifier-blog/2022/12/21/easily-verify-your-rust-in-ci-with-kani.html">Easily verify your Rust in CI with Kani and Github Actions</a>.
Next we’ll take a look at a real world instance where Kani was able to catch a bug well before it made it to production.</p>
<h2 id="catching-a-bug-in-packet-number-decoding">Catching a bug in packet number decoding</h2>
<p>Another learning that QUIC took from TCP was the importance of knowing the ordering in which packets are transmitted.
In TCP, a receiver must infer the order a packet was transmitted, making it harder to determine when a packet has been lost, versus just re-ordered or delayed.
This introduces ambiguity into the RTT calculation I described earlier, as it is not always clear if the packet took longer to be acknowledged because the network slowed down or if it had been lost and subsequently retransmitted.
To address these issues, QUIC assigns a monotonically increasing packet number to every QUIC packet, included retransmissions, that explicitly indicates the order in which a packet was transmitted.</p>
<p>The packet number is a value ranging from 0 to 2^62-1.
While such a large range is necessary for supporting long running connections that may send many packets, it also would consume 8 bytes of every packet.
8 bytes might not sound like much, but with s2n-quic being used to send billions of packets back and forth, it adds up and ultimately increases the overhead of using the QUIC protocol.
Therefore, the packet number is encoded in 1 to 4 bytes, following a process that truncates the most significant bits of the packet number based on some assumptions about how wide a range of packet numbers could be in flight at a given time.</p>
<p>We noticed that s2n-quic’s function for decoding the packet number, <code class="language-plaintext highlighter-rouge">decode_packet_number</code>, was showing up in the CPU flame graphs that the s2n-quic CI generates based on a range of common traffic patterns.
If you haven’t seen a flame graph before, the important thing to take away is the wider the box containing each function is, the more relative CPU time that particular function is using.
From the flame graph below we can see the <code class="language-plaintext highlighter-rouge">s2n_quic_core::packet::number::decode_packet_number</code> function is consuming the bulk of the CPU time needed to <code class="language-plaintext highlighter-rouge">unprotect</code> the packet, representing about 2% of the total CPU used for processing a packet.</p>
<p><img src="https://model-checking.github.io/kani-verifier-blog/assets/images/s2n-quic-decode-packet-number-flame-graph-before.png" alt="decode_packet_number flame graph before optimization" /></p>
<p>The <a href="https://datatracker.ietf.org/doc/html/rfc9001#section-9.5">QUIC RFC</a> requires this packet decoding process be free of timing side channels, which our initial implementation ensured by using volatile read operations at the expense of additional CPU utilization.
As was the case with the RTT estimator, there is room for optimization here.
We found that by supplying some additional compiler instructions we could refactor <code class="language-plaintext highlighter-rouge">decode_packet_number</code> in a way that would fulfill the constant-time requirement from the RFC while avoiding the expensive volatile read operations.
With these optimizations, the CPU usage of <code class="language-plaintext highlighter-rouge">decode_packet_number</code> became so small it barely showed up in the flame graph at all:</p>
<p><img src="https://model-checking.github.io/kani-verifier-blog/assets/images/s2n-quic-decode-packet-number-flame-graph-after.png" alt="decode_packet_number flame graph after optimization" /></p>
<p>All good, right? Not so fast…</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>cargo kani <span class="nt">--harness</span> packet::number::tests::rfc_differential_test <span class="nt">--tests</span>
...
SUMMARY:
<span class="k">**</span> 1 of 760 failed <span class="o">(</span>1 unreachable<span class="o">)</span>
Failed Checks: assertion failed: candidate_pn <<span class="o">=</span> VarInt::MAX.as_u64<span class="o">()</span>
File: <span class="s2">"/s2n-quic/quic/s2n-quic-core/src/packet/number/mod.rs"</span>, line 235, <span class="k">in </span>packet::number::decode_packet_number
VERIFICATION:- FAILED
Verification Time: 2.8690908s
</code></pre></div></div>
<p>The Kani proof on the <code class="language-plaintext highlighter-rouge">rfc_differential_test</code> has failed!</p>
<p><code class="language-plaintext highlighter-rouge">rfc_differential_test</code> is another example of a differential test using Bolero, like we saw for testing the smoothed RTT calculation:</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">#[test]</span>
<span class="nd">#[cfg_attr(kani,</span> <span class="nd">kani::proof,</span> <span class="nd">kani::solver(kissat))]</span>
<span class="k">fn</span> <span class="nf">rfc_differential_test</span><span class="p">()</span> <span class="p">{</span>
<span class="nn">bolero</span><span class="p">::</span><span class="nd">check!</span><span class="p">()</span>
<span class="nf">.with_type</span><span class="p">()</span>
<span class="nf">.cloned</span><span class="p">()</span>
<span class="nf">.for_each</span><span class="p">(|(</span><span class="n">largest_pn</span><span class="p">,</span> <span class="n">truncated_pn</span><span class="p">)|</span> <span class="p">{</span>
<span class="k">let</span> <span class="n">largest_pn</span> <span class="o">=</span> <span class="nf">new</span><span class="p">(</span><span class="n">largest_pn</span><span class="p">);</span>
<span class="k">let</span> <span class="n">space</span> <span class="o">=</span> <span class="n">largest_pn</span><span class="nf">.space</span><span class="p">();</span>
<span class="k">let</span> <span class="n">truncated_pn</span> <span class="o">=</span> <span class="n">TruncatedPacketNumber</span> <span class="p">{</span>
<span class="n">space</span><span class="p">,</span>
<span class="n">value</span><span class="p">:</span> <span class="n">truncated_pn</span><span class="p">,</span>
<span class="p">};</span>
<span class="k">let</span> <span class="n">rfc_value</span> <span class="o">=</span> <span class="nf">rfc_decoder</span><span class="p">(</span>
<span class="n">largest_pn</span><span class="nf">.as_u64</span><span class="p">(),</span>
<span class="n">truncated_pn</span><span class="nf">.into_u64</span><span class="p">(),</span>
<span class="n">truncated_pn</span><span class="nf">.bitsize</span><span class="p">(),</span>
<span class="p">)</span>
<span class="nf">.min</span><span class="p">(</span><span class="nn">VarInt</span><span class="p">::</span><span class="n">MAX</span><span class="nf">.as_u64</span><span class="p">());</span>
<span class="k">let</span> <span class="n">actual_value</span> <span class="o">=</span> <span class="nf">decode_packet_number</span><span class="p">(</span><span class="n">largest_pn</span><span class="p">,</span> <span class="n">truncated_pn</span><span class="p">)</span><span class="nf">.as_u64</span><span class="p">();</span>
<span class="nd">assert_eq!</span><span class="p">(</span><span class="n">actual_value</span><span class="p">,</span> <span class="n">rfc_value</span><span class="p">,</span> <span class="s">"diff: {:?}"</span><span class="p">,</span>
<span class="n">actual_value</span><span class="nf">.checked_sub</span><span class="p">(</span><span class="n">rfc_value</span><span class="p">)</span>
<span class="nf">.or_else</span><span class="p">(||</span> <span class="n">rfc_value</span><span class="nf">.checked_sub</span><span class="p">(</span><span class="n">actual_value</span><span class="p">))</span>
<span class="p">);</span>
<span class="p">});</span>
<span class="p">}</span>
</code></pre></div></div>
<p>This test runs both the <code class="language-plaintext highlighter-rouge">decode_packet_number</code> implementation from above as well as an alternate implementation (<code class="language-plaintext highlighter-rouge">rfc_decoder</code>) that exactly follows the <a href="https://www.rfc-editor.org/rfc/rfc9000.html#name-sample-packet-number-decodi">pseudocode</a> the QUIC specification provides for decoding packet numbers.
If the result from the two implementations differ, the test will fail.
In this case, the test failed even before <code class="language-plaintext highlighter-rouge">decode_packet_number</code> produced a result, as the following assertion no longer held true:</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">debug_assert!</span><span class="p">(</span><span class="n">candidate_pn</span> <span class="o"><=</span> <span class="nn">VarInt</span><span class="p">::</span><span class="n">MAX</span><span class="nf">.as_u64</span><span class="p">());</span>
</code></pre></div></div>
<p>It turns out that a simple <code class="language-plaintext highlighter-rouge">min</code> operation that had been applied to <code class="language-plaintext highlighter-rouge">candidate_pn</code> in the unoptimized code had been removed during the optimization process.
Kani saved us from merging this potential bug! One quick fix later, and the verification now succeeds:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>cargo kani <span class="nt">--harness</span> packet::number::tests::rfc_differential_test <span class="nt">--tests</span>
...
SUMMARY:
<span class="k">**</span> 0 of 760 failed <span class="o">(</span>1 unreachable<span class="o">)</span>
VERIFICATION:- SUCCESSFUL
Verification Time: 2.7241826s
</code></pre></div></div>
<p>For this example, using Bolero to run <code class="language-plaintext highlighter-rouge">rfc_differential_test</code> as a fuzz test also reveals the bug after running for less than a minute.
In the next example, we’ll see that this is not always the case.</p>
<h2 id="fitting-stream-frames-into-a-quic-packet">Fitting Stream frames into a QUIC packet</h2>
<p>Continuing on the theme of using CPU and bytes as efficiently as possible, the QUIC protocol uses variable-length encoding for most integers written into QUIC packets and frames.
This allows for a wide range of integers to be represented while requiring less bytes for smaller values.
This is accomplished by repurposing the first 2 bits of the first byte of an integer to indicate the total length of the integer, which may be either 1, 2, 4 or 8 bytes.
For example, if the first 2 bits of the first byte are <code class="language-plaintext highlighter-rouge">00</code>, this means the integer is 1 byte long and may represent a number in the range 0 to 63 (2^6 - 1) in the remaining 6 bits of the first byte.
If the first 2 bits of the first byte are <code class="language-plaintext highlighter-rouge">01</code>, the integer is 2 bytes long and thus can represent integers from 0 to 2^14 - 1, with 6 usable bits left in the first byte and 8 bits in the second byte, for a total of 14 bits.</p>
<p>An interesting property arises when this variable-length encoding is used to encode the length of data contained within a frame, as is done with the <code class="language-plaintext highlighter-rouge">Stream</code> frame that carries the majority of data transmitted over a QUIC connection.
s2n-quic tries to fit as much data as possible in each <code class="language-plaintext highlighter-rouge">Stream</code> frame, based on how much remaining capacity is available in the QUIC packet the frame is being written to.
Since the size of the variable-length integer used to represent the Stream data length depends on the length of the value it is encoding, trying to fit one more byte of data into a packet may end up also increasing the size of the variable-length integer, resulting in the <code class="language-plaintext highlighter-rouge">Stream</code> frame not fitting anymore! For example, say that a packet has 65 bytes of available capacity.
If we try to fit 63 bytes of data in this packet, this will consume 64 bytes from the packet, as the length of 63 can be encoded in 1 byte, as we saw above.
We have 65 bytes to work with though, so why not try to fit one more byte? A length of 64 requires 2 bytes to encode, so now the total amount we’ve consumed is 2 bytes + 64 bytes = 66 bytes, more than the available capacity.</p>
<p>The logic s2n-quic uses to determine how much data to try to fit in a <code class="language-plaintext highlighter-rouge">Stream</code> frame is contained in the function <code class="language-plaintext highlighter-rouge">try_fit</code>.
With complicated logic like this, we have a Bolero test harness to validate the correctness of the logic:</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">#[test]</span>
<span class="nd">#[cfg_attr(kani,</span> <span class="nd">kani::proof,</span> <span class="nd">kani::solver(kissat))]</span>
<span class="k">fn</span> <span class="nf">try_fit_test</span><span class="p">()</span> <span class="p">{</span>
<span class="nn">bolero</span><span class="p">::</span><span class="nd">check!</span><span class="p">()</span>
<span class="nf">.with_type</span><span class="p">()</span>
<span class="nf">.cloned</span><span class="p">()</span>
<span class="nf">.for_each</span><span class="p">(|(</span><span class="n">stream_id</span><span class="p">,</span> <span class="n">offset</span><span class="p">,</span> <span class="n">length</span><span class="p">,</span> <span class="n">capacity</span><span class="p">)|</span> <span class="p">{</span>
<span class="nf">model</span><span class="p">(</span><span class="n">stream_id</span><span class="p">,</span> <span class="n">offset</span><span class="p">,</span> <span class="n">length</span><span class="p">,</span> <span class="n">capacity</span><span class="p">);</span>
<span class="p">});</span>
<span class="p">}</span>
</code></pre></div></div>
<p>The <code class="language-plaintext highlighter-rouge">model</code> function called in this test constructs a <code class="language-plaintext highlighter-rouge">Stream</code> frame with the given inputs, tries to fit it into a packet with the given <code class="language-plaintext highlighter-rouge">capacity</code>, and makes several assertions about the result.
Let’s first run this test harness using the libfuzzer fuzzing engine:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>cargo bolero <span class="nb">test </span>frame::stream::tests::try_fit_test <span class="nt">--engine</span> libfuzzer
<span class="c">#65536 pulse cov: 119 ft: 175 corp: 16/330b lim: 659 exec/s: 21845 rss: 63Mb</span>
<span class="c">#131072 pulse cov: 119 ft: 175 corp: 16/330b lim: 1308 exec/s: 21845 rss: 69Mb</span>
<span class="c">#262144 pulse cov: 119 ft: 175 corp: 16/330b lim: 2609 exec/s: 21845 rss: 82Mb</span>
<span class="c">#524288 pulse cov: 119 ft: 175 corp: 16/330b lim: 4096 exec/s: 22795 rss: 107Mb</span>
<span class="c">#1048576 pulse cov: 119 ft: 175 corp: 16/330b lim: 4096 exec/s: 22310 rss: 157Mb</span>
<span class="c">#2097152 pulse cov: 119 ft: 175 corp: 16/330b lim: 4096 exec/s: 22550 rss: 258Mb</span>
<span class="c">#4194304 pulse cov: 119 ft: 175 corp: 16/330b lim: 4096 exec/s: 22429 rss: 460Mb</span>
<span class="c">#8388608 pulse cov: 119 ft: 175 corp: 16/330b lim: 4096 exec/s: 22429 rss: 522Mb</span>
<span class="c">#16777216 pulse cov: 119 ft: 175 corp: 16/330b lim: 4096 exec/s: 22429 rss: 525Mb</span>
</code></pre></div></div>
<p>The last line of output indicates that libfuzzer has tried more than 16 million different inputs at a rate of 4096 executions per second, and so far no assertions have failed.
This took over 10 minutes to run, and letting it run even longer still failed to find any failures.
Let’s see what Kani thinks:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>cargo kani <span class="nt">--harness</span> frame::stream::tests::try_fit_test <span class="nt">--tests</span>
SUMMARY:
<span class="k">**</span> 1 of 561 failed <span class="o">(</span>1 unreachable<span class="o">)</span>
Failed Checks: assertion failed: frame.encoding_size<span class="o">()</span> <span class="o">==</span> capacity
File: <span class="s2">"/s2n-quic/quic/s2n-quic-core/src/frame/stream.rs"</span>, line 322, <span class="k">in </span>frame::stream::tests::model
VERIFICATION:- FAILED
Verification Time: 20.149725s
</code></pre></div></div>
<p>Kani found a failure in only 20 seconds!</p>
<p>It turns out in this case that the bug was the <code class="language-plaintext highlighter-rouge">model</code> function itself being overly strict, but it could have just as easily been a bug in the <code class="language-plaintext highlighter-rouge">try_fit</code> implementation.
This example highlights yet another learning for s2n-quic: the value of defense in depth.
We’re strong believers in the Swiss cheese model, in which each layer of cheese may contain many holes (such as the fuzz test above missing out on the failure), but combining the layers together greatly reduces the chance that a hole reaches all the way through.
A single testing technique may not catch every issue, but layering fuzz testing, Kani verification, and all the other testing methodologies s2n-quic employs vastly improves the likelihood that at least one test will catch a bug before it reaches production.</p>
<h2 id="conclusion">Conclusion</h2>
<p>s2n-quic has a very high bar for software quality, as any bug in a transport protocol implementation can have drastic consequences for the customers that rely on the library as the foundation for the applications and services they are building.
I’ve shown above how Kani helps us meet that bar with a vanishingly small amount of effort by easily integrating with our existing Bolero fuzz test harnesses and automatically running as part of our continuous integration test suite.
And even better than catching bugs in the wild, Kani has helped s2n-quic catch bugs before they make it any further than a pull request, even when a fuzz test was unable to.
As of now, s2n-quic has over thirty Bolero harnesses with Kani enabled, spanning many critical parts of the codebase.
As we continue to add more and more proofs and increase code coverage, Kani gives us the confidence to continue optimizing and improving s2n-quic while ensuring the correctness our customers require.</p>
<h2 id="author-bio">Author bio</h2>
<p>Wesley Rosenblum is a Senior Software Development Engineer at AWS.
Over his 16 year career at Amazon he has designed and implemented software ranging from automated inventory management services to open source encryption libraries.
He has spent the past 3 years working on s2n-quic, with a particular focus on the loss recovery and congestion control algorithms that are critical for performance and reliable data delivery.</p>s2n-quic is a Rust implementation of the QUIC protocol, a transport protocol designed for fast and secure communication between hosts in a network. QUIC is relatively new, but it builds upon and learns from TCP (Transmission Control Protocol), the transport protocol that has been the standard for communication on the Internet for decades. TCP is the underlying protocol for HTTP/1.1 and HTTP/2, but for the latest generation HTTP/3, QUIC is used instead. In that respect, QUIC can be thought of as a next-generation version of TCP with new features such as stream multiplexing and connection migration. However, the improvements over TCP go beyond just new features and functionality. Notably, QUIC is secure by default; the specification requires TLS be used within the QUIC handshake to establish encryption and authentication keys, and subsequently every QUIC packet is protected using these keys. With TCP, confidentiality and integrity were not part of the protocol design, and thus TLS was layered on top of TCP, as is done in HTTPS.Writing Code with ChatGPT? Improve it with Kani.2023-05-01T00:00:00+00:002023-05-01T00:00:00+00:00https://model-checking.github.io//kani-verifier-blog/2023/05/01/writing-code-with-chatgpt-improve-it-with-kani<p>Kani is a verification tool that can help you prove properties about your Rust code.
To learn more about Kani, check out <a href="https://model-checking.github.io/kani/kani-tutorial.html">the Kani tutorial</a> and our <a href="https://model-checking.github.io/kani-verifier-blog/">previous blog posts</a>.</p>
<p>Recently, <a href="https://openai.com/blog/chatgpt">ChatGPT</a>—OpenAI’s AI chatbot built on top of Large Language Models (LLMs)—has taken the world by storm.
People are using ChatGPT to write blog posts, do homework, and even write code!
That’s right: LLMs like ChatGPT can be used to write code, since code is essentially just another kind of text.
All you need to do is provide a text prompt describing the program you want, and ChatGPT will generate the code for you, along with an explanation.</p>
<p>We were also eager to explore the possibilities of using LLMs for daily coding tasks, so we decided to put ChatGPT to the test.
In this blog post, we’ll share our experiences with using ChatGPT and Kani to solve example coding problems, and explore how these technologies could revolutionize the way we approach software development.
The experiments we did include the following three tasks:</p>
<ol>
<li>Writing a prompt to generate <em>unverified</em> code with ChatGPT.</li>
<li>Checking the generated code with Kani to see if it’s correct.</li>
<li>Using ChatGPT and Kani iteratively to converge to code that passes verification.</li>
</ol>
<p>Our experiments show that, while ChatGPT is easy to use and can efficiently generate code from text description, the generated code does not always do what it is supposed to do.
This is where verification tools like Kani can help detect bugs, corner cases and unexpected behavior in the generated code, thereby helping users of ChatGPT write more proven code.</p>
<h2 id="code-generation-using-chatgpt">Code Generation using ChatGPT</h2>
<p>We demonstrate the three tasks with a toy problem of writing a Rust function to compute the average of two integers.</p>
<p>Let’s start our first task: <strong>writing code using a prompt with ChatGPT</strong>.
For the toy problem, the prompt is also very simple and straightforward.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Q: Can you write me a Rust function that computes the integer average of two integers?
</code></pre></div></div>
<p>Giving the above prompt, ChatGPT answered us with the following function,</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">fn</span> <span class="nf">integer_average</span><span class="p">(</span><span class="n">a</span><span class="p">:</span> <span class="nb">i32</span><span class="p">,</span> <span class="n">b</span><span class="p">:</span> <span class="nb">i32</span><span class="p">)</span> <span class="k">-></span> <span class="nb">i32</span> <span class="p">{</span>
<span class="p">(</span><span class="n">a</span> <span class="o">+</span> <span class="n">b</span><span class="p">)</span> <span class="o">/</span> <span class="mi">2</span>
<span class="p">}</span>
</code></pre></div></div>
<p>an explanation,</p>
<blockquote>
<p>In this function, we define a function called integer_average that takes two integer arguments a and b.
The function then adds the two arguments together and divides the result by 2 to compute the integer average.</p>
</blockquote>
<p>and even a driver function to call and test the average function.</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">fn</span> <span class="nf">main</span><span class="p">()</span> <span class="p">{</span>
<span class="k">let</span> <span class="n">a</span> <span class="o">=</span> <span class="mi">4</span><span class="p">;</span>
<span class="k">let</span> <span class="n">b</span> <span class="o">=</span> <span class="mi">8</span><span class="p">;</span>
<span class="k">let</span> <span class="n">result</span> <span class="o">=</span> <span class="nf">integer_average</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">);</span>
<span class="nd">println!</span><span class="p">(</span><span class="s">"The average of {} and {} is {}"</span><span class="p">,</span> <span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">,</span> <span class="n">result</span><span class="p">);</span>
<span class="p">}</span>
</code></pre></div></div>
<p>From the example, ChatGPT’s code generation capabilities are thoroughly impressive.
Not only does it produce code that looks good at first glance, but it also provides explanations that give us greater confidence that the code is doing exactly what we intended it to do.
In fact, we can even request ChatGPT to generate a driver function and some test cases using follow-up prompts to further test the generated code.
However, as verification folks, we know that we still can’t be completely sure that the code is correct without <strong>verification</strong>.</p>
<h2 id="verification-of-generated-code">Verification of Generated Code</h2>
<p>So, naturally, our next task is: <strong>verifying the generated programs with Kani</strong>.
That is, we want to run Kani on the generated function <code class="language-plaintext highlighter-rouge">integer_average</code> to see if there is any bug in it.</p>
<p>We can simply transform the driver function into a Kani harness by substituting the test cases with <code class="language-plaintext highlighter-rouge">kani::any()</code>.</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">fn</span> <span class="nf">integer_average</span><span class="p">(</span><span class="n">a</span><span class="p">:</span> <span class="nb">i32</span><span class="p">,</span> <span class="n">b</span><span class="p">:</span> <span class="nb">i32</span><span class="p">)</span> <span class="k">-></span> <span class="nb">i32</span> <span class="p">{</span>
<span class="p">(</span><span class="n">a</span> <span class="o">+</span> <span class="n">b</span><span class="p">)</span> <span class="o">/</span> <span class="mi">2</span> <span class="c">// <------ line 2</span>
<span class="p">}</span>
<span class="nd">#[kani::proof]</span>
<span class="k">fn</span> <span class="nf">main</span><span class="p">()</span> <span class="p">{</span>
<span class="k">let</span> <span class="n">a</span> <span class="o">=</span> <span class="nn">kani</span><span class="p">::</span><span class="nf">any</span><span class="p">();</span>
<span class="k">let</span> <span class="n">b</span> <span class="o">=</span> <span class="nn">kani</span><span class="p">::</span><span class="nf">any</span><span class="p">();</span>
<span class="k">let</span> <span class="n">result</span> <span class="o">=</span> <span class="nf">integer_average</span><span class="p">(</span><span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">);</span>
<span class="nd">println!</span><span class="p">(</span><span class="s">"The average of {} and {} is {}"</span><span class="p">,</span> <span class="n">a</span><span class="p">,</span> <span class="n">b</span><span class="p">,</span> <span class="n">result</span><span class="p">);</span>
<span class="p">}</span>
</code></pre></div></div>
<p>This harness verifies that, for any <code class="language-plaintext highlighter-rouge">i32</code> integers <code class="language-plaintext highlighter-rouge">a</code> and <code class="language-plaintext highlighter-rouge">b</code>, the function <code class="language-plaintext highlighter-rouge">integer_average</code> cannot trigger any of the <a href="https://model-checking.github.io/kani/tutorial-kinds-of-failure.html">bad behaviors that Kani checks for</a>.</p>
<p>We ran Kani on the above program</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>> kani integer_average.rs
</code></pre></div></div>
<p>and got the verification result</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>...
SUMMARY:
** 1 of 4 failed
Failed Checks: attempt to add with overflow
File: "...", line 2, in integer_average
VERIFICATION:- FAILED
Verification Time: 0.6991158s
</code></pre></div></div>
<p>The code is actually buggy!
Although the average of two <code class="language-plaintext highlighter-rouge">i32</code> integers is always in the range of <code class="language-plaintext highlighter-rouge">i32</code>, the intermediate sum <code class="language-plaintext highlighter-rouge">a + b</code> may overflow.
A counterexample is when <code class="language-plaintext highlighter-rouge">a</code> equals <code class="language-plaintext highlighter-rouge">i32::MAX</code> and <code class="language-plaintext highlighter-rouge">b</code> equals any positive integer, the sum of them will be greater than <code class="language-plaintext highlighter-rouge">i32::MAX</code>, and hence the overflow.</p>
<p>The examples above illustrate that, although code generation is efficient and accessible, it’s crucial to exercise caution and avoid blindly trusting the generated code, as it may contain errors.
We believe there are two factors that contribute to this unreliability.
Firstly, most of the programs in the training corpus of LLMs are unverified programs<sup id="fnref:footnote-training-corpus" role="doc-noteref"><a href="#fn:footnote-training-corpus" class="footnote" rel="footnote">1</a></sup>.
Secondly, text prompts can be ambiguous, leading to the production of flawed code.
Fortunately, program verifiers such as Kani can provide valuable information about generated code, such as verification results, and counterexamples.
Armed with this information, developers can locate and fix bugs more effectively.
But what about the potential for language models like ChatGPT to leverage such insights to improve code generation accuracy?
It’s an intriguing question that merits further exploration.</p>
<h2 id="code-generation-with-verifiers-in-the-loop">Code Generation with Verifiers in the Loop</h2>
<p>The last task we explore in this post is: <strong>using ChatGPT and Kani iteratively to converge to code that passes verification.</strong>.</p>
<p>Recall that we found the overflow error in the expression <code class="language-plaintext highlighter-rouge">a + b</code> in the generated function <code class="language-plaintext highlighter-rouge">integer_average</code> with Kani.
We shared this information with ChatGPT.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Q: The expression (a + b) / 2 may overflow.
</code></pre></div></div>
<p>ChatGPT agreed with us and fixed the overflow with a new implementation using <code class="language-plaintext highlighter-rouge">checked_add</code>:</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">fn</span> <span class="nf">integer_average</span><span class="p">(</span><span class="n">num1</span><span class="p">:</span> <span class="nb">i32</span><span class="p">,</span> <span class="n">num2</span><span class="p">:</span> <span class="nb">i32</span><span class="p">)</span> <span class="k">-></span> <span class="nb">i32</span> <span class="p">{</span>
<span class="k">match</span> <span class="n">num1</span><span class="nf">.checked_add</span><span class="p">(</span><span class="n">num2</span><span class="p">)</span> <span class="p">{</span>
<span class="nf">Some</span><span class="p">(</span><span class="n">sum</span><span class="p">)</span> <span class="k">=></span> <span class="n">sum</span> <span class="o">/</span> <span class="mi">2</span><span class="p">,</span>
<span class="nb">None</span> <span class="k">=></span> <span class="p">{</span>
<span class="c">// Handle overflow gracefully</span>
<span class="nd">println!</span><span class="p">(</span><span class="s">"Overflow occurred!"</span><span class="p">);</span>
<span class="nn">std</span><span class="p">::</span><span class="nn">i32</span><span class="p">::</span><span class="n">MAX</span> <span class="c">// Fallback value</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>We reran Kani on this new implementation.
Kani verified that this new version is correct!
However, the function is no longer complete because it doesn’t compute the average for <em>all</em> <code class="language-plaintext highlighter-rouge">i32</code> integers as we specified.
For example, it does not compute the average for the input <code class="language-plaintext highlighter-rouge">integer_average(std::i32::MAX, 1)</code>.
So, we tried again with the following prompt to generate a complete version of <code class="language-plaintext highlighter-rouge">integer_average</code>.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Q: Can you give me a fix that does not throw any error?
</code></pre></div></div>
<p>This time, ChatGPT found us an implementation that accepts all <code class="language-plaintext highlighter-rouge">i32</code> integers as input.</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">fn</span> <span class="nf">integer_average</span><span class="p">(</span><span class="n">num1</span><span class="p">:</span> <span class="nb">i32</span><span class="p">,</span> <span class="n">num2</span><span class="p">:</span> <span class="nb">i32</span><span class="p">)</span> <span class="k">-></span> <span class="nb">i32</span> <span class="p">{</span>
<span class="n">num1</span> <span class="o">/</span> <span class="mi">2</span> <span class="o">+</span> <span class="n">num2</span> <span class="o">/</span> <span class="mi">2</span> <span class="o">+</span> <span class="p">(</span><span class="n">num1</span> <span class="o">%</span> <span class="mi">2</span> <span class="o">+</span> <span class="n">num2</span> <span class="o">%</span> <span class="mi">2</span><span class="p">)</span> <span class="o">/</span> <span class="mi">2</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Kani didn’t report any failure on this implementation.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>> kani integer_average.rs
...
SUMMARY:
** 0 of 6 failed
VERIFICATION:- SUCCESSFUL
Verification Time: 0.5309424s
</code></pre></div></div>
<h2 id="a-more-practical-example-program-optimization">A More Practical Example: Program Optimization</h2>
<p>In this section, we will repeat the three tasks on a more practical and challenging problem.</p>
<p>First, we wrote a Rust function that computes the lowest unset bit in a <code class="language-plaintext highlighter-rouge">u32</code> integer.</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">fn</span> <span class="nf">lowest_unset_bit_ori</span><span class="p">(</span><span class="n">x</span><span class="p">:</span> <span class="nb">u32</span><span class="p">)</span> <span class="k">-></span> <span class="nb">Option</span><span class="o"><</span><span class="nb">u8</span><span class="o">></span> <span class="p">{</span>
<span class="k">let</span> <span class="k">mut</span> <span class="n">mask</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span>
<span class="k">for</span> <span class="n">i</span> <span class="n">in</span> <span class="mi">0</span><span class="o">..</span><span class="mi">32</span> <span class="p">{</span>
<span class="k">if</span> <span class="n">x</span> <span class="o">&</span> <span class="n">mask</span> <span class="o">==</span> <span class="mi">0</span> <span class="p">{</span>
<span class="k">return</span> <span class="nf">Some</span><span class="p">(</span><span class="n">i</span><span class="p">);</span>
<span class="p">}</span>
<span class="n">mask</span> <span class="o"><<=</span> <span class="mi">1</span><span class="p">;</span>
<span class="p">}</span>
<span class="nb">None</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Our implementation uses a loop to iterate through all 32 bits in the input.
And we know that this could be slower than using some <a href="https://graphics.stanford.edu/~seander/bithacks.html">bit twiddling hacks</a>.
In this case, our goal is to optimize the function with ChatGPT.
This is the prompt we used, which includes the implementation we wrote:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Q: Optimize this Rust function:
fn lowest_unset_bit(x: u32) -> Option<u8> {
let mut mask = 1;
for i in 0..32 {
if x & mask == 0 {
return Some(i);
}
mask <<= 1;
}
None
}
</code></pre></div></div>
<p>After a few seconds, ChatGPT gave us a solution</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">fn</span> <span class="nf">lowest_unset_bit_opt</span><span class="p">(</span><span class="n">x</span><span class="p">:</span> <span class="nb">u32</span><span class="p">)</span> <span class="k">-></span> <span class="nb">Option</span><span class="o"><</span><span class="nb">u8</span><span class="o">></span> <span class="p">{</span>
<span class="k">if</span> <span class="n">x</span> <span class="o">==</span> <span class="nn">std</span><span class="p">::</span><span class="nn">u32</span><span class="p">::</span><span class="n">MAX</span><span class="p">{</span>
<span class="k">return</span> <span class="nb">None</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">let</span> <span class="n">y</span> <span class="o">=</span> <span class="n">x</span> <span class="p">|</span> <span class="p">(</span><span class="n">x</span> <span class="o">+</span> <span class="mi">1</span><span class="p">);</span>
<span class="k">let</span> <span class="n">z</span> <span class="o">=</span> <span class="n">y</span> <span class="o">^</span> <span class="p">(</span><span class="n">y</span> <span class="o">+</span> <span class="mi">1</span><span class="p">);</span>
<span class="nf">Some</span><span class="p">((</span><span class="n">z</span><span class="nf">.trailing_zeros</span><span class="p">()</span> <span class="o">%</span> <span class="mi">32</span><span class="p">)</span> <span class="k">as</span> <span class="nb">u8</span><span class="p">)</span>
<span class="p">}</span>
</code></pre></div></div>
<p>along with an explanation.</p>
<h4 id="equivalence-checking">Equivalence Checking</h4>
<p>Demonstrating that one program is an optimization of another requires proving two things: that the programs are equivalent and that the optimized version performs better.
Although proving the performance improvement is challenging, we can at least verify the equivalence of the two implementations using Kani.
We put the implementation we wrote and the implementation ChatGPT generated together and wrote a harness function for checking the equivalence of two the implementation of <code class="language-plaintext highlighter-rouge">lowest_unset_bit</code>.
In the harness function, we let the input <code class="language-plaintext highlighter-rouge">x</code> be any <code class="language-plaintext highlighter-rouge">u32</code> integer, call both implementations on <code class="language-plaintext highlighter-rouge">x</code>, and check that the outputs returned by them are the same.</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">// Original implementation.</span>
<span class="k">fn</span> <span class="nf">lowest_unset_bit_ori</span><span class="p">(</span><span class="n">x</span><span class="p">:</span> <span class="nb">u32</span><span class="p">)</span> <span class="k">-></span> <span class="nb">Option</span><span class="o"><</span><span class="nb">u8</span><span class="o">></span> <span class="p">{</span>
<span class="k">let</span> <span class="k">mut</span> <span class="n">mask</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span>
<span class="k">for</span> <span class="n">i</span> <span class="n">in</span> <span class="mi">0</span><span class="o">..</span><span class="mi">32</span> <span class="p">{</span>
<span class="k">if</span> <span class="n">x</span> <span class="o">&</span> <span class="n">mask</span> <span class="o">==</span> <span class="mi">0</span> <span class="p">{</span>
<span class="k">return</span> <span class="nf">Some</span><span class="p">(</span><span class="n">i</span><span class="p">);</span>
<span class="p">}</span>
<span class="n">mask</span> <span class="o"><<=</span> <span class="mi">1</span><span class="p">;</span>
<span class="p">}</span>
<span class="nb">None</span>
<span class="p">}</span>
<span class="c">// Optimized implementation generated by ChatGPT.</span>
<span class="k">fn</span> <span class="nf">lowest_unset_bit_opt</span><span class="p">(</span><span class="n">x</span><span class="p">:</span> <span class="nb">u32</span><span class="p">)</span> <span class="k">-></span> <span class="nb">Option</span><span class="o"><</span><span class="nb">u8</span><span class="o">></span> <span class="p">{</span>
<span class="k">if</span> <span class="n">x</span> <span class="o">==</span> <span class="nn">std</span><span class="p">::</span><span class="nn">u32</span><span class="p">::</span><span class="n">MAX</span><span class="p">{</span>
<span class="k">return</span> <span class="nb">None</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">let</span> <span class="n">y</span> <span class="o">=</span> <span class="n">x</span> <span class="p">|</span> <span class="p">(</span><span class="n">x</span> <span class="o">+</span> <span class="mi">1</span><span class="p">);</span> <span class="c">// <------ line 18 </span>
<span class="k">let</span> <span class="n">z</span> <span class="o">=</span> <span class="n">y</span> <span class="o">^</span> <span class="p">(</span><span class="n">y</span> <span class="o">+</span> <span class="mi">1</span><span class="p">);</span>
<span class="nf">Some</span><span class="p">((</span><span class="n">z</span><span class="nf">.trailing_zeros</span><span class="p">()</span> <span class="o">%</span> <span class="mi">32</span><span class="p">)</span> <span class="k">as</span> <span class="nb">u8</span><span class="p">)</span>
<span class="p">}</span>
<span class="nd">#[kani::proof]</span>
<span class="k">fn</span> <span class="nf">check</span><span class="p">()</span> <span class="p">{</span>
<span class="k">let</span> <span class="n">x</span><span class="p">:</span> <span class="nb">u32</span> <span class="o">=</span> <span class="nn">kani</span><span class="p">::</span><span class="nf">any</span><span class="p">();</span>
<span class="nd">assert_eq!</span><span class="p">(</span><span class="nf">lowest_unset_bit_ori</span><span class="p">(</span><span class="n">x</span><span class="p">),</span> <span class="nf">lowest_unset_bit_opt</span><span class="p">(</span><span class="n">x</span><span class="p">));</span> <span class="c">// <------ line 26</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Running Kani on the program with the command</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>> kani lowest_unset_bit.rs
</code></pre></div></div>
<p>gave us the verification result</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>...
SUMMARY:
** 2 of 70 failed
Failed Checks: attempt to add with overflow
File: "...", line 18, in lowest_unset_bit_opt
Failed Checks: assertion failed: lowest_unset_bit_ori(x) == lowest_unset_bit_opt(x)
File: "...", line 26, in check
VERIFICATION:- FAILED
Verification Time: 0.84979576s
</code></pre></div></div>
<p>The generated implementation is incorrect!
There were two failed checks.
The first one <code class="language-plaintext highlighter-rouge">Failed Checks: attempt to add with overflow</code> says that the addition <code class="language-plaintext highlighter-rouge">x + 1</code> may overflow.
The other one says that equivalence check may fail—the two implementations are actually not equivalent.</p>
<p>To better understand the bug, we asked Kani to produce a counterexample with the <code class="language-plaintext highlighter-rouge">--concrete-playback</code> option (discussed in <a href="https://model-checking.github.io/kani-verifier-blog/2022/09/22/internship-projects-2022-concrete-playback.html">our previous post</a>).</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>> cargo kani --enable-unstable --concrete-playback=print | grep "//" | awk '{print $2}'
2147483647
</code></pre></div></div>
<p>The counterexample is when <code class="language-plaintext highlighter-rouge">x</code> equals <code class="language-plaintext highlighter-rouge">2147483647</code> (<code class="language-plaintext highlighter-rouge">i32::MAX</code>).</p>
<p>We reported this counterexample to ChatGPT and asked for a fix.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Q: The optimized version does not produce the same value as the original for x = 2147483674.
</code></pre></div></div>
<p>It gave as a new optimization.</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">fn</span> <span class="nf">lowest_unset_bit_opt_2</span><span class="p">(</span><span class="n">x</span><span class="p">:</span> <span class="nb">u32</span><span class="p">)</span> <span class="k">-></span> <span class="nb">Option</span><span class="o"><</span><span class="nb">u8</span><span class="o">></span> <span class="p">{</span>
<span class="k">if</span> <span class="n">x</span> <span class="o">==</span> <span class="nn">std</span><span class="p">::</span><span class="nn">u32</span><span class="p">::</span><span class="n">MAX</span><span class="p">{</span>
<span class="k">return</span> <span class="nb">None</span><span class="p">;</span>
<span class="p">}</span>
<span class="nf">Some</span><span class="p">((</span><span class="n">x</span> <span class="o">^</span> <span class="p">(</span><span class="n">x</span> <span class="o">+</span> <span class="mi">1</span><span class="p">))</span><span class="nf">.trailing_zeros</span><span class="p">()</span> <span class="k">as</span> <span class="nb">u8</span><span class="p">)</span>
<span class="p">}</span>
</code></pre></div></div>
<p>We reran Kani to check if the new version is equivalent to the original implementation with the same harness function.
This time, there was only one failed check.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>...
SUMMARY:
** 1 of 70 failed
Failed Checks: assertion failed: lowest_unset_bit_ori(x) == lowest_unset_bit_opt(x)
File: "...", line 22, in check
VERIFICATION:- FAILED
Verification Time: 0.84979576s
</code></pre></div></div>
<p>The overflow has been fixed, but the equivalence failure persists.
Similarly, we produced another counterexample <code class="language-plaintext highlighter-rouge">x = 4293918719</code> with Kani, reported it to ChatGPT, and got another optimization.</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">fn</span> <span class="nf">lowest_unset_bit_3</span><span class="p">(</span><span class="n">x</span><span class="p">:</span> <span class="nb">u32</span><span class="p">)</span> <span class="k">-></span> <span class="nb">Option</span><span class="o"><</span><span class="nb">u8</span><span class="o">></span> <span class="p">{</span>
<span class="k">if</span> <span class="n">x</span> <span class="o">==</span> <span class="nn">std</span><span class="p">::</span><span class="nn">u32</span><span class="p">::</span><span class="n">MAX</span><span class="p">{</span>
<span class="k">return</span> <span class="nb">None</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">let</span> <span class="n">y</span> <span class="o">=</span> <span class="o">!</span><span class="n">x</span><span class="p">;</span>
<span class="nf">Some</span><span class="p">(</span><span class="n">y</span><span class="nf">.trailing_zeros</span><span class="p">()</span> <span class="k">as</span> <span class="nb">u8</span><span class="p">)</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Kani successfully verified that this optimization is equivalent to the original implementation.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>> kani lowest_unset_bit.rs
...
SUMMARY:
** 0 of 68 failed
VERIFICATION:- SUCCESSFUL
Verification Time: 5.308244s
</code></pre></div></div>
<p>The result saves us the effort of manually reasoning about the bit hack and convincing ourselves that the two implementations are equivalent.</p>
<h2 id="summary">Summary</h2>
<p>In this post, we showed how to generate programs with ChatGPT, how to verify generated code with Kani, and how to generate code that passes verification with both ChatGPT and Kani.
With increasingly powerful code generation capabilities, software development is becoming more automated and accessible.
However, as developers spend less time on implementation details, corner cases, and safety issues, their code becomes more susceptible to errors.
We believe program verifiers like Kani will play a critical role in ensuring the correctness of such code, revealing potential bugs, and helping developers fix them.</p>
<h2 id="footnote">Footnote</h2>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:footnote-training-corpus" role="doc-endnote">
<p>For example, <a href="https://arxiv.org/pdf/2107.03374.pdf">CodeX</a>, which powers GitHub Copilot, is training on publicly available code from GitHub. <a href="#fnref:footnote-training-corpus" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>Kani is a verification tool that can help you prove properties about your Rust code. To learn more about Kani, check out the Kani tutorial and our previous blog posts.How Kani helped find bugs in Hifitime2023-03-31T00:00:00+00:002023-03-31T00:00:00+00:00https://model-checking.github.io//kani-verifier-blog/2023/03/31/how-kani-helped-find-bugs-in-hifitime<p><a href="https://github.com/model-checking/kani">Kani</a> is a verification tool
that can help you systematically test properties about your Rust code.
To learn more about Kani, check out <a href="https://model-checking.github.io/kani/kani-tutorial.html">the Kani
tutorial</a>
and our <a href="https://model-checking.github.io/kani-verifier-blog/">previous blog
posts</a>.</p>
<p><a href="https://docs.rs/hifitime/latest/hifitime/">Hifitime</a> is a scientifically accurate time management library that provides nanosecond precision of durations and time computations for 65 thousand centuries in most time scales (also called time systems). It is useful wherever a monotonic clock is needed, even in the presence of leap second or remote time corrections (like for Network Time Protocol or Precise Time Protocol). It is suitable for desktop applications and for embedded systems without a floating point unit (FPU). It also supports the typical features of date time management libraries, like the <a href="https://docs.rs/hifitime/latest/hifitime/efmt/format/struct.Format.html">formatting and parsing of date times</a> in a <code class="language-plaintext highlighter-rouge">no-std</code> environment using the typical C89 tokens, or a <a href="https://docs.rs/hifitime/latest/hifitime/prelude/struct.Duration.html#method.approx">human friendly approximation</a> of the difference between dates. For scientific and engineering applications, it is mostly tailored to astronomy (with the inclusion of the UT1 time scale), astrodynamics (with the ET and TDB time scales), and global navigation satellite systems (with the GPS, Galileo, and BeiDou time scales).</p>
<p>The purpose of this blog post is to show how Kani helped solve a number of important non-trivial bugs in hifitime. For completeness’ sake, let’s start with <a href="#why-time-matters">an introduction of the notion of time</a>, how hifitime <a href="#avoiding-loss-of-precision">avoids loss of precision</a>, and finally <a href="#importance-of-kani-in-hifitime">why Kani is crucial</a> to ensuring the correctness of hifitime.</p>
<p>This is our first blog post from an external contributor. We are very excited for this contribution and thank <a href="#author-bio">Christopher Rabotin</a> for his time, interest, and copacetic use of Kani.</p>
<h2 id="why-time-matters">Why time matters</h2>
<p><em>Time is just counting subdivisions of the second, it’s a trivial and solved problem.</em> Well, it’s not quite trivial, and often incorrectly solved.</p>
<p>As explained in <a href="https://bookshop.org/p/books/beyond-measure-the-hidden-history-of-measurement-from-cubits-to-quantum-constants-james-vincent/18124818">“Beyond Measure” by James Vincent</a>, measuring the passing of time has been important for societies for millennia. From using the flow of the Nile River to measure the passage of a year, to tracking the position of the Sun in the sky to delimit the start and end of a day, the definition of the time has changed quite a bit.</p>
<p>On Earth, humans follow a local time, which is supposed to be relatively close to the idea that noon is when the Sun is at its height in the sky for the day. We’ve also decided that the divisions of time must be fixed: for example, an hour lasts 3600 seconds regardless of whether it’s 1 am or 2 pm. If we only follow this fixed definition of an hour, then time drifts because the Earth does not in fact complete a full rotation on itself in <em>exactly</em> 24 hours (nor in one sidereal day of 23 hours 56 minutes and 4.091 seconds, which is also an approximation of the UT1 time scale). For “human time” (Universal Coordinated Time, <code class="language-plaintext highlighter-rouge">UTC</code>) to catch up with the time with respect to the stars (<code class="language-plaintext highlighter-rouge">UT1</code>), we regularly introduce “leap seconds.” Like leap years, where we introduce an extra day every fourth year (with rare exceptions), leap seconds introduce an extra second <em>every so often</em>, as <a href="https://www.ietf.org/timezones/data/leap-seconds.list">announced by the IERS</a>. When scientists report to the IERS that the Earth is about to drift quite a bit compared to <code class="language-plaintext highlighter-rouge">UTC</code>, IERS announces that on a given day at least six months in the future, there will be an extra second in that day to allow for the rotation of the Earth to catch up. In practice, this typically means that UTC clocks “stop the counting of time” for one second. In other words, UTC is <em>not</em> a continuous time scale: that second we didn’t count in UTC still happened for the universe! As the Standards Of Fundamental Astronomy (SOFA) eloquently puts it:</p>
<blockquote>
<p>Leap seconds pose tricky problems for software writers, and consequently there are concerns that these events put safety-critical systems at risk. The correct solution is for designers to base such systems on TAI or some other glitch-free time scale, not UTC, but this option is often overlooked until it is too late.
– “SOFA Time Scales and Calendar Tools”, Document version 1.61, section 3.5.1</p>
</blockquote>
<p>As seen in the figure below, the deviation of the UTC time scale compared to the other time scales increases over time. Moreover, the time between two corrections is not equally spaced. In other words, UTC is not a monotonic clock and attempting to use UTC over a leap second adjustment will cause problems.</p>
<p><img src="https://model-checking.github.io/kani-verifier-blog/assets/images/utc-time-scale-drift.png" alt="UTC drifts with respect to other time scales" /></p>
<p>Luckily there are a few glitch-free time scales to choose from. Hifitime uses TAI, the international atomic time. This allows quick synchronization with UTC time because both UTC and TAI “tick” at the same rate as they are both Earth based time scales. In fact, with general relativity, we understand that time is affected by gravity fields. This causes the additional problem that a second does not tick with the same frequency on Earth as it does in the vacuum between Mars and Jupiter (which is more than twice as far apart as Mars is from the Sun by the way). As such, the position of planets published by NASA JPL are provided in a time scale in which the influence of the gravity of Earth has been removed and where the “ticks” of one second are fixed at the solar system barycenter: the Dynamic Barycentric Time (TDB) time scale. As you may imagine, this causes lots of problems when converting from human time to any astronomical time.</p>
<p>The figure below shows how the effect of gravity on the duration of a second compared to Earth based clocks, like TT (blue line fixed at 32.184 s) and TAI (reference Earth time, not shown since it would be a line at exactly zero).</p>
<p><img src="https://model-checking.github.io/kani-verifier-blog/assets/images/tai-vs-et-vs-tdb.png" alt="TT is fixed to TAI but TDB and ET are periodic" /></p>
<h2 id="avoiding-loss-of-precision">Avoiding loss of precision</h2>
<p>The precision of a floating point value depends on its magnitude (as represented by the exponent) and on how it was computed from other values (rounding modes, error accumulation, use of extended precision for intermediate computations in CPU, etc.). Moreover, some microcontrollers do not have a floating point unit (FPU), meaning that any floating point operation is emulated through software, adding a considerable number of CPU cycles for any computation.</p>
<p>Dates, or “epochs,” are <em>simply</em> the duration past a certain reference epoch. Operations on these epochs, like adding 26.7 days, will cause rounding errors (even sometimes in a single operation like on this <a href="https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=b760579f103b7192c20413ebbe167b90"><code class="language-plaintext highlighter-rouge">f32</code></a>).</p>
<p>Time scales have different reference epochs, and to correctly convert between different time scales, we must ensure that we don’t lose any precision in the computation of these offsets. That’s especially important given that one of the most common scale in astronomy is the Julian Date, whose reference epoch is 4713 BC (or -4712). Storing such a number with an <code class="language-plaintext highlighter-rouge">f64</code> would lead to very large rounding errors.</p>
<p>SOFA solves this by using a tuple of double precision floating point values for all computations, an approach used in hifitime version 2 but discarded in version 3. In fact, to avoid loss of precision, hifitime stores all durations as a tuple of the number of centuries and the number of nanoseconds into this century. For example <code class="language-plaintext highlighter-rouge">(0, 1)</code> is a duration of one nanosecond. Then, the Epoch is a structure containing the TAI duration since the TAI reference epoch of 01 January 1900 at noon and the time scale in which the Epoch was initialized. This approach guarantees no more and no less than one nanosecond of precision on all computations for 32,768 centuries on either side of the reference epoch. (<a href="https://github.com/nyx-space/hifitime/issues/186">And <em>maybe</em> hifitime should be even more precise?</a>)</p>
<h2 id="importance-of-kani-in-hifitime">Importance of Kani in hifitime</h2>
<p>The purpose of hifitime is to convert between time scales, and these drift apart from each other. Time scales are only used for scientific computations, so it’s important to correctly compute the conversion between these time scales. As previously noted, to ensure exactly one nanosecond precision, hifitime stores durations as a tuple of integers, as these aren’t affected by successive rounding errors. The drawback of this approach is that we have to rewrite the arithmetics of durations on this (centuries, nanosecond) encoding and guarantee it satisfies expected properties.</p>
<p>This is where Kani comes in very handy. From its definition, <code class="language-plaintext highlighter-rouge">Duration</code>s have a minimum and maximum value. On an unsigned 64-bit integer, we can store enough nanoseconds for four full centuries and a bit (but less than five centuries worth of nanoseconds). We want to normalize all operations such that the nanosecond counter stays within one century, unless we’ve reached the maximum (or minimum) number of centuries in which case, we want those nanoseconds to continue counting until the <code class="language-plaintext highlighter-rouge">u64</code> bound is reached. Centuries are stored in a <em>signed</em> 16-bit integer, and are the only field that stores whether the duration is positive or negative.</p>
<p>At first sight, it seems like a relatively simple task to make sure that this math is correct. It turns out that handling edge cases near the min and max durations, or when performing operations between very large and very small durations requires special attention, or even when crossing the boundary of the reference epoch. For example, the TAI reference epoch is 01 January 1900, so when subtracting one nanosecond from 01 January 1900 at midnight, the internal duration representation goes from <code class="language-plaintext highlighter-rouge">centuries: 0, nanoseconds: 0</code> to <code class="language-plaintext highlighter-rouge">centuries: -1, nanoseconds: 3_155_759_999_999_999_999</code>, as there are <code class="language-plaintext highlighter-rouge">3155759999999999999 + 1 = 3155760000000000000</code> nanoseconds in one century.</p>
<p>With Kani, we can check for all permutations of a definition of a <code class="language-plaintext highlighter-rouge">Duration</code>, and ensure that the decomposition of a <code class="language-plaintext highlighter-rouge">Duration</code> into its parts of days, hours, minutes, seconds, milliseconds, microseconds, and nanoseconds, never causes any overflow, underflow, or other unexpected behaviors. In hifitime, this is done simply by implementing <code class="language-plaintext highlighter-rouge">Arbitrary</code> for <code class="language-plaintext highlighter-rouge">Duration</code>, and calling the <code class="language-plaintext highlighter-rouge">decompose()</code> function on <em>any</em> duration. This test is beautiful in its simplicity: small code footprint for mighty guarantees, such could be the motto of Kani!</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">#[cfg(kani)]</span>
<span class="k">impl</span> <span class="n">Arbitrary</span> <span class="k">for</span> <span class="n">Duration</span> <span class="p">{</span>
<span class="nd">#[inline(always)]</span>
<span class="k">fn</span> <span class="nf">any</span><span class="p">()</span> <span class="k">-></span> <span class="n">Self</span> <span class="p">{</span>
<span class="k">let</span> <span class="n">centuries</span><span class="p">:</span> <span class="nb">i16</span> <span class="o">=</span> <span class="nn">kani</span><span class="p">::</span><span class="nf">any</span><span class="p">();</span>
<span class="k">let</span> <span class="n">nanoseconds</span><span class="p">:</span> <span class="nb">u64</span> <span class="o">=</span> <span class="nn">kani</span><span class="p">::</span><span class="nf">any</span><span class="p">();</span>
<span class="nn">Duration</span><span class="p">::</span><span class="nf">from_parts</span><span class="p">(</span><span class="n">centuries</span><span class="p">,</span> <span class="n">nanoseconds</span><span class="p">)</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="c">// (...)</span>
<span class="nd">#[cfg(kani)]</span>
<span class="nd">#[kani::proof]</span>
<span class="k">fn</span> <span class="nf">formal_duration_normalize_any</span><span class="p">()</span> <span class="p">{</span>
<span class="k">let</span> <span class="n">dur</span><span class="p">:</span> <span class="n">Duration</span> <span class="o">=</span> <span class="nn">kani</span><span class="p">::</span><span class="nf">any</span><span class="p">();</span>
<span class="c">// Check that decompose never fails</span>
<span class="n">dur</span><span class="nf">.decompose</span><span class="p">();</span>
<span class="p">}</span>
</code></pre></div></div>
<p>In the test above, if the call to <code class="language-plaintext highlighter-rouge">decompose</code> ever causes any overflow, underflow, division by zero, etc. then Kani will report an error and the exact value and bits inputted that caused the problem. This feedback is great, because one can just plugin those values into another test and debug it precisely. One will note that this test does not actually check the output value. That’s for two reasons. First, testing values would require developing either a formal specification or an independent implementation of the <code class="language-plaintext highlighter-rouge">decompose</code> function against which to check. Both options require quite a bit of effort, and the <code class="language-plaintext highlighter-rouge">decompose</code> function is really only used for displaying a duration in a human-readable format. Second, the purpose of the Hifitime Kani tests is to ensure that the aren’t any unsound operations. Finally, Hifitime has plenty of values that are explicitly tested for in the rest the tests.</p>
<h3 id="example-of-bug-found-with-kani">Example of bug found with Kani</h3>
<p>In a <a href="https://github.com/nyx-space/hifitime/blob/5f3a5bb06172a5d7eee98530543ac18036126af2/src/duration.rs#L285">previous version of Hifitime</a>, the <code class="language-plaintext highlighter-rouge">normalize</code> function would add two 64-bit unsigned integers to determine whether the duration being normalized would saturate to the <code class="language-plaintext highlighter-rouge">MIN</code> or <code class="language-plaintext highlighter-rouge">MAX</code>. Hifitime ensures that huge or tiny durations are saturated and don’t overflow. The exact erroneous code was as follows, and initially looks like it should be correct since we compute the remainder of the division by the number of nanoseconds in a century before adding it to the number of nanoseconds:</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">let</span> <span class="n">rem_nanos</span> <span class="o">=</span> <span class="k">self</span><span class="py">.nanoseconds</span><span class="nf">.rem_euclid</span><span class="p">(</span><span class="n">NANOSECONDS_PER_CENTURY</span><span class="p">);</span>
<span class="k">if</span> <span class="k">self</span><span class="py">.centuries</span> <span class="o">==</span> <span class="nn">i16</span><span class="p">::</span><span class="n">MIN</span> <span class="p">{</span>
<span class="k">if</span> <span class="n">rem_nanos</span> <span class="o">+</span> <span class="k">self</span><span class="py">.nanoseconds</span> <span class="o">></span> <span class="nn">Self</span><span class="p">::</span><span class="n">MIN</span><span class="py">.nanoseconds</span> <span class="p">{</span>
<span class="c">// We're at the min number of centuries already, and we have extra nanos, so we're saturated the duration limit</span>
<span class="o">*</span><span class="k">self</span> <span class="o">=</span> <span class="nn">Self</span><span class="p">::</span><span class="n">MIN</span><span class="p">;</span>
<span class="p">}</span>
<span class="c">// Else, we're near the MIN but we're within the MIN in nanoseconds, so let's not do anything here.</span>
<span class="p">}</span> <span class="c">// ...</span>
</code></pre></div></div>
<p>With that code, Kani found a counter example via the <code class="language-plaintext highlighter-rouge">formal_normalize_any</code> test that triggered an overflow when the number of centuries was set to its absolute minimum and the number of nanoseconds was zero. In fact, if the number of centuries is minimized, <code class="language-plaintext highlighter-rouge">Duration</code> allows incrementing the number of nanoseconds until the maximum 64-bit unsigned integer (this allows <code class="language-plaintext highlighter-rouge">Duration</code> to cover a large time span). The report clearly displayed the input values that caused the overflow, and the exact line where the overflow(s) occurred.</p>
<ul>
<li>File src/duration.rs
<ul>
<li>Function duration::Duration::normalize
<ul>
<li>Line 285: [trace] attempt to add with overflow</li>
<li>Line 291: [trace] attempt to add with overflow</li>
</ul>
</li>
</ul>
</li>
</ul>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(...)
Step 8962: Function MISSING, File MISSING, Line 0
extra_centuries = 5ul (00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000101)
Step 8963: Function MISSING, File MISSING, Line 0
-> core::num::<impl u64>::rem_euclid
Step 8964: Function MISSING, File MISSING, Line 0
self = 17480859535802999432ul (11110010 10011000 01111101 00100001 01011000 00001001 10100010 10001000)
Step 8965: Function MISSING, File MISSING, Line 0
rhs = 3155760000000000000ul (00101011 11001011 10000011 00000000 00000100 01100011 00000000 00000000)
Step 8966: Function MISSING, File MISSING, Line 0
<- core::num::<impl u64>::rem_euclid
Step 8967: Function MISSING, File MISSING, Line 0
rem_nanos = 1702059535802999432ul (00010111 10011110 11101110 00100001 01000010 00011010 10100010 10001000)
Step 8968: Function duration::Duration::normalize, File src/duration.rs, Line 285
if rem_nanos + self.nanoseconds > Self::MIN.nanoseconds { // We're at the min number of centuries already, and we have extra nanos, so we're saturated the duration limit *self = Self::MIN;
failure: duration::Duration::normalize.assertion.1: attempt to add with overflow
</code></pre></div></div>
<p>Now, the <a href="https://github.com/nyx-space/hifitime/blob/ef3777b7270898b9de1644e123917ed338e2126a/src/duration.rs#L287"><code class="language-plaintext highlighter-rouge">normalize</code></a> function is not only simpler but also correct.</p>
<h2 id="conclusion">Conclusion</h2>
<p>Kani has helped fix at least eight different categories of bugs in a single pull request: <a href="https://github.com/nyx-space/hifitime/pull/192">https://github.com/nyx-space/hifitime/pull/192</a>. Most of these were bugs near the boundaries of a <code class="language-plaintext highlighter-rouge">Duration</code> definition: around the zero, maximum, and minimum durations. But many of the bugs were on important and common operations: partial equality, negation, addition, and subtraction operations. These bugs weren’t due to lax testing: there are over 74 integration tests with plenty of checks within each.</p>
<p>One of the great features of Kani is that it performs what is known as symbolic execution of programs, where inputs are modelled as symbolic variables covering whole ranges of values at once. All program behaviors possible under these inputs are analyzed for defects like arithmetic overflows or underflows, signed conversion overflow or underflow, etc. If a defect is possible for some values of the inputs, Kani will generate a counter example trace with concrete values triggering the defect.</p>
<p>Thanks to how Kani analyzes a program, tests can either have explicit post-conditions or not. A test with explicit post-conditions includes an assertion: execute a set of instructions and then check something. This is a typical test case.</p>
<p>Kani can also test code where there is no explicit condition to check. Instead, only the successive operations of a function call are executed, and each are tested by Kani for failure cases by analyzing the inputs and finding cases where inputs will lead to runtime errors like overflows. This approach is how most of the bugs in hifitime have been found.</p>
<p>Tests without explicit post-conditions effectively ensure that sanity of the operations in a given function call. Explicit tests provide the same while also checking for conditions after the calls. If either of these tests fail, Kani can provide a test failure report outlining the sequence of operations, and the binary representation of each intermediate operation, to help the developer gain an understanding of why their implementation is incorrect.</p>
<p>The overhead to implement tests in Kani is very low, and the benefits
are immense. Hifitime has only eleven Kani tests, but that covers all
of the core functionality. Basically, write a Kani verification like a
unit test, add some assumptions on the values if desired, run the
model verifier, and you’ve formally verified this part of the code
(given the assumptions). Amazing!</p>
<h2 id="author-bio">Author bio</h2>
<p><a href="https://chrisrabotin.me/">Chris Rabotin</a> is a senior guidance, navigation, and controls (GNC) engineer at Rocket Lab USA.
His day to day revolves around trajectory design and orbit determination of Moon-bounder missions, and developing GNC algorithms in C++ that run on spacecraft and lunar landers. On his free time, he architects and develops high fidelity astrodynamics software in Rust and Python (<a href="https://nyxspace.com/">Nyx Space</a>) and focuses way too much on testing and validation of its results. Chris has over twenty years of experience in Python and picked up Rust in 2017 after encountering yet another memory overrun in a vector math library in C.</p>Kani is a verification tool that can help you systematically test properties about your Rust code. To learn more about Kani, check out the Kani tutorial and our previous blog posts.Kani Internship Projects 2022: Function Stubbing2023-02-28T00:00:00+00:002023-02-28T00:00:00+00:00https://model-checking.github.io//kani-verifier-blog/2023/02/28/kani-internship-projects-2022-stubbing<p>Kani is a verification tool that can help you systematically test properties about your Rust code.
To learn more about Kani, check out <a href="https://model-checking.github.io/kani/kani-tutorial.html">the Kani tutorial</a> and our <a href="https://model-checking.github.io/kani-verifier-blog/">previous blog posts</a>.</p>
<p>Today we’re continuing a series of posts on the internship projects carried out
in our team during 2022. The Kani team is proud to be part of the AWS Automated
Reasoning Group, which every year hosts a number of interns to work on
automated-reasoning projects for tools like Kani. More details on AWS Automated
Reasoning areas of work and available locations can be found <a href="https://2023arinternships.splashthat.com/">here</a>.
If you’re a Masters or PhD student interested in Automated Reasoning, please
consider applying to the following openings:</p>
<ul>
<li><a href="https://www.amazon.jobs/en/jobs/2173429/2023-applied-science-internship-automated-reasoning-united-states">2023 Applied Science Internship (Master’s student)</a></li>
<li><a href="https://www.amazon.jobs/en/jobs/2173372/2023-applied-science-internship-automated-reasoning-united-states">2023 Applied Science Internship (PhD student)</a></li>
</ul>
<p>In this post, we will talk about the internship project titled <em>Improving Kani’s Usability with Function and Method Stubbing</em>.</p>
<blockquote>
<p>This internship project was executed by <a href="https://people.seas.harvard.edu/~bembenek/">Aaron
Bembenek</a>. Aaron joined the Kani
team as an Applied Scientist Intern while finishing his graduate studies at the
<a href="https://seas.harvard.edu/">Harvard John A. Paulson School of Engineering and Applied
Sciences</a>. We are very grateful to Aaron for his hard
work on this project, and wish him the best in his PhD defense!</p>
</blockquote>
<h2 id="function-stubbing-in-kani">Function Stubbing in Kani</h2>
<p><strong>Function stubbing</strong> refers to users writing stub (mock) functions that they substitute for the real implementation during verification.
Although definitions for <em>mocking</em> (commonly used in testing) and <em>stubbing</em> may slightly differ, we often use both terms interchangeably.</p>
<p>In general, we have identified three reasons where users may consider stubbing out a function:</p>
<ol>
<li><strong>Unsupported features:</strong> The code under verification contains features that Kani does not support, such as inline assembly.</li>
<li><strong>Bad performance:</strong> The code under verification contains features that Kani supports, but it leads to bad verification performance (for example, deserialization code).</li>
<li><strong>Compositional reasoning:</strong> The code under verification contains code that has been verified separately.
Stubbing the code that has already been verified with a less complex version that soundly models its behavior can result in reduced verification workloads<sup id="fnref:footnote-contracts" role="doc-noteref"><a href="#fn:footnote-contracts" class="footnote" rel="footnote">1</a></sup>.</li>
</ol>
<p>Note that stubbing tries to solve a usability problem for Kani users by enabling them to verify code that otherwise would be impossible or impractical to verify.</p>
<p>To enable stubbing in the analysis, one must add the attribute <code class="language-plaintext highlighter-rouge">#[kani::stub(<function_name>, <stub_name>)]</code> to the harness function.
In addition, Kani must be called with options <code class="language-plaintext highlighter-rouge">--enable-unstable --enable-stubbing --harness <harness_name></code>.
Note that <code class="language-plaintext highlighter-rouge">--harness</code> is needed because the stubbing feature is limited to a single harness at the moment.</p>
<h3 id="an-example-stubbing-randrandom">An example: stubbing <code class="language-plaintext highlighter-rouge">rand::random</code></h3>
<p>Let’s see a simple example where we use the <a href="https://docs.rs/rand/latest/rand/fn.random.html"><code class="language-plaintext highlighter-rouge">rand::random</code></a> function
to generate an encryption key.</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">#[cfg(kani)]</span>
<span class="nd">#[kani::proof]</span>
<span class="k">fn</span> <span class="nf">encrypt_then_decrypt_is_identity</span><span class="p">()</span> <span class="p">{</span>
<span class="k">let</span> <span class="n">data</span><span class="p">:</span> <span class="nb">u32</span> <span class="o">=</span> <span class="nn">kani</span><span class="p">::</span><span class="nf">any</span><span class="p">();</span>
<span class="k">let</span> <span class="n">encryption_key</span><span class="p">:</span> <span class="nb">u32</span> <span class="o">=</span> <span class="nn">rand</span><span class="p">::</span><span class="nf">random</span><span class="p">();</span>
<span class="k">let</span> <span class="n">encrypted_data</span> <span class="o">=</span> <span class="n">data</span> <span class="o">^</span> <span class="n">encryption_key</span><span class="p">;</span>
<span class="k">let</span> <span class="n">decrypted_data</span> <span class="o">=</span> <span class="n">encrypted_data</span> <span class="o">^</span> <span class="n">encryption_key</span><span class="p">;</span>
<span class="nd">assert_eq!</span><span class="p">(</span><span class="n">data</span><span class="p">,</span> <span class="n">decrypted_data</span><span class="p">);</span>
<span class="p">}</span>
</code></pre></div></div>
<p>At present, Kani fails to verify this example because its <a href="https://doc.rust-lang.org/nomicon/ffi.html">FFI</a> support requires improvements (see <a href="https://github.com/model-checking/kani/issues/1781">issue #1781</a> for more details).
In other words, the code under verification contains unsupported features.</p>
<p>However, the stubbing feature allows us to work around this limitation<sup id="fnref:footnote-limitation" role="doc-noteref"><a href="#fn:footnote-limitation" class="footnote" rel="footnote">2</a></sup> as follows:</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">#[cfg(kani)]</span>
<span class="k">fn</span> <span class="n">stub_random</span><span class="o"><</span><span class="n">T</span><span class="p">:</span> <span class="nn">kani</span><span class="p">::</span><span class="n">Arbitrary</span><span class="o">></span><span class="p">()</span> <span class="k">-></span> <span class="n">T</span> <span class="p">{</span>
<span class="nn">kani</span><span class="p">::</span><span class="nf">any</span><span class="p">()</span>
<span class="p">}</span>
<span class="nd">#[cfg(kani)]</span>
<span class="nd">#[kani::proof]</span>
<span class="nd">#[kani::stub(rand::random,</span> <span class="nd">stub_random)]</span>
<span class="k">fn</span> <span class="nf">encrypt_then_decrypt_is_identity</span><span class="p">()</span> <span class="p">{</span>
<span class="k">let</span> <span class="n">data</span><span class="p">:</span> <span class="nb">u32</span> <span class="o">=</span> <span class="nn">kani</span><span class="p">::</span><span class="nf">any</span><span class="p">();</span>
<span class="k">let</span> <span class="n">encryption_key</span><span class="p">:</span> <span class="nb">u32</span> <span class="o">=</span> <span class="nn">rand</span><span class="p">::</span><span class="nf">random</span><span class="p">();</span>
<span class="k">let</span> <span class="n">encrypted_data</span> <span class="o">=</span> <span class="n">data</span> <span class="o">^</span> <span class="n">encryption_key</span><span class="p">;</span>
<span class="k">let</span> <span class="n">decrypted_data</span> <span class="o">=</span> <span class="n">encrypted_data</span> <span class="o">^</span> <span class="n">encryption_key</span><span class="p">;</span>
<span class="nd">assert_eq!</span><span class="p">(</span><span class="n">data</span><span class="p">,</span> <span class="n">decrypted_data</span><span class="p">);</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Here, the <code class="language-plaintext highlighter-rouge">#[kani::stub(rand::random, stub_random)]</code> attribute indicates to Kani that it should replace <code class="language-plaintext highlighter-rouge">rand::random</code> with the stub <code class="language-plaintext highlighter-rouge">stub_random</code>.
This is a sound replacement: the value returned by <code class="language-plaintext highlighter-rouge">kani::any</code> captures all possible <code class="language-plaintext highlighter-rouge">u32</code> values returned by <code class="language-plaintext highlighter-rouge">rand::random</code>.</p>
<p>Now, let’s run it through Kani:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>cargo kani <span class="nt">--enable-unstable</span> <span class="nt">--enable-stubbing</span> <span class="nt">--harness</span> encrypt_then_decrypt_is_identity
</code></pre></div></div>
<p>The verification result is composed of a single check: the assertion corresponding to <code class="language-plaintext highlighter-rouge">assert_eq!(data, decrypted_data)</code>.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>RESULTS:
Check 1: encrypt_then_decrypt_is_identity.assertion.1
- Status: SUCCESS
- Description: "assertion failed: data == decrypted_data"
- Location: src/main.rs:18:5 in function encrypt_then_decrypt_is_identity
SUMMARY:
** 0 of 1 failed
VERIFICATION:- SUCCESSFUL
</code></pre></div></div>
<p>Kani verifies the assertion successfully, avoiding any issues that appear if we attempt to verify the code without stubbing.</p>
<h3 id="another-example-stubbing-factorial">Another example: stubbing <code class="language-plaintext highlighter-rouge">factorial</code></h3>
<p>Let’s see another example where we compute <a href="https://en.wikipedia.org/wiki/Binomial_coefficient">binomial coefficients</a>.
This computation is often expressed as <code class="language-plaintext highlighter-rouge">n choose k</code> (implemented by <code class="language-plaintext highlighter-rouge">choose</code> in below),
and it represents the number of ways to choose <code class="language-plaintext highlighter-rouge">k</code> elements from a set of <code class="language-plaintext highlighter-rouge">n</code> elements.
In this example, we’re interested in verifying the property <code class="language-plaintext highlighter-rouge">(n choose k) == (n choose (n - k))</code>.</p>
<p>Note that <code class="language-plaintext highlighter-rouge">choose</code> makes use of a recursive <code class="language-plaintext highlighter-rouge">factorial</code> function that uses the method <code class="language-plaintext highlighter-rouge">checked_mul</code> to prevent overflow errors.</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">fn</span> <span class="nf">factorial</span><span class="p">(</span><span class="n">n</span><span class="p">:</span> <span class="nb">u64</span><span class="p">)</span> <span class="k">-></span> <span class="nb">Option</span><span class="o"><</span><span class="nb">u64</span><span class="o">></span> <span class="p">{</span>
<span class="k">if</span> <span class="n">n</span> <span class="o">==</span> <span class="mi">0</span> <span class="p">{</span>
<span class="k">return</span> <span class="nf">Some</span><span class="p">(</span><span class="mi">1</span><span class="p">);</span>
<span class="p">}</span>
<span class="n">n</span><span class="nf">.checked_mul</span><span class="p">(</span><span class="nf">factorial</span><span class="p">(</span><span class="n">n</span> <span class="o">-</span> <span class="mi">1</span><span class="p">)</span><span class="nf">.unwrap</span><span class="p">())</span>
<span class="p">}</span>
<span class="k">fn</span> <span class="nf">choose</span><span class="p">(</span><span class="n">n</span><span class="p">:</span> <span class="nb">u64</span><span class="p">,</span> <span class="n">k</span><span class="p">:</span> <span class="nb">u64</span><span class="p">)</span> <span class="k">-></span> <span class="nb">u64</span> <span class="p">{</span>
<span class="k">let</span> <span class="n">fact_n</span> <span class="o">=</span> <span class="nf">factorial</span><span class="p">(</span><span class="n">n</span><span class="p">)</span><span class="nf">.unwrap</span><span class="p">();</span>
<span class="k">let</span> <span class="n">fact_k</span> <span class="o">=</span> <span class="nf">factorial</span><span class="p">(</span><span class="n">k</span><span class="p">)</span><span class="nf">.unwrap</span><span class="p">();</span>
<span class="k">let</span> <span class="n">fact_n_minus_k</span> <span class="o">=</span> <span class="nf">factorial</span><span class="p">(</span><span class="n">n</span> <span class="o">-</span> <span class="n">k</span><span class="p">)</span><span class="nf">.unwrap</span><span class="p">();</span>
<span class="n">fact_n</span> <span class="o">/</span> <span class="p">(</span><span class="n">fact_k</span> <span class="o">*</span> <span class="n">fact_n_minus_k</span><span class="p">)</span>
<span class="p">}</span>
<span class="nd">#[cfg(kani)]</span>
<span class="nd">#[kani::proof]</span>
<span class="nd">#[kani::unwind(</span><span class="mi">22</span><span class="nd">)]</span>
<span class="k">fn</span> <span class="nf">verify_choose</span><span class="p">()</span> <span class="p">{</span>
<span class="k">let</span> <span class="n">n</span> <span class="o">=</span> <span class="nn">kani</span><span class="p">::</span><span class="nf">any</span><span class="p">();</span>
<span class="k">let</span> <span class="n">k</span> <span class="o">=</span> <span class="nn">kani</span><span class="p">::</span><span class="nf">any</span><span class="p">();</span>
<span class="nn">kani</span><span class="p">::</span><span class="nf">assume</span><span class="p">(</span><span class="n">n</span> <span class="o"><</span> <span class="mi">21</span><span class="p">);</span>
<span class="nn">kani</span><span class="p">::</span><span class="nf">assume</span><span class="p">(</span><span class="n">k</span> <span class="o"><</span> <span class="mi">21</span><span class="p">);</span>
<span class="nn">kani</span><span class="p">::</span><span class="nf">assume</span><span class="p">(</span><span class="n">n</span> <span class="o">>=</span> <span class="n">k</span><span class="p">);</span>
<span class="nd">assert_eq!</span><span class="p">(</span><span class="nf">choose</span><span class="p">(</span><span class="n">n</span><span class="p">,</span> <span class="n">k</span><span class="p">),</span> <span class="nf">choose</span><span class="p">(</span><span class="n">n</span><span class="p">,</span> <span class="n">n</span> <span class="o">-</span> <span class="n">k</span><span class="p">));</span>
<span class="p">}</span>
</code></pre></div></div>
<p>This example takes more than 15 minutes to verify on an <a href="https://aws.amazon.com/ec2/instance-types/">AWS EC2 <code class="language-plaintext highlighter-rouge">m5a.4xlarge</code> instance</a>.
The code under verification shows bad performance due to the recursive definition of the <code class="language-plaintext highlighter-rouge">factorial</code> function.</p>
<p>But we can avoid the recursion by pre-computing the values<sup id="fnref:footnote-values" role="doc-noteref"><a href="#fn:footnote-values" class="footnote" rel="footnote">3</a></sup> that fit into a <code class="language-plaintext highlighter-rouge">u64</code> value and replacing <code class="language-plaintext highlighter-rouge">factorial</code> with another function that gets those values.</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">const</span> <span class="n">FACT</span><span class="p">:</span> <span class="p">[</span><span class="nb">u64</span><span class="p">;</span> <span class="mi">21</span><span class="p">]</span> <span class="o">=</span> <span class="p">[</span>
<span class="mi">1</span><span class="p">,</span>
<span class="mi">1</span><span class="p">,</span>
<span class="mi">2</span><span class="p">,</span>
<span class="mi">6</span><span class="p">,</span>
<span class="mi">24</span><span class="p">,</span>
<span class="mi">120</span><span class="p">,</span>
<span class="mi">720</span><span class="p">,</span>
<span class="mi">5040</span><span class="p">,</span>
<span class="mi">40320</span><span class="p">,</span>
<span class="mi">362880</span><span class="p">,</span>
<span class="mi">3628800</span><span class="p">,</span>
<span class="mi">39916800</span><span class="p">,</span>
<span class="mi">479001600</span><span class="p">,</span>
<span class="mi">6227020800</span><span class="p">,</span>
<span class="mi">87178291200</span><span class="p">,</span>
<span class="mi">1307674368000</span><span class="p">,</span>
<span class="mi">20922789888000</span><span class="p">,</span>
<span class="mi">355687428096000</span><span class="p">,</span>
<span class="mi">6402373705728000</span><span class="p">,</span>
<span class="mi">121645100408832000</span><span class="p">,</span>
<span class="mi">2432902008176640000</span><span class="p">,</span>
<span class="p">];</span>
<span class="k">fn</span> <span class="nf">factorial</span><span class="p">(</span><span class="n">n</span><span class="p">:</span> <span class="nb">u64</span><span class="p">)</span> <span class="k">-></span> <span class="nb">Option</span><span class="o"><</span><span class="nb">u64</span><span class="o">></span> <span class="p">{</span>
<span class="k">if</span> <span class="n">n</span> <span class="o">==</span> <span class="mi">0</span> <span class="p">{</span>
<span class="k">return</span> <span class="nf">Some</span><span class="p">(</span><span class="mi">1</span><span class="p">);</span>
<span class="p">}</span>
<span class="n">n</span><span class="nf">.checked_mul</span><span class="p">(</span><span class="nf">factorial</span><span class="p">(</span><span class="n">n</span> <span class="o">-</span> <span class="mi">1</span><span class="p">)</span><span class="nf">.unwrap</span><span class="p">())</span>
<span class="p">}</span>
<span class="k">fn</span> <span class="nf">choose</span><span class="p">(</span><span class="n">n</span><span class="p">:</span> <span class="nb">u64</span><span class="p">,</span> <span class="n">k</span><span class="p">:</span> <span class="nb">u64</span><span class="p">)</span> <span class="k">-></span> <span class="nb">u64</span> <span class="p">{</span>
<span class="k">let</span> <span class="n">fact_n</span> <span class="o">=</span> <span class="nf">factorial</span><span class="p">(</span><span class="n">n</span><span class="p">)</span><span class="nf">.unwrap</span><span class="p">();</span>
<span class="k">let</span> <span class="n">fact_k</span> <span class="o">=</span> <span class="nf">factorial</span><span class="p">(</span><span class="n">k</span><span class="p">)</span><span class="nf">.unwrap</span><span class="p">();</span>
<span class="k">let</span> <span class="n">fact_n_minus_k</span> <span class="o">=</span> <span class="nf">factorial</span><span class="p">(</span><span class="n">n</span> <span class="o">-</span> <span class="n">k</span><span class="p">)</span><span class="nf">.unwrap</span><span class="p">();</span>
<span class="n">fact_n</span> <span class="o">/</span> <span class="p">(</span><span class="n">fact_k</span> <span class="o">*</span> <span class="n">fact_n_minus_k</span><span class="p">)</span>
<span class="p">}</span>
<span class="nd">#[cfg(kani)]</span>
<span class="k">fn</span> <span class="nf">stub_factorial</span><span class="p">(</span><span class="n">n_64</span><span class="p">:</span> <span class="nb">u64</span><span class="p">)</span> <span class="k">-></span> <span class="nb">Option</span><span class="o"><</span><span class="nb">u64</span><span class="o">></span> <span class="p">{</span>
<span class="k">let</span> <span class="n">n</span> <span class="o">=</span> <span class="n">n_64</span> <span class="k">as</span> <span class="nb">usize</span><span class="p">;</span>
<span class="k">if</span> <span class="n">n</span> <span class="o"><</span> <span class="n">FACT</span><span class="nf">.len</span><span class="p">()</span> <span class="p">{</span>
<span class="nf">Some</span><span class="p">(</span><span class="n">FACT</span><span class="p">[</span><span class="n">n</span><span class="p">])</span>
<span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
<span class="nb">None</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="nd">#[cfg(kani)]</span>
<span class="nd">#[kani::proof]</span>
<span class="nd">#[kani::unwind(</span><span class="mi">22</span><span class="nd">)]</span>
<span class="nd">#[kani::stub(factorial,</span> <span class="nd">stub_factorial)]</span>
<span class="k">fn</span> <span class="nf">verify_choose</span><span class="p">()</span> <span class="p">{</span>
<span class="k">let</span> <span class="n">n</span> <span class="o">=</span> <span class="nn">kani</span><span class="p">::</span><span class="nf">any</span><span class="p">();</span>
<span class="k">let</span> <span class="n">k</span> <span class="o">=</span> <span class="nn">kani</span><span class="p">::</span><span class="nf">any</span><span class="p">();</span>
<span class="nn">kani</span><span class="p">::</span><span class="nf">assume</span><span class="p">(</span><span class="n">n</span> <span class="o">></span> <span class="mi">0</span> <span class="o">&&</span> <span class="n">n</span> <span class="o"><</span> <span class="mi">21</span><span class="p">);</span>
<span class="nn">kani</span><span class="p">::</span><span class="nf">assume</span><span class="p">(</span><span class="n">k</span> <span class="o">></span> <span class="mi">0</span> <span class="o">&&</span> <span class="n">k</span> <span class="o"><</span> <span class="mi">21</span><span class="p">);</span>
<span class="nn">kani</span><span class="p">::</span><span class="nf">assume</span><span class="p">(</span><span class="n">n</span> <span class="o">>=</span> <span class="n">k</span><span class="p">);</span>
<span class="nd">assert_eq!</span><span class="p">(</span><span class="nf">choose</span><span class="p">(</span><span class="n">n</span><span class="p">,</span> <span class="n">k</span><span class="p">),</span> <span class="nf">choose</span><span class="p">(</span><span class="n">n</span><span class="p">,</span> <span class="n">n</span> <span class="o">-</span> <span class="n">k</span><span class="p">));</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Let’s run this harness with Kani!</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>cargo kani <span class="nt">--enable-unstable</span> <span class="nt">--enable-stubbing</span> <span class="nt">--harness</span> verify_choose
</code></pre></div></div>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>SUMMARY:
** 0 of 10 failed
VERIFICATION:- SUCCESSFUL
Verification Time: 13.64521s
</code></pre></div></div>
<p>Now Kani verifies the example successfully, and in less 15 seconds!
This time, Kani completes the verification faster because our stub avoids recursion altogether.</p>
<h2 id="risks-of-stubbing">Risks of Stubbing</h2>
<p>Stubbing is a feature that comes with great power and, as such, it should be used with caution.
It’s the developer’s responsibility to <strong>ensure that a stub replacing another function soundly models its behavior</strong>.
This is normally not the case in testing, where developers writing <em>mock-ups</em> tend to setup a concrete version of an object (an <em>under-approximation</em> of the model).</p>
<p>For example, let’s suppose you’re attempting to stub a call to <a href="https://docs.rs/serde_json/latest/serde_json/de/fn.from_slice.html"><code class="language-plaintext highlighter-rouge">serde_json::from_slice</code></a> in your harness.
This isn’t a strange situation since deserialization often leads to bad performance.</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">#[derive(Deserialize)]</span>
<span class="nd">#[cfg_attr(kani,</span> <span class="nd">derive(kani::Arbitrary))]</span>
<span class="k">pub</span> <span class="k">struct</span> <span class="n">MyStruct</span> <span class="p">{</span>
<span class="c">// -- snip -- </span>
<span class="p">}</span>
<span class="nd">#[cfg(kani)]</span>
<span class="k">fn</span> <span class="n">stub_deserialize</span><span class="o"><</span><span class="n">S</span><span class="p">,</span> <span class="n">T</span><span class="o">></span><span class="p">(</span><span class="mi">_</span><span class="n">data</span><span class="p">:</span> <span class="o">&</span><span class="p">[</span><span class="nb">u8</span><span class="p">])</span> <span class="k">-></span> <span class="nn">serde_json</span><span class="p">::</span><span class="n">Result</span><span class="o"><</span><span class="n">T</span><span class="o">></span>
<span class="k">where</span>
<span class="n">T</span><span class="p">:</span> <span class="nn">kani</span><span class="p">::</span><span class="n">Arbitrary</span><span class="p">,</span>
<span class="p">{</span>
<span class="nf">Ok</span><span class="p">(</span><span class="nn">kani</span><span class="p">::</span><span class="nf">any</span><span class="p">())</span>
<span class="p">}</span>
<span class="nd">#[cfg(kani)]</span>
<span class="nd">#[kani::proof]</span>
<span class="nd">#[kani::stub(serde_json::from_slice,</span> <span class="nd">stub_deserialize)]</span>
<span class="k">fn</span> <span class="nf">verify_with_deserialization</span><span class="p">()</span> <span class="p">{</span>
<span class="k">let</span> <span class="n">data</span> <span class="o">=</span> <span class="nf">symbolic_slice</span><span class="p">();</span>
<span class="k">let</span> <span class="n">my_struct</span> <span class="o">=</span> <span class="nn">serde_json</span><span class="p">::</span><span class="nn">from_slice</span><span class="p">::</span><span class="o"><</span><span class="n">MyStruct</span><span class="o">></span><span class="p">(</span><span class="o">&</span><span class="n">data</span><span class="p">);</span>
<span class="c">// -- snip -- </span>
<span class="p">}</span>
</code></pre></div></div>
<p>Here, we’re using <code class="language-plaintext highlighter-rouge">kani::any()</code> to generate a symbolic <code class="language-plaintext highlighter-rouge">MyStruct</code>.
This is easy to do since we’re deriving <code class="language-plaintext highlighter-rouge">kani::Arbitrary</code> for <code class="language-plaintext highlighter-rouge">MyStruct</code>.</p>
<p>In principle, this may look like an <strong>over-approximation</strong> because we generate a symbolic value for <code class="language-plaintext highlighter-rouge">MyStruct</code>.
But actually, we’re missing something:
<a href="https://docs.rs/serde_json/latest/serde_json/type.Result.html"><code class="language-plaintext highlighter-rouge">serde_json::Result<T></code></a> can also return a <code class="language-plaintext highlighter-rouge">serde_json::Error</code> if the deserialization fails.
However, our <code class="language-plaintext highlighter-rouge">stub_deserialize</code> function assumes that this case will never happen!
Therefore, the stub we wrote cannot be considered a <strong>sound model</strong> of <code class="language-plaintext highlighter-rouge">serde_json::from_slice</code>, and it’s dangerous to use it in our harnesses.</p>
<p>We hope this convinces you about the risks of stubbing.
Always keep in mind that stubs are essentially additional assumptions in your harnesses.
Because of that, we recommend our users to only use stubbing when necessary.
Note that you can also write harnesses for your stubs if you need more assurance.</p>
<h2 id="considered-designs">Considered Designs</h2>
<p>The experimentation for this feature mainly focused on the program transformation required for stubbing functions, methods and types.
While the feature had been scoped down to function stubbing, we wanted to keep an eye open for any alternative that allowed us do type stubbing<sup id="fnref:footnote-type-stubbing" role="doc-noteref"><a href="#fn:footnote-type-stubbing" class="footnote" rel="footnote">4</a></sup>.
For functions and methods, the transformation boils down to replacing all calls to the original function/method with calls to the replacement function/method.</p>
<p>That said, the transformation can also be applied at different stages of the compilation step.
As you may know, the Rust language uses multiple <a href="https://rustc-dev-guide.rust-lang.org/overview.html#intermediate-representations">Intermediate Representations</a> (IRs) to represent and perform analyses on Rust programs.
In total, we considered five approaches:</p>
<ul>
<li>Conditional compilation<sup id="fnref:footnote-conditional" role="doc-noteref"><a href="#fn:footnote-conditional" class="footnote" rel="footnote">5</a></sup></li>
<li>Source-to-source transformation</li>
<li>AST-to-AST transformation</li>
<li>HIR-to-HIR transformation</li>
<li>MIR-to-MIR transformation</li>
</ul>
<p>In summary, we tried to answer the following question for each approach:</p>
<blockquote>
<p>Does the approach allow stubbing of local/external functions/methods/types?</p>
</blockquote>
<table>
<thead>
<tr>
<th> </th>
<th>Conditional compilation</th>
<th>Source-to-source transformation</th>
<th>AST-to-AST transformation</th>
<th>HIR-to-HIR transformation</th>
<th>MIR-to-MIR transformation</th>
</tr>
</thead>
<tbody>
<tr>
<td>Local functions</td>
<td>Yes</td>
<td>Yes</td>
<td>Yes</td>
<td>Yes</td>
<td>Yes</td>
</tr>
<tr>
<td>Local methods</td>
<td>Yes</td>
<td>Yes</td>
<td>Yes</td>
<td>Yes</td>
<td>Yes</td>
</tr>
<tr>
<td>Local types</td>
<td>Yes</td>
<td>Yes</td>
<td>Yes</td>
<td>Yes</td>
<td>No</td>
</tr>
<tr>
<td>External functions</td>
<td>No</td>
<td>Yes</td>
<td>Maybe</td>
<td>Maybe</td>
<td>Yes</td>
</tr>
<tr>
<td>External methods</td>
<td>No</td>
<td>Yes</td>
<td>Maybe</td>
<td>Maybe</td>
<td>Yes</td>
</tr>
<tr>
<td>External types</td>
<td>No</td>
<td>Yes</td>
<td>Maybe</td>
<td>Maybe</td>
<td>No</td>
</tr>
</tbody>
</table>
<p>The main disadvantage of the conditional compilation approach is that it cannot be applied to external code.</p>
<p>In a source-to-source transformation, we’d rewrite the source code before it gets to the compiler.
This approach is the most flexible, allowing us to basically stub any code (functions, methods, types).
However, it requires all source code to be available, and in general it’s difficult to work with (e.g., unexpanded macros).</p>
<p>AST-to-AST and HIR-to-HIR transformations are close to the source code, so they’re quite flexible too.
The main downside is that they’d require modifications to the Rust compiler (to plug in new AST/HIR passes).
Moreover, they’d introduce new issues while compiling dependencies<sup id="fnref:footnote-alternatives" role="doc-noteref"><a href="#fn:footnote-alternatives" class="footnote" rel="footnote">6</a></sup>.</p>
<p>Therefore, <strong>the approach we followed was the MIR-to-MIR transformation</strong>.
While it doesn’t allow stubbing types, the MIR-to-MIR approach presents many advantages:</p>
<ul>
<li>It operates over a relatively simple IR.</li>
<li>The Rust compiler already has good support for plugging in MIR-to-MIR transformations.</li>
<li>It’s possible to integrate this transformation with <a href="https://model-checking.github.io/kani-verifier-blog/2022/09/22/internship-projects-2022-concrete-playback.html">concrete playback</a>.</li>
<li>Our team is already familiar with the MIR interface.</li>
</ul>
<h2 id="summary">Summary</h2>
<p>In this post, we’ve showed the stubbing feature in action and briefly discussed the designs we considered.
If you enjoyed this post, the <a href="https://model-checking.github.io/kani/rfc/rfcs/0002-function-stubbing.html">RFC for Function Stubbing</a> includes other topics that you may find interesting:</p>
<ul>
<li>The rules to determine <a href="https://model-checking.github.io/kani/rfc/rfcs/0002-function-stubbing.html#stub-compatibility-and-validation">stub compatibility and validation</a>.</li>
<li>A <a href="https://model-checking.github.io/kani/rfc/rfcs/0002-function-stubbing.html#comparison-to-function-contracts">comparison to function contracts</a>.</li>
<li>The list of <a href="https://model-checking.github.io/kani/rfc/rfcs/0002-function-stubbing.html#future-possibilities">future possibilities for stubbing</a>.</li>
</ul>
<p>Please go ahead and try the stubbing feature yourself!</p>
<p>The documentation for stubbing is available <a href="https://model-checking.github.io/kani/reference/stubbing.html">here</a>.
Please let us know if you <a href="https://github.com/model-checking/kani/issues/new?assignees=&labels=bug&template=bug_report.md">find any issues</a> while
using the feature.</p>
<p>Overall, we had a lot of fun working with <a href="https://people.seas.harvard.edu/~bembenek/">Aaron</a> on this project, and we’re sure
it’ll improve Kani’s usability in the future!</p>
<h2 id="references">References</h2>
<ol>
<li><a href="https://model-checking.github.io/kani/rfc/rfcs/0002-function-stubbing.html">RFC for Function Stubbing</a></li>
<li><a href="https://model-checking.github.io/kani/reference/stubbing.html">Documentation for Stubbing</a></li>
</ol>
<h3 id="footnotes">Footnotes</h3>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:footnote-contracts" role="doc-endnote">
<p>In this case, function contracts are a good alternative to stubbing. You can find a comparison to function contracts in <a href="https://model-checking.github.io/kani/rfc/rfcs/0002-function-stubbing.html#comparison-to-function-contracts">this section of the RFC for function stubbing</a>. Unfortunately, functions contracts aren’t available in Kani at the moment. <a href="#fnref:footnote-contracts" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:footnote-limitation" role="doc-endnote">
<p>Another option is to supply an alternative implementation for verification using <code class="language-plaintext highlighter-rouge">#[cfg(kani)]</code>. In fact, this is considered the baseline approach for stubbing. Note, however, that this approach isn’t valid for external code. <a href="#fnref:footnote-limitation" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:footnote-values" role="doc-endnote">
<p>We only need to pre-compute 21 values because the factorial computation overflows for any value beyond 21. <a href="#fnref:footnote-values" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:footnote-type-stubbing" role="doc-endnote">
<p>Type stubbing would be a powerful technique to have in Kani. It could allow us to provide verification-friendly stubs for frequently used types (e.g., <code class="language-plaintext highlighter-rouge">Vec</code>), among many other things. <a href="#fnref:footnote-type-stubbing" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:footnote-conditional" role="doc-endnote">
<p>Conditional compilation refers to making use of <code class="language-plaintext highlighter-rouge">#[cfg(kani)]</code> and <code class="language-plaintext highlighter-rouge">#[cfg(not(kani))]</code> to guard the code that’s used for verification and standard compilation, respectively. Note that we could’ve used it in <a href="#an-example-stubbing-randrandom">our first example</a>, by specifying <code class="language-plaintext highlighter-rouge">#[cfg(not(kani))]</code> for the <code class="language-plaintext highlighter-rouge">rand::random</code> call and <code class="language-plaintext highlighter-rouge">#[cfg(kani)]</code> for the <code class="language-plaintext highlighter-rouge">kani::any</code> call. <a href="#fnref:footnote-conditional" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:footnote-alternatives" role="doc-endnote">
<p>The advantages and disadvantages of these approaches are better explained in <a href="https://model-checking.github.io/kani/rfc/rfcs/0002-function-stubbing.html#rationale-and-alternatives-stubbing-mechanism">this section of the RFC for function stubbing</a>. <a href="#fnref:footnote-alternatives" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>Kani is a verification tool that can help you systematically test properties about your Rust code. To learn more about Kani, check out the Kani tutorial and our previous blog posts.Checking Code Reachability and Sanity Checking Proof Harnesses with kani::cover2023-01-30T00:00:00+00:002023-01-30T00:00:00+00:00https://model-checking.github.io//kani-verifier-blog/2023/01/30/reachability-and-sanity-checking-with-kani-cover<p>Kani is a verification tool that can help you prove properties about your Rust code.
To learn more about Kani, check out <a href="https://model-checking.github.io/kani/kani-tutorial.html">the Kani tutorial</a> and our <a href="https://model-checking.github.io/kani-verifier-blog/">previous blog posts</a>.</p>
<p>In today’s blog post, we will talk about two applications of the new <code class="language-plaintext highlighter-rouge">kani::cover</code> macro that was introduced in <a href="https://github.com/model-checking/kani/releases/tag/kani-0.18.0">Kani 0.18</a>:</p>
<ol>
<li>Determining whether certain lines of code are reachable, and generating tests that cover them.</li>
<li>Sanity checking proof harnesses to ensure that what they’re verifying matches our expectations.</li>
</ol>
<p>Let’s start by explaining what <code class="language-plaintext highlighter-rouge">kani::cover</code> is.</p>
<h2 id="what-is-kanicover">What is <code class="language-plaintext highlighter-rouge">kani::cover</code>?</h2>
<p>In a nutshell, <code class="language-plaintext highlighter-rouge">kani::cover</code> is a macro that instructs Kani to check whether a certain condition at a certain point in the code can be satisfied from a given harness by at least one execution.
For each <code class="language-plaintext highlighter-rouge">kani::cover</code> call, Kani analyzes the code and arrives at one of two possible outcomes:</p>
<ol>
<li>It finds a possible execution of the program that satisfies the condition</li>
<li>It proves that no such execution is possible from the given harness</li>
</ol>
<p>Let’s go through an example.
Suppose we’d like to find out if there’s a 16-bit integer greater than 8 whose cube, computed with wrapping multiplication, can yield the value 8.
We can write a harness with <code class="language-plaintext highlighter-rouge">kani::cover</code> to find out the answer:</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">#[kani::proof]</span>
<span class="k">fn</span> <span class="nf">cube_value</span><span class="p">()</span> <span class="p">{</span>
<span class="k">let</span> <span class="n">x</span><span class="p">:</span> <span class="nb">u16</span> <span class="o">=</span> <span class="nn">kani</span><span class="p">::</span><span class="nf">any</span><span class="p">();</span>
<span class="k">let</span> <span class="n">x_cubed</span> <span class="o">=</span> <span class="n">x</span><span class="nf">.wrapping_mul</span><span class="p">(</span><span class="n">x</span><span class="p">)</span><span class="nf">.wrapping_mul</span><span class="p">(</span><span class="n">x</span><span class="p">);</span>
<span class="k">if</span> <span class="n">x</span> <span class="o">></span> <span class="mi">8</span> <span class="p">{</span>
<span class="nn">kani</span><span class="p">::</span><span class="nd">cover!</span><span class="p">(</span><span class="n">x_cubed</span> <span class="o">==</span> <span class="mi">8</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>If we run Kani on this example, it tells us that the cover property is satisfiable:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>kani test.rs
Checking harness cube_value...
<span class="c"># -- snip --</span>
RESULTS:
Check 1: cube_value.cover.1
- Status: SATISFIED
- Description: <span class="s2">"cover condition: x_cubed == 8"</span>
- Location: test.rs:6:9 <span class="k">in function </span>cube_value
SUMMARY:
<span class="k">**</span> 1 of 1 cover properties satisfied
</code></pre></div></div>
<p>meaning that there is indeed a 16-bit integer that satisfies this condition.
We can use Kani’s <a href="https://model-checking.github.io/kani-verifier-blog/2022/09/22/internship-projects-2022-concrete-playback.html">concrete playback feature</a> to find a particular value, which will give us one of the following values: 16386, 32770, or 49154.</p>
<p>On the other hand, if we ask the same question replacing 8 with 4 or 27, Kani will tell us that no 16-bit integer satisfies this condition:</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">#[kani::proof]</span>
<span class="k">fn</span> <span class="nf">cube_value</span><span class="p">()</span> <span class="p">{</span>
<span class="k">let</span> <span class="n">x</span><span class="p">:</span> <span class="nb">u16</span> <span class="o">=</span> <span class="nn">kani</span><span class="p">::</span><span class="nf">any</span><span class="p">();</span>
<span class="k">let</span> <span class="n">x_cubed</span> <span class="o">=</span> <span class="n">x</span><span class="nf">.wrapping_mul</span><span class="p">(</span><span class="n">x</span><span class="p">)</span><span class="nf">.wrapping_mul</span><span class="p">(</span><span class="n">x</span><span class="p">);</span>
<span class="k">if</span> <span class="n">x</span> <span class="o">></span> <span class="mi">27</span> <span class="p">{</span>
<span class="nn">kani</span><span class="p">::</span><span class="nd">cover!</span><span class="p">(</span><span class="n">x_cubed</span> <span class="o">==</span> <span class="mi">27</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>RESULTS:
Check 1: cube_value.cover.1
- Status: UNSATISFIABLE
- Description: "cover condition: x_cubed == 27"
- Location: test.rs:6:9 in function cube_value
SUMMARY:
** 0 of 1 cover properties satisfied
</code></pre></div></div>
<p>Thus, Kani along with <code class="language-plaintext highlighter-rouge">kani::cover</code> can help us answer such questions.</p>
<p>The examples given in <a href="https://model-checking.github.io/kani-verifier-blog/">previous blog posts</a> primarily relied on Rust’s <code class="language-plaintext highlighter-rouge">assert</code> macro to express properties for Kani.
Is there a relationship between <code class="language-plaintext highlighter-rouge">assert</code> and <code class="language-plaintext highlighter-rouge">kani::cover</code>?
The next section addresses this question.</p>
<h2 id="how-does-kanicover-relate-to-assert">How does <code class="language-plaintext highlighter-rouge">kani::cover</code> relate to <code class="language-plaintext highlighter-rouge">assert</code>?</h2>
<p>Let’s look at how Kani interprets <code class="language-plaintext highlighter-rouge">kani::cover</code> and <code class="language-plaintext highlighter-rouge">assert</code>:</p>
<ul>
<li>As explained above, for <code class="language-plaintext highlighter-rouge">kani::cover!(condition)</code>, Kani checks if there’s an execution from a given harness that satisfies <code class="language-plaintext highlighter-rouge">condition</code> — in which case, the cover property is satisfiable, or concludes that no such execution exists — in which case the cover property is unsatisfiable.</li>
<li>On the other hand, for <code class="language-plaintext highlighter-rouge">assert!(condition)</code>, Kani checks if there’s an execution from a given harness that <em>violates</em> <code class="language-plaintext highlighter-rouge">condition</code> — in which case, the assertion fails, or concludes that no such input value exists — in which case the assertion holds.</li>
</ul>
<p>These descriptions sound very similar, don’t they?
Indeed, they are just inverses of each other!
In fact, Kani more or less models <code class="language-plaintext highlighter-rouge">kani::cover!(condition)</code> as <code class="language-plaintext highlighter-rouge">assert!(!condition)</code> under the hood (with some caveats).
If it finds an execution that violates the assertion (i.e. causes <code class="language-plaintext highlighter-rouge">!condition</code> to be false), then this execution satisfies the condition, and hence the cover property.
If on the other hand, it proves that no such execution from the given harness can cause the assertion to be violated, then it has proven that the cover property is unsatisfiable in that harness!</p>
<p>Let’s go through a couple of applications of <code class="language-plaintext highlighter-rouge">kani::cover</code> to see how it can be used in practice.</p>
<h2 id="application-1-using-kanicover-to-check-code-reachability">Application 1: Using <code class="language-plaintext highlighter-rouge">kani::cover</code> to check code reachability</h2>
<p>While developing and reviewing code, we sometimes spot certain branches in the code that we’re not certain if they’re reachable.
The following code snippet gives one such example where it’s not clear whether <code class="language-plaintext highlighter-rouge">construct_ip</code> can return a localhost IPv6 address:</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">use</span> <span class="nn">std</span><span class="p">::</span><span class="nn">net</span><span class="p">::{</span><span class="n">IpAddr</span><span class="p">,</span> <span class="n">Ipv4Addr</span><span class="p">,</span> <span class="n">Ipv6Addr</span><span class="p">};</span>
<span class="k">fn</span> <span class="nf">process_ip</span><span class="p">(</span><span class="n">host_id</span><span class="p">:</span> <span class="nb">u8</span><span class="p">)</span> <span class="p">{</span>
<span class="k">let</span> <span class="n">ip</span><span class="p">:</span> <span class="n">IpAddr</span> <span class="o">=</span> <span class="nf">construct_ip</span><span class="p">(</span><span class="n">host_id</span><span class="p">);</span>
<span class="k">match</span> <span class="n">ip</span> <span class="p">{</span>
<span class="nn">IpAddr</span><span class="p">::</span><span class="nf">V4</span><span class="p">(</span><span class="n">v4_addr</span><span class="p">)</span> <span class="k">=></span> <span class="p">{</span>
<span class="k">if</span> <span class="n">v4_addr</span> <span class="o">==</span> <span class="nn">Ipv4Addr</span><span class="p">::</span><span class="n">LOCALHOST</span> <span class="p">{</span>
<span class="c">// ...</span>
<span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
<span class="c">// ...</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="nn">IpAddr</span><span class="p">::</span><span class="nf">V6</span><span class="p">(</span><span class="n">v6_addr</span><span class="p">)</span> <span class="k">=></span> <span class="p">{</span>
<span class="k">if</span> <span class="n">v6_addr</span> <span class="o">==</span> <span class="nn">Ipv6Addr</span><span class="p">::</span><span class="n">LOCALHOST</span> <span class="p">{</span>
<span class="c">// Do we expect to ever receive a localhost V6 address?</span>
<span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
<span class="c">// ...</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>If we have <a href="https://en.wikipedia.org/wiki/Code_coverage">code coverage</a> set up, we can check if there are tests that hit that line of code.
But if there aren’t, or if we don’t run code coverage, how do we determine if this condition is possible?
Kani along with <code class="language-plaintext highlighter-rouge">kani::cover</code> can help answer this question.</p>
<p>The way we go about this is to write a harness that invokes the code in question and inject a <code class="language-plaintext highlighter-rouge">kani::cover</code> call at the line of code we want to reach, e.g.:</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">use</span> <span class="nn">std</span><span class="p">::</span><span class="nn">net</span><span class="p">::{</span><span class="n">IpAddr</span><span class="p">,</span> <span class="n">Ipv4Addr</span><span class="p">,</span> <span class="n">Ipv6Addr</span><span class="p">};</span>
<span class="k">fn</span> <span class="nf">process_ip</span><span class="p">(</span><span class="n">host_id</span><span class="p">:</span> <span class="nb">u8</span><span class="p">)</span> <span class="p">{</span>
<span class="k">let</span> <span class="n">ip</span><span class="p">:</span> <span class="n">IpAddr</span> <span class="o">=</span> <span class="nf">construct_ip</span><span class="p">(</span><span class="n">host_id</span><span class="p">);</span>
<span class="k">match</span> <span class="n">ip</span> <span class="p">{</span>
<span class="nn">IpAddr</span><span class="p">::</span><span class="nf">V4</span><span class="p">(</span><span class="n">v4_addr</span><span class="p">)</span> <span class="k">=></span> <span class="p">{</span>
<span class="k">if</span> <span class="n">v4_addr</span> <span class="o">==</span> <span class="nn">Ipv4Addr</span><span class="p">::</span><span class="n">LOCALHOST</span> <span class="p">{</span>
<span class="c">// ...</span>
<span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
<span class="c">// ...</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="nn">IpAddr</span><span class="p">::</span><span class="nf">V6</span><span class="p">(</span><span class="n">v6_addr</span><span class="p">)</span> <span class="k">=></span> <span class="p">{</span>
<span class="k">if</span> <span class="n">v6_addr</span> <span class="o">==</span> <span class="nn">Ipv6Addr</span><span class="p">::</span><span class="n">LOCALHOST</span> <span class="p">{</span>
<span class="c">// Do we expect to ever receive a localhost V6 address?</span>
<span class="nn">kani</span><span class="p">::</span><span class="nd">cover!</span><span class="p">();</span> <span class="c">// <------ Check if this line is reachable</span>
<span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
<span class="c">// ...</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="nd">#[kani::proof]</span>
<span class="k">fn</span> <span class="nf">kani_cover_harness</span><span class="p">()</span> <span class="p">{</span>
<span class="k">let</span> <span class="n">host_id</span><span class="p">:</span> <span class="nb">u8</span> <span class="o">=</span> <span class="nn">kani</span><span class="p">::</span><span class="nf">any</span><span class="p">();</span>
<span class="nf">process_ip</span><span class="p">(</span><span class="n">host_id</span><span class="p">);</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Notice that we’ve used <code class="language-plaintext highlighter-rouge">kani::cover!()</code> without an argument, which is a shorthand for <code class="language-plaintext highlighter-rouge">kani::cover!(true)</code>.</p>
<p>If we run Kani on this harness, we can get one of two possible outcomes:</p>
<ol>
<li>Kani reports the cover property to be unreachable, which indicates that <code class="language-plaintext highlighter-rouge">construct_ip</code> can never return a localhost V6 address for any <code class="language-plaintext highlighter-rouge">host_id</code>.
Since our harness covers the whole input space of <code class="language-plaintext highlighter-rouge">construct_ip</code>, we can conclude that the code under this branch is dead.
In this case, one could place an <code class="language-plaintext highlighter-rouge">unreachable!()</code> call at that line to enable the compiler to optimize this branch out, or replace the entire if condition with an assertion, e.g.</li>
</ol>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="nn">IpAddr</span><span class="p">::</span><span class="nf">V6</span><span class="p">(</span><span class="n">v6_addr</span><span class="p">)</span> <span class="k">=></span> <span class="p">{</span>
<span class="nd">assert_ne!</span><span class="p">(</span><span class="n">v6_addr</span><span class="p">,</span> <span class="nn">Ipv6Addr</span><span class="p">::</span><span class="n">LOCALHOST</span><span class="p">);</span>
<span class="c">// handle non-localhost IPs</span>
<span class="c">// ...</span>
<span class="p">}</span>
</code></pre></div></div>
<ol>
<li>Kani reports the cover property to be satisfiable.
In this case, one can use Kani’s <a href="https://model-checking.github.io/kani-verifier-blog/2022/09/22/internship-projects-2022-concrete-playback.html">concrete playback</a> feature to extract a unit test that covers this line, thereby increasing code coverage.</li>
</ol>
<h2 id="application-2-using-kanicover-to-sanity-check-proof-harnesses">Application 2: Using <code class="language-plaintext highlighter-rouge">kani::cover</code> to sanity check proof harnesses</h2>
<p>A green “Verification Successful” outcome from Kani is always pleasant to see, as it indicates that all properties in a harness were proven.
However, it sometimes leaves a person with doubts on whether the proof harness they wrote is <em>vacuous</em>, e.g. does it cover the entire input space of the function under verification?
For example, for an array input, does the proof harness include the case of an empty array?
Or for an input that is a pair of integers, does the harness permit them to have the same value?</p>
<p>The <code class="language-plaintext highlighter-rouge">kani::cover</code> macro can help answer these questions.
Let’s go through an example to clarify what we mean by that.</p>
<p>Suppose we’re given a function to verify that generates arrays of a given size such that <em>every slice</em> of the array satisfies the following property: the last element of the slice is greater than or equal to the sum of all the previous elements in the slice.
For instance, the following array satisfies this property: <code class="language-plaintext highlighter-rouge">[5, 5, 20, 32]</code> because:</p>
<ul>
<li>For the <code class="language-plaintext highlighter-rouge">[5, 5, 20, 32]</code> slice, <code class="language-plaintext highlighter-rouge">5 + 5 + 20 <= 32</code>.</li>
<li>For the <code class="language-plaintext highlighter-rouge">[5, 5, 20]</code> and <code class="language-plaintext highlighter-rouge">[5, 20, 32]</code> slices , <code class="language-plaintext highlighter-rouge">5 + 5 <= 20</code> and <code class="language-plaintext highlighter-rouge">5 + 20 <= 32</code>.</li>
<li>For the <code class="language-plaintext highlighter-rouge">[5, 5]</code>, <code class="language-plaintext highlighter-rouge">[5, 20]</code>, and <code class="language-plaintext highlighter-rouge">[20, 32]</code> slices, <code class="language-plaintext highlighter-rouge">5 <= 5</code>, <code class="language-plaintext highlighter-rouge">5 <= 20</code>, and <code class="language-plaintext highlighter-rouge">20 <= 32.</code></li>
<li>The property also holds trivially for all slices of length 0 and 1.</li>
</ul>
<p>Suppose we want to verify this function with Kani.
Assuming its signature is as follows:</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">fn</span> <span class="n">generate_array_with_sum_property</span><span class="o"><</span><span class="k">const</span> <span class="n">N</span><span class="p">:</span> <span class="nb">usize</span><span class="o">></span><span class="p">(</span><span class="n">seed</span><span class="p">:</span> <span class="nb">i32</span><span class="p">)</span> <span class="k">-></span> <span class="p">[</span><span class="nb">i32</span><span class="p">;</span> <span class="n">N</span><span class="p">]</span>
</code></pre></div></div>
<p>we can write a harness for N = 5 that looks as follows:</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">#[kani::proof]</span>
<span class="nd">#[kani::unwind(</span><span class="mi">5</span><span class="nd">)]</span>
<span class="k">fn</span> <span class="nf">check_generate_array_with_sum_property</span><span class="p">()</span> <span class="p">{</span>
<span class="c">// 1. Call the function to generate an array passing it any seed value</span>
<span class="k">let</span> <span class="n">arr</span><span class="p">:</span> <span class="p">[</span><span class="nb">i32</span><span class="p">;</span> <span class="mi">5</span><span class="p">]</span> <span class="o">=</span> <span class="nf">generate_array_with_sum_property</span><span class="p">(</span><span class="nn">kani</span><span class="p">::</span><span class="nf">any</span><span class="p">());</span>
<span class="c">// 2. Create any slice of the array</span>
<span class="k">let</span> <span class="n">slice</span> <span class="o">=</span> <span class="nf">any_slice</span><span class="p">(</span><span class="o">&</span><span class="n">arr</span><span class="p">);</span>
<span class="c">// 3. Verify that the slice satisfies the sum property</span>
<span class="k">assert</span><span class="o">!</span><span class="p">(</span><span class="nf">sum_property_holds_for_slice</span><span class="p">(</span><span class="n">slice</span><span class="p">));</span>
<span class="p">}</span>
</code></pre></div></div>
<p>where <code class="language-plaintext highlighter-rouge">sum_property_holds_for_slice</code> can be implemented as follows:</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">fn</span> <span class="nf">sum_property_holds_for_slice</span><span class="p">(</span><span class="n">s</span><span class="p">:</span> <span class="o">&</span><span class="p">[</span><span class="nb">i32</span><span class="p">])</span> <span class="k">-></span> <span class="nb">bool</span> <span class="p">{</span>
<span class="c">// slices of length 0 and 1 trivially satisfy the property</span>
<span class="k">if</span> <span class="n">s</span><span class="nf">.is_empty</span><span class="p">()</span> <span class="p">||</span> <span class="n">s</span><span class="nf">.len</span><span class="p">()</span> <span class="o">==</span> <span class="mi">1</span> <span class="p">{</span>
<span class="k">return</span> <span class="k">true</span><span class="p">;</span>
<span class="p">}</span>
<span class="c">// compute the sum of all elements except for the last</span>
<span class="k">let</span> <span class="k">mut</span> <span class="n">sum</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="k">for</span> <span class="n">i</span> <span class="n">in</span> <span class="mi">0</span><span class="o">..</span><span class="n">s</span><span class="nf">.len</span><span class="p">()</span> <span class="o">-</span> <span class="mi">1</span> <span class="p">{</span>
<span class="n">sum</span> <span class="o">+=</span> <span class="n">s</span><span class="p">[</span><span class="n">i</span><span class="p">];</span>
<span class="p">}</span>
<span class="c">// return whether the sum is smaller than or equal to the last element</span>
<span class="n">sum</span> <span class="o"><=</span> <span class="o">*</span><span class="n">s</span><span class="nf">.last</span><span class="p">()</span><span class="nf">.unwrap</span><span class="p">()</span>
<span class="p">}</span>
</code></pre></div></div>
<p>The question now is: how do we implement the <code class="language-plaintext highlighter-rouge">any_slice</code> function, such that it can return any possible slice of the provided array?
Here’s one possible implementation:</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">fn</span> <span class="nf">any_slice</span><span class="p">(</span><span class="n">arr</span><span class="p">:</span> <span class="o">&</span><span class="p">[</span><span class="nb">i32</span><span class="p">])</span> <span class="k">-></span> <span class="o">&</span><span class="p">[</span><span class="nb">i32</span><span class="p">]</span> <span class="p">{</span>
<span class="k">let</span> <span class="n">start</span><span class="p">:</span> <span class="nb">usize</span> <span class="o">=</span> <span class="nn">kani</span><span class="p">::</span><span class="nf">any</span><span class="p">();</span>
<span class="k">let</span> <span class="n">end</span><span class="p">:</span> <span class="nb">usize</span> <span class="o">=</span> <span class="nn">kani</span><span class="p">::</span><span class="nf">any</span><span class="p">();</span>
<span class="nn">kani</span><span class="p">::</span><span class="nf">assume</span><span class="p">(</span><span class="n">end</span> <span class="o"><</span> <span class="n">arr</span><span class="nf">.len</span><span class="p">());</span>
<span class="nn">kani</span><span class="p">::</span><span class="nf">assume</span><span class="p">(</span><span class="n">start</span> <span class="o"><=</span> <span class="n">end</span><span class="p">);</span>
<span class="o">&</span><span class="n">arr</span><span class="p">[</span><span class="n">start</span><span class="o">..</span><span class="n">end</span><span class="p">]</span>
<span class="p">}</span>
</code></pre></div></div>
<p>The implementation assigns both the start and end of the slice to <code class="language-plaintext highlighter-rouge">kani::any()</code>.
It then applies two constraints to <code class="language-plaintext highlighter-rouge">start</code> and <code class="language-plaintext highlighter-rouge">end</code> using <code class="language-plaintext highlighter-rouge">kani::assume</code>:</p>
<ol>
<li>The first assumption restricts <code class="language-plaintext highlighter-rouge">end</code> to be less than the length of the array so that the ranges are within the array bounds.</li>
<li>The second assumption restricts <code class="language-plaintext highlighter-rouge">start</code> to be less than or equal to <code class="language-plaintext highlighter-rouge">end</code> to rule out invalid ranges.</li>
</ol>
<p>Finally, it returns the slice of the array between <code class="language-plaintext highlighter-rouge">start</code> and <code class="language-plaintext highlighter-rouge">end</code>.</p>
<p>We’re now ready to run Kani.
Suppose we do, and we get the reassuring “Verification Successful” message.
This might leave us wondering: does <code class="language-plaintext highlighter-rouge">any_slice</code> really cover all possible slices of the given array?
This is where <code class="language-plaintext highlighter-rouge">kani::cover</code> can help.</p>
<p>We can use <code class="language-plaintext highlighter-rouge">kani::cover</code> to check whether <code class="language-plaintext highlighter-rouge">any_slice</code> can generate some of the cases we expect it to.
If it can’t, that typically means that the proof harness may not be covering all the possible space of inputs that you intend it to cover.
An example of what <code class="language-plaintext highlighter-rouge">kani::cover</code> can be used for is to check whether <code class="language-plaintext highlighter-rouge">any_slice</code> <em>can</em> return an empty slice.
The updated harness is as follows:</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="k">let</span> <span class="n">arr</span><span class="p">:</span> <span class="p">[</span><span class="nb">i32</span><span class="p">;</span> <span class="mi">5</span><span class="p">]</span> <span class="o">=</span> <span class="nf">generate_array_with_sum_property</span><span class="p">(</span><span class="nn">kani</span><span class="p">::</span><span class="nf">any</span><span class="p">());</span>
<span class="k">let</span> <span class="n">slice</span> <span class="o">=</span> <span class="nf">any_slice</span><span class="p">(</span><span class="o">&</span><span class="n">arr</span><span class="p">);</span>
<span class="nn">kani</span><span class="p">::</span><span class="nd">cover!</span><span class="p">(</span><span class="n">slice</span><span class="nf">.is_empty</span><span class="p">());</span> <span class="c">// <---- Check if slice can be empty</span>
<span class="k">assert</span><span class="o">!</span><span class="p">(</span><span class="nf">sum_property_holds_for_slice</span><span class="p">(</span><span class="n">slice</span><span class="p">));</span>
</code></pre></div></div>
<p>If we run Kani, we get the following:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>SUMMARY:
** 0 of 75 failed
** 1 of 1 cover properties satisfied
VERIFICATION:- SUCCESSFUL
</code></pre></div></div>
<p>which tells us that Kani found an input value that resulted in <code class="language-plaintext highlighter-rouge">any_slice</code> returning an empty slice.
Indeed, we can easily see that <code class="language-plaintext highlighter-rouge">start == 0</code> and <code class="language-plaintext highlighter-rouge">end == 0</code> satisfy both assumptions and result in an empty slice.</p>
<p>Another interesting corner case is whether <code class="language-plaintext highlighter-rouge">any_slice</code> can return a slice that spans the entire array.
We can easily check this case using <code class="language-plaintext highlighter-rouge">kani::cover</code> as follows:</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="k">let</span> <span class="n">arr</span><span class="p">:</span> <span class="p">[</span><span class="nb">i32</span><span class="p">;</span> <span class="mi">5</span><span class="p">]</span> <span class="o">=</span> <span class="nf">generate_array_with_sum_property</span><span class="p">(</span><span class="nn">kani</span><span class="p">::</span><span class="nf">any</span><span class="p">());</span>
<span class="k">let</span> <span class="n">slice</span> <span class="o">=</span> <span class="nf">any_slice</span><span class="p">(</span><span class="o">&</span><span class="n">arr</span><span class="p">);</span>
<span class="nn">kani</span><span class="p">::</span><span class="nd">cover!</span><span class="p">(</span><span class="n">slice</span><span class="nf">.is_empty</span><span class="p">());</span>
<span class="nn">kani</span><span class="p">::</span><span class="nd">cover!</span><span class="p">(</span><span class="n">slice</span><span class="nf">.len</span><span class="p">()</span> <span class="o">==</span> <span class="n">arr</span><span class="nf">.len</span><span class="p">());</span> <span class="c">// <------ Check if slice can span entire array</span>
<span class="k">assert</span><span class="o">!</span><span class="p">(</span><span class="nf">sum_property_holds_for_slice</span><span class="p">(</span><span class="n">slice</span><span class="p">));</span>
</code></pre></div></div>
<p>If we run the updated harness with Kani, we get the following result:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> ** 1 of 2 cover properties satisfied
</code></pre></div></div>
<p>Oh, oh! Kani found the new cover property to be unsatisfiable! We can find the specific unsatisfiable cover property by looking at the results section, which will have:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Check 2: check_generate_array_with_sum_property.cover.2
- Status: UNSATISFIABLE
- Description: "cover condition: slice.len() == arr.len()"
- Location: test.rs:18:5 in function check_generate_array_with_sum_property
</code></pre></div></div>
<p>This result indicates that <code class="language-plaintext highlighter-rouge">any_slice</code> cannot return a slice with the full length of the array.
Thus, our proof harness may miss bugs in <code class="language-plaintext highlighter-rouge">generate_array_with_sum_property</code>!
Indeed, if <code class="language-plaintext highlighter-rouge">generate_array_with_sum_property</code> were to return an array that doesn’t satisfy the property, e.g. <code class="language-plaintext highlighter-rouge">[5, 8, 20, 57, 70]</code>, (due to the <code class="language-plaintext highlighter-rouge">[20, 57, 70]</code> slice), verification still succeeds!</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">#[kani::proof]</span>
<span class="nd">#[kani::unwind(</span><span class="mi">5</span><span class="nd">)]</span>
<span class="k">fn</span> <span class="nf">check_generate_array_with_sum_property</span><span class="p">()</span> <span class="p">{</span>
<span class="k">let</span> <span class="n">arr</span><span class="p">:</span> <span class="p">[</span><span class="nb">i32</span><span class="p">;</span> <span class="mi">5</span><span class="p">]</span> <span class="o">=</span> <span class="nf">generate_array_with_sum_property</span><span class="p">(</span><span class="nn">kani</span><span class="p">::</span><span class="nf">any</span><span class="p">());</span>
<span class="k">let</span> <span class="n">arr</span> <span class="o">=</span> <span class="p">[</span><span class="mi">5</span><span class="p">,</span> <span class="mi">8</span><span class="p">,</span> <span class="mi">20</span><span class="p">,</span> <span class="mi">57</span><span class="p">,</span> <span class="mi">70</span><span class="p">];</span>
<span class="k">let</span> <span class="n">slice</span> <span class="o">=</span> <span class="nf">any_slice</span><span class="p">(</span><span class="o">&</span><span class="n">arr</span><span class="p">);</span>
<span class="nn">kani</span><span class="p">::</span><span class="nd">cover!</span><span class="p">(</span><span class="n">slice</span><span class="nf">.is_empty</span><span class="p">());</span>
<span class="nn">kani</span><span class="p">::</span><span class="nd">cover!</span><span class="p">(</span><span class="n">slice</span><span class="nf">.len</span><span class="p">()</span> <span class="o">==</span> <span class="n">arr</span><span class="nf">.len</span><span class="p">());</span>
<span class="k">assert</span><span class="o">!</span><span class="p">(</span><span class="nf">sum_property_holds_for_slice</span><span class="p">(</span><span class="n">slice</span><span class="p">));</span>
<span class="p">}</span>
</code></pre></div></div>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
SUMMARY:
** 0 of 75 failed (1 unreachable)
** 1 of 2 cover properties satisfied
VERIFICATION:- SUCCESSFUL
</code></pre></div></div>
<p>How did this happen?
The bug is in <code class="language-plaintext highlighter-rouge">any_slice</code>, specifically due to those two lines:</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="nn">kani</span><span class="p">::</span><span class="nf">assume</span><span class="p">(</span><span class="n">end</span> <span class="o"><</span> <span class="n">arr</span><span class="nf">.len</span><span class="p">());</span>
<span class="o">&</span><span class="n">arr</span><span class="p">[</span><span class="n">start</span><span class="o">..</span><span class="n">end</span><span class="p">]</span>
</code></pre></div></div>
<p>The assumption is intended to guarantee that the range falls within the array.
However, the function uses the “half-open” <a href="https://doc.rust-lang.org/std/ops/struct.Range.html">range</a>, which excludes the <code class="language-plaintext highlighter-rouge">end</code>, thus, the returned slice can never include the last element of the array!
This is an example of <em>overconstraining</em>, where the harness rules out inputs of interest (in this case slices that span the entire array), causing the proof to be incomplete.</p>
<p>There are two ways to fix <code class="language-plaintext highlighter-rouge">any_slice</code>.
We can either relax the assumption to use “less than or equal” instead of the strict “less than”, or we can use the <a href="https://doc.rust-lang.org/std/ops/struct.RangeInclusive.html">inclusive range</a>.
Let’s do the latter.
The updated function is as follows:</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">fn</span> <span class="nf">any_slice</span><span class="p">(</span><span class="n">arr</span><span class="p">:</span> <span class="o">&</span><span class="p">[</span><span class="nb">i32</span><span class="p">])</span> <span class="k">-></span> <span class="o">&</span><span class="p">[</span><span class="nb">i32</span><span class="p">]</span> <span class="p">{</span>
<span class="k">let</span> <span class="n">start</span><span class="p">:</span> <span class="nb">usize</span> <span class="o">=</span> <span class="nn">kani</span><span class="p">::</span><span class="nf">any</span><span class="p">();</span>
<span class="k">let</span> <span class="n">end</span><span class="p">:</span> <span class="nb">usize</span> <span class="o">=</span> <span class="nn">kani</span><span class="p">::</span><span class="nf">any</span><span class="p">();</span>
<span class="nn">kani</span><span class="p">::</span><span class="nf">assume</span><span class="p">(</span><span class="n">end</span> <span class="o"><</span> <span class="n">arr</span><span class="nf">.len</span><span class="p">());</span>
<span class="nn">kani</span><span class="p">::</span><span class="nf">assume</span><span class="p">(</span><span class="n">start</span> <span class="o"><=</span> <span class="n">end</span><span class="p">);</span>
<span class="o">&</span><span class="n">arr</span><span class="p">[</span><span class="n">start</span><span class="o">..=</span><span class="n">end</span><span class="p">]</span> <span class="c">// <-------- Notice the new equal sign before `end`</span>
<span class="p">}</span>
</code></pre></div></div>
<p>If we rerun Kani, we get the following result:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> ** 0 of 83 failed (1 unreachable)
** 1 of 2 cover properties satisfied
VERIFICATION:- SUCCESSFUL
</code></pre></div></div>
<p>What?! Why is a cover property still unsatisfiable, and which one is it?
The detailed results section indicates that this time, the <code class="language-plaintext highlighter-rouge">slice.is_empty()</code> condition is the one that is unsatisfiable:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>RESULTS:
Check 1: check_generate_array_with_sum_property.cover.1
- Status: UNSATISFIABLE
- Description: "cover condition: slice.is_empty()"
- Location: blog_example.rs:17:5 in function check_generate_array_with_sum_property
Check 2: check_generate_array_with_sum_property.cover.2
- Status: SATISFIED
- Description: "cover condition: slice.len() == arr.len()"
- Location: blog_example.rs:18:5 in function check_generate_array_with_sum_property
</code></pre></div></div>
<p>Indeed, since we switched to using the inclusive range (<code class="language-plaintext highlighter-rouge">start..=end</code>), it is no longer possible to return an empty slice, this time because of the second assumption, <code class="language-plaintext highlighter-rouge">kani::assume(start <= end)</code> since an empty slice using the inclusive range is only possible if <code class="language-plaintext highlighter-rouge">start > end</code>.
So this time, the first cover property we wrote saved us!
We can fix this by relaxing the assumption.
The updated <code class="language-plaintext highlighter-rouge">any_slice</code> becomes:</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">fn</span> <span class="nf">any_slice</span><span class="p">(</span><span class="n">arr</span><span class="p">:</span> <span class="o">&</span><span class="p">[</span><span class="nb">i32</span><span class="p">])</span> <span class="k">-></span> <span class="o">&</span><span class="p">[</span><span class="nb">i32</span><span class="p">]</span> <span class="p">{</span>
<span class="k">let</span> <span class="n">start</span><span class="p">:</span> <span class="nb">usize</span> <span class="o">=</span> <span class="nn">kani</span><span class="p">::</span><span class="nf">any</span><span class="p">();</span>
<span class="k">let</span> <span class="n">end</span><span class="p">:</span> <span class="nb">usize</span> <span class="o">=</span> <span class="nn">kani</span><span class="p">::</span><span class="nf">any</span><span class="p">();</span>
<span class="nn">kani</span><span class="p">::</span><span class="nf">assume</span><span class="p">(</span><span class="n">end</span> <span class="o"><</span> <span class="n">arr</span><span class="nf">.len</span><span class="p">());</span>
<span class="nn">kani</span><span class="p">::</span><span class="nf">assume</span><span class="p">(</span><span class="n">start</span> <span class="o"><=</span> <span class="n">end</span> <span class="o">+</span> <span class="mi">1</span><span class="p">);</span>
<span class="o">&</span><span class="n">arr</span><span class="p">[</span><span class="n">start</span><span class="o">..=</span><span class="n">end</span><span class="p">]</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Alternatively, we can revert to using the half-open range, and relax the first assumption as follows:</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">fn</span> <span class="nf">any_slice</span><span class="p">(</span><span class="n">arr</span><span class="p">:</span> <span class="o">&</span><span class="p">[</span><span class="nb">i32</span><span class="p">])</span> <span class="k">-></span> <span class="o">&</span><span class="p">[</span><span class="nb">i32</span><span class="p">]</span> <span class="p">{</span>
<span class="k">let</span> <span class="n">start</span><span class="p">:</span> <span class="nb">usize</span> <span class="o">=</span> <span class="nn">kani</span><span class="p">::</span><span class="nf">any</span><span class="p">();</span>
<span class="k">let</span> <span class="n">end</span><span class="p">:</span> <span class="nb">usize</span> <span class="o">=</span> <span class="nn">kani</span><span class="p">::</span><span class="nf">any</span><span class="p">();</span>
<span class="nn">kani</span><span class="p">::</span><span class="nf">assume</span><span class="p">(</span><span class="n">end</span> <span class="o"><=</span> <span class="n">arr</span><span class="nf">.len</span><span class="p">());</span> <span class="c">// <-------- Use less than or equal</span>
<span class="nn">kani</span><span class="p">::</span><span class="nf">assume</span><span class="p">(</span><span class="n">start</span> <span class="o"><=</span> <span class="n">end</span><span class="p">);</span>
<span class="o">&</span><span class="n">arr</span><span class="p">[</span><span class="n">start</span><span class="o">..</span><span class="n">end</span><span class="p">]</span>
<span class="p">}</span>
</code></pre></div></div>
<p>With either of the last two <code class="language-plaintext highlighter-rouge">any_slice</code> implementations, if we rerun Kani, both cover properties are satisfied:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> ** 2 of 2 cover properties satisfied
</code></pre></div></div>
<p>and verification is successful.
Also, if we use an array the doesn’t satisfy the property (e.g. <code class="language-plaintext highlighter-rouge">[5, 8, 20, 57, 70]</code>), verification will fail as expected:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> Failed Checks: assertion failed: sum_property_holds_for_slice(slice)
File: "/home/ubuntu/examples/cover/blog_example.rs", line 20, in check_generate_array_with_sum_property
</code></pre></div></div>
<p>We can easily add more cover properties that check for other cases of interest, e.g. all possible slice lengths between 0 and 5.</p>
<h2 id="summary">Summary</h2>
<p>In this post, we discussed two applications of the <code class="language-plaintext highlighter-rouge">kani::cover</code> macro. We’d like to hear your feedback on it, and whether there are related features you would like to see implemented to allow you to make good use of <code class="language-plaintext highlighter-rouge">kani::cover</code>.</p>
<h2 id="references">References:</h2>
<ol>
<li>Cover Statement RFC: <a href="https://model-checking.github.io/kani/rfc/rfcs/0003-cover-statement.html">https://model-checking.github.io/kani/rfc/rfcs/0003-cover-statement.html</a></li>
<li>Documentation for <code class="language-plaintext highlighter-rouge">kani::cover</code>: <a href="https://model-checking.github.io/kani/crates/doc/kani/macro.cover.html">https://model-checking.github.io/kani/crates/doc/kani/macro.cover.html</a></li>
</ol>Kani is a verification tool that can help you prove properties about your Rust code. To learn more about Kani, check out the Kani tutorial and our previous blog posts.Easily verify your Rust in CI with Kani and Github Actions2022-12-21T00:00:00+00:002022-12-21T00:00:00+00:00https://model-checking.github.io//kani-verifier-blog/2022/12/21/easily-verify-your-rust-in-ci-with-kani<p>Formal verification tools like Kani check that certain classes of bugs will not occur in your program under any circumstance.
Kani uses mathematical techniques to explore your code over all inputs, meaning that while testing is a great way to catch the <em>first</em> bug, formally verifying your code with Kani is a great way to validate that you’ve caught the <em>last</em> bug.
To learn more about Kani, and how you can use it to prevent bugs in your Rust code, you can <a href="https://model-checking.github.io/kani/">read the Kani documentation</a>, or <a href="https://model-checking.github.io/kani-verifier-blog">consult the real-world examples on our blog</a>.</p>
<p>Code, however, is seldom static.
As code evolves, how can we ensure that it remains correct; i.e. how do we catch the <em>next</em> bug?
Many codebases already contain test suites and other checks that can run automatically.
Developers can use a platform like <a href="https://github.com/features/actions">GitHub Actions</a> to run these tests upon every code change, as part of continuous integration, and even <a href="https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/defining-the-mergeability-of-pull-requests/about-protected-branches#require-status-checks-before-merging">ensure that the tests pass before the code can be merged</a>.
This assures developers that a code change does not introduce a bug for inputs covered by the test suite.</p>
<p>For developers looking for a higher assurance of software quality, Kani proofs can be validated automatically as part of continuous integration.
To this end, we are excited to announce the <a href="https://github.com/marketplace/actions/kani-rust-verifier">Kani Rust Verifier Action</a> on the GitHub Marketplace.</p>
<h2 id="how-to-use-the-kani-github-action">How to use the Kani GitHub action</h2>
<p>If you have a <a href="https://doc.rust-lang.org/cargo/">Rust Cargo</a> project in a GitHub repo, using the Kani CI action is as simple as adding it to your GitHub Actions workflow file.
Your Kani checks will appear as part of the same workflow as your existing tests, as shown in this <a href="https://github.com/aws/s2n-quic">s2n-quic</a> CI report:</p>
<p><img src="https://model-checking.github.io/kani-verifier-blog/assets/images/kani-in-s2n-quic-ci.png" alt="s2n-quic CI report showing both tests and Kani proofs verified on a pull request" /></p>
<p>To get the latest version, visit the <a href="https://github.com/marketplace/actions/kani-rust-verifier">Kani Verifier action</a> on the marketplace, and click on the green “Use Latest Version” button, which will give you a yaml snippet you can paste into your <code class="language-plaintext highlighter-rouge">.yml</code> file.
<img src="https://model-checking.github.io/kani-verifier-blog/assets/images/kani-verifier-action.png" alt="A list of tests and checks run on a pull request as part of continuous integration, with all checks required and passing" /></p>
<p>For example, for Kani 0.17, your <code class="language-plaintext highlighter-rouge">kani-ci.yml</code> file might look like this:</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">name</span><span class="pi">:</span> <span class="s">Kani CI</span>
<span class="na">on</span><span class="pi">:</span>
<span class="na">pull_request</span><span class="pi">:</span>
<span class="na">push</span><span class="pi">:</span>
<span class="na">jobs</span><span class="pi">:</span>
<span class="na">run-kani</span><span class="pi">:</span>
<span class="na">runs-on</span><span class="pi">:</span> <span class="s">ubuntu-20.04</span>
<span class="na">steps</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s1">'</span><span class="s">Checkout</span><span class="nv"> </span><span class="s">your</span><span class="nv"> </span><span class="s">code.'</span>
<span class="na">uses</span><span class="pi">:</span> <span class="s">actions/checkout@v3</span>
<span class="c1"># You can get the latest version from</span>
<span class="c1"># https://github.com/marketplace/actions/kani-rust-verifier</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s1">'</span><span class="s">Run</span><span class="nv"> </span><span class="s">Kani</span><span class="nv"> </span><span class="s">on</span><span class="nv"> </span><span class="s">your</span><span class="nv"> </span><span class="s">code.'</span>
<span class="na">uses</span><span class="pi">:</span> <span class="s">model-checking/kani-github-action@v0.17</span>
</code></pre></div></div>
<p>For more advanced use cases, we provide facilities to override the working directory, as well as to configure the Kani command itself.
For example, the <a href="https://github.com/aws/s2n-quic">s2n-quic</a> project uses <a href="https://github.com/aws/s2n-quic/blob/c221530beb37addf81c7ed58461a945750a1b251/.github/workflows/ci.yml#L613">the following CI configuration</a>, which overrides the working directory, and enables the <code class="language-plaintext highlighter-rouge">--tests</code> option of <code class="language-plaintext highlighter-rouge">cargo kani</code>.</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">on</span><span class="pi">:</span>
<span class="na">push</span><span class="pi">:</span>
<span class="na">branches</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">main</span>
<span class="na">pull_request</span><span class="pi">:</span>
<span class="na">branches</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">main</span>
<span class="c1"># ...</span>
<span class="na">jobs</span><span class="pi">:</span>
<span class="na">kani</span><span class="pi">:</span>
<span class="na">runs-on</span><span class="pi">:</span> <span class="s">ubuntu-latest</span>
<span class="na">steps</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">uses</span><span class="pi">:</span> <span class="s">actions/checkout@v3</span>
<span class="na">with</span><span class="pi">:</span>
<span class="na">submodules</span><span class="pi">:</span> <span class="no">true</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Kani run</span>
<span class="na">uses</span><span class="pi">:</span> <span class="s">model-checking/kani-github-action@v0.17</span>
<span class="na">with</span><span class="pi">:</span>
<span class="na">working-directory</span><span class="pi">:</span> <span class="s">quic/s2n-quic-core</span>
<span class="na">args</span><span class="pi">:</span> <span class="s">--tests</span>
</code></pre></div></div>
<p>For full details, consult <a href="https://model-checking.github.io/kani/install-github-ci.html">the documentation</a>.</p>
<h2 id="when-to-run-the-kani-action">When to run the Kani action</h2>
<p>CI systems such as GitHub Actions offer multiple options for when workflows, such as the Kani action, should execute.
Options include validating every push to every PR, validating once on a PR right before merge, or validating on a fixed time interval, such as nightly.
In general, the sooner verification runs, the better: it’s easier to debug and fix issues when they’re found right away, rather than after they’re merged.</p>
<p>Our experience is that many Kani proofs can complete quickly on standard CI hardware.
Proofs that run in a similar time-frame to existing tests make sense to run on a similar cadence, e.g.
if you run your unit-tests on every code-push, then validate your Kani proofs on every code-push as well.</p>
<p>Some proofs, on the other hand, may be memory and compute intensive.
In these cases, your organization may prefer nightly jobs to reduce cost and avoid introducing latency to the PR process.
Our suggestion is to start with running proofs on every pull request, but start moving some of them to a separate nightly run if they begin to take too much time.</p>
<h2 id="caveats">Caveats</h2>
<p>The Kani GitHub Action executes on standard GitHub Action Runners; these machines <a href="https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners#supported-runners-and-hardware-resources">provide 7GB of RAM and 2 processing cores</a>.
Formal verification can be memory and compute intensive: if you find the Kani action running out of memory and CPU, we suggest using <a href="https://docs.github.com/en/actions/using-github-hosted-runners/using-larger-runners">large runners</a> or <a href="https://aws.amazon.com/codebuild/">AWS CodeBuild</a>.</p>
<p>To test drive Kani yourself, check out our <a href="https://model-checking.github.io/kani/getting-started.html">“getting started” guide</a>.
We have a one-step install process and examples, so you can start proving your code today.
If you are running into issues with Kani or have feature requests or suggestions, we’d <a href="https://github.com/model-checking/kani/issues">love to hear from you</a>.</p>Formal verification tools like Kani check that certain classes of bugs will not occur in your program under any circumstance. Kani uses mathematical techniques to explore your code over all inputs, meaning that while testing is a great way to catch the first bug, formally verifying your code with Kani is a great way to validate that you’ve caught the last bug. To learn more about Kani, and how you can use it to prevent bugs in your Rust code, you can read the Kani documentation, or consult the real-world examples on our blog.