Jekyll2022-08-28T12:14:22+00:00https://www.keiruaprod.fr/blog/feed.xmlKeiruaProdTechnical blog related to keiruaprod.fr Freelance web development, Ruby On Rails, Python, Rust, Remote workAméliorer les perfs d’un programme avec Rust.2022-08-28T00:00:00+00:002022-08-28T00:00:00+00:00https://www.keiruaprod.fr/blog/2022/08/28/amelioration-de-perfs-rust<p>J’ai optimisé un bout de code avec Rust et j’ai gagné un facteur ~670 dans les performances.</p>
<ul>
<li>la version initiale en python prend 30s pour 1 million d’itérations</li>
<li>la version optimisée en Rust prend 0.4s pour 10 million d’itérations</li>
</ul>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Summary
<span class="s1">'./target/release/kero_fold_par_iter 1000000'</span> ran
7.01 ± 0.28 <span class="nb">times </span>faster than <span class="s1">'./target/release/kero_simple_port 1000000'</span>
667.27 ± 25.77 <span class="nb">times </span>faster than <span class="s1">'python main.py'</span>
</code></pre></div></div>
<p>Nous allons voir dans cet article comment j’ai optimisé le code, en pilotant les améliorations grâce à des outils.</p>
<p>J’ai fait plusieurs choses:</p>
<ul>
<li>écriture d’une première version Python</li>
<li>réécrire le code en Rust</li>
<li>benchmarker avec Criterion</li>
<li>paralléliser avec Rayon</li>
<li>micro-optimiser avec perf+flamegraph</li>
<li>visualisation des résultats avec <code class="highlighter-rouge">plotters</code></li>
</ul>
<h1 id="le-problème-et-première-version-en-python">Le problème et première version en Python</h1>
<p>Mon problème : calculer des probabilités pour un bout du jeu de plateau <code class="highlighter-rouge">Kero</code>.
Pour cela, j’ai écrit une simulation de Monte Carlo.</p>
<p>Le principe est simple:</p>
<ul>
<li>on simule plein de fois un processus aléatoire</li>
<li>la loi des grands nombres nous dit en gros que si on fait suffisament de simulations, la distribution des résultats devrait être représentative du résultat analytique.</li>
</ul>
<p>Concrètement, si on prend l’exemple d’un dé à 6 faces: Si vous lancez beaucoup de fois un dé à 6 faces (disons, un million), vous devriez avoir à peu prêt autant de fois des 1, des 2… que des 6. De cette distribution des résultats, on peut déduire des probabilités, par exemple celle d’obtenir un dé de valeur 6.</p>
<p>Dans cet exemple très simple on pourrait calculer le résultat analytique facilement, mais il y a des cas où ca n’est pas possible, ou bien où il est plus simple de faire des estimations. Exemple pour un problème précédent: https://www.keiruaprod.fr/blog/2020/12/10/better-odds-against-my-3yo-kid.html</p>
<p>J’ai donc écrit une première version:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">simulate_refill</span><span class="p">(</span><span class="n">nb_dices</span><span class="o">=</span><span class="mi">8</span><span class="p">):</span>
<span class="c1"># on simule des lancers de dés, le code est peu important
</span> <span class="c1"># …
</span> <span class="k">return</span> <span class="n">nb_rolls</span>
<span class="n">dices_rolls</span> <span class="o">=</span> <span class="n">defaultdict</span><span class="p">(</span><span class="nb">int</span><span class="p">)</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">1_000_000</span><span class="p">):</span>
<span class="n">nb_rolls</span> <span class="o">=</span> <span class="n">simulate_refill</span><span class="p">()</span>
<span class="n">dices_rolls</span><span class="p">[</span><span class="n">nb_rolls</span><span class="p">]</span> <span class="o">+=</span> <span class="mi">1</span>
</code></pre></div></div>
<p>Cette première version prend ~30s:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span><span class="nb">time </span>python main.py
real 0m29,254s
user 0m29,219s
sys 0m0,031s
</code></pre></div></div>
<p>À noter que <code class="highlighter-rouge">time</code> est une très mauvaise manière de calculer les performances, mais ça suffit à avoir un ordre d’idée au départ.</p>
<p>À noter aussi qu’à ce stade j’avais la réponse à ma question sur Kero, mais j’étais curieux de voir jusqu’où Rust pouvait raisonnablement nous amener.</p>
<h2 id="réécriture-en-rust">Réécriture en Rust</h2>
<p>J’ai réécrit la boucle principale en Rust, en stockant les résultats dans une <code class="highlighter-rouge">HashMap</code> de la librairie standard. Le code est très proche, il faut inclure la crate <code class="highlighter-rouge">rand</code> et implémenter <code class="highlighter-rouge">simulate_refill</code> mais ce n’est pas le sujet de l’article.</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">pub</span> <span class="k">fn</span> <span class="nf">generate_rolls_counts</span><span class="p">(</span><span class="n">nb_simulations</span><span class="p">:</span> <span class="nb">u32</span><span class="p">)</span> <span class="k">-></span> <span class="n">HashMap</span><span class="o"><</span><span class="nb">u32</span><span class="p">,</span> <span class="nb">u32</span><span class="o">></span> <span class="p">{</span>
<span class="k">let</span> <span class="k">mut</span> <span class="n">rng</span> <span class="o">=</span> <span class="nn">rand</span><span class="p">::</span><span class="nf">thread_rng</span><span class="p">();</span>
<span class="k">let</span> <span class="k">mut</span> <span class="n">dice_rolls</span><span class="p">:</span> <span class="n">HashMap</span><span class="o"><</span><span class="nb">u32</span><span class="p">,</span> <span class="nb">u32</span><span class="o">></span> <span class="o">=</span> <span class="nn">HashMap</span><span class="p">::</span><span class="nf">new</span><span class="p">();</span>
<span class="k">for</span> <span class="mi">_</span> <span class="n">in</span> <span class="mi">0</span><span class="o">..</span><span class="n">nb_simulations</span> <span class="p">{</span>
<span class="k">let</span> <span class="n">rolls</span> <span class="o">=</span> <span class="nf">simulate_refill</span><span class="p">(</span><span class="o">&</span><span class="k">mut</span> <span class="n">rng</span><span class="p">,</span> <span class="mi">8</span><span class="p">);</span>
<span class="o">*</span><span class="n">dice_rolls</span><span class="nf">.entry</span><span class="p">(</span><span class="n">rolls</span><span class="p">)</span><span class="nf">.or_insert</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="o">+=</span> <span class="mi">1</span><span class="p">;</span>
<span class="p">}</span>
<span class="n">dice_rolls</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Grossièrement avec <code class="highlighter-rouge">time</code>, on est déjà plus rapide d’un facteur 100.</p>
<p>On peut aussi écrire cette fonction différement, avec une approche fonctionnelle en utilisant <a href="https://doc.rust-lang.org/std/iter/trait.Iterator.html#method.fold"><code class="highlighter-rouge">fold</code></a> de la librairie standard.</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">pub</span> <span class="k">fn</span> <span class="nf">generate_rolls_counts_fold</span><span class="p">(</span><span class="n">nb_simulations</span><span class="p">:</span> <span class="nb">u32</span><span class="p">)</span> <span class="k">-></span> <span class="n">HashMap</span><span class="o"><</span><span class="nb">u32</span><span class="p">,</span> <span class="nb">u32</span><span class="o">></span> <span class="p">{</span>
<span class="k">let</span> <span class="k">mut</span> <span class="n">rng</span> <span class="o">=</span> <span class="nn">rand</span><span class="p">::</span><span class="nf">thread_rng</span><span class="p">();</span>
<span class="k">let</span> <span class="n">dice_rolls</span><span class="p">:</span> <span class="n">HashMap</span><span class="o"><</span><span class="nb">u32</span><span class="p">,</span> <span class="nb">u32</span><span class="o">></span> <span class="o">=</span> <span class="p">(</span><span class="mi">0</span><span class="o">..</span><span class="n">nb_simulations</span><span class="p">)</span>
<span class="nf">.into_iter</span><span class="p">()</span>
<span class="nf">.fold</span><span class="p">(</span><span class="nn">HashMap</span><span class="p">::</span><span class="nf">new</span><span class="p">(),</span> <span class="p">|</span><span class="k">mut</span> <span class="n">acc</span><span class="p">,</span> <span class="mi">_</span><span class="p">|{</span>
<span class="k">let</span> <span class="n">rolls</span> <span class="o">=</span> <span class="nf">simulate_refill</span><span class="p">(</span><span class="o">&</span><span class="k">mut</span> <span class="n">rng</span><span class="p">,</span> <span class="mi">8</span><span class="p">);</span>
<span class="o">*</span><span class="n">acc</span><span class="nf">.entry</span><span class="p">(</span><span class="n">rolls</span><span class="p">)</span><span class="nf">.or_insert</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="o">+=</span> <span class="mi">1</span><span class="p">;</span>
<span class="n">acc</span>
<span class="p">});</span>
<span class="n">dice_rolls</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Le résulat est le même, le style est différent (itératif vs fonctionnel), quelle est la fonction la plus rapide ?</p>
<p>Pour y répondre correctement, il faut benchmarker.</p>
<h2 id="benchmarking">Benchmarking</h2>
<p>Il y a plusieurs problèmes au fait de faire des benchmarks naifs avec <code class="highlighter-rouge">time</code>: la gestion du cache (dans quel état est-il? chaud, pas chaud?), la precision des résulats (la seconde), le nombre de résultats (1 seul).</p>
<p>Une solution plus pérenne c’est d’utiliser un outil dédié. Rust dispose d’une <code class="highlighter-rouge">crate</code>, <a href="https://bheisler.github.io/criterion.rs">criterion</a>, qui sert à ça. <a href="https://bheisler.github.io/criterion.rs/book/getting_started.html">La doc</a> est très bonne. Une des fonctionnalités: <a href="https://bheisler.github.io/criterion.rs/book/user_guide/comparing_functions.html">comparer deux implémentations</a>.</p>
<p>Je passe l’installation et la configuration, venons en directement au fichier de benchmark:</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">// benches/simulation.rs</span>
<span class="k">use</span> <span class="nn">criterion</span><span class="p">::{</span><span class="n">criterion_group</span><span class="p">,</span> <span class="n">criterion_main</span><span class="p">,</span> <span class="n">Criterion</span><span class="p">,</span> <span class="n">BenchmarkId</span><span class="p">};</span>
<span class="k">use</span> <span class="nn">kero</span><span class="p">::</span><span class="o">*</span><span class="p">;</span>
<span class="k">fn</span> <span class="nf">bench_generate_rolls</span><span class="p">(</span><span class="n">c</span><span class="p">:</span> <span class="o">&</span><span class="k">mut</span> <span class="n">Criterion</span><span class="p">)</span> <span class="p">{</span>
<span class="k">let</span> <span class="k">mut</span> <span class="n">group</span> <span class="o">=</span> <span class="n">c</span><span class="nf">.benchmark_group</span><span class="p">(</span><span class="s">"Generate rolls"</span><span class="p">);</span>
<span class="k">for</span> <span class="n">i</span> <span class="n">in</span> <span class="p">[</span><span class="mi">100_000</span><span class="p">]</span><span class="nf">.iter</span><span class="p">()</span> <span class="p">{</span>
<span class="n">group</span><span class="nf">.bench_with_input</span><span class="p">(</span><span class="nn">BenchmarkId</span><span class="p">::</span><span class="nf">new</span><span class="p">(</span><span class="s">"Linear"</span><span class="p">,</span> <span class="n">i</span><span class="p">),</span> <span class="n">i</span><span class="p">,</span>
<span class="p">|</span><span class="n">b</span><span class="p">,</span> <span class="n">i</span><span class="p">|</span> <span class="n">b</span><span class="nf">.iter</span><span class="p">(||</span> <span class="nf">generate_rolls_counts</span><span class="p">(</span><span class="o">*</span><span class="n">i</span><span class="p">)));</span>
<span class="n">group</span><span class="nf">.bench_with_input</span><span class="p">(</span><span class="nn">BenchmarkId</span><span class="p">::</span><span class="nf">new</span><span class="p">(</span><span class="s">"Fold"</span><span class="p">,</span> <span class="n">i</span><span class="p">),</span> <span class="n">i</span><span class="p">,</span>
<span class="p">|</span><span class="n">b</span><span class="p">,</span> <span class="n">i</span><span class="p">|</span> <span class="n">b</span><span class="nf">.iter</span><span class="p">(||</span> <span class="nf">generate_rolls_counts_fold</span><span class="p">(</span><span class="o">*</span><span class="n">i</span><span class="p">)));</span>
<span class="p">}</span>
<span class="n">group</span><span class="nf">.finish</span><span class="p">();</span>
<span class="p">}</span>
<span class="nd">criterion_group!</span><span class="p">(</span><span class="n">benches</span><span class="p">,</span> <span class="n">bench_generate_rolls</span><span class="p">);</span>
<span class="nd">criterion_main!</span><span class="p">(</span><span class="n">benches</span><span class="p">);</span>
</code></pre></div></div>
<p>Avec ca, un <code class="highlighter-rouge">cargo bench</code> nous permet</p>
<h2 id="parallèliser">Parallèliser</h2>
<p>Le problème de nos deux versions, c’est que tout se fait dans un même thread, séquentiellement.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>cargo add rayon
Updating <span class="s1">'https://github.com/rust-lang/crates.io-index'</span> index
Adding rayon v1.5.3 to dependencies
</code></pre></div></div>
<p>On ne peut pas simplement remplacer <code class="highlighter-rouge">iter</code> par <code class="highlighter-rouge">par_iter</code> comme c’est parfois le cas : chaque thread se retrouverait à écrire dans la même HashMap, ce qui pose des problèmes de concurrence d’accès.</p>
<p>Une solution: chaque thread va créer une Hashmap, puis elles vont être combinées avec <code class="highlighter-rouge">reduce</code> pour retourner une unique Hashmap.</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">pub</span> <span class="k">fn</span> <span class="nf">generate_rolls_counts_fold_par_iter</span><span class="p">(</span><span class="n">nb_simulations</span><span class="p">:</span> <span class="nb">u32</span><span class="p">)</span> <span class="k">-></span> <span class="n">HashMap</span><span class="o"><</span><span class="nb">u32</span><span class="p">,</span> <span class="nb">u32</span><span class="o">></span> <span class="p">{</span>
<span class="k">let</span> <span class="n">dice_rolls</span><span class="p">:</span> <span class="n">HashMap</span><span class="o"><</span><span class="nb">u32</span><span class="p">,</span> <span class="nb">u32</span><span class="o">></span> <span class="o">=</span> <span class="p">(</span><span class="mi">0</span><span class="o">..</span><span class="n">nb_simulations</span><span class="p">)</span>
<span class="nf">.into_par_iter</span><span class="p">()</span>
<span class="nf">.fold</span><span class="p">(</span><span class="nn">HashMap</span><span class="p">::</span><span class="n">new</span><span class="p">,|</span><span class="k">mut</span> <span class="n">acc</span><span class="p">,</span> <span class="mi">_</span><span class="p">|{</span>
<span class="k">let</span> <span class="k">mut</span> <span class="n">rng</span> <span class="o">=</span> <span class="nn">rand</span><span class="p">::</span><span class="nf">thread_rng</span><span class="p">();</span>
<span class="k">let</span> <span class="n">rolls</span> <span class="o">=</span> <span class="nf">simulate_refill</span><span class="p">(</span><span class="o">&</span><span class="k">mut</span> <span class="n">rng</span><span class="p">,</span> <span class="mi">8</span><span class="p">);</span>
<span class="o">*</span><span class="n">acc</span><span class="nf">.entry</span><span class="p">(</span><span class="n">rolls</span><span class="p">)</span><span class="nf">.or_insert</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="o">+=</span> <span class="mi">1</span><span class="p">;</span>
<span class="n">acc</span>
<span class="p">})</span>
<span class="nf">.reduce_with</span><span class="p">(|</span><span class="k">mut</span> <span class="n">m1</span><span class="p">,</span> <span class="n">m2</span><span class="p">|</span> <span class="p">{</span>
<span class="k">for</span> <span class="p">(</span><span class="n">k</span><span class="p">,</span> <span class="n">v</span><span class="p">)</span> <span class="n">in</span> <span class="n">m2</span> <span class="p">{</span>
<span class="o">*</span><span class="n">m1</span><span class="nf">.entry</span><span class="p">(</span><span class="n">k</span><span class="p">)</span><span class="nf">.or_default</span><span class="p">()</span> <span class="o">+=</span> <span class="n">v</span><span class="p">;</span>
<span class="p">}</span>
<span class="n">m1</span>
<span class="p">})</span>
<span class="nf">.unwrap</span><span class="p">();</span>
<span class="n">dice_rolls</span>
<span class="p">}</span>
</code></pre></div></div>
<h2 id="flamegraph">flamegraph</h2>
<p>Si on veut aller plus loin dans le détail, on peut utiliser un
<a href="https://github.com/flamegraph-rs/flamegraph">flamegraph</a>.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>cargo <span class="nb">install </span>flamegraph
cargo flamegraph <span class="nt">--bin</span> kero_fold_par_iter
</code></pre></div></div>
<p>Personnellement à ce stade j’étais déjà content.</p>
<h2 id="benchmarking-des-binaires-avec-hyperfine">Benchmarking des binaires avec hyperfine</h2>
<p><a href="https://github.com/sharkdp/hyperfine">hyperfine</a> propose une autre approche du benchmarking.</p>
<p>Dans des scénarios plus compliqués, il peut être pertinent de faire les benchmarks sur les binaires complets, et pas juste sur les fonctions de la librairie. Cela peut permettre d’avoir des tests plus réalistes, en comparant les temps d’exécution de deux versions. Ca permet plusieurs choses, comme comparer des programmes dans différents langages. Il est aussi possible que l’on ait pas accès au code source de l’autre version.</p>
<p>Ici, cela nous permet de comparer la version initiale en python avec différentes améliorations. On peut voir que le gain est d’environ un facteur 667 entre la version initiale et une version parallélisée en Rust.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>cargo build <span class="nt">--release</span>
<span class="nv">$ </span>hyperfine <span class="s2">"python main.py 1000000"</span> <span class="s2">"./target/release/kero_simple_port 1000000"</span> <span class="s2">"./target/release/kero_fold_par_iter 1000000"</span>
Benchmark 1: python main.py
⠙ Current estimate: 33.236 s ████████████████████████████████████████████████████████████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ Time <span class="o">(</span>mean ± σ<span class="o">)</span>: 33.010 s ± 0.764 s <span class="o">[</span>User: 32.990 s, System: 0.018 s]
Range <span class="o">(</span>min … max<span class="o">)</span>: 31.800 s … 34.150 s 10 runs
Benchmark 2: ./target/release/kero_simple_port 1000000
Time <span class="o">(</span>mean ± σ<span class="o">)</span>: 346.8 ms ± 8.8 ms <span class="o">[</span>User: 340.7 ms, System: 6.1 ms]
Range <span class="o">(</span>min … max<span class="o">)</span>: 338.3 ms … 369.8 ms 10 runs
Benchmark 3: ./target/release/kero_fold_par_iter 1000000
Time <span class="o">(</span>mean ± σ<span class="o">)</span>: 49.5 ms ± 1.5 ms <span class="o">[</span>User: 574.4 ms, System: 14.1 ms]
Range <span class="o">(</span>min … max<span class="o">)</span>: 45.2 ms … 53.1 ms 58 runs
Summary
<span class="s1">'./target/release/kero_fold_par_iter 1000000'</span> ran
7.01 ± 0.28 <span class="nb">times </span>faster than <span class="s1">'./target/release/kero_simple_port 1000000'</span>
667.27 ± 25.77 <span class="nb">times </span>faster than <span class="s1">'python main.py'</span>
</code></pre></div></div>J’ai optimisé un bout de code avec Rust et j’ai gagné un facteur ~670 dans les performances.Batch resizing images using a Nautilus extension2022-08-19T00:00:00+00:002022-08-19T00:00:00+00:00https://www.keiruaprod.fr/blog/2022/08/19/batch-resizing-files-with-nautilus<p>Something I do very often is resizing images. I wrote about it before, and for a long time I’ve settled on a custom bash script.</p>
<p>I’ve recently found a somewhat better option: integrating a button in Nautilus, the Ubuntu file explorer.</p>
<h2 id="possible-but-nope">Possible but nope</h2>
<p>There a many ways to integrate custom code inside Nautilus in order to extend it. For example:</p>
<ul>
<li>writing a custom C program. <a href="https://github.com/videogame-hacker/nautilus-png-convert">Example</a>. That’s the historic way to write plugins, but the thing is: it’s hard to write and maintain C. I have experience with that 10+ years ago, and I don’t want to relearn old stuff.</li>
<li>writing the same program in Rust. <a href="https://github.com/talklittle/tmsu-nautilus-rs">Example</a>. A bit better for me, better dependency management but the ecosystem is fragile and I fear that the <1.0.0 libs will break and force me to spend time fixing library stuff when I update my system.</li>
<li>a nautilus shell script. The low tech solution I’ll settle on for now.</li>
</ul>
<h1 id="nautilus">Nautilus</h1>
<p>Nautilus proposes a simple way to create scripts on files:</p>
<ul>
<li>it exposes a <code class="highlighter-rouge">NAUTILUS_SCRIPT_SELECTED_FILE_PATHS</code> environment variable that contain the list of selected files</li>
<li>you can write a executable script and store it in <code class="highlighter-rouge">~/.local/share/nautilus/scripts</code></li>
</ul>
<p>This way, you can write an executable file in whatever language you like (bash, python, rust…) in order to perform tasks on said files.</p>
<h1 id="our-script">Our script</h1>
<p>Save this as <code class="highlighter-rouge">resize-image.py</code>, make it executable (<code class="highlighter-rouge">chmod +x resize-image.py</code>) and copy it to <code class="highlighter-rouge">~/.local/share/nautilus/scripts</code>. You can now select a few image files in Nautilus, right click on them -> scripts -> resize image.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">#! /usr/bin/env python3
</span>
<span class="c1"># resize an image to 25% of its original size, and store the result in
# the `small` subdirectory
# select the image files you want to resize, right click, and under Script
# choose resize-image
</span>
<span class="kn">import</span> <span class="nn">os</span>
<span class="kn">import</span> <span class="nn">subprocess</span>
<span class="k">def</span> <span class="nf">new_filename</span><span class="p">(</span><span class="n">filewithpath</span><span class="p">,</span> <span class="n">subdirectory</span><span class="o">=</span><span class="s">'small'</span><span class="p">):</span>
<span class="n">directory</span> <span class="o">=</span> <span class="n">os</span><span class="p">.</span><span class="n">path</span><span class="p">.</span><span class="n">dirname</span><span class="p">(</span><span class="n">filewithpath</span><span class="p">)</span>
<span class="n">filename</span> <span class="o">=</span> <span class="n">os</span><span class="p">.</span><span class="n">path</span><span class="p">.</span><span class="n">basename</span><span class="p">(</span><span class="n">filewithpath</span><span class="p">)</span>
<span class="n">target_directory</span> <span class="o">=</span> <span class="n">os</span><span class="p">.</span><span class="n">path</span><span class="p">.</span><span class="n">join</span><span class="p">(</span><span class="n">directory</span><span class="p">,</span> <span class="n">subdirectory</span><span class="p">)</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">os</span><span class="p">.</span><span class="n">mkdir</span><span class="p">(</span><span class="n">target_directory</span><span class="p">)</span>
<span class="k">except</span> <span class="nb">FileExistsError</span> <span class="k">as</span> <span class="n">e</span><span class="p">:</span>
<span class="c1"># The directory already exists and yes, this exception is poorly named
</span> <span class="k">pass</span>
<span class="n">target_filename</span> <span class="o">=</span> <span class="n">os</span><span class="p">.</span><span class="n">path</span><span class="p">.</span><span class="n">join</span><span class="p">(</span><span class="n">target_directory</span><span class="p">,</span> <span class="n">filename</span><span class="p">)</span>
<span class="k">return</span> <span class="n">target_filename</span>
<span class="k">def</span> <span class="nf">main</span><span class="p">():</span>
<span class="k">for</span> <span class="n">filepath</span> <span class="ow">in</span> <span class="n">os</span><span class="p">.</span><span class="n">getenv</span><span class="p">(</span><span class="s">'NAUTILUS_SCRIPT_SELECTED_FILE_PATHS'</span><span class="p">,</span> <span class="s">''</span><span class="p">).</span><span class="n">splitlines</span><span class="p">():</span>
<span class="n">small_filename</span> <span class="o">=</span> <span class="n">new_filename</span><span class="p">(</span><span class="n">filepath</span><span class="p">)</span>
<span class="n">subprocess</span><span class="p">.</span><span class="n">run</span><span class="p">([</span><span class="s">"convert"</span><span class="p">,</span> <span class="s">"-resize"</span><span class="p">,</span> <span class="s">"25%"</span><span class="p">,</span> <span class="n">filepath</span><span class="p">,</span> <span class="n">small_filename</span><span class="p">])</span>
<span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">'__main__'</span><span class="p">:</span>
<span class="c1"># alternatively you can test this script by hand by setting the env var yourself:
</span> <span class="c1"># export NAUTILUS_SCRIPT_SELECTED_FILE_PATHS='/home/some/full/path/test.png'; python resize-image.py
</span> <span class="n">main</span><span class="p">()</span>
</code></pre></div></div>Something I do very often is resizing images. I wrote about it before, and for a long time I’ve settled on a custom bash script.Casser un hash de CTF avec z32022-06-15T20:44:00+00:002022-06-15T20:44:00+00:00https://www.keiruaprod.fr/blog/2022/06/15/casser-un-hash-de-ctf-avec-z3<p>Un classique des compétitions de sécurité: on vous donne un algorithme de hashage (de manière plus ou moins directe), une valeur hachée, et il faut trouver le mot de passe qui a mené à cette valeur hachée.</p>
<p>On va voir dans ce post comment résoudre ce type de problèmes avec z3, un solveur SMT (Satisfiability Modulo Theories) dont j’ai déjà parlé. En gros on lui donne des contraintes, et il trouve un ensemble de valeurs qui satisfait ces contraintes.</p>
<h2 id="lalgo-à-casser">L’algo à casser</h2>
<p>Pour l’algo de hashage à casser, j’ai porté en Python l’exercice de cette <a href="https://youtu.be/0DJZ2Bt5eyE?t=2022">fin de vidéo</a> :</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># Example adapted from:
# https://youtu.be/0DJZ2Bt5eyE?t=2022
</span><span class="k">def</span> <span class="nf">hash</span><span class="p">(</span><span class="n">s</span><span class="p">:</span><span class="nb">str</span><span class="p">)</span> <span class="o">-></span> <span class="nb">int</span><span class="p">:</span>
<span class="n">n</span> <span class="o">=</span> <span class="mi">7</span>
<span class="k">for</span> <span class="n">c</span> <span class="ow">in</span> <span class="n">s</span><span class="p">:</span>
<span class="n">n</span> <span class="o">=</span> <span class="n">n</span><span class="o">*</span><span class="mi">31</span><span class="o">+</span><span class="nb">ord</span><span class="p">(</span><span class="n">c</span><span class="p">)</span>
<span class="k">return</span> <span class="n">n</span><span class="o">%</span><span class="p">(</span><span class="mi">2</span><span class="o">**</span><span class="mi">32</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">check_hash</span><span class="p">(</span><span class="n">s</span><span class="p">:</span><span class="nb">str</span><span class="p">)</span> <span class="o">-></span> <span class="nb">bool</span><span class="p">:</span>
<span class="n">TARGET_HASH</span><span class="o">=</span><span class="mi">593779930</span>
<span class="k">return</span> <span class="nb">hash</span><span class="p">(</span><span class="n">s</span><span class="p">)</span> <span class="o">==</span> <span class="n">TARGET_HASH</span>
<span class="k">if</span> <span class="n">__name__</span><span class="o">==</span><span class="s">"__main__"</span><span class="p">:</span>
<span class="kn">import</span> <span class="nn">sys</span>
<span class="k">print</span><span class="p">(</span><span class="n">sys</span><span class="p">.</span><span class="n">argv</span><span class="p">[</span><span class="mi">1</span><span class="p">])</span>
<span class="k">print</span><span class="p">(</span><span class="n">check_hash</span><span class="p">(</span><span class="n">sys</span><span class="p">.</span><span class="n">ARGV</span><span class="p">[</span><span class="mi">1</span><span class="p">]))</span>
</code></pre></div></div>
<p>On est donc face à un bout de code qui hashe un mot de passe, et renvoie un entier.</p>
<h2 id="casser-ce-hash">Casser ce hash</h2>
<p>On a besoin de la librairie python z3:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pip <span class="nb">install </span>z3-solver
</code></pre></div></div>
<p>Ensuite, on va bruteforcer la longueur, créer un solveur pour chaque longueur, et tenter de trouver une solution. Une fois qu’on en a trouvée une (solver.check() == sat), on reconstruit le mot de passe.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">z3</span> <span class="kn">import</span> <span class="o">*</span>
<span class="n">TARGET_HASH</span><span class="o">=</span><span class="mi">593779930</span>
<span class="k">def</span> <span class="nf">hash</span><span class="p">(</span><span class="n">s</span><span class="p">)</span> <span class="o">-></span> <span class="nb">int</span><span class="p">:</span>
<span class="n">n</span> <span class="o">=</span> <span class="mi">7</span>
<span class="k">for</span> <span class="n">c</span> <span class="ow">in</span> <span class="n">s</span><span class="p">:</span>
<span class="n">n</span> <span class="o">=</span> <span class="n">n</span><span class="o">*</span><span class="mi">31</span><span class="o">+</span><span class="n">c</span>
<span class="k">return</span> <span class="n">n</span><span class="o">%</span><span class="p">(</span><span class="mi">2</span><span class="o">**</span><span class="mi">32</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">break_at_length</span><span class="p">(</span><span class="n">target_length</span><span class="p">:</span> <span class="nb">int</span><span class="p">)</span> <span class="o">-></span> <span class="n">Solver</span><span class="p">:</span>
<span class="s">"""Coeur de la résolution:
on crée un solveur, on ajoute nos contraintes:
- une longueur fixée
- des lettres entre a et z
- le hash de cette entrée doit valoir le hash cible
"""</span>
<span class="n">s</span> <span class="o">=</span> <span class="n">Solver</span><span class="p">()</span>
<span class="c1"># Le mot de passe
</span> <span class="n">chars</span> <span class="o">=</span> <span class="p">[</span><span class="n">Int</span><span class="p">(</span><span class="sa">f</span><span class="s">"c_</span><span class="si">{</span><span class="n">i</span><span class="si">}</span><span class="s">"</span><span class="p">)</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">target_length</span><span class="p">)]</span>
<span class="c1"># Chaque élément du mot de passe doit être une lettre en minuscule
</span> <span class="k">for</span> <span class="n">c</span> <span class="ow">in</span> <span class="n">chars</span><span class="p">:</span>
<span class="n">s</span><span class="p">.</span><span class="n">add</span><span class="p">(</span><span class="n">And</span><span class="p">(</span><span class="n">c</span> <span class="o">>=</span> <span class="nb">ord</span><span class="p">(</span><span class="s">'a'</span><span class="p">),</span> <span class="n">c</span> <span class="o"><=</span> <span class="nb">ord</span><span class="p">(</span><span class="s">'z'</span><span class="p">)))</span>
<span class="c1"># On veut que le hash obtenu avec ce mot de passe soit celui de la cible
</span> <span class="n">s</span><span class="p">.</span><span class="n">add</span><span class="p">(</span><span class="nb">hash</span><span class="p">(</span><span class="n">chars</span><span class="p">)</span><span class="o">==</span><span class="n">TARGET_HASH</span><span class="p">)</span>
<span class="k">return</span> <span class="n">s</span><span class="p">,</span> <span class="n">chars</span>
<span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">"__main__"</span><span class="p">:</span>
<span class="c1"># comme on ne connait pas la longueur du mode de passe,
</span> <span class="c1"># une option c’est de bruteforcer: tant qu’on a pas trouvé
</span> <span class="c1"># une solution valide, on teste une longueur plus grande
</span> <span class="n">length</span> <span class="o">=</span> <span class="mi">1</span>
<span class="k">while</span> <span class="bp">True</span><span class="p">:</span>
<span class="n">s</span><span class="p">,</span> <span class="n">chars</span> <span class="o">=</span> <span class="n">break_at_length</span><span class="p">(</span><span class="n">length</span><span class="p">)</span>
<span class="k">if</span> <span class="n">s</span><span class="p">.</span><span class="n">check</span><span class="p">()</span> <span class="o">!=</span> <span class="n">sat</span><span class="p">:</span>
<span class="k">print</span><span class="p">(</span><span class="sa">f</span><span class="s">"Nothing found at length </span><span class="si">{</span><span class="n">length</span><span class="si">}</span><span class="s">"</span><span class="p">)</span>
<span class="n">length</span><span class="o">+=</span><span class="mi">1</span>
<span class="k">else</span><span class="p">:</span>
<span class="k">break</span>
<span class="n">m</span> <span class="o">=</span> <span class="n">s</span><span class="p">.</span><span class="n">model</span><span class="p">()</span>
<span class="k">print</span><span class="p">(</span><span class="sa">f</span><span class="s">"password length: </span><span class="si">{</span><span class="n">length</span><span class="si">}</span><span class="s">"</span><span class="p">)</span>
<span class="c1"># On transforme notre modèle z3 en une string
</span> <span class="k">print</span><span class="p">(</span><span class="s">""</span><span class="p">.</span><span class="n">join</span><span class="p">([</span><span class="nb">chr</span><span class="p">(</span><span class="n">m</span><span class="p">[</span><span class="n">c</span><span class="p">].</span><span class="n">as_long</span><span class="p">())</span> <span class="k">for</span> <span class="n">c</span> <span class="ow">in</span> <span class="n">chars</span><span class="p">]))</span>
</code></pre></div></div>
<p>On teste:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ python solve-cyberseed.py
Nothing found at length 1
Nothing found at length 2
Nothing found at length 3
Nothing found at length 4
Nothing found at length 5
password length: 6
dragon
</code></pre></div></div>
<p>On a trouvé le mot de passe (<code class="highlighter-rouge">dragon</code>), trop cool !</p>Un classique des compétitions de sécurité: on vous donne un algorithme de hashage (de manière plus ou moins directe), une valeur hachée, et il faut trouver le mot de passe qui a mené à cette valeur hachée.Puzzle : Je ne sais pas quels sont les chiffres2022-06-14T00:00:00+00:002022-06-14T00:00:00+00:00https://www.keiruaprod.fr/blog/2022/06/14/puzzle-je-ne-sais-pas<p>Je suis tombé sur un puzzle récemment : <a href="https://news.ycombinator.com/item?id=31293611">I don’t know the numbers</a></p>
<p>L’intitulé est très cool et à la lecture, la première chose qu’on se dit, c’est <code class="highlighter-rouge">what the fuck</code>:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Two numbers are chosen randomly, both are positive integers smaller than 100.
Sandy is told the sum of the numbers,
while Peter is told the product of the numbers.
Then, this dialog occurs between Sandy and Peter:
Peter: I don't know the numbers.
Sandy: I don't know the numbers.
Peter: I don't know the numbers.
Sandy: I don't know the numbers.
Peter: I don't know the numbers.
Sandy: I don't know the numbers.
Peter: I don't know the numbers.
Sandy: I don't know the numbers.
Peter: I don't know the numbers.
Sandy: I don't know the numbers.
Peter: I don't know the numbers.
Sandy: I don't know the numbers.
Peter: I don't know the numbers.
Sandy: I don't know the numbers.
Peter: I do know the numbers.
What are the numbers?
</code></pre></div></div>
<p>Dans la suite (mega spoiler), on va résoudre ce problème avec un peu de code.</p>
<h2 id="résoudre-ce-problème">Résoudre ce problème</h2>
<p>Alors, que se passe-t-il ?</p>
<p>Hé bien le plus simple pour démarrer, c’est de commencer par visualiser ce genre de problèmes:</p>
<ul>
<li>on va générer des paires de nombres</li>
<li>on va regarder les valeurs qui permettent d’obtenir ces paires de nombres</li>
</ul>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">pprint</span> <span class="k">as</span> <span class="n">pp</span>
<span class="kn">from</span> <span class="nn">collections</span> <span class="kn">import</span> <span class="n">defaultdict</span>
<span class="n">max_value</span> <span class="o">=</span> <span class="mi">100</span>
<span class="n">products</span> <span class="o">=</span> <span class="n">defaultdict</span><span class="p">(</span><span class="nb">set</span><span class="p">)</span>
<span class="n">sums</span> <span class="o">=</span> <span class="n">defaultdict</span><span class="p">(</span><span class="nb">set</span><span class="p">)</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="n">max_value</span><span class="p">):</span>
<span class="k">for</span> <span class="n">j</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="n">max_value</span><span class="p">):</span>
<span class="n">pair</span> <span class="o">=</span> <span class="p">(</span><span class="nb">min</span><span class="p">(</span><span class="n">i</span><span class="p">,</span><span class="n">j</span><span class="p">),</span> <span class="nb">max</span><span class="p">(</span><span class="n">i</span><span class="p">,</span><span class="n">j</span><span class="p">))</span>
<span class="n">products</span><span class="p">[</span><span class="n">i</span><span class="o">*</span><span class="n">j</span><span class="p">].</span><span class="n">add</span><span class="p">(</span><span class="n">pair</span><span class="p">)</span>
<span class="n">sums</span><span class="p">[</span><span class="n">i</span><span class="o">+</span><span class="n">j</span><span class="p">].</span><span class="n">add</span><span class="p">(</span><span class="n">pair</span><span class="p">)</span>
</code></pre></div></div>
<p>Qu’est ce qu’on vient de faire ?
Hé bien, avec ce bout de code, on vient de générer deux dictionnaires:</p>
<ul>
<li>un pour la liste des sommes des paires de nombres</li>
<li>un pour la liste des produits des paires de nombres</li>
</ul>
<p>On utilise un <code class="highlighter-rouge">set</code> plutôt qu’une liste pour éliminer les doublons.
On trie la paire de valeurs pour gérer facilement le cas (j, i) qui est le même que (j, i), car i+j == j+i et i<em>j == j</em>i.</p>
<p>Voici le début du dictionnaire des produits:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="o">>>></span> <span class="n">pp</span><span class="p">.</span><span class="n">pprint</span><span class="p">(</span><span class="n">products</span><span class="p">)</span>
<span class="n">defaultdict</span><span class="p">(</span><span class="o"><</span><span class="k">class</span> <span class="err">'</span><span class="nc">set</span><span class="s">'>,
{1: {(1, 1)},
2: {(1, 2)},
3: {(1, 3)},
4: {(1, 4), (2, 2)},
5: {(1, 5)},
6: {(2, 3), (1, 6)},
# …
</span></code></pre></div></div>
<p>On peut voir qu’il y a deux manières d’obtenir 4 : 1<em>4, et 2</em>2.
Pareil pour 6: 2<em>3, et 1</em>6</p>
<h2 id="comprendre-les-contraintes">Comprendre les contraintes</h2>
<p>Comment on utilise ça pour résoudre le problème ? On peut remarquer qu’à chaque tour, les personnages nous donnent un indice: il n’ont pas assez d’élément pour décider. Il semble y avoir une progression, car au bout d’un moment, Peter trouve la réponse.</p>
<p>Peter connait le produit qu’il doit atteindre, mais il y forcément plus d’une manière de l’obtenir car sinon, il pourrait décider.</p>
<p>Par exemple, si la valeur que Peter cherche est 5, il pourrait dès le début conclure: il y a une seule manière d’obtenir 5, c’est <code class="highlighter-rouge">1*5</code>.</p>
<p>Maintenant, comment avancer ? Nous allons propager les contraintes.</p>
<h2 id="propagation-des-contraintes">Propagation des contraintes</h2>
<p>On sait que tout un tas de paires ne sont pas des solutions: par exemple, on vient de voir que (1, 5) n’est pas la solution, car si c’était le cas, on peut répondre.</p>
<p>On peut donc:</p>
<ul>
<li>Rechercher toutes les paires “solitaires” qui ne sont pas des solutions à l’étape actuelle</li>
<li>supprimer toutes ces valeurs dans les listes du dictionnaire des produits et dans celles du dictionnaire des sommes</li>
</ul>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">find_single</span><span class="p">(</span><span class="n">d</span><span class="p">):</span>
<span class="s">"""
Retourne la liste des valeurs qui sont seules dans le dictionnaire
"""</span>
<span class="k">return</span> <span class="p">[</span><span class="nb">next</span><span class="p">(</span><span class="nb">iter</span><span class="p">(</span><span class="n">v</span><span class="p">))</span> <span class="k">for</span> <span class="n">v</span> <span class="ow">in</span> <span class="n">d</span><span class="p">.</span><span class="n">values</span><span class="p">()</span> <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">v</span><span class="p">)</span> <span class="o">==</span> <span class="mi">1</span><span class="p">]</span>
<span class="n">single_pairs</span> <span class="o">=</span> <span class="n">find_single</span><span class="p">(</span><span class="n">products</span><span class="p">)</span>
<span class="c1"># Réduction de l’espace de recherche
</span><span class="n">products</span> <span class="o">=</span> <span class="p">{</span><span class="n">x</span><span class="p">:</span><span class="n">y</span><span class="o">-</span><span class="nb">set</span><span class="p">(</span><span class="n">single_pairs</span><span class="p">)</span> <span class="k">for</span> <span class="n">x</span><span class="p">,</span><span class="n">y</span> <span class="ow">in</span> <span class="n">products</span><span class="p">.</span><span class="n">items</span><span class="p">()</span> <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">y</span><span class="o">-</span><span class="nb">set</span><span class="p">(</span><span class="n">single_pairs</span><span class="p">))</span> <span class="o">>=</span> <span class="mi">1</span><span class="p">}</span>
<span class="n">sums</span> <span class="o">=</span> <span class="p">{</span><span class="n">x</span><span class="p">:</span><span class="n">y</span><span class="o">-</span><span class="nb">set</span><span class="p">(</span><span class="n">single_pairs</span><span class="p">)</span> <span class="k">for</span> <span class="n">x</span><span class="p">,</span><span class="n">y</span> <span class="ow">in</span> <span class="n">sums</span><span class="p">.</span><span class="n">items</span><span class="p">()</span> <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">y</span><span class="o">-</span><span class="nb">set</span><span class="p">(</span><span class="n">single_pairs</span><span class="p">))</span> <span class="o">>=</span> <span class="mi">1</span><span class="p">}</span>
</code></pre></div></div>
<h2 id="solution-complète-en-python">Solution complète en Python</h2>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">collections</span> <span class="kn">import</span> <span class="n">defaultdict</span>
<span class="kn">import</span> <span class="nn">pprint</span> <span class="k">as</span> <span class="n">pp</span>
<span class="k">class</span> <span class="nc">ProductSumPuzzle</span><span class="p">:</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">max_value</span><span class="p">):</span>
<span class="bp">self</span><span class="p">.</span><span class="n">sums</span> <span class="o">=</span> <span class="n">defaultdict</span><span class="p">(</span><span class="nb">set</span><span class="p">)</span>
<span class="bp">self</span><span class="p">.</span><span class="n">products</span> <span class="o">=</span> <span class="n">defaultdict</span><span class="p">(</span><span class="nb">set</span><span class="p">)</span>
<span class="c1"># Generate the candidate lists for the products and sums
</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="n">max_value</span><span class="p">):</span>
<span class="k">for</span> <span class="n">j</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="n">max_value</span><span class="p">):</span>
<span class="n">pair</span> <span class="o">=</span> <span class="p">(</span><span class="nb">min</span><span class="p">(</span><span class="n">i</span><span class="p">,</span><span class="n">j</span><span class="p">),</span> <span class="nb">max</span><span class="p">(</span><span class="n">i</span><span class="p">,</span><span class="n">j</span><span class="p">))</span>
<span class="bp">self</span><span class="p">.</span><span class="n">products</span><span class="p">[</span><span class="n">i</span><span class="o">*</span><span class="n">j</span><span class="p">].</span><span class="n">add</span><span class="p">(</span><span class="n">pair</span><span class="p">)</span>
<span class="bp">self</span><span class="p">.</span><span class="n">sums</span><span class="p">[</span><span class="n">i</span><span class="o">+</span><span class="n">j</span><span class="p">].</span><span class="n">add</span><span class="p">(</span><span class="n">pair</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">reduce_search_space</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">single_pairs</span><span class="p">):</span>
<span class="bp">self</span><span class="p">.</span><span class="n">products</span> <span class="o">=</span> <span class="p">{</span><span class="n">x</span><span class="p">:</span><span class="n">y</span><span class="o">-</span><span class="nb">set</span><span class="p">(</span><span class="n">single_pairs</span><span class="p">)</span> <span class="k">for</span> <span class="n">x</span><span class="p">,</span><span class="n">y</span> <span class="ow">in</span> <span class="bp">self</span><span class="p">.</span><span class="n">products</span><span class="p">.</span><span class="n">items</span><span class="p">()</span> <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">y</span><span class="o">-</span><span class="nb">set</span><span class="p">(</span><span class="n">single_pairs</span><span class="p">))</span> <span class="o">>=</span> <span class="mi">1</span><span class="p">}</span>
<span class="bp">self</span><span class="p">.</span><span class="n">sums</span> <span class="o">=</span> <span class="p">{</span><span class="n">x</span><span class="p">:</span><span class="n">y</span><span class="o">-</span><span class="nb">set</span><span class="p">(</span><span class="n">single_pairs</span><span class="p">)</span> <span class="k">for</span> <span class="n">x</span><span class="p">,</span><span class="n">y</span> <span class="ow">in</span> <span class="bp">self</span><span class="p">.</span><span class="n">sums</span><span class="p">.</span><span class="n">items</span><span class="p">()</span> <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">y</span><span class="o">-</span><span class="nb">set</span><span class="p">(</span><span class="n">single_pairs</span><span class="p">))</span> <span class="o">>=</span> <span class="mi">1</span><span class="p">}</span>
<span class="k">def</span> <span class="nf">find_single</span><span class="p">(</span><span class="n">d</span><span class="p">):</span>
<span class="k">return</span> <span class="p">[</span><span class="nb">next</span><span class="p">(</span><span class="nb">iter</span><span class="p">(</span><span class="n">v</span><span class="p">))</span> <span class="k">for</span> <span class="n">v</span> <span class="ow">in</span> <span class="n">d</span><span class="p">.</span><span class="n">values</span><span class="p">()</span> <span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">v</span><span class="p">)</span> <span class="o">==</span> <span class="mi">1</span><span class="p">]</span>
<span class="k">if</span> <span class="n">__name__</span><span class="o">==</span> <span class="s">"__main__"</span><span class="p">:</span>
<span class="n">MAX_VALUE</span> <span class="o">=</span> <span class="mi">100</span>
<span class="n">puzzle</span> <span class="o">=</span> <span class="n">ProductSumPuzzle</span><span class="p">(</span><span class="n">MAX_VALUE</span><span class="p">)</span>
<span class="c1"># Les 14 "je ne sais pas" réduisent l’espace de recherche, tour à tour avec les produits et les sommes:
</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">7</span><span class="p">):</span>
<span class="c1"># remove product pairs that have single elements
</span> <span class="n">puzzle</span><span class="p">.</span><span class="n">reduce_search_space</span><span class="p">(</span><span class="n">find_single</span><span class="p">(</span><span class="n">puzzle</span><span class="p">.</span><span class="n">products</span><span class="p">))</span>
<span class="c1"># remove sum pairs that have single elements
</span> <span class="n">puzzle</span><span class="p">.</span><span class="n">reduce_search_space</span><span class="p">(</span><span class="n">find_single</span><span class="p">(</span><span class="n">puzzle</span><span class="p">.</span><span class="n">sums</span><span class="p">))</span>
<span class="c1"># Finalement, un des produits a une unique paire de valeurs
</span> <span class="c1"># qui mène à ce produit:
</span> <span class="n">pair</span> <span class="o">=</span> <span class="n">find_single</span><span class="p">(</span><span class="n">puzzle</span><span class="p">.</span><span class="n">products</span><span class="p">)[</span><span class="mi">0</span><span class="p">]</span>
<span class="n">prod</span> <span class="o">=</span> <span class="n">pair</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="o">*</span><span class="n">pair</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span>
<span class="k">print</span><span class="p">(</span><span class="n">pair</span><span class="p">,</span> <span class="n">prod</span><span class="p">)</span>
</code></pre></div></div>Je suis tombé sur un puzzle récemment : I don’t know the numbersRevealJS pour des prez vite fait2022-04-27T00:00:00+00:002022-04-27T00:00:00+00:00https://www.keiruaprod.fr/blog/2022/04/27/template-revealjs<p>Pour des présentations vite fait j’utitise souvent revealJS. Ca me permet d’avoir un support pour ne rien oublier, un trace écrite à partager et c’est rapide à mettre en place.</p>
<p>On peut récupérer la dernière version de revealJS là: https://revealjs.com/ (c’est nécessaire pour les fichiers dans <code class="highlighter-rouge">dist/</code> et <code class="highlighter-rouge">plugin/</code>)</p>
<p>Voici un template de base.</p>
<p>Le contenu est en markdown directement dans le HTML car c’est plus simple, et quand je change ma présentation j’ai juste à tout effacer.</p>
<div class="language-html highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt"><html></span>
<span class="nt"><head></span>
<span class="nt"><meta</span> <span class="na">charset=</span><span class="s">"utf-8"</span><span class="nt">></span>
<span class="nt"><title></span>Présentation<span class="nt"></title></span>
<span class="nt"><link</span> <span class="na">rel=</span><span class="s">"stylesheet"</span> <span class="na">href=</span><span class="s">"dist/reveal.css"</span><span class="nt">></span>
<span class="nt"><link</span> <span class="na">rel=</span><span class="s">"stylesheet"</span> <span class="na">href=</span><span class="s">"dist/theme/black.css"</span><span class="nt">></span>
<span class="nt"><script </span><span class="na">src=</span><span class="s">"dist/reveal.js"</span><span class="nt">></script></span>
<span class="nt"><script </span><span class="na">src=</span><span class="s">"plugin/markdown/markdown.js"</span><span class="nt">></script></span>
<span class="nt"></head></span>
<span class="nt"><body></span>
<span class="nt"><div</span> <span class="na">class=</span><span class="s">"reveal"</span><span class="nt">></span>
<span class="nt"><div</span> <span class="na">class=</span><span class="s">"slides"</span><span class="nt">></span>
<span class="nt"><section</span> <span class="na">data-markdown</span><span class="nt">></span>
<span class="nt"><textarea</span> <span class="na">data-template</span><span class="nt">></span>
## Page 1
Votre premier slide
---
## Page 2
- Un truc important
- Un autre truc important
---
## ✨ Et voilà ! ✨
<span class="nt"></textarea></span>
<span class="nt"></section></span>
<span class="nt"></div></span>
<span class="nt"></div></span>
<span class="nt"><script></span>
<span class="nx">Reveal</span><span class="p">.</span><span class="nx">initialize</span><span class="p">({</span>
<span class="na">plugins</span><span class="p">:</span> <span class="p">[</span> <span class="nx">RevealMarkdown</span> <span class="p">]</span>
<span class="p">});</span>
<span class="nt"></script></span>
<span class="nt"></body></span>
<span class="nt"></html></span>
</code></pre></div></div>Pour des présentations vite fait j’utitise souvent revealJS. Ca me permet d’avoir un support pour ne rien oublier, un trace écrite à partager et c’est rapide à mettre en place.Daniel Vassallo et son “portfolio des petits paris”2022-03-14T00:00:00+00:002022-03-14T00:00:00+00:00https://www.keiruaprod.fr/blog/2022/03/14/daniel-vassallo<p>Daniel Vassalo a vendu ses planches pour découper la viande à 600 dollars.</p>
<p>Plus de stock en deux jours. Comment? Storytime.</p>
<p>Pour comprendre comment on en arrive à être sold-out en 2j sur un produit que je n’achèterais jamais, il faut remonter le fil de ce qu’il a fait ces dernières années.</p>
<p>Daniel Vassallo a quitté son job chez Amazon très bien payé (500k$/an !), pour trouver plus de sens. D’après lui, “seule la motivation intrinsèque tient dans le temps”. Il a écrit <a href="https://dvassallo.medium.com/only-intrinsic-motivation-lasts-92c0497cf97c">un article dessus</a> qui l’a lancé sur les réseaux sociaux.</p>
<p>Sa philosophie part d’un <strong>constat atypique</strong>:</p>
<ul>
<li>avoir un job salarié est risqué, on peut se faire virer à tout moment et tout perdre</li>
<li>monter une boite et espérer que ça marche va a l’encontre des statistiques</li>
</ul>
<p><strong>Sa solution</strong>: avoir plusieurs projets pour se diversifier et minimiser les chances qu’un d’entre eux se plante. Il a alors diversifié ses revenus à travers plusieurs projets, comme:</p>
<ul>
<li>un livre sur AWS, un des services clé d’Amazon</li>
<li>un livre sur comment grossir sur Twitter</li>
<li>un service SaaS</li>
<li>une communauté autour de sa philosophie de diversification et de minimalisme</li>
<li>ses résultats financiers (oui oui)</li>
<li>…et récemment, des planches à découper.</li>
</ul>
<p>Des planches à découper ? bah… oui. Il aime les fabriquer, travailler le bois est un de ses hobbies. Avec plus de <a href="https://twitter.com/dvassallo">110k abonnés sur Twitter</a> dont beaucoup très engagés, quand il a proposé d’en vendre à un prix élevé, son stock initial est parti en quelques jours. On peut <a href="https://dvassallo.gumroad.com/?tags=cutting-boards">en acheter actuellement</a>.</p>
<p>Il appelle cette approche du business <strong>“le portfolio des petits paris”</strong>.</p>Daniel Vassalo a vendu ses planches pour découper la viande à 600 dollars.Bannière Twitter dynamique2022-03-12T00:00:00+00:002022-03-12T00:00:00+00:00https://www.keiruaprod.fr/blog/2022/03/12/banniere-dynamique-twitter<p>Projet du weekend: je me suis bricolé une bannière Twitter dynamique.</p>
<p>C’est à la mode sur Twitch/YT, j’avais envie de la même chose, on va voir comment j’ai codé ça.</p>
<p><img src="/assets/pictures/twitter-banner/banniere-dynamique.png" alt="" /></p>
<p>Sur Twitter c’est pas répandu mais ce n’est pas une idée nouvelle.</p>
<p>J’ai vu récemment <a href="https://twitter.com/tdinh_me">@tdinh_me</a> et <a href="https://twitter.com/acemtp">@acemtp</a> se lancer là dedans, il y en a sans doute plein d’autres.</p>
<p>Pour faire ça il nous faut 3 trucs:</p>
<ul>
<li>🧠 Des accès à l’API Twitter : les clés récupérer les derniers followers et mettre à jour la bannière</li>
<li>🎨 Savoir générer une image avec du code</li>
<li>🖥 Un serveur qui s’occupe de faire les mises à jour</li>
</ul>
<h2 id="lapi">L’API</h2>
<p>L’API Twitter est mal documentée sur https://developer.twitter.com, il y a 2 versions.
On doit utiliser l’API version 1, pour laquelle on veut les droits en lecture/écriture. Je me suis fait mordre avec juste l’accès lecture.
La version 2 ne permet pas de mettre à jour la bannière de profil. C’est dommage car c’est elle que Twitter compte privilégier à l’avenir.</p>
<p>Après avoir créé le projet correctement, on obtient des identifiants qui permettent alors d’effectuer des requêtes vers l’API.</p>
<p>On range les valeurs dans un fichier .env, pour ne pas inclure les clés directement dans le code versionné du projet.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># config.env</span>
<span class="nv">API_KEY</span><span class="o">=</span><span class="s2">"do"</span>
<span class="nv">API_KEY_SECRET</span><span class="o">=</span><span class="s2">"re"</span>
<span class="nv">ACCESS_TOKEN</span><span class="o">=</span><span class="s2">"mi"</span>
<span class="nv">ACCESS_TOKEN_SECRET</span><span class="o">=</span><span class="s2">"fa"</span>
</code></pre></div></div>
<p>Passons au code Python : on veut récupérer la photo de profil des 5 derniers followers.</p>
<p>J’ai utilisé dotenv pour récupérer mes identifiants Twitter dans les variables d’environnement, et Tweepy pour solliciter l’API.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">os</span>
<span class="kn">import</span> <span class="nn">tweepy</span>
<span class="kn">from</span> <span class="nn">dotenv</span> <span class="kn">import</span> <span class="n">load_dotenv</span>
<span class="c1"># Load environment variables from .env file
</span><span class="n">load_dotenv</span><span class="p">(</span><span class="s">"config.env"</span><span class="p">)</span>
<span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">"__main__"</span><span class="p">:</span>
<span class="n">auth</span> <span class="o">=</span> <span class="n">tweepy</span><span class="p">.</span><span class="n">OAuthHandler</span><span class="p">(</span><span class="n">os</span><span class="p">.</span><span class="n">getenv</span><span class="p">(</span><span class="s">"API_KEY"</span><span class="p">),</span> <span class="n">os</span><span class="p">.</span><span class="n">getenv</span><span class="p">(</span><span class="s">"API_KEY_SECRET"</span><span class="p">))</span>
<span class="n">auth</span><span class="p">.</span><span class="n">set_access_token</span><span class="p">(</span><span class="n">os</span><span class="p">.</span><span class="n">getenv</span><span class="p">(</span><span class="s">"ACCESS_TOKEN"</span><span class="p">),</span> <span class="n">os</span><span class="p">.</span><span class="n">getenv</span><span class="p">(</span><span class="s">"ACCESS_TOKEN_SECRET"</span><span class="p">))</span>
<span class="n">api</span> <span class="o">=</span> <span class="n">tweepy</span><span class="p">.</span><span class="n">API</span><span class="p">(</span><span class="n">auth</span><span class="p">)</span>
<span class="n">latest_followers</span> <span class="o">=</span> <span class="n">api</span><span class="p">.</span><span class="n">get_followers</span><span class="p">(</span><span class="n">count</span><span class="o">=</span><span class="mi">5</span><span class="p">,</span> <span class="n">skip_status</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
<span class="k">print</span><span class="p">([</span><span class="n">follower</span><span class="p">.</span><span class="n">profile_image_url_https</span> <span class="k">for</span> <span class="n">follower</span> <span class="ow">in</span> <span class="n">latest_followers</span><span class="p">])</span>
</code></pre></div></div>
<p>On tient quelque chose \o/</p>
<h2 id="générer-la-bannière">Générer la bannière</h2>
<p>On peut maintenant générer la bannière complète :</p>
<ul>
<li>Les dimensions requises sont 1500x500</li>
<li>J’ai utilisé en fond une image avec “Suivez moi” et une flèche, dessinées dans GIMP</li>
</ul>
<p><img src="/assets/pictures/twitter-banner/fond-transparent.png" alt="" /></p>
<p>J’ai utilisé la librairie PIL pour générer dynamiquement le reste :</p>
<ul>
<li>Arrière plan avec un gradient (moche). Un bout de code trouvé sur le net et un peu d’aléatoire.</li>
<li>J’ai transformé les images de profil carrées via un masque rond et de l’antialiasing.</li>
<li>Du texte</li>
</ul>
<p>Quand on est satisfait, on génère un fichier PNG et on l’envoie via l’API twitter</p>
<p>Il y a quelques optimisations, par exemple je mets en cache les images déjà téléchargées. En pseudo-code, l’esprit est le suivant:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># Load environment variables from .env file
</span><span class="n">load_dotenv</span><span class="p">(</span><span class="s">"config.env"</span><span class="p">)</span>
<span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">"__main__"</span><span class="p">:</span>
<span class="n">auth</span> <span class="o">=</span> <span class="n">tweepy</span><span class="p">.</span><span class="n">OAuthHandler</span><span class="p">(</span><span class="n">os</span><span class="p">.</span><span class="n">getenv</span><span class="p">(</span><span class="s">"API_KEY"</span><span class="p">),</span> <span class="n">os</span><span class="p">.</span><span class="n">getenv</span><span class="p">(</span><span class="s">"API_KEY_SECRET"</span><span class="p">))</span>
<span class="n">auth</span><span class="p">.</span><span class="n">set_access_token</span><span class="p">(</span><span class="n">os</span><span class="p">.</span><span class="n">getenv</span><span class="p">(</span><span class="s">"ACCESS_TOKEN"</span><span class="p">),</span> <span class="n">os</span><span class="p">.</span><span class="n">getenv</span><span class="p">(</span><span class="s">"ACCESS_TOKEN_SECRET"</span><span class="p">))</span>
<span class="n">api</span> <span class="o">=</span> <span class="n">tweepy</span><span class="p">.</span><span class="n">API</span><span class="p">(</span><span class="n">auth</span><span class="p">)</span>
<span class="n">latest_followers</span> <span class="o">=</span> <span class="n">api</span><span class="p">.</span><span class="n">get_followers</span><span class="p">(</span><span class="n">count</span><span class="o">=</span><span class="mi">5</span><span class="p">,</span> <span class="n">skip_status</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
<span class="n">download_images</span><span class="p">(</span><span class="n">latest_followers</span><span class="p">)</span>
<span class="n">image</span> <span class="o">=</span> <span class="n">generate_header</span><span class="p">(</span><span class="n">latest_followers</span><span class="p">,</span> <span class="n">WIDTH</span><span class="p">,</span> <span class="n">HEIGHT</span><span class="p">)</span>
<span class="n">image</span><span class="p">.</span><span class="n">save</span><span class="p">(</span><span class="n">header_filename</span><span class="p">)</span>
<span class="n">api</span><span class="p">.</span><span class="n">update_profile_banner</span><span class="p">(</span><span class="n">header_filename</span><span class="p">)</span>
</code></pre></div></div>
<h2 id="déploiement">Déploiement</h2>
<p>On a maintenant besoin de faire la mise à jour à intervalle régulier.
Le plus simple, c’est cron, un outil linux qui sert à dire “execute telle opération à telle fréquence”</p>
<p>Mes machines pro/perso ne tournant pas en continu, il nous faut un serveur allumé en continu. J’ai des machines “labo” qui me servent pour ce genre de bricolages. J’utilise ansible pour gérer la configuration de mes outils. Oldie but very goodie.</p>
<p>J’ai donc déployé ce projet avec ansible sur la machine distante, en lui indiquant:</p>
<ul>
<li>de copier le code, les images et les fonts</li>
<li>d’installer les outils et librairies python nécessaires</li>
<li>d’ajouter un cron toutes les minutes</li>
</ul>
<div class="language-yml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nn">---</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Deploiement</span>
<span class="na">hosts</span><span class="pi">:</span> <span class="s">bot_machine</span>
<span class="na">become</span><span class="pi">:</span> <span class="s">yes</span>
<span class="na">become_user</span><span class="pi">:</span> <span class="s">root</span>
<span class="na">tasks</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s2">"</span><span class="s">Installe</span><span class="nv"> </span><span class="s">les</span><span class="nv"> </span><span class="s">dépendances"</span>
<span class="na">apt</span><span class="pi">:</span>
<span class="na">pkg</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">python3</span>
<span class="pi">-</span> <span class="s">python3-venv</span>
<span class="pi">-</span> <span class="s">vim</span>
<span class="na">state</span><span class="pi">:</span> <span class="s">latest</span>
<span class="na">update_cache</span><span class="pi">:</span> <span class="no">true</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s2">"</span><span class="s">Crée</span><span class="nv"> </span><span class="s">les</span><span class="nv"> </span><span class="s">répertoire</span><span class="nv"> </span><span class="s">nécessaires"</span>
<span class="na">file</span><span class="pi">:</span>
<span class="na">path</span><span class="pi">:</span> <span class="s2">"</span><span class="s">"</span>
<span class="na">state</span><span class="pi">:</span> <span class="s">directory</span>
<span class="na">with_items</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s1">'</span><span class="s">/twitter-banner'</span>
<span class="pi">-</span> <span class="s1">'</span><span class="s">/twitter-banner/cache'</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s2">"</span><span class="s">Envoie</span><span class="nv"> </span><span class="s">le</span><span class="nv"> </span><span class="s">code"</span>
<span class="na">copy</span><span class="pi">:</span> <span class="s">src= dest=</span>
<span class="na">with_items</span><span class="pi">:</span>
<span class="pi">-</span> <span class="pi">{</span> <span class="nv">src</span><span class="pi">:</span> <span class="s1">'</span><span class="s">twitter-header-bot-with-followers.py'</span><span class="pi">,</span> <span class="nv">dest</span><span class="pi">:</span> <span class="s1">'</span><span class="s">/twitter-banner/bot.py'</span> <span class="pi">}</span>
<span class="pi">-</span> <span class="pi">{</span> <span class="nv">src</span><span class="pi">:</span> <span class="s1">'</span><span class="s">twitter-header-follow.png'</span><span class="pi">,</span> <span class="nv">dest</span><span class="pi">:</span> <span class="s1">'</span><span class="s">/twitter-banner/twitter-header-follow.png'</span> <span class="pi">}</span>
<span class="pi">-</span> <span class="pi">{</span> <span class="nv">src</span><span class="pi">:</span> <span class="s1">'</span><span class="s">requirements.txt'</span><span class="pi">,</span> <span class="nv">dest</span><span class="pi">:</span> <span class="s1">'</span><span class="s">/twitter-banner/requirements.txt'</span> <span class="pi">}</span>
<span class="pi">-</span> <span class="pi">{</span> <span class="nv">src</span><span class="pi">:</span> <span class="s1">'</span><span class="s">fonts'</span><span class="pi">,</span> <span class="nv">dest</span><span class="pi">:</span> <span class="s1">'</span><span class="s">/twitter-banner/'</span> <span class="pi">}</span>
<span class="pi">-</span> <span class="pi">{</span> <span class="nv">src</span><span class="pi">:</span> <span class="s1">'</span><span class="s">config.env'</span><span class="pi">,</span> <span class="nv">dest</span><span class="pi">:</span> <span class="s1">'</span><span class="s">/twitter-banner/config.env'</span> <span class="pi">}</span>
<span class="pi">-</span> <span class="pi">{</span> <span class="nv">src</span><span class="pi">:</span> <span class="s1">'</span><span class="s">cron.sh'</span><span class="pi">,</span> <span class="nv">dest</span><span class="pi">:</span> <span class="s1">'</span><span class="s">/twitter-banner/cron.sh'</span> <span class="pi">}</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s2">"</span><span class="s">Installe</span><span class="nv"> </span><span class="s">les</span><span class="nv"> </span><span class="s">dépendances</span><span class="nv"> </span><span class="s">du</span><span class="nv"> </span><span class="s">projet</span><span class="nv"> </span><span class="s">dans</span><span class="nv"> </span><span class="s">un</span><span class="nv"> </span><span class="s">environnement</span><span class="nv"> </span><span class="s">virtuel"</span>
<span class="na">pip</span><span class="pi">:</span>
<span class="na">requirements</span><span class="pi">:</span> <span class="s">/twitter-banner/requirements.txt</span>
<span class="na">virtualenv</span><span class="pi">:</span> <span class="s">/twitter-banner/venv</span>
<span class="na">virtualenv_command</span><span class="pi">:</span> <span class="s">/usr/bin/python3 -m venv</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s2">"</span><span class="s">Ajoute</span><span class="nv"> </span><span class="s">notre</span><span class="nv"> </span><span class="s">bot</span><span class="nv"> </span><span class="s">dans</span><span class="nv"> </span><span class="s">un</span><span class="nv"> </span><span class="s">cron"</span>
<span class="s">ansible.builtin.cron</span><span class="pi">:</span>
<span class="na">name</span><span class="pi">:</span> <span class="s2">"</span><span class="s">Bot"</span>
<span class="na">minute</span><span class="pi">:</span> <span class="s2">"</span><span class="s">*"</span>
<span class="na">hour</span><span class="pi">:</span> <span class="s2">"</span><span class="s">*"</span>
<span class="na">job</span><span class="pi">:</span> <span class="s2">"</span><span class="s">/bin/bash</span><span class="nv"> </span><span class="s">/twitter-banner/cron.sh"</span>
</code></pre></div></div>Projet du weekend: je me suis bricolé une bannière Twitter dynamique.Documentez les étapes manuelles de vos scripts2022-03-05T13:08:00+00:002022-03-05T13:08:00+00:00https://www.keiruaprod.fr/blog/2022/03/05/documentez-vos-scripts-bash<p>La petite astuce qui va vous faire kiffer de lancer vos scripts: faites des scripts bash qui vous indiquent les étapes suivantes.</p>
<p>Quand il y a des étapes manuelles pas automatisables à prévoir, c’est mieux qu’une doc.
Si c’est automatisable… vous complétez le script hein ;)</p>
<p>Voici quelques exemples :</p>
<h2 id="lancer-tous-vos-services">Lancer tous vos services</h2>
<p>Une fois que tous les services sont lancés, on est perdu car on ne sait pas forcément sur quels ports ils tournent. Un petit récap bien pratique à l’usage !</p>
<p>Docker en noie certains au milieu des logs, et certains trucs sont sur une URL donnée qui n’apparait pas dans les logs.</p>
<p><img src="/assets/pictures/document-your-bash/make-run.png" alt="" /></p>
<h2 id="création-de-machines-à-la-volée">Création de machines à la volée</h2>
<p>On crée des containers à la volée pour certaines taches avec des noms complexes.</p>
<ul>
<li>se connecter dessus c’est pénible -> “voila comment faire”</li>
<li>on se souvient jamais de la commande pour supprimer la machine -> “voila comment faire”</li>
</ul>
<p><img src="/assets/pictures/document-your-bash/make-restore.png" alt="" /></p>
<p>La raison pour laquelle on ne va pas plus loin dans l’automatisation:</p>
<ul>
<li>on fait plein de trucs manuels sur ces machines</li>
<li>on a tout un tas de scripts d’ETL qui importent des fichiers mal formés générés à la main, ce qui fait que chaque semaine ca casse pour une raison différente. Si on pouvait automatiser ou dégager ça… on le ferait avec joie.</li>
</ul>
<h2 id="import-dune-base-locale">Import d’une base locale</h2>
<p>Après import d’un base local, il faut peut-être:</p>
<ul>
<li>relancer les containers</li>
<li>éventuellement à remettre à zéro les mots de passe</li>
</ul>
<p>Le script bash fait ce rappel et liste les commandes à jouer si besoin</p>
<p><img src="/assets/pictures/document-your-bash/make-temp-machine.png" alt="" /></p>
<p>(Dans cet exemple, on pourrait automatiser ces 2 commandes supplémentaires dans le script, mais on ne le fait pas pour la mauvaise raison que tout le monde n’utilise pas docker).</p>
<h2 id="comment-faire-">Comment faire ?</h2>
<ul>
<li>Mettez les infos dans un cat multiligne</li>
<li>Allez-y sur les emoji ;)</li>
</ul>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">cat</span> <span class="o"><<</span> <span class="no">EOF</span><span class="sh">
🎉 Le déploiement est terminé 🎉
Vous pouvez maintenant:
- ✈️ Aller sur la machine:
clever ssh --alias </span><span class="nv">$IMPORT_APP_NAME</span><span class="sh">
- 🔨 Jouer un script d’import, par ex:
cd app_* && ./scripts/imports-asp.sh
- 🍺 Supprimer la machine:
clever delete --alias </span><span class="nv">$IMPORT_APP_NAME</span><span class="sh"> --yes
</span><span class="no">EOF
</span></code></pre></div></div>La petite astuce qui va vous faire kiffer de lancer vos scripts: faites des scripts bash qui vous indiquent les étapes suivantes.Des hooks pre-commit pour git avec… pre-commit2022-02-23T10:23:00+00:002022-02-23T10:23:00+00:00https://www.keiruaprod.fr/blog/2022/02/23/pre-commit<p>Dans plein de projets, il y a des hooks pre-commit pour gérer le linter et empêcher d’ajouter dans le dépot du code invalide selon le standard du projet. Ces scripts sont généralement des fichiers bash qu’on appelle dans <code class="highlighter-rouge">.git/hooks/pre-commit</code>.</p>
<p>Inconvénient de cette approche:</p>
<ul>
<li>Ces fichiers ne sont pas versionnés, ou mal, car ce qui est dans <code class="highlighter-rouge">.git</code> n’est pas versionné</li>
<li>Tout le monde écrit son propre script, ou non, ou doit le copier depuis ailleurs</li>
<li>En leur absence, on fait échouer la CI, on rale car elle prend du temps et qu’il faut recommencer</li>
</ul>
<h2 id="une-alternative">Une alternative</h2>
<p><a href="https://pre-commit.com/">pre-commit</a> est pas mal cool pour gérer les outils de pre-commit:</p>
<ul>
<li>on l’installe localement via <code class="highlighter-rouge">pip install pre-commit</code></li>
<li>on liste ce qu’on veut dans un .yml. Notre projet utilise <strong>black</strong>, <strong>isort</strong>, et <strong>flake8</strong>, ce sont des outils utiles pour Python.</li>
<li>On install les scripts à jouer via <code class="highlighter-rouge">pre-commit install</code></li>
</ul>
<p>Chaque commit peut être refusé s’il ne respecte pas les règles:</p>
<p><img src="/assets/pictures/pre-commit/pre-commit-result.png" alt="" /></p>
<p>la valeur ajoutée c’est que:</p>
<ul>
<li>c’est versionné (dont tout le monde a les même hooks)</li>
<li>on ne casse plus la CI pour des erreurs de linter</li>
</ul>
<h2 id="configuration-des-différents-outils">Configuration des différents outils</h2>
<p>Sous Python, il n’y a pas de consensus dans l’écosystème sur la configuration et c’est toujours la galère de se souvenir quelle configuration va dans quel fichier. On va aussi voir que la longueur max est dupliquée dans les 3 outils…. Voici une config que j’utilise :</p>
<h3 id="black">Black</h3>
<p>black est un linter configuré via <code class="highlighter-rouge">pyproject.toml</code></p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[tool.black]
line-length = 119
</code></pre></div></div>
<h3 id="isort-et-flake8">isort et flake8</h3>
<p>isort et flake8 sont configurés via <code class="highlighter-rouge">setup.cfg</code></p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code># setup.cfg
# - https://timothycrosley.github.io/isort/
[isort]
combine_as_imports = True
ensure_newline_before_comments = True
force_grid_wrap = 0
include_trailing_comma = True
lines_after_imports = 2
line_length = 119
multi_line_output = 3
use_parentheses = True
# - https://www.flake8rules.com
[flake8]
max-line-length = 119
</code></pre></div></div>Dans plein de projets, il y a des hooks pre-commit pour gérer le linter et empêcher d’ajouter dans le dépot du code invalide selon le standard du projet. Ces scripts sont généralement des fichiers bash qu’on appelle dans .git/hooks/pre-commit.Redirection HTTP vers HTTPS2022-02-04T17:37:00+00:002022-02-04T17:37:00+00:00https://www.keiruaprod.fr/blog/2022/02/04/rediriger-http-vers-https<p>Je veux régulièrement rediriger tout le trafic HTTP vers la version HTTPS, voici la version <code class="highlighter-rouge">.htaccess</code>:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>RewriteEngine On
RewriteCond %{HTTPS} off
RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI} [R=301,L]
</code></pre></div></div>Je veux régulièrement rediriger tout le trafic HTTP vers la version HTTPS, voici la version .htaccess: