Philip Cristiano2024-03-26T00:26:24+00:00https://philipcristiano.com/atom.xml2024-03-26T00:26:24+00:002024-03-26T00:26:24+00:00Philip Cristianohttps://philipcristiano.com/notes/2024/7be5151d-47f9-4fb7-972f-0c69fb27400f/
<p>Having an EV charger at vacation house is such a quality-of-life improvement. We didn’t seek out a house with a charger, but now I think we always will. </p>
2024-03-18T22:06:36+00:002024-03-18T22:06:36+00:00Philip Cristianohttps://philipcristiano.com/notes/2024/22765d6f-daed-4754-90d8-05a86798d25e/
<p>So I don't forget...</p>
<p>As seen on <a href="https://www.reddit.com/r/tailwindcss/comments/kj146o/comment/ggu0hd1/">Reddit</a> ...:</p>
<p>Don't forget to set the viewport when starting a new site</p>
<p><code><meta name="viewport" content="width=device-width, initial-scale=1.0"></code> </p>
2024-03-16T23:11:18+00:002024-03-16T23:11:18+00:00Philip Cristianohttps://philipcristiano.com/notes/2024/ae2ac49a-ecf1-4885-8302-4526a69a5915/
<p>Been having a good time lately playing <a href="https://www.nytimes.com/games/strands">Strands</a></p>
2024-02-26T18:19:36+00:002024-02-26T18:19:36+00:00Philip Cristianohttps://philipcristiano.com/notes/2024/seitan-vegeballs/
<p>By far the best vegan meatballs we've made have been the <a href="https://itdoesnttastelikechicken.com/vegan-italian-seitan-meatballs/">It Doesn't Taste Like Chicken Vegan Italian Meatballs</a>. Previous recipes we've tried have been black bean based. They tend not to hold together well enough. These ones don't fall apart!</p>
Automated Nomad Docker Image Updates2024-02-20T14:50:26+00:002024-02-20T14:50:26+00:00Philip Cristianohttps://philipcristiano.com/posts/2024/nomad-docker-update/
<p>For a few years now I've grumbled at updating Docker images in my Nomad homelab. Nomad isn't as popular as Kubernetes or Docker Compose and isn't supported in Dependabot.</p>
<p>Eventually I found <a href="https://github.com/hashicorp/nomad/issues/13061#issuecomment-1192472280">this comment</a></p>
<p><img src="https://philipcristiano.com/posts/2024/nomad-docker-update/comment.png" alt="Using a Dockerfile for a Nomad Docker image" /></p>
<p>I didn't think this was a great solution to my problem as I split up the registry from the repo/image so that I can pull images from my own repository. This solves the problem of Dependabot updating images though!</p>
<p>There was also an annoyance that I still need to copy these images into my Docker Registry. I've been using <a href="https://github.com/regclient/regclient">regclient</a>'s <code>regctl image copy</code> command as part of a <a href="https://github.com/philipcristiano/nixos-cluster-config/blob/c60e0ab1b1536db9857a137e7947df5a56336bb8/services/regctl/regctl.nomad">Nomad job</a> that makes this a bit easier.</p>
<h3 id="tada">Tada</h3>
<p>If the Dockerfile now has a <code>FROM [IMAGE]</code> in the service directory the deploy process now looks like:</p>
<ul>
<li><code>awk</code> the image out <code>awk '/FROM/ {print $2}' Dockerfile</code></li>
<li>Dispatch the <code>regctl</code> job with the <code>IMAGE</code></li>
<li>Deploy the service job passing in the <code>IMAGE</code> as a variable.</li>
</ul>
<p><a href="https://github.com/dependabot/dependabot-core/issues/2178">Dependabot</a> doesn't seem to do great with monorepos without lots of copying.</p>
<p><a href="https://github.com/apps/renovate">Renovate</a> does though!</p>
<p><a href="https://github.com/philipcristiano/nixos-cluster-config/blob/424743df6fca82ff0b34908eb0550e66ea10392d/services/hello_idc/deploy.sh">Finally, a service that can autoupdate!</a></p>
<p>Merges on Github do not automatically deploy to my homelab so the final deploy takes 2 more commands to deploy (<code>git pull</code> and <code>bash deploy</code>) but this no longer requires any manual commits!</p>
2024-02-15T16:17:43+00:002024-02-15T16:17:43+00:00Philip Cristianohttps://philipcristiano.com/notes/2024/llm/
<p>I recently tried running <a href="https://github.com/bentoml/OpenLLM">OpenLLM</a> in my homelab. It was super nice to setup and run but pretty poor performance running models on CPU. A <a href="https://huggingface.co/spaces/zhangtao103239/NeuralHermes-2.5-Mistral-7B-GGUF-Chat">GGUF model</a> on HF was running pretty quickly but OpenLLM doesn't seem to support GGUF.</p>
<p>Running <a href="https://github.com/oobabooga/text-generation-webui">text-generation-webui</a> with a <a href="https://huggingface.co/TheBloke/Mistral-7B-Instruct-v0.2-GGUF">Mistral GGUF</a> model has been really nice. The performance on an <a href="https://www.amd.com/en/products/apu/amd-ryzen-3-5300u">AMD Ryzen 3 5300U</a> is super usable!</p>
Towards a more useful Matrix Synapse healthcheck2024-02-13T00:00:00+00:002024-02-13T00:00:00+00:00Philip Cristianohttps://philipcristiano.com/posts/2024/synapse-healthcheck/
<p>I've been running <a href="https://matrix-org.github.io/synapse/latest/welcome_and_overview.html">Synapse</a> for about a year at this point and it's been fairly consistently the software that gives me the most trouble for my homelab.</p>
<p>A part I've started to address recently is the ineffectiveness of the <a href="https://matrix-org.github.io/synapse/latest/reverse_proxy.html#health-check-endpoint">healthcheck endpoints</a>. It always works! Which is a problem when the service isn't working and the healthcheck says it is. This is commonly when my Postgres server moves for some reason, Synapse never reconnects. It's a <a href="https://github.com/matrix-org/synapse/issues/11473">known</a> <a href="https://github.com/element-hq/synapse/issues/11473">issue</a>.</p>
<p>I've worked around this with a healthcheck script that will probe the Synapse API</p>
<pre data-lang="bash" style="background-color:#2b303b;color:#c0c5ce;" class="language-bash "><code class="language-bash" data-lang="bash"><span style="color:#bf616a;">curl -fv</span><span> http://127.0.0.1:{{ PORT }}/_matrix/client/v3/publicRooms</span><span style="color:#bf616a;"> -H </span><span>"</span><span style="color:#a3be8c;">Authorization: Bearer {{ TOKEN }}</span><span>"
</span></code></pre>
<p>The <code>TOKEN</code> needs to be a <a href="https://webapps.stackexchange.com/questions/131056/how-to-get-an-access-token-for-element-riot-matrix">user token</a> which also gives this a bootstrapping problem where this check can only be in place after the server works. And a user changing their password will invalidate this token and take the server offline...</p>
<p>After a few days though it's solved my problem of Synapse breaking but pretending to be fine.</p>
2024-02-11T21:07:38+00:002024-02-11T21:07:38+00:00Philip Cristianohttps://philipcristiano.com/notes/2024/electrs/
<p>New <a href="https://hub.docker.com/layers/philipcristiano/electrs/0.10.3/images/sha256-911e3e5c0e37e1f0795222e84ffd59d764467cbeaec5bd0c14299e8043c44105?context=explore">Docker image for electrs</a> with the <a href="https://github.com/romanz/electrs/blob/master/RELEASE-NOTES.md#0103-feb-10-2024">0.10.3</a> release.</p>
2024-02-08T00:45:45+00:002024-02-08T00:45:45+00:00Philip Cristianohttps://philipcristiano.com/replies/2024/19-first-reply/
<a class="u-in-reply-to" href="https://aaronparecki.com/2018/06/30/11/your-first-webmention">In reply to</a><p>Trying out this guide to sending webmentions</p>
2024-02-06T15:56:59+00:002024-02-06T15:56:59+00:00Philip Cristianohttps://philipcristiano.com/notes/2024/15-note/
<p>Adding <a href="https://philipcristiano.com/notes/">notes</a> as something separate from posts. As I start to add <a href="https://indieweb.org/note">IndieWeb</a> things!</p>
Send Webmentions with Github Actions2024-02-05T00:00:00+00:002024-02-05T00:00:00+00:00Philip Cristianohttps://philipcristiano.com/posts/2024/publish-webmentions-with-github-actions/
<p>As I start <a href="https://philipcristiano.com/posts/2024/webmentions/">working with webmentions</a> I needed to find a way to publish webmentions as part of the build/release process for this site. I'm currently using <a href="https://docs.github.com/en/actions">Github Actions</a> to build the site and upload to <a href="http://netlify.com/">Netlify</a>.</p>
<p><a href="https://webmention.app/">webmention.app</a> came up frequently when I searched around for how to publish webmentions. It supports RSS/Atom as a feed, although the docs suggest using <a href="https://ifttt.com">IFTTT</a> to trigger webhooks. Github Actions can do that though! For whatever reason webmention.app didn't seen to find any links in my feed. As I was trying to figure out why by <a href="https://webmention.app/docs#using-the-command-line">using the command line</a> I discovered that the CLI version was able to find the links in my feed!</p>
<p>Adding this as a step post-release</p>
<pre data-lang="yaml" style="background-color:#2b303b;color:#c0c5ce;" class="language-yaml "><code class="language-yaml" data-lang="yaml"><span>- </span><span style="color:#bf616a;">name</span><span>: </span><span style="color:#a3be8c;">Send Webmentions
</span><span> </span><span style="color:#bf616a;">run</span><span>: </span><span style="color:#b48ead;">|
</span><span style="color:#a3be8c;"> npm install @remy/webmention
</span><span style="color:#a3be8c;"> npx webmention ${{ secrets.WEBMENTION_TARGET_URL }} --limit=0 --send
</span></code></pre>
<p>In my pull-requests I have a variation of this, removing <code>--send</code> and using the temporary Netlify URL for the PR so I can see what webmentions would be sent.</p>
<p>As part of using my atom feed for this I now only include the last 10 posts in my feed to avoid sending lots of old webmentions, most of which didn't seem to work as the links are dead.</p>
Starting Webmentions2024-02-04T00:00:00+00:002024-02-04T00:00:00+00:00Philip Cristianohttps://philipcristiano.com/posts/2024/webmentions/
<p>Is anyone using <a href="https://indieweb.org/Webmention">webmentions</a>? I've added <a href="https://webmention.io/">Webmention.io</a> for hosting my webmentions at the moment as this is currently a static site. If you're using them please try and mention this page and I can hopefully see in my RSS reader!</p>
<h2 id="testing-things">Testing things</h2>
<p><a href="https://webmention.rocks/test/1">Webmention test 1</a></p>
<p>Also trying to send them as part of my site build process. Maybe <a href="https://webmention.rocks/update/1">this</a> will work?</p>
<p><a href="https://webmention.rocks/update/1/part/2">Second part</a>!</p>
Prefetching Docker Images2023-10-11T00:00:00+00:002023-10-11T00:00:00+00:00Philip Cristianohttps://philipcristiano.com/posts/2023/10/15-prefetching-docker-images/
<p>While running Nomad I've been running into a bootstrapping/critical path problem. I have a Docker Registry running in the cluster and pulling an image requires:</p>
<ul>
<li><a href="https://docs.docker.com/registry/">Docker Registry</a></li>
<li><a href="https://traefik.io/">Traefik</a></li>
<li><a href="https://github.com/mayuresh82/gocast">GoCast</a></li>
<li><a href="https://min.io/">Minio</a></li>
</ul>
<p>The Registry is required to serve the image.</p>
<p>Traefik routes the requests to the Registry as well as requesting Lets Encrypt certificates</p>
<p>GoCast announces the floating IP for Traefik</p>
<p>Minio stores the images for the Registry</p>
<h2 id="problems-updating-images">Problems updating images</h2>
<p>Separate from bootstrapping, just updating the image of many of these will require everything to already be running, just to pull the next image. There is an <a href="https://github.com/hashicorp/nomad/issues/6380">open bug</a> to address this in Nomad, but it doesn't seem like it's going to be resolved anytime soon.</p>
<p>When updating Traefik I run into a condition that GoCast has created the floating IP addr on the host but Traefik isn't running. The floating IP won't work while Traefik is running-but-not-serving. GoCast BGP is working correctly in that the floating IP is not accounced to the network, but the updating host still can't reach the other-host instances of the floating IP. I'm not sure if <a href="https://github.com/mayuresh82/gocast/issues/25">leaving the addr in place is a feature or a bug</a>.</p>
<p>A way around this would be to run multiple instances of Traefik on each host. As currently setup though I need to bind multiple instances of Traefik to the same ports and <a href="https://github.com/traefik/traefik/issues/9823">SO_REUSEPORT isn't supported</a>. With GoCast I could map the floating IP ports to container ports and not require host networking (thus avoiding the port collision) but that may be quite burdensome to manage. I also haven't tried running multiple ports with GoCast NAT'ing.</p>
<h2 id="solving-part-of-the-problem">Solving part of the problem</h2>
<p>For the Traefik case of not being able to pull the image there are some workarounds. Manually pulling, or <a href="https://github.com/hashicorp/nomad/issues/6380#issuecomment-1121425133">system batch jobs</a> could solve this but is fairly manual.</p>
<p><a href="https://github.com/regclient/regclient">regclient</a> has a daemon mode that can pull/sync images to registries, but it <a href="https://github.com/regclient/regclient/issues/568">doesn't support pushing to a Docker Engine</a>.</p>
<h2 id="docker-prefetch-image">Docker Prefetch Image</h2>
<p>I've started on a <a href="https://github.com/philipcristiano/docker-prefetch-image/">tool to prefetch Docker images</a> based on a config file. Updating the config file appropriately to match the image used in Nomad Jobs is still a problem. This uses the <a href="https://docs.docker.com/engine/api/v1.43/#tag/Image/operation/ImageCreate">Docker Engine API</a> via the Rust <a href="https://docs.rs/docker-api/latest/docker_api/index.html">docker_api</a> crate to pull the image to the host.</p>
<p>Nomad Consul Template though can populate the config file from Consul to avoid manual file updates thought which isn't terrible. I'm not sure if there is a nice way to integrate with the Nomad API to watch what images might be needed and pull the in in advance of any job using it.</p>
<p>This has solved my case for updating parts of the critical path of Docker Image hosting. It doesn't fully solve the bootstrapping case though where none of the services are running yet. An idea though is to extend the config/API calls to have the "expected" image tag Nomad would look for and a "source". If the "expected" image cannot be pulled, try the "source" and tag it locally as the "expected" tag. This would allow prefetching all images required for bootstrapping the system!</p>
What I want for a Queue System2023-07-25T00:00:00+00:002023-07-25T00:00:00+00:00Philip Cristianohttps://philipcristiano.com/posts/2023/07/25-what-i-want-for-a-queue-system/
<p>I've been interested in queue systems since first learning about and using
RabbitMQ in ... 2009 (woah... it's been a minute). What I've learned through
most of this is that:</p>
<ul>
<li>People won't care as much as me</li>
<li>You can't make people care</li>
<li>If they don't care then it's even easier to make mistakes.</li>
</ul>
<p>Redis is quite popular as a queue system and I've joined multiple
companies/teams where Redis and <a href="https://python-rq.org/">Python-RQ</a> were used
for async tasks. Redis is wonderful and is a great solution to many problems
(including async tasks!) but in the cases I have seen, it's been mostly an
incomplete, improper solution.</p>
<p>Google Pub/Sub is pretty wonderful generally and the pattern I love most is
combining Pub/Sub and Cloud Run for HTTP delivery of events. There are some
limitations with this pattern but I love most of all the removal of
many problems developers can cause.</p>
<p>Event -> HTTP -> Service makes handling events much easier.</p>
<ul>
<li>It's difficult to run tasks for hours from a single HTTP request</li>
<li>Handling of events requires little knowledge of a particular library</li>
<li>Much of the complexity doesn't need to be in the app</li>
<li>Removing the complexity from the app makes it easier for more apps to use it, without lots of work</li>
</ul>
<p>I can't run Pub/Sub and Cloud Run at my house though.</p>
<h3 id="what-i-want-from-a-queue-system">What I want from a queue system</h3>
<ul>
<li>HTTP and/or GRPC submission of events to the queue system</li>
<li>HTTP and/or GRPC push to a service</li>
<li>Possible to run in a home environment, but not the-worst-idea in a larger environment</li>
<li>Back-pressure. When too many events are in the system the publishing will slow down.</li>
<li>Easy to run and not worry about it</li>
<li>Small idle footprint in memory/CPU</li>
<li>Horizontal scalability. If it's ever used in production somewhere, adding capacity should be easy.</li>
</ul>
<h3 id="what-i-don-t-need-from-a-queue-system">What I don't need from a queue system</h3>
<ul>
<li>Super high throughput. 10k events per second is wonderful... but if it can do 100 and scales out, I'm not too worried.</li>
<li>Perfect durability. I'll assume that at some point data might be lost and those outliers are OK.</li>
<li>Perfect deliverability. I normally add end-to-end checks for data that a dropped event will only cause a delay, not a consistency problem.</li>
</ul>
<h3 id="what-to-do-about-it">What to do about it</h3>
<p>I haven't found exactly what I'm looking for in other systems. Since I'm mostly
scratching a self-hosting itch at the moment I'm looking to throw together a
sample system to solve my problem, never expecting it to go beyond that
(although maybe it will be useful for someone else?)</p>
<p>As I learn Rust, connecting <a href="https://docs.rs/axum/latest/axum/">Axum</a>, <a href="https://doc.rust-lang.org/rust-by-example/std_misc/channels.html">Rust
channels</a> and
<a href="https://docs.rs/reqwest/latest/reqwest/">Reqwest</a> should get me pretty far.</p>
<p>And the real goal here is to use a simple enough system similar to what I'd
recommend for production use cases with cloud services (and not my homegrown
thing).</p>
Nomad Events Logger2023-07-15T00:00:00+00:002023-07-15T00:00:00+00:00Philip Cristianohttps://philipcristiano.com/posts/2023/07/15-nomad-events-logger/
<p>As part of learning Rust I built a tool to read events from the <a href="https://developer.hashicorp.com/nomad/tutorials/integrate-nomad/event-stream">Nomad Events
API</a>
and log them to stdout. This allows an easy, low-resource way to pull Nomad
cluster events into your log processing stream.</p>
<p>Low-resource as in ~4MB of memory for the Docker container!</p>
<p><a href="https://github.com/philipcristiano/nomad-events-logger">Nomad Events Logger</a>
is deployable as a Docker image, and if I get around to it, a native binary as
well. At the moment I run ~everything in Docker in my Nomad cluster so let me
know if you want other formats.</p>
Paperless-ngx Celery won't consume documents2023-06-23T00:00:00+00:002023-06-23T00:00:00+00:00Philip Cristianohttps://philipcristiano.com/posts/2023/06/23-paperless-ngx-celery-wont-consume/
<p>When running <a href="https://docs.paperless-ngx.com/">Paperless-ngx</a> I ran into a problem where the Celery process in Docker (as part of supervisord) would start, supervisor would report it running, but the Celery process appeared to do nothing.</p>
<p>The last related lines I would see were:</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>INFO success: celery entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
</span><span>[INFO] [paperless.management.consumer] Adding [REDACTED] to the task queue.
</span></code></pre>
<p>I'm not sure what part of Celery does this, maybe it's just Paperless? But eventually I found a <code>.__celery.lock</code> file in the Paperless data directory. Removing that allowed everything to work again.</p>
<p>This was likely caused with Nomad terminating the process and the lock file not getting cleaned up. I now have my Nomad job remove <code>.*.lock</code> files before starting Paperless.</p>
More Declarative Containers with NixOS2021-04-28T00:00:00+00:002021-04-28T00:00:00+00:00Philip Cristianohttps://philipcristiano.com/posts/2021/04/28-more-nixos-declarative-containers/
<p>In newer versions of NixOS it's possible to use Docker directly in your <a href="https://nixos.wiki/wiki/NixOS_Containers"><code>/etc/nixos/configuration.nix</code></a>!</p>
<p>Example from that page:</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span> { config, pkgs, ... }:
</span><span> {
</span><span> config.docker-containers = {
</span><span> hackagecompare = {
</span><span> image = "chrissound/hackagecomparestats-webserver:latest";
</span><span> ports = ["127.0.0.1:3010:3010"];
</span><span> volumes = [
</span><span> "/root/hackagecompare/packageStatistics.json:/root/hackagecompare/packageStatistics.json"
</span><span> ];
</span><span> cmd = [
</span><span> "--base-url"
</span><span> "\"/hackagecompare\""
</span><span> ];
</span><span> };
</span><span> };
</span><span> }
</span></code></pre>
<p>I've moved to this format as it's a bit cleaner and simpler to use for syncing container images than <code>rkt</code> wound up being.</p>
Risks With Git Tag Triggered Deploys2020-04-16T00:00:00+00:002020-04-16T00:00:00+00:00Philip Cristianohttps://philipcristiano.com/posts/2017/04/in-progres-risks-with-git-tag-triggered-deploys/
<p>Git workflows can come in many flavors. Once the code hits a continuous
integration system your workflow will need to trigger a deploy to production. A
common way of handling this is to create a Git tag that will trigger the deployment.
Using a Git tag to trigger the deployment can lead to increased risk against safely
deploying your code.</p>
<p>These risks can be countered in multiple ways, but these are patterns I've seen
in the deployment process for various services.</p>
<h2 id="tags-can-be-pushed-by-anyone-with-write-access">Tags can be pushed by anyone with write access</h2>
<p>Your process may allow anyone trigger a deploy to production. In many ways this
is a good thing. In GitHub though, certain branches can be protected in order
to enforce a certain workflow such as requiring each pull request receive
approval from 1 other person.</p>
<p>Tags in Github do now have such a protection. Anyone with write access could push
a tag, bypassing the Github workflow.</p>
<h2 id="tags-do-not-have-an-order">Tags do not have an order</h2>
<p>Any commit in the repository can be tagged. There is little difference (to Git)
between a tag on the latest commit and a tag on a commit from 3 months ago. If
your process relies on some semantic meaning for these tags you will have to
encode that information and handle it in your deployment automation.</p>
Declarative Containers with NixOS2018-10-11T00:00:00+00:002018-10-11T00:00:00+00:00Philip Cristianohttps://philipcristiano.com/posts/2018/10/nixos-declarative-containers/
<p>I spent some time recently attempting to setup some software on a NixOS system I have at home. It looks like declarative containers were removed in an earlier version of NixOS as they weren't quite ready for use. After <a href="https://github.com/NixOS/nixpkgs/issues/37553">some searching</a> I was able to find an example with <code>rkt</code>!</p>
<p>Setting up a container can be as simple as adding this to your <code>/etc/nixos/configuration.nix</code>:</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>virtualisation.rkt.enable = true;
</span><span>
</span><span>systemd.services."rkt-nginx" = {
</span><span> description = "Nginx (rkt)";
</span><span> wantedBy = [ "multi-user.target" ];
</span><span> serviceConfig = {
</span><span> Slice = "machine.slice";
</span><span> ExecStart = ''\
</span><span> ${pkgs.rkt}/bin/rkt run --insecure-options=image \
</span><span> --net=host \
</span><span> docker://nginx
</span><span> '';
</span><span> KillMode = "mixed";
</span><span> Restart = "always";
</span><span> };
</span><span>};
</span></code></pre>
OmniosCE Networking on OVH2018-03-12T00:00:00+00:002018-03-12T00:00:00+00:00Philip Cristianohttps://philipcristiano.com/posts/2018/03/omniosce-networking-on-ovh/
<p>I recently found that my DHCP leasing on OVH was unreliable. The address worked
at one point, but after a few months/reboots I found that the instance could
not longer obtain a lease. After a few attempts to release/renew, I decided to set a static IP.</p>
<p>The <a href="http://wiki.omniosce.org/GeneralAdministration">General Administration
page</a> has general information
about setting this. The IP from your <a href="https://ca.ovh.com/manager/">OVH control
panel</a> for the specific server is needed. From
that information the routing gateway can be determined.</p>
<p>The gateway is the same as the IP of the server with the last octet replaced
with <code>254</code>. If the IP is <code>10.2.3.4</code>, the gateway is <code>10.2.3.254</code>. To set this on the host:</p>
<pre style="background-color:#2b303b;color:#c0c5ce;"><code><span>ipadm create-addr -T static -a $SERVER_IP/32 ixgbe0/v4
</span><span>route -p add default $GATEWAY_IP
</span></code></pre>