Last year at the Open Science Retreat (#OSR24NL) I have been introduced to nanopubs by @egonw and created my first nanopub declaring citations for a paper using CiTOs (citation ontologies).

Now, travelling to #OSR25CH, due to issues with the train network foreseen with plenty of time, I used the opportunity and created a new one ( https://w3id.org/np/RAicq7k9QHX8EG8ho7Baib9GxHsU18O0tyFCZ4tbbGGPA ) for my latest publication on teaching #Snakemake on #HPC systems.

The teaching material is — again — in desperate need of additions and overhaul, but that is for another day.

This feels pretty huge! The #Snakemake docs now have an LLM powered chatbot (provided for free by
@gurubaseio). It can be seen as a replacement for the normal doc search. This can help a lot, in particular when you are unsure about the keywords to use for a doc search.

There have been several new #Snakemake releases in the last days. Apart from various fixes, the most important new feature is the ability to annotate input file access patterns. This enables storage backends tailor their provisioning mechanism for each file: https://snakemake.readthedocs.io/en/stable/snakefiles/storage.html#access-pattern-annotation

does "snakemake --config config=in.yml -n --export-cwl "out.cwl works ? I don't have any ouput ( snakemake 9.1.3)

> Removing output files of failed job product2animation since they might be corrupted

Thanks for nothing #SnakeMake for deleting a very precious output file, resulting from an hours/days-long process because it *might* be corrupted

I though we were past this nonsense of build-systems deleting files at will, especially with a science-focused project such as snakemake!?

Ugh, #SnakeMake apparently has a hard requirement of all input (and previously-made output-) files to be present on-disk. Can't even do a dry-run `snakemake -n`.

I want a build system that:

- is as powerful and flexible as #SCons
- as readable and concise as #SnakeMake
- has a fricking progress bar+ETA
- is #datalad / #gitannex agnostic (knows that files can be fetched from elsewhere
- remembers how long building things takes
- balances that to decide if rebuilding locally instead of fetching gigabytes via slow internet is favorable
- integrates well with #nix for reproducibility

#SnakeMake has some nasty caching going on, where it sometimes reuses old versions of code you have long overwritten. Removing the .snakemake/ folder solves it. Also this behavior that if there's an error in your notebook, Snakemake will just *not* save the processed notebook so you can't actually have a look at the error.
Makes it kinda annoying to debug.

It is beyond me why build systems like #make #scons #snakemake et al. all don't have a flag like `--pattern '*2022-*'` that will make only those files it knows how to make matching a given pattern.

(using shell brace expansion and/or globbing is risky and doesn't work fully if not all the files exist yet)

Today is the day of closed pull request for #Snakemake. The #SnakemakeHackathon2025 participants worked at full speed!

We decided to write a white-paper summarizing our achievements rather than posting individual things. Suffice to say, that also the documentation made a great leap towards better readability!

The second day of the #SnakemakeHackathon2025 is over. Today I have merged 11 fantastic pull requests in the in main snakemake repo alone. And we have our first #Snakemake release during the hackathon, adding integration with xonsh for scripting and helpers for extracting params from input files.

I've used #Snakemake for a long time, but clearly I haven't been following their updates closely. This whole list of semantic helpers are great and I wish I had known about them earlier.

https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html#semantic-helpers

#Snakemake has an update best practices guide: https://snakemake.readthedocs.io/en/stable/snakefiles/best_practices.html

#Snakemake 8.29 has been released. It provides various improvements to the integrated reporting, as well as many small bug fixes: https://snakemake.readthedocs.io/en/stable/project_info/history.html

There are many HPC admins who prohibit using considerable CPU time on login nodes. This is understandable.

I want to take this opportunity to provide a data point. My student has measured the accumulated CPU time (user + system) for a 9 h (precise: 33343 s) run of a Snakemake workflow. It was 225 s or about 0.67 % - including jobs which were carried out on this login node, e.g. `mv`, `ln` or download of data.

There is certainly room for improvements. There will ever be room for improvements.

But my dear fellow admins: Running Snakemake on login nodes as a shepherd of jobs, will impair nobodies work.

Over time, I will certainly gather more and different statistics. And will invest time in necessary improvements. Regarding CPU time for checking job status, however, I believe to have demonstrated, that this is a pretty high hanging fruit.

Tomorrow? Trip to Dresden. Towards a course, I give about reproducible computing with #Snakemake on #HPC clusters.

I will take a few #FediScience flyers with me. Just in case someone is interested in #OpenScience communication, here on #Mastodon.

@johanneskoester Random #snakemake question for you - should with work with SLURM where the default shell is Dash not Bash?

I have some code and workflows which works on SLURM cluster A where /bin/sh is an alias for bash, but fail on SLURM cluster B where /bin/sh is an alias for dash:

WorkflowError:
SLURM sbatch failed. The error message was /bin/sh: 1: Syntax error: "(" unexpected

I don't have any brackets in my workflows, but there are some in the data files…

Yesterday, I learned that we have only 16 - in letters: sixteen(!) - people attending my course in Dresden, so far.

So, if you are interested in creating sustainable data analysis workflows using #Snakemake on #HPC systems, this course might be something for you:

On the 26th and 27th, I will be teaching a 2-day course in #Saxony, #Dresden. See https://tu-dresden.de/zih/hochleistungsrechnen/nhr-training/bd-hpda/snakemake for details and registration.

When reading this call (https://mast.hpc.social/@sneuwirth/113945330638974358) from @sneuwirth I think, I could tell a number of tales from my experience with developing the #Slurm plugin for #Snakemake (and other software over the years).

Particularly, issue reports telling what peculiarities fellow admins thought of (e.g. "we are not allowed to do this" and "we not that". "We have this setting" and "we that".) To develop #HPC software is a challenge in its own right. To support workflow software seems to develop into a nice challenge.

Could certainly deliver a funny few minutes. Perhaps I should collect anecdotes and write an article sometime?

The #snakemake workflow catalog has since recently >300 registered workflows with standardized and unified usage instructions . Furthermore, Michael Jahn from Emmanuelle Charpentier's lab is now the new maintainer of the catalog: https://snakemake.github.io/snakemake-workflow-catalog