Dependency Management

This post was started on 2020-06-18. It was published on 2020-06-18.

I’ve been gradually fleshing out parts of the Colophon system. An immediate issue that I’ve been running into is dependency management. For example, making a change to a post needs to trigger the related indexes to rebuild. However, Pollen as it stands doesn’t make this easy. In particular, it rebuilds pages only when the corresponding source file has changed (the posts/index.html is rebuilt when the posts/index.html.pm file is changed.

To rebuild things like index pages on demand, we need two things. First, on the Pollen side we need to be able to rebuild pages at will, even if the corresponding source file hasn’t changed. Second, we need a way to track dependencies: the index page depends on all the pages that it indexes, and changing any of them (or adding one) should cause the index page to rebuild.

For the first part, the Pollen render command supports a --force option. However, according to the documentation it alters the timestamp of the source file (and I suppose uses the rest of the production path). This is workable, but less than ideal. It would be better if the --force option directly rebuilt the pages as directed. Unfortunately this is tied into Pollen’s caching system, so it’ll require interacting with the Pollen code at a much lower level.

For the second part, an immediate option would be to use a build management system like make to do the heavy lifting. For example, an index page would be generated by a rule such as:

index.html: $(INPUT)/index.html.pm $(POSTS)
        raco pollen render -f $(INPUT)/index.html.pm

By making the index pages a dependency of the top-level target(s), running make will correctly render any new post pages, and rebuild the index. Of course, by placing this logic in a makefile, the Pollen server won’t rebuild the indexes when a new page is added. But this is fine for now: the Pollen server is most useful for checking the rendering of in-progress pages. A makefile is a good way to handle larger rebuilds and deployments, so it seems reasonable to make it responsible for rebuilding the indexes.

Using Pollen’s --force option and a makefile is a good way to capture dependencies and drive rebuilds. But let’s think about a less janky, more integrated system for doing dependency-driven rebuilds.

Pollen’s --force is fine. Updating timestamps is a little inconvenient, but probably not problematic for most things. And it doesn’t require messing with Pollen’s caching system. Maybe we can write a wrapper, in Racket, to call the appropriate Pollen code?

Luckily, there is a make dependency manager written in Racket. I haven’t dug into it, but it seems capable of tracking dependencies between files and executing Racket code to build the target from the dependencies. The logical next step seems to be to implement index generation using make.