[{"url":"/2026/08/conventional-commits-all-the-way-to-a-released-version/","title":"Conventional Commits all the way to a released version","summary":"Wiring commit-driven releases so the version number actually means something — the misconfiguration that made every release a minor bump for months, the separation between version calculator and changelog generator, and the CI permissions that block every first attempt","date":"2026-08-03","tags":["devops","git","automation","changelog"],"cover":"yellow","body":"For the better part of a year my releases looked automated. Merge to main, and a pipeline computed a version, wrote a changelog section, tagged the repository and published the artefact. Nobody bumped a number by hand. Nobody wrote release notes. It felt solved.\nThen someone asked why version 0.7.0 — a minor bump, implying new functionality — contained nothing but a documentation change.\nIt happened because the version number was never actually derived from the commits. It was derived from a single line of configuration that said \u0026ldquo;on main, increment the minor component\u0026rdquo;, and it had said that since the day the pipeline was written. Every release for months had been a minor bump: bug fixes, docs, refactors, everything. The numbers were incrementing, but they carried no information at all.\nThat is the failure mode this post is really about. Wiring up commit-driven releases is a weekend\u0026rsquo;s work and every tool has a quickstart. Wiring them up so that the number means something — and confirming that it does — is the part that gets skipped.\nThe idea in one line The commit message is the only input. Everything downstream is derived from it:\nWhat Derived from Whether the commit is allowed at all the format Which changelog section the entry lands in the type (feat → Features, fix → Fixes) The entry text and its scope prefix the subject The version number the type as well — feat → minor, fix → patch, ! → major So a developer writing this:\ntext Copy 1 feat(api): add heartbeat endpoint has, in that one line, requested a minor version bump and written their own release note. Nothing else in the pipeline needs a human decision.\nThe moving parts, and the one distinction that matters Four files and a set of repository permissions:\nFile Role .githooks/commit-msg Rejects a commit whose subject is not a Conventional Commit. Local only. A version-calculator config Computes the next SemVer from branch and tag history. A changelog-generator config Turns commits into a changelog section. CHANGELOG.md The output. Manual notes on top, generated sections below. I used GitVersion for the number and git-cliff for the text, but the tools are interchangeable. What is not interchangeable is the separation between them:\nThe version calculator owns the number. The changelog generator owns the text. The generator is handed the number and never computes one itself.\nGet this wrong — let both tools work out a version independently — and they will eventually disagree, usually on the release where it matters. Pass the computed number in explicitly (git-cliff --tag \u0026quot;$VERSION\u0026quot;) and the question cannot arise.\nOne commit message, and everything downstream derives from it. The commit-msg hook is the only place a human sees a rejection. Step one: enforce the format, and know what your enforcement is worth A commit-msg hook is about thirty lines of shell. It takes the first non-blank, non-comment line and tests it against a single regex:\nbash Copy 1 pattern=\u0026#39;^(feat|fix|docs|style|refactor|perf|test|build|ci|chore|revert)(\\([a-zA-Z0-9._/-]\u0026#43;\\))?!?: .\u0026#43;\u0026#39; Two details are worth copying. First, let machine-generated subjects through untouched — you did not author them, and failing them makes rebasing miserable:\nbash Copy 123 if printf \u0026#39;%s\u0026#39; \u0026#34;$subject\u0026#34; | grep -qE \u0026#39;^(Merge |Revert \u0026#34;|fixup!|squash!|amend!)\u0026#39;; then exit 0 fi Second, make the failure message teach the convention rather than merely refusing. Print the expected shape, the allowed types and three worked examples. A contributor who hits this hook once should not need to look anything up.\nKeep the hook in a version-controlled directory rather than .git/hooks, so it is reviewed like any other file. The cost is that git does not enable it automatically — every contributor must run git config core.hooksPath .githooks once per clone, and you must put that line in CONTRIBUTING.md.\nAnd then accept that the hook is advisory, not a gate. It is per-clone local configuration. A contributor who never runs that command, uses a GUI client that bypasses hooks, or commits through the web editor will push non-conventional subjects — which then silently vanish from your changelog. If you want real enforcement, add a CI job on merge requests that validates the commits in the MR range server-side, and make it blocking. The hook is for fast feedback; the CI job is for correctness.\nStep two: make the type drive the number This is the step I got wrong, so it gets the most space.\nOut of the box, GitVersion knows nothing about Conventional Commits. Its built-in bump triggers look for an explicit trailer that essentially nobody writes:\nBump Default trigger Major +semver: breaking or +semver: major Minor +semver: feature or +semver: minor Patch +semver: fix or +semver: patch If your config omits the bump-message patterns, no commit ever matches, every release falls through to the branch\u0026rsquo;s default increment:, and the number degrades into a counter. Nothing fails. No test catches it. The pipeline stays green for months.\nThe fix is three lines:\nyaml Copy 123 major-version-bump-message: \u0026#39;(^|\\n)\\w\u0026#43;(\\([^)]*\\))?!:|(^|\\n)BREAKING[ -]CHANGE:\u0026#39; minor-version-bump-message: \u0026#39;(^|\\n)feat(\\([^)]*\\))?:\u0026#39; patch-version-bump-message: \u0026#39;(^|\\n)(fix|perf|refactor)(\\([^)]*\\))?:\u0026#39; The tool tests every commit since the last tag against these in order and applies the highest match found across the whole set.\nTwo regex subtleties cost me time:\nThe patterns are matched against the entire commit message as one string, not line by line. A bare ^ therefore only ever reaches the subject — which is fine for the type, but means a BREAKING CHANGE: footer in the body is never seen. Hence (^|\\n). The \\([^)]*\\) scope group must be optional and non-greedy about parentheses, or a scoped commit like feat(api): fails to match while the unscoped feat: succeeds. That produces the maddening symptom of the bump working only sometimes. Types you deliberately leave unmatched — docs, chore, ci, style, test, build, revert — fall through to the branch default, which should be Patch.\nDo not add a \u0026ldquo;no bump\u0026rdquo; pattern for them. If every merge to main cuts a release, a no-bump result recomputes the existing version and the tag push fails with \u0026ldquo;already exists\u0026rdquo;. A docs-only release should be a patch, not a paradox.\nVerify it against your own changelog, not against the docs Here is the two-minute check that would have saved me months. Open your changelog. Take the last five releases. For each one, look at the commit types that went into it and confirm the version component that moved is the one you would expect.\nMine failed immediately and obviously: a fix-only release that cut a minor, two docs-only releases that also cut minors. The evidence had been sitting in the repository the whole time, formatted as a table, and nobody had read it as a test result.\nDo this check after any change to the version configuration, and once as an audit even if you have not changed anything.\nStep three: make the type drive the text The same commits, grouped differently. The changelog config maps each type to a section:\ntoml Copy 1234567891011 conventional_commits = true filter_unconventional = true commit_parsers = [ { message = \u0026#34;^feat\u0026#34;, group = \u0026#34;\u0026lt;!-- 0 --\u0026gt;Features\u0026#34; }, { message = \u0026#34;^fix\u0026#34;, group = \u0026#34;\u0026lt;!-- 1 --\u0026gt;Fixes\u0026#34; }, { message = \u0026#34;^perf\u0026#34;, group = \u0026#34;\u0026lt;!-- 2 --\u0026gt;Performance\u0026#34; }, { message = \u0026#34;^refactor\u0026#34;, group = \u0026#34;\u0026lt;!-- 3 --\u0026gt;Refactoring\u0026#34; }, { message = \u0026#34;^docs\u0026#34;, group = \u0026#34;\u0026lt;!-- 4 --\u0026gt;Documentation\u0026#34; }, # ... ] The \u0026lt;!-- N --\u0026gt; prefixes are a sort key — the template strips the comment before rendering, so sections appear in a deliberate order rather than alphabetically or by first occurrence. It looks like a hack because it is one, but it is the documented one.\nNote filter_unconventional = true: anything that does not parse is dropped. Combined with an advisory-only hook, that means a badly-formed commit does not break the release, it just quietly does not appear in it. Silent omission is the characteristic failure of this whole setup, and it is worth saying out loud to your team.\nSplice, do not regenerate The obvious invocation regenerates the entire changelog file from history — destroying every hand-written note in it. Render only the new section and splice it in:\nbash Copy 1 git-cliff --config cliff.toml --tag \u0026#34;$VERSION\u0026#34; --unreleased --strip all \u0026gt; new_section.md Flag Why --tag \u0026quot;$VERSION\u0026quot; Stamps the calculated number on. The generator never invents one. --unreleased Only commits since the last tag. --strip all Drops header and footer, leaving a section body ready to insert. Then insert it above the first versioned heading rather than below the ## [Unreleased] heading — anchoring on the version pattern means the same script works on a repository that has no releases yet.\nDecide explicitly what happens to the manual [Unreleased] block on release. Some of our repositories leave it alone, so notes accumulate until someone prunes them; others reset it to a placeholder. Either is defensible. Not deciding means contributors do not know whether their hand-written note will survive the next release, so they stop writing them.\nStep four: the release job, and its four load-bearing lines The job stamps the version into the source, regenerates the changelog, commits, tags and pushes. Most of it is unremarkable sed. Four lines are not, and all four are easy to lose when copying between projects:\nLine Why it is essential git fetch --tags --force The CI clone usually has no tags. Without them the generator cannot find the previous release, and --unreleased yields your entire history as one section. git checkout -b \u0026quot;$BRANCH\u0026quot; CI checks out a detached HEAD at a commit SHA. You cannot push a detached HEAD to a branch. git remote set-url origin https://… The default remote uses the job\u0026rsquo;s own token, which is never permitted to push. Swapping in the bot credential is what makes the push possible at all. -o ci.skip on both pushes Otherwise the release commit and the tag each trigger a fresh pipeline, which tags again — an infinite release loop. A fifth, if your runner image is Alpine or otherwise musl-based: installing git-cliff via pip tries to compile the Rust crate from source and fails, because there is no musllinux wheel. Download the prebuilt static binary for the architecture instead. This wastes an afternoon exactly once.\nStep five: the permissions, which are the actual blocker Everything above is files you can copy. This part is not in any file, and it is what will break your first attempt.\nA CI job pushing to a protected main has to clear two independent server-side gates, and fixing the first only reveals the second. Configure both up front.\nGate one — protected branch push permission. main is protected, so by default nobody may push to it, including your bot. Add the bot to the branch\u0026rsquo;s allowed-to-push list. If you also protect tags, add it to allowed-to-create there too, or the branch push will succeed and the tag push will fail a second later.\nHere is the thing that cost me the most time, across two separate projects, months apart:\nThe CI variable holding your token is not a username. I had a variable called something like RELEASE_BOT, and searching for that string in the member picker returns nothing, because no such user exists. A group access token authenticates as an auto-generated bot user named group_\u0026lt;groupId\u0026gt;_bot_\u0026lt;hash\u0026gt;; a project token uses project_\u0026lt;projectId\u0026gt;_bot_\u0026lt;hash\u0026gt;. Search the member list for bot.\nPrefer adding that specific bot user over granting a whole role. Granting Maintainers the right to push to main solves your pipeline problem by removing your branch protection, which is not the trade you meant to make.\nGate two — push rules. A \u0026ldquo;reject unverified users\u0026rdquo; rule requires the committer email on every pushed commit to be a verified address of the pushing user. My job committed under a friendly vanity identity but pushed as the token bot, whose only verified address is its own no-reply one. Rejected.\nIf you cannot turn that rule off, derive the bot\u0026rsquo;s real address at runtime instead of hardcoding one — it changes when the token is rotated:\nbash Copy 123 BOT_EMAIL=$(curl -sf --header \u0026#34;PRIVATE-TOKEN: ${TOKEN}\u0026#34; \u0026#34;${API_URL}/user\u0026#34; \\ | sed -n \u0026#39;s/.*\u0026#34;email\u0026#34;:\u0026#34;\\([^\u0026#34;]*\\)\u0026#34;.*/\\1/p\u0026#39;) git config user.email \u0026#34;${BOT_EMAIL:-fallback@example.com}\u0026#34; Push rules are typically inherited from a parent group, so check there as well as on the project. A rule set one level up is the usual explanation for why one repository releases cleanly and its sibling does not with identical pipeline configuration.\nRead the error to tell the gates apart The messages are similar enough to conflate, and each points somewhere different:\nError Meaning not allowed to push code to protected branches The bot is a member but is not in the allowed-to-push list → gate one not allowed to push code to this project The bot is a member but has too low a role The project you were looking for could not be found The bot is not a member at all Authentication failed / HTTP Basic: Access denied The token is missing, wrong or expired You cannot push commits for '\u0026lt;email\u0026gt;' Permissions are fine — a push rule is rejecting the identity → gate two Three more things that will bite Squash merges rewrite the subject. If you squash on merge, the platform builds the squashed subject from the merge request title, not from your carefully-typed commits. An MR titled \u0026ldquo;Fix the thing\u0026rdquo; produces a non-conventional subject that gets filtered out of the changelog — and, now that the type also drives the number, quietly comes out as a patch when it should have been a minor. One bad title destroys both signals at once. Either enforce Conventional Commit syntax on MR titles, or turn squashing off and rely on the commits.\nHand-edits to generated sections are lost. Only the manual block is yours. A generated release section is regenerated from commit messages, so editing it achieves nothing durable — amend the commit message instead, before it reaches main.\nToken rotation breaks every project simultaneously. Rotating a group access token mints a new bot user with a different hash. The old entry in the allowed-to-push list still names the dead bot, so every release-automated project in the group starts failing at once with a permissions error and no code change to explain it. When you rotate, note the new bot username and update every project\u0026rsquo;s protected-branch and protected-tag lists.\nWhen SemVer is not the right key One of our repositories is not software. It holds a single schema-validated architecture document, and a change to it is reviewed under a governance ticket rather than shipped as a version. \u0026ldquo;Version 2.4.0 of the design document\u0026rdquo; would be meaningless to every human involved.\nThe interesting outcome is that almost all of the machinery still applied — I changed the key, not the pipeline. The branch is named after the governance ticket, the document\u0026rsquo;s version field is stamped with that same ticket ID, and the changelog section is headed with it too. Commit types still drive which group each entry lands in. The one thing that goes away is the version calculator, because the ticket ID replaces the computed number.\nTwo adaptations were worth the effort and would transfer to any repository:\nGenerate locally, verify in CI. Instead of a job that commits the changelog back to the protected branch, a contributor runs the generator locally and a CI job re-runs it with a --check flag and fails if the result differs. That deletes the entire permissions problem from step five — no bot, no protected-branch exception, no ci.skip, no release loop. If you do not need a version stamped into published artefacts, this is the better design, and I would now reach for it first. Regenerating replaces, rather than appends. Re-running on the same branch removes the existing section for that key before splicing the new one in, so an amended commit produces a corrected section instead of a duplicate. I also made both scripts degrade gracefully: if git or the generator is unavailable, they skip with a zero exit rather than failing. A tooling gap should never falsely block a pipeline that is otherwise fine.\nThe checklist Copy the hook, the version config and the changelog config to the repository root. Keep the bump-message patterns. Without them the types do nothing, and nothing tells you. Create CHANGELOG.md with a preamble and a manual [Unreleased] heading, and decide what happens to that block on release. Add the pipeline jobs. Preserve the four load-bearing lines. Add the bot user to protected-branch push and protected-tag create. Check push rules on the project and the parent group. Document git config core.hooksPath .githooks in CONTRIBUTING.md, and add a blocking CI job that validates commit format server-side. Merge a feat: commit and confirm a minor bump. Then merge a docs: commit and confirm a patch. Confirm the tag, the changelog section and the published artefact all carry the same number. Step eight is the one to insist on. Everything else can be green while the system is quietly meaningless — and a version number that means nothing is worse than no version number at all, because people trust it.\nFurther reading Conventional Commits v1.0.0 Keep a Changelog Semantic Versioning "},{"url":"/2026/08/snapshot-first-why-your-restore-tool-should-never-trust-the-live-environment/","title":"Snapshot-first: why your restore tool should never trust the live environment","summary":"A design constraint for disaster recovery tooling — resolve every restore against a configuration snapshot captured before the incident, never against a live query — and what falls out of it: pure-function planning, fully offline tests, and drills cheap enough to run continuously","date":"2026-08-03","tags":["security","devops","azure","databricks","architecture"],"cover":"cobalt","body":"Most disaster recovery runbooks contain a sentence like this one:\nRecreate the external location pointing at the original storage path.\nIt reads fine in a review. It is useless at 02:00, because it quietly assumes you can still find out what the original storage path was — and the system you would normally ask is the one that just broke.\nI want to describe the design constraint I ended up adopting to remove that assumption entirely: a restore resolves against a snapshot of the environment captured before the incident, never against a live query. It sounds like a small implementation detail. It is not. Adopting it changes what your recovery code can be tested against, what your drills cost, and — in at least one case I will come to — whether a particular class of recovery is possible at all.\nMy examples are Azure Databricks with Unity Catalog over ADLS Gen2, but the principle transfers to any platform where the metadata layer and the data layer can fail independently.\nThe failure mode this fixes Consider a plausible incident: someone drops a schema containing a few hundred Unity Catalog managed tables. The underlying storage has soft delete enabled, so the data is still there. You are, technically, fine.\nThen you go and look at the storage account, and find this:\ntext Copy 123456 abfss://unity-catalog@example.dfs.core.windows.net/3f2504e0-4f89-11d3-9a0c-0305e82c3301/\rtables/\r9b2e4c71-0a5d-4f6e-9c3a-2b1d8e7f4a05/\rc4a0f2b8-77e1-4a9d-8b3c-6f0d5e1a2b94/\re17d3a92-5c4b-4e88-9f2a-1d0c7b6e5a43/\r... 400 more There is nothing here that says which directory was sales.orders and which was sales.order_lines. Unity Catalog knew the mapping. Unity Catalog is the thing that lost it. Every one of those GUID directories holds a perfectly intact Delta table that you cannot identify, and no amount of storage-layer recovery will tell you the names.\nThe recovery is trivially easy if you wrote the mapping down beforehand, and effectively impossible if you did not. That asymmetry is the whole argument. There is a category of recovery metadata whose only viable capture point is before the incident, and a restore tool that queries live state has, by construction, no access to it.\nWhat a snapshot is — and what it is not A snapshot is a configuration capture, not a data backup. It records the shape and intended state of the environment: what exists, where it points, who can read it, and how it is protected. It does not contain a single row of business data.\nThat distinction matters because it sets expectations correctly in both directions. A snapshot alone cannot restore anything — it always pairs with the file or Delta backup that holds the actual bytes. But the backup alone cannot restore anything usefully either, because it restores a pile of anonymous directories. You need both, and it is worth drawing an explicit coverage matrix showing which recovery scenario needs which half.\nA snapshot is also not a monitoring system. It is written on a schedule, retained for a long time, and never overwritten in place. Its value is precisely that it is old.\nWhat to capture The rule of thumb: capture anything you would have to look up in order to rebuild, and anything whose absence you would only notice under pressure.\nFor the Databricks metadata layer:\nArtefact Why a restore needs it Catalogs and schemas, with owners Recreate the container hierarchy in the right order Catalog/schema/object grants Replay permissions; otherwise a restored object is invisible to its users Table inventory with physical storage locations The name → path map, including managed GUID paths External locations and their URLs Re-register external tables at the correct paths Storage credentials and identity bindings Reconnect the metadata layer to storage Volumes Same problem as tables, for non-tabular data Groups and members; users Grant replay is meaningless without the principals Deployed job/pipeline definitions Rebuild the workloads that populate the restored tables For the Azure layer underneath:\nArtefact Why a restore needs it Storage accounts: resource locks, soft-delete and versioning settings, public network access Establishes what recovery options actually exist Private endpoints and their connection state Re-establish the classic connectivity path Serverless network connectivity rules Re-establish the serverless path, which is separate Key Vault and secret-scope inventory (names, not values) Know what has to be repopulated, and by whom The two connectivity rows deserve a note, because it is a trap I have watched people fall into. A storage account with public network access disabled is typically reachable by two independent private paths: customer-VNet private endpoints for classic compute, and an account-level network connectivity rule for serverless compute. Repairing one does not repair the other, and a successful read over the classic path proves nothing about the serverless path. They are separate rows in the snapshot for exactly that reason — capturing them together is what makes the difference visible before someone assumes it away.\nStructure on disk Keep it boring and filesystem-shaped:\ntext Copy 1234567891011 output/snapshots/{env}/{YYYY-MM-DD_HHmm}/\rstorage_accounts.json # Azure artefacts at the root\rprivate_endpoints.json\rprod/ # Databricks artefacts, per stage\rcatalogs.json\rexternal_locations.json\rstorage_credentials.json\rgroups.json\rprod_2026-07-02_1002_table_inventory.csv\rtest/\r... Three properties are worth being deliberate about:\nTimestamped directory names that sort lexicographically by time. YYYY-MM-DD_HHmm gives you \u0026ldquo;the latest snapshot\u0026rdquo; as a sorted(...)[-1], with no metadata index to maintain or corrupt. Immutable directories. A snapshot run creates a new directory; it never edits an old one. This is what lets you diff two points in time. Plain JSON and CSV. During an incident someone will want to open these in an editor or grep them. Do not make that hard. Retention is a real decision, not an afterthought. Ninety days of CI artefacts is a reasonable starting point, but the retention window is also your maximum blast radius for a slow corruption: if a bad change went in four months ago and every snapshot you still hold records the damaged state, the snapshot cannot tell you what \u0026ldquo;correct\u0026rdquo; looked like.\nThe capture side Each artefact gets its own small extraction script. They are boring on purpose — query the SDK, normalise, write JSON:\nextract_catalogs.py python Copy 1234567891011121314151617181920212223242526 \u0026#34;\u0026#34;\u0026#34;Capture catalogs, schemas, and their grants.\u0026#34;\u0026#34;\u0026#34;\rfrom databricks.sdk import WorkspaceClient\rfrom databricks.sdk.service.catalog import SecurableType\rdef extract_catalogs(client: WorkspaceClient) -\u0026gt; list[dict]:\rcatalogs = []\rfor catalog in client.catalogs.list():\rgrants = client.grants.get(\rsecurable_type=SecurableType.CATALOG, full_name=catalog.name\r)\rcatalogs.append(\r{\r\u0026#34;name\u0026#34;: catalog.name,\r\u0026#34;owner\u0026#34;: catalog.owner,\r\u0026#34;schemas\u0026#34;: [\r{\u0026#34;name\u0026#34;: s.name, \u0026#34;owner\u0026#34;: s.owner}\rfor s in client.schemas.list(catalog_name=catalog.name)\r],\r\u0026#34;grants\u0026#34;: [\r{\u0026#34;principal\u0026#34;: a.principal, \u0026#34;privileges\u0026#34;: [p.value for p in a.privileges]}\rfor a in (grants.privilege_assignments or [])\r],\r}\r)\rreturn catalogs Two conventions that pay for themselves:\nWrap the payload. Write {\u0026quot;generated\u0026quot;: \u0026quot;\u0026lt;iso timestamp\u0026gt;\u0026quot;, \u0026quot;catalogs\u0026quot;: [...]} rather than a bare list. You will want to know when a file was written without trusting the directory name, and you will eventually want to add a field beside the payload.\nRecord permission failures rather than omitting them. If the caller lacks the rights to read something, write a marker — {\u0026quot;access\u0026quot;: \u0026quot;permission_denied\u0026quot;} — instead of an empty list or nothing at all. An empty list is indistinguishable from \u0026ldquo;there genuinely aren\u0026rsquo;t any\u0026rdquo;, and that ambiguity will cost someone an hour during an incident. This happens more than you would expect: some account-level settings are only readable by an account administrator, so a snapshot taken with workspace-level credentials legitimately cannot see them.\nThe read side The restore tool never touches the snapshot files directly. It goes through one small reader class, which is where all the format tolerance lives:\nsnapshot.py python Copy 12345678910111213141516171819202122232425262728 class Snapshot:\r\u0026#34;\u0026#34;\u0026#34;Read access to one snapshot directory.\u0026#34;\u0026#34;\u0026#34;\rdef __init__(self, path: Path, stage: str = \u0026#34;prod\u0026#34;) -\u0026gt; None:\rif not path.is_dir():\rraise SnapshotError(f\u0026#34;Snapshot directory does not exist: {path}\u0026#34;)\rself.path = path\rself.stage = stage\rdef _unwrap(self, name: str, key: str) -\u0026gt; list[dict]:\r\u0026#34;\u0026#34;\u0026#34;Load an artefact written as {\u0026#34;generated\u0026#34;: ..., \u0026#34;\u0026lt;key\u0026gt;\u0026#34;: [...]}.\rA bare list is also accepted, so a snapshot-format change does not\rbreak a restore.\r\u0026#34;\u0026#34;\u0026#34;\rdata = json.loads((self.path / self.stage / name).read_text())\rreturn data.get(key, []) if isinstance(data, dict) else data\rdef catalogs(self) -\u0026gt; list[dict]:\rreturn self._unwrap(\u0026#34;catalogs.json\u0026#34;, \u0026#34;catalogs\u0026#34;)\rdef managed_path_map(self) -\u0026gt; dict[str, str]:\r\u0026#34;\u0026#34;\u0026#34;Map fully-qualified table name → physical storage path.\u0026#34;\u0026#34;\u0026#34;\rreturn {\rt.full_name: t.storage_location\rfor t in self.table_details()\rif t.is_managed and t.storage_location\r} That _unwrap tolerance is not fussiness. The snapshot format will change, and the snapshots you most need are the old ones. A reader that only understands this month\u0026rsquo;s format has quietly expired your entire archive.\nWhat falls out of the constraint This is the part that surprised me. I adopted snapshot sourcing for correctness, but most of the benefit turned up somewhere else entirely.\nResolution becomes a pure function If a restore scenario reads only files, then \u0026ldquo;work out what we would do\u0026rdquo; is a pure function from snapshot to plan. So express it that way: every scenario exposes a plan_* function that takes a context and returns an ordered list of steps, and nothing executes while resolving.\nscenarios.py python Copy 123456789101112131415161718192021 def plan_reregister_external_table(context: RestoreContext, table: str) -\u0026gt; RestorePlan:\rplan = RestorePlan(scenario=\u0026#34;3\u0026#34;, title=\u0026#34;Re-register external table\u0026#34;, target=table)\rdetail = context.snapshot.table_details_by_name().get(table)\rif detail is None:\rplan.notes.append(f\u0026#34;{table} is absent from the snapshot inventory.\u0026#34;)\rplan.add(\r\u0026#34;Identify the table\u0026#39;s storage path by hand before proceeding\u0026#34;,\rmanual_reason=\u0026#34;Not in the snapshot — the path must not be guessed.\u0026#34;,\r)\rreturn plan\rplan.add(\r\u0026#34;Recreate the table at its recorded location\u0026#34;,\rcommand=(\rf\u0026#34;CREATE TABLE {quote_full_name(table)} \u0026#34;\rf\u0026#34;LOCATION {quote_location(detail.storage_location)}\u0026#34;\r),\raction=lambda: context.sql(...),\r)\rreturn plan Tests become fully offline Because resolution is pure, the entire test suite runs with no cloud credentials. A pytest fixture builds a miniature snapshot under tmp_path and the tests assert on plans:\ntest_scenarios.py python Copy 1234 def test_missing_table_degrades_to_a_manual_step(tiny_snapshot):\rplan = plan_reregister_external_table(ctx(tiny_snapshot), \u0026#34;sales.not_captured\u0026#34;)\rassert plan.steps[0].is_manual\rassert \u0026#34;must not be guessed\u0026#34; in plan.steps[0].manual_reason And crucially you can test the things that actually go wrong in a restore — that a schema is never created before its catalog, that a grant is never replayed before the object it applies to, that validation happens before consumers are re-pointed and not after. In a restore, the order of operations is the correctness. Ordering is almost impossible to test against a live environment and trivial to test against a fixture.\n\u0026ldquo;Missing data\u0026rdquo; gets a correct home When a plan needs a field the snapshot does not contain, there is exactly one right answer: fix the extraction script, take a new snapshot, and carry on. The wrong answer — inferring it, defaulting it, reconstructing it from a naming convention — is now visibly wrong, because the code has no live client to infer it from.\nMake the degradation explicit and unpleasant. Never guess a storage path, particularly a managed table\u0026rsquo;s GUID path: a plausible-looking wrong path in a recovery plan is worse than no plan, because it will be run. Emit a step with no action and a manual_reason instead.\nThat leads to the guardrail I would keep even if I kept nothing else: a plan containing any manual step refuses to execute.\nplan.py python Copy 12345678 def execute(self) -\u0026gt; None:\rif self.has_manual_steps:\rraise RuntimeError(\r\u0026#34;Plan contains manual steps and cannot be executed automatically. \u0026#34;\r\u0026#34;Run it as a dry run and perform the manual steps by hand.\u0026#34;\r)\rfor step in self.steps:\rstep.run() Dry run is the default; --execute is opt-in and refused while anything is unresolved. A step only loses its manual flag once a drill has been recorded for it — automation is earned, scenario by scenario, rather than assumed at design time.\nDrills stop being expensive A dry run exercises the whole resolution path — snapshot lookup, path mapping, dependency ordering, SQL generation — while touching nothing. You can run every scenario against last night\u0026rsquo;s snapshot on a laptop, in seconds, and see the exact commands each one would issue. That is a rehearsal cheap enough to run continuously, which means the runbook stops rotting between annual DR tests.\nDiffs explain drift Two immutable, timestamped captures of the same environment are a diff waiting to happen:\nbash Copy 12 diff-snapshots output/snapshots/prod/2026-06-30_0908 \\\routput/snapshots/prod/2026-07-02_1002 A newly-failing configuration assertion becomes a concrete line of output — this grant was removed, this endpoint went from Approved to Pending — instead of a debate. This is also the natural home for compliance evidence: you are not asserting that permissions were correct on a given date, you are holding the capture.\nThe trap: the latest snapshot is not necessarily a good one The obvious default — restore from the most recent snapshot — is wrong, and worth guarding against explicitly.\nA scheduled capture keeps running throughout an incident. In a corruption or accidental-deletion event, the newest snapshot may faithfully record the damaged state: the grant already removed, the table already dropped, the endpoint already broken. Restoring from it reproduces the incident with great fidelity.\nSo make \u0026ldquo;latest\u0026rdquo; a convenience and \u0026ldquo;explicit\u0026rdquo; a first-class option:\nlocate.py python Copy 1234567 def find_snapshot_dir(root: Path, env: str, explicit_dir: Path | None = None) -\u0026gt; Path:\r\u0026#34;\u0026#34;\u0026#34;Locate the snapshot to restore from.\rArgs:\rexplicit_dir: An exact snapshot directory, bypassing the \u0026#34;latest\u0026#34; lookup.\rUse this whenever the newest snapshot is not known-clean.\r\u0026#34;\u0026#34;\u0026#34; And say so loudly in the runbook, at the top, where someone under pressure will actually read it. The first question of any restore is not \u0026ldquo;what do I run\u0026rdquo; but \u0026ldquo;which snapshot am I trusting, and do I know it predates the incident?\u0026rdquo;\nWhere this does not apply Three honest limits.\nSome checks genuinely cannot be static. Whether a TCP endpoint is currently accepting connections is not a property you can capture in JSON. Keep those as live checks, mark them clearly as such in the code, and accept that they are a different kind of test.\nA snapshot is not a data backup, and the failure mode of forgetting that is severe. Draw the coverage matrix, keep it current, and make sure every scenario in the runbook names both halves of what it needs.\nCapture is bounded by the credentials it runs with. Some settings need account-level rights that a workspace-scoped identity does not have. Decide deliberately whether to elevate the snapshot job or to accept a documented blind spot — but write the blind spot down, in the snapshot, as a marker.\nGetting started If you want to try this on an existing platform, the smallest useful increment is not the whole framework. It is one CSV.\nCapture the table inventory — fully-qualified name, table type, physical storage location — on a schedule, and keep it. This is the single highest-value artefact, because it is the one that is impossible to reconstruct afterwards. Add grants next, then external locations and storage credentials. Write one scenario as a plan_* function that reads that inventory and returns steps it refuses to execute. Print it. Show it to whoever owns the runbook. Add the Azure protection settings — locks, soft delete, public network access — so the snapshot can also answer \u0026ldquo;what recovery options do we actually have?\u0026rdquo; Diff two snapshots and see what has drifted since you started. Step 1 alone converts one impossible recovery into a routine one. Everything after it is compounding interest.\nThe pattern described here came out of building recovery tooling for a Unity Catalog data platform on Azure. The specifics — artefact names, scenario numbering — are implementation detail; the constraint is the transferable part. During an incident, the live environment is not a source of truth about intended state. Write it down first.\n"},{"url":"/2026/08/teaching-an-ai-reviewer-to-review-a-design-document-not-code/","title":"Teaching an AI reviewer to review a design document, not code","summary":"What happened when I pointed a shared AI merge-request reviewer at an architecture repository containing no code — separating the mechanical gates from the judgement checks, forbidding fabrication, and the style rule that would have broken the build","date":"2026-08-03","tags":["ai","architecture","devops","automation"],"cover":"tangerine","body":"I switched on a shared AI merge-request reviewer across a set of our repositories. On the code repos it behaved roughly as advertised: it read the diff, made some reasonable points, made some obvious ones, and occasionally caught something worth catching.\nThen I pointed it at the architecture repository — and it had almost nothing useful to say.\nThat repository is documentation-as-data. There is no application code in it at all. It holds a single schema-validated YAML file describing a platform\u0026rsquo;s architecture, which CI renders into a Markdown view, an HTML site, and a PDF for the governance portal. A merge request against it changes prose and structured metadata, not behaviour.\nThe reviewer\u0026rsquo;s first pass on that repo asked where the unit tests were.\nThis is not a failing of the tool. Given a diff and no other information, an AI reviewer infers what kind of review to perform, and the overwhelming prior is \u0026ldquo;this is code\u0026rdquo;. If the artefact is not code, you have to say so — and, more usefully, you have to say what a good review of that artefact actually looks like. So this post is about doing exactly that: what I told it, what worked, and the recommended fix that cost me a broken build.\nStart from the human review, not from the tool The instinct is to configure an AI reviewer by thinking about what an AI reviewer can do. That produces a shopping list of generic checks and a lot of noise.\nThe better question is: when a competent architecture reviewer opens this merge request, what do they actually do? Write that down first, in plain language, and only then translate it into configuration.\nFor a design document it turned out to split cleanly in two:\nMechanical gates — objective things that must pass, and where a wrong answer is simply wrong. Judgement checks — the things that make the review worth a human\u0026rsquo;s time: internal contradictions, missing governance records, claims that no longer match reality. Those two halves want very different instructions, and getting the reviewer to understand which mode it is in for a given finding turned out to matter more than any individual rule.\nHalf one: the mechanical gates These duplicate what CI already enforces, which sounds redundant. It is worth doing anyway, because a failed pipeline job tells the author that something is wrong, and the reviewer tells them what and why in a comment attached to the line.\nyaml Copy 12345678 focus_areas: - \u0026#34;The document must remain valid against the design schema and pass yamllint\u0026#34; - \u0026#34;Preserve the schema declaration, YAML hierarchy, key names, and multiline block scalars (|) with their indentation\u0026#34; - \u0026#34;Do not drop business identifiers (application, service, capability, or governance IDs) unless explicitly requested\u0026#34; - \u0026#34;Do not rename section-title values that begin with markdown heading markers (e.g. ### 1 Overview) unless asked\u0026#34; The structural-preservation rules exist because of how this document is authored. It is a schema-bound YAML file, so a well-meaning tidy-up — reflowing a block scalar, renaming a section title, dropping an identifier that looks like clutter — is almost always a mistake, and almost always invisible in review until the render breaks. Telling the reviewer these are load-bearing turns \u0026ldquo;why is this weird ID here?\u0026rdquo; into \u0026ldquo;do not remove this\u0026rdquo;.\nI also encode the repository\u0026rsquo;s change-management convention, because it is mechanical and frequently forgotten:\nyaml Copy 12345 - \u0026#34;The document version field must equal the governance ticket ID in the branch name (e.g. feature/DESIGN-1234 -\u0026gt; DESIGN-1234)\u0026#34; - \u0026#34;The document date must be the date of the most recent change, in DD/Mon/YYYY\u0026#34; - \u0026#34;CHANGELOG.md must contain a single section matching the branch\u0026#39;s ticket ID; do not hand-edit it — it is generated\u0026#34; Note the last clause. Half the value of these instructions is stopping the reviewer from recommending a change that will be overwritten by automation.\nHalf two: the judgement checks This is where an AI reviewer earns its place on a document repo, because these checks are genuinely tedious for a human and genuinely hard to do well when you are the twelfth reviewer of a hundred-page document.\nThe document contradicting itself A long design document accretes contradictions. Section 2 says there are no virtual machines outside the managed platforms; section 5 describes a standalone VM. One section states an RTO of four hours, another eight. An availability-zone count differs between the narrative and a diagram.\nyaml Copy 12345 - \u0026#34;Flag statements that contradict each other elsewhere in the document (e.g. \u0026#39;no VMs outside the managed platforms\u0026#39; vs a described standalone VM; conflicting RPO/RTO values or availability-zone counts)\u0026#34; - \u0026#34;A component, interface, or connectivity path shown in a diagram must be reflected in the relevant narrative section, and vice versa\u0026#34; An LLM reading a whole document at once is genuinely good at this, and a human reviewer reading a diff is genuinely bad at it — because the contradiction is usually between the diff and a section nobody opened.\nThe body contradicting a recorded decision This is the check I would keep if I could keep only one.\nArchitecture decision records exist so that a choice, once made and justified, does not get quietly re-litigated. But nothing enforces the link: the decisions live in one section of the document, and the narrative that must obey them lives in twenty others. A change can perfectly plausibly describe using technology X for a workload where a decision record explicitly chose Y over X — and it will pass every mechanical gate.\nyaml Copy 123 - \u0026#34;Every claim in the body must be consistent with the recorded architectural decisions; if the body does X but a decision record chose Y over X, cite both the section and the decision record\u0026#34; Checking this by hand means holding the decision list in your head while reading the change. Checking it mechanically is impossible. It is exactly the shape of problem a language model is suited to, and it is the finding type that has produced the most \u0026ldquo;oh — good catch\u0026rdquo; reactions.\nAbsence: should this change have produced a new record? A diff shows you what changed. The hardest review question is what should have changed and did not.\nyaml Copy 1234567891011 - \u0026#34;When a change introduces something architecturally new (a new technology, integration pattern, external dependency, or network path, or a materially different way of doing something an existing decision record covers), check whether a new decision record is warranted — recommend one, do NOT invent its content\u0026#34; - \u0026#34;When a change introduces or exposes a risk not already recorded (new external dependency, new single point of failure, DR limitation, security or compliance exposure), recommend a new risk entry with a candidate description, mitigation, residual rating and owner — leave numbering to the author\u0026#34; - \u0026#34;Flag existing risks whose target remediation date has passed, or whose mitigation this change has invalidated\u0026#34; That last one is a small gift. Risk registers rot silently; a reviewer that reads the whole register on every merge request will notice a remediation date that slipped past two quarters ago.\nStale questionnaire answers Most governance documents contain a compliance questionnaire — a set of fixed questions with short answers, often booleans with explanatory notes. These go stale faster than anything else in the document, because they were filled in once and nobody re-reads them.\nyaml Copy 1234 - \u0026#34;Sanity-check the production-environment answers against the rest of the document — e.g. a change that adds a VM while the servers question still answers \u0026#39;None\u0026#39;, compute answers that no longer match the described platform, or a boolean answer whose accompanying notes say the opposite\u0026#34; \u0026ldquo;A boolean whose notes say the opposite\u0026rdquo; is a real pattern, and a genuinely embarrassing one to have found in an audit rather than in review.\nThe instruction that matters most: recommend, never fabricate An AI that invents a plausible architecture decision record is far worse than an AI that says nothing. It produces something that reads like governance, will be skimmed and approved, and records a justification nobody actually made.\nSo this is stated repeatedly and in several places:\nyaml Copy 12345678 coding_standards: | - Do not fabricate decision-record options, risk numbers, or reference URLs. Recommend that the author adds the governance record and links it, rather than inventing its content. - This is documentation-as-data, not code: quote the relevant key or section number in each finding, and do NOT rewrite the document unless asked. - When a claim cannot be confirmed from available evidence, mark it \u0026#34;unverified, please confirm\u0026#34; rather than asserting it is correct. That third line generalises well beyond this use case. The default failure mode of an AI reviewer on a factual claim is confident agreement. Giving it an explicit, low-effort way to express uncertainty — a fixed phrase it is told to use — converts a silent false negative into a visible prompt for the author.\nProse style, and the trap that broke the build The document is written in British English, held to a specific style guide. That is a reasonable thing to ask a language model to check, and it does it well:\nyaml Copy 12345678910 - \u0026#34;Prose is British English: Oxford -ize endings (realize, organize, recognize), keeping the fixed -ise set (advertise, comprise, exercise, supervise, surprise)\u0026#34; - \u0026#34;Single quotation marks as primary; doubles only for a quote within a quote\u0026#34; - \u0026#34;Spaced en dash for parenthetical asides, unspaced en dash for ranges (1939-45); never an em dash\u0026#34; - \u0026#34;Use the serial comma before the final item in a list of three or more\u0026#34; - \u0026#34;Spell out numbers one to ninety-nine in running prose; numerals for 100 and above, except measurements, percentages, versions and identifiers\u0026#34; - \u0026#34;Flag inconsistent spelling, hyphenation, or capitalisation of the same term across the document\u0026#34; Then it suggested a fix that would have failed the schema validation.\nThe document contains fields whose values are constrained to a fixed schema enumeration. One of them is a criticality tier whose exact enum string includes a hyphenated range — something of the shape Tier 2: RTO 4-8 hours. Applied naively, the \u0026ldquo;unspaced en dash for ranges\u0026rdquo; rule says that hyphen should be an en dash. Make that change and the value no longer matches the enum, and the schema validation job fails.\nThe fix is an exemption list, and it needs to be explicit rather than implied:\nyaml Copy 12345 - \u0026#34;Exempt machine-formatted values from prose style rules: schema URNs, version and date fields, section-title markers (### 1.9), code and identifier strings, resource names, URLs, and any value constrained to a fixed schema enumeration (e.g. a criticality tier containing \u0026#39;RTO 4-8 hours\u0026#39; — keep the hyphen, it must match the enum exactly)\u0026#34; The general lesson is worth stating plainly, because it applies to every AI style check you will ever configure: when you ask a model to enforce style, you must also tell it where style does not apply. A document is not uniformly prose. It contains identifiers, enums, paths, and formatted values that look like text and are not. The model cannot infer the boundary, and the cost of getting it wrong is asymmetric — a missed style nit is trivial, a \u0026ldquo;corrected\u0026rdquo; enum value is a broken pipeline.\nWorth noting too: style findings are the highest-volume and lowest-value category, so say so. Mine are explicitly ranked as subordinate to schema, governance and consistency findings, and capped.\nConfiguration mechanics worth knowing A handful of small things that made a disproportionate difference:\nIgnore your rendered outputs. This repo commits the generated Markdown and PDF alongside the source. Without ignore patterns, the reviewer reviews the same change three times and leaves comments on generated files the author cannot meaningfully fix.\nyaml Copy 12345 ignore_patterns: - \u0026#34;docs/design.md\u0026#34; - \u0026#34;docs/design.pdf\u0026#34; - \u0026#34;reviews/*.html\u0026#34; - \u0026#34;**/.venv/**\u0026#34; Turn off code examples. Most reviewers default to illustrating findings with code snippets. On a document repo that produces nonsense; you want quoted keys and section numbers.\nyaml Copy 1 require_code_examples: false Cap suggestions per category. Uncapped, style findings drown everything else and reviewers learn to skim the whole comment.\nyaml Copy 12345 max_suggestions_per_type: style: 8 consistency: 12 governance: 12 security: 15 Give per-file-type instructions. The design source, the changelog, and the ordinary markdown docs each want different treatment — the changelog in particular should be left alone because it is generated.\nWhat it is good at, and what it is not Good at: whole-document consistency, cross-referencing claims against a decision register, noticing absence, spotting stale dates, and applying a style guide uniformly. All the things that are boring, mechanical-feeling, and that human reviewers skip when the document is long.\nNot good at: knowing whether the architecture is right. It has no opinion worth having on whether the chosen technology suits the workload, whether the risk appetite is appropriate, or whether the design will survive contact with the operational team. That is the human review, and it always was.\nOccasionally wrong in predictable ways. Mine reliably flags passing a secret from one CI variable into another as a hardcoded-credential risk. It is a false positive — the platform masks by value, so the secret stays redacted — but it is a reasonable false positive, and the right response is to resolve it in the thread rather than contort the configuration to suppress it. Configuring away every false positive eventually configures away the true ones too.\nIt also caught something I would have got wrong: that the Conventional Commits specification puts the breaking-change ! immediately before the colon — feat(scope)!:, not feat!(scope):. A small thing, but exactly the kind of spec detail that a reviewer with the specification in its training data will beat a human on.\nThe transferable version If you are pointing an AI reviewer at something that is not code:\nTell it what the artefact is. A paragraph of project description does more than any individual rule. Write down what your human reviewers actually do, then encode that — rather than accepting the tool\u0026rsquo;s defaults. Separate the mechanical gates from the judgement checks, and let the mechanical ones duplicate CI. The explanation is the value, not the pass/fail. Look for absence, not just change. \u0026ldquo;Should this have produced a governance record?\u0026rdquo; is the highest-value question and the one no linter can ask. Forbid fabrication explicitly, and give the model a stock phrase for uncertainty. If you ask for style, define where style does not apply — enums, identifiers, and formatted values will otherwise get \u0026ldquo;corrected\u0026rdquo; into a broken build. Ignore generated files, cap the noisy categories, and check the config is somewhere the tool will actually find it. The end state is a little bit different from a typical code review. The goal is not an AI that approves merge requests. The goal is a left-shift to save my time and that of others by capturing some categories of mistakes early. If the AI reviewer is happy with my latest changes, I can at least be reasonably confident that any human reviewer it goes to opens a document which is already schema-valid, internally consistent, correctly stamped and stylistically clean — and can spend their entire attention on whether the architecture is any good.\n"},{"url":"/2026/07/a-changelog-you-never-write-conventional-commits-lefthook-and-git-cliff/","title":"A changelog you never write: Conventional Commits, Lefthook and git-cliff","summary":"How I standardised commit messages and changelogs across a polyglot workspace — Conventional Commits enforced by a shared shell hook, Lefthook to manage the hooks, Keep a Changelog as the format, and git-cliff to generate the changelog from history and keep it current in CI","date":"2026-07-17","tags":["devops","git","automation","changelog"],"cover":"mint","body":"Over a couple of evenings I standardised the developer tooling across every repository in my workspace — this blog, Velostevie (my cycling blog), and a handful of client, QA and publishing projects. Eleven repos in five languages, each with its own idea of what a commit message should look like and none of them with a changelog worth reading.\nThe goal was maintainability: consistent commit history, a changelog in every project, and hooks that keep both honest without anyone having to remember they exist. This post is the pattern that came out of it — Conventional Commits enforced by a shared hook, Lefthook to install and run the hooks, Keep a Changelog as the format, and git-cliff to generate that changelog from the commit history. The payoff is a changelog you never write by hand and never forget to update.\nThe problem with changelogs A CHANGELOG.md is the first file to fall out of date. It lives outside the code, nothing breaks when it is wrong, and updating it is a separate act of discipline from the work it describes. So it rots — or, more often, it never gets started, and \u0026ldquo;what changed in this release?\u0026rdquo; becomes an archaeology exercise across git log and merged PRs.\nThe information is already there. Every change is a commit, and a commit already has a message. The problem is that free-form commit messages aren\u0026rsquo;t structured enough to generate anything from. \u0026ldquo;Fixing up dependencies\u0026rdquo;, \u0026ldquo;Latest content\u0026rdquo;, \u0026ldquo;Typo correction\u0026rdquo; — I found all of these in my own history — tell a tool nothing about whether a change was a feature, a fix, or a chore, and nothing about what to put under which heading.\nThe fix is to treat the commit message as structured data: adopt a grammar strict enough that a tool can parse it, enforce that grammar at commit time, and then generate the changelog from the parsed history. Three moving parts:\nConventional Commits — the grammar. A commit-msg hook (managed by Lefthook) — the enforcement. git-cliff — the generator that turns the history into a Keep a Changelog document. This is the same instinct that runs through the rest of this blog: do the work at build time, not by hand, and not at the moment someone is trying to cut a release.\nConventional Commits: the grammar Conventional Commits is a tiny specification for the first line of a commit message:\ntext Copy 1 \u0026lt;type\u0026gt;[(scope)][!]: \u0026lt;description\u0026gt; A type from a fixed set, an optional scope in parentheses, an optional ! to flag a breaking change, then a colon and a description. A few real examples from these repos:\ntext Copy 1234 feat(nav): add mobile menu fix: correct hero lede fallback ci: lint with ruff instead of flake8 chore(release): 1.0.0 The types I settled on are the eleven from @commitlint/config-conventional — the most widely adopted ruleset, descended from Angular\u0026rsquo;s original commit convention: feat, fix, docs, style, refactor, perf, test, build, ci, chore and revert. That is enough vocabulary to describe any change without agonising over the boundary. feat and fix are the two that carry semantic weight — they map onto minor and patch version bumps — and everything else is bookkeeping.\nThe value only appears once the format is enforced. A convention that is merely documented is a convention that half your commits ignore, and half is worse than none because now your generator produces a changelog with holes in it.\nEnforcing it with a dependency-free hook The obvious tool here is commitlint, and if every repo were a Node project I would have used it. But only four of my eleven repos have a package.json; the rest are Python, or Go, or plain content. Adding a Node toolchain to a Python OCR pipeline purely to lint commit messages is the sort of incidental complexity that standardisation is supposed to remove, not create.\nSo the enforcement is a single POSIX sh script with no dependencies, identical in every repo. Here is the core of it:\n.githooks/commit-msg sh Copy 123456789101112131415161718192021222324252627282930313233343536373839 #!/bin/sh # commit-msg hook — enforce Conventional Commits across all projects. msg_file=\u0026#34;$1\u0026#34; commit_source=\u0026#34;$2\u0026#34; # Skip auto-generated merge/squash messages. case \u0026#34;$commit_source\u0026#34; in merge | squash) exit 0 ;; esac # Header = first non-comment, non-blank line. header=$(grep -v \u0026#39;^#\u0026#39; \u0026#34;$msg_file\u0026#34; | sed \u0026#39;/^[[:space:]]*$/d\u0026#39; | head -n 1) # Skip commit kinds that have their own conventions. case \u0026#34;$header\u0026#34; in \u0026#34;Merge \u0026#34;* | \u0026#34;Revert \u0026#34;* | \u0026#34;fixup! \u0026#34;* | \u0026#34;squash! \u0026#34;*) exit 0 ;; esac types=\u0026#39;feat|fix|docs|style|refactor|perf|test|build|ci|chore|revert\u0026#39; if printf \u0026#39;%s\u0026#39; \u0026#34;$header\u0026#34; | grep -Eq \u0026#34;^(${types})(\\([a-z0-9._/-]\u0026#43;\\))?(!)?: .\u0026#43;\u0026#34;; then # Description must not end with a full stop. desc=$(printf \u0026#39;%s\u0026#39; \u0026#34;$header\u0026#34; | sed -E \u0026#39;s/^[^:]\u0026#43;: //\u0026#39;) case \u0026#34;$desc\u0026#34; in *.) echo \u0026#34;✖ commit-msg: no trailing full stop in the description\u0026#34; \u0026gt;\u0026amp;2; exit 1 ;; esac else echo \u0026#34;✖ commit-msg: header must follow Conventional Commits:\u0026#34; \u0026gt;\u0026amp;2 echo \u0026#34; \u0026lt;type\u0026gt;[(scope)][!]: \u0026lt;description\u0026gt;\u0026#34; \u0026gt;\u0026amp;2 echo \u0026#34; allowed types: ${types}\u0026#34; \u0026gt;\u0026amp;2 exit 1 fi # Hard limit 100 characters on the header (72 preferred). len=$(printf \u0026#39;%s\u0026#39; \u0026#34;$header\u0026#34; | wc -m | tr -d \u0026#39; \u0026#39;) [ \u0026#34;$len\u0026#34; -gt 100 ] \u0026amp;\u0026amp; { echo \u0026#34;✖ commit-msg: header is $len chars; keep it ≤ 100\u0026#34; \u0026gt;\u0026amp;2; exit 1; } exit 0 The real version also warns (without blocking) on headers over 72 characters and on body lines over 72, and enforces the blank line between header and body. But the shape above is the whole idea: a regex for the header, a check that the description doesn\u0026rsquo;t end in a full stop, and a length gate. Forty lines of shell, no node_modules, works the same whether the repo is Hugo or Python.\nOne honest limitation: a commit-msg hook runs locally only. Commits made through the GitHub web UI, and Dependabot\u0026rsquo;s automated commits, never touch it. That is a gap I accept — those commits are already well-formed (Dependabot writes build(deps): by default), and the alternative, a CI check that rejects PRs on message format, is more friction than it is worth for a personal workspace. If you are on a team, add that CI check; the same regex works there.\nLefthook: managing the hooks A hook script in a file does nothing until Git knows to run it. The naïve approach is git config core.hooksPath .githooks, pointing Git at a committed directory. That works, and it is where I started, but it has a sharp edge I walked straight into.\nThe core.hooksPath trap Git consults exactly one hooks path. If you set core.hooksPath, Git ignores .git/hooks entirely. So the moment you also want a proper hook manager — to run linters on staged files, a build on push, several commands per hook — you discover that the manager installs into .git/hooks, which Git is now ignoring because you told it to look at .githooks instead. The hooks silently never fire.\nThe resolution is to pick one owner. I moved to Lefthook and let it own .git/hooks, then had Lefthook invoke the shared commit-msg script as a command. You must unset core.hooksPath for this to work — the two mechanisms are mutually exclusive, not complementary.\nOne config, three hooks Lefthook is a single Go binary configured by one YAML file. Here is the whole thing for this blog:\nlefthook.yml yaml Copy 123456789101112131415161718192021222324 pre-commit: parallel: true commands: eslint: glob: \u0026#34;*.{js,mjs,cjs}\u0026#34; run: npx eslint {staged_files} stylelint: glob: \u0026#34;*.{css,scss,sass}\u0026#34; run: npx stylelint {staged_files} markdownlint: glob: \u0026#34;*.md\u0026#34; exclude: - CLAUDE.md run: npx markdownlint-cli2 {staged_files} pre-push: commands: build: run: npm run build commit-msg: commands: conventional: run: .githooks/commit-msg {1} Three things worth drawing out:\npre-commit lints only staged files. The {staged_files} template, filtered by glob, means a commit that touches one SCSS file runs Stylelint on that file and nothing else. Commits stay fast, and — importantly — the linters don\u0026rsquo;t retroactively rewrite files you didn\u0026rsquo;t touch. Legacy code is left alone until you next edit it. The heavy job goes on pre-push, not pre-commit. A full Hugo build takes seconds, not milliseconds. Running it on every commit would make committing painful; running it on push catches a broken build before it ever reaches CI, which is exactly when you want to know. commit-msg just calls the shared script. Lefthook passes the message-file path through as {1}. The polyglot enforcement and the per-language linting live in the same config, but the Conventional Commits logic stays in the one portable shell script that every repo shares. Installation is wired into whatever each ecosystem already runs. For the Node repos that is the prepare script, so npm install sets the hooks up with no extra step:\npackage.json json Copy 12345 { \u0026#34;scripts\u0026#34;: { \u0026#34;prepare\u0026#34;: \u0026#34;lefthook install || true\u0026#34; } } For the Python repos, brew install lefthook plus a line in the README covers it. The point is that a fresh clone plus the normal install command leaves you with working hooks and no manual \u0026ldquo;don\u0026rsquo;t forget to run lefthook install\u0026rdquo; ritual.\nKeep a Changelog: the format With commit messages now structured, the destination needs a shape too. Keep a Changelog is the de facto standard: a reverse-chronological list of released versions, each grouped under headings — Added, Changed, Fixed, Removed, Deprecated, Security — with a link to the compare view for each release. It reads like this:\nCHANGELOG.md markdown Copy 1234567891011 ## [1.0.0] - 2026-07-10 ### Added - Add Plausible analytics and fix npm audit vulnerabilities - Add SEO improvements: meta tags, Open Graph, Twitter Cards, JSON-LD ### Changed - Modernise tooling and image shortcode - Document security scanning setup in CLAUDE.md The reason to keep a curated file at all, rather than pointing people at git log, is that a changelog is written for humans deciding whether to upgrade, not for machines reconstructing history. It is grouped, de-noised, and phrased for a reader. The trick is to generate that human-readable document from machine-readable commits — which is exactly what git-cliff does.\ngit-cliff: generating the changelog git-cliff is a changelog generator (written in Rust, distributed as a single binary) that parses your commit history according to Conventional Commits and renders it through a template. One cliff.toml per repo configures both the parsing and the output.\nThe template half emits the Keep a Changelog structure — the header, the per-release ## [version] - date blocks, and the compare-view links in the footer. The interesting half is the commit_parsers array, which maps commits onto the changelog\u0026rsquo;s section headings:\ncliff.toml toml Copy 1234567891011121314 [git] conventional_commits = true filter_unconventional = false commit_parsers = [ { message = \u0026#34;^Merge \u0026#34;, skip = true }, { message = \u0026#34;^.*: add\u0026#34;, group = \u0026#34;Added\u0026#34; }, { message = \u0026#34;^.*: remove\u0026#34;, group = \u0026#34;Removed\u0026#34; }, { message = \u0026#34;^fix\u0026#34;, group = \u0026#34;Fixed\u0026#34; }, { message = \u0026#34;^.*: fix\u0026#34;, group = \u0026#34;Fixed\u0026#34; }, { message = \u0026#34;^.*\u0026#34;, group = \u0026#34;Changed\u0026#34; }, ] sort_commits = \u0026#34;oldest\u0026#34; Each commit is tested against the parsers in order and lands in the first group it matches. fix: commits become Fixed, anything mentioning \u0026ldquo;add\u0026rdquo; becomes Added, and the catch-all ^.* sweeps everything else into Changed. The { message = \u0026quot;^Merge \u0026quot;, skip = true } line at the top is doing real work: it drops the \u0026ldquo;Merge pull request #204…\u0026rdquo; commits that a PR-based workflow generates, which would otherwise be noise in every release.\nGenerating or updating the changelog is then one command:\nbash Copy 1 git cliff --output CHANGELOG.md The baseline gotcha The first time you run git-cliff on an existing repo, it parses the entire history back to the first commit — including all the pre-Conventional-Commits mess. My baseline changelogs came out with \u0026ldquo;Initial commit\u0026rdquo; and \u0026ldquo;Fixing up dependencies\u0026rdquo; duplicated across groups, because the heuristic parsers do their best with messages that predate the convention. That is fine for a one-time baseline: you accept a slightly untidy first entry, and everything from that point forward is clean because every new commit is conventional. If it bothers you, hand-edit the baseline once; it is a document, not a build artefact.\nClosing the loop: automate it in CI Everything so far still has one manual step: someone has to remember to run git cliff and commit the result before cutting a release. That is the weak link — the exact discipline that lets changelogs rot in the first place. So the last move is to take the human out of it and regenerate the changelog in CI, on merge to main.\ngit-cliff can compute the next version from the commits it sees (a feat implies a minor bump, a fix a patch, a ! a major) with --bump, so the whole thing is one action:\n.github/workflows/release.yml yaml Copy 12345678910111213141516171819202122232425262728293031323334 name: Release on: push: branches: [main] permissions: contents: write jobs: changelog: # Don\u0026#39;t retrigger on the release commit this job itself pushes. if: \u0026#34;!contains(github.event.head_commit.message, \u0026#39;chore(release):\u0026#39;)\u0026#34; runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 with: fetch-depth: 0 - name: Generate changelog and compute next version id: cliff uses: orhun/git-cliff-action@v4 with: args: --bump --output CHANGELOG.md - name: Commit, tag and push run: | version=\u0026#34;${{ steps.cliff.outputs.version }}\u0026#34; git config user.name \u0026#34;github-actions[bot]\u0026#34; git config user.email \u0026#34;github-actions[bot]@users.noreply.github.com\u0026#34; git add CHANGELOG.md git commit -m \u0026#34;chore(release): ${version#v}\u0026#34; git tag \u0026#34;$version\u0026#34; git push --follow-tags On every merge to main, git-cliff regenerates the changelog, works out the next version from the commits, and the job commits the updated CHANGELOG.md, tags the release, and pushes. The if: guard is essential — without it the job\u0026rsquo;s own chore(release): commit would trigger the workflow again, and you would have an infinite loop of releases.\nThere is one wrinkle if main is branch-protected (mine is — it requires passing checks before anything merges). A protected branch won\u0026rsquo;t accept a direct push from CI, so instead of pushing to main you regenerate the changelog on the branch that is about to be merged. On this blog the flow is push to dev → CI opens a PR to main → merge; the natural home for the git-cliff step is on that dev push, so the changelog update travels into main as part of the reviewed PR rather than being forced in afterwards. Either way, the principle holds: the changelog is regenerated by a machine from structured commits, and no human is trusted to remember.\nTeach your AI assistant the convention One more place the convention should live: your AI coding assistant\u0026rsquo;s project instructions. If you use Claude Code, that is CLAUDE.md at the repo root; for GitHub Copilot it is .github/copilot-instructions.md. The assistant reads the file at the start of every session, so a convention written there is one the assistant follows without being told — including when it writes your commit messages for you.\nThe section I added to every repo is short:\nCLAUDE.md markdown Copy 1234567891011121314 ## Commit messages All commits follow Conventional Commits, enforced by a commit-msg hook: \u0026lt;type\u0026gt;[(scope)][!]: \u0026lt;description\u0026gt; - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore, revert. - Description in the imperative mood, no trailing full stop; header 72 characters or fewer. - The type drives the changelog and the version bump (fix → patch, feat → minor, ! → major) — choose it for what the change means to a reader, not for what is convenient. - Never bypass the hooks with --no-verify. Two things make this worth the five minutes. First, it dissolves the main objection to adopting a commit grammar — that writing to a format is friction — because the assistant drafts a well-formed message from the staged diff and you just review it. Second, the note about types driving the version bump matters more than it looks: once git-cliff derives semver from your commits, the choice between fix and chore is a versioning decision, and the assistant should know that when it proposes one.\nThe instructions and the hook are complementary, not redundant: the instructions file makes the assistant produce conforming messages, and the hook remains the guarantee for every commit regardless of who — or what — wrote it. Guidance in the instructions, enforcement in the hook.\nWhat you actually get Put together, the pieces reinforce each other:\nConventional Commits give every change a machine-readable shape. The shared commit-msg hook makes that shape non-optional, in one portable script that doesn\u0026rsquo;t care what language the repo is written in — and writing to the format is barely any work now that an AI coding assistant such as Claude Code will draft a well-formed Conventional Commit straight from the staged diff, so conforming costs essentially nothing at the keyboard. Lefthook installs and runs that hook — plus staged-file linting and a pre-push build — from one config, on a plain install. Keep a Changelog gives the output a human-readable structure. git-cliff generates that structure from the history, and CI keeps it current on every merge. The result across eleven repos is that commit messages are consistent enough to be useful, every project has a changelog, and — the part that actually matters — none of it depends on anyone remembering to do it. The discipline lives in the tooling, which is the only place discipline reliably survives.\nIf you adopt one piece, make it the commit convention: it is the input everything else feeds on, and it costs nothing but a forty-line hook.\n"},{"url":"/2026/07/shipping-the-release-pipeline-git-cliff-in-github-actions-across-ten-repos/","title":"Shipping the release pipeline: git-cliff in GitHub Actions across ten repos","summary":"A follow-up to the Conventional Commits post — rolling out automated semantic versioning, changelog updates and GitHub Releases with git-cliff in GitHub Actions across ten repositories, and the gotchas found on the way: release commits polluting the changelog, GITHUB_TOKEN loop protection, and a branch-protected main","date":"2026-07-17","tags":["devops","git","automation","ai"],"cover":"mint","body":"The previous post ended with a recommendation: close the loop by regenerating the changelog in CI, so no human is ever trusted to remember. At the time that step was the one piece I hadn\u0026rsquo;t actually built. This post is the report from building it — the same evening, across all ten repositories in my workspace — and the three or four wrinkles that only showed up once the workflow met real repos.\nThe goal, precisely: on every push or merge to main, compute the next semantic version from the Conventional Commit history, regenerate CHANGELOG.md for that version, tag it, and publish a GitHub Release whose notes are that version\u0026rsquo;s changelog section. Version first, changelog second — the version number is derived from the commits, and the changelog is written under it.\nThe workflow Nine of the ten repos commit directly to an unprotected main, and they all share one workflow file. The core of it:\n.github/workflows/release.yml yaml Copy 1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556 name: Release on: push: branches: [main] permissions: contents: write jobs: release: # Belt and braces: never react to our own release commit. if: ${{ !startsWith(github.event.head_commit.message, \u0026#39;chore(release):\u0026#39;) }} runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 with: fetch-depth: 0 - name: Install git-cliff uses: taiki-e/install-action@v2 with: tool: git-cliff - name: Compute next version id: version run: | next=\u0026#34;$(git cliff --bumped-version)\u0026#34; latest=\u0026#34;$(git describe --tags --abbrev=0 2\u0026gt;/dev/null || echo \u0026#39;\u0026#39;)\u0026#34; echo \u0026#34;next=$next\u0026#34; \u0026gt;\u0026gt; \u0026#34;$GITHUB_OUTPUT\u0026#34; if [ -n \u0026#34;$next\u0026#34; ] \u0026amp;\u0026amp; [ \u0026#34;$next\u0026#34; != \u0026#34;$latest\u0026#34; ]; then echo \u0026#34;release=true\u0026#34; \u0026gt;\u0026gt; \u0026#34;$GITHUB_OUTPUT\u0026#34; else echo \u0026#34;release=false\u0026#34; \u0026gt;\u0026gt; \u0026#34;$GITHUB_OUTPUT\u0026#34; fi - name: Update changelog, tag and release if: steps.version.outputs.release == \u0026#39;true\u0026#39; env: VERSION: ${{ steps.version.outputs.next }} GH_TOKEN: ${{ github.token }} run: | set -euo pipefail git cliff --bump --output CHANGELOG.md git cliff --bump --unreleased --strip all --output RELEASE_NOTES.md git config user.name \u0026#34;github-actions[bot]\u0026#34; git config user.email \u0026#34;41898282\u0026#43;github-actions[bot]@users.noreply.github.com\u0026#34; git add CHANGELOG.md git commit -m \u0026#34;chore(release): ${VERSION#v}\u0026#34; git tag -a \u0026#34;$VERSION\u0026#34; -m \u0026#34;$VERSION\u0026#34; git push origin HEAD:main git push origin \u0026#34;$VERSION\u0026#34; gh release create \u0026#34;$VERSION\u0026#34; --title \u0026#34;$VERSION\u0026#34; --notes-file RELEASE_NOTES.md git cliff --bumped-version does the semver arithmetic: it reads the conventional commits since the last tag and applies the standard rules — a fix implies a patch bump, a feat a minor, a ! (breaking change) a major. Comparing the result against the latest existing tag gives a clean no-op path: if nothing releasable has landed, the job exits without touching anything. The second git cliff invocation, with --unreleased --strip all, emits just the new version\u0026rsquo;s section — no Keep a Changelog header, no link footer — which is exactly what you want as GitHub Release notes.\nEverything runs on the built-in GITHUB_TOKEN. No personal access tokens, no secrets to create or rotate in any of the ten repos.\nGotcha one: your own release commits poison the changelog The first dry run produced a changelog with a mysterious bullet reading, in its entirety, - 0.1.0. That is the previous release commit — chore(release): 0.1.0 — being swept up by the catch-all parser and rendered under Changed with its type prefix stripped. Harmless once; with automation cutting a release on every merge, the changelog would steadily fill with bullets that just say version numbers.\nThe fix is two more skip parsers at the top of cliff.toml, alongside the existing one for merge commits:\ncliff.toml toml Copy 123456 commit_parsers = [ { message = \u0026#34;^Merge \u0026#34;, skip = true }, { message = \u0026#39;^chore\\(release\\)\u0026#39;, skip = true }, { message = \u0026#39;^chore\\(changelog\\)\u0026#39;, skip = true }, # ... the grouping parsers follow ] The general lesson: any commit the pipeline itself generates must be invisible to the pipeline\u0026rsquo;s parser, or the machinery\u0026rsquo;s own bookkeeping leaks into the document it maintains.\nGotcha two: the infinite loop that never happens The obvious worry with a workflow that pushes a commit to main on push to main is recursion. The workflow\u0026rsquo;s if: guard — skip when the head commit is a chore(release): — addresses it, but it turns out to be belt and braces rather than load-bearing: pushes made with the built-in GITHUB_TOKEN deliberately do not trigger workflows. GitHub designed exactly this footgun away. The release commit lands on main and nothing fires.\nThat protection has a flip side worth knowing about. If a repo has other workflows on push to main — a Cloudflare Pages deploy, say — those don\u0026rsquo;t fire for the release commit either. The updated CHANGELOG.md won\u0026rsquo;t be deployed to the site until the next ordinary push. For a changelog that is a non-issue; if the pipeline ever touched something user-facing, it would matter, and the fix would be a deploy key or app token in place of GITHUB_TOKEN.\nGotcha three: the branch-protected repo Nine repos take the workflow above unchanged. The tenth — this blog — has a protected main that only accepts changes through a reviewed PR from dev, with four passing checks. A workflow cannot push a release commit to that main, and I didn\u0026rsquo;t want to weaken the protection or mint a bypass token just for a changelog.\nThe answer is to split the job in two along the existing flow:\nchangelog.yml runs on push to dev. It regenerates CHANGELOG.md with the next computed version (git cliff --bump) and commits it back to dev if it changed. The changelog update then rides into main inside the same auto-created PR as the work it describes — reviewed, checked, and merged like everything else. release.yml runs on push to main — which, on this repo, means \u0026ldquo;when the PR merges\u0026rdquo;. It never pushes a commit. It recomputes the bumped version, and if that tag doesn\u0026rsquo;t exist yet, creates the annotated tag and publishes the GitHub Release. A tag is its own ref; branch protection on main doesn\u0026rsquo;t stand in its way. The division of labour falls out naturally: the document travels through the front door with the code, and only the immutable markers — tag and Release — are created on the far side. No secrets, no bypass, and the protection stays intact.\nPermissions, quietly One repo had its Actions default token permission set to read-only rather than write. I expected that to be a manual settings fix, but it wasn\u0026rsquo;t: a workflow-level permissions: contents: write block requests the elevated scope explicitly, and GitHub grants it regardless of the repo default. Declaring permissions in the workflow — which you should do anyway, as documentation of intent — also makes the workflow portable across repos with different defaults. All ten repos run the pipeline with zero settings changes.\nCanary first Rolling an untested workflow to ten repositories simultaneously is how you get ten identical failures. I applied the pipeline to one repo first — the smallest, least consequential one — watched the Actions run complete, and verified all three artefacts: the chore(release): 0.1.1 commit on main, the v0.1.1 tag, and the GitHub Release with correctly grouped notes. Only then did the other nine get the same files.\nThe first automated run across the fleet:\nRepository First automated release xxxxxx_archive_directory (canary) v0.1.1 codespace v0.1.1 xxxxxx-qa-playwright v1.0.1 xxxxxx-qa-cypress v1.0.1 armenian-institute-publishing v0.1.1 scatterpub-toolchain v0.1.1 scatterpub-toolchain-example v0.1.1 ocr-pipeline-demo v0.1.1 velostevie v2.0.1 scattercode.dev v1.1.0, via the dev → main PR Every repo had accumulated a few conventional commits since its baseline tag, so every repo cut a patch release on its first run — a nice end-to-end proof that the whole chain works, from commit message grammar to published Release notes.\nTeaching your AI assistant to maintain it Here is the part I have started doing for every convention I adopt, and it deserves its own habit: write the convention into the AI assistant\u0026rsquo;s project instructions. This entire rollout was done in a Claude Code session, and the reason it could be is that each repo\u0026rsquo;s CLAUDE.md already documented the commit format and the hooks — the assistant read the rules and worked within them, writing every one of the rollout\u0026rsquo;s own commits as well-formed Conventional Commits. The release pipeline itself is the newest convention, so it gets the same treatment.\nAn automated release pipeline changes what a contributor — human or machine — must not do: hand-edit the changelog, or tag versions manually. Unless you say so, an eager assistant asked to \u0026ldquo;cut a release\u0026rdquo; will happily do both. So the project instructions need a section like this — CLAUDE.md for Claude Code, .github/copilot-instructions.md for GitHub Copilot, same content either way:\nCLAUDE.md markdown Copy 12345678910111213141516 ## Releases and changelog Releases are fully automated — do not perform them by hand. - Every push to `main` runs `.github/workflows/release.yml`: git-cliff computes the next semantic version from the Conventional Commit history, regenerates `CHANGELOG.md`, tags the version, and publishes a GitHub Release. - Never edit `CHANGELOG.md` manually — it is generated by git-cliff from `cliff.toml`. To change what appears in it, fix the commit messages or the `commit_parsers` in `cliff.toml`. - Never create version tags or GitHub Releases manually. - Version bumps follow the commit types: `fix` → patch, `feat` → minor, a `!` breaking change → major. Choose commit types accordingly. - Any commit generated by tooling must use `chore(release):` or `chore(changelog):` so the changelog parser skips it. And if you are starting from a repo that doesn\u0026rsquo;t have the pipeline yet, the ask is a single prompt: \u0026ldquo;Add automated releases with git-cliff: on push to main, compute the next semver from conventional commits, regenerate CHANGELOG.md, tag, and publish a GitHub Release. Add skip parsers to cliff.toml so release commits stay out of the changelog, and use the built-in GITHUB_TOKEN so the release commit doesn\u0026rsquo;t retrigger workflows.\u0026rdquo; Every constraint in that sentence is one of this post\u0026rsquo;s gotchas — which is really the point: the prompt encodes the experience, and the instructions file keeps it encoded for every future session.\nThe last sentence of the previous post said the discipline should live in the tooling, because that is the only place discipline reliably survives. Instruction files are the same move one level up: the knowledge of how the tooling works also shouldn\u0026rsquo;t live in anyone\u0026rsquo;s head — not mine, and not in whatever the assistant happens to infer. Write it down where the machines read it.\n"},{"url":"/2026/05/from-scans-to-published-ebook-building-a-digital-publication-pipeline/","title":"From Scans to Published Ebook: Building a Digital Publication Pipeline","summary":"How I built a composable pipeline to take physical book scans through OCR, automated cleaning, AI copy-editing, and Vellum layout — and packaged it as a reusable toolchain with a step-by-step tutorial.","date":"2026-05-10","tags":["python","publishing","ocr","vellum","claude","git"],"cover":"mint","body":"The Armenian Institute holds a collection of out-of-print books that deserve a second life. The texts exist as physical copies and sometimes as flat scans — but not in any form that can be copy-edited, laid out for modern ebook formats, or distributed digitally. Getting from a shelf of scanned pages to a published epub involves a chain of steps, each of which introduces a different kind of problem. I built a pipeline to automate as much of that chain as possible, and packaged it as a reusable toolchain that anyone can adopt for their own digitisation project.\nThe result is scatterpub-toolchain, an open-source set of Python scripts and Claude Code skills for taking physical book scans to a finished ebook. There is also a companion example project with real sample scans and a step-by-step tutorial that gets you from zero to a Word document ready for Vellum import in around thirty minutes.\nThe Problem With Digitising Literary Texts Digitising a technical document is relatively forgiving. A few garbled words in a user manual are easy to spot because the text is structured and factual. Literary prose is harder. A missed letter can produce a word that looks plausible in isolation but is wrong in context: bom for born, Westem for Western. A fused word — ofMezre, onthe — reads as noise in a technical document but might almost pass in a novel.\nThe pipeline has to handle three distinct categories of error:\nMechanical OCR artefacts — running headers, invisible characters, hyphenated line-breaks — that are structural and regular enough for a script to catch. Contextual OCR errors — dropped characters, character substitutions, fused words — that require reading in context. Style and editorial issues — punctuation inconsistencies, British versus American spelling, capitalisation — that require applying a specific style guide. Each category needs a different tool.\nThe Pipeline text Copy 1234567891011121314 Scanned page PDFs ↓ ocr-to-markdown.py Raw Markdown ← running headers, invisible chars, line-break hyphens ↓ clean-ocr.py --join-hyphens --reflow Clean Markdown ← mechanically corrected, reflowed paragraphs ↓ Claude Code: /copyeditor (OCR artefact pass) AI-corrected MD ← fused words, dropped chars, proper noun fixes ↓ copy into draft/, manual review Draft Markdown ← human-edited, then imported into Vellum ↓ Vellum (layout and design) ↓ Claude Code: /copyeditor (style review) Final Vellum file ← copy-edited against Hart\u0026#39;s Rules or CMOS ↓ md-to-docx.py (or Vellum direct export) Published ebook / Word document Step 1: OCR with marker-pdf scripts/ocr-to-markdown.py takes a folder of per-page PDFs and produces a single Markdown file. The default OCR engine is marker-pdf, a machine-learning model that significantly outperforms traditional OCR engines on book typography.\nbash Copy 123 python3 toolchain/scripts/ocr-to-markdown.py \\ \u0026#34;publishing/tell-me-bella/ocr/scans/clean\u0026#34; # → publishing/tell-me-bella/ocr/tell-me-bella-raw.md The raw output has several predictable problems. Running headers — the page number and book title repeated at the top of every page — appear as stray paragraphs. Invisible Unicode characters creep in (soft hyphens, non-breaking spaces, zero-width joiners). Words split across lines by the typesetter become news-\\npaper in the OCR output.\nGetting clean scans The quality of the raw OCR output depends almost entirely on the quality of the input scans. Professional book scanning equipment presses a sheet of glass against the page to flatten it; a hand-held camera over a slightly warped page produces curved baselines that confuse any OCR engine. Practical tips that made a significant difference:\nAlign pages in Acrobat before OCR. I crop each scan to a clean rectangular block of text with no opposite-page bleed visible. Put black paper behind the page. Text on the verso side bleeds through the page under bright scanning light. Black paper behind the page eliminates this entirely. Scan each page individually. A spread scan with the gutter in frame gives the OCR engine two columns of text and a curved spine to deal with. Step 2: Mechanical Cleaning with clean-ocr.py The cleaning script addresses artefacts that have a regular enough structure to catch programmatically.\nbash Copy 1234 python3 toolchain/scripts/clean-ocr.py \\ \u0026#34;publishing/tell-me-bella/ocr/tell-me-bella-raw.md\u0026#34; \\ --join-hyphens --reflow # → publishing/tell-me-bella/ocr/tell-me-bella-clean.md --join-hyphens A word split across a line break by the typesetter becomes two fragments joined by a hyphen: news-\\npaper. The script detects this pattern and joins them: news-paper. The hyphen is kept so a human can review whether to remove it (newspaper) or retain it (news-paper).\n--reflow OCR preserves the typeset line-breaks from the printed page. A paragraph in the original book might span seven lines of text; the raw output has seven separate lines. --reflow joins consecutive non-blank lines into a single long line, so each paragraph becomes one continuous string — standard Markdown prose style and much easier to read and search.\npython Copy 123456789101112131415 def reflow_paragraphs(text): lines = text.split(\u0026#39;\\n\u0026#39;) result, buffer = [], [] for line in lines: stripped = line.rstrip() if stripped == \u0026#39;\u0026#39;: if buffer: result.append(\u0026#39; \u0026#39;.join(buffer)) buffer = [] result.append(\u0026#39;\u0026#39;) else: buffer.append(stripped) if buffer: result.append(\u0026#39; \u0026#39;.join(buffer)) return \u0026#39;\\n\u0026#39;.join(result) Typography normalisation The script also runs three typography passes that are always on:\nSmart quotes — straight ' and \u0026quot; converted to curly equivalents. The algorithm uses the preceding character to determine direction: a quote preceded by a space, newline, or opening bracket opens; otherwise it closes. YAML front matter is excluded. En-dash normalisation — - (space-hyphen-space) converted to – (spaced en dash), the Hart\u0026rsquo;s Rules style for parenthetical asides. Ellipsis normalisation — ... converted to … (the Unicode ellipsis character). Step 3: AI OCR Correction with Claude Code The cleaning script catches structural artefacts, but some errors require reading in context. The copyeditor skill has a dedicated OCR artefact pass for this:\ntext Copy 123456 /copyeditor Please apply an OCR artefact correction pass to publishing/tell-me-bella/ocr/tell-me-bella-clean.md and write the corrected version to publishing/tell-me-bella/ocr/tell-me-bella-ai-clean.md The skill knows what to look for:\nError type Example Cause Fused words ofMezre, OldRomanRoad Word boundary lost by OCR Dropped characters bom for born, Westem for Western r, n, rn cluster missed d as cl Sadcller for Saddler Typeface ambiguity Spurious characters in names Tow:vanda, Ktikor OCR inserting noise into proper nouns Digit spacing 189 3 for 1893 OCR splitting a number The AI pass is positioned after the script-clean step so Claude sees reflowed, artefact-free text — it only has to think about contextual errors, not mechanical ones.\nStep 4: Vellum for Layout After manual review and editing, the draft Markdown is imported into Vellum for layout. Vellum handles the visual design, chapter structure, and multi-format export (epub for all the major retailers, PDF for print).\nThe .vellum file format is an NSKeyedArchiver binary property list. Newer versions of Vellum save as a ZIP archive containing content.vellumcontent; older versions save as a macOS package directory with the same file at its root. The extract-vellum.py script handles both:\npython Copy 123456789 def _load_plist(vellum_path: Path) -\u0026gt; dict: if vellum_path.is_dir(): content_path = vellum_path / \u0026#39;content.vellumcontent\u0026#39; with open(content_path, \u0026#39;rb\u0026#39;) as f: return plistlib.load(f) else: with zipfile.ZipFile(vellum_path) as z: with z.open(\u0026#39;content.vellumcontent\u0026#39;) as f: return plistlib.load(io.BytesIO(f.read())) io.BytesIO wraps the ZIP member bytes so plistlib.load() gets a seekable binary file object — the ZipExtFile returned by z.open() is binary but not seekable, and plistlib needs to seek.\nInside the plist, the object graph is an NSKeyedArchiver archive. Python\u0026rsquo;s standard library plistlib handles binary plists natively. The traversal pattern is:\npython Copy 123456 objects = plist[\u0026#39;$objects\u0026#39;] # flat array of all objects def deref(objects, ref): if isinstance(ref, plistlib.UID): return objects[ref.data] # .data, not .integer — attribute changed between Python versions return ref Each chapter node has a typeName ('foreword', 'chapter', 'epilogue', etc.), a title, and a text field containing an NSAttributedString whose plain text lives under its NSString key.\nStep 5: AI Copy-Edit Review Once the book is in Vellum and close to final, the copyeditor skill produces an HTML annotation report. The .vellum file is the source of truth at this stage — it is extracted to Markdown, reviewed, and any agreed changes go back into Vellum:\nbash Copy 123 python3 toolchain/scripts/extract-vellum.py \\ \u0026#34;publishing/tell-me-bella/Tell me, Bella.vellum\u0026#34; \\ \u0026#34;publishing/tell-me-bella/draft/tell-me-bella.md\u0026#34; text Copy 1234 /copyeditor Please produce a full copy-edit review of publishing/tell-me-bella/draft/tell-me-bella.md The review is written to a self-contained HTML file with embedded CSS that opens in any browser without a build step.\nThe style guide system The skill selects its style guide from the language field in book.md:\nmarkdown Copy 12345 --- title: \u0026#34;Tell me, Bella\u0026#34; author: Vahan Totovents language: en-GB --- language Style guide en-GB Hart\u0026rsquo;s Rules (British English) en-US Chicago Manual of Style, 18th edition The two guides differ on several points that matter in literary texts:\nRule Hart\u0026rsquo;s (en-GB) CMOS (en-US) Primary quotes Single: 'like this' Double: \u0026quot;like this\u0026quot; Parenthetical dashes Spaced en dash – Unspaced em dash — -ize spellings Oxford -ize: realize, organize CMOS -ize (same) The issue taxonomy Every flagged issue is categorised and colour-coded in the HTML report:\nLabel Colour Use for TYPO Red Spelling errors, wrong or missing words PUNCT Orange Quotation marks, dashes, ellipsis STYLE Yellow Spelling variants, capitalisation, numbers CONSISTENCY Blue Same word or name formatted differently across the book QUERY Purple Ambiguous phrasing — flagged for a human to decide The HTML review report: colour-coded issue cards with context snippets and suggested corrections The QUERY category is particularly important for translated texts. Some unusual phrasings may be deliberate choices of the translator rather than errors. Flagging them as QUERY rather than correcting them keeps the editorial decision with the human.\nIterating to clean Each round of review produces fewer issues as the human works through them in Vellum. Over three passes on Tell me, Bella, the count went from 41 issues to 18 to 29 (the third pass catching subtler things the first two missed). The workflow is:\nExtract fresh Markdown from Vellum Run the copy-edit review Work through the HTML report and make agreed changes in Vellum Repeat The Toolchain as a Submodule The scripts and skills are packaged as scatterpub-toolchain, designed to be embedded in a book project repository as a git submodule. This keeps the toolchain versioned separately from the book content, and lets multiple book projects share the same toolchain while pinning independently to a known-good commit.\nbash Copy 12 git submodule add https://github.com/scattercode/scatterpub-toolchain.git toolchain git submodule update --init Claude Code skills are discovered by the IDE from .claude/skills/. Symlinking the skills from the submodule wires them up automatically for every contributor without any post-clone setup:\nbash Copy 12 mkdir -p .claude/skills ln -s ../../toolchain/.claude/skills/copyeditor .claude/skills/copyeditor Symlinks are tracked by git, so a fresh git clone --recurse-submodules of a book project gets everything in place.\nThe Example Project scatterpub-toolchain-example is a fork-ready project with real sample scans and a TUTORIAL.md covering all six parts of the pipeline:\nPart What it covers 1 Set up: clone, submodule init, Homebrew tools, Poetry, virtual environment 2 OCR the scans 3 Clean the raw output 4 AI OCR correction pass 5 AI copy-edit review 6 Generate a Word document The tutorial takes around thirty minutes end to end. The sample scans are real pages from a public-domain text, so the OCR output is genuinely imperfect and the cleaning and correction steps produce visible, meaningful changes.\nThe example project deliberately contains no Armenian Institute book data. This separation means the tutorial is self-contained and forkable — anyone can follow it without access to the original book files.\nWhat the Pipeline Doesn\u0026rsquo;t Do Page numbering. OCR text has no reliable page-number information. Running headers are stripped by the clean script, but the text is treated as a continuous stream. For footnotes that reference page numbers in the original, these need manual attention.\nIllustrations. The pipeline processes text only. Images in the original scan are ignored by the OCR engine. If a book contains plates or illustrations that need to appear in the digital edition, these must be handled separately.\nRight-to-left text. The pipeline was built for left-to-right English and Armenian Latin-transliterated text. RTL scripts (Armenian in the original script, Arabic, Hebrew) would need a different OCR configuration and different clean-up heuristics.\nLessons The clean step and the AI step do different things — keep them separate It is tempting to ask the AI to do everything: fix the mechanical artefacts and the contextual errors in one pass. In practice it is better to do the mechanical cleaning first. The script is faster and cheaper for what it can do, and it gives the AI a cleaner signal — Claude does not have to distinguish between a running header and a genuine title, or between an invisible character and a meaningful dash.\nThe .vellum file is the source of truth once layout begins All editorial changes after the Vellum import go back into Vellum, not into any intermediate Markdown file. The Markdown extract is a disposable snapshot for review purposes — it is generated fresh at the start of each review cycle. Treating the extract as editable breaks this constraint and introduces drift between the Markdown and the Vellum file.\nA style guide in code is a forcing function for consistency The copy-editor skill applies the same rules to every chapter, every session, regardless of how many times the book has been reviewed. A human editor working alone will be more rigorous on chapter one than chapter seven. The AI pass catches the same class of issues throughout — it does not get tired or start skipping edge cases near the end of a long document.\n"},{"url":"/2026/05/the-css-specificity-trap-that-killed-my-paragraph-spacing/","title":"The CSS specificity trap that killed my paragraph spacing","summary":"How a routine margin reset overrode the owl selector and made all my prose paragraphs run together — and the one-line fix.","date":"2026-05-04","tags":["css"],"cover":"pink","body":"I was looking at a freshly styled blog post and something felt wrong. The text was readable, the line height was fine, but the paragraphs looked wrong — like there was no gap between them. There was a gap, technically, but it was the same as the gap between lines in the same paragraph. The page felt like one continuous block of text.\nThe layout had looked fine in the mockup. Something had broken it when I wired it up to real content.\nThe setup The prose styles were built on the owl selector — a pattern for adding spacing between sibling elements without touching individual components:\ncss Copy 123 .prose \u0026gt; * \u0026#43; * { margin-top: var(--s-5); /* 24px */ } This adds a top margin to every direct child of .prose that follows another child: headings after paragraphs, paragraphs after headings, blockquotes, code blocks, all of it. One rule, no element-specific exceptions.\nThere was also a reset to kill the browser\u0026rsquo;s default paragraph margin:\ncss Copy 123 .prose p { margin: 0; } Browsers add margin-block-start and margin-block-end to \u0026lt;p\u0026gt; elements by default. If you don\u0026rsquo;t zero them out, they stack with whatever spacing your design adds, and you get gaps that are slightly too large and inconsistent across browsers.\nSo: owl selector adds spacing, margin reset kills the browser default. Except it also killed the owl selector\u0026rsquo;s spacing. Every \u0026lt;p\u0026gt; inside .prose had margin-top: 0, full stop.\nWhy it happened CSS specificity is calculated as three columns: ID selectors, class/attribute/pseudo-class selectors, and element/pseudo-element selectors. A higher number in any column beats a lower one to its left.\nRule IDs Classes Elements Specificity .prose \u0026gt; * + * 0 1 0 (0, 1, 0) .prose p 0 1 1 (0, 1, 1) .prose p has one more element selector than the owl selector, so it wins — regardless of which rule appears later in the source. Both rules target the same \u0026lt;p\u0026gt; element inside .prose. The reset wins, and the owl selector\u0026rsquo;s margin-top is overridden.\nThe common misconception is that source order is what matters. It does, but only as a tiebreaker when specificity is equal. Here they\u0026rsquo;re not equal, so source order is irrelevant.\nThe fix Add a third rule that is more specific than the reset, and only fires between adjacent paragraphs:\ncss Copy 123 .prose p \u0026#43; p { margin-top: var(--s-6); /* 32px */ } Rule IDs Classes Elements Specificity .prose \u0026gt; * + * 0 1 0 (0, 1, 0) .prose p 0 1 1 (0, 1, 1) .prose p + p 0 1 2 (0, 1, 2) .prose p + p wins over both. The reset still kills the browser default margin on every \u0026lt;p\u0026gt; (which is what it\u0026rsquo;s there for), and the p + p rule re-adds spacing only between consecutive paragraphs — which is exactly the case the owl selector was supposed to handle.\nI used --s-6 (32px) rather than the owl selector\u0026rsquo;s --s-5 (24px) to give paragraph breaks a bit more weight than other element transitions. Paragraphs after paragraphs need a clearer visual break than, say, a paragraph after a heading. That distinction was there in the original design and was worth preserving.\nThe general lesson The \u0026ldquo;reset to zero, then re-add where needed\u0026rdquo; pattern is common in CSS. It\u0026rsquo;s a sensible approach — clear out browser defaults, then apply your own spacing intentionally. The trap is when the reset selector is more specific than the rule that re-adds spacing.\nBefore writing element { margin: 0 }, check what selectors are responsible for adding that margin back. If the re-add rule has lower specificity than the reset, the re-add will silently lose every time, and you\u0026rsquo;ll spend a while wondering why the spacing you thought you defined isn\u0026rsquo;t showing up.\nThe owl selector in particular is vulnerable to this: it\u0026rsquo;s deliberately low-specificity (one class selector, two universal selectors) so it doesn\u0026rsquo;t get in the way. Any element-level reset inside the same scoping class will outrank it.\n"},{"url":"/2026/05/building-an-about-page-in-hugo-without-touching-single.html/","title":"Building an about page in Hugo without touching single.html","summary":"How to use Hugo's layout key to give a standalone page its own template, rather than bending a shared layout with conditionals.","date":"2026-05-02","tags":["hugo","devops"],"cover":"yellow","body":"The temptation You have a working layouts/_default/single.html for article pages. It renders a hero image, an eyebrow label, a date, and a comments section. Now you need an About page — same fonts, same nav, same footer, but none of that article-specific structure.\nThe tempting path: add a conditional.\ngo-html-template Copy 1234 {{ if ne .Type \u0026#34;about\u0026#34; }} \u0026lt;div class=\u0026#34;article-hero-image\u0026#34;\u0026gt;...\u0026lt;/div\u0026gt; \u0026lt;div class=\u0026#34;eyebrow\u0026#34;\u0026gt;Route report · {{ .Date.Format \u0026#34;January 2006\u0026#34; }}\u0026lt;/div\u0026gt; {{ end }} Don\u0026rsquo;t. Every conditional you add to a shared layout is a claim that two fundamentally different things are the same thing. single.html accumulates special cases over time, and eventually you\u0026rsquo;re reading a template full of if branches trying to reconstruct which of five page types you\u0026rsquo;re on.\nThe layout front matter key Hugo has a cleaner answer. Any content file can declare the template it wants:\nyaml Copy 1234 --- title: Hello. layout: about --- Hugo looks up layouts/_default/about.html and uses it for this page. single.html is never involved. The about page gets its own template, does exactly what it needs to do, and nothing else changes.\nThe layout file go-html-template Copy 12345678910111213141516171819202122232425 {{ define \u0026#34;main\u0026#34; }} \u0026lt;div class=\u0026#34;prose\u0026#34;\u0026gt; \u0026lt;header class=\u0026#34;about-header\u0026#34;\u0026gt; \u0026lt;div class=\u0026#34;eyebrow\u0026#34;\u0026gt;About\u0026lt;/div\u0026gt; \u0026lt;h1 class=\u0026#34;about-title\u0026#34;\u0026gt;{{ .Title }}\u0026lt;/h1\u0026gt; \u0026lt;/header\u0026gt; {{- with .Params.portrait -}} {{- $img := resources.Get (strings.TrimLeft \u0026#34;/\u0026#34; .src) -}} {{- if $img -}} {{- $portrait := $img.Resize \u0026#34;680x webp\u0026#34; -}} \u0026lt;figure class=\u0026#34;about-portrait\u0026#34;\u0026gt; \u0026lt;img src=\u0026#34;{{ $portrait.RelPermalink }}\u0026#34; alt=\u0026#34;{{ $.Params.portrait.caption | default $.Title }}\u0026#34; loading=\u0026#34;lazy\u0026#34;\u0026gt; {{- with $.Params.portrait.caption -}} \u0026lt;figcaption\u0026gt;{{ . }}\u0026lt;/figcaption\u0026gt; {{- end -}} \u0026lt;/figure\u0026gt; {{- end -}} {{- end }} {{ .Content }} \u0026lt;/div\u0026gt; {{ end }} {{ define \u0026quot;main\u0026quot; }} plugs into the baseof.html base template — the about page still gets the nav, footer, and any globally-loaded scripts. It just has a different main block.\nThe portrait image is optional. resources.Get returns nil if the file doesn\u0026rsquo;t exist; the {{ if $img }} guard means the figure is simply omitted rather than causing a build error. The page renders correctly before you have a photo to put on it.\nThe content file yaml Copy 12345678910 --- title: Hello. description: One sentence for the HTML meta description. layout: about portrait: src: /images/about/portrait.jpg caption: Somewhere alongside the Loire. --- Body text here, written in Markdown as normal. description is kept for \u0026lt;meta name=\u0026quot;description\u0026quot;\u0026gt; but the layout doesn\u0026rsquo;t render it on the page — the design goes straight from h1 to photo to prose. One field, two uses, no duplication.\nWhen to use a dedicated layout This pattern is worth reaching for whenever a page differs structurally from the norm rather than just in content. Good candidates:\nAn About page (no hero, no date, no comments) A Contact page (a form, no article body) An index page that needs a custom header or grid If a page needs a single field suppressed, a conditional in the shared template is probably fine. If it needs multiple sections replaced or a different overall structure, give it its own layout. The distinction is: am I configuring the template, or am I fighting it?\n"},{"url":"/2026/05/adding-a-hover-preview-tooltip-to-leaflet-markers/","title":"Adding a hover-preview tooltip to Leaflet markers","summary":"How to build a floating thumbnail tooltip for Leaflet photo markers — shared DOM element, edge-flip positioning, hover delays, and keyboard accessibility.","date":"2026-05-01","tags":["javascript","leaflet"],"cover":"cobalt","body":"Leaflet\u0026rsquo;s bindTooltip is fine for text labels but limited for richer previews. This is how to build a floating thumbnail tooltip that appears when hovering a photo marker, stays within the map bounds, and works with the keyboard.\nOne element, not many The natural instinct is to create a tooltip element per marker. Don\u0026rsquo;t. With many markers on the map, that\u0026rsquo;s many hidden elements in the DOM, each needing positioning logic run on every hover.\nA better approach: one shared element, repositioned and repopulated on demand.\njavascript Copy 123456789101112 var tip = document.createElement(\u0026#39;div\u0026#39;); tip.className = \u0026#39;velo-preview\u0026#39;; tip.setAttribute(\u0026#39;aria-hidden\u0026#39;, \u0026#39;true\u0026#39;); tip.innerHTML = \u0026#39;\u0026lt;img class=\u0026#34;velo-preview__img\u0026#34; alt=\u0026#34;\u0026#34; /\u0026gt;\u0026#39; \u0026#43; \u0026#39;\u0026lt;div class=\u0026#34;velo-preview__body\u0026#34;\u0026gt;\u0026#39; \u0026#43; \u0026#39;\u0026lt;div class=\u0026#34;velo-preview__caption\u0026#34;\u0026gt;\u0026lt;/div\u0026gt;\u0026#39; \u0026#43; \u0026#39;\u0026lt;/div\u0026gt;\u0026#39;; map.getContainer().appendChild(tip); var tipImg = tip.querySelector(\u0026#39;.velo-preview__img\u0026#39;); var tipCaption = tip.querySelector(\u0026#39;.velo-preview__caption\u0026#39;); Append it to the map container, not the document body, so position coordinates are relative to the map.\nHover delays Firing immediately on mouseenter feels jittery — graze across a cluster of markers and tooltips flash in and out. A short delay smooths this out:\njavascript Copy 12345678910111213141516171819202122 var HOVER_IN_DELAY = 80; // ms before showing var HOVER_OUT_DELAY = 200; // ms before hiding var hoverInTimer = null; var hoverOutTimer = null; var activeUrl = null; function scheduleShow(m, btnEl) { clearTimeout(hoverOutTimer); clearTimeout(hoverInTimer); if (activeUrl !== null) { showFor(m, btnEl); // already showing something — swap immediately } else { hoverInTimer = setTimeout(function () { showFor(m, btnEl); }, HOVER_IN_DELAY); } } function scheduleHide() { clearTimeout(hoverInTimer); clearTimeout(hoverOutTimer); hoverOutTimer = setTimeout(hide, HOVER_OUT_DELAY); } When moving between adjacent markers, activeUrl !== null causes an immediate swap rather than waiting for the in-delay again. The out-delay gives the user a moment to move from the marker to the tooltip without it disappearing.\nEdge-flip positioning Anchoring the tooltip at a fixed offset from the marker breaks near the edges of the map. Measure the tooltip dimensions and flip when it would overflow:\njavascript Copy 123456789101112131415161718192021222324252627282930313233343536373839 function showFor(m, btnEl) { // Populate content tipImg.src = m.thumb; tipImg.alt = m.caption; tipCaption.textContent = m.caption; // Measure marker position relative to map container var containerRect = map.getContainer().getBoundingClientRect(); var btnRect = btnEl.getBoundingClientRect(); var mx = btnRect.left - containerRect.left \u0026#43; btnRect.width / 2; var my = btnRect.top - containerRect.top \u0026#43; btnRect.height / 2; // Measure tooltip height while invisible tip.style.visibility = \u0026#39;hidden\u0026#39;; tip.classList.add(\u0026#39;is-visible\u0026#39;); var th = tip.offsetHeight || 168; tip.classList.remove(\u0026#39;is-visible\u0026#39;); tip.style.visibility = \u0026#39;\u0026#39;; var W = containerRect.width; var H = containerRect.height; var TW = 200; // fixed tooltip width from CSS var tx = mx \u0026#43; 14; var ty = my - th - 12; if (tx \u0026#43; TW \u0026gt; W - 8) { tx = mx - TW - 14; } // flip left if (ty \u0026lt; 8) { ty = my \u0026#43; 14; } // flip below // Clamp within container tx = Math.max(8, Math.min(W - TW - 8, tx)); ty = Math.max(8, Math.min(H - th - 8, ty)); tip.style.left = tx \u0026#43; \u0026#39;px\u0026#39;; tip.style.top = ty \u0026#43; \u0026#39;px\u0026#39;; tip.classList.add(\u0026#39;is-visible\u0026#39;); tip.setAttribute(\u0026#39;aria-hidden\u0026#39;, \u0026#39;false\u0026#39;); activeUrl = m.url; } The key step is measuring the tooltip\u0026rsquo;s height while it\u0026rsquo;s invisible. Apply the is-visible class (which gives it display: block or equivalent), read offsetHeight, then remove it before setting the final position and showing it for real. Without this, the height measurement returns 0 and vertical positioning is wrong.\nButton markers for keyboard access Change the marker inner element from a \u0026lt;div\u0026gt; to a \u0026lt;button\u0026gt;:\njavascript Copy 12345678 icon: L.divIcon({ className: \u0026#39;photo-marker\u0026#39;, html: \u0026#39;\u0026lt;button class=\u0026#34;photo-marker-label\u0026#34; type=\u0026#34;button\u0026#34; \u0026#39; \u0026#43; \u0026#39;aria-label=\u0026#34;Photo \u0026#39; \u0026#43; (i \u0026#43; 1) \u0026#43; \u0026#39;: \u0026#39; \u0026#43; escapeHtml(m.caption) \u0026#43; \u0026#39;\u0026#34;\u0026gt;\u0026#39; \u0026#43; (i \u0026#43; 1) \u0026#43; \u0026#39;\u0026lt;/button\u0026gt;\u0026#39;, iconSize: [22, 22], iconAnchor: [11, 11] }) A \u0026lt;button\u0026gt; is focusable by default, responds to Enter and Space, and exposes a role of button to screen readers. Wire focus/blur to the same show/hide functions as mouseenter/mouseleave and the tooltip works with keyboard navigation for free.\nPrefetch thumbnails Hover-in delay is 80ms, but image loading might take longer on a slow connection, producing a blank flash in the tooltip. Prefetch all thumbnail URLs on map load:\njavascript Copy 123 markers.forEach(function (m) { if (m.thumb) { var img = new Image(); img.src = m.thumb; } }); The browser caches the images. By the time the hover fires and tipImg.src is set, the image is already available — the tooltip appears populated.\nDismiss on pan and zoom The tooltip\u0026rsquo;s position is calculated relative to a static marker position. When the map moves, the marker moves but the tooltip doesn\u0026rsquo;t — it hangs in the wrong place. Dismiss it:\njavascript Copy 12 map.on(\u0026#39;movestart zoomstart\u0026#39;, hide); map.on(\u0026#39;click\u0026#39;, hide); "},{"url":"/2026/05/touch-events-and-focus-on-mobile-the-two-tap-trap/","title":"Touch events and focus on mobile — the two-tap trap","summary":"Why the 'first tap previews, second tap acts' pattern is broken on touch devices, and what to do instead.","date":"2026-05-01","tags":["javascript","mobile"],"cover":"tangerine","body":"The pattern that seems reasonable You have a UI element — a map marker, a card, a thumbnail — where hovering reveals a preview and clicking performs an action. On desktop this works cleanly: mouseenter shows the preview, click performs the action.\nOn touch devices there\u0026rsquo;s no hover, so you adapt: first tap shows the preview, second tap performs the action. The implementation usually looks something like this:\njavascript Copy 1234567891011 var stickyUrl = null; btnEl.addEventListener(\u0026#39;focus\u0026#39;, function () { showPreview(); }); btnEl.addEventListener(\u0026#39;blur\u0026#39;, function () { hidePreview(); stickyUrl = null; }); btnEl.addEventListener(\u0026#39;click\u0026#39;, function () { if (stickyUrl !== null) { openLightbox(stickyUrl); // second tap } else { stickyUrl = m.url; // first tap — show preview, remember URL } }); Reasonable enough. First tap sets stickyUrl and shows the preview. Second tap finds stickyUrl set and opens the lightbox.\nIt doesn\u0026rsquo;t work.\nWhy it breaks On mobile, the browser fires a blur event after every tap. The moment the user lifts their finger, the element loses focus. Your blur handler runs, clears stickyUrl, and resets everything — before the second tap can register.\nThe sequence of events for two taps on mobile is actually:\nFirst tap: focus → click (stickyUrl set ✓) Finger lifts: blur (stickyUrl cleared ✗) Second tap: focus → click (stickyUrl is null, shows preview again) The lightbox never opens. The user taps forever.\nThis is not a bug you can easily reproduce on a desktop browser\u0026rsquo;s mobile emulator — device emulation doesn\u0026rsquo;t faithfully reproduce mobile focus behaviour. You need a real device or browser stack to catch it.\nThe fix The two-tap pattern assumes focus can persist between taps on touch. It can\u0026rsquo;t. The fix is to stop trying.\nThe hover preview is inherently a pointer feature: on touch there is no hover, so the preview adds friction rather than value. Showing a preview on first tap forces the user to tap twice to do what they came to do.\nRemove the two-tap logic entirely. One tap, one action:\njavascript Copy 12345 btnEl.addEventListener(\u0026#39;mouseenter\u0026#39;, function () { scheduleShow(m, btnEl); }); btnEl.addEventListener(\u0026#39;mouseleave\u0026#39;, function () { scheduleHide(); }); btnEl.addEventListener(\u0026#39;focus\u0026#39;, function () { scheduleShow(m, btnEl); }); btnEl.addEventListener(\u0026#39;blur\u0026#39;, function () { scheduleHide(); }); btnEl.addEventListener(\u0026#39;click\u0026#39;, function () { openLightbox(m.url); }); mouseenter and mouseleave handle the hover preview on pointer devices — they never fire on touch. click opens the lightbox on all devices. The preview still works for desktop users; mobile users get a direct tap-to-action.\nIf you want to call hide() before opening the lightbox — to cleanly dismiss any visible preview — do it at the start of the action function:\njavascript Copy 1234 function openLightbox(url) { hide(); // dismiss preview before lightbox opens // … open the lightbox … } The broader rule Don\u0026rsquo;t rely on focus persisting between separate user interactions on touch devices. Desktop users have a cursor that maintains hover/focus state continuously; touch users interact in discrete, stateless taps. Design for the touch model — one tap, one outcome — and layer hover enhancements on top for pointer devices.\nThe test for whether a pattern works on touch: if removing the hover/focus event listeners entirely would break the intended flow, the flow is designed for desktop and needs a touch alternative (or to be simplified).\n"},{"url":"/2026/05/validating-hugo-front-matter-with-nodetest/","title":"Validating Hugo front matter with node:test","summary":"A lightweight, zero-dependency test that walks your Hugo content tree and catches broken image paths before they reach production.","date":"2026-05-01","tags":["hugo","testing","devops"],"cover":"mint","body":"The silent failure problem Hugo doesn\u0026rsquo;t error on a missing image in front matter. If image: /images/articles/2025/foo/hero.jpg refers to a file that doesn\u0026rsquo;t exist, the build succeeds, the template gets nil back from resources.Get, and the page renders without a hero image. No warning. No clue.\nOn a site with dozens of articles and hundreds of image references, a single mistyped path is easy to miss. It might go live, or it might sit there broken until someone notices the blank space in a browser.\nThe fix: a one-file test Node 24 includes a built-in test runner — node:test — that needs no framework, no config, and no additional dependencies. A single file can walk the entire content tree and fail fast on any broken reference.\njavascript Copy 1234567891011121314151617181920212223242526272829303132333435363738394041 // tests/content-images.test.mjs import { test } from \u0026#39;node:test\u0026#39;; import assert from \u0026#39;node:assert/strict\u0026#39;; import { readFile } from \u0026#39;node:fs/promises\u0026#39;; import { existsSync } from \u0026#39;node:fs\u0026#39;; import { glob } from \u0026#39;node:fs/promises\u0026#39;; import { join, resolve } from \u0026#39;node:path\u0026#39;; const ROOT = resolve(import.meta.dirname, \u0026#39;..\u0026#39;); const ASSETS = join(ROOT, \u0026#39;assets\u0026#39;); const CONTENT = join(ROOT, \u0026#39;content\u0026#39;); function extractPaths(yaml) { const paths = []; // image: /images/articles/... const image = yaml.match(/^image:\\s*(.\u0026#43;)$/m); if (image) paths.push(image[1].trim()); // thumbnail: // url: /images/articles/... const thumb = yaml.match(/^\\s\u0026#43;url:\\s*(.\u0026#43;)$/m); if (thumb) paths.push(thumb[1].trim()); return paths; } test(\u0026#39;all front matter image references resolve to existing files in assets/\u0026#39;, async () =\u0026gt; { const files = await Array.fromAsync(glob(\u0026#39;articles/**/*.md\u0026#39;, { cwd: CONTENT })); const broken = []; for (const rel of files) { const src = await readFile(join(CONTENT, rel), \u0026#39;utf8\u0026#39;); const match = src.match(/^---\\n([\\s\\S]*?)\\n---/); if (!match) continue; for (const ref of extractPaths(match[1])) { const abs = join(ASSETS, ref.replace(/^\\//, \u0026#39;\u0026#39;)); if (!existsSync(abs)) broken.push(`${rel}: ${ref}`); } } assert.deepEqual(broken, [], `Broken image references:\\n${broken.join(\u0026#39;\\n\u0026#39;)}`); }); Run it:\nbash Copy 1 node --test tests/content-images.test.mjs Output when everything passes:\ntext Copy 1 ✔ all front matter image references resolve to existing files in assets/ (10ms) Output when something\u0026rsquo;s broken:\ntext Copy 123 ✗ all front matter image references resolve to existing files in assets/ AssertionError: Broken image references: articles/2025/canal-des-deux-mers/2025-09-05_cdm_day_05/index.md: /images/articles/2025/cdm/cdm_day_05/hero.jpg Integrating with the rest of your tests Add it to package.json:\njson Copy 123 \u0026#34;scripts\u0026#34;: { \u0026#34;test:content-images\u0026#34;: \u0026#34;node --test tests/content-images.test.mjs\u0026#34; } If you\u0026rsquo;re using Playwright, exclude it from Playwright\u0026rsquo;s discovery — it\u0026rsquo;s a node:test file, not a Playwright spec, and Playwright will try to run it as one if it matches the filename pattern:\ntypescript Copy 12345 // playwright.config.ts export default defineConfig({ testIgnore: [\u0026#39;**/content-images.test.mjs\u0026#39;], // … }); Run order: this test needs no server and no build, so it fits alongside ESLint and Stylelint in the fast, server-free check stage — run it before the Playwright tests that require a running dev server.\nExtending it The same pattern extends to any front-matter field that references a file. GPX tracks, thumbnail images, og:image overrides — add a regex for each field and a file-existence check. The test stays fast regardless of how many fields you add, because it\u0026rsquo;s just filesystem lookups, not HTTP requests or Hugo builds.\nFor a site with structured YAML front matter, you could replace the regex extraction with a proper YAML parser (js-yaml or yaml), but the regex approach covers the common simple cases without any extra dependency.\n"},{"url":"/2026/04/rules-engines-on-the-jvm-in-2026/","title":"Rules engines on the JVM in 2026","summary":"Drools is no longer the only game in town. A look at Easy Rules, RuleBook, and when you should reach for a rules engine at all.","date":"2026-04-28","tags":["rules","java","architecture"],"cover":"cobalt","body":"Rules engines occupy a strange corner of the Java ecosystem. They solve a real problem — externalising business logic that changes faster than your release cycle — but the dominant choice for years, Drools, has always carried significant weight: a steep learning curve, a KIE workbench nobody asked for, and a community that seems perpetually one Red Hat acquisition away from abandonment.\nIn 2026 the picture is a bit more interesting. Here is what I have been using and thinking about.\nThe contenders Easy Rules is the lightweight option. It is an annotation-driven framework that feels like writing plain Java, not a DSL. You define a rule as a POJO, annotate the condition and action methods, and register it with an engine. Five minutes to productive. The trade-off is expressiveness: it has no conflict resolution beyond priority ordering, no forward-chaining inference, and no fact pattern matching. If you need those things, Easy Rules is not your tool.\nRuleBook takes a fluent, functional approach. Rules are defined as lambdas in a chain. It is readable and testable. Like Easy Rules, it trades power for simplicity.\nDrools still wins on raw capability. RETE algorithm, backward chaining, complex event processing, a full rule language (DRL). If you are genuinely doing expert-system-style inference over a large fact base, nothing else comes close on the JVM. The cost is complexity, and the 8.x stream is navigating a messy transition to the cloud-native KOGITO platform.\nWhen to reach for one The honest answer is: not as often as you think.\nA database query or a feature flag will handle most conditional logic that looks like it needs a rules engine. The pattern that actually benefits is where you have a large, frequently-changing set of business rules that non-developers need to own — underwriting rules, pricing bands, compliance checks. The rules engine earns its keep when the alternative is a release cycle per business change.\nIf your \u0026ldquo;rules\u0026rdquo; are ten conditionals that a developer will touch twice a year, you do not need a framework. Write the conditions, test them, ship them.\nMy current default For most projects I reach for Easy Rules first. The annotation model maps well to how business analysts describe rules, it is straightforward to test, and its limitations become apparent quickly enough that you will know if you need to escalate to Drools before you are too deep.\nDrools gets the call when the problem is genuinely rule-heavy — trading limit validation, insurance underwriting, the sort of thing that arrives as a 40-page specification and changes monthly.\nThe JVM rules engine landscape is not exciting in 2026, but it is functional. Pick the simplest tool that solves the problem.\n"},{"url":"/2026/04/adding-copyright-watermarks-to-images-with-hugos-asset-pipeline/","title":"Adding copyright watermarks to images with Hugo's asset pipeline","summary":"How to stamp a copyright notice onto every image at Hugo build time using images.Text — including the font trap, the shadow technique for readability, and how to keep multiple shortcodes in sync.","date":"2026-04-27","tags":["hugo","devops"],"cover":"cobalt","body":"Hugo\u0026rsquo;s extended image processing pipeline includes an images.Text filter that can stamp text onto images at build time. This post shows how to use it to add a copyright watermark — covering the font requirement, a shadow technique for legibility on varied backgrounds, and a non-obvious consistency requirement when the same image is processed in more than one template.\nPrerequisites: images must be in assets/ Hugo\u0026rsquo;s image processing only works on resources in the assets/ directory. Files in static/ are served as-is and cannot be processed.\nIf your images are in static/images/, you\u0026rsquo;ll need to move them to assets/images/ first. Once there, use resources.Get and resources.Match instead of path string construction, and use .RelPermalink or .Permalink on the resulting resource instead of building URLs manually.\nThe basic pattern For a gallery lightbox image at 1920px:\ngo-html-template Copy 123456789 {{- $base := .Resize \u0026#34;1920x webp\u0026#34; -}} {{- $wm := images.Text \u0026#34;© 2025 Stephen Masters\u0026#34; (dict \u0026#34;color\u0026#34; \u0026#34;#ffffff\u0026#34; \u0026#34;size\u0026#34; 16 \u0026#34;font\u0026#34; $font \u0026#34;x\u0026#34; (sub $base.Width 260) \u0026#34;y\u0026#34; (sub $base.Height 26) ) -}} {{- $full := $base | images.Filter $wm -}} images.Text returns a filter. images.Filter applies it to the image and returns a new image resource. The original is unchanged.\nThe x and y parameters are the pixel coordinates of the top-left corner of the text, measured from the top-left of the image. To position in the bottom-right corner, subtract from the image\u0026rsquo;s .Width and .Height after resizing — you need to resize first to know the dimensions.\nThe font trap: Hugo\u0026rsquo;s default font is ASCII-only Here\u0026rsquo;s the problem that trips almost everyone:\nHugo\u0026rsquo;s default font for images.Text is Go\u0026rsquo;s basicfont.Face7x13 — a small bitmap font that covers printable ASCII (characters 0x20–0x7E). The copyright symbol © is Unicode U+00A9. It is not ASCII. If you use the default font, the © character will not render — you\u0026rsquo;ll get a blank or the character will be silently dropped.\nTo use ©, you must provide a TrueType font via the font parameter:\ngo-html-template Copy 12345678 {{- $font := resources.Get \u0026#34;fonts/watermark.ttf\u0026#34; -}} {{- $wm := images.Text \u0026#34;© 2025 Stephen Masters\u0026#34; (dict \u0026#34;color\u0026#34; \u0026#34;#ffffff\u0026#34; \u0026#34;size\u0026#34; 16 \u0026#34;font\u0026#34; $font \u0026#34;x\u0026#34; (sub $base.Width 260) \u0026#34;y\u0026#34; (sub $base.Height 26) ) -}} A good choice is DejaVu Sans — open-source (Bitstream Vera / SIL licence, freely redistributable), wide Unicode coverage, and a reasonable visual weight for a watermark. Place the .ttf file at assets/fonts/watermark.ttf.\nMaking it legible: the shadow technique A plain white watermark on a white or light background is invisible. A dark watermark on a dark background is equally invisible. Since photos vary widely in tone and colour, any single-colour text will disappear somewhere.\nThe solution is a drop shadow: apply two text filters in sequence — a dark semi-transparent layer offset by one pixel, then the main white text on top.\ngo-html-template Copy 123456789 {{- $font := resources.Get \u0026#34;fonts/watermark.ttf\u0026#34; -}} {{- $year := .Page.Date.Format \u0026#34;2006\u0026#34; -}} {{- $copyright := printf \u0026#34;© %s Stephen Masters\u0026#34; $year -}} {{- $base := .Resize \u0026#34;1920x webp\u0026#34; -}} {{- $wmX := sub $base.Width 260 -}} {{- $wmY := sub $base.Height 26 -}} {{- $shadow := images.Text $copyright (dict \u0026#34;color\u0026#34; \u0026#34;#000000cc\u0026#34; \u0026#34;size\u0026#34; 16 \u0026#34;font\u0026#34; $font \u0026#34;x\u0026#34; (add $wmX 1) \u0026#34;y\u0026#34; (add $wmY 1)) -}} {{- $text := images.Text $copyright (dict \u0026#34;color\u0026#34; \u0026#34;#ffffff\u0026#34; \u0026#34;size\u0026#34; 16 \u0026#34;font\u0026#34; $font \u0026#34;x\u0026#34; $wmX \u0026#34;y\u0026#34; $wmY ) -}} {{- $full := $base | images.Filter $shadow $text -}} images.Filter accepts multiple filters and applies them in order. The dark shadow (80% opacity, 1px down-right) provides contrast against light areas; the full-white text sits on top and reads against dark areas.\nUsing the article date for the year Rather than hardcoding the year, use the article\u0026rsquo;s date front matter field. This means 2024 articles automatically get \u0026ldquo;© 2024\u0026rdquo; and 2025 articles get \u0026ldquo;© 2025\u0026rdquo;:\ngo-html-template Copy 12 {{- $year := .Page.Date.Format \u0026#34;2006\u0026#34; -}} {{- $copyright := printf \u0026#34;© %s Stephen Masters\u0026#34; $year -}} In shortcode context .Page.Date is available directly. In a layout template (e.g. _default/single.html) use $.Date.\nWhich images to watermark Not every processed image needs a watermark. The priority is the full-size images that are actually worth copying:\nImage type Size Watermarked Gallery lightbox 1920px Yes — primary sharing target Inline article images 1200px Yes Article hero 1400px Yes Gallery thumbnails 800px No — too small to be useful Route card thumbnails 640px No — too small The multi-shortcode consistency requirement This is the non-obvious part.\nOn Velostevie, the same gallery images are processed in two places:\ngallery.html shortcode — produces thumbnail + lightbox versions; the lightbox data-src URL points to the processed image gpxmap.html shortcode — produces a full-size version for each GPS-tagged photo; the marker URL in data-photo-markers JSON points to the processed image When a user clicks a map marker, JavaScript matches the marker URL against the gallery\u0026rsquo;s data-src to open the lightbox. This match must succeed.\nHugo\u0026rsquo;s image pipeline caches processed images by their source file plus their processing operations. If gallery.html applies a watermark and gpxmap.html does not (or applies different parameters), they produce different processed images with different URLs — and the click-through silently fails.\nThe fix: both shortcodes must apply identical filter parameters. Same font, same colour, same size, same offsets. Then Hugo produces the same cached image resource in both places, and the URLs match.\ngo-html-template Copy 123456 {{- /* In both gallery.html AND gpxmap.html — identical */ -}} {{- $wmX := sub $base.Width 260 -}} {{- $wmY := sub $base.Height 26 -}} {{- $shadow := images.Text $copyright (dict \u0026#34;color\u0026#34; \u0026#34;#000000cc\u0026#34; \u0026#34;size\u0026#34; 16 \u0026#34;font\u0026#34; $font \u0026#34;x\u0026#34; (add $wmX 1) \u0026#34;y\u0026#34; (add $wmY 1)) -}} {{- $text := images.Text $copyright (dict \u0026#34;color\u0026#34; \u0026#34;#ffffff\u0026#34; \u0026#34;size\u0026#34; 16 \u0026#34;font\u0026#34; $font \u0026#34;x\u0026#34; $wmX \u0026#34;y\u0026#34; $wmY ) -}} {{- $full := $base | images.Filter $shadow $text -}} Since both shortcodes are on the same page, .Page.Date.Format \u0026quot;2006\u0026quot; produces the same year in both. The image path is the same. The operations are identical. Hugo returns the same cached file.\nCache invalidation When you change watermark parameters (colour, size, position), Hugo does not automatically reprocess the cached images. The cache at resources/_gen/images/ stores the result of each unique combination of source image + processing operations. Changing a filter parameter changes the cache key, so in theory a fresh build would produce the new version.\nIn practice, the dev server can serve stale content. To force a clean rebuild:\nbash Copy 12 rm -rf resources/_gen/images/ npm run start This is especially important during watermark tuning — if the text looks wrong and you\u0026rsquo;ve already made template changes, clearing the cache is the first thing to try.\nSizing the x offset The x parameter positions the left edge of the text. To avoid the text wrapping at the image edge, you need to leave enough room for the full text width.\nWith DejaVu Sans at 16px, \u0026ldquo;© 2025 Stephen Masters\u0026rdquo; is approximately 230px wide. Using sub $base.Width 260 places the left edge at 1660px on a 1920px image, leaving 260px of space to the right edge — comfortably more than 230px. If you change the font, size, or name, you may need to adjust this offset.\nThere is no programmatic way to get the rendered text width from images.Text — you can only work empirically. Add 20–30px of buffer beyond your estimate and inspect the result.\nSummary Images must be in assets/ to use Hugo\u0026rsquo;s processing pipeline Hugo\u0026rsquo;s default basicfont is ASCII-only — use a TrueType font for © A drop shadow (two sequential filters) gives readability on any background Use .Page.Date.Format \u0026quot;2006\u0026quot; for an automatically correct copyright year When the same image is processed in multiple templates, all must apply identical filter parameters or processed-image URLs will diverge Clear resources/_gen/images/ when changing filter parameters to avoid serving stale cached images "},{"url":"/2026/04/embedding-gps-photo-markers-at-build-time-with-hugo/","title":"Embedding GPS photo markers at build time with Hugo","summary":"How to replace browser-side EXIF GPS reading with a pre-build Node script that embeds coordinates directly in the HTML — faster maps, no async loading, no browser EXIF parsing.","date":"2026-04-27","tags":["hugo","javascript","devops"],"cover":"cobalt","body":"The problem with reading GPS in the browser An earlier version of the Velostevie map read GPS coordinates from image EXIF metadata in the browser using exifr. The flow was:\nHugo shortcode emits a list of photo URLs as a data-photos attribute JavaScript fetches each image from the server exifr extracts the GPS coordinates from the EXIF data Leaflet markers are placed once all reads complete This works, but it has a significant cost: the browser has to download every image just to read its metadata. On a page with thirty gallery photos that might mean thirty HTTP requests firing before a single marker appears. The map loads blank and fills in gradually as the GPS reads complete.\nThere\u0026rsquo;s also an architectural smell: the browser is doing work that could be done once, at build time. Coordinates don\u0026rsquo;t change. The same GPS data is computed fresh on every page load.\nA better approach: extract GPS before Hugo runs The site already runs a Node script before every build to prepare data. The pattern for moving GPS extraction to build time is:\nPre-build: a Node script reads GPS EXIF from all images and writes a JSON data file Build: the Hugo shortcode reads that JSON and embeds coordinates directly in the HTML Runtime: the browser reads coordinates synchronously from the DOM — no fetches, no async, instant markers Step 1: the pre-build script scripts/extract-gps.mjs walks the image directory and writes data/photo-gps.json:\njavascript Copy 123456789101112131415161718192021222324252627282930313233 import { readdir, stat, writeFile } from \u0026#39;fs/promises\u0026#39;; import { join, relative } from \u0026#39;path\u0026#39;; import exifr from \u0026#39;exifr\u0026#39;; const ASSETS_DIR = new URL(\u0026#39;../assets\u0026#39;, import.meta.url).pathname; const OUT_FILE = new URL(\u0026#39;../data/photo-gps.json\u0026#39;, import.meta.url).pathname; async function walk(dir) { const entries = await readdir(dir, { withFileTypes: true }); const files = []; for (const entry of entries) { const full = join(dir, entry.name); if (entry.isDirectory()) files.push(...await walk(full)); else if (/\\.(jpg|jpeg|png)$/i.test(entry.name)) files.push(full); } return files; } const files = await walk(join(ASSETS_DIR, \u0026#39;images\u0026#39;)); const result = {}; for (const file of files) { try { const gps = await exifr.gps(file); if (gps?.latitude \u0026amp;\u0026amp; gps?.longitude) { const key = relative(ASSETS_DIR, file).replace(/\\\\/g, \u0026#39;/\u0026#39;); result[key] = { lat: gps.latitude, lng: gps.longitude }; } } catch { /* no GPS — skip */ } } await writeFile(OUT_FILE, JSON.stringify(result, null, 2)); console.log(`Wrote ${Object.keys(result).length} GPS entries to data/photo-gps.json`); The keys are paths relative to assets/ with no leading slash — matching how Hugo\u0026rsquo;s resources.Match reports resource names (after stripping the leading / with strings.TrimLeft \u0026quot;/\u0026quot; .Name).\nWire it into the build in package.json:\njson Copy 12345 \u0026#34;scripts\u0026#34;: { \u0026#34;extract-gps\u0026#34;: \u0026#34;node scripts/extract-gps.mjs\u0026#34;, \u0026#34;prestart\u0026#34;: \u0026#34;npm run -s mod:vendor \u0026amp;\u0026amp; npm run -s extract-gps\u0026#34;, \u0026#34;prebuild\u0026#34;: \u0026#34;npm run clean:public \u0026amp;\u0026amp; npm run -s mod:vendor \u0026amp;\u0026amp; npm run -s extract-gps\u0026#34; } prestart and prebuild run automatically before npm run start and npm run build, so data/photo-gps.json is always fresh when Hugo runs. The Cloudflare Pages build command also needs to include the step explicitly:\nbash Copy 1 npm ci \u0026amp;\u0026amp; hugo mod vendor \u0026amp;\u0026amp; node scripts/extract-gps.mjs \u0026amp;\u0026amp; hugo --gc --minify Step 2: the Hugo shortcode layouts/shortcodes/gpxmap.html now reads from site.Data[\u0026quot;photo-gps\u0026quot;] and embeds all the data it needs at build time:\ngo-html-template Copy 123456789101112131415161718192021222324 {{- $dir := .Get \u0026#34;gallery\u0026#34; -}} {{- $photoMarkers := slice -}} {{- if and (not $isSection) $dir -}} {{- $gpsData := index $.Site.Data \u0026#34;photo-gps\u0026#34; -}} {{- $images := resources.Match (printf \u0026#34;%s/*\u0026#34; $dir) -}} {{- range $images -}} {{- $filename := path.Base .Name -}} {{- if not (hasPrefix $filename \u0026#34;.\u0026#34;) -}} {{- $key := strings.TrimLeft \u0026#34;/\u0026#34; .Name -}} {{- $gps := index $gpsData $key -}} {{- if $gps -}} {{- $full := .Resize \u0026#34;1920x webp\u0026#34; -}} {{- $base := strings.TrimSuffix (path.Ext $filename) $filename -}} {{- $caption := replace (strings.Trim (replaceRE \u0026#34;^[0-9]\u0026#43;\u0026#34; \u0026#34;\u0026#34; $base) \u0026#34;_\u0026#34;) \u0026#34;_\u0026#34; \u0026#34; \u0026#34; -}} {{- $marker := dict \u0026#34;url\u0026#34; $full.Permalink \u0026#34;lat\u0026#34; $gps.lat \u0026#34;lng\u0026#34; $gps.lng \u0026#34;caption\u0026#34; $caption -}} {{- $photoMarkers = $photoMarkers | append $marker -}} {{- end -}} {{- end -}} {{- end -}} {{- end -}} {{- with $photoMarkers }} \u0026lt;div class=\u0026#34;gpx-map\u0026#34; data-photo-markers=\u0026#34;{{ jsonify . }}\u0026#34;\u0026gt;\u0026lt;/div\u0026gt; {{- end -}} For each image that has a GPS entry in the data file, we build a {url, lat, lng, caption} object and serialise the whole array to JSON in the data-photo-markers attribute. Hugo does this work once at build time and caches it.\nKey gotcha with resources.Match: .Name on a matched resource is the full path relative to assets/ with a leading / — e.g. /images/articles/2025/foo/gallery/bar.png. path.Base .Name gives you just the filename. strings.TrimLeft \u0026quot;/\u0026quot; .Name gives you the key for the JSON lookup (e.g. images/articles/2025/foo/gallery/bar.png). Use the strings.TrimLeft cutset string argument order — cutset first. strings.TrimLeft .Name \u0026quot;/\u0026quot; is wrong and silently returns an empty string (it treats the entire path as the cutset to strip from /).\nStep 3: JavaScript reads synchronously assets/js/gpxmap.js no longer imports exifr or fires any async GPS reads:\njavascript Copy 12345678910111213141516171819202122232425262728293031323334 function addPhotoMarkers(map, el, onBoundsReady) { var markers = []; try { markers = JSON.parse(el.dataset.photoMarkers || \u0026#39;[]\u0026#39;); } catch (e) {} var photoBounds = L.latLngBounds(); markers.forEach(function (m, i) { var marker = L.marker([m.lat, m.lng], { icon: L.divIcon({ className: \u0026#39;photo-marker\u0026#39;, html: \u0026#39;\u0026lt;span class=\u0026#34;photo-marker-label\u0026#34;\u0026gt;\u0026#39; \u0026#43; (i \u0026#43; 1) \u0026#43; \u0026#39;\u0026lt;/span\u0026gt;\u0026#39;, iconSize: [22, 22], iconAnchor: [11, 11] }) }).addTo(map); photoBounds.extend([m.lat, m.lng]); marker.bindTooltip(m.caption, { direction: \u0026#39;top\u0026#39;, offset: [0, -14] }); marker.on(\u0026#39;click\u0026#39;, function () { var triggers = document.querySelectorAll(\u0026#39;.lb-trigger[data-src]\u0026#39;); for (var j = 0; j \u0026lt; triggers.length; j\u0026#43;\u0026#43;) { try { if (decodeURIComponent(triggers[j].dataset.src) === decodeURIComponent(m.url)) { triggers[j].click(); break; } } catch (e) {} } }); }); if (onBoundsReady) onBoundsReady(photoBounds.isValid() ? photoBounds : L.latLngBounds()); } JSON.parse on a data- attribute is synchronous. All markers are placed in a single synchronous loop. onBoundsReady is called immediately at the end — no async waiting.\nBefore and after Before After GPS data source Read from EXIF in browser Embedded in HTML at build time Browser requests One per gallery image (to read EXIF) Zero Marker appearance Gradual, async Instant, synchronous exifr dependency Required in browser Only in pre-build Node script Build time No change Slightly longer (one EXIF read per image) The trade-off is explicitly in favour of the reader: build time goes up marginally, page load speed improves significantly.\nWhat doesn\u0026rsquo;t get a marker Images without GPS metadata simply don\u0026rsquo;t appear in data/photo-gps.json and are silently skipped. This is correct behaviour for indoor photos (château interiors, restaurants) where the camera didn\u0026rsquo;t record location, and for photos exported without location metadata.\nTo audit which gallery images are missing GPS, scripts/check-gps.sh uses exiftool to check each file directly:\nbash Copy 1234567891011 #!/usr/bin/env bash ASSETS_DIR=\u0026#34;$(cd \u0026#34;$(dirname \u0026#34;$0\u0026#34;)/..\u0026#34; \u0026amp;\u0026amp; pwd)/assets\u0026#34; missing=0 while IFS= read -r -d \u0026#39;\u0026#39; img; do gps=$(exiftool -GPSLatitude \u0026#34;$img\u0026#34; 2\u0026gt;/dev/null) if [[ -z \u0026#34;$gps\u0026#34; ]]; then echo \u0026#34;NO GPS: ${img#\u0026#34;$ASSETS_DIR/\u0026#34;}\u0026#34; ((missing\u0026#43;\u0026#43;)) fi done \u0026lt; \u0026lt;(find \u0026#34;$ASSETS_DIR/images\u0026#34; -path \u0026#34;*/gallery/*\u0026#34; -type f \\( -iname \u0026#34;*.jpg\u0026#34; -o -iname \u0026#34;*.jpeg\u0026#34; -o -iname \u0026#34;*.png\u0026#34; \\) -print0 | sort -z) echo \u0026#34;$missing image(s) missing GPS metadata.\u0026#34; Summary Moving GPS extraction to build time eliminated all browser-side EXIF reads. The map now renders its markers synchronously from data already embedded in the HTML — no waiting, no progressive loading. The pre-build Node script runs automatically before every npm run start and npm run build, so data/photo-gps.json is always up to date.\nThis is a specific application of a general principle: if computation can happen at build time rather than in the browser, do it there. The build runs once; the page loads for every reader.\n"},{"url":"/2026/04/hugo-image-processing-gotchas-what-the-docs-dont-warn-you-about/","title":"Hugo image processing gotchas: what the docs don't warn you about","summary":"A collection of non-obvious traps in Hugo's image processing pipeline: the ASCII-only default font, the strings.TrimLeft argument order, stale image caches, and why two shortcodes processing the same image can produce different URLs.","date":"2026-04-27","tags":["hugo","devops"],"cover":"tangerine","body":"Hugo\u0026rsquo;s image processing pipeline is powerful, but it has some sharp edges that are easy to hit and hard to diagnose because they all fail silently. This is a collection of the ones I\u0026rsquo;ve run into while building Velostevie.\n1. The default font for images.Text is ASCII-only Hugo\u0026rsquo;s images.Text filter uses Go\u0026rsquo;s basicfont.Face7x13 by default — a small bitmap font covering printable ASCII (0x20–0x7E). If you include any non-ASCII character in your text, it will not render. There is no error. The character is silently dropped or produces a blank glyph.\nThe most common casualty: the copyright symbol ©, which is U+00A9.\ngo-html-template Copy 12 {{- /* This produces \u0026#34;2025 Stephen Masters\u0026#34; with a gap where © should be */ -}} {{- $wm := images.Text \u0026#34;© 2025 Stephen Masters\u0026#34; (dict \u0026#34;size\u0026#34; 14) -}} Fix: provide a TrueType font via the font parameter.\ngo-html-template Copy 12 {{- $font := resources.Get \u0026#34;fonts/watermark.ttf\u0026#34; -}} {{- $wm := images.Text \u0026#34;© 2025 Stephen Masters\u0026#34; (dict \u0026#34;size\u0026#34; 14 \u0026#34;font\u0026#34; $font) -}} The font must be a resource in assets/. DejaVu Sans is a good choice for watermarks: open-source, comprehensive Unicode support, freely redistributable.\n2. strings.TrimLeft takes the cutset first This one is a classic Go template trap. Hugo\u0026rsquo;s strings.TrimLeft signature is:\ntext Copy 1 strings.TrimLeft CUTSET STRING The cutset (the set of characters to strip) comes first. The string to operate on comes second.\ngo-html-template Copy 123456 {{- /* Correct — strips leading \u0026#34;/\u0026#34; from .Name */ -}} {{- $key := strings.TrimLeft \u0026#34;/\u0026#34; .Name -}} {{- /* Wrong — treats .Name as the cutset, strips those characters from \u0026#34;/\u0026#34; */ -}} {{- /* Returns \u0026#34;\u0026#34; because every character in \u0026#34;/\u0026#34; is in the cutset. */ -}} {{- $key := strings.TrimLeft .Name \u0026#34;/\u0026#34; -}} The wrong version returns an empty string and produces no error. I hit this when building the GPS data lookup: the key came back empty, every GPS lookup returned nil, and no photo markers appeared. The fix was trivial once found, but finding it took a while.\nThis affects strings.TrimLeft, strings.TrimRight, and strings.Trim — all three take the cutset first.\n3. Changing filter parameters doesn\u0026rsquo;t automatically invalidate the dev server cache Hugo caches processed images in resources/_gen/images/. The cache key is derived from the source image and the processing operations applied. When you change filter parameters (font, size, colour, position), the cache key changes — so a new build will produce a new image.\nHowever, the dev server (hugo server) does not always detect that filter parameters have changed and re-run the template. In practice, if you change your images.Text parameters and the watermark looks wrong (or unchanged), the server may still be serving the old processed file from cache.\nFix: clear the image cache and restart.\nbash Copy 12 rm -rf resources/_gen/images/ npm run start This forces Hugo to reprocess every image from scratch. The first build after clearing will be slow; subsequent builds only reprocess changed files.\n4. Two templates processing the same image can produce different URLs Hugo\u0026rsquo;s image pipeline is deterministic: the same source file + the same operations = the same output file at the same URL. This is how the cache works, and it\u0026rsquo;s usually what you want.\nThe trap: if the same image is processed in two different templates with different operations, you get two different output files at two different URLs — and any code that expects them to match will fail silently.\nOn Velostevie, gallery images are processed in two places:\ngallery.html shortcode: image.Resize \u0026quot;1920x webp\u0026quot; + watermark filter → URL goes into data-src on lightbox trigger buttons gpxmap.html shortcode: image.Resize \u0026quot;1920x webp\u0026quot; + watermark filter → URL goes into data-photo-markers JSON, used by the map to open the lightbox when a marker is clicked The JavaScript match is: decodeURIComponent(marker.url) === decodeURIComponent(trigger.dataset.src). If the two shortcodes produce different URLs for the same image, this comparison silently fails and clicking a map marker does nothing.\nFix: ensure both templates apply identical processing steps in the same order with the same parameters.\ngo-html-template Copy 1234567891011 {{- /* gallery.html */ -}} {{- $base := .Resize \u0026#34;1920x webp\u0026#34; -}} {{- $shadow := images.Text $copyright (dict \u0026#34;color\u0026#34; \u0026#34;#000000cc\u0026#34; \u0026#34;size\u0026#34; 16 \u0026#34;font\u0026#34; $font \u0026#34;x\u0026#34; (add $wmX 1) \u0026#34;y\u0026#34; (add $wmY 1)) -}} {{- $text := images.Text $copyright (dict \u0026#34;color\u0026#34; \u0026#34;#ffffff\u0026#34; \u0026#34;size\u0026#34; 16 \u0026#34;font\u0026#34; $font \u0026#34;x\u0026#34; $wmX \u0026#34;y\u0026#34; $wmY) -}} {{- $full := $base | images.Filter $shadow $text -}} {{- /* gpxmap.html — identical */ -}} {{- $base := .Resize \u0026#34;1920x webp\u0026#34; -}} {{- $shadow := images.Text $copyright (dict \u0026#34;color\u0026#34; \u0026#34;#000000cc\u0026#34; \u0026#34;size\u0026#34; 16 \u0026#34;font\u0026#34; $font \u0026#34;x\u0026#34; (add $wmX 1) \u0026#34;y\u0026#34; (add $wmY 1)) -}} {{- $text := images.Text $copyright (dict \u0026#34;color\u0026#34; \u0026#34;#ffffff\u0026#34; \u0026#34;size\u0026#34; 16 \u0026#34;font\u0026#34; $font \u0026#34;x\u0026#34; $wmX \u0026#34;y\u0026#34; $wmY) -}} {{- $full := $base | images.Filter $shadow $text -}} Since both templates are on the same page, variables like $copyright (derived from .Page.Date) and $wmX/$wmY (derived from $base.Width/$base.Height) will have the same values in both. Hugo returns the same cached image resource and the URLs match.\n5. resources.Match returns full paths with a leading slash When you call resources.Match \u0026quot;images/gallery/*\u0026quot;, the .Name property on each result is the full path relative to assets/, with a leading / — e.g. /images/gallery/foo.png, not foo.png.\nThis matters when you need to use the path as a lookup key in a data file (where the key was written without a leading slash) or when extracting just the filename.\ngo-html-template Copy 123456789 {{- range $images -}} {{- /* .Name is \u0026#34;/images/gallery/foo.png\u0026#34; */ -}} {{- /* Filename only */ -}} {{- $filename := path.Base .Name -}} {{- /* \u0026#34;foo.png\u0026#34; */ -}} {{- /* Key for data lookup (no leading slash) */ -}} {{- $key := strings.TrimLeft \u0026#34;/\u0026#34; .Name -}} {{- /* \u0026#34;images/gallery/foo.png\u0026#34; */ -}} {{- end -}} Remember: strings.TrimLeft \u0026quot;/\u0026quot; .Name — cutset first (see gotcha 2).\nSummary Gotcha Symptom Fix Default font is ASCII-only © and other non-ASCII chars silently absent Provide a TrueType font via font parameter strings.TrimLeft argument order Empty string returned, lookups fail silently Cutset first: strings.TrimLeft \u0026quot;/\u0026quot; .Name Dev server caches stale images Watermark changes don\u0026rsquo;t appear rm -rf resources/_gen/images/ then restart Different operations = different URLs Marker click-through silently fails Keep all templates that process the same image in sync resources.Match returns full paths GPS/data lookups fail, captions wrong Use path.Base .Name for filename, strings.TrimLeft \u0026quot;/\u0026quot; .Name for keys All five of these fail silently. None produce a Hugo build error. The only diagnostic is to add logging or inspect the generated HTML to check what\u0026rsquo;s actually in the processed attributes.\n"},{"url":"/2026/04/building-a-gps-photo-map-with-hugo-leaflet-and-exifr/","title":"Building a GPS photo map with Hugo, Leaflet, and exifr","summary":"How to build an interactive map for a Hugo static site that reads GPS coordinates directly from image EXIF data and plots photo markers alongside a GPX route — with all the gotchas.","date":"2026-04-26","tags":["hugo","javascript","leaflet","devops"],"cover":"cobalt","body":"On my cycling blog Velostevie each trip article includes an interactive map showing the GPX route and numbered markers for every photo taken along the way. Clicking a marker opens the photo in a lightbox. The whole thing is a Hugo static site — no server, no database — so the map has to be built from static files.\nThis post walks through the architecture: a Hugo shortcode that wires up the data, a vanilla JavaScript IIFE that uses Leaflet for the map and exifr to extract GPS coordinates from image EXIF metadata, and the non-obvious gotchas I ran into along the way.\nThe finished map — GPX polyline with numbered photo markers on OpenStreetMap tiles What we\u0026rsquo;re building The end result looks like this:\nA Leaflet map is embedded in each article page. If the article directory contains a .gpx file, the route is drawn as a polyline. If the article has a gallery/ folder, each photo that has GPS metadata embedded gets a numbered circular marker at its location on the map. Clicking a marker opens the photo in the site\u0026rsquo;s lightbox. If there\u0026rsquo;s no GPX file, the map still renders and fits itself to the bounds of the photo markers. The shortcode is called like this in the article\u0026rsquo;s index.md:\nhugo Copy 1 {{\u0026lt; gpxmap gallery=\u0026#34;images/articles/2025/canal-des-deux-mers/2025-09-01_cdm_day_01/gallery\u0026#34; \u0026gt;}} Architecture overview The design is split cleanly across two phases:\nPhase Where What happens Build time Hugo shortcode (gpxmap.html) Finds GPX files and photo paths, encodes them as data-* attributes on a \u0026lt;div\u0026gt; Runtime JavaScript (gpxmap.js) Reads those attributes, initialises Leaflet, fetches GPX, reads EXIF GPS from photos Hugo templates run at build time with no access to the browser. JavaScript runs in the browser with no access to Hugo\u0026rsquo;s template context. The data-* attributes on the map \u0026lt;div\u0026gt; are the handoff point between the two.\nThe Hugo shortcode The full shortcode lives at layouts/shortcodes/gpxmap.html:\ngo-html-template Copy 1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950 {{- $isSection := eq .Page.Kind \u0026#34;section\u0026#34; -}} {{- $gpxFiles := .Page.Resources.Match \u0026#34;*.gpx\u0026#34; -}} {{- /* On section pages, aggregate GPX files from all child articles */ -}} {{- if $isSection -}} {{- range .Page.RegularPages.ByDate -}} {{- range .Resources.Match \u0026#34;*.gpx\u0026#34; -}} {{- $gpxFiles = $gpxFiles | append . -}} {{- end -}} {{- end -}} {{- end -}} {{- /* On single pages, build photo URL list from gallery param */ -}} {{- $dir := .Get \u0026#34;gallery\u0026#34; -}} {{- $photoUrls := slice -}} {{- if and (not $isSection) $dir -}} {{- $files := readDir (printf \u0026#34;static/%s\u0026#34; $dir) -}} {{- range $files -}} {{- if not (hasPrefix .Name \u0026#34;.\u0026#34;) -}} {{- $photoUrls = $photoUrls | append (printf \u0026#34;%s/%s\u0026#34; $dir .Name) -}} {{- end -}} {{- end -}} {{- end -}} {{- if or $gpxFiles $photoUrls -}} {{- $urls := slice -}} {{- range $gpxFiles -}} {{- $urls = $urls | append .Permalink -}} {{- end -}} {{- $absPhotoUrls := slice -}} {{- range $photoUrls -}} {{- $absPhotoUrls = $absPhotoUrls | append (absURL .) -}} {{- end -}} \u0026lt;div class=\u0026#34;gpx-block\u0026#34;\u0026gt; \u0026lt;div class=\u0026#34;gpx-map\u0026#34; {{- with $urls }} data-gpx-files=\u0026#34;{{ delimit . \u0026#34;|\u0026#34; }}\u0026#34;{{- end }} {{- with $absPhotoUrls }} data-photos=\u0026#34;{{ delimit . \u0026#34;|\u0026#34; }}\u0026#34;{{- end }}\u0026gt;\u0026lt;/div\u0026gt; {{- if and (not $isSection) $gpxFiles -}} \u0026lt;div class=\u0026#34;gpx-download\u0026#34;\u0026gt; {{- range $i, $f := $gpxFiles -}} {{- $label := \u0026#34;Download GPX\u0026#34; -}} {{- if gt (len $gpxFiles) 1 -}} {{- $label = printf \u0026#34;Download GPX (%d of %d)\u0026#34; (add $i 1) (len $gpxFiles) -}} {{- end -}} \u0026lt;a href=\u0026#34;{{ $f.Permalink }}\u0026#34; download=\u0026#34;{{ $f.Name }}\u0026#34; class=\u0026#34;gpx-download-link\u0026#34;\u0026gt;↓ {{ $label }}\u0026lt;/a\u0026gt; {{- end -}} \u0026lt;/div\u0026gt; {{- end -}} \u0026lt;/div\u0026gt; {{- end -}} A few things worth noting:\nGPX files are page bundle resources. They live in the same directory as index.md and are accessed via .Page.Resources.Match \u0026quot;*.gpx\u0026quot;. Their .Permalink gives an absolute URL that the browser can fetch().\nPhoto paths are read from the filesystem. readDir lists the contents of static/\u0026lt;gallery\u0026gt;/ at build time. Each path is then converted to an absolute URL using absURL.\nSection pages aggregate GPX from all children. The shortcode can be dropped on a series _index.md to show the whole route across all days.\nThe shortcode renders nothing if there\u0026rsquo;s no data. If there are no GPX files and no gallery, the \u0026lt;div\u0026gt; is not emitted at all.\nGotcha 1: absURL and leading slashes This one cost me most of the debugging time.\nabsURL takes a path and prepends the site\u0026rsquo;s baseURL. The trap is that if you pass a path with a leading /, Hugo treats it as absolute from the domain root and strips the base URL subpath. This matters when the site lives at a subpath (e.g. GitHub Pages at https://username.github.io/repo-name/).\ngo-html-template Copy 1234567 {{- /* Wrong — leading slash strips the subpath */ -}} {{- absURL \u0026#34;/images/articles/foo/bar.png\u0026#34; -}} {{- /* → https://username.github.io/images/articles/foo/bar.png */ -}} {{- /* Correct — path-relative, subpath is preserved */ -}} {{- absURL \u0026#34;images/articles/foo/bar.png\u0026#34; -}} {{- /* → https://username.github.io/repo-name/images/articles/foo/bar.png */ -}} Since the gallery paths come from readDir they don\u0026rsquo;t start with /, so the fix was simply not to prepend one.\nGotcha 2: canonifyURLs doesn\u0026rsquo;t touch data-* attributes Hugo\u0026rsquo;s canonifyURLs = true setting rewrites root-relative URLs in standard HTML attributes (href, src, etc.) to absolute URLs. It does not touch data-* attributes. Any URL passed to JavaScript via a data- attribute must be made absolute in the template itself — as we do above with absURL.\nGotcha 3: Go template URL-encodes attributes whose name contains \u0026quot;url\u0026quot; Go\u0026rsquo;s html/template package has a security rule: any HTML attribute whose name contains the substring url is treated as a URL context and its value is URL-encoded. This will silently mangle a pipe-delimited list of paths.\ngo-html-template Copy 12345 {{- /* Dangerous — Go will URL-encode the value because the name contains \u0026#34;url\u0026#34; */ -}} \u0026lt;div data-photo-urls=\u0026#34;{{ delimit $absPhotoUrls \u0026#34;|\u0026#34; }}\u0026#34;\u0026gt;\u0026lt;/div\u0026gt; {{- /* Safe — no \u0026#34;url\u0026#34; substring in the attribute name */ -}} \u0026lt;div data-photos=\u0026#34;{{ delimit $absPhotoUrls \u0026#34;|\u0026#34; }}\u0026#34;\u0026gt;\u0026lt;/div\u0026gt; The fix is to choose attribute names that don\u0026rsquo;t contain url. I use data-gpx-files and data-photos.\nLoading the scripts The Leaflet CSS, Leaflet JS, exifr, and gpxmap.js should only load on pages that actually use the shortcode — there\u0026rsquo;s no point adding that weight to every page.\nHugo\u0026rsquo;s .HasShortcode method makes this easy. In layouts/_default/baseof.html:\ngo-html-template Copy 12345678910 \u0026lt;head\u0026gt;{{ partial \u0026#34;head.html\u0026#34; . }}\u0026lt;/head\u0026gt; \u0026lt;body\u0026gt; ... {{- if .HasShortcode \u0026#34;gpxmap\u0026#34; }} \u0026lt;script src=\u0026#34;/leaflet/leaflet.js\u0026#34; defer\u0026gt;\u0026lt;/script\u0026gt; \u0026lt;script src=\u0026#34;/exifr/exifr-lite.umd.js\u0026#34; defer\u0026gt;\u0026lt;/script\u0026gt; {{ $gpxMapJs := resources.Get \u0026#34;js/gpxmap.js\u0026#34; | minify | fingerprint }} \u0026lt;script src=\u0026#34;{{ $gpxMapJs.RelPermalink }}\u0026#34; integrity=\u0026#34;{{ $gpxMapJs.Data.Integrity }}\u0026#34; defer\u0026gt;\u0026lt;/script\u0026gt; {{- end }} \u0026lt;/body\u0026gt; And the Leaflet CSS in layouts/partials/head.html:\ngo-html-template Copy 123 {{- if .HasShortcode \u0026#34;gpxmap\u0026#34; }} \u0026lt;link rel=\u0026#34;stylesheet\u0026#34; href=\u0026#34;/leaflet/leaflet.css\u0026#34;\u0026gt; {{- end }} Leaflet and exifr are served locally from static/leaflet/ and static/exifr/ — not from a CDN. This keeps the site self-contained and avoids third-party dependencies.\nImportant: do not add crossorigin=\u0026quot;\u0026quot; to locally-served script tags. For same-origin resources, it triggers a CORS preflight that will fail. The attribute is only needed for cross-origin resources.\nThe JavaScript The full script is an IIFE (Immediately Invoked Function Expression) — no ES modules, no bundler required.\njavascript Copy 123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115 (function () { function parseGPX(xmlText) { var parser = new DOMParser(); var doc = parser.parseFromString(xmlText, \u0026#39;application/xml\u0026#39;); var pts = doc.getElementsByTagName(\u0026#39;trkpt\u0026#39;); return Array.from(pts).map(function (pt) { return [parseFloat(pt.getAttribute(\u0026#39;lat\u0026#39;)), parseFloat(pt.getAttribute(\u0026#39;lon\u0026#39;))]; }); } function addPhotoMarkers(map, el, onBoundsReady) { var raw = el.dataset.photos; if (!raw || typeof exifr === \u0026#39;undefined\u0026#39;) { if (onBoundsReady) onBoundsReady(L.latLngBounds()); return; } var urls = raw.split(\u0026#39;|\u0026#39;).filter(Boolean); var photoBounds = L.latLngBounds(); var remaining = urls.length; function done() { remaining--; if (remaining === 0 \u0026amp;\u0026amp; onBoundsReady) onBoundsReady(photoBounds); } urls.forEach(function (url, i) { exifr.gps(url).then(function (gps) { if (!gps || !gps.latitude || !gps.longitude) { done(); return; } var filename = decodeURIComponent(url.split(\u0026#39;/\u0026#39;).pop()); var caption = filename.replace(/\\.[^.]\u0026#43;$/, \u0026#39;\u0026#39;).replace(/[_-]\u0026#43;/g, \u0026#39; \u0026#39;); var num = i \u0026#43; 1; var marker = L.marker([gps.latitude, gps.longitude], { icon: L.divIcon({ className: \u0026#39;photo-marker\u0026#39;, html: \u0026#39;\u0026lt;span class=\u0026#34;photo-marker-label\u0026#34;\u0026gt;\u0026#39; \u0026#43; num \u0026#43; \u0026#39;\u0026lt;/span\u0026gt;\u0026#39;, iconSize: [22, 22], iconAnchor: [11, 11] }) }).addTo(map); photoBounds.extend([gps.latitude, gps.longitude]); marker.bindTooltip(caption, { direction: \u0026#39;top\u0026#39;, offset: [0, -14] }); marker.on(\u0026#39;click\u0026#39;, function () { var triggers = document.querySelectorAll(\u0026#39;.lb-trigger\u0026#39;); for (var j = 0; j \u0026lt; triggers.length; j\u0026#43;\u0026#43;) { try { if (decodeURIComponent(triggers[j].dataset.src) === decodeURIComponent(url)) { triggers[j].click(); break; } } catch (e) { /* malformed URI — skip */ } } }); done(); }).catch(function () { done(); }); }); } function initMap(el) { var map = L.map(el); L.tileLayer(\u0026#39;https://{s}.tile.openstreetmap.org/{z}/{x}/{y}.png\u0026#39;, { attribution: \u0026#39;\u0026amp;copy; \u0026lt;a href=\u0026#34;https://www.openstreetmap.org/copyright\u0026#34;\u0026gt;OpenStreetMap\u0026lt;/a\u0026gt; contributors\u0026#39;, maxZoom: 19 }).addTo(map); var gpxRaw = el.dataset.gpxFiles; var gpxUrls = gpxRaw ? gpxRaw.split(\u0026#39;|\u0026#39;).filter(Boolean) : []; if (gpxUrls.length === 0) { addPhotoMarkers(map, el, function (photoBounds) { if (photoBounds \u0026amp;\u0026amp; photoBounds.isValid()) { map.fitBounds(photoBounds, { padding: [40, 40] }); } }); return; } var colors = [\u0026#39;#2563eb\u0026#39;, \u0026#39;#dc2626\u0026#39;]; var bounds = L.latLngBounds(); var pending = gpxUrls.length; addPhotoMarkers(map, el, null); gpxUrls.forEach(function (url, i) { fetch(url) .then(function (res) { return res.text(); }) .then(function (text) { var coords = parseGPX(text); if (coords.length) { var poly = L.polyline(coords, { color: colors[i % colors.length], weight: 3, opacity: 0.85 }).addTo(map); bounds.extend(poly.getBounds()); } }) .finally(function () { pending--; if (pending === 0 \u0026amp;\u0026amp; bounds.isValid()) { map.fitBounds(bounds, { padding: [20, 20] }); } }); }); } if (typeof L !== \u0026#39;undefined\u0026#39;) { document.querySelectorAll(\u0026#39;.gpx-map\u0026#39;).forEach(initMap); } })(); GPX parsing: getElementsByTagName, not querySelectorAll GPX files declare a default XML namespace (e.g. xmlns=\u0026quot;http://www.topografix.com/GPX/1/1\u0026quot;). In a namespaced document, querySelectorAll('trkpt') finds nothing because CSS selectors don\u0026rsquo;t match namespaced elements without a namespace prefix. getElementsByTagName('trkpt') ignores the namespace and works correctly.\nURL normalisation when matching photos to lightbox triggers The marker click handler needs to find the matching lightbox trigger element for the photo. Both the data-photos attribute on the map \u0026lt;div\u0026gt; and the data-src attribute on lightbox triggers carry URLs — but one may have literal spaces in filenames and the other may have %20. The comparison silently fails unless both sides are normalised with decodeURIComponent.\nAsync GPS reads and fitBounds exifr.gps(url) is asynchronous. In photo-only mode (no GPX file), fitBounds must not be called until all GPS reads have completed — otherwise you\u0026rsquo;re fitting to an empty or incomplete bounds object. The onBoundsReady callback pattern ensures fitBounds only runs once all the promises have settled.\nGotcha 4: exifr — use the full build, not the lite build The exifr library comes in two builds: a lite build and a full build. The lite build supports JPEG EXIF data but not PNG GPS. If your gallery images are PNGs (as mine are, exported from an iPhone), you must use the full build.\nThe file in this project is named exifr-lite.umd.js but is actually the full build — I replaced the lite build with node_modules/exifr/dist/full.umd.js and kept the original filename. Worth checking if you\u0026rsquo;re copying this pattern.\nEmbedding GPS metadata in photos For photo markers to appear, images need GPS coordinates embedded in their EXIF data. Modern smartphone photos include this automatically if location services are enabled during capture. If you\u0026rsquo;re exporting from a photo management app, make sure the export includes location metadata.\nTo check which images are missing GPS data, I wrote a small shell script:\nbash Copy 12345678910 #!/usr/bin/env bash # scripts/check-gps.sh # Lists gallery images that are missing GPS metadata. find static/images -type f \\( -iname \u0026#34;*.jpg\u0026#34; -o -iname \u0026#34;*.jpeg\u0026#34; -o -iname \u0026#34;*.png\u0026#34; \\) | while read -r f; do lat=$(exiftool -s3 -GPSLatitude \u0026#34;$f\u0026#34; 2\u0026gt;/dev/null) if [ -z \u0026#34;$lat\u0026#34; ]; then echo \u0026#34;NO GPS: $f\u0026#34; fi done Run it from the project root:\nbash Copy 1 ./scripts/check-gps.sh Deployment note: Cloudflare Pages vs GitHub Pages This site originally deployed to GitHub Pages, which serves Hugo sites at a subpath (https://username.github.io/repo-name/). That subpath causes all the URL generation headaches described above.\nI switched to Cloudflare Pages, which serves the site at a clean root domain — https://velostevie.com/ — with no subpath. This eliminates an entire class of URL problems. Cloudflare also deploys automatically on every push to main with no extra workflow configuration needed.\nIf you are deploying a Hugo site with data-* attributes carrying URLs and you have flexibility over your hosting, Cloudflare Pages is the simpler choice.\nSummary Here\u0026rsquo;s the full approach in brief:\nHugo shortcode runs at build time: finds .gpx page bundle resources and reads the gallery directory listing, converts both to absolute URLs, emits them as data-gpx-files and data-photos on a \u0026lt;div\u0026gt;. baseof.html uses .HasShortcode \u0026quot;gpxmap\u0026quot; to conditionally load Leaflet, exifr, and the map script — only on pages that need it. gpxmap.js reads the data attributes, initialises a Leaflet map, fetches and parses each GPX file, then calls exifr.gps() on each photo to get its coordinates and place a marker. Marker clicks trigger the lightbox via decodeURIComponent-normalised URL matching. The trickiest parts were all URL-related: absURL with path-relative inputs, canonifyURLs not touching data-* attributes, Go template URL-encoding attribute names, and URL normalisation in JavaScript. Once those were understood the architecture itself is fairly straightforward.\nThe source is available on GitHub at stephen-masters/velostevie.\nClicking a photo marker opens the photo in the lightbox "},{"url":"/2021/10/moving-to-hugo/","title":"Moving to Hugo","summary":"Switching the blog to Hugo on GitHub Pages after years on various platforms.","date":"2021-10-02","tags":["devops"],"cover":"yellow","body":" Over quite a few years this blog has spent time on a number of sites such as gratiartis.org and scattercode.co.uk. I\u0026rsquo;m now trying to simplify things, so I\u0026rsquo;m switching to Hugo on GitHub Pages.\nThis latest iteration involves a move to stephen-masters.github.io.\n"},{"url":"/2021/03/computing-in-armenia/","title":"Computing in Armenia","summary":"Armenia's pivotal role in developing computers in the Soviet Union — written in preparation for an Armenian Institute event on Women in Science.","date":"2021-03-16","tags":["armenia","history"],"cover":"pink","body":" As a Westerner, all of the history of computing I learned growing up was focused on developments in the UK and USA. Recently however, in preparation for an Armenian Institute event on Armenian Women in Science and Innovation, I have been reading up on the parallel developments within the Soviet Union and Armenia\u0026rsquo;s pivotal role.\nAs a companion to the event, I put together a short article about the history of History of computing in Armenia.\nComputing In Armenia - From Soviet Military Mainframes To Incubators And Startups A few years ago, I visited Bletchley Park, and I went to have a look at one of the bombes - the electro-mechanical devices that were used to decipher the German Enigma machine messages. An old lady in a wheelchair rolled up next to me and I struck up a conversation in which she started telling me about how she used to program it by setting up various configurations of patch cables. All of her fellow bombe operators were women from the Wrens (WRNS - Women\u0026rsquo;s Royal Naval Service).\nAs the war progressed, the Germans developed new ciphers that were harder to decipher than those produced by the original Enigma machine. Colossus the first programmable, electronic, digital computer, was designed to crack them. The operating team for Collossus was made up of 272 Wrens (Women\u0026rsquo;s Royal Navy Service) and only 27 men. In the USA, ENIAC, the first general purpose digital computer was developed to help develop artillery firing tables. The team of 6 who programmed ENIAC were all women.\nNowadays, if you look at the world\u0026rsquo;s five largest tech companies, only 14% of the software engineers are women. Silicon Valley has developed a reputation for a \u0026ldquo;bro culture\u0026rdquo; in the tech industry, and many large tech companies have reputations as highly toxic environments for women.\nSomething seems to have gone wrong.\nEven if we broaden our view beyond software engineers the percentage of women employed by tech companies seems to be around 20% in the USA and UK.\nHowever, as I discovered more recently, things seem to be quite different in Armenia. Armenia\u0026rsquo;s technology sector has been growing rapidly in recent years - by 33% in 2018. Reportedly, around 30% of the people working in this sector are women, and at many newer companies, this percentage is around 50% or more.\nArmenia has a long history in computing, and a much larger role in the history of Soviet computing than many would imagine for such a small country. For example, somewhere between 30% and 40% of Soviet military computers were built in Armenia.\nThis history seems to begin with Andronik Iosifyan. Born in 1905 in the Kalbajar district of Artsakh, he became director of the All-Union Scientific Research Institute of Electromechanics (AUSRIE) in Moscow. Iosifyan specialised in designing electronics, and used his skills to design electrical systems for missiles, nuclear submarines, satellites and spacecraft, such as the first Soviet Meteor weather satellites. The electronics for the Soyuz spacecraft and Mir space station were developed under his leadership.\nVictor Hambartsumyan, known as the founder of theoretical physics in the Soviet Union, was looking for designs for a computer that might be assembled at the Yerevan Scientific Research Institute of Mathematical Machines (YerSRIMM). He travelled to Moscow to meet Iosifyan, in the hope of securing such a design. Sergey Korolev, the lead designer of the first Soviet spaceships and satellites was also part of this meeting. Iosifyan knew Isaak Bruk, who had designed a minicomputer called the M-3 for scientific calculations, and arranged to build three at AUSRIE between 1957-1958. One of these stayed at AUSRIE, one went to Korolev and the other to Sergey Mergelyan at YerSRIMM.\nYerSRIMM had been established in 1956, with the mathematician Mergelyan as its founding director. Receiving the M-3 computer in Yerevan enabled Mergelyan and his team to accelerate their work in computing and they designed a new computer called Aragats between 1958-60, based on the M-3.\nThe Hrazdan/Razdan family of computers were designed at YerSRIMM between 1958 and 1965. This was the first semiconductor computer in the Soviet Union. Manufactured from 1961, the Razdan-2 could perform 5000 operations per second, and the Razdan-3 released in 1966 could perform in the order of 30,000 operations per second. The Razdan computers were large - designed to occupy a 50 square metre room - and were mostly used for military purposes. A Razdan-3 can still be seen in the Computer Science Museum in Szeged, Hungary.\nLater, the Nairi minicomputer, was developed to be used to solve scientific, engineering and economic problems. This was a smaller machine, designed to be operated by a single person, and some were in use in Moscow railway stations. A number of iterations of Nairi were developed, with those in the 1980s being designed to be compatible with DEC PDP-11 computers.\nSadly, the breakup of the Soviet Union seems to have led to a lack of support and funding for research. In 1996, disappointed by the situation, Mergelyan left Armenia to join his son in Sacramento, California. Through the 90s, it seems that much was lost, but by the late 90s and early 2000s, efforts were being made to revive the industry.\nFortunately, in recent years, the technology industry in Armenia has been experiencing a very positive outlook. In 2015, the technology industry was responsible for 5% of GDP, and it was realised that this industry is relatively unaffected by Armenia’s geopolitical situation, being landlocked and with two of its borders closed to trade. New laws were introduced, making it much easier to found, operate, and grow a tech startup in Armenia. In 2014, it was reported that the IT sector was growing at a rate of 20% per year; in 2018, it grew by 33%. Technology incubators have been set up, funded by Silicon Valley venture capital funds with the express aim of supporting Armenian startup businesses, and there are already success stories. The Armenian technology industry seems to have a bright future ahead.\nThe Armenian Institute will be hosting an event on Thursday 18th March 2021, to explore the current situation with a panel of speakers who are all involved in this exciting growth industry in Armenia. Please join us to discover more about what is happening and what the future looks like for innovation in Armenia.\n"},{"url":"/2020/08/digitizing-surmelian/","title":"Digitizing Surmelian","summary":"In 2020, the Armenian Institute republished I Ask You, Ladies and Gentlemen by Leon Surmelian. This is the story of how I got involved in digitising it.","date":"2020-08-06","tags":["book","armenia"],"cover":"lilac","body":" I Ask You, Ladies and Gentlemen, by Leon Surmelian was a bestseller when it was published in 1945, but for some reason it went out of print and was never republished. As it is such a beautiful book, at the Armenian Institute, we recently republished it. In order to do so, I had my first serious experience of digitizing a book via scanning and OCR.\nI wrote a short article about that experience of Digitizing Surmelian.\nIf you would like to buy a copy of the book, there are a few options. Some are better than others, depending on where you live.\nArmenian Institute store Amazon Kindle Abril Books (Los Angeles) National Association for Armenian Studies and Research (NAASR) (Massachusetts) "},{"url":"/2017/03/using-the-kie-workbench-api-to-create-a-project/","title":"Using the KIE Workbench API to create a project","summary":"The KIE Workbench REST API lets you automate project setup in a fresh container — useful when VirtualBox keeps changing your IP address.","date":"2017-03-20","tags":["java","rules"],"cover":"lilac","body":"I was playing around with the KIE Workbench Docker image and came across an issue whereby the container would become unusable if the IP address of the host changed. My sandbox is VirtualBox, running Ubuntu 16.04, so this would happen all the time. I needed some way to be able to blow away an existing container and start up a new one with the project I had been working on.\nThis turned out to be a bit fiddly. For example, I couldn’t clone the Git repository for my project and push it into a new container. The new container didn’t have a repository to push to. Similarly, it wasn’t enough to copy the myproject.git file out of the original image and into the new one. It clearly takes more than that.\nA process that I found, which did work was to start up a new container, go into the Workbench web application and create an organisation, repository and project, with the same names as in the previous container. I could then pull that repository into the project I had already cloned. Subsequent to a little bit of merging, I would then be able to push the repository back to the remote Workbench repository.\nAs you might imagine, creating the organisation, repository and project with the exact same names was a bit tedious and error prone, when done manually. I wasn’t happy. However, I spotted that Workbench provides a REST API for a small set of actions, such as creating an organisation, creating a repository and creating a project. There’s some documentation here:\nhttps://docs.jboss.org/drools/release/6.5.0.Final/drools-docs/html/ch20.html\nSo I dived in and tried it out. Unfortunately things went a pear-shaped rather quickly. It would appear that the API might have changed a little bit without the documentation being updated. But a little dig through the Workbench API code showed me that it wasn’t massively out of sync. It’s actually quite easy. You just need to understand the order that things need to be done and a couple of undocumented properties.\nFirst, create an organisation:\ncurl -X POST -H \u0026quot;Content-Type: application/json\u0026quot; --user admin:admin \\ 127.0.0.1:8080/drools-wb/rest/organizationalunits \\ -d '{ \u0026quot;name\u0026quot;: \u0026quot;com.sctrcd.kiewb\u0026quot;, \\ \u0026quot;description\u0026quot;: \u0026quot;Example Workbench Organisation\u0026quot;, \\ \u0026quot;owner\u0026quot;: \u0026quot;Scattercode\u0026quot;, \\ \u0026quot;defaultGroupId\u0026quot;: \u0026quot;com.sctrcd.kiewb\u0026quot; }' Note that the defaultGroupId is not mentioned in the documentation. Without it, you will find that an organisation seems to be created and can be seen in the Workbench web interface. However, there will be a couple of problems with it. For one, if you try making a GET request for it, the API will not be able to find it. You will receive a 404 response. Similarly, if you try to create a repository associated with the organisation, the creation will fail with a 404, when the API tries to find the organisation. But if you include the defaultGroupId, all will be well.\nSecond, create a repository associated with that organisation:\ncurl -X POST -H \u0026quot;Content-Type: application/json\u0026quot; --user admin:admin \\ 127.0.0.1:8080/drools-wb/rest/repositories \\ -d '{ \u0026quot;name\u0026quot;: \u0026quot;rulesrepo\u0026quot;, \\ \u0026quot;description\u0026quot;: \u0026quot;Example rules repo\u0026quot;, \\ \u0026quot;userName\u0026quot;: null, \u0026quot;password\u0026quot;: null, \u0026quot;gitURL\u0026quot;: null, \\ \u0026quot;requestType\u0026quot;: \u0026quot;new\u0026quot;, \\ \u0026quot;organizationalUnitName\u0026quot;: \u0026quot;com.sctrcd.kiewb\u0026quot; }' Note here, that the documentation implies that you can create a repository without associating it with an organisation. This is something you can do in the Workbench web interface, but through the API, it fails. So you should include the organizationalUnitName in the call to create the repository. There’s not much point in a repository without an organisation, anyway.\nFinally, create a project in the repository:\ncurl -X POST -H \u0026quot;Content-Type: application/json\u0026quot; --user admin:admin \\ 127.0.0.1:8080/drools-wb/rest/repositories/rulesrepo/projects/ \\ -d '{ \u0026quot;name\u0026quot;: \u0026quot;rulesproject\u0026quot;, \\ \u0026quot;description\u0026quot;: \u0026quot;Example rules project\u0026quot; }' Now you’re done. If you have a local Git project, cloned from the original Workbench container, and you have started the new container on the same ports and host, then you can just run a Git pull.\nAt this point, the Git pull will leave merge issues on a handful of files. I’m currently trying to think of my preferred process to correct this. I may just script up taking a copy of the files (5 of them), copying them back after the pull, committing and pushing.\nNow that you know the steps, I should mention that I created a script to perform all three steps. If you want to base a script of your own on it, feel free. Here it is:\nkie-workbench-rest-api-create-org.sh gist · stephen-masters/d7dea1aa7318ad5f20119727daa8afb2 sh Copy 123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183 #!/bin/bash # -------------------------------------------------------------------------------- # # Script to demonstrate using the KIE (Drools) Workbench REST API to: # # create an organistion. # create a repository associated with the organisation. # create a project in the repository. # # Based on the documentation here: # https://docs.jboss.org/drools/release/6.5.0.Final/drools-docs/html/ch20.html # # At time of writing, the official documentation seems to be a little bit behind # the current state of the API. Therefore, if you use the example entities provided # in the documentation, the API calls will not work. # # Some of the values hardcoded below (URL, username, password), are based # on those defined in the Drools Workbench Showcase Docker image: # https://hub.docker.com/r/jboss/drools-workbench-showcase/ # I would recommend turning those into arguments or environment variables if you # intend to make use of this script. I have done that here, just to keep everything # for the example in one place. Please, don\u0026#39;t keep the hardcoded password! # # -------------------------------------------------------------------------------- # -------------------------------------------------------------------------------- # First, we create an organisation # -------------------------------------------------------------------------------- API_RESPONSE=`curl -X POST -H \u0026#34;Content-Type: application/json\u0026#34; --user admin:admin \\ 127.0.0.1:8080/drools-wb/rest/organizationalunits \\ -d \u0026#39;{ \u0026#34;name\u0026#34;: \u0026#34;com.sctrcd.kiewb\u0026#34;, \\ \u0026#34;description\u0026#34;: \u0026#34;Example Workbench Organisation\u0026#34;, \\ \u0026#34;owner\u0026#34;: \u0026#34;Scattercode\u0026#34;, \\ \u0026#34;defaultGroupId\u0026#34;: \u0026#34;com.sctrcd.kiewb\u0026#34; }\u0026#39;` echo \u0026#34;API_RESPONSE: \u0026#34; echo \u0026#34;$API_RESPONSE\u0026#34; echo \u0026#34;\u0026#34; JOB_STATE=`echo $API_RESPONSE | jq -c \u0026#39;. | {status}\u0026#39;` JOB_ID=`echo $API_RESPONSE | jq -c \u0026#39;. | {jobId}\u0026#39;` JOB_ID=${JOB_ID#\u0026#39;{\u0026#34;jobId\u0026#34;:\u0026#34;\u0026#39;} JOB_ID=${JOB_ID%\u0026#39;\u0026#34;}\u0026#39;} echo \u0026#34;JOB_STATE: $JOB_STATE\u0026#34; echo \u0026#34;JOB_ID: $JOB_ID\u0026#34; if [ \u0026#34;$JOB_STATE\u0026#34; != \u0026#39;{\u0026#34;status\u0026#34;:\u0026#34;APPROVED\u0026#34;}\u0026#39; ] then echo \u0026#34;Request rejected. Request state: $JOB_STATE\u0026#34; exit 1 fi # All jobs are async. We need to keep checking the state of the job until it is flagged as SUCCESS or fails. while [[ $JOB_STATE == \u0026#39;{\u0026#34;status\u0026#34;:\u0026#34;APPROVED\u0026#34;}\u0026#39; || $JOB_STATE == \u0026#39;{\u0026#34;status\u0026#34;:\u0026#34;ACCEPTED\u0026#34;}\u0026#39; ]]; do JOB_STATE=`curl 127.0.0.1:8080/drools-wb/rest/jobs/$JOB_ID --user admin:admin | jq -c \u0026#39;. | { status }\u0026#39;` echo \u0026#34;JOB_STATE: $JOB_STATE\u0026#34; sleep 1s done if [ \u0026#34;$JOB_STATE\u0026#34; != \u0026#39;{\u0026#34;status\u0026#34;:\u0026#34;SUCCESS\u0026#34;}\u0026#39; ] then echo \u0026#34;Request accepted, but failed. Job state: $JOB_STATE\u0026#34; if [ \u0026#34;$JOB_STATE\u0026#34; != \u0026#39;{\u0026#34;status\u0026#34;:\u0026#34;BAD_REQUEST\u0026#34;}\u0026#39; ] then # A BAD_REQUEST state indicates that the resource is already there. exit 1 fi fi echo \u0026#34;Request succeeded. Job state: $JOB_STATE\u0026#34; # -------------------------------------------------------------------------------- # Now that we have an organisation, we can create a repository. # -------------------------------------------------------------------------------- API_RESPONSE=`curl -X POST -H \u0026#34;Content-Type: application/json\u0026#34; --user admin:admin \\ 127.0.0.1:8080/drools-wb/rest/repositories \\ -d \u0026#39;{ \u0026#34;name\u0026#34;: \u0026#34;rulesrepo\u0026#34;, \\ \u0026#34;description\u0026#34;: \u0026#34;Example rules repo\u0026#34;, \\ \u0026#34;userName\u0026#34;: null, \u0026#34;password\u0026#34;: null, \u0026#34;gitURL\u0026#34;: null, \\ \u0026#34;requestType\u0026#34;: \u0026#34;new\u0026#34;, \\ \u0026#34;organizationalUnitName\u0026#34;: \u0026#34;com.sctrcd.kiewb\u0026#34; }\u0026#39;` echo \u0026#34;\u0026#34; echo \u0026#34;API_RESPONSE: \u0026#34; echo \u0026#34;$API_RESPONSE\u0026#34; echo \u0026#34;\u0026#34; JOB_STATE=`echo $API_RESPONSE | jq -c \u0026#39;. | {status}\u0026#39;` JOB_STATE=`echo $API_RESPONSE | jq -c \u0026#39;. | {status}\u0026#39;` JOB_ID=`echo $API_RESPONSE | jq -c \u0026#39;. | {jobId}\u0026#39;` JOB_ID=${JOB_ID#\u0026#39;{\u0026#34;jobId\u0026#34;:\u0026#34;\u0026#39;} JOB_ID=${JOB_ID%\u0026#39;\u0026#34;}\u0026#39;} echo \u0026#34;JOB_STATE: $JOB_STATE\u0026#34; echo \u0026#34;JOB_ID: $JOB_ID\u0026#34; if [ \u0026#34;$JOB_STATE\u0026#34; != \u0026#39;{\u0026#34;status\u0026#34;:\u0026#34;APPROVED\u0026#34;}\u0026#39; ] then echo \u0026#34;Request rejected. Request state: $JOB_STATE\u0026#34; exit 1 fi # All jobs are async. We need to keep checking the state of the job until it is flagged as SUCCESS or fails. while [[ $JOB_STATE == \u0026#39;{\u0026#34;status\u0026#34;:\u0026#34;APPROVED\u0026#34;}\u0026#39; || $JOB_STATE == \u0026#39;{\u0026#34;status\u0026#34;:\u0026#34;ACCEPTED\u0026#34;}\u0026#39; ]]; do JOB_STATE=`curl metis:8080/drools-wb/rest/jobs/$JOB_ID --user admin:admin | jq -c \u0026#39;. | { status }\u0026#39;` echo \u0026#34;JOB_STATE: $JOB_STATE\u0026#34; sleep 1s done if [ \u0026#34;$JOB_STATE\u0026#34; != \u0026#39;{\u0026#34;status\u0026#34;:\u0026#34;SUCCESS\u0026#34;}\u0026#39; ] then echo \u0026#34;Request accepted, but failed. Job state: $JOB_STATE\u0026#34; if [ \u0026#34;$JOB_STATE\u0026#34; != \u0026#39;{\u0026#34;status\u0026#34;:\u0026#34;BAD_REQUEST\u0026#34;}\u0026#39; ] then # A BAD_REQUEST state indicates that the resource is already there. exit 1 fi fi echo \u0026#34;Request succeeded. Job state: $JOB_STATE\u0026#34; # -------------------------------------------------------------------------------- # Now that we have a repository, lets create a project in it. # -------------------------------------------------------------------------------- API_RESPONSE=`curl -X POST -H \u0026#34;Content-Type: application/json\u0026#34; --user admin:admin \\ 127.0.0.1:8080/drools-wb/rest/repositories/rulesrepo/projects/ \\ -d \u0026#39;{ \u0026#34;name\u0026#34;: \u0026#34;rulesproject\u0026#34;, \\ \u0026#34;description\u0026#34;: \u0026#34;Example rules project\u0026#34; }\u0026#39;` echo \u0026#34;\u0026#34; echo \u0026#34;API_RESPONSE: \u0026#34; echo \u0026#34;$API_RESPONSE\u0026#34; echo \u0026#34;\u0026#34; JOB_STATE=`echo $API_RESPONSE | jq -c \u0026#39;. | {status}\u0026#39;` JOB_STATE=`echo $API_RESPONSE | jq -c \u0026#39;. | {status}\u0026#39;` JOB_ID=`echo $API_RESPONSE | jq -c \u0026#39;. | {jobId}\u0026#39;` JOB_ID=${JOB_ID#\u0026#39;{\u0026#34;jobId\u0026#34;:\u0026#34;\u0026#39;} JOB_ID=${JOB_ID%\u0026#39;\u0026#34;}\u0026#39;} echo \u0026#34;JOB_STATE: $JOB_STATE\u0026#34; echo \u0026#34;JOB_ID: $JOB_ID\u0026#34; if [ \u0026#34;$JOB_STATE\u0026#34; != \u0026#39;{\u0026#34;status\u0026#34;:\u0026#34;APPROVED\u0026#34;}\u0026#39; ] then echo \u0026#34;Request rejected. Request state: $JOB_STATE\u0026#34; exit 1 fi # All jobs are async. We need to keep checking the state of the job until it is flagged as SUCCESS or fails. while [[ $JOB_STATE == \u0026#39;{\u0026#34;status\u0026#34;:\u0026#34;APPROVED\u0026#34;}\u0026#39; || $JOB_STATE == \u0026#39;{\u0026#34;status\u0026#34;:\u0026#34;ACCEPTED\u0026#34;}\u0026#39; ]]; do JOB_STATE=`curl metis:8080/drools-wb/rest/jobs/$JOB_ID --user admin:admin | jq -c \u0026#39;. | { status }\u0026#39;` echo \u0026#34;JOB_STATE: $JOB_STATE\u0026#34; sleep 1s done if [ \u0026#34;$JOB_STATE\u0026#34; != \u0026#39;{\u0026#34;status\u0026#34;:\u0026#34;SUCCESS\u0026#34;}\u0026#39; ] then echo \u0026#34;Request accepted, but failed. Job state: $JOB_STATE\u0026#34; if [ \u0026#34;$JOB_STATE\u0026#34; != \u0026#39;{\u0026#34;status\u0026#34;:\u0026#34;BAD_REQUEST\u0026#34;}\u0026#39; ] then # A BAD_REQUEST state indicates that the resource is already there. exit 1 fi fi echo \u0026#34;Request succeeded. Job state: $JOB_STATE\u0026#34; "},{"url":"/2016/01/multiple-databases-with-spring-boot-and-spring-data-jpa/","title":"Multiple databases with Spring Boot and Spring Data JPA","summary":"Connecting a Spring Boot application to two separate databases with Spring Data JPA, working around Boot's default autowiring behaviour.","date":"2016-01-05","tags":["java","spring"],"cover":"mint","body":"A little while back I knocked up a post describing how to enable a Spring application to connect to multiple data sources. At the time, I had only just heard about Spring Boot at the SpringOne 2GX conference in Santa Clara, so the examples didn’t take advantage of that and also didn’t work around some of the autowiring that it does.\nRecently, I was working on a little ETL project to migrate data from one database to another with a different structure, so I returned to this problem and the following is the result.\nFirst, if you want to get hold of a working (including some simple tests) example project, here it is:\nhttps://github.com/gratiartis/multids-demo/tree/now-with-spring-boot\nAs previously, when you define an entity manager, you can define where it should scan for entities and repository classes. The classes can be named individually, but it is easiest if you put your domain entities and repository classes into their own packages and point the entity manager factory at the package. In this example, I used:\ncom.sctrcd.multids.foo.domain com.sctrcd.multids.foo.repo com.sctrcd.multids.bar.domain com.sctrcd.multids.bar.repo I suspect that it’s certainly possible to get around it, but I found that due to Spring Boot trying to inject beans based on default names, it was easiest to set up one of the data sources to use the defaults and the other to use bean names that I defined. As you can see in the application.yml below:\napplication.yml gist · stephen-masters/ce9990e6a4d04a53e799 yml Copy 123456789101112131415161718192021222324252627282930 spring: datasource: url: jdbc:mysql://localhost/foo_schema username: root password: d4t4b4s3sForLif3 driverClassName: com.mysql.jdbc.Driver test-on-borrow: true test-while-idle: true validation-query: select 1; maxActive: 1 jpa: show-sql: false generate-ddl: false properties: hibernate: dialect: org.hibernate.dialect.MySQL5InnoDBDialect ddl-auto: validate hbm2ddl: import_files: bar: datasource: url: jdbc:mysql://localhost/bar_schema username: root password: d4t4b4s3sForLif3 driverClassName: com.mysql.jdbc.Driver test-on-borrow: true test-while-idle: true validation-query: select 1; maxActive: 1 … the spring.datasource.url, spring.datasource.username and spring.datasource.password properties are all defined for the ‘default’ datasource. I define some additional non-conventional properties for the additional schema. We will see how those are picked up shortly.\nBeyond the application.yml configuration, all we need to do is define @Configuration beans which will pick up the properties. First, a @Configuration to wire up the ‘default’ data source. This defines each bean as @Primary, to ensure that they are the beans picked up by anything which does not specify a @Qualifier:\nFooDbConfig.java gist · stephen-masters/2c202c741c30cab102e6 java Copy 12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152 package com.sctrcd.multidsdemo; import javax.persistence.EntityManagerFactory; import javax.sql.DataSource; import org.springframework.beans.factory.annotation.Qualifier; import org.springframework.boot.autoconfigure.jdbc.DataSourceBuilder; import org.springframework.boot.context.properties.ConfigurationProperties; import org.springframework.boot.orm.jpa.EntityManagerFactoryBuilder; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; import org.springframework.context.annotation.Primary; import org.springframework.data.jpa.repository.config.EnableJpaRepositories; import org.springframework.orm.jpa.JpaTransactionManager; import org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean; import org.springframework.transaction.PlatformTransactionManager; import org.springframework.transaction.annotation.EnableTransactionManagement; @Configuration @EnableTransactionManagement @EnableJpaRepositories( entityManagerFactoryRef = \u0026#34;entityManagerFactory\u0026#34;, basePackages = { \u0026#34;com.sctrcd.multidsdemo.foo.repo\u0026#34; }) public class FooConfig { @Primary @Bean(name = \u0026#34;dataSource\u0026#34;) @ConfigurationProperties(prefix=\u0026#34;spring.datasource\u0026#34;) public DataSource dataSource() { return DataSourceBuilder.create().build(); } @Primary @Bean(name = \u0026#34;entityManagerFactory\u0026#34;) public LocalContainerEntityManagerFactoryBean entityManagerFactory( EntityManagerFactoryBuilder builder, @Qualifier(\u0026#34;dataSource\u0026#34;) DataSource dataSource) { return builder .dataSource(dataSource) .packages(\u0026#34;com.sctrcd.multidsdemo.foo.domain\u0026#34;) .persistenceUnit(\u0026#34;foo\u0026#34;) .build(); } @Primary @Bean(name = \u0026#34;transactionManager\u0026#34;) public PlatformTransactionManager transactionManager( @Qualifier(\u0026#34;entityManagerFactory\u0026#34;) EntityManagerFactory entityManagerFactory) { return new JpaTransactionManager(entityManagerFactory); } } Second a @Configuration to wire up the additional datasource. It is essentially identical to the ‘default’ configuration, except that it defines non-conventional names for the data source, entity manager factory and transaction manager and scans different packages for the entities and repositories. It also defines the named transaction manager in the @EnableJpaRepositories annotation and does not define the beans as @Primary.\nMultiDsBarDbConfig.java gist · stephen-masters/db8643cdd89714de494b java Copy 12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849 package com.sctrcd.multidsdemo; import javax.persistence.EntityManagerFactory; import javax.sql.DataSource; import org.springframework.beans.factory.annotation.Qualifier; import org.springframework.boot.autoconfigure.jdbc.DataSourceBuilder; import org.springframework.boot.context.properties.ConfigurationProperties; import org.springframework.boot.orm.jpa.EntityManagerFactoryBuilder; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; import org.springframework.data.jpa.repository.config.EnableJpaRepositories; import org.springframework.orm.jpa.JpaTransactionManager; import org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean; import org.springframework.transaction.PlatformTransactionManager; import org.springframework.transaction.annotation.EnableTransactionManagement; @Configuration @EnableTransactionManagement @EnableJpaRepositories( entityManagerFactoryRef = \u0026#34;barEntityManagerFactory\u0026#34;, transactionManagerRef = \u0026#34;barTransactionManager\u0026#34;, basePackages = { \u0026#34;com.sctrcd.multidsdemo.bar.repo\u0026#34; }) public class BarConfig { @Bean(name = \u0026#34;barDataSource\u0026#34;) @ConfigurationProperties(prefix=\u0026#34;bar.datasource\u0026#34;) public DataSource barDataSource() { return DataSourceBuilder.create().build(); } @Bean(name = \u0026#34;barEntityManagerFactory\u0026#34;) public LocalContainerEntityManagerFactoryBean barEntityManagerFactory( EntityManagerFactoryBuilder builder, @Qualifier(\u0026#34;barDataSource\u0026#34;) DataSource barDataSource) { return builder .dataSource(barDataSource) .packages(\u0026#34;com.sctrcd.multidsdemo.bar.domain\u0026#34;) .persistenceUnit(\u0026#34;bar\u0026#34;) .build(); } @Bean(name = \u0026#34;barTransactionManager\u0026#34;) public PlatformTransactionManager barTransactionManager( @Qualifier(\u0026#34;barEntityManagerFactory\u0026#34;) EntityManagerFactory barEntityManagerFactory) { return new JpaTransactionManager(barEntityManagerFactory); } } Beyond those configuration classes, everything is just the standard setup for a Spring Boot / Spring Data JPA application, so if you have an application connecting to a single database already, there isn’t a lot of modification to support connecting to additional databases.\n"},{"url":"/2015/02/a-minimal-spring-boot-drools-web-service/","title":"A minimal Spring Boot Drools web service","summary":"Just the essentials: a Spring Boot application exposing a Drools rules engine as an HTTP API, nothing more.","date":"2015-02-06","tags":["java","spring","rules"],"cover":"cobalt","body":"A little while back, I knocked up Qzr to demonstrate using Spring Boot with the Drools rules engine. However, I also wanted to play around with a few more technologies (AngularJS and Spring HATEOAS), so it’s a bit large for just demonstrating exposing Drools rules as an HTTP web service.\nA few folks found it difficult to pick out the essentials of running Drools in a Spring Boot application, so I thought I’d have a go at creating a simpler application, which does nothing more than that.\nHence, the Bus Pass Web Service\nAs might be guessed from the project name, for the rules, I took my cues from the Drools Bus Pass example in the Drools project. I cut the rules down a little bit and reduced the code by replacing some of the Java fact classes with DRL declared types. I prefer this for facts which are only referenced from within the DRL.\nAssuming that you have a reasonably recent install of Maven and the JDK (I have tested with 8), you should be able to do the following from the command line.\nBuild the application:\nmvn clean package Run the application:\njava -jar target/buspass-ws-1.0.0-SNAPSHOT.jar Then send a request to the API using curl or your favourite web browser. The rules state that if you request a bus pass for a person with age less than 16, you should see a ChildBusPass. For someone 16 or over, you should see an AdultBusPass.\nFor example, opening http://127.0.0.1:8080/buspass?name=Steve\u0026amp;age=15 gives me:\n{\u0026quot;person\u0026quot;:{\u0026quot;name\u0026quot;:\u0026quot;Steve\u0026quot;,\u0026quot;age\u0026quot;:15},\u0026quot;busPassType\u0026quot;:\u0026quot;ChildBusPass\u0026quot;} … and opening http://127.0.0.1:8080/buspass?name=Steve\u0026amp;age=16 gives me:\n{\u0026quot;person\u0026quot;:{\u0026quot;name\u0026quot;:\u0026quot;Steve\u0026quot;,\u0026quot;age\u0026quot;:16},\u0026quot;busPassType\u0026quot;:\u0026quot;AdultBusPass\u0026quot;} The full source code is on GitHub, so that you can browse through it. I don’t intend to change it much now, other than to add a few comments. The following are some of the key features, that you should know about.\nFirst of all, it’s a Maven project, so I hope you’re familiar with that. The following XML is extracted from the pom.xml. Note that to enable Spring Boot, I have imported the Spring platform Bill of Materials and defined spring-boot-starter-web as a dependency. By including the spring-boot-maven-plugin, the Maven build will generate an executable jar, which will run up an embedded Tomcat instance to host the web application. You don’t need to have a web server installed on your machine, to run this application.\nThe Drools functionality is enabled by defining kie-ci as a dependency. This brings in the Drools API, and sets up classpath scanning so that it can find the rules in your application.\nspring-platform-bom.xml gist · stephen-masters/b60c730070304e6c4163 xml Copy 123456789101112131415161718192021222324252627282930313233343536373839 \u0026lt;!-- Transitively bring in the Spring IO Platform Bill-of-Materials `pom.xml` --\u0026gt; \u0026lt;dependencyManagement\u0026gt; \u0026lt;dependencies\u0026gt; \u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;io.spring.platform\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;platform-bom\u0026lt;/artifactId\u0026gt; \u0026lt;version\u0026gt;1.1.1.RELEASE\u0026lt;/version\u0026gt; \u0026lt;type\u0026gt;pom\u0026lt;/type\u0026gt; \u0026lt;scope\u0026gt;import\u0026lt;/scope\u0026gt; \u0026lt;/dependency\u0026gt; \u0026lt;/dependencies\u0026gt; \u0026lt;/dependencyManagement\u0026gt; \u0026lt;dependencies\u0026gt; \u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;org.springframework.boot\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;spring-boot-starter-web\u0026lt;/artifactId\u0026gt; \u0026lt;/dependency\u0026gt; \u0026lt;!-- ... --\u0026gt; \u0026lt;dependency\u0026gt; \u0026lt;groupId\u0026gt;org.kie\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;kie-ci\u0026lt;/artifactId\u0026gt; \u0026lt;version\u0026gt;${kie.version}\u0026lt;/version\u0026gt; \u0026lt;/dependency\u0026gt; \u0026lt;/dependencies\u0026gt; \u0026lt;build\u0026gt; \u0026lt;plugins\u0026gt; \u0026lt;plugin\u0026gt; \u0026lt;groupId\u0026gt;org.springframework.boot\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;spring-boot-maven-plugin\u0026lt;/artifactId\u0026gt; \u0026lt;executions\u0026gt; \u0026lt;execution\u0026gt; \u0026lt;goals\u0026gt; \u0026lt;goal\u0026gt;repackage\u0026lt;/goal\u0026gt; \u0026lt;/goals\u0026gt; \u0026lt;/execution\u0026gt; \u0026lt;/executions\u0026gt; \u0026lt;/plugin\u0026gt; \u0026lt;/plugins\u0026gt; \u0026lt;/build\u0026gt; Having kie-ci in the project means that Drools will scan for rules based on certain conventions. It will look for a file called kmodule.xml in src/main/resources/META-INF/.\nkmodule.xml gist · stephen-masters/781abd092397bf3d3a44 xml Copy 123456789 \u0026lt;?xml version=\u0026#34;1.0\u0026#34; encoding=\u0026#34;UTF-8\u0026#34;?\u0026gt; \u0026lt;kmodule xmlns=\u0026#34;http://jboss.org/kie/6.0.0/kmodule\u0026#34; xmlns:xsi=\u0026#34;http://www.w3.org/2001/XMLSchema-instance\u0026#34;\u0026gt; \u0026lt;kbase name=\u0026#34;BusPassKbase\u0026#34; packages=\u0026#34;com.sctrcd.buspassws.rules\u0026#34;\u0026gt; \u0026lt;ksession name=\u0026#34;BusPassSession\u0026#34; /\u0026gt; \u0026lt;/kbase\u0026gt; \u0026lt;/kmodule\u0026gt; The kmodule.xml defines the package where the rules for your knowledge base can be found. Based on the definition above, it will scan for rules (.drl files and others) in src/main/resources/com/sctrcd/buspassws/rules. I won’t explain the rules. Feel free to go take a look at them yourself. As can be seen in the XML, this also defines a knowledge session called BusPassSession. This means that you can now start a knowledge session like so:\ngistfile1.java gist · stephen-masters/e246df0650984490e7be java Copy 12 KieContainer kieContainer = KieServices.Factory.get().getKieClasspathContainer(); KieSession kieSession = kieContainer.newKieSession(\u0026#34;BusPassSession\u0026#34;); The heart of a Spring Boot application is its main class, which causes your application to be bootstrapped.\nBusPassApp.java gist · stephen-masters/35d5321e6c2373b579b9 java Copy 12345678910111213 @SpringBootApplication public class BusPassApp { public static void main(String[] args) { ApplicationContext ctx = SpringApplication.run(BusPassApp.class, args); } @Bean public KieContainer kieContainer() { return KieServices.Factory.get().getKieClasspathContainer(); } } This is standard Spring Boot stuff, but the addition we have here is to define a bean, which references the Drools KieClasspathContainer. In doing this, we have a reference to the container, which we can inject into our application beans. This is exactly what we do with the BusPassService.\nBusPassService.java gist · stephen-masters/cff5e162df0f465d3a6c java Copy 1234567891011121314151617181920212223242526 @Service public class BusPassService { private final KieContainer kieContainer; @Autowired public BusPassService(KieContainer kieContainer) { log.info(\u0026#34;Initialising a new bus pass session.\u0026#34;); this.kieContainer = kieContainer; } /** * Create a new session, insert a person\u0026#39;s details and fire rules to * determine what kind of bus pass is to be issued. */ public BusPass getBusPass(Person person) { KieSession kieSession = kieContainer.newKieSession(\u0026#34;BusPassSession\u0026#34;); kieSession.insert(person); kieSession.fireAllRules(); BusPass busPass = findBusPass(kieSession); kieSession.dispose(); return busPass; } // ... } As you can see, we are now exposing Drools functionality in our Spring Boot application. A service bean is injected with a reference to the Drools KieContainer. Subsequently, whenever a call is made to the getBusPass method, we instantiate a new KieSession (note the session name, which matches that defined in kmodule.xml), insert details about a person, fire rules, and see what kind of bus pass they should be given.\nFinally, we need a controller.\nBusPassController.java gist · stephen-masters/cf6db31dd6f643da4c19 java Copy 12345678910111213141516171819202122232425 @RestController public class BusPassController { private static Logger log = LoggerFactory.getLogger(BusPassController.class); private final BusPassService busPassService; @Autowired public BusPassController(BusPassService busPassService) { this.busPassService = busPassService; } @RequestMapping(value = \u0026#34;/buspass\u0026#34;, method = RequestMethod.GET, produces = \u0026#34;application/json\u0026#34;) public BusPass getBusPass( @RequestParam(required = true) String name, @RequestParam(required = true) int age) { Person person = new Person(name, age); log.debug(\u0026#34;Bus pass request received for: \u0026#34; \u0026#43; person); BusPass busPass = busPassService.getBusPass(person); return busPass; } } By annotating the controller class as @RestController, Spring will set it up as a bean and ensure that anything returned from a method is marshalled. As the getBusPass method has been defined as producing application/json, Spring will automatically use Jackson to marshal the response to JSON.\nThe @RequestMapping annotation indicates that you can reach the URL at /buspass. For instance, if you run up the application as it is, this means that you can send GET requests to http://127.0.0.1:8080/buspass. The @RequestParam annotations indicate that you need to send querystring arguments, providing values for “name” and “age”.\nAll that remains is to try it out. Please do let me know if you spot anything that you think could be improved.\n"},{"url":"/2013/10/bundling-project-dependencies-with-the-shade-plugin/","title":"Bundling project dependencies with the Shade plugin","summary":"The Maven Shade plugin bundles all dependencies into a single fat jar — handy for deploying libraries into Drools Guvnor or running self-contained FitNesse fixtures.","date":"2013-10-16","tags":["java"],"cover":"yellow","body":"Have you ever struggled with the mass of .jar files that you can find in a directory of Java libraries? You have no idea what version each of them is and what its dependencies are. You want to put your own application jar in there, but you know that will mean needing to get hold of 20 other jar files to deal with its dependencies.\nI certainly have trouble with this. I use the Drools Guvnor web application to manage business rules, but this sometimes requires that I place my own libraries in that web application’s lib directory. Some of these libraries are actually minimal Spring applications which need to do data access and invoke web services. This means that each of them does require multiple additional Jar files. It becomes difficult to keep track of what libraries I have added to that directory and what I need to add.\nHowever I have come across a decent way of dealing with this problem. The shade plugin enables me to build a single .jar file containing all dependencies for an application. This way, I’m able to ensure that all the dependencies I tested against in my build are definitely the ones that have been deployed.\nThe following is a basic example of configuring the shade plugin in your pom.xml:\nmaven-shade-plugin-example.xml gist · stephen-masters/3609269 xml Copy 123456789101112131415161718192021222324252627282930313233 \u0026lt;plugin\u0026gt; \u0026lt;groupId\u0026gt;org.apache.maven.plugins\u0026lt;/groupId\u0026gt; \u0026lt;artifactId\u0026gt;maven-shade-plugin\u0026lt;/artifactId\u0026gt; \u0026lt;version\u0026gt;1.4\u0026lt;/version\u0026gt; \u0026lt;executions\u0026gt; \u0026lt;execution\u0026gt; \u0026lt;phase\u0026gt;package\u0026lt;/phase\u0026gt; \u0026lt;goals\u0026gt; \u0026lt;goal\u0026gt;shade\u0026lt;/goal\u0026gt; \u0026lt;/goals\u0026gt; \u0026lt;configuration\u0026gt; \u0026lt;transformers\u0026gt; \u0026lt;transformer implementation=\u0026#34;org.apache.maven.plugins.shade.resource.AppendingTransformer\u0026#34;\u0026gt; \u0026lt;resource\u0026gt;META-INF/spring.handlers\u0026lt;/resource\u0026gt; \u0026lt;/transformer\u0026gt; \u0026lt;transformer implementation=\u0026#34;org.apache.maven.plugins.shade.resource.AppendingTransformer\u0026#34;\u0026gt; \u0026lt;resource\u0026gt;META-INF/spring.schemas\u0026lt;/resource\u0026gt; \u0026lt;/transformer\u0026gt; \u0026lt;/transformers\u0026gt; \u0026lt;filters\u0026gt; \u0026lt;filter\u0026gt; \u0026lt;artifact\u0026gt;*:*\u0026lt;/artifact\u0026gt; \u0026lt;excludes\u0026gt; \u0026lt;exclude\u0026gt;META-INF/*.SF\u0026lt;/exclude\u0026gt; \u0026lt;exclude\u0026gt;META-INF/*.DSA\u0026lt;/exclude\u0026gt; \u0026lt;exclude\u0026gt;META-INF/*.RSA\u0026lt;/exclude\u0026gt; \u0026lt;/excludes\u0026gt; \u0026lt;/filter\u0026gt; \u0026lt;/filters\u0026gt; \u0026lt;/configuration\u0026gt; \u0026lt;/execution\u0026gt; \u0026lt;/executions\u0026gt; \u0026lt;/plugin\u0026gt; I also find it particularly handy for FitNesse where I’m able to run a quick script to download the latest version of an artifact from a repository, and I know that what I’m getting includes all the dependencies I need. If a dependency version changes, or is added, I don’t need to alter my deployment script.\n"},{"url":"/2013/01/a-web-service-powered-by-spring-and-drools/","title":"A web service powered by Spring and Drools","summary":"A reference project wiring Spring and Drools without the heavyweight KIE integration — hand-cranked and straightforward.","date":"2013-01-23","tags":["java","spring","rules"],"cover":"cobalt","body":"For the past few years I have been designing and building web services which make use of decision management technology such as Drools and FICO Blaze Advisor. The past year or so has all been about using Drools Guvnor to enable business users (legal and operations teams) manage rules, and using the Drools rules engine to evaluate trade requests against those rules.\nMy preference in setting up web services is to use the Spring Framework to configure my application and manage its various components. However, I struggled to find much information online about how best to wire up a Spring web application to make use of Drools for rules evaluation. The Drools documentation does include a chapter on Spring integration, but I found that it didn’t seem to make the integration any simpler, and forced dependencies on older versions of Spring that I didn’t want to use. In the end, I decided to hand-crank the integration in my application, and it turned out to be quite easy to do.\nSo in the hope that it might be useful to someone else, I have knocked up an example project, which configures web services, which are backed by services that each make use of a Drools knowledge base to make decisions. That project can be found at GitHub:\nhttps://github.com/stephen-masters/sctrcd-fx-web\nFeel free to grab a copy and play around with it. It’s built with Maven, and generates a Java web application, which makes use of Spring and Drools to provide a number of web services that could be part of a foreign exchange payments system. I won’t talk about it in depth now though, as it makes use of a variety of technologies. I will soon be posting more, talking about different specific aspects of that project.\n"},{"url":"/2013/01/a-bigdecimal-accumulator-for-drools/","title":"A BigDecimal accumulator for Drools","summary":"Drools's built-in sum accumulator silently converts BigDecimals to doubles. A custom accumulator to fix that before it corrupts your financial calculations.","date":"2013-01-17","tags":["java","rules"],"cover":"cobalt","body":"Working in the financial industry, I have become rather strict about avoiding doubles in Java. The trouble is that they are a floating point representation of a number, which is just an approximation of the real value. This can lead to some unusual results.\nFor instance, according to this, 0.34 + 0.01 is not equal to 0.35.\ndouble x = 0.35; double y = 0.34 + 0.01; System.out.println(x + \u0026quot; : \u0026quot; + y + \u0026quot; : \u0026quot; + (x == y)); 0.35 : 0.35000000000000003 : false Those inaccuracies might seem very small, but it’s surprisingly easy for them to start impacting a real world application. Imagine you wanted to sell dollars and buy Iranian Rial. You would be getting almost 20,000 Rial for every dollar. At that rate, imprecise floating point values could easily impact the final amount being sent. Although with the current US trade sanctions against Iran, that could be the least of your problems.\nIf you are running reports on a history of transactions, then a large number of smaller transactions can add up to large enough values that the imprecise doubles start affecting your totals. Even if you’re not dealing in huge numbers, things can go wrong easily enough. If your process takes a number through a sequence of multiplications or divisions, errors can be magnified. If you have a business rule to always round up, then according to our calculation above, 0.34 + 0.01 = 0.36.\nHowever, that’s enough about why doubles are bad for financial calculations. What caught me by surprise was running accumulate functions in Drools. I was writing code to react to currency exposures being at particular limits, which were all being added up from nice healthy BigDecimal values. However, the numbers I was getting from the ‘sum’ accumulate function were not equal to the numbers my unit tests were expecting. A little investigation showed that the sum accumulator was converting all my nice fixed precision BigDecimal numbers into doubles. Oh dear…\nSo after a little further investigation, I established that there are no BigDecimal accumulators for Drools, and a bug has been open since 2008. The only workaround mentioned was to write your own sumBigDecimal accumulator function.\nThis didn’t seem like great progress, but I thought it seemed like a good opportunity to learn a new corner of Drools, so I knocked together this BigDecimalAccumulator implementation:\nBigDecimalAccumulator.java gist · stephen-masters/4088798 java Copy 123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139 package uk.co.scattercode.drools.accumulators; import java.io.IOException; import java.io.ObjectInput; import java.io.ObjectOutput; import java.io.Serializable; import java.math.BigDecimal; import org.drools.runtime.rule.TypedAccumulateFunction; /** * This is a \u0026#39;sum\u0026#39; accumulator for a collection of {@link BigDecimal}. * Out-of-the-box Drools has sum accumulators which are able to add up * BigDecimals, but do so by converting them to doubles. So the number you get * out at the end is not good if your application is doing anything * financial. * * To use this, create a package builder configuration file: * \u0026lt;pre\u0026gt; * META-INF/drools.packagebuilder.conf * \u0026lt;/pre\u0026gt; * Put the following in it: * \u0026lt;pre\u0026gt; * drools.accumulate.function.sumbd = uk.co.scattercode.drools.accumulators.BigDecimalAccumulator * \u0026lt;/pre\u0026gt; * And you can now do things like: * \u0026lt;pre\u0026gt; * accumulate( * CurrencyExposure( currency in (\u0026#34;GBP\u0026#34;, \u0026#34;EUR\u0026#34;, \u0026#34;NOK\u0026#34;), * $exp : exposure * ), * $sumGbpEurNok : sumbd( $exp ) * ) * \u0026lt;/pre\u0026gt; * * @author Stephen Masters */ public class BigDecimalAccumulator implements TypedAccumulateFunction { /** * Session-specific data required by the accumulator is stored in a * {@link BigDecimalSum} context, which is instantiated by this * method. */ @Override public Serializable createContext() { return new BigDecimalSum(); } /** * Initializes the accumulator with an empty list of {@link BigDecimal}. */ @Override public void init(Serializable context) throws Exception { BigDecimalSum accumulator = (BigDecimalSum) context; accumulator.init(); } /** * Adds the value to the accumulator sum. */ @Override public void accumulate(Serializable context, Object value) { BigDecimalSum accumulator = (BigDecimalSum) context; accumulator.add((BigDecimal) value); } /** * Subtracts the value from the accumulator sum. */ @Override public void reverse(Serializable context, Object value) throws Exception { BigDecimalSum accumulator = (BigDecimalSum) context; accumulator.subtract((BigDecimal) value); } /** * Yes, this accumulator does implement the reverse method.. */ @Override public boolean supportsReverse() { return true; } /** * Returns the current \u0026#39;sum\u0026#39; held in the accumulator. */ @Override public Object getResult(Serializable context) throws Exception { BigDecimalSum accumulator = (BigDecimalSum) context; return accumulator.sum; } /** * Returns the class of the object returned by getResult. */ @Override public Class\u0026lt;BigDecimal\u0026gt; getResultType() { return BigDecimal.class; } /** * Required to support {@link Externalizable} interface so that data can be * shared across sessions. However, we don\u0026#39;t need to do that, so this method * is empty. */ @Override public void readExternal(ObjectInput in) throws IOException, ClassNotFoundException { } /** * Required to support {@link Externalizable} interface so that data can be * shared across sessions. However, we don\u0026#39;t need to do that, so this method * is empty. */ @Override public void writeExternal(ObjectOutput out) throws IOException { } /** * Session-specific data required by the accumulator is stored in a * an instance of this class. */ private static class BigDecimalSum implements Serializable { /** Generated serialVersionUID */ private static final long serialVersionUID = -3852330030144129793L; BigDecimal sum = BigDecimal.ZERO; void init() { this.sum = BigDecimal.ZERO; } void add(BigDecimal augend) { this.sum = sum.add(augend); } void subtract(BigDecimal subtrahend) { this.sum = sum.subtract(subtrahend); } } } I am a bit puzzled that I’m not finding examples of this all over the place, as Drools has seen a lot of uptake in the financial industry, and it seems like an obvious thing that anybody using Drools for financial rules and calculations would need. Maybe there are loads of private repositories out there, each with their own implementations?\nAnyway, in the absence of BigDecimal accumulator functionality in core Drools, feel free to grab this for your own applications.\n"},{"url":"/2013/01/getting-the-latest-snapshot-from-sonatype-nexus/","title":"Getting the latest snapshot from Sonatype Nexus","summary":"A Ruby script to parse the Nexus REST API and reliably fetch the latest snapshot artifact, sidestepping the timestamped filename problem.","date":"2013-01-15","tags":["java","devops"],"cover":"tangerine","body":"Sonatype Nexus is a repository for build artifacts, which is particularly handy if you have a Maven project. Once you have your Maven project configured, every time you run mvn deploy Maven will do a bit of building and then upload the resulting artifacts (.jar, .war, …) to the repository. If you browse Nexus you will then be able to find those artifacts with a unique name and download them.\nThis is all great, but if your project is running on a snapshot version, then every time you deploy to Nexus, the artifact file name will be appended with date and an ever-incrementing number. For my purposes, I wanted to be able to go on to a Linux test server where I have Apache Tomcat installed and grab the latest .war file. Maybe I need to relax more, but I was getting a bit irritated with having to manually find the snapshot in Nexus, copy the link and then fire off a curl -O -L http://\u0026hellip; command every time I updated the project.\nFortunately it turns out that Nexus provides a REST API for searching. Unfortunately, it only returns the name of an artifact without the time-stamp. I think that this is intended to be a ‘good thing’ with Nexus automatically resolving the latest snapshot based on requesting the snapshot with no time-stamp. Unfortunately, requesting that artifact from the location indicated by the API results in a ‘not found’ response.\nTherefore I knocked up a little Ruby script, which will go to the URI at which a full artifact list can be found, which includes resources such as poms, jars, sha1 and md5 hashes. It then parses the response XML to narrow down the results and selects the most recent artifact that matches the search criteria. Finally it will download the artifact.\nThe latest version of the script can be found as a gist on GitHub:\nget_latest_snapshot.rb gist · stephen-masters/1852106 rb Copy 123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219 #--------------------------------------------------------------------- # # Parse the response from Sonatype Nexus in order to determine the # correct URI for the most recent snapshot of an artifact. # # Usage: # ruby get_latest_snapshot.rb \\ # -n http://localhost:8080/nexus \\ # -g uk.co.scattercode \\ # -a my-artifact \\ # -v 1.0.0 \\ # -c jar-with-dependencies \\ # -p jar # # Raison d\u0026#39;etre: Sonatype Nexus provides an API for searching for artifacts. # Maven snapshot builds are generated with a time-stamp on them. The API # returns the name of the artifact without the time-stamp. I think that this # is intended to be a \u0026#39;good thing\u0026#39; with Nexus automatically resolving the # latest snapshot based on requesting the snapshot with no time-stamp. # Unfortunately, requesting that artifact from the location indicated by the # API results in a \u0026#39;not found\u0026#39; response. Therefore this script is intended # to go to the URI at which the full artifact list can be found, which # includes resources such as poms, jars, sha1 and md5 hashes. It narrows # down the results and selects the most recent artifact that matches the # search criteria. # # Find the latest version of this script here: # https://gist.github.com/1852106 # #--------------------------------------------------------------------- require \u0026#39;getoptlong\u0026#39; require \u0026#39;net/http\u0026#39; require \u0026#39;rexml/document\u0026#39; require \u0026#39;open-uri\u0026#39; # # Let folks know what args they could use. # def show_help puts \u0026lt;\u0026lt;-EOF Usage: ruby get_latest_snapshot.rb [OPTION] ... -h, --help: show help --file [file], -f [file]: File to get API response from instead of URL. --nexus [host], -n [host]: The base URL of the Nexus server. --artifact, -a: The name of the artifact. --version, -v: The version of the artifact (1.0.0, 1.0.0-SNAPSHOT, ...). --classifier, -c: The classifier, which gets appended to the name. As defined by \u0026#39;descriptorRef\u0026#39; in Maven assembly plugin. --package, -p: The package type (jar, war, ear, ...). EOF end # # Download the artifact. # def download(uri, filename) puts \u0026#34;Downloading \\n from uri: #{uri} \\n to file: #{filename}\u0026#34; open(filename, \u0026#39;wb\u0026#39;) do |fo| fo.print open(uri).read end puts \u0026#34;I think I just downloaded: #{filename}\u0026#34; end # # Determine the appropriate filename. # def filename(artifact, version, classifier, package) if /-SNAPSHOT/.match(version) vnum = /.\u0026#43;(?=-SNAPSHOT)/.match(version).to_s else vnum = version end if classifier == nil filename = \u0026#34;#{artifact}-#{vnum}.#{package}\u0026#34; else filename = \u0026#34;#{artifact}-#{vnum}-#{classifier}.#{package}\u0026#34; end return filename end opts = GetoptLong.new( [\u0026#39;--help\u0026#39;, \u0026#39;-h\u0026#39;, GetoptLong::NO_ARGUMENT], [\u0026#39;--file\u0026#39;, \u0026#39;-f\u0026#39;, GetoptLong::OPTIONAL_ARGUMENT], [\u0026#39;--nexus\u0026#39;, \u0026#39;-n\u0026#39;, GetoptLong::OPTIONAL_ARGUMENT], [\u0026#39;--group\u0026#39;, \u0026#39;-g\u0026#39;, GetoptLong::REQUIRED_ARGUMENT], [\u0026#39;--artifact\u0026#39;, \u0026#39;-a\u0026#39;, GetoptLong::REQUIRED_ARGUMENT], [\u0026#39;--version\u0026#39;, \u0026#39;-v\u0026#39;, GetoptLong::REQUIRED_ARGUMENT], [\u0026#39;--classifier\u0026#39;, \u0026#39;-c\u0026#39;, GetoptLong::OPTIONAL_ARGUMENT], [\u0026#39;--package\u0026#39;, \u0026#39;-p\u0026#39;, GetoptLong::REQUIRED_ARGUMENT] ) file = nil nexus = nil group = nil artifact = nil version = nil classifier = nil package = nil opts.each do |opt, arg| case opt when \u0026#39;--help\u0026#39; show_help when \u0026#39;--file\u0026#39; file = arg when \u0026#39;--nexus\u0026#39; nexus = arg when \u0026#39;--group\u0026#39; group = arg when \u0026#39;--artifact\u0026#39; artifact = arg when \u0026#39;--version\u0026#39; version = arg when \u0026#39;--classifier\u0026#39; classifier = arg when \u0026#39;--package\u0026#39; package = arg end end puts \u0026lt;\u0026lt;-EOF Args as follows: file = #{file} nexus = #{nexus} group = #{group} artifact = #{artifact} version = #{version} classifier = #{classifier} package = #{package} EOF # # Now we get to the meat of the script. # # I\u0026#39;m going to ignore the search API and just query the \u0026#39;directory\u0026#39; # in which Nexus should be holding the artifacts. if file != nil # We have been given a file with the directory contents XML. # Most likely for test purposes... puts \u0026#34;Getting directory XML from file: #{file}\u0026#34; xml = File.open(file) else url=\u0026#34;#{nexus}/service/local/repositories/snapshots/content/#{group}/#{artifact}/#{version}/\u0026#34; puts \u0026#34;Gettting directory XML from URL: #{url}\u0026#34; xml = Net::HTTP.get_response( URI.parse( url ) ).body end doc = REXML::Document.new(xml) most_recent_uri = nil most_recent_snapshot_id = nil doc.elements.each(\u0026#34;//resourceURI\u0026#34;) {|r| uri = r.text # puts \u0026#34;Looking at uri: #{uri}\u0026#34; # Filter out hashes and irrelevant artifacts. if classifier == nil match = /\\d\u0026#43;.#{package}$/ else match = /#{classifier}.#{package}$/ end if uri =~ match if classifier == nil seq = /(?\u0026lt;=-)\\d\u0026#43;(?=.#{package}$)/.match(uri).to_s.to_i else seq = /(?\u0026lt;=-)\\d\u0026#43;(?=-#{classifier}.#{package}$)/.match(uri).to_s.to_i end puts \u0026#34;Found matching uri with sequence ID: #{seq}\u0026#34; if most_recent_uri == nil || seq \u0026gt; most_recent_snapshot_id most_recent_uri = uri most_recent_snapshot_id = seq end end } if most_recent_uri == nil puts \u0026#34;Unable to find an artifact matching those criteria.\u0026#34; else puts \u0026#34;The most recent snapshot of that artifact is here: \\n #{most_recent_uri}\u0026#34; end download(most_recent_uri, filename(artifact, version, classifier, package)) nexus_response.xml gist · stephen-masters/1852106 xml Copy 123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377 \u0026lt;!-- The response from Nexus will look something like this. --\u0026gt; \u0026lt;content\u0026gt; \u0026lt;data\u0026gt; \u0026lt;content-item\u0026gt; \u0026lt;resourceURI\u0026gt; http://127.0.0.1:8080/nexus/service/local/repositories/snapshots/content/uk/co/scattercode/myartifact/1.0.0-SNAPSHOT/myartifact-1.0.0-20120216.162545-1.pom.md5 \u0026lt;/resourceURI\u0026gt; \u0026lt;relativePath\u0026gt; /uk/co/scattercode/myartifact/1.0.0-SNAPSHOT/myartifact-1.0.0-20120216.162545-1.pom.md5 \u0026lt;/relativePath\u0026gt; \u0026lt;text\u0026gt; myartifact-1.0.0-20120216.162545-1.pom.md5 \u0026lt;/text\u0026gt; \u0026lt;leaf\u0026gt;true\u0026lt;/leaf\u0026gt; \u0026lt;lastModified\u0026gt;2012-02-16 16:25:45.0 GMT\u0026lt;/lastModified\u0026gt; \u0026lt;sizeOnDisk\u0026gt;32\u0026lt;/sizeOnDisk\u0026gt; \u0026lt;/content-item\u0026gt; \u0026lt;content-item\u0026gt; \u0026lt;resourceURI\u0026gt; http://127.0.0.1:8080/nexus/service/local/repositories/snapshots/content/uk/co/scattercode/myartifact/1.0.0-SNAPSHOT/myartifact-1.0.0-20120216.163318-2.jar.md5 \u0026lt;/resourceURI\u0026gt; \u0026lt;relativePath\u0026gt; /uk/co/scattercode/myartifact/1.0.0-SNAPSHOT/myartifact-1.0.0-20120216.163318-2.jar.md5 \u0026lt;/relativePath\u0026gt; \u0026lt;text\u0026gt; myartifact-1.0.0-20120216.163318-2.jar.md5 \u0026lt;/text\u0026gt; \u0026lt;leaf\u0026gt;true\u0026lt;/leaf\u0026gt; \u0026lt;lastModified\u0026gt;2012-02-16 16:33:18.0 GMT\u0026lt;/lastModified\u0026gt; \u0026lt;sizeOnDisk\u0026gt;32\u0026lt;/sizeOnDisk\u0026gt; \u0026lt;/content-item\u0026gt; \u0026lt;content-item\u0026gt; \u0026lt;resourceURI\u0026gt; http://127.0.0.1:8080/nexus/service/local/repositories/snapshots/content/uk/co/scattercode/myartifact/1.0.0-SNAPSHOT/myartifact-1.0.0-20120216.163318-2.pom.md5 \u0026lt;/resourceURI\u0026gt; \u0026lt;relativePath\u0026gt; /uk/co/scattercode/myartifact/1.0.0-SNAPSHOT/myartifact-1.0.0-20120216.163318-2.pom.md5 \u0026lt;/relativePath\u0026gt; \u0026lt;text\u0026gt; myartifact-1.0.0-20120216.163318-2.pom.md5 \u0026lt;/text\u0026gt; \u0026lt;leaf\u0026gt;true\u0026lt;/leaf\u0026gt; \u0026lt;lastModified\u0026gt;2012-02-16 16:33:18.0 GMT\u0026lt;/lastModified\u0026gt; \u0026lt;sizeOnDisk\u0026gt;32\u0026lt;/sizeOnDisk\u0026gt; \u0026lt;/content-item\u0026gt; \u0026lt;content-item\u0026gt; \u0026lt;resourceURI\u0026gt; http://127.0.0.1:8080/nexus/service/local/repositories/snapshots/content/uk/co/scattercode/myartifact/1.0.0-SNAPSHOT/myartifact-1.0.0-20120216.162545-1.pom \u0026lt;/resourceURI\u0026gt; \u0026lt;relativePath\u0026gt; /uk/co/scattercode/myartifact/1.0.0-SNAPSHOT/myartifact-1.0.0-20120216.162545-1.pom \u0026lt;/relativePath\u0026gt; \u0026lt;text\u0026gt; myartifact-1.0.0-20120216.162545-1.pom \u0026lt;/text\u0026gt; \u0026lt;leaf\u0026gt;true\u0026lt;/leaf\u0026gt; \u0026lt;lastModified\u0026gt;2012-02-16 16:25:45.0 GMT\u0026lt;/lastModified\u0026gt; \u0026lt;sizeOnDisk\u0026gt;3872\u0026lt;/sizeOnDisk\u0026gt; \u0026lt;/content-item\u0026gt; \u0026lt;content-item\u0026gt; \u0026lt;resourceURI\u0026gt; http://127.0.0.1:8080/nexus/service/local/repositories/snapshots/content/uk/co/scattercode/myartifact/1.0.0-SNAPSHOT/myartifact-1.0.0-20120216.163318-2-jar-with-dependencies.jar \u0026lt;/resourceURI\u0026gt; \u0026lt;relativePath\u0026gt; /uk/co/scattercode/myartifact/1.0.0-SNAPSHOT/myartifact-1.0.0-20120216.163318-2-jar-with-dependencies.jar \u0026lt;/relativePath\u0026gt; \u0026lt;text\u0026gt; myartifact-1.0.0-20120216.163318-2-jar-with-dependencies.jar \u0026lt;/text\u0026gt; \u0026lt;leaf\u0026gt;true\u0026lt;/leaf\u0026gt; \u0026lt;lastModified\u0026gt;2012-02-16 16:33:20.0 GMT\u0026lt;/lastModified\u0026gt; \u0026lt;sizeOnDisk\u0026gt;26403601\u0026lt;/sizeOnDisk\u0026gt; \u0026lt;/content-item\u0026gt; \u0026lt;content-item\u0026gt; \u0026lt;resourceURI\u0026gt; http://127.0.0.1:8080/nexus/service/local/repositories/snapshots/content/uk/co/scattercode/myartifact/1.0.0-SNAPSHOT/myartifact-1.0.0-20120216.163318-2-jar-with-dependencies.jar.sha1 \u0026lt;/resourceURI\u0026gt; \u0026lt;relativePath\u0026gt; /uk/co/scattercode/myartifact/1.0.0-SNAPSHOT/myartifact-1.0.0-20120216.163318-2-jar-with-dependencies.jar.sha1 \u0026lt;/relativePath\u0026gt; \u0026lt;text\u0026gt; myartifact-1.0.0-20120216.163318-2-jar-with-dependencies.jar.sha1 \u0026lt;/text\u0026gt; \u0026lt;leaf\u0026gt;true\u0026lt;/leaf\u0026gt; \u0026lt;lastModified\u0026gt;2012-02-16 16:33:23.0 GMT\u0026lt;/lastModified\u0026gt; \u0026lt;sizeOnDisk\u0026gt;40\u0026lt;/sizeOnDisk\u0026gt; \u0026lt;/content-item\u0026gt; \u0026lt;content-item\u0026gt; \u0026lt;resourceURI\u0026gt; http://127.0.0.1:8080/nexus/service/local/repositories/snapshots/content/uk/co/scattercode/myartifact/1.0.0-SNAPSHOT/maven-metadata.xml.md5 \u0026lt;/resourceURI\u0026gt; \u0026lt;relativePath\u0026gt; /uk/co/scattercode/myartifact/1.0.0-SNAPSHOT/maven-metadata.xml.md5 \u0026lt;/relativePath\u0026gt; \u0026lt;text\u0026gt;maven-metadata.xml.md5\u0026lt;/text\u0026gt; \u0026lt;leaf\u0026gt;true\u0026lt;/leaf\u0026gt; \u0026lt;lastModified\u0026gt;2012-02-16 16:53:46.0 GMT\u0026lt;/lastModified\u0026gt; \u0026lt;sizeOnDisk\u0026gt;32\u0026lt;/sizeOnDisk\u0026gt; \u0026lt;/content-item\u0026gt; \u0026lt;content-item\u0026gt; \u0026lt;resourceURI\u0026gt; http://127.0.0.1:8080/nexus/service/local/repositories/snapshots/content/uk/co/scattercode/myartifact/1.0.0-SNAPSHOT/myartifact-1.0.0-20120216.162545-1.jar \u0026lt;/resourceURI\u0026gt; \u0026lt;relativePath\u0026gt; /uk/co/scattercode/myartifact/1.0.0-SNAPSHOT/myartifact-1.0.0-20120216.162545-1.jar \u0026lt;/relativePath\u0026gt; \u0026lt;text\u0026gt; myartifact-1.0.0-20120216.162545-1.jar \u0026lt;/text\u0026gt; \u0026lt;leaf\u0026gt;true\u0026lt;/leaf\u0026gt; \u0026lt;lastModified\u0026gt;2012-02-16 16:25:45.0 GMT\u0026lt;/lastModified\u0026gt; \u0026lt;sizeOnDisk\u0026gt;15900\u0026lt;/sizeOnDisk\u0026gt; \u0026lt;/content-item\u0026gt; \u0026lt;content-item\u0026gt; \u0026lt;resourceURI\u0026gt; http://127.0.0.1:8080/nexus/service/local/repositories/snapshots/content/uk/co/scattercode/myartifact/1.0.0-SNAPSHOT/maven-metadata.xml \u0026lt;/resourceURI\u0026gt; \u0026lt;relativePath\u0026gt; /uk/co/scattercode/myartifact/1.0.0-SNAPSHOT/maven-metadata.xml \u0026lt;/relativePath\u0026gt; \u0026lt;text\u0026gt;maven-metadata.xml\u0026lt;/text\u0026gt; \u0026lt;leaf\u0026gt;true\u0026lt;/leaf\u0026gt; \u0026lt;lastModified\u0026gt;2012-02-16 16:53:46.0 GMT\u0026lt;/lastModified\u0026gt; \u0026lt;sizeOnDisk\u0026gt;1216\u0026lt;/sizeOnDisk\u0026gt; \u0026lt;/content-item\u0026gt; \u0026lt;content-item\u0026gt; \u0026lt;resourceURI\u0026gt; http://127.0.0.1:8080/nexus/service/local/repositories/snapshots/content/uk/co/scattercode/myartifact/1.0.0-SNAPSHOT/maven-metadata.xml.sha1 \u0026lt;/resourceURI\u0026gt; \u0026lt;relativePath\u0026gt; /uk/co/scattercode/myartifact/1.0.0-SNAPSHOT/maven-metadata.xml.sha1 \u0026lt;/relativePath\u0026gt; \u0026lt;text\u0026gt;maven-metadata.xml.sha1\u0026lt;/text\u0026gt; \u0026lt;leaf\u0026gt;true\u0026lt;/leaf\u0026gt; \u0026lt;lastModified\u0026gt;2012-02-16 16:53:46.0 GMT\u0026lt;/lastModified\u0026gt; \u0026lt;sizeOnDisk\u0026gt;40\u0026lt;/sizeOnDisk\u0026gt; \u0026lt;/content-item\u0026gt; \u0026lt;content-item\u0026gt; \u0026lt;resourceURI\u0026gt; http://127.0.0.1:8080/nexus/service/local/repositories/snapshots/content/uk/co/scattercode/myartifact/1.0.0-SNAPSHOT/myartifact-1.0.0-20120216.163318-2.pom.sha1 \u0026lt;/resourceURI\u0026gt; \u0026lt;relativePath\u0026gt; /uk/co/scattercode/myartifact/1.0.0-SNAPSHOT/myartifact-1.0.0-20120216.163318-2.pom.sha1 \u0026lt;/relativePath\u0026gt; \u0026lt;text\u0026gt; myartifact-1.0.0-20120216.163318-2.pom.sha1 \u0026lt;/text\u0026gt; \u0026lt;leaf\u0026gt;true\u0026lt;/leaf\u0026gt; \u0026lt;lastModified\u0026gt;2012-02-16 16:33:18.0 GMT\u0026lt;/lastModified\u0026gt; \u0026lt;sizeOnDisk\u0026gt;40\u0026lt;/sizeOnDisk\u0026gt; \u0026lt;/content-item\u0026gt; \u0026lt;content-item\u0026gt; \u0026lt;resourceURI\u0026gt; http://127.0.0.1:8080/nexus/service/local/repositories/snapshots/content/uk/co/scattercode/myartifact/1.0.0-SNAPSHOT/myartifact-1.0.0-20120216.163318-2-jar-with-dependencies.jar.md5 \u0026lt;/resourceURI\u0026gt; \u0026lt;relativePath\u0026gt; /uk/co/scattercode/myartifact/1.0.0-SNAPSHOT/myartifact-1.0.0-20120216.163318-2-jar-with-dependencies.jar.md5 \u0026lt;/relativePath\u0026gt; \u0026lt;text\u0026gt; myartifact-1.0.0-20120216.163318-2-jar-with-dependencies.jar.md5 \u0026lt;/text\u0026gt; \u0026lt;leaf\u0026gt;true\u0026lt;/leaf\u0026gt; \u0026lt;lastModified\u0026gt;2012-02-16 16:33:23.0 GMT\u0026lt;/lastModified\u0026gt; \u0026lt;sizeOnDisk\u0026gt;32\u0026lt;/sizeOnDisk\u0026gt; \u0026lt;/content-item\u0026gt; \u0026lt;content-item\u0026gt; \u0026lt;resourceURI\u0026gt; http://127.0.0.1:8080/nexus/service/local/repositories/snapshots/content/uk/co/scattercode/myartifact/1.0.0-SNAPSHOT/myartifact-1.0.0-20120216.163318-2-tests.jar.sha1 \u0026lt;/resourceURI\u0026gt; \u0026lt;relativePath\u0026gt; /uk/co/scattercode/myartifact/1.0.0-SNAPSHOT/myartifact-1.0.0-20120216.163318-2-tests.jar.sha1 \u0026lt;/relativePath\u0026gt; \u0026lt;text\u0026gt; myartifact-1.0.0-20120216.163318-2-tests.jar.sha1 \u0026lt;/text\u0026gt; \u0026lt;leaf\u0026gt;true\u0026lt;/leaf\u0026gt; \u0026lt;lastModified\u0026gt;2012-02-16 16:33:19.0 GMT\u0026lt;/lastModified\u0026gt; \u0026lt;sizeOnDisk\u0026gt;40\u0026lt;/sizeOnDisk\u0026gt; \u0026lt;/content-item\u0026gt; \u0026lt;content-item\u0026gt; \u0026lt;resourceURI\u0026gt; http://127.0.0.1:8080/nexus/service/local/repositories/snapshots/content/uk/co/scattercode/myartifact/1.0.0-SNAPSHOT/myartifact-1.0.0-20120216.163318-2.jar.sha1 \u0026lt;/resourceURI\u0026gt; \u0026lt;relativePath\u0026gt; /uk/co/scattercode/myartifact/1.0.0-SNAPSHOT/myartifact-1.0.0-20120216.163318-2.jar.sha1 \u0026lt;/relativePath\u0026gt; \u0026lt;text\u0026gt; myartifact-1.0.0-20120216.163318-2.jar.sha1 \u0026lt;/text\u0026gt; \u0026lt;leaf\u0026gt;true\u0026lt;/leaf\u0026gt; \u0026lt;lastModified\u0026gt;2012-02-16 16:33:18.0 GMT\u0026lt;/lastModified\u0026gt; \u0026lt;sizeOnDisk\u0026gt;40\u0026lt;/sizeOnDisk\u0026gt; \u0026lt;/content-item\u0026gt; \u0026lt;content-item\u0026gt; \u0026lt;resourceURI\u0026gt; http://127.0.0.1:8080/nexus/service/local/repositories/snapshots/content/uk/co/scattercode/myartifact/1.0.0-SNAPSHOT/myartifact-1.0.0-20120216.163318-2.pom \u0026lt;/resourceURI\u0026gt; \u0026lt;relativePath\u0026gt; /uk/co/scattercode/myartifact/1.0.0-SNAPSHOT/myartifact-1.0.0-20120216.163318-2.pom \u0026lt;/relativePath\u0026gt; \u0026lt;text\u0026gt; myartifact-1.0.0-20120216.163318-2.pom \u0026lt;/text\u0026gt; \u0026lt;leaf\u0026gt;true\u0026lt;/leaf\u0026gt; \u0026lt;lastModified\u0026gt;2012-02-16 16:33:18.0 GMT\u0026lt;/lastModified\u0026gt; \u0026lt;sizeOnDisk\u0026gt;3872\u0026lt;/sizeOnDisk\u0026gt; \u0026lt;/content-item\u0026gt; \u0026lt;content-item\u0026gt; \u0026lt;resourceURI\u0026gt; http://127.0.0.1:8080/nexus/service/local/repositories/snapshots/content/uk/co/scattercode/myartifact/1.0.0-SNAPSHOT/myartifact-1.0.0-20120216.162545-1-jar-with-dependencies.jar.sha1 \u0026lt;/resourceURI\u0026gt; \u0026lt;relativePath\u0026gt; /uk/co/scattercode/myartifact/1.0.0-SNAPSHOT/myartifact-1.0.0-20120216.162545-1-jar-with-dependencies.jar.sha1 \u0026lt;/relativePath\u0026gt; \u0026lt;text\u0026gt; myartifact-1.0.0-20120216.162545-1-jar-with-dependencies.jar.sha1 \u0026lt;/text\u0026gt; \u0026lt;leaf\u0026gt;true\u0026lt;/leaf\u0026gt; \u0026lt;lastModified\u0026gt;2012-02-16 16:25:48.0 GMT\u0026lt;/lastModified\u0026gt; \u0026lt;sizeOnDisk\u0026gt;40\u0026lt;/sizeOnDisk\u0026gt; \u0026lt;/content-item\u0026gt; \u0026lt;content-item\u0026gt; \u0026lt;resourceURI\u0026gt; http://127.0.0.1:8080/nexus/service/local/repositories/snapshots/content/uk/co/scattercode/myartifact/1.0.0-SNAPSHOT/myartifact-1.0.0-20120216.162545-1-tests.jar \u0026lt;/resourceURI\u0026gt; \u0026lt;relativePath\u0026gt; /uk/co/scattercode/myartifact/1.0.0-SNAPSHOT/myartifact-1.0.0-20120216.162545-1-tests.jar \u0026lt;/relativePath\u0026gt; \u0026lt;text\u0026gt; myartifact-1.0.0-20120216.162545-1-tests.jar \u0026lt;/text\u0026gt; \u0026lt;leaf\u0026gt;true\u0026lt;/leaf\u0026gt; \u0026lt;lastModified\u0026gt;2012-02-16 16:25:45.0 GMT\u0026lt;/lastModified\u0026gt; \u0026lt;sizeOnDisk\u0026gt;3588\u0026lt;/sizeOnDisk\u0026gt; \u0026lt;/content-item\u0026gt; \u0026lt;content-item\u0026gt; \u0026lt;resourceURI\u0026gt; http://127.0.0.1:8080/nexus/service/local/repositories/snapshots/content/uk/co/scattercode/myartifact/1.0.0-SNAPSHOT/myartifact-1.0.0-20120216.162545-1.jar.md5 \u0026lt;/resourceURI\u0026gt; \u0026lt;relativePath\u0026gt; /uk/co/scattercode/myartifact/1.0.0-SNAPSHOT/myartifact-1.0.0-20120216.162545-1.jar.md5 \u0026lt;/relativePath\u0026gt; \u0026lt;text\u0026gt; myartifact-1.0.0-20120216.162545-1.jar.md5 \u0026lt;/text\u0026gt; \u0026lt;leaf\u0026gt;true\u0026lt;/leaf\u0026gt; \u0026lt;lastModified\u0026gt;2012-02-16 16:25:45.0 GMT\u0026lt;/lastModified\u0026gt; \u0026lt;sizeOnDisk\u0026gt;32\u0026lt;/sizeOnDisk\u0026gt; \u0026lt;/content-item\u0026gt; \u0026lt;content-item\u0026gt; \u0026lt;resourceURI\u0026gt; http://127.0.0.1:8080/nexus/service/local/repositories/snapshots/content/uk/co/scattercode/myartifact/1.0.0-SNAPSHOT/myartifact-1.0.0-20120216.162545-1-tests.jar.md5 \u0026lt;/resourceURI\u0026gt; \u0026lt;relativePath\u0026gt; /uk/co/scattercode/myartifact/1.0.0-SNAPSHOT/myartifact-1.0.0-20120216.162545-1-tests.jar.md5 \u0026lt;/relativePath\u0026gt; \u0026lt;text\u0026gt; myartifact-1.0.0-20120216.162545-1-tests.jar.md5 \u0026lt;/text\u0026gt; \u0026lt;leaf\u0026gt;true\u0026lt;/leaf\u0026gt; \u0026lt;lastModified\u0026gt;2012-02-16 16:25:45.0 GMT\u0026lt;/lastModified\u0026gt; \u0026lt;sizeOnDisk\u0026gt;32\u0026lt;/sizeOnDisk\u0026gt; \u0026lt;/content-item\u0026gt; \u0026lt;content-item\u0026gt; \u0026lt;resourceURI\u0026gt; http://127.0.0.1:8080/nexus/service/local/repositories/snapshots/content/uk/co/scattercode/myartifact/1.0.0-SNAPSHOT/myartifact-1.0.0-20120216.162545-1-jar-with-dependencies.jar.md5 \u0026lt;/resourceURI\u0026gt; \u0026lt;relativePath\u0026gt; /uk/co/scattercode/myartifact/1.0.0-SNAPSHOT/myartifact-1.0.0-20120216.162545-1-jar-with-dependencies.jar.md5 \u0026lt;/relativePath\u0026gt; \u0026lt;text\u0026gt; myartifact-1.0.0-20120216.162545-1-jar-with-dependencies.jar.md5 \u0026lt;/text\u0026gt; \u0026lt;leaf\u0026gt;true\u0026lt;/leaf\u0026gt; \u0026lt;lastModified\u0026gt;2012-02-16 16:25:49.0 GMT\u0026lt;/lastModified\u0026gt; \u0026lt;sizeOnDisk\u0026gt;32\u0026lt;/sizeOnDisk\u0026gt; \u0026lt;/content-item\u0026gt; \u0026lt;content-item\u0026gt; \u0026lt;resourceURI\u0026gt; http://127.0.0.1:8080/nexus/service/local/repositories/snapshots/content/uk/co/scattercode/myartifact/1.0.0-SNAPSHOT/myartifact-1.0.0-20120216.163318-2-tests.jar \u0026lt;/resourceURI\u0026gt; \u0026lt;relativePath\u0026gt; /uk/co/scattercode/myartifact/1.0.0-SNAPSHOT/myartifact-1.0.0-20120216.163318-2-tests.jar \u0026lt;/relativePath\u0026gt; \u0026lt;text\u0026gt; myartifact-1.0.0-20120216.163318-2-tests.jar \u0026lt;/text\u0026gt; \u0026lt;leaf\u0026gt;true\u0026lt;/leaf\u0026gt; \u0026lt;lastModified\u0026gt;2012-02-16 16:33:19.0 GMT\u0026lt;/lastModified\u0026gt; \u0026lt;sizeOnDisk\u0026gt;3588\u0026lt;/sizeOnDisk\u0026gt; \u0026lt;/content-item\u0026gt; \u0026lt;content-item\u0026gt; \u0026lt;resourceURI\u0026gt; http://127.0.0.1:8080/nexus/service/local/repositories/snapshots/content/uk/co/scattercode/myartifact/1.0.0-SNAPSHOT/myartifact-1.0.0-20120216.162545-1-jar-with-dependencies.jar \u0026lt;/resourceURI\u0026gt; \u0026lt;relativePath\u0026gt; /uk/co/scattercode/myartifact/1.0.0-SNAPSHOT/myartifact-1.0.0-20120216.162545-1-jar-with-dependencies.jar \u0026lt;/relativePath\u0026gt; \u0026lt;text\u0026gt; myartifact-1.0.0-20120216.162545-1-jar-with-dependencies.jar \u0026lt;/text\u0026gt; \u0026lt;leaf\u0026gt;true\u0026lt;/leaf\u0026gt; \u0026lt;lastModified\u0026gt;2012-02-16 16:25:46.0 GMT\u0026lt;/lastModified\u0026gt; \u0026lt;sizeOnDisk\u0026gt;26403555\u0026lt;/sizeOnDisk\u0026gt; \u0026lt;/content-item\u0026gt; \u0026lt;content-item\u0026gt; \u0026lt;resourceURI\u0026gt; http://127.0.0.1:8080/nexus/service/local/repositories/snapshots/content/uk/co/scattercode/myartifact/1.0.0-SNAPSHOT/myartifact-1.0.0-20120216.163318-2.jar \u0026lt;/resourceURI\u0026gt; \u0026lt;relativePath\u0026gt; /uk/co/scattercode/myartifact/1.0.0-SNAPSHOT/myartifact-1.0.0-20120216.163318-2.jar \u0026lt;/relativePath\u0026gt; \u0026lt;text\u0026gt; myartifact-1.0.0-20120216.163318-2.jar \u0026lt;/text\u0026gt; \u0026lt;leaf\u0026gt;true\u0026lt;/leaf\u0026gt; \u0026lt;lastModified\u0026gt;2012-02-16 16:33:18.0 GMT\u0026lt;/lastModified\u0026gt; \u0026lt;sizeOnDisk\u0026gt;15946\u0026lt;/sizeOnDisk\u0026gt; \u0026lt;/content-item\u0026gt; \u0026lt;content-item\u0026gt; \u0026lt;resourceURI\u0026gt; http://127.0.0.1:8080/nexus/service/local/repositories/snapshots/content/uk/co/scattercode/myartifact/1.0.0-SNAPSHOT/myartifact-1.0.0-20120216.162545-1.jar.sha1 \u0026lt;/resourceURI\u0026gt; \u0026lt;relativePath\u0026gt; /uk/co/scattercode/myartifact/1.0.0-SNAPSHOT/myartifact-1.0.0-20120216.162545-1.jar.sha1 \u0026lt;/relativePath\u0026gt; \u0026lt;text\u0026gt; myartifact-1.0.0-20120216.162545-1.jar.sha1 \u0026lt;/text\u0026gt; \u0026lt;leaf\u0026gt;true\u0026lt;/leaf\u0026gt; \u0026lt;lastModified\u0026gt;2012-02-16 16:25:45.0 GMT\u0026lt;/lastModified\u0026gt; \u0026lt;sizeOnDisk\u0026gt;40\u0026lt;/sizeOnDisk\u0026gt; \u0026lt;/content-item\u0026gt; \u0026lt;content-item\u0026gt; \u0026lt;resourceURI\u0026gt; http://127.0.0.1:8080/nexus/service/local/repositories/snapshots/content/uk/co/scattercode/myartifact/1.0.0-SNAPSHOT/myartifact-1.0.0-20120216.162545-1.pom.sha1 \u0026lt;/resourceURI\u0026gt; \u0026lt;relativePath\u0026gt; /uk/co/scattercode/myartifact/1.0.0-SNAPSHOT/myartifact-1.0.0-20120216.162545-1.pom.sha1 \u0026lt;/relativePath\u0026gt; \u0026lt;text\u0026gt; myartifact-1.0.0-20120216.162545-1.pom.sha1 \u0026lt;/text\u0026gt; \u0026lt;leaf\u0026gt;true\u0026lt;/leaf\u0026gt; \u0026lt;lastModified\u0026gt;2012-02-16 16:25:45.0 GMT\u0026lt;/lastModified\u0026gt; \u0026lt;sizeOnDisk\u0026gt;40\u0026lt;/sizeOnDisk\u0026gt; \u0026lt;/content-item\u0026gt; \u0026lt;content-item\u0026gt; \u0026lt;resourceURI\u0026gt; http://127.0.0.1:8080/nexus/service/local/repositories/snapshots/content/uk/co/scattercode/myartifact/1.0.0-SNAPSHOT/myartifact-1.0.0-20120216.163318-2-tests.jar.md5 \u0026lt;/resourceURI\u0026gt; \u0026lt;relativePath\u0026gt; /uk/co/scattercode/myartifact/1.0.0-SNAPSHOT/myartifact-1.0.0-20120216.163318-2-tests.jar.md5 \u0026lt;/relativePath\u0026gt; \u0026lt;text\u0026gt; myartifact-1.0.0-20120216.163318-2-tests.jar.md5 \u0026lt;/text\u0026gt; \u0026lt;leaf\u0026gt;true\u0026lt;/leaf\u0026gt; \u0026lt;lastModified\u0026gt;2012-02-16 16:33:19.0 GMT\u0026lt;/lastModified\u0026gt; \u0026lt;sizeOnDisk\u0026gt;32\u0026lt;/sizeOnDisk\u0026gt; \u0026lt;/content-item\u0026gt; \u0026lt;content-item\u0026gt; \u0026lt;resourceURI\u0026gt; http://127.0.0.1:8080/nexus/service/local/repositories/snapshots/content/uk/co/scattercode/myartifact/1.0.0-SNAPSHOT/myartifact-1.0.0-20120216.162545-1-tests.jar.sha1 \u0026lt;/resourceURI\u0026gt; \u0026lt;relativePath\u0026gt; /uk/co/scattercode/myartifact/1.0.0-SNAPSHOT/myartifact-1.0.0-20120216.162545-1-tests.jar.sha1 \u0026lt;/relativePath\u0026gt; \u0026lt;text\u0026gt; myartifact-1.0.0-20120216.162545-1-tests.jar.sha1 \u0026lt;/text\u0026gt; \u0026lt;leaf\u0026gt;true\u0026lt;/leaf\u0026gt; \u0026lt;lastModified\u0026gt;2012-02-16 16:25:45.0 GMT\u0026lt;/lastModified\u0026gt; \u0026lt;sizeOnDisk\u0026gt;40\u0026lt;/sizeOnDisk\u0026gt; \u0026lt;/content-item\u0026gt; \u0026lt;/data\u0026gt; \u0026lt;/content\u0026gt; "},{"url":"/2013/01/multiple-databases-with-spring-data-repositories/","title":"Multiple databases with Spring Data repositories","summary":"Configuring a Spring application with two separate data sources using Spring Data JPA — separate @Configuration classes and EntityManagerFactory beans for each schema.","date":"2013-01-15","tags":["java","spring"],"cover":"mint","body":"The Spring Data project keeps making it easier to do database access in Spring applications, and one of the neatest improvements of recent times is that by defining an interface which extends JpaRepository and referencing a JPA entity, an implementation will automatically be injected with all the usual CRUD methods: findAll(), findOne(id), save(entity), delete(id), etc.\nRecently I was working on a project, where I had taken full advantage of this, and for which I needed to add domain objects from an additional database. Unfortunately, as soon as I added references to entities in a different database I started experiencing troubles. For instance:\nNot an managed type: class com.sctrcd.multidsdemo.domain.bar.Bar … which was being caused by my repository being injected with the entityManager and transactionManager for the other database. Here I walk through how I resolved the problems and got things working.\nAfter naming my beans to ensure that I would not be referencing the wrong one, I started seeing:\nNo bean named 'entityManagerFactory' is defined. … because the repository implementation defaults to a by-name search for a bean called “entityManagerFactory”.\nI was struggling to find any documentation of how to do it right, so to help me work through the steps of such a configuration, I created a minimal demo project at GitHub, containing two entities with those traditional names: Foo and Bar.\nhttps://github.com/gratiartis/multids-demo\nHere I shall explain the configuration that I ended up with in the hope that readers might understand that it can be done, and that it’s actually quite easy and requires very little code, if you know how!\nFirst of all, we set up two JPA entities, Foo and Bar:\n@Entity public class Foo { /* Constructors, fields and accessors/mutators */ } @Entity public class Bar { /* Constructors, fields and accessors/mutators */ } Associated with these we create two repositories: FooRepository and BarRepository. Thanks to the awesomeness of Spring Data, we can get ourselves some pretty full-featured repositories purely by defining interfaces which extend JpaRepository:\npublic interface FooRepository extends JpaRepository\u0026lt;Foo, Long\u0026gt; {} public interface BarRepository extends JpaRepository\u0026lt;Bar, Long\u0026gt; {} We need to ensure that each of these maps to a table in its own database. To achieve this, we will need two separate entity managers, each of which has a different datasource. However, in a Spring Java config @Configuration class, we can only have one @EnableJpaRepositories annotation and each such annotation can only reference one EntityManagerFactory. To achieve this, we create two separate @Configuration classes: FooConfig and BarConfig.\nEach of these @Configuration classes defines a DataSource based on an embedded HSQL database. The following is the BarConfig. FooConfig is identical except for some different names and package paths.\nBarConfig.java gist · stephen-masters/7530207 java Copy 123456789101112131415161718192021222324252627282930313233343536373839404142434445 @Configuration @EnableTransactionManagement @EnableJpaRepositories( entityManagerFactoryRef = \u0026#34;barEntityManagerFactory\u0026#34;, transactionManagerRef = \u0026#34;barTransactionManager\u0026#34;, basePackages = { \u0026#34;com.sctrcd.multidsdemo.integration.repositories.bar\u0026#34; }) public class BarConfig { @Autowired JpaVendorAdapter jpaVendorAdapter; /** * Primary because if we have activated embedded databases, we do not want * the application to connect to an external database. */ @Bean(name = \u0026#34;barDataSource\u0026#34;) public DataSource dataSource() { return new EmbeddedDatabaseBuilder() .setName(\u0026#34;bardb\u0026#34;) .setType(EmbeddedDatabaseType.HSQL) .build(); } @Bean(name = \u0026#34;barEntityManager\u0026#34;) public EntityManager entityManager() { return entityManagerFactory().createEntityManager(); } @Bean(name = \u0026#34;barEntityManagerFactory\u0026#34;) public EntityManagerFactory entityManagerFactory() { LocalContainerEntityManagerFactoryBean lef = new LocalContainerEntityManagerFactoryBean(); lef.setDataSource(dataSource()); lef.setJpaVendorAdapter(jpaVendorAdapter); lef.setPackagesToScan(\u0026#34;com.sctrcd.multidsdemo.domain.bar\u0026#34;); lef.setPersistenceUnitName(\u0026#34;barPersistenceUnit\u0026#34;); lef.afterPropertiesSet(); return lef.getObject(); } @Bean(name = \u0026#34;barTransactionManager\u0026#34;) public PlatformTransactionManager transactionManager() { return new JpaTransactionManager(entityManagerFactory()); } } Each configuration should define a DataSource, EntityManager, EntityManagerFactory and PlatformTransactionManager. You need to make sure that @Entity beans for different data sources are in different packages. We then need to put the correct references in the @EnableJpaRepositories annotation for each @Configuration class.\n@Configuration @EnableTransactionManagement @EnableJpaRepositories( entityManagerFactoryRef = \u0026quot;barEntityManagerFactory\u0026quot;, transactionManagerRef = \u0026quot;barTransactionManager\u0026quot;, basePackages = {\u0026quot;com.sctrcd.multidsdemo.integration.repositories.bar\u0026quot;}) public class BarConfig { // ... } @Configuration @EnableTransactionManagement @EnableJpaRepositories( entityManagerFactoryRef = \u0026quot;barEntityManagerFactory\u0026quot;, transactionManagerRef = \u0026quot;barTransactionManager\u0026quot;, basePackages = { \u0026quot;com.sctrcd.multidsdemo.integration.repositories.bar\u0026quot; }) public class BarConfig { // ... } As you can see, each of these @EnableJpaRepositories annotations defines a specific named EntityManagerFactory and PlatformTransactionManager. They also specify which repositories should be wired up with those beans. In the example, I have put the repositories in database-specific packages. It is also possible to define each individual repository by name, by adding includeFilters to the annotation, but by segregating the repositories by database, I believe that things should end up more readable.\nAt this point you should have a working application using Spring Data repositories to manage entities in two separate databases. Feel free to grab the project from the link above and run the tests to see this happening. And please do let me know if you can spot any good opportunities for improvement.\nUpdate Since writing the post above, I have had the opportunity to implement a multiple datasource solution in a Spring Boot application. As a few people asked about it, here’s a follow-up post describing what needs to be done to implement multiple data sources in a Spring Boot application.\n"},{"url":"/2011/05/playing-around-with-apache-camel/","title":"Playing around with Apache Camel","summary":"First impressions of Apache Camel for implementing Enterprise Integration Patterns — a content-based router and a CSV-to-XML transform with almost no code.","date":"2011-05-12","tags":["java"],"cover":"mint","body":"I have been playing around with Apache Camel for a couple of projects recently, and so far I’m very impressed. Camel is one of a number of frameworks that seem to have sprung up over the past few years in response to the book Enterprise Integration Patterns by Gregor Hohpe and Bobby Woolf. It attempts to provide mechanisms to support all the patterns described in the book. And it does so very well, from what I have experienced so far. So I thought I would mention a couple of things I have done with it.\nA simple content-based router The problem I was trying to solve was that a legacy application was designed to listen to a WebSphere MQ queue, which would contain requests for a variety of operations. A new application had been developed to handle a subset of these operations. I couldn’t have both applications listening to the same queue, so I needed to divert particular operation request messages to a separate new queue.\nI needed to put together a simple content-based router that would inspect the header of each incoming message and route the message to a different destination depending on the operation name. I was able to implement this by defining a route which used XPath to select an endpoint based on XML attributes. This could be done in the Camel context XML file and the project contained no code outside this file.\ncamel-content-based-router.xml gist · stephen-masters/4546092 xml Copy 123456789101112131415161718 \u0026lt;camelContext xmlns=\u0026#34;http://camel.apache.org/schema/spring\u0026#34;\u0026gt; \u0026lt;route\u0026gt; \u0026lt;from uri=\u0026#34;webspheremq:ANY.OPERATION.REQUEST\u0026#34; /\u0026gt; \u0026lt;choice\u0026gt; \u0026lt;when\u0026gt; \u0026lt;xpath\u0026gt;/REQUEST/HEADER[@OperationName=\u0026#39;Old.Request\u0026#39;]\u0026lt;/xpath\u0026gt; \u0026lt;to uri=\u0026#34;activemq:OLD.REQUEST\u0026#34; /\u0026gt; \u0026lt;/when\u0026gt; \u0026lt;when\u0026gt; \u0026lt;xpath\u0026gt;/REQUEST/HEADER[@OperationName=\u0026#39;New.Request\u0026#39;]\u0026lt;/xpath\u0026gt; \u0026lt;to uri=\u0026#34;activemq:NEW.REQUEST\u0026#34; /\u0026gt; \u0026lt;/when\u0026gt; \u0026lt;otherwise\u0026gt; \u0026lt;to uri=\u0026#34;activemq:DEAD.LETTER.QUEUE\u0026#34; /\u0026gt; \u0026lt;/otherwise\u0026gt; \u0026lt;/choice\u0026gt; \u0026lt;/route\u0026gt; \u0026lt;/camelContext\u0026gt; A CSV to XML transform Here, I was dealing with integrating two off the shelf applications, the aim being to facilitating exporting a document and meta data from one system and importing it into the other. When exporting a document from the first application, a CSV would be generated in a directory on the filesystem. The other application provided an import adapter, which required an XML trigger file. I needed a small application to follow the following steps:\nListen for CSV files being dropped in a directory on the filesystem. Split the CSV up into separate requests for each document being exported. Generate an XML trigger file for each request. Drop the XML trigger file into a directory ready for import by the downstream system. As well as providing middleware messaging adapters, Camel also supports defining endpoints that are directories on the filesystem, so it can automatically create a listener for a directory. To deal with the first two steps, I made use of opencsv to parse the CSV, but as I soon discovered, Camel also provided CSV unmarshallers.\nI extended the Camel RouteBuilder and using the fluent DSL for Java, defined my routes and created a Splitter class that would take the unmarshalled CSV and output a list of messages to an internal queue., this looked a bit like the following. I then defined a route to pick the individual metadata messages off the internal queue and use a Processor to generate XML in the required format.\nCamelCsvToXml.java gist · stephen-masters/4546103 java Copy 12345678 from(\u0026#34;C:/router/export/csv/\u0026#34;) .unmarshal().csv() .split().method(\u0026#34;org.gratiartis.router.Splitter\u0026#34;, \u0026#34;split\u0026#34;) .to(\u0026#34;jms:DOCUMENT.METADATA.QUEUE\u0026#34;); from(\u0026#34;jms:DOCUMENT.METADATA.QUEUE\u0026#34;) .processRef(\u0026#34;org.gratiartis.router.Processor\u0026#34;) .to(\u0026#34;C:/router/out/xml/\u0026#34;); Further reading Camel is very comprehensive and is also one of the best documented projects out there. I keep trying to implement something myself and then finding that there’s already something that will do the job for me.\nThese are probably the best starting points for info on any particular integration pattern:\nCamel enterprise integration patters Camel architecture And this was one of the better tutorial introductions to it:\nCamel integration tutorial If Camel sounds good, you should also take a look at Spring Integration. It’s another framework based on the patterns in the Enterprise Integration Patterns book, but has that Spring tendency towards implementation through bean annotations. But you don’t have to pick one or the other; the Camel project has developed the camel-spring-integration library to provide a bridge from Camel components to Spring Integration endpoints.\n"},{"url":"/2008/04/weblogic-scripting-tool-scripts/","title":"WebLogic Scripting Tool scripts","summary":"Useful WLST resources and Gist examples for scripting WebLogic domain creation and server administration.","date":"2008-04-05","tags":["java"],"cover":"cobalt","body":"I mentioned my use of the WebLogic Scripting Tool a little while back. I have noticed since then that a number of folks visiting this site are looking for example scripts. I have obviously written a number myself and I promise I’ll try to get around to posting them here. However until I get myself in gear, I thought I would point you at some useful examples that are already out there. I’ll expand this post as I find more…\nFirst, make sure you sign up on http://dev2dev.bea.com/. That’s the BEA site supporting developers where you will find provides news, tutorials and samples to help you get going with WebLogic.\nThey have a number of projects and code samples that you will be able to get at. My recommendation for getting started is to go to the CodeShare section and have a look around at all the good stuff that has been provided by generous developers around the world.\nThere’s a WLST project maintained by the guys who developed it in the first place. This contains a whole load of scripts.\nAnd to give you more of a head start, you can go to the Code Samples area of dev2dev and search for WLST, which will give you a list of samples worth looking at. Currently these include a Server Health Monitor (artifact S198), which will periodically check your runtime heap and execute queues and log the state to file.\nAlso, if you are looking at WLST for its server monitoring capabilities, you should know that BEA Guardian is now free!\nUpdate A few folks have asked about getting hold of some example scripts. Unfortunately most of what I have written is for work, so it is tied to work environments and not mine to share. However, I have created some simple scripts for sharing, that cover scripting the creation of a domain. These are available as GitHub gists for creating WebLogic domains:\nconfig.properties gist · stephen-masters/589660 properties Copy 123456789101112131415161718192021222324252627282930313233343536 bea_home=/path/to/bea java_home=/path/to/bea/jdk160_18 domain_template=/path/to/bea/weblogic11/common/templates/domains/wls.jar admin_server_name=my_admin_server admin_url=t3\\://myserver\\:7001 admin_user=weblogic admin_password=w3blogicpwd admin_port=7001 admin_port_secure=7002 domain_name=my_domain my_server_1_port=8001 my_server_2_port=8011 my_domain_log4j_file=/path/to/bea/user_projects/domains/my_domain/log4j.xml # ---------------------------------------------------------------------- # Datasources # ---------------------------------------------------------------------- my_ds_name=my_datasource my_ds_targets=my_server_1,my_server_2 my_ds_jndi=my.datasource my_ds_url=jdbc\\:oracle\\:thin\\:@dbserver\\:dbport\\:dbname my_ds_user=dbusername my_ds_password=dbuserpassword # ---------------------------------------------------------------------- # Applications # ---------------------------------------------------------------------- my_app_name=my_app my_app_targets=my_server_1 my_app_path=/path/to/my_app_1.0.ear create_domain.py gist · stephen-masters/589660 py Copy 1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556575859606162636465666768697071727374757677787980818283 \u0026#34;\u0026#34;\u0026#34;Stage 1 domain creation script. Read a template domain and make some offline modifications to it to define admin server details and create some managed servers. A follow-up online phase is needed once the servers have been started up. \u0026#34;\u0026#34;\u0026#34; execfile(\u0026#34;wlst_util_offline.py\u0026#34;) #======================================================================================= # Open a domain template. #======================================================================================= print \u0026#34;Reading: \u0026#34; \u0026#43; domain_template readTemplate(domain_template) #======================================================================================= # Configure the Administration Server. #======================================================================================= print \u0026#34;Moving to Server/AdminServer\u0026#34; cd(\u0026#39;Server/AdminServer\u0026#39;) print \u0026#34;Setting Name to: \u0026#34; \u0026#43; admin_server_name set(\u0026#39;Name\u0026#39;, admin_server_name) print \u0026#34;Setting ListenAddress to nothing.\u0026#34; set(\u0026#39;ListenAddress\u0026#39;, \u0026#39;\u0026#39;) print \u0026#34;Setting ListenPort to \u0026#34; \u0026#43; admin_port set(\u0026#39;ListenPort\u0026#39;, int(admin_port)) create(admin_server_name, \u0026#39;SSL\u0026#39;) cd(\u0026#39;SSL/\u0026#39; \u0026#43; admin_server_name) set(\u0026#39;Enabled\u0026#39;, \u0026#39;True\u0026#39;) set(\u0026#39;ListenPort\u0026#39;, int(admin_port_secure)) #======================================================================================= # Define the password for user weblogic. You must define the password before you # can write the domain. #======================================================================================= print \u0026#34;Admin password: \u0026#34; \u0026#43; admin_password cd(\u0026#39;/\u0026#39;) cd(\u0026#39;Security/base_domain/User/weblogic\u0026#39;) cmo.setPassword(admin_password) #======================================================================================= # Set Options: # - CreateStartMenu: Enable creation of Start Menu shortcut. # - ServerStartMode: Set mode to development. # - JavaHome: Sets home directory for the JVM used when starting the server. # - OverwriteDomain: Overwrites domain, when saving, if one exists. #======================================================================================= setOption(\u0026#39;CreateStartMenu\u0026#39;, \u0026#39;true\u0026#39;) setOption(\u0026#39;ServerStartMode\u0026#39;, \u0026#39;prod\u0026#39;) setOption(\u0026#39;JavaHome\u0026#39;, bea_home \u0026#43; \u0026#39;/jdk160_18\u0026#39;) setOption(\u0026#39;OverwriteDomain\u0026#39;, \u0026#39;true\u0026#39;) #======================================================================================= # Write the domain and close the domain template. #======================================================================================= writeDomain(bea_home \u0026#43; \u0026#39;/user_projects/domains/\u0026#39; \u0026#43; domain_name) closeTemplate() #======================================================================================= # Reopen the domain. #======================================================================================= readDomain(bea_home \u0026#43; \u0026#39;/user_projects/domains/\u0026#39; \u0026#43; domain_name) print \u0026#39;Creating managed servers...\u0026#39; create_managed_server(\u0026#39;my_server_1\u0026#39;, my_server_1_port, \u0026#39;\u0026#39;, my_domain_log4j_file) create_managed_server(\u0026#39;my_server_2\u0026#39;, my_server_2_port, \u0026#39;\u0026#39;, my_domain_log4j_file) #======================================================================================= # We\u0026#39;re done with configuring the domain. Close the domain template. #======================================================================================= updateDomain() closeDomain() exit() wlst_util_offline.py gist · stephen-masters/589660 py Copy 123456789101112131415161718192021222324252627282930313233343536373839 \u0026#34;\u0026#34;\u0026#34;A collection of utility methods used in creating a domain. \u0026#34;\u0026#34;\u0026#34; import os def create_managed_server(server_name, port, listen_address, log4jfile): \u0026#34;\u0026#34;\u0026#34;Creates a managed server on the domain.\u0026#34;\u0026#34;\u0026#34; print \u0026#39;Creating managed server: server_name=\u0026#39; \u0026#43; server_name \\ \u0026#43; \u0026#39;, port=\u0026#39; \u0026#43; str(port) \\ \u0026#43; \u0026#39;, listen_address=\u0026#39; \u0026#43; listen_address \\ \u0026#43; \u0026#39;, log4jfile=\u0026#39; \u0026#43; log4jfile cd(\u0026#39;/\u0026#39;) create(server_name, \u0026#39;Server\u0026#39;) cd(\u0026#39;/Servers/\u0026#39; \u0026#43; server_name) set(\u0026#39;ListenPort\u0026#39;, int(port)) set(\u0026#39;ListenAddress\u0026#39;, listen_address) set(\u0026#39;Machine\u0026#39;, get_machine_name()) create(server_name, \u0026#39;ServerStart\u0026#39;) def get_machine_name(): \u0026#34;\u0026#34;\u0026#34;Determines the physical machine name.\u0026#34;\u0026#34;\u0026#34; # HOSTNAME is usual on UNIX, but COMPUTERNAME is the Windows location. if os.environ.has_key(\u0026#39;HOSTNAME\u0026#39;): return os.getenv(\u0026#39;HOSTNAME\u0026#39;) elif os.environ.has_key(\u0026#39;COMPUTERNAME\u0026#39;): return os.getenv(\u0026#39;COMPUTERNAME\u0026#39;) else: return \u0026#39;UNKNOWN\u0026#39; def create_machine(machineName): \u0026#34;\u0026#34;\u0026#34;Creates a \u0026#39;machine\u0026#39; on the domain against which servers can be allocated. This is only used in clustered environments. \u0026#34;\u0026#34;\u0026#34; cd(\u0026#39;/\u0026#39;) create(machineName, \u0026#39;Machine\u0026#39;) "},{"url":"/2007/02/weblogic-scripting-tool/","title":"WebLogic Scripting Tool","summary":"BEA's Jython-based scripting tool for automating WebLogic server administration — create domains, manage deployments, and handle disaster recovery without restarting.","date":"2007-02-05","tags":["java"],"cover":"cobalt","body":"According to the BEA documentation, the WebLogic Scripting Tool is a command-line scripting interface that system administrators and operators use to monitor and manage WebLogic Server instances and domains. It allows you to write scripts in Jython that are able to connect to a running WebLogic domain and make modifications to the configuration with no need to restart anything. It can also be used for creating and modifying a domain in its offline mode. It comes as standard with WebLogic 9.2 and a version is available for 8.1. It is recommended and supported by BEA for automating WebLogic server administration. I am currently developing WLST scripts to improve the development and deployment process.\nI see it as having the following potential benefits:\nStreamlining development – As it can be executed from an Ant build and cause an application version to be undeployed and replaced with a new version on a running server, all without intervention. Improving deployments – Manual steps in a deployment are slow and unreliable. At some stage they are guaranteed to go wrong. The scripted nature of this means that a deployment can be tested against multiple environments and proven before going live. You know that the deployment method for production is the one that produced your test environments. Faster, more reliable disaster recovery – Scripts can be developed to handle a number of failures. i.e. If a database fails and needs to be run from a DR server, scripts can be written in advance to re-create all connection pools pointed at the DR location. This way, the disaster recovery process is fast and reliable. The person initiating the fail-over only needs to know where to find the appropriate scripts. They do not need to know the steps themselves. Monitoring – Scripts can be written (many already exist) to connect to the running server and monitor it. This can include things such as checking whether message queues are live, testing connection pools, monitoring the JVM heap and various other tasks. Useful links for getting started This page has only existed for a very short time, so I haven’t had much opportunity to develop my own content. However, there is already a lot of good documentation out there that would help someone get started with WLST. Here I present my bucket of links that I have found useful.\nWebLogic Scripting Tool WLST project home at Dev2Dev Environment proving with WLST WLST Online and Offline command summary Using WLST offline Automating WebLogic platform application provisioning WLNav – Interview with developers at Dev2Dev Jython Source code I have already written a number of WLST objects and scripts to make my life easier. I need to work on pulling them out into the web site in a manner that I’m happy with, but in the meantime if you are interested, please get in touch and I can send you what I have so far.\nUpdate A few folks have asked about getting hold of some example scripts. Unfortunately most of what I have written is for work, so it is tied to work environments and not mine to share. However, I have created some simple scripts for sharing, that cover scripting the creation of a domain. These are available as GitHub gists for creating WebLogic domains:\nconfig.properties gist · stephen-masters/589660 properties Copy 123456789101112131415161718192021222324252627282930313233343536 bea_home=/path/to/bea java_home=/path/to/bea/jdk160_18 domain_template=/path/to/bea/weblogic11/common/templates/domains/wls.jar admin_server_name=my_admin_server admin_url=t3\\://myserver\\:7001 admin_user=weblogic admin_password=w3blogicpwd admin_port=7001 admin_port_secure=7002 domain_name=my_domain my_server_1_port=8001 my_server_2_port=8011 my_domain_log4j_file=/path/to/bea/user_projects/domains/my_domain/log4j.xml # ---------------------------------------------------------------------- # Datasources # ---------------------------------------------------------------------- my_ds_name=my_datasource my_ds_targets=my_server_1,my_server_2 my_ds_jndi=my.datasource my_ds_url=jdbc\\:oracle\\:thin\\:@dbserver\\:dbport\\:dbname my_ds_user=dbusername my_ds_password=dbuserpassword # ---------------------------------------------------------------------- # Applications # ---------------------------------------------------------------------- my_app_name=my_app my_app_targets=my_server_1 my_app_path=/path/to/my_app_1.0.ear create_domain.py gist · stephen-masters/589660 py Copy 1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556575859606162636465666768697071727374757677787980818283 \u0026#34;\u0026#34;\u0026#34;Stage 1 domain creation script. Read a template domain and make some offline modifications to it to define admin server details and create some managed servers. A follow-up online phase is needed once the servers have been started up. \u0026#34;\u0026#34;\u0026#34; execfile(\u0026#34;wlst_util_offline.py\u0026#34;) #======================================================================================= # Open a domain template. #======================================================================================= print \u0026#34;Reading: \u0026#34; \u0026#43; domain_template readTemplate(domain_template) #======================================================================================= # Configure the Administration Server. #======================================================================================= print \u0026#34;Moving to Server/AdminServer\u0026#34; cd(\u0026#39;Server/AdminServer\u0026#39;) print \u0026#34;Setting Name to: \u0026#34; \u0026#43; admin_server_name set(\u0026#39;Name\u0026#39;, admin_server_name) print \u0026#34;Setting ListenAddress to nothing.\u0026#34; set(\u0026#39;ListenAddress\u0026#39;, \u0026#39;\u0026#39;) print \u0026#34;Setting ListenPort to \u0026#34; \u0026#43; admin_port set(\u0026#39;ListenPort\u0026#39;, int(admin_port)) create(admin_server_name, \u0026#39;SSL\u0026#39;) cd(\u0026#39;SSL/\u0026#39; \u0026#43; admin_server_name) set(\u0026#39;Enabled\u0026#39;, \u0026#39;True\u0026#39;) set(\u0026#39;ListenPort\u0026#39;, int(admin_port_secure)) #======================================================================================= # Define the password for user weblogic. You must define the password before you # can write the domain. #======================================================================================= print \u0026#34;Admin password: \u0026#34; \u0026#43; admin_password cd(\u0026#39;/\u0026#39;) cd(\u0026#39;Security/base_domain/User/weblogic\u0026#39;) cmo.setPassword(admin_password) #======================================================================================= # Set Options: # - CreateStartMenu: Enable creation of Start Menu shortcut. # - ServerStartMode: Set mode to development. # - JavaHome: Sets home directory for the JVM used when starting the server. # - OverwriteDomain: Overwrites domain, when saving, if one exists. #======================================================================================= setOption(\u0026#39;CreateStartMenu\u0026#39;, \u0026#39;true\u0026#39;) setOption(\u0026#39;ServerStartMode\u0026#39;, \u0026#39;prod\u0026#39;) setOption(\u0026#39;JavaHome\u0026#39;, bea_home \u0026#43; \u0026#39;/jdk160_18\u0026#39;) setOption(\u0026#39;OverwriteDomain\u0026#39;, \u0026#39;true\u0026#39;) #======================================================================================= # Write the domain and close the domain template. #======================================================================================= writeDomain(bea_home \u0026#43; \u0026#39;/user_projects/domains/\u0026#39; \u0026#43; domain_name) closeTemplate() #======================================================================================= # Reopen the domain. #======================================================================================= readDomain(bea_home \u0026#43; \u0026#39;/user_projects/domains/\u0026#39; \u0026#43; domain_name) print \u0026#39;Creating managed servers...\u0026#39; create_managed_server(\u0026#39;my_server_1\u0026#39;, my_server_1_port, \u0026#39;\u0026#39;, my_domain_log4j_file) create_managed_server(\u0026#39;my_server_2\u0026#39;, my_server_2_port, \u0026#39;\u0026#39;, my_domain_log4j_file) #======================================================================================= # We\u0026#39;re done with configuring the domain. Close the domain template. #======================================================================================= updateDomain() closeDomain() exit() wlst_util_offline.py gist · stephen-masters/589660 py Copy 123456789101112131415161718192021222324252627282930313233343536373839 \u0026#34;\u0026#34;\u0026#34;A collection of utility methods used in creating a domain. \u0026#34;\u0026#34;\u0026#34; import os def create_managed_server(server_name, port, listen_address, log4jfile): \u0026#34;\u0026#34;\u0026#34;Creates a managed server on the domain.\u0026#34;\u0026#34;\u0026#34; print \u0026#39;Creating managed server: server_name=\u0026#39; \u0026#43; server_name \\ \u0026#43; \u0026#39;, port=\u0026#39; \u0026#43; str(port) \\ \u0026#43; \u0026#39;, listen_address=\u0026#39; \u0026#43; listen_address \\ \u0026#43; \u0026#39;, log4jfile=\u0026#39; \u0026#43; log4jfile cd(\u0026#39;/\u0026#39;) create(server_name, \u0026#39;Server\u0026#39;) cd(\u0026#39;/Servers/\u0026#39; \u0026#43; server_name) set(\u0026#39;ListenPort\u0026#39;, int(port)) set(\u0026#39;ListenAddress\u0026#39;, listen_address) set(\u0026#39;Machine\u0026#39;, get_machine_name()) create(server_name, \u0026#39;ServerStart\u0026#39;) def get_machine_name(): \u0026#34;\u0026#34;\u0026#34;Determines the physical machine name.\u0026#34;\u0026#34;\u0026#34; # HOSTNAME is usual on UNIX, but COMPUTERNAME is the Windows location. if os.environ.has_key(\u0026#39;HOSTNAME\u0026#39;): return os.getenv(\u0026#39;HOSTNAME\u0026#39;) elif os.environ.has_key(\u0026#39;COMPUTERNAME\u0026#39;): return os.getenv(\u0026#39;COMPUTERNAME\u0026#39;) else: return \u0026#39;UNKNOWN\u0026#39; def create_machine(machineName): \u0026#34;\u0026#34;\u0026#34;Creates a \u0026#39;machine\u0026#39; on the domain against which servers can be allocated. This is only used in clustered environments. \u0026#34;\u0026#34;\u0026#34; cd(\u0026#39;/\u0026#39;) create(machineName, \u0026#39;Machine\u0026#39;) "}]