← HDC technology spike

Specification

We are performing a technology spike to iterate on key design decisions needed to be explored before committing to a future development process. Specifically we are:

Because you are working within a technology spike you will be generating multiple iterations of the same solution switching to other technologies and then listing their pros/cons of each one to help with final evaluation.

We will create all iterations of the site and then create a landing page which will allow us to navigate these different static sites.

Aims

Progress so far, and requirements for subsequent spikes

All three planned SSG iterations are now built: spikes/astro/, spikes/nextjs/ (static export), spikes/eleventy/ — each a full vertical slice (SSG + data model + search + filters + CMS + styling + markdown-page rendering), each with its own EVALUATION.md. See /REPORT.md at the repo root for the formal cross-spike conclusion and recommendation — that document, not this one, is the place to look for "which SSG should we pick and why."

Several artifacts are shared infrastructure across all three iterations, not duplicated per SSG:

Every SSG iteration reads directly from /data and /schema rather than keeping its own copy.

The Astro spike also surfaced requirements that weren't explicit in earlier drafts of this document but apply to every iteration, not just Astro:

Selecting a suitable CMS/static site engine

Possible CMS/static site options. This must be able to generate content based on submitted data e.g. YAML, can be easily modified by a non-technical individual, a trivial context switch to managing, scale to the needs of HDC (portal, search) and our get rid of WordPress goal (tables, images, videos)

Integrate search

Search engine (in-process JavaScript) to support HDC. Solution needs to offer filters, search over non-displayed attributes, free text, scale to 100s easily within the browser.

You will need to review the current offering from HDC for the filters and review what has been said below. Ensure the design of the system allows us to iterate on each of these as individual loaders for the search. Potentially have a separate instance of the SSG to provide this functionality.

Search MUST be able to search within a document NOT limited to what is provided in the first page of the interface e.g. searching for "Sangya" should bring back the "Cohort Discovery Service" due to the contactTeamPerson is "Sangya Pundir".

We want a go button on the main page next to the search box and a reset so we can reset all options.

Each introduction page (landing page) is effectively an implementation of a search engine which will then find hits on the other pages. So produce a page for each implementation and integrate as best you can. Write a short summary per engine about how easy/hard it was to integrate. Each page for search MUST include the filters below since this is attempting to evaluate how well the products work.

Filters

Review https://ukhealthdata.org/health-data-connect/ and the screenshot for details of the filters. We should support filters and data that do not appear in the summary listings. This means we support

Provide a way to clear the active filters too

CMS gui like options

To support the removal of WordPress we need to support non-technical people to develop pages and deposit custom formats and data including images and videos. We have identified the following two example CMS solutions providing low cost editors over a GitHub repo

Goals for this spike are

Note additional constraints defined by the CMS issue (you are free to let this be a human interaction)

We should provide a link to which ever CMS is used on the home page.

Styling

Reuse the existing styling developed for HDC. Take it from their website and integrate.

Release method

We want an automated release pattern with no devops involvement required. Already we assume this will be deployed via GCP's bucket HTML system with a small CDN over the top. The ingest of new records should take no additional effort to perform. Consider how we might support pre-scanning of entries.

Converting the HDC CSV data into something more useful

Take the example data and convert this into something better. Design a schema and look for the enums present. Define what they are.

Demonstrate Markdown rendering

Introduce evaluation.md as a page into the site alongside a summary markdown page which briefly describes the aims of this project. Render these as two separate pages reusing the correctly styling. You are testing the CMS' ability to allow a user to drop a markdown file and have it render sensibly without your input. Otherwise if you need to custom create pages for each file this is not easy.

Demonstrate alternative layouts and images

One key factor was altering layouts and bringing in images. We have the images folder and the markdown folder offering an outline.md which refers to images. You need to create a new layout which is different from the HDC portal and then demonstrate how we can alter the layout/rendering. This might include sharing the overall HEAD and structure of the site or creating a fresh stack again. Either way you MUST work with the SSG's tooling to ensure you are not cheating the system. In the case of one where there is no default markdown rendering you are responsible for creating a sensible infrastructure to support that automatic drop in rendering (as done above).

Individual environments/rendering

Explore how each SSG could offer a way to provide alternative environments based on the target deployment infrastructure. Almost like feature flag based rendering e.g. displaying something is on a dev/preprod/prod or a rebadged site.

Render the whole record

Entry detail pages MUST render every field present in the entry record — nothing in the data file may be omitted from the page, in each and every SSG implementation. Earlier iterations deliberately left a handful of fields unrendered (contactTeamPerson, originatorOrgs, adoptionEvidence, contact/adoption-pack/maturity fields) to demonstrate the "search over non-displayed attributes" requirement — that demonstration served its purpose, but a real portal shows the whole record. The "Sangya" search test remains in force, now as a regression check that search covers the full record.

Search-index-driven listings with pagination

The homepage listing must be rendered from the search index/corpus, not rendered in full and then hidden. The first iterations of every spike server-rendered every entry card into the homepage and used client JS to hide non-matches — acceptable at 13 entries, but the wrong architecture at the 100s of entries this system must scale to, and it quietly makes the DOM (not the search index) the real data source for filtering.

Requirements, for each and every SSG implementation:

Future planning

Data modelling and ValueSet usage

Spike one: Data schema definition

Review the data model from HDC (see web site for example). Identify possible gaps in the model compared to our own data model in gateway. Also vice versa. Identify value sets. Look at the data model we want to use to enforce this (can we use schemata and traser). Enforce using JSONSchema but be aware we might need to allow other data formats to be compared against the schema.

Identify possible links back to schema.org or similar entities for this work and we might want to embed that into the pages (semantic web support).

Spike two: ValueSets

A value set is a defined list of authorised values, codes, or terms used to validate data entry and ensure consistency. They are a lot like enums and in fact enums do the same but the key component here is they are publicly available, can be reused by others and almost stand alone as a standard. The can also semantically link out to other controlled vocabularies.

Goals for this are