Server-first Web Components with DSD, HTMX, and Islands

Over the last several years, browsers have made huge strides in bringing native components to HTML. In 2020, the first Web Component features reached parity across all major browsers, and in the years since, the list of capabilities has continued to grow. In particular, early this year, streaming Declarative Shadow DOM (DSD) finally reached universal support when Firefox shipped its implementation in February. It is this strategic addition to the HTML standard that unlocks a number of new, powerful server possibilities.

In this article, we’ll look at how to leverage existing server frameworks people already use, to level up to native components, without piling on mounds of JavaScript. While I’ll be demonstrating these techniques with Node.js, Express, and Handlebars, nearly every modern web framework today supports the core concepts and extensibility mechanisms I’ll be showing. So, whether you’re working with Node.js, Rails, or even C# and .NET, you’ll be able to translate these techniques to your stack.

If you are new to Web Components or if you find the following exploration interesting, I’d like to recommend you to my own Web Component Engineering course. Its 13 modules, 170+ videos, and interactive learning app will guide you through DOM APIs, Web Components, modular CSS, accessibility, forms, design systems, tools, and more. It’s a great way to up your web standards game.

Background

Before we jump into the code, it may be worth reminding ourselves of a few things.

As I mentioned, the first Web Component features were universally supported by browsers by early 2020. This included a few core capabilities:

• Inert HTML and basic templating through the <template> element.
• The ability to define new HTML tags with the customElements.define(...) API.
• Platform protected encapsulation of HTML and CSS, along with DOM composition, provided by Shadow DOM and <slot>.
• Basic theming through shadow-piercing CSS Properties.

By combining these standards with a few smaller ones, anyone could create fully native, interoperable components on the web. However, with the exception of CSS Properties, using all of these APIs required JavaScript. For example, notice how much JavaScript code is involved in creating a simple card component:

class UICard {
  static #fragment = null;
  #view = null;

  constructor() {
    super();
    this.attachShadow({ mode: "open" });
  }

  connectedCallback() {
    if (this.#view === null) {
      this.#view = this.#createView();
      this.shadowRoot.appendChild(this.#view);
    }
  }

  #createView() {
    if (UICard.#fragment === null) {
      // find the template and styles in the DOM
      const template = document.getElementById("ui-card");
      UICard.#fragment = document.adoptNode(template.content);
    }

    return UICard.#fragment.cloneNode(true);
  }
}

customElements.define("ui-card", UICard);

This is an unfortunate amount of boilerplate code, particularly for a component consisting only of basic HTML and CSS, with no real behavior. The code above works to find the template, clone it, create the Shadow DOM, and append the cloned template to it. So, unfortunately, the browser can’t render this card until after the JavaScript loads, parses, and runs. But what if there was a way to do all of this entirely in HTML, with no JavaScript?

Enter Declarative Shadow DOM (DSD). With DSD we have an HTML-first mechanism for declaring instances of a component, along with its Shadow DOM content and styles. The entire card component can be implemented with only the following HTML:

<ui-card>
  <template shadowrootmode="open">
    <style>
      :host {
        display: block;
        contain: content;
        box-sizing: border-box;
        box-shadow: var(--shadow-raised);
        border: var(--stroke-thicknessMinus1) var(--stroke-style) var(--color-layerBorder);
        border-radius: var(--border-radius);
        background: var(--background, transparent);
        color: var(--foreground, var(--color-onLayerBase));
      }
    </style>

    <slot></slot>
  </template>

  Card content goes here...
</ui-card>

That’s it. Put that HTML in your page and the <ui-card> will render with an automatically attached shadow root, fully encapsulated HTML and CSS, and slot-based composition of the card’s content. This is all accomplished through the addition of the shadowrootmode attribute, which tells the browser not to create a <template> but to instead attach a shadow root and stream the contents of the template element into the root it just created. This feature is implemented at the level of the HTML parser itself. No JavaScript required!

At first glance, this is fantastic. We can create basic components with no JavaScript, encapsulated HTML and CSS, slot-based composition, etc. But what if we have 100 cards on the page? We no longer have our HTML and CSS in just one place where our component is defined, but instead it’s duplicated everywhere we use the card. Yikes!

But this is no new problem. It’s as old as the web itself and is one of the reasons that web frameworks were created in the first place. And as we’ll see throughout the rest of this article, we can use all our standard server-side tools to not only solve this problem, but make further improvements to the solution as well.

The Demo App

We need a simple web app to help us demonstrate how all the pieces come together. So, I threw together a little list/detail UI based on the Star Wars API.

Along the left side of the screen, there is a list of a few Star Wars films. The list and the individual items are all implemented as Web Components, rendered entirely on the server. On the right of the screen, we have a detail view of the film that’s currently selected in the list. The detail view is also implemented as a Web Component, rendered on the server. As the list selection changes, declarative HTMX attributes in the list component HTML trigger AJAX requests for the associated film detail component from the server.

I call these types of Web Components “Server-first” because all rendering is done on the server. There is no client-side JavaScript code for rendering. In fact, there is hardly any JavaScript at all in this solution. As we’ll see shortly, we only need a few small pieces to enable the HTMX integration and progressively enhance our list with selection styling.

If you want to get the full code and try things out yourself, you can find it in this GitHub repo. The readme file contains instructions for setting everything up and running the app.

The structure of the demo is kept relatively simple and standard. At the root there are two folders: client and server. In the client folder, you will find page-level css, the images, and the JavaScript. In the server folder, I have broken down the code into controllers and views. This is also where the mock data resides as well as some core infrastructure code we’ll be going over shortly.

General Approach

For the demo, I’ve followed a fairly standard MVC-style approach. The server is comprised of two controllers:

• / – The home controller loads the list of films and renders the “home” view, displaying the list and the default selection.
• /films/:id – The films controller handles requests for specific films. When it is invoked via an AJAX request, it loads the film details data and returns a partial view, including only the film details. When it is invoked normally, it renders the entire home view, but with the specified film selected in the list and its detail view off to the side.

Each controller invokes the “backend” as needed, builds up a view model, and then passes that data along to the view engine, which renders the HTML. To improve ergonomics and enable some of the more advanced features of the architecture, Handlebars partial views and HTML helpers are used.

There’s nothing exceptional about this. It’s all standard fare for anyone using MVC frameworks since Rails emerged on the scene in the early 2000s. However, the devil is in the details…

Rendering Web Components

In this architecture, Web Component templates (views) and styles, as well as their data inputs, are fully defined on the server, not the client. To accomplish this, we use a partial view per component, much like one would do in a traditional MVC architecture. Let’s take a look at the Handlebars server code for the same <ui-card> we discussed previously.

<ui-card>
  <template shadowrootmode="open">
    {{{shared-styles "./ui-card.css"}}}
    <slot></slot>
  </template>

  {{>@partial-block}}
</ui-card>

This code defines our <ui-card> partial. It doesn’t have any data inputs, but it can render child content with Handlebars’ {{>@partial-block}} syntax. The other interesting thing to note is the shared-styles custom HTML helper. We’ll look at that in detail later. For now, just know that it automatically includes the CSS located in the specified file.

With the basic card defined, now we can build up more interesting and complex components on the server. Here’s a <structured-card> that has specific opinions about how header, body, and footer content should be structured and styled in card form.

<structured-card>
  <template shadowrootmode="open">
    {{{shared-styles "./structured-card.css"}}}

    {{#>ui-card}}
      <div part="content">
        <slot name="header"></slot>
        <slot></slot>
        <slot name="footer"></slot>
      </div>
    {{/ui-card}}
  </template>

  {{>@partial-block}}
</structured-card>

We follow the same basic pattern as <ui-card>. The main difference being that the <structured-card> actually composes the <ui-card> in its own Shadow DOM with {{#>ui-card}}...{{/ui-card}} (because it’s a Handlebars partial block itself).

Both these components are still highly generic. So, let’s now look at the film-card, which is just a standard partial view. It doesn’t define a Web Component, but instead uses the <structured-card> by merging it with film data:

{{#>structured-card}}
  <h3 slot="header">{{film.title}}</h3>
  <span slot="footer">Released {{film.release_date}}</span>
{{/structured-card}}

Now that we have a partial that can render film cards, we can put those together in a list. Here’s a slightly more advanced <film-list> Web Component:

<film-list>
  <template shadowrootmode="open">
    {{{shared-styles "./film-list.css"}}}

    <ul hx-boost="true" hx-target="global #film-detail">
      {{#each films}}
        <li>
          <a href="/films/{{id}}">
            {{>film-card film=.}}
          </a>
        </li>
      {{/each}}
    </ul>
  </template>
</film-list>

Here, we can see how the <film-list> has a films array as input. It then loops over each film in the array, rendering it with a link that encompasses our film-card, which internally renders the film data with a <structured-card>.

If you’ve been building MVC apps for a while, you may recognize that these are all familiar patterns for decomposing and recomposing views. However, you probably noticed a few significant changes.

1. First, each partial that serves as a Web Component has a single root element. That root element is a custom HTML tag of our choosing, following the platform custom element naming rules (i.e. names must include a hyphen). Examples: <film-list>, <structured-card>, <ui-card>.
2. Second, each Web Component contains a <template> element with the shadowrootmode attribute applied. This declares our Shadow DOM and enables us to provide the specific HTML that will get rendered therein, whenever we use the component.
3. Third, each component uses a custom Handlebars HTML helper called shared-styles to include the styles within the Shadow DOM, ensuring that the component is always delivered to the browser with its required styles, and that those styles are fully encapsulated.
4. Finally, components that are intended to be wrappers around other content use <slot> elements (a web standard), combined with Handlebar’s special {{>@partial-block}} helper to allow the server view engine to properly render the wrapped HTML as a child of the custom element tag.

So, the pattern roughly looks like this:

<tag-name>
  <template shadowrootmode="open">
    {{{shared-styles "./tag-name.css"}}}

    Component HTML goes here. 
    Add <slot></slot> and the partial block helper below if you need to render child content.
  </template>

  {{>@partial-block}}
</tag-name>

Those are the basic steps that enable us to author server-rendered Web Components. Hopefully, you can see from the several examples above how we can use these simple steps to create a wide variety of components. Next, let’s dig a little deeper into the technical details of the server and client infrastructure that make styling and dynamic behaviors work smoothly.

Tricks of the Trade: Sharing Styles

When we first looked at our manual DSD-based <ui-card> component, we inlined the styles. As a reminder, it looked like this:

<ui-card>
  <template shadowrootmode="open">
    <style>
      :host { ...host styles here... }
    </style>

    <slot></slot>
  </template>

  Card content goes here...
</ui-card>

One big problem with this HTML is that every time we have an instance of the card, we have to duplicate the styles. This means the server is sending down CSS for every single card instance, instead of just once, shared across all instances. If you have 100 cards, you have 100 copies of the CSS. That’s certainly not ideal (though some of the cost can be mitigated with GZIP compression).

ASIDE: Currently, W3C/WHATWG is working on a new web standard to enable declaratively sharing styles in DSD as well as representing multiple style sheets in the same file. Once this standards work is complete and shipped in browsers, the solution presented below will no longer be needed.

We can solve this problem though. There are two pieces to the puzzle:

• We need a way for the server to send the styles for the first instance of any component it renders, and know not to send the styles for any successive instances of the same component in the same HTTP request.
• We need a way for the browser to capture the styles sent by the server and propagate them across all instances of the same component.

To accomplish this, we’ll create a simple shared style protocol. In order to explain it, let’s look at what we’ll have the server send when it needs to render two card components:

<ui-card>
  <template shadowrootmode="open">
    <style style-id="./ui-card.css">
      :host { ...host styles here... }
    </style>
    <shared-styles style-id="./ui-card.css"></shared-styles>

    <slot></slot>
  </template>

  Card 1 Content Here.
</ui-card>

<ui-card>
  <template shadowrootmode="open">
    <shared-styles style-id="./ui-card.css"></shared-styles>

    <slot></slot>
  </template>

  Card 2 Content Here.
</ui-card>

Notice that the first card instance has the inline <style> element, with a special attribute added: style-id. This is followed by a special custom element called <shared-styles> that also has an attribute referencing the same style-id. In the second instance of the card, we don’t have the repeated <style> element anymore. We only have the <shared-styles> element, referencing the same style-id.

The first part of getting this working is in how we implement the <shared-styles> custom element in the browser. Let’s take a look at the code:

const lookup = new Map();

class SharedStyle extends HTMLElement {
  connectedCallback() {
    const id = this.getAttribute("style-id");
    const root = this.getRootNode();
    let styles = lookup.get(id);

    if (styles) {
      root.adoptedStyleSheets.push(styles);
    } else {
      styles = new CSSStyleSheet();
      const element = root.getElementById(id);
      styles.replaceSync(element.innerHTML);
      lookup.set(id, styles);
    }

    this.remove();
  }
}

customElements.define("shared-styles", SharedStyle);

The <shared-styles> element will have its connectedCallback() invoked by the browser as it streams the HTML into the DSD. At this point, our element will read its own style-id attribute and use it to look up the styles in its cache.

If the cache already has an entry for the id:

1. The element adds the styles to the adoptedStyleSheets collection of the containing shadow root.
2. Then, the <shared-styles> removes itself from the DSD.

If the styles are not present in the cache:

1. First, the element constructs a CSSStyleSheet instance.
2. Second, it locates the <style> element inside the containing DSD using the id.
3. Third, the <style> element’s contents are used to provide the styles for the CSSStyleSheet.
4. Fourth, the style sheet is cached.
5. And finally, the <shared-styles> element removes itself from the DSD.

There are a couple other details of the implementation worth pointing out:

• We use this.getRootNode() to find the shadow root that the <shared-styles> element is inside of. If it’s not inside of a shadow root, this API will return the document, which also has an adoptedStyleSheets collection.
• If it is the first time <shared-styles> is seeing a particular style-id, it doesn’t need to push the styles into the adoptedStyleSheets of the root because an in-line <style> element is already present, fulfilling the same purpose.

Now that we have the client side of our protocol implemented, we need a way for the server to generate this code. This is the role of the {{{shared-styles}}} Handlebars HTML helper that we’ve been using. Let’s look at the implementation of that:

// excerpted from the server handlebars configuration
helpers: {
  "shared-styles": function(src, options) {
    const context = getCurrentContext();
    const stylesAlreadySent = context.get(src);
    let html = "";

    if (!stylesAlreadySent) {
      const styles = loadStyleContent(src);
      context.set(src, true)
      html = `<style id="${src}">${styles}</style>`;
    }

    return html + `<shared-styles style-id="${src}"></shared-styles>`;
  }
}

Whenever the shared-styles helper is used, it performs the following steps:

1. Get the current request context (more on this later).
2. Check the request context to see if the style source has already been emitted during this request.
3. If it has previously been requested, return only the HTML for the <shared-styles> element, with the src as the style-id.
4. If it has NOT previously been requested, load the CSS and emit it into a <style> element, following that with the <shared-styles> element, both set up with the same style-id.

This simple HTML helper lets us track requests for the same styles across the entire request, emitting the correct HTML depending on the current request state.

The last requirement is to track the request context across controllers, views, and async functions, when we wouldn’t otherwise have access to it. For this, we’re going to use the Node.js async_hooks module. However, once standardization is complete, AsyncContext will be an official part of JavaScript, and the best approach for this piece of the puzzle.

Here’s how we leverage async_hooks:

import { AsyncLocalStorage } from "async_hooks";

const als = new AsyncLocalStorage();

export const getCurrentContext = () => als.getStore();
export const runInNewContext = (callback) => als.run(new Map(), callback);
export const contextMiddleware = (req, res, next) => runInNewContext(next);

The AsyncLocalStorage class enables us to provide state, in this case a Map, that is available to anything that runs within a callback. In the above code, we create a simple Express middleware function that ensures that all request handlers are run within the context, and receive a unique per-request instance of the Map. This can then be accessed with our getCurrentContext() helper at any time within the HTTP request. As a result, our shared-styles Handlebars HTML helper is able to track what styles it has already sent to the client within a given request, even though it doesn’t have direct access to Express’s request objects.

With both the server and client pieces in place, we can now share styles across components, without duplication, always ensuring that exactly one copy of the needed CSS is provided to the browser for a given request.

Tricks of the Trade: Handle Common Behavior with HTMX

If you’ve built a few web sites/apps in your life, it’s likely you’ve noticed many of them share a core set of needs. For example, making basic HTTP requests, changing out DOM nodes, history/navigation, etc. HTMX is a small JavaScript library that provides a declarative mechanism for attaching many common behaviors to HTML elements, without the need to write custom JavaScript code. It fits great with Web Components, and is an especially good companion when focusing on server rendering.

In our demo application, we use HTMX to AJAX in the film details whenever an item in the <film-list> is clicked. To see how that’s setup, let’s look again at the HTML for the <film-list> Web Component:

<film-list>
  <template shadowrootmode="open">
    {{{shared-styles "./film-list.css"}}}

    <ul hx-boost="true" hx-target="global #film-detail">
      {{#each films}}
        <li>
          <a href="/films/{{id}}">
            {{>film-card film=.}}
          </a>
        </li>
      {{/each}}
    </ul>
  </template>
</film-list>

HTMX can be instantly identified by its hx- prefixed attributes, which add the common behaviors I mentioned above. In this example, we use hx-boost to tell HTMX that any child <a> should have its href dynamically fetched from the server. We then use hx-target to tell HTMX where we want it to put the HTML that the server responds with. The global modifier tells HTMX to look in the main document, rather than the Shadow DOM. So, HTMX will execute the query selector “#film-detail” in document scope. Once located, the HTML returned from the server will be pushed into that element.

This is a simple but common need in web sites, one that HTMX makes easy to accomplish, and can be fully specified in our server HTML with no need to worry about custom JavaScript. HTMX provides a robust library of behaviors for all sorts of scenarios. Definitely, check it out.

Before we move on, it’s important to note that there are a couple of tricks to getting the above HTMX markup working with Web Components. So, let’s go over those quickly.

First, by default, HTMX searches the global document for its hx- attributes. Because our Web Components are using Shadow DOM, it won’t find them. That’s no problem, all we need to do is call htmx.process(shadowRoot) to enable that (more on where we hook this up later).

Second, when HTMX performs an AJAX and processes the HTML to insert into the DOM, it uses some old browser APIs which don’t handle DSD. I am hopeful that HTMX will soon be updated to use the latest standards, but in the meantime, we can solve this very easily with the following steps:

1. Use a MutationObserver to watch the DOM for any changes that HTMX makes by adding nodes.
2. Any time an element is added, process the DSD ourselves by turning templates into shadow roots.

This can be accomplished with a small amount of code as follows:

function attachShadowRoots(root) {
  root.querySelectorAll("template[shadowrootmode]").forEach(template => {
    const mode = template.getAttribute("shadowrootmode");
    const shadowRoot = template.parentNode.attachShadow({ mode });
    shadowRoot.appendChild(template.content);
    template.remove();
    attachShadowRoots(shadowRoot);
  });
}

// For even more advanced techniques, see Devel without a Causes's
// excellent post on streaming fragments:
// https://blog.dwac.dev/posts/streamable-html-fragments/
new MutationObserver((records) => {
  for (const record of records) {
    for (const node of record.addedNodes) {
      if (node instanceof HTMLElement) {
        attachShadowRoots(node);
      }
    }
  }
}).observe(document, { childList: true, subtree: true });

Finally, HTMX has a very particular way that it manages history. It converts the previous page into an HTML string and stores it in local storage before navigating. Then, when the user navigates back, it retrieves the string, parses it, and pushes it back into the DOM.

This approach is the default, and is often not sufficient for many apps, so HTMX provides various configuration hooks to turn it off or customize it, which is exactly what we need to do. Otherwise, HTMX won’t handle our DSD correctly. Here are the steps we need to take:

1. Whenever we navigate to a new page, we clear out HTMX’s history cache by calling localStorage.removeItem('htmx-history-cache').
2. We then instruct HTMX to refresh the page whenever it navigates and can’t find an entry in its cache. To configure that we set htmx.config.refreshOnHistoryMiss = true; at startup.

That’s it! Now we can use any HTMX behavior in our Shadow DOM, handle history/navigation, and ensure AJAX’d server HTML renders its DSD correctly.

Tricks of the Trade: Handle Custom Behavior with Web Component Islands

While HTMX can handle many common behavior scenarios, we often still need a little bit of custom JavaScript. In fact, minimally we need the custom JavaScript that enables HTMX for Shadow DOM.

Thanks to our use of custom elements, this is extremely simple. Any time we want to add custom behavior to a component, we simply create a class, register the tag name, and add any JS we want. For example, here’s how we could write a bit of code to enable HTMX across an arbitrary set of custom elements.

function defineHTMXComponent(tag) {
  customElements.define(tag, class {
    connectedCallback() {
      htmx.process(this.shadowRoot);
    }
  });
}

With that tiny bit of code, we can do something like this:

["film-list" /* other tags here */ ].forEach(defineHTMXComponent);

Now, in the case of our <film-list>, we don’t want to do this because we want to add other behavior. But, for any custom element where we only need HTMX enabled, we just add its tag to this array.

Turning our attention more fully to <film-list>, let’s look at how we can setup a little JavaScript “island” to ensure that whatever route we’re visiting gets styled properly:

export class FilmList extends HTMLElement {
  #links;

  connectedCallback() {
    this.#links = Array.from(this.shadowRoot.querySelectorAll("a"));

    htmx.process(this.shadowRoot);
    globalThis.addEventListener("htmx:pushedIntoHistory", this.#selectActiveLink);

    this.#selectActiveLink();
  }

  #selectActiveLink = () => {
    for (const link of this.#links) {
      if (link.href.endsWith(location.pathname)) {
        link.classList.add("active");
      } else {
        link.classList.remove("active");
      }
    }

    localStorage.removeItem('htmx-history-cache');
  }
}

customElements.define("film-list", FilmList);

Now we can see everything coming together. Here’s what happens:

• When we define the element, the browser will “upgrade” any custom elements it finds in the DOM that match our specified tag name of “film-list”, applying the behavior we’ve specified in our class.
• Next, the browser will call the connectedCallback(), which our code uses to enable HTMX on its shadow root, listen for HTMX history changes, and select the link that matches the current location. Whenever HTMX changes the history, the same active link code will also get run.
• Whenever we set the active link, we remember to clear out the HTMX history cache, so it doesn’t store HTML in local storage.

And that’s all the custom JavaScript in the entire app. Using standard custom elements, we’re able to define small islands of JavaScript that apply the custom behavior we need only where we need it. Common behavior is handled by HTMX, and many components don’t need JavaScript at all.

Wrapping Up

Hopefully, you’ve been able to see how straight forward it can be to server render Web Components with DSD, leverage common behaviors declaratively, and incrementally add custom JavaScript “islands” as your application evolves. The steps are essentially:

1. Use your server framework’s view engine to create partials for each Web Component.
2. Choose a unique custom tag as the root of your partial and declare a template using shadowrootmode if you want to enable DSD. Reminder: You don’t have to use Shadow DOM. Simply having the custom tag enables custom element islands. You only need Shadow DOM if you want HTML and CSS encapsulation and composition.
3. Use your server framework’s HTML helper mechanism to deliver shared styles for your DSD component.
4. Leverage HTMX attributes in server HTML as needed for common behaviors, being sure to register the tag to enable HTMX in Shadow DOM.
5. When custom code is needed, simply define a custom element that matches your tag name to enable your JavaScript island.

We don’t need to adopt complex JavaScript frameworks to take advantage of modern browser features. Leveraging the patterns present in any mature MVC server framework, we can evolve our codebases to use Web Components, DSD, and Islands. This enables us to be agile in our development, incrementally adopting and evolving our applications in response to what matters most: our customers.

Don’t forget to check out the demo on GitHub and if you want to dig deeper into Web Components and related web standards, have a look at my Web Component Engineering course.

Cheers!