Indexing your Posts to JSON

This component consists of two main sub-components. A build-time component that builds a searchable index (in JSON), and a client-side component that uses the index to find relevant entries.

Create a JSON Index at build-time

I created a javascript module that only works “server-side”, or better termed, build-time. This is because it uses the node.js module fs which is not available client-side.

It has a method addEntry(entry) that is used to collect post information. Note that this index works with my frontmatter definition for my posts, you can see the structure later in this post.

The other method is save(path) that will serialize the index to a file.

//-- /src/components/search/Indexer.js
import fs from 'fs';
let Indexer = {
  entries: [],
  addEntry: function (entry) {
    let isProd = import.meta.env.PROD;
    if (isProd) {
      this.entries.push(entry);
    }
  },
  save: function (path) {
    let isProd = import.meta.env.PROD;
    if (isProd) {
      fs.writeFileSync(
        path, 
        JSON.stringify(
          { entries: this.entries }, false, 2
        )
      );
    }
  }
};
export default Indexer;

Another thing that makes this Indexer module only work at “build-time” is the usage of true. The module is looking if it is running in DEV (the dev server) or PROD (the build-time).

So the index file is only really created when building your site.

Using the Indexer

In my astro page where I list my post entries, I added the following:

//-- /src/pages/blog/index.astro
--- // frontmatter
import BaseLayout from './../../layouts/BaseLayoutNoContainer.astro';
import Indexer from './../../components/search/Indexer.js';
const posts = Astro.fetchContent('./*.md');
// filter posts based on existing frontmatter.published
let filtered = posts.filter((post) => {
  if (!post.published) return false;
  return post.title;
});
// sort filtered posts by published desc
filtered.sort((a, b) => {
  if (a.published > b.published) return -1;
  return 1;
});
// add entries to the search index, grabbing data from frontmatter
filtered.forEach((post) => {
  let entry = {
    title: post.title,
    summary: post.summary,
    tags: '',
    published: post.published,
    image: post.og.basic.image,
    slug: post.url
  };
  if (post.tags) entry.tags = post.tags.join(' ');
  Indexer.addEntry(entry);
});
// serialize to JSON file in /public/data
Indexer.save('./public/data/searchIndex.json');
--- // markup excluded

This will create the following JSON file:

{
  "entries": [
    {
      "title": "Table of Contents Component",
      "summary": "Builds a table of contents from the markdown headers",
      "tags": "astro markdown",
      "published": "2022-02-22",
      "image": "https://cdn.ixmage.com/...",
      "slug": "/blog/table-of-contents"
    },
    {
      "title": "Free usable cloud Elasticsearch",
      "summary": "Playground for quick and dirty Proofs of Concept",
      "tags": "elasticsearch free",
      "published": "2022-02-21",
      "image": "https://cdn.discordapp.com/...png",
      "slug": "/blog/es-bonsai"
    },
    ... # more entries
  ]
}

And this is the data that will be “searched” on the client-side.

The frontmatter for my posts looks something like this:

layout: ./../../layouts/PostLayout.astro
title: 
summary: 
published: '2001-01-01'
tags: []
og:
  site_name: Oma Chambers
  basic:
    title: 
    type: Article
    image: https://.../default-img.png
    description:

The astro component

The astro component consists of a text input-field, a search button, and a div where the search results will be displayed.

I called the component SearchLocalIndex.astro, as I plan to build another one that will use an index in the cloud, but that is for a later post.

Let’s inspect the code in 4 sections, frontmatter+, markup, javascript, and style. Combining these 4 sections together into a file completes the SearchLocalIndex.astro file.

Frontmatter+

---
import idx_data from './../../public/data/searchIndex.json';
---

Not much here. I am importing the JSON index into a build-time variable idx_data.

Markup

<div class="search-component mt-4 mb-4" style="background:#f0f0f0; padding:10px; border-radius:5px;">
  <input type="text" id="search-term" />
  <span class="btn btn-sm btn-primary" id="action-search">Search</span><br/>
  <small class="text-muted">This search component looks in a post's title, summary, tags</small>
  <div id="search-results"></div>
</div>

So here are the text input, the search button, and an empty div for results.

All of these elements have their respective ids that will be referenced in the next section because Vanilla Javascript.

Javascript

<script type="text/javascript" define:vars={{idx_data}}>
  // grab what the user typed into the text input
  function getSearchTerm() {
    return document.getElementById('search-term').value.toLowerCase();
  };
  // the simple search logic
  function search() {
    var term = getSearchTerm();
    var results = []; //<-- collect results here
    // iterate the index entries
    idx_data.entries.forEach((entry) => {
      // look for a hit in the entry title
      //-- worth 1 point
      if (entry.title.toLowerCase().indexOf(term) >= 0) {
          if (!entry.score) {
            entry.score = 1;
            results.push(entry);
          } else {
            entry.score += 1;
          }
      }
      // look for a hit in the entry summary
      //-- worth 1 point
      if (entry.summary.toLowerCase().indexOf(term) >= 0) {
          if (!entry.score) {
            entry.score = 1;
            results.push(entry);
          } else {
            entry.score += 1;
          }
      }
      // look for a hit in the entry tags
      //-- worth 1 point
      if (entry.tags.toLowerCase().indexOf(term) >= 0) {
          if (!entry.score) {
            entry.score = 1;
            results.push(entry);
          } else {
            entry.score += 1;
          }
      }
    });
    // sort the hits based on score desc
    results = results.sort((a, b) => { if (a.score < b.score) return 1; return -1; });
    var markup = '';
    // build the whole html for each result
    results.forEach((item) => {
      markup += '<div class="search-item">';
      markup += '<div class="float-end" style="margin:-10px -10px 10px 10px;"><img src="'+item.image+'" height="72" /></div>';
      markup += '<a href="'+item.slug+'"><b>'+item.title+'</b></a> <span class="text-muted">('+item.score+')</span><br/>';
      markup += '<span>'+item.summary+'</span><br/>';
      markup += '</div>';
    });
    // put the html markup into the empty results div
    document.getElementById('search-results').innerHTML = markup;
    // reset entry scores for next search
    idx_data.entries.forEach((entry) => {
      entry.score = 0;
    });
  };
  // attach click event on button
  document.getElementById('action-search').addEventListener('click', function () {
    search();
  });
  // attach keyup event when user types, search is activated after 3 characters
  document.getElementById('search-term').addEventListener('keyup', function () {
    var term = getSearchTerm();
    if (term.length > 3) search();
  });
</script>

Remember in the frontmatter we imported the index data to idx_data (?)

Well, here I am using define:vars={{idx_data}} to bring that whole JSON into our javascript section. This data is used in the search() function.

Hopefully the /* commentary */ in the code helps in the understanding of what the component does.

Style

<style global>
.search-item { display:inline-block; padding:10px; border:1px solid #abd; border-radius:5px; margin:5px; overflow:auto; }
</style>

Some miscellaneous styling.

Conclusion

The working demo: Type “astro” w/o quotes to see some results.

Search
This search component looks in a post's title, summary, tags

Including the component here

// at the top of my MD file frontmatter:
---
setup: |
  import SearchLocalIndex from './../../components/SearchLocalIndex.astro'
...
---
// and then somewhere in the MD
<SearchLocalIndex />

And the same approach to use the component in an .astro file.

You can also see this component in action at the blog’s index page.

What next

I already started on a v2 where the index file will be posted to a cloud elasticsearch and searched from there.

Reactions:

Back to Post List