Code Browser with Gatsby

Published on December 23, 2023

At first, I was content with the ability to insert code snippets into these articles. gatsby-remark-prismjs makes that process pretty straightforward.

But what if an example is more complex than that? Web developers typically turn to tools like CodeSandbox for this purpose. However, the downside there is that the example code needs to be versioned using e.g. CodeSandbox and cannot be stored alongside our source code.

We'll be taking a fairly deep dive into Gatsby to solve this problem. The general workflow with Gatsby is to use source plugins to add nodes to the object store, then run transformers over them to do useful things (e.g. generating thumbnails or parsing JSON), then finally to query the results in your page components.

First Step: Sourcing and Transforming

There is already a gatsby-source-filesystem to handle the common case of sourcing arbitrary files, but unless they are transformed we cannot access the file contents directly using this plugin alone.

Enter gatsby-transformer-plaintext, which does exactly what you might think: transforms sourced files with a MIME type of text/plain by adding the file's contents into the Gatsby object store.

Unfortunately, this plugin only works for text files and is not configurable. So I forked it into gatsby-transformer-source-code. The new plugin allows you to specify the MIME types you want to process, then puts the file contents in a SourceCode node attached to each File node in the Gatsby store.

Now we can configure Gatsby like this:

module.exports = {
  plugins: [
    {
      resolve: 'gatsby-source-filesystem',
      options: {
        name: 'snippets',
        path: `${__dirname}/src/snippets`
      }
    },
    {
      resolve: 'gatsby-transformer-source-code',
      options: {
        mimeTypes: ['application/javascript', 'text/jsx', 'text/x-scss']
      }
    }
  ]
};

This lays the foundation for what we will be building. Now the source code files (and their contents) are in Gatsby's object store, waiting for us to query them.

Second Step: Provide Data When Creating Pages

Because of Gatsby's design, we cannot use the useStaticQuery hook to fetch our code snippet data - it does not support passing/referencing variables. We need to fetch the content dynamically based on URL, so we have to provide some metadata to the page renderer which will then be fed back into a GraphQL query to provide the data at the page component level.

This blog generates a page for an article, each of which is contained in an MDX file. Here is a summarized version of that code from gatsby-node.js:

exports.createPages = async ({ actions, graphql, reporter }) => {
  const mdxComponent = resolve('src/components/mdxArticle.js');
  const { createPage } = actions;
  const result = await graphql(`
    query {
      allMdx {
        nodes {
          internal {
            contentFilePath
          }
          frontmatter {
            path
          }
        }
      }
    }
  `);
  result.data.allMdx.nodes.forEach((node) => {
    const {
      frontmatter: { path },
      internal: { contentFilePath }
    } = node;

    if (!path) {
      reporter.warn(
        `Did not find a path in the frontmatter of ${contentFilePath}`
      );
      return;
    }

    createPage({
      // this "URL" is for gatsby-plugin-mdx
      component: `${mdxComponent}?__contentFilePath=${contentFilePath}`,
      path,
      context: {
        // this variable will be accessible in the pageQuery
        pathGlob: `${path.substring(1)}/**/*`
      }
    });
  });
};

This queries the MDX nodes provided by gatsby-plugin-mdx and creates a page for each, critically passing pathGlob along as a page context variable. Without this, we would not be able to filter for "snippets that relate to the page in question."

Third Step: Querying The Data

Now, in the src/components/mdxArticle.js file, we will use the following page query:

query ($pathGlob: String!) {
  allFile(
    filter: {
      sourceInstanceName: { eq: "snippets" }
      relativeDirectory: { glob: $pathGlob }
    }
  ) {
    nodes {
      path: relativePath
      code: childSourceCode {
        ... on SourceCode {
          content
        }
      }
    }
  }
}

Each article will now load any snippets that live under a path matching the article URL. For example, if an article has the URL /react/some-article-name, then ./src/snippets/react/some-article-name will be searched for code.

Now we have the data at the page level. However, this is an MDX page - the actual markdown content is passed as the children prop to the page component. Because of this, we will have to leverage a React context to span the gap.

Fourth Step: Integrating With MDX

import CodeBrowser from 'src/components/codeBrowser';

export default function MdxArticle({ data, children }) {
  const {
    mdx: { frontmatter },
    allFile: { nodes: snippets }
  } = data;

  return (
    <ArticleContainer {...frontmatter}>
      <MDXProvider components={{ CodeBrowser }}>
        <SnippetProvider
          snippets={snippets.map((snippet) => ({
            ...snippet,
            path: snippet.path.replace(`${frontmatter.path.substring(1)}/`, '')
          }))}
        >
          {children}
        </SnippetProvider>
      </MDXProvider>
    </ArticleContainer>
  );
}

We use the MDXProvider that gatsby-plugin-mdx offers to add in a custom component called <CodeBrowser />. Then we pass the snippet data we received from GraphQL to the SnippetProvider, which simply copies them into an otherwise empty context.

import Prism from 'prismjs';
import { uniq } from 'lodash-es';
import PropTypes from 'prop-types';
import { useState, useEffect, useMemo } from 'react';
import { Row, Tab, Col, Card, Container } from 'react-bootstrap';

import DirectoryTree from 'components/directoryTree';
import useSnippets from 'hooks/useSnippets';

const getDirName = (path) => path.substring(0, path.lastIndexOf('/'));

const getExtension = (path) => path.substring(path.lastIndexOf('.') + 1);

export default function CodeBrowser({ id }) {
  const { snippets } = useSnippets();
  const [activeDocument, setActiveDocument] = useState(null);

  useEffect(() => {
    const activeDoc = snippets.find(
      (doc) => doc.path.replace(`${id}/`, '') === activeDocument
    );

    if (activeDoc) {
      Prism.highlightElement(
        document.getElementById(`doc-${activeDoc.path.replace(`${id}/`, '')}`)
      );
    }
  }, [activeDocument]);

  const snippetTree = useMemo(() => {
    const result = {
      path: '.',
      files: [],
      children: []
    };
    const strippedSnippets = snippets.map((doc) => ({
      ...doc,
      path: doc.path.replace(`${id}/`, '')
    }));
    const directories = uniq(
      strippedSnippets.map((doc) => getDirName(doc.path))
    );

    for (const dir of directories) {
      const segments = dir.split('/');

      if (!segments.join('')) {
        continue;
      }

      for (let i = 0; i < segments.length; i++) {
        let node = result;
        const path = segments.slice(0, i + 1);

        for (const segment of path) {
          const nextNode = node.children.find((dir) => dir.path === segment);

          if (nextNode) {
            node = nextNode;
          } else {
            node.children.push({
              path: segment,
              files: [],
              children: []
            });
          }
        }
      }
    }

    for (const doc of strippedSnippets) {
      let node = result;
      const segments = getDirName(doc.path).split('/');

      for (const segment of segments) {
        const nextNode = node.children.find((dir) => dir.path === segment);

        if (nextNode) {
          node = nextNode;
        }
      }

      node.files.push(doc);
    }

    return result;
  }, [snippets]);

  return (
    <Tab.Container
      id={id}
      activeKey={activeDocument}
      onSelect={(path) => setActiveDocument(path)}
    >
      <Row className="g-2">
        <Col xs={3}>
          <Card body>
            <Card.Title>Files</Card.Title>
            <DirectoryTree activeDocument={activeDocument} node={snippetTree} />
          </Card>
        </Col>
        <Col xs={9}>
          <Tab.Content>
            {snippets.map((doc) => (
              <Tab.Pane
                eventKey={doc.path.replace(`${id}/`, '')}
                key={doc.path}
              >
                <Container fluid>
                  <Row className="g-0">
                    <Col xs={12} className="bg-primary rounded-top">
                      <h5 className="my-3 ms-3">
                        {doc.path.replace(`${id}/`, '')}
                      </h5>
                    </Col>
                  </Row>
                  <Row>
                    <Col xs={12}>
                      <pre
                        id={`doc-${doc.path.replace(`${id}/`, '')}`}
                        className={`language-${getExtension(
                          doc.path
                        )} mt-0 rounded-bottom`}
                        style={{ maxHeight: 650, overflowY: 'auto' }}
                      >
                        {doc.code.content}
                      </pre>
                    </Col>
                  </Row>
                </Container>
              </Tab.Pane>
            ))}
          </Tab.Content>
        </Col>
      </Row>
    </Tab.Container>
  );
}

CodeBrowser.propTypes = {
  id: PropTypes.string.isRequired
};

This component renders a directory/file structure on the left with the selected source code on the right.

In our MDX file, we can now simply use the following JSX to embed our code sample.

<CodeBrowser id="example-one" />

Now we can show complete code samples with a file tree browser. Here's the code we just wrote above, showing itself:

Files

gatsby-config.js

module.exports = {
  plugins: [
    {
      resolve: 'gatsby-source-filesystem',
      options: {
        name: 'articles',
        path: `${__dirname}/articles`
      }
    },
    {
      resolve: 'gatsby-source-filesystem',
      options: {
        name: 'snippets',
        path: `${__dirname}/snippets`
      }
    },
    {
      resolve: 'gatsby-transformer-source-code',
      options: {
        mimeTypes: ['application/javascript']
      }
    }
  ]
};

gatsby-node.js

const { resolve } = require('path');

exports.createPages = async ({ actions, graphql, reporter }) => {
  const mdxComponent = resolve('src/components/mdxArticle.js');
  const { createPage } = actions;
  const result = await graphql(`
    {
      allMdx {
        nodes {
          internal {
            contentFilePath
          }
          frontmatter {
            path
          }
        }
      }
    }
  `);

  if (result.errors) {
    reporter.panicOnBuild('Error while running GraphQL markdown query.');
    return;
  }

  let counter = 0;

  result.data.allMdx.nodes.forEach((node) => {
    const {
      frontmatter: { path },
      internal: { contentFilePath }
    } = node;

    if (!path) {
      reporter.warn(
        `Did not find a path in the frontmatter of ${contentFilePath}`
      );
      return;
    }

    counter++;
    createPage({
      component: `${mdxComponent}?__contentFilePath=${contentFilePath}`,
      path,
      context: {
        pathGlob: `${path.substring(1)}/**/*`
      }
    });
  });

  reporter.info(`Created ${counter} markdown pages!`);
};

exports.onCreateWebpackConfig = ({ actions }) => {
  actions.setWebpackConfig({
    resolve: {
      modules: [resolve(__dirname, 'src'), 'node_modules']
    }
  });
};

articles/demo.mdx

---
path: /demo
date: 2023-12-20
description: Demo
title: Demo
---

This is an example article.

<CodeBrowser id="first" />

src/components/articleContainer.jsx

import Prism from 'prismjs';
import { useEffect } from 'react';
import PropTypes from 'prop-types';
import { format, parseISO } from 'date-fns';
import { Container, Row, Col } from 'react-bootstrap';

import Layout from 'components/layout';

export default function ArticleContainer({
  title,
  description,
  date,
  children
}) {
  useEffect(() => {
    Prism.manual = true;
    Prism.highlightAll();
  }, []);

  return (
    <Layout title={title} description={description}>
      <Container>
        <Row>
          <Col md="12" className="mb-2">
            <h1>{title}</h1>
            {Boolean(date) && (
              <h4>Published on {format(parseISO(date), 'yyyy-MM-dd')}</h4>
            )}
          </Col>
          <hr />
          <Col xs={12}>{children}</Col>
        </Row>
      </Container>
    </Layout>
  );
}

ArticleContainer.propTypes = {
  title: PropTypes.string.isRequired,
  description: PropTypes.string,
  date: PropTypes.string,
  children: PropTypes.node.isRequired
};

src/components/codeBrowser.jsx

import Prism from 'prismjs';
import { uniq } from 'lodash-es';
import PropTypes from 'prop-types';
import { useState, useEffect, useMemo } from 'react';
import { Row, Tab, Col, Card, Container } from 'react-bootstrap';

import DirectoryTree from './directoryTree.jsx';
import useSnippets from '../hooks/useSnippets';

const getDirName = (path) => path.substring(0, path.lastIndexOf('/'));

const getExtension = (path) => path.substring(path.lastIndexOf('.') + 1);

export default function CodeBrowser({ id }) {
  const { snippets } = useSnippets();
  const [activeDocument, setActiveDocument] = useState(null);

  useEffect(() => {
    const activeDoc = snippets.find(
      (doc) => doc.path.replace(`${id}/`, '') === activeDocument
    );

    if (activeDoc) {
      Prism.highlightElement(
        document.getElementById(`doc-${activeDoc.path.replace(`${id}/`, '')}`)
      );
    }
  }, [activeDocument]);

  const snippetTree = useMemo(() => {
    const result = {
      path: '.',
      files: [],
      children: []
    };
    const strippedSnippets = snippets.map((doc) => ({
      ...doc,
      path: doc.path.replace(`${id}/`, '')
    }));
    const directories = uniq(
      strippedSnippets.map((doc) => getDirName(doc.path))
    );

    for (const dir of directories) {
      const segments = dir.split('/');

      if (!segments.join('')) {
        continue;
      }

      for (let i = 0; i < segments.length; i++) {
        let node = result;
        const path = segments.slice(0, i + 1);

        for (const segment of path) {
          const nextNode = node.children.find((dir) => dir.path === segment);

          if (nextNode) {
            node = nextNode;
          } else {
            node.children.push({
              path: segment,
              files: [],
              children: []
            });
          }
        }
      }
    }

    for (const doc of strippedSnippets) {
      let node = result;
      const segments = getDirName(doc.path).split('/');

      for (const segment of segments) {
        const nextNode = node.children.find((dir) => dir.path === segment);

        if (nextNode) {
          node = nextNode;
        }
      }

      node.files.push(doc);
    }

    return result;
  }, [snippets]);

  return (
    <Tab.Container
      id={id}
      activeKey={activeDocument}
      onSelect={(path) => setActiveDocument(path)}
    >
      <Row className="g-2">
        <Col xs={3}>
          <Card body>
            <Card.Title>Files</Card.Title>
            <DirectoryTree activeDocument={activeDocument} node={snippetTree} />
          </Card>
        </Col>
        <Col xs={9}>
          <Tab.Content>
            {snippets.map((doc) => (
              <Tab.Pane
                eventKey={doc.path.replace(`${id}/`, '')}
                key={doc.path}
              >
                <Container fluid>
                  <Row className="g-0">
                    <Col xs={12} className="bg-primary rounded-top">
                      <h5 className="my-3 ms-3">
                        {doc.path.replace(`${id}/`, '')}
                      </h5>
                    </Col>
                  </Row>
                  <Row>
                    <Col xs={12}>
                      <pre
                        id={`doc-${doc.path.replace(`${id}/`, '')}`}
                        className={`language-${getExtension(
                          doc.path
                        )} mt-0 rounded-bottom`}
                        style={{ maxHeight: 650, overflowY: 'auto' }}
                      >
                        {doc.code.content}
                      </pre>
                    </Col>
                  </Row>
                </Container>
              </Tab.Pane>
            ))}
          </Tab.Content>
        </Col>
      </Row>
    </Tab.Container>
  );
}

CodeBrowser.propTypes = {
  id: PropTypes.string.isRequired
};

src/components/directoryTree.jsx

import PropTypes from 'prop-types';
import { Fragment, useState } from 'react';
import { Row, Col, Nav } from 'react-bootstrap';
import { FontAwesomeIcon } from '@fortawesome/react-fontawesome';
import {
  faFolder,
  faCaretDown,
  faCaretRight,
  faFile
} from '@fortawesome/free-solid-svg-icons';

const getFileName = (path) => path.substring(path.lastIndexOf('/') + 1);

export default function DirectoryTree({ activeDocument, node, level = 0 }) {
  const { path, children, files } = node;
  const isFile = !Array.isArray(files);

  const [expanded, setExpanded] = useState(level === 0);

  return (
    <Fragment>
      <Row onClick={isFile ? null : () => setExpanded((prevVal) => !prevVal)}>
        <Col xs={1}>
          {!isFile && (
            <FontAwesomeIcon icon={expanded ? faCaretDown : faCaretRight} />
          )}
        </Col>
        <Col xs={11} style={level > 0 ? { paddingLeft: (level + 1) * 12 } : {}}>
          <Nav.Item className={activeDocument === path && 'text-primary'}>
            <Nav.Link eventKey={isFile ? path : null}>
              <FontAwesomeIcon
                icon={isFile ? faFile : faFolder}
                className="me-1"
              />{' '}
              {path ? getFileName(path) : '.'}
            </Nav.Link>
          </Nav.Item>
        </Col>
      </Row>
      {expanded &&
        [...(children || []), ...(files || [])].map((child) => (
          <DirectoryTree
            key={child.path}
            node={child}
            level={level + 1}
            activeDocument={activeDocument}
          />
        ))}
    </Fragment>
  );
}

DirectoryTree.propTypes = {
  activeDocument: PropTypes.string,
  node: PropTypes.object.isRequired,
  level: PropTypes.number
};

src/components/mdxArticle.jsx

// eslint-disable-next-line no-unused-vars
import React from 'react';
import { graphql } from 'gatsby';
import PropTypes from 'prop-types';
import { MDXProvider } from '@mdx-js/react';

import ArticleContainer from './articleContainer.jsx';
import CodeBrowser from './codeBrowser.jsx';
import SnippetProvider from './snippetProvider.jsx';

export default function MdxArticle({ data, children }) {
  const {
    mdx: { frontmatter },
    allFile: { nodes: snippets }
  } = data;

  return (
    <ArticleContainer {...frontmatter}>
      <MDXProvider components={{ CodeBrowser }}>
        <SnippetProvider
          snippets={snippets.map((snippet) => ({
            ...snippet,
            path: snippet.path.replace(`${frontmatter.path.substring(1)}/`, '')
          }))}
        >
          {children}
        </SnippetProvider>
      </MDXProvider>
    </ArticleContainer>
  );
}

MdxArticle.propTypes = {
  data: PropTypes.shape({
    mdx: PropTypes.shape({
      frontmatter: PropTypes.shape({
        path: PropTypes.string.isRequired,
        title: PropTypes.string.isRequired,
        description: PropTypes.string,
        date: PropTypes.string
      })
    }),
    allFile: PropTypes.shape({
      nodes: PropTypes.arrayOf(
        PropTypes.shape({
          path: PropTypes.string.isRequired,
          code: PropTypes.shape({
            content: PropTypes.string.isRequired
          })
        })
      )
    })
  }),
  children: PropTypes.node.isRequired
};

export const pageQuery = graphql`
  query ($path: String!, $pathGlob: String!) {
    mdx(frontmatter: { path: { eq: $path } }) {
      frontmatter {
        path
        title
        description
        date
      }
    }

    allFile(
      filter: {
        sourceInstanceName: { eq: "snippets" }
        relativeDirectory: { glob: $pathGlob }
      }
    ) {
      nodes {
        path: relativePath
        code: childSourceCode {
          ... on SourceCode {
            content
          }
        }
      }
    }
  }
`;

src/components/snippetProvider.jsx

import PropTypes from 'prop-types';

import { SnippetContext } from '../hooks/useSnippets';

export default function SnippetProvider({ snippets, children }) {
  return (
    <SnippetContext.Provider value={{ snippets }}>
      {children}
    </SnippetContext.Provider>
  );
}

SnippetProvider.propTypes = {
  snippets: PropTypes.arrayOf(PropTypes.object),
  children: PropTypes.node.isRequired
};

src/hooks/useSnippets.js

import { createContext, useContext } from 'react';

export const SnippetContext = createContext({
  snippets: []
});

export default function useSnippets() {
  return useContext(SnippetContext);
}

snippets/demo/first/index.js

export default {
  add: (a, b) => a + b
};