Hosting on IOOS GitHub Pages Workflow

Introduction

This document describes the deployment of the IOOS documentation web pages generated with a Hugo static page generator. The Hugo generator was picked out after some research and experiments mostly for the following reasons:

  • portability, i.e. it is a self-contained application that does not require installation;
  • no prerequisite installation of a tool/language like Ruby, Python, or Java;
  • direct processing of the Markdown content;
  • very satisfactory results with a bare minimal modification of the default themes and templates;
  • relatively simple workflow of the generated web site deployment onto GitHub Pages.

The following workflow description is mostly based on the Hosting on GitHub Pages tutorial, which has been updated to reflect IOOS specifics.

IOOS Documentation Repositories

The IOOS DMAC Documentation Strategy requires that all DMAC-related documentation is converted into Markdown, and stored in a specially structured GitHub repository. In general, the repository must contain a “doc” directory that holds all documents in Markdown, and a “README.md” file that briefly describes the repository:

   ioos/[some repository]/
    ├── doc/
    |    ├── DOC1.md
    |    ├── DOC2.md
    |    |    └── images/
    |    └── DOC3.md  
    └── README.md

In some cases the repository may have more complex structure, and contain not only Markdown files; for example, the sos-guidelines repository includes an additional SOS Templates section with a set of XML files:

   ioos/sos-guidelines/
    ├── doc/
    |    ├── testing/
    |    |    └── sos_test_list_github_notoc_summary.md
    |    └── wsdd/ 
    |         ├── sos_wsdd_github_notoc.md    
    |         └── sos_wsdd_github_toc.md
    ├── template/
    |    └── milestone 1.0/
    |          ├── OM-GetObservation.xml
    |          ├── SML-DescribeSensor-Network.xml
    |          ├── SML-DescribeSensor-Station.xml
    |          ├── SOS-GetCapabilities.xml
    |          ├── SWE-MultiStation-TimeSeries.xml
    |          ├── SWE-MultiStation-TimeSeries_QC.xml
    |          ├── SWE-SingleStation-SingleProperty-TimeSeries.xml
    |          ├── SWE-SingleStation-TimeSeriesProfile.xml
    |          └── SWE-SingleStation-TimeSeriesProfile_QC.xml
    └── README.md

Structure of IOOS Documentation Website

By default, Hugo expects the content of the site to be structured in some specific way, and uses that same structure to render the website.

The top level of a source directory will typically have the following elements:

   ioos/[some-repository-name]/
    ├── archtypes/
    ├── content/
    ├── layouts/
    ├── static/
    └── config file

The content directory by default holds the documents that are the actual content of the website (i.e. are rendered by Hugo), and sub-directories of the content directory form the sections. The real content holding directory may be changed with a contentdir value in the Hugo config file.

The static directory is used to hold the common website elements, such as Style Sheets, Java Scripts, and images, etc., which will be copied directly into the final site when rendered. Although Hugo does not require any specific structure for static directory, it is usually organized in the following way:

   static/
    ├── css/
    ├── js/
    └── images/

NOTES:

  1. Currently, Hugo can render only Markdown files, so all documents in the contentdir (e.g. ‘content’) must be in Markdown format for proper rendering. As the original documents may be in various formats, it is necessary to keep the contentdir separate from the original document directory (e.g. ‘doc’).
  2. All images and diagrams from the original documents that should appear on the rendered web pages must be copied into the static/images directory for the proper website rendering. The links to the images in the Markdown version of the document have to be modified accordingly.

Site Config File

The config file defines the framework of the website generated by Hugo. It defines site-wide settings (like the websites baseurl), and allows to modify to some extent the default website structure. For example, the sos-guidelines website has file config.toml with the following contents

baseurl = "http://abirger.github.io/sos-guidelines"
CanonifyUrls = true
contentdir = "content"
layoutdir = "layouts"
publishdir = "public"
languageCode = "en-us"
[params]
    Title = "IOOS SOS Guidelines"
    Tagline = "Docs, codes, scripts, templates"
[params.Github]
    Head = "master"
    Url = "https://github.com/ioos/sos-guidelines"

For detailed explanation of how to build the Hugo config file, and the exact meaning of the parameters and operators refer to the Configuration section of Hugo documentation.

HTML Templates

For IOOS DMAC documentation web pages, the Lanyon theme has been chosen because it has a sliding sidebar controlled by an unobtrusive clickable button, which allows using almost all available screen space for large documents like XML listings and such. The default theme was slightly modified in order to increase its expressiveness, add IOOS logo, and further expand the document viewing area.

NOTES:

  1. The Hugo requires that the home page should be defined as the index.html file in the layouts directory.
  2. To show the documents’ summary on the website home page, each document in contentdir directory has to start from the summary text separated from the rest of the document with the line that contains the <!–more–> comment.
  3. If the home page should contain just a single document (e.g. about the site), that document must have a url = "index.html" line in the front matter.
  4. The IOOS logo is in the ‘ioos_blue2.png’ file located in ‘static/images/’ directory.

For a detailed description of the Hugo themes, templates and layouts refer to the Hugo documentation.

Hugo Metadata in Content Files

To ensure a proper rendering of the website content, each document in the contentdir should include at the top a metadata section. That metadata section is called a front matter, and should be written in YAML, TOML or JSON format. The front matter tells Hugo some details of how to render the content file. For example, the following front matter is included in the SOS WSDD file:

+++
draft = false
title = "IOOS SOS v1.0 WSDD"
description = "IOOS SOS v1.0 Web Service Description"
type = "post"
sidebar = true
date = 2014-08-04T08:09:56Z
weight = "1"
+++

In the following example of the front matter, the line url = "index.html" instructs Hugo that this file should be rendered as a home page:

+++
weight = "0"
type = "post"
draft = false
sidebar = true
Title = "ABOUT: Biological Data Enrollment Process"
date = 2014-08-04T07:56:39Z
url = "index.html"
+++

For a detailed description of the metadata section supported formats and variables refer to the Hugo Front Matter documentation.

Content and Non-content Menu in Sidebar

By default, Hugo will put the document title into sidebar document menu. That behavior can be changed with the sidebar = false line in the front matter.

Apart from the content menu entries, the sidebar menu may contain entries that aren’t attached to any document in contentdir. This takes place in the sitewide config file. For example, to add the menu item that links to the video from a Webinar that is hosted somewhere else, the following lines must be added to the sitewide config file (name and url are mandatory while weight and identifier are optional):

[[menu.extramedia]]
    name = "Webinar: IOOS Biological Data Enrollment Process (video)"
    url = "https://mmancusa.webex.com/mmancusa/ldr.php?RCID=a08a97a20c0328828c0bc2f4e07f0fa2"
    weight = -110
    identifier = "video"

NOTE:

The original templates from Lanyon theme have been modified to allow adding items to the extramedia menu. Adding other non-content menus will require further modification of the .\layouts\partial\sidebar.html template.

For more detailed information on menus definition, refer to a corresponding part of Hugo documentation.

Workflow for IOOS Documentation Website deployment on GitHub

The Hugo IOOS DMAC Documentation website can be created from any GitHub documentation repository by following a few simple procedures below.

Create and Debug a Hugo Site

  1. Clone the repository from github.com/IOOS, or create the repository in the local directory.
  2. Copy the files from IOOS Hugo Skeleton into the local repository directory.
  3. Copy or move the Markdown files into ./content directory, and all images into .static/images directory.
  4. Edit the ./config.toml file as needed to fit the project. Set up baseurl = "http://ioos.github.io/repository-name".
  5. Add front matter to each Markdown document in the ./content directory, and edit documents as needed.
  6. Edit ./layouts/index.html template and other templates in ./layouts directory as needed to fit the project.
  7. Edit the ./README.md file as needed to fit the project.
  8. Run Hugo with options in the local repository root directory: hugo server --watch
  9. Repeat steps 5 to 7 as needed to produce the satisfactory looking website.

The command hugo server –watch starts a web server available at “http://localhost:1313/repository-name/”, generates website content in the ./public directory, watches source files for changes, recreates the web content as needed, and automatically reload any open pages in browser. After the website development and debugging is completed, the final version of the site should be deployed on GitHub Pages.

Deploy the Site to GitHub Pages

GitHub Pages will serve up a website for any project repository that has a branch called gh-pages with a valid index.html file at that branch’s root. For the organization/user repository (e.g. ioos.github.io), however, the GitHub Pages serve content of the master branch itself. That may make a deployment of the generated website to GitHub a convoluted process, and unnecessary clutter up the default and deploy repository branches.

A simple and visual deployment workflow based on the Hugo’s Hosting on GitHub Pages tutorial helps untangle the deployment process, and keep an adequate separation between the default and deploy branches. The workflow allows a direct push of the entire content of the generated website to the correct remote repository’s branch. It does not stipulate for the commit of generated files to the remote default branch, does not make superfluous commits or deploys when the generated files do not change, and allows keeping a linear history on the deploy branch.

The core element of the workflow is the Bash script deploy.sh that renders automatic the deployment process apart from generating the website with Hugo: it creates a deploy branch of the remote repository, and commits the website content to it. Although the script requires some initial configuration, the IOOS documentation GitHub Pages are so simple and highly similar that it can be used for each of them without any modification; each repository includes the pre-configured copy of the script.

With this script, the deployment process can be reduced to a few steps:

configuration

Edit the following variables within the script as needed to fit the project:

  • deploy_directory: root of the tree of files to deploy; must match the value of the publishdir in the site config file. For IOOS websites the value is always public.
  • deploy_branch: branch to commit files to and push to origin. For projects the value is gh-pages.
  • default_username, default_email: identity to use for git commits if none is set already. Usually no need to set up.
  • repo: repository to deploy to (must be readable and writable). If script runs in the repo root directory, the variable sets up automatically.

setup

Ensure configuration variables are correct in the script and run

              deploy.sh -s

Option -s or --setup makes the script perform one-time setup to prepare the repository for deployments: creates deploy_branch (i.e. gh-pages), initializes it with the contents of deploy_directory (i.e. public), and pushes it to remote repository.

run

Do the following every time you want to deploy:

  • check out the master branch or commit of the source (useful if somebody else might make changes to the source that you are not aware of). The script will use this commit to generate a message when it makes its own commit on the deploy branch.
  • run Hugo without options to generate the files in deploy_directory (i.e. publish).
  • make sure no uncommitted changes left in git’s index, otherwise the script will abort. Although it is OK to have uncommitted files in the work tree as the script does not touch the work tree, it is strongly recommended to commit and push all changes to the remote default branch before running the script.
  • make sure you are in the project’s root directory
  • run deploy.sh with or without any of the following options:
    • -v or --verbose echo commands as they are executed; it is recommended for debugging if something goes wrong;
    • -e or --allow-empty allow deployment of an empty public directory (by default, the script will abort if deploy_directory is empty).

NOTES:

  1. To allow running Bash scripts on Windows, some Bash shell software for Windows has to be installed. For example, Bash shell is included into a standard GitHub for windows installation package.
  2. The above procedure assumes that the script runs from the site <root> directory, i.e. the directory, where the content (or another directory defined as contentdir) and layout sub-directories are located).
  3. As GitHub Pages render the organization/user sites in a different way, the deploy_branch value for the ioos.github.io repository must be master instead of gh-pages (for the same reason, a new-website branch has been set up as the default working branch instead of master). To avoid confusion, the name of the pre-configured script for the ioos.github.io site has changed to deploy-io.sh.