Dada Portal

Dada Portal
Login

I build thomaslevine.com with this software.

I store this in a separate repository from the website articles in order to avoid cluttering the other repository's history, as I am likely to switch the software yet again within a few years. I try to put files in the other repository if I think I will keep them through website software changes, and I put them in here if I don't think I will keep them. The main exception to this rule is the CSS styles, because I found it easier to put them in here.

This software will probably work for you only if you are okay with having a website that looks a lot like mine. Otherwise, you may still find the implementation to be interesting.

Notable features

Dadaportal renders a website as a directory of files that can be copied to a server. I have designed dadaportal specifically for my own needs.

Here are fundamental website compilation features.

These following features rely on the website being served by an appropriately configured Apache server.

The build process is designed to be very fast.

Dadaportal also provides some utilities for reviewing the content of a website.

The remaining features are included in the same package because it is convenient, but I think they would make more sense in a separate package because they are very specific to my particular website.

Install

I intend for websites to be built in one environment and for them to be served from another. In the present document I expect that you are always logged in to the building environment; when I direct you to run commands in the serving environment, they are specified as commands run remotely from the building environment.

If this is your first time using dadaportal, just set up the building environment, and think about about the serving environment later.

Building environment

You must install Python3. The remaining dependencies are Python packages.

Download the source code, and navigate to the source code directory.

wget https://thomaslevine.com/scm/dadaportal/tarball/dadaportal.tar.gz
tar xzf dadaportal.tar.gz
cd dadaportal

Some of the Python dependencies are annoying to build; consider installing PIL and lxml from binary packages rather than compiling them from source.

Install. With pip, you can do this.

pip3 install .

Alternatively, you can use a different package manager. This is what I do with OpenBSD pkg.

pkg_add py3-{lxml,jinja2,docutils,markdown}

Without either, you can install the Python packages listed in setup.py, and then you can run, for example, python3 setup.py install.

Serving environment

To use all of the features, your web server must have Apache and fossil and it must recognize .htaccess files. But you might not care about all of the features, so other configurations might be just fine for you.

If you don't have fossil, you can generate an installation script that compiles fossil on the server, though you'll need a C compiler on the server. Do this after you have configured the website in the building environment; when you are ready, look at the directions for NearlyFreeSpeech and Super Dimensional Fortress.

Starting a website

A website is specified by a configuration file and by a few directories and files referenced by the configuration file.

Write a configuration file.

A configuration file is a list of field settings. Each line is a setting. The first colon or equal sign on the line separates the field name from its value. Empty lines and lines starting with "#" are ignored.

The following fields are allowed.

root-directory
Local root directory for source files, relative the directory containing the configuration file.
root-address
Web address corresponding to the "root-directory"
fossil-route
Web address in which to put the fossil server, relative "root-address"
fossil-repositories
Directory in which to the fossil server should expect fossil repositories
fossil-executable
Shebang line for the fossil executable
fossil-debug
File to which fossil should write debugging information
redirects
File containing legacy redirects, specified as a CSV file with two columns (source and target) and no header
extra-link-checks
This apparently does something, but I think it is redundant and obselete.
404
File containing the 404 page, in any of the available page formats
output
Local root directory for output files, relative the directory containing the configuration file.

I personally have two configuration files, one for local testing and one for publication on NearlyFreeSpeech. This is my local.conf,

redirects: conf/redirects.csv

fossil-route: scm
fossil-executable: /usr/bin/env fossil
fossil-repositories: /home/tlevine/clear/fossil
fossil-debug: /tmp/fossil.log

root-directory: src
root-address: http://localhost/~tlevine

404: conf/404.md
extra-link-checks: conf/links.csv
output: /home/tlevine/public_html

and this is my nfsn.conf.

redirects: conf/redirects.csv

fossil-route: scm
fossil-executable: /home/protected/fossil
fossil-repositories: /home/protected/r

root-directory: src
root-address: https://thomaslevine.com

404: conf/404.md
extra-link-checks: conf/links.csv
output: out

The configure command can write the configuration file for you.

Write a page.

A page that someone looks at on the website corresponds to a directory that you edit in the website source directory. Put the main text of the website in one of the following files, formatted to match the file extension.

When you build the website, dadaportal will render the file as HTML and put it in the base HTML page template. Aside from compiling the source file, dadaportal will include your file verbatim. That is, you can't do any templating here. I had that this sort of templating in previous website softwares, and I didn't like it because I had to change the template references every time I changed the software.

Metadata section

The page can contain metadata. It should be specified as a configuration file with the following field names, placed at the top of the index file and separated from the rest of the file by a line containing only "-" characters.

title
Title used in HTML metadata
description
Description used in HTML metadata
Anything starting with "x-"
This is ignored, but I allow it because I had a lot of metadata fields before
centered
If this field is present, the article text will be centered rather than left-aligned. Pages with lots of pictures sometimes look better this way.
nobread
If this field is present, no header nor footer will appear on the page. I use this for the home page and the 404 error page.

Here's an example from my website.

$ head src/index.md
title: Thomas Levine
description: Dada
centered:
nobread:
------

* [Data Music](!/data-music/)
* [Government open data](!/open-data/)
* [Computing](!/computing/)
* [Software products](!/software/)

The page directory is otherwise allowed to contain any other sort of file. You could include videos, images, and other text files, and you could refer to these in the index file. With one exception, subdirectories are directories, so they are considered pages. The exception is directories named "ignore"; these are ignored.

The "ignore" directory

The "ignore" directory is my compromise to allow for the inclusion of large files without making file transfers very slow and without using lots of space on the web server. If I want to generate a file based on a large source file, I put the source file in an "ignore" directory, and I generate the other file outside the "ignore" directory. For example, here is how I generate a short, small video from a long one-hour stream (in Makefile format).

# https://thomaslevine.com/scm/thomaslevine.com/artifact/13723949f2618096

hopping-cities.webm: hopping-cities.mkv
    ffmpeg -i hopping-cities.mkv hopping-cities.webm

presentation.webm: ignore/presentation.mp4
    ffmpeg -i ignore/presentation.mp4 presentation.webm
presentation.mp4: ignore/stream.flv
    ffmpeg -i ignore/stream.flv -ss 01:26:36 -t 3:50 ignore/presentation.mp4
ignore/stream.flv:
    youtube-dl -o ignore/stream.flv -f 0 'http://www.ustream.tv/recorded/91035940'
    touch ignore/stream.flv

The "ignore" directory is in the same tree, so it is backed up with the other files, but dadaportal does not copy it to the output directory, so I do not publish it to the web server.

How Dadaportal determines the title and description of a page

Dadaportal gets page titles from one of three sources.

  1. The title field in the metadata of the page.
  2. The first h1 tag in the body of a page.
  3. The slug of the web page in title case and with hyphens replaced by spaces

It uses the first of these sources that is available.

Dadaportal gets page descriptions from one of two sources.

  1. The description field in the metadata of the page.
  2. The first p tag in the body of a page.

It uses the first of these sources that is available. If neither is available, it leaves the description empty.

Publication commands

Once you have written the configuration file, you can manage the dada portal with the commands below. Here I present the main ideas of the commands. Each command has options that I have not discussed below, so see the command-line help for details.

Build the website

Compile the website like so.

dadaportal build blahblah.conf

The configuration file specifies the location of your source documents and the location where the output will go. You should configure Apache to serve that location and to evaluate the .htaccess file.

Without apache, you could instead serve the website locally like this.

cd output
python3 -m http.server

Check links

Web pages contain links to other pages. To ensure that the links are good, I run this command.

dadaportal check local.conf

It checks for the following sorts of problems.

Then I go through the various articles and correct all the problems.

Utilities

The utilities below do not depend on the configuration file. They are helpful when I am editing articles on my website and when I am setting up hosting for my website. As with the main commands above, I present only the main ideas of the utilities, and you need to look at the command-line help for details.

Write a configuration file.

Fill out all the fields in a configuration file.

dadaportal skeleton configure blahblah.conf

Write a directory listing.

If I have multiple pages under one directory, I usually write an index file linking to them. Consider, for example, the following files.

$ ls ./src/!/favorites/
a-picture.jpg
foods/
index.md
movies/
thomasville/
words/

The index file is index.md, and contains links to the other files. The listing tool will get this file started. Suppose that index.rst does not exist; I could create it like so.

$ dadaportal skeleton listing ./src/!/favorites/
.. image:: a-picture.jpg
    :alt: A Picture

`Foods <foods/>`_
`Movies <movies/>`_
`Thomasville <thomasville/>`_
`Words <words/>`_

Note that the tool ignores index.md. I made the tool do this because I do not think the index file should go in the listing.

I purposely do not run this automatically inside the build command, for the following reasons.

  1. I prefer to have the flexibility to organize each listing very differently.
  2. I have to open the listed files to get some of the information that I need to compose the listings. The main thing is titles. The build command would thus have to open multiple files in order to compose the listing, and this would make the build step slower.
  3. I could have a system where the an automated listing is produced only when the manual listing is not available, but I didn't see any benefit in such a system. The check command tells me when listings are missing, and it is easy enough to create the listings as I need them.

Set permissions to protect against accidental deletion.

To make things run faster, I do not add large files from this filesystem tree to fossil, and I only back up the tree once every few days. I am thus concerned about losing these files by accidentally deleting them.

I protect against such an accident with the permissions tool. Run it like so.

dadaportal skeleton permissions dadaportal.conf

It looks for files of particular file formats that I am unlikely to want to edit, and it marks them all read-only (chmod 444).

Setting up a fossil server on NearlyFreeSpeech

After writing the configuration file, prepare the server like this.

dadaportal skeleton nfsn dadaportal.conf > install.sh
scp install.sh your-nfsn-host:/home/private
ssh your-nfsn-host 'sh /home/private/install.sh'

This configuration places a fossil executable in /home/protected/fossil, and it expects fossil repositories to be stored in /home/protected/r.

Setting up a fossil server on Super Dimensional Fortress

After writing the configuration file, prepare the server like this.

dadaportal skeleton nfsn dadaportal.conf > install.sh
scp install.sh user@sdf.org:
ssh user@sdf.org 'sh install.sh'

This configuration places a fossil executable in ~/fossil, and it expects fossil repositories to be stored in ~/r.

Developing

I version the source code in fossil. Clone like so.

fossil clone https://thomaslevine.com/scm/dadaportal dadaportal.fossil
mkdir dadaportal
cd dadaportal
fossil open ../dadaportal.fossil