— part of check-in
on branch trunk
We use Scott2 to organize data on permits related to earthmoving on
wetlands. It programmatically collects data from various sources and
assists people in their interactions with the data.
We support the following styles of interaction with Scott2.
People fill out various structured data fields based on the primary
data that are programmatically collected, and flag permit
applications that should be reviewed.
Professionals thoroughly read public notices about permit
applications; they determine whether to submit a comment on the
each application and to begin other processes related to
scrutinizing this application.
After one submits something like a comment or freedom of information
request, one must wait several weeks for a response. The software
helps people keep track of when responses are due or when they
otherwise need to take further actions.
People can download data for further analysis with statistical
software, geographic information systems, &c. The permit application
data on their own can show trends in permitting. To study how
wetlands development relates to other outcomes, (for example, public
health, environmental quality, economic development) people can
combine the data from Scott2 with other data sources.
Scott2 has very good support for managing data for Louisiana and rough
support for the rest of the United States. While it is already usable
for other regions, we need to work with appropriate experts from the
other regions who better understand what data are available and how they
Through substantial usability research and through cataloging of the
various primary data sources, we have developed a streamlined system for
processing data related to wetland permit applications. The data
collection tools are the core of Scott2; I (Tom) think that a tool like
Scott2 could be used internally at USACE to make their own processes
more efficient and to create their own rich, open data in the first
place, so that we don't have to rely on unofficial data from groups like
the Gulf Restoration Network.
How it works
This application is designed to match the workflow of wetlands
professionals like Scott.
Maps USACE More data souces
\ | /
\ | /
\ | /
\ | /
Database of wetland-related
/ \ \
/ \ \
/ Proofread data,
/ review projects,
| and track progress.
Output as maps,
Scott2 loads data automatically from various sources into a
central database of projects.
The most notable source is the public notices posted on the
respective websites of the various districts of the
United States Army Corps of Engineers.
An computer program tries to pick out the most important information
from these documents, such as the locations and acreages of proposed
People then look over the data to correct anything that the computer got
wrong. This part doesn't require particularly much knowledge about
wetlands or on administrative processes surrounding permits, so it is
the easiest thing for new volunteers to help out on.
In practice, they may not fix *all* of the data in the database;
they are most focused on identifying reckless developments and
stopping those, so it is less important that data about small
developments and restorations be complete.
Acting on the data
Wetlands professionals then review the projects in the database
and decide whether to take action in response to any. If they take
action, they usually follow the following process, which is connected
to (CITE THE PROCESS).
A side effect of this process is that we wind up with all of the data in
a nice spreadsheet, and this makes it easy to make graphs and maps and
stuff. For example, Kate Gruzd made a chart with this sort of data.
Run this on debian. ::
apt-get install python3 python3-pip python3-gdal
pip3 install .
Run this on Fedora. ::
dnf install python3 python3-pip gdal-python3 python3-plyvel python3-bcrypt python3-lxml
pip3 install .
Run this on nix. ::
Data are all these things.
* Automatically extracted data
* Manually extracted data (computer-assisted)
* Drafts of comments
* Communications with the army corps of engineers
The main web server
Get usace public notice data
reify data and serialize and stuff
I designed the leveldb schema such that schema migrations should rarely be
needed. Unfortunately, this means that when they are needed, I forget how to
do them safely, and I wound up corrupting the database once.
To make backups of manually entered data, it should be sufficient to download
the "update history" file; this is a spreadsheet with one row per field that
has been edited by a person. For convenience, you may also download the
archive of the leveldb; this is how the data are stored in the live server,
so it doesn't require a data import step for running a new copy of the
You should save at least the "update history" file regularly.
Save both files before touching anything on the server.