scott2

Artifact Content
Login

Artifact 6b4d8c17261fe36c87085eefa38d3d6679479f68:


We use Scott2 to organize data on permits related to earthmoving on
wetlands. It programmatically collects data from various sources and
assists people in their interactions with the data.

We support the following styles of interaction with Scott2.

Screen
    People fill out various structured data fields based on the primary
    data that are programmatically collected, and flag permit
    applications that should be reviewed.
Review
    Professionals thoroughly read public notices about permit
    applications; they determine whether to submit a comment on the
    each application and to begin other processes related to
    scrutinizing this application.
Tracking
    After one submits something like a comment or freedom of information
    request, one must wait several weeks for a response. The software
    helps people keep track of when responses are due or when they
    otherwise need to take further actions.
High-level analysis
    People can download data for further analysis with statistical
    software, geographic information systems, &c. The permit application
    data on their own can show trends in permitting. To study how
    wetlands development relates to other outcomes, (for example, public
    health, environmental quality, economic development) people can
    combine the data from Scott2 with other data sources.

Scott2 has very good support for managing data for Louisiana and rough
support for the rest of the United States. While it is already usable
for other regions, we need to work with appropriate experts from the
other regions who better understand what data are available and how they
are reported.

Through substantial usability research and through cataloging of the
various primary data sources, we have developed a streamlined system for
processing data related to wetland permit applications. The data
collection tools are the core of Scott2; I (Tom) think that a tool like
Scott2 could be used internally at USACE to make their own processes
more efficient and to create their own rich, open data in the first
place, so that we don't have to rely on unofficial data from groups like
the Gulf Restoration Network.

How it works
--------------------
This application is designed to match the workflow of wetlands
professionals like Scott.

::

    Maps         USACE     More data souces
        \            |            /
          \          |          /
            \        |        /
              \      |      /
        Database of wetland-related
           development projects
                   /   \   \
                 /       \   \        
               /      Proofread data,
             /        review projects,         
            |         and track progress.      
            |            
            |
            |
    Output as maps, 
    other graphs,   
    and spreadsheets

Data import
~~~~~~~~~~~~~
Scott2 loads data automatically from various sources into a
central database of projects.
The most notable source is the public notices posted on the
respective websites of the various districts of the
United States Army Corps of Engineers.

Screening
~~~~~~~~~~~~~~~~~~~~~~~~~~~
An computer program tries to pick out the most important information
from these documents, such as the locations and acreages of proposed
developments.

People then look over the data to correct anything that the computer got
wrong. This part doesn't require particularly much knowledge about
wetlands or on administrative processes surrounding permits, so it is
the easiest thing for new volunteers to help out on.

In practice, they may not fix *all* of the data in the database;
they are most focused on identifying reckless developments and
stopping those, so it is less important that data about small
developments and restorations be complete.

Acting on the data
~~~~~~~~~~~~~~~~~~~~
Wetlands professionals then review the projects in the database
and decide whether to take action in response to any. If they take
action, they usually follow the following process, which is connected
to (CITE THE PROCESS).

1. 


High-level analysis
~~~~~~~~~~~~~~~~~~~~
A side effect of this process is that we wind up with all of the data in
a nice spreadsheet, and this makes it easy to make graphs and maps and
stuff. For example, Kate Gruzd made a chart with this sort of data.


Installing
---------------
Run this on debian. ::

    apt-get install python3 python3-pip python3-gdal
    pip3 install .

Run this on Fedora. ::

    dnf install python3 python3-pip gdal-python3 python3-plyvel python3-bcrypt python3-lxml
    pip3 install .

Run this on nix. ::

    nix-shell default.nix

Data
--------
Data are all these things.

* Automatically extracted data
* Manually extracted data (computer-assisted)
* Drafts of comments
* Communications with the army corps of engineers

Package structure
-----------------

server
    The main web server
usace_public_notices
    Get usace public notice data
db
    reify data and serialize and stuff  

Other references
----------------------
* http://healthygulf.org/our-work/water-wetlands/stopping-wetland-destruction
* http://healthygulf.org/sites/healthygulf.org/files/A_Guide_to_Protecting_Wetlands_in_the_Gulf_of_Mexico.pdf
* http://opc.deq.state.ms.us/faq.aspx#02

On backups
------------
I designed the leveldb schema such that schema migrations should rarely be
needed. Unfortunately, this means that when they are needed, I forget how to
do them safely, and I wound up corrupting the database once.

To make backups of manually entered data, it should be sufficient to download
the "update history" file; this is a spreadsheet with one row per field that
has been edited by a person. For convenience, you may also download the
archive of the leveldb; this is how the data are stored in the live server,
so it doesn't require a data import step for running a new copy of the
server.

You should save at least the "update history" file regularly.
Save both files before touching anything on the server.