Blooming Cacti

Hugo Bound?

Joe Martin — Thu, 15 Nov 2018 01:59:43 -0600

I've been thinking about switching my blogs from being generated with Pelican to being generated with Hugo. Pelican takes nearly 30 seconds to rebuild my main blog, while Hugo promises to work much more quickly.

Pelican is written in Python and needs to be installed in a virtualenv, with a whole passel of Python modules. Hugo is written in Go and compiled to a single, static executable. That one file can be dropped anywhere on a server and run without any further dependencies. That kind of simplicity appeals to me, over my current setup which has Pelican running in a Docker container.

My current setup is more baroque than just running Pelican in a container. All of my blog posts are stored in Git. To publish something, I commit the new post to Git and push the update to the repository on BitBucket. BitBucket then pings my server with a notification that something was changed. My server pulls the updates from the Git repository, uses the Dockerized install of Pelican to rebuild the site, and then pushes the updated HTML to the web server. There's a lot of moving parts there. And most of the time I don't even remember how they work or where they're all installed.

Before I can possibly move to Hugo, I need to document how my current setup works, where everything is installed, and how the pieces communicate with each other. I also need to document which features of Pelican I'm actually using (and which plugins), so that I can be sure that Hugo has equivalent functionality.

(I hope Betteridge's Law doesn't apply to this blog post.)

Picking a Development Language

Joe Martin — Tue, 06 Jun 2017 16:39:34 -0500

Note: I wrote this post 2 years ago, in a Houston hotel room. I recently realized that I never actually posted it to the blog.

I’m thinking about ditching PHP and using Python for web development. @marcoarment linked to PHP: a fractal of bad design. It’s a comprehensive rant about everything that’s wrong with PHP.

Summary:

PHP is full of surprises: mysql_real_escape_string, E_ALL

PHP is inconsistent: strpos, str_rot13

PHP requires boilerplate: error-checking around C API calls, ===

PHP is flaky: ==, foreach ($foo as &$bar)

PHP is opaque: no stack traces by default or for fatals, complex error reporting

I’ve seen most of these myself. I’ve been ignoring on the theory that PHP runs everywhere and it’s the language I know best. But, realistically, I’m going to be running my stuff on a VPS from now on. So the “runs anywhere” logic is moot. I also don’t know it all that well. If I’m going to be doing more programming, it makes sense to invest my time in a language that will be better long term instead of one that’s horrible.

I’ve resisted committing to a modern language like Python or Ruby. I like that I can create one-off PHP scripts in a single new file, drop them on the server, and have them executed easily. I don’t like having to create a separate Python / Ruby application for each idea that I have.

Today, I had an epiphany: create one application for all of my one-off ideas. Implement each idea in a separate method (or module) and use application URLs to route between them. If I have a new idea, I only need to create a new method and then redeploy the application. In some ways, that will be simpler than creating new one-off scripts all over the place. For one thing, all of my scripts will be collected in one place, making maintenance easier.

Why Python? It’s the language of choice for Editorial, my favorite scriptable iOS editor. I can run Python scripts on iOS, via Pythonista. The highly recommended Sublime Text editor uses Python as its scripting engine. Sublime Text runs on Mac, Windows, and Linux. Python looks to be my best choice for a universal scripting / development language across all of my text editors and platforms.

The author of the rant recommended three Python projects for web development.

Flask for the web stuff
Pyramid - medium level
Django, which is a complex monstrosity that works well for building sites like Django's

Linode: Final Migration Steps

Joe Martin — Tue, 06 Jun 2017 16:39:34 -0500

This post is a bit of a grab bag, of the final things that I did to set up my new Linode server.

Videohub Migration

I had created a very small, personal video hosting application using PHP and symfony. It was pretty easy to move but I’d forgotten some of the details of how it worked, so I had to figure out a few extra steps to get it working.

I rsynced the project over to Linode.
I symlinked the videohub2/web directory to the public_html directory that I’d created for the application.
I edited the symfony file, to change the PHP path to /usr/bin/php.
I cleaned the symfony cache, to remove cached references to the old server.
```
./symfony cc
```
I set the timezone, at the top of web/index.php, to stop PHP from complaining about a missing default time zone.
```
date_default_timezone_set('America/Chicago');
```
Then I got a fun error from PHP, preventing the app from running at all.
```
PHP Fatal error:  session_start(): Failed to initialize storage module: files (path: )
```
I figured out that PHP didn’t have anywhere to save session files. I solved this by adding a new config line to my PHP-FPM pool definition and creating a new directory. (This happened before I finalized my PHP application creation script.)
```
php_admin_value[session.save_path] = /srv/www/videohub/tmp
```
Finally, I hit my last error.
```
Unable to open PDO connection [wrapped: could not find driver]
```
I’d semi-forgotten that the app was built with a SQLite database and that PHP needed SQLite extensions, to access the database. I solved the error by installing the PHP extension for SQLite.
```
apt-get install sqlite3 php5-sqlite
```

After I restarted PHP (to pick up the new extensions), the app worked beautifully.

Sending Mail

I don’t want to run a full mail server on my Linode, but I did want the ability to send out emails, so that I could get notifications from my cron jobs, in the event that anything went wrong.

After a quick search of the Linode Library, I found a guide to creating a gateway with Postfix on Debian 6 (Squeeze). I followed the steps in that guide (up until the “create mail directories” step) and ended up with a perfectly functional outgoing mail server.

Logrotate

I needed to ensure that none of my (many) nginx and PHP log files grew too large. Debian comes with logrotate preinstalled. Adding my log files to logrotate’s configuration was pretty easy. Out of the box, logrotate loads its configurations from any files that are in /etc/logrotate.d. I created a new file there, called www.

I rotates files weekly, as long as they’re at least 1 megabyte. The rotated logs are created under the www-data user / group and compressed with bzip2. As the logs are rotated, I’m keeping only the last 12 logs (about 3 months worth). Each rotated file is named with the date it was rotated.

/srv/www/*/log/*.log {
        size 1M
        copytruncate
        create 0660 www-data www-data
        rotate 12
        compress
        dateext
        weekly
        missingok
        compresscmd /bin/bzip2
        compressext .bz2
}

It works well.

Offsite Backups

Finally, I setup a script to use rsync to copy both the /srv/www and database backup folders over to Strongspace, nightly. In the event of disaster, I’ll never lose more than a day’s worth of changes to the sites.

#!/bin/bash

RSYNC="/usr/bin/rsync"
OPTIONS="-rltvz --delete"
SERVER="df@df.strongspace.com"
MYDATE="`date +%d | sed 's/0*//'`"

SRC="/var/backup/"
DEST="/strongspace/df/home/JoesFiles/mark_copy"
BACKUPDIR="/home/df/Backups/Joe/mark_copy/Backup-$((${MYDATE}/2))"
EXCLUDE="/root/excludes.txt"
${RSYNC} ${OPTIONS} --exclude-from "${EXCLUDE}" "${SRC}" "${SERVER}":"${DEST}"

SRC="/srv/www/"
DEST="/strongspace/df/home/JoesFiles/mark_copy"
BACKUPDIR="/home/df/Backups/Joe/mark_copy/Backup-$((${MYDATE}/2))"
EXCLUDE="/root/excludes.txt"
${RSYNC} ${OPTIONS} --exclude-from "${EXCLUDE}" "${SRC}" "${SERVER}":"${DEST}"

And, with that, I’m done with my migration to Linode. It was fun but now I’m ready to get back to actually enhancing my sites rather than just moving my sites.

Migrating Wordpress Multisite

Joe Martin — Tue, 06 Jun 2017 16:39:34 -0500

Using my new PHP setup method, I migrated a bunch of PHP applications over to my Linode. But I saved the hardest challenge for last: migrating my Wordpress multisite install. There are 5 active sites inside of that install and about 3 inactive sites, as well as a decent amount of uploaded media. I wanted to figure out all of the other wrinkles first, before I tackled the most important PHP application. As if all of that wasn’t enough, I decided that I wanted to move the Wordpress installation from its previous domain (desertflood.com) to its current domain (wordflood.net).

After reading through multiple sites about Wordpress migrations, I decided to take the migration in two steps.

Move the site using the existing domain and using my /etc/hosts file to access the existing domain via the new IP address.
Change the domain on the new Wordpress install.

Moving to the New Server

After all of the worry, the actual move turned out to be fairly straightforward. I was already using automysqlbackup to create nightly backups of my MySQL databases and rsync to create offsite backups of my files. Thanks to that, and shell access on both servers, it was fairly trivial to copy both the Wordpress files and database to the new server.

I had already created a home directory and user account for the new Wordpress install, using my setup script. After transferring my files from the old server to the new server and copying the database backup from the old server to the new server, I was able to load the backup into my new database, using the mysql command line tool.

rsync -rav jmartin@litho.joyent.us:domains/desertflood.com/web/public/ /srv/www/wordpress/public_html
scp jmartin@litho.joyent.us:/users/home/jmartin/backups/db/latest/jmartin_mjm_wp3_week.34.2012-08-25_01h03m.sql.bz2 /home/wordpress
bzip2 -d /home/wordpress/jmartin_mjm_wp3_week.34.2012-08-25_01h03m.sql.bz2
cat /home/wordpress/jmartin_mjm_wp3_week.34.2012-08-25_01h03m.sql | mysql -u wordpress -p wordpress

I was already using nginx on my Joyent shared account. That being the nginx setup easy as well. I copied the nginx configuration file from the old server, to merge it into the configuration file for the new server.

scp jmartin@litho.joyent.us:etc/nginx/sites/desertflood.conf .
ln -s /etc/nginx/sites-available/wordpress /etc/nginx/sites-enabled

Then, on my laptop, I pointed my domains to the new server, using /etc/hosts.

66.175.215.121 desertflood.com www.desertflood.com thosemartins.desertflood.com thosemartins.us minorthoughts.desertflood.com minorthoughts.com www.minorthoughts.com www.thosemartins.us

After I edited my wp-config.php file to point to the new database, I was able to access Wordpress on the new server.

Moving to the New Domain

I was worried about this too but it turned out to be almost as easy as moving the actual Wordpress installation. There were just three steps.

Edit the wp-config.php file to change the DOMAIN_CURRENT_SITE parameter to the new domain.
Update the MySQL database. The MySQL database had references to the old domain scattered all throughout it. My natural inclination would be to just do a global search and replace on the database backup, before loading it into MySQL. However, Wordpress has a lot of serialized data in the database. You can’t just change values there, without breaking those columns.

Fortunately, the internet has already solved that problem for me. Interconnect IT mentions it in their guide to migrating a Wordpress/WPMU/BuddyPress website. They developed a tool for safely searching and replacing in the database, using PHP. I temporarily installed the “Safe Search and Replace” script and used it to change all instances of desertflood.com to wordflood.net in the database.
Finally, in the ‘Domain Mapping’ section of Wordpress’s Network Admin, I changed the Server IP Address to the IP of the new server.

And, just like that, Wordpress was live on the new domain.

WP Super Cache / MultiSite / nginx

One final note. On Monday, I decided to activate the WP Super Cache plugin, to add that extra bit of oomph to the site’s loading speed. I’ve used it before but it’s been a while. When I turned it on, I ran into a nasty bug that shows up when using WPSC 1.1 with nginx. The plugin assumes that the environment variable SERVER_NAME is configured, but nginx doesn’t set that environment variable.

The end result is that Wordpress caches whichever site the administrator last visited. Then nginx serves out the content for that site to visitors for every other Wordpress site. In effect, the administrator’s last visited site gets cloned across all of the Wordpress sites.

Fortuantely, donncha posted a development version of WPSC that fixes the SERVER_NAME problem. Once I installed it, my caching ills went away.

Implementing a Locked Down PHP Site

Joe Martin — Tue, 06 Jun 2017 16:39:34 -0500

In my last post I talked about a plan for securely setting up PHP. After making the plan, I had two goals: test it to see if it would work and automate the setup.

Testing the Setup

I started with a simple site: a small Dokuwiki installation. It was an easy site to test because it didn’t require a database connection.

Run nginx as a separate user account, with read access to the web root.

Done. nginx is already setup to run as the www-data user and the web files are not group writeable.
Create a new user, dokuwiki.

I set up the user account with a user private group as well as membership to the www-data group. This will let me log in as the dokuwiki user to edit the dokuwiki site (and only the dokuwiki site). I also created a directory for the dokuwiki site, changed the user ownership to the dokuwiki user and the group ownership to the www-data group. Finally, I made the whole thing user writeable and group readable.
```
adduser dokuwiki
usermod -a -G www-data dokuwiki
mkdir /srv/www/dokuwiki
chown -R dokuwiki:www-data /srv/www/dokuwiki
chmod -R 2755 /srv/www/dokuwiki
```
Set up a separate PHP instance for each application and use the open_basedir directive to lock PHP down to the specific application’s folder.

I setup my VPS to use the Dotdeb repository for Debian and installed the php5-fpm package. I copied the default configuration, to make one specific for Dokuwiki, and edited it.
```
apt-get install php5 php-pear php5-mysql php5-suhosin php5-cli php5-cgi php5-fpm
sudo cp /etc/php5/fpm/pool.d/www.conf /etc/php5/fpm/pool.d/dokuwiki.conf 
sudo vim /etc/php5/fpm/pool.d/dokuwiki.conf
```
I made the following changes to my dokuwiki.conf file. This sets up a PHP “pool” just for Dokuwiki. The pool runs as the dokuwiki user, so that PHP will have read-write access to the dokuwiki folders.
```
; pool name
[dokuwiki]

user = dokuwiki
group = www-data
listen = /var/run/php5-fpm/dokuwiki.sock

pm.max_children = 2    ; this is an infrequently used site
pm.start_servers = 1
pm.min_spare_servers = 1
pm.max_spare_servers = 1
pm.max_requests = 500
request_terminate_timeout = 30s

chdir = /srv/wwww/dokuwiki/public_html/
php_admin_flag[log_errors] = on
php_admin_value[error_log] = /srv/wwww/dokuwiki/log/fpm-php.err.log
php_admin_value[open_basedir] = /srv/wwww/dokuwiki/
php_admin_value[session.save_path] = /srv/www/dokuwiki/tmp
security.limit_extensions = .php .php3 .php4 .php5
```
The chdir directive ensures that PHP starts from the dokuwiki web directory. The php_admin_flag and php_admin_value directives allow me to set PHP options directly, without needing to create a separate php.ini file for each application. With this setup, PHP will log errors to an application specific log file, limit applications to reading and writing files in the application’s directory, and limit PHP to executing files with various PHP extensions.

I can’t be sure until someone actually hacks me, of course, but I think this is pretty locked down. Attackers can’t use PHP to read / write other files on the server. And, because I’m using a specific user account for each application, exploiting security holes to gain shell access only gains you shell access for this particular account and application. I may not be able to prevent an individual application from being hacked, but I can limit the damage to that one application.
Set up nginx to serve the site

After I saved the configuration file, I started up php5-fpm, to launch the persistently running PHP processes: /etc/init.d/php5-fpm start. Next I created an nginx sites file, to enable nginx to serve the application: /etc/nginx/sites-enabled/dokuwiki.
```
location ~ \.php$ {
    include /etc/nginx/fastcgi_params;
    fastcgi_pass /var/run/php5-fpm/dokuwiki.sock
    fastcgi_index index.php;
    fastcgi_param SCRIPT_FILENAME /srv/www/dokuwiki/public_html$fastcgi_script_name;
}
```
Notice that nginx and and PHP-FPM are communicating through Unix sockets, using a unique socket for each PHP application. I tested the configuration and it worked perfectly.

Automating the Setup

I created a shell script to automate each of these steps for me. It needs to be run by the root user. The first parameter is the username that you want the script to create and the second is the name of the application that you’re setting up. The only real rule is that the app name can’t have an '@' symbol in it. I saved this script as phpapp.sh.

First it will create the new user account (and user private group) and add the user account to the www-data group. It will create all of the necessary application folders under /srv/www and set the ownership and permissions appropriately. It will also create a PHP-FPM configuration file (using the default file as a base) and an nginx sites file, setting up the correct Unix socket in each.

#!/bin/sh
# params
# $1 = user
# $2 = app name
# $3 = domain

USER=$1
APP=$2
DOMAIN=$3
APATH="/srv/www/${APP}"
CONF="/etc/php5/fpm/pool.d/${APP}.conf"
NGINXCONF="/etc/nginx/sites-available/${APP}"

adduser ${USER}
usermod -a -G www-data ${USER}
mkdir -p "${APATH}/public_html"
mkdir -p "${APATH}/tmp"
mkdir -p "${APATH}/log"
chown -R ${USER}:www-data "${APATH}"
chmod -R 2755 "${APATH}"

cp /etc/php5/fpm/pool.d/www.old "${CONF}"
sed -i "s/\[www\]/\[${APP}\]/" "${CONF}"
sed -i "s@listen = 127.0.0.1:9000@listen = /var/run/php5-fpm/${APP}.sock@" "${CONF}"
sed -i "s@user = www-data@user = ${USER}@" "${CONF}"
sed -i "s/;listen.allowed_clients/listen.allowed_clients/" "${CONF}"
sed -i "s@chdir = /@chdir = ${APATH}/public_html/@" "${CONF}"
sed -i 's/;security.limit_extensions/security.limit_extensions/' "${CONF}"
sed -i "s/;php_admin_flag\[log_errors\] = on/php_admin_flag\[log_errors\] = on/" "${CONF}"
sed -i "s@;php_admin_value\[error_log\] = /var/log/fpm-php.www.log@php_admin_value\[error_log\] = ${APATH}/log/fpm-php.err.log@" "${CONF}"
echo "php_admin_value[open_basedir] = ${APATH}/" >> "${CONF}"
echo "php_admin_value[session.save_path] = ${APATH}/tmp" >> "${CONF}"

cp /etc/nginx/sites-available/template "${NGINXCONF}"
sed -i "s@/srv/www/app/@/srv/www/${APP}/@" "${NGINXCONF}"
sed -i "s@/var/run/php5-fpm/app.sock@/var/run/php5-fpm/${APP}.sock@" "${NGINXCONF}"
sed -i "s/server_name template.com/server_name ${DOMAIN}/" "${NGINXCONF}"

After running this script, you just need to restart PHP-FPM and reload nginx and your application will be good to go, safely sandboxed in its own directory and account.

Locking Down PHP

Joe Martin — Tue, 06 Jun 2017 16:39:34 -0500

The most important sites that I need to migrate from Joyent to Linode are PHP sites, primarily my Wordpress multisite network. I’ve been dragging my feet on it though, because I want to be paranoid about security and I couldn’t figure out how to be properly paranoid with PHP.

After all, the web server needs to have read access to all of the files. The PHP interpreter needs to have read access to all of the files. And, for several sites, the PHP interpreter needs to have write access to both files and folders. But if the PHP interpreter has write access to the files then one poorly written script can give an attacker write access to those files as well. The last thing I want is for a bug in one PHP application to cause destruction across all of my PHP applications.

Today, I think I finally figured out how I want my PHP setup to work. I was searching the Linode forums for inspiration and I found this post from hybinet. Extrapolating from that post, there are several things that I want to do.

Setup nginx to run under its own user account, as a member of a group that has read access (but not write access) to the entire web hierarchy.
Create a separate user account for each PHP application. Make that application’s files readable and writeable by that user and readable by the nginx group.
Setup a separate PHP instance for each application. Have PHP run as that user, with read and write access to that user’s files and that user’s files only.
Use the open_basedir directive to lock PHP down to the specific application’s folder. Don’t let it exit that file hierachy for any reason.

With this setup, if an application gets hacked, the attacker will have access to one specific user account and one specific, limited, set of files. A successful attack against one application won’t automatically result in all applications being attacked.

The big question I had was how to manageably set up a separate PHP instance for each application and manageably limit PHP to a specific application root. That’s why I was ecstatic when I learned about PHP-FPM (FastCGI Process Manager). It’s a new (as of PHP 5.3.3 and PHP 5.4) FastCGI implementation. Among its features is the ability to start workers with different uid/gid/chroot/environments and the ability to start workers with different php.ini files.

I can easily configure the PHP-FPM configuration file with entries for each application, limiting the user account and application roots. Then I can configure nginx to direct PHP scripts for each application to that application’s specific pool of PHP-FPM workers. The result will be a centrally managed PHP setup that can have limited security for each application.

The theory sounds fantastic. Now I just have to work on actually setting it up for a test PHP application.

Your Password is Weak

Joe Martin — Tue, 06 Jun 2017 16:39:34 -0500

When I set up a new account for my parents, they're always annoyed by my password choice: long, nonsense strings of letters, numbers, and punctuation characters. There's a reason for that though: if you can think up a password, someone else can crack it with hardly any effort at all.

Ars Technica goes into the gory details.

The average Web user maintains 25 separate accounts but uses just 6.5 passwords to protect them, according to a landmark study (PDF) from 2007. ... such password reuse, combined with the frequent use of e-mail addresses as user names, means that once hackers have plucked login credentials from one site, they often have the means to compromise dozens of other accounts, too.

Newer hardware and modern techniques have also helped to contribute to the rise in password cracking. Now used increasingly for computing, graphics processors allow password-cracking programs to work thousands of times faster than they did just a decade ago on similarly priced PCs that used traditional CPUs alone. A PC running a single AMD Radeon HD7970 GPU, for instance, can try on average an astounding 8.2 billion password combinations each second, depending on the algorithm used to scramble them. Only a decade ago, such speeds were possible only when using pricey supercomputers.

At any given time, Redman is likely to be running thousands of cryptographically hashed passwords though a PC containing four of Nvidia's GeForce GTX 480 graphics cards. It's an "older machine," he conceded, but it still gives him the ability to cycle through as many as 6.2 billion combinations every second.

What can you do to protect yourself? Use long, randomly generated passwords. And never, ever reuse a password.

Even powerful computation engines have trouble cracking longer passwords using brute force. Assuming such an attack checks for all combinations of all 95 letters, numbers, and symbols available on a standard English-language keyboard, it takes a matter of hours for a desktop computer with an Intel Core i7 980x processor to brute-force crack any five character password. Increasing the password length by just one character requires about a day; bumping the length by one more character, though, dramatically increases the cracking time to more than 10 days. Rob Graham, the Errata Security CEO who calculated the requirements, refers to this limitation as the "exponential wall of brute-force cracking."

So what can the average person do to pick a passcode that won't be toppled in a matter of hours? Per Thorsheim, a security advisor who specializes in passwords for a large company headquartered in Norway, said the most important attribute of any passcode is that it be unique to each site.

"For most sites, you have no idea how they store your password," he explained. "If they get breached, you get breached. If your password at that site is unique, you have much less to worry about."

It's also important that a password not already be a part of the corpus of the hundreds of millions of codes already compiled in crackers' word lists, that it be randomly generated by a computer, and that it have a minimum of nine characters to make brute-force cracks infeasible. Since it's not uncommon for people to have dozens of accounts these days, the easiest way to put this advice into practice is to use program such as 1Password or PasswordSafe. Both apps allow users to create long, randomly generated passwords and to store them securely in a cryptographically protected file that's unlocked with a single master password.

If you need help generating a strong master password, you could try my online Diceware pass phrase generator.

Moving Lessn

Joe Martin — Tue, 06 Jun 2017 16:39:34 -0500

The last site that I’m going to move this weekend is a small: my personal URL shortener, using Lessn. It’s a good small test, because it uses PHP and it will let me make sure that I can get PHP working, without having to mess with a large, complex site.

I’m not going to detail all of the setup. For the most part, I’m taking the basic nginx / PHP setup stragith from Linode’s Nginx and PHP-FastCGI on Debian 6 guide.

Here are the MySQL steps, to create the Lessn database and a secure user for it.

mysqladmin create lessn

CREATE USER 'lessn'@'localhost' IDENTIFIED BY 'xxxxx';
GRANT ALL ON lessn.* TO 'lessn'@'localhost';

Finally, I could load the database backup, from Litho.

scp jmartin@litho.joyent.us:/users/home/jmartin/backups/db/latest/jmartin_lessn_2012-08-19_01h03m.Sunday.sql.bz2 .
bzip2 -d jmartin_lessn_2012-08-19_01h03m.Sunday.sql.bz2
cat jmartin_lessn_2012-08-19_01h03m.Sunday.sql | mysql -u lessn -p lessn

I also need to copy the site files over, from Litho, and configure them for their new home.

rsync -ravn jmartin@litho.joyent.us:/users/home/jmartin/domains/jmdf.im/web/public /srv/www/jmdf.im/public_html
chown -R www-data:www-data /srv/www/jmdf.im
vim /srv/www/jmdf.im/public_html/-/config.php

The final change I had to make was to enable URL rewriting, for lessn, in /etc/nginx/sites-available/jmdf.im.

location / {
    index index.php index.html index.htm;
    # if the requested file exists, return it immediately
    if (-e $request_filename) {
            break;
    }

    # all other requests go to Wordpress
    if (!-e $request_filename) {
            rewrite ^(.*)$ /index.php?token=$1 last;
    }
}

After I reloaded nginx with the new configuration, everything worked perfectly.

Migrating "Timetrack" from Joyent to Linode

Joe Martin — Tue, 06 Jun 2017 16:39:34 -0500

I decided to start my migration by migrating my Rails application, timetrack. This migration would have been a lot easier except that this particular application is using 2-year old tools and still needs a lot of tweaking.

The main dependencies are MySQL and Ruby 1.8.7.

User Account and Ruby

I started by creating a new user account, just for the timetrack application.

useradd timetrack

I installed rvm, then tried to install ruby 1.8.7, from the ‘timetrack’ user account.

rvm install 1.8.7

The rvm script gave me a list of Debian dependencies to install first.

/usr/bin/apt-get install build-essential openssl libreadline6 libreadline6-dev curl git-core zlib1g zlib1g-dev libssl-dev libyaml-dev libsqlite3-dev sqlite3 libxml2-dev libxslt-dev autoconf libc6-dev ncurses-dev automake libtool bison subversion pkg-config

After that, ruby 1.8.7 installed cleanly. Although I did get a warning that just compiling it on modern Linux required two extra patches and that I should strongly consider installing ruby 1.9, for better security.

Install MySQL

Installing MySQL was easy.

apt-get install mysql-server libmysqlclient-dev

I tuned it, using the parameters from Linode.

I created a new database, specifically for this application, then created a new database specific MySQL user account.

mysqladmin create timetrack

CREATE USER 'timetrack'@'localhost' IDENTIFIED BY 'xxxx';
GRANT ALL ON timetrack.* TO 'timetrack'@'localhost';

Finally, I could load the database backup, from Litho.

mysql -u timetrack -p timetrack < jmartin_desertflood_timetrack_2012-08-19_01h03m.Sunday.sql

Deploy the Application

Capistrano is quite powerful and should have made this the easiest part of the setup. But I never did build a really good deployment recipe, so it was only a partial help. I started by updating the config/deploy.rb file, to point to my new server and user account. Then I had to setup my 'timetrack' user account, to be able to access GitHub. Without it, I was getting errors on the cap deploy step.

Create a new SSH key
```
ssh-keygen
```
Copy the contents of '.ssh/id_rsa.pub' to my Github keys. Then do an initial checkout, to make sure that SSH is okay with Github’s server.
```
git clone git@github.com:jmartindf/timetrack.git
```

Setup the required directories.

mkdir -p /home/timetrack/apps/timetrack/releases
mkdir shared/pids
mkdir shared/log

Deploy the app
```
cap deploy
```

Update local configurations

vim config/database.yml
vim config/unicorn.conf.rb

Install required gem dependencies
```
bundle install
```

Start the app, to see if it works.

unicorn -E production -D -o 127.0.0.1 -c /home/timetrack/apps/timetrack/current/config/unicorn.conf.rb

And, it works. Amazing.

Start at boot

It’s all well and good to start it manually, but I really need it to start at boot. After some Googling, I found an init.d script for unicorn, that I could use as a basis for my own init.d script.

After tweaking it a bit, this is what it looks like. The key points are that it uses rvm and bundler, to get the dependencies right. (As mentioned before, I’m using some old gems and an old version of ruby.) It also uses the chdir and chuid flags, to start-stop-daemon, to get everything running with the right permissions in the right folder. I’ve tested it and it does all work.

#! /bin/sh

PATH=/sbin:/usr/sbin:/bin:/usr/bin
DESC="Timetrack server"
NAME=timetrack
RVM=/home/timetrack/.rvm/bin/rvm
RVM_EXEC_ARGS="1.8.7 exec"
RAILS_DIR=/home/timetrack/apps/timetrack/current
DAEMON=$RVM
DAEMON_ARGS="$RVM_EXEC_ARGS bundle exec unicorn -c $RAILS_DIR/config/unicorn.conf.rb -E production -D"
PIDFILE=$RAILS_DIR/tmp/pids/timetrack.pid
USER=timetrack
SCRIPTNAME=/etc/init.d/$NAME

# Exit if the package is not installed
[ -x "$DAEMON" ] || exit 0

# Read configuration variable file if it is present
[ -r /etc/default/$NAME ] && . /etc/default/$NAME

# Load the VERBOSE setting and other rcS variables
. /lib/init/vars.sh

# Define LSB log_* functions.
# Depend on lsb-base (>= 3.0-6) to ensure that this file is present.
. /lib/lsb/init-functions

# unicorn needs some environment vars
. /etc/environment

# Create /var/run dir if missing
# if [ ! -d /var/run/unicorn ]; then
#   mkdir -m 0755 /var/run/unicorn
#   chown $USER /var/run/unicorn
# fi

symlink_resque() {
    ln -sfT `$RVM $RVM_EXEC_ARGS which resque | tail -1`/server/public $RAILS_DIR/public/resque
}

#
# Function that starts the daemon/service
#
do_start()
{
    # Return
    #   0 if daemon has been started
    #   1 if daemon was already running
    #   2 if daemon could not be started
    symlink_resque
    start-stop-daemon --start --quiet --pidfile $PIDFILE --chuid $USER --exec $DAEMON --test $RAILS_DIR --chdir $RAILS_DIR > /dev/null \
        || return 1
    start-stop-daemon --start --quiet --pidfile $PIDFILE --chuid $USER --exec $DAEMON --chdir $RAILS_DIR -- \
        $DAEMON_ARGS \
        || return 2
    # Add code here, if necessary, that waits for the process to be ready
    # to handle requests from services started subsequently which depend
    # on this one.  As a last resort, sleep for some time.
}

#
# Function that stops the daemon/service
#
do_stop()
{
    # Return
    #   0 if daemon has been stopped
    #   1 if daemon was already stopped
    #   2 if daemon could not be stopped
    #   other if a failure occurred
    start-stop-daemon --stop --quiet --retry=TERM/30/KILL/5 --pidfile $PIDFILE
    RETVAL="$?"
    [ "$RETVAL" = 2 ] && return 2
    # Ensure the pidfile is cleared
    rm -f $PIDFILE
    return "$RETVAL"
}

#
# Function that sends a SIGHUP to the daemon/service
#
do_reload() {
    symlink_resque
    start-stop-daemon --stop --signal 1 --quiet --pidfile $PIDFILE
    return 0
}

#
# Function that sends a SIGUSR2 to the daemon/service
#
do_graceful_reload() {
    symlink_resque
    start-stop-daemon --stop --signal USR2 --quiet --pidfile $PIDFILE
    return 0
}

case "$1" in
  start)
    [ "$VERBOSE" != no ] && log_daemon_msg "Starting $DESC" "$NAME"
    do_start
    case "$?" in
        0|1) [ "$VERBOSE" != no ] && log_end_msg 0 ;;
        2) [ "$VERBOSE" != no ] && log_end_msg 1 ;;
    esac
    ;;
  stop)
    [ "$VERBOSE" != no ] && log_daemon_msg "Stopping $DESC" "$NAME"
    do_stop
    case "$?" in
        0|1) [ "$VERBOSE" != no ] && log_end_msg 0 ;;
        2) [ "$VERBOSE" != no ] && log_end_msg 1 ;;
    esac
    ;;
  status)
       status_of_proc "$DAEMON" "$NAME" && exit 0 || exit $?
       ;;
  reload)
    log_daemon_msg "Reloading $DESC" "$NAME"
    do_reload
    log_end_msg $?
    ;;
  graceful-reload)
    log_daemon_msg "Gracefully reloading $DESC" "$NAME"
    do_graceful_reload
    log_end_msg $?
    ;;
  restart)
    log_daemon_msg "Restarting $DESC" "$NAME"
    do_stop
    case "$?" in
      0|1)
        do_start
        case "$?" in
            0) log_end_msg 0 ;;
            1) log_end_msg 1 ;; # Old process is still running
            *) log_end_msg 1 ;; # Failed to start
        esac
        ;;
      *)
        # Failed to stop
        log_end_msg 1
        ;;
    esac
    ;;
  *)
    echo "Usage: $SCRIPTNAME {start|stop|status|restart|graceful-reload|reload}" >&2
    exit 3
    ;;
esac

:

I activated the timetrack init.d script, for the default run levels.

update-rc.d timetrack defaults

Nginx

My app is running, but it’s not on a port that the outside world can access. That’s where nginx comes in. I like nginx as a lightweight web server and in this case, I can easily use it to proxy traffic back and forth between my app. I just needed to create a very simple configuration file, for nginix.

vim /etc/nginx/sites-available/timetrack.desertflood.com
cd /etc/nginx/sites-enabled
ln -s /etc/nginx/sites-available/timetrack.desertflood.com
/etc/init.d/nginx start

Here’s what the file looks like.

upstream timetrack {
  server 127.0.0.1:10100;
}

server {
  listen       80;
  server_name  timetrack2.desertflood.com;
  location / {
  proxy_set_header  X-Real-IP        $remote_addr;
  proxy_set_header  X-Forwarded-For  $proxy_add_x_forwarded_for;
  proxy_set_header  Host             $http_host;
  proxy_redirect    off;
  proxy_pass        http://timetrack;
  }
}

And, just like that, I’ve migrated my first site from Joyent to Linode.

Moving to Linode

Joe Martin — Tue, 06 Jun 2017 16:39:34 -0500

After Joyent’s decision to terminate my lifetime hosting account I had two options: I could stay at Joyent, for 5 years, using a SmartMachine (their Virtual Private Server). Or I could take a refund of my original purchases ($600) and go elsewhere. I decided to go elsewhere.

Right now, I don’t want to be at Joyent. The way that Jason decided to terminate the lifetime hosting, the short time frame he gave us to migrate, and the original promotional offer (just 1 year on a SmartMachine) just rubbed me the wrong way. I had a lot of trust and affection banked for Joyent. But, at least right now, the trust is gone and the affection is greatly diminished.

As I thought about my options, I thought back to all of the negative aspects of hosting with Joyent. I thought back over the number of products that were promised and then failed to materialize. TextDrive was going to have a truly awesome control panel, but it never came. Then there was going to be a next generation statistics system, but that too never came. The Joyent Connector was going to be the future of group webmail, until it rotted on the vine. Shared Accelerators were the future until suddenly they were the past and they stopped selling shared hosting entirely. Accelerators became SmartMachines, and so on.

Joyent’s documentation could be a challenge too. Sometimes things were well documented on Wikis. Sometimes they weren’t. You took your chances. The community was great for figuring things out but, lately, the community just hasn’t been there around Joyent’s SmartMachines. The target customer either knows what they’re doing or can afford to pay for support. Those learning on their own are a bit left out. Then too, Joyent’s communication has always been ad-hoc. We joyeurs found out about many changes, on the forum. There was always a risk that you might miss changes entirely if you stopped participating in the forum.

All in all, I felt that it was time for me to move on. As I thought about my options, I grew increasingly attracted to Linode. I’ve been on a shared, managed hosting server for the past 7 years. But my real dream has always been to have my own server, out on the internet. I’ve always wanted to have a box that I can fully control and configure, to my own design.

Linode gives me that. For just $20 a month, I can have a Virtual Private Server of my very own. I can reboot it, wipe it, re-image it with different operating systems, move it between data centers, clone it, and more. I can install any server software I want and leave out anything that I don’t want. And Linode has great documentation too. Some if it is more basic than I need and some of it is more advanced than I need. That makes for a great foundation for learning and tinkering.

I’m taking a refund of my $600 from Joyent and I’ll be spending it at Linode, over the next couple of years. I can’t wait to get started.

The End of An Era

Joe Martin — Tue, 06 Jun 2017 16:39:34 -0500

For the past 7 years, I've hosted all of my websites at Joyent. Over the next 7 weeks, that will change. It wasn't my choice, but this is my story.

The Marriage

In 2005 I was a college senior. At that point, I'd already had the wordflood.net domain name for at least 5 years and I'd been using various web hosts that entire time. As I'd learned more about the internet and hosting, I'd switched providers frequently, to take advantage of what I was learning.

In March 2005, I found TextDrive. TextDrive was founded by Dean Allen, first and foremost a writer, as a web host for people who cared about writing, who cared about good design, and who didn't want to deal with corporate headaches. The first 200 users provided the funding for the initial servers and the entire setup was community driven and funded. I quickly switched my hosting to TextDrive and felt at home.

TextDrive had a great community of users. The founders (Dean Allen and Jason Hoffman) were the first two users and first two financial backers. They were very active in the forums and in almost every way were just two more members of the community. The community learned from each other, laughed together, and encouraged each other. Like almost every community, we had a host of inside jokes and our own cultural references. I loved it.

TextDrive offered a "VCIII" lifetime hosting plan, in September 2005. I would have to pay $399 for it, but it would guarantee me an account on their servers for as long as TextDrive existed. Given the existing community and the active participation by the founders, I wanted in.

I had graduated from college in May 2005 and gotten married in July 2005. For me, $399 was a lot of money. My wife wasn't really enthusiastic about the purchase. I convinced her by pointing out that it was a lifetime deal. It would get the current hosting amount off of our monthly budget and it would give me a "free" place to tinker, experiment, and learn for a long, long time.

In November 2005, a company called Joyent purchased TextDrive. The TextDrive customers were extremely nervous about this acquistion. We had a great company, a great community, and a great host. Would Joyent ruin it? Would they kick us out? Were our lifetime hosting accounts over now that TextDrive no longer officially existed?

David Young, Joyent's CEO, acted quickly to demonstrate good faith. He not only promised to honor the existing lifetime accounts, he doubled down by offering the Mixed Grill lifetime package, for $499. It included TextDrive hosting, Strongspace for file storage, and Joyent Connector for small group email. And, "how long is it good for?", we asked. "As long as we exist", David told us.

My wife and I had recently been talking about the need for online file storage, for backups. Strongspace would be perfect for that. And Joyent's Connector product looked like a good idea for email too. When Dean Allen decided that existing VC customers could upgrade to the Mixed Grill lifetime package for just $199, we were in. We scraped up the money from our budget and sent it in.

Since then, I've had nearly seven years to enjoy my hosting package and the Joyent community. Over time, most of the discussion migrated from the TextDrive forums to Twitter. But we all (mostly) stayed in contact, linked by our appreciation for Joyent and the early community we'd shared.

I learned a lot too. TextDrive gave us command prompt access to the servers. We had the freedom to install, configure, and manage our own software. I learned Ruby on Rails and hosted my own app on my account. I compiled my own copy of nginx and used it for my own PHP web apps. I installed multiple versions of Wordpress and learned how to create and manage a Wordpress multi-site server. I ran my own websites and those of my family on the server. It was a great time to tinker and learn.

Along the way, Strongspace was sold to ExpanDrive, but Joyent ensured that our lifetime accounts would stay intact and endure. The Joyent Connector was discontinued and the hosting servers were migrated from FreeBSD servers to shiny, new Solaris servers. Joyent grew, expanded, and moved into enterprise hosting. Joyent discontinued Shared Hosting for new customers but continued to honor their agreements with their lifetime customers.

The Break Up

Thursday morning, Joyent gave their loyal, lifetime customers a nasty surprise.

Dear Joseph,

We've been analyzing customer usage of Joyent’s systems and noticed that you are one of the few customers that are still on our early products and have not migrated to our new platform, the Joyent Cloud.

For many business reasons, including infrastructure performance, service quality and manageability, these early products are nearing their End of Life. We plan to sunset these services on October 31, 2012 and we'd like to walk you through a few options.

We understand this might be an inconvenience for you, but we have a plan and options to make this transition as easy as possible. We’ve been developing more functionality on our new cloud infrastructure, the Joyent Cloud, for our customers who care about performance, resiliency and security. Now’s the time to take advantage of all the new capabilities you don’t have today. Everyone that’s moved to our new cloud infrastructure has been pleased with the results.

We appreciate and value you as one of Joyent's lifetime Shared Hosting customers. As this service is one of our earliest offerings, and has now run its course, your lifetime service will end on October 31, 2012. However, we believe that you will enjoy the new functionalities of the Joyent Cloud. To show you our appreciation, as one of Joyent's lifetime Shared Hosting customers, we'd like to offer you a free 512MB SmartMachine on the Joyent Cloud for one year. Use this promotional code to redeem the offer.

Promotional Code: xxxxxxxxxxxx

Please review the Terms and Conditions for the Joyent Cloud One Year Free 512 MB Machine Promotion by visiting this link.

To find out more about the Joyent Cloud and your options, please follow this link to our migration center for additional details.

Sincerely,

Jason Hoffman
Founder and CTO
Joyent
jason@joyent.com

It was surprising to get this message at all. It was shocking to get it from Jason. He was one of us, he'd been there since the beginning, and he'd helped so many of us. More than a few customers had been promoted to TextDrive or Joyent employees. More than a few people had visited San Francisco and had beers or tacos with Jason. It was a personal relationship that we trusted. And, suddenly, it was over. Our lifetime accounts were no longer good for the lifetime of Joyent.

We would have been willing to migrate from our existing servers to new servers, using Joyent's latest infrastructure and products. We realize that servers get old and architectures change. We'd already migrated once before and we were willing to do it as many times as needed, over the years. But that offer wasn't extended to us. Instead, we were invited to "upgrade" to a paid, monthly account using Joyent's latest products. That offer seemed insulting, to say the least.

It felt like a kick in the gut. It felt like a sudden breach with an old friend. We were thrown out, with a token promotional code and a mere 10 weeks to migrate all of our sites to a new server. The news spread quickly, the outrage spread faster, and by the end of Thursday, Jason was negotiating with his customers, over the best resolution for everyone involved.

Jason has now offered everyone their choice of either 5 years free hosting on a Joyent SmartMachine or a full refund of their original purchase. I haven't decided which option I'll take, but I'll need to decide soon and then start migrating my own sites.

It's the end of an era for me and I'm saddened by that.

Other Reactions

I'm Looking For a New Email Host

Joe Martin — Tue, 06 Jun 2017 16:39:34 -0500

I’ve been increasingly thinking of ditching my GMail account. I have a couple of reasons for wanting to do that but they mainly boil down to one thing: I don’t like how comfortable Google is with delivering products that don’t pay attention to details.

For instance:

You can’t rename a domain in Google Apps. You have to delete the domain and start over with the new name.
You can’t rename an admin user in Google Apps. You have to create a new user account and start over. (If the user was the initial user in the domain, it won’t even work to demote the admin user and then rename him.)
GMail likes to add a “Sender” header anytime you send a message from an alternate email address. You can prevent this (setup instructions) but it’s a bit clunky and requires you to have an external SMTP server.
GMail IMAP is wonky, making me reluctant to use my GMail account with a desktop email client.
You can’t use a Google Apps account with Google Plus.
On a more general note, Google Plus is overly dramatic in its requirement that people to use real names and not pseudonyms.

All of these things add up to give me a general feeling of using a half-baked product from a company that doesn’t sweat the details. And I don’t like it.

There are a lot of things that I do like about GMail.

I can host multiple personal domains.
I can have e-mail addresses setup at my domain (for family members) that either deliver mail to a GMail account or else deliver mail to an outside account (so that I can provide a “forwarding address”).
I get generous email quotas.
The service is free.
I get mobile push email.
Google provides a very good, nearly instantaneous, email search.
There is a great conversation view, in the web interface.
I really like the entire Inbox / everything else paradigm that Gmail created and I like being able to one-click Archive my messages.
I can create unlimited message filters to automatically file messages and mark them as read or starred, even when my computer isn’t running.
Robust spam filtering.

But, given that I no longer trust Google to provide a high quality product, I’m looking to move to a new email provider. I’m willing to pay for my service. Here’s the baseline feature set I’m looking for.

Let me host email for multiple domains.
Let me create multiple aliases either for a mailbox at the host or to forward to an outside mailbox.
Robust spam filtering.
Generous email quotas.
Server side message filters, to sort and file my email.

Here’s the features that I’d like to see.

A good, web-based, email search.
A web-based conversation view.
Push email, for mobile clients.

I'll be looking around to see who can sell me that, at a price that's reasonable for just 2-3 mailboxes.

My Kindle Wishlist

Joe Martin — Tue, 06 Jun 2017 16:39:34 -0500

Amazon is all set to announce an update to their Kindle product list, tomorrow morning. The rumor mill is pretty certain that this will involve the debut of some kind of a tablet device and not an update to their existing eInk Kindles.

I’m really only interested in the eInk Kindles. I like the book-like readability of the screens and I’m really not interested in reading from an LCD screen for a long period of time. I’ve owned every Kindle so far (each iteration of the 9” model) and I’m pretty pleased with how the hardware has developed. In general, in day to day use, I don’t have any complaints about my Kindle. But I do have a small wish list, for the upcoming 4th generation eInk Kindle.

I’ll probably upgrade even if Amazon doesn’t implement these features but I’d sure love if it they did implement some (or all) of them.

Easy Hardware Transitions

Apple has a very easy process for moving from an old iPod touch to a new iPod touch. Plug the new iPod touch into your computer, go into iTunes and choose the “Restore from Backup” option. That will move over all of your applications, settings, music, books, movies, TV shows, etc. In a few minutes (or a few hours, if you have a lot of media loaded) your new iPod touch will be setup identically to your old iPod touch.

Amazon has nothing remotely comparable to this, to make it easy to upgrade from an old Kindle to a new Kindle. There are two things going on here.

DRM books. If you’ve purchased books from Amazon, the Digital Rights Management locks those books to your specific piece of hardware. You can copy the digital files from the old Kindle to the new Kindle, but you won’t be able to open or read the books once they’re on the new Kindle.

The only way to get the books onto the new Kindle is to download them, one at a time, from Amazon. For digital pack rats, this is a colossal pain. The only saving grace is that the Kindle will remember which Collection the book was in and what your bookmark was, so you won't have to reorganize it once it is downloaded.
Non-DRM books (personal files). These books aren't locked to your specific piece of hardware, so you can copy them from one Kindle to another. However, the Kindle doesn’t remember anything about them, so you lose both your current bookmarks and your organization. After copying the books over, you can read them but you have to spend an hour or so putting everything back into the proper Collections.

Unique Screensavers

My wife and I each have a Kindle. They look physically identical and, in the beginning, it was easy to pick up the wrong Kindle when I was ready to read. Then I discovered the Kindle screensaver hack and we each loaded a single, custom, screensaver that differentiates our Kindles. Mine reflects my book personality.

This is something that shouldn’t require a hack to do. I’d like to see Amazon create an easy tool, even a web based one, to pick a photo and convert it into personalized Kindle screensaver.

I’ve heard that the Kobo eReader will default to showing the cover of whichever book you’re currently reading. I think this is a great idea too and it would really give each Kindle its own personality.

Better Magazine Management

I like reading magazines on my Kindle. I have a lot of back issues of Jim Baen’s Universe Clarkesworld, Orson Scott Card’s Intergalactic Medicine Show, and Instapaper. Unfortunately, magazines live in their own little world and can’t be added to any of your existing Collections. That’s a real pity, since I have a “Currently Reading” Collection that I use to organize everything that I’m, well, currently reading and it’s impossible to add my current (or past) magazine issues to that Collection.

Here’s an idea I’d love to see implemented: associate a magazine subscription to a Collection. Any new issues for that magazine should automatically be added to that Collection, ready for reading. But I should also be free to move back issues to a different Collection.

Fast Book Switching

This is another area where Amazon should take a cue from Apple. iOS allows you to double-tap the home button to get to a list of recently used applications. That allows you to quickly switch from one application to another, without having to go all the way out to the home screen.

I’m generally reading 4-6 books at any one time and they’re not always next to each other on the home screen. Right now, to switch between books, I have to press the “Home” button, look for the book on a specific page of the home screen or navigate to a specific collection, and then open the book.

It’d be nice if the Kindle supported some form of fast switching (a pop-up menu? a list at the bottom of the window) that would allow me to quickly toggle between various recently read books or magazines. The current system isn’t bad, exactly, but I think it could be improved upon.

Easier Collection Management

This is yet another area where Amazon could take a page from Apple’s playbook. I really like using iTunes to manage my iPod touch. (iTunes itself could use some work but that’s a topic for another post.) iTunes makes it easy to create playlists, organize media, load and unload applications, etc.

The Kindle has nothing comparable. The only way to manage your books, magazines, and documents is to do the management on the Kindle itself. It works but it’s exceedingly clunky and slow. It’s hard to manage multiple items at once and the screen refresh delays makes the experience an exercise in patience.

Amazon should provide some way to organize your content using a desktop or web application that can push your changes out to the Kindle itself.

Another option, maybe a better one, is to provide something for existing applications (perhaps Calibre) to hook into. Calibre is already a suberb application for organizing eBooks and its limited only by the fact that it can’t manage Kindle Collections. If Amazon provided a way for it to manage Collections, Calibre could easily become the iTunes of the Kindle ecosystem.

More Customized Books

Kindle books are too uniform. Amazon should provide a way for publishers to customize font faces, margins, line spacing, etc so that publishers can provide books with a unique look and feel. One of the joys of physical books is the look that each book can have. With the current Kindles, that joy is sadly missing.

Additionally, the Kindle needs to support more formatting features. For example, the Kindle doesn’t provide any way for a book to support block quoting. In many non-fiction books, it’s very difficult to separate out the block-quoted content from the author’s commentary. I think that needs to change in order for eBooks to reach their full potential.

ePub Support

Finally, I’d really like to see Amazon support the ePub file format. It’s the de-facto standard for every other eReader on the market. Right now, I have to keep all of my books in both ePub and MOBI/Kindle formats, to maintain compatibility with all readers. I don’t necessarily expect Amazon to distribute their own books as ePub files but it’d be nice to be able to load my other non-Amazon books as ePub files, without needing to convert them first.

WhisperSync Plus

I buy a lot of my eBooks directly from the publishers: Baen Books and Pyr, among others. I can load those books onto my Kindle and my iPod touch (running the Kindle app) but I can’t synchronize my last-read position between the devices. That only works for books that were purchased through the Amazon Kindle store.

I realize that WhisperSync is a big advantage for Amazon’s store and that they’re not likely to give it up just for my convenience. But what if Amazon created a "WhisperSync Plus" service, available as an annual subscription ($25 a year? $50 a year? part of an Amazon Prime account?) that would allow members to use WhisperSync for personal documents? I would gladly pay for that service and I'd like to think that other people would too.

Updating VideoHub: Design Needs

Joe Martin — Tue, 06 Jun 2017 16:39:34 -0500

I generate my video files from iMovie and dump them into a folder. I generally generate both a high and a low quality video. I need a way to painlessly:

Detect the video quality of the files
Generate a stub post for the file, with appropriate height / width values, and links to both the high and low quality files.
Generate separate RSS feeds for the high and low quality videos.
Generate the VideoJS embed code for the video, to include in the post.
Geneate a VideoJS embed file, suitable for embedding in an iFrame, so that the videos can be embedded on Those Martins or other sites.

I think I can do all of that with Jekyll but I need to learn it and figure out which pieces do what.

I think that each individual post should have some YML Front Matter to detail the metadata for each video. It may look something like this:

Video Data:

Name
Height
Width
High Quality File
Low Quality File
Preview Image

From that data I need to generate several files:

An embed file, including the HTML/JS code necessary to show the video in a browser. It should be suitable both the actual video posts as well as being used from an iframe, to embed the video on a different site.
The actual video post file
The low quality RSS feed
The high quality RSS feed

I can then edit the post, to add a description / narrative for the video. After that, I generate the static site and let rsync take care of pushing it to the server.

The Video Tag and Embed

Through Liquid Extensions, Jekyll has support for including fragments into a larger page. So, the embed files could be generated into _includes/video.html. Then, the post file could include it and the file for the actual embed page could also include it.

{% include video.html %}

Getting the Video Data

I’ve been looking at Jekyll plugins but I’m not sure that they’re right for scanning my video directory, looking for new videos, getting the video’s metadata and then generating the embed fragments. I might be able to make it work but it also seems more natural to just write a standalone program that will do all of that work and then let Jekyll handle the actual site generation.

Generating the RSS Feeds

It doesn’t look like Jekyll has a built-in method for generating RSS feeds. I found an example RSS template, courtesy of recursive. I’m thinking that I should just create two RSS templates: one to pull the low quality data from the posts and one to pull the high quality data from the posts.

Other Things

To make it easy to move the video files around, I need to have a global host / path prefix, for where the videos live. I should be able to define that in a site-wide configuration file, so that all of my URLs generate automatically.

Updating VideoHub: Introduction

Joe Martin — Tue, 06 Jun 2017 16:39:34 -0500

I have an aversion to depending on third party websites to serve my content. I don't mind using them (how else would I actually have a site on the internet, if I didn't have a web host) but I don't like being absolutely dependent on them.

So, YouTube has always presented me with a bit of a conundrum. It's very convenient but if I were to use it extensively and it were to change or go down, all of my hosted videos would disappear.

Several years, I created a site I called VideoHub to get around this problem. It was designed to do two things:

Show a new post for each family movie I created, allowing me to post them and allowing others to view them. It is, of course, RSS enabled so that you can subscribe to new videos.
Generate a podcast/vodcast compatible RSS feed, so that you could subscribe to the videos in iTunes and auto download them. For geographically remote family members, this is an absolute necessity.
Serve both a high quality and low quality vodcast feed. For geographically remote family members with poor internet connections, this is also an absolute necessity.

The site still works well for all of these goals. But it’s gotten a bit long in the tooth and needs both maintenance and enhancement. It needs maintenance because it doesn’t work well in a world of iPads and iPhones and is still heavily dependent on flash. It needs enhancement because it’s still too much of a pain to add new videos to the site.

I’m leaning towards rewriting the site in Jekyll. Because Jekyll creates static HTML pages, I can speed up the site eliminate a PHP dependency. I can code the site’s logic in Ruby, without having to actually run Ruby on my server.

I think can code scripts that will do much of the work of generating a new Jekyll post and then upload everything to the server. That will make creating a new post a much more fire and forget experience.

I’ll also switch to using VideoJS, or something like it, to embed the video files. This will give me direct MP4 playback for Safari, Chrome, and IE9+. I have no intention of creating, uploading, and hosting OGM versions of my videos, which creates a problem for users of Firefox and Opera. VideoJS will give me a Flash fallback for these browsers.

Now, I just need to find the time to make it happen.

We're Moving...

Joe Martin — Tue, 06 Jun 2017 16:39:34 -0500

We are moving... to a new site.

If you like reading about our family life (and why wouldn't you?), you'll need to update your bookmarks. We're moving to Those Martins.

All new family posts will be at Those Martins. wordflood.net will stick around but I'll be changing the content entirely, to be technology focused and not family focused.

So change your bookmarks (or feed readers) to point to thosemartins.us.