Words

I didn’t take notes on how I was building willangley.org in 2019, because I was only planning to build it once; I planned to deal with things going wrong with backups, and things going right with in-place capacity upgrades.

Now it’s prompting me to upgrade, from Ubuntu 18.04 LTS to Ubuntu 20.04 LTS. It’s sometimes necessary to rebuild servers when an in place upgrade doesn’t work, so I might need to build this twice after all; restoring to before a failed upgrade is likely to bring you to a state where the upgrade will fail again .

This is a good reminder to figure out what I did and write it down, even if I don’t upgrade immediately.

Development environment

I’ve been using Local to run WordPress on my machine since before I knew if I’d go live with a WordPress site, or how I’d serve it if I did.

I don’t develop PHP at work any more and lean heavily on an editor (usually either Sublime Text or Visual Studio Code) to remind me what I’m looking at, and this makes them work well.

I’ve tried to make my site portable between both my server and Local.

Server

I mostly followed the 2019 version of Hosting WordPress Yourself, with some exceptions that follow below.

Machine

willangley.org runs on a single DigitalOcean Basic droplet, with DigitalOcean backups taking weekly snapshots on the server.

PHP

I’m using PHP 7.2, since it’s packaged with Ubuntu, rather than adding a PPA to get a more modern version.

I’ve also added several extensions to PHP to get one plugin or another working. Running dpkg -l | grep php says I’ve got: php-common php-igbinary php-imagick php-intl php-pear php-redis php7.2-bcmath php7.2-cli php7.2-common php7.2-curl php7.2-dev php7.2-fpm php7.2-gd php7.2-imap php7.2-intl php7.2-json php7.2-mbstring php7.2-mysql php7.2-opcache php7.2-readline php7.2-soap php7.2-xml php7.2-xmlrpc php7.2-zip pkg-php-tools installed now, and will likely need them again on a new server too. (Some, but not all, of them are installed by the base PHP install.)

SSH

I followed the guide , set up an SSH key pair, and turned off passwords.

Since then I’ve installed Tailscale on the server, and firewalled off SSH from most of the Internet. Then Jetpack complained, so I opened up SSH back up for their IP address ranges only.

This firewall is applied at the DigitalOcean Networking level, rather than running on the droplet. It’ll apply to my new server automatically when I add it to the web-serving firewall group.

DNS

DNS for my domain lives on Google Domains, but my server lives in DigitalOcean.

In Google Domains, I have an NS record for do.willangley.org that delegates them to DigitalOcean, and I have A records for @ and www that point directly to the IP address of my server in DigitalOcean.

In DigitalOcean, I have an NS record for DigitalOcean, and an A record for the server running willangley.org.

I’d been trying to avoid troubleshooting mail forwarding for @willangley.org by moving as little as possible, because I use a @willangley.org address as a login for some services. But this split setup is seeming like a mistake in hindsight and I’ll likely back it out.

Monitoring

Is set up through the DigitalOcean agent. CPU looks good,

memory meh but probably fine.

Object Cache

I skipped using Redis initially, then came back when I installed the AMP plugin and it asked for an object cache.

I used WP Redis instead of Redis Object Cache like the tutorial recommended, because I wanted to use Redis on a Unix domain socket and Redis Object Cache doesn’t support that. Unix domain sockets mean one less thing to manage passwords and firewall rules for.

Page Cache

I set up WP Super Cache for page caching, instead of handling caching in nginx and using Nginx Cache like the tutorial recommended.

I can’t remember why I did this but it might’ve had something to do with using Local for development instead of a copy of the server.

Email

I ignored the part about sending email from a WordPress plugin. This makes a lot of sense if you’re hosting WordPress for clients, and want them to get WordPress emails and you to get system emails.

But I’m not; I want both WordPress and system emails to go to me. So I followed the Sending email with SendGrid instructions to send mail from Postfix instead.

Backups

I ignored the part about tarball backups, and signed up for Jetpack backups instead.

Since I was already planning to use other Jetpack features this made sense to me. It still largely does, although it’s never fit in super well with Local (protip: Local will create a database for you when it’s creating a site, but the .sql script that Jetpack Backups gives you expects not to have this. Drop the tables in phpMyAdmin, then run the script, and things will work cleanly) and iterating on my theme a few weeks ago didn’t seem to prompt real-time backups.

Automatic security updates

I’ve configured unattended-upgrades to perform security updates and to restart automatically. I did not request a specific reboot time, since I’d rather have brief downtime than run an old kernel for a day until the time arrives again.

Additionally, I’ve set it to use minimal steps to avoid blocking shutdown, and to mail me on upgrades.

Unattended-Upgrade::MinimalSteps "true";

Unattended-Upgrade::Mail "root";

This reminds me that my server is taking care of itself, and if something goes wrong I’ll hopefully have an email with the bad upgrade in my inbox before an email from Jetpack downtime monitoring telling me that things are broken.

WordPress build

I did this locally first, rather than starting with wp-cli on the server, and copied it over later. I’ve got some plugins with config settings in them, and a customized theme, that both seemed easier to develop this way than across the Internet.

The smallest Basic droplet is powerful enough to serve a pretty good amount of traffic, but Visual Studio Code Remote Development will bring it to its knees in a hurry.

I disabled comments before serving my site. This isn’t a permanent decision – I think they can add value – but I’m only going to think about enabling them if I get to blogging regularly enough I think I’ll be able to sustain a community.

Jetpack is enabled. I’m actively using Anti-Spam (on the contact form only), Backups, Brute Force Login Protection, Contact Form, Downtime Monitoring, Image Optimization, Scan, and Stats.

Automatic updates are turned for WordPress, plugins, and themes too.

Planning to build on 20.04

Now that I’ve poked at this a bit I’m pretty sure it’ll translate to the new version of Hosting WordPress Yourself, Set Up a DigitalOcean Virtual Server for WordPress on Ubuntu 20.04.

Even though 18.04 will be supported for another two and a half years, I’m hoping to go to 20.04 sooner. WordPress is already recommending PHP 7.4, and I’d rather upgrade than spend time getting backports working.

But this time I’ll write down what I’m doing so I don’t have to reverse engineer it all over again when 22.04 comes out.

willangley.org, and some short-lived predecessor blogs, ran without a server for about a decade total: intermittently between 2004 and 2009, and then steadily from 2014 through the middle of 2019. Most of this time was spent running as a static site with Jekyll on GitHub Pages.

I really enjoyed writing in Markdown. Sublime Text is a great Markdown editor; Marked 2 is a great Markdown previewer. You can get a lot of writing done quickly. And the environment really does make it easier to concentrate by hiding formatting controls away.

But I hit a wall almost as soon as I wanted to share something that wasn’t writing.

Hitting a wall

I’d picked up an old Pentax K1000 and started taking photos on film in 2015, and by 2018 I had several albums that my friends had enjoyed seeing in person and I wanted to share with the world.

In September 2018 I spent three days putting an image gallery on my site to share the photos for one roll of film, and was incredibly frustrated at the end.

I had no idea how to extend Jekyll, since I had never learned Ruby. I thought “well, let’s switch to something I can extend,” and knew Go, so I spent two days migrating my site to Hugo on essentially a reflex.

This was a bad decision, and although I didn’t realize it, it was documented as such at the time. I spent the third day in a tarpit of slow thumbnail rebuilds as I kludged together a {{< gallery >}} shortcode to scale my scanned photos down to thumbnails and show them as a film strip.

I checked in the images, realized I was going to run out of room in the GitHub repo that held my site if I shot a roll of film every weekend for nine months, and then put off dealing with it until spring 2019.

Searching for something better

I spent a long time looking in Spring 2019, and ran down a lot of dead ends: letting myself have ideas about things I wanted, not writing them down, trying an approach that might fulfill one idea, and then realizing I’d hedged myself out of another thing I wanted.

I talked to some very smart engineers at Google as I was looking, and it didn’t help; we’d all read the same things that had steered us wrong.

Solving my problem

I got out of this by talking to different people, who I knew from outside of work, and who made websites for a living. They used WordPress.

It was hard to get myself to try this, because it’s got tons of what Silicon Valley culture doesn’t like in a program. It’s 20 years old, complicated, made of PHP, and you need to run a server to use it. And I’d personally spent some years saying “friends don’t let friends run WordPress,” after helping recover a hacked WordPress site and move it to Squarespace.

I was wrong to repeat that. Even if I couldn’t teach someone how to run their small business from WordPress in 2014, I might be able to learn enough to run my personal website in 2019. And not trying it was feeling increasingly like a mistake of values – like not making a painting because someone might rip the canvas.

I found a guide on Hosting WordPress Yourself that looked reasonable, and it worked.

When I got stuck theming the website, I asked my friend what they did for that. I was a little surprised by their answer, so I asked at the local WordPress meetup‘s help desk, and they said the same thing: use the Genesis Framework. So I did.

That worked too. Even though I hadn’t written PHP for 10 years, I was able to finish theming my site within a day.

Retrospective

I caught up on all of the photo albums I was sitting on and then some, and my site’s been running smoothly ever since: downtime monitoring says my site’s gone down for an hour and a half of downtime in a year and a half of running, so I’m north of 99.9% uptime.

There’ve been a few kinks, mostly in the spaces I know I didn’t quite follow the guides that I was working from; now that my server is prompting me to upgrade it I need to figure out what I did, write it down, and get ready to build it again.

But even with that, not-running a server cost me more than running a server ever has.

Why?

I wanted to get to my WiFi router from the Internet, but didn’t want it serving the Internet. Putting it on a VPN is a good way to do this, and Tailscale makes VPNs easy enough to run and use that there’s no good reason not to.

When?

This past weekend. I’d just gotten back to Brooklyn and installed OpenWRT 19.07, which included a new enough kernel to support Wireguard.

This was possible before – Tailscale’s been shipping ARM builds for a while, and OpenWRT’s been shipping 19.07 for a while too – but I didn’t try it before now.

How?

I get out a MacBook Pro that I’ve already set up Tailscale on, and read through the Tailscale docs for setting up Tailscale with static binaries to know how the install is supposed to flow.

It probably won’t be possible to follow them exactly, because OpenWRT is different, so I start playing around.

Installing Tailscale on the router

I look up my router’s hardware specs, then find the tarball for my router in Tailscale’s stable track, and download it to my machine:

% curl -O https://pkgs.tailscale.com/stable/tailscale_1.0.5_arm.tgz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 11.9M  100 11.9M    0     0  3433k      0  0:00:03  0:00:03 --:--:-- 3433k
%

I’m not sure how much free RAM and flash I have on my router, but I know I haven’t installed anything else, and guess that 11.9M is relatively small enough that I don’t need to worry about it.

So I copy the tarball over:

% scp tailscale_1.0.5_arm.tgz root@192.168.1.1:/tmp

then SSH to the router and try to unpack the tarball:

# cd /tmp
# tar xvf tailscale_1.0.5_arm.tgz
tar: invalid tar magic
# 

and it doesn’t work. OpenWRT uses busybox tar, so I fumble around for a bit (xkcd.com/1168 applies) and eventually figure out:

# tar x -zvf tailscale_1.0.5_arm.tgz

Then I try running the binaries to make sure they’re actually the right ones 😛 :

# cd tailscale_1.0.5_arm
# ./tailscale version
1.0.5-g31b5dec0a
# ./tailscaled --help
<...>

Looks it. Then I try to start tailscaled for real to figure out its dependencies.

# ./tailscaled
logtail started
Program starting: v1.0.5-g31b5dec0a, Go 1.14.4-ts56db765: []string{"/usr/sbin/tailscaled"}
LogID: 025f6648f94bcd1d40b0ee005ac472e123f7aeb2813874a71a7ba121da3d1827
logpolicy: using system state directory "/var/lib/tailscale"
2.6M/14.9M Starting userspace wireguard engine with tun device "tailscale0"
2.9M/16.6M Linux kernel version: 4.14.195
2.9M/16.8M is CONFIG_TUN enabled in your kernel? `modprobe tun` failed with:
2.9M/16.8M CreateTUN: can't create TUN device; /dev/net/tun does not exist
2.9M/16.8M wgengine.New: can't create TUN device; /dev/net/tun does not exist
flushing log.
logger closing down
logtail: dialed "log.tailscale.io:443" in 415ms
logtail: upload: log upload of 659 bytes compressed failed: Post "https://log.tailscale.io/c/tailnode.log.tailscale.io/0055d7ffaa6c80ecda7fef05574c36d025a42ead7d7409fce3b7218f84628e4a": x509: certificate signed by unknown authority
logtail: backoff: 12 msec
logtail: dialed "log.tailscale.io:443" in 413ms
logtail: upload: log upload of 659 bytes compressed failed: Post "https://log.tailscale.io/c/tailnode.log.tailscale.io/0055d7ffaa6c80ecda7fef05574c36d025a42ead7d7409fce3b7218f84628e4a": context canceled
#

There are many lines of error messages but really two errors:

• `modprobe tun` failed means that Tailscale asked the kernel for a TUN device and got back “huh?”
• x509: certificate signed by unknown authority means that Tailscale tried to make a TLS connection but couldn’t figure out who was on the other end. When this happens with professional software (like Tailscale) it usually means no certificates are installed at all.

Both of these are pretty standard things to get from a package manager, but everyone calls them different names. I poke around OpenWRT’s packages and determine I probably need kmod-tun and ca-bundle, so I install them and try again.

# opkg update
# opkg install ca-bundle kmod-tun

After this, tailscaled is able to start:

# ./tailscaled
<...>

so I copy it to Flash.

# cp tailscale tailscaled /usr/sbin

Running as a service

Tailscale comes with a simple systemd unit file. OpenWRT can’t use this directly, since it uses procd instead. I read the procd docs and, after about an hour of trial and error, port enough of it to get things working:

It’s my first time working with procd, so I’m pretty happy with how it turned out.

Gists have awkwardly long URLs, so I make a redirect to it that will let me download it with a short URL and upload it to my router easily:

% curl -LO https://willangley.org/tailscale-procd
% less tailscale-procd
% scp tailscale-procd root@192.168.1.1:/tmp

I SSH to the router, install the init script, and run it to start tailscaled.

# cp /tmp/tailscale-procd /etc/init.d/tailscale
# chmod +x /etc/init.d/tailscale
# /etc/init.d/tailscale start

Log in to Tailscale

Once tailscaled is running, I run

# tailscale up

to get a login link, and click it to log in.

After logging in, I go to the Tailscale admin console and look for my router’s hostname; since I’ve never changed it, it’s OpenWRT.

I copy its IP address and make sure I can ping it from the machine I’m working from. (This would fail if the machine wasn’t on Tailscale, or was on a different Tailscale network.)

% ping 100.114.61.77
PING 100.114.61.77 (100.114.61.77): 56 data bytes
64 bytes from 100.114.61.77: icmp_seq=0 ttl=64 time=43.378 ms
64 bytes from 100.114.61.77: icmp_seq=1 ttl=64 time=7.895 ms
64 bytes from 100.114.61.77: icmp_seq=2 ttl=64 time=7.046 ms
64 bytes from 100.114.61.77: icmp_seq=3 ttl=64 time=5.387 ms
^C
--- 100.114.61.77 ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 5.387/15.926/43.378/15.875 ms

Once I confirm the connection works, I disable key expiry for the router to avoid it falling off of my network.

Surviving reboot

Now that I know it’s working, I enable the Tailscale service so it launches on every boot.

# /etc/init.d/tailscale enable

And check to make sure it has succeeded:

# ls /etc/rc.d/*tailscale
/etc/rc.d/S80tailscale

This took me several tries to get right, because I forgot to add the START=80 variable to my init script; without it, OpenWRT didn’t know where to link it into the startup order.

But now that it’s there, I’m good to go! At least until the next sysupgrade, when all of this will get wiped away.

Future Work

There’s an open Tailscale issue, tailscale/tailscale#724, to package Tailscale for OpenWRT. If this happens soon, there won’t be much to do here 🙂

If it takes longer, I’ll probably try to get OpenWRT to not wipe this away with a sysupgrade – I think this’ll involve some combination of editing /etc/sysupgrade.conf and using sysupgrade -k on future upgrades.

I thought about building custom OpenWRT images with Tailscale pre-authenticated keys baked in, too; Google Cloud Build makes systems like this easy to spin up nowadays, and I think I could do it in O(days) worth of work from experience. But it’s probably overkill unless I wind up with a fleet of these somehow.

Appendix

I broke my router’s firmware once doing this.

It wasn’t even a step that I needed to do; I was trying to figure out how much space I’d consumed with packages, and ran a command from a snippet in a Google search that was supposed to remove unused packages without clicking it to read the rest of the page.

The snippet looked reasonable, but if I’d clicked in, I would have seen the command I was about to run was going to remove the WiFi drivers.

But as I said, I didn’t, and wound up needing to rummage for a USB-to-Ethernet adapter and Ethernet cable to reinstall my firmware.

It’s easy to search for resume tips, but too many of them come back; it’s unlikely they’re all right.

So I started looking for scientific papers in the area to see if anyone had studied it systematically. Eventually the Google keywords [evidence based job search] turned up a review paper that looked useful and that I had access to:

Liu, Songqi, Jason L. Huang, and Mo Wang. 2014. “Effectiveness of Job Search Interventions: A Meta-Analytic Review.” Psychological Bulletin 140 (4): 1009–41. https://doi.org/10.1037/a0035923.

Working outwards from there, I’ve found that we’re pretty sure about:

Writing

Competency statements (Bright and Hutton, 2000) work. They work well enough that copy-pasting them into resumes meaningfully boosted how managers rated them in a systematic survey.

Length

I’ve not been able to find a decision rule for “how long should I make my resume,” but it’s increasingly seeming like the answer is “as short as possible.”

Recruiters sometimes only turned out to have shortlisted one page resumes (Lipovsky, 2014), and studies that varied design (Wilgenburg, 2017; Uskaurs, 2018) used one page resumes exclusively.

Bright and Hutton, which varied content and left design fixed, used two page resumes and a cover letter.

No one asked recruiters to look at three page resumes, presumably because the response rate would have been too poor to publish.

Design

Design risks rarely paid off when studied, although the relationship between design and outcomes is difficult to describe. Reportedly, it may depend on:

• fashion, with (Arnulf et al., 2010) finding creative resume designs uniformly underperforming traditional ones, and later studies turning up more qualified answers
• intelligibility, with (Lipovsky, 2014) finding recruiters selected organized resumes
• the length of time a recruiter looks at your resume, with shorter review times favoring splashy resumes (Wilgenburg, 2017)
• the position applied for, with creative positions favoring creative resumes, and others favoring restrained, one column layouts (ibid; Uskars, 2018)

The best we can say is there’s more evidence for investing time in writing and proofreading than in visual design. Unless you’re applying for a design position, focus on making an easy to follow resume, with one column or one column and a sidebar, and move on.

References

Arnulf, Jan Ketil, Lisa Tegner, and Øyunn Larssen. 2010. “Impression Making by Résumé Layout: Its Impact on the Probability of Being Shortlisted.” European Journal of Work and Organizational Psychology 19 (2): 221–30. https://doi.org/10.1080/13594320902903613

Azrin, N. H., T. Flores, and S. J. Kaplan. 1975. “Job-Finding Club: A Group-Assisted Program for Obtaining Employment.” Behaviour Research and Therapy 13 (1): 17–27. https://doi.org/10.1016/0005-7967(75)90048-0.

Bright, Jim E. H., and Sonia Hutton. 2000. “The Impact of Competency Statements on Résumés for Short‐listing Decisions.” International Journal of Selection and Assessment 8 (2): 41–53. https://doi.org/10.1111/1468-2389.00132.

Higgins, Chad A., and Timothy A. Judge. 2004. “The Effect of Applicant Influence Tactics on Recruiter Perceptions of Fit and Hiring Recommendations: A Field Study.” The Journal of Applied Psychology 89 (4): 622–32. https://doi.org/10.1037/0021-9010.89.4.622.

Lipovsky, Caroline. 2014. “The CV as a Multimodal Text.” Visual Communication 13 (4): 429–58. https://doi.org/10.1177/1470357213497869.

Liu, Songqi, Jason L. Huang, and Mo Wang. 2014. “Effectiveness of Job Search Interventions: A Meta-Analytic Review.” Psychological Bulletin 140 (4): 1009–41. https://doi.org/10.1037/a0035923.

Uskaurs, Matiss. 2018. “The Impact of a Resume on a Job Interview and First Impression.” Edited by Thomas Sabel. Vaasa University of Applied Sciences. https://www.semanticscholar.org/paper/834c681f76b20b6928f17f3218834c7aa5941e05.

Wilgenburg, Florence van. 2017. “The Impact of Creativity in Resume Shortlisting.” Edited by dr Ruud Koolen. Masters in Communication and Information Sciences, Tilburg University. http://arno.uvt.nl/show.cgi?fid=145069