Thursday, October 10, 2019

LUC server back online(?)

Well, weirdly, my original old server seems to be back online. I'm going to continue this rebuilding process, especially because I was about to get into PHP, which is where my memories of what actually worked and what didn't become extremely fuzzy. But at least this means I have a viable version ready to go for DHCS in a few weeks!

Wednesday, October 9, 2019

H2B SOTU-db Step 3: Apache

This series of posts documents the process of getting SOTU-db back online after losing access to the LUC servers upon which it was originally hosted.

Now that I've got my VM provisioned and running R, the next step is to turn it into a web server by installing Apache. The goal for the completion of this step will be to visit and have it display a basic HTML page, which I'll use to post a temporary notice the SOTU-db is offline for now. 

Before I do that, I'm going to activate UFW, the Uncomplicated Firewall by following the steps on this Digital Ocean Ubuntu setup guide (starting with Step Seven). It's pretty... uncomplicated, so I won't go into detail here. Basically, I'm just allowing OpenSSH connections from anywhere for now, then after installing Apache we'll give it permissions as well. It looks like activating UFW this was successful, so I'll move on.

Fortunately, Apache is extremely popular and is easy to access from the standard Ubuntu package repositories. Installation should be as simple as:
sudo apt update
sudo apt install apache2
And indeed, the installation process went very smoothly. However, I was frustrated to see that I couldn't seem to access the "It works!" page that was supposed to be generated as part of the install. The file is in /var/www/html on the VM like it's meant to be, and I added "Apache Full" to the UFW allow-list. After some running around in circles, I eventually realized that I had left blank two checkboxes in the Google Compute Engine dashboard for my instance: under a small section called "Firewalls," I had left "allow HTTP traffic" and "allow HTTPS traffic" unchecked. So, I don't think my requests were ever even reaching my VM. Once I checked those boxes and restarted the VM, I had no problem accessing the "It works!" page.

Next, I opened up Atom and wrote a quick new index.html file, letting users know that SOTU-db is down and directing them to the dev blog.

I also promoted the VM's public IP address from an ephemeral to a static IP, then went into my Google Domains dashboard and added an A record linking the domain to that IP. This way, is connected to my VM and, unlike with the old LUC VM, should allow "sotu-db" to remain in the address bar as long as a user is on the site (instead of being redirected to like previously).

So, we're now up and running with
1. a VM
2. R installed
3. a static page served to visitors to Try it out!

Next steps will be to begin migrating the project data itself onto the server!

H2B SOTU-db Step 2: Installing R

This series of posts documents the process of getting SOTU-db back online after losing access to the LUC servers upon which it was originally hosted.

After getting my Ubuntu virtual machine up and running, I'll need to install some software onto the machine that is not contained in the SOTU-db project repo. The two main programs I'll need are the web server Apache and the statistical analysis package R. I've decided to install R first.

Installing R

Installing R is pretty straightforward once you know the fundamentals of installing software in Ubuntu. The command line installation guide provided by CRAN (the organization that distributes R) is pretty helpful. I'll walk through it here:
  1. add the CRAN repositories to sources.list
Basically, this step tells Ubuntu about the remote servers where it can find R. So, first I'll open an SSH shell into my VM, through the Google Cloud Engine dashboard. Then, I'll navigate over to the folder I want with cd /etc/apt/. Now comes the tricky part... which text editor do I use? How long until I get stuck or accidentally erase the whole file? Let's try nano... sudo nano ./sources.list. That works... the Cloud Engine console even allows me to paste from the Windows clipboard with CTRL+SHIFT+V. So, I'll just plop in the source listing that corresponds to my version of Ubuntu:
deb xenial-cran35/
and save, or CTRL+O for "WRITE OUT" just to remind me why I hate Linux text editors, then CTRL+X to exit, and we should be good on step 1!

STEP 2: Install the "Complete R System"
This step is really simple:

sudo apt-get update
sudo apt-get install r-base

STEP 3: Go back a step, add security keys and another repo

This is where the R setup process always drives me crazy, and is an issue with countless Linux installation guides: running the apt-get update command from Step 2, above, results in a warning that the public key for the CRAN repo that we added in Step 1 is not available. If you scroll down the R installation guide, it tells you how to address this, and also mentions that "Installation and compilation of R or some of its packages may require Ubuntu packages from the “backports” repositories..." So:

STEP 3A: Add the Backports repo

I'm adding this to the same sources.list file as before, which now has two entries that look like this:

STEP 3B: Add public keys for R packages
Keep this step simple by just running
sudo apt-key adv --keyserver --recv-keys E298A3A825C0D65DFD57CBB651716619E084DAB9
Now we can go back to Step 2 and run those commands again. I'm actually still getting a couple warnings on the update, but they seem ignorable (such as backports not having a "release" file, which seems to make sense to me). And install r-base ran without any errors.

STEP 4: Install r-base-dev
Finally, we're going to install some additional developer packages. The guide says "Users who need to compile R packages from source [e.g. package maintainers, or anyone installing packages with install.packages()] should also install the r-base-dev package," and that's us, so here we go. This should be a simple matter of running
sudo apt-get install r-base-dev
Hm, interestingly, it's telling me I already have the newest version of r-base-dev installed. I wonder if adding in the backports repo before installing r-base had something to do with it?

STEP 5: Profit
Now I can run R (not r) and see that the program runs successfully. Next stop, Apache!

How to build SOTU-db: Step 1

This series of posts documents the process of getting SOTU-db back online after losing access to the LUC servers upon which it was originally hosted.

Step 1: Get a VM

The first step is actually procuring a server upon which to run SOTU-db. I looked at Azure and Google and decided on Google, partially because they have a simpler interface and a good amount of credits for new users. But I also wanted to see about using Firebase more, and figured that integration would be easier if I used Google for the VM (they call this service the Google Compute Engine). 

I decided to start low in specs, figuring I could scale up if needed, so right now I just have a single VM running Ubuntu 16.04 with 1 vCPU and 3.75GB memory. I assume I'll need to scale that up, but am hoping I can do that after getting more of the platform set up properly. Since initializing the VM, the only steps I have taken have been:
  • run apt update & upgrade
  • point my domain to the VM's public address, which I should probably undo and point to this blog until at least getting the web server running
Step 2, coming up, will be to install either Apache or R; I'll probably glance through the documentation of each to see if doing those in a particular order will be helpful, but I don't think it will really make a difference. 

Tuesday, October 8, 2019

Fall 2019 Update

Good news: I've been invited to present SOTU-db at the 2019 Chicago Colloquium for Digital Humanities and Computer Science!

Bad news: SOTU-db is offline, as our host servers at Loyola seem to have been taken offline.

This means I have about one month to get SOTU-db back up and running. Ideally, I would also like to ask and address another contemporary research question, but that might not be realistic given the timeframe.

So, the plan going forward is going to be to essentially rebuild SOTU-db's web presence on a new VM, document the process here in blog posts, and get everything up and running again by November 9. Stay tuned!

Thursday, February 28, 2019

February 2019 Development Update

It's been an eventful couple of weeks for me, in Chicago, and in State of the Union world, but not so much for SOTU-db.

My new position at the Chicago History Museum is in full swing, including some recent and upcoming weekend events. It's also been a wild few weeks of weather as the "polar vortex" and plenty of snow have blanketed Chicago. Together with a busy year in my personal life, this has prevented me from getting as much work done on SOTU-db in recent weeks as I would have liked.

Sadly, this work-slowdown coincided with one of the more interesting periods of SOTU news in recent memory. Originally scheduled for January 29, President Trump's 2019 "State of the Union" address was postponed when Speaker Pelosi delivered a Jan. 16 letter to Mr. Trump postponing the address until the conclusion of the then-ongoing partial federal government shutdown. By January 25, the federal government was fully re-opened (quite possibly temporarily - stay tuned) and the SOTU was quickly rescheduled for Tuesday, February 5.

"The Daily" podcast by the New York Times ran an episode about the State of the Union the morning after Trump's SOTU, which included this wonderful exchange:
Michael: "I really don't like this language people use: 'SOTU.'"
Mark: "Yeah, it's really awful --"
Michael [crosstalk]: "awful, so let's not use it..."
Mark: "... it's kind of a dismal evening anyway, and calling it SOTU just sort of deepens the gloom that hangs over the whole thing."
So that was fantastic - but the overall episode tried to address how the last four POTUSs have responded in their SOTU addresses to losing their party's majority in Congress. It was definitely worth a listen and speaks to one of the original questions SOTU-db was meant to address: do State of the Union addresses actually matter, and is there a correlation between what a president says in a SOTU with their actual policy decisions? The podcast suggests that they do matter, and that the correlation does exist.

Sadly that must be it for this post - it's already Feb 28 so I'm out of time for my monthly update. I'm hopeful that I can get back to a more regular SOTU-db update schedule once History Fair season wraps up for this year.

Tuesday, January 22, 2019

January 2019 Development Update

A quick update on SOTU-db:

  • SOTU-db's servers were offline for several weeks around the new year, but have now been restored
  • I successfully defended SOTU-db as my Master's capstone project, and have now completed my MA in Digital Humanities
  • I recently took a new full-time position working with the Chicago Metro History Fair (blog post about the new role here)
  • Due to a combination of the above three factors, development of SOTU-db has been at a standstill in recent weeks
I'm hoping to resume work on SOTU-db now at a pace that reflects its status as a hobby/passion project. I'm not committing to further development milestones right now, but I plan to post on this blog at least once a month with the latest updates. I will likely also be making much heavier use of commit messages and version tags on GitHub. I still fully intend to bring SOTU-db into a "full release," but it could be quite a while before that happens.

It's not realistic to think many more features will be completed or bugs will be squashed before the 2019 SOTU address, whether postponed due to the partial government shutdown or not. Nevertheless, I would like to be able to promote the site via hashtags whenever the SOTU actually is delivered, so for now I think a focus on clarity and usability will be my main focus, so that new users can figure out what the site is doing.