Administration Blog It is better than bad. Fri, 06 Feb 2015 13:59:49 +0000 en-US hourly 1 Google Mail: Operation Cold Shoulder Fri, 06 Feb 2015 12:48:14 +0000 Continue reading Google Mail: Operation Cold Shoulder]]> ATTENTION CONSERVATION NOTICE: For a short period of time, I’m cutting us off from Google Mail as much as possible.   You might not get mail you would like to get. and associated domains are increasingly being filtered and blocked by Gmail– to the point where it is now severely affecting the day to day operations of things that I am responsible for.

The reason for this is kind of complex and manifold.   First, in the middle of last year, a security hole in a WordPress install was exploited on our webserver, turning it into a spammer.  The break in, discovery, and patching all occurred on June 17th of last year, but hundreds of thousands of messages were attempted in that time, a fair percentage of which are to gmail addresses.

Additionally, we send a lot of email to Google Mail which it doesn’t like.   All of our active mailing lists include email addresses hosted there.  Some of those lists can be high traffic, which Google (according to research) doesn’t care for.   Also, many of you who have full accounts on our servers are blind-forwarding all mail you receive to Gmail addresses, which means we are sending every message you receive, including spam, to those servers.    As you might imagine, that’s not well thought of by Google.

So, for now, I’m minimizing the server’s interactions with Google for a little while.   I’ll be periodically testing to see how we’re doing, but I would not be surprised at two weeks.   This works out a few ways:

  1. If you’re forwarding your mail: I’ve turned that off.   For now, you’ll need to log into the webmail client and check for mail on your account here.   Once we’re talking to Google Mail again, I’ll need to show you how to set up filters so that you’re not forwarding spam to your gmail account.
  2. If you’re on a mailing list: I’ve temporarily disabled mail to your address.   Once testing shows that we’re improving– on the order of days, I hope, I’ll turn it back on and let you know.   In the meantime, if you’d like a temporary (or permanent) account on my server for getting access to the list, we can set that up.  Contact me: you’re either getting this via email, in which case you can reply, or you’re looking at the blog, where you can leave a comment.
  3. If you’re a user and you’re sending mail to Google Mail: It’ll might fail, and it’ll point to this post.

I really didn’t want to do it this way, but nothing else I’ve tried so far is working.   I’m at the point now where I can’t communicate with people on Google Mail, so I might as well take advantage of that.

I’m going over the mail server thoroughly (again) to make sure that we’re well configured, and I’m going to make some changes regardless.

]]> 0
Domain Screw-Up Thu, 01 Jan 2015 22:23:55 +0000 There was a foul-up with the domain today, so it was unavailable for most of it.   Mail, unfortunately, will have bounced.    It may take up to a day or so for new DNS changes to propagate, but it was solved this morning.

]]> 0
MariaDB 10.0 Installation Today Sat, 23 Aug 2014 23:03:50 +0000 Continue reading MariaDB 10.0 Installation Today]]> We’re doing a major database upgrade today.   The expected upgrade path is to dump all databases, upgrade, and then reimport.   Since this is essentially completely rebuilding the DB structure, I’m taking this time to bring the db server more in line with how we do things.   Actual downtime should be measured in minutes, but changes may be lost if they are committed today.

]]> 0
Flashcache Disabled, Indra Shutdown Sat, 22 Feb 2014 12:11:34 +0000 Continue reading Flashcache Disabled, Indra Shutdown]]> I’ve disable the SSD flashcache plugin, for a couple of reasons explained after the break.

More importantly, I’ve decided to shut down Indra, the virtual machine what provides shell access to people logging into   The reasons are very simple– no one uses it. has evolved since 1998, and where almost everything was done via shell back in the day, these days most people use web interfaces.   If they need shell, they use their local shells, for the most part.   Even I use a VM on my local network rather than log into indra.   So to save resources, and maintenance time on my part, I’m shutting it down.

Should we need to bring it back, it’d be relatively trivial.   SSH services (the only thing really handled by indra) will now be directed to kali, the web server.

Flashcache Rationale:

First and foremost, it was turning into a huge bottleneck for disk I/O.   Disabling it has improved disk performance on Copernicium significantly.  Secondly, I may be swapping out flashcache for bcache as it seems to have much better writethrough performance.


]]> 0
Webmail Updated, Fixed Thu, 20 Jun 2013 09:59:39 +0000 Continue reading Webmail Updated, Fixed]]> Looks like whatever made Apple block my domain is gone.   Hooray for that.

I’ve also updated webmail and associated interfaces.   You can check it out here.   Changes:

  • Upgraded to the latest software
  • HTTPS is now enforced
  • Mail client has HTML5 notifications
  • keyboard shortcuts
  • drag and drop upload
  • you can now change your mail password and forwards
  • and a bit more…


]]> 0
Mail to is being blocked Sun, 16 Jun 2013 00:31:12 +0000 I’ve submitted a question to the blocking authority, but no answers so far.  Please let people know.  I’ll update again when I know what the hell is going on.

]]> 0
Mail and Web Updates Wed, 12 Jun 2013 04:20:31 +0000 Continue reading Mail and Web Updates]]> New Mail SSL Certificate

I have purchased an SSL certificate for   So, in your mailer, as long as you use “” as your mail server, it will no longer complain about a self-signed certificate.

This also means that SSL is now fully functional for the webmail client.   Since we no longer have the self-signed certificate concern, I will be disabling webmail access via standard HTTP.   Finally.

Why hadn’t I heard of SNI years ago?

Web Server Update

The hard part of upgrading the server is done now.   So that’s good.    Aggregate downtime was about five minutes, hooray for virtualization.

I’m going to be adding some other things to the web server (in anticipation of wrapping the media streaming and distribution functions out of the media server), but that will have downtimes measured in milliseconds.

]]> 0
Web Server Will Be Slow, Intermittent Outages Tue, 11 Jun 2013 12:18:43 +0000 Continue reading Web Server Will Be Slow, Intermittent Outages]]> I’m upgrading the web server today.   In fact, the web server we’re using right now is the backup image of the actual web server.

There should be a couple of outage about a minute or two long, but web performance is going to be slow as I make changes in LVM snapshots, and make backup copies.

This should all be done by tomorrow, the 11th of June.

]]> 0
Database Server Updated Tue, 26 Mar 2013 11:32:36 +0000 Database server has been updated, which is why the web server was showing a maintenance page for a little under an hour tonight.   No data lost, mail queued, everything went fairly painlessly.

Next is the mail server.

]]> 0
Cn Got Weird, Upgrades Begin Mon, 11 Mar 2013 09:54:43 +0000 Continue reading Cn Got Weird, Upgrades Begin]]> Copernicium got strange.   Copernicium, or Cn, is the server in Arizona that all the main VMs run on.

I can’t say it crashed, because I rebooted it.   It just got hung up on a bad XFS filesystem and wouldn’t properly remount.   I needed to have someone at the facility go up and press the button.

Sort of my bad, but I was attempting to begin upgrading to fix some problems on the network, so I need to work on this.   Total outage about eight minutes.

UPDATE:  Now that that’s out of the way, I’m beginning site-wide upgrades.   I’ve done a test upgrade of of indra and it is working, so I’ll start with that one.

]]> 0