Gmail and Mail.app on Mavericks – A Middle Ground

There’s been a lot of complaining about how Mavericks breaks Gmail compatibility in the built in Mail application from the likes of Gruber and Marco based on Joe Kissel’s piece on TidBITS.  Kissel eventually decided to switch away from Gmail as he outlines in this follow up tutorial.

Some background: I’ve been using Gmail and Mail.app together for many years, since before Gmail offered POP3 or IMAP support.  My setup has been very stable and reliable over the past 7+ year that I’ve used it, so I thought it would be a good time to share how it works.  It came about because I wanted to be able to access my Gmail messages via several different computers but they had yet to offer support for the IMAP protocol (or even POP3 at that point).

I created a separate email account on a server that supported traditional IMAP.  I used a private domain hosted from DreamHost, but any mail server with IMAP support will work.  You also don’t have to worry about spam filtering or making sure your domain is not black or gray listed by the big guys (a common problem with self hosted email) because this account will only be used by you.

Once I had my own IMAP compatible email server, I set Gmail to forward all messages to this private account and then archive its own copy.  In a sense, Gmail is being used as an intermediary and front-end for my privately hosted email back-end.  Finally, I set all my mail clients like Mail.app on my Mac and iPhone to use my private IMAP server for incoming mail while they used Gmail’s SMTP servers for all outgoing mail.

And that’s really all you have to do.  People can email you at your Gmail or Google Domain hosted email address just like they always have, but you’ll get their messages from your standard IMAP server.  Outgoing email, since it’s sent via Gmail SMTP, shows all the standard addresses and headers from Gmail so recipients can’t tell the difference. It’s also archived to your Gmail Sent Mail folder/tag.

Basically, it’s a nice middle ground approach to using Gmail on your Mac: more convenient than a full switch to another email provider and more sane than Gmail’s own IMAP.

Some of the benefits of this setup:

  1. No addresses to change, use the same email addresses you always have.
  2. All sent and received items are archived in your Gmail account for reference later.  If you ever can’t find a local copy of an email, it’s easy to log into the Gmail web interface and dig it up from your full history.
  3. Similar to point #2 you can be a little more aggressive about cleaning and purging your inbox, knowing that Gmail has a backup copy of your email.
  4. Since your email will come via Gmail, it’s very unlikely your messages will be filtered by spam blocking tools.
  5. If Gmail ever disable’s IMAP support, this setup will continue working perfectly.
  6. You get Gmail’s spam filtering before things reach your inbox.
  7. You can use this to combine multiple external addresses into one account, and, with, some server-side rules on your IMAP host, filter messages accordingly.

 

Some of the downsides:

  1. You need somewhere to host your IMAP inbox, if you already have a web server this should be pretty easy but otherwise you’ll need to pay a few dollars a month for a service to host it for you.
  2. There are 2 points of failure between getting your email.  This hasn’t been a large problem for me, but if you had frequent downtimes on your IMAP host it could be.
  3. It’s a bit confusing to set up because you can’t use the Gmail setup assistants in OS X and iOS.  Rather, you can use them but you have to modify the domains later.  Alternatively, once you set it up on your Mac you can sync the account settings to iOS.
  4. You loose access to the extra features Gmail puts on top of email, like tags.
  5. The Gmail web interface will not reflect the state of your inbox.  You’ll have to use the one from your IMAP host, but that one might not be able to send messages via SMTP to Gmail (this depends on your provider).
  6. If Gmail ever disables forwarding, this setup will break.  I consider this less likely than them removing IMAP support.
  7. It’s harder to use Gmail-centric email clients like Mailbox with this setup, but that’s true if you just run your own IMAP server too.
  8. The NSA is going to be able to read your mail even easier.

 

Update: Apple released Mail Update for Mavericks as I was typing this post.  I think it should fix some of the Mail.app bugs and regressions in the latest OS.  Instead it should now work just like things did in Mountain Lion.  However, this doesn’t fix the non-standard IMAP implementation Gmail provides, so I still recommend my solution.

ToWatchList API & TWL_Downloader

I’ve been working on the ToWatchList API recently and I’m happy to announce the public availability of the basic API.

This API allows you to send URLs to the ToWatchList scanning tool.  The scanner will then parse the page and report back numbers on how many videos were found.

I’ve also created a GitHub project called the TWL_Downloader which can automatically download the latest videos you add on ToWatchList.

If you use the API or scripts for anything useful, please let me know.

Mr. Reader Now Supports ToWatchList

I’m pleased to announce that the iPad RSS reader client app, Mr. Reader now supports ToWatchList.  This allows you to send links and posts from your RSS feeds to be scanned by ToWatchList for videos without even leaving the reading interface.

Mr. Reader is available in the iTunes App Store. The changelog for the the new release (version 2.1) is also available.

Developing using Coda 2 & Git – Part 2.5: Background Info & Coda Issues

In Part 2 of my posts on using Coda 2 and git, I promised more info on how to get them working well together. I’ve spent the past few months working on my site and trying to get them working, but, to be honest, I haven’t had much success.

The problem isn’t any config issue, it’s just that Coda 2 only will integrate with locally hosted git repositories. So if you develop your site by running a webserver on your local machine, you’re fine. I don’t know how common this is; it’s certainly something I could do technically but it brings its own technical hurdles: Do you also host your database locally as well? Does everything work the same when you put it back up remotely? I’m not really satisfied with local-hosting my dev work because I don’t have good answers to those questions.

What Coda 2’s git tools really needs in order to be useful to me is support remote git repositories.

One workaround I’ve used is to work on my site within a git repo locally and then frequently upload the content to the remote dev server (which is not tracked by a git repo). The problem with this is you’re constantly saving and then taking another action to push out your changes and see what they look like. Frankly, it’s a pain. It would be much easier if you could tell Coda 2 to just upload your file as soon as it’s saved locally, and indeed this feature is on a Panic Survey for possible future Coda development.

In the meantime, I’ve been learning to manipulate git repositories remotely via the command line (again Pro Git is a good resource here). It’s not as simple as it could/should be but it’s worked decently well. I’ve also got a lot of help from the Gitbox app which makes Git much simpler and easier to understand without sacrificing much power. SourceTree is a free alternative that’s more complex but has a lot more options and tools. However, for beginners and ease of use I think GitBox is worth the money because it will save you valuable time figuring things out.

Anyway, stay tuned for Part 3 eventually.  I’ll discuss Coda 2’s git more and hopefully present a better solution than just using the command line.

Developing using Coda 2 & Git – Part 2: Configuring your Dev Server & Git Repo

In Part 1 of this article, we discussed setting up your git repository server.  This post will explain how to use that server to deploy primary and development versions of your site.

We’re going to start on a non-git part of the setup: creating your development subdomain: Whenever you’re working on a new subdomain the first thing to do is actually create the A-Record with your hosting provider because these records can take a while to propagate around DNS. If you’re not familiar with A-Records they’re basically just a way for DNS to know where to send requests for any subdomains of your site; here’s Linode’s information on them other hosting providers will obviously have different settings but you’re basically looking to modify your DNS settings.  For my development purposes, I created a ‘dev’ subdomain with the same IP as my main server and left TTL as Default.  (I also created another one for this ‘blog’ subdomain, but that’s obviously not part of this guide.)

While those new DNS records are being spread around the globe you’ll want to create separate Apache virtual hosts for your dev subdomain (again some Linode documentation).  Your new Dev host should look something like this:

#DEV
<VirtualHost *:80>
# Admin email, Server Name (domain name), and any aliases
ServerAdmin webmaster@<yourURL>
ServerName dev.<yourURL>
# Index file and Document Root (where the public files are located)
DirectoryIndex index.html index.php
DocumentRoot ~<yourUsername>/public/<yourURL>/dev/
# Log file locations
LogLevel warn
ErrorLog ~<yourUsername>/public/<yourURL>/log/error.dev.log
CustomLog ~<yourUsername>/public/<yourURL>/log/access.dev.log combined
</VirtualHost>

Once you’re done with that remember to restart the Apache2 service so it sees your new config files.  Assuming your DNS updates have propagated, you should now be able to see something when you point your browser at dev.<yourURL>.  If you don’t, work on the steps below and check again in a few hours.

At this point I’m going to assume you already have some sort of working website, no matter how basic, that you want to keep running as is while you tinker with your dev version on the subdomain.  If you don’t already have a site just create a basic index.html page and put it up at your main (non-subdomained) URL to get yourself started.

Now you need to use the gitolite-admin control repo you set up in the first part of this guide to create a new repository for your site.  I just called mine ‘site’, but a more descriptive name is probably better.  Change your working directory on your Linux server to the DocumentRoot path to your dev site which you set in your Apache config files earlier, and run the following command:

git clone git@<yourGitServer>:site

This should clone the empty repository to your new development subdomain. Now copy all the files from your production site’s DocumentRoot path to your new development site DocumentRoot, add them all to git, commit them, tag your commit, and push them back to the repository.

cd <devDocumentRoot>
cp -r <prductionDocumentRoot>/* <prductionDocumentRoot>/.* .
git add .
git commit -a -m "Initial Commit"
git tag "v1.0"
git push

If everything went right, the repo that you created in part 1 should now contain the entire contents of your site. To test pulling it to your production site, create a new folder in the same one as the DocumentRoot of this site, cd into it, and try to clone your site’s repo with the command:

cd <prductionDocumentRoot>/..
mkdir .gitversion
cd .gitversion
git clone git@<yourGitServer>:site

You should end up with a second copy of all the files in your site with the addition of the .git folder which has the file history and your repo information, so go ahead and swap this new git-ified version of your site with the main version:

mv <siteFolder> <siteFolder>.non-git-version
mv <siteFolder>.gitversion <siteFolder>

If everything still works when you try your site, you’re in great shape.  Hopefully your subdomain is working as an identical copy of your site now as well.  If it still isn’t run the following command on your local machine to check what your DNS server is returning:

host <yourURL>
host dev.<yourURL>

Both commands should return the same IP address, if not, something is wrong with your config or your DNS servers haven’t updated yet.

I’d suggest a few small .htaccess file tweaks at this point.  First, make sure your .git files can’t be seen by people visiting your site (you should probably apply this to both your dev and primary site’s .htaccess files, or change one and push/pull it to the other with git for practice).  Second, on your dev site, you’ll want to limit access by IP address to just your own IPs so people can’t see your less-secure development code.  Finally, tell git to ignore your .htaccess files in the future so that you dev version with IP restrictions doesn’t get pushed to your main site.

Wow, that took a while.  Stay tuned for Part 3 where I finally describe how to set up Coda 2 with this mess.

Update: still no Part 3, but Part 2.5 is up.

Developing using Coda 2 & Git – Part 1: Configuring your Git Repo Server

I started developing this site on a locally hosted machine with just a local git repository.  This worked great but wasn’t suitable when I wanted to deploy it.  I’m new to git but I very useful when I was developing, so I wanted to find a way to continue using it to help me manage my code.  If you’re not a bit familiar with git this is probably not the right place to start. (For that, I suggest reading this free Pro Git book or just playing with a test repo you can tinker with safely.)

My ideal setup and process for development seemed simple but I found it very difficult to set up and get working the way I wanted it, so I hope this post helps someone in a similar situation.  Here’s what I wanted:

  1. A primary ToWatchList site which wouldn’t get changed unless I specifically pulled a version of my site from my git repository.
  2. A secondary development site where I could tinker and hack at stuff without breaking anything for my normal users hosted under a subdomain of my main site.
  3. A secure git repository that would still be fully accessible from any computer I chose to work from.
  4. Integration with the new git version control features of Coda 2 which I’m greatly enjoying using.

Seems simple but it turned out to be a bigger pain to set up that it should have been.

Here are the problems I had:

  1. My VPS hosting provider Linode (that’s an affiliate link, I highly recommend them, thanks) has excellent documentation with instructions on how to set up many common hosting features.  However their info on git was a little out of date, primarily because they recommended using gitosis as a git server though it has been deprecated by the community (and removed from Ubuntu 12.04 Server repositories) in favor of gitolite.
  2. Actually installing gitolite can be a confusing process until you really wrap your head around what it’s doing and how it works.
  3. Coda 2 has only the most minimal documentation on how to use it with git.  I had to try several setups before I was even able to get it working correctly.

Let’s work through those issues.  As I said before, I’m using Ubuntu server but the info should be equally applicable to other distros.

First: You should use gitolite not gitosis. That’s why I’m writing this guide. Alternatively,  you can probably host your code on github but I wanted to run my own server, so we’ll be following that path.

Second: gitolite is confusing for two related because it’s install process is clunky if you’re installing it from the provided repositories (something I try to do whenever possible) and because it’s mostly controlled via git rather than many config files.

Rachel Aurand has written an nice procedure that I’m going to crib off of here to get things started with the install (remember these steps are for Ubuntu 12.04 but should be similar on other distros).

First make sure you have the regular git software installed:

sudo apt-get install git-core

You should probably also set a few global settings for git like this:

git config --global user.name "Your Name"
git config --global user.email your@email.com

There are lots of other config tweaks you can make that will make things more pleasant when using git, here’s a great interactive tutorial (that requires a free sign-up) and another nice guide to get you started. Also, this post has some good starting settings.  In particular, I recommend setting core.excludesfile, help.autocorrect, color.ui and repack.usedeltabaseoffset, along with disabling core.legacyheaders as shown below but there’s lots more good tweaks.

git config --global help.autocorrect true
git config --global color.ui auto
git config --global repack.usedeltabaseoffset true
git config --global core.legacyheaders false

Next you’re going to need a ssh public/private key for each machine or user you would like to access git repo from, including the machine you’re hosting your repos from if you’d like to access them locally.  Here’s the basic command for generating them:

ssh-keygen -t rsa -C "name@computer"

You can accept the defaults from that command in most cases.  If you do, the .pub files you’ll need will be located under ~/.ssh/id_rsa.pub. Name copies these .pub files so you’ll remember each user/machine they’re associated with and upload/move them to the /tmp directory of your repository server.

Now go ahead and gitolite on your server:

sudo apt-get install gitolite

Because it doesn’t have great setup scripts, that install might complain or warn about not having the right user account set up, no worries, let’s do that now. Create a user to “run” gitolite with on your server:

sudo adduser \
--system \
--shell /bin/bash \
--gecos 'git version control' \
--group \
--disabled-password \
--home /home/git \
git

Now switch to the newly created user, and set the path, and move to its home directory:

sudo su git
echo "PATH=$HOME/bin:$PATH" > ~/.bashrc
cd

No we need to run the gitolite setup command with the public key you copied to the tmp directory for the machine you’re running the repository on:

gl-setup /tmp/<YourLocalServer>.pub

If it shows you any config files during this step, simply exit them without editing.  This setup script creates two repositories: gitolite-admin.git and testing.git repo along with some config files for gitolite-admin.  First let’s clone the gitolite-admin repo to the account on the repository server where we generated the Check to see that everything works by cloning the repo on the machine with the public/private key.

git clone git@:gitolite-admin

You can now edit the config file under ./gitolite-admin/conf/ and add any other .pub key files you generated to ./gitolite-admin/keydir/. Syntax for the conf files is super simple but here’s the documentation for adding repos as well as some notes about using keys, which, remember, represent users.

Once you’ve set stuff up to your liking you should be able to add/update all the new files and modifications and push them back to the server for updates with these commands from within the gitolite-admin folder:

git add .
git commit -a -m "Initial Setup"
git push"

At this point you should probably clone the testing and gitolite-admin repositories to any other machines you plan on developing on to get a feel for how things work, particularly if you’re not an experienced git user.  Once you’re comfortable it’s time to set things up for your development subdomain, which I’ll handle in Part 2.