G Suite legacy free edition retirement - What to do

I've had a few of my domains pointed to G Suite since its inception.  During that beta period, it was free to try out and anyone who got in then has been able to stay free all this time.

Months ago I started getting notifications that looked like they were saying it would no longer be free and that I'd have to sign up and start being billed.  I didn't really want to pay for this for three domains since they're basically just toy domains, not really doing anything interesting or worthwhile.

I finally started researching my options and found this little tidbit buried in one of the G Suite legacy free edition support pages...

I used my account for personal use

...this handy little button.

For non-commercial use, you can still continue for free. You don't have access to any of the premium services, but all the core services (Gmail, Calendar, Drive, Meet, Sites, etc) remain free.

Now I'm curious as to whether a regular account, not "legacy", could use this option to declare their account is just for personal use as well.  🤔

Putty and double bastion host tunneling

I've always found complex ssh tunneling to be a pain in Windows.  Unfortunately, I'm stuck with it due to company mandate.  This post is as much about sharing the info as it is about documenting it so that I can come back and refer to it when I forget everything here.

So, the use case is this.  You have a host that you have to hop through in order to get to other hosts, but there may be more hosts that you can't get to unless you hop through yet another host.

For example, if I want to access my Linux workstation at the office, I have to hop through two hosts, my ssh bastion host and from there to another host that also has an IP on the office network, and from there to my desktop.

Even to experienced tunnelers, that's daunting, especially from Windows.

So, here goes.  Here's what the values in my example mean:

My workstation: 172.16.0.99
The public facing bastion host: bastion.example.com
The internal server: server1.example.com
My username on my workstation: myusername
My username when accessing company servers: companyusername
My WSL username: myWSLusername

Putty Configuration

1. Create a new session in Putty.
2. Hostname: 172.16.0.99
3. Port: 22
4. Category → Connection → Data
• Auto-login username: myusername
5. Category → Connection → Proxy
• Proxy type: Local
• Telnet command, or local proxy command:
  c:\progra~1\putty\plink.exe -ssh -agent -A -l companyusername server1.example.com -nc %host:%port -proxycmd "c:\progra~1\putty\plink.exe companyusername@bastion.example.com -l companyusername -agent -A -nc server1.example.com:22"
6. Category → Connection → SSH
• Enable compression
• Notice that I didn't use the compression option -C in my plink commands in the previous step.  When tunneling ssh traffic, you should only enable compression in one place so that you're not compressing traffic only to have other segments attempting to compress the already compressed data.
7. Category → Connection → SSH → Auth
• Attempt authentication using Pageant
• Allow agent forwarding
8. Category → Connection → SSH → X11
• With WSL and VcXsrv X Server installed, you can run gui apps
• X11 Forwarding: Enable X11 forwarding
• X display location: 127.0.0.1:0.0 (Look in VcXsrv server's log to confirm this value.)
• Remote X11 authentication protocol: MIT-Magic-Cookie-1
• X authority file for local display:
  %LOCALAPPDATA%\Packages\CanonicalGroupLimited.Ubuntu18.04onWindows_79rhkp1fndgsc\LocalState\rootfs\home\myWSLusername\.Xauthority
• (You might to run from WSL bash shell xauth generate $DISPLAY initially to get the x authority file seeded.)
9. Save new Putty session
10. Launch new Putty session

There are certainly other ways of doing this that can be a little bit simpler.  One of the reasons I used this method is because my bastion host strips X11 traffic.  It's not configured for it and doesn't have any of the required x related dependencies needed (xorg-x11-server-utils et. al.).  Doing it the way I have creates a tunnel that simply passes all traffic through to the next host keeping me well below the application layer from the perspective of the bastion host.

Workstation File Access

The other advantage to this method is that I can use this same Putty session to make additional ssh tunnels that go right to my workstation.  So, in my Putty config detailed above, I also have a local tunnel 20202:172.16.0.99:22 that gives me direct ssh access to my workstation by ssh'ing to 127.0.0.1:20202 here on my laptop.

So, software like WinSCP can now access my workstation over this tunnel and behaves as if it were direct access.  I use Mountain Duck, which is not free software. The end result is that it allows me to map a drive right to my Linux workstation at the office from my Windows 10 laptop at home.

(Additionally, I use the RDP tunneling method described in my previous post to make it so that RDP sessions end up originating from my workstation.)

Limitations

While drive/file access works quite well, this is not fast enough to do X11 well at all.  So as much as I'd like to run gvim directly from my workstation over the tunnel, it's just not fast enough.  

But, having the option to do it was helpful for me.  There were a couple apps that I really just needed some of the settings out of so that I could set up the Windows versions of those apps to work the same way.

OpenSSH

I haven't actually explored the capabilities of the OpenSSH that's now included with Windows 10, but regardless, for non-Windows users, note that there is a newer ProxyJump directive.  This let's you chain together any number of bastion hosts.  So, following my earlier example, you can do something like in an .ssh/config file entry:

Host workstation-tunnel
    ProxyJump companyusername@bastion.example.com,companyusername@server1.example.com
    Hostname 172.16.0.99
    User myusername
    ForwardAgent yes
    ForwardX11 yes
    ForwardX11Trusted yes
    Compress yes
    PubkeyAuthentication yes

Then you just ssh workstation-tunnel and you're good!

SSH SOCKS Proxying with Putty

I'm writing this during the COVID-19 lockdown.  My company's VPN is getting hit really hard since everyone is working from home.  Anything we can do to stay off of it is helpful.

We also keep a host available with SSH exposed publicly (public key auth only).  So, I use that host as an SSH SOCKS proxy and it works great for keeping me off the VPN.

So, if you're in a similar position or simply would like to use SSH as a sort of pseudo-VPN, these instructions might be helpful.

Non-Windows users can do the same thing, you just need to use the ssh command to connect to the remote host and use the -D parameter.  Something like: ssh -D 1337 yourhost

Putty Configuration

1. Create a new session in Putty
2. Hostname: yourhost
3. Port: 22
4. Go under Category → Connection → Data
• Auto-login username: <your username>
5. Category → Connection → Proxy
• Leave this off
6. Category → Connection → SSH
• Enable compression
7. Category → Connection → SSH → Auth
• Attempt to authenticate using Pageant
• Allow agent forwarding
8. Category → Connection → SSH → Tunnels
• Source port: 1337
• Destination: yourhost
• Radio button: Dynamic
• Click Add
• (Just shows D1337, this ok)
9. Save the new Putty session
10. Launch the new Putty session

Proxy Configuration

Now, to actually use the proxy, you can go a couple ways.  Originally, I was doing it the manual way, but I found the Chrome extension SOCKS proxy which works great.  It's hassle free and even make it so that DNS requests go over the proxy.  The source code is very small and easily reviewed so you can see it's not doing anything nefarious.

If you can't or won't install an extension, here's the manual method.

1. Run the inetcpl.cpl control panel. (NOT the new Windows 10 Proxy Settings page.)
2. Go under the Connections tab
3. LAN settings button
4. Uncheck automatic detection
5. Check Use a proxy server for your LAN
6. Advanced button.
7. Fill in ONLY the SOCKS information (not http, secure, or ftp. Uncheck Use the same proxy for all protocols)
• Socks: 127.0.0.1
• Port: 1337

DNS Considerations

Now, if you don't have to worry about resolving any private DNS records, you're good to go.  My company has whole zones that aren't resolvable from the public internet.  For these, DNS queries have to originate from the company network.  Chrome, by default, will not send DNS requests over the SOCKS proxy, so there's an additional step required.

I suggest copying your existing Chrome icon and giving it a different name.  Edit this icon and append to the end of the Target: field, after the final quote (not inside the quotes) the following:

--proxy-server="socks5://127.0.0.1:1337" --host-resolver-rules="MAP * ~NOTFOUND , EXCLUDE 127.0.0.1"

I haven't test it myself, but I've heard that Firefox automatically pushes DNS requests over the proxy.

Limitations

So one of the big limitations of this is that it doesn't really help in a heavy Active Directory environment where your PC has to communicate with things over a domain, such as shared drives.

RDP

However, you can tunnel RDP through your SSH host as well.  Configure additional tunnels, one per RDP destination. Back in your new Putty session:

• Category → Connection → SSH → Tunnels
• Source port: 38001
• (This is a made up value of no significance. You'll have to make one up for each RDP destination.)
• Destination: rdphost:3389
• Relaunch your Putty session
• Open RDP
• Use the destination address: 127.0.0.1:38001
• Repeat the port forwards with different port numbers for each RDP host you to access.

Using Wireshark on a remote host

In a large environment, troubleshooting problems with network packet traces usually means you're logged into a remote host running tcpdump.  Even after you develop some skill with pcap-filter syntax, wielding tcpdump is clunky and it usually looks like you're trying to view The Matrix encoded.

There are other console based tools like tshark, but few of them are as useful and as user-friendly as Wireshark which can render and parse network packets in an extremely readable and comprehensive fashion.

The problem is that Wireshark is a graphical interface.  Running it on a remote host means you'll have to install it and all supporting dependencies and libraries on the remote host and then ssh X tunneling it back to your desktop.  For many reasons, this may not work well.  Or, you may not even be able to install Wireshark on the remote host for any number of reasons.

One workaround used by a lot of people is to capture some network output with tcpdump writing to a file, then fetch that capture file to your desktop and open it up in Wireshark.  It's definitely handy that pcap is so portable that this is possible, but this method lacks the ability to watch network traffic in real-time.

So how can you achieve the holy grail and use Wireshark locally on your desktop to watch live traffic on a remote host?

Enter socat - Multipurpose relay.

The socat utility is a swiss army knife of basically all possible types if input/ouput.  One of its supported i/o types is named pipes.

In short, we can use socat as the middleman to read from a remote named pipe to a local named pipe. Then, we take advantage of Wireshark's ability to read right from a named pipe and read that local named file.

Here's the steps using the example username jsmith, example remote host name srv1, and example network interface name eth0.

On the remote host:

1. Create a temp dir for your named pipe file.
• sudo mkdir /tmp/fifo
• sudo chown jsmith /tmp/fifo
• sudo chmod 700 /tmp/fifo
2. Create the named pipe
• sudo mkfifo /tmp/fifo/pcappipe
3. Kick off tcpdump, writing to that pipe.
• sudo tcpdump -i eth0 -s 0 -U -w /tmp/fifo/pcappipe not port 22

Notice the temp dir permissions.  You need to be able to read the named pipe as the non-root user with which you're going to use to log in.

Also notice the pcap filter 'not port 22'.  You can alter this of course, but if you don't specifically exclude your ssh traffic, tcpdump is going to pick up all of the traffic from you being logged in as well as the part where we remotely read from the named pipe which takes place over ssh.

Next, on your local desktop, run socat like so:

socat -b 67108864 \
    EXEC:"stdbuf -i0 -o0 -e0 ssh -x -C -t srv1 cat /tmp/fifo/pcappipe",pty,raw \
    PIPE:/home/jsmith/localpcappipe,wronly=1,noatime

This tells socat to ssh into the remote host and cat the named pipe (sending the data to STDOUT).  It reads from that and writes it to the named pipe file in your home directory.

The buffer tuning was important to making it as live as possible as well as more stable.  Plus, this can be somewhat of a brittle process and socat can end up crashing easily.  The buffer tuning helps make things much more stable and reliable.

Next, run wireshark, as root.

sudo wireshark -s 0 -k -i /home/jsmith/localpcappipe

Profit!


Normal ssh rules apply.  So, if you can't ssh directly to your remote host, configure your .ssh/config file accordingly.

I need to tunnel through an intermediary jump host as well, so this is what I do in my .ssh/config file:

Host srv1
    ProxyCommand           ssh jumpsrv1 /usr/bin/nc %h 22
    User                   jsmith
    IdentityFile           ~/.ssh/id_rsa
    Compression            yes
    PubkeyAuthentication   yes
    Port                   22
    Protocol               2
    EscapeChar             none
    ServerAliveInterval    30

(I know I know, there's a new ProxyJump directive...  I don't change my .ssh/config that often.)

Google Reader to TT-RSS - coping with 3rd party apps and services

Here's a guide to replacing Google Reader with Tiny Tiny RSS, but with more detail on the greater tragedy (travesty?) of Google's continued bulldozing of open internet standards with the termination of Reader, the interaction with other services and software on which we used to depend.  I'm starting with a couple of simple but important services, though I'd like to flesh this out over time.

Infrastructure

The problem we all face isn't installing the software (granted, TT-RSS is a non-trivial install, but I consider that to be outside the scope of this guide). The problem we face is the need for hosting. And most of the hosting solutions that will meet our needs require us to configure a hosted server of some kind.

My solution was to start with a Bitnami stack.  Specifically, I started with the LAMP stack.  However, the Bitnami portion of this process will soon become very simple. Tiny Tiny RSS won the regular contest Bitnami holds to decide on new stacks and is expected to be released soon.  Keep an eye on it here: http://bitnami.com/product/tiny-tiny-rss

Next, I signed up a free "developer" class account with Bitnami Cloud Hosting.

Bitnami will allow you to deploy your stack to the Amazon EC2 service automatically and provides documentation for getting your Amazon account set up and ready:  http://wiki.bitnami.com/Amazon_cloud/prepare_aws_account

If this is your first foray into AWS, the first year of your EC2 micro-instance is free.

Software

Now, onto the part where you actually use TT-RSS.

There's plenty of howto links around for installing TT-RSS itself, so I keep the redundancy to a minimum.  The official guide is over here: http://tt-rss.org/redmine/projects/tt-rss/wiki/InstallationNotes

The guide for how to set up automatic updating of your feeds: http://tt-rss.org/redmine/projects/tt-rss/wiki/UpdatingFeeds

I'm not going to bother with details on how I rigged this up in my Bitnami stack as most of it will be soon be moot with the introduction of the TT-RSS stack. If you're a good Linux user and you do want to wait for the bitnami release, here's the summary with no training wheels.

• Run gnu screen.
• I used /opt/bitnami/mysql/bin/mysql_setpermission to set up the tt-rss db and configure a user to be able to access it.
• Extract the tt-rss source tarball into /opt/bitnami/apache2/html/ and rename directory to tt-rss.
• Slurp in the mysql schema.
• Configure tt-rss according to the install guide.
• Because the update_daemon2.php script needs to be run as the daemon user, I added a minimal ~daemon/.bashrc (just the path from ~bitnami/.bashrc):
• PATH="/opt/bitnami/memcached/bin:/opt/bitnami/varnish/bin:/opt/bitnami/redis/bin:/opt/bitnami/nodejs/bin:/opt/bitnami/mercurial/bin:/opt/bitnami/perl/bin:/opt/bitnami/git/bin:/opt/bitnami/nginx/sbin:/opt/bitnami/frameworks/laravel/app/Console:/opt/bitnami/frameworks/cakephp/app/Console:/opt/bitnami/frameworks/codeigniter/bin:/opt/bitnami/frameworks/symfony/bin:/opt/bitnami/frameworks/zendframework/app/Console:/opt/bitnami/sphinx/bin:/opt/bitnami/sqlite/bin:/opt/bitnami/apps/django/bin:/opt/bitnami/php/bin::/opt/bitnami/java/bin:/opt/bitnami/mysql/bin:/opt/bitnami/postgresql/bin:/opt/bitnami/apache2/bin:/opt/bitnami/python/bin:/opt/bitnami/subversion/bin:/opt/bitnami/ruby/bin:/opt/bitnami/common/bin:$PATH"
  export PATH
• In another screen window, become the daemon user
• sudo su - daemon -c bash
• Run the update script.
• cd /opt/bitnami/apache2/html/tt-rss/
• ./update_daemon2.php

That's pretty much it.  Filling in the blanks is left to the more experienced user.

On to the more important pieces...

Exporting from Google Reader

Here is an excellent guide for how to export your Google Reader data:
How to painlessly export your Google Reader feeds

Importing your rss subscriptions to TT-RSS

Extract the file you downloaded from Google Takeout.  Inside, the important item is the subscriptions.xml file.  Log into your TT-RSS instance and go into Preferences, then click into your Feeds tab. Expand OPML, the 2nd accordion item. Click the Import my OPML button and select the subscriptions.xml file you extracted. If you're interested in preserving your starred items from Reader, expand Import starred or shared items from Google Reader, the last accordion item. Select the starred.json file you extracted and click the Import my Starred Items button.

Android

App

The official TT-RSS App has been under very active development and is maturing very quickly.  It's been perfectly stable and usable for me. The only downside is the lack of a launcher widget allow you to read feed article titles right on your homescreen.

Widget

My solution for the lack of a widget is a little complicated, but it's working great. It's made possible because TT-RSS is not just a feed reader, it is also a feed publisher. In your TT-RSS instance, put the feeds you want to appear in your widget into a category.  I used the Favorites for my category name. After you create and populate your new category, click on it.  Positioned underneath the Preferences link, you'll see Favorites and the RSS icon. Click directly on the RSS icon and it will pop up a long link.  Copy down this url, you'll need it later.

Next, install Simple RSS Widget.

Yahoo Pipes

You'll be feeding Simple RSS Widget from Yahoo Pipes.  Log into your Yahoo Pipes account and create a new pipe.

Since I have several feeds under my Favorites category in TT-RSS, I wanted to give each item title a little visual distinction so I would know from which feed the individual article came.  This made the Yahoo Pipe I created a bit more complicated than it needed to be, but the end result is exactly what I wanted.

From the Sources section, Drag out an Item Builder object.

I populated mine with the following attributes:

title = Favorites
description = Items from my Favorites category
link = http://pipes.yahoo.com/<username>/<placeholder>?_render=rss (replace this with the link to your resulting yahoo pipe when you're done.
author = aharrison

From the Operators section, drag out a Loop object. Back under Sources, drag out a Fetch Data object and drop it in the middle of the Loop object you just dragged out. For the url, you're going to drop in the long url you copied earlier. However, you're going to modify the url. Your url will look something like:


http://my-ttrss-instance.bitnamiapp.com/tt-rss/public.php?op=rss&id=26&is_cat=1&key=abcdef1234568901234567890123456789012345

Modify it slightly:

http://my-ttrss-instance.bitnamiapp.com/tt-rss/public.php?op=rss&id=26&is_cat=1&format=json&view-mode=adaptive&key=abcdef1234568901234567890123456789012345

Add the item I put in bold and leave everything else the same.  In the Path to item list field, type: articles

Also worth noting, if you're using https:// for your TT-RSS instance, you need to drop the s to allow Yahoo Pipes to hit your feed without ssl. Likely this is simply because the default ssl certficates in your bitnami stack are just self-signed examples. If you want Yahoo Pipes to be able to use ssl, you're probably going to have to buy your own ssl certificate and rig it up inside your Bitnami stack. (If you really want to do this, here's an old guide that will help you get there: How to activate ssl / Security

Inside your Loop object, choose the radio button emit and the all dropdown choice.

Click and drag the botton of your Item Builder object to the top of your Loop object to connect them together.

If you don't want to bother altering the item titles to reflect the feed, you can now connect the bottom of this Loop object to the top of the Pipe Output object and you're done.  If you also want the indication of which feed the item is from, proceed.

Drag out a Rename object.  Connect the Loop object to this new Rename object.

For the mapping, choose item.link from the field choices, then Copy As from the drop down, and type in the last field: titleprefix

Drag out a Regex object.  It should read:

In item.titleprefix replace ^http[:][/][/](.*?)[.](com|net|org).*$ with [$1]

Connect the Rename object to this new Regex object.

Drag out another Loop object. Connect the Regex object to this new Loop object.

Grab another String Builder object and drop it in the middle of the new Loop object. Populate it like so:

• item.titleprefix
• <blank spot>
• item.title

Grab another String Builder object, but this time, drop it outside the Loop object, not inside.
Inside the new String Builder object, just enter ' - ' (without the quotes).  That's a space, a dash, and another space.

From the bottom of this new String Builder object, click and drag to connect it to the little bubble to the right of your empty field inside your Loop object's String Builder object. Now in your inner String Builder object, instead of the empty field, it should turn gray and read 'text [wired]' because the empty field will now pull from the outer String Builder object sitting off to the side.

To finish with our Loop object, instead of the emit results radio button, choose the assign result to radio button and select item.title from the choices.

Click and drag from the bottom of this Loop object and connect it to the top of the Pipe Output object.

You're done creating your Pipe, so click Save and then go back to your pipes page.

Click on your new pipe.  Inside, you'll see a link Get as RSS. Right click it and copy it.

Edit the pipe you just created.  Remember back up in the Item Builder object where I said you'd replace the url?  Paste the url there.

Also, send this link to your Android device so that you'll be able to configure it there as well.

Take that link, and configure it inside the Simple RSS Widget you installed. Now when you drop the widget onto your homescreen, you'll see all of your favorite items.

I made an example of this pipe available for reference:
http://pipes.yahoo.com/ahinmaine/examplettrssfavorites

ifttt.com

Another frustration of the Reader retirement is all the awesome things you can do with your feeds at ifttt.com.  Fortunately, all is not lost. For example, I had an ifttt trigger to push any of my starred items to Pocket automatically. Instead, you can use the simple version of the above Yahoo Pipe example to accomplish the same thing. For your Fetch Data object, use this example url instead.

http://my-ttrss-instance.bitnamiapp.com/tt-rss/public.php?op=rss&id=-1&format=json&view-mode=adaptive&is_cat=&key=abcdef1234568901234567890123456789012345

Notice the id is -1, this automatically chooses your starred items feed. And instead of is_cat being 1, leave it blank.

Why the Pipes?

You may have already wondered, why would I use Yahoo Pipes to access feeds I've published in my TT-RSS instance rather than accessing them by my TT-RSS url directly?  Simple, if you have multiple apps and services polling your TT-RSS instance, this is going to greatly increase the amount of traffic to your EC2 instance. There are usage limits to these micro-instances and you'll hit them a lot faster if you let it get hit from lots of different sources. Yahoo Pipes is an effective way of keeping your costs down. For example, if your Simple RSS Widget is set to poll every 5 minutes, Yahoo Pipes is NOT going to hit your TT-RSS instance every time, it's just going to serve up the results since Pipes last polled it.

The Pipes Regex object

Yes, I know the regular expression I used for an article title prefix isn't very robust.  I'm lazily grabbing the hostname from the domain of the url of the feed. Improving the appearance of your feeds by having fun with Pipes is left as an exercise for the reader.

Vim slick trick of the day...

I've always wanted to know how to do this and happened to stumbled across it in a forum.
Here's how you take the piped output from two commands and vimdiff them together.

vimdiff <(cat filename1) <(cat filename2)

Ridiculous, I know.  But the point is the commands can be anything allowed by the shell.

vimdiff <(grep foo filename1 | grep -v bar | sort -u) <(grep foo filename2 | grep -v bar | sort -u)

The Complete Idiots Guide to Correctly Validating Your Customer's Email Addresses

Update August 2020:  

I have turned this blog post into a site of its own.  Please visit guide.mlz.me for a more updated version of this post.

_____________

Quick tip for openSUSE users of App::perlbrew...

Ever since I installed 11.2 on my ultra40 a few months ago, perlbrew has been busted for me.  After failing, the build log always ended with:

ODBM_File.xs:124: error: too few arguments to function ‘dbmclose’

I finally took the trouble to fix it.  After some googling, I saw that others on opensuse had seen this while compiling on their own, but didn't readily see any search results for how to easily fix it with an automated builder like perlbrew.  Turns out the secret sauce isn't that difficult since perlbrew with happily pass through any perl building arguments:

perlbrew install perl-5.12.2 -D noextensions=ODBM_File

Efficient command line file searching...

I recently replied to the blog post: One-liner: Finding files that include a match with this response.