Linux user's field notes on Windows setup for DevOps tasks

My main work platform has been Mac OS X for years. Since year and a half I’ve switched to Linux. I’m quite happy with this decision. I know Linux using it professionally, so the transition from Mac to PC was quick and painless. Lately, however, I had to spend significant time with an operating system that I’ve been using only occasionally before. Windows… back in the days online communities I was participating in were calling Microsoft’s creation “MustDie”. A lot has changed since nineties. Opensource movement is not constrained by Linux/BSD world anymore. Hyped technologies like docker support Windows too.

Still, it took a couple of months for me to figure out how to make my experience with Windows at work enjoyable and efficient. Recipes I’m sharing in this article are the result of my daily usage of Windows 7 for about a year. I’m a cloud consultant, so the focus here is on setting up and using my typical toolset for “devops stuff” on Windows. I will also mention a couple of painful Windows-specific gotchas and how I’ve dealt with them.

Here is a story of a linux user’s journey to a foreign land of Windows.

Disclaimer: everything described below has been tested and proven to work on Windows 7. I believe most of the material is applicable to newer OS releases, but, as always, YMMV.

Rewinding a year back in time. Here I have my freshly installed Windows 7 and I’m not sure what to begin with. As a minimum survival kit I need to be able to access AWS Console and run aws cli. I decide to start with a browser. Windows 7 ships with a years old version of Internet Explorer. Unsurprisingly, I’m going to use it just to download the Real Thing. In Windows world, you typically install software by double-clicking on a setup.exe, which copies files somewhere to C:\Program Files, makes registry changes and drops an icon on your desktop. At least some of these actions require “root” privileges. From the very beginning of my Windows journey I’ve decided to avoid doing privileged actions as much as possible.

Luckily, there are other people that think it makes sense not to mess around with system files and registry and there’s a wonderful Portable Apps website offering lots and lots of no-root-required software packages. So the first step of my windows setup is installing the portable firefox.

But then as it comes down to aws cli, the choice doesn’t look so simple to me. Should I use AWS Tools for PowerShell or stay with the tried and true pip install awscli, and if so – which shell, which terminal is a good choice for me? Oh, and I need to install python somehow… to begin with.

To me it sounds like a good deal of work for a package manager. I used to use brew on a mac and on my linux machine I have a pacman/yaourt combo for that. The concept of a package manager is so natural to many people that it’s no wonder there’s a bunch of different projects in this field for Windows too. I want something brew-like, so I choose scoop. scoop also satisfies my requirement of being able to work without “root” permissions. Now you probably wonder, why did I use portable apps – not scoop – to install firefox? The thing is, scoop is mostly targeted at commandline applications, just like brew. Apart from that, I like the self-update feature of firefox.

Installation process for scoop is as simple as for brew. Just follow the instructions on the official website. I my case I have to configure proxies beforehand, so in a powershell terminal I run

$env:HTTP_PROXY="http://x.x.x.x:9400"
$env:HTTPS_PROXY="http://x.x.x.x:9400"

and then proceed to the scoop installation command.

With the package manager in place, now I go on with the essentials. In my consulting I’m not doing any windows serverside work, so powershell is just not my thing. I’d like to have bash handy along with the good old company of grep, sed & awk. Powershell terminal emulator itself is also far from ideal for me.

scoop bucket add extras
scoop install cmder

I’m an old-school guy and I typically use midnight commander for my file management tasks. Although I can readily install mc with scoop, I think that Far Manager - which follows the same philosophy as mc - is a better choice for Windows. Far Manager also integrates nicely with cmder. So my next command is

scoop install far

Quite surprising to me is the fact that there is no separate package for bash in scoop. Probably, you are not supposed to use bash for the sake of bash – instead it is bundled with other software. Ok, so then I’m installing a bunch of goodies in one go

scoop install git-with-openssh git

Hey! Two git packages? That’s right, quite a weird thing, one for me and one for scoop to update itself. Don’t ask.

Ok, back to the shell. To make use of bash, I need to configure now my terminal emulator, cmder. The goal is to fire up a new bash session instead of cmd.exe every time when I start cmder or open a new tab in it. To do that, I start cmder, click on a “sandwich” button (lower right corner) and go to “Settings” in the pop-up menu. In a tree-like list on the left I choose “Startup -> Tasks”. Now I need to create a new item by pressing the “+” button. The Group name is going to be bash::scoop and in the Commands field I put this:

cmd /c ""%CMDER_ROOT%\..\..\git-with-openssh\current\usr\bin\bash.exe" --login -i"

As I’m already here, I would love to be able make Far accessible from cmder. Fortunately, no need to type any commands in this case – I’m just pressing the “Add/refresh default tasks” button and then the “Add new tasks” in a popup window. I can now start bash or Far with the green plus button (there is a tiny black triangle to the right of it to choose what to run in a new tab).

And – not to forget to make bash::scoop the default task! In a tree-like menu choose “General” and select bash::scoop in a drop-down menu called “Choose your startup task”.

Ok, almost done with the terminal emulator. I don’t like that by default Ctrl-w closes an active tab. It makes more sense to me if this hotkey means just “remove last word”. In Settings menu I choose Keys & Macro, type into the filter field ctrl+w and enter, select the first item in the filtered list and remove W from the Choose hotkey field. Nice!

Good bye, powershell, I’m going to continue now with bash.

And now I need a text editor to do quick edits here and there. I’m a happy emacs user for many years, but in this case I would rather opt for vim: vim is good enough in its default configuration and, well, it’s just convenient to fire it up from a console.

scoop install vim

The only thing I’d love to change in my vim’s default configuration is its behavior on pasting text. Not sure why, but when when you right-click to paste
a multiline snippet in vim,
  it aligns
    all the lines
      like this. The fix is easy:

echo 'set paste' >> ~/.vimrc

And to make Far Manager use vim for file edits, I bring up Far in cmder, then go to F9 -> Options -> Editor Settings and in Editor command field I type

vim "!.!"

Now if I press F4 when a file is highlighted in Far, that file is opened in vim.

Not bad. I have now a terminal, a file manager and an editor usable on Windows.

And now with my shiny new windows terminal emulator I’m going to clone a bunch of git repositories. I use passphrase-protected ssh key to authorize on git servers, so here is a snippet I’m adding to my ~/.bashrc in order to type my ssh passphrase only once – when I start cmder after Windows boot:

SSH_ENV="$HOME/.ssh_environment"

start_agent() {
  echo "Initializing new SSH agent..."
  touch $SSH_ENV
  chmod 600 "${SSH_ENV}"
  ssh-agent | sed 's/^echo/#echo/' >> "${SSH_ENV}"
  . "${SSH_ENV}" > /dev/null
  ssh-add
}

if [ -f "${SSH_ENV}" ]; then
  . "${SSH_ENV}" > /dev/null
  kill -0 $SSH_AGENT_PID 2>/dev/null || {
      start_agent
  }
else
    start_agent
fi

Here are two more important tips related to git on windows – backed by my half a year of experience of daily usage.

Windows has quite different file permissions model than Unix-like systems. Once I’ve started working with a git repository shared with folks working on Linux, I was amazed at my git status showing tons of unstaged changes when I just edited a single line in one file. Looking at the issue closely, it appeared that git was trying to change file mode for each and every file in the repository. This is clearly not something I want. To make windows chill on file modes it just doesn’t understand, I’m issuing this command in cmder:

git config --global core.filemode false

Now git status is showing only the real changes.

Tip number two for git on windows: I’m disabling fancy git prompt in bash as it slows down my shell.

echo 'export PS1=\'$ \'' >> ~/.bashrc

That’s it for git. I’m back to installing aws cli. And I would need some python now:

scoop install python

Once this command finishes, I’m restarting cmder. I’m going to do it a couple more times later. This is needed for the terminal to pick up an updated PATH environment variable.

Now my shell is able to find pip, so I can continue with aws cli installation:

pip install awscli 

My complete CloudFormation development toolbox consist of many other useful utilities. Here I am installing the first batch of them:

pip install cfn-flip aws-shell yamllint
scoop install jp jq

• cfn-flip is a yaml <-> json converter for CloudFormation templates
• aws-shell is kind of an aws REPL, useful for exploration
• yamllint is a linter for yaml files; I use it in my development workflow to spot simple mistakes like wrong indentation
• jp this is my helper tool for authoring jmespath queries for aws cli; I fetch an unfiltered json, store it to a file and then pass it through jp until I come up with a query that produces the result I want
• jq is a popular commandline tool for parsing json; it’s syntax is different from jmespath

Yamllint is not the only tool I use to validate my CloudFormation templates. For every project I have a makefile that describes a continuous deployment pipeline. I use to develop CloudFormation code in an iterative way, sometimes with a lot of experimentation. I want to know whether my last edit is correct – as soon as possible. So here’s what happens when I type make deploy in my terminal:

“Hey, dude, wait a minute…” - a thoughtful reader may ask. - “Is it ‘CPP’ written there? As in ‘C Preprocessor’?”

Yes, no mistake. I use cpp as an engine for a simple templating language: mostly for #include’s, but there is more to that (details to come in a separate post).

Ok, so now I’m going to install the rest of the tools mentioned in the diagram above.

scoop install make grep coreutils gcc

As it should be clear by now, the gcc package is a part of my CloudFormation Development Kit because of C preprocessor.

cfn-lint is a second element in the validation part of my CloudFormation delivery pipeline. cfn-lint checks that all referenced resources are defined, looks for missing required properties, examines intrinsic functions for correctness. cfn-lint even provides links to corresponding articles in AWS documentation along with error messages. Extremely useful utility.

cfn-lint is written in javascript:

scoop install nodejs

I’m restarting cmder to make which npm happy.

npm install -g cfn-lint

Next in the pipeline is cfn-nag. This guy is a security-focused static analyzer. And it’s written in ruby:

scoop install ruby

This command suddenly triggers a Windows Firewall popup, suggesting that I should cut ruby.exe from internet. I disagree and allow access explicitly. Once again, I’m restarting cmder for it to re-read the PATH environment variable.

gem install --no-ri --no-rdoc cfn-nag

Please note: the gem is called cfn-nag (with a dash) and the actual application is cfn_nag_scan (with underscores).

Ok, I’m done with my CloudFormation stuff.

Here’s one more AWS-related tool though:

pip install awslogs

awslogs is like a tail -f for CloudWatch logs. I’ve used this utility for quite some time on linux and macOS without any issues. However, when I run a command like this on Windows

awslogs get -wGS /aws-glue/crawlers

I am greeted with an obsure python exception, last line of which contains a weird-looking file path

botocore.errorfactory.InvalidParameterException: An error occurred (InvalidParameterException) when calling the FilterLogEvents operation: 1 validation error detected: Value ‘C:/Users/ignis/scoop/apps/git-with-openssh/current/aws-glue/crawlers’ at ‘logGroupName’ failed to satisfy constraint: Member must satisfy regular expression pattern: [.-_/#A-Za-z0-9]+

This C:/Users/ignis/scoop/apps/git-with-openssh/current/aws-glue/crawlers is definitely out of place here. And it looks like my shell has just converted a log group name with a slash in front into a non-existing file path. Well, it seems that unix tools on windows do use some black magic.

Bash for windows is a part of a software package called MSYS. MSYS tries to convert what looks like a POSIX path into a windows path. Luckily, there is a way to disable this behavior:

MSYS_NO_PATHCONV=1 awslogs get -wGS  /aws-glue/crawlers

I’m almost done with my Windows setup for devops. Here’s a little bonus: two extremely helpful Windows GUI programs that I use in my consulting work.

First one records gif screencasts:

scoop install licecap

And this one is a screenshot tool ala macOS skitch:

scoop install greenshot

I hope this post was useful for you. I personally find that the OS from Microsoft is a totally viable platform for devops work (although I must admit I’m not using docker on Windows). It has its quirks, yes, but I believe it’s possible to find solutions for them.

Thank you for reading.