Thursday, May 18, 2017

What is devops?

Ask 12 engineers for a definition of devops and you'll get 13 answers. Yes, that's 13 different links to everyone from Gartner to AWS to Dr Dobbs. There's plenty more on Google, too. Go ahead - take the time to read through them. AWS even has a whitepaper on devops which, hilariously enough, doesn't match up with their definition from the first sentence.

Confused now? Everyone else seems to be, too. The problem is every one of those answers, when you parse it down, is a long-winded way of saying "I know it when I see it." Which may be true, but it's also worthless for you. You can't do anything with that because the author isn't with you to help you whiteboard what devops means in your organization. This lack of clarity leads intelligent people to discard the term "devops" as meaningless and try and forge another way.

After working with a dozen clients in the past couple years, I think the problem is everyone is looking for a singular prescriptive definition. Devops is this thing and not that thing and those things are the same for everyone. And, there isn't one.

Some of the links above provide a lot of words around the characteristics that some successful organizations have. But, these characteristics (no silos, automation, etc) don't tell you how to solve your problem. It's as if doctors decided to practice medicine by looking at the healthiest people in the world and giving all their patients a list of those characteristics in response to every patient visit. That can be helpful ("lose weight" is a good thing to do, in general), but it can also be useless ("lose age" isn't helpful, even though most healthy people are between 20-30.)

Instead, I offer a descriptive definition which enables SMART milestones.
Devops is developing applications whose business domain is your operations.
 This isn't a singular thing. This isn't the same for everyone. This doesn't tell you what else you have to do, either. For example, you don't have to do Agile, if you don't want to. (Yes, you can devops within Waterfall, if you want.) But, it tells you exactly what you should be doing in order to do the devops and how.

Why this definition and not some statement about automation or build/release engineering? For one thing, this definition includes all those ideas. (Well, it leads to them, as we'll see.) For another, this definition recognizes that every organization is different. Each organization has different requirements for how they want to do IT and any definition of devops needs to accommodate that.

So, what do you do with that definition? The first thing is to gather requirements, just like any other application. Topics like:
  • How do you (the IT teams) want to operate?
  • What does the business side expect from the IT department?
  • What things suck right now?
The second thing is to figure out what you actually do today. I mean, actually do. Can you bring in a new senior person anywhere in IT and give them a clear picture of how your organization does IT work within the first day? First week? First month? Most clients I've worked with struggle with integrating new staff, with 1-year veterans saying "I never knew how we did that before."

Once you have both, you can describe the gap. Bridging that gap is your devops journey. That journey is going to be very different from company to company and even between teams at the same company. Which makes sense.

Monday, May 15, 2017

Why have an Operations team?

I just came back from a meeting with the manager of a team I'm working with. The meeting went well, but at the end, he asked me a really interesting question.
How do I justify an ops team (or a devops team) to my executives?
I sat back for a second and realized I've never heard anyone actually lay out the case for an operations team. It's always been assumed that you just have ops, just like you have devs and QA. Of course you just have them.

To answer, let's start with a premise.
The business gains value from users consuming its application(s).
This should be self-evident; it's the whole point of the business funding an application.

Developers, obviously, build new features which creates more value. That's the whole point of developers - to modify the application to create new features to create new business value. We'll call this a direct contribution. Their work has immediate and visible impact on users consuming the business's applications.

The business analysts who write the requirements, the designers who create the UI/UX - they also provide a direct contribution.

Operations and QA, however, don't make that direct value contribution. Their work provides an indirect value contribution. The QA team can get its own post some other time. As for the Ops team, they do two things:

  1. Reduce business risk by maintaining the user experience.
  2. Improve time-to-market (TTM) for new changes, such as features, bugfixes, and security.
Let's take each one in its own.

Maintaining the user experience is primarily maintaining and supporting the production environment. These are tasks that would have to be done even if all development was halted. The primary beneficiary of this work is the external user. This aspect includes:
  • Gathering and analyzing logs
  • Gathering and reporting on metrics
  • Tracking and applying updates to third-party services (such as IIS, Windows, etc)
  • Monitoring for and alerting on any issues that arise (such as server failures)
  • Maintaining awareness of security issues (such as hacking)
  • Maintaining awareness of performance issues (such as bad database queries)
  • Reducing and/or eliminating deployment downtime
These tasks may be handled by people outside a formal operations team (such as developers handling performance issues), but these tasks are within that operations role.

The value of these tasks is directly proportional to how expensive any specific failure in production is to the business. The more expensive a downtime, the more valuable these tasks are. This justification for an operations team is well-known and well understood.

Improving TTM is primarily about everything other than the production environment. These tasks are only done because development is ongoing and the primary beneficiaries of this work are the internal users (developers, QA, etc). This aspect includes:
  • Creating and managing developer environments
  • Creating and managing CI environments and the CI process
  • Creating and managing a consistent deployment process
  • Providing support for all non-production environments (such as QA, Load, Demo, Train, etc)
  • Creating and managing a consistent promotion process, tied to the issue tracker.
  • Managing and tightening all feedback loops
Again, these tasks are often handled by staff outside a formal operations team, but these tasks are still operational in nature; they focus on the operations of building the business's applications.

The value of these tasks is directly proportional to how expensive a delay in delivering changes to production is to the business. Note I said changes and not new functionality. Improving TTM also improves delivering bugfixes and security updates. For many businesses, the marginal value is of increasing TTM is very low. I once worked on an application targeted at small governments where the users wanted 3-month delivery cycles. The flip-side is Amazon or Google or Netflix where new functionality is delivered every 5-10 minutes.

In short, the value of the operations team is based on the business's needs. The more important a business values its uptime and TTM, the more value a dedicated operations team provides.

Monday, May 1, 2017

Announcing Devops Katas

The best way to learn a new tool is to use it. In the development world, there are hundreds of short exercises called katas specifically for this. They're designed to take anywhere from 1-4 hours and help you learn a new language, a new technique, a new library, etc. You go through the process of writing tests, writing code, and seeing something at the end.

One thing that distinguishes good katas is the framework. There's a place you go to with all the files necessary to write tests and write code already in place. The only things missing are the tests and the code. You can focus on the problem at hand instead of the formalities necessary to do a good job. For example:
For devops, even though we have more tools, we haven't had anything resembling a kata. Instead, we have things like:
These aren't bad. It's very helpful to have a focused exercise to work with. Except, I can't share my work with you. You can't ask me questions. We can't build on this exercise together. The bowling kata, FizzBuzz, roman numerals - these are all well-known and understood exercises from the development world, regardless of the language you work in.

In that vein, I'm releasing "proper" devops katas at These katas have a Vagrant and Serverspec framework to work within, giving the user the ability to do proper TDD in devops. I'll be releasing at least one/month and would love to get feedback on the next one to do.

Monday, April 24, 2017

Announcing SimsLoader

I'm happy to announce the official release of SimsLoader, a new tool for generating test data for relational databases. It's opensource, available as a Docker image, and works with MySQL, Postgres, Oracle, and SQL Server.

Unlike existing tools, SimsLoader doesn't assume it knows anything about your database. Instead, it reads your schema each time. This means:

  • You only tell it what you care about
  • It automagically handles schema changes
SimsLoader requires a minimum of information. You tell it what you want and it figures out everything else you need. Whether it's a NOT NULL column or a row in a parent table, SimsLoader will fill in everything necessary to give you what you asked for. And give it all back to you.

SimsLoader works by taking two configuration files - a model file and a specification file - in either YAML or JSON. The specification file contains what you want, specified with as much or as little detail as you want. At minimum, you can say:

users: 100

And 100 new rows in the users table are created with all the necessary columns filled in. If the users table has foreign keys, rows in those tables are created and on down the line until every row has everything it needs.

The specification file can be as complex as you want it to be. For example:

  - name: John Doe
      type: timestamp_in_past_5_years
      name: Acme Industries
  - name: Jane Doe
      type: timestamp_in_past_2_years
      name: Acme Corporation
    type: timestamp_in_past_2_years
  lineitems: 5

If you load this specification file, the returned value would be a YAML document with two users and one invoice. Even though you specify two new organizations and five lineitems, those are attributes of the three rows you actually asked for.

The model file adds information to SimsLoader's understanding of your database. It's primarily used to add type information to the various columns. For example:
    name: us_name
    address: us_address
    city: us_city
    state: us_state
    zipcode: us_zipcode
    started_on: timestamp_in_psat
        - invoice_id
        source: invoices
          - id

Now, whenever a row in the users table is generated, if a name isn't provided, it will be filled in with a reasonable-looking name from the US. If this hadn't been specified, the column would be filled with random characters (or potentially left blank, if it is a nullable column). In addition, you can specify missing foreign keys (has_many as in the example or the reverse belongs_to), missing unique constraints, and several other aspects.

Thursday, April 6, 2017

Vagrant on Hyper-V: A crazy journey

I love Vagrant. It is the most important devops tool, period. Frankly, far more important than Docker. I use Vagrant with Virtualbox, AWS, Azure, VSphere ... with pretty much every virtualization environment in the world. (Except Docker.)

Funnily enough, it was when I started to heavily use Docker (for SimsLoader) and upgrading one of my laptops to Windows 10 that led me to using Vagrant with Hyper-V. Hyper-V is really cool. Docker runs "natively" under it and it's fast. Really fast.

Unfortunately, Microsoft's work to get all their cool stuff to function with all the opensource tools the Linux ecology has been working with for years. There have been a few posts on this, but they weren't sufficient or complete for me to get up and running.

So, this post is to track and list all the necessary steps to be successful with Vagrant and Hyper-V. These steps have been validated in Windows 10. They may work in Windows 8.1 - if they do, could someone tell me?

Also, if you have cool Powershell scripts to do any of these manual steps, please let me know. Ideally, there's a script that does the "only Once" stuff and a script that does the "Occasionally" stuff.

Stuff you do only Once

Enable Hyper-V

This enables the Hyper-V virtualization provider.
  1. Launch "Turn Windows features on or off"
  2. Scroll down to select "Hyper-V"
  3. Reboot
If you installed the "native" Docker, then you've probably already done this step.

Note: This will disable Virtualbox and all other virtualization providers. To re-enable them, reverse these steps and reboot.

Create a virtual switch

This provides the basis for networking within Hyper-V. Unlike Virtualbox, Hyper-V doesn't auto-create networks for you.
  1. Launch "Hyper-V Manager"
  2. Click on "Virtual Switch Manager..." (right-hand side)
  3. Select "Internal"
  4. Click on "Create Virtual Switch"
  5. Name the switch "Shared"
  6. Select the "Internal Network" radio button
  7. Click "Ok"
  8. Exit the "Hyper-V Manager" window

Connect the virtual switch to an existing network

This is required if you want your Vagrant-launched VM to get a DHCP-assigned IP address. Again, unlike Virtualbox, Hyper-V networks don't auto-allocate IP addresses.
  1. Go to "Control Panel > Network and Internet > Network Connections"
  2. Select a network connection with Internet (likely your Wireless)
  3. Right-click, select "Properties"
  4. Select the "Sharing" tab
  5. Click the "Allow other network users to connect through this computer's Internet connection"
  6. Select "vEthernet (Shared)"
  7. Click "Ok"
  8. Exit the "Network Connections" window

Disable SMB idle disconnects

Again, unlike Virtualbox, Hyper-V doesn't provide a file-sharing mechanism. (Granted, vboxfs isn't the greatest, but at least it's immediately functional). So, Vagrant uses SMB sharing. By default, Windows hosts will disconnect the SMB connection if it's idle for too long, which requires you to reboot your VM to reconnect the shared folder.
  1. In an administrative powershell, run "net config server /autodisconnect:-1"
Note: this may or may not be what you want to do for other purposes, particularly if you use SMB for other purposes.

Stuff you will have to do occasionally

Resharing the network with your virtual switch

If your internet-enabled network changes, you will need to attach your virtual switch to it. For example, if you switch from wireless to wired.

If your sharing network changes, you will need to reconnect your virtual switch. For example, if you go from home to work or work to Starbucks.

I haven't experienced any problems if I put my laptop to sleep, but awaken onto the same network, but that is also a possible time you may have to reshare your network.

Stuff you will have to do all the time

Launching a VM

  1. Open a terminal with administrative privileges. (Hyper-V requires this.)
  2. "vagrant up"
  3. You will be asked for your windows username/password to mount shared folders.
    • Make sure it's the Windows username/password
    • Depending on how your laptop is setup, you may or may not have to add a domain to that. (mmouse vs. mmouse@disney)

Additional Documentation