Monday, April 24, 2017

Announcing SimsLoader

I'm happy to announce the official release of SimsLoader, a new tool for generating test data for relational databases. It's opensource, available as a Docker image, and works with MySQL, Postgres, Oracle, and SQL Server.

Unlike existing tools, SimsLoader doesn't assume it knows anything about your database. Instead, it reads your schema each time. This means:

  • You only tell it what you care about
  • It automagically handles schema changes
SimsLoader requires a minimum of information. You tell it what you want and it figures out everything else you need. Whether it's a NOT NULL column or a row in a parent table, SimsLoader will fill in everything necessary to give you what you asked for. And give it all back to you.

SimsLoader works by taking two configuration files - a model file and a specification file - in either YAML or JSON. The specification file contains what you want, specified with as much or as little detail as you want. At minimum, you can say:

users: 100

And 100 new rows in the users table are created with all the necessary columns filled in. If the users table has foreign keys, rows in those tables are created and on down the line until every row has everything it needs.

The specification file can be as complex as you want it to be. For example:

  - name: John Doe
      type: timestamp_in_past_5_years
      name: Acme Industries
  - name: Jane Doe
      type: timestamp_in_past_2_years
      name: Acme Corporation
    type: timestamp_in_past_2_years
  lineitems: 5

If you load this specification file, the returned value would be a YAML document with two users and one invoice. Even though you specify two new organizations and five lineitems, those are attributes of the three rows you actually asked for.

The model file adds information to SimsLoader's understanding of your database. It's primarily used to add type information to the various columns. For example:
    name: us_name
    address: us_address
    city: us_city
    state: us_state
    zipcode: us_zipcode
    started_on: timestamp_in_psat
        - invoice_id
        source: invoices
          - id

Now, whenever a row in the users table is generated, if a name isn't provided, it will be filled in with a reasonable-looking name from the US. If this hadn't been specified, the column would be filled with random characters (or potentially left blank, if it is a nullable column). In addition, you can specify missing foreign keys (has_many as in the example or the reverse belongs_to), missing unique constraints, and several other aspects.

Thursday, April 6, 2017

Vagrant on Hyper-V: A crazy journey

I love Vagrant. It is the most important devops tool, period. Frankly, far more important than Docker. I use Vagrant with Virtualbox, AWS, Azure, VSphere ... with pretty much every virtualization environment in the world. (Except Docker.)

Funnily enough, it was when I started to heavily use Docker (for SimsLoader) and upgrading one of my laptops to Windows 10 that led me to using Vagrant with Hyper-V. Hyper-V is really cool. Docker runs "natively" under it and it's fast. Really fast.

Unfortunately, Microsoft's work to get all their cool stuff to function with all the opensource tools the Linux ecology has been working with for years. There have been a few posts on this, but they weren't sufficient or complete for me to get up and running.

So, this post is to track and list all the necessary steps to be successful with Vagrant and Hyper-V. These steps have been validated in Windows 10. They may work in Windows 8.1 - if they do, could someone tell me?

Also, if you have cool Powershell scripts to do any of these manual steps, please let me know. Ideally, there's a script that does the "only Once" stuff and a script that does the "Occasionally" stuff.

Stuff you do only Once

Enable Hyper-V

This enables the Hyper-V virtualization provider.
  1. Launch "Turn Windows features on or off"
  2. Scroll down to select "Hyper-V"
  3. Reboot
If you installed the "native" Docker, then you've probably already done this step.

Note: This will disable Virtualbox and all other virtualization providers. To re-enable them, reverse these steps and reboot.

Create a virtual switch

This provides the basis for networking within Hyper-V. Unlike Virtualbox, Hyper-V doesn't auto-create networks for you.
  1. Launch "Hyper-V Manager"
  2. Click on "Virtual Switch Manager..." (right-hand side)
  3. Select "Internal"
  4. Click on "Create Virtual Switch"
  5. Name the switch "Shared"
  6. Select the "Internal Network" radio button
  7. Click "Ok"
  8. Exit the "Hyper-V Manager" window

Connect the virtual switch to an existing network

This is required if you want your Vagrant-launched VM to get a DHCP-assigned IP address. Again, unlike Virtualbox, Hyper-V networks don't auto-allocate IP addresses.
  1. Go to "Control Panel > Network and Internet > Network Connections"
  2. Select a network connection with Internet (likely your Wireless)
  3. Right-click, select "Properties"
  4. Select the "Sharing" tab
  5. Click the "Allow other network users to connect through this computer's Internet connection"
  6. Select "vEthernet (Shared)"
  7. Click "Ok"
  8. Exit the "Network Connections" window

Disable SMB idle disconnects

Again, unlike Virtualbox, Hyper-V doesn't provide a file-sharing mechanism. (Granted, vboxfs isn't the greatest, but at least it's immediately functional). So, Vagrant uses SMB sharing. By default, Windows hosts will disconnect the SMB connection if it's idle for too long, which requires you to reboot your VM to reconnect the shared folder.
  1. In an administrative powershell, run "net config server /autodisconnect:-1"
Note: this may or may not be what you want to do for other purposes, particularly if you use SMB for other purposes.

Stuff you will have to do occasionally

Resharing the network with your virtual switch

If your internet-enabled network changes, you will need to attach your virtual switch to it. For example, if you switch from wireless to wired.

If your sharing network changes, you will need to reconnect your virtual switch. For example, if you go from home to work or work to Starbucks.

I haven't experienced any problems if I put my laptop to sleep, but awaken onto the same network, but that is also a possible time you may have to reshare your network.

Stuff you will have to do all the time

Launching a VM

  1. Open a terminal with administrative privileges. (Hyper-V requires this.)
  2. "vagrant up"
  3. You will be asked for your windows username/password to mount shared folders.
    • Make sure it's the Windows username/password
    • Depending on how your laptop is setup, you may or may not have to add a domain to that. (mmouse vs. mmouse@disney)

Additional Documentation