tag:blogger.com,1999:blog-24582938135338906022024-02-06T21:25:43.658-05:00~Streamlined~The process of application delivery is convoluted and often filled with bottlenecks. A better process often yields better results.Rob Kinyonhttp://www.blogger.com/profile/07966252008096453177noreply@blogger.comBlogger38125tag:blogger.com,1999:blog-2458293813533890602.post-59671054069660102122017-05-18T10:52:00.000-04:002017-05-18T10:52:10.415-04:00What is devops?<a href="http://www.itskeptic.org/content/define-devops">Ask</a> <a href="https://www.quora.com/What-is-DevOps">12</a> <a href="https://devops.com/surprise-broad-agreement-on-the-definition-of-devops/">engineers</a> <a href="https://www.devopsguys.com/2015/03/16/5-definitions-of-devops/">for</a> <a href="http://www.drdobbs.com/architecture-and-design/what-exactly-is-devops/240009147">a</a> <a href="https://theagileadmin.com/what-is-devops/">definition</a> <a href="http://searchitoperations.techtarget.com/definition/DevOps">of</a> <a href="https://en.wikipedia.org/wiki/DevOps">devops</a> <a href="https://www.versionone.com/devops-101/what-is-devops/">and</a> <a href="http://www.webopedia.com/TERM/D/devops_development_operations.html">you'll</a> <a href="https://aws.amazon.com/devops/what-is-devops/">get</a> <a href="http://www.gartner.com/it-glossary/devops/">13</a> <a href="https://newrelic.com/devops/what-is-devops">answers</a>. Yes, that's 13 different links to everyone from Gartner to AWS to Dr Dobbs. There's plenty more on Google, too. Go ahead - take the time to read through them. AWS even has a <a href="https://pages.awscloud.com/ln_ImportWINServertoEC2_ps.html?sc_channel=PS&sc_campaign=PS_DevOps_Solution&sc_publisher=google&sc_medium=lg_solution_devops_nb&sc_content=devops_generic_e&sc_detail=devop&sc_category=devops_solution&sc_segment=193744002465&sc_matchtype=p&sc_country=US&s_kwcid=AL!4422!3!193744002465!p!!g!!devop&ef_id=WRtHjQAABDt4xns-:20170516184013:s">whitepaper on devops</a> which, hilariously enough, doesn't match up with their definition from the first sentence.<br />
<br />
Confused now? Everyone else seems to be, too. The problem is every one of those answers, when you parse it down, is a long-winded way of saying <a href="http://corporate.findlaw.com/litigation-disputes/movie-day-at-the-supreme-court-or-i-know-it-when-i-see-it-a.html">"I know it when I see it."</a> Which may be true, but it's also worthless for you. You can't <u>do</u> anything with that because the author isn't with you to help you whiteboard what devops means in <u>your</u> organization. This lack of clarity leads intelligent people to discard the term "devops" as meaningless and try and forge another way.<br />
<br />
After working with a dozen clients in the past couple years, I think the problem is everyone is looking for a singular <u>prescriptive</u> definition. Devops is <i>this thing</i> and not <i>that thing</i> and those things are the same for everyone. And, there isn't one.<br />
<br />
Some of the links above provide a lot of words around the characteristics that some successful organizations have. But, these characteristics (no silos, automation, etc) don't tell you how to solve your problem. It's as if doctors decided to practice medicine by looking at the healthiest people in the world and giving all their patients a list of those characteristics in response to every patient visit. That can be helpful ("lose weight" is a good thing to do, in general), but it can also be useless ("lose age" isn't helpful, even though most healthy people are between 20-30.)<br />
<br />
Instead, I offer a <u>descriptive</u> definition which enables <a href="https://en.wikipedia.org/wiki/SMART_criteria">SMART</a> milestones.<br />
<blockquote class="tr_bq">
Devops is developing applications whose business domain is your operations.</blockquote>
This isn't a singular <i>thing</i>. This isn't the same for everyone. This doesn't tell you what else you have to do, either. For example, you don't have to do Agile, if you don't want to. (Yes, you can devops within Waterfall, if you want.) But, it tells you exactly what you should be doing in order to <i>do the devops</i> and how.<br />
<br />
Why this definition and not some statement about automation or build/release engineering? For one thing, this definition includes all those ideas. (Well, it <i>leads</i> to them, as we'll see.) For another, this definition recognizes that every organization is different. Each organization has different requirements for how they want to do IT and any definition of devops needs to accommodate that.<br />
<br />
So, what do you do with that definition? The first thing is to gather requirements, just like any other application. Topics like:<br />
<ul>
<li>How do <u>you</u> (the IT teams) want to operate?</li>
<li>What does the business side expect from the IT department?</li>
<li>What things suck right now?</li>
</ul>
<div>
The second thing is to figure out what you actually do today. I mean, <u>actually</u> do. Can you bring in a new senior person anywhere in IT and give them a clear picture of how your organization does IT work within the first day? First week? First month? Most clients I've worked with struggle with integrating new staff, with 1-year veterans saying "I never knew how we did that before."<br />
<br />
Once you have both, you can describe the gap. Bridging that gap is your devops journey. That journey is going to be very different from company to company and even between teams at the same company. Which makes sense.</div>
Rob Kinyonhttp://www.blogger.com/profile/07966252008096453177noreply@blogger.com0tag:blogger.com,1999:blog-2458293813533890602.post-5185786779182376362017-05-15T09:26:00.000-04:002017-05-15T11:02:31.556-04:00Why have an Operations team?I just came back from a meeting with the manager of a team I'm working with. The meeting went well, but at the end, he asked me a really interesting question.<br />
<blockquote class="tr_bq">
How do I justify an ops team (or a devops team) to my executives?</blockquote>
I sat back for a second and realized I've never heard anyone actually lay out the case for an operations team. It's always been <i>assumed </i>that you just have ops, just like you have devs and QA. Of course you just have them.<br />
<br />
To answer, let's start with a premise.<br />
<blockquote class="tr_bq">
The business gains value from users consuming its application(s).</blockquote>
This should be self-evident; it's the whole point of the business funding an application.<br />
<br />
Developers, obviously, build new features which creates more value. That's the whole <u>point</u> of developers - to modify the application to create new features to create new business value. We'll call this a direct contribution. Their work has immediate and visible impact on users consuming the business's applications.<br />
<br />
The business analysts who write the requirements, the designers who create the UI/UX - they also provide a direct contribution.<br />
<br />
Operations and QA, however, don't make that direct value contribution. Their work provides an <i>indirect</i> value contribution. The QA team can get its own post some other time. As for the Ops team, they do two things:<br />
<br />
<ol>
<li>Reduce business risk by maintaining the user experience.</li>
<li>Improve time-to-market (TTM) for new changes, such as features, bugfixes, and security.</li>
</ol>
<div>
Let's take each one in its own.</div>
<div>
<br /></div>
<div>
Maintaining the user experience is primarily maintaining and supporting the production environment. These are tasks that would have to be done even if all development was halted. The primary beneficiary of this work is the external user. This aspect includes:</div>
<div>
<ul>
<li>Gathering and analyzing logs</li>
<li>Gathering and reporting on metrics</li>
<li>Tracking and applying updates to third-party services (such as IIS, Windows, etc)</li>
<li>Monitoring for and alerting on any issues that arise (such as server failures)</li>
<li>Maintaining awareness of security issues (such as hacking)</li>
<li>Maintaining awareness of performance issues (such as bad database queries)</li>
<li>Reducing and/or eliminating deployment downtime</li>
</ul>
<div>
These tasks may be handled by people outside a formal operations team (such as developers handling performance issues), but these tasks are within that operations <i>role</i>.<br />
<br />
The value of these tasks is directly proportional to how expensive any specific failure in production is to the business. The more expensive a downtime, the more valuable these tasks are. This justification for an operations team is well-known and well understood.</div>
</div>
<div>
<br /></div>
<div>
Improving TTM is primarily about everything <u>other</u> than the production environment. These tasks are only done because development is ongoing and the primary beneficiaries of this work are the internal users (developers, QA, etc). This aspect includes:</div>
<div>
<ul>
<li>Creating and managing developer environments</li>
<li>Creating and managing CI environments and the CI process</li>
<li>Creating and managing a consistent deployment process</li>
<li>Providing support for all non-production environments (such as QA, Load, Demo, Train, etc)</li>
<li>Creating and managing a consistent promotion process, tied to the issue tracker.</li>
<li>Managing and tightening all feedback loops</li>
</ul>
<div>
Again, these tasks are often handled by staff outside a formal operations team, but these tasks are still <i>operational</i> in nature; they focus on the <i>operations</i> of building the business's applications.<br />
<br />
The value of these tasks is directly proportional to how expensive a delay in delivering changes to production is to the business. Note I said <i><u><b>changes</b></u></i> and not <i><u>new functionality</u></i>. Improving TTM also improves delivering bugfixes and security updates. For many businesses, the marginal value is of increasing TTM is very low. I once worked on an application targeted at small governments where the <i>users</i> wanted 3-month delivery cycles. The flip-side is Amazon or Google or Netflix where new functionality is delivered every 5-10 minutes.<br />
<br />
In short, the value of the operations team is based on the business's needs. The more important a business values its uptime and TTM, the more value a dedicated operations team provides.</div>
</div>
Rob Kinyonhttp://www.blogger.com/profile/07966252008096453177noreply@blogger.com0tag:blogger.com,1999:blog-2458293813533890602.post-74044636938721212042017-05-01T15:15:00.002-04:002017-05-01T15:15:28.781-04:00Announcing Devops KatasThe best way to learn a new tool is to use it. In the development world, there are hundreds of short exercises called <i>katas</i> specifically for this. They're designed to take anywhere from 1-4 hours and help you learn a new language, a new technique, a new library, etc. You go through the process of writing tests, writing code, and seeing something at the end.<div>
<br /></div>
<div>
One thing that distinguishes good katas is the framework. There's a place you go to with all the files necessary to write tests and write code <u>already in place</u>. The only things missing are the tests and the code. You can focus on the problem at hand instead of the formalities necessary to do a good job. For example:</div>
<div>
<ul>
<li><a href="http://cyber-dojo.org/">http://cyber-dojo.org/</a></li>
<li><a href="http://www.butunclebob.com/ArticleS.UncleBob.TheBowlingGameKata">http://www.butunclebob.com/ArticleS.UncleBob.TheBowlingGameKata</a></li>
</ul>
<div>
<div>
For devops, even though we have more tools, we haven't had anything resembling a kata. Instead, we have things like:</div>
</div>
</div>
<div>
<ul>
<li><a href="https://github.com/devtoprod/devopskatas">https://github.com/devtoprod/devopskatas</a></li>
<li><a href="http://devopsy.com/blog/2013/08/16/devops-kata-single-line-of-code/">http://devopsy.com/blog/2013/08/16/devops-kata-single-line-of-code/</a></li>
</ul>
<div>
These aren't <u>bad</u>. It's very helpful to have a focused exercise to work with. Except, I can't share my work with you. You can't ask me questions. We can't <i>build</i> on this exercise together. The bowling kata, FizzBuzz, roman numerals - these are all well-known and understood exercises from the development world, regardless of the language you work in.</div>
</div>
<div>
<br /></div>
<div>
In that vein, I'm releasing "proper" <a href="https://github.com/greenfishbluefish/devops-katas">devops katas</a> at https://github.com/greenfishbluefish/devops-katas. These katas have a Vagrant and Serverspec framework to work within, giving the user the ability to do proper TDD in devops. I'll be releasing at least one/month and would love to get feedback on the next one to do.</div>
Rob Kinyonhttp://www.blogger.com/profile/07966252008096453177noreply@blogger.com0tag:blogger.com,1999:blog-2458293813533890602.post-87323343421499804262017-04-24T20:38:00.000-04:002017-04-24T20:38:33.558-04:00Announcing SimsLoaderI'm happy to announce the official release of <a href="https://hub.docker.com/r/robkinyon/sims_loader/">SimsLoader</a>, a new tool for generating test data for relational databases. It's opensource, available as a Docker image, and works with MySQL, Postgres, Oracle, and SQL Server.<br />
<br />
Unlike existing tools, SimsLoader doesn't assume it knows anything about your database. Instead, it reads your schema each time. This means:<br />
<br />
<ul>
<li>You only tell it what you care about</li>
<li>It automagically handles schema changes</li>
</ul>
SimsLoader requires a minimum of information. You tell it what you want and it figures out everything else you need. Whether it's a NOT NULL column or a row in a parent table, SimsLoader will fill in everything necessary to give you what you asked for. And give it all back to you.<br />
<br />
SimsLoader works by taking two configuration files - a model file and a specification file - in either YAML or JSON. The specification file contains what you want, specified with as much or as little detail as you want. At minimum, you can say:<br />
<br />
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: "andale mono" , "lucida console" , "monaco" , "fixed" , monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 100%;"><code>users: 100
</code></pre>
<br />
And 100 new rows in the users table are created with all the necessary columns filled in. If the users table has foreign keys, rows in those tables are created and on down the line until every row has everything it needs.<br />
<br />
The specification file can be as complex as you want it to be. For example:<br />
<br />
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: "andale mono" , "lucida console" , "monaco" , "fixed" , monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 100%;"><code>users:
- name: John Doe
started_on:
type: timestamp_in_past_5_years
organization:
name: Acme Industries
- name: Jane Doe
started_on:
type: timestamp_in_past_2_years
organization:
name: Acme Corporation
invoices:
created:
type: timestamp_in_past_2_years
lineitems: 5
</code></pre>
<br />
If you load this specification file, the returned value would be a YAML document with two users and one invoice. Even though you specify two new organizations and five lineitems, those are attributes of the three rows you actually asked for.<br />
<br />
The model file adds information to SimsLoader's understanding of your database. It's primarily used to add type information to the various columns. For example:<br />
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: "andale mono" , "lucida console" , "monaco" , "fixed" , monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 100%;"><code>users:
columns:
name: us_name
address: us_address
city: us_city
state: us_state
zipcode: us_zipcode
started_on: timestamp_in_psat
has_many:
invoices:
columns:
- invoice_id
foreign:
source: invoices
columns:
- id
</code></pre>
<br />
Now, whenever a row in the users table is generated, if a name isn't provided, it will be filled in with a reasonable-looking name from the US. If this hadn't been specified, the column would be filled with random characters (or potentially left blank, if it is a nullable column). In addition, you can specify missing foreign keys (<span style="color: #3d85c6; font-family: Arial, Helvetica, sans-serif;">has_many</span> as in the example or the reverse <span style="color: #3d85c6; font-family: Arial, Helvetica, sans-serif;">belongs_to</span>), missing unique constraints, and several other aspects.Rob Kinyonhttp://www.blogger.com/profile/07966252008096453177noreply@blogger.com0tag:blogger.com,1999:blog-2458293813533890602.post-63811362499121618312017-04-06T08:31:00.000-04:002017-04-06T08:31:08.350-04:00Vagrant on Hyper-V: A crazy journeyI love <a href="http://streamlined-book.blogspot.com/2013/08/use-vagrant-for-great-good.html">Vagrant</a>. It is the most important devops tool, period. Frankly, far more important than Docker. I use Vagrant with Virtualbox, AWS, Azure, VSphere ... with pretty much every virtualization environment in the world. (Except Docker.)<br />
<br />
Funnily enough, it was when I started to heavily use Docker (for <a href="https://hub.docker.com/r/robkinyon/sims_loader/">SimsLoader</a>) <u>and</u> upgrading one of my laptops to Windows 10 that led me to using Vagrant with <a href="https://www.microsoft.com/en-us/cloud-platform/server-virtualization">Hyper-V</a>. Hyper-V is really cool. Docker runs "natively" under it and it's <u>fast</u>. Really fast.<br />
<br />
Unfortunately, Microsoft's work to get all their cool stuff to function with all the opensource tools the Linux ecology has been working with for years. There have been a <a href="https://lewiswalsh.com/node-on-ubuntu-on-vagrant-on-hyperv/">few</a> <a href="http://www.hurryupandwait.io/blog/running-an-ubuntu-guest-on-hyper-v-assigned-an-ip-via-dhcp-over-a-wifi-connection">posts</a> <a href="https://blog.areflyen.no/2012/10/10/setting-up-internet-access-for-hyper-v-with-nat-in-windows-8/">on</a> this, but they weren't sufficient or complete for me to get up and running.<br />
<br />
So, this post is to track and list all the necessary steps to be successful with Vagrant and Hyper-V. These steps have been validated in Windows 10. They <u>may</u> work in Windows 8.1 - if they do, could someone tell me?<br />
<br />
Also, if you have cool Powershell scripts to do any of these manual steps, please let me know. Ideally, there's a script that does the "only Once" stuff and a script that does the "Occasionally" stuff.<br />
<h3>
Stuff you do only Once</h3>
<h4>
Enable Hyper-V</h4>
<div>
This enables the Hyper-V virtualization provider.</div>
<div>
<ol>
<li>Launch "Turn Windows features on or off"</li>
<li>Scroll down to select "Hyper-V"</li>
<li>Reboot</li>
</ol>
<div>
If you installed the "native" Docker, then you've probably already done this step.</div>
<div>
<br /></div>
Note: This will disable Virtualbox and all other virtualization providers. To re-enable them, reverse these steps and reboot.<br />
<h4>
Create a virtual switch</h4>
</div>
<div>
This provides the basis for networking within Hyper-V. Unlike Virtualbox, Hyper-V doesn't auto-create networks for you.</div>
<div>
<ol>
<li>Launch "Hyper-V Manager"</li>
<li>Click on "Virtual Switch Manager..." (right-hand side)</li>
<li>Select "Internal"</li>
<li>Click on "Create Virtual Switch"</li>
<li>Name the switch "Shared"</li>
<li>Select the "Internal Network" radio button</li>
<li>Click "Ok"</li>
<li>Exit the "Hyper-V Manager" window</li>
</ol>
<h4>
Connect the virtual switch to an existing network</h4>
</div>
<div>
This is required if you want your Vagrant-launched VM to get a DHCP-assigned IP address. Again, unlike Virtualbox, Hyper-V networks don't auto-allocate IP addresses.</div>
<div>
<ol>
<li>Go to "Control Panel > Network and Internet > Network Connections"</li>
<li>Select a network connection with Internet (likely your Wireless)</li>
<li>Right-click, select "Properties"</li>
<li>Select the "Sharing" tab</li>
<li>Click the "Allow other network users to connect through this computer's Internet connection"</li>
<li>Select "vEthernet (Shared)"</li>
<li>Click "Ok"</li>
<li>Exit the "Network Connections" window</li>
</ol>
<h4>
Disable SMB idle disconnects</h4>
</div>
<div>
Again, unlike Virtualbox, Hyper-V doesn't provide a file-sharing mechanism. (Granted, vboxfs isn't the greatest, but at least it's immediately functional). So, Vagrant uses SMB sharing. By default, Windows hosts will disconnect the SMB connection if it's idle for too long, which requires you to reboot your VM to reconnect the shared folder.</div>
<div>
<ol>
<li>In an administrative powershell, run "net config server /autodisconnect:-1"</li>
</ol>
</div>
<div>
Note: this may or may not be what you want to do for other purposes, particularly if you use SMB for other purposes.</div>
<h3>
Stuff you will have to do occasionally</h3>
<h4>
Resharing the network with your virtual switch</h4>
<div>
If your internet-enabled network changes, you will need to attach your virtual switch to it. For example, if you switch from wireless to wired.</div>
<div>
<br /></div>
<div>
If your sharing network changes, you will need to reconnect your virtual switch. For example, if you go from home to work or work to Starbucks.</div>
<div>
<br /></div>
<div>
I haven't experienced any problems if I put my laptop to sleep, but awaken onto the same network, but that is also a possible time you may have to reshare your network.</div>
<h3>
Stuff you will have to do all the time</h3>
<h4>
Launching a VM</h4>
<div>
<ol>
<li>Open a terminal with <u>administrative privileges</u>. (Hyper-V requires this.)</li>
<li>"vagrant up"</li>
<li>You will be asked for your windows username/password to mount shared folders.</li>
<ul>
<li>Make sure it's the Windows username/password</li>
<li>Depending on how your laptop is setup, you may or may not have to add a domain to that. (mmouse vs. mmouse@disney)</li>
</ul>
</ol>
<h3>
Additional Documentation</h3>
</div>
<div>
<ul>
<li><a href="https://www.vagrantup.com/docs/hyperv/">https://www.vagrantup.com/docs/hyperv/</a></li>
<li><a href="https://www.vagrantup.com/docs/hyperv/configuration.html">https://www.vagrantup.com/docs/hyperv/configuration.html</a></li>
<li><a href="https://www.vagrantup.com/docs/synced-folders/smb.html">https://www.vagrantup.com/docs/synced-folders/smb.html</a></li>
</ul>
</div>
Rob Kinyonhttp://www.blogger.com/profile/07966252008096453177noreply@blogger.com0tag:blogger.com,1999:blog-2458293813533890602.post-8175555817249792622015-12-16T22:39:00.004-05:002015-12-16T22:39:45.786-05:00Evaluating a Devops teamEnd of year has rolled around. Along with new budgets starting, old budgets ending, and bad holiday parties where you spend two hours avoiding <i>that</i> guy, this is the season of annual evaluations.<br />
<br />
You're the manager of a devops team, possibly one newly-formed this year. This devops things is all new. The team operates completely differently from the other sysadmin / operations teams you've seen. So, how do you evaluate them?<br />
<br />
An operations team is traditionally measured by things like:<br />
<ul>
<li>Production uptime percentage (high)</li>
<li>Frequency and length of production downtimes (low)</li>
</ul>
<div>
Everything is focused around the production environment and how well it stays available. These goals lead the team to fear and avoid changes to production. They also lead the team to disregard everything <u>other</u> than production as less important, or even unimportant. Not every operations team is like this, but every human will eventually conform to the actions that are rewarded and avoid the ones that aren't.<br />
<br />
Our devops team, though, wants to accomplish different things. This team wants to deliver changes to production rapidly, even daily or hourly. They talk about reconstructing production on every deployment. Reconstructing lower environments on every deployment, too. They also talk about production in a very different way, not focused on external users.<br />
<br />
If we want our fledgling devops team to keep doing these things, as different as they are from the traditional operations team, then we need to change what we measure. Or, rather, add to that list. That list is important. Operations teams <u>are</u> responsible for production uptime and need to be measured on that. But, the traditional mistake is <u>only</u> measuring on that.<br />
<br />
Let's add a few more items to that list.<br />
<br />
<ul>
<li>Cost of Deployment</li>
<ul>
<li>Time to deploy (low)</li>
<li>Mean time between deployments (low)</li>
<li>Number of people required for a deployment (low)</li>
</ul>
<li>Non-production friction</li>
<ul>
<li>Uptime percentage (high)</li>
<li>Frequency and length of downtimes (low)</li>
</ul>
<li>Number of manual tasks (low)</li>
</ul>
<div>
The last one may be unnecessary - it often falls out of what a devops team is trying to do. This team you sometimes struggle to understand isn't an operations team - it's a development team building an application which, when used, results in a new application environment. Yes, we measure them on operational efficiency, but it's much more than that.</div>
<div>
<br /></div>
<div>
Operational efficiency becomes a measurement of the applications the devops team has created, similar to signups and engagement metrics for the web applications we work so hard to deploy. Instead, we start to measure them in how well the application performs its job.</div>
</div>
Rob Kinyonhttp://www.blogger.com/profile/07966252008096453177noreply@blogger.com0tag:blogger.com,1999:blog-2458293813533890602.post-64280938016304653352015-09-24T17:26:00.002-04:002015-09-24T17:26:43.463-04:00Opensource isn't a right and it's not freeThe trigger for this post is <a href="https://groups.google.com/forum/#!topic/columbusrb/sqN5ZQ2DsG4">a reasonable request from a Windows user</a>. Well, it's reasonable on its face. <a href="http://www.oxforddictionaries.com/words/he-or-she-versus-they">They</a> put forward two questions, paraphrased below:<br />
<br />
<ol>
<li>Why is is so hard for opensource developers to get their stuff working on Windows?</li>
<li>Why don't opensource developers care about the Windows population?</li>
</ol>
<div>
When it comes to operating systems, Linux and OSX are practically kissing cousins. Linux grows out of Minix and OSX grows out of NetBSD. There's been significant cross-pollenation and agreement. Both are POSIX-compliant, have relatively sane package systems, and provide a consistent command-line interface and a common configuration mechanism. Filepaths differ, but that's a simple case-statement. The same tools work the same.</div>
<div>
<br /></div>
<div>
Windows, on the other hand, is different. Very different. The entire architecture of the thing is alien. It has different assumptions for what an OS does and how someone will use it. It puts things in different places. It has different ways of reading and setting configuration. It has <b>four</b> different commandlines, each of them completely different. It has <u>three</u> unrelated packaging systems, none of which put stuff anywhere like the others and which don't even acknowledge the others' existences.</div>
<div>
<br /></div>
<blockquote class="tr_bq">
That should answer the "Why don't it work right?" question.</blockquote>
<div>
<br /></div>
<div>
90% of all web applications are deployed onto some form of Linux. (Rarely on OSX, but that's the fault of Apple's licensing, not a problem with the platform. IIS may have had its day in the sun, but that sun has practically set.) Given Linux's historical problems with providing a good GUI, OSX has proven to be a great development platform for web applications destined for Linux. Given its great support for virtual machines, it also becomes a great platform for developing mobile applications (XCode and various VM solutions for Android). That hits the developer tetrafecta.</div>
<div>
<br /></div>
<div>
Because of this, most developers are very familiar with Linux and are likely using an OSX laptop. They will only use Windows for testing in IE.</div>
<div>
<br /></div>
<blockquote class="tr_bq">
This is what people are paid to do.</blockquote>
<div>
<br /></div>
<div>
Let's talk about who are opensource developers. The average OSS developer (insofar as there is an average) is someone who's paid to do development using OSS tools during the day. They will usually work on these modules to initially scratch an itch for work - the initial release is usually a donation from an employer. The developer then starts to take pride in it at home. Most OSS developers realize that these modules act as advertisement for their skills. (Every job offer I've received in the past 10+ years has referenced my OSS work in some way.)</div>
<div>
<br /></div>
<div>
There is a vast world out there of things that I could learn. Whole new ways of</div>
<div>
<ul>
<li>Storing and managing data</li>
<li>Interacting with huge datasets</li>
<li>Doing operations</li>
<li>Doing development</li>
<li>Thinking about parallel execution</li>
</ul>
<div>
I don't have a mousefart's chance in a windstorm to even keep up with what I already do for work. So why am I going to spend time learning how to manage a complex system that I will <u>never</u> use for work?</div>
</div>
<div>
<br /></div>
<div>
I'm not. And that's why I don't care if my opensource stuff works on Windows. If you want to make it work, then great. I love pull requests. If your employer wants to pay me to support it on Windows, we can discuss my rate.</div>
<div>
<br /></div>
<div>
But you don't have the right to make me work for free to accomplish something I don't care about. That's just not how things work.</div>
Rob Kinyonhttp://www.blogger.com/profile/07966252008096453177noreply@blogger.com0tag:blogger.com,1999:blog-2458293813533890602.post-56112414471259573142015-09-17T11:31:00.003-04:002015-09-17T11:31:15.713-04:00Devops - The toolchestI gotten the same question from a few people about <a href="http://streamlined-book.blogspot.com/2015/09/devops-where-do-i-start.html">Devops - Where do I start?</a>. "What exactly do you mean by everything?" Yes, there's a list in the post, but the question started to evolve over a few conversations to the related question "What is the full list of tools that a devops team needs to have an answer for".<br />
<br />
So, treat this list as a checklist. Your IT operations may not need an example of one or another of these items, but you need to have thought about why you don't need it.<br />
<br />
Caveats:<br />
<ol>
<li>I'm not interested in vim/emacs-type debates. So, questions of Puppet vs. Chef (in specific) aren't going to be in here. But, "configuration management" is going to be one of the line items. </li>
<li>My experience is primarily focused on web application development. Nearly everything is still correct for mobile, desktop, or embedded development. If there are differences in tool categories, I'd love to hear about them.</li>
</ol>
<h4>
Backoffice Operations</h4>
This is the list of tools/services that are necessary to just run a company today. It doesn't matter what your company does or even if it writes its own software.<br />
<ul>
<li>DNS</li>
<ul>
<li>Depending on how other things are set up, you may not need this explicitly</li>
</ul>
<li>Firewalls</li>
<ul>
<li>Depending on how other things are set up, you may not need this explicitly</li>
</ul>
<li>Mail (both receiving and sending)</li>
<ul>
<li>Something like Google Mail</li>
<li>You may use something additional for sending emails from an application</li>
</ul>
<li>Document Management</li>
<ul>
<li>Something like Google Docs</li>
</ul>
<li>Instant Messaging</li>
<ul>
<li>Something like HipChat or Slack. Your teams are going to be doing it anyways, so best get in front of it.</li>
</ul>
<li>Conferencing</li>
<ul>
<li>Google Hangouts, Skype, or even Appear.in. Your teams are going to be doing it anyways, so best get in front of it.</li>
</ul>
<li>Monitoring</li>
<ul>
<li>Yes, you want to monitor even these things</li>
<li>Depending on your topologies, you may have multiple monitoring tools for different purposes, such as Pingdom plus Icinga plus CloudWatch.</li>
</ul>
<li>Alerting / Escalation</li>
<ul>
<li>While Nagios may do alerting, it's best to have a specific tool to handle that. I like PagerDuty.</li>
</ul>
<li>Dashboards</li>
<ul>
<li>The best teams have an up-or-not dashboard for every service the company depends on, whether it's internal or external. It cuts down on the emails that say "Is X down?".</li>
<li>Dashboards make executives happy. Non-executives love happy executives. Therefore, non-executives love dashboards, especially ones that executives can manipulate.</li>
</ul>
</ul>
<h4>
Development Tools</h4>
<div>
This is a list of the internal-facing tools that enable your ability to do development.</div>
<div>
<ul>
<li>Source Control Management - specifically a DVCS like Git or Mercurial</li>
<li>Issue Tracker</li>
<ul>
<li>This needs to be linked to your SCM so that commits will change issue status</li>
</ul>
<li>Pull Request / Code Review tool</li>
<ul>
<li>This should be the only tool that can merge to master</li>
<li>This should also update the issue tracker</li>
</ul>
<li>Job Runner (aka, Continuous Integration / CI)</li>
<ul>
<li>Runs tests upon pull requests (on create and update)</li>
<li>Runs packaging upon merge to master</li>
</ul>
<li>Deployables repository</li>
<ul>
<li>Maven, Yum, Rubygems - there's lots of package types and you need to have a place to put them.</li>
</ul>
<li>Deployment process</li>
<ul>
<li>This is Chef / Puppet / Salt / OpsWorks / whatever.</li>
<li>This needs to be integrated into however you have constructed your SCM process</li>
<li>This needs to update your issue tracker</li>
<li>Ideally, anybody in the company should be able to push a button and the button does the right thing.</li>
</ul>
</ul>
<h4>
Production Tools</h4>
</div>
<div>
When you're running your web application, these are the services you need to consider. You will also need to consider the development environment version of each of these. For example, if you're using S3 as your file store, do you provide a development S3 bucket (with all the attendant issues of using a shared resource for multiple developers) or do you use something like <a href="https://github.com/jubos/fake-s3">fake-s3</a>?</div>
<div>
<ul>
<li>Load Balancer</li>
<ul>
<li>This includes SSL termination (you don't want to terminate SSL at your web application)</li>
</ul>
<li>CDN / Static files</li>
<ul>
<li>All your HTML, CSS, Javascript, images, and videos belong here.</li>
<li>This is different from your application caching layer, such as Squid (though you may reuse the same tool).</li>
</ul>
<li>Application servers</li>
<ul>
<li>Where all your code goes.</li>
<li>You may have multiple tiers of this, depending on your application's topologies.</li>
</ul>
<li>Metrics gathering</li>
<ul>
<li>Something like New Relic or Librato.</li>
<li>You may do monitoring on these metrics (for example, N internal server errors per M seconds)</li>
</ul>
<li>Application Caching</li>
<ul>
<li>This may be response caching (such as Squid) or data caching (such as Memcache)</li>
<li>It may be ephemeral caching (such as Memcache) or semi-permanent caching (such as Redis)</li>
</ul>
<li>Relational Database(s)</li>
<ul>
<li>More than just production, it's also the development choice and how those interact.</li>
</ul>
<li>Key-Value Database(s)</li>
<ul>
<li>More than just production, it's also the development choice and how those interact.</li>
</ul>
<li>Backups</li>
<ul>
<li>This includes how to test backups, including testing them in production</li>
<li>This includes disaster-recovery and offsite storage</li>
</ul>
<li>Data destruction policies</li>
<ul>
<li>This includes how to remove data from backups so that it's guaranteed to be destroyed</li>
</ul>
<li>Non-production environment construction</li>
<ul>
<li>Non-production environments cannot be the same as production. Exactly what tradeoffs are you making and why?</li>
<li>Developer environments are even less like production. How are you ensuring the developer environment is as close as possible so that "It works on my machine" is never uttered.</li>
</ul>
</ul>
</div>
Rob Kinyonhttp://www.blogger.com/profile/07966252008096453177noreply@blogger.com0tag:blogger.com,1999:blog-2458293813533890602.post-44357283800277360812015-09-11T15:03:00.003-04:002015-09-11T15:03:52.024-04:00Devops - The "Process"™<a href="http://streamlined-book.blogspot.com/2015/09/devops-where-do-i-start.html">Last post</a>, I laid out the argument for using source control in operations. To summarize, put all the things in source control so you can control them. Except tools don't control what people do - only processes can do that. So, let's work through what the process should be, now that everything is in a known place.<br />
<br />
First, what are we looking for in this process? It's really hard to know if you've achieved something if we don't know what we're trying to achieve. Sounds pretty obvious, I know, but think back - how many projects have you worked on where that <i>wasn't</i> done? How successful were those projects?<br />
<br />
What I want out of my operations process is this:<br />
<ol>
<li>A guarantee that every piece of infrastructure was:</li>
<ol>
<li>created solely from things in source control.</li>
<li>changed solely from things in source control.</li>
</ol>
<li>A guarantee that I can find out:</li>
<ol>
<li>what changed</li>
<li>why it changed</li>
<li>when it changed</li>
<li>who reviewed/approved the change</li>
<li>when it was applied to each instance in each environment</li>
<ol>
<li>and which instances in which environments it hasn't been applied to</li>
</ol>
<li>what is the dependency graph between this and other changes</li>
</ol>
</ol>
<div>
Oh, and it has to be unobtrusive and not get in the way and be easy to understand. Not so hard.</div>
<div>
<br /></div>
<div>
These guarantees look very similar to the guarantees that the standard Agile development processes provide. The first is obviously implemented - devs aren't allowed to touch deployed servers. So, any change they want to see in the application <i style="font-weight: bold;">must</i> come from things in source control. (This is different from most non-Agile processes where code changes to deployed servers are sometimes emailed (or even IM'ed!) from dev to ops and implemented by hand because changes are so infrequent.)</div>
<div>
<br /></div>
<div>
The second is implemented through discipline. There is nothing in Agile that says all changes must be associated with an issue, confined within a branch, and go through a multi-person development/review process. But, every Agile team I have ever seen or heard of does it that way because doing it any other way has led to uncomfortable situations and unanswerable questions. It's so ubiquitous that tools now treat this as "Agile mode". Github's pull request process even creates an issue# just to ensure that there is a record in the issue tracker.</div>
<div>
<br /></div>
<div>
In order to apply this process to operations, we will need discipline of our own. Any operations team, to do its job, will be applying changes manually. This gets the job done <u>now</u> and is easy to reason about. Not a bad thing. Except, how do you know that what was done is what was defined in source control? This is where the discipline comes in.</div>
<div>
<br /></div>
<div>
Ideally, all changes to deployed servers happen via script. If those scripts are executed from a place separate from the deployed servers, that's even better. (For example, <i style="font-weight: bold;">only</i> using the AWS SDK to touch your AWS infrastructure.) The script can be a Puppet/Chef/Salt thing or <a href="http://www.slideshare.net/RobKinyon/ruby-the-language-of-devops">Ruby scripts</a> or even Bash. It doesn't matter, so long as the computer is the one actually doing the changes. The scripted-ness is checked in and you treat it like application code. Including deployment.</div>
<div>
<br /></div>
<div>
<span style="color: #666666;"><b>In short, you treat deployments of <i>your</i> changes exactly as you treat deployments of the applications under your management. Which makes sense because an application is more than just code - it's also the infrastructure.</b></span></div>
<div>
<br /></div>
<div>
<span style="color: #6aa84f; font-family: Georgia, Times New Roman, serif;">"That's great in an ideal world, Rob, but nothing I have is scripted. It's all checklists. Now what?"</span></div>
<div>
<br /></div>
<div>
Checklists are scripts that run against a human virtual machine. If you look at the checklist, you should be able to replace many of the steps with "execute this script". Maybe even condense 2-3 steps into one script. Over time (and this is definitely a journey, not a destination), each checklist will condense into "invoke these N scripts". Which, itself, is just one script.</div>
<div>
<br /></div>
<div>
<span style="color: #6aa84f; font-family: Georgia, Times New Roman, serif;">"That great, but I don't even have checklists. My people just know what to do."</span></div>
<div>
<br /></div>
<div>
If <i>they</i> "just know what to do," then <i>you</i> do not. Do not know what they do, do not know they have done it, do not have control. You're responsible that they do it. And, most importantly, you're responsible that new people can learn it. If it isn't written down, how are new people learning their job?</div>
Rob Kinyonhttp://www.blogger.com/profile/07966252008096453177noreply@blogger.com0tag:blogger.com,1999:blog-2458293813533890602.post-75099516898340252812015-09-08T09:45:00.002-04:002015-09-08T09:45:23.438-04:00Devops - Where do I start?Last post, I laid out a <a href="http://streamlined-book.blogspot.com/2015/09/questions-to-ask-operations.html" target="_blank">series of questions</a> every operations team should be able to answer. Everyone may agree that this list of operational capabilities is good, but getting from here to there is far more complicated. What should we do first?<br />
<br />
The absolute first thing every operations team must do is get everything into source control. If it's not in source control, you cannot audit it, review it, or manage it. If a change happens, you don't know how, when, what, or why and there's no chain of custody showing what the approvals were. In short, you do not control it. Not controlling the stuff that makes your stuff isn't sane devops.<br />
<br />
This implies there needs to be solid source control. First choice is what to use. Always use a distributed version control system (aka, DVCS) - <a href="https://git-scm.com/" target="_blank">Git</a> or <a href="https://mercurial.selenic.com/" target="_blank">Mercurial</a> if at all possible. Using a DVCS has two massive advantages over a centralized version control system, like Subversion, Perforce, or CVS (no links to bad choices!). First, DVCS's are far better tools for managing development changesets - lots of discussion about that around the web. The better reason, for operations, is that every clone can be used as the new master if everything else goes pear-shaped. Remember - the first question asks if you are capable of recovering everything else if you have your production backups <i>and the latest checkout of source control</i>. You cannot recover source control itself with the latest checkout of Subversion or CVS. You can do so with Git or Mercurial.<br />
<br />
If at all possible, use a service. (This, btw, is going to be a theme I'll expand on more later.) <a href="https://github.com/" target="_blank">GitHub</a>, <a href="https://aws.amazon.com/codecommit/" target="_blank">AWS's CodeCommit</a>, <a href="https://bitbucket.org/" target="_blank">Atlassian's BitBucket</a>, or <a href="https://cloud.google.com/tools/cloud-repositories/docs/" target="_blank">Google's Cloud Source Repositories</a> are all excellent choices, along with many others. They all provide private repositories and are extremely scalable and secure. For nearly every organization, these services are capable enough. If you're already using Google's AppEngine or the AWS suite of services, the choice is pretty simple. If you're using the <a href="https://confluence.atlassian.com/cloud/" target="_blank">hosted Atlassian suite</a>, BitBucket again seems to be an easy choice. Github is an excellent choice in most other scenarios. Sometimes, for corporate reasons, you have to host internally. In that case, you should strongly consider using <a href="https://about.gitlab.com/" target="_blank">GitLab</a> or <a href="https://www.atlassian.com/software/stash" target="_blank">Atlassian Stash</a>. In all cases, you should be using the same tools as your developers.<br />
<br />
Once you picked a tool and a method for hosting, the next step is to get everything into it. Literally and truly everything. If it's a script, check it in. If it's a configuration file, check it in. If it's a Chef recipe, Puppet manifest, Salt pillar, or any other file for a similar tool - yup, check it in. All the secrets (GPG-encrypted first, of course). All the scripts. All the configuration. <u style="font-style: italic;">Everything</u>.<br />
<br />
If you don't have an automated way of building it, then write down how to build and check <i>that</i> in. Unless you have a really good reason not to, use a text-based markup language. It's important to use text because text is diff-able by Git/Mercurial. If it's not diff-able, then it becomes very difficult to see the differences when someone wants to change something. I prefer <a href="https://help.github.com/articles/github-flavored-markdown/">GitHub-flavored Markdown</a>, but there's <a href="http://redcloth.org/textile">plenty</a> of <a href="http://www.wikicreole.org/">other</a> <a href="http://docutils.sourceforge.net/rst.html">good</a> <a href="http://perldoc.perl.org/perlpod.html">choices</a>. Use the one that makes the most sense with the rest of your tooling landscape.<br />
<br />
<i><u>Note</u>:</i> I recommend both text (for diff-ability) and GPG-encryption (for secrets). GPG encryption is inherently not diff-able. For secrets, this is good. For instructions and scripts, you want diff-ability.<br />
<br />
What's the minimal list of servers/services/activities that you need to check into source control? You guessed it - everything.<br />
<ul>
<li>DNS / Network definitions</li>
<li>LDAP / IAM / User authentication lists</li>
<li>Mail servers (if you manage mail internally, otherwise treat it as an external service)</li>
<li>Monitoring <u>and</u> alerting definitions</li>
<ul>
<li>Especially if you use something like PagerDuty for alerting</li>
</ul>
<li><a href="https://www.gnupg.org/gph/en/manual.html#AEN111" target="_blank">GPG-encrypted</a> master passwords for external services</li>
<li><a href="https://www.gnupg.org/gph/en/manual.html#AEN111"><span id="goog_472368838"></span>GPG-encrypted</a> keys (and other authentication methods) for external services</li>
<li><a href="https://www.gnupg.org/gph/en/manual.html#AEN111">GPG-encrypted</a> SSL certificates</li>
<li>Server construction methods (for your application)</li>
<ul>
<li>Including where and how to get the base images</li>
</ul>
<li>Application deployment methods</li>
<li>Service construction methods (for your internally-hosted supporting services)</li>
<ul>
<li>For example, CI (Jenkins, Stash, etc)</li>
</ul>
<li>Any and all desktop support (including VPN clients, etc)</li>
<li>Anything else you are responsible for</li>
</ul>
<div>
(<u><i>Note</i></u>: While it doesn't explicitly say it in the documentation, but <a href="http://stackoverflow.com/a/597200/1732954">you can GPG-encrypt a file for multiple recipients</a> and <i>any</i> of them can decrypt it. This is a good thing. Note that you will need to re-key all the secrets whenever someone leaves, not just re-encrypt them. You needed to do this anyway. You only need to re-encrypt them when someone new comes in.)<br />
<br />
All of these things may not all live in the same repository. But, don't hesitate to put it into some repository just because you don't know the perfect place. You can always move it later. And, the movement itself will document the growing understanding you have of how to manage the infrastructure.<br />
<br />
If you don't know how to rebuild a server, take your best guess and stick that in the repository. As you learn more, you will update what's there. The repository logs will be a trail of exactly what you had to do in order to get all the information you currently have.<br />
<br />
The next post will talk about what to <i><u>do</u></i> with this repository once you've built it.</div>
Rob Kinyonhttp://www.blogger.com/profile/07966252008096453177noreply@blogger.com0tag:blogger.com,1999:blog-2458293813533890602.post-87975268829470703592015-09-03T12:49:00.002-04:002015-09-08T09:46:33.943-04:00Questions to ask Operations(. . . or, if you're the devops team, questions you should be asking yourself)<br />
<br />
Devops teams exist to make the answers to the following questions a resounding "Absolutely!". If there is any question in this list that you're not 110% confident in the "Absolutely!", then that's the next thing to work on. And, yes, this list is ordered from most important to least important.<br />
<ol>
<li>Can we <u><i>confidently</i></u><i> </i>rebuild our production environment from source control and backups of data?</li>
<ol>
<li>In under an hour?</li>
<li>Including all monitoring, alerting, and metrics gathering?</li>
</ol>
<li>Can we <u style="font-style: italic;">confidently</u> terminate one person's access?</li>
<ol>
<li>In under 10 minutes?</li>
<li>With one command?</li>
</ol>
<li>Can we <u style="font-style: italic;">confidently</u> create a instance of the application?</li>
<ol>
<li>That is a structural clone of production?</li>
<li>With reasonable <u>fake</u> data?</li>
<li>In one command?</li>
<li>On a laptop?</li>
</ol>
<li>Can we <u style="font-style: italic;">confidently</u> turn off any one server in production at any time?</li>
<ol>
<li>With zero impact or visibility to users?</li>
<li>Including your:</li>
<ol>
<li>database master?</li>
<li>session store?</li>
</ol>
</ol>
<li>Can we <u style="font-style: italic;">confidently</u> tell anyone to take 3 months leave to care for a sick family member?</li>
<ol>
<li>Without ever calling them once?</li>
</ol>
<li>Can we <u style="font-style: italic;">confidently</u> hire into any spot and have that person fully authenticated and authorized?</li>
<ol>
<li>With nothing missing?</li>
<li>In their first hour?</li>
<li>Before they even show up?</li>
</ol>
<li>Can we <u style="font-style: italic;">confidently</u> hire someone into IT and have them make a change to production?</li>
<ol>
<li>In their first week?</li>
<li>In their first day?</li>
</ol>
<li>Can we <u style="font-style: italic;">confidently</u> say that what is reviewed in QA is <b><u><i>EXACTLY</i></u></b> what can go to production?</li>
<li>Can we <u style="font-style: italic;">confidently</u> let anyone promote from one environment to the next?</li>
<ol>
<li>With a button?</li>
<li>Showing them exactly what will be promoted?</li>
<ol>
<li>As issue numbers linked from your issue tracker?</li>
</ol>
<li>With rollbacks?</li>
</ol>
<li>Do you have tests of your infrastructure?</li>
<ol>
<li>Including monitoring, alerting, and metrics gathering?</li>
<li>Including external interfaces?</li>
<li>Run as part of a CI service?</li>
<li>With automated coverage statistics?</li>
<ol>
<li>Over 90%?</li>
</ol>
</ol>
</ol>
<div>
Implicit in every question is the follow-up "How do you know?" If you ask yourself these questions and cannot point to where you did that yesterday (or the last time, in the case of authn/z changes), then you're treating your infrastructure as magic.<br />
<br />
<a href="http://streamlined-book.blogspot.com/2015/09/devops-where-do-i-start.html">Next post discusses where to start</a>.<br />
<ol></ol>
</div>
Rob Kinyonhttp://www.blogger.com/profile/07966252008096453177noreply@blogger.com0tag:blogger.com,1999:blog-2458293813533890602.post-69597552679015733352015-08-27T17:15:00.001-04:002015-08-27T17:15:29.337-04:00The Packager DSL - The second user story<ol>
<li><a href="http://streamlined-book.blogspot.com/2015/07/why-use-dsl.html" target="_blank">DSLs - An Overview</a></li>
<li><a href="http://streamlined-book.blogspot.com/2015/08/creating-packager-dsl-initial-steps.html" target="_blank">The Packager DSL - The first user story</a></li>
<li><b>The Packager DSL - The second user story</b></li>
</ol>
<div>
Our main user is an odd person. So far, all we've made is a DSL that can create empty packages with a name and a version. No files, no dependencies, no before/after scripts - nothing. But, instead of asking for anything <i>useful</i>, the first response we get is "Huh - I didn't think 'abcd' was a legal version number." Some people just think in weird ways.</div>
<div>
<br /></div>
<div>
Our second user story:</div>
<blockquote class="tr_bq">
Throw an error whenever the version isn't an acceptable version string by whatever criteria are normally employed.</blockquote>
<div>
Before we dig into the user story itself, let's talk about why this isn't a "bug" or a "defect" or any of the other terms normally bandied about whenever deployed software doesn't meet user expectations. Every time the user asks us to change something, it doesn't matter whether we call it a "bug", "defect", "enhancement", or any other word. It's still a <i>change to the system as deployed</i>. Underneath all the fancy words, we need to treat every single change with the same processes. Bugfixes don't get a special pass to production. Hotfixes don't get a special pass to production. Everything is "just a change", nothing less and nothing more.</div>
<div>
<br /></div>
<div>
In addition, the "defect" wasn't in our implementation. It was in the first user story if it was anywhere. The first user story didn't provide any restrictions on the version string, so we didn't place any upon it. And that was correct - you should never do more than the user story requires. If you think that a user story is incomplete in its description, you should go back to the user and negotiate the story. Otherwise, the user doesn't know what they're going to receive. Even worse, you might add something to the story that the user <i>does not want</i>.</div>
<div>
<br /></div>
<div>
Really, this shouldn't be considered a defect in specification, either. That concept assumes an all-knowing specifier that is able to lay out fully-formed and fully-correct specifications that never need updating. Which is ridiculous on its face. No-one can possibly be that person and <a href="https://en.wikipedia.org/wiki/Waterfall_model" target="_blank">no-one should ever be forced to try</a>. This much tighter feedback loop between specification to production to next specification is one of the key concepts behind Agile development. (The <a href="https://en.wikipedia.org/wiki/DevOps#History_of_the_term_.22DevOps.22" target="_blank">original name for devops</a> was <a href="http://theagileadmin.com/what-is-devops/" target="_blank">agile systems administration</a> or agile operations.)</div>
<div>
<br /></div>
<div>
All of this makes sense. When you first conceive of a thing, you have a vague idea of how it should work. So, you make the <a href="https://en.wikipedia.org/wiki/You_aren%27t_gonna_need_it" target="_blank">smallest possible thing that could work</a> and use it. While you start with a few ideas of where to go next, the moment you start using it, you realize other things that you never thought of. All of them (older ideas and newer realizations) become user stories and we (the developers and the users together) agree on what each story means, then prioritizes them in collaboration. Maybe the user wants X, but the developers point out Y would be really quick to do, so everyone agrees to do Y first to get it out of the way. It's an ongoing conversation, not a series of dictates.</div>
<div>
<br /></div>
<div>
All of which leads us back to our user story about bad version strings. The first question I have is "what makes a good or bad version string?" The answer is "whatever criteria are normally employed". That means it's up to us to come up with a first pass. And, given that we're building this in a Ruby world, the easiest thing would be to see what Ruby does.</div>
<div>
<br /></div>
<div>
Ruby's interface with version strings would be in its packages - gems. Since everything in Ruby is an object, then we would look at <a href="http://ruby-doc.org/stdlib-2.0.0/libdoc/rubygems/rdoc/Gem.html" target="_blank">Gem</a> and its version-handling class <a href="http://ruby-doc.org/stdlib-2.0.0/libdoc/rubygems/rdoc/Gem/Version.html" target="_blank">Gem::Version</a>. Reading through that documentation, it looks like the Ruby community has given a lot of thought to the issue. More thought than I would have realized was necessary, but it's good stuff. More importantly, if we use Gem::Version to do our version string validation, then we have a ready-documented worldview of how we expect version strings to be used.</div>
<div>
<br /></div>
<div>
Granted, we will have to conform to whatever the package formats our users will want to generate require. FreeBSD may require something different from RedHat and maybe neither is exactly what Gem::Version allows. At that point, we'll have failing tests we can write from user stories saying things like "For Debian, allow <i>this</i> and disallow <i>that</i>." For now, let's start with throwing an error on values like "bad" (and "good"). Anything more than that will be another user story.</div>
<div>
<br /></div>
<div>
As always, the first thing is to write a test. Because we can easily imagine needing to add more tests for this as we get more user stories, let's make a new file at <span style="background-color: #f3f3f3; font-family: Courier New, Courier, monospace; font-size: x-small;">spec/dsl/version_spec.rb</span>. That way, we have a place to add more variations.</div>
<div>
<br /></div>
<pre>describe Packager::DSL do
context "has version strings that" do
it "rejects just letters" do
expect {
Packager::DSL.execute_dsl {
package {
name 'foo'
version 'abcd'
type 'whatever'
}
}
}.to raise("'abcd' is not a legal version string")
end
end
end
</pre>
<div>
<br /></div>
<div>
Once we have our failing test, let's think about how to fix this. We have three possible places to put this validation. We may even choose to put pieces of it in multiple places.</div>
<div>
<ol>
<li>The first is one we've already done - adding a validation to the <span style="background-color: #f3f3f3; font-family: Courier New, Courier, monospace; font-size: x-small;">:package</span> entrypoint. That solution is good for doing validations that require knowing everything about the package, such as the version string and the type together.</li>
<li>The second is to add a Packager::Validator class, similar to the DSL and Executor classes we already have. This is most useful for doing validation of the entire DSL file, for example if multiple packages need to be coordinated.</li>
<li>The third is to create a new type, similar to the String coercion type we're currently using for the version.</li>
</ol>
</div>
<div>
Because it's the simplest and is sufficient for the user story, let's go with option #3. I'm pretty sure that, over time, we'll need to exercise option #1 as well, and possibly even #2. But, YAGNI. If we have to change it, we will. That's the beauty of well-built software - it's cheap and easy to change as needed.<br />
<br />
<pre>class Packager::DSL < DSL::Maker
add_type(VersionString = {}) do |attr_name, *args|
unless args.empty?
begin
___set(attr_name, Gem::Version.new(args[0]).to_s)
rescue ArgumentError
raise "'#{args[0]}' is not a legal version string"
end
end
___get(attr_name)
end
add_entrypoint(:package, {
...,
:version => VersionString,
...,
}) ...
end
</pre>
<br />
Note how we use Gem::Version to do the validation, but we don't save it as a Gem::Version object. We could keep the value as such, but there's no real reason (yet) to do so. <span style="background-color: #f3f3f3; font-family: Courier New, Courier, monospace; font-size: x-small;">___get()</span> and <span style="background-color: #f3f3f3; font-family: Courier New, Courier, monospace; font-size: x-small;">___set()</span> (with three underscores each) are provided by DSL::Maker to do the sets and gets. The attr_name is provided for us. So, if we wanted, we could reuse this type for many different attributes.<br />
<br />
We could add more tests, documenting exactly what we've done. But, I'm happy with this. Ship it!<br />
<br />
There's something else we haven't discussed about this option in particular. Type validations happen immediately when the item is set. Since values can be reused within a definition, they are safely reusable. For example (once we have defined how to include files in our packages), we could have something like:<br />
<br />
<pre>package {
name 'some-product'
version '1.2.44'
files {
source "/downloads/vendor/#{name}/#{version}/*"
dest "/place/for/files"
}
}
</pre>
<br />
If we defer validation until the end, we aren't sure we've caught all the places an invalid value may have been used in another value. This is why we attempt a "defense in depth", a concept normally seen in <a href="https://en.wikipedia.org/wiki/Information_security" target="_blank">infosec</a> circles. We want to make sure the version value is as correct as we can make it with just the version string. Then, later, we want to make sure it's even more correct once we can correlate it with the package format (assuming we ever get a user story asking us to do so).</div>
Rob Kinyonhttp://www.blogger.com/profile/07966252008096453177noreply@blogger.com0tag:blogger.com,1999:blog-2458293813533890602.post-62315449764649507202015-08-26T15:54:00.001-04:002015-08-26T15:54:06.478-04:00Creating the Packager DSL - Retrospective<ol>
<li><a href="http://streamlined-book.blogspot.com/2015/07/why-use-dsl.html" target="_blank">Why use a DSL?</a></li>
<li><a href="http://streamlined-book.blogspot.com/2015/08/why-create-your-own-dsl.html" target="_blank">Why create your own DSL?</a></li>
<li><a href="http://streamlined-book.blogspot.com/2015/08/what-makes-good-dsl.html" target="_blank">What makes a good DSL?</a></li>
<li><a href="http://streamlined-book.blogspot.com/2015/08/creating-your-own-dsl-parsing.html" target="_blank">Creating your own DSL - Parsing</a></li>
<li><a href="http://streamlined-book.blogspot.com/2015/08/creating-your-own-dsl-parsing-with-ruby.html" target="_blank">Creating your own DSL - Parsing (with Ruby)</a></li>
<li><a href="http://streamlined-book.blogspot.com/2015/08/creating-packager-dsl-initial-steps.html" target="_blank">Creating the Packager DSL - Initial steps</a></li>
<li><a href="http://streamlined-book.blogspot.com/2015/08/creating-packager-dsl-first-feature.html" target="_blank">Creating the Packager DSL - First feature</a></li>
<li><a href="http://streamlined-book.blogspot.com/2015/08/creating-packager-dsl-executor.html" target="_blank">Creating the Packager DSL - The executor</a></li>
<li><a href="http://streamlined-book.blogspot.com/2015/08/creating-packager-dsl-cli.html" target="_blank">Creating the Packager DSL - The CLI</a></li>
<li><a href="http://streamlined-book.blogspot.com/2015/08/creating-packager-dsl-integration.html" target="_blank">Creating the Packager DSL - Integration</a></li>
<li><b>Creating the Packager DSL - Retrospective</b></li>
</ol>
<div>
We've finished the first user-story. To review:</div>
<div>
<blockquote class="tr_bq">
<span style="background-color: white; line-height: 18.2000007629395px;"><span style="font-family: inherit;">I want to run a script, passing in the name of my DSL file. This should create an empty package by specifying the name, version, and package format. If any of them are missing, print an error message and stop. Otherwise, an empty package of the requested format should be created in the directory I am in.</span></span></blockquote>
Our user has a script to run and a DSL format to use. While this is definitely not the end of the project by any means, we can look back on what we've done so far and learn a few things. We're also likely to find a list of things we need to work on and refactor as we move along.<br />
<br />
It helps to come up with a list of what we've accomplished in our user story. As we get further along in the project, this list will be much smaller. But, the first story is establishing the <a href="http://alistair.cockburn.us/Walking+skeleton" target="_blank">walking skeleton</a>. So far, we have:<br />
<br />
<ol>
<li>The basics of a Ruby project (gemspec, Rakefile, Bundler, and project layout)</li>
<li>A DSL parser (using <a href="https://rubygems.org/gems/dsl_maker" target="_blank">DSL::Maker</a> as the basis)</li>
<ol>
<li>Including verification of what is received</li>
</ol>
<li>An intermediate data structure representing the packaging request (using Ruby Structs)</li>
<li>An Executor (that calls out to <a href="https://rubygems.org/gems/fpm" target="_blank">FPM</a> to create the package)</li>
<li>A CLI handler (using <a href="https://rubygems.org/gems/thor" target="_blank">Thor</a> as the basis)</li>
<ol>
<li>Including verification of what is received</li>
</ol>
<li>Unit-test specifications for each part (DSL, Executor, and CLI)</li>
<ol>
<li>Unit-tests by themselves provide 100% code coverage</li>
</ol>
<li>Integration-test specifications to make sure the whole functions properly</li>
<li>A development process with user stories, TDD, and continuous integration</li>
<li>A release process with Rubygems</li>
</ol>
<div>
That's quite a lot. We should be proud of ourselves. But, there's always improvements to be made.</div>
<div>
<br /></div>
<div>
The following improvements aren't new features. Yes, an empty package without dependencies is almost completely worthless. Those improvements will come through user stories. These improvements are ones we've seen in how we've built the guts of the project. Problems we've noticed along the way that, left unresolved, will become <a href="https://en.wikipedia.org/wiki/Technical_debt" target="_blank">technical debt</a>. We need to list them out now so that, as we work on user stories, we can do our work with an eye to minimizing these issues. In no particular order:</div>
<div>
<ul>
<li>No error handling in the call to FPM</li>
<ul>
<li>Calling an external command can be fraught with all sorts of problems.</li>
</ul>
<li>Version strings aren't validated</li>
<li>There's no whole-package validator between the DSL and the Executor</li>
<li>We probably need a Command structure to make the Executor easier to work with</li>
<li>We probably want to create some shared contexts in our specs to reduce boilerplate</li>
<ul>
<li>This should be done as we add more specs</li>
</ul>
</ul>
<div>
It's important to be continually improving the codebase as you complete each new user story. Development is the act of changing something. Good developers make sure that the environment they work in is as nimble as possible. Development becomes hard when all the pieces aren't working together.</div>
</div>
<div>
<br /></div>
<div>
Over the next few iterations, we'll see how this improved codebase works in our favor when we add the next user stories (validating the version string, dependencies, and files).</div>
</div>
Rob Kinyonhttp://www.blogger.com/profile/07966252008096453177noreply@blogger.com0tag:blogger.com,1999:blog-2458293813533890602.post-67145145402136607312015-08-25T20:23:00.000-04:002015-08-25T20:23:01.329-04:00Packager DSL 0.0.1 ReleasedThe <a href="https://rubygems.org/gems/packager-dsl" target="_blank">Packager DSL</a> has been released to Rubygems. (I was going to name it "packager", but that was already taken.) Please download it and take it for a spin. It'll be under pretty active development, so any bugs you find or missing features you need should be addressed rapidly.Rob Kinyonhttp://www.blogger.com/profile/07966252008096453177noreply@blogger.com0tag:blogger.com,1999:blog-2458293813533890602.post-18825238978389247342015-08-24T15:23:00.000-04:002015-08-24T15:23:30.968-04:00Creating the Packager DSL - Integration<ol>
<li><a href="http://streamlined-book.blogspot.com/2015/07/why-use-dsl.html" target="_blank">Why use a DSL?</a></li>
<li><a href="http://streamlined-book.blogspot.com/2015/08/why-create-your-own-dsl.html" target="_blank">Why create your own DSL?</a></li>
<li><a href="http://streamlined-book.blogspot.com/2015/08/what-makes-good-dsl.html" target="_blank">What makes a good DSL?</a></li>
<li><a href="http://streamlined-book.blogspot.com/2015/08/creating-your-own-dsl-parsing.html" target="_blank">Creating your own DSL - Parsing</a></li>
<li><a href="http://streamlined-book.blogspot.com/2015/08/creating-your-own-dsl-parsing-with-ruby.html" target="_blank">Creating your own DSL - Parsing (with Ruby)</a></li>
<li><a href="http://streamlined-book.blogspot.com/2015/08/creating-packager-dsl-initial-steps.html" target="_blank">Creating the Packager DSL - Initial steps</a></li>
<li><a href="http://streamlined-book.blogspot.com/2015/08/creating-packager-dsl-first-feature.html" target="_blank">Creating the Packager DSL - First feature</a></li>
<li><a href="http://streamlined-book.blogspot.com/2015/08/creating-packager-dsl-executor.html" target="_blank">Creating the Packager DSL - The executor</a></li>
<li><a href="http://streamlined-book.blogspot.com/2015/08/creating-packager-dsl-cli.html" target="_blank">Creating the Packager DSL - The CLI</a></li>
<li><b>Creating the Packager DSL - Integration</b></li>
</ol>
<div>
Our user story:<br />
<blockquote class="tr_bq">
<span style="background-color: white; line-height: 18.2000007629395px;"><span style="font-family: inherit;">I want to run a script, passing in the name of my DSL file. This should create an empty package by specifying the name, version, and package format. If any of them are missing, print an error message and stop. Otherwise, an empty package of the requested format should be created in the directory I am in.</span></span></blockquote>
Our progress:<br />
<br />
<ul>
<li>We can parse the DSL into a Struct. We can handle name, version, and package format. If any of them are missing, we raise an appropriate error message.</li>
<li>We can create a package from the parsed DSL</li>
<li>We have a script the executes everything</li>
</ul>
<div>
So, we're done, right? Not quite. We have no idea if it actually works. Sure, you can run it manually and see the script works. But, that's useful only when someone remembers to do it and remembers how to interpret it. Much better is a test that runs every single time in our CI server (so no-one has to remember to do it) and knows how to interpret itself. In other words, a spec.<br />
<br />
We have tests for each of the pieces <i>in isolation</i>. That was deliberate - we want to make sure that each piece works without involving more complexity than is necessary. But, our user story doesn't care about that. It says the user wants to execute a script and get their package. The user doesn't care about the Parser vs. the Executor (or anything else we've written). Those distinctions are for <i>us</i>, the developers, to accommodate the inevitable changes that will happen. A developer's job (and <i>raison d'etre</i>, frankly) is to create change. Without change, a developer has nothing to do. So, we organize our project to make it as easy as possible to make change.<br />
<br />
But, at the end of the day, it's the <i>integration</i> of these pieces that matters. So, we need to create end-to-end integration tests that show how all the pieces will work together. Where the unit tests we've written so far test the inner workings of each unit, the integration tests will test the coordination of all the units together. We are interested in checking that the output of one piece fits the expected input of the next piece.<br />
<br />
Said another way, our unit tests should provide 100% code coverage (which you can see with <span style="background-color: #f3f3f3;"><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">rspec spec/{cli,dsl,executor}</span></span>). The integration tests will provide 100% user-expectation coverage.<br />
<br />
As always, first thing we do is write a test. We have a subdirectory in spec/ for the unit tests for each component. Let's add another one called spec/integration with a file called empty_spec.rb and contents of<br />
<br />
<pre>require 'tempfile'
describe "Packager integration" do
let(:dsl_file) { Tempfile.new('foo').path }
it "creates an empty package" do
append_to_file(dsl_file, "
package {
name 'foo'
version '0.0.1'
package_format 'dir'
}
")
Packager::CLI.start('execute', dsl_file)
expect(File).to exist('foo.dir')
expect(Dir['foo.dir/*'].empty?).to be(true)
end
end
</pre>
<br />
Take a file with an empty package definition, create a package in the directory we're in, then verify. Seems pretty simple. We immediately run into a problem - no package is created. If you remember back when <a href="http://streamlined-book.blogspot.com/2015/08/creating-packager-dsl-executor.html" target="_blank">when we were creating the executor</a>, we never actually call out to FPM. It's relatively simple to add in an #execute method to the Executor which does a system() call. That should make this test pass.<br />
<br />
But, that's not enough. After you run it, do a <span style="background-color: #f3f3f3;"><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">git status</span></span>. You'll immediately see another problem - the package was created in the directory you ran rspec in. Which sucks. But, it's fixable.<br />
<br />
In the same way we have tempfiles, we have temp directories. Sysadmins used to bash are familiar with <span style="font-family: Courier New, Courier, monospace; font-size: x-small;">mktemp</span> and <span style="font-family: Courier New, Courier, monospace; font-size: x-small;">mktemp -d</span>. Ruby has Tempfile and Dir.mktmpdir, respectively. So, let's run the test within a temporary directory - that should solve the problem.<br />
<br />
<pre>require 'tempfile'
require 'tmpdir'
describe "Packager integration" do
let(:dsl_file) { Tempfile.new('foo').path }
it "creates an empty package" do
Dir.mktmpdir do |tempdir|
Dir.chdir(tempdir) do
# Rest of test
end
end
end
end
</pre>
<br />
That keeps the mess out of the main directory. Commit and push.<br />
<br />
Though, when I look at what's been written, the tempdir handling is both manually-done (the author has to remember to do it every time) and creates two more levels of indentation. The manual part means it's possible for someone to screw up. The indentations part means that it's harder to read what's happening - there's boilerplate in every test. Which is somewhat ironic given that the whole point of this process is to create a DSL - removing the boilerplate. We can do better. Red-Green-Refactor doesn't just apply to the code. (Or, put another way, tests are also code.)<br />
<br />
RSpec allows us to do things before-or-after all-or-each of the specs in a context. Let's take advantage of this to ensure that every integration test will always happen within a tempdir.<br />
<br />
<pre>require 'fileutils'
require 'tempfile'
require 'tmpdir'
describe "Packager integration" do
let(:dsl_file) { Tempfile.new('foo').path }
let(:workdir) { Dir.mktmpdir }
before(:all) { @orig_dir = Dir.pwd }
before(:each) { Dir.chdir workdir }
after(:each) {
Dir.chdir @orig_dir
FileUtils.remove_entry_secure workdir
}
it "creates an empty package" do
# Rest of test
end
end
</pre>
<br />
A few notes here.<br />
<br />
<ol>
<li>When we used the block versions of Dir.mktmpdir and Dir.chdir, the block cleaned up whatever we did (e.g., changed back to the original directory). When we use the direct invocation, we have to do our own cleanup.</li>
<li>before(:all) will always run before before(:each) (guaranteed by RSpec).</li>
<li>We don't want to use let() for the original directory. let() is lazy, meaning it only gets set the first time it's invoked. Instead, we set an attribute of the test (as provided helpfully to us by RSpec).</li>
<ol>
<li>We could have used let!() instead (which is eager), but it's too easy to overlook the !, so I don't like to use it. Sometimes, subtlety is overly so.</li>
</ol>
<li>Tests should be runnable in any order. And this includes all the other tests in all the other spec files. You should never assume that any two tests will ever run in a specific order or even that any test will run in a specific test run. So, we always make sure to change directory back to the original directory (whatever that was). There's nothing here that assumes anything about the setup.</li>
<li>FileUtils has many ways to remove something. #remove_entry_secure is the most conservative, so the best for something that needs to be accurate more than it needs to be fast.</li>
<li>We need to leave the tempdir that we're in before trying to remove it. On some OSes, the OS will refuse to remove a directory if a process has it as its working directory.</li>
</ol>
<div>
</div>
<br />
<a href="http://streamlined-book.blogspot.com/2015/08/creating-packager-dsl-cli.html" target="_blank">prev</a></div>
</div>
Rob Kinyonhttp://www.blogger.com/profile/07966252008096453177noreply@blogger.com0tag:blogger.com,1999:blog-2458293813533890602.post-23041191934350254902015-08-21T17:07:00.001-04:002015-08-21T17:08:20.439-04:00Creating the Packager DSL - The CLI<ol>
<li><a href="http://streamlined-book.blogspot.com/2015/07/why-use-dsl.html" target="_blank">Why use a DSL?</a></li>
<li><a href="http://streamlined-book.blogspot.com/2015/08/why-create-your-own-dsl.html" target="_blank">Why create your own DSL?</a></li>
<li><a href="http://streamlined-book.blogspot.com/2015/08/what-makes-good-dsl.html" target="_blank">What makes a good DSL?</a></li>
<li><a href="http://streamlined-book.blogspot.com/2015/08/creating-your-own-dsl-parsing.html" target="_blank">Creating your own DSL - Parsing</a></li>
<li><a href="http://streamlined-book.blogspot.com/2015/08/creating-your-own-dsl-parsing-with-ruby.html" target="_blank">Creating your own DSL - Parsing (with Ruby)</a></li>
<li><a href="http://streamlined-book.blogspot.com/2015/08/creating-packager-dsl-initial-steps.html" target="_blank">Creating the Packager DSL - Initial steps</a></li>
<li><a href="http://streamlined-book.blogspot.com/2015/08/creating-packager-dsl-first-feature.html" target="_blank">Creating the Packager DSL - First feature</a></li>
<li><a href="http://streamlined-book.blogspot.com/2015/08/creating-packager-dsl-executor.html" target="_blank">Creating the Packager DSL - The executor</a></li>
<li><b>Creating the Packager DSL - The CLI</b></li>
</ol>
<div>
Our user story:<br />
<blockquote class="tr_bq">
<span style="background-color: white; line-height: 18.2000007629395px;"><span style="font-family: inherit;">I want to run a script, passing in the name of my DSL file. This should create an empty package by specifying the name, version, and package format. If any of them are missing, print an error message and stop. Otherwise, an empty package of the requested format should be created in the directory I am in.</span></span></blockquote>
Our progress:<br />
<br />
<ul>
<li>We can parse the DSL into a Struct. We can handle name, version, and package format. If any of them are missing, we raise an appropriate error message.</li>
<li>We can create a package from the parsed DSL</li>
</ul>
<div>
We still need to:</div>
<div>
<ul>
<li>Provide a script that executes everything</li>
</ul>
<div>
Writing this script, on its face, looks pretty easy. We need to:</div>
<div>
<ol>
<li>Receive the filename from the commandline arguments</li>
<li>Pass the contents of that filename to Packager::DSL.parse_dsl()</li>
<li>Pass the contents of that to Packager::Executor</li>
</ol>
</div>
<div>
A rough version (that works) could look like:<br />
<br /></div>
<div>
<pre>#!/usr/bin/env ruby
require 'packager/dsl'
require 'packager/executor'
filename = ARGV[0]
items = Packager::DSL.parse_dsl(IO.read(filename))
Packager::Executor.new.create_package(items)
</pre>
</div>
<div>
<br />
You can create a file with a package declaration (such as the one in our spec for the DSL) and pass it to this script and you will have an empty package created. All done, right?<br />
<br />
Not quite.<br />
<br />
The first problem is testing executables is hard. Unlike classes and objects which live in the same process, the testing boundary of a script is a process boundary. Process boundaries are much harder to work with. Objects can be invoked and inspected over and over, in any order. Invocations of an executable are one-shot. Outside the results, there's nothing to inspect once you've invoked the script. If we could minimize the script and move most of the logic into some objects, that would make testing it so much easier. And, we can measure our code coverage of it.<br />
<br />
The second (and bigger) problem is writing good executables is hard. Very very hard. Good executables have options, error-handling, and all sorts of other best practices. It is nearly impossible to write a good executable that handles all the things, even if you're an expert.<br />
<br />
Again, the good Ruby opensource developers have provided a solution - <a href="http://whatisthor.com/" target="_blank">Thor</a>. With Thor, we can move all the logic into a Packager::CLI class and our executable in <span style="background-color: #f3f3f3;"><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">bin/packager</span></span> becomes<br />
<br />
<div>
<pre>#!/usr/bin/env ruby
$:.unshift File.expand_path('../../lib', __FILE__)
require 'rubygems' unless defined? Gem
require 'packager/cli'
Packager::CLI.start
</pre>
</div>
<br />
Almost all of that is cribbed from other Ruby projects, meaning we can directly benefit from their experience. The executable is now 8 lines (including whitespace). We can visibly inspect this and be extremely certain of its correctness. Which is good because we really don't want to have to test it. The actual CLI functionality moves into classes and objects, things we can easily test.<br />
<br />
First thing first - we need a test. Thor scripts tend to function very similarly to git, with invocations of the form "<script> <command> <flags> <parameters>". So, in our case, we're going to want "packager create <DSL filenames>". This translates into the #create method on the Packager::CLI class. The filenames will be passed in as the arguments to the #create method. We don't have any flags, so we'll skip that part (for now).<br />
<br />
A note on organization - we have tests for the DSL, the Executor, and now the CLI. We can see wanting to write many more tests for each of those categories as we add more features, so let's take the time right now to reorganize our <span style="background-color: #f3f3f3; font-family: Courier New, Courier, monospace; font-size: x-small;">spec/</span> directory. RSpec will recursively descend into subdirectories, so we can create <span style="background-color: #f3f3f3; font-family: Courier New, Courier, monospace; font-size: x-small;">spec/dsl</span>, <span style="background-color: #f3f3f3;"><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">spec/executor</span></span>, and <span style="background-color: #f3f3f3; font-family: Courier New, Courier, monospace; font-size: x-small;">spec/cli</span> directories. <span style="background-color: #f3f3f3;"><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">git mv</span></span> the existing DSL and Executor specs into the appropriate directories (renaming them to be more meaningful), run RSpec to make sure everything is still found, then commit the change. You can pass rspec the name of a file or a directory, if you want to run just a subset of the tests. So, if you're adding just a DSL piece, you can run those tests to make sure they pass without having to do the entire thing.<br />
<br />
Back to the new CLI test. The scaffolding for this looks like<br />
<br />
<pre>describe Packager::CLI do
subject(:cli) { Packager::CLI.new }
describe '#create' do
end
end
</pre>
<br />
The nested describe works exactly as you'd expect. (RSpec provides many ways to organize things, letting you choose which works best for the situation at hand.)<br />
<br />
The first test, as always, is the null test. What happens if we don't provide any filenames? Our script should probably print something and stop, ideally setting the exit code to something non-zero. In Thor, the way to do that is to <span style="background-color: #f3f3f3;"><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">raise Thor::Error, "Error string"</span></span>. (I wish they'd call that Thor::Hammer, but you can't have everything.) So, the first test should expect the error is raised.<br />
<br />
<pre> it "raises an error with no filenames" do
expect {
cli.create
}.to raise_error(Thor::Error, "No filenames provided for create")
end
</pre>
<br />
Run that, see it fail, then let's create <span style="background-color: #f3f3f3;"><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">packager/cli.rb</span></span> to look like<br />
<br />
<pre>class Packager
class CLI < Thor
def create()
raise Thor::Error, "No filenames passed for create"
end
end
end
</pre>
<br />
Again, we're writing just enough code to make the tests pass. Now, let's pass in a filename to #create. Except, where do we get the file?<br />
<br />
One possibility is to create a file with what we want, save it somewhere inside <span style="background-color: #f3f3f3;"><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">spec/</span></span>, add it to the project, then reference that filename as needed.<br />
<br />
There are a few subtle problems with that approach.<br />
<br />
<ol>
<li>The file contents are separated from the test(s) using them.</li>
<li>These files have to be packaged with the gem in order for client-side tests to work.</li>
<li>There could be permission issues with writing to files inside the installation directory.</li>
<li>Developers end up wanting to keep the number of these files small, so shoehorn as many possible cases within each file as possible.</li>
</ol>
<div>
Fortunately, there's a much better approach. Ruby, like most languages, has a library for creating and managing tempfiles. Ruby's is called <a href="http://ruby-doc.org/stdlib/libdoc/tempfile/rdoc/Tempfile.html" target="_blank">Tempfile</a>. Adding this to our test file results in</div>
<div>
<br /></div>
<pre>require 'tempfile'
describe Packager::CLI do
subject(:cli) { Packager::CLI.new }
let(:package_file) { Tempfile.new('foo').path }
def append_to_file(filename, contents)
File.open(filename, 'a+') do |f|
f.write(content)
f.flush
end
end
describe '#create' do
it "raises an error with no filenames" do
expect {
cli.create
}.to raise_error(Thor::Error, "No filenames provided for create")
end
it "creates a package with a filename" do
append_to_file(package_file, "
package {
name 'foo'
version '0.0.1'
type 'dir'
}
")
cli.create(package_file)
end
end
end
</pre>
<div>
<br /></div>
<div>
We create a tempfile and the filename stored in the <span style="background-color: #f3f3f3; font-family: Courier New, Courier, monospace; font-size: x-small;">package_file</span> 'let' variable. That's just an empty file, though. We then want to put some stuff in it, so we create the append_to_file helper method. (This highlights something important - we can add methods as needed to our tests.) Then, we use it to fill the file with stuff and pass the filename to Packager::CLI#create.<br />
<br />
Note: We have to flush to disk to ensure that when we read from the file, the contents are actually in the file instead of the output buffer.<br />
<br />
We have our input filename (and its contents) figured out. What should we expect to happen? We could look at whether a package was created in the directory we invoked the CLI. That <i>is</i> what our user story requires. And, we will want to have that sort of integration test, making sure everything actually functions the way a user will expect it to function. But, that's not <u>this</u> test. (Not to mention the Executor doesn't actually invoke FPM yet!)<br />
<br />
These tests are meant to exercise each class <i>in isolation</i> - these are <u>unit tests</u>. Unit tests exercise the insides of one <i>and only one</i> class. If we were to see if a package is created, we're actually testing three classes - the CLI as well as the DSL and Executor classes. That's too many moving parts to quickly figure out what's gone wrong when something fails. By having tests which <i>only</i> focus on the internals of the CLI, DSL, and Executor classes by themselves as well as the integration of all the parts, we can easily see which part of our system is in error when tests start to fail. Is it the integration and CLI tests? Is it just the integration tests? Is it just the DSL? All of these scenarios immediately point out the culprit (or culprits).<br />
<br />
Given that the CLI is going to invoke the DSL and Executor, we want to catch the invocation of the #parse_dsl and #create_package methods. We don't want to actually <i>do</i> what those methods do as part of this test. Frankly, the CLI object doesn't care what those methods do. It only cares that the methods function, whatever that means.<br />
<br />
RSpec has a concept called stubbing. (This is part of a larger concept in testing called "mocking". RSpec provides mocks, doubles, and stubs, as do many other libraries like <a href="https://rubygems.org/gems/mocha" target="_blank">mocha</a>.) For our purposes, what we can do is say "The next time method X is called on an instance of class Y, do <this> instead." Stubs (and mocks and doubles) will be uninstalled at the end of every spec, so there's no danger of it leaking or affecting anything else. With stubs, our happy-day test now looks like<br />
<br />
<pre> it "creates a package with a filename" do
contents = "
package {
name 'foo'
version '0.0.1'
type 'dir'
}
"
append_to_file(package_file, contents)
expect(Packager::DSL).to receive(:parse_dsl).with(contents).and_return(:stuff)
expect_any_instance_of(Packager::Executor).to receive(:create_package).with(:stuff).and_return(true)
cli.create(package_file)
end
</pre>
This looks like an awful mouthful. And, it may seem odd to create expectations before we call <span style="background-color: #f3f3f3;"><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">cli.create</span></span>. But, if you think about it for a second and read the two expectations out loud, it can make sense. All of our tests so far have been "We expect X <u>is</u> true." What we're now saying is "We expect X <u>will</u> be true." Which works out.<br />
<br />
As for the formatting, you can do the following:<br />
<br />
<pre> expect(Packager::DSL).to(
receive(:parse_dsl).
with(contents).
and_return(:stuff)
)
</pre>
<br />
Note the new parentheses for the .to() method and the periods at the end of one line (instead of the beginning of the next). These are required for how Ruby parses. You could also use backslashes, but I find those too ugly. This, to me, is a good compromise. Please feel free to experiment - the goal is to make it readable for you and your maintainers, not me or anyone else in the world.<br />
<br />
Our #create method now changes to<br />
<br />
<pre> def create(filename=nil)
raise Thor::Error, "No filenames passed for create" unless filename
items = Packager::DSL.parse_dsl(IO.read(filename))
Packager::Executor.new.create_package(items)
end
</pre>
<br />
and we're done. Remember to do a <span style="background-color: #f3f3f3;"><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">git status</span></span> to make sure you're adding all the new files we've been creating to source control.<br />
<br />
<a href="http://streamlined-book.blogspot.com/2015/08/creating-packager-dsl-executor.html" target="_blank">prev</a></div>
</div>
</div>
</div>
Rob Kinyonhttp://www.blogger.com/profile/07966252008096453177noreply@blogger.com0tag:blogger.com,1999:blog-2458293813533890602.post-3500285614226333502015-08-18T16:32:00.000-04:002015-08-18T16:32:05.314-04:00Creating the Packager DSL - The executor<ol>
<li><a href="http://streamlined-book.blogspot.com/2015/07/why-use-dsl.html" target="_blank">Why use a DSL?</a></li>
<li><a href="http://streamlined-book.blogspot.com/2015/08/why-create-your-own-dsl.html" target="_blank">Why create your own DSL?</a></li>
<li><a href="http://streamlined-book.blogspot.com/2015/08/what-makes-good-dsl.html" target="_blank">What makes a good DSL?</a></li>
<li><a href="http://streamlined-book.blogspot.com/2015/08/creating-your-own-dsl-parsing.html" target="_blank">Creating your own DSL - Parsing</a></li>
<li><a href="http://streamlined-book.blogspot.com/2015/08/creating-your-own-dsl-parsing-with-ruby.html" target="_blank">Creating your own DSL - Parsing (with Ruby)</a></li>
<li><a href="http://streamlined-book.blogspot.com/2015/08/creating-packager-dsl-initial-steps.html" target="_blank">Creating the Packager DSL - Initial steps</a></li>
<li><a href="http://streamlined-book.blogspot.com/2015/08/creating-packager-dsl-first-feature.html" target="_blank">Creating the Packager DSL - First feature</a></li>
<li><b>Creating the Packager DSL - The executor</b></li>
</ol>
<div>
Our user story:<br />
<blockquote class="tr_bq">
<span style="background-color: white; line-height: 18.2000007629395px;"><span style="font-family: inherit;">I want to run a script, passing in the name of my DSL file. This should create an empty package by specifying the name, version, and package format. If any of them are missing, print an error message and stop. Otherwise, an empty package of the requested format should be created in the directory I am in.</span></span></blockquote>
Our progress:<br />
<br />
<ul>
<li>We can parse the DSL into a Struct. We can handle name, version, and package format. If any of them are missing, we raise an appropriate error message.</li>
</ul>
<div>
We still need to:</div>
<div>
<ul>
<li>Create a package from the parsed DSL</li>
<li>Provide a script that executes everything</li>
</ul>
<div>
Since the script is the umbrella, creating the package is the next logical step. To create the package, we'll defer to <a href="https://github.com/jordansissel/fpm" target="_blank">FPM</a>. FPM doesn't have a Ruby API - it is designed to be used by sysadmins and requires you to build a directory of what you want and invoke a script.</div>
</div>
<div>
<br /></div>
<div>
The first seemingly-obvious approach is to directly embed the things to do directly in the parser where we are currently creating the Package struct. That way, we do things right away instead of building some intermediate thing we're just going to throw away. Sounds like a great idea. And, it would be horrible.</div>
<div>
<br /></div>
<div>
The best programs are written in reusable chunks that each <a href="https://en.wikipedia.org/wiki/Unix_philosophy#Do_One_Thing_and_Do_It_Well" target="_blank">do one and only one thing and do it well</a>. This is true for operating systems and especially true for programs. In software, we call it <a href="https://en.wikipedia.org/wiki/Coupling_(computer_programming)" target="_blank">coupling</a>, or the degree one unit is inextricably-linked to other units. And, we want our units to be coupled as little as possible.</div>
<div>
<br /></div>
<div>
Our one unit right now (the parser) handles understanding the DSL as a string. We have two other responsibilities - creating a package and handling the interaction with the command-line. Unless we have a good reason otherwise, let's treat each of those as a separate unit. (There are occasionally good reasons to couple things together, but it's best to <a href="http://www.goodreads.com/quotes/16419-know-the-rules-well-so-you-can-break-them-effectively" target="_blank">know why the rule's there before you go about breaking it</a>.)</div>
<div>
<br /></div>
<div>
Now, we have two different units that, when taken together in the right order, will work together to take a DSL string and create a package. They will need to communicate one to the next so that the package-creation unit creates the package described by the DSL-parsing unit. We could come up with some crazy communication scheme, but the parser already produces something (the Package struct). <a href="https://en.wikipedia.org/wiki/You_aren%27t_gonna_need_it" target="_blank">That should be sufficient for now</a>. When that changes, we can refactor with confidence because we will keep our 100% test coverage.<br />
<br />
Before anything else, we'll need to install FPM. So, add it to the gemspec and <span style="background-color: #f3f3f3;"><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">bundle install</span></span>.<br />
<br />
Next, we need to write a test (always write the test first). The first part of the test (the input) is pretty easy to see - we want to create a Packager::Struct::Package with a name, version, and format set. Figuring out what the output should be is . . . a little more complicated. We don't want to test how FPM works - we can assume (barring counterexample) that FPM works exactly as advertised. But, at some point, we need to make sure that <u>our usage</u> of FPM is what we want it to be. So, we will need to test that the output FPM creates from our setup is what we want.<br />
<br />
The problem here is FPM delegates the actual construction of the package to the OS tools. So, it uses RedHat's tools to build RPMs, Debian's tools to be DEBs, etc. More importantly, the tools to <i>parse</i> those package formats only exist on those systems. Luckily for this test, we can ignore this problem. The command for creating an empty package is extremely simple - you can test it easily yourself on the commandline. But, we need to keep it in the back of our mind for later - sooner rather than later.<br />
<br />
Since we're testing the executor (vs. the parser), we should put our test in a second spec file. The test would look something like:<br />
<br />
<pre>describe Packager::Executor do
it "creates an empty package"
executor = Packager::Executor.new(:dryrun => true)
input = Packager::DSL::Package.new('foo', '0.0.1', 'unknown')
executor.create_package(input)
expect(executor.command).to eq([
'fpm',
'--name', input.name,
'--version', input.version,
'-s', 'empty',
'-t', input.package_format,
])
end
end
</pre>
<br />
A few things here:<br />
<ol>
<li>Unlike Packager::DSL where we run with class methods (because of how DSL::Maker works), we're creating an <i>instance</i> of the Packager::Executor class to work with. This allows us to set some configuration to control how the executor will function without affecting global state.</li>
<li>FPM does not support the "unknown" package format. We're testing that we get out what we put in.</li>
<li>The FPM command already looks hard to work with. Arrays are positional, but the options to FPM don't have to be. We will want to change that to be more testable.</li>
<li>Creating that Packager::DSL::Package object is going to become very confusing very quickly for the same reasons as the FPM command - it's an Array. Positional arguments become hard to work with over time.</li>
</ol>
<div>
You should run the spec to make sure it fails. The Packager::Executor code in lib/packager/executor.rb would look like:
<br />
<br />
<pre>class Packager::DSL
attr_accessor :dry_run, :command
def initialize(opts)
@dry_run = opts[:dry_run] ? true : false
@command = [] # Always initialize your attributes
end
def create_package(item)
command = [
'fpm',
'--name', item.name,
'--version', item.version,
'-s', 'empty',
'-t', item.package_format,
]</pre>
<pre>
</pre>
<pre> return true
end
end</pre>
</div>
</div>
</div>
<div>
<br />
Make sure to add the appropriate <span style="background-color: #f3f3f3; font-family: Courier New, Courier, monospace; font-size: x-small;">require</span> statement in either lib/packager.rb or spec/spec_helper.rb and <span style="background-color: #f3f3f3; font-family: Courier New, Courier, monospace; font-size: x-small;">rake spec</span> should pass. Add and commit everything, then push. We're not done, but we're one big step closer.<br />
<br />
<a href="http://streamlined-book.blogspot.com/2015/08/creating-packager-dsl-first-feature.html" target="_blank">prev</a></div>
Rob Kinyonhttp://www.blogger.com/profile/07966252008096453177noreply@blogger.com0tag:blogger.com,1999:blog-2458293813533890602.post-65092956402628390962015-08-17T12:04:00.000-04:002015-08-17T12:04:16.210-04:00Creating the Packager DSL - Initial steps<ol>
<li><a href="http://streamlined-book.blogspot.com/2015/07/why-use-dsl.html" target="_blank">Why use a DSL?</a></li>
<li><a href="http://streamlined-book.blogspot.com/2015/08/why-create-your-own-dsl.html" target="_blank">Why create your own DSL?</a></li>
<li><a href="http://streamlined-book.blogspot.com/2015/08/what-makes-good-dsl.html" target="_blank">What makes a good DSL?</a></li>
<li><a href="http://streamlined-book.blogspot.com/2015/08/creating-your-own-dsl-parsing.html" target="_blank">Creating your own DSL - Parsing</a></li>
<li><a href="http://streamlined-book.blogspot.com/2015/08/creating-your-own-dsl-parsing-with-ruby.html" target="_blank">Creating your own DSL - Parsing (with Ruby)</a></li>
<li><b>Creating the Packager DSL - Initial steps</b></li>
<li style="margin: 0px 0px 0.25em; padding: 0px;"><a href="http://streamlined-book.blogspot.com/2015/08/creating-packager-dsl-first-feature.html" target="_blank">Creating the Packager DSL - First feature</a></li>
</ol>
<div>
The Packager DSL is, first and foremost, a software development project. In order for a software project to function properly, there's a lot of scaffolding that needs to be set up. This post is about that. (Please feel free to skip it if you are comfortable setting up a <a href="https://www.ruby-lang.org/" target="_blank">Ruby</a> project.) When this post is completed, we will have a full <a href="http://blog.codeclimate.com/blog/2014/03/20/kickstart-your-next-project-with-a-walking-skeleton/" target="_blank">walking skeleton</a>. We can then add each new feature quickly. We'll stop when we're just about to add the first DSL-ish feature.<br />
<br />
If you're creating your own DSL with your own project name, just put it everywhere you see packager and Packager accordingly.<br />
<h4>
Repository choice</h4>
</div>
<div>
Distributed version control systems (aka, DVCS) provide significant capabilities in terms of managing developer contributions over previous centralized systems (such as Subversion and CVS). Namely, Git (and <a href="https://mercurial.selenic.com/" target="_blank">Mercurial</a> or <a href="http://bazaar.canonical.com/" target="_blank">Bazaar</a>) have extremely lightweight branches and very good merging strategies. There's a ton of good work on the net about choosing one of these systems and how to use it.</div>
<div>
<br /></div>
<div>
I use <a href="https://git-scm.com/" target="_blank">Git</a>. For my opensource work, I use <a href="https://github.com/" target="_blank">GitHub</a>. If I'm within an enterprise, <a href="https://about.gitlab.com/" target="_blank">Gitlab</a> or <a href="https://www.atlassian.com/software/stash" target="_blank">Stash</a> are excellent repository management solutions. The minimum scaffolding for any project in Git is:</div>
<div>
<ul>
<li>.gitignore file</li>
<ul>
<li>These files are never to be checked into the repository. Usually intermediate files for various developer purposes (coverage, build artifacts, etc). The contents of this file are usually language-specific.</li>
</ul>
<li>.gitattributes file</li>
<ul>
<li>Ensure line-endings are consistent between Windows, Linux, and OSX.</li>
</ul>
</ul>
</div>
<h4>
Repository scaffolding</h4>
<div>
The minimum scaffolding for a Ruby project is:</div>
<div>
<ul>
<li>lib/ directory</li>
<ul>
<li>Where our library code lives. Leave this empty (for now).</li>
</ul>
<li>spec/ directory</li>
<ul>
<li>Where our tests (or specifications) live. Leave this empty (for now).</li>
</ul>
<li>bin/ directory</li>
<ul>
<li>Where our executables live. Leave this empty (for now).</li>
</ul>
<li>.rspec file</li>
<ul>
<li>Default options for rspec (Ruby's specification runner).</li>
</ul>
<li>Rakefile file</li>
<ul>
<li>Ruby's version of Makefile</li>
</ul>
<li>packager.gemspec file</li>
<ul>
<li>How to package our project.</li>
</ul>
<li>Gemfile file</li>
<ul>
<li>Dependency management and repository location. I write mine so that it will delegate to the gemspec file. I hate repeating myself, especially when I don't have to.</li>
</ul>
</ul>
<div>
You are welcome to copy any or all of these files from the <a href="https://github.com/robkinyon/ruby-packager" target="_blank">ruby-packager</a> project and edit them accordingly or create them yourself. There is ridiculous amounts of documentation on this process and each of the tools (<a href="https://rubygems.org/gems/rake" target="_blank">Rake</a>, <a href="http://rspec.info/" target="_blank">Rspec</a>, <a href="https://rubygems.org/" target="_blank">Gem</a>, and <a href="http://bundler.io/" target="_blank">Bundler</a>) all over the web. <a href="https://www.ruby-lang.org/" target="_blank">Ruby</a>, moreso than most software communities, has made a virtue of clear, clean, and comprehensive documentation.</div>
</div>
<div>
<br /></div>
<div>
You also need to have installed <a href="http://bundler.io/" target="_blank">Bundler</a> using <span style="background-color: #f3f3f3; font-family: Courier New, Courier, monospace; font-size: x-small;">gem install bundler</span>. Bundler will then make sure your dependencies are always up-to-date with the simple <span style="background-color: #f3f3f3;"><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">bundle install</span></span> command. I tend to install <a href="http://rvm.io/" target="_blank">RVM</a> first, just to make sure I have can upgrade (or downgrade!) my version of Ruby as needed.</div>
<div>
<br /></div>
<div>
At this point, go ahead and add all of these files to your checkout and commit. I like using the message <span style="background-color: #f3f3f3; font-family: Courier New, Courier, monospace; font-size: x-small;">"Create initial commit"</span>. (If you use git, the empty lib/ and spec/ directories won't be added. Don't worry - they will be added before we're finished with Day 1.)</div>
<div>
<br /></div>
<div>
<b><u>Note:</u></b> If your DSL is for use within an enterprise, please make sure that you know where to download and install your gems from. Depending on the enterprise, you probably already have an internal gems repository and a standard process for requesting new gems and new versions. You can set the location of that internal gems repository within the Gemfile instead of <span style="background-color: #f3f3f3;"><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">https://rubygems.org</span></span>. If you have to set a proxy in order to reach rubygems.org, <i><b>please</b></i> talk your IT administrator first.<br />
<h4>
Documentation</h4>
</div>
<div>
It may seem odd to consider documentation before writing anything to document (and we will revisit this when we have an actual API to document), but there are two documentation files we should create right away:</div>
<div>
<ul>
<li>README.md</li>
<ul>
<li>This is the file GitHub (and most other repository systems) will display when someone goes to the repository. This should describe your project and give a basic overview of how to use it.</li>
</ul>
<li>Changes / Changelog</li>
<ul>
<li>Development is the act of changing software. Let's document what we've changed and when. I prefer the name Changes, but several other names will work.</li>
</ul>
</ul>
<div>
I prefer to use <a href="http://daringfireball.net/projects/markdown/" target="_blank">Markdown</a> (specifically, <a href="https://help.github.com/articles/github-flavored-markdown/" target="_blank">GitHub-flavored Markdown</a>) in my README files. Hence, the <span style="background-color: #f3f3f3; font-family: Courier New, Courier, monospace; font-size: x-small;">.md</span> suffix. Please use whatever markup language you prefer and which will work within your repository system.</div>
</div>
<div>
<br /></div>
<div>
Your changelog should be a reverse-sorted (newest-first) list of all the changes grouped by version. The goal is to provide a way for users to determine when something was changed and why. So, wherever possible, you should provide a link to the issue or task that describes what changed in detail. The best changelog is just a list of versions with issue descriptions linking to the issue.</div>
<div>
<br /></div>
<div>
Finally, <span style="background-color: #f3f3f3; font-family: 'Courier New', Courier, monospace; font-size: x-small;">git add . && git commit -am "Create documentation stubs"</span></div>
<h4>
SDLC</h4>
<div>
Before we write any software, let's talk quickly about how we're going to get the changes into the software and out the door. While the first set of changes are going to be driven by what we want, we definitely want a place for our users to open requests. GitHub provides a very simple <a href="https://github.com/robkinyon/ruby-packager/issues" target="_blank">issue tracker</a> with every project, so packager will use that. If you're within an enterprise, you probably already have a bug tracker to use. If you don't, I recommend <a href="https://www.atlassian.com/software/jira" target="_blank">Jira</a> (and the rest of the <a href="https://www.atlassian.com/" target="_blank">Atlassian</a> suite) if you're a larger organization and <a href="http://trac.edgewall.org/" target="_blank">Trac</a> if you're not.</div>
<div>
<br /></div>
<div>
Either way, every change to the project should have an associated issue in your issue tracker. That way, there's a description of why the change was made associated with the blob that is the change.</div>
<div>
<br /></div>
<div>
Speaking of changes, use <a href="https://help.github.com/articles/using-pull-requests/" target="_blank">pull requests</a> wherever possible. Pull requests do two very important things:</div>
<div>
<ol>
<li>They promote code reviews, the single strongest guard against bugs.</li>
<li>They make it easy to have a blob that is the change for an issue.</li>
</ol>
<div>
I will confess, however, that when I'm working by myself on a project, I tend to commit to master without creating issues. But, I like knowing that the process can be easily changed to accomodate more than one developer.</div>
</div>
<h4>
Release process</h4>
<div>
Our release process will be pretty simple - we'll use the gem format. We can add a couple lines to our Rakefile and Ruby will handle all the work for us.</div>
<blockquote class="tr_bq">
require 'rubygems/tasks'<br />
Gem::Tasks.new</blockquote>
<div>
We will also need to add it to our gemspec</div>
<blockquote class="tr_bq">
s.add_development_dependency 'rubygems-tasks', '~> 0'</blockquote>
<div>
and <span style="background-color: #f3f3f3; font-family: 'Courier New', Courier, monospace; font-size: x-small;">bundle install</span>. That provides us with the release Rake task (among others). Please read the <a href="https://github.com/postmodern/rubygems-tasks" target="_blank">documentation</a> for what else it provides and what it does.</div>
<div>
<br /></div>
<div>
<b><u>Note:</u></b> If your DSL is for use within an enterprise, please make sure that you know where and how you are going to release your gem. This will likely be the place you will download your dependencies from, but not always. <i><b>Please</b></i> talk your IT administrator first.</div>
<h4>
First passing spec</h4>
<div>
I am a strong proponent of <a href="https://en.wikipedia.org/wiki/Test-driven_development" target="_blank">TDD</a>. Since we're starting from scratch, let's begin as we mean to continue. So, first, we add a spec/spec_helper.rb file that looks like:</div>
<div>
<blockquote class="tr_bq">
require 'simplecov'<br />
SimpleCov.configure do<br />
add_filter '/spec/'<br />
add_filter '/vendor/'<br />
minimum_coverage 100<br />
refuse_coverage_drop<br />
end<br />
SimpleCov.start<br />
<br />
require 'packager'</blockquote>
</div>
<div>
Then, add a spec/first_spec.rb file with:</div>
<div>
<blockquote class="tr_bq">
describe "DSL" do<br />
it "can compile"<br />
expect(true).to be(true)<br />
end<br />
end</blockquote>
(This is a throwaway test, and that's okay. We will get rid of it when we have something better to test.)<br />
<br />
<span style="background-color: #f3f3f3; font-family: 'Courier New', Courier, monospace; font-size: x-small;">rake spec</span> fails right away because simplecov isn't installed. You'll need to add simplecov to the packager.gemspec and run <span style="background-color: #f3f3f3; font-family: 'Courier New', Courier, monospace; font-size: x-small;">bundle install</span> to install simplecov.<br />
<br />
We haven't created lib/packager.rb yet, so when we execute <span style="background-color: #f3f3f3; font-family: Courier New, Courier, monospace; font-size: x-small;">rake spec</span>, it will fail (compile error). So, let's create lib/packager.rb with:<br />
<blockquote class="tr_bq">
class Packager<br />
end</blockquote>
<span style="background-color: #f3f3f3; font-family: 'Courier New', Courier, monospace; font-size: x-small;">rake spec</span> now passes. <span style="background-color: #f3f3f3; font-family: 'Courier New', Courier, monospace; font-size: x-small;">git add . && git commit -am "Create initial test"</span> to mark the success.<br />
<br />
We also have 100% test coverage from the start. While this isn't a silver bullet that cures cancer, it <u>does</u> tell us when we're writing code that may not always work like we think it will. By forcing every line of code to be executed at least once by something our test suite (which is what 100% coverage guarantees), we are forced to write at least one test to reach every nook and cranny. Hopefully, we'll be adult enough to recognize when there's a use-case that isn't tested, even if the code under question is executed.<br />
<h4>
Continuous Integration</h4>
</div>
<div>
Finally, before we start adding actual features, let's end with continuous integration. If you're on GitHub, then use <a href="https://travis-ci.org/" target="_blank">Travis</a>. Just copy <a href="https://github.com/robkinyon/ruby-packager/blob/master/.travis.yml" target="_blank">the one</a> in ruby-packager and it will work properly (JRuby needs some special handling for code coverage). Travis will even run the test suite on pull requests and note the success (or lack thereof) right there. In order to enable Travis to run against your GitHub repository, you will need to register with Travis and point-and-click on its site to set things up. There is plenty of documentation (including StackOverflow Q&A) on the process.</div>
<div>
<br /></div>
<div>
Otherwise, talk to your devops or automation team in order to set up integration with <a href="https://jenkins-ci.org/" target="_blank">Jenkins</a>, <a href="https://www.atlassian.com/software/bamboo" target="_blank">Bamboo</a>, or whatever you're using in your enterprise. Whatever you do, it should be set up to run the whole test suite on <i><b><u>every single push</u></b></i> on <b><u><i>every single branch</i></u></b>. More importantly, it should act as a veto for pull requests (if that's supported by your tooling).</div>
<h4>
Summary</h4>
<div>
It may not seem like we've actually <i>done</i> anything, but we've done quite a lot. A development project isn't about writing code (though that's a necessary part). It's about managing requests for change and delivering them in a sane and predictable fashion. Everything we've done here is necessary to support exactly that.</div>
<div>
<br /></div>
<div>
<table style="width: 100%;"><tbody>
<tr><td><a href="http://streamlined-book.blogspot.com/2015/08/creating-your-own-dsl-parsing-with-ruby.html" target="_blank">prev</a></td><td><div style="text-align: right;">
<a href="http://streamlined-book.blogspot.com/2015/08/creating-packager-dsl-first-feature.html" target="_blank">next</a></div>
</td></tr>
</tbody></table>
</div>
Rob Kinyonhttp://www.blogger.com/profile/07966252008096453177noreply@blogger.com0tag:blogger.com,1999:blog-2458293813533890602.post-24551247093115923002015-08-14T16:23:00.000-04:002015-08-18T15:29:21.812-04:00Creating the Packager DSL - First feature<ol>
<li><a href="http://streamlined-book.blogspot.com/2015/07/why-use-dsl.html" target="_blank">Why use a DSL?</a></li>
<li><a href="http://streamlined-book.blogspot.com/2015/08/why-create-your-own-dsl.html" target="_blank">Why create your own DSL?</a></li>
<li><a href="http://streamlined-book.blogspot.com/2015/08/what-makes-good-dsl.html" target="_blank">What makes a good DSL?</a></li>
<li><a href="http://streamlined-book.blogspot.com/2015/08/creating-your-own-dsl-parsing.html" target="_blank">Creating your own DSL - Parsing</a></li>
<li><a href="http://streamlined-book.blogspot.com/2015/08/creating-your-own-dsl-parsing-with-ruby.html" target="_blank">Creating your own DSL - Parsing (with Ruby)</a></li>
<li><a href="http://streamlined-book.blogspot.com/2015/08/creating-packager-dsl-initial-steps.html" target="_blank">Creating the Packager DSL - Initial steps</a></li>
<li><b>Creating the Packager DSL - First feature</b></li>
</ol>
<div>
In our <a href="http://streamlined-book.blogspot.com/2015/08/creating-packager-dsl-initial-steps.html" target="_blank">last segment</a>, we did everything necessary to set up the <a href="https://github.com/robkinyon/ruby-packager" target="_blank">Packager DSL repository</a>, but it doesn't actually <i>do</i> anything. Let's make a start on fixing that.</div>
<h3>
User Story</h3>
<div>
First, we need a description of what we're going to do. This, in Agile, would be the User Story. Well, first-er, we need to decide what we're going to do. For that, let's look at why our project exists at all.</div>
<div>
<br /></div>
<div>
The Packager DSL's purpose is to provide an easy way to describe what should go into a package, primary focusing on OS-packages (RPM, DEB, MSI, etc). Eventually, it should be usable to describe (nearly) every type and construction of OS package that's reasonable to expect to use. So, any package you might install on your RedHat, Ubuntu, or Windows server should be describable with this DSL. We'll defer the actual package construction to <a href="https://github.com/jordansissel/fpm" target="_blank">fpm</a>. But, all the work of collecting (and validating!) the files, figuring out versions, and constructing the invocation of fpm - that's what we'll do.</div>
<div>
<br /></div>
<div>
For our first feature, let's build an empty package. And that's the first stab at a user story.</div>
<blockquote class="tr_bq">
I want to create an empty package.</blockquote>
<div>
In talking with our client (yes, I talk to myself a lot), the first question I tend to ask is "what happens if I don't receive X?" and I don't treat myself in client-mode any differently. So, what happens if we get something like <span style="background-color: #f3f3f3;"><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">package {}</span></span>. That seems a bit off. Package filenames are usually constructed as:</div>
<blockquote class="tr_bq">
<name>-<version>-<other stuff>.<extension></blockquote>
<div>
Name and version seem to be key, so let's amend the story to require name and version.</div>
<blockquote class="tr_bq">
I want to create an empty package by specifying the name and version. If either is missing, stop and inform the user.</blockquote>
<div>
Which immediately begs the question of "How do we <i><u>stop and inform the user</u></i>?" Which then leads to the question of how we're even running the DSL. The easiest thing to do is run a script, so let's do that. If we write things properly, then we can easily change the UI from a command-line script to something fancier.</div>
<div>
<blockquote class="tr_bq">
I want to invoke a script, passing in the name of my DSL file. This should create an empty package by specifying the name and version. If either is missing, print an error message and stop. Otherwise, a package should be created in the directory I am in.</blockquote>
</div>
<div>
Hmm. "a package should be created" - what kind of package? RPM? DEB? Something else?</div>
<div>
<div>
<blockquote class="tr_bq">
I want to run a script, passing in the name of my DSL file. This should create an empty package by specifying the name, version, and package format. If any of them are missing, print an error message and stop. Otherwise, an empty package of the requested format should be created in the directory I am in.</blockquote>
</div>
</div>
<div>
<h3>
First test</h3>
The first thing we need to do is write a failing test. In TDD, this is known as <a href="http://blog.cleancoder.com/uncle-bob/2014/12/17/TheCyclesOfTDD.html" target="_blank">Red</a>-<a href="http://www.jamesshore.com/Blog/Red-Green-Refactor.html" target="_blank">Green</a>-<a href="http://agileinaflash.blogspot.com/2009/02/red-green-refactor.html" target="_blank">Refactor</a> (all three linked posts are good reading). We want to write the smallest test that can fail, then the smallest code that can pass, then take the opportunity to clean anything up (if necessary). A couple sidebars are important here.<br />
<h4>
Failing tests</h4>
Tests are worthless if they only pass. Tests <u><b><i>exist to fail</i></b></u>. If they don't fail, then they cannot warn you when something isn't working. As a result, the first thing we want to test is the test itself, otherwise we cannot trust it. When you write a test, it's really important to to see the test fail first.<br />
<br />
Also, in this process, each test is the next step in the journey of creating the product. If we've followed TDD, then every feature and each line of code is already tested. We're wanting to test something that hasn't been written yet - it's the next step. We're describing what should happen <i>once</i> we've taken the step. If that doesn't fail, then we have no confidence that we've properly described where we're planning to go.<br />
<br />
If you do not see the test fail, then several things are going wrong, probably a combination of:<br />
<br />
<ul>
<li>Your test is worthless because it won't fail when it should in the future.</li>
<li>You don't understand the system well enough to push it beyond its edges.</li>
<li>You haven't constructed your test infrastructure well enough to exercise the problem at hand.</li>
<li>You haven't described the problem at hand properly.</li>
</ul>
<h4>
Immediate Refactoring</h4>
<div>
Refactoring, as a whole, becomes much simpler with a robust and comprehensive test suite. Refactoring immediately, though, is less obviously beneficial. It's great to be able to have an opportunity to rethink your implementation right after you have it working. But, the biggest gain in my experience is that by rethinking your implementation, you end up thinking of more edge cases. Each of these becomes another test. By continuously refactoring, you keep driving the development process forward.<br />
<h3>
The test</h3>
</div>
</div>
<div>
The first test we want to write is the simplest thing that we could pass in. That would be <span style="background-color: #f3f3f3;"><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">package {}</span></span>.</div>
<pre class="ruby">
describe Packager::DSL do
it "fails on an empty package" do
expect {
Packager::DSL.parse_dsl("package {}")
}.to raise("Every package must have a name")
end
end
</pre>
<div>
When we run this with <span style="background-color: #f3f3f3;"><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">rake spec</span></span>, it fails with a compilation error because the Packager::DSL class doesn't exist. Which makes sense - we're just now taking the first step into Packager-dsl-istan. The test tells us what the first steps should be (and in what order):</div>
<div>
<ol>
<li>Create the class</li>
<li>Subclass it from DSL::Maker (which provides parse_dsl)</li>
<li>Create an entrypoint called "package"</li>
<li>Add a validation that always fails with "Every package must have a name"</li>
</ol>
<div>
Yes - always fails. We don't have a test for the success path yet, so the <i>simplest</i> code that could possibly work (while still driving us forward) is for the validation to always raise an error. We'll fix that as soon as we know how to make it succeed.</div>
</div>
<div>
<br /></div>
<div>
To make this work, we need to create a lib/packager/dsl.rb file with</div>
<pre>
require 'dsl/maker'
class Packager
class DSL < DSL::Maker
add_entrypoint('package') do
end
add_validation('package') do
return "Every package must have a name"
end
end
end
</pre>
<div>
<span style="background-color: #f3f3f3; font-family: 'Courier New', Courier, monospace; font-size: x-small;">rake spec</span> still fails stating it cannot find Packager::DSL. Huh?! Ahh ... we forgot to load it in our spec_helper. We can either add a require statement in spec/spec_helper.rb or we can add it in lib/packager.rb. Either one is fine - you can always move it later if you find you need to.<br />
<br />
Now, <span style="background-color: #f3f3f3; font-family: 'Courier New', Courier, monospace; font-size: x-small;">rake spec</span> gives a different error - it cannot find DSL::Maker. We're not creating that - we're going to use it from RubyGems. So, let's add it to our gemspec file. (Remember - our Gemfile is delegating to our gemspec.) We want to make sure we're using at least 0.1.0 (the latest version as of this writing).<br />
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;"><span style="background-color: white; line-height: 16.7999992370605px; white-space: pre;">s.add_dependency </span><span class="pl-s" style="background-color: white; box-sizing: border-box; line-height: 16.7999992370605px; white-space: pre;"><span class="pl-pds" style="box-sizing: border-box;">'</span>dsl_maker<span class="pl-pds" style="box-sizing: border-box;">'</span></span><span style="background-color: white; line-height: 16.7999992370605px; white-space: pre;">, </span><span class="pl-s" style="background-color: white; box-sizing: border-box; line-height: 16.7999992370605px; white-space: pre;"><span class="pl-pds" style="box-sizing: border-box;">'</span>~> 0.1<span class="pl-pds" style="box-sizing: border-box;">'</span></span><span style="background-color: white; line-height: 16.7999992370605px; white-space: pre;">, </span><span class="pl-s" style="background-color: white; box-sizing: border-box; line-height: 16.7999992370605px; white-space: pre;"><span class="pl-pds" style="box-sizing: border-box;">'</span>>= 0.1.0<span class="pl-pds" style="box-sizing: border-box;">'</span></span></span></blockquote>
After a quick <span style="background-color: #f3f3f3;"><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">bundle install</span></span>, <span style="background-color: #f3f3f3; font-family: 'Courier New', Courier, monospace; font-size: x-small;">rake spec</span> now runs cleanly. We also want to delete the other spec file we created when we added our scaffolding. So, <span style="background-color: #f3f3f3;"><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">git add . && git rm spec/first_spec.rb</span></span>. We've removed a spec, so let's make sure we still have 100% cover with <span style="background-color: #f3f3f3; font-family: 'Courier New', Courier, monospace; font-size: x-small;">rake spec</span>. Once confirmed, <span style="background-color: #f3f3f3;"><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">git commit -m "Create first spec with a real DSL"</span></span>.<br />
<h4>
Writing for failure first</h4>
<div>
In the same way that we want to write the test first and see it fail, we want to write for failure (or sad-day) situations first. Writing for success (or happy-day) is actually pretty easy - it's the way most people think of the world. It's the way our user story was written and what most people are going to be looking for when they evaluate your work. But, if you think back to the software you've used, the best software was the one that caught and managed the <i>errors</i> the best. There is nothing more frustrating that a program that blithely lets you get into a huge mess that it should've (at least) warned you about.</div>
<div>
<br /></div>
<div>
So, the best way to write software is to try and figure out all the different ways a person can screw up and plug those holes. You'll miss some - <a href="https://en.wikiquote.org/wiki/Rick_Cook#The_Wizardry_Compiled_.281989.29" target="_blank">the universe is always making a better idiot</a>. But, you'll catch most of them, and that's what matters.</div>
<h3>
Adding the attributes</h3>
</div>
<div>
Let's add a success. Let's allow a DSL that has a package with just a name succeed. We're going to have to amend this success when we add version and type, but the neat thing about code is that it's so moldy. (Err ... changeable.)</div>
<div>
<br /></div>
<div>
What does it mean for a DSL to succeed? The immediate straight-line to solve the user story would be to invoke FPM directly. While tempting, it's too big of a step to take. Right now, we're focused on parsing the DSL. So, let's make sure we're doing that right before worrying about integration with another library. For now, let's just create a data structure that represents what we put into the DSL. Ruby provides a very neat thing called a <a href="http://ruby-doc.org/core-2.2.0/Struct.html" target="_blank">Struct</a> which allows us to define a limited data structure without too much effort. Let's add another specification into spec/empty_package_spec.rb</div>
<pre>
it "succeeds with a name" do
items = Packager::DSL.execute_dsl {
package {
name "foo"
}
}
expect(items[0]).to be_instance_of(Packager::DSL::Package)
expect(items[0].name).to eq("foo")
end
</pre>
<div>
(I prefer using execute_dsl() because it allows Ruby to make sure my package definitions compile. You can use parse_dsl() instead.) Make sure it fails, then change lib/packager/dsl.rb to be</div>
<pre>
class Packager
class DSL < DSL::Maker
Package = Struct.new(:name)
add_entrypoint('package', {
:name => String,
}) do
Package.new(name)
end
add_validation('package') do |item|
return "Every package must have a name" unless item.name
end
end
end
</pre>
<div>
That should pass both the new test <u>and</u> the old test. Commit everything with a good message. Adding the version and type attributes should be a similar sequence of activities. Make sure to exercise the discipline of adding each one separately, ensuring that you have a failing test, then minimal code, then passing tests.<br />
<br />
You may have to amend tests to make them pass with the new code. That's expected as the requirements change. Just make sure that the tests you have to amend are ones you either expected to amend or which make sense to amend. Don't just blindly "make the tests pass". That makes the test suite worthless.<br />
<h3>
Summary</h3>
</div>
<div>
We haven't finished the user story, but we've gotten to the point where we can parse a DSL with validations and have a predictable data structure at the end of it. Looking down the road, we will take that data structure and invoke fpm with it. Then, we'll see some really neat benefits to having the separation between parsing and execution.</div>
<div>
<br /></div>
<div>
<a href="http://streamlined-book.blogspot.com/2015/08/creating-packager-dsl-initial-steps.html" target="_blank">prev</a></div>
Rob Kinyonhttp://www.blogger.com/profile/07966252008096453177noreply@blogger.com0tag:blogger.com,1999:blog-2458293813533890602.post-1632423218231831002015-08-11T18:37:00.000-04:002015-08-17T12:05:21.200-04:00Creating your own DSL - Parsing (with Ruby)<ol>
<li><a href="http://streamlined-book.blogspot.com/2015/07/why-use-dsl.html" target="_blank">Why use a DSL?</a></li>
<li><a href="http://streamlined-book.blogspot.com/2015/08/why-create-your-own-dsl.html" target="_blank">Why create your own DSL?</a></li>
<li><a href="http://streamlined-book.blogspot.com/2015/08/what-makes-good-dsl.html" target="_blank">What makes a good DSL?</a></li>
<li><a href="http://streamlined-book.blogspot.com/2015/08/creating-your-own-dsl-parsing.html" target="_blank">Creating your own DSL - Parsing</a></li>
<li><b>Creating your own DSL - Parsing (with Ruby)</b></li>
<li><a href="http://streamlined-book.blogspot.com/2015/08/creating-packager-dsl-initial-steps.html" target="_blank">Creating the Packager DSL - Initial steps</a></li>
<li><a href="http://streamlined-book.blogspot.com/2015/08/creating-packager-dsl-first-feature.html" target="_blank">Creating the Packager DSL - First feature</a></li>
</ol>
<div>
<a href="https://www.ruby-lang.org/" target="_blank">Ruby</a> is a full-featured generic programming language with all the standard bells and whistles. So, it may seem odd that I'm suggesting it is also useful for parsing DSLs. But, Ruby has a few interesting features (both in syntax and semantics) which make it ideal for parsing DSLs:</div>
<div>
<ul>
<li>All functions are method calls on an implicit object</li>
<li>Code blocks as the last parameter of a method call</li>
<li>Greedy implicit binding rules</li>
</ul>
<div>
In short, Ruby makes it possible for a DSL author to create a class that, with a little help, will treat all key-value pairs as method invocations on an object. This includes nested blocks (though we need to create a new class for each level of descent). Because these nested blocks are values of some key, that key is the method call that receives the code block. And, because Ruby doesn't require a lot of symbols like parentheses and the like, the DSL ends up looking very clean and non-program-ish. (All the DSL examples in this series are parsable in Ruby.)</div>
<div>
<br /></div>
<div>
Most importantly, all of the features of full Ruby language (such as branching and looping) are available for free. The DSL author doesn't have to do anything special - it's just there. The DSL user simply has to be pointed at the standard Ruby documentation (and all the internet resources) to know how to solve any problem. If the DSL author desires, they can even elide over the use of Ruby and document Ruby's syntax as part of their language. There's no requirement to trumpet the use of Ruby in your DSL.</div>
<div>
<br /></div>
<div>
Ruby's only contribution to DSL parsing is a function called <i>instance_exec(). </i>It is the magic sauce that makes a code block act as if all the functions are method calls on the object of our choosing (vs whatever object is appropriately in scope). It's also extremely low-level, which is an obstacle to DSL authors.</div>
<div>
<br /></div>
<div>
The <a href="https://rubygems.org/gems/docile" target="_blank">docile</a> gem (aka, a Ruby package) provides a very nice way of mapping classes (and their objects) to different levels while enhancing the error-handling. It's definitely more usable than <i>instance_exec()</i> on its own. But, you still have to know far more about Ruby classes and objects that a DSL author should. It also doesn't provide any facilities for validation or production, leaving those as exercises for the reader.</div>
</div>
<div>
<br /></div>
<div>
The <a href="https://rubygems.org/gems/dsl_maker" target="_blank">dsl_maker</a> gem wraps <a href="https://rubygems.org/gems/docile" target="_blank">docile</a> with a more explicit way of declaring the DSL structure and validation. In essence, it provides a quasi-DSL for declaring DSL parsing and validation rules. It also allows the DSL author to work with concepts that map more closely to DSL creation, such as types and nested structures without having to maintain the mapping between DSL nesting and Ruby classes.</div>
<div>
<br /></div>
<div>
Over the next few posts, I'm going to walk through the creation of a non-trivial DSL designed to describe how a package should be created. I will discuss creating and maintaining the parser, validator, and executor. I will also discuss distribution, testing, and all the other aspects of a good software project and how those things are handled within a DSL. You can follow along at the <a href="https://github.com/robkinyon/ruby-packager" target="_blank">packager GitHub repository</a>.<br />
<br />
<table style="width: 100%;"><tbody>
<tr>
<td><a href="http://streamlined-book.blogspot.com/2015/08/creating-your-own-dsl-parsing.html" target="_blank">prev</a></td>
<td style="text-align: right;"><a href="http://streamlined-book.blogspot.com/2015/08/creating-packager-dsl-initial-steps.html" target="_blank">next</a></td>
</tr>
</tbody></table>
</div>
Rob Kinyonhttp://www.blogger.com/profile/07966252008096453177noreply@blogger.com0tag:blogger.com,1999:blog-2458293813533890602.post-18963596815600635342015-08-10T09:00:00.000-04:002015-08-17T12:05:17.282-04:00Creating your own DSL - Parsing<ol>
<li><a href="http://streamlined-book.blogspot.com/2015/07/why-use-dsl.html" target="_blank">Why use a DSL?</a></li>
<li><a href="http://streamlined-book.blogspot.com/2015/08/why-create-your-own-dsl.html" target="_blank">Why create your own DSL?</a></li>
<li><a href="http://streamlined-book.blogspot.com/2015/08/what-makes-good-dsl.html" target="_blank">What makes a good DSL?</a></li>
<li><b>Creating your own DSL - Parsing</b></li>
<li><a href="http://streamlined-book.blogspot.com/2015/08/creating-your-own-dsl-parsing-with-ruby.html" target="_blank">Creating your own DSL - Parsing (with Ruby)</a></li>
<li><a href="http://streamlined-book.blogspot.com/2015/08/creating-packager-dsl-initial-steps.html" target="_blank">Creating the Packager DSL - Initial steps</a></li>
<li><a href="http://streamlined-book.blogspot.com/2015/08/creating-packager-dsl-first-feature.html" target="_blank">Creating the Packager DSL - First feature</a></li>
</ol>
<div>
So far, we've talked about the whys and wherefores of DSLs. If you've made it this far, you probably agree that DSLs are a good idea. You have probably identified a spot in your processes where a DSL would make life so much easier. So, let's get started.</div>
<div>
<br /></div>
<div>
At its heart, creating a programming language (or DSL) is dealing with these three activities:</div>
<div>
<ol>
<li>Parsing the program file into a data structure</li>
<ul>
<li>If there are errors in syntax, inform the user here.</li>
</ul>
<ol>
</ol>
<li>Validating the data structure</li>
<ul>
<li>If there are errors in semantics, inform the user here.</li>
</ul>
<ol>
</ol>
<li>Executing the requested activities</li>
<ul>
<li>If there are errors in what was attempted, inform the user here.</li>
</ul>
</ol>
</div>
<div>
Before we can design our language, we need to pick what our parsing process will be. The parsing step is what deals with user interface. The parser we choose will strongly shape what kind of language we can create for our users. If we pick a very simplistic parser, that means we are constrained to a possibly unusable DSL. It doesn't matter what kind of wonderful things our DSL can do if no-one is able to work with the language itself. (For example, compare <a href="https://en.wikipedia.org/wiki/COBOL" target="_blank">COBOL</a> and <a href="https://en.wikipedia.org/wiki/BASIC" target="_blank">BASIC</a> with <a href="https://www.java.com/" target="_blank">Java</a>, <a href="https://www.python.org/" target="_blank">Python</a>, and <a href="https://www.ruby-lang.org/" target="_blank">Ruby</a>.) On the other hand, if we pick a very complex parser, we may never end up creating the language at all.<br />
<br />
There are hundreds of tools for creating programming languages. (Make no mistake - you're doing just that.) The problem with most of them (and the reason most DSLs are never created) is that they're far too complicated (such as writing a <a href="https://en.wikipedia.org/wiki/Parsing" target="_blank">parser</a> and <a href="https://en.wikipedia.org/wiki/Lexical_analysis" target="_blank">lexer</a> to generate an <a href="https://en.wikipedia.org/wiki/Abstract_syntax_tree" target="_blank">AST</a>). Most developers simply will not be able to reframe their simple DSL in terms of tokens and similar parsing terms.<br />
<br />
The good news is that most DSLs do not need the full treatment of a parser+lexer. Most DSLs are better ways of describing nested data structures. So, maybe we should try something like <a href="https://en.wikipedia.org/wiki/JSON" target="_blank">JSON</a> or <a href="https://en.wikipedia.org/wiki/yaml" target="_blank">YAML</a>. And some tools (such as <a href="http://www.ansible.com/home" target="_blank">Ansible</a> and <a href="http://saltstack.com/" target="_blank">Salt</a>) do just that. They use a YAML parser to handle step 1. This definitely solves the problem of parsing - just let someone else do it. :)</div>
<div>
<br />
YAML and JSON, though, while very easy for the DSL author to work with, is a really poor interface for the DSL <i>user</i>. Data structures become extremely brittle the moment you use them for anything non-trivial. This matters because the user is the group that will be working within the DSL 100x more than the author will be working within the DSL definition. We should optimize (wherever possible) for the most usage.<br />
<br />
The first major issue is no variables. Every programming language (and DSLs are no exception) works in problemspaces that want to reuse the same values. When using just YAML, the user ends up having to re-specify the same value over and over. When (not if!) that value changes, the user will invariably forget one place to change the value.<br />
<br />
An enterprising DSL author might decide to create a section in their document called "variables" (or somesuch) and allow the user to specify a hashtable of key-value pairs for use elsewhere in the DSL. This <i>would</i> work, but it becomes very cumbersome to work <i>with</i>. Now, the user has to specify "This is a variable lookup" which means the author has to provide special tokens (or <i>sigils</i>) for doing that. Now, the user cannot just "use YAML". It's YAML+. And every DSL author will have their own unique '+'. No-one wants to invest in non-transferrable skills.<br />
<br />
The second, larger issue is no control mechanisms. Programming languages provide three key control mechanisms:<br />
<ol>
<li>Branching (if-then-else, switch/case)</li>
<li>Looping (for loops, while loops)</li>
<li>Abstraction (subroutines)</li>
</ol>
<div>
Without these, data structures balloon in size because the user ends up having to repeat themselves. An example can illustrate best:<br />
<br /></div>
<div>
<table style="width: 100%;"><tbody>
<tr>
<td>server {<br />
name "www1.place.com"<br />
ip "192.168.50.1"<br />
...<br />
}<br />
server {<br />
name "www2.place.com"<br />
ip "192.168.50.2"<br />
...<br />
}<br />
server {<br />
name "www3.place.com"<br />
ip "192.168.50.3"<br />
...<br />
}</td>
<td>(1 .. 3).each do |id|<br />
server {<br />
name "www#{id}.place.com"<br />
ip "192.168.50.#{id}"<br />
...<br />
}<br />
end</td>
</tr>
</tbody></table>
</div>
<div>
<br />
Now, imagine the server definition runs to 50 lines, all of them identical except for the two lines in the example. And instead of 3 webservers, your product is in heavy use and has 10 servers. Or 30. Which would you prefer to maintain, as a user? Your users will feel the same way.<br />
<br />
The maintainers of Salt and Ansible immediately recognized this problem and provide an interesting solution. Their DSL files aren't actually YAML. They are <a href="http://jinja.pocoo.org/" target="_blank">Jinja2</a> files that render to YAML. Jinja2 is a templating language that provides variables and the control structures missing from YAML. Obviously, some products (Salt and Ansible) feel this is a workable solution. Their users must feel the same way, or they wouldn't use the product.<br />
<br />
I don't agree. This forces users to learn <i>two</i> languages - YAML and Jinja2. That's twice the barrier to entry and twice the opportunity for the user to make an error. DSLs are meant to <i>reduce</i> the barriers to entry and <i>reduce</i> the potential for error. But, we still want someone else to handle all the parsing (because parsing is hard and error-prone). We need an easy way to set key-value pairs (because that's most of what we want), but we still want variables and all the valuable control structures. While 90% of all the usage of the type of DSL we're writing could be satisfied by basic YAML, we want the escape hatch of a full-on programming language <i>for when we need it</i>. There has to be a better tool<br />
<br />
That better tool is <a href="https://www.ruby-lang.org/" target="_blank">Ruby</a>.<br />
<table style="width: 100%;"><tbody>
<tr><td><a href="http://streamlined-book.blogspot.com/2015/08/what-makes-good-dsl.html" target="_blank">prev</a></td><td><div style="text-align: right;">
<a href="http://streamlined-book.blogspot.com/2015/08/creating-your-own-dsl-parsing-with-ruby.html" target="_blank">next</a></div>
</td></tr>
</tbody></table>
</div>
</div>
Rob Kinyonhttp://www.blogger.com/profile/07966252008096453177noreply@blogger.com0tag:blogger.com,1999:blog-2458293813533890602.post-90282050714701816262015-08-05T11:01:00.000-04:002015-08-17T12:05:11.315-04:00What makes a good DSL?<ol>
<li><a href="http://streamlined-book.blogspot.com/2015/07/why-use-dsl.html" target="_blank">Why use a DSL?</a></li>
<li><a href="http://streamlined-book.blogspot.com/2015/08/why-create-your-own-dsl.html" target="_blank">Why create your own DSL?</a></li>
<li><b>What makes a good DSL?</b></li>
<li><a href="http://streamlined-book.blogspot.com/2015/08/creating-your-own-dsl-parsing.html" target="_blank">Creating your own DSL - Parsing</a></li>
<li><a href="http://streamlined-book.blogspot.com/2015/08/creating-your-own-dsl-parsing-with-ruby.html" target="_blank">Creating your own DSL - Parsing (with Ruby)</a></li>
<li><a href="http://streamlined-book.blogspot.com/2015/08/creating-packager-dsl-initial-steps.html" target="_blank">Creating the Packager DSL - Initial steps</a></li>
<li><a href="http://streamlined-book.blogspot.com/2015/08/creating-packager-dsl-first-feature.html" target="_blank">Creating the Packager DSL - First feature</a></li>
</ol>
If someone wanted to create a human language which was really good at managing reindeer herds, it would make more sense to just use Sami. It has a lot of <i>busat</i> (expressive terseness) in that domain. Likewise, it makes the most sense to just use Inuit terms when building a human language focused on dealing with matters of snow and ice. These <i>domain-specific</i> languages (DSLs) focus their primary areas of terseness on specific areas (reindeer, snow/ice, etc), rendering them less suitable for other areas (tropical seafaring, web application devleopment, etc).<br />
<br />
The same goes for programming languages. If you work on modern web applications, you already use several DSLs, such as SQL and CSS (for set manipulation and defining visitors on trees, respectively). These DSLs have two qualities that elevate them to the top of the DSL game. They are:<br />
<br />
<ol>
<li>Intensely Focused (Do only one thing, but do it well)</li>
<li>Expressively Terse (Just the facts)</li>
</ol>
<h4>
Intensely Focused</h4>
<div>
<a href="https://en.wikipedia.org/wiki/SQL" target="_blank">SQL</a> is the standard language for interacting with data in a relational database, such as Oracle or MySQL. Relational databases store their data in sets. At its heart, SQL is a set manipulation DSL. It has roughly 40-60 keywords (depending on the dialect and how you count). But, every single one of those keywords has a single purpose focusing on one (and only one) of the basic set operations. Where SQL has keywords that do non-set things (such as collations or engine-specific hints), that's where people complain the most about how complicated SQL is. While dealing with set theory can be difficult to master, no-one complains about how SQL implements it.</div>
<div>
<br /></div>
<div>
<a href="https://en.wikipedia.org/wiki/Cascading_Style_Sheets" target="_blank">CSS</a> is even more focused. There are hundreds of properties, each with their own specific set of acceptable values (potentially depending on what type of node is affected), and that's what most people see in CSS. But, CSS isn't a DSL for setting values for properties - it's a DSL for creating actions to take when walking trees. In short, CSS is a massive <a href="https://en.wikipedia.org/wiki/Visitor_pattern" target="_blank">Visitor</a> pattern definer.</div>
<div>
<br /></div>
<div>
But, CSS doesn't allow you to take just any action - you are only allowed to set properties on nodes. It isn't a generic visitor pattern definer - it is focused on one type of visitor action. There may be hundreds of properties, but they all follow the exact same pattern of <span style="color: #999999; font-family: Courier New, Courier, monospace;"><b>name: value [modifier];</b></span>. This allows it to be more generic when it comes to the matching rules for which nodes in the tree are affected, which is the true power of CSS.</div>
<h4>
Expressively Terse</h4>
<div>
Terseness is a quality of using as few words as possible to say what you want to say. It's expressive only if every word we use is exactly the right word for the job. Or, if the concept expressed by the word is exactly the right concept. You have to truly understand what your DSL is focused on doing.</div>
<div>
<br /></div>
<div>
Both SQL and CSS are extreme terse. There is one and only one keyword or operand for each concept. Every concept is mapped clearly and cleanly to the problemspace at hand. If you removed any keyword or operand, you would cripple the DSL's ability to solve problems.</div>
<div>
<br /></div>
<div>
CSS, in specific, is extremely terse. Syntactically, the only interesting things happen in the selectors. But, even with that complexity, there is only way to specify a specific path to a node. (Depending on your structure, you may be able to specify multiple ways to get to a node, but there's only one way to specify each way.)<br />
<br />
<table style="width: 100%;"><tbody>
<tr><td><a href="http://streamlined-book.blogspot.com/2015/08/why-create-your-own-dsl.html" target="_blank">prev</a></td><td style="text-align: right;"><a href="http://streamlined-book.blogspot.com/2015/08/creating-your-own-dsl-parsing.html" target="_blank">next</a></td></tr>
</tbody></table>
</div>
Rob Kinyonhttp://www.blogger.com/profile/07966252008096453177noreply@blogger.com0tag:blogger.com,1999:blog-2458293813533890602.post-7657149722008912622015-08-03T09:25:00.000-04:002015-08-17T12:05:06.762-04:00Why create your own DSL?<ol>
<li><a href="http://streamlined-book.blogspot.com/2015/07/why-use-dsl.html" target="_blank">Why use a DSL?</a></li>
<li><b>Why create your own DSL?</b></li>
<li><a href="http://streamlined-book.blogspot.com/2015/08/what-makes-good-dsl.html" target="_blank">What makes a good DSL?</a></li>
<li><a href="http://streamlined-book.blogspot.com/2015/08/creating-your-own-dsl-parsing.html" target="_blank">Creating your own DSL - Parsing</a></li>
<li><a href="http://streamlined-book.blogspot.com/2015/08/creating-your-own-dsl-parsing-with-ruby.html" target="_blank">Creating your own DSL - Parsing (with Ruby)</a></li>
<li><a href="http://streamlined-book.blogspot.com/2015/08/creating-packager-dsl-initial-steps.html" target="_blank">Creating the Packager DSL - Initial steps</a></li>
<li><a href="http://streamlined-book.blogspot.com/2015/08/creating-packager-dsl-first-feature.html" target="_blank">Creating the Packager DSL - First feature</a></li>
</ol>
DSLs are great. The problem, though, is that there are many domains that have been too small for someone to write a DSL for. SQL and CSS exist because millions of develpers need to access relational data and style web pages. There are dozens of domains that could use a DSL, from packaging to orchestration to HTTP route management to cross-product configuration management. Let alone all the domains that are specific to your organization, from the corporation down to the team.<br />
<br />
Just because there isn't a DSL doesn't mean you aren't programming in that domain. For example, you may have to manage a heterogenous environment of Apache and Nginx web servers for various legacy reasons. They may have server-specific configurations, but they have to share a set of configuration. Someone has to ensure that a change to the Nginx configurations is both made to the Apache configurations <i>and</i> that the change is translated properly <i>and</i> that the change occurs in the same deployment.<br />
<br />
No-one will ever create a publicly-available generic DSL for managing web server configurations. There just isn't a large enough population who have to maintain a heterogenous web server environment. And even if there was such a DSL, it wouldn't be quite as useful for your needs. It would be <i>generic</i> - everything available as the lowest common denonimator. Which reduces the <i>busat</i> of the DSL for your purposes.<br />
<br />
Busat ("expressive terseness") is not an objective measure equivalent for all people in all places. Expressiveness is directly related to the receiver's ability to understand what was communicated. If you can limit your listeners to only those who agree on specific terms, then you can be terser while remaining as expressive. If you have a jargon, then a DSL can take advantage of that.<br />
<br />
Compare the following examples:<br />
<br />
<table style="width: 100%;"><tbody>
<tr>
<td>server {<br />
hostname "host1.domain.com"<br />
ssl {<br />
key_file "/etc/ssl/key_file"<br />
ca_file "/etc/ssl/ca_file"<br />
pem_file "/etc/ssl/pem_file"<br />
}<br />
....<br />
}</td>
<td>purpose :internal_web {<br />
ssl_root_directory "/etc/ssl"<br />
domain "domain.com"<br />
# Other internal_web things<br />
....<br />
}<br />
<br />
server "host1" {<br />
purpose :internal_web<br />
....<br />
}</td>
</tr>
</tbody></table>
<br />
If your audience can all agree on what "internal_web" means, then that's strictly better. It describes exactly what you're doing, why you're doing it, and changes become much easier to vet for correctness. But, unless you're willing to write your own DSL, you would never be able to collapse the boilerplate in the configuration.<br />
<br class="Apple-interchange-newline" />
It's highly unlikely you specifically have to maintain both Apache and Nginx configurations to do the same thing. But, it's guaranteed that your organization or team has processes unique to it. Some special snowflake way of looking at <i>something</i> in the development process. Something that's just really annoying to manage in the standard language. Some good places to look are:<br />
<ul>
<li>Packaging and orchestration (or most other devops/operations activities)</li>
<li>Configuration file generation</li>
<ul>
<li>web servers</li>
<li>monitoring</li>
<li>datastores</li>
</ul>
<li>Configuration value management across environments</li>
<li>Anything that has to interact with multiple different systems</li>
<li>Anything repetitive (such as CSS, for which there is <a href="http://lesscss.org/" target="_blank">Less</a>)</li>
</ul>
<table style="width: 100%;"><tbody>
<tr><td><a href="http://streamlined-book.blogspot.com/2015/07/why-use-dsl.html" target="_blank">prev</a></td><td style="text-align: right;"><a href="http://streamlined-book.blogspot.com/2015/08/what-makes-good-dsl.html" target="_blank">next</a></td></tr>
</tbody></table>
Rob Kinyonhttp://www.blogger.com/profile/07966252008096453177noreply@blogger.com0tag:blogger.com,1999:blog-2458293813533890602.post-205274670170353202015-07-29T13:05:00.003-04:002015-08-17T12:05:00.511-04:00Why use a DSL?<ol>
<li><b>Why use a DSL?</b></li>
<li><a href="http://streamlined-book.blogspot.com/2015/08/why-create-your-own-dsl.html" target="_blank">Why create your own DSL?</a></li>
<li><a href="http://streamlined-book.blogspot.com/2015/08/what-makes-good-dsl.html" target="_blank">What makes a good DSL?</a></li>
<li><a href="http://streamlined-book.blogspot.com/2015/08/creating-your-own-dsl-parsing.html" target="_blank">Creating your own DSL - Parsing</a></li>
<li><a href="http://streamlined-book.blogspot.com/2015/08/creating-your-own-dsl-parsing-with-ruby.html" target="_blank">Creating your own DSL - Parsing (with Ruby)</a></li>
<li><a href="http://streamlined-book.blogspot.com/2015/08/creating-packager-dsl-initial-steps.html" target="_blank">Creating the Packager DSL - Initial steps</a></li>
<li><a href="http://streamlined-book.blogspot.com/2015/08/creating-packager-dsl-first-feature.html" target="_blank">Creating the Packager DSL - First feature</a></li>
</ol>
Languages exist to communicate ideas. Most of us are familiar with generic human languages like English, Swahili, Japanese - even created languages like Esperanto and Lojban. These are able to express any idea humans can possibly come up with in a way other humans can understand. In programming terms, all human languages are <a href="https://en.wikipedia.org/wiki/Turing_completeness" target="_blank">Turing-complete</a>.<br />
<br />
Sometimes, though, some ideas are easier to express in certain languages vs. others. Supposedly, <a href="https://en.wikipedia.org/wiki/Eskimo_words_for_snow" target="_blank">Eskimos have 50+ words for snow</a> and <a href="http://www.washingtonpost.com/national/health-science/there-really-are-50-eskimo-words-for-snow/2013/01/14/e0e3f4e0-59a0-11e2-beee-6e38f5215402_story.html" target="_blank">the Sami have nearly 1000 words for reindeer</a>. Given how important those topics are in those cultures, that would a lot of sense. People working together in those domains would be able to communicate more quickly because the same effort communicates more concepts. For example, "busat" (in Sami) translates to "male reindeer with a single, very large testicle". I have no idea how often this occurs, but that's probably a unique identifier in most reindeer herds.<br />
<br />
We see this as well in programming languages. Programmers were writing object-oriented programs in ANSI C for years before Bjarne Stroustrup created C++. It's <i>easier</i> to write OO programs in C++ than in C. In C, you have to be extremely disciplined to make sure that you're adhering to public vs. private interfaces, that you invoke the "methods" properly (passing the invocant as the first parameter, passing the correct invocant to the right method, etc), and lots of other bookkeeping. It's just exhausting to keep track of all of that, especially across a large codebase. In C++, the language not only reduces the bookkeeping you have to do, but it also reduces the number of characters you have to read (and type, but read is more important).<br />
<br />
Like Sami, there are programming languages that trade expressibility in one domain for another. I suspect most of the words for computing and the internet in Sami are borrowed from English (as they are in many other languages). All the usable words are already taken for other purposes. "Scripting" languages, like Ruby and Python and Javascript, make a similar set of tradeoffs. They give up the ability to write programs that execute extremely quickly (like programs written in C would do) in order to make it easy for humans to write the programs. Programs written in these languages are often much shorter (10-100x shorter) than the equivalent in C or Java. They are much more expressive when it comes to specific domains of computing. No-one would write an <a href="http://www.foo.be/docs/tpj/issues/vol5_2/tpj0502-0009.html" target="_blank">operating system in Perl</a>, but these languages excel at manipulating text and talking to databases at faster-than-human-reaction-time speeds.<br />
<br />
Expressive terseness (aka, "busat") is really important in programming because the hardest part of doing development is working within existing code. Depending on whose percentages you want to use, the maintenance phase of a project is anywhere from 60%-90% of the time and cost of that project. Maintenance, first and foremost, is an effort in reading comprehension. You can't fix a bug unless you understand the code where the bug lives, what code is connected to it, and how the various execution paths wend through that code (and the code around it). This is a lot easier to do when you're dealing with 50 lines of code than 500 (assuming equal <a href="https://en.wikipedia.org/wiki/Cyclomatic_complexity" target="_blank">cyclomatic complexities</a>). The business-level concepts are easier to see and there are fewer places for bugs to hide.<br />
<br />
SQL and CSS are good examples of DSLs that take complex domains (set manipulation and style metadata, respectively) and allow the developer to express exactly and minimally what they are trying to accomplish. Querying sets - writing joins, projections, and all the other logic that SQL provides - is extremely complicated. Doing this in any standard programming language can run to hundreds and thousands of lines with lots of cyclomatic complexity. Plenty of places for bugs to live. A DSL makes it easier to express the desire to do these three set conjunctions (using these indices for lookup), then project these 5 data points (with these manipulations), ordered in this way.<br />
<br />
DSLs also make it much easier for people working in different languages (or even business domains) to collaborate and learn from each other within the domain. There are hundreds of forums, discussion boards, and blogs on SQL or CSS tips, tricks, and improvements. These tips work regardless of what programming language you use.<br />
<div style="text-align: right;">
<a href="http://streamlined-book.blogspot.com/2015/08/why-create-your-own-dsl.html" target="_blank">next</a></div>
Rob Kinyonhttp://www.blogger.com/profile/07966252008096453177noreply@blogger.com0tag:blogger.com,1999:blog-2458293813533890602.post-35196409676993990612015-07-05T11:21:00.000-04:002015-07-05T11:26:32.069-04:00What is production?In <a href="http://streamlined-book.blogspot.com/2015/06/what-is-application.html" target="_blank">What is an application?</a>, I propose a definition for "application" as "A set of capabilities provided to a user to enable them to satisfy their desires." But, there are many other terms that are undefined. Over the next several posts, I'll define each one. The most important (after application) is "production", so I'll start there.<br />
<br />
Let's do this with a thought experiment. Pretend that your application only has a production, however you define it. This is where your users come and where you make your money (assuming you do). There is only the one instance and, because there's only one, no-one needs a name for it. It's just "the application" - there's nothing to confuse it with. Anytime you need to make a change, you go make it in "the application" and your users immediately see it. Sounds good, right?<br />
<br />
Of course, no-one works like this, and for good reason. Some changes are small enough that they can be made directly where your users are interacting, but the vast majority of them are not. Most changes require several hours (if not days) of work, often collaborating between multiple people and are built in stages you don't want your users to see.<br />
<br />
So, we distinguish between where users go for the "live" application and where developers work to make changes. Stand up a clone of production, except it doesn't have live users going to it, and call it "development". Developers can make changes to it knowing they are safe from affecting the business. Production remains the place where users satisfy their desires.<br />
<br />
So far, it seems pretty clear what production vs. development is. Production is where users go (but not developers) and development is where developers go (but not users). And, from a developer's perspective, that would be enough.<br />
<br />
There are more stakeholders in an application than just users and developers. At minimum, you have the business owners. They define what the application is meant to do - what desires the user is attempting to satisfy and what capabilities the user will have to do so. If communication was perfect, then the business owners could tell the developers "Do this" and be assured that the necessary changes would happen exactly as they intended. This also assumes developers will never make mistakes. In real life, neither statement is remotely true. Review of work requested is a fact of life. Business owners need to assure and control the quality of what they pay for. Hence, the name "QA" (or, sometimes, "QC", for quality control).<br />
<br />
Some organizations choose to have such review occur within the development instance. This makes a lot of sense for smaller, newer, and/or slower projects who either cannot or do not need the ongoing cost of a separate instance. In most other projects, the shortcomings of this plan become obvious very quickly. Ongoing development makes it difficult to determine if a failure is because of the work under review or the unstable nature of the development instance. Business owners are uncertain what would happen to the production instance if they approve the work done for a request. Will the change for that request work properly when users try to exercise the new capability? Were the failures in that change or in something else?<br />
<br />
We have development, QA/QC, and production. It's pretty obvious what "production" is - it's where the users are and it has to be stable with a managed and defined process for change.<br />
<br />
So, where does a demonstration/demo or training environment fit? It's not <i>the</i> production, but it needs to be stable for a smaller set of users and a limited window of time. This is where a lot of organizations stumble, attempting to tie the demo or training instance to either the existing production (slow-changing) or QA (quick-changing) environments. Except, the business needs usually require a middle-ground between the two.<br />
<br />
Which leads to the better definition of "production". Or, rather, splitting out what constitutes "production" into different knobs we can apply to other environments.<br />
<br />
The first knob is change management. Different environments will change within different change control regimens. This knob is based on who decides when the environment changes. Development changes whenever a developer edits a file. QA changes whenever a developer finishes some work. Production, however, changes whenever the business feels a feature is both ready for use and appropriate for release. A demo or training environment will be similarly managed by the business, not the development teams.<br />
<br />
The second knob is the stringency of review. We've already seen how changes to production will usually go through a QA environment first before a user will see it in production. Demo and training environments also need similar review because users will be in these environments.<br />
<br />
So, what's the difference between production, training, and demo? From a developer's perspective, often nothing. They're all strongly controlled environments with reviewed changes pushed when the business wants them.<br />
<br />
All of this discussion leads to this:<br />
<ol>
<li>Production is where users live.</li>
<li>Production is where change control is at its maximum (whatever that is).</li>
<li>Production is where data robustness is at its maximum. (To be discussed in a later post.)</li>
<li>Production is where availability is at its maximum. (To be discussed in a later post.)</li>
<li>Multiple environments can share aspects of Production and should be treated as such in those axes.</li>
</ol>
Rob Kinyonhttp://www.blogger.com/profile/07966252008096453177noreply@blogger.com0