Configuration Management and Rapid Deployment

Configuration Management and Rapid Deployment

This is the first time I can recall that I’ve decided to go ahead and include work-related stuff into my personal blog, but it fits exactly into the discussion I’ve started regarding dreams and interests. Truth be told, I love my job. I’m a Senior Systems Administrator at Kestra Financial, Inc. and it’s one of the most compelling experiences job wise I’ve had in my life. I’ve always loved all things tech and about 9 years ago, I realized that my biggest passion is the back end of web development. The servers and the background stuff that keeps everything ticking. In my current role at Kestra, my job spans all of the following: Federation Services, Single Sign On applications, Storage management (Disk-based SAN / Flash-based SAN / CIFS), VMWare management, Server and Application Monitoring and alerting, IIS Management and maintenance, and interfacing directly with the Application Engineers themselves to ensure all of the environments (DEV / QA / PROD) are running optimally and to assist them in the deployment of their code.

The largest part of my time is working with the application engineers (developers). This is a pretty great group of people who are incredibly talented at what they do and they’re consistently pushing the envelope to develop creative applications to service our customers and provide a best of breed experience. This pushing of the envelope comes with challenges on the Infrastructure side as we work hand in hand with them to keep pace and optimize the servers / SQL / storage processes to ensure everything is tuned performance wise to give a consistently excellent user experience. Like all good teams, we mostly succeed, but sometimes we experience challenges and this requires a good bit of troubleshooting and experimentation in our QA environment to fix issues and bring constant improvement to all of our processes.

One of my goals is to develop a configuration management and deployment methodology that allows for us to deploy code as soon as it’s been approved by the QA department – during the business day. This should be possible (as long as there aren’t schema changes on the database side). So, while we ask the developers to ensure schema changes are minimized and that backwards compatibility should be maintained whenever reasonable, we still can’t easily deploy mid-day without interrupting users. Due to the sheer amount of interaction and integration in our platform, it is not a simple thing to just deploy a single container or service model without it affecting a lot of other elements. While many would say that this means we need to look to redevelop everything in a containerized solution with multiple miniature services that can be deployed on the fly (a la The Phoenix Project), it’s an unreasonable expectation when you have a mountain of tightly integrated code that is working well and providing a rich user experience. It’s unreasonable to expect a business to pay for such a pie in the sky concept that provides no immediate benefit to the end users and does nothing tangible to move specific business goals forward.

But, that doesn’t mean I can’t dream and it doesn’t mean I can’t work toward utilizing systems that will assist us in streamlining our deployment processes from an infrastructure perspective without requiring major changes in development processes or philosophy. So, here’s my dream scenario of a finding a Configuration Management tool that will allow me to do all of the following:

  • Continue our current use of Micorosft’s VSRM to manage all code releases (this is tightly integrated into our workflow and incredibly successful)
  • Use some steady state configuration tool that ensures we can deploy Windows VMs with the latest approved patches and configured perfectly every single time (PowerShell DSC is the first thing I’ll be working on to try and implement this) – this would have to include domain membership
  • Leverage those definitions to build these VMs on the fly during a deployment window (ensuring all VMs during a deployment are freshly created)
  • Script Load Balancer functions to spin up new Virtual IPs internally that correspond to these newly created VMs
  • Use Flash snap-shotting to spin out new SQL Database copies of the production content attached to new SQL VMs spun up using the processes outlined above (XIO is the first thing I’ll be trying to accomplish this)
  • Have a full Virtual Lab Environment that can be logged into by the QA Team that specifically points to these new VMs for testing (using a custom DNS server) – this entire staging environment would be walled off network-wise from the primary production IPs so that we can guarantee valid testing

I’ve taken a look at Chef and I’m also looking into Ansible. I’m certain I’ll need to create some extensive customization and SSH/PowerShell scripting to arrive at my destination. I’m hoping to post my successes and challenges as I proceed.

Tomorrow will be another uber-geeky post about a PowerShell issue I’m facing with IIS object enumeration… stay tuned fellow Windows geeks.