Update: Heading to DevOps Enterprise Summit!

Thanks to contributions from friends & family, support from the conference and support from my new place of work, I’ll be able to attend the DevOps Enterprise Summit in San Francisco next week!  I really appreciate everyone helping make such a short notice trip possible!

I look forward to joining a panel discussion with some great talent to discuss the work we did on our DevOps audit defense toolkit.

Check out the conference and speakers here,  catch me on twitter if you will be there, would love to meet in person!


Good Bye Luminex, Hello HomeAway

October was a month of change for me. I’ve said good-bye to my great friends at Luminex Corporation and started a new job with HomeAway.com working in the cloud engineering team. This opportunity will afford me more time to dive deeper into large scale platforms, distributed systems and more DevOps patterns & processes but I still plan on completing much of the work I started with my blog about Enterprise DevOps Patterns and Enterprise DevOps Audit Defense.

I have a huge back-log of posts i’ve written up that I just need to cleanup and get published and I look forward to continuing the discussion of large “Enterprise” applications in a “DevOps” flow.

More to come and I hope with a lot more frequency 🙂

DevOps in the Enterprise – Oracle Financials “Flow”

Is your eBusiness suite environment full of multiple development instances all running their own mini projects of sorts with unique efforts and no alignment?  Do you catch yourself thinking that you need more environments because you have more projects that need to go on? Are you adding more projects to the mix because you’re stuffing your projects with fill time and dependencies and seeing that as idle time to do more?

Lets talk about FLOW.

In my experience, the typical Oracle Financials flow is chaos, managed chaos. Its heavily based upon very inflexible and highly managed plans and projects. These plans and projects often have layers of people management, process management, requirements, expectations, deadlines, goals and ambitions that are based upon assumptions and waste – processes that add no value to our end goal of delivering or supporting the financials applications stack. Come to think of it, I’ve been in this industry for 10+ years and its always been a struggle against unrecognized waste! We build our procedures on waste without really knowing it.. almost habitually!

15 Years of eBusiness suite deployments and the bulk of my work has been waste.

Let that settle in. (I feel anxious just thinking about it!)

I’m not here to point fingers, lay blame or say it was all wrong, I just need this to sink in before you can move to understanding flow, understanding your own waste and understand what it means to operate with “DevOps” value or “lean” operations. We can’t possibly work towards DevOps patterns in Enterprise systems until you realize what your own “Flow” of work is and what waste really is.

So about this waste, lets get into more details so you understand why its called “waste”.


Warning: This is mostly thoughts flowing out of my head.. maybe not the most cohesive.. most grammatically or syntactically correct, but just thoughts & experiences & ideas. I hope these get you thinking about your “waste” more than they are meant to document all waste.

Learned helplessness – Procedures are in place to trigger punishment, there is no reward for stopping production to do things right, there is no value consensus in the project or collaboration & reflection on getting things done.  Someone else has planned things out according to someone’s idea of a deliverable that someone else forecast and the core value sold to the business it that the project will be done by a specified time and budget. Where is the value in this?

Overproduction – The tendency of projects to keep people busy by losing focus on what the customer has requirements for.  In general over production is producing goods for which there are no orders, but in an Enterprise IT shop, this could translate to having too many environments operating that doesn’t fit your main flow essentially overproducing in order to keep the appearance of utilization.  This also contains Excess Inventory, adding new expensive instances that you then have to have capital appropriations for because the storage, CPU and memory is expensive.  You start passing those costs on to your customers or absorbing them as cost of doing the project / business. You start building Unnecessary movement with too many flows of work because now you have to maintain, patch and clone these environments on disparate schedules thus, creating more slippage in projects, more waste time in cycles.  All of which adds up to Big Buffers,  Defects, and Unused Employee Creativity.  I’ll let those speak for themselves 🙂

With poor flow you build insurmountable technical debt;  we start choosing the “least” concerning option, the “least” risky option, the “safest” route – and generally, not even based upon a consensus but someone actually signing off on the risk as a personal responsibility or PUSHING responsibility up/down the “People who are paid enough” chain. I quote all these “safe” and “least” routes because they’re often done on assumptions and learned behaviors more than they are chosen on  facts and the discussions thereof.

How then do you then start to approach your eBusiness suite flow?

The conceptual practices that people generally speak of in DevOps circles stem from the works of Kevin Behr, Gene Kim & George Spafford (and many.. many others!) through the metaphor of the Three ways they hint of in the Phoenix Project.  I say metaphors because conceptualizing these is fairly simple, but applying them is a different story and they don’t really “Speak” to the typical “Enterprise” shop. (if you ask me..) – BTW, let me be clear, this is mostly paraphrasing the Three ways!

  • The First Way – Create a fast flow of work that moves from development into IT.
  • The Second Way – Amplify feedback loops, so we can fix quality at the source and avoid rework.
  • The Third Way – Create a culture that simultaneously fosters experimentation, learning from failures, and understanding that repetition and practice are the prerequisites to mastery.

I believe the approach to thoughtfully applying and understanding these three ways successfully is to learn from Operations Management, Lean processes and to borrow heavily from lean JIT manufacturing and the Toyota Production System (TPS).

How does Lean workflow improve our flow?

Here are seven traits of lean operations, made possible by well studied and observed lean practitioners..

  1. Builds in Quality
  2. Flexibility
  3. Higher Productivity
  4. Frees up floor space
  5. Improves Safety
  6. Improves Morale
  7. Reduces cost of inventory

Did you read those and ponder on them a bit? Did you think about them in your own environment? I wanted to get them out there to get you thinking about them so the next part makes more sense.. It can be confusing to talk principles, culture and concepts when you don’t feel any bearing to them yourself.

Developing lean / DevOps values & principles.

Principle 1.  Long term philosophy even at the expense of short term goals. If your goal is to generate value to your customers then you can’t achieve that by taking short cuts to appease near term goals. I had a big paragraph describing concepts here, but deleted it because this will make more sense as you get through the rest of the principles. Simply put, your financial application investment is a HUGE long term investment, don’t put it at risk for near term gain! Keep that in mind as you build your values & culture!

Principle 2. Create continuous process flow to bring problems to the surface.  Those project plans, are they full of hand-offs, waits & idle time? (Waste?) Make flow evident in your organization so the project itself is improved iteratively to optimize its own value.  In a continuous flow, you don’t promote your problems to the next “Cell” for them to work on, you surface them to fix them!

Principle 3. Use Pull systems to avoid overproduction Provide your customers with what they want, when they want it and in the amount they want.  Minimize your WIP (Work in Progress/Process) so you can be responsive to day-by-day shifts in demand rather than relying on schedules.  Too much WIP and you start losing your flow and spiral out of control.

Principle 4. level out the workload Eliminate waste, eliminate overburden.  Remove the painful “start & stop” of implementations – strive to work through to completion smaller, more iterative processes.  The more you start and stop work because you have so many inter-dependencies, the more you’re breaking up the workload, the more you’re adding waste & re-work. Projects & WIP generally get out of hand if you don’t value the leveling out of workloads!

Principle 5. Build a culture of stopping to fix problems – get quality right the first time. Don’t build up technical debt! The quality of service we provide is our value proposition to our customers and the business.  Use quality assurance methods available to help get quality right.  This is one of the areas where I’ll mention tools as you can build a process around tools to remove complexity, remove dependencies (decouple!), remove guess work and improve the flow.  Continuous integration environments, Unit testing, Desired State Configs. These are all concepts that embrace the culture of getting quality right the first time and embracing a culture of stopping to fix problems.  “Prod” is no longer just the production instance of your financials application, it is the entire flow!

Principle 6. Standardize tasks Build foundations for continuous improvement and employee empowerment.  Use stable, repeatable methods everywhere to maintain the predictability, regular timing and output of your processes.  Those “controls” you have in place that you thought did this, actually get in the way.  Rethink those!  The tasks are more important than the controls because the tasks empower people to improve them, collaborate on them, test them and analyze them. Obviously I’m not saying to throw out controls, companies have them in place for Audit, legislative, legal and financial accountability reasons, but use them wisely.  Look at your values and see how your “controls” are part of the value chain!

Principle 7. Use visual controls so no problems are hidden. Many people recommend Kanban style “pull” visuals so you can visually represent the flow, but you can expand these into providing visual data across your flows. We often call them “KPI’s”, we call them “Dashboards”,  We call them “Portals”, They’re representations of flow!  Don’t hide problems by keeping people unaware from them, shoving them off to other groups or pushing the blame around, keep them front and center so you know what you have to apply your efforts on!

Principle 8. Use reliable and thoroughly tested technology that serve your people and processes. Isn’t this the expectation we have in buying and paying for Oracle eBusiness Suite? We often imply reliability and thoroughness by forcing long lead times, forcing compliance, forcing validation, forcing checkpoints and buffering time in our projects. We’re doing it wrong!  Build up reliable OS platforms, so you can do OS patching without your Business Analyst having to approve something they have no bearing on. (You know, those change requests that get promoted to 5 different people!).  Build up reliable infrastructure so you can have alerting, collect metrics, analyze data and visualize your flow.  You want to have the trust to do things right as well as the trust to be able to fix things that break. If you’re not innovating, its time to review your process!

Principle 9 – Grow leaders who thoroughly understand the work, live the philosophy and teach it to others. I personally believe that growing an agile / lean / DevOps culture is much more powerful than buying one. The best way to grow people is to empower them and the best way to empower them is to motivate them by making them part of the process, showing them how they contribute to the process and reflecting their value.

Principle 10. Develop exceptional people and teams who follow your company’s philosophy What is your goal? Are you working with these values? Are you looking to “iterate to innovate”?   Do you have a policy of “If you’re not innovating, its time to review your process”? Are you understanding these values? Are you developing them in your teams & company culture?

Principle 11. Respect your extended network of partners and suppliers by challenging them and helping them improve Work with your partners, help them understand your requirements and your flow. Make sure Oracle knows your patterns, make sure your OS vendor knows your strategy, make sure you’re choosing people who support YOUR business.  Don’t let “best practices” or other buzz phrases get in the way. Your best practices are the practices you foster, you measure, you analyze, you validate, you test and you improve. Validate what your partners are doing through the same process you validate your own organization!

Principle 12. Go and see for your self to thoroughly understand the situation! Don’t leave people hanging! Simple as that. If there is a problem, go and discuss it. If something breaks, you should go and stop the flow, see the problem for yourself, get the right people involved to fix it and take ownership.

Principle 13. Make decisions slowly by consensuses, thoroughly considering all options; implement decisions rapidly. If you ask me, this is the best principle ever because its a complete reversal of experience I’ve had where there is very little discussion on doing this well.  Reverse your patterns by striving for excellence in design & implementation, increase your flow so the implementation of decisions is rapid! Often times people assume a project manager has done all of this in advance and the entire project is implementation, that seems entirely backwards and counter productive to me but a trap I see all to often! Make sure stakeholders are holding their stake! 😀

Principal 14. Become a learning organization through relentless reflection and continuous improvement This is it folks. The culture you develop should be based on relentless reflection and continuous improvement. This means experimentation, A/B Analysis, trial and error. This means blameless postmortems. This means collaboration. This means cross domain participation.  This is “DevOps” folks!

What do these principals do for your flow?

These principals help you develop new values and new learned behaviors.  They really have clarity when you tie them back to those seven traits I bullet pointed earlier.

Builds in Quality – Your principals create a work ethic based on quality products to your customer.  Smaller batch sizes, lower risk, automated testing, easier to verify. You’re actually increasing your throughput even through a single flow!

Flexibility – Reduced lead times, faster deployments, faster fixes.  You’re now empowering people to be creative, flexible because the process is creative.

Higher Productivity – Single flow means that the work is getting done and done right. You have less over production, less environment/instances, less coordination, less process management, more focus on agility and very little non value add work.

Frees up floor space – While we don’t typically have “floor space” in the manufacturing sense, we do have “floor” space in the technology.  Single flow means less VM sprawl, server sprawl, less capital tied up in idle hardware. We’re no local focusing on local optimums of optimizing infrastructure use and VM use by using MORE of it to make it appear utilized.

Improves Safety – People usually don’t get hurt physically in eBusiness suite deployments and upgrades but safety can be distilled in many ways. Safety can be seen as stability, Safety can be seen as respect,  partnerships.  Safety can be the protocol by which you solve the insatiable appetite of managers to strive for consistency.

Improves Morale – People are focused on single flows, people feel empowered to make a difference, people are challenged to do better, people feel the reward of a job well done.  People appreciate adding value and creating value!

Reduces cost of inventory – Just like freeing up floor space, you can reduce sprawl, hardware, processes and controls, reducing the cost of inventory, reducing the cost of WIP and reducing the cost of your flow. You don’t need to have environments on standby, don’t need to have excess capacity just in case.  You can use these excess capacities for other purposes that add value!  Use them to experiment, test, verify. Use them for proof of concepts.  The idle time you measure should be idle time you allocate to improvement. If there is one thing you want to stock up on, its an excess of pride, excess of trust, excess of ownership. The ROI on that is through the roof!

The Three ways

Once you start understanding the metaphors, values and principals of lean operations you can start building your own metaphor for the “Three ways”.  Understand your flow,  make feedback part of your process and wrap it all up with culture!

My Story

If after reading my opening statements you feel I’m being snarky in a way, it’s probably because I am.  Not only have I ran into exceedingly costly and ever failing efforts and projects but as a person in this failure loop, its hurt me personally.  Its built up bad behaviors, bad experiences, bad attitudes and so much more!  Poor flow projects turn collaboration inside out, people don’t want to talk, they don’t want to work together, blame takes over, people fight for the wrong types of responsibility and the demands on your technical staff to pull of thankless miracle after miracle become detrimental to not only their attitudes, but their behaviors and even their health. We start striving for the wrong goals, we start striving for deadlines and dates, we hide risk behind the successes of deadlines, we build technical debt not creating a process that allows us to improve our deficiencies and do things “The right way” and when we say the “right way” – that doesn’t mean “My way as I see fit” but the “Collaborative way with continuous improvement in mind so that we know we’re all working towards creating and adding value”.

The end result of determining your flow isn’t just a beautiful “Self-healing”, “self-maturing”,”reflective” project and infrastructure – it’s a cultural shift for your organization that is downright empowering!

I don’t have all the answers, but what I wanted to do was offer an honest critique of where we are failing and ways that we can fix it, building upon the successful values of others. Isn’t that really where our flow stems from? I didn’t invent this, don’t claim to be a master of it, and i’m always having to practice it and think about it.  I believe in sharing it and writing about it since that is part of my philosophy and how I learn as well!

As always, comments/feedback is HIGHLY appreciated! I know this was a long one so thanks for staying with it!


The Phoenix Project

The Toyota Way

Taiichi Ohno

Too many great resources to list, but that is of course, a great start!

DevOps in the Enterprise – Oracle Financials

This post is more of an introduction to a journey of sorts to get thinking about DevOps in an Oracle eBusiness (Financials/CRM) type infrastructure and I plan on developing these posts over time with more details; incorporating feedback on the goal, designs, components and concerns and just as importantly, I hope to collaborate on these ideas to incorporate non Oracle tools.  It’s extremely important (at least to me!) to leverage the existing tools whenever and however possible so that Oracle doesn’t remain a perpetual silo no one else wants to touch!

Back story about me: I started out supporting Oracle eBusiness Suite in the late 90s, running 10.7 NCA, 11.0.3, every release of 11.5.x and currently supporting a 12.1 environment while trying to plan ahead to R12.2.  I’ve supported Oracle eBusiness on AIX, HPUX, Linux, Windows and Solaris environments over the years – through custom processes, oracle best practices to integrating with ITIL tools (Mercury Interactive at the time). I even spent a while working for Oracle themselves supporting implementations for higher education & healthcare verticals. I’ve seen it, I’ve done it, I’ve broken it, I’ve fixed it, and I have a lot of stories to share about it 🙂

Lets get the wheels spinning!


Achieve a process of continuous integration (or more aptly “continuous validation”) within the eBusiness suite platform to achieve DevOps value objectives.

Concerns: (a few of many..)

Just how do you do DevOps with eBusiness suite (and similar COTS systems)?

  • How do you unit test?
  • How do you load test?
  • Do you push code to production?
  • How do you provide developers access?
  • What does the process look and feel like?
  • What about Business Analysts?
  • Audits?
  • Performance?
  • Availability?
  • feedback loops!


The components I use to build my “Enterprise Devops Pattern” for eBusiness Suite.

  • Oracle eBusiness Suite
  • Oracle RDBMS
  • Oracle iAS/Weblogic
  • Oracle Application Testing Suite
  • Oracle Cloud Control
  • Oracle Linux
  • Puppet, PuppetDB & Foreman
  • VMware


What are the common processes we can leverage, optimize and built trust upon to greatly reduce  systems complexity?

  • Patching
  • Project work
  • Customizations
  • Cloning
  • Automation
  • Interfaces
  • Integrations
  • ….

Putting it together

How do we build a system that embraces trust, is reliable and performant? How do we test and prove our value stream?

  • Measuring
  • Experimenting
  • Audit defense
  • Monitoring.. testing..
  • Feedback loops
  • Reporting..

As you can see, just by starting to list out bullet points, the complexity of the eBusiness suite starts to rear its ugly head.  One quickly realizes that while the eBusiness suite is comprised of hundreds of connected “Applications” (AR, AP, GL, CRM, and FA so on and so forth) it really is its own “system” that Oracle has built and bolted on to over the years.  By thinking of it in terms of systems, we can start to see how we can apply DevOps values into the system to solve issues that plague the system by and large – Complexity, Mean times to do anything,  long project cycles, operational silos, skill silos, deep reliance on vendor support and so much more.

I hope to see people join me on this journey, share their ideas & experiences and challenge my assumptions.  The end goal is absolutely a “cookbook” of ideas to bridge the Oracle Financials “monstrosity” (for lack of better word) into a DevOps value stream.

In opening the proverbial flood gates, I want to speak to you – the people implementing, supporting, planning, patching eBusiness suite and see what you are doing. I want to hear from Vendors working on solutions, from groups developing custom solutions to open source projects that can be used to help provide the tools for change and help consolidate some of this data to report back and share.

So please, contact me through my Blog,  LinkedIn, Twitter, Google Plus, e-mail or phone.  Let’s talk! BTW, This work will be shared here and on the DevOps Enterprise Patterns group I’ve volunteered to assist Gene Kim with as well. Please join us there!


DevOps Enterprise Patterns

Devops Audit Defense Toolkit

Contact Me!

Tagged , , , , , ,

Phoenix Project, the alternate Brent universe

I’ve read, and re-read the Phoenix Project and absolutely love it, learning something new each time and picking up new ideas to keep my brainstorming going, but one thought that has plagued me through to the end every time, is the “guardian angel” to management and the IT person staying the IT person.

How different would the story be if Brent was the benefactor of Erik’s advice?

In my years of experience, I often find the Brents of the industry aren’t wholly self-made but often easily fall into traps that we’re not taught to get out of.  It’s easy for management to rely on us, keeping us as the choke point of many critical projects without realizing it or facing it and we burn out, get angry, go BOFH.  We’re often managed into systems, managed into context and managed into a story and we often try and logic reason ourselves out of this with no end in sight. What if there was a different story?

How different would the Phoenix Project be if Brent was able to learn some leadership, learn some collaboration and develop his persuasion skills to help motivate and transform the organization and himself from within?  Would this story better help others facing similar situations? Would it help develop more leadership roles from the Brent types over the world? Would it help many of us Brent types “see the Forrest from the trees” more clearly?

The book is very good, don’t get me wrong, but I just have a gut feeling every time I read it that I wish Brent could have been developed, fostered and nurtured, not as a follower of management making changes but as the catalyst for management making changes and as a growth path for Brent himself, because I feel that better parallels my experience.  Obviously in the story Brent does benefit from the changes and we can all learn from the story, but in the end, it feels like a management win and it left me wondering how I could do something as if I’d just meet Erik.

I parallel this thought to many of the discussions I see on twitter and my own sorted experiences to help make changes at an organization.  In some tweets, I see people struggling to do what I’d call applied DevOps, replying to others to quit a job because they can’t convince management/peers of the logic of DevOps, people describing you must to this, do that, and run this or be that, and lots of frustration or indifference towards DevOps values by everyone else but them.  I’ve just been insanely curious about trying to parallel the story of the Phoenix Project to myself, my own short comings, my own experiences and those I observe (such as the mentioned tweets and indifference) and how to approach this. It was an awaking of sorts after reading MANY stories and books on leadership and growth that not only are a lot of “Brents” trying to make these changes in their organizations  but ironically they do so in a very “Brent” way  and here is how I’d answer that story.

For me, the new story steams from a few books I’ve read (mentioned below) and an excellent class available from the Teaching Company – Transformational Leadership and how I thought this could make a new Story for Brent if Brent meet Erik and hand the skills to transform.

How to sell DevOps at your org if Brent was the benefactor of Erik’s advice.
The hard way – The Brent way of convincing organizations of devops.

  • Strongly stated position – speak to things as facts – applied “DevOps” – get focused on mimicking successful organizations more than anything else. We’re good at having strong views and strongly stated positions.
  • Assertive Supporting Arguments – “This is the way, we must do this or else…” – Our logic is what we feel makes us assertive and how we speak objectively to something as if that will win people over.
  • Closing the deal – resisting compromise, using only logic/data and extreme passion to make out point.  Combining everything above, we often trap ourselves into an all or none type system/belief.

Would the book have made much more sense to many of us if Brent was developed to learn better collaboration? Be a better leader?  Would you be able to recognize the hard way above as the hard way? After all, most leaders are taught that the better way of persuasion (and leadership) is, collaborative persuasion. Collaborative persuasion helps us achieve the very goals we convey and yet, we don’t speak or really practice to this much, if any at all. While I felt the leadership story in Phoenix Project was very mature and bold, I’d just like to see more parallels of how Brent could have been fostered to lead the transition too and I believe many of us do it the hard way, or Brent way.

The Collaborative Way – Collaborative persuasion – using the very concept we’re trying to convey as a solution as the solution.

  • Establish Credibility – Don’t over estimate your own self!  (Big thing for us Brent types!) – Conduct experiments & develop pilots to share what we’re trying to sell. Build credibility as DevOps applies to YOUR organization. Begin Somewhere!
    Build TRUST! Would Brent be able to experiment  (Vs jump to applying) with the advice of Erik? Could we translate that to Management/Leadership? Could we do so without falling back to our logical bias and do it collaborative for small wins and tactical leadership positions? Automation may be Brent’s goal idealistically (it is after all, following our logical trend) but could we step back and be happy showing a Kanban process and the wins thereof or simply starting with communication? Could we lead the small wins to a strong catalyst for cultural changes? Sometimes culture defies logic and that’s tough for us Brent types to understand or often don’t comprehend at all.
  • Framing for common ground – It’s important to collaborate, it’s important to explain things and frame them in such a way to express ourselves better.  Instead of saying “this must be the way, look, everyone else is succeeding at it” frame the topic so that the focus is drawn to the attention of where you desire to be. Lead people to positive results.  Framing is how you can steer the DevOps story to be better applicable and connected to your organization.
  • Connect emotionally – Is this possible for Brents of the world? Can we adopt our strategy to a broad audience in a collaborative way that connects with management and works through these gatekeepers? Can you focus on the divergent thinking to make it happen? Are we presenting REAL solutions or are we presenting problems? I know it’s terribly easy for us Brents to focus on problems and solutions as if their logically black and white. Can you connect emotionally to your peers and speak to them as if you put on their shoes?
  • Evidence – how can we build stories, examples and metaphors?  How can we make this evidence connect emotionally? Frame it for common ground? Build trust and establish credibility?  Evidence absolutely comes natural for many of us “logical Brents” but the hard part is connecting it to the collaborative persuasion process and realizing that this collaborative persuasion it was grows us from a Brent who may have learned something and wants to achieve it into a Brent who becomes a leader to enact it.

Brent types can transform to be leaders. As leaders, we don’t need formality to express our desires, we need strategy, history, allies, solutions and we need to work with and through the gatekeepers to tell the DevOps story. I think once we learn this, we also learn that perhaps there isn’t an absolute way for everything, that collaboration can lead to new ideas, new thoughts and new models of getting things done. We can not only grow to become leaders, but grow in our understanding and interpretations of the values of DevOps and how those values benefit the growth of your organizations and just as much, if not more importantly the growth of ourselves.

So while the Phoenix Project didn’t focus on the Brent solution as many of us are or have been over the years, I hope this post invigorates you, Brent or not, to think differently, think like a leader and realize that what you are trying to achieve doesn’t have to be an effort of management directly, but an effort that when done right can come directly from you. I implore you twitters not to tell people to quit their job, give up, find something else to do or beg and chew your way up or simply try and mimic or apply logic as if that’s the only way –  as DevOps values aren’t something we believe in and force upon others, they’re  stories that connect emotionally and speak elegantly to us to learn, practice, share and collaborate on. We need to do better selling that story to management & leaders or better develop ourselves to become those management & leaders.

Thanks for your time.  I realize not every Brent wants to be a leader and may be quite content with an organization that delivers DevOps values for him/her, but at the same time, I think there are a lot of us who do want more and I hope this helps you think of how to achieve more and how to lead that effort in a very DevOps way.

Books & Resources I’ve drawn much of this from and wouldn’t want to do without 🙂

PS, I’m terribly new to writing, feel free to leave any feedback about my style, grammar, spelling or any methods to improve.  I’m mostly sharing my stories and thoughts but I’m always open to improve.  Thanks!

As always, feedback appreciated.  Feel free to find me on twitter as well.

Tagged , , , , ,

Devops – It’s about critical thinking & the evolutionary “WHY” of Silos.

I believe one of the best things to ever come out of DevOps movement that no one seems to be describing is essentially an explosion of critical thinking and reasoning skills, The maturing of IT if you will. Some people prescribe different views or distill it into different methods such as C.A.M.S (Culture, Automation, Measure, Share), some people say it’s the destruction of silos, some say it’s just a cool cultural shift and an awakening of sorts. I’d like to take you on a short journey and focus on two things for this topic – the concept of critical thinking and silos.

I found this blog post by Ben Kepes VERY interesting, where he discusses the odd effect of DevOps teams becoming silo teams in and of themselves and how that could appear to be negative or wrong. While I think it is an interesting point to bring up, I think its made without looking at the “WHY” it happens and more of a conceptual view of “WHY” it shouldn’t happen. So lets put on our critical thinking caps and “think here for a minute” about the whys of silos 🙂

From my perspective, the biggest reasons we see silos aren’t vanishing (and quite possibly are being replaced by new silos) is that the silos are the result of real differences of domain. By this I mean, the domain knowledge of your operations, development, analysts, QA, DBA’s, sysadmins, engineers or any group are essentially domains of expertise. When it comes to cultural changes, the best change we can do isn’t to squish these silos down and pretend they don’t exist, but to align them better and give them strategic significance. If a silo really is a moving target, lets align them so they’re all moving in the same direction!

It seems to me, we’re more focused on prescribing ideas rather than deriving solutions. We see improvements because we aligned developers and ops into a team and we think “wow, getting rid of those silos worked” but maybe the silos weren’t the problem to begin with, maybe it was a failure of alignment, workflow, poor cooperation and poor business decisions. By aligning people and connecting them to not only costs of doing business but the revenue of doing business the entire “WIP” chain can see the impact of their work as a whole, as a unit, and as a team (Something to Measure!). Much like the comparisons of WIP from an IT shop to WIP of a manufacturing floor as described in Gene Kim’s excellent book “The Phoenix Project” we have to remind ourselves that even though the mfr floor looks like its running on its own and well oiled and doing small and simple tasks – those small and simple tasks are not just the experience of repetition, but silos of domain knowledge and experience that you don’t NEED to squish and quite frankly, do better when left to be developed and fostered.

I had my first real world experience of this in the early 2000s when I worked at a manufacturing plant. I was a technical lead for implementing an upgraded ERP/MRP system and one day, the manufacturing union went on strike and we full-time employees were led out into the floor to run the plant to finish delivery of critical items. I never thought much about it other than how interesting of an experience it was to go through a labor strike and realize how specialized some of the manufacturing jobs are and how they can appear simple but be rather complex in the end.. I realized I couldn’t assemble tubes like they did, my ceramic painting skills were terrible and the output/workflow of my experience was disastrous at best. Again, after reading “The Phoenix Project” I had a few epiphanies and differing views of the “flattening of silos” that I just couldn’t figure out how to express for a while and it’s just barely coming out here. It dawned on me that the people working this assembly line have a domain knowledge of their own. The people fixing the assembly line have a domain knowledge of their own. When the line breaks, no one expects the line workers to come in during their off hours and fix it because they need to be bothered to learn something about what broke, The line maintenance and facilities crews do their thing. Both domain experts of workers and repairers also have their pipeline, their WIP, their process improvement and they coordinate and match these skills, not distill them and flatten them. So why have such an expectation in IT?

So long story short, I believe we need to do better at understanding the “silos” we complain about. Again, the existence of silos in and of themselves doesn’t mean that they’re all silos of control, silos of despair & silos of differences and if they are these described silos, well, there lies your problem. You can fix the flows, you can align the groups and you can integrate teams and get them working together without artificially ignoring domain knowledge the same way you can prevent domain knowledge creating artificial silos.

The “Why” of Silos isn’t necessarily a bad thing by itself. Have you ever put your critical thinking cap on and thought about YOUR personal “silos”? Are you able to be “culturally adept” but still hold on to personal prejudices against all things commercial or all things not invented here? Do you let your hatred of specific vendors / ideas / practices tunnel vision your beliefs? Are you using silos as a big stick to ignore domain knowledge and experience? Are we really being critical thinkers?

Think critically about your organization. Understand how important your role in IT is to the business and its bottom line. You’re all on the same team, and there may be many silos, but as long as all the silos are on the same “farm”, they really shouldn’t be a problem. In fact, you may soon realize that when your silos are there exactly when and where you need them and this may be more strategic and important than tearing them down and having everything scattered across the field.. metaphorically speaking 🙂

We evolve into silos of domain expertise as a natural progression of our careers, aspirations and experiences. These silos can be very powerful if used wisely and correctly recognized and aligned by your organization. Perhaps DevOps is its own silo, especially if you imply DevOps with specific ideas, tools, solutions and services and that’s why I think silos are less important than we realize and the real concept we should be focusing on is “critical thinking” because it’s that skill that really matters. It’s critical thinking that will give you the gut feeling of what works and what doesn’t and where you need to align yourself and your peers within your organization.

Hope to write down more thoughts on this later.. what do you think?

Tagged , , , , ,

Puppet & Oracle Enterprise Linux – When Modules Fail

Puppet & Oracle Linux – Fixing those failing manifests!

  • OS: Oracle Enterprise Linux 6.3 / 6.4
  • Puppet: Puppet 3.1.1
  • Facter 1.6 or Higher

Puppet runs great on OEL 6, there really isn’t any pain to implementing it until you get to installing some modules – a lot of them simply don’t know what OEL or OracleLinux is. The fix however, is pretty straightforward and easy if you run current versions of facter and puppet.

First thing you want to do, is validate that you have a current version of facter that has osfamily fact.

[bmiller@puppet ~]$ facter | grep osfamily
osfamily => RedHat
[bmiller@puppet ~]$

If you have the osfamily support, you will want to edit the manifest that is failing. We will use The Foreman for an example.  The Foreman 1.1 has multiple setup files that implement the following code: (you will often find these in install.pp manifests or if you run “puppet agent -t -noop” it should show the failing manifest during a puppet run)

class apache::install {
  case $::operatingsystem {
    redhat,centos,fedora,Scientific: {
      $http_package = 'httpd'
    Debian,Ubuntu: {
      $http_package = 'apache2'
    default: {
      fail("${::hostname}: This module does not support operatingsystem ${::operatingsystem}")

  package { $http_package:
    ensure => installed,
    alias  => 'httpd'
This code fails miserably that “OracleLinux  isn’t a supported Operating System. The fix? Instead of looking for each and every type of RedHat Distro and adding more OS Variants to check for (some have a LONG list), use facter to get the $osfamily.  It is as simple as this code snippet:
class apache::install {
  case $::osfamily {
    RedHat: {
      $http_package = 'httpd'
    Debian: {
      $http_package = 'apache2'
    default: {
      fail("${::hostname}: This module does not support operatingsystem ${::osfamily}")

  package { $http_package:
    ensure => installed,
    alias  => 'httpd'

I’ve submitted the fixes to the foreman, so future versions can install without a hitch and I hope other developers start implementing the “osfamily” check in lieu of “operatingsystem” checks. The osfamily check makes cleaner code and happier admins 🙂

Hope this helps other people running into issues on OEL / Oracle Linux / OracleLinux (Or whatever they want to call it!)

%d bloggers like this: