[{"content":"What do I find most surprising about too many IT shops in 2024?\nIt\u0026rsquo;s the still prevalent attitude of \u0026ldquo;treat us like a black box\u0026rdquo; to their business while also entirely out of touch with the end customer.\nI get it. Back in the day, which is further back than I am going to disclose, I also believed IT should be a black box. Just give us your requirements, and we\u0026rsquo;ll make it happen. Over the years, I understood that \u0026ldquo;we are all the business.\u0026rdquo; You need to be able to trace how you, the engineer, the architect, or the manager directly impact the customer experience to know if you are building and working on the right things.\nMy friend Gary Kay introduced me to the Steel Thread concept, which traces the most essential execution paths through a software architecture. But we don\u0026rsquo;t stop there. Extend it beyond the architecture to where the software creates value for your customer. The customer end of the steel thread gives you the starting place for your KPIs.\nIf you haven\u0026rsquo;t heard of the Steel Thread concept, it\u0026rsquo;s okay. Wikipedia deleted the page in 2013 because it is \u0026ldquo;a colloquial term and not notable within Software Engineering, the page\u0026rsquo;s only category.\u0026rdquo; And this is why you can\u0026rsquo;t unquestioningly trust Wikipedia. Check out this white paper for those diving deeper into the Steel Thread concept.\nSo, to those IT shops focused inwardly on your challenges and the list of requirements given to you, it\u0026rsquo;s time to spend some time with your business partners and go see how your tech is (or isn\u0026rsquo;t!) used in the real world.\n","permalink":"https://opsrob.com/post/black-box/","summary":"\u003cp\u003eWhat do I find most surprising about too many IT shops in 2024?\u003c/p\u003e\n\u003cp\u003eIt\u0026rsquo;s the still prevalent attitude of \u0026ldquo;treat us like a black box\u0026rdquo; to their business while also entirely out of touch with the end customer.\u003c/p\u003e\n\u003cp\u003eI get it. Back in the day, which is further back than I am going to disclose, I also believed IT should be a black box. Just give us your requirements, and we\u0026rsquo;ll make it happen. Over the years, I understood that \u0026ldquo;we are all the business.\u0026rdquo; You need to be able to trace how you, the engineer, the architect, or the manager directly impact the customer experience to know if you are building and working on the right things.\u003c/p\u003e","title":"What do I find most surprising about too many IT shops in 2024?"},{"content":"\u0026ldquo;My door is always open.\u0026rdquo; But is that really enough?\nYou\u0026rsquo;ve heard this one before and maybe have used it yourself. Even assuming sincerity on the part of the leader with the open door, it\u0026rsquo;s still problematic.\nI was reminded of this when chatting with Chris Blackburn, and the topic of Gemba walks came up. In Japanese (and Lean thinking), Gemba is the place of actual work. It\u0026rsquo;s where the value creation happens.\nOne of the most critical things leaders should do is go on Gemba walks. Go hang out with the people who are creating value in your organization. Better yet, work alongside them. Seeing a leader in their space who is genuinely curious will build trust over time and encourage value creators to start sharing concerns and ideas and asking questions. However, Gemba walks are not the time for the leader to bring solutions; you are listening and asking questions. Be curious!\nAnd now you see why having an open door doesn\u0026rsquo;t work. Making people leave where they create value for the location of someone more senior, regardless of how approachable they are, puts the responsibility on the value creator—the opposite of servant leadership. Social power dynamics puts the person walking through the door (physical or virtual) on their back foot, not a great way to get the best ideas or top concerns.\nBut what about remote work? During the pandemic, I realized I was missing the touchpoints that happened in person, which allowed me to maintain a deep context with the actual work. The solution? Lots of Zoom time. I use short recurring meetings every other month (will vary depending on the size of the org) with everyone in my tree, not just skip level. Getting this invite will be nerve-wracking for the team members, so it is critical to be clear about your intention in the invite.\nMany leaders mistakenly think they must maintain a sense of all-knowing in front of their organization. If this sounds like you or you dread getting asked a question you don\u0026rsquo;t have the answer to, it\u0026rsquo;s ok! Being authentically curious and honest when you don\u0026rsquo;t know something goes a long way to building trust with your team. Gemba walks (or Zoom calls) are a great way to practice.\nGo Gemba!\n","permalink":"https://opsrob.com/post/open.door/","summary":"\u003cp\u003e\u0026ldquo;My door is always open.\u0026rdquo; But is that really enough?\u003c/p\u003e\n\u003cp\u003eYou\u0026rsquo;ve heard this one before and maybe have used it yourself. Even assuming sincerity on the part of the leader with the open door, it\u0026rsquo;s still problematic.\u003c/p\u003e\n\u003cp\u003eI was reminded of this when chatting with Chris Blackburn, and the topic of Gemba walks came up. In Japanese (and Lean thinking), Gemba is the place of actual work. It\u0026rsquo;s where the value creation happens.\u003c/p\u003e","title":"My Door is Always Open"},{"content":" It doesn\u0026rsquo;t have to be the individual genius that drives innovation\u0026ndash;community and location can do more to foster creativity and innovation.\nWhile kicking off the IT Revolution\u0026rsquo;s Enterprise Technology Leadership Summit last month, Gene Kim introduced me to scenius, a word suggested by Brian Eno in 1996. Brian said, \u0026ldquo;Scenius stands for the intelligence and the intuition of a whole cultural scene. It is the communal form of the concept of the genius.\u0026rdquo;\nIt\u0026rsquo;s often something you don\u0026rsquo;t realize you have until it is gone. I encourage you to dig into the inspiration behind scenius.\nA few other takeaways:\nLauren Woods delivered the opening keynote. One of her points was, \u0026ldquo;Changing altitude changes your perspective and helps you rise above the turbulence.\u0026rdquo; Yes! The ability to change perspective is a deeply undervalued quality experienced and intensely curious senior leaders can bring to an organization. Too many leaders (and organizations) believe that navigating into and through turbulence is why they are there. Nope- the best leaders know to go above and beyond.\nI found myself nodding again when Michael Nygard flashed \u0026ldquo;Data - The Land that DevOps Forgot\u0026rdquo; on the screen. The challenges of escaping 24-hour-long batch cycles are sociotechnical, even when it is about data. Remember Nygard\u0026rsquo;s Change Management Basics:\n\u0026ldquo;Once you are tired of saying it, people are just starting to listen.\u0026rdquo; \u0026ldquo;Cost attribution feedback loop needed to help the same person understand cost+value.\u0026rdquo; \u0026ldquo;Don\u0026rsquo;t create a legacy world and a separate promised land\u0026rdquo; (I\u0026rsquo;m looking at you, Mode 1 and Mode 2) \u0026ldquo;Change happens in the whitespace of the organization.\u0026rdquo; Only expect change if you are creating sufficient slack in the system. Next up\u0026hellip;don\u0026rsquo;t let your finance model drive bad behavior (and thus outcomes). It should be a tattoo on my back\u0026ndash;I am always down for geeking out on procurement and its impact on change. Elizabeth Ayer takes this to another level. 18F\u0026rsquo;s updated Derisking Guide is coming this month, and I encourage you to check it out. Until then, she asks us to keep in mind:\n\u0026ldquo;Many times, needs are more strategic than they (the buyers) think. A bias towards demanding certainty worsens this, resulting in contracts that stifle the ability to adapt over time.\u0026rdquo; \u0026ldquo;Buyers must contract so value is being delivered to customers, not just to the buyer\u0026hellip;which drives structural problems.\u0026rdquo; If I\u0026rsquo;m buying consulting services to build and supply a product, the contract needs to include the point of view of the end customers of that product, not just the buyer who is having the product built. Steve Smith gave an entertaining talk going into three ways you are screwing up Platform Engineering:\nPower Tools. Implementing core capabilities with heavy-weight tools (K8s, Istio, Kafka). Config and Integration complexity creep in Technology Anarchy. The platform enables autonomy but doesn\u0026rsquo;t offer technical alignment. Ticketing Hell. Your platform is an Ops Service Desk behind the scenes. Ticketing happens because teams (intentionally or unintentionally) silo themselves away. Please stop it. I wasn\u0026rsquo;t expecting to get much new out of \u0026ldquo;The Science Behind Team Cognitive Load,\u0026rdquo; but Dr Laura Weis proved me very wrong. If you ever have a chance to see her present, do it. I now have a list of theories to dig into, such as the Capacity Theory of Attention, Attentional Control Theory, Cognitive Load Theory, and Information Theory. All the while considering her warning, \u0026ldquo;Information Overload is a form of pollution in our environment.\u0026rdquo;\nRemember that guy who coined the term DevOps? Yup, that was Patrick Debois. He has been digging into all things AI. In \u0026ldquo;Every AI Engineer Deserves an AI Platform (team),\u0026rdquo; he reviewed the current state of AI platforms in the enterprise and some gotchas. I\u0026rsquo;m coming around to the idea that coding \u0026ldquo;agents\u0026rdquo; may become more mainstream in the enterprise sooner rather than later. Patrick offers some ironies to think about during this transition:\n\u0026ldquo;The developer\u0026rsquo;s role is changing, becoming more of a manager of code\u0026hellip;more like an ops person.\u0026rdquo; \u0026ldquo;As agents complete more of the app, you will experience a gradual loss of situational awareness.\u0026rdquo; \u0026ldquo;There\u0026rsquo;s a big problem with never producing code. How do you learn enough to become a reviewer for the agent?\u0026rdquo; \u0026ldquo;Learning will take the time saved in producing. (For example, chaos days to learn how it fails and practice troubleshooting).\u0026rdquo; I enjoyed catching up with friends and hearing what other folks are doing in the enterprise. While I had a nagging sense of \u0026ldquo;all of this has happened before, and all of this will happen again,\u0026rdquo; I remember the importance of us doing the hard work to move the plot forward. Or, as Lauren Woods put it, \u0026ldquo;Choose effective over doing it right.\u0026rdquo;\n","permalink":"https://opsrob.com/post/etls-reflections/","summary":"\u003cimg src=\"/img/etls.png\" alt=\"Dr Laura Weis presenting\" width=\"400\"\u003e\n\u003cp\u003eIt doesn\u0026rsquo;t have to be the individual genius that drives innovation\u0026ndash;community and location can do more to foster creativity and innovation.\u003c/p\u003e\n\u003cp\u003eWhile kicking off the IT Revolution\u0026rsquo;s Enterprise Technology Leadership Summit last month, Gene Kim introduced me to scenius, a word suggested by Brian Eno in 1996. Brian said, \u0026ldquo;Scenius stands for the intelligence and the intuition of a whole cultural scene. It is the communal form of the concept of the genius.\u0026rdquo;\u003c/p\u003e","title":"Enterprise Technology Leadership Summit Reflections"},{"content":"Exciting New Chapter Ahead!\nAfter nearly eight incredible years at Slalom, I decided to embark on a new journey last month.\nNo drama here – this wasn’t another layoff or a falling out. Instead, the timing was perfect for self-authoring and spending the summer with my daughters while they still find me somewhat interesting. I feel that stepping back occasionally is a critical and undervalued growth move for senior leaders who can take the time.\nAfter an extended tour of duty throughout the Nordstrom technology organization, I joined Slalom in 2016 specifically to broaden my industry experience and learn from amazing people. Mission accomplished!\nDuring my time at Slalom, I had the privilege to work with:\nDiverse industries: Airlines, semiconductors, government (federal and state), non-profits, utilities, retail, video gaming, real estate investment trusts, automotive, entertainment, telecommunication services, and more.\nEngineering teams of all sizes: Helping them overcome years of technical debt and organizational scar tissue.\nSenior and executive leaders: Accelerating their journey towards adopting an agile, engineering-focused culture and the modern operating models that enable it.\nSlalom’s teams: Adopting SRE practices and a “we build it, we run it” mentality for operating and enhancing client\u0026rsquo;s products in production.\nSo, what’s next? I’m not sure, but maybe you have an idea? I’ll likely take a break from consulting to return to an industry-focused senior leadership role in a technology organization facing growth or velocity challenges that benefit from rethinking business as usual.\nIf you are facing scaling, transformation, or delivery challenges in a tech organization, I’d love to chat—whether you have a role or not! (Keep in mind, I am actually spending time enjoying the summer with my family, so responses might be a bit delayed 😀)\n","permalink":"https://opsrob.com/post/exciting-new-chapter/","summary":"\u003cp\u003eExciting New Chapter Ahead!\u003c/p\u003e\n\u003cp\u003eAfter nearly eight incredible years at Slalom, I decided to embark on a new journey last month.\u003c/p\u003e\n\u003cp\u003eNo drama here – this wasn’t another layoff or a falling out. Instead, the timing was perfect for self-authoring and spending the summer with my daughters while they still find me somewhat interesting. I feel that stepping back occasionally is a critical and undervalued growth move for senior leaders who can take the time.\u003c/p\u003e","title":"Exciting new chapter ahead!"},{"content":"I was chatting with Ben Grossman-Kahn M.Ed., and was reminded of his Goats and Fences empowerment lesson, which I still frequently share when coaching leaders to empower their teams.\nCheck out the less than 3-minute video Ben and team put together back in the day:\n","permalink":"https://opsrob.com/post/goats-and-fences/","summary":"\u003cp\u003eI was chatting with Ben Grossman-Kahn M.Ed., and was reminded of his Goats and Fences empowerment lesson, which I still frequently share when coaching leaders to empower their teams.\u003c/p\u003e\n\u003cp\u003eCheck out the less than 3-minute video Ben and team put together back in the day:\u003c/p\u003e\n\u003cdiv style=\"position: relative; padding-bottom: 56.25%; height: 0; overflow: hidden;\"\u003e\n\t\t\t\u003ciframe allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share; fullscreen\" loading=\"lazy\" referrerpolicy=\"strict-origin-when-cross-origin\" src=\"https://www.youtube.com/embed/IRRVgqvdgi8?autoplay=0\u0026amp;controls=1\u0026amp;end=0\u0026amp;loop=0\u0026amp;mute=0\u0026amp;start=0\" style=\"position: absolute; top: 0; left: 0; width: 100%; height: 100%; border:0;\" title=\"Empowerment Lesson: Goats and Fences\"\u003e\u003c/iframe\u003e\n\t\t\u003c/div\u003e","title":"Goats and Fences"},{"content":"As a big fan of Matthew Skelton and Manuel Pais\u0026rsquo;s Team Topologies book, I was excited when Emma Button pointed me toward this awesome repo of Team Topologies graphics/icons for all of your favorite tools (including paper!).\n","permalink":"https://opsrob.com/post/team-shape-templates/","summary":"\u003cp\u003eAs a big fan of Matthew Skelton and Manuel Pais\u0026rsquo;s Team Topologies book, I was excited when Emma Button pointed me toward \u003ca href=\"https://github.com/TeamTopologies/Team-Shape-Templates\"\u003ethis awesome repo of Team Topologies graphics/icons\u003c/a\u003e for all of your favorite tools (including paper!).\u003c/p\u003e","title":"Team Topologies graphics repo"},{"content":"Lately, the most common request I get from CIOs is about what they should focus on and what other companies are doing that they should be getting a start on.\nI have the same answer for most of them. The biggest unlock they can provide is not centered around a technology or specific process. It’s around viewing technology delivery as a sociotechnical problem space to be holistically optimized.\nThis means (over time) shifting the entire technology and product organization to become a value stream-aligned org with a pull-through work model (vs. submitting tickets and pushing into a queue). Do this by centering team models around four types, as outlined in the book “Team Topologies” by Matthew Skelton and Manuel Pais:\nStream-Aligned – these are fully-kitted teams providing technology directly to end users/customers from a product perspective, not a functional technology silo. Platform – These teams reduce the cognitive load of stream-aligned teams by providing self-service capabilities. Enabling – These teams temporarily partner with Stream aligned and/or Platform teams to explore novel/new problem spaces, think of them as consultant teams that help the above two teams become self-sufficient in something new. Complicated Sub-System (optional and should only be a few) – These support a complex functional technology. Usually, this is something like a mainframe.\nLayer on top of that the concepts of Slowification, Simplification, and Amplification from Gene Kim and Steven J. Spear\u0026rsquo;s latest book, “Wiring the Winning Organization” and you can dramatically increase the time to value, moving from an internal perception of a slow-moving behemoth to a proactively responsive organization.\nSlowification – Makes solving problems easier for the organization to do (and therefore faster). Simplification – Break down problem spaces to make the problems themselves easier to solve. Amplification – Make it obvious there are problems that demand attention and whether they’ve been seen and solved.\nOf course, this is all easier said than done in any reasonably complicated organization. But, failure to make this happen means you likely aren’t taking care of your people as well as you’d like, and you will lose ground to competitors doing this right.\n","permalink":"https://opsrob.com/post/devex-in-action/","summary":"\u003cp\u003eLately, the most common request I get from CIOs is about what they should focus on and what other companies are doing that they should be getting a start on.\u003c/p\u003e\n\u003cp\u003eI have the same answer for most of them. The biggest unlock they can provide is not centered around a technology or specific process. It’s around viewing technology delivery as a sociotechnical problem space to be holistically optimized.\u003c/p\u003e\n\u003cp\u003eThis means (over time) shifting the entire technology and product organization to become a value stream-aligned org with a pull-through work model (vs. submitting tickets and pushing into a queue). Do this by centering team models around four types, as outlined in the book “Team Topologies” by Matthew Skelton and Manuel Pais:\u003c/p\u003e","title":"Common CIO request- What should I focus on?"},{"content":" Last week, I was privileged to be given an early copy of the latest book by Gene Kim, Wiring the Winning Organization. This happily coincided with a solo weekend trip to the Washington coast I had already planned\u0026ndash;what I would soon learn was my version of personal slowification, simplification, and amplification.\nThis latest book is aimed at the most senior leaders of an organization, those responsible for the wiring of its sociotechnical system, and I can\u0026rsquo;t recommend it to them enough.\nMiswire your organization by failing to complete social circuits, introducing latency and errors through complexity, or ignoring essential sensors, and the people you are accountable for will feel the futility of working in a broken system\u0026hellip;with results to match. Get it right, and your people will experience a, perhaps, once-in-a-lifetime experience of working in a well-functioning, self-stabilizing system and achieving exceptional outcomes.\nThis book dives into these sociotechnical systems and wiring them for success through the lens of slowification, simplification, and amplification:\nSlowification: Pausing your operational tempo to reflect and learn. Simplification: Break problems into smaller pieces to solve them more straightforwardly. Amplification: Make it impossible to miss that a problem needs attention, and make it clear that the problem has been seen and solved. Throughout the book, you\u0026rsquo;ll notice familiar practices in the many case studies illustrating these three mechanisms. However, this is the first time I\u0026rsquo;ve seen the dials needed for successfully modifying a sociotechnical system presented holistically in such an easy-to-consume way. And, for those like me who love to geek out on the underlying fundamentals, the appendix offers a deeply researched list of source materials that will keep my \u0026ldquo;To Read\u0026rdquo; queue well populated.\nBut, back to my trip to the coast to disconnect. In the past, I always thought taking a break to be with my thoughts was simply an opportunity to recharge with some intentionality. What I found fascinating was that I could apply the mechanisms of slowification, simplification, and amplification to this time.\nBy slowing down and pausing my operational tempo- disconnecting from work and home- I made space to reflect on the last several months and focus on what I could learn from my recent experiences. It became possible to simplify what had previously seemed like intractable problems, and I could ensure I was amplifying the most critical signals and coming up with a response to them.\nOverall, it was a successful weekend, and I promised myself to stay clear of the tyranny of maintaining operational tempo and instead pausing to reflect.\nWiring the Winning Organization by Gene Kim and published by IT Revolution will be released on November 21st. You can preorder it from Amazon.\n","permalink":"https://opsrob.com/post/slowification-simplification-amplification/","summary":"\u003cimg src=\"/img/wwo.jpg\" alt=\"bookcover\" width=\"200\"\u003e\n\u003cp\u003eLast week, I was privileged to be given an early copy of the latest book by Gene Kim, Wiring the Winning Organization. This happily coincided with a solo weekend trip to the Washington coast I had already planned\u0026ndash;what I would soon learn was my version of personal slowification, simplification, and amplification.\u003c/p\u003e\n\u003cp\u003eThis latest book is aimed at the most senior leaders of an organization, those responsible for the wiring of its sociotechnical system, and I can\u0026rsquo;t recommend it to them enough.\u003c/p\u003e","title":"Slowification, Simplification, and Amplification"},{"content":"I participated in a panel discussion with Fedscoop about how government agencies are using the public cloud to modernize.\nRead more here: FedScoop\nYouTube ","permalink":"https://opsrob.com/post/leveraging-cloud/","summary":"\u003cp\u003eI participated in a panel discussion with Fedscoop about how government agencies are using the public cloud to modernize.\u003c/p\u003e\n\u003cp\u003eRead more here: \u003ca href=\"https://fedscoop.com/video/leveraging-the-cloud-to-modernize-applications/\"\u003eFedScoop\u003c/a\u003e\u003c/p\u003e\n\u003ch2 id=\"youtube\"\u003eYouTube\u003c/h2\u003e\n\u003cdiv style=\"position: relative; padding-bottom: 56.25%; height: 0; overflow: hidden;\"\u003e\n\t\t\t\u003ciframe allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share; fullscreen\" loading=\"lazy\" referrerpolicy=\"strict-origin-when-cross-origin\" src=\"https://www.youtube.com/embed/kFswUSvzsUw?autoplay=0\u0026amp;controls=1\u0026amp;end=0\u0026amp;loop=0\u0026amp;mute=0\u0026amp;start=0\" style=\"position: absolute; top: 0; left: 0; width: 100%; height: 100%; border:0;\" title=\"Driving Cloud-First Strategies in the Public Sector\"\u003e\u003c/iframe\u003e\n\t\t\u003c/div\u003e","title":"Leveraging the cloud to modernize applications"},{"content":"Originally posted on Medium\nIntroduction In my role as a Cloud and SRE Practice Lead at Slalom Build, I am fortunate to talk to a wide range of organizations, from smaller mid-market companies all the way to astoundingly large and complex enterprises, all from an equally wide range of industries.\nThere is no doubt about it, Site Reliability Engineering (SRE) is the latest hot topic. These companies are looking to reduce the impact and risk of failure that can come from moving quickly at scale with increasingly complex systems.\nWhat is SRE? It is a specific implementation of DevOps that applies a software engineering mindset towards solving traditional operations problems, with a focus on creating reliable and scalable technology.\nMuch like DevOps, it turns out no one is following a common team blueprint when it comes to SRE. However, we are seeing several distinct patterns emerge as organizations adopt SRE.\nPlease note- This post is not about gatekeeping and declaring there is only one true way to approach SRE. After being through too many “DevOps is not a team or title” debates, I have mellowed out considerably when it comes to these things. It is more important that you go with what works for your context and stay focused on the outcomes.\nSRE Implementation Patterns With that in mind, lets get into the more common team structures emerging in organizations adopting Site Reliability Engineering:\nThe Google Model A dedicated engineering team focused on running and scaling a product or platform. Team is made of highly skilled software and systems engineers who are both directly updating the product code base for reliability and building associated tooling to support the product. Team is on-call for the product. Can hand on-call support back to the feature teams if reliability falls below an agreed upon threshold (Google uses an Error Budget model for driving these conversations). We Are Now SRE Ops, DevOps, and platform teams have rebranded and reimagined themselves as Site Reliability Engineers. Goal is to put a stronger engineering emphasis around improving reliability and scalability. These teams typically do not make significant changes to product code, but do play a heavy role in the underlying infrastructure, tooling, platforms, or day to day support of the product. Team is on-call and typically rely on a development team for escalating application specific issues. SRE Center of Practice A centralized team focused on creating and advocating for reliability tools and processes. On-call responsibilities are limited to non-external customer facing tooling. Is an internal consulting arm to help with adopting SRE patterns and tooling, but does not have direct product accountability. These teams are uniquely positioned to see the forest for the trees while also staying sharp on the latest technologies, trends, and research in the SRE space. Embedded SRE SRE engineers are embedded into cross-functional teams that own the the end-to-end lifecycle of a product, from build through decommission. This can take two shapes. First is a matrix SRE organization where engineers belong to a single capability and are also embedded full time within a product team (this is the Slalom Build SRE model). Alternatively, product teams may hire their own dedicated SRE engineers. The SRE engineer role is a hands on reliability/scalability Subject Matter Expert that helps the team adopt the engineering practices and tooling to ensure right-sized scalability and reliability throughout the lifecycle. The entire team participates in an on-call rotation. The Long Tail I still get into a significant number of conversations where teams or entire organizations have not yet heard of SRE. Although it’s obviously the new hotness, we are still in the early stages of adoption. It is an exciting time for all of us to share what we are learning for the benefit those starting down the path. Consider This If you are embarking on an SRE implementation, how do you ensure you achieve the reliability and scalability benefits you are looking for? A couple things to keep in mind:\nMake it Humane My absolute favorite aspect of the SRE community today is the dialog around the interaction points between humans and the complex systems they are supporting. Make improving the on-call experience and reducing the inherent stress of dealing with large-scale production systems a key tenet of your SRE program. Customer Outcome Driven Maintain a direct line of sight to customer impacts (positive and negative), regardless of whether your SRE engineers are directly embedded in a product team, or building and supporting a platform used by product teams. A common red flag is a team so focused on their platform and associated tooling that they can’t quantify or get visibility into the external customer impact they are having. In fact, this was the drive for the creation of Google’s Customer Reliability Engineering team…to establish a bridge between actual customer products and the underlying Google Compute Platform teams. Focus on Rightsizing SRE teams must have real discussions with their internal customers, product teams, and business partners around the cost of downtime and the cost of uptime. Every additional nine has a real cost that needs to be quantified and made visible. Make both missing and greatly surpassing Service Level Objectives (SLOs) undesirable. This is a new concept for most traditional production support teams that consider all downtime an unattractive or unacceptable risk. Exceeding your SLO over too many consecutive periods might be an indication that you have over-invested in reliability at the expense of other customer features. Transparency through real SLOs, created in partnership with the business, tracked, and regularly evaluated for relevancy are a must have in the SRE world. Too many teams sidestep or only pay lip service to this during their journey to SRE. Ownership and Accountability One of the most painful parts of traditional ops on-call is the complete decoupling of product development and the running of that product. Getting called at 3am is bad enough, not being empowered to fix the underlying causes is much worse. Ensure a solid feedback path into the product teams backlog is in place for operational fixes and improvements. Make sure the engineers accountable for production have ownership over reliability and scalability fixes, or at least very strong input into their prioritization. Include both Software and Systems Engineering Expertise Transitioning to SRE is most successful with a mix of Software and System Engineering expertise on the team. This means substantially updated or additional job descriptions are usually needed. If you are looking to transition a traditional Ops team to an SRE model, push hard to bring in at least a pair of software engineers. A pair creates enough critical mass to ensure their voices are heard, they can bounce ideas off of each other, and creates capacity for two-way knowledge sharing with the rest of the team. The same works in reverse, bring in a pair of Systems Engineers into a traditional Software Engineering team transitioning to SRE. Be Uncomfortable You should experience this as an uncomfortable transition, at first. Otherwise, you aren’t pushing yourselves forward enough to get the benefits you are looking for and instead will only get the “feel good” effects of being associated with SRE. Your business partners will be even more uncomfortable. They are taking a leap of faith by fundamentally changing core support processes into a new model they may not yet fully understand. This is ok, but be sure to have empathy and treat them as equal partners in this journey. How to move to SRE? “Start small” advice definitely applies. Look for both technology and business teams excited about the opportunity SRE brings and put a strong focus on quick wins. Have a bias towards action, measure your progress and impact to inform your next action. Make sure you are adjusting direction as you learn. Be transparent about what is working, what isn’t, and where you have learned something unexpected. Look for upcoming posts to share more tactical starting points, as well as why Slalom Build chose Embedded SRE as the right model for us.\nSpecial thanks to Arielle Allen, Sascha Bates, Jeremiah Dangler, Joel Forman, Jeff Knecht, Dan Mazur, and Kevin McClelland, and for their help making this post better.\n","permalink":"https://opsrob.com/post/shapes-of-sre/","summary":"\u003cp\u003e\u003cem\u003eOriginally posted on \u003ca href=\"https://medium.com/slalom-build/the-many-shapes-of-site-reliability-engineering-468359866517\"\u003eMedium\u003c/a\u003e\u003c/em\u003e\u003c/p\u003e\n\u003cp\u003e\u003cimg alt=\"laptop displaying metrics dashboard\" loading=\"lazy\" src=\"/img/laptop-dashboard.jpg\"\u003e\u003c/p\u003e\n\u003ch2 id=\"introduction\"\u003eIntroduction\u003c/h2\u003e\n\u003cp\u003eIn my role as a Cloud and SRE Practice Lead at \u003ca href=\"https://https://www.slalombuild.com/\"\u003eSlalom Build\u003c/a\u003e, I am fortunate to talk to a wide range of organizations, from smaller mid-market companies all the way to astoundingly large and complex enterprises, all from an equally wide range of industries.\u003c/p\u003e\n\u003cp\u003eThere is no doubt about it, Site Reliability Engineering (SRE) is the latest hot topic. These companies are looking to reduce the impact and risk of failure that can come from moving quickly at scale with increasingly complex systems.\u003c/p\u003e","title":"The Many Shapes of SRE"},{"content":"Introduction All sorts of reasons are used to justify Enterprise Architecture. At the top of the list is usually a desire for high efficiency and minimization of risk. In other words, Enterprise Architecture knows best. We repeatedly end up in the model because, ultimately, you can\u0026rsquo;t have the wild west in large technology organizations. Or\u0026hellip;can you?\nYes you can, with Simon Wardley\u0026rsquo;s Pioneers, Settlers, and Town Planners model. Take advantage of organizational theft to ensure amazing people in each of these three groups contribute to increasing the velocity and on-going sustainability of your organization. While many know about this model, the mechanics (like theft, exciting!) are counter intuitive and it is often understandably difficult to relate this to real-world problems.\nThis talk aims to provide real-world tooling/technical practice adoption examples and key things you need to know to begin shifting your complex organization to this model. And yes, even Enterprise Architecture has a place in the new frontier. I consider this one of the best organizational constructs out there for achieving velocity and managing technical risk/debt. Let\u0026rsquo;s get it out there and keep testing it!\nYouTube Slideshare Settlers of DevOps - DevOpsDays Boston 2017 from Rob Cummings ","permalink":"https://opsrob.com/post/pioneers-settlers-town-planners/","summary":"\u003ch2 id=\"introduction\"\u003eIntroduction\u003c/h2\u003e\n\u003cp\u003eAll sorts of reasons are used to justify Enterprise Architecture. At the top of the list is usually a desire for high efficiency and minimization of risk. In other words, Enterprise Architecture knows best. We repeatedly end up in the model because, ultimately, you can\u0026rsquo;t have the wild west in large technology organizations. Or\u0026hellip;can you?\u003c/p\u003e\n\u003cp\u003eYes you can, with Simon Wardley\u0026rsquo;s Pioneers, Settlers, and Town Planners model. Take advantage of organizational theft to ensure amazing people in each of these three groups contribute to increasing the velocity and on-going sustainability of your organization. While many know about this model, the mechanics (like theft, exciting!) are counter intuitive and it is often understandably difficult to relate this to real-world problems.\u003c/p\u003e","title":"Settlers of DevOps - Down with the Tyranny of Architecture Governance"},{"content":"Introduction I had the privilege of supporting two teams this past year, one that was an operations team learning development practices, and a development team that was discovering operations while introducing continuous delivery. Both approached Chef from different viewpoints, but in the end were united by this common tool set.\nThis talk, given at ChefConf 2014, covers what I wish I had known a year ago about leading change (with specific examples), including:\nThe ups and downs your team will experience Who to put on your full stack team Why empowering teams is hard Where to focus the adoption of new ideas How to make those ideas stick. YouTube Slideshare Chef Conf DevOps Roller Coaster from robc77 ","permalink":"https://opsrob.com/post/the-devops-roller-coaster/","summary":"\u003ch2 id=\"introduction\"\u003eIntroduction\u003c/h2\u003e\n\u003cp\u003eI had the privilege of supporting two teams this past year, one that was an operations team learning development practices, and a development team that was discovering operations while introducing continuous delivery. Both approached Chef from different viewpoints, but in the end were united by this common tool set.\u003c/p\u003e\n\u003cp\u003eThis talk, given at ChefConf 2014, covers what I wish I had known a year ago about leading change (with specific examples), including:\u003c/p\u003e","title":"The DevOps Roller Coaster"},{"content":"Introduction I had the honor of delivering one of the ChefConf 2013 keynotes. It was my first time speaking at a conference and although I was extremely nervous at the beginning, it was great fun and I learned a ton.\nThis talk discussed the challenges of trying to push a fundamental or disruptive change into large organizations, why it is difficult, why that is actually a good thing, and then showed one way to go about it along with two real-life examples from Nordstrom.\nYouTube SlideShare ","permalink":"https://opsrob.com/post/chefconf-2013-keynote-level-up-your-change/","summary":"\u003ch2 id=\"introduction\"\u003eIntroduction\u003c/h2\u003e\n\u003cp\u003eI had the honor of delivering one of the ChefConf 2013 keynotes. It was my first time speaking at a conference and although I was extremely nervous at the beginning, it was great fun and I learned a ton.\u003c/p\u003e\n\u003cp\u003eThis talk discussed the challenges of trying to push a fundamental or disruptive change into large organizations, why it is difficult, why that is actually a good thing, and then showed one way to go about it along with two real-life examples from Nordstrom.\u003c/p\u003e","title":"ChefConf 2013 Keynote - Level Up Your Change"}]