Our GlobalNOC Renewal Program has been going for over a year.  Over this time, we’ve seeing both deep changes and interesting challenges.  So, where do we stand?  What’s gone well, what didn’t, and what did we learn from it all?  Here’s how we think we’ve done so far.

What went well: our best accomplishments for 2019

1.) We developed GlobalNOC OS to be our compass

“GlobalNOC OS” is what we call the set of foundational elements that define what we want GlobalNOC to be. We really see these things as our guide for making the most important decisions going forward for the next decade or more.  For example, to decide how we want to change our organizational structure, we need to know what kind of culture we want, and what outcomes we’re looking for.  To decide whether to take on a new service, we have to see how it aligns with our strategy and vision of the future.

It includes:

  • why we exist (mission)
  • what it would really look like if we were completely successful (vision)
  • what kind of place we want GlobalNOC to be (culture)
  • our core strategy for who we serve, what they need, and how we can deliver that (value strategy)

2.) We became “One GlobalNOC”

“Working with GlobalNOC is like working with a bunch of different organizations with 1 logo.”  Ouch.

When we started the GlobalNOC Renewal Program, we interviewed many of our clients about what they see when they work with us.  During this, someone mentioned something that’s really stuck with us ever since.  They said, “Working with GlobalNOC is like working with a bunch of different organizations with 1 logo.”  Ouch. So, we’ve worked really hard to fix this, to really become One GlobalNOC, not just to our partners, but also from the inside.

One GlobalNOC to our partners

We want to get much better at making it easy for our clients to work with us as a single partner.  This lead to 3 things:

  1. Client Relationship Manager: We want our clients to be understood. We want to get everything we do for them aligned.  So in 2019 we tried an experiment.  We assigned someone to dive in with one of our network partners (OSHEAN) as their client relationship manager.  The CRM coordinated everything we did across all GlobalNOC teams with what they learned is important to OSHEAN.  We wanted to see if this kind of role would pay off.  Would it make us more responsive? Would we waste less of everyone’s time working on the things they didn’t want?  Every indication was that the experiment was a success. We were able to clean up requests that weren’t really needed and focus ourselves in on the things that really mattered.
  2. GlobalNOC Days: We also want to build a sense of community among the networks we support.  So in 2019 we hosted the first ever GlobalNOC Days meeting in Indianapolis.  We brought  about 50 people together to talk about what everyone is doing and what they all really need from us.  The facetime was valuable for us to hear from everyone and for them to hear from each other. It was crystal clear that we needed to focus on giving them portals they could use for their own customers and on moving to a more modern service-specific notification system.
  3. A new GlobalNOC User Group: We see a future where every client benefits from the work for other clients.  Where the things we learn to develop a new process or tool for one network benefits every other client. So to make this happen, we relaunched the GlobalNOC User Group as a forum for regular conversation.  We shared regular status updates, demo’d new developments, and discussed changes to how we work.

One GlobalNOC to ourselves

But it’s not enough for us to just work cohesively with others.  We also need to be cohesive from the inside; “One GlobalNOC” is more than skin deep. As we grow GlobalNOC, we need to be able to keep our identity and culture intact, to work well across all of our teams.  It’s right there in our cultural rules: “We succeed together”. We believe that the best GlobalNOC won’t come from having a few geniuses, but from a team of people doing more together than they could apart.  We used a number of things to create this sense of common purpose and cohesion:

  • All-Staff Meetings that matter: It was important for us to get everyone together, not to talk AT them, but to generate conversation.  So, we started having the entire org meet every 6 weeks or so for lively discussion with lots of break-out sessions.
  • Pizza -> Learning: If you want to really spark curiosity and learning, serve pizza.  We ran 11 technical sharing sessions in 2019.  In these “Pizza Talks” our experts brought people from across the org together over lunch. We covered topics from Segment Routing to Time Series Data to a Post-mortem on a complex outage.
  • AMAs for work:  In one of our staff meetings, people asked if we might try something like Reddit’s prevalent Ask Me Anything format to share information.  It was super-easy to try and ended up working out great.  We’ve done this not just on projects but also had a leadership AMA where anyone could ask the org leadership any questions, nothing off-limits

3.) We moved to an automated engineering future

Despite all the advances we’ve made to network operations over the 20+ years, we were still doing most of our work configuring networks the same way we always have (and the way most networks in R&E do), by logging into a CLI and typing commands one by one.  It was time for this to change. This isn’t just inefficient; it’s also holding the networks back.  The network of the future is part of the larger IT service system, where containers, cloud, and *aaS have taken over.  The network should be just as quick and flexible as the rest of the system. And it all needs to work together.

So, we’re committed to transforming the way we configure network infrastructure and services to be automated, on-demand, and orchestrated with everything else.  Not just for the networks we support directly, but also for our entire R&E networking community.  We’re committed to becoming automation-first, and to sharing everything we learn along the way with the community.

We made pretty good strides toward this:

  • hired Grover Browning to lead the initiative
  • developed a strong automation backend system based on Git and Ansible AWX integrated with our existing network database
  • worked with the networks with full GlobalNOC engineering support to standardize and automate their core infrastructure configurations

While we didn’t meet our hopes to have the majority of our work automated by the end of 2019 (see lessons below!), we’re confident that we have everything setup to get there in 2020.

4.) We created a feedback culture

Finally, we worked on a number of things to make GlobalNOC a better place for people to grow and to work.  The future GlobalNOC should be a workplace every employee is proud to work for: full of opportunity, challenge, and support.  But when we asked everyone how we were doing,  we found we fell short of this.  One of the lowest scoring areas was in the quality and quantity of feedback to help us improve.

We realized quickly that fixing this needed to be a top priority. If we hope to continue to grow and retain our expertise, we need to have a culture of continuous learning and improvement.  There’s no way we can do this without everyone learning first how to both give and take feedback.

To fix this, we worked both on peer to peer and manager feedback.  We trained everyone on effective feedback, asked every person to give structured feedback to their teammates, and created a single method for review and feedback conversations between managers and their staff.  Based on the 50% drop in dissatisfaction in this area, we think we made good progress here.

Challenges: what lessons did we learn?

Lesson 1: Everything takes longer than you think it will

We knew this going in.  We took it into consideration.  And yet, things still took longer. They always do. On the list of things that took much more effort than we thought:

  • Building a community among our clients
  • Figuring out how to get input from users
  • Creating a leadership team comfortable talking about difficult things with each other
  • Defining culture and strategy
  • Standardizing network configurations

Because of this, there were things we’d hoped to do in 2019 that have shifted to 2020 (structural org changes, completion of automation, helping staff with career advancement, and many others).  Which leads to lesson 2…

Lesson 2: Progress > perfection

This is one of our 5 culture rules, and has almost become a mantra for us. In an org conditioned to look for perfect uptime, it’s not surprising that we tend to worry about risk.  It bleeds over from operating networks into every other decision we make.  So, when it comes to trying new ways of doing things in the organization, we’ve often struggled to move forward without knowing how things will turn out.

But this results in too many discussions and too little activity.  And after all, there are clear and daunting risks to doing nothing. So, we’ve slowly learned to be ok with uncertainty.  Instead of using risk as a way not to do something, we try to use it to develop safe-to-try experiments.

Lesson 3: Doing less = doing more

Closely related to lesson #1. Given the fact that you will never get as much done as you’d hoped, what do you do? Two things:

First, the most obvious thing: get very good at prioritizing.  Whatever is at the bottom of the list probably won’t get done. So it better not include anything that’s truly critical.

Second, the less obvious thing: break efforts down as small as possible.  Just like an agile software company will look for minimal viable products, we need to look for minimal viable improvements.

Going forward, instead of many initiatives with many parallel projects, we’re focusing on a small number of projects at a time, each with clear limited scope.

Lesson 4: Better communication is possible

Lastly, it might seem obvious to say that communication of change is critical and that it requires a concentrated effort.  I know I thought it when reading about it in John Kotter’s “Leading Change”.  And yet, even in an organization of only 120 people, we still struggled with it.

Great communicators have an appreciation for positioning. They understand the people they’re trying to reach and what they can and can’t hear. They send their message in through an open door rather than trying to push it through a wall

-John Kotter, “Leading Change”

The lessons? We’ve found 3 critical elements to effectively communicating. People need to hear things many times, from many people, and done in a way that’s relevant to them.  One email on an organizational change will not be enough; it’s unfair to expect it.  It also won’t be enough just to hear it from one leader. They will need to hear it from their manager and co-workers as well.  And most importantly, it needs to come in a form that is meaningful to them.  For instance, instead of just telling someone that we value progress over perfection, we should say that we’ve decided to move forward on their project as a trial, in keeping with our desire to become more experimental.