What it Takes to be Genuinely Data Driven in Software Engineering

Transcript

Shane Hastie: Hey, folks. Before starting today’s podcast, I wanted to share that QCon Plus, our online software development conference, returns from November 30 to December 8. Learn from senior software leaders at early adopter companies, as they share their real world technical and leadership experiences, to help you adopt the right patterns and practices. Learn more at qconplus.com. We hope to see you there.

Shane Hastie: Good day, folks. This is Shane Hastie for the InfoQ Engineering Culture podcast. Today, I have the privilege of sitting down, across many miles, with Andrew Lau. Andrew, you are the CEO of Jellyfish, and probably a good starting point is, who’s Andrew, and what’s Jellyfish?

Introductions [00:51]

Andrew Lau: Shane, thank you for having me here. It’s great to talk to you and your community, and thanks for letting us share my story around. Okay, so who am I? Gee. So, I’m hailing from Cambridge, Massachusetts, right across the water from Boston. I’m actually a Bay Area, California native, who’s been in the more volatile weather for the last 25 years.

But to know me, I was trained as a software engineer, I came to the East Coast of the US for school, and I had no intention of staying. It wasn’t the cold winters that I sought, but life happened, is the actual answer. Transit engineer… Put my finger, almost by accident, into the startup scene. Met my current co-founders of Jellyfish 22 years ago, and I can talk to that in a second. But that’s what actually got me to both be in the startup scene, and stay here, geographically, where I am.

And so, I’ve gone on the journey from being an engineering manager, learning how to do pre-sales, pro services, engineering management, to find myself running big engineering teams, to eventually, entrepreneuring, and finding my way, and growing teams and businesses along the way. I probably wear multiple hats in life, doing all those things. And for fun, to know me personally, gee, I play tennis. And the last few years, a lot of people have been baking bread. I’ve been working on my, well prior life I was a barbecue judge, so I’ve actually been working on my Texas brisket game. So some people have been sitting at home watching bread rise. I’ve been sitting at home watching my smoker run. So that’s knowing me a little bit on this.

And to your second question, what is Jellyfish? Well, Jellyfish is a software company, five years old. Jellyfish makes an engineering management platform to really help leaders of engineering teams kind of help their teams get more successful and connected to the business.

We’ve been really lucky and during the last few years and I say that lucky in the sense that, look, it’s been a hard few years for all of us and I come here today with empathy. Shane, I hope you and your family are doing well. I hope listeners, your families are as well. It’s just been a tumultuous last few years, but the silver lining for us is just the growth that we’ve seen. We were just a handful of people before the pre Covid times and today we’re almost couple hundred. And so that change has been really a positive part of it, but it’s also been hectic, amazing and crazy to get the chance to work with such amazing people along the way.

Shane Hastie: An interesting background there and definitely we’ll probably delve into some of those things, but let’s start with something that came out of Jellyfish is this state of engineering management report. Tell us a little bit about that.

State of Engineering Management Report [03:25]

Andrew Lau: So, we have this unique seat from where we’re sitting where we have the privilege, and I say privilege, by working with hundreds of companies, the leading edge, the upper echelon of teams that are really defining the way that we do engineering at scale as an industry. And we have the privilege of working and helping with them and learning from them. But also that because of this we actually can see trends move through the actual teams, right?. And because of this earlier this spring, we put out an annual report, a state of engineering management and we’re really able to get a feel of what’s changing in these teams and what’s relevant along the way. And so those are… That really is the nature of our report and it’s because we’re sitting in this place where we can actually see what’s actually happening across the teams.

Shane Hastie: You can see what happens across the teams. So what are some of those insights? What are the things that you were seeing in this report?

Andrew Lau: Shane, with the state of engineering manage report, we actually look across the hundreds of companies we’re actually talking to, which allows us to see how hundreds of thousands of engineers are doing their work. And in doing so, we can actually see the trends in the world of engineering management leadership. And so the things that are actually in that report tie into how we can look at data driven teams actually outpace like data deficient teams, those that actually are focusing on innovation, quality and speed to market. Those are all areas where we actually can see the correlation moving that those actually focus on those things can actually accelerate those pieces.

If you cared a lot about making sure you invest in innovation, we can see how teams that actually focus on that are able to drive that up. We actually see teams that actually are focused on quality and actually how the implications of quality. Where poor quality actually influences the amount of time being spent actually repairing and maintaining and how you can actually manage that going forward once you start looking at it.

And the last area that I would actually say is actually even things are speed to market. So the time it takes from the time you conceive something to the time you actually can get it out there and you can see how teams, when they actually start looking at this data and how they behave differently across there. The report’s available, you’re welcome to actually pull it and take a look. A lot of great insights there. And I think part of it is because it’s a sampling of being able to see what trends play out in these kind of contemporary organizations that are really are driving the way forward for all of us. And so we’re privileged to actually get a chance to actually both see that and learn from these folks marching forward.

Shane Hastie: So what does it really mean to be data driven as an engineering manager today?

What does being data driven really mean? [05:57]

Andrew Lau: Great question. So, look, I grew up as an engineer, eventually made my way to engineering management. I would say that in my experience, a lot of it is what makes one successful. Well you get the knows of what’s actually happening on your team. You get the knows of when things are stuck, who actually needs help when a project’s actually in trouble and how can you actually nudge it forward. You also find out the knows of actually what’s happening from the business side and what they need.

I actually think after years of that, those are the things that you instinctually learn, which is great. However, I think instinct alone often isn’t sufficient because a few things at scale as you get bigger, as your team gets bigger, instinct alone can be dangerous. And look, we all know we’re wrong sometimes. We have gut feels. We have a quick read. I think we should feel that muscle. I think that’s an asset for all of us.

But sometimes you should also double check with data in that stuff. And I think especially in this day and age, I think we have to be careful of actually not realizing that we might actually have biases if we’re not careful. We all think we know who the best engineer is that helps save these projects. But sometimes we should lift the covers a little bit and realize is someone’s stuck? Are they not able to get through something because the PR’s are not getting helped? Is someone not actually getting the actual ramp up that they actually need?

And so when I come back to data driven, I think I’m going to paraphrase it wrong, but there’s an old analogy which is you can’t improve what you don’t measure, right? And I think data is part of your tool belt.

So what do we all bring as leaders? We bring, well we’re in technology space. We bring our technology chops forward and our own experiences when we’re actually doing it. We bring our own war stories and experiences that we’ve been through on the good days and the bad days and team members and friends that have had done well or actually hit walls. We bring all those with us. Those who alone are the kind of historical tool belts. I think what we should bring forward too is actually bring some data to validate your gut feel and assumptions along the way. Because in some ways as we scale, all of us… Data brings that kind of the bionic tool belt to help us make decisions better, right? Either because it validates that your gut is actually right or sometimes when we’re lost in not knowing where our team is needing help to bring kind of insight into figuring out, hey where should we turn? Where should we actually dive into? Or actually where should we actually correct the business? Because it doesn’t understand what’s actually happening.

And so to me data is actually just part of contemporary leadership, which is because we’re awash in it, we actually have to integrate in the way that we do things today. So that’s what data driven means to me.

Shane Hastie: You make the point there that we are awash in data making sense of those floods.

Making sense of the floods of data available to us [08:35]

Andrew Lau: Yeah, I was talking to someone recently, we have KPIs, we have all the KPIs, all the KPIs. We have all the data and I’m like well if you have all the data, the problem is then you almost have nothing because we’re so  awash in it today, right? It’s easy to say you have all the data. I think it’s actually important to figure out what it is to look at. And I think when you actually think about this actually you should…

The area that we all can help each other on is actually understanding what to look at. Because today you could open any charting package, you could open any data stream, we’re awash in it and the question is what do you pay attention to? And I think for us there’s a few things. We’ve learned from some experienced individuals, people that we work with that like which things they pay attention to and we aim to actually make those data streams more accessible to those that are actually earlier in their pathway that don’t know those things.

We also are able to look at the corpus of all the things that are happening and kind of bring some data science to actually highlight what what’s correlating in these different factors, which things are actually the pieces that actually you should pay attention to. And so when I think about our role in this and all of our roles, it’s not that we need more data for data’s sake. We need to figure out which data pieces we should be looking at, which things we need to actually refine and which things we actually need to focus on. And then last bit is to make it easy to get to .but I think that’s the journey that we’re all on together to actually evolve this industry forward

Shane Hastie: As an engineering manager, which are the data gems, the things that really can help me understand what’s going on with my team and how do we make this data safe?

Andrew Lau: Okay, so my turn to actually turn the tables, Shane. What does safe mean to you? Because I feel like that’s a leading question. I want to make sure I get from you what safe means to you? ‘Cause you’re teaching us here,

Shane Hastie: How do we use the data in a way that the insights are useful and the people who the data is from have trust that it’s being used in a way that’s not going to be used against me. We go back to the 1940’s and before. W. Edwards Deming give a manager a numeric target and he will meet it even if it means killing the organization to do so.

Andrew Lau: You’re right. I would say a couple things to that. So this is fun by the way. Look, I think we can be terrible leaders with or without data. I’ll start with there first. So it is not because of data that actually makes us terrible leaders. And I actually think of this first and foremost when we think about leadership, core leadership, data’s an enabler, data’s not the thing.

Good leadership is not dependent on data [11:08]

I often talk about, before we talk about anything when it comes to data, we as a leader we need to talk about culture. And culture can be this soft, squishy word. Sometimes you can talk about enduring, it’s the unwritten rules. But I think when it comes to technology teams, it’s like actually culture actually implies what matters to us. And when I say us as a team, us as people delivering and you actually phrased it as against, I mean look frankly any company that actually is talking about against probably isn’t in the right place anyways because why are we all here at the companies we’re at? I don’t know anybody that actually wants their company to fail or at least I hope not.

So when I look at this, it’s all about making sure that if there is conflict it’s because we’re not understanding each other. We’re not rowing in the same direction proverbially.

The importance of alignment and clarity of purpose [11:57]

And so go back to culture. I actually think the most important part in leadership is to make clear what we care about and why. When I see disagreements actually happening and technology companies. I first started, is there a misalignment and lack of understanding because who the hell actually wants to sink their ship? I don’t know, that’s a crazy person in my view. More than likely I look at this and I see teams where the interns are working hard, jamming late into the night on a legacy code base fixing bugs of something they didn’t even make. And all the while the business is screaming, why isn’t this new feature delivered because we promised it yesterday.

Okay, well let me pause on that part of it. There’s a fundamental misalignment there. These guys and gals are staying late at night fixing a piece of software they didn’t actually fix. The business is yelling about delivering this other thing. Both of those are valuable for the company clearly, but somehow both parties doesn’t know what the other one wants and why. They’re not actually having a conversation. And so back to culture, I think it’s important to actually align on what matters to us. So Shane, you and I, we could care like, hey we need to just get things on the floor out as fast as possible. We might care about lead time and velocity like nobody’s business because all we want is get stuff out there. That’s all that matters. So if that were the case then I would say we should agree on that and we should actually make sure that we’re on the same page around why lead time and velocity of things that we actually look at and we care about.

Okay, it could be that you and I are like, hey we really got to spend time on innovation, this new thing. We need to put away everything else we’re doing. We need to agree on that. And so we should make sure that we’re protecting our time to work on those things. And so we should measure to make sure what else are we doing and let’s make sure we get it off the plate.

We should have this conversation first before we talk about any data because you’re right, we are awash with data in the world and data can take you many places if you don’t talk where you’re going first, right? It is but a tool to accomplish what you’re doing. I come back to it’s part of your Swiss army knife. It’s part of what you do once you’ve decided as a team, as a leader where you want to go. It is not magic on its own but I think it’s actually just a superpower to enhance what you’re already going in.

Shane Hastie: Shifting gear a tiny bit. Looking back at your career, you’ve gone from as you said, engineer to engineering manager to business person to entrepreneur. What’s that journey and what are the different perspectives that you take at each step in that journey?

The journey to engineering manager and entrepreneur [14:27]

Andrew Lau: I appreciate you asking because you and I were kibitzing a second ago about the languages we all started in. I started with C and you said you’re a COBOL guy, right? And so both of these things are probably lost annals of time. I don’t even know if any of my team members today actually can even talk about a Malik or a Calic anymore in these things. But who knows? I did start as an engineer. I was a systems engineer. I lived in kind of network stacks and the good old days of making sure my memory was free and rolled out. And look, I really enjoyed those days. Probably if I find some moment of retirement that probably is the life of me that can go back to that kind of hands on bit of it. I think that there’s something cathartic about actually working there in that way.

But I actually then eventually moved on to engineering management and I found that role just different. I actually think that when people go into engineering management, I think people think that’s a step forward. I don’t think it’s a step forward for everybody. I think there are times when it’s not fun and I actually look at this like, I’m a dad now of two girls and I kind of think of actually management as parenting. You don’t know you’re going to love it till you’re in it. We can intellectualize it all day long but you have to learn whether you find derivative satisfaction in leading others or when you’re coding yourself, you make a thing, you do a thing in that part of it. And so you kind of learn your way through that.

And then as I kind of progressed in my career, I learned around kind of large scale management, more executive management and I learned that the tillage of actually needing how to translate to business people that actually don’t understand how engineering actually works and that actually becomes the role.

Earlier my career I used to think that oh, you get into these roles and then you get to tell people what to do. Well you know, don’t tell anyone what to do actually in reality. You spend a lot of time bridging is the role and you’re trying to connect the dots and making two pieces actually fit together.

And then entrepreneuring on the other side, on the pure business side, having been through the other journey, I’m actually seeing that, look, there are business implications in this stuff and I empathize with not understanding on the other side. And I guess the benefit of going through all those journeys is I get to look at my other self and actually realize I didn’t actually understand the perspective from the other side of the table at the time. And I try the best I can to realize, to speak to myself back then to understand two sides of it. This is what I need from the other side of the table.And I think to me that’s actually, I joke with my engineer leader either the best job or the worst job. The worst job is that I know the challenges he’s facing and I could tell him, did you do this? Did you didn’t do that? Okay, I know I can be a pain in the ass.

On the flip side, I come here with empathy on it and I understand that your stereotypical business CEO is a pain in the butt yelling this, this, this. And at the very least I try to come back and I should contextualize it. Hey, this is the why. This is what I saw when I was in that seat the other side and I empathize with you. It is a thing but this is a business reality and our job together is to connect those pieces, right?

And I think at the same time I look back to my old engineer self and I would say, hey, there were days that I hated too. I hated working on somebody else’s code that was horrible and then we needed to ship a date on this thing and I hated time commitments that someone else made. But at the same time with that empathy I can actually say, well if a time commitment was made it was because of this and this is why the business needs this and if we don’t do this, this is what’s actually happening and dear engineer tell me what else could we actually do differently to make sure that we get this or tell me what’s going to give. And the engineer that’s stuck late at night working on bugs, I see you and I understand that sucks and that wasn’t your code and that’s not your fault. And so please bring it up to me because the business would rather actually have you working on new features and great things.

So surface that for me so that we can actually talk about what we can actually put away so you can get to those other things. I think that to me being in all these seats has actually helped me actually see the other side. And I hope and you can go connect with my engineering team to figure out, am I the good or bad version of me? But I try to show up to actually figure out can I speak in a way that I actually can bridge both sides of it. At least to both contextualize why I’m saying what I’m saying and try to elicit what’s holding us back from doing those things. And I think being in multiple seats helps us see the constraints of each party along the way.

Shane Hastie: Thank you for that. So culture. It’s a wide ranging term. With Jellyfish with the organizations that you have been a part of and being a founder for, how do you define the culture and maintain the culture?

Understanding culture [18:54]

Andrew Lau: First of all, culture’s a hard word to define. You called it out. On the worst of times people think of it as foosball tables and yoga balls. That’s not culture. We can actually talk about the artifacts and outcomes that it is the unwritten rules. It’s the fabric that holds us together. These are analogies but they’re not actually defining it. So if you ever find a way to describe to me culture in succinct words, Shane, please email me and teach me how to talk about it.

I take it for granted that culture’s vividly important. It is not the foosball tables and yoga balls. I think that’s a media cleansed way of what culture actually is. I actually think of it as the way that we behave. It’s the norms that we actually do things. I do like that it’s the unwritten rules. So we don’t actually have lots of process. It’s process without process. It’s the values, it’s all of those things.

I do think it’s really important. It’s what keeps high performing teams heading in the same direction without a lot of direction and berating. And I actually think that I am definitely a student of it more than anything I know it’s what kept me at companies over time saying that we actually have to develop. So to me personal philosophy, I don’t think culture can be mandated. And we’re definitely in the middle of learning this over the last couple years. So maybe we were actually have a handful of people. Maybe you could pretend that one could mandate culture like dear team, have fun, right? Okay, well guess what? That doesn’t work. Last I checked, no one has fun when you say to have fun. And this is extra true actually at the couple hundred people that we’re actually at now. And so the other thing to recognize that culture can be stagnant, it actually has to evolve because the thing that’s stuck in the past is never going to come with a new team. And teams have to own their culture. And so for me philosophically, culture has to be refreshed and built by the teams. It can’t be proclaimed. It actually has to be organically developed at the team level. And my role in this is to actually shepherd it, to nudge it to be the guardrails. I can tamp down things that are bad but I can still click a fire in things that are good but the kernel of it actually has to start on the team first.

Leadership’s role is to spot things that are good and give them fuel [20:52]

And so my role in this is to guide to spot things that are good and give it fuel. And fuel can be attention, time could be money, encouragement, teaching. Those are all things that I look for because I actually think that’s the most important part of actually that culture does evolve. Culture is present and culture is owned by the team themselves, not proclaimed from somebody from far away.

And so to make it super tangible, I’ll tell a silly story. Really early on in the company someone came in and I don’t know if you’ve ever seen, there’s a YouTube show called Hot Ones about eating chicken wings. And I know this is going to take your podcast in funny directions here, but it’s about chicken wings. And I remember a woman, I’m not going to out her ’cause she’s awesome. She just joined the team and she came to work one day. She’s like, I saw this thing about the show of Hot Ones. I think we should do it at work. And then the conversation was like, no I don’t think we should, I don’t think I’m going to do any work. I think you should do this. What I didn’t mean by that is, it is not like it’s on her, it’s actually it’s about her. And so the encouragement was actually like you should do it.

And so behind the scenes, yes we make sure that we’re buying the hot sauces. We make sure we’re picking up the wings at the grocery store for her and Falafels in her case because she’s vegetarian too and we make sure that she’s sending out the emails to nudge people to show up and to remind them. But she shows up and she throws this amazing party. Everyone had a great time, burnt their faces off, remembers it for years to come. And she did this amazing and trivia. But the key part is it’s about her. It’s about that culture. It’s about the initiative there. And that’s a very minor, silly story. It’s about chicken wings but it’s actually about how we play a role, all of us and actually encouraging each other to bring it forward, to be yourself and to bring it to the team and let it blossom in that way. And I know most of your content on your podcast is actually about engineering and teams and that stuff and we’re talking about chicken wings. But I actually think culture is a big part of what we do as leaders and teams. That’s the way we think about it at our house.

Shane Hastie: Another part of culture is rituals. How do we define build and allow these rituals to be positive experiences for our teams?

Rituals as a part of culture [22:56]

Andrew Lau: I think rituals go part of culture. It’s part of the lore of being there, part of why you’re there. It also makes you feel at home, makes you feel part of the team. It’s part of the identity along the way. I think they are important. And I go back to my matching point around culture, which is I think they evolve organically.

Rituals are almost the best things that we remember from our past that we keep doing forward. I think can’t pre-think a ritual before it actually happens. So you can’t proclaim culture from on high, it doesn’t work. Rituals are the things that happen to survive because people like them. I’ll share a fun story. One of our rituals at our team is that if you look from the outside you would say, oh you guys do an all meeting every Thursday at 4:00 PM. I was like, yeah, we kind of do but we don’t call it that. It has a strange name. It’s called demos. Okay, you’re like demos, what is that?

So it’s an outgrowth. It’s an outgrowth of when we were under 10 people, we used to sit around a big red table. We were probably seven folks we’re all working on different things and we’d sit around a table and we used to demonstrate what we’re actually working on with each other just every Thursday. So we sit down, sit around table, grab a beer maybe and just pass the screen share around and just show what we’re actually working on. And back then it was probably more about code, more about product. And that evolved as we grew too as we actually added a sales team, a finance team, marketing team. And it evolved to being something that we actually, I took turns sharing what are we doing in different departments. I remember actually saying, hey, here’s a new legal contract we actually showed and my co-founder, David, used to make fun of me. You just showed a word document that doesn’t count. But it was actually good.

People actually found value in actually learning what each of us actually working on. And there are times when the world is tough that we take a moment to actually share that hey, here’s our bank account balance, we’re going to be okay team. But either way it became a thing that we rotated. And so we still continue that ritual today at 200 people and we now go department to department and we actually pass the mic and each week we describe what we’re actually working on. And for my turn when I get to do it, we still have tradition which is every board meeting we do, which is every quarter. We then run the Thursday after a board meeting. We run the same slides with the whole team and for two reasons. One because I believe it’s deeply aligning to actually understand what we’re marching towards and why as a team. We execute better for it.

But also back to the term culture. I think we have a culture of curiosity in our team. We have a culture of actually curiosity, especially around entrepreneurship. Not everybody wants to be an entrepreneur tomorrow, but they’re all really curious about it. And so we’re there to demystify what actually happens. When I was earlier in my career, it’s like what happens in these boards? It’s really scary, complicated. It’s not. We have to approve things. We have to legal stamp things. But it is I think valuable to actually unfurl those things. And so that’s an example of a ritual that we took from when we were smaller to when we’re actually bigger because people still wanted it. We still check in every Thursday. It’s like, do you still want this? We still keep doing that, right? You made me think of recently I’ve been unfurling, where did this ritual come from?

And so my co-founders and I met 23 years ago. They hired me in 1999 and I realized that this whole demos concept actually goes back to a thing we used to do. That ritual back at that company we used to call dogs and demos. We would do hot dogs and demos, same concept. And I realized that that actually concept itself was actually born in a company that I was actually with when I crossed paths. My co-founder, Dave, in 1999 called Inc. to me, which became Yahoo’s search when they used to demos on a monthly basis, similar thing, grab a beer and watch a demo.

And I learned as I pulled that thread, that ritual actually goes all the way back to the mid nineties to SGI. So silicon graphics, so Shane, you and I can date ourselves that we can nod at each other and actually know what SGI was. Those were the super coolest workstations back in the day. It is super cool to, for me to actually realize that these rituals do carry on from generation to generation just because when things work, you keep doing them right? And we learn from that part of it. And that’s how rituals actually established because we propagate forward what worked. And you can’t proclaim that beforehand. You have to do it post facto.

Shane Hastie: Andrew, some really interesting conversations and great stuff here. If people want to continue the conversation, where do they find you?

Andrew Lau: Well, you can find me on LinkedIn. You can find me on Twitter. I guess the handles amlau, A-M-L-A-U and please come check us out jellyfish.co. We’re hiring. Anyone actually that has contributions around culture or metrics or thinking of engineering leadership and how that actually makes a dent in the world. We’d love to meet and trade notes. We’re here as students of all of us and love to learn from you. So, Shane would love the chance to actually meet some of your crew here and really appreciate you having us here.

Shane Hastie: Thank you so much.

Mentioned