Chef and DevOps at Equifax
Equifax Inc. is a global leader in consumer, commercial and workforce information solutions that provide businesses of all sizes and consumers with insight and information they can trust. Equifax organizes and assimilates data on more than 820 million consumers and 91 million businesses worldwide.
In this article we feature Jim Grill, Senior Director of IT Software Engineering and Automation at Equifax, to talk about how the company is adopting a DevOps culture using Chef.
Equifax has been using Chef since April 2014. It's critical to their release pipeline and for delivering applications and infrastructure rapidly and safely. Given its importance, Jim's team is dedicated to bringing the love of all things Chef, as well as DevOps culture in general, to every operations and development team at Equifax. They're doing that through training, community building, and personal interaction.
"We started with some internal training programs of our own, then brought in Chef to do some training," Jim says. “We also had Chef trainers teach some Equifax people so they could deliver Chef classes themselves. We now have two people who are capable of delivering the Chef Essentials and the Intermediate classes.
"We try to conduct the classes the way Chef does. We keep a class to a dozen people or less—more than eight and less than 12. We hold classes in both our Auburn and Atlanta offices. We always do them in sets of three. If anyone misses a class, they can go to the Auburn or Atlanta office, and there's always that third one to make sure we catch everybody. Every 60 days we start the classes up again."
"We also onboard our development teams onto Chef through our boot camp, which is something we've developed in-house,” Jim continues. “That's where we introduce them to test-driven development, as well. We generally start right from the very beginning with a simple example cookbook that we choose. We walk them through the entire process of how you create a cookbook, test it, how you write the tests, run Test Kitchen, how to commit changes, and how to do pull requests (PRs).
"We build a pipeline for them, and we show them that their PR triggered an automated build. By the time someone takes a look to see if they can merge the PR, automated tests have already executed. If the tests passed, the cookbook maintainers can merge in the changes. We show them how to create a release candidate, how to branch and merge. All along the way, Jenkins is automatically building and testing.
"Boot camp takes about a week or two. We don't put a time limit on it. The ultimate goal is, when you're done, you have your own Chef org, you own your own pipeline, you have your own cookbooks, you've put a cookbook into our private Supermarket, and moved a change all the way into production. You leave with the keys, you drive out the other side, ready to go.
"We've graduated six or seven different teams now. We keep a regular cadence going. There's almost always somebody going through the boot camp. We're an established company with a long history – over 100 years. We have a lot of products so it'll take a lot of work to get around to everybody and modernize everything but that's exactly what we're doing as fast as we can.
"Once you've completed boot camp, you're part of the Chef family. People call you 'Chef,' and say 'Ohai' to you in the hallway."
Jim's team also offers space where people can come and work with them on particular problems and projects. "We have a team room we call the ICU—Intensive Collaboration Unit, and we have extra seats and tables that we call touchdown stations. We have room for 10 or more guests to bring their laptops in there and work with us. They don't really need an appointment or that sort of thing. At any given time, we'll be working with two or three different teams.
"We'll also accept what we call mini-challenges. For example, there are challenges to automate releases or there are challenges to get into Chef. That's usually where people start. Once they're done with us, they can go on to our release management team, who also work out of the ICU. People can put together all the pieces so that they're deploying their infrastructure and their code all at once, in a fully automated way, so we don't have manual deployments and that sort of stuff going on."
Jim's own experience in how to train people demonstrates how much DevOps depends on members in the community sharing with each other. "I'd like to give a little shout out to Target,” he says. “Their dojo really inspired our ICU. We heard them speak in, I think, 2014 and talked to them later about their idea. We were experiencing the same challenges scaling out DevOps practices."
Another of Jim's goals is to build a strong internal Chef community. Jim says, "We use HipChat and have a room that's dedicated to Chef. My team also operates two channels for other purposes, but we get Chef questions there as well. We encourage anyone to join in. That includes people who are already using Chef and people who want to but haven't started yet. One thing I knew very early on was I did not want to be the Chef police. I did not want to be the Chef cop. I want to be an enabler. I want to teach people. I want to give them the keys. I want them to love it. I don't want them to feel like I won't let them do things with Chef. Almost all our repositories wide open. Anybody inside the company can see them and contribute.
"I'm also working on getting a Chef guild started. Everyone who's been to the boot camp and who are active users of Chef will be invited. Other people will be able to come, too. We're going to meet regularly, probably at least monthly. I'm just going to wing it at first and see what people want to talk about and see if we can turn our meetings into a place where people can exchange tips and tricks, ask questions, talk about neat stuff they're doing with Chef, and make important announcements. For example, we have a base cookbook, as I'm sure many organizations do, that has a lot of the fundamental stuff that most corporations are going to have. All our hardening stuff is in there. There are a lot of things you need to know to get monitoring working correctly, and other attributes you have to provide. There are links in the comments to where you can go to see what the attributes are for and why you need them. You get the idea.
"The base cookbook's pretty critical but we haven't been very good about letting people know when there's a new release. We put it in our Chef chat room and we try to send out emails but people are busy. They don't notice or they don't realize that we've done it. Sometimes people will be having a problem and we can tell it's because they're using an old version. That's our fault. I'm hoping the guild will help solve that problem."
DevOps at scale
When Jim thinks about broadening the adoption of Chef and DevOps, one of the issues he has to deal with is that Equifax is a very large company. "I think our size is part of the challenge, and I think it changes what DevOps means to us,” he says. I used to have a very purist idea of what I thought DevOps meant, but I heard Adam Jacob talk about DevOps Kungfu and how it can be different things for different people and I thought, 'OK, that's different. I'm going to check out that repo and put my name in there and I'm going to go think about this because I think he's right.'
"There's just so many people. We're easily outnumbered 50:1, maybe more, by application developers. They're in different countries, they have different cadences, some are Agile, some are not. Some are in line to onboard onto Chef with us, and for some of them, quite honestly, it's going to be a year or two before we get to everybody and maybe longer.
"I think the best way to implement DevOps at this huge company is to give people the freedom and the power to do their jobs. Self-service is a good example. Self-service is the way, I believe, you can really help the development team and help the business when you're severely outnumbered. My teams can't possibly work with everybody as much as we would love to. The best I can do is provide self-service where I can and help them with the tools they need to be successful.
"We also build cross-functional teams for Chef, automation and DevOps adoption, which helps. That's something else we learned from Target. They talked about the scalability problem. You can't have one team do it all. We all know that's wrong. You just end up creating another island, another silo. Target talked about scaling skills in their dojo. My boss and I looked at each other and nodded. We both took the same note at that same moment.
"On the engineering side of IT, we mix people. We put storage people, network people, coders, all on the same teams. They are put in charge of what we call our platforms. Hyper-V is a platform as is OpenStack. We have a few other platforms as well. A platform is infrastructure that's a pattern, that's consumable, repeatable, and that we're good at.
"On the operations side of IT, people are highly specialized. There are systems administrators, there are storage administrators, there are database administrators, there are network administrators, there are highly skilled people for everything. We need experts in operations. It’s on the engineering side that we really focus on cross-function.
"If I had to sum it up, to me, DevOps means enabling people and then getting out of their way."
Next in this series: Bringing IT Talent to Equifax
Learn how Equifax is working with Auburn University to train a new generation of automation engineers.Read the article