Case Study: Should an Algorithm Tell You Who to Promote?

February 28, 2018 Jeffrey T. Polzer

feb18-28-nicholas-blechman-algor
nicholas blechman for hbr

Aliyah Jones was having trouble paying attention to the farewell toasts. Although she was sad to see her longtime colleague, Anne Bank, go, she was more consumed with trying to figure out who should replace her.

As a VP of sales and marketing for Becker-Birnbaum International, a global consumer products company, Aliyah knew she needed a talented marketing director to support her division’s portfolio of 34 products. After working with HR to narrow down the list of candidates, she had two finalists, both internal: Molly Ashworth, a brand manager on her team in the cleaning division, and Ed Yu, a rising star from BBI’s beauty division.

Aliyah liked Molly and respected her work. Two years earlier, Molly had spearheaded a new subscription service for BBI cleaning products, which had, despite a slow start, shown strong growth in the past two quarters. Customers seemed to love the convenience, and the R&D, marketing, and executive teams had gotten excited about the service as a platform to test new offerings. Having mentored Molly through the pitch and launch of the service, Aliyah was intimately familiar with her protégé’s strengths and weaknesses and was certain that she was ready for the next challenge.

But soon after the position had been posted, Christine Jenkins, a corporate VP of HR, had come to Aliyah with Ed’s résumé. Like Molly, he’d joined BBI right out of business school and been quickly tapped as a high potential. He also had his own BBI success story: As a brand manager in the beauty group, he had revived its 20-year-old FreshFace makeup-removal product line, increasing sales 60% in three years. Perhaps more important to Christine, he’d been recommended as a 96% match for the job by HR’s new people-analytics system, which she had championed. (Molly had been an 83% match, according to the algorithm.) The goal of the HR initiative was to use data analytics to inform hiring, promotion, and compensation decisions. Ed was flagged not just because of his stellar performance but also because of his interest in transferring from New York to BBI’s headquarters in London. Aliyah was happy to have two insiders in contention — she had come up the ranks herself at BBI — but that made the decision more difficult.

As the COO made another well-deserved toast to Anne, Aliyah thought back to her interviews with Ed and Molly.

Meeting Ed Yu

“I’m sorry I’m so late,” Ed said, looking a little discombobulated. “My Uber driver insisted he knew a shortcut from Heathrow — but he was wrong.”

It was hard not to draw an immediate comparison to Molly, who was always buttoned up and calm, but Aliyah reminded herself to keep an open mind.

“No problem,” she said. “Shall we get started?”

“Absolutely,” Ed said eagerly.

“Tell me what interests you about this job.”

Ed explained that while he was proud of the growth FreshFace had experienced under his leadership, he was ready for a new challenge. He’d enjoyed diving deep into one product, but felt his skills were better suited for a position that would allow him to work across programs and direct a larger portfolio.

Sharp, clear answer, Aliyah thought. “What have you learned in beauty that would apply here in cleaning?” she asked.

This was an important question. BBI’s executive team had issued an imperative that the divisions share more best practices and improve collaboration. In fact, she was getting pressure from her boss to spend more time with her counterparts in other divisions.

Ed explained how he thought the approach his division had taken to in-field customer research, which he credited with boosting FreshFace sales, could work in cleaning. Partnering with anthropologists was something Aliyah’s team had talked about but hadn’t yet tried out.

He also asked about the new subscription program, referencing a recent white paper on trends in subscription business models. He’d clearly done his homework, was smart and ambitious, knew BBI’s business well, and seemed eager to learn. But his answers and even questions seemed a bit rehearsed, stiff even. Aliyah didn’t sense the dynamism or entrepreneurial mindset that she knew Molly had. Maybe he’s nervous, she told herself. Or maybe that’s just who he is.

Aliyah didn’t doubt he could do the job. But she didn’t feel excited about hiring him.

Molly’s “Interview”

Setting Molly’s interview up for the same day as Ed’s seemed like a great idea when she’d suggested it to Christine and, given the noon time-slot, it had been only natural to meet at their usual lunch spot near the office. But as soon as Aliyah walked into the café, she realized how unfair these back-to-backs were to Ed.

It was impossible not to hug Molly hello and ask for a quick update on her projects and family. They even ordered the same thing: curried egg salad on rocket.

But as soon as the waitress left, Molly got down to business: “I know we e-mail 10 times a day and have lunch often, but I’d like to treat this as a formal interview.”

Aliyah smiled. “Of course.”

As Christine had advised her to do, she asked questions that were the same or at least similar to the ones she’d asked Ed.

“Tell me why you’re interested in this job,” she started. It was awkward. Aliyah knew the answer to that already, but to Molly’s credit, she proceeded as if they weren’t close colleagues. With each response, Aliyah was reminded of the potential she’d always seen in her. She demonstrated deep knowledge of the business and had good suggestions for collaborating across marketing programs and building on the success of the subscription program. She was as polished and thoughtful as Ed, but she also seemed more warm and self-aware.

Knocked it out of the park, Aliyah thought, as they walked back to the office. She almost said it out loud but stopped herself. The decision wasn’t made yet. But looking at the smile on Molly’s face, Aliyah knew her protégé was feeling confident that it might go in her favor.

The Algorithm

The day after Anne’s farewell party, Aliyah met with Christine and Brad Bibson, a data scientist on the people analytics team.

“I know you were leaning toward Molly after we debriefed the interviews,” Christine said, “but we wanted to share some more data.”

Brad handed over two colorful diagrams.

“These are network analyses based on Molly’s and Ed’s e-mail and meeting history at BBI. With their permission and without looking at the content of their e-mails or calendars, we analyzed who they had been in contact with across the firm over the past six months.”

Looking at the diagrams, it was clear right away that Ed was connected to not just his beauty division colleagues but also key people in other groups. In contrast, Molly’s network was concentrated mainly within cleaning products.

“I didn’t know we were doing this kind of analysis,” Aliyah said.

“We’ve just started looking at networks,” Brad said, “and we think they can reveal some useful information.”

“I know one chart isn’t going to sway your decision,” Christine said, “but better to have the information, right? You wouldn’t launch a new product or a new campaign without data; HR decisions should be approached the same way.” It was a pitch that Christine had made countless times while stumping for the new initiative. “We feel confident that decisions made on the basis of our algorithms are reasoned, strong, and less-biased by personal feelings toward employees.”

Brad coughed, and Aliyah noticed he was shifting in his seat. “I assume you agree, Brad?”

“Of course,” he said, watching for Christine’s reaction. “But as a data scientist, I also encourage healthy skepticism. Our algorithm is brand new. It’s been used to inform three promotion decisions so far, but it’s too early to tell how those people are doing in the jobs. I don’t want to give the impression that we’re 100% confident.”

Christine looked annoyed. “I appreciate your caution, Brad, but we’ve heard from the hiring managers that the unexpected recommendations they got from the algorithm really changed the way they thought about the position and the ideal candidate profile. And we’re confident in the data — we’ve been testing the system for months now.”

Aliyah sighed. “It’d be easier for me to trust the algorithm if I understood it better.” She knew she wasn’t alone in her hesitation: Christine’s team had gotten a lot of questions about the methodology, despite the companywide training sessions.

“I’d be happy to talk more about how the algorithm works, but right now, you should focus on the two candidates you have in front of you. The point of the system isn’t to replace your judgment. It’s meant to surface candidates you wouldn’t otherwise know about and help you make a more informed decision.”

“It’ll help you make a less-biased decision too,” Brad chimed in, “by relying more on the data and less on gut instinct.”

Aliyah wondered whether Brad was hinting that she was unfairly favoring Molly for the job. She had worried about that herself and cared deeply about making an objective decision. Would trusting the new system help her do that?

“But the algorithm’s not completely neutral either, right? You’re still relying on information — performance reviews, résumés—that conceivably has bias baked into it,” Aliyah said.

“Fair point,” Christine conceded, “and we’ve worked hard to control for that. But as a data-driven company, we have to extend our approach to the most important part of our business: people.”

“It feels like you’re pushing Ed for this position,” Aliyah said.

“It may seem like I’m promoting my own agenda,” Christine said, “but remember, I have to take a broader view. We ran analysis to show which high-potentials are at risk of leaving BBI, Ed was near the top of the list. There’s unlikely to be an opening in beauty products, and we want to keep him.”

“But what about Molly? She’ll be devastated if she doesn’t get this job, and I’m sure she’ll start looking too.”

“Our analysis didn’t flag her as a flight risk,” Brad said. “But you could be right.”

Decision Time

A week later, Aliyah wasn’t any closer to making a decision. She’d been avoiding Molly and had put Ed’s résumé and Brad’s analyses in a drawer. There was no question that Ed looked good on paper and had interviewed well. He’d impressed her. But she knew intuitively that Molly was ready for the job.

She had to ask herself, though: Did she favor Molly unfairly because of their close relationship? And was Christine right that Molly would stay at BBI even if she was passed over?

Aliyah needed to make a decision. Should she trust the algorithm or her instincts?

Question: Should Aliyah hire Molly or Ed?

If you’d like your comment to be considered for publication in a forthcoming issue of HBR, please remember to include your full name, company or university affiliation, and email address.

Read more...

Previous Article
Study: When CEOs’ Equity Is About to Vest, They Cut Investment to Boost the Stock Price
Study: When CEOs’ Equity Is About to Vest, They Cut Investment to Boost the Stock Price

It’s more (and more-rigorous) evidence of short-termism.

Next Article
Make Tools Like Slack Work for Your Company
Make Tools Like Slack Work for Your Company

Tsedal Neeley, a professor at Harvard Business School, and Paul Leonardi, a management professor at UC Sant...

Get the Latest People Performance & Human Capital Resources

Download