Jobvite – Inside The Job Description Grader With Zach Linder and Morgan Llewellyn

William  00:30

Ladies and gentlemen, this is William Tincup, and you’re listening to the RecruitingDaily podcast. Today we have Zach and Morgan – they are alumni of the podcast. So it’s actually great to actually have them back on. They’re both from Jobvite, of course, we’re gonna be talking about a new product that they’ve created, and that they’re releasing to the market. It’s called a Job Description Grader. And so it’s gonna be kind of a fun conversation, because we’ve done this thing before, and also talking about a new technology that they’ve created. So Zach, why don’t you go first to introduce yourself and Morgan, you? And then what have you talked about Jobvite? And we’ll jump into the Grader?

 

Zach  01:16

Yeah, absolutely. Hey, so I’m Zach Linder. I’m the VP of Analytics and Machine learning at Jobvite and oversee all of our really cool data initiatives, and really try to get data in the hands of our users and customers in ways that haven’t been previously available. Morgan, throw it to you.

 

Morgan  01:38

Guys, yeah, Morgan Llewellyn, Chief Data Scientist here at Jobvite. And I head up the AI team and you know, work with Zach and a lot of other great folks to really help bring the vision of AI into this world and really excited to be here today. Thank you.

 

William  01:55

I love both of y’all’s titles. I am title envious, I have to admit. And real quickly, won’t y’all give us the bubble up on Jobvite?

 

Zach  02:04

Sure thing. Yeah, so Jobvite is a full service talent acquisition platform, we have services like an applicant tracking system, CRM, internal mobility, referrals, management, as well as intelligent messaging that enables our customers to reach out to their candidates via solutions, like texting and using drip campaigns, as well as the complete analytic stack that surfaces all this great data and information that, that you get out of the application as a user and in ways that help you improve your day to day operations.

 

William  02:44

And “comma” and a new product. And, and a Job Description Grader. So let’s talk about the beginning. Where did this idea, you know, without getting into the weeds, where did this idea come from?

 

Zach  03:01

Yes, so last year, Jobvite acquired Morgan’s company, Predictive Partner. And we’ve been firing on all cylinders since the team joined Jobvite. And we were so excited about all the things that we’ve gotten there, they’re cranking out all this great work. And then we’re, we’re trying to figure out how do we get this into the hands of the users faster? Right? And what can we do to show how – how we can really change the way that we’re looking at data and using data and helping us make better decisions. And if we think about the TA space, you know, it’s it’s resumes and job descriptions, and resumes are our personal documents, but job descriptions are very public. And so we thought, let’s start with job descriptions. Let’s look at those. Let’s infer those in ways that, you know, there are definitely some similar products out there, but in ways that might be a little more unique or novel. And let’s see what we can do. And – and before we even put it in the application, how do we get it into the hands of just the broader market and see what the uptake is there and, and really show some of the power and promise that Morgan and team have been building over the last six, eight months.

 

William  04:14

I love that.

 

Morgan  04:16

To piggyback on that, yeah, just real quick, I think there’s a relevant and important point there, right? When we think about the Job Description Grader, there’s, there’s really a mission behind this product, right? We’re releasing it, you know, not just to customers, but to everyone, right? It’s a complimentary product available for everyone. And I think that’s really important because it shows the dedication and the mission, the shared mission that we have at Jobvite over improving D&I and addressing D&I and providing tools in a meaningful way. And I think that’s what is just so heartwarming about this product. And what’s so great about this product is it is complimentary, and it’s our way to try and give back to the community to evolve and further the conversation.

 

William  05:01

Thank you for saying that because I was actually gonna be one of my questions was this within the product and something that’s you have to have to be a Jobvite customer but you already answered that. I want to start with, okay, when we’re grading and when users are grading it, so job descriptions aren’t going away, right? We know this because internally, people if they do if they’re doing linking it to competency and competency models and, and, and compensation and all these other things internally, then the externally facing job description or job ad that becomes a part of that. We’ve got to attract talent to the right things. Historically, these are documents, my hands are bloody. So I’ll just I’ll just be honest, historically, these two documents have not been very well put together, probably wrought with biases, and probably poor grammar, and, and other errors. So let’s take a little walk through, kind of, what does the user do when they go to the job Grader – Job Description Grader do they upload? Or does it? Does it? Is it something that it’s a Chrome extension? How do they – How do they get a grade? on their job description? And then what does it do after that?

 

Zach  06:17

Yes, sir. Thanks. So it’s super easy. And we’ll shoot the link over to you that we can attach after this. But it’s, it’s basically copy and paste, right? So file uploading could be a challenge. And it just, it’s so much easier to copy and paste, right. So finding a job description could come from a web page, could come from a Word doc, could come from a PDF, highlight it, drop it into the body of the page, put a little bit of information around it, like what is the job title? What industry are you specifically looking for? That helps with some of the analysis that Morgan will hit on later, and then you hit “go” basically, and then then we return the score. And we’ve got a few metrics that we look at. Sentiment is one, word count, readability, any insensitive words, as well as both gender and racial bias that might get picked up. So those are the key components that we look at. And, and then based upon how you did on that we provide an overall A-B-C-D-E-F score, and give you the opportunity. If you want to go back and edit in line, we highlight some areas that might have caused a flag of some type, but give you the opportunity to go back and make those changes and rescore it again.

 

William  07:37

So if people are listening, or as people are listening, they’re thinking of Textio they’re thinking of Grammarly, and then they are thinking one step beyond those things, correct?

 

Zach  07:48

Yep, for sure. 

 

William  07:49

And so by returning a grade and also giving recommendations in line, then people can then change that description, and then copy and paste it and then they can bring it somewhere else. And then obviously, post it to Indeed or Ziprecruiter, or wherever they or their own career page, wherever they want to. So first of all, is all that, do I have all that correct?

 

Zach  08:12

You do, I do have to make one minor clarification. We don’t provide recommendations – we’ll highlight the text that we have flagged. We’re saving the recommendations for the in app version. So that’s, that’s the one differentiator there. But yeah, we highlight it all, we make sure that we’ve called out the specific things we tried to be descriptive. In the text, make sure you understand if your word description is too verbose? Or is it not enough words? Or how’s your sentiment? We even use super fun emojis to give you the starry eyes if it’s awesome. So it’s a fun product to use. And it looks fantastic as well.

 

William  08:53

I love that. So one of the things I want to show about as you’re as you’re as you’re, we’re developing this, and Morgan, you kind of hit on this a little bit. But the idea of, you know, reducing bias, you talked about D&I, these job descriptions that we’ve put, you know, forth in the past have been wrought. With biases, though, maybe we knew about maybe we didn’t, but by highlighting by running through the Grader, it’s highlighting some of those things. Can you take us inside the back end of that to, you know, talk a little bit about how it helps people or reduce bias?

 

Morgan  09:29

Yeah, I’d be happy to. But let me let me pass it back to Zach and let’s talk about like a real world. This is an example of us at Jobvite using it and getting insight into bias because I think what we’re doing at Jobvite is, you know, fundamentally different than what a lot of other kind of maybe similar tools might be doing. And so I think kind of highlighting that – that difference and where this you know, where this kind of fits in within the ecosystem is important to have that conversation. So okay, so Zach – Do you want to kind of kick us off with, like how you use it?

 

Zach  10:02

Yeah. So, like most people, probably, once this was out and running, I tested a job description that we were actively hiring for. And so by actively hiring, I mean, the individual that we did end up hiring started yesterday, right? So this is a very real world use case. I run my job description through, and I get a C, which – So first of all, I’m like, What in the world have I done wrong? How’s it a C? like, my sentiment is not where it should be. But I think one of the really interesting parts was that it highlighted particular texts that I wasn’t expecting. And it indicated a bit of gender bias because I was looking for experience with Linux shell scripting. And, and that’s highlighted, and again, indicating a bit of bias and, and so first of all, I call Morgan, like, how in the world is this a C? I’m like, this isn’t a negative sentiment, what’s going on? And, and he basically says, hey, it’s, it’s just the math, right? It’s just, I’m not telling you, you’re good or bad. I’m just saying that this is how you compare to others. But I immediately wanted me to make a better version of this. Right. So how do we go do that? But then all Morgan, talk about like, why is that highlighted in there? And what is what caused that particular shell scripting component to be flagged? 

 

Morgan  11:21

Yeah, so this is where our approach is different from what you might see out there. And we’ve ensured and kind of, you know, other tools in it really is we’re not a, we’re not a dictionary based approach to bias, right? We don’t – we don’t flag a word and say that the word analysis or analyze is a male word versus a female word. We abstract away from that. And what we really do is we look at resumes, and we look at job descriptions. And we say, okay, is this the piece of text in this job description? Is it predominantly mapping to male or female resumes, to different race resumes, you know, different races. And if it’s predominantly mapping to those different races, then we’re able to be able to identify that there could potentially be bias there. And so it’s not, you know, it’s not that a single word is an issue, it’s that the people you’re looking for, right? The people you’re specifically calling out in your, you know, by your skills and your experiences that you’re asking for, those people tend to come from, you know, a more limited pool are some populations with specific characteristics. And that’s really what we’re doing differently is, we’re not living in a dictionary that says this word has been historically, you know, genderized, and therefore, it’s bad. It’s when we look at the population of available candidates and applicants, we see differences in how the skill or the experience, you’re, you’re asking for maps to the underlying population, which is, which is fundamentally different than really important.

 

William  12:53

I like that. First of all, thanks for explaining that. Where does that data come from? The not to get too geeky for the folks that but you’re if you’re basing that on resume data, is that through LinkedIn? Or is that kind of looking at Indeed, and other kinds of aggregators? like where do you? Where are you basing the data of resumes off of?

 

Morgan  13:19

Yeah, so at Jobvite, we have a large population that says – So we have a large population of resumes. So we’re looking at kind of real, underlying, you know, kind of trends in resumes, right across industries, within industries across job titles, and kind of getting back to something that Zach mentioned at the beginning, that’s one of the important things that we’re doing here. When you, when you specify your industry, we can help, you know, guide that, you know, kind of guide your feedback within the industry, particularly the industry like word count or sentiment, right. Some of the, you know, depending on your industry, you might be a more you know, kind of just the facts, industry and that will show up in your results.

 

William  14:03

And so you can anonymously y’all have anonymized the resume data inside Jobvite. And and I say anonymize, you’re not getting down into the name per se, I guess you can use that data. And to then make it personal to the role potentially title of the role in the industry? Is that my understanding that ish perfectly?

 

Morgan  14:29

Yes, it’s less about anonymization of the of the right we’re not doing anything with a specific person, right, looking at the accounts and trends and skills and experiences and you know, how those, you know, and similarity of those things, right? So it’s not about an individual’s resume. It’s really being able to aggregate that information through, you know, Advanced Data Science, AI, and be able to understand while you were asking for, you know, shell scripting, that’s Also similar to these other things, and these other things happen to be, you know, held by males and Zach’s instance, right. That’s really what we’re doing.

 

William  15:08

That makes it really special to me because, you know, by by looking at the data from from an industry, pure industry perspective, again, you know, it’s like, you know, I used to tell the story of HR and finance, right when you’re talking to HR use a lot of adjectives. And when you’re talking finance, don’t. And, and so what I love about this is y’all have y’all have you’re using the data to then again, make sure that people understand that within this industry, within this maybe this job class, this is what’s going to work with this audience.

 

Morgan  15:45

Yes, exactly. Right. Yep. Go ahead, Zach.

 

Zach  15:48

I think that’s absolutely right. It’s that benchmarking, right, because you don’t want to compare job description for a nurse with a manufacturing line operator, right, that they’re gonna be completely different in construct. And they’re really not even the same document. But but you really want to hone in on what are what are similar job descriptions? And then what kind of resumes am I going to see those are going to be similar? And really isolate those and be able to use that benchmarking to understand what are what are your peers in this space doing? And how do I stack up to them?

 

William  16:20

I really like that, I really, I’m glad we unpack that, because I really like that. And that’s just gonna get smarter over time. That ultimately, we’re going to know more about what works. And I say what works is just some amusing a complaint vernacular of just saying, you know, here’s what works here, as opposed to what works over here. So from a process perspective, and second blend that you, guinea pigs yourself, for a position cuz I think this is what every recruiter should do, is before they put it out into the open market, whatever channels that they use, they should, you know, grade it and see where they’re at. And, and I guess that’s, you know, just cut, copy, run the run the, the Grader and then make changes, and then obviously paste that and take that out and then go go with it from there. Where do you see this code? Where do you see? I mean, you know, not, first of all, what you’ve built is fantastic. And I love I love what you’ve built. Where do you see this going next?

 

Zach  17:23

Yeah, so a couple of things. The I think one underlying theme is that this is like any other machine learning AI technology, right? It’s meant to be a tool in the bag. If you get a grade and the grade comes back as B, because maybe you’re just a little more wordy than the than the benchmark is. It doesn’t mean you need to go shave four or five words from your job description, right? This is where the machine tops out its intelligence, and the human intelligence has to take over and say, let’s not trim the words, it’s good enough. This is this is a good job description. Let’s go with that. It’s meant to provide that insight, like I got to see because it wasn’t positive enough. And like what the world how I’m pretty positive guy, what do we need to do to add positivity to this thing, right? And so that to the real world example is this job description. I’m also embarrassed to say that was open for over 100 days, right. And so I don’t know that changing it with or grading it early on, because we didn’t have it ourselves. I don’t know that it would have changed it. But I do know that our new hire started Monday literally yesterday, right? And so like it is a it’s something that you can use to really provide that insight and do the best you can with with what the technology will will give to us. So that’s that’s one part. I think that where this goes next is this is probably about where we’re going to be for for the publicly available version. Right now our attention has been shifted to what do we do in app inside of Jobvite for all of our customers, and really expand on the recommendations and the the suggestions that we’ll provide. So in that use case that we’ve talked about with shell scripting, an example of what the recommendations will be is, here’s some other common languages that are held by a more diverse audience, right. So that way, if we’re being too limiting with some of the terms, and I had no idea that shell scripting would have skewed more male than any other term, but what are some other words and examples of skills that I can include in that job description to be to be more diverse in nature and just be more open and accurate and so that way I’m getting the diverse population that I want to apply and not shutting any doors for anyone. So it’s that type of thing that we really want to add.

 

William  19:35

I love that. So now let’s just kinda pivot to the in app stuff that y’all will you know, obviously, it’s the roadmap and we are developing. I want to ask about your three things in particular. One is multiple languages. As you know, your customers are all over the world. Do you see this kind of going into kind of a future again, the in app, not the free version, but the stuff that you do in app. Do you see this becoming something that’s that you do across multiple languages?

 

Morgan  20:04

So that that’s something I’ll take a stab at. Right? So this is really, I think, another kind of benefit of our – what the underlying population looks like approach – versus a dictionary based approach, right. So if you want to extend a dictionary based approach in other languages, you need to figure out the genderized words, across every language. And there’s some languages out there that don’t have gender in that way. Like they don’t have a concept of a male word and a female word. And so you’ve got to figure out other ways to do it. Our approach because we are basing everything on what’s the underlying resumes and tying back, here’s the language you’re looking for, does it tie back to, you know, different subpopulations in your resumes? It does extend and it extends naturally, you know, just as long as you have sufficient numbers of kind of applicants at that language level. So yeah, it’s a it’s a natural extension.

 

William  20:56

Okay. Good, good, good. And either /or answering this one, cultural differences. Which is separate than languages, obviously. Right. So people that are from different cultures, probably consume content and job descriptions and job ads in different ways. In the future, is there a way to track that? Is there a way to make sure that we’re doing the right thing? By cultural differences?

 

Morgan  21:24

That’s a – that’s a really good question.

 

William  21:27

I don’t have the answer, by the way. 

 

Morgan  21:30

I don’t either. Right. Let me just say, I don’t either. Um, but I think it’s also a really important question. And something we do think about. And one of the reasons why we do think about this is everything that’s going on in the self selection process of, I see a job description, I decided to apply to it. Right. Everything that’s in that job description also impacts things that happened subsequently, think of candidate matching that we’re doing a job, right? How do you take a job description, and then understand if that applicants a good fit for the position based on skills and experience and things like that. So there’s a lot of things that happen after the job description that we need to think about. And so anything that is being done to kind of, you know, impact how those candidates are scored, or narrow the sub population of the kind of irrelevant people, right, you want to broaden that funnel here. And so anything that’s being done, I think, is something we should take a look at. And again, that’s where I think our approach is a little bit different. So being able to map back to these different subpopulations, whether they be cultural, whether they be, you know, ethnic, racial, gender, what have you been able to map? Here’s a job description, right, this particular phrase maps to, you know, kind of these, you know, these demographics, right, our cultural identities, that’s, that is something that, you know, is on the table and something that we are really thinking about, you know, even as basic as, you know, kind of, you know, here within the US think about region, right? From the north, south, east west. Right.

 

William  23:09

Right, the 11 regions, I think they broke it down into 11 territories, or 11 regions of the United States. At one point, I’m thinking, I think it was the New Yorker that did that. But

 

Morgan  23:19

I think you raise a really important question, and I think it just continues to underline inclusion is not a destination, right? We can’t raise our flag and say, done, we have a, you know, we’ve achieved gender and racial, you know, kind of inclusion check mark. Right? There are so – there are so many different dimensions and, you know, considerations to have here.

 

William  23:44

I tend to describe it to people with peeling an onion, the more and it’s just an it’s an endless onion, just, we’re gonna find out more to keep the word keep peeling away, we’re gonna find out more about ourselves and about other people.

 

Morgan  23:56

Exactly. And so what we’ve done here at Jobvite is we’ve developed a framework to allow us to continually add kind of, you know, what rings to the onion in that analogy, right, so that we can continue to peel back and go deeper. We’ve started with race and gender, but now we want to go deeper, we can do more, we should do more. I think as a society, we’ve got to do more.

 

William  24:20

So – So three quick things. One is as an extension of that, as people with disabilities I’ve been doing a lot of work there recently, and not really thinking about it, quite frankly, thinking about people that are blind people, you know, that that, that can’t hear or, you know, go through all the different disabilities, you know, you’re – we’re fixing with the Grader we’re fixing a problem. But we’re also kind of highlighting for folks that with disabilities that there’s there’s still there’s still more to go right. So, again, in app, down the road, you’re probably going to be dealing with that like cultural differences like like multilingual down the road. What is what is the with – First of all, what you’ve created for all recruiters all hiring managers is wonderful. And I want to make sure that that’s live ready to go. People can use it today, right? For sure. Absolutely good. So we’ll link out to that make sure that people have that. In app, you know, as you’ve developed the free version for everyone, and trying to make things better for all of, you know, all candidates and all recruiters, the in application, you Zach, you said that, you know, eventually you really want to have recommendations in the engine inside of that as well. So not just highlighting, here’s, here’s what’s wrong, but also kind of giving them ways to fix it in app, right.

 

Zach  25:44

Yeah, yeah, absolutely. Yeah, I think that’s where the, we don’t want this to be a laborious process where we highlight some things, you go back, you edit, you get the score to get it, right. How do we just highlight it? And here’s some options, click on those options, and your score improves, right? And then how does this not be gaming the system? But how do you just begin to think about these things? More completely. Right. So you’re talking about the the ADA type compliance things? Does that belong on all job descriptions? Maybe? Yes, maybe? No? Did you forget it? Do you need to be reminded to include it on there? Or is that a requirement for the company? Is it not? Like, what are the other types of things? It’s not just about code? It’s not just about the specific words, it’s like about the overall package right? This is your your lens into the world of how you want these candidates to view your company. And it’s it’s not just about the the inclusiveness of the job description. But how do we make it a really good valuable document if you if you think about to like it. Some job descriptions might say that you looking for, you know, a team player or a great work ethic. But I can tell you that nobody’s resume says that they’re 10 out of 10 team player, right? And so is that is that a valuable component to the job description. Maybe it is from a marketing perspective, but it’s definitely not from finding a good candidate, right. And so maybe we could highlight even some of those types of data points on the job description and say that this does not help or hurt your job description as far as being able to match to the appropriate candidates, cuz that’s ultimately what we want, right? We want to use the job description to find those candidates that are good matches, and we got other candidate matching technology on the back end to that will help pair those up. But if you’re putting a lot of words in the job description that don’t help you find the candidate or the candidate find you. Maybe we should question Why are they even there?

 

William  27:42

Right, right. They’re not adding value. So they’re just, they’re just extra. I know, I get it, we went a little bit long. I need to get y’all on to your next thing. First of all, I love what you built for the public. And for all the right – all the reasons that y’all have talked about, but also I can’t wait to kind of see what your customers think of, you know, the in application usage of the Grader and see where they kind of push you to kind of move it next. So guys, you’ve been on the show before. Thank you so much for carving out time. I know you’re busy. And just really appreciate it and thanks for everyone for listening to the RecruitingDaily podcast. 

 

Zach  28:18

Absolutely. We really appreciate the time.