Sawtooth Software, Inc. Conference
February 2-5, 1999 San Diego, CA

Matching Candidates with Job Openings Using Web-based Adaptive Conjoint

Introduction
Traditional Recruiting
Futurestep Recruiting: Phase 1 -- Finding Candidates
Futurestep Recruiting: Phase 2 -- Assessment of Candidates
Futurestep Recruiting: Phase 3 -- Placement
How have candidates reacted to the conjoint interview?
Changes required of ACA data collection
    Web Delay
    Reversal of preferences from priors
What lies ahead?

The widespread availability of Web access has led to re-evaluation of traditional relationships between companies and their customers. Industries which have a heavy information component, such as news and financial services, are experiencing large shifts to online delivery.

The business logic of online job search and recruiting for middle-management and professional positions is that the Internet allows Futurestep to use sophisticated assessment methodologies effectively with a large database of management candidates. Candidates, including those who may not be actively looking for career changes, register with Futurestep because of the personal information and feedback they receive. Recruiting professionals manage the process with employers, particularly the job specification step, preparation of an offer and negotiation.

Futurestep was the first company to use ACQNET, a Web adaptation of Sawtooth Software's ACA interviewing module. Over 70,000 ACQNET interviews have been completed to date. In the Futurestep process, direct interview questions are used to identify likely candidates for potential jobs. Structured questionnaires also assess decision-making and communication styles, and career motivators. The ACA results are used to identify potential candidates in the early stages of a search by eliminating those who would be unlikely to accept offers within the hiring company's set ranges: Is this candidate ready to move? Which job factors are the most important to this candidate? The ACA results are also used in the final stages of a search to provide a client company with information about how to tailor an offer that appeals to a candidate's preferences.

Since the analysis of conjoint results is conducted at an individual level, minor adjustments were necessary in the ACA interviewing algorithm and analysis.

Whether traditional or not, recruiting firms generally follow three stages in conducting a search:

• FINDING means sourcing a large pool of potential candidates. It is usually done by industry Researchers. As much as 50% of a firm's resources may be devoted to this stage, which is repeated for each new search conducted.

• ASSESSMENT of candidates means reviewing the pool and understanding the top candidates beyond their objective experience levels. This gives the recruiter a chance to probe and test the way candidates handle themselves in the context of the client's needs.

• PLACEMENT of candidates includes articulating to both the candidate and employer why this is a good match, plus negotiating an offer.

The job search and recruiting business has traditionally relied on a "telephone tree" of Researchers to source potential candidates for job openings. This concentrates over half of the resources of a firm on simply obtaining a list of potentially qualified candidates. In a typical recruiting firm, dozens of Researchers may contact hundreds of candidates by telephone for a given search. Many of the contacts may be wasted, because they surface neither a potential candidate nor a referral to someone else in the telephone tree.

Researchers identify a large pool of potential candidates with qualifications that match the job specifications. The particular skills and experiences required for the job are assessed at this stage.

Next, in the traditional firm, Senior Associates or Partners review the qualifications and select a small number, perhaps 2-6 candidates, for additional interviewing. This is a time-consuming process that usually involves a personal contact with each candidate.

Finally, one or more candidates are presented to the employer, who will often interview several of the candidates. At this stage, the recruiting firm seeks to solidify the match: both sides should be convinced of the appeal. A specific offer will be made, usually with input from the recruiting firm. The inefficiency at this stage occurs when there are protracted, iterative rounds of interviews with client managers or rounds of offer negotiations and/or a candidate ultimately declines the offer -- a "slow-no."

To reduce the inefficiency of the Finding step, a large database of qualified candidates was developed. Futurestep launched a nation-wide radio and print advertising campaign. The Wall Street Journal is an important partner with Futurestep because of its recognition among potential candidates and clients. Within a month there were over 50,000 registrants. These individuals are generally mid-career managers and higher.

Why do candidates come to Futurestep? Because they receive:

•  Complimentary career and market value feedback
•  Guaranteed confidentiality of candidate's information
•  Credibility of Korn/Ferry International and the Wall Street Journal
•  Consideration for exclusive, non-advertised positions from blue chip organizations as well as smaller, high-growth firms.

Some of the feedback from candidates illustrates their level of involvement with the process.

"This whole process ... provides an opportunity to establish a personal relationship in a high-tech world. The business world rarely sees technology as an enabler and technology rarely sees beyond its owns bits & bytes."

"Thank you! What a great way to get to know your business style. The more I know, the better I can communicate my needs in a job interview."

"This is an outstanding and leading edge tool for career development."

Futurestep is part of a sea change in the business of searching for jobs (stated from the employee's perspective) and hiring (employer's perspective). It is changing because of the amount of information available to candidates and employers. As we have seen in other markets, such as home mortgages and foreign exchange, when more information is made available in a standardized format to both buyers and sellers, "liquidity" improves due to a greater transparency in pricing. Transaction costs decrease and there is an overall greater level of transactions.

Also, as uncertainty is diminished due to greater transparency of information, the decision process is shortened. Overall, the effects on the employment marketplace may eventually be as large as the introduction of employment advertising!

In both the traditional and Futurestep approaches there is a requirement to verify the information that has been submitted and check that the candidate in-person matches the description on paper. Futurestep uses a proprietary video-conferencing system to permit recruiters to interview potential candidates in their own homes. To keep the comparison between candidates more consistent, the questions are asked from a common script. In fact, the interviews are recorded and can be used in future searches as well. Once Futurestep has selected one or more candidates to present to the employer, a computer CD is prepared that contains the recorded assessment interviews. In addition to the ACA data collected, Futurestep uses a battery of validated self-assessment instruments to better match personal preferences in decision-making and managerial styles. This data is matched with information provided by hiring managers and potential peers of the candidate who fill out a set of "mirror" instruments prior to the search.

Having identified several potential candidates for a search -- usually 2-3 individuals -- the next challenge for the recruiting firm is to market the job to the candidate, and vice versa.

•  The wealth of standardized information available to the recruiter and employer about each candidate's approach to work environments gives the employer a clear basis for selecting the top candidate.

•  Conjoint utilities provide the values that the candidate places on the most important aspects of the job. This helps the recruiter and employer in several ways:

Having conjoint results also lets the recruiter highlight to the candidate those factors of the job that would be most important to him/her, rather than spending time on unimportant characteristics.

Overall, another feature of having the conjoint data collected in advance is that candidates' answers aren't tailored to their perceptions of a particular job opening. It serves as a natural check on the enthusiasm with which candidates, and sometimes recruiters, approach a search.

When combined with the other structured questionnaires, it gives the candidate and recruiter a richer language with which to discuss a particular job as well as the candidate's needs and values. Since this type of standardized data collection is not typical in the recruitment industry, training for Futurestep recruiters has been built into the process.

The Futurestep conjoint design consists of 11 attributes with a total of 40 levels. The interview itself asks for respondent input just as it would in ACA Version 4 for DOS:

Except for one attribute, all preference rankings are set a priori.
Each attribute is rated on importance using Sawtooth's suggested 4-point scale.
15 pairs are presented, 10 with 2 attributes and 5 with 3 attributes each.
4 calibration concepts are used.

Comments from candidates who completed the conjoint portion show that they are very involved in this part of the process, and generally find it an accurate reflection of how they would make choices. There is little drop-off in participation at this stage. The conjoint is the last portion of the registration, so candidates have generally invested over 45 minutes up to this point. Although exact statistics aren't available, over 90% of respondents who get to the end of the prior section complete the conjoint part too.

The comments below highlight the need for more complete explanations of the conjoint process than might be required in a "normal" market research interview.

"The characteristics described above are the person I am. I did not believe the questions asked could reveal so much. This indeed is very interesting and exciting. Thank you."

"Thanks for the Desired Job Characteristics - a good reflection in my case EXCEPT: major event = too high and making an impact = too low; probably more like 8 and 8 respectively. Thanks again."

"It was surprising and revealing to me that my career questionnaire feedback was in accordance with my career experiences to date. I was very impressed that my job characteristics (as defined by your assessment) is almost a perfect match with the job that I am doing. It was also heartening to read about the job characteristics of positions that I am applying for actually dovetail with my desired job characteristics. As a result, I would say that your evaluations are 'on the money.' Thank you very much."

"I essentially agree with the results and explanations associated with each area except in the section "Desired Job Characteristics." I am disappointed in that my responses indicate that I favor personal wealth twice as much as making an impact. Although achieving wealth is important, I believe that it should follow making a positive impact, especially in a new position. I guess I didn't answer the questions with this in mind. Is it possible to change my responses in order to make this point? I found this to be a very enlightening exercise."

"Very pleased. The analysis "felt" right on target. It also brought up a career option/class that I didn't know existed, which matched really well with the characteristics of my ideal job."

"I really enjoyed the exercise - I think the results are right on and consistent with other profiling techniques I have experienced. In this case, I think the attributes in the "job characteristics" report were well chosen - had to be via focus groups." [Editor: No focus groups.]

"I have some [Job Characteristics] that are different by only 1% or 2%. Are these statistically significant differences?"

"I would like to comment on the "relocation" weighting in the job characteristics section. Because I live within five miles of Chicago, and am willing to work at locations within a 20 mile radius of one of the major employment centers in the country, my reluctance to relocate should not be weighted in the same way as someone who lives in a much less urban area. Does your analysis take geographic location of the respondent into consideration?" [Editor: Yes it does.]

"I somewhat disagree with results of Desired Job Characteristics. That may be how my answers fell out, so to speak, however if I obtain and successfully perform a job that provides more opportunity and responsibility, I, personally, would only assume that I will impact the performance results and the bottom line, in a positive manner, significantly."

"Absolutely amazing! You have drawn a very accurate picture of my personality, characteristics and personal goals."

"It was a pleasant experience registering to Korn/Ferry's Futurestep site on the Internet. I was specially impressed by the immediate "Desired Job Characteristics" feedback upon conclusion of the registration process."

Web Delay
In the DOS version of ACA, calculations can be made after each entry by the respondent. Even on an original IBM PC (8088), there is an insignificant delay as ACA does its computations.

Since ACQNET has avoided client-side execution, the results of each page must be sent to the server for processing, no matter how trivial. This imposes a transmission delay for Web-based ACA of approximately 3 seconds. There are two places in ACA which this affects:

• During the priors section, ACA uses either a preference rating or ranking for levels within attributes. In the DOS version, preference rankings are simplified for the respondent by use of a disappearing list. After a choice is made, that answer is removed from the screen. This makes it possible to ask the respondent to rank order a large number of levels.
  
  To do this on the Web without client-side execution would require a page to be submitted and returned for each judgment. Instead, ACQNET implements preference rankings by asking the respondent to put the rank order (1 to n) into a text box. Rankings on all levels of all attributes are obtained before the page is submitted. While ACQNET allows the researcher to specify either preference rankings or ratings, ratings are recommended if there are more than 3 levels per attribute.

• The pairs section presents a different kind of problem. In DOS ACA, utilities are re-calculated after the respondent enters a judgment on each pair, which can affect the left-to-right balancing of the next pair. However, doing re-calculations during the pairs section requires a balance between the respondent's patience with delays and the additional accuracy obtained if ACA does frequent re-calculations.
  
  ACQNET addresses the need to re-calculate utilities during the pairs without stretching the patience of the respondent. It does so by presenting a variable number of pairs on one Web page -- currently 3 at a time. When the respondent submits a page of answers to pairs, ACQNET re-calculates the utilities and selects the next set of pairs to present.

Reversal of preferences from priors
There is another characteristic of ACA that was not particularly suited to individual level analysis. Preferences for levels within attributes are either set aprior (E.g., Higher cost is always less appealing than lower cost) or asked in the priors questions ("Which do you prefer?"). Answers to the pairs questions, however, can actually lead to utilities that contradict the ordering of preferences from the prior section. In the DOS version, this is permitted and the final utilities sometimes contain reversals. Our experience with aggregate-level analysis of ACA is that such reversals have no measurable effect on simulation results.

In individual level analysis, however, such reversals can become confusing to respondents and analysts. When the utilities for two levels are reversed in preference, ACA will not properly balance a pair that contains those levels. This means that the respondent could be presented with the following type of choice:

In this situation, some respondents pause and ask of themselves (or an interviewer, if present) "There's no trade-off here. What should I do?"

A preliminary analysis of two previously collected ACA data sets was performed to determine how often a pair was presented in which either the Left or Right side contained all the preferred levels. In one ACA study there were practically no reversals: 4 out of a total of 2,595 pairs presented to 145 respondents contained no trade-off between left and right. However in another study there were 102 non-trade-off pairs out of 1,918 pairs judged by 100 respondents. In other words, 5% of the pairs involved decisions where there was no trade-off.

Additionally, respondents seem to use the extreme scale values when rating pairs where there is no trade-off. Out of the 106 reversals noted above, 74% of the pairs involved were rated either 1, 2, 8 or 9. This may indicate that respondents were rating their certainty of preference rather than the strength of preference.

Further analysis is being considered to determine the situations that are likely to produce preference reversals.

For the Futurestep application, a decision was made that the preference rank orders from the priors section were to be accepted. Of the 11 Futurestep attributes, 10 were rank ordered a priori (e.g., more salary is always preferred to less). There was one attribute that required the respondent to make a judgment: whether relocation to a more desirable location was preferred to no relocation. For this application, it was determined that the respondent's judgments of preferences during the priors would be more accurate than preferences derived from pairs.

ACQNET contains an option to restrict the judgments in the pairs section from reversing the ordering of preferences for levels from the priors. If a reversal is found, the utilities for the two levels are averaged, then the tie is broken by adding .01 to the utility for the level that was preferred in the priors.

• Futurestep isn't the answer for all businesses.

Small business employers are least likely to have well-developed job specifications and are also quite cost sensitive. Their adoption of this method for staffing would require an increased level of formalization for the business as well as lower cost delivery by the recruiting firm.

The Futurestep model works well where there is a well-defined skill set or product knowledge required. If a job requires a high degree of personal fit, individualized assessment or personal relationships for hiring, the standardized Futurestep approach is unlikely to be adopted. Examples would be employment by a movie studio, creative fields such as design or advertising, or any job that requires an audition.

• Futurestep isn't for all candidates, but since online job searching significantly expands a candidate's opportunities, it is expected to be adopted with increasing frequency. Candidates will still use multiple methods of job searching, including networking, classifieds and Internet job postings.

• Further development of tools and training for recruiters in the "language of conjoint" will be required. Currently under development with Futurestep is a method to specify jobs in terms of the attributes and levels for each potential candidate. This is starting as a subjective process with a long-term goal of systematizing this analytic step.

• There is also an opportunity to better understand employer's trade-offs for a candidate: what would drive a greater dollar offer, or other benefits? What would make a candidate exceptional -- such that an employer would willingly pay more? ACA has the ability to make the dialogue between recruiter and employer more systematic and less based on assumptions. As conjoint has forced researchers to be explicit about product attributes, so would employers communicate their trade-offs with greater accuracy.

• What's ahead for ACA on the Web? We see several trends:
  ... More access by researchers and non-researchers.
  ... Online design and testing of ACA via the Web is already available. Prototyping and testing can be greatly simplified.
  ... Humongous data sets can be developed, increasing the opportunities for subgroup and segmentation analyses.
  ... When conjoint is used with individual feedback, it places greater demands on researchers to provide clear and meaningful explanations of individual conjoint results.