Understanding Skills Scarcity in An External Market — Part 2
AI Trust and COVID-19 Impact
In Part I of this blog, I introduced the concept of Skills Scarcity, described a data-driven approach to measure its value, and illustrated two exemplary use cases. In Part 2, I will cover the AI trust that we built around Skills Scarcity, as well as sharing some insights on how scarcity values have been impacted by COVID-19.
AI Trust
AI trust is an integral part of any AI asset development. At the end of the day, without human’s trust on AI prediction, recommendation or insights, such AI output has no values. Consequently, to gain end users’ trust in the Skills Scarcity data, and in particular, to help end users better understand the computation logic behind the Skills Scarcity data as well as better interpret the scarcity values, we have provided both AI Explainability and an AI Factsheet. These are two important aspects of AI trust, where AI Explainability helps make the AI output explainable and an AI Factsheet provides facts about the AI model for data transparency.
AI Explainability
As the name implies, AI Explainability answers the question of why: why a particular pair of Country and Job Role Specialty (JRS) has a scarcity value of High?
We achieve AI Explainability by providing a set of reasons that support the predicted Skills Scarcity level in a straightforward and intuitive way. Specifically, we first define the following six categories of reasons based on the factors used in the predictive model, which are easy to understand and also intuitive for end users to resonate with: external supply, pay competitiveness, resource movement, ease in filling role, emerging skill, and external market data.
We then identify evidence from the data which supports the predicted scarcity values along the above six dimensions, based on the behavior patterns and data characteristics associated with different skills scarcity levels learned through model training. As an illustration, one example of such learned pattern could be: The majority of Country-JRS pairs which are tagged with High Scarcity value tend to have a higher “voluntary attrition rate”, as compared to the average value of this feature among all IBM skills.
The output of this process is a list of top three reasons for each Country-JRS pair which support its scarcity value. As an example, the table below shows the top three reasons explaining why the scarcity value for JRS “Data scientist: advanced analytics” in US is High.
In our latest release of the scarcity data, we provided the reasons for scarcity along with the data to business units for their review. We found that the availability of these reasons has greatly improved units’ confidence in the data. In particular, the three benefits that resulted from the explanation of data points (as opposed to our previous release without explanation) are: 1) the whole unit review process has been greatly expedited, 2) far fewer questions or inquiries about the data were received, and 3) far fewer scarcity data overrides were submitted by the units. This shows a huge improvement on user trust and workload reduction over the previous release where such AI explainability was not yet available.
AI Factsheet
An AI Factsheet is a collection of relevant information about the creation and deployment of an AI model or an AI service. Such information can include the purpose of building the model, the intended domain or use cases, datasets that are used for model training, testing and validation, and various aspects about the model such as the input, output, model type, performance metrics and how to deploy the models. Any information related to AI trust, such as the AI fairness assessment on detecting potential bias in prediction, model robustness to adversarial attack and AI explainability, can also be included into the Factsheet.
There are several advantages to document all the information related to an AI model or AI service in one place. On one hand, it allows the model or service consumers to determine if the model is appropriate for their situations. On the other hand, from an AI Governance perspective, it enables an enterprise to specify and enforce policies around how an AI model or service should be constructed and deployed. This can prevent undesirable situations during a model’s lifecycle, such as a model trained with unapproved datasets, models having biases, or models having unexpected performance variations.
Below is an excerpt of the AI Factsheet on Skills Scarcity, which touched upon the purpose, intended domain, and training data aspect of the model.
AI FactSheet of Skills Scarcity
Overview
This document is a FactSheet accompanying the Skills Scarcity offering and aims at increasing trust in AI services through supplier’s declarations of conformity. It documents the process of training the Skills Scarcity model as well as its performance measurement, use cases and explainability.
Purpose
At IBM, we have developed a skill planning platform called Skills Value Framework (SVF). One of its goals is to understand the supply level or scarcity of a skill in the external marketplace, which is referred to as Skills Scarcity. Skills Scarcity has been applied to various enterprise programs to assist decision making on employee skill development and compensation strategy.
…
Intended Domain
Skills Scarcity indicates whether the market supply of a specific skill is abundant or not, and by skill, we mean Job Role Specialty (JRS), as defined in IBM’s Expertise Taxonomy. An example of JRS is “Data Scientist: Advanced Analytics”, where “Data Scientist” is a Job Role, and “Advanced Analytics” is a Specialty.
Skills Scarcity is measured for every JRS in every country. A scarcity value can be Low, Medium or High.
…
Training Data
The training data contains a set of Country and JRS pairs, along with a list of features extracted to represent each pair (see section below), and a label of its scarcity level (i.e. High, Medium or Low) which is annotated by SMEs from Talent Acquisition team. We usually refer to SME annotation as “ground truth”.
If you are interested to learn more about IBM’s AI Factsheet 360 work, please check it out here.
COVID-19 Impact on Skills Scarcity
COVID-19 has brought unprecedented impact to many aspects of our lives. In this section, I will cover the analysis we performed to understand how the pandemic has impacted the Skills Scarcity. The analysis was conducted using the Hiring Difficulty Score (HDS) data provided by an external vendor, which indicates how difficult it is to fill a job position. To refresh your memory, HDS is one of the key inputs to our Skills Scarcity measurement, and is calculated for each pair of Country and JRS. The value of HDS ranges from 1 to 99, where 1 represents the easiest job to fill and 99 represents the most difficult job to fill.
Below are two charts showing the distribution of Country-JRS pairs by the change rate of their HDS values over a one-year period for both pre-COVID and into-COVID time. In particular, the left chart shows the distribution for the pre-COVID period which ranges from Q1 to Q4 2019. Its x-axis indicates the level of HDS value change during this 1-year period and the y-axis indicates the % of Country-JRS pairs falling into each HDS change bucket. As you can see, the majority of the pairs had maintained the same HDS values, i.e. with a 0% value change, from Q1 to Q4, which indicates a fairly stable hiring conditions in the external market in 2019.
Now, if we look at the chart on the right which depicts the picture during the COVID-19 time, i.e. from Q4 2019 to Q3 2020, we notice that this time, around 60% of Country-JRS pairs had their HDS values reduced by 0–50%. This shows a big difference as compared to that in the pre-COVID time.
The question that immediately pops up is, why has the pandemic situation caused the decrease of hiring difficulty in an external market? or alternatively, why it became easier to hire skills during the pandemic? Well, the answer is, it is due to the dramatic drop of company hiring starting from March 2020, as shown in the figure below. Such external hiring freeze has significantly reduced skills demand in the market, which consequently resulted in the decrease of hiring difficulty. While it is true that the overall employees’ voluntary attrition had also dropped during the same period due to the uncertainty of job market, yet such attrition drop was not at the same scale as the decrease of company hiring.
Here are some additional interesting observations from looking at the quarterly HDS data change during the period of Q4 2019 — Q3 2020:
- From Q4 2019 to Q1 2020, 40% of Country-JRS pairs had HDS decreased, indicating the significant impact by COVID-19
- From Q1 to Q2 2020, only about 5% of pairs had HDS changed, which is equally split in both change directions, indicating the sustained impact of COVID-19
- From Q2 to Q3 2020, 14% pairs had HDS decreased, yet 18% had HDS increased, indicating the gradual recovery of hiring from COVID-19. This is consistent with the observation on the rising hiring trend from July to September in the figure above
In a word, as skills demand is an important aspect in determining the level of skills scarcity in an external market, it has thus been significantly impacted by the COVID-19 situation. Yet as companies gradually get back to the normal hiring status, the Scarcity data is also gradually going back to the pre-COVID level.
Conclusion
In this blog, I covered the AI Explainability on Skills Scarcity and described the AI Factsheet built to provide data transparency. I also discussed the impact of COVID-19 on Skills Scarcity data. Nevertheless, whether it is before, during or post COVID-19, Skills Scarcity would always remain to be a valuable and important data input to a variety of enterprise programs including skills planning and compensation investment.
