Measuring digital skills
The first report that results from the "From Digital Skills to Tangible Outcomes project" proposes an instrument to measure five types of Internet skills. To come up with such an instrument, we took a critical look at the existing digital skills literature. Moreover, our own experience and work related to digital skills helped us in building an elaborate skills framework including specific skill indictors. Two main theoretical approaches the proposed framework was built on were the skill distinctions as set out by Van Deursen and Van Dijk and measures tested by Helsper and Eynon. For several types of skills proposed in these approaches we were able to define corresponding items. We ensured that all items reflected typical Internet use that everyone might imagine him or herself doing. Furthermore, we avoided contextual items related to specific platforms or activities. This should allow these items to be used for a considerable amount of time because they are not dependent on what type of activity is trending or on new platforms becoming popular. The only exceptions are the items that were introduced regarding mobile skills, as a consequence these items might have to be adjusted or integrated into other skills as mobile platforms become more mainstream. All items used a scale that gave statements about things that a person was able to do with answer formats that ranged from ‘Not at all true of me’ to ‘Very true of me,’ and furthermore included a ‘I do not understand what this means’ option.
After the development of a first full survey instrument, we used a three-fold approach to test the validity and reliability of the latent skill constructs and the corresponding items. The first step consisted of cognitive interviews held in both the UK and the Netherlands. Based on the cognitive interview results, we made several amendments to the proposed skill items to improve clarity.
The second step consisted of a pilot survey of digital skills, both in the UK and in the Netherlands (at this stage we also measured uses and outcomes of Internet use, the results of this will be reported later in 2014). The result of the second step was a final theoretical, empirically and cross nationally consistent framework consisting of five types of digital skills: Operational, Information Navigation, Social, Creative and Mobile skills. We suggested longer scales for most of these, consisting of between six to ten items, and shorter scales consisting of five items. The Mobile skills scale (the least theoretically grounded) consisted of only three items.
We recommend the use of the shorter five item scales in larger research projects that need to include a variety of skills. However, if researchers want to focus on skills only or on a specific skill in relation to other variables, they should use the longer scales which offer more variance and the opportunity to study them in detail by, for example, distinguishing different operational skills. If researchers or evaluators do not care about distinguishing different skills, it would be possible to take two items (the highest loading or most relevant ones) from each short scale and create one ten item scale. However, we strongly recommend against this approach.
During the final step, we examined the consistency of the five Internet skill scales and their characteristics when measured in a representative sample survey of Dutch Internet users. This step revealed that reliability and validity of the scales as well as indicators of convergent and discriminant characteristics were good. As a result of this work, we recommend the use of the Operational, Information Navigation, Social, Creative and Mobile skills scales in general population research. While we believe these scales are a significant contribution to research that measures digital skills, it is important that further research is carried out to understand the different relationships these skills have with one another within different socio-demographic groups. As noted in section 5, we suspect that there might be underlying differences in experience and meaning between these groups that underlie slight variations in the digital skills landscape, even when all the skills are valid ways of classifying the abilities people need to use Information and Communication Technologies.
As noted above, the key purpose of this report is to put forward a set of valid digital skills measures that are of value to survey researchers working in this field. We very much welcome feedback and comments from readers who are interested in testing these scales in a range of countries.