menu
The Value of Analysing Qualitative Text-Based Written Scouting Assessments

Football, probably more than any other sport, is a game of opinions.

No matter whether you are a player, coach, scout, manager, chairman or supporter, there is every chance that you are going to interpret key incidents in a game or the performance of a certain player differently based on what you see with the naked eye, which is undoubtedly going to be influenced by pre-existing views and your previous experiences.

However, when you are working in the professional business of scouting and recruitment, assessing the performance of hundreds if not thousands of players every month, what steps can you, as a collective department, take to ensure that every stakeholder in the decision-making process applies the same level of consistency to their assessments, taking into account the club’s key requirements?

Given the significant financial investments that are necessary in the recruitment of a player, the one thing every club must ensure is that there aren’t any individuals involved in the process who apply a different value, whether positive or negative, to certain criteria in an individual assessment. If one scout is very quick to praise, whilst another is more reluctant to give a positive assessment, there is a danger that there will be inconsistencies in the interpretation of reports, which could affect a key decision.

In amongst all of this, the structure of the assessment template is absolutely crucial to ensure you get the desired outcome.

In a wide variety of business sectors, the recruitment candidate selection criteria can incorporate a blend of numerical-based quantitative assessments and subjective qualitative factors, such as a candidate’s performance at a job interview. Whilst the job interview may be less of a factor in football, assessment templates may still comprise a combination the quantitative, for example a numerical questionnaire which position-specific criteria, along with some kind of qualitative written assessment or verdict.

From a quantitative standpoint, it is far easier to make direct comparisons with score-based matrices across several different reports and identify discrepancies in the weighting of individual scores, however is it possible to do the same and identify discrepancies in the words and phrases which are used frequently by scouts when they submit their written assessments?

To try to find out, I have recently completed a project for Scout7, which has comprised a text mining study of old scouting reports compiled by the company’s Consultancy operation. These reports were written several years ago and are now of little value, as all of the players assessed are no longer active in the professional game.

Some of the findings were highly significant and highlight the value of regularly checking the written testimonies entered into free text boxes in assessment templates.

In this Blog, I am going to highlight just a small number of the report findings, which I believe could be the first of its kind to be applied to professional football.

A General Summary of the Study

In total I analysed over 140 reports and each report was encoded so that a scout’s comments were broken down into five categories based on recurring words and phrases:

Findings

Positive vs Negative Mentions, by Category

The first definitive conclusion I found, following the text mining, was a higher proportion of positive mentions in reports, in all categories, compared to negative ones. On average, scouts commented positively three times as much as they did negatively, as highlighted by the table below.

Coding Category

Positive Mentions

Negative Mentions

Positivity Ratio

Composure & Confidence

35

10

78%

Creativity

7

6

54%

Dynamism & Work rate

46

13

78%

Pace

30

18

63%

Passing

47

18

72%

OVERALL

165

65

72%

Positive vs Negative Mentions, by Scout

In terms of making direct comparisons by scout, we were able to unearth some highly informative findings, which suggested large discrepancies in the number of positive and negative comments made by each scout (note that all the names in the table are aliases).

Name

No of reports

Positive Mentions

Negative Mentions

Positivity Ratio

Alex Cole

23

21

0

100%

Chris Hoburn

20

36

5

88%

Terry Dean

12

14

4

78%

Jon Robinson

8

6

2

75%

David Whitehouse

9

10

6

63%

Chris Cunliffe

12

14

9

61%

Alex Dempsey

20

24

17

59%

Simon Timmis

16

12

10

55%

Andrew Armstrong

8

6

5

55%

What makes these findings even more significant is that we also found a direct link between the number of positive mentions and the final subjective ‘rating’ the scout gave a player (ranging from A-D, with A being high). We found that the more positive statements a scout made, the higher the rating he gave the player.

Category Mentions, Broken Down by Position

As well as comparing the positive with the negative, we also studied how often each category was mentioned based on the position of the player each scout was assessing. The results make it clear that the scouts mentioned certain criteria for some positions more than they did for others:

Coding Category

Defender

Forward

Midfielder

Composure & Confidence

33%

16%

33%

Creativity

3%

6%

12%

Dynamism & Work rate

22%

33%

50%

Pace

53%

28%

56%

Passing

42%

11%

52%

As you can see, the scouts referred to pace and passing for defenders and midfielders far more regularly than they did for forwards, whilst dynamism and work rate was mentioned more in midfielder assessments compared to defenders and forwards. Of course these findings may reflect the key attributes of these positions, however from a management perspective it may suggest that scouts need to consciously look into attributes that they have previously neglected in reports.

Scouting Terms Used, by Rating

In addition to categorising the various terms used by scouts, we have also analysed the number of times specific words appeared in reports and how they can varied based on the final A-D rating given by the scout.

Bars represent the % of documents using the term

Once again, you can see there is a marked increase in the number of times terms are used in reports where the scout awards a higher rating. Some terms, notably ‘either foot’ and ‘ability’ are hardly ever referred to in players awarded a ‘C’ grade or below.

Scouting Terms Used by Position

As part of the study, we were keen to see if certain terms were position specific. The following table highlights how frequently terms were used based on the position of the player being assessed.

Bars represent the % of documents using the term

One of the stand out conclusions was how terms relating to possession and movement dominated when scouts were assessing midfielders, whilst several other terms used regularly for defenders and forwards were omitted.

When assessing forwards, understandably the term ‘goal’ scored highly, however a large proportion of all-round terms were also used too. With defenders, key attributes specific to the position appear highly, notably ‘defend’, ‘position’, ‘aerial’, ‘pace’ and the possession based terms.

Conclusion

From studying just a very small sample of written reports, we were able to see fluctuations in scouting terminology being used, both in a positive and negative context, based on the position being assessed, the final rating and the author of the report. We also found differences in the word count from each scout and a correlation between the number of words used and the final rating being awarded, with higher grades having a bigger number of words.

Of course the outcomes of such an exercise can only be properly measured based on the understanding the very specific instructions that were given to the scouts. For example, we have highlighted how the number of positive mentions vastly outweighed the negative, but that may be a direct result of how the scouts have been briefed on what they have been told to report on. The same goes for the specific terms used by position – again the scout may have been told to report on attributes which are specific to position and that may reflect in the final outcomes.

That said, given the importance of good recruitment and the impact a report can have on decision making, there is clearly a value of regularly mining written assessments as part of a wider scouting audit, if only to confirm that each contributor is submitting reports to their brief.

Another key conclusion, if indirect, also highlights the importance of structuring your assessment templates so that you get your desired outcomes. This may relate to how you structure questionnaires, the choice of criteria, the ranges, the questions asked, the choice of answers available and the guidelines in place if you choose to include a written testimony. If you don’t get that right, it can make it very difficult to interpret the information that comes in.

If text mining of scouting reports is of interest to you, or other aspects of scouting analytics including the setting up of templates, please feel free to contact me. All my details can be found on my website, which includes a range of other consultancy services.

If you would like to read my full report into the Scout7 analysis, which encompassing the full findings of the text mining study, please email Scout7 and they will send you a copy via post.

Dr. Garry Gelade is a Member of the American Psychological Association, and a Charter Member of the British Psychological Society.

He is the founder of Business Analytic, an independent consultancy specializing in organizational research, sports analytics and applied statistics. He is an Associate of the Centre for Research in Applied Creativity, Ontario, a past Visiting Researcher at City University (Cass) Business School, and the author of twenty-seven peer-reviewed scientific articles.

by Dr Garry Gelade (Guest Blog) Founder & Director, Business Analytic

Published 29 January 2016

Tags