EVALUATION IN NHS24
The project “Health in my Language” is now in its third year. As one of the partners, NHS 24 has responsibility for the programme's evaluation work package in Scotland. James Sheary, NHS 24 Partnership and Engagement Officer leads on this evaluation which has focused on Polish and Romanian translations.
In Scotland, the Polish and Romanian communities are very different. The Polish community has become well established over the last 20 years, with a large number of Polish people coming to Scotland to work and study. Most Polish residents are bilingual, speaking Polish and English. The Romanian community, on the other hand have not been as well established, and literacy levels are extremely poor. However, both communities have not integrated very well into Scottish society and statistics show a lack of understanding of the health care system.
In year 1, the evaluation consisted of one exercise: Fine Semantic Analysis. This was carried out by professional translators, due to the complexity of the task. In year 2, the system was at a stage of development that allowed for non-professionals to be involved in the evaluation.
Year 2 evaluation
The first task was to identify the target audience. Having done this, James researched how best to engage with the Polish and Romanian communities. He contacted a number of third and public sector organisations who support the Polish and Romanian communities, as well as schools and universities with a large number of Polish and Romanian students. He then set out to explain to the organisations what HimL is and what the benefits are. Service users were then encouraged to contact James. He then either instructed or met with participants to carry out exercise sessions.
Evaluation excercises for Year 2:
In this task participants were shown a number of sentences taken from different sources of health information and machine translated using different translation engines, and the HimL translation engine. Evaluators were asked to rank the accuracy of each of the translations.
Polish evaluators were mostly educated to university level and computer literate. They required very little assistance to complete the task.
Romanian evaluators were mostly educated to secondary school level and computer literate. Some required additional assistance in carrying out the task.
The Gap-Filling exercise would have consisted of a number of individual statements/paragraphs, taken from sources of health information such as NHS Inform and Cochrane, and translated using the HimL engine. A brief statement would follow with key words omitted. How well the annotator was able to fill the gap would indicate how accurate the source paragraph was. This was postponed for year 2.
3. Online user survey
Designed on SurveyMonkey, this 10 question survey consisted of 2 parts. The first part asked about how people access and use of health information and NHS services. The second part included links to test website with health information from www.nhsinform.scot translated using the HimL translation engine. Respondents are asked to access the site and feedback on the accuracy of the translations.
Launched earlier this month (April 2017) this survey will be live for around 10 weeks. It has been disseminated to a wide range of third sector and public sector organisations that support the Polish and Romanian communities such as Glasgow Health and Social Care Partnership, Renfrewshire Polish Association, NHS Scotland Territorial Health Boards.