Improving the lives of those living with a mental health condition

Situation

Mental Health is a significant challenge for health services in the Netherlands. Depression has the highest burden of disease of all health conditions (8.2%), including cardiovascular diseases and cancer (1). To address this growing demand, professionals have identified that digital health can be useful in treatment, enhancing and extending services.  But they haven’t used it extensively, as there has been uncertainty around which products are good. Professionals wanted digital health to be assessed before recommending it to patients. 

Yet at the same time, people experiencing mental health conditions have been increasingly using apps. This has typically been approached informally by trial and error or asking peers. But this approach takes time, requires persistence and isn’t informed by evidence. 

So in 2019, the Ministry of Mental Health Services put in place the funding to establish a programme to evaluate health apps, putting in place the tools to enable digital health to be safely part of mental health recovery services. With this funding at the end of 2019 ‘MIND’ – in co-creation with ‘de Nederlandse ggz’ – started the development of www.ggzappwijzer.nl, which was eventually launched in the beginning of 2021. 

Approach

To be guided by evidence MIND reviewed scientific papers to understand if apps really do help. Once it was confirmed that there are good apps that do provide a positive impact, they researched how people find and choose apps. 

As there is very little research into how people choose health apps, the team conducted experiments themselves. They discovered that searches result in hundreds of app suggestions, with no trusted information to indicate which are good.

Focus groups revealed the information people want when choosing an app. They don’t just want a score, they want the facts behind this. Basically they want to know three things:

1. Is it useful for me? There is no perfect one app that meets all needs for all people, so people want to understand if it will help their specific need.  
2. Is it safe? Most people care if their personal information will be secure. 
3. Can I trust it? As health is important, people want to know if an app has been checked for clinical assurance.

Also people are interested in the opinion of others about an app. Do peers and/or professionals give some kind of appreciation or depreciation of an app?

Whilst researching the sector, the team learned about the Organisation for the Review of Care and Health Apps (ORCHA), and the hundreds of assessments it had already conducted on mental health apps. The team felt that the ORCHA assessment was objective and thorough; they liked how the assessment publishes not only a score but also the additional information sought. Further research revealed that every question the MIND team had developed to assess an app, ORCHA had already asked it. In fact ORCHA asked more questions the team couldn’t possibly answer within a reasonable balance between costs and benefits.

ORCHA’s comprehensive base of reviews, together with its work with the NHS, led MIND to work with ORCHA. This collaboration was more appropriate for the programme than trying to reinvent the wheel. 

The programme decided to utilise ORCHA’s objective reviews and layer on top its own qualitative assessment information. For this additional layer, MIND arranged for 4 to 8 people to test an app for two weeks, after which each participant answers a structured questionnaire containing 20 (of quantitative and qualitative) questions. Questions ranged from ‘How easy was this app for you to keep using?’, ‘To what extent would you recommend this app to other people?’ to ‘What improvements do you suggest to improve the app?’. Participants included both professionals and people with mental health conditions. The feedback has been consistent from both groups, each bringing a different but each other reinforcing perspective. The individual testing reports were transformed by the MIND team into one consolidated summary, which functions as a preview of what someone can expect from this app.

Results 

As part of this programme, almost 100 digital health products have been through the ORCHA evaluation and 20 of these have been through the additional MIND structured questionnaire layer. It is hoped this second figure will reach 50 by the end of this year. The evaluated products are hosted on a website that can be easily searched by everybody who is interested in mental health apps. The website is also useful for health care professionals to give advice about using a mental health app.

Since its launch in January 2021, the site has been visited 35,000 times and feedback has been extremely positive. People welcome the trusted information on the apps that are in the library. This has been especially important during lock-down, when an increasing number of apps from un-reputable sources have been launched.

To reach more people, the library of reviews is now also integrated into QULI, one of the major personal health records in use in the Netherlands, as part of the Medmij programme.

The combined ORCHA assessment and MIND reviews have also enabled the team to have informed discussions with developers of digital health. They have provided developers with clear and actionable advice on how their product can be improved.  

Commenting on the programme, Rimmert Brandsma, Project Leader, eHealth, MIND, said: 

“The relationship between MIND, de Nederlandse ggz and ORCHA has also been very successful. There is huge synergy and I feel the end product is better having all partners working actively together. 

“We look forward to the next steps, of making multilingual versions of the website and increasing the number of apps assessed, possibly including AI, VR and wearables into the assessment structure. This will help us to support even more people to find the best digital health.” 

(1) https://www.oecd.org/els/health-systems/MMHC-Country-Press-Note-Netherlands.pdf