Usability and Acceptability of ASSESS MS: Assessment of Motor Dysfunction in Multiple Sclerosis Using Depth-Sensing Computer Vision

Background Sensor-based recordings of human movements are becoming increasingly important for the assessment of motor symptoms in neurological disorders beyond rehabilitative purposes. ASSESS MS is a movement recording and analysis system being developed to automate the classification of motor dysfunction in patients with multiple sclerosis (MS) using depth-sensing computer vision. It aims to provide a more consistent and finer-grained measurement of motor dysfunction than currently possible. Objective To test the usability and acceptability of ASSESS MS with health professionals and patients with MS. Methods A prospective, mixed-methods study was carried out at 3 centers. After a 1-hour training session, a convenience sample of 12 health professionals (6 neurologists and 6 nurses) used ASSESS MS to capture recordings of standardized movements performed by 51 volunteer patients. Metrics for effectiveness, efficiency, and acceptability were defined and used to analyze data captured by ASSESS MS, video recordings of each examination, feedback questionnaires, and follow-up interviews. Results All health professionals were able to complete recordings using ASSESS MS, achieving high levels of standardization on 3 of 4 metrics (movement performance, lateral positioning, and clear camera view but not distance positioning). Results were unaffected by patients’ level of physical or cognitive disability. ASSESS MS was perceived as easy to use by both patients and health professionals with high scores on the Likert-scale questions and positive interview commentary. ASSESS MS was highly acceptable to patients on all dimensions considered, including attitudes to future use, interaction (with health professionals), and overall perceptions of ASSESS MS. Health professionals also accepted ASSESS MS, but with greater ambivalence arising from the need to alter patient interaction styles. There was little variation in results across participating centers, and no differences between neurologists and nurses. Conclusions In typical clinical settings, ASSESS MS is usable and acceptable to both patients and health professionals, generating data of a quality suitable for clinical analysis. An iterative design process appears to have been successful in accounting for factors that permit ASSESS MS to be used by a range of health professionals in new settings with minimal training. The study shows the potential of shifting ubiquitous sensing technologies from research into the clinic through a design approach that gives appropriate attention to the clinic environment.


Outline of session
• Training takes roughly 1 hour and is split into four parts:

Simulated examination (45 minutes)
3. Patient management aspects (2 minutes) 4. Pitfalls (10 minutes) • The session can be extended by giving trainees an opportunity to practice at the end.
• The recommended number of trainees is 2-3.

Conceptual introduction
• Introduce ASSESS-MS: a 'talking camera' system intended to support the consistent measurement of movement ability for clinical trials by capturing a series of movements performed by a patient.
• Explain the two issues, relevant for clinical trials, that the device is trying to solve: o High inter-and intra-rater variability when assessments are performed by humans.
High score variance increases the sample sizes required to demonstrate improvements in trials and may also threaten the validity of inferences based on human-assigned ratings.
o Limited granularity of the resulting scores that makes it hard to detect subtle changes in disease state. Detecting small improvements, sooner, would be useful for clinical trials that currently have to run over many months.
• Explain how the system aims to address these issues by offering a reproducible, standardised way to capture movement and analysis techniques that can identify subtle changes in disease.
• Explain the components of the system:

Simulated examination
• Explain that the easiest way to get to know the system is to walk through an examination. Ask for a volunteer and get them to sit on the examination couch.
• Give movement protocol handouts to trainees.
• Orient ASSESS-MS so that both patient and clinician screens are visible to trainees, including trainee taking part in examination.
• Provide a conceptual overview of the examination, with a flow that goes from seated to standing:

Introduction to navigation
• Once positioning is satisfactory, explain principles of navigating using touch interface: o Option to press the Done button on the positioning screen, or o Using the navigation arrows at the top of the screen.
• Explain that arrows are always available to move forwards and backwards as required. Explain that this can be used to skip examinations if they cannot be performed at all.
• Explain that the interface is very sensitive and a light touch is all that is needed.

Introduction/distractions screen
• Explain that there is a video that you should show the patient that summarises the principles of the system. Play the video.
• State that the clinician screen shows a different view and this is of things that are distracting the camera. Explain briefly how shiny things interfere with how K inect sees depth. I l lustrate the effect on the positioning screen using a shiny object and the principle of eliminating objects framed in green.
• Discuss common potential sources of distraction (e.g. wheelchairs, crutches) and strategies for removing them with the patient's permission.

Introduction to examination interface
• Advance to the TAT int roduction screen.
• Explain, and illustrate, how each test is divided into two parts: o An int roductory screen with a video for the patient. The screen also has a small positioning display which should be checked periodically to make sure the patient is still centred.
o One or more test screens to actually perform the test. These are accessed by tapping Begin. Some tests have multiple variants -for example eyes open or eyes closed.
• Explain that the t rainee doesn't need to worry about remembering all the variants, the system will automatically step through them.

Performing an examination (TAT)
• Explain, and perform, the standard sequence common to all the tests, using TAT as the example: o Preparation: explain that for TAT feet need to be off the f loor. Show how this information is displayed on the clinician screen (and on the handout).
o Play the int roductory video. Explain that this can be repeated as many times as required.
o Br iefly check patient's understanding.
o Ensure that the patient is in the neutral starting position. This is always, for seated movements, with hands on lap.
o Click Begin and illustrate how the test variant is clearly shown (eyes opened for TAT).
o Press Record to start the examination. Emphasise the importance of not starting the movement until the beep sounds and the screen border goes red.
o Explain how the test must be performed for 5 seconds. Timing is up to the clinician, although the system provides a timer.
o Press Record again to stop the examination.
• Emphasise the simplicity of the process: the simple sequence that was just completed is the same for all the examinations.
• Show how the user interface has automatically advanced to the next variant of the TAT test (eyes closed). I nt roduce the progress bar.
• Explain, and show, what to do if the recording needs to be repeated. Explain that patients are often keen for their first test and i t is easy to miss the start of the first movement. Discuss the balance of repeating tests to get them r ight but also being sensitive to patients if they are having difficulty understanding the instructions.

Standing and walking examinations
• Address positioning for standing/walking (2.8 and 5m distances, respectively).
• Address importance of protecting patient against falling and possibility of omitting tests if patient is unable to stand/walk at all.
• Int roduce remote control as a way of ensuring clinician can stay with patient to support them if necessary. Explain clinician should always stand and t ravel on patient's left.
• Address importance of protecting patient against falling and possibility of omitting tests if patient is unable to stand/walk at all.

Patient management
• Navigate to the Start screen and explain how to initiate an examination by pressing Start.
• Walk through the patient data screen.
o Explain how patient, location and room numbers will be provided.
o Explain data validation and consequences of incorrect data entry (box is highlighted red, cannot advance to next screen) • Advance to positioning screen which is where t raining started

Pitfalls
• Use the pitfalls PowerPoint presentation to recap and highlight common issues.
The presentation notes detail topics that could be addressed.

SIX SITTING MOVEMENTS (1.9 metres)
All movements begin and end with the hands in the lap.

Ataxia
Stretch out the arms to the side for five seconds.
• The arms should be as straight as the patient is able to make them.
• This test is done with the feet OFF the ground: raise the bed

Finger to Nose Test
Stretch out the arm to the side and touch the nose three times ending.
• The arm should start to the side (NOT the front).
• This test is done with feet ON the ground: lower the bed

Finger to Finger Test
Stretch out the arm to the side and touch the nose three times ending.
• The arm should start to the side (NOT the front).

Drawing Squares
Draw two imaginary squares. Start with the hands at eye level, draw down, out, up, and in.
• Start DOWN, and OUT • Draw four sides of a square only

Turning Pages Forward and Backwards
Turn 3 consecutive pages forward in the text of the book with the left hand.
• Place a notebook in front of the patient.

Drinking From a Cup
Reach out and grasp the cup, take a sip, puts the cup down, puts the hand back in the lap. Repeat.
• Place a small table in front of the patient and place a cup of water that is ¾ full in at arm's length in front of shoulder.

Romberg
Put the feet together, lift arms to shoulder height and turn palms up. Hold for 5 seconds and then ask patient to close their eyes for another 5 seconds. Stand to the patient's left.
• Wave at camera when patient closes eyes

Rotating on the Spot
Turn on the spot in the direction shown.

Hopping on One Foot
The patient tu rns on a forty-five degree angle and hops 10 t imes on the foot closest to the camera.
• Stand to the patient's left the fi rst t ime and to their r ight the second time.

Normal Walking
Walk to the camera, turn around and walk back to the starting position, and tu rn again and walk towards the camera. Stand to the patient's left.

Tightrope Walking
Walk towards the camera, placing one foot in front of the other.
Over the course of six t raining sessions int roducing clinical staff to the ASSESS-MS system, K H had a chance to refine the way in which the principles of M L underpinning the system operation were explained. The motivation for this explanation was to reinforce messages about the importance of movement conduct standardisation that was a main focus of the t raining process.
The same two videos were used to assist explanation, one showing a single depth video, described as "An example of the one of the videos that the t rainees had just captured", and a composite video showing the same movement performed by multiple individuals, described as "How the system sees the world." Over the t raining sessions K H attempted to actively refine the explanation in response to whether, and how easily, each of the t rainees appeared to grasp the concepts being communicated. Strategies that appeared to be less successful were: Using the second panel to explain that the computer had learned the movements of people with a wide range of functional ability; that consequently it knew what it is that people with a score of zero look like, what it is that people with a score of one look like, and so on, and therefore when given data from someone new it can use this information to place them in the appropriate group. This explanation requires an abstraction -the idea that there are characteristics of those with a particular scorewhich did not appear to resonate with trainees. Nor did it directly address our emphasis on the need for standardisation.
Using the first panel to build agreement with the trainees that it is easy for a human to interpret movement because of our familiarity with what normal motion looks like at both an intuitive and a more explicitly descriptive level. We can attend to specific features that fit with our conceptual understanding of how dysfunction can affect the movement, whereas for a computer there would be no sense of 'normal movement' other than that derived from what it has seen of healthy and unhealthy movements, illustrated by the variation seen in the composite second panel.
The most successful strategy involved a highly simplified conception of the ML process: That the ML functions simply by asking the question "Which of these many patients that I have learned from in the past does this new patient most look like?" This was particularly successful in respect of reinforcing the need for standardisation because of the consequence of this form of explanation: "If a patient looks like score X then they will be labelled as X. This doesn't matter if they look like X because they actually have dysfunction or they look like X because they didn't perform the movement correctly." The four trainees offered this explanation appeared to easily grasp their role, and that of the supporting UI, in trying to avoid the second contingency, often completing the sentence as it was being said and nodding in agreement: "We need to make sure when they can't do a movement right, it's because the patient really can't do it, not because they didn't 'get it'."