Approximating true cost of faulty grip control from a true user view / experience [cost versus benefit aspect]

The true cost of a faulty grip of a prosthetic arm/device grip was not established so far [check:link].

We have even contemporary researchers to propose that for some prosthetic arm, “83% accuracy” is somehow “very high”1. To end these fairy tales and unfounded claims once and for all, here is the executive summary for all that find anything TLDR (too long to read):

A useful grip error rate for a prosthetic arm is at or below 0,03%.

The ratio between those researchers’ “very low” (17%) error rates and my own realistic estimate of at least a reasonably low grip error rate (0.03% not even being lush, from a practical view, see below) shows that these particular researchers cited above 2 could be seen as being off in their estimate by a factor of ~566  (17 / 0.03 = ~566).

Now listen up, I am not saying they are off in measuring an accuracy of 83% / error rate of 17%, no, they probably and in all expectation got that measured and documented right, and well documented in their protocols and all – I am saying they are totally off by calling that “very good” or the error “very low”. If anything, their result may be termed “very good in relation to …” or, better, “an improvement over …”, but do not believe for a second that when using a prosthetic arm as unilateral transradial amputee, a prosthetic arm error rate of 17% is very good, good, or even acceptable.

I propose to classify this judgement of “very good” as being off by quite a bit, not being just a wee bit. They say that 17% errors is good when in fact, my counter proposal backed by a whole wake of blood, tears and money as well as cold rational thought (below,..) is that 0.03% error is just about good. So we have 17% on one hand – the researchers – and 0,03 on the other – me, the practician. Of course they are wrong!

So to illustrate this magnitude of deviation in calling 17% grip error “very low”, we may compare car speeds in the light of the current Swiss freeway speed limit (120 km/h). Now, if you were to call anyone going just about ~400 km/h (to propose some speed figure that is relatively fast) in Switzerland “insane”, where the legal freeway tempo limit is nowhere higher than 120 km/h, I could not possibly add anything sensible to that except saying “but they were not even going ~400 km/h, not even ~4’000 km/h, and not even ~40’000 km/h, but actually even more”. Current land speed record (22.06.2021) around ~1200 km / h – but these people propose prosthetic grip error rates as being not just tolerable but good in a range that would claim a land speed of ~68’000 km/h as “reasonable” vice versa a speed limit of 120 km/h. So they clearly assume a world, with physical laws even, that is different from ours – not just by a bit but by dimensions. This is really very simple to visualize.

In summary, that type of wrong estimate would be like having a tempo limit on a freeway of 120 km/h but our researchers would instead going at a speed of ~566 times that much, which would be around some ~68’000 km/h and *then* call that reasonable  – i.e., erring on the side of higher risk by quite a bit, by a gigantic bit even, and then – which tops it off entirely – acting as if we all chill. We all chill dudes, at 68000 km/h in a car on Swiss freeways? That’s what we are?

That is about where it’s at. That is why I say R&D people do live on a different planet — so, maybe we should continue reading after all?

A particularly interesting question is, who lets them claim and propose these things.

Who, possibly by name and affiliation, are the people that are directly responsible for this claim to be printed without annotation or comment?

After all, it is only a prosthetic arm research if the result actually is a prosthetic arm, and not just an overcooked soft spaghetti, to use another analogy.

Because if their sole interest is for me to mainly spend time crashing plates and glasses, or, to crash expensive digital cameras – that can be had without cumbersome fruitless research. Then we can just meet and destroy the targeted items without further ado. I just go in, throw the required items on the floor or even put them in the trash bin as is, and we call it a day. Because once users find out that that is what their prosthetic arm does, and “bionic” myoelectric devices do have such tendencies, this happens [link].

Goal and assumptions

Goal

I want to use a simple to calculate example / model to put prosthetic grip failure and associated cost of dropped items into such a relationship, using assumed but halfway realistic figures, to illustrate:

  • why using (not so much wearing, but actually using) a prosthetic arm can be damaging and costly beyond the damage to the prosthetic arm itself,
  • why the decision to thus not use such a prosthetic arm can be very, very rational,
  • why the massive difference between required and available grip error rates play a considerable role in this context,
  • and why finding other ways to build and test a prosthetic arm are better than what is presented by R&D these days,
  • and last but not the least, why so many users of “bionic” myoelectric arms seem to complain of overuse problems and problems of asymmetry – that can only mean that they may wear but really will not use (as in: for grips) their prosthesis, thus overusing their other arm.

Assumptions

As I try to use a simple model, this comes with a load of restrictions as to generalisation, but it comes with practical advantages.

I will, for purposes of illustration here, assume that

  • all faulty grips result in an object drop
  • all object drops result in object damage unless specified otherwise
  • and if these are my own objects, I lose them and have to pay for them eventually

There are things in daily life where drops are free of charge and harmless. I can drop these, often many times over, and no one cries over spilt milk:

  • Vacuum cleaner hose drops – unnerving but harmless.
  • Clothes that drop during folding or hanging up.
  • Car tires, these may even bounce back up who knows.

Other things, not so funny. From my own experience these are the worst:

  • Dishes.
  • Camera/cell phone.

So let us drop some across 1 year, realistically, and sum up the cost, shall we?

Dish-washer

To approximate true cost of error rates, assume that I unload a dishwasher with 30 items that could break every two days, and each costs 8.00 USD. The total sum of dishes moved per year costs 43 800 USD and dropping 1% of these costs 438 USD.

So the actual cost of lost porcelain and glass in USD is 438 USD per percent of a recognition rate of less than 100%.

This means that an error rate of 21.3% (see study below) costs the amputee an equivalent of about 9230 USD per year.

That is why we recommend R&D to target error rates that range in industrial manufacturing error target ranges of five sigma or six sigma 3.

So a device offering three-sigma grip accuracy (93,3% – error rate of 6,7%) in field use or everday real life application may be entirely out of reach for myoelectric prostheses, and can only be achieved with a body-powered arm (which is capable of doing a lot better than that even), and still will add 2934 USD in wasted porcelain and glass alone to the amputee’s household bill.

Even a four sigma quality (entirely unachievable for myoelectric control) (an error rate of 0,6%) will still set you back by 263 USD in dropped and damaged dishes per year.

Five sigma (99,97% correct recognition – 0,03 % error rate) will cost around 13 USD in dropped dishes per year – that is acceptable but … not perfect. This will cause still almost two broken items per year, it amounts to about eight lost cups or plates or glasses in five years, possibly a whole set of plates or so.

So that is why five sigma is not good but a minimal standard, six sigma would be ok.

Camera

But why not add more real-life aspects to further sharpen the senses.

If I shoot about 340 digital photos a month, quite a realistic number, I just checked, this will be 4080 shots per year. So if my camera costs 400 USD and it dies after 3 falls, then I divide the total shots taken by 300, and multiply the result with 400. That is what 1% error rate of grip control costs me, given above assumptions – camera drops each time the grip does not work – and with this model it will be 5440 USD cost for 1% of error rate at that level of digital image capturing. So a four sigma error for grip controls will cost me an equivalent of 0,6% if I take photos at that rate, still over 3000 bucks. With five sigma, still, about 163 bucks per year in damaged cameras. And I did drop cell phones and cameras and dishes due to prosthesis failure. And it was expensive. And, that is how I know. So why you think I use body-powered arms? Really, seriously. They win Cybathlon 2016, 2020, these considerations .. and you all do not get it. At some stage of arguing this it won’t be me any more.

Because in real life, many arm amputees clearly realize that and stop taking many camera pictures, or they instead start wearing the camera around their neck using a strap. But most online forum discussions were around not at all using a digital camera. At that stage you do not have to worry that 85% or more arm amputees also do not use a prosthetic arm. The only attitude that helps here is Nihilism.

Cost summary

So a manufacturer even managing to offer five sigma grip control errors, that is, errors of 0.03% or less, is not to be overly proud – and yet, this is entirely unthinkable for myoelectric control, none of these studies even manage 1 sigma.

Object / Sigma-level 1 Sigma, 69% error 2 Sigma, 31% error 3 Sigma, 6,7% error / 93.3 % accuracy 4 Sigma, 0,6% error / 99,4% accuracy 5 Sigma, 0,03% error / 99.96% accuracy 6 Sigma, 0.000003% error / 99.999997% accuracy
Dishwasher model, 1% cost 438 USD 30’222 USD 13’578 USD 2’935 USD 263 USD 13.20 USD <0.01 USD
Camera model, 1% cost 5440 USD 375’360 USD 168’640 USD 36’448 USD 3’264 USD 163 USD ~0.01 USD
Sum ~405’582K USD 182’218 USD 39’383 USD 3’527 USD 176.20 USD ~0.01 USD

That means that per 1% of grip error, assuming the error is to let go in 0.5% and to not let go on 0.5%, total annual cost is half of (438+5440) / 2= 5878 /2 = 2939 USD. For easier calculation the rule of thumb may be that 1% of grip error entails a material loss of around 3000 USD per year.

Based on this estimate, 1% of grip error costs the user the equivalent of ~3’000 USD per year.

Realistic daily myoelectric control error rates are around 5 to 30% which – under assumption of consequential use of such a prosthesis for both dishwasher activities and camera use for photography, the annual cost for collateral damage will be 14’695 to 88’170 USD.

In research laboratories, the myoelectric error rates, down to 0,6% with 99,4% accuracy are still deep in the ranges of entirely unacceptable levels of error! No one can pay for that type of lifestyle. There are R&D representatives that state “myoelectric control is here to stay”. A study with a proud report of 81.5% accuracy [1] thus bases on technology that entails an error rate for control of 18.5% costing an estimate, based on these consideration, for collateral dropped item damages of over 55’000 USD. I leave it up to you to think about the question whether that is, in fact, ridiculous.

These people have gnawed on grip error rates of myoelectric controls for over 40 years, with results getting trend-wise even a bit worse (possibly due to increased background chatter and bad shielding, who knows).

The burden of all prosthetic grip control error rates in excess of ~0,03% is carried by the other “healthy” anatomically intact arm.

No wonder some of the dedicated myoelectric arm wearers just have the most extensive overuse problems of their other arm – because one really ends up doing the work with that arm that works reliably, because one won’t crash dishes or cameras costing hundreds our thousands of dollars per year, after all.

So you can get your myoelectric and “bionic” or Hero arm, or even Hy5 or Taska hand, but do not fool yourself – you will pay for faulty grip consequences, one way or another. You either overuse your other arm, or you will experience serious restrictions, or, you will have to accept damaged goods – or all of these.

If the manufacturers respect CE-mark requirements, they will clearly state the limitations of their products in a way that it is very clear and apparent to the user or customer. If manufacturers even respect the client that may have to round up insane payments for myoelectric devices, they will bend over backwards to avoid that the client buys a product that does NOT serve them in a satisfying fashion. If manufacturers continue to claim use advantages of myoelectric technology that is simply not there, then the user backlash will be different, and not so forgiving. If that is new to you, you missed out on the simplest of all add-up calculations. And this all has been going on for a while already. Not all users are deep into technical trouble shooting, many users even do not have any concept or grasp of really why they were sold this expensive technology to begin with, so they keep struggling. In a world where CE-marking lays down the terms so they are respected, only there though, such disappointments will become a rarity.

And do not get me wrong – I am not saying R&D should not try to achieve a benchmark of 0,03% error rate as realistic everyday use grip error rate result, even for myoelectric technology if they think this type of technology is “here to stay”. If they think they can work that out, hey.

Just the last 40 years showed they just cannot work that out, no way, they are not even close. They are off by A MULTIPLICATION FACTOR of over 100. Worse, they think that 17% error rate is “very low”. As illustrated above, that is like saying driving 68 000 km/h on a freeway that has 120 km/h as a limit is very acceptable. What were they smoking? They live on a different planet altogether, that is what is going on.

Depending on what you work, that was clear from day 1. It is sky. It hits you instantly. It is an at once realization. You may only believe differently if you neither try it out nor listen – so I know what it takes to get to the point where you actually believe a myoelectric arm to be better. But inability to listen or understand or experience does not alter significant facts, and so I never felt it necessary to be as wasteful (CO2 still a thing?) as Cybathlon to “show the general public” what does not at all boil down as a difference in subtleties, but as being off, for error rates, by a factor of about ~10’333’333. Why did you think it was cool for me and others to keep feet on table and watch this from far?

But nothing will change. At some stage – right away? after a decade? – we will know also what cost pure irrationality has. We pay the cost for inconsiderate research subject choices. We did, and we still do. Remember (and you read that here first too): chances are super high that nothing will change in the next 20 years there either.

Examples of studies and their control success rates

User costs are estimated from an approach where the design/setup is actually used/employed, and where based on the error rates, items are dropped and damaged. A more realistic assumption would be that a user drops an item up to maybe 5 times, then stops using the device. But having a device that isn’t used because it is so fault ridden defies any purpose. To illustrate a consequential use and its outcome, the true cost of ownership is therefore a sensible estimate.

Only once a designer is held liable for the costs they incur by faulty design, which is possible once we can establish they provided the faulty designs intentionally, they may realize that the path they take is wrong not only for the user but for themselves.

  •  “The average success rates were around 80% for tasks involving basic functions (..)” [2]. From that, estimated annual user cost is around 20 x 3000 ~ 60’000 USD. Ten users wearing a device for ten years in total would cost the designer 600’000 USD in material damage/reparation payments.

 

 

[1] A. M. Simon, K. L. Turner, L. A. Miller, B. K. Potter, M. D. Beachler, G. A. Dumanian, L. J. Hargrove, and T. A. Kuiken, “User performance with a transradial multi-articulating hand prosthesis during pattern recognition and direct control home use,” IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 31, pp. 271-281, 2022.
[Bibtex]
@article{simon2022user,
  title={User performance with a transradial multi-articulating hand prosthesis during pattern recognition and direct control home use},
  author={Simon, Ann M and Turner, Kristi L and Miller, Laura A and Potter, Benjamin K and Beachler, Mark D and Dumanian, Gregory A and Hargrove, Levi J and Kuiken, Todd A},
  journal={IEEE Transactions on Neural Systems and Rehabilitation Engineering},
  volume={31},
  pages={271--281},
  year={2022},
  publisher={IEEE}
}
[2] A. Gigli, “Interaction-Driven Approaches for Efficient and Autonomous Calibration of Myoelectric Controllers,” PhD Thesis, 2024.
[Bibtex]
@phdthesis{gigli2024interaction,
  title={Interaction-Driven Approaches for Efficient and Autonomous Calibration of Myoelectric Controllers},
  author={Gigli, Andrea},
  year={2024},
  school={FAU Erlangen}
}

Footnotes

  1. From Cracchiolo, M., Valle, G., Petrini, F., Strauss, I., Granata, G., Stieglitz, T., Rossini, P.M., Raspopovic, S., Mazzoni, A. and Micera, S., 2020. Decoding of grasping tasks from intraneural recordings in trans-radial amputee. Journal of neural engineering, 17(2), p.026034: Abstract – Objective. A major challenge in neuroprosthetics is the restoration of sensory-motor hand functions in upper-limb amputees. Neuroprostheses based on the direct re-connection of the peripheral nerves may be an interesting approach for re-establishing the natural and effective bidirectional control of hand prostheses. Recent results have shown that transverse intrafascicular multi-channel electrodes (TIMEs) can restore natural and sophisticated sensory feedback. However, the potential of using TIME-recorded motor intraneural signals to decode grasping tasks has not as yet been explored. Approach. In this study, we show that several hand-movement intentions can be decoded from intraneural signals recorded using four TIMEs implanted in the median and ulnar nerves of an upper limb amputee. Experimental sessions were performed over a week, from day 16 to day 23 after the surgical operation. Intraneural activity was recorded during several hand motor tasks imagined by the subject and processed offline. Main results. We obtained a very high decoding accuracy considering 11 class states (up to 83%). These results confirm that neural signals recorded by multi-channel intraneural electrodes can be used to decode several movement intentions with high accuracy. Moreover, we were able to use same TIME channels for decoding over one week within the first month, even if the stability has to be confirmed during long-term experiments. Significance. Therefore, TIMEs could be used in the future to achieve a complete bidirectional approach exploiting neural pathways, to make a more natural and intuitive new generation of hand prostheses that have a closer resemblance to a healthy hand.
  2. From Cracchiolo, M., Valle, G., Petrini, F., Strauss, I., Granata, G., Stieglitz, T., Rossini, P.M., Raspopovic, S., Mazzoni, A. and Micera, S., 2020. Decoding of grasping tasks from intraneural recordings in trans-radial amputee. Journal of neural engineering, 17(2), p.026034
  3. From https://www.brighthubpm.com/six-sigma/84858-three-sigma-vs-six-sigma/:

    • 1 Sigma: 690K errors per million (31% accuracy).
    • 2 Sigma: 308K errors per million (69% accuracy).
    • 3 Sigma: 66.8K errors per million (93.3% accuracy).
    • 4 Sigma: 6.2K errors per million (99.4% accuracy).
    • 5 Sigma: 233 errors per million (99.97% accuracy).
    • 6 Sigma: 3.4 errors per million (99.999997% accuracy)


Cite this article:
Wolf Schweitzer: swisswuff.ch - Approximating true cost of faulty grip control from a true user view / experience [cost versus benefit aspect]; published 30/05/2021, 19:10; URL: https://www.swisswuff.ch/tech/?p=11986.

BibTeX 1: @MISC{schweitzer_wolf_1738963318, author = {Wolf Schweitzer}, title = {{swisswuff.ch - Approximating true cost of faulty grip control from a true user view / experience [cost versus benefit aspect]}}, month = {May}, year = {2021}, url = {https://www.swisswuff.ch/tech/?p=11986}

BibTeX 2: @MISC{schweitzer_wolf_1738963318, author = {Wolf Schweitzer}, title = {{Approximating true cost of faulty grip control from a true user view / experience [cost versus benefit aspect]}}, howpublished = {Technical Below Elbow Amputee Issues}, month = {May}, year = {2021}, url = {https://www.swisswuff.ch/tech/?p=11986} }