Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Letter
  • Published:

Cortical control of a prosthetic arm for self-feeding

Nature volume 453, pages 1098–1101 (2008)Cite this article

Abstract

Arm movement is well represented in populations of neurons recorded from the motor cortex1,2,3,4,5,6,7. Cortical activity patterns have been used in the new field of brain–machine interfaces8,9,10,11 to show how cursors on computer displays can be moved in two- and three-dimensional space12,13,14,15,16,17,18,19,20,21,22. Although the ability to move a cursor can be useful in its own right, this technology could be applied to restore arm and hand function for amputees and paralysed persons. However, the use of cortical signals to control a multi-jointed prosthetic device for direct real-time interaction with the physical environment (‘embodiment’) has not been demonstrated. Here we describe a system that permits embodied prosthetic control; we show how monkeys (Macaca mulatta) use their motor cortical activity to control a mechanized arm replica in a self-feeding task. In addition to the three dimensions of movement, the subjects’ cortical signals also proportionally controlled a gripper on the end of the arm. Owing to the physical interaction between the monkey, the robotic arm and objects in the workspace, this new task presented a higher level of difficulty than previous virtual (cursor-control) experiments. Apart from an example of simple one-dimensional control23, previous experiments have lacked physical interaction even in cases where a robotic arm16,19,24 or hand20 was included in the control loop, because the subjects did not use it to interact with physical objects—an interaction that cannot be fully simulated. This demonstration of multi-degree-of-freedom embodied prosthetic control paves the way towards the development of dexterous prosthetic devices that could ultimately achieve arm and hand function at a near-natural level.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Figure 1: Behavioural paradigm.
Figure 2: Unfiltered kinematic and spike data.
Figure 3: Movement quality.
Figure 4: Unit modulation.

Similar content being viewed by others

References

  1. Georgopoulos, A. P., Kalaska, J. F., Crutcher, M. D., Caminiti, R. & Massey, J. T. in Dynamic Aspects of Neocortical Function (eds Edelman, G. M., Gall, W. E. & Cowan, W. M.) 501–524 (Wiley & Sons, New York, 1984)

    Google Scholar 

  2. Georgopoulos, A. P., Kettner, R. E. & Schwartz, A. B. Primate motor cortex and free arm movements to visual targets in three-dimensional space. II. Coding of the direction of movement by a neuronal population. J. Neurosci. 8, 2928–2937 (1988)

    Article  CAS  Google Scholar 

  3. Schwartz, A. B. Direct cortical representation of drawing. Science 265, 540–542 (1994)

    Article  ADS  CAS  Google Scholar 

  4. Schwartz, A. B. & Moran, D. W. Motor cortical activity during drawing movements: Population representation during lemniscate tracing. J. Neurophysiol. 82, 2705–2718 (1999)

    Article  CAS  Google Scholar 

  5. Moran, D. W. & Schwartz, A. B. Motor cortical activity during drawing movements: Population representation during spiral tracing. J. Neurophysiol. 82, 2693–2704 (1999)

    Article  CAS  Google Scholar 

  6. Moran, D. W. & Schwartz, A. B. Motor cortical representation of speed and direction during reaching. J. Neurophysiol. 82, 2676–2692 (1999)

    Article  CAS  Google Scholar 

  7. Wessberg, J. et al. Real-time prediction of hand trajectory by ensembles of cortical neurons in primates. Nature 408, 361–365 (2000)

    Article  ADS  CAS  Google Scholar 

  8. Wolpaw, J. R., Birbaumer, N., McFarland, D. J., Pfurtscheller, G. & Vaughan, T. M. Brain-computer interfaces for communication and control. Clin. Neurophysiol. 113, 767–791 (2002)

    Article  Google Scholar 

  9. Schwartz, A. B. Cortical neural prosthetics. Annu. Rev. Neurosci. 27, 487–507 (2004)

    Article  CAS  Google Scholar 

  10. Leuthardt, E. C., Schalk, G., Moran, D. & Ojemann, J. G. The emerging world of motor neuroprosthetics: A neurosurgical perspective. Neurosurgery 59, 1–14 (2006)

    Article  Google Scholar 

  11. Schwartz, A. B., Cui, X. T., Weber, D. J. & Moran, D. W. Brain-controlled interfaces: Movement restoration with neural prosthetics. Neuron 52, 205–220 (2006)

    Article  CAS  Google Scholar 

  12. Kennedy, P. R. & Bakay, R. A. E. Restoration of neural output from a paralyzed patient by a direct brain connection. Neuroreport 9, 1707–1711 (1998)

    Article  CAS  Google Scholar 

  13. Kennedy, P. R., Bakay, R. A., Moore, M. M., Adams, K. & Goldwaithe, J. Direct control of a computer from the human central nervous system. IEEE Trans. Rehabil. Eng. 8, 198–202 (2000)

    Article  CAS  Google Scholar 

  14. Serruya, M. D., Hatsopoulos, N. G., Paninski, L., Fellows, M. R. & Donoghue, J. P. Instant neural control of a movement signal. Nature 416, 141–142 (2002)

    Article  ADS  CAS  Google Scholar 

  15. Taylor, D. M., Helms Tillery, S. I. & Schwartz, A. B. Direct cortical control of 3D neuroprosthetic devices. Science 296, 1829–1832 (2002)

    Article  ADS  CAS  Google Scholar 

  16. Carmena, J. M. et al. Learning to control a brain-machine interface for reaching and grasping by primates. PLoS Biol. 1, 193–208 (2003)

    Article  CAS  Google Scholar 

  17. Musallam, S., Corneil, B. D., Greger, B., Scherberger, H. & Andersen, R. A. Cognitive control signals for neural prosthetics. Science 305, 258–262 (2004)

    Article  ADS  CAS  Google Scholar 

  18. Wolpaw, J. R. & McFarland, D. J. Control of a two-dimensional movement signal by a noninvasive brain-computer interface in humans. Proc. Natl Acad. Sci. USA 101, 17849–17854 (2004)

    Article  ADS  CAS  Google Scholar 

  19. Lebedev, M. A. et al. Cortical ensemble adaptation to represent velocity of an artificial actuator controlled by a brain-machine interface. J. Neurosci. 25, 4681–4693 (2005)

    Article  CAS  Google Scholar 

  20. Hochberg, L. R. et al. Neuronal ensemble control of prosthetic devices by a human with tetraplegia. Nature 442, 164–171 (2006)

    Article  ADS  CAS  Google Scholar 

  21. Wahnoun, R., He, J. & Helms Tillery, S. I. Selection and parameterization of cortical neurons for neuroprosthetic control. J. Neural Eng. 3, 162–171 (2006)

    Article  ADS  Google Scholar 

  22. Santhanam, G., Ryu, S. I., Yu, B. M., Afshar, A. & Shenoy, K. V. A high-performance brain-computer interface. Nature 442, 195–198 (2006)

    Article  ADS  CAS  Google Scholar 

  23. Chapin, J. K., Moxon, K. A., Markowitz, R. S. & Nicolelis, M. A. L. Real-time control of a robot arm using simultaneously recorded neurons in the motor cortex. Nature Neurosci. 2, 664–670 (1999)

    Article  CAS  Google Scholar 

  24. Helms Tillery, S. I., Taylor, D. M. & Schwartz, A. B. The general utility of a neuroprosthetic device under direct cortical control. Proc. 25th Annu. Int. Conf. IEEE EMBS 3, 2043–2046 (2003)

    Google Scholar 

  25. Brockwell, A. E., Rojas, A. L. & Kass, R. E. Recursive bayesian decoding of motor cortical signals by particle filtering. J. Neurophysiol. 91, 1899–1907 (2004)

    Article  CAS  Google Scholar 

  26. Sanchez, J. C., Erdogmus, D., Nicolelis, M. A. L., Wessberg, J. & Principe, J. C. Interpreting spatial and temporal neural activity through a recurrent neural network brain-machine interface. IEEE Trans. Neural Syst. Rehabil. Eng. 13, 213–219 (2005)

    Article  Google Scholar 

  27. Yu, B. M. et al. Mixture of trajectory models for neural decoding of goal-directed movements. J. Neurophysiol. 97, 3763–3780 (2007)

    Article  Google Scholar 

  28. Schwartz, A. B., Taylor, D. M. & Helms Tillery, S. I. Extraction algorithms for cortical control of arm prosthetics. Curr. Opin. Neurobiol. 11, 701–707 (2001)

    Article  CAS  Google Scholar 

  29. Morasso, P. Spatial control of arm movements. Exp. Brain Res. 42, 223–227 (1981)

    Article  ADS  CAS  Google Scholar 

  30. Soechting, J. F. Effect of target size on spatial and temporal characteristics of a pointing movement in man. Exp. Brain Res. 54, 121–132 (1984)

    Article  CAS  Google Scholar 

Download references

Acknowledgements

We thank S. Clanton and M. Wu for help with software and hardware development, S. Chase for discussions and I. Albrecht for the illustration in Fig. 1a. Support contributed by NIH-NINDS-N01-2-2346 and NIH NSØ5Ø256.

Author information

Authors and Affiliations

  1. Department of Neurobiology, School of Medicine, E1440 BST, Lothrop Street, University of Pittsburgh, Pittsburgh, Pennsylvania 15213, USA,

    Meel Velliste & Andrew B. Schwartz

  2. Department of Bioengineering, 749 Benedum Hall, University of Pittsburgh, Pittsburgh, Pennsylvania 15261, USA,

    Sagi Perel, M. Chance Spalding, Andrew S. Whitford & Andrew B. Schwartz

  3. Center for the Neural Basis of Cognition, University of Pittsburgh and Carnegie Mellon University, Pittsburgh, Pennsylvania 15213, USA,

    Sagi Perel, M. Chance Spalding, Andrew S. Whitford & Andrew B. Schwartz

  4. Department of Physical Medicine and Rehabilitation, University of Pittsburgh, Pittsburgh, Pennsylvania 15213, USA,

    Andrew B. Schwartz

  5. McGowan Institute for Regenerative Medicine, University of Pittsburgh, Pittsburgh, Pennsylvania 15219, USA,

    Andrew B. Schwartz

  6. Robotics Institute, Carnegie Mellon University, Pittsburgh, Pennsylvania 15213, USA,

    Andrew B. Schwartz

Authors
  1. Meel Velliste
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sagi Perel
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. M. Chance Spalding
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Andrew S. Whitford
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Andrew B. Schwartz
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Andrew B. Schwartz.

Ethics declarations

Competing interests

The authors have applied for a patent on parts of the work described in this paper.

Supplementary information

Supplementary information

The file contains Supplementary Methods, Supplementary Data, Supplementary Tables 1-4, Supplementary Figures 1-13 with Legends, and Legends to Supplementary Movies 1-5. (PDF 585 kb)

Supplementary information

The file contains Supplementary Movie 1 showing continuous self-feeding by monkey A Continuous self-feeding by monkey A showing 7 consecutive successful trials. The monkey's cortical control is 4-dimensional, including 3 dimensions of endpoint control plus gripper control. (WMV 8052 kb)

Supplementary information

The file contains Supplementary Movie 2 showing continuous self-feeding by monkey P Continuous self-feeding by monkey P showing 6 consecutive trials (5 successful). Monkey's cortical control is 3-dimensional, i.e. endpoint control. The gripper is controlled as a dimension dependent on endpoint movement: it opens when the arm moves forward and closes when the arm is held stable or moved backward. (MPG 12550 kb)

Supplementary information

The file contains Supplementary Movie 3 showing target tracking As the monkey makes a reach toward an initial target with the prosthetic arm, the target is displaced so that a direct move to target would knock the food off the presentation device. The monkey then moves the arm endpoint in a curved path to avoid the collision, and successfully obtains the food. (WMV 1661 kb)

Supplementary information

The file contains Supplementary Movie 4 showing an emergent behaviour: finger licking When a target is presented, the monkey ignores the target and instead moves the arm so as to be able to lick the gripper fingers. This emergent behaviour is outside the task requirements and is a result of embodied control. (WMV 3192 kb)

Supplementary information

The file contains Supplementary Movie 5 showing an emergent behaviour: using the arm to push food into the mouth. When a marshmallow ends up barely between the monkey's lips at the end of a successful reaching and retrieval, the animal is unable to get the food into its mouth without a helping "hand", so it uses the robotic arm to push the food into its mouth. (WMV 7833 kb)

Rights and permissions

Reprints and permissions

About this article

Cite this article

Velliste, M., Perel, S., Spalding, M. et al. Cortical control of a prosthetic arm for self-feeding. Nature 453, 1098–1101 (2008). https://doi.org/10.1038/nature06996

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/nature06996

This article is cited by

Comments

By submitting a comment you agree to abide by our Terms and Community Guidelines. If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing