Care-Providing Robot FRIEND
Encyclopedia
The care-providing robotic
system FRIEND (Functional Robot arm with user-frIENdly interface for Disabled people) is a semi-autonomous robot
designed to support disabled
and elderly
people in their daily life activities, like preparing and serving a meal, or reintegration in professional life. FRIEND make it possible for such people, e.g. patients which are paraplegic
, have muscle diseases or serious paralysis
, e.g. due to strokes, to perform special tasks in daily life self-determined and without help from other people like therapists or nursing staff.
The robot FRIEND is the third generation of such robots developed at the Institute of Automation (IAT) of University of Bremen within different research projects. Within the last project AMaRob, an interdisciplinary consortium
, consisting of technicians, designers as well as therapists and further representatives of various interest groups, influences the development of FRIEND. Besides covering the various technical aspects, also design aspects were included as well as requirements from daily practice given by therapists, in order to develop a care-providing robot that is suitable for daily life activities. The AMaRob project was founded by the German Federal Ministry of Education and Research ("BMBF - Bundesministerium für Bildung und Forschung") within the "Leitinnovation Servicerobotik".
platform Nemo Vertical, which is an electrical wheelchair from Meyra. This basic platform has been equipped with various additional components, which are described in the following.
as well as in professional life.
enables the user to prepare and eat a meal. A special meal-tray has been designed which can be gripped by the manipulator. First the meal-tray is fetched from a refrigerator
which is equipped with an automatic door opener. Then the manipulator puts the meal-tray in a remote controlled microwave oven
to cook the meal. After cooking the meal-tray is fetched from the microwave oven and placed on the tray of the wheelchair. After that FRIEND supports the user to eat the meal. With a special designed spoon the manipulator can take a bit meal and feed the user. After the eating procedure the meal-tray is put away.
The handles are designed in such a way that they can be robustly recognized by the vision system
and easily grasped by manipulator's gripper.
service desk. Professional scenarios are very important for disabled people from the viewpoint of re-integration into daily life activities. With FRIEND the user can manage tasks like lending and return of books.
s for public telephone boxes
. A keypad has to be taken from a keypad magazine by the gripper and put into a test adapter to check the correct working of pad's electronics. After that each keypad has to be inspected visually by the user to detect cracks or similar damages. Based on the results the keypad have to be sorted.
With respect to a semi-autonomous service robotic system this means that the user's cognitive capabilities are taken into account, whenever a robust and reliable technical solution is unavailable. It is obvious that the acceptance of such a system will be low in general. But for people that are dependent on a personnel assistant, like the disabled or elderly, this approach offers the opportunity to decrease this dependency and therefore to increase their quality of life
. A service robot has to be able to pursue a certain mission goal as commanded from its user but also needs to react flexibly to dynamic changes within the workspace. To meet these requirements, hybrid multi-layer control architectures
have been successfully applied. These architectures usually consist of three layers:
The software framework
MASSiVE (Multi-layer Architecture for Semi-autonomous Service robots with Verified task Execution) is a special kind of hybrid multi-layer control architecture which is tailored to the requirements of semi-autonomous and distributed systems, like the care-providing robot FRIEND, acting in environments with distributed smart components. These intelligent wheelchair mounted manipulator systems allow to benefit from the inclusion of the user's cognitive capabilities into task execution and consequently lower the system complexity compared to a fully autonomous system. The semi-autonomous control requires a sophisticated integration of a human-machine-interface (HMI)
which is able to couple input devices according to the user's impairment, for example a haptic suit, eye-mouse, speech-recognition, chin joystick or a brain-computer interface (BCI). The resulting MASSiVE control architecture with special emphasis on the HMI component is depicted in Fig. 2. Here, the deliberator has been moved to the sequencer component, and the HMI has direct access to control the actuators in the reactive layer during user interactions (e.g., to move the camera, until the desired object to be manipulated is in field of view).
Besides the focus on semi-autonomous system control, the MASSiVE framework includes a second main paradigm, namely the pre-structuring of task knowledge. This task planner input is specified offline in a scenario and model driven approach with the help of so-called process-structures on two levels of abstraction, the abstract level and the elementary level. After specification and before being used for task execution, the task knowledge is verified offline, to guarantee a robust runtime behavior. This development process model provides a structured guidance and enforce consistency throughout the whole process, so that uniform implementation
s and maintainability are achieved. Furthermore it guides through development and test of system core functionality (skills). The whole paradigm is depicted in Fig. 3.
The tasks is selected and started by the user via the HMI, depicted in Fig. 4, on a high level of abstraction, e.g. "cook meal". After initial monitoring to define the current state of the system and the environment, the tasks execution is performed and a list of elementary operations are created which can be executed autonomously by the system. These elementary operations consists of, e.g. image processing algorithm
to recognize objects in the environment or manipulative algorithms to calculate a special trajectory to grasp an object.
Besides these layers, a world model is included in the control architecture that contains the current system's perspective on the world according to the task to be executed. Due to the hybrid architecture a separation of world-model data into two categories is mandatory: The deliberator operates with symbolic object representations (e.g. "C" for the representation of a cup), while the reactive layer deals with the sensor percepts taken from these objects, so-called sub-symbolic information. Examples for sub-symbolic information are the color
, size
, shape
, location or weight
of an object.
In order to cope with the above described problems in the care-providing robot FRIEND the special vision system ROVIS (RObust machine VIsion for Service robotics) was developed. The structure of ROVIS is depicted in Fig. 5. There are two main ROVIS components: hardware and object recognition and reconstruction chain. The connection between ROVIS and the overall robot control system is represented by the World Model where ROVIS stores the processed visual information. At the initialization phase of ROVIS the extrinsic camera parameters needed for 3D object reconstruction and the transformation between stereo camera and manipulator which is necessary for vision based object manipulation are calculated by the Camera Calibration module. The object recognition and reconstruction chain consists of robust algorithms for object recognition and 3D reconstruction for the purpose of reliable manipulation motion planning and grasping in unstructured environments and variable illumination conditions. Therefore an accuracy of 5mm for the estimated 3D pose is necessary which enforces very good and precise algorithms. In ROVIS, robustness must be understood as the capability to the system to adapt to varying operational conditions and is realized through the inclusion feedback
mechanisms at the image processing level and also between different hardware and software components of the vision system. A core part of the system is the automatic, closed-loop calculation of an image Region of Interest (ROI)
on which vision methods are applied. By using a ROI the performance of object recognition and 3D reconstruction can be improved since the scene complexity
is reduced.
Within ROVIS there are several methods used for object recognition, e.g. robust region based color segmentation
and robust edge detection
. The first one is for objects with uniform color and without texture (e.g. bottle
, glass, handles
) and the second one for objects with textures (e.g. book
s). In order to recognize objects in FRIEND's environment robustly special feature are extracted which are used to identify objects and to determine their pose. For big objects like refrigerator or microwave an improved SIFT (Scale Invariant Feature Transform) algorithm is used, which was developed at the Institute of Automation at the University of Bremen
is to find a collision free trajectory
. Nevertheless, in service robotics, other aspects must be additionally taken into account like safety and smoothness of the trajectories. In order to suffice all these requirements for FRIEND a sensor-based motion planning for the manipulator was developed. The procedure is based on Cartesian space
information, and therefore computationally efficient as well as with a high precision. The object grasping frames are calculated on-line which increases the flexibility
of the system during execution. All implemented algorithms are suitable for real-time applications in service robotics but also for industrial usage.
The motion planning is performed based on a Mapped Virtual Reality
(MVR), displayed in Fig. 6. This 3D-model of the environment is built from the information within the world model which was perceived by the sensors. Before any motion of the robot arm is performed the trajectory from the current to the goal configuration is calculated within this 3D-model. During motion on-line collision checking is done, since obstacles can move during motion. Obstacles are in this case all objects of the environment, which are not included in the current motion, also in some tasks the user in the wheelchair. It is important that there is at any time no danger for the user.
At the wrist of the manipulator a force-torque-sensor is installed. The information from this sensor is used to detect collision
s or for fine-tuning
during manipulative operations, e.g. when the robot arm should put a gripped object in a small opening. This ensures robustness during execution.
In general the trajectories calculated from the motion planning algorithms are robot like, i.e. clipped and jerky. In order to enhance this and make the movement of the manipulator more pleasant to the user the trajectories are smoothed and the quality is enhanced. Therefore the used robot arm is helpful. Since six degrees of freedom are sufficient to define a 3D pose in the environment (three degrees for position and three degrees for orientation) and FRIEND's manipulator has seven degrees of freedom, one degree of freedom can be chosen optional, i.e. the manipulator can turn its elbow joint by 360 degrees without changing gripper's pose. This can be used to find the best trajectory from a start to a goal configuration among an infeasible set of possible configurations. This seventh degree of freedom is also used to solve and avoid dead-locks during the motion process and to keep a minimal distance to obstacles.
Robotics
Robotics is the branch of technology that deals with the design, construction, operation, structural disposition, manufacture and application of robots...
system FRIEND (Functional Robot arm with user-frIENdly interface for Disabled people) is a semi-autonomous robot
Robot
A robot is a mechanical or virtual intelligent agent that can perform tasks automatically or with guidance, typically by remote control. In practice a robot is usually an electro-mechanical machine that is guided by computer and electronic programming. Robots can be autonomous, semi-autonomous or...
designed to support disabled
Disability
A disability may be physical, cognitive, mental, sensory, emotional, developmental or some combination of these.Many people would rather be referred to as a person with a disability instead of handicapped...
and elderly
Old age
Old age consists of ages nearing or surpassing the average life span of human beings, and thus the end of the human life cycle...
people in their daily life activities, like preparing and serving a meal, or reintegration in professional life. FRIEND make it possible for such people, e.g. patients which are paraplegic
Paraplegia
Paraplegia is an impairment in motor or sensory function of the lower extremities. The word comes from Ionic Greek: παραπληγίη "half-striking". It is usually the result of spinal cord injury or a congenital condition such as spina bifida that affects the neural elements of the spinal canal...
, have muscle diseases or serious paralysis
Paralysis
Paralysis is loss of muscle function for one or more muscles. Paralysis can be accompanied by a loss of feeling in the affected area if there is sensory damage as well as motor. A study conducted by the Christopher & Dana Reeve Foundation, suggests that about 1 in 50 people have been diagnosed...
, e.g. due to strokes, to perform special tasks in daily life self-determined and without help from other people like therapists or nursing staff.
The robot FRIEND is the third generation of such robots developed at the Institute of Automation (IAT) of University of Bremen within different research projects. Within the last project AMaRob, an interdisciplinary consortium
Consortium
A consortium is an association of two or more individuals, companies, organizations or governments with the objective of participating in a common activity or pooling their resources for achieving a common goal....
, consisting of technicians, designers as well as therapists and further representatives of various interest groups, influences the development of FRIEND. Besides covering the various technical aspects, also design aspects were included as well as requirements from daily practice given by therapists, in order to develop a care-providing robot that is suitable for daily life activities. The AMaRob project was founded by the German Federal Ministry of Education and Research ("BMBF - Bundesministerium für Bildung und Forschung") within the "Leitinnovation Servicerobotik".
Systems
FRIEND is built from reliable industrial components. It is based on the wheelchairWheelchair
A wheelchair is a chair with wheels, designed to be a replacement for walking. The device comes in variations where it is propelled by motors or by the seated occupant turning the rear wheels by hand. Often there are handles behind the seat for someone else to do the pushing...
platform Nemo Vertical, which is an electrical wheelchair from Meyra. This basic platform has been equipped with various additional components, which are described in the following.
- Robot Arm / Manipulator: The Light Weight Robotic Arm 3 (LWA3) is a 7 degrees of freedom manipulatorManipulatorIn robotics a manipulator is a device used to manipulate materials without direct contact. The applications were originally for dealing with radioactive or biohazardous materials, using robotic arms, or they were used in inaccessible places. In more recent developments they have been used in...
of SchunkSCHUNKSCHUNK Intec, Inc. is a multinational manufacturing company, based principally in Germany and in the United States. The company operates worldwide in more than 50 countries with wholly owned subsidiaries, five manufacturing locations in Germany, USA and China, and distribution partners all over the...
mounted on an automated panning arm. So the arm can park behind the seat in order to navigate FRIEND in narrow passages. The robot arm is equipped with the prosthetic hand "SensorHand Speed" from Otto BockOtto BockOtto Bock is a German prosthetics company situated in Duderstadt. It was founded in 1919 by its namesake prosthetist, Otto Bock. It was created in response to the large number of injured veterans from World War I....
which has built-in slip sensors in order to detect the slipping of gripped object and adapt the force accordingly. At the robot's wristWristIn human anatomy, the wrist is variously defined as 1) the carpus or carpal bones, the complex of eight bones forming the proximal skeletal segment of the hand;...
a force-torque sensor is mounted to perform force-torque-based reactive manipulative operations and to detect collisions.
- TFT-Display: The TFT display provides visual information to the user and is also mounted on a panning arm.
- Intelligent Tray: In front of the user an intelligent tray is available on which objects can be placed down by the manipulator. This tray is based on infra-red (IR)InfraredInfrared light is electromagnetic radiation with a wavelength longer than that of visible light, measured from the nominal edge of visible red light at 0.74 micrometres , and extending conventionally to 300 µm...
devices to acquire precise information about object locations, which should be manipulated.
- Stereo Camera System: A Bumblebee 2 stereo camera systemStereo cameraA stereo camera is a type of camera with two or more lenses with a separate image sensor or film frame for each lens. This allows the camera to simulate human binocular vision, and therefore gives it the ability to capture three-dimensional images, a process known as stereo photography. Stereo...
with built-in calibrationCalibrationCalibration is a comparison between measurements – one of known magnitude or correctness made or set with one device and another measurement made in as similar a way as possible with a second device....
, synchronizationSynchronizationSynchronization is timekeeping which requires the coordination of events to operate a system in unison. The familiar conductor of an orchestra serves to keep the orchestra in time....
and stereo projective calculation features is used to acquire information of the environment. It is mounted at the top of the system on a pan-tilt-head unit, which itself is installed on a special rack behind the seat.
- Computer System: A high-end PCPersonal computerA personal computer is any general-purpose computer whose size, capabilities, and original sales price make it useful for individuals, and which is intended to be operated directly by an end-user with no intervening computer operator...
unit is mounted on the wheelchair platform behind the user. The mounting as depicted is still in a prototype state.
- Input devices: There are several input devices which are available for FRIEND or under development: chin joystick, hand joystick, speech control (in- and output), brain-computer interface (BCI)Brain-computer interfaceA brain–computer interface , sometimes called a direct neural interface or a brain–machine interface , is a direct communication pathway between the brain and an external device...
and eye control. The input devices are adapted according to the impairments of the user or his preferences.
- Infra-red Communication and Appliances: An infra-red control unit, development by IGEL , for communication with various appliances in the robot's environment is integrated underneath the pan-tilt-head unit. Thus, e.g. an automatic door opening mechanism in the refrigerator and the microwave, the configuration and control of the microwave itself or various consumer electronic components can be operated wireless.
Usage
Within the AMaRob (AMaRob web page ) project three scenarios were developed that support disabled and elderly people in their Activities of Daily Life (ADL)Activities of daily living
Activities of Daily Living is a term used in healthcare to refer to daily self-care activities within an individual's place of residence, in outdoor environments, or both...
as well as in professional life.
ADL
This scenarioScenario
A scenario is a synoptical collage of an event or series of actions and events. In the Commedia dell'arte it was an outline of entrances, exits, and action describing the plot of a play that was literally pinned to the back of the scenery...
enables the user to prepare and eat a meal. A special meal-tray has been designed which can be gripped by the manipulator. First the meal-tray is fetched from a refrigerator
Refrigerator
A refrigerator is a common household appliance that consists of a thermally insulated compartment and a heat pump that transfers heat from the inside of the fridge to its external environment so that the inside of the fridge is cooled to a temperature below the ambient temperature of the room...
which is equipped with an automatic door opener. Then the manipulator puts the meal-tray in a remote controlled microwave oven
Microwave oven
A microwave oven is a kitchen appliance that heats food by dielectric heating, using microwave radiation to heat polarized molecules within the food...
to cook the meal. After cooking the meal-tray is fetched from the microwave oven and placed on the tray of the wheelchair. After that FRIEND supports the user to eat the meal. With a special designed spoon the manipulator can take a bit meal and feed the user. After the eating procedure the meal-tray is put away.
The handles are designed in such a way that they can be robustly recognized by the vision system
Machine vision
Machine vision is the process of applying a range of technologies and methods to provide imaging-based automatic inspection, process control and robot guidance in industrial applications. While the scope of MV is broad and a comprehensive definition is difficult to distil, a "generally accepted...
and easily grasped by manipulator's gripper.
Library
The first professional scenario is situated at a libraryLibrary
In a traditional sense, a library is a large collection of books, and can refer to the place in which the collection is housed. Today, the term can refer to any collection, including digital sources, resources, and services...
service desk. Professional scenarios are very important for disabled people from the viewpoint of re-integration into daily life activities. With FRIEND the user can manage tasks like lending and return of books.
Workshop
The second professional scenario takes place in a rehabilitation workshop. The realized scenario is representative for many quality control tasks in industry and consists of checking the functionality of keypadKeypad
A keypad is a set of buttons arranged in a block or "pad" which usually bear digits, symbols and usually a complete set of alphabetical letters. If it mostly contains numbers then it can also be called a numeric keypad...
s for public telephone boxes
Telephone booth
A telephone booth, telephone kiosk, telephone call box or telephone box is a small structure furnished with a payphone and designed for a telephone user's convenience. In the USA, Canada and Australia, "telephone booth" is used, while in the UK and the rest of the Commonwealth it is a "telephone...
. A keypad has to be taken from a keypad magazine by the gripper and put into a test adapter to check the correct working of pad's electronics. After that each keypad has to be inspected visually by the user to detect cracks or similar damages. Based on the results the keypad have to be sorted.
Architecture
The realization of semi-autonomy is based on the central idea to include human tasks into task execution. A good explanation of this principle is given in the following quotation:- "When we think of interfaces between human beings and computers, we usually assume that the human being is the one requesting that a task be completed, and the computer is completing the task and providing the results. What if this process were reversed and a computer program could ask a human being to perform a task and return the results? [Mechanical Turk - amazon.com]"
With respect to a semi-autonomous service robotic system this means that the user's cognitive capabilities are taken into account, whenever a robust and reliable technical solution is unavailable. It is obvious that the acceptance of such a system will be low in general. But for people that are dependent on a personnel assistant, like the disabled or elderly, this approach offers the opportunity to decrease this dependency and therefore to increase their quality of life
Quality of life
The term quality of life is used to evaluate the general well-being of individuals and societies. The term is used in a wide range of contexts, including the fields of international development, healthcare, and politics. Quality of life should not be confused with the concept of standard of...
. A service robot has to be able to pursue a certain mission goal as commanded from its user but also needs to react flexibly to dynamic changes within the workspace. To meet these requirements, hybrid multi-layer control architectures
Three-layer architecture
The Three-Layer Architecture is a hybrid reactive/deliberative robot architecture developed by R. James Firby that consists of three layers: a reactive feedback control mechanism, a reactive plan execution mechanism, and a mechanism for performing time-consuming deliberative computations.- See also...
have been successfully applied. These architectures usually consist of three layers:
- A deliberative layer, which contains a task planner to generate a sequence of operations to reach a certain goal with respect to the user's input command.
- A reactive layer, which has access to the system's sensors and actuators and provides reactive behavior which is robust even under environmental disturbances, for example with the help of closed-loop control.
- A sequencer that mediates between deliberator and reactive layer i.e. activates or deactivates reactive operations according to the deliberator's specification.
The software framework
Software framework
In computer programming, a software framework is an abstraction in which software providing generic functionality can be selectively changed by user code, thus providing application specific software...
MASSiVE (Multi-layer Architecture for Semi-autonomous Service robots with Verified task Execution) is a special kind of hybrid multi-layer control architecture which is tailored to the requirements of semi-autonomous and distributed systems, like the care-providing robot FRIEND, acting in environments with distributed smart components. These intelligent wheelchair mounted manipulator systems allow to benefit from the inclusion of the user's cognitive capabilities into task execution and consequently lower the system complexity compared to a fully autonomous system. The semi-autonomous control requires a sophisticated integration of a human-machine-interface (HMI)
User interface
The user interface, in the industrial design field of human–machine interaction, is the space where interaction between humans and machines occurs. The goal of interaction between a human and a machine at the user interface is effective operation and control of the machine, and feedback from the...
which is able to couple input devices according to the user's impairment, for example a haptic suit, eye-mouse, speech-recognition, chin joystick or a brain-computer interface (BCI). The resulting MASSiVE control architecture with special emphasis on the HMI component is depicted in Fig. 2. Here, the deliberator has been moved to the sequencer component, and the HMI has direct access to control the actuators in the reactive layer during user interactions (e.g., to move the camera, until the desired object to be manipulated is in field of view).
Besides the focus on semi-autonomous system control, the MASSiVE framework includes a second main paradigm, namely the pre-structuring of task knowledge. This task planner input is specified offline in a scenario and model driven approach with the help of so-called process-structures on two levels of abstraction, the abstract level and the elementary level. After specification and before being used for task execution, the task knowledge is verified offline, to guarantee a robust runtime behavior. This development process model provides a structured guidance and enforce consistency throughout the whole process, so that uniform implementation
Implementation
Implementation is the realization of an application, or execution of a plan, idea, model, design, specification, standard, algorithm, or policy.-Computer Science:...
s and maintainability are achieved. Furthermore it guides through development and test of system core functionality (skills). The whole paradigm is depicted in Fig. 3.
The tasks is selected and started by the user via the HMI, depicted in Fig. 4, on a high level of abstraction, e.g. "cook meal". After initial monitoring to define the current state of the system and the environment, the tasks execution is performed and a list of elementary operations are created which can be executed autonomously by the system. These elementary operations consists of, e.g. image processing algorithm
Image processing
In electrical engineering and computer science, image processing is any form of signal processing for which the input is an image, such as a photograph or video frame; the output of image processing may be either an image or, a set of characteristics or parameters related to the image...
to recognize objects in the environment or manipulative algorithms to calculate a special trajectory to grasp an object.
Besides these layers, a world model is included in the control architecture that contains the current system's perspective on the world according to the task to be executed. Due to the hybrid architecture a separation of world-model data into two categories is mandatory: The deliberator operates with symbolic object representations (e.g. "C" for the representation of a cup), while the reactive layer deals with the sensor percepts taken from these objects, so-called sub-symbolic information. Examples for sub-symbolic information are the color
Color
Color or colour is the visual perceptual property corresponding in humans to the categories called red, green, blue and others. Color derives from the spectrum of light interacting in the eye with the spectral sensitivities of the light receptors...
, size
Size
The word size may refer to how big something is. In particular:* Measurement, the process or the result of determining the magnitude of a quantity, such as length or mass, relative to a unit of measurement, such as a meter or a kilogram...
, shape
Shape
The shape of an object located in some space is a geometrical description of the part of that space occupied by the object, as determined by its external boundary – abstracting from location and orientation in space, size, and other properties such as colour, content, and material...
, location or weight
Weight
In science and engineering, the weight of an object is the force on the object due to gravity. Its magnitude , often denoted by an italic letter W, is the product of the mass m of the object and the magnitude of the local gravitational acceleration g; thus:...
of an object.
Image processing
The main problem with service robotic systems such as the care-providing robot FRIEND is that they have to operate in dynamic surroundings where the state of the environment is unpredictable and changes stochastically, hence two main problems have been encountered when developing image processing systems for service robotics: unstructured environment and variable illumination conditions. They have to cope with a large amount of visual information and for the implementation of the vision system a high degree of complexity is necessary. A second major problem in robot vision is the wide spectrum of illumination conditions that appear during the on-line operation of the machine vision system, since colors are one important attribute in object recognition. The human visual system has the ability to compute color descriptors that stay constant even in variable illumination conditions, which is not the case for machine vision systems. A key requirement in this field is the reliable recognition of objects in the robot's camera image, extraction of object features from the images and, based on the extracted features, subsequent correct object localization in a complex 3D environment so that these information can be used for reliable object grasping and manipulation.In order to cope with the above described problems in the care-providing robot FRIEND the special vision system ROVIS (RObust machine VIsion for Service robotics) was developed. The structure of ROVIS is depicted in Fig. 5. There are two main ROVIS components: hardware and object recognition and reconstruction chain. The connection between ROVIS and the overall robot control system is represented by the World Model where ROVIS stores the processed visual information. At the initialization phase of ROVIS the extrinsic camera parameters needed for 3D object reconstruction and the transformation between stereo camera and manipulator which is necessary for vision based object manipulation are calculated by the Camera Calibration module. The object recognition and reconstruction chain consists of robust algorithms for object recognition and 3D reconstruction for the purpose of reliable manipulation motion planning and grasping in unstructured environments and variable illumination conditions. Therefore an accuracy of 5mm for the estimated 3D pose is necessary which enforces very good and precise algorithms. In ROVIS, robustness must be understood as the capability to the system to adapt to varying operational conditions and is realized through the inclusion feedback
Feedback
Feedback describes the situation when output from an event or phenomenon in the past will influence an occurrence or occurrences of the same Feedback describes the situation when output from (or information about the result of) an event or phenomenon in the past will influence an occurrence or...
mechanisms at the image processing level and also between different hardware and software components of the vision system. A core part of the system is the automatic, closed-loop calculation of an image Region of Interest (ROI)
Region of interest
A Region of Interest, often abbreviated ROI, is a selected subset of samples within a dataset identified for a particular purpose.For example:* on a waveform , a time or frequency interval...
on which vision methods are applied. By using a ROI the performance of object recognition and 3D reconstruction can be improved since the scene complexity
Complexity
In general usage, complexity tends to be used to characterize something with many parts in intricate arrangement. The study of these complex linkages is the main goal of complex systems theory. In science there are at this time a number of approaches to characterizing complexity, many of which are...
is reduced.
Within ROVIS there are several methods used for object recognition, e.g. robust region based color segmentation
Segmentation (image processing)
In computer vision, segmentation refers to the process of partitioning a digital image into multiple segments . The goal of segmentation is to simplify and/or change the representation of an image into something that is more meaningful and easier to analyze...
and robust edge detection
Edge detection
Edge detection is a fundamental tool in image processing and computer vision, particularly in the areas of feature detection and feature extraction, which aim at identifying points in a digital image at which the image brightness changes sharply or, more formally, has discontinuities...
. The first one is for objects with uniform color and without texture (e.g. bottle
Bottle
A bottle is a rigid container with a neck that is narrower than the body and a "mouth". By contrast, a jar has a relatively large mouth or opening. Bottles are often made of glass, clay, plastic, aluminum or other impervious materials, and typically used to store liquids such as water, milk, soft...
, glass, handles
Handle (grip)
A handle is a part of, or attachment to, an object that can be moved or used by hand. The design of each type of handle involves substantial ergonomic issues, even where these are dealt with intuitively or by following tradition...
) and the second one for objects with textures (e.g. book
Book
A book is a set or collection of written, printed, illustrated, or blank sheets, made of hot lava, paper, parchment, or other materials, usually fastened together to hinge at one side. A single sheet within a book is called a leaf or leaflet, and each side of a leaf is called a page...
s). In order to recognize objects in FRIEND's environment robustly special feature are extracted which are used to identify objects and to determine their pose. For big objects like refrigerator or microwave an improved SIFT (Scale Invariant Feature Transform) algorithm is used, which was developed at the Institute of Automation at the University of Bremen
Motion planning
Primary task of the motion planningMotion planning
Motion planning is a term used in robotics for the process of detailing a task into discrete motions....
is to find a collision free trajectory
Trajectory
A trajectory is the path that a moving object follows through space as a function of time. The object might be a projectile or a satellite, for example. It thus includes the meaning of orbit—the path of a planet, an asteroid or a comet as it travels around a central mass...
. Nevertheless, in service robotics, other aspects must be additionally taken into account like safety and smoothness of the trajectories. In order to suffice all these requirements for FRIEND a sensor-based motion planning for the manipulator was developed. The procedure is based on Cartesian space
Cartesian coordinate system
A Cartesian coordinate system specifies each point uniquely in a plane by a pair of numerical coordinates, which are the signed distances from the point to two fixed perpendicular directed lines, measured in the same unit of length...
information, and therefore computationally efficient as well as with a high precision. The object grasping frames are calculated on-line which increases the flexibility
Flexibility (engineering)
Flexibility is used as an attribute of various types of systems. In the field of engineering systems design, it refers to designs that can adapt when external changes occur. Flexibility has been defined differently in many fields of engineering, architecture, biology, economics, etc...
of the system during execution. All implemented algorithms are suitable for real-time applications in service robotics but also for industrial usage.
The motion planning is performed based on a Mapped Virtual Reality
Virtual reality
Virtual reality , also known as virtuality, is a term that applies to computer-simulated environments that can simulate physical presence in places in the real world, as well as in imaginary worlds...
(MVR), displayed in Fig. 6. This 3D-model of the environment is built from the information within the world model which was perceived by the sensors. Before any motion of the robot arm is performed the trajectory from the current to the goal configuration is calculated within this 3D-model. During motion on-line collision checking is done, since obstacles can move during motion. Obstacles are in this case all objects of the environment, which are not included in the current motion, also in some tasks the user in the wheelchair. It is important that there is at any time no danger for the user.
At the wrist of the manipulator a force-torque-sensor is installed. The information from this sensor is used to detect collision
Collision
A collision is an isolated event which two or more moving bodies exert forces on each other for a relatively short time.Although the most common colloquial use of the word "collision" refers to accidents in which two or more objects collide, the scientific use of the word "collision" implies...
s or for fine-tuning
Fine-tuning
In theoretical physics, fine-tuning refers to circumstances when the parameters of a model must be adjusted very precisely in order to agree with observations. Theories requiring fine-tuning are regarded as problematic in the absence of a known mechanism to explain why the parameters happen to...
during manipulative operations, e.g. when the robot arm should put a gripped object in a small opening. This ensures robustness during execution.
In general the trajectories calculated from the motion planning algorithms are robot like, i.e. clipped and jerky. In order to enhance this and make the movement of the manipulator more pleasant to the user the trajectories are smoothed and the quality is enhanced. Therefore the used robot arm is helpful. Since six degrees of freedom are sufficient to define a 3D pose in the environment (three degrees for position and three degrees for orientation) and FRIEND's manipulator has seven degrees of freedom, one degree of freedom can be chosen optional, i.e. the manipulator can turn its elbow joint by 360 degrees without changing gripper's pose. This can be used to find the best trajectory from a start to a goal configuration among an infeasible set of possible configurations. This seventh degree of freedom is also used to solve and avoid dead-locks during the motion process and to keep a minimal distance to obstacles.