Deafblind Robodrummer

Deafblind Robodrummer has no eyes and no ears, all he got is his sense of touch and rhythm. Cluelessly he scans the environment with pegs in his hands, that he uses as drumsticks, looking for objects to drum on.
When he feels the presence of drum-able objects close to him, he plays a beat. It's those moments that bring a glimpse of joy to his Land of Silence and Darkness.

 

Description

Deafblind Robodrummer (DBRD) is equipped with a torso, to which two arms and a head are attached. The arms can turn on the shoulders and the wrist is able to move vertically. This allows DBRD to turn towards a drum and beat it. As there are no further sensors or input devices, the robot is dependant on sensing his environments through the servo motors.

After initialisation, DBRD begins to look for drum-able objects, by moving his arms as far out as possible and lowering the sticks by moving the wrists. If he feels the resistance of an object, he remembers the position of the wrist at which it was felt, moves the wrist upward and turns the shoulder slightly, continuing the probing procedure. He continues until the two arms meet in the middle and the entire environment has been scanned.

If the scanning procedure was successful, DBRD now has a list of positions at which an object has been felt, however he does not know yet where an object begins or ends and how many there are around him. Therefore he marks the first occurrence of a measurement with a "b" (begin) and all the subsequent measurements with "i" (inside) until the measurement returned false (no drum encountered). Positions where no object was detected are marked with "o" (outside).

The measurements tagged with "b" are combined with their neighbouring "i" and averaged to find the center of the objects, at which position DBRD imagines to be a drum. Below is a table illustrating the process for some sample measurements with position shifts of 0.5.

Positions-2.5-2-1.5-1-0.500.51
Measurements2.13122.2132false1.8311.79321.8123falsefalse
Interpretationbiobiioo
ConstructionDrum 1 Drum 2

When all measurements have been analysed, DBRD has internalised the positions of the possible drums for both left and right arms. He now can use as much objects as are around him to play his beats.

Issues Encountered

DBRD continues playing for a predefined amount of time. Originally, the idea was to make him go on for as long as there are objects to drum on (objects might move away during drumming or be removed maliciously). However, when hitting a drum, thereby executing a fast down- and upwards motion with the wrist, the forces acting on the wrist are as high when an object has been hit as they are when the object is missed. This is because in both cases the speed of the wrist drops to zero instantaneously. DBRD therefore is unable to tell if a hit was successful or not.

The lack of sensitivity in the servo motors was a general problem in the process. As visible in the video, DBRD has to execute a lot of force, almost lifting himself up in order to detect an object. A threshold defines at which point the torque applied to the motor represents an object. If it is chosen too high, the robot ignores the object and might fall, if it is too low, too many false positives are registered due to normal fluctuations in torque when the motors are moving.

The speed and step size at which DBRD explores his environment have been chosen in order to minimise false positives, while at the same time keeping the duration of the scanning procedure to a bearable level. With the current setting, small objects are missed and objects too close to each other would not be recognised separately. Reducing the size of the steps the robot arms move horizontally would make it possible to also detect smaller objects. A faster vertical movement however proved unsuccessful, as the forces on the motor in normal motion become too strong.

When developing the software, some minor issues have come up, mainly due to conflicts arising from the parallel exploration of the environment with both arms. This was solved by encapsulating the scanning process in separate classes and instantiating them for both arms individually.

How to continue

An interesting continuation of the project, would be to add some mobility to the robot, make him able to move around and gather objects to use as a drum set. At the moment DBRD relies on suitable objects being placed in his surrounding. He is also not able to distinguish rigid objects, that make a sound, from flexible ones, that are less interesting to drum on. It might be possible to improve the algorithm for detection to make this differentiation. However, it could be necessary to add a force resistant sensor to the hands of the robot, but this would be outside the interest of this project, which looks at the possibility of using only the servos for robot localisation and orientation.


 

Code

The main code is in the file DrummerBoy.u, which also creates an instance of the main class. The robot automatically resets and after calling the function start() begins the scanning procedure. The class DrummerBoy creates two instances of the class DrumSet, one for each arm. All the functions for scanning and detecting objects are within the class DrumSet. For each detected object, an instance of the class Drum is called. The class Drum is used to store the horizontal and vertical position of the drums. Below is a diagram illustrating the classes and their processes.

Download Source Code

DrummerBoy.u

load("dynamixel.u");
var Global.d = Dynamixel.new;

class DrummerBoy {

	load("drumset.u");
	
	/* Initialisation */

	function init() {
		var this.hitDistance = 0.5; // Force with which the drums will be hit
		var this.hitSpeed = 0.1; // Delay after drum has been hit
		var this.duration = 20; // Number of beats to play
		
		var this.drumTag = Tag.new;
		this.drumTag.freeze;
		
		// Initialise motors
		var this.head = Dynamixel.Device.new(d,4,Dynamixel.DeviceDesc.AX12);
		var this.rightWrist = Dynamixel.Device.new(d,1,Dynamixel.DeviceDesc.AX12);
		var this.rightShoulder = Dynamixel.Device.new(d,2,Dynamixel.DeviceDesc.AX12);
		var this.leftWrist = Dynamixel.Device.new(d,11,Dynamixel.DeviceDesc.AX12);
		var this.leftShoulder = Dynamixel.Device.new(d,12,Dynamixel.DeviceDesc.AX12);
		var this.torso = Dynamixel.Device.new(d,8,Dynamixel.DeviceDesc.AX12);
		var this.kick = Dynamixel.Device.new(d,10,Dynamixel.DeviceDesc.AX12);
		
		// Initialise center and extreme positions of robot limbs
		this.initPositions();
		
		// Reset positions
		this.setSpeed(1);
		this.reset();
		sleep(1s);
		this.setSpeed(0);
	};
	
	function initPositions() {
		
			// Set min and max positions as well as center positions for all joints
		
			var this.head.initPos = 0;
			var this.head.maxLeft = 1.5;
			var this.head.maxRight = -1.5;

			var this.rightWrist.initPos = 0;
			var this.rightWrist.maxUp = -1.5;
			var this.rightWrist.maxDown = 1.5;

			var this.rightShoulder.initPos = 1.5;
			var this.rightShoulder.maxRight = 0;
			var this.rightShoulder.maxLeft = 2.5;

			var this.leftWrist.initPos = 0;
			var this.leftWrist.maxUp = 1.5;
			var this.leftWrist.maxDown = -1.5;

			var this.leftShoulder.initPos = -1.5;
			var this.leftShoulder.maxRight = -2.5;
			var this.leftShoulder.maxLeft = 0;

			var this.torso.initPos = 0;
			var this.torso.maxRight = -1.5;
			var this.torso.maxLeft = 1.5;

			var this.kick.initPos = 0;
			var this.kick.maxUp = -1.5;
			var this.kick.maxDown = 1;
	};
	
	/* Main behaviour */
	
	function start() {
		reset();
		this.drumTag.freeze;
		
		// Create drumset object for left and right arm
		var this.leftDrumSet = DrumSet.new(this.leftShoulder,this.leftWrist,"left"),
		var this.rightDrumSet = DrumSet.new(this.rightShoulder,this.rightWrist,"right");
		
		echo("Found "+this.rightDrumSet.drums.size()+" drum(s) on the right");
		echo("Found "+this.leftDrumSet.drums.size()+" drum(s) on the left");
		
		// Start drumming if drums have been found
		
		if(this.rightDrumSet.drums.size()>0 || this.leftDrumSet.drums.size()>0) {
		
			this.drumTag.unfreeze;
			this.drumTag :while (1 == 1){
				var countdown = this.duration;
				every(2s) {
					this.drum();
					countdown--;
					if(countdown<=0) {
						this.drumTag.freeze;
						break;
					}
				};
			};
		};
	};
	
	function drum() {
		var i;
		var drumToHit;
		
		if(this.rightDrumSet.drums.size()>0) {
			
			// Pick a random drum and hit it
			i = random(this.rightDrumSet.drums.size());
			echo("Hitting drum "+i+" on the right");
			drumToHit = this.rightDrumSet.drums[i];
			
			// Lift wrist up to initial position
			this.rightWrist.targetPos = this.rightWrist.initPos;
			sleep(0.2s);
			
			// Turn shoulder to position of the drum
			this.rightShoulder.targetPos = drumToHit.getShoulderPos();
			sleep(0.2s);
			
			// Place wrist above drum
			this.rightWrist.targetPos = drumToHit.getWristPos()-this.hitDistance;
			sleep(0.2s);
			
			// Hit it
			this.hitRight();
		};
		if(this.leftDrumSet.drums.size()>0) {
			i = random(this.leftDrumSet.drums.size());
			echo("Hitting drum "+i+" on the left");
			drumToHit = this.leftDrumSet.drums[i];
			this.leftWrist.targetPos = this.leftWrist.initPos;
			sleep(0.2s);
			this.leftShoulder.targetPos = drumToHit.getShoulderPos();
			sleep(0.2s);
			this.leftWrist.targetPos = drumToHit.getWristPos()+this.hitDistance;
			sleep(0.2s);
			this.hitLeft();
		};
	};
	
	function stop() {
		this.drumTag.freeze;
	};
	
	/* Commands */
	
	function hitLeft() {
		// Execute hit motion with left wrist
		var curPos = this.leftWrist.position;
		this.leftWrist.targetPos = curPos-this.hitDistance;
		sleep(this.hitSpeed);
		this.leftWrist.targetPos = curPos;
	};
	
	function hitRight() {
		// Execute hit motion with right wrist
		var curPos = this.rightWrist.position;
		this.rightWrist.targetPos = curPos+this.hitDistance;
		sleep(this.hitSpeed);
		this.rightWrist.targetPos = curPos;
	};
	
	function hitKick() {
		// Execute hit motion with kick
		var curPos = this.kick.position;
		this.kick.targetPos = curPos+this.hitDistance;
		sleep(this.hitSpeed);
		this.kick.targetPos = curPos;
	};
	
	/* Utilities */
	
	function pos2Deg(var position, var left, var right) {
		// Convert motor position to degrees
		return (position-left)/(right-left)*(-90-90)+90;
	};
	
	function deg2Pos(var degree, var maxLeft, var maxRight, var leftDeg, var rightDeg) {
		// Convert degrees to motor position
		return (degree-rightDeg)/(leftDeg-rightDeg)*(maxRight-maxLeft)+maxLeft;
	};
	
	function reset() {
		// Initialise all limbs
		var delay = 0.1;
		this.rightShoulder.targetPos = 1.5;
		sleep(delay);
		this.rightWrist.targetPos = 0;
		sleep(delay);
		this.leftWrist.targetPos = 0;
		sleep(delay);
		this.leftShoulder.targetPos = -1.5;
		sleep(delay);
		this.torso.targetPos = 0;
		sleep(delay);
		this.kick.targetPos = 0;
		sleep(delay);
		this.head.targetPos = 0;
	};
	
	function setSpeed(var speed) {
		this.head.targetSpeed = speed;
		this.leftShoulder.targetSpeed = speed;
		this.leftWrist.targetSpeed = speed;
		this.rightShoulder.targetSpeed = speed;
		this.rightWrist.targetSpeed = speed;
		this.torso.targetSpeed = speed;
		this.kick.targetSpeed = speed;
	};
};

var drummer = DrummerBoy.new;

			

DrumSet.u

class DrumSet {
	load("drum.u");
	
	function init(shoulder,wrist,side) {
		var this.torqueThreshold = 80; // Sensitivity for drum detection

		// Initialise drum scan
		var this.drums = [];
		var drumSet = scan(shoulder,wrist,side);
		
		// Setup drum set
		if(drumSet.size()) {
			for(var i=0;i<drumSet.size();i++) {
				this.drums += [Drum.new(drumSet[i][0],drumSet[i][1])]
			};
		};
	};
	
	function scan(shoulder,wrist,side) {
		var step = 0.25; // Shoulder scan intervals
		var gap = 0.3; // Buffer space

		var init;
		var goal;
		var positions = [];
		var detects = [];
		
		if(side=="left") {
			// Start from left position, move to right
			init = shoulder.maxLeft;
			goal = shoulder.maxRight+gap;
			
			// Inverse step size for left shoulder
			step = -1*step;
			
			// Scan step by step from left to right
			for(var i = init;i>goal;i+=step) {
				shoulder.targetPos = i;
				positions += [i];
				// Detect drum at current position
				detects += [detect(wrist,side)];
				sleep(0.5s);
			};
		} else {
			init = shoulder.maxRight;
			goal = shoulder.maxLeft-gap;
			
			// Scan step by step from right to left
			for(var i = init;i<goal;i+=step) {
				shoulder.targetPos = i;
				positions += [i];
				// Detect drum at current position
				detects += [detect(wrist,side)];
				sleep(0.5s);
			};
		};
		// Return the evaluated scans
		return evaluateScans(positions,detects);
	};
	
	function detect(wrist,side) {
		var step = 0.1; // Steps to lower the wrist
		var ignore = 5; // The first few steps are ignored
		var gap = 1; // Leave some space before reaching maximum position
				
		// Don't change these variables
		var detected = false;
		var ok = true;
		var position = 0;
		var torque = 0;
		
		// Pick limb
		if(side=="left") {
			// If we deal with the left wrist, we have to inverse the direction
			step = -1*step; 
		};
		
		for(var i=0;i<60;i= i+1) {
			// Ignore first steps
			if(ignore>0) {
				ignore--;
			};
			torque = wrist.torque%1024; // prevent wraparound
			
			// If torque exceeds thresholds, drum has been detected
			if(torque>this.torqueThreshold && !ignore) {
				detected = true;
				position = wrist.position;
				break;
			};
			
			// If position is going to exceed maximum position, exit loop
			if(side=="left" && wrist.position+(step*gap)<wrist.maxDown) {
				ok = false;
			} else if (side!="left" && wrist.position+(step*gap)>wrist.maxDown) {
				ok = false;
			};
			if(ok) {
				wrist.targetPos = wrist.position+step;
			} else {
				break;
			};
			sleep(0.3s);
		};

		// reset position
		wrist.targetPos = wrist.initPos;
		
		// return position if drum has been detected
		if(detected) {
			return position;
		} else {
			return false;
		}
	};
	
	
	function evaluateScans(positions,detects) {
		// Array sizes of positions and detects have to match
		if(positions.size()!=detects.size()) {
			echo("Incompatible list sizes");
			return false;
		};
		var out = true;
		var inside = false;
		var drums = [];
		var drumSet = [];
		var p;
		for(var i=0;i<positions.size();i++) {
			// Mark the beginning, inside and outside positions of the drum
			if(detects[i] && out) {
				drums += ["b"];
				out = false;
				inside = true;
			}  else if (detects[i] && inside) {
				drums +=["i"];
			} else if (!detects[i]) {
				drums +=["o"];
				out = true;
				inside = false;
			};
		};
		// Construct the drum position from the measurements 
		for(var i=0;i<drums.size();i++) {
			p = 0;
			var j;
			var k;
			// If the beginning of a drum has been detected
			if(drums[i]=="b") {
				if(i<drums.size()-1) {
					// Look for the end of it
					for(j=i+1;j<drums.size();j++) {
						if(drums[j]=="o") {
							break;
						}
					};
					j--;
					// Sum up the positions that comprise the drum
					for(k=i;k<=j;k++) {
						p+=positions[k];
					};
				} else {
					j = i;
				};
				// And take the average of all the positions 
				// to get the center of the drum
				p = p/(j-i+1);
				// Add the shoulder and wrist position to the drum set
				drumSet += [[p,detects[i]]];
			}
		};
		return drumSet;	
	};
}
			

Drum.u

class Drum {
	function init(x,y) {
		var this.x = x;
		var this.y = y;
	};
	
	function getShoulderPos() {
		return this.x;
	};
	
	function getWristPos() {
		return this.y;
	};
};				
			
 

Photos

_DSC9145 _DSC9141 _DSC9135 _DSC9137

 

References

BDRD is by far not the first percussionist robot, some of the ones around even do somersaults. From the german robocross machine comes StickyBoy. The four-armed two-legged metal robo drummer, runs on compressed air and can be programmed via MIDI.

A particularly witty one is the Yellow Drum Machine by Frits Lyneborg. With its belt tracks it looks more like a digger than a robot, but they enable it to move around and look for objects to drum on. If it finds interesting sounding objects, it records the sound and composes a rhythm with the samples.

In contrast to BDRD however, the Yellow Drum Machine uses ultrasonic sensors to locate objects. Most current robots use ultrasound, cameras or laser to orient themselves, at the moment there are a few who rely solely on touch. A recent paper titled Biomimetic vibrissal sensing for robots examines the use of whiskers as an additional sense for robot orientation. At the time of writing however, BDRD does not have many deafblind colleagues.

Credits

This project was realised as part of the course Creative Robotics at Goldsmiths, lead by Patrick Tresset. The robot was built in collaboration with Thibault Lelievre, Thomas Tanay and Dean Spiridonov.