Imagine if you were working with a robot that could leave you written messages, or draw diagrams to explain concepts. Such a scenario has come a step closer to reality, as a university student has taught a robot how to copy what we write, and what we draw.
Atsunobu Kotani, an undergrad at Rhode Island’s Brown University, started by creating a deep-learning algorithm that enabled a robot to fairly-accurately determine which pen strokes had been utilized – and in what order – to produce handwritten words and hand-drawn images.
Utilizing a pen held in its arm, the robot was subsequently able to reproduce a variety of words that it hadn’t previously seen. These included the word “hello,” written by different people in 10 languages that employed different characters – the languages included Greek, Hindi, Urdu, Chinese and Yiddish. The robot’s ability to do so was a little surprising, as the algorithm had initially been trained solely using Japanese characters.
Along with the handwriting, the robot also reproduced some rough line sketches, including one of the Mona Lisa.
“A lot of the existing work in this area requires the robot to have information about the stroke order in advance,” says Kotani’s advisor, Asst. Prof. Stefanie Tellex. “If you wanted the robot to write something, somebody would have to program the stroke orders each time. With what Atsu has done, you can draw whatever you want and the robot can reproduce it. It doesn’t always do the perfect stroke order, but it gets pretty close.”