PaperActive
PaperActive is a project I've started to create a drawing application (think of blank sheet of paper, whiteboard, chalkboard, etc.) to aid in learning and exploration. The application will understand what you are drawing and bring it to life and make it interactive.
The inspiration for this project came from two people.
Michael Nielsen
Watch this video by Michael Nielsen where discusses neural networks and the universal approximation theorem. Notice how the application he uses understands something about neural networks. When he draws a circle the app recognizes that he is drawing a neuron. His imperfect circle becomes a perfect circle. But it's not just a circle. It has properties, including a weight and bias. He can then draw coordinate axes and link the neuron to the axes and visually see a graph of the neuron's function.
After watching that video I had to know more about the application he was using. It turns out that Nielsen created it himself. It was a prototype he called Magic Paper, and it was inspired by another system created by a man named Ken Perlin.
For Nielsen this was just an experiment and something he does not intend to improve upon. This is reflected in the quality of the code he wrote. It is not designed well enough to become a more robust system. It was meant to be just enough to complete his research. The algorithm he uses to recognize what is drawn involves comparing distances between its vertices and the vertices in predefined "glyphs" or symbols. This algorithm is not sophisticated enough to reliably distinguish between things such as a circle and a square. You can see this problem in the above video around the 3:27 mark.
He created the system to explore neural networks. It would be exciting if the system used a neural network to recognize what is drawn!
Ken Perlin
Ken Perlin created a system with students at NYU called Chalktalk. Their goal was to create an educational system that combined the narratives that take place at the chalkboard with the power of a computer program. He explains that if you only have a chalkboard, then whatever is drawn on it "just sits there," and if you only have a computer program, then the student can just pull it up on their smartphone and they are not engaged in the narrative happening at the chalkboard. Combining the two can help a teacher teach more effectively.
This is one of the coolest videos I've seen, but after researching a little deeper there are things I would like to do better.
After reading the documentation I feel that the system has a very difficult user interface. Instead of a teacher writing naturally as they would on a chalkboard, they have to memorize strokes and perform them in a predetermined order and direction. Perlin had to draw the pendulum exactly the way you saw him do it. He had to start at the top with the base, then draw the rope, and then end with the weight (circle). Any other order would have confused the system.
In many cases the direction of the strokes are just as important. Drawing a line from left to right is not the same as from right to left. I think this has a lot to do with solving Nielson's problem of distinguishing a circle from a square. This would be easier to do if you required the user to draw circles in a counter-clockwise direction and squares clockwise. But a better system would let the teacher draw it however they wanted.
Another issue with the interface is the controls. It's hard to tell in the above video because you can't see Perlin's screen or his hands, but each object has an invisible radial menu associated with it that Perlin has memorized. You have to click at invisible areas at specific angles around objects in order to interact with them in different ways. Watch the video below at around the 00:11 mark to see what I mean. Notice that before Perlin can connect the pendulum to the graph he has to tap his pen somewhere off to the side of the pendulum.
You can see him do this again around the 00:57 mark to connect the triangle to the graph. It looks very simple because he has had so much practice at it. Every unique interaction (drawing, moving, scaling, animating, etc.) requires tapping a different location. If you turn up the volume and watch fullscreen you can see and hear a lot of taps with the pen. All of these are necessary and calculated.
Notice all the similarities in the two videos? Notice how he draws many of the same objects, usually in the same order, and interacts with them in the exact same ways? Perin's demos are not spontaneous and improvised. They are very rehearsed. I think this is because the system is difficult to work with and not that flexible.
A better system would have controls that were intuitive. Imagine you were using this system on a tablet that supported multi-touch. What would feel like a natural way to connect the pendulum to the graph? How many fingers do you think you should use? What motion do you think you should make? The system should have controls that reflect what users will naturally feel to be true. You shouldn't have to spend significant time learning the controls.
Goals
Here are some of my project goals:
- Train a neural network (or system of networks) to recognize what is being drawn so that the user can draw in whatever way feels natural to them.
- Use intuitive controls, supporting multi-touch devices.
- Create a simple API for developers, creating a common interface to hide the complexity and differences between a 2d and webg canvas.
One last goal: Nielsen felt that Perlin's system was designed only to explain and educate, but he wanted to know if a similar system could be built for thinking and exploration. He said,
Explanatory media are often highly polished, presenting a finished product from which distracting elements have been eliminated. By contrast, exploration involves distractions, wrong paths, and dead ends.
when education and real practice are separated, it can result in busy work on poorly-motivated “educational” problems. An interesting project would be to create a single environment which can be used for both education and research.