An Introduction to Canvas in GNU Emacs

1. Frequently Asked Questions

“Is this related to HTML5 Canvas?”

No. It has nothing to do with any web framework or technology. We call it “canvas” because it allows the user to draw arbitrarily on to a surface in an Emacs buffer. So at a high-level, both the HTML5 and Emacs canvas do the same thing, but they have zero relation in their implementation or how they work. HTML5 Canvas also has a more rich API with drawing functions and such, right now Emacs’ canvas API integrates with the existing image API and just provides access to the underlying pixel buffer via dynamic module API. See Canvas via Dynamic Modules.
“How is this different from SVG that Emacs has support for since a long time?”

Good question! So, at the level of an Emacs buffer the canvas looks and behaves just like any other Emacs image, including SVGs. Where it differs from SVG and other images is:
- a canvas’ :data is just ARGB32 pixel data (in strings or vectors), so no specific compressed format.
- the ability to access an Emacs Lisp image object from a lower level and be able to manipulate it
- update the image without having to call a full redisplay
“Isn’t this still limited by Emacs’ redisplay engine?”

Not entirely. We have a function called canvas-refresh that is to be called every time you wish to update the canvas, and this function doesn’t use/call Emacs’ redisplay, it has its own redrawing path that only updates the specific glyphs in the buffer that pertain to that canvas. Thus, whereas a full redisplay would cost more because it will try to update all the glyphs that changed, canvas-refresh only updates what needs to be. Since Emacs 25 we have double-buffering so if that’s enabled in the frame you’d still need to call redisplay to flip the buffers, but that won’t slow down your canvas. If you’re not convinced, see the demos below that run smooth at 25, 30, and 60FPS without causing any lag to the rest of the Emacs session. Or, look at this where I display a 2K60 FPS and a 720p 30FPS video side by side without Emacs missing a beat!
“Is this hardware-accelerated? If not, why are you not using the GPU for this?”

Because we simply don’t need it :D! A canvas simply exposes a pixel buffer, and Emacs simply redisplays it as it changes. You can get your pixels hardware accelerated outside of Emacs if you wish to, and then put them in canvas’ pixel buffer. This will give you hardware acceleration where you need it the most, pixel calculation, texture math, etc. Ideally, an introduction of a GPU powered redisplay within Emacs itself would be great and certainly help but one can still reap much of the benefits of HW acceleration from current Emacs via canvas.

2. What Is a Canvas?

Before answering that question, we should ask: “what is an image in GNU Emacs?” Everybody has seen Emacs display dazzling images since decades, and of varying formats. How does Emacs do that? Well, at the level of an Emacs buffer we have something called Text Properties (also see Overlay Properties), as its name suggests it is a property list (aka plist) for a character position or a string within a buffer. Among the many properties a text can have, there’s a special one called the display property. This special property is responsible for how the text gets displayed. This is exactly what we use to display images! You use create-image to create an image object, which is exactly a plist, and then you set this image object (called an image specification) to be the display property of some text or overlay (which are like text properties but without the "text", it’s an object that belongs to a particular buffer with specific beginning and end and along with properties just like for usual text).

This is briefly how the API of how images work in GNU Emacs. The new canvas feature works in full compatibility with this API. In short, at the level of an Emacs buffer, a canvas is just another image type! Indeed, compiling the latest GNU Emacs from source and evaluating: (image-type-available-p 'canvas) should return a t.

The immediate question then is: if Emacs already supported images, and if canvas just follows the same API, then what was the point of it? Why have a new image type, when we have XBM, PPM, PNG, GIF, SVG, etc.? Well, the two-fold short answer is:

You need to pass image data via strings to create these objects and for most of the above formats it’s expensive to do so (except SVGs)
Any arbitrary external image data cannot be directly displayed without throttling Emacs strings or objects, etc. (i.e., there’s no low-level access to the image data)

With SVGs you can mostly do really good things as long as they are simple enough, the moment you lead to have complex graphics at high refresh rate, Emacs starts throttling. I wanted in 2025 was exactly a way around this, a way to access the image Emacs is displaying at pixel-level so that I can update it in-place by fiddling with the pixels without having to pass strings or Emacs Lisp objects through the garbage collector. And this is what a canvas provides.

So, there are two levels at which a canvas can be viewed and used:

purely from Emacs Lisp as an image object
as an ARGB32 pixel buffer via dynamic module API

This allows for full integration with the existing Image API of Emacs while not sacrificing on low-level efficiency of being able to manipulate the pixel buffer directly. Now I will demonstrate how one can use the canvas in these two ways separately and/or simultaneously.

3. Creating and Using A Canvas

3.1. Canvas via Emacs Lisp Only

Like usual Emacs images, one can create images either by using create-image or manually building the image spec plist. For a canvas, the latter approach would look like this:

(setq test-canvas `(image :type canvas
                          :id test
                          :data-width 10
                          :data-height 10
                          :data nil))

This is what a canvas image’s specification looks like, the only difference between this and other images (other than the obvious :type canvas) is that canvas images need an :id property, this is used to uniquely identify each canvas.

If you used instead: (setq test-canvas (create-image nil 'canvas t)) to create the canvas, it would look almost the same:

(image :type canvas :data nil :scale default :id g138)

Since, of course, we didn’t provide any data in either of the cases, they are not technically valid canvas images, but that is how it looks like. To make things more interesting, remember that the "data" that a canvas accepts is ARGB32 pixel arrays. This can be done via Emacs Lisp through either unibyte strings or vectors. The latter is a bit easier to showcase, so I will go continue with that.

First, let’s create our array of pixels. We use make-vector:

;; A red square of 10x0 size
(setq rect-vec (make-vector (* 10 10) #xFFFF0000))

This creates a vector of 10x10 size containing each element as #xFFFF0000 (red) 32-bit ARGB pixel. It’s literally an pixel by pixel array vector that contains the color red. This is valid data for our canvas, we can now create it manually:

(setq rect-canvas `(image :type canvas
                          :id rect
                          :data-width 10
                          :data-height 10
                          :data ,rect-vec))

Now we can simply display this:

(insert (propertize "#" 'display rect-canvas))

After evaluating the above, you’ll see a tiny red square in your buffer! Voila, we now have a small canvas!

3.1.1. Note on Canvas Refresh

This might not seem too useful, because you can make a small rectangle via SVGs much quickly. The real value of canvas arises with the use of canvas-refresh function using which you can update the canvas without requiring a full redisplay. It only touches the glyphs which cover the particular canvas and updates them immediately. This is significantly cheaper than calling a full redisplay that will try to update all the changes.

Also, canvas-refresh has an additional optional argument RELOAD-DATA which if non-nil will reload any new updated data (either via the :data property or change in :file) from the canvas. It will be demonstrated later how to use this function properly.

3.2. Canvas via Dynamic Modules

As suggested in the introduction, canvases can be used either via Emacs Lisp and/or via dynamic modules. To know more about dynamic modules in Emacs and how to use them effectively, please consult my previous article on it. Emacs 32 provides a simple API for dealing with canvases from dynamic modules. The idea is simple, once a canvas has been created you can simply call the canavs_data function on it and it will provide you with the pixel buffer associated with the canvas. Once again, this buffer is only valid for ARGB32 pixel data.

static emacs_value
Fcanvas_update(emacs_env* env, ptrdiff_t nargs, emacs_value args[], void* data) {
  emacs_value canvas = env->args[0];
  uint32_t *canvas_pixel = env->canvas_data(env, canvas); // accessing canvas' pixel buffer
  if (canvas_pixel)
    {
      // do some pixel manipulation to the canvas pixel
    }
}

env->funcall(env, env->intern(env, "canvas-refresh"), (emacs_value[])[rect_canvas, Qnil]);

The above dynamic module function, when embedded in a legitimate dynamic module, compiled and loaded via module-load, would result in updating the canvas repeatedly by changing the colors (or whatever the pixel manipulation code does). This is how we can surpass Emacs Lisp’s limitations by accessing a low-level representation of the canvas we can manipulate directly. And since canvas-refresh avoids throttling Emacs’ full redisplay we do not encounter any slowness at all! And since we don’t use :data via Emacs Lisp image spec, whenever canvas is used via dynamic modules the RELOAD-DATA argument of canvas-refresh is to be nil.

4. Demonstrations

Here we showcase a few neat graphical animations you can do with canvas, both in Emacs Lisp and dynamic modules. For our previous demonstrations, consult Tushar’s blog post where he collects all of the experiments I did with and without Emacs. Also, look at the Minad's emacs-canvas-patch which has some basic demos (including a fancy mode-line one!). The first two of the demos below would be purely in Emacs Lisp and the last one would be in C via a dynamic module.

4.1. A Bouncing Ball

Let’s start with creating and setting up our canvas. We’ll go with dark violet (#xFF8B00FF) for the background:

(setq W 250
      H 250
      BG #xFF8B00FF)
(setq ball-canvas-vec (make-vector (* W H) BG))
(setq ball-canvas `(image :type canvas
                          :id rect
                          :data-width ,W
                          :data-height ,H
                          :data ,ball-canvas-vec))
(insert (propertize "#" 'display ball-canvas))

This is exactly what we did in the previous section, just in violet! Now is the interesting part, we want this 250x250 pixel canvas to have a ball bouncing. Well, first we prepare the ball, it will have some radius, an initial position and some initial velocity (and color!):

(setq ball-radius 10)
(setq bx (/ W 2)
      by (/ H 2)) ; start from the center
(setq dx 4
      dy 3) ; initial velocity
(setq ball-color #xFFFFFF00) ; yellow

Now we need to actually draw the ball, now I will ask you to remind yourself of some analytical geometry for this! The way we draw the ball is we take a point to be the center of the circle and check if we can build a circle around a certain bounding box. If ball-radius is 15, thus the diameter is 30, so we need at least a 30x30 square to fit the circle. Now to draw the actual circle, we’ll use the equation for a circle:

$x^{2}+y^{2}\leq r^{2}$

Putting this in Emacs Lisp, we get:

(defun draw-ball (cx cy color)
  (let ((r2 (* ball-radius ball-radius))) ; Radius squared
    ;; the nested loop checks for the circle within the square
    (dotimes (y-off (1+ (* 2 ball-radius)))
      (dotimes (x-off (1+ (* 2 ball-radius)))
        (let* ((px (+ cx (- x-off ball-radius)))
               (py (+ cy (- y-off ball-radius)))
               (dx-local (- px cx)) ; Horizontal distance from center
               (dy-local (- py cy)) ; Vertical distance from center
               (dist2 (+ (* dx-local dx-local) (* dy-local dy-local)))) ; squared distance using pythagorean theorem
          ;; check if pixel is inside the circle AND inside the canvas bounds
          (when (and (>= px 0) (< px W) (>= py 0) (< py H) (<= dist2 r2))
            (aset ball-canvas-vec (+ (* py W) px) color)))))))

The above function should make sense, while it is not the most efficient way to draw a ball, it’s relatively simple enough for the demonstration. Now we simply need to check the physics for making sure it stays within the canvas:

(defun ball-bounce ()
  (setq bx (+ bx dx) by (+ by dy))
  ;; compare with ball-radius so the *edge* of the circle bounces, not the center.
  (when (<= bx ball-radius) (setq dx (abs dx)) (setq bx ball-radius))
  (when (>= bx (- W ball-radius)) (setq dx (- (abs dx))) (setq bx (- W ball-radius)))
  (when (<= by ball-radius) (setq dy (abs dy)) (setq by ball-radius))
  (when (>= by (- H ball-radius)) (setq dy (- (abs dy))) (setq by (- H ball-radius))))

And now we simply do it in a loop!

(setq ball-frame 0)
(setq max-ball-frames 2000)
(defun ball-loop ()
  (if (>= ball-frame max-ball-frames)
      (cancel-timer ball-timer)
    (setq ball-frame (1+ ball-frame))
    (draw-ball bx by BG) ; make old ball invisible (turn it into background color)
    (ball-bounce)
    (draw-ball bx by ball-color)
    (canvas-refresh ball-canvas t)))
(setq ball-timer (run-with-timer 0 0.02 'ball-loop))

Evaluate it all and you would have a yellow ball bouncing inside a violet square. It automatically stops after 2000 frames, but you can have it go on indefinitely as well! Here’s what it will look like:

4.2. Chaotic Double Pendulum

One of the things that fascinated me when I was studying classical mechanics for the first time was the double pendulum. The math underlying this: the Euler-Lagrange equation and variational calculus in general was also very fascinating to me. If you think it’s not, I suggest you look into the history and math behind the Brachistochrone curve. It’s evident from all this, that I want to put this into visualization using canvas!

The two equations in Lagrangians which describes the chaotic system:

\displaystyle\ddot{\theta}_{1}=\frac{-m_{2}\cos(\theta_{1}-\theta_{2})\,l_{1}% \dot{\theta}_{1}^{2}\sin(\theta_{1}-\theta_{2})+m_{2}\cos(\theta_{1}-\theta_{2% })\,g\sin(\theta_{2})-m_{2}l_{2}\dot{\theta}_{2}^{2}\sin(\theta_{1}-\theta_{2}% )-(m_{1}+m_{2})g\sin(\theta_{1})}{l_{1}\left(m_{1}+m_{2}-m_{2}\cos^{2}(\theta_% {1}-\theta_{2})\right)}

\displaystyle\ddot{\theta}_{2}=\frac{(m_{1}+m_{2})\left(l_{1}\dot{\theta}_{1}^% {2}\sin(\theta_{1}-\theta_{2})+\dfrac{\dot{\theta}_{2}^{2}\sin(\theta_{1}-% \theta_{2})\cos(\theta_{1}-\theta_{2})\,m_{2}l_{2}}{m_{1}+m_{2}}+\cos(\theta_{% 1}-\theta_{2})\,g\sin(\theta_{1})-g\sin(\theta_{2})\right)}{l_{2}\left(m_{1}+m% _{2}\sin^{2}(\theta_{1}-\theta_{2})\right)}

Now let’s create the canvas as usual with black background and display it like before:

(setq W 250
      H 250
      BG #xFF101010)
(setq canvas-vec (make-vector (* W H) BG))
(setq canvas `(image :type canvas :id dp :data-width ,W :data-height ,H :data ,canvas-vec))
(insert (propertize "#" 'display canvas))

We set some constants: gravity, length of rods, and delta time:

(setq g 9.8
      L 60.0
      dt 0.05)

We set some initial conditions for the angles $\theta_{1}$ and $\theta_{2}$ ; angular velocities $w_{1}$ and $w_{2}$ , and a fixedd anchor point:

(setq th1 2.5
      th2 2.5
      w1 0.0
      w2 0.0)
(setq pivot-x 125
      pivot-y 80)

We need some helpers, we already have the ball one from before and one for drawing a line/rod and to set a pixel in the vector:

(defun set-px (x y color)
  (when (and (>= x 0) (< x W) (>= y 0) (< y H))
    (aset canvas-vec (+ (* (truncate y) W) (truncate x)) color)))

(defun draw-line (x0 y0 x1 y1 color)
  (let* ((x0 (truncate x0)) (y0 (truncate y0))
         (x1 (truncate x1)) (y1 (truncate y1))
         (dx (abs (- x1 x0))) (dy (abs (- y1 y0)))
         (sx (if (< x0 x1) 1 -1)) (sy (if (< y0 y1) 1 -1))
         (err (- dx dy)) e2)
    (while (not (and (= x0 x1) (= y0 y1)))
      (set-px x0 y0 color)
      (setq e2 (* 2 err))
      (when (> e2 (- dy))
        (setq err (- err dy))
        (setq x0 (+ x0 sx)))
      (when (< e2 dx)
        (setq err (+ err dx))
        (setq y0 (+ y0 sy))))
    (set-px x0 y0 color)))

(defun draw-circ (cx cy r color)
  (let ((r2 (* r r)))
    (dotimes (y (* 2 r)) (dotimes (x (* 2 r))
                           (let* ((px (+ (- cx r) x)) (py (+ (- cy r) y))
                                  (dx (- px cx)) (dy (- py cy)))
                             (when (<= (+ (* dx dx) (* dy dy)) r2)
                               (set-px px py color)))))))

And now the main procedure that takes care of the pendulum movment:

(defun pendulum-physics ()
  (let* ((dth (- th1 th2))
         (den (- 3.0 (cos (* 2.0 dth))))
         (num1 (+ (* (- g) 3.0 (sin th1))
                  (* (- g) (sin (- th1 (* 2.0 th2))))
                  (* (- 2.0) (sin dth) (+ (* w2 w2 L) (* w1 w1 L (cos dth))))))
         (a1 (/ num1 (* L den)))
         (num2 (* 2.0 (sin dth)
                  (+ (* 2.0 w1 w1 L)
                     (* 2.0 g (cos th1))
                     (* w2 w2 L (cos dth)))))
         (a2 (/ num2 (* L den))))
    (setq w1 (+ w1 (* a1 dt)) w2 (+ w2 (* a2 dt)))
    (setq th1 (+ th1 (* w1 dt)) th2 (+ th2 (* w2 dt)))))

The exercise is left to the reader to make sure the above code follows the math! Specifically compare the two equations we introduced initially with the above procedure. And now the main game loop:

(defun pendulum-loop ()
  (dotimes (i 12) (pendulum-physics))
  (fillarray canvas-vec BG)
  (let* ((x1 (+ pivot-x (* L (sin th1))))
         (y1 (+ pivot-y (* L (cos th1))))
         (x2 (+ x1 (* L (sin th2))))
         (y2 (+ y1 (* L (cos th2)))))
    (draw-line pivot-x pivot-y x1 y1 #xFF888888)
    (draw-line x1 y1 x2 y2 #xFF888888)
    (draw-circ x1 y1 8 #xFF00FF00)
    (draw-circ x2 y2 8 #xFFFF0000)
    (set-px pivot-x pivot-y #xFFFFFFFF))
  (canvas-refresh canvas t))
(setq my-timer (run-with-timer 0 0.05 'pendulum-loop))

And once evaluated, it will look something like this:

One can enable interactivity here by making the mouse be able to drag the pendulum and set the initial positions, this is very much doable by using Emacs’ track-mouse functionality. I am not including it here because it’ll be too complex.

4.3. Points in 3D Space

This demo is borrowed from Alexey Kutepov, aka tsoding. They built a graphics library called olive.c. The following demo is Dots3D example from olive.c. Since this is a dynamic module, all the pixel manipulation math would be happening in C and on the Emacs Lisp side we will take care of creating and displaying the canvas and just calling the dynamic module function in a timer to update the canvas. While the original demo didn’t have any mouse interactivity, we can include it in ours quite easily.

After doing the mandatory int plugin_is_GPL_compatible we define some constants in the dynamic module:

#define WIDTH 960
#define HEIGHT 720
#define BACKGROUND_COLOR 0xFF181818
#define GRID_COUNT 10
#define GRID_PAD (0.5f/GRID_COUNT)
#define GRID_SIZE ((GRID_COUNT - 1)*GRID_PAD)
#define CIRCLE_RADIUS 5
#define Z_START 0.25f

We’ve just inherited them from the original demo’s code. Now we need one helper to draw the actual circles/points in this grid/space. We can just convert into C the code we wrote earlier for drawing circles:

void draw_circle(uint32_t *pixels, int cx, int cy, int r, uint32_t color) {
  int r2 = r * r;
  for (int y = -r; y <= r; ++y) {
    for (int x = -r; x <= r; ++x) {
      if (x*x + y*y <= r2) {
        int px = cx + x; int py = cy + y;
        if (px >= 0 && px < WIDTH && py >= 0 && py < HEIGHT)
          pixels[py * WIDTH + px] = color;
      }
    }
  }
}

Now we need to write the actual module function that will be called from Emacs. Firstly, what should this function’s signature be? Since it’s a module function, it has to be like this:

static emacs_value render(emacs_env* env, ptrdiff_t nargs, emacs_value args[], void* data)

But what should be its arguments when called? Well, firstly it needs the canvas where it will render to. It will also need as arguments the angles from which to render the whole grid, because the grid must be rotating, so the angles will be changing continuously. So let’s get those arguments, and get access to the canvas’ pixel buffer as well:

static emacs_value render(emacs_env* env, ptrdiff_t nargs, emacs_value args[], void* data) {
    emacs_value canvas = args[0];
    float angle_x = env->extract_float(env, args[1]);
    float angle_y = env->extract_float(env, args[2]);

    uint32_t* pixels = env->canvas_data(env, canvas);
    if (!pixels) return Qnil;
}

Now the background must be painted, for which we’ll just loop through all the pixels and set them to BACKGROUND_COLOR. We’ll also get some float valuees for the upcoming math. We clearly need some camera math to make it move as well, and since we are in 3D we’ll need a 3-level nested for loop. So here’s what the final function looks like:

static emacs_value render(emacs_env* env, ptrdiff_t nargs, emacs_value args[], void* data) {
    emacs_value canvas = args[0];
    float angle_x = env->extract_float(env, args[1]);
    float angle_y = env->extract_float(env, args[2]);

    uint32_t* pixels = env->canvas_data(env, canvas);
    if (!pixels) return Qnil;

    for(int i = 0; i < WIDTH * HEIGHT; ++i) pixels[i] = BACKGROUND_COLOR;

    float cos_x = cosf(angle_x), sin_x = sinf(angle_x);
    float cos_y = cosf(angle_y), sin_y = sinf(angle_y);

    float camera_distance = 0.8f;  // How far the camera is from the center (Lower = closer)
    float focal_length = 800.0f;   // The "zoom" multiplier (Higher = more zoomed in)

    for (int ix = 0; ix < GRID_COUNT; ++ix) {
        for (int iy = 0; iy < GRID_COUNT; ++iy) {
            for (int iz = 0; iz < GRID_COUNT; ++iz) {
                float x = ix*GRID_PAD - GRID_SIZE/2.0f;
                float y = iy*GRID_PAD - GRID_SIZE/2.0f;
                float z = Z_START + iz*GRID_PAD;

                float px = x;
                float py = y;
                float pz = z - (Z_START + GRID_SIZE/2.0f);

                // 1. Rotate around Y axis (Yaw)
                float x1 = px * cos_y + pz * sin_y;
                float z1 = -px * sin_y + pz * cos_y;
                float y1 = py;

                // 2. Rotate around X axis (Pitch)
                float y2 = y1 * cos_x - z1 * sin_x;
                float z2 = y1 * sin_x + z1 * cos_x;
                float x2 = x1;

                // 3. Apply camera distance
                float z_cam = z2 + camera_distance;

                // 4. Perspective projection using focal length
                float screen_x = (x2 / z_cam) * focal_length + WIDTH / 2.0f;
                float screen_y = (y2 / z_cam) * focal_length + HEIGHT / 2.0f;

                uint32_t r = ix*255/GRID_COUNT;
                uint32_t g = iy*255/GRID_COUNT;
                uint32_t b = iz*255/GRID_COUNT;
                uint32_t color = 0xFF000000 | (r << 16) | (g << 8) | b;

                draw_circle(pixels, (int)screen_x, (int)screen_y, CIRCLE_RADIUS, color);
            }
        }
    }

    return Qnil;
}

After this we can just initialize the module:

int emacs_module_init(struct emacs_runtime *rt) {
    if ((size_t)rt->size < sizeof (*rt)) return 1;
    emacs_env* env = rt->get_environment(rt);
    if ((size_t)env->size < sizeof (*env)) return 2;
    Qnil = env->make_global_ref(env, env->intern(env, "nil"));

    env->funcall(env, env->intern(env, "defalias"), 2,
                 (emacs_value[]){
                     env->intern(env, "dots3d-render"),
                     env->make_function(env, 3, 3, render, "Render dots3d", 0)
                 });
    return 0;
}

This now needs to be compiled into a shared object, do not forget to add emacs-module.h wherever gcc looks for includes:

gcc -O2 -I%ssrc dots3d.c -o /tmp/dots3d.so -fPIC -shared -lm

Now we can use this from Emacs Lisp, some preliminary stuff:

(module-load "/tmp/dots3d.so")
(declare-function dots3d-render "ext:dots3d.c")

(switch-to-buffer (get-buffer-create "*dots3d*"))

(defvar dots3d-canvas)
(defvar dots3d-frame 0)
(defvar dots3d-time 0.0)
(defvar dots3d-last-time 0.0)

(defvar dots3d-angle-x 0.0)
(defvar dots3d-angle-y 0.0)
(defvar dots3d-auto-rotate t)

(setq dots3d-time (float-time))
(setq dots3d-last-time (float-time))

We setup the main canvas:

(setq dots3d-canvas '(image :type canvas
                            :data-width 960
                            :data-height 720
                            :margin (20 . 20)
                            :scale 1
                            :id dots3d))

We need to disable the mode-line and cursor:

(setq-local cursor-type nil
            mode-line-position nil
            mode-line-modified nil
            mode-line-mule-info nil
            mode-line-remote nil)

And we display the canvas:

(insert (propertize "#" 'display dots3d-canvas))

Now the main function that takes care of interacting with the space. We basically use track-mouse to update the canvas by calling the render function with new angles, every time we drag on the canvas. To be noted, we need to stop the auto-rotate while we are dragging.

(defun dots3d-start-drag (event)
  (interactive "e")
  (setq dots3d-auto-rotate nil)
  (let* ((start-pos (posn-object-x-y (event-start event)))
         (last-x (car start-pos))
         (last-y (cdr start-pos)))
    (when (and last-x last-y)
      (track-mouse
        (let (evt pos mx my)
          (while (progn
                   (setq evt (read-event))
                   (mouse-movement-p evt))
            (setq pos (posn-object-x-y (event-start evt)))
            (when (and (car pos) (cdr pos))
              (setq mx (car pos) my (cdr pos))
              (setq dots3d-angle-y (+ dots3d-angle-y (* (- mx last-x) 0.01)))
              (setq dots3d-angle-x (+ dots3d-angle-x (* (- my last-y) 0.01)))
              (setq last-x mx last-y my)
              (dots3d-render dots3d-canvas dots3d-angle-x dots3d-angle-y)
              (canvas-refresh dots3d-canvas))))))
    (setq dots3d-auto-rotate t)))
(local-set-key [down-mouse-1] 'dots3d-start-drag)

And the final render function in Emacs Lisp can be extremely simple, you just call the render function with slightly adjusted angles and we call it in a timer so that it keeps rotating:

(defun dots3d-update ()
    (let* ((time (float-time))
           (dt (- time dots3d-last-time)))
      (switch-to-buffer (get-buffer-create "*dots3d*"))
      (setq dots3d-last-time time)
      (when dots3d-auto-rotate
        (setq dots3d-angle-y (+ dots3d-angle-y (* dt 0.5))))
      (dots3d-render dots3d-canvas dots3d-angle-x dots3d-angle-y)
      (canvas-refresh dots3d-canvas)))
(run-with-timer nil (/ 1 60.0) 'dots3d-update)

And after evaluation, you should see as below and be able to move the grid by dragging it:

5. Conclusion & Further Work

As is evident, this feature has enabled a new universe of graphical capabilities within Emacs. Of such possibilities, we have only scratched the surface in a few places. We are very excited to see what kinds of applications hackers can build using this. Here are some possible applications that can be built on top of canvas in Emacs (some of them we might work on ourselves):

A full-fledged document reader in Emacs. (Emacs Reader, the project which started it all.)
A full-fledged video player
A drawing pad (screw M$ Paint)
A framework to make video games
Scientific plotting (integration with Calc)
Org babel integrations
Rendering LaTeX snippets directly.
A nice color picker like HTML’s color attribute
2D & 3D Graph viewer
Image & Video editors
A visual programming environment

And more… we’ve opened a discussion on such graphical applications within Emacs and how to make them composable, feel free to contribute there.