Monday, November 4, 2019

Trimify2x: a Post-Processing Program to Improve Sandify Sand Table Patterns

Update 5/3/20:

Sandify now contains code that will minimize sand table perimeter travel, so my Perl program, described below, is now obsolete.

Sandify now has edge motion minimization built in, and automatically applied.  If you use the switch in the green rectangle, above, additional minimization that ignores the pattern sequence will be applied. I don't use that for the sand table, but it could be useful for a machine that uses an airbrush to paint patterns on walls, floors, etc.

Original content of this post:

The Spice Must Flow sand table is one of my favorite projects.  It uses a magnetically driven steel ball to draw patterns in sand.  The patterns that drive the table are stored as gcode files generated by a great program called Sandify.

A pattern created using Sandify, drawn on The Spice Must Flow.

I like to run the table fast, typically with maximum speed set to 500 mm/sec, so that it's interesting to watch as it draws patterns.  Here's a short video of it erasing at 1m/sec:

Many people who have seen it say that they could stand and watch it all day, and I have to agree.  It's mesmerizing.  I've been working on making the mechanism run quietly at 500 mm/sec and largely succeeded, but there was one remaining problem.

Something I noticed almost as soon as the table was working over a year ago was that when the pattern being drawn exceeds the size of the table, the ball zips around on the edges of the table, sometimes back and forth along one or two edges, for quite a while before it gets back to drawing lines on the table. It's a problem because it takes a lot of extra time, increases wear on the mechanism, and it's not nearly as interesting to watch as lines being drawn on the table.

Here's a video that shows some of the wasted back-and-forth motion along the edges (watch from about 0:50 to 1:30):

The Spice Must Flow from Mark Rehorst on Vimeo.

As in a 3D printer, the ball's motion on the sand table is subject to acceleration and deceleration.  That means that short line segments never reach the maximum firmware-allowed speed.  All that crawling around on the edges of the table can really slow things down because it is very often short back and forth motion before the ball finally goes back out on the table and starts drawing in the sand.

I decided to write a program that will read the Sandify pattern file and strip out the lines that have the ball running back and forth along the edges.  The edited version of the pattern file will generate the exact same pattern with minimal time wasted along the edges of the table.

Sandify Pattern File Structure

The Sandify pattern files are just text files that start with a header that includes the file name, the parameters that are used by the Sandify program to generate the patterns including the dimensions of the pattern.  The rest of the file is the actual pattern in the form of a long series of G01 commands (basic gcode) with X,Y coordinates.

Here's a typical, but shortened, Sandify pattern file:

; Created by Sandify


;   Sandify Version: 0.1.3

;   Machine Type: Rectangular
;     MinX: 5 MaxX: 710 MinY: 5 MaxY: 1570
;   Content Type: Shapes
;     Starting Size: 800
;     Offset: X: 382 Y: 261
;     Selected Shape: Polygon
;       Polygon Sides: 4
;     Number of Loops: 165
;     Spin: true
;       Spin Value: -4
;       Spin Switchbacks: 0
;     Grow: true
;       Grow Value: -2
;     Track: true
;       Track Count: -37
;       Track Size: 0.8
;       Track Grow: false
;   Path Reversed: false

; filename: 'square1.gcode'


G28 Y
G28 X
G01 F30000; END PRE
G01 X710.000 Y1570.000
G01 X154.163 Y1570.000
G01 X153.366 Y1570.000
G01 X132.599 Y489.079
G01 X710.000 Y460.418
There are about 1100 of the G1 command lines (this is a very small pattern file) and then the file comes to an end:

G01 X5.000 Y587.353
G01 X5.000 Y36.396
G01 X5.000 Y5.000
G01 X294.080 Y5.000
G01 X710.000 Y5.000
G01 X710.000 Y1570.000
G01 X5.000 Y1570.000
G01 X5.000 Y705.184


G04 S60
; square1.gcode; END POST

The line highlighted in yellow lists the dimensions of the pattern and will be useful later...
The patterns that move the ball are always a series of G01 commands which makes it easy to recognize those lines.  Each line such as "G01 X738.000 Y973.789" consists of a command (G01 - equivalent of a go-to statement) and coordinates of the point to which the ball should move (738.000, 973.789) from wherever the previous command left it.  If MinX=5, G01 commands that move the ball to/along the left edge will always contain X5.000, likewise the other axes.

After months spent thinking about how to process the files in a single pass, nothing was done. It seemed to require detecting all the possible transitions from one corner or edge to another, etc., and was a bit overwhelming. I was familiar with this uncomfortable state from my engineering days- it's the dreaded "analysis paralysis" in which one has too much information and too large a task to do anything.  

Stage 1

One day, after generating a particularly nice pattern for The Spice Must Flow (see photo above), I became really annoyed with the amount of time the ball spent moving around at the edges of the table.  I decided to try to manually edit the pattern file to cut that stuff out.

I looked for sequences of statements where the ball just went back and forth along an edge of the table.  What I realized was that those sequences are very easy to spot visually, so editing them out was also very easy.  I manually edited a couple pattern files (about an hour each!), stripping out the excess edge movement, first on one edge, then the next, etc., then ran the edited files on the table to see if they really improved performance.  Oh yeah!  The ratio of productive drawing time to useless edge movement was increased dramatically.

Since the files consist mainly of G01 commands, all I had to do was look for sequences like the one highlighted in blue and yellow, where X5.000 represents the left edge of the pattern:

G01 X15.000 Y23.000

G01 X5.000 Y100.000
G01 X5.000 Y230.000
G01 X5.000 Y110.000
G01 X5.000 Y321.000
G01 X5.000 Y16.000
G01 X15.000 Y21.000

and transform it to this by chopping out the stuff in yellow:

G01 X15.000 Y23.000

G01 X5.000 Y100.000
G01 X5.000 Y16.000
G01 X15.000 Y21.000

Some of those sequences are 100 statements long, and there may be hundreds or even thousands of those sequences in any given pattern file.  NotePad++ made it particularly easy to find the sequences by using the Search > Mark > Mark All function.

Sometimes the way out of analysis paralysis is to just start working on a piece of the problem and see where it takes you.  Buoyed by my new-found confidence in my ability to make an improvement, I decided to try to automate the edits I had done manually. A little research pointed me at Perl, a programming language specifically developed to process text files. I didn't know anything about programming in Perl, so I got a couple books and used some on-line searches.  As I studied the books and ran small test files, I kept thinking about how to scan the Sandify pattern files, what needed to be kept, and what could be thrown away.  It took about 4 days to get the first stage of the program working.

The first thing the program does is locate the statement in the header that contains the maximum and minimum values for each axis in the pattern. Next, it starts looking for and trimming sequences along pattern edges.

It's actually pretty easy (in hindsight, isn't everything?) to find those sequences that contain multiple statements along each edge- just examine three lines at a time and write the second line to the output file unless all three lines contain the edge value being checked.  That ensures that single and double instances of an edge statement pass through to the output file, but if there are three or more in sequence, only the first and last are written to the output file.  For more details, download the program and take a look at it.  I was very liberal with comments so it should be easy to see what's going on.

There is probably a way to have the program clean up all four edges in a single pass using an array, but I'm not much of a Perl programmer, so I wrote separate passes for each edge value, just as I had manually edited the files, creating intermediate files along the way and then deleting them just before the end of the program.

Processing the pattern files this way leads to a reduction in the file size (not that it matters much), a significant speed up in pattern completion, and much less boring edge motion.

Stage 2

The patterns I generate for the table frequently start with a very large base shape offset beyond the table's boundaries, and shrink and rotate it as the pattern cycles, letting it get to zero size and then allowing it to grow after that.  As the pattern grows it overwrites the stuff that was drawn as the pattern was shrinking.  That process leads to beautiful and interesting patterns that look different every few minutes throughout the drawing process but often contain a lot of unproductive, boring, edge and corner to corner motion.

Once all the excess edge motion was trimmed out, the pattern files now contained sequences of statements with corner coordinates.  Those sequences contain the remaining unproductive motion. The trimming program didn't add those corner statements - they were in the file the whole time, but buried among the useless motion along the edges. Now it was time to minimize unnecessary motion through the corners.  

If the ball leaves the table on one edge and reenters the table on the opposite edge, there are two paths it can take along the edges, both going through two corners of the table/pattern.  One way will usually be shorter than the other.

Two-corner path with the ball exiting the table on the left edge and reentering along the right edge.  Blue square is the edges of the pattern/table. Green is the ball.  Orange is the time wasting long path, black is the more desirable short path. 

Likewise, if the ball leaves the table along one edge and comes back on the table along an adjacent edge, there are two paths it can take.  One path goes the long way through three corners, and the other, shorter path, goes through only one corner.

A 3-corner path.  The path through the 4th corner (black) is always shorter than the path through 3 corners.

If the ball leaves the table on one edge and returns to the table along the same edge, sometimes it takes the long way between the two points and travels through all four corners of the table.  That's a silly waste of time!

A 4-corner path orange), a big time waster.  Simply deleting the 4-corner sequence forces the shorter (black) path.

Of course, sometimes the ball has to pass through a single corner and that's OK (the alternative is to take the longer, 3-corner path around the table- nope!).  There's no wasted motion there.

The second part of the optimizer program searches for the sequences of corner statements illustrated above, leaving single corners alone, checks two corner sequences and replaces the longer path with the shorter path through the other two corners, replaces three corner sequences with the shorter path through the fourth corner, and deletes sequences of four corners because they start and end on the same edge.

In the case of multiple loops around the table, as in the 7-corner path (orange) illustrated below, the first four corners will be deleted, and the remaining 3-corner path will be converted to the 1-corner path (black), greatly reducing the time wasted zipping around the edges of the table.

A 7-corner path (orange) that wastes a lot of time gets converted to the 1 corner path by first deleting the first four corners and then converting the remaining 3-corner path to the much faster 1-corner path (black).

I've run many test files and pattern and the trimming program seems to work reliably.  There's a minimal amount of wasted time zipping around the edges of the table now, which makes it much more interesting to watch as the patterns are drawn.

You can get the post-processing program here...

(((note- there's an updated version linked near the bottom of this post)))))

You'll need Perl to run the program.  I used Strawberry Perl for Windows, but the program should work with any recent Perl 5 on any OS.  To run the program, just open the Perl command line app, point it at the directory where you keep the target Sandify pattern file, and type "perl" Type in the input file name when it asks, hit "enter", and you'll find the trimmed pattern in a file called clean.gcode.  That's it.

A Sample of the Results

The picture at the top of this post is a pattern file that starts with a heart shape that is larger than the table area.  With each cycle of the pattern, the heart shrinks a bit bringing it onto the table, first along only one edge.  With each cycle, the ball travels around the perimeter of the table including a little back-and-forth motion along one edge.  Here's video of the start of the pattern, as generated by Sandify:

Untrimmed pattern running on sand table from Mark Rehorst on Vimeo.

That looping around the table goes on for quite a while before it starts drawing anything at the opposite end of the table.

The next video is the start of the same pattern after it has been optimized using the  There are essentially two optimizations seen here.  First, the back-and-forth motion along one edge is eliminated.  The second, and much more important optimization in this case, is that looping around the edges of the table is eliminated, greatly speeding up pattern generation and improving watchability of the table.

The same pattern with wasted edge motion trimmed out from Mark Rehorst on Vimeo.

This is just one example of the improvement that comes from trimming the Sandify pattern files using  The original file, heartfuzz1.gcode was 359 kB long.  The trimmed file, heartfuzz1_clean.gcode is just 193 kB long.  The difference represents wasted motion along the edges of the table.  But how much wasted motion?  On my table, with acceleration set to 1000 mm/sec^2 and speed set to 500 mm/sec, the untrimmed file takes 59:20 to complete.  The trimmed file requires only 42:03 to complete the same pattern.  That's a difference of more than 17 minutes of the ball moving around the edges of the table, producing the exact same pattern.

I'm going to try to make a couple time-lapse videos of a pattern, one untrimmed and the other trimmed, and combine them into a single side by side video.  If and when I get it done, I'll post it here.

Making Sequences of Patterns

The RepRap Firmware that runs on the Duet controller boards has enabled the M98 gcode command.  That command allows you to create playlists of patterns to run on a sand table.  You put your pattern files (already cleaned by Trimify) on the uSD card on the controller board in the default /gcodes directory.  Then you create a sequence file and put it into the same directory.

The sequence file should look something like this:

M98 P"0:/gcodes/wipe0.gcode"

M98 P"0:/gcodes/heartfuzz1_clean.gcode"
G04 S90
M98 P"0:/gcodes/wipe0.gcode"
M98 P"0:/gcodes/mishegoss1_clean.gcode"
G04 S90
M98 P"0:/gcodes/wipe0.gcode"
M98 P"0:/gcodes/mishegoss2_clean.gcode"
G04 S90

This sequence will erase the table using the wipe0.gcode pattern, then run the heartfuzz1 pattern, then wait 90 seconds, then erase, then run the mishegoss1 pattern, etc.  You can make the sequence as long as you like.  You can even create files that call different sequences.  Unfortunately, there doesn't seem to be any way to automatically randomize the pattern sequence, so you'll have to do that yourself.

Name the sequence file something like "sequence_01.gcode".  You can then select that file to run using the web interface on the controller. 

UPDATE!  11/25/19

I have discovered that the "clean" pattern files occasionally still contain some useless motion along the corners/edges.  It seems to occur when the useless stuff in the pattern bends around a single corner.  Running Trimify a second time using the "clean" output file as input has fixed it in every case (which isn't many) I've seen.  I have run Trimify multiple times on some test files, and it has never damaged the files, so I will be modifying Trimify to automatically perform two passes on the input file and will post a link to the new version here as soon as it is done.

UPDATE!  11/30/19

The residue problem is now solved (I think), and I've made a couple additional tweaks to the program which is now called

The first change is that the output file is now named from whatever the input file is.  If you enter "mystars.gcode" for the input file name, the output file name will be "mystars_clean.gcode".  The second change is that program now makes two full passes over the input file and I believe completely eliminates any "residue" left from the first pass.

The compare plugin in notepad++ was extremely useful for checking operation of the program.  It compares and highlights differences between two files.  Here's a comparison of a sandify pattern called jazz2.gcode and the output of trimify called "clean.gcode" (click on the image and open in a new tab to make the text readable):

This is the notepad++ comparison of the original jazz2.gcode file (right) with the trimify output file clean.gcode (left).  All the stuff that's highlighted in green on the input file is gone in the output file.  Notice that line numbers 1570 and 1574 both contain G01 X710.000 Y5.000 and both end up in sequence in the output file.  
This is a comparison of the output from the original trimify - clean.gcode (left) and the output from trimify2x - jazz2_clean.gcode (right).  The useless motion that sends the ball to the corner of the table at X710.000 Y5.000 is now eliminated.  In this particular input file, that was the only change needed.  I have seen other pattern files with more of this sort of thing.
 In this particular example, the second pass of the maxx trimming is what takes out the duplicate and useless G01 X710.000 Y5.000 lines.  Trimify2x runs the whole process twice including all the edge and corner trimming and substitution.  I suspect that the corner trimming and substitution may never do anything on a second pass, but it was easy to include it so I did. is every bit as crude as  I was going to try to set it up to loop but found it much easier to just append a copy of the original stuff to the end of the program, then make a couple minor tweaks to the input and output file names.

I am not planning on making any more changes to the program unless someone points out a bug. is located here.

Here's a better example of before and after trimify2x.  First, the original pattern straight out of Sandify:

Sandify pattern running on The Spice Must Flow sand table. from Mark Rehorst on Vimeo.

And here is the result of running trimify2x on that pattern file:

Sandify pattern file cleaned by Trimify2x from Mark Rehorst on Vimeo.

This particular pattern went from taking 2 hours down to taking only 57 minutes.  Over an hour was wasted at the edges of the table in the original pattern file.

This is what that entire pattern looks like when it's finished:


Trimify (and trimify2x) may give unexpected results if you string together a bunch of pattern files and then try to optimize them in one go.  The program only looks for the MinX, etc., declaration statement once.  If you change the dimensions from one pattern to another, the optimizer will probably screw things up.  It's best to optimize the patterns individually, before stringing them together.

Trimify (and trimify2x) is every bit as crudely coded as you would expect a first-time Perl programmer's code to be.  There's no error checking, range limiting, etc.  If you give it bad input it will try to process it.  I have found that occasionally I get compiler warnings about a variable getting assigned a string instead of an expected numeric value.  I have looked at the files that cause that warning to be generated and can't see any reason for the warning.  The resulting trimmed output files look and work fine on the table, so I'm not  really motivated to try to track down the source.

Now that the excess edge and corner motion has been eliminated, the edges of the pattern are sometimes left looking a little ragged once the pattern is complete.  That can easily be fixed by simply sending the ball on a single trip around the edges of the table.

No comments:

Post a Comment

Leave comments or a questions here and I'll try to post a response as soon as I can.