Filesystem Walking: Cancellation, Pt. 2

Read first: Filesystem Walking: Diving Deeper Than Beginner Tutorials, Pt. 1

Part 1 provided a good implementation of a filesystem walker using the Generator concurrency pattern. Before the walker can be used in real world applications or libraries two issues need to be addressed. In this Part 2, We’ll be exploring cancellation using the context package.

Where We Left Off (better maybe?, eh. not exactly)

The previous implementation in Part 1 had all the perks. Decoupled file processors, non-blocking, rate limited, and clearly implemented.

func Walker(ctx context.Context, path string) <-chan string {
	rch := make(chan string)
	go func() {
		defer close(rch) // Close when walk is complete to signal walking has finished
		
		// Use WalkDir intoduced in Go 1.16 which is faster than Walk
		err := filepath.WalkDir(path, func(path string, d os.DirEntry, err error) error {
			if err != nil {
				return err
			}
			if !d.IsDir() {
				rch <- path
			}

			return nil
		})
		if err != nil {
			log.Printf("encountered error: %v\n", err)
		}
	}()

	return rch
}

The required features still missing:

  • No cancellation. Walk will continue to put results on the channel until completely done
  • Does not follow symlinks

Wait, it can be stopped? (cancellation)

Using the context.Context it should be pretty straight forward to cancel the job. If a context with cancel or deadline is provided to the walker a select can be used to act on the context.Done() channel or the results channel

func Walker(ctx context.Context, path string) <-chan string {
	rch := make(chan string) // channel to send results and return to caller.
	go func() {
		defer close(rch) // Close the channel to signal readers the walk is complete
		
		err := filepath.WalkDir(path, func(path string, d os.DirEntry, err error) error {
			if err != nil {
				return err
			}
			
			// only send files but check ctx.Done() for dir and file
			if !d.IsDir() {
				select {	 
				case <-ctx.Done(): // return an error to end the filepath.WalkDir if done.
					return ctx.Err()
				case rch <- path:
					return nil
				}
			} else {
				// check for done even if item is not a file
				select {
				case <-ctx.Done():
					return ctx.Err()
				default:
					return nil
				}
			}
		})
		if err != nil {
			log.Printf("encountered error: %v\n", err)
		}
	}()

	return rch
}

Using Walker, for real? already?

There is still a small issue of the walker not resolving symlinks on filesystems that support the feature. However, now is a great time to check out how to actually use this walker in a small app.

In this example a context with deadline is created in 1 second and the Walker() function from above is run. Try both versions of walker and we’ll see right away context cancellation works.

package main

import (
	"context"
	"fmt"
	"os"
	"path/filepath"
	"time"
)

func main() {
	ctx, cancel := context.WithDeadline(context.Background(), time.Now().Add(1*time.Second))
	defer cancel()

	for p := range Walker(ctx, "/home/user/") {
		fmt.Printf("%s\n", p)
	}
}

Conclusion

With cancellation complete the file walker is ready to be used. Just watch out for symlinks which is/will be covered in Part 3

Filesystem Walking: Diving Deeper Than Beginner Tutorials, Pt. 1 - Previous Article
Canel operations - Go Blog
Go Concurrency Patterns: Context - Go Blog
Context for Cancellation and Timeouts in Go - Tutorial I liked