Go vendoring tools, glide, and local cache naming

Go vendoring tools, glide, and local cache naming

Prologue

I recently started migrating my various libraries on Github from names like github.com/lestrrat/go-foobar to github.com/lestrrat-go/foobar where github.com/lestrrat-go is an organization.

I did this after learning Olivier’s technique (seen in below tweet), mainly because I started accumulating way too many repositories under my regular username github.com/lestrrat .

I did this by first transferring the original project to an organization, and modifying the import paths in that transferred location (from github.com/lestrrat/go-foobar to github.com/lestrrat-go/foobar).

One minor note: while Github automatically redirects access to transferred repositories, in Go the path names matter. So merely letting moving the repository location is going to leave users possibly with code checked out in the old path, while the contents of the library may have a different path in them. To avoid this, you would need to leave a copy of the old version in the old repository path and mark it as archived, which is what I did. e.g.: https://github.com/lestrrat/go-msgpack and https://github.com/lestrrat-go/msgpack

Enter glide, and Cache Keys Collide

Now this is where the fun part begins. I use glide for vendoring.

(Please don’t post comments saying I should be using some other vendoring tool X. As of this writing (Mar 2018) I personally do not find any advantage in migrating, as I really can’t tell which one of those tools are going to be The One yet. I’ve been using glide for a long time, and with all of its quirks, it works)

For performance, glide keeps a local cache of repository clones in ~/.glide/cache . Here’s the path where github.com/lestrrat-go/foobar will be cached under the following path:

~/.glide/cache/src/https-github.com-lestrrat-go-foobar

Now, remember the old repository, before the transfer? It’s github.com/lestrrat/go-foobar . This will be cached under the following path:

~/.glide/cache/src/https-github.com-lestrrat-go-foobar

Ooooops. The cache key (the path) is not unique, even when we have different paths… One simple solution is to just escape the repository URL via url.PathEscape instead of doing manual translation, or maybe use a SHA Digest based on the repository path, …but oh well.

Implications, and Countermeasures

Now, normally this is not a problem, but if you happen to be vendoring one of my libraries using glide, and you update them to the new repository paths, you may have a problem.

The first time you switch paths from github.com/lestrrat/go-foobar to github.com/lestrrat-go/foobarand run glide install, glide may complain that it can’t update the specified repository because the remote URL does not match or some such.

In that case, what you should be doing is to remove the affected package from ~/.glide/cache/src or if you don’t care, run glide cc to clear the cache.

Even after you do this, you may still find glide repeating the same error message, but for a different package (of mine, probably). If that’s the case, then you probably have other packages, possibly because github.com/lestrrat/go-foobar uses the new path, but you still have the old path referenced from somewhere. In this case, you should make sure that you are using the same path (version) everywhere, and clear the cache again.

Repeat this until you are done, and you should be fine.

Why Aren’t You Sending Issues To Glide?

Well I will (updated: I did). But you should also know that glide is in sort of a freeze mode because of dep. I can totally understand if the maintainer does not want to work on it. In that case, the best thing for me to do is to leave this memo out in the open, so people can possibly google for it.

Happy hacking!