I feel stupid for not being aware of a global git ignore earlier. It’s a good approach to globally ignore those editor-specific swap files, OS-specific crap (like .DS_Store in OS X) etc. From usevim:
A better approach is to use a global option. Run git config –global core.excludesfile ~/.gitignore. Then you can add *.sw?, and perhaps *~ as well. I like to add .DS_Store because Windows developers don’t need to worry about that nonsense, and they should probably add Thumbs.db.
The trick is to add globally-ignored files into some place like ~/.gitignore and specifying this in core.excludesfile.