If you need to add rel="fancybox" in the standard wordpress gallery, then this is a lesson for you.
Firstly what we need to do, its enqueue fancybox script, add image sizes for thumbnail and big picture and change implementation of gallery shortcode by default.
So let's do it.