Will I need to have different OpenCL optimizations for different devices?

"Not necessarily. Will you add a small #ifdef in your code to run 50% faster?.. Will you duplicate a 1000-line file for that? Would you do it for only 10% speedup? Or, maybe you would prefer adding the optimization unconditionally and pay 10% slowdown on other devices for 50% improvement?.. It is totally your decision. In some cases, you will need to make the tradeoff between cross device performance and maintainability of your OpenCL application". 

=> OpenCL* Design and Programming Guide for the Intel® Xeon Phi™ Coprocessor