Making Nginx + PHP (FPM) + APC work well is more of an art than technical skill. When you add memcached / redis and varnish and maybe a varying amount of available system memory, it gets even funnier.
In particular, both NGinx and APC come with very conservative if not downright bad default settings. It's important to give APC and memcached enough room. The last (Xen) VPS I prepared, has Nginx in high availability configuration + PHP FPM + APC + memcached + varnish + MySQL 5.6 (in a master-slave setup) and it runs ISPConfig3, WP 3.6 with W3TC, a wiki and Magento 1.7 + another famous e-commerce software I am not authorized to disclose + phpmyadmin + awstats + squirrel mail and more. It works like a charm, including in the admins.
Therefore if my configuration works, then yours should too.
If it does not, I'd check the amount of memory given to APC as it seems to struggle dealing with demanding tasks like phpmyadmin and WP admin are. There is also a strong case for checking Nginx parameters as they command over PHP's ones and Nginx comes with absolutely insufficient defaults for enhanced, large web apps.