Почему MySQL не может открыть hosts.allow/hosts.deny?

Given that Nexenta was OpenSolaris-based at that time, you may be running into this issue. It does seem like file descriptor limits aren't being applied in a consistent manner throughout the system.

Are you actually using hosts.allow/deny in your setup?

What does lsof output look like?