Have you tried creating a new group and adding members you want to have full access? Then change all the permissions of the single forum/category to "no access" for the people who won't be allowed to see it? And the same for the new group - but they get full access? If that makes sense? :S