You should probably start taking things apart. Grab the manual.

Clean out the dispenser so it has no residue. Clean out the impeller housing, spray arms, etc. Basically clean the whole machine out very well, run some vinegar through it and then try again with the detergent.

I use a green/eco (store-brand) powder that has no phosphate and it works very well. At least as well as the Kirkland (Costco) brand I used for the first year that did have phosphate. I've also tried Seventh Generation and that worked well too. These detergents have always seemed to wash better than the name-brand (Cascade, electrosol, etc.) ever did in my old dishwasher or those of my parents. My previous one was a POS GE, but my parents have always had some nice Bosch unit.