If you're interested in a B&B, we stay here when we go to Victoria, the owners are fantastic, rooms are amazing, breakfasts are delicious, rates are decent. We usually stay in the honeymoon suite, has great views, bed is super comfy, they have other suites that are more in your budget. All the rooms are self contained, have fridges, toasters, microwave and kick ass bathrooms. It is near Royal Roads University.
http://www.birdsofafeather.ca/rooms
Red Fish Blue Fish on the warf downtown has awesome food, gotta get there early to beat the line ups though. My favorite is BBQ Fanny Bay oyster Sandwich
http://www.redfish-bluefish.com/
As mentioned, Nautical Nellies is another awesome place
If you like Dim Sum, the best I've ever had is in China Town at Don Mee
Best Pho I've ever had and I've tried many places is Pho Vuong