Preheat the oven to 350 degrees F. Butter two loaf pans.
Whisk together the flours (you can also use a combination of ap and whole wheat flour. Just make sure that it adds up to 3 1/2 cups), baking powder, cinnamon, salt, and nutmeg.
In another bowl, cream the butter and sugar together. Add the mashed bananas, eggs, and vanilla extract.
Stir the dry ingredients into the wet. Add the nuts, if you're using them.
Divide the batter between the two loaf pans and bake for 50-60 minutes, until a toothpick comes out clean.