Garden club backs out of Colonial Garden