Borage - look weird but tastes like cucumber.  This underappreciated annual herb is a gem.  It prefers warm temps, so direct-sow the seed or place transplants into the garden when nighttime temperatures are consistently above 50 degrees.  Once this is planted you will never have to plant it again-unless you move it. 
Use:  Use the new leaves and flowrs, which have a delicate cucumber flavor, as a garnish in salads, drinks or froze ice cubes.

Garlic chives - pack a pungent punch.  With their flat, wide leaves, garlic chives are slow to emerge in spring.  They have a wonderful garlic flavor, great green color, and showy white flowers that look stunning in a garden planter.  The cut flowers hold well in a vase bouquets.  Buy this herb as a transplant, or because it divides easily, encourage a friend who has it to give you a hunk.
Use:  Chop the leaves and use fresh or dried in soups, salads, and marinades to add some garlicky onion flavor.  The flowers make a great garnish.

Stevia - the sweetest herb of all.  The leaves of this plant are sweeter than sugar.  Because it loves warm weather, it must be planted when temps are 50 degrees or above.  Be sure to cut off the white flowers to encourage this herb to make more sweet leaves.  You can plant directly into a garden pot and then bring it indoors when it gets cold.

Here are some great ideas:
Infusion:  1 C water

              4 stems stevia
Bring the water to boil, add stevia leaves, allow to steep until cool.  Strain, and refigerate for up to 5 days or freeze to use later.

Another fun idea:
Mix the infusion (above) to taste with unsweetened tea or lemonade.  Serve over ice, and garnish with a sprig.

Anise hyssop - is as pretty as it is tasty.  This anise scented perennial has long, showy purple flowers, which keep coming throughout most of the summer.  If you are going to eat this herb make sure you buy the edible not ornamental perennial so you won't run the risk of it being treated with chemicals.  Anise hyssop is easily grown from seed.

Use:  The leaves have a natural sweetness and can be steeped in warm honey and served with raspberries.