Preheat the oven to 325 degrees. Grease and flour a 9×5 inch loaf pan (or 3 small pans; 3 1/2 x 6-inch pans).
Combine the milk and lavender in a small saucepan over medium heat. Heat to a simmer, then remove from heat, and allow to cool slightly.
With a hand mixer, in a medium mixing bowl, cream together the butter and sugar until smooth. Beat in the eggs and lemon zest until the mixture is light and fluffy; add in Greek yogurt, and vanilla.
In a separate bowl, combine the flour, baking powder, and salt; gently mix into the creamed mixture alternately with the milk and lavender, until just blended.
Pour the batter into the prepared pan(s).
Bake for 45-50 minutes (loaf pan). For the 3 small pans, bake for 30 minutes.
To prepare glaze, combine powdered sugar, lemon juice and vanilla. Poke holes with a fork into each loaf (about 6-8 times per loaf). Spread glaze over hot cake. Cool in pan 20 minutes on a wire rack; remove each loaf from the pan. Cool completely on wire rack.