Cut angel food cake into 1/2-inch thick layers. Cut out 32 rounds with the same diameter as the push pop molds. For easier cutting, press out cake rounds with a round cookie cutter. Set aside.
Beat butter in large bowl with electric mixer on medium speed until light and fluffy. Add lemon extract; mix well. Gradually add confectioners' sugar, beating until well blended after each addition and scraping sides and bottom of bowl frequently. Add milk and red food color; beat until light and fluffy. Fill piping bag fitted with a small plain tip with frosting.
For each Push Pop, pipe a small amount of frosting into push pop mold. Place 1 cake round into tube, pressing into the frosting. Top with a small amount of frosting and a second cake round. Repeat layering for a total of 4 cake layers. Top with frosting.