Mix fig jam and rosemary. Set aside. Butter 1 side of each of the bread slices. Spread jam mixture on each of 4 bread slices with buttered side down. Layer each with Brie cheese. Top with remaining bread slices with buttered side up.
Place sandwiches in large skillet or griddle on medium-low heat. Cook 3 to 4 minutes per side or until bread is browned and cheese is melted. Cook in batches if necessary.