Whisk flour, baking soda, baking powder, salt, and pumpkin pie spice in a large bowl. In a separate bowl, whisk pumpkin puree, oil, both sugars, eggs, milk, and vanilla until smooth. Fold wet into dry until just combined.

Stir melted butter, ⅓ cup brown sugar, and cinnamon together until it resembles wet sand.

Line a 12-cup muffin tin with liners. Spoon a tablespoon of batter into each cup, top with a heaping teaspoon of cinnamon mixture, then cover with remaining batter. Add any leftover cinnamon mixture on top. Swirl gently with a toothpick in figure-eight motions.

Preheat oven to 375°F (190°C). Bake muffins for 18 to 22 minutes, until a toothpick comes out with moist crumbs. Cool in the pan 5 minutes, then transfer to a wire rack to cool completely.

Beat cream cheese until smooth, then beat in powdered sugar, 1 tablespoon milk, and vanilla until creamy. Add more milk if needed to drizzle. Spoon glaze over cooled muffins and let set 10 minutes.