To put it another way, Pogba is the CM hovering around the box that Pjanic or Pirlo passes to. He is not Pjanic or Pirlo. The more he's on the ball in midfield, the less he can use his physical advantages (height, strength, jumping) and skills (shot, first touch dribble) around the box.
I guarantee every defender is happy to see Pogba on the ball in midfield instead of having to watch him out of the corner of their periphery for 90 minutes. Is he going to stay on the 45 of the box and smash it into the corner. Is he going to instead sprint from that area right onto the penalty spot and smash me into the net. Is he going to get behind me when the #9 drops into midfield?