In the factory manual (link at top of forum) it says about heating the cover up using a special tool (I was just going to warm it up with a hair drier to help things flex without breaking). Ambiguously it doesn't mention removing the button but it does mention replacing it
I don't want to go forcing it and breaking a clip (already done this with the incredibly fragile rear boot lock trim to adjust the very poorly adjusted boot latch from the factory!).