Ok, I understand now.
The most important part is this: "3500fps vs 2500fps". FPS scales non-linearly. This means that having such big difference means nothing. If your fps was 60 then the same change might have changed it to 59.
To really see how much difference there is you would have to create so many text objects that you get about 60 or 30 fps and then change them to Labels. And then instead of measuring fps just measure the time per frame (which is not reliable while getting thousands of fps).
A certain performance loss is to be expected. Label is so much more than a simple sf::Text, although just drawing it is optimized so that there is practically no difference. The only thing that Label::draw does more than Text::draw is executing a few if statements to skip parts of the drawing, it will only takes a lot more time when you e.g. set a background color or disable auto-scaling. This part can't be optimized much more.
There are also other parts in the gui where you might loose some performance, e.g. every frame the elapsed time will be updated and all widgets will receive an event for this in case they need to do something. Things like these can also take some time. The goal of the gui is to be simple, not to be the most performant one (although functions like the draw function which are executed a lot are usually more optimized than e.g. loading widgets).
To summarize: Less performance is expected but the big fps drop says nothing about how big the performance difference really is.