That is very unusual to see that.... I get zero difference here between direct to modem and using the router....
Do you have QoS set up on the router ? if so, disable it and try again....
Also, don't use a single test as your guide....even moments apart you can get varied readings....use an average...just my opin... 8-)
Are you running a Gb ethernet port on the pc ? What modem are you using ? What is the router's WAN port speed set to ? Typically use AUTO....
On speedtest.net make sure you are seeing the same IP for the tests....I had an instance where using direct to modem, it would report one IP address....and SOMEHOW, as yet unexplained, when using the DIR, it would grab a totally different IP from comcast....