I had to borrow a phone to set up a mobile hotspot.
It has the same speeds inside the wireguard tunnel as when I tested from my wired connection (250 kbps TCP, 170 kbps UDP).
The loss reported by iperf is dependent on the bandwidth that i test with. But as I increase bandwidth from the client the loss grows towards 100%.

I tried testing in reverse (sending from VPS to devices on different networks) with surprising results:

• TCP, wireguard: 5-10 mbps
• UDP, wireguard: 50 mbps
• TCP, no wireguard: 45 mbps
• UDP, no wireguard: 250 mbps (saturates download speed on client when compared to speedtest.net)

Thanks! I had not read the manpage close enough, I guess.
When specifying the bandwidth, I can saturate the connection using UDP to around 15 Mbits/s. (That this speed is much much lower than the 300-500 Mbits my connection and the VPS is capable of is a problem for a different time, I think).

What I also realized is, that I had not put the iperf server in UDP-mode, so my results reported in another comment are wrong.
I read the results from the client, but the server did not respond.
When running the iperf server in UDP-mode, I get 15 Mbits/s outside the wireguard tunnel and 180 Kbits/s inside the tunnel. With TCP-mode I get 10-15 Mbits/s outside the thunnel and 250 Kbits/s inside the tunnel.

I am currently running mtr from multiple devices:

• in wireguard tunnel: single hop 10.1.1.1, loss% 0,1, avg ping 27.4 ms
• outside wg between same devices:
  • ISP supplied modem/router 55% loss, avg ping 1.6 ms
  • multiple hops without loss, avg ping 16-20 ms
  • random intermediary 30 % loss, avg ping 20 ms
  • endpoint, 5% loss, avg ping 25 ms

It looks like across the board my ISP modem / router is dropping 50-80 % of packets, and that packet loss is ramping up from 4% to 80 percent after a few minutes of running mtr.
It also looks like my VPS endpoint climbs to 20% packet loss over time (5-15 minutes of testing).

Can I use this information to probe further into the devices I have access to (ISP modem and VPS)?

MTU is 1440.

I’m on linux on all devices. I tested packet fragmentation using:
ping -M do -s [packet size] [ip]
No fragmentation using packet size 1472 and below (before wireguard overhead).

I just tested again:

• no wireguard, tcp: 9.75 Mbits/s
• no wireguard, udp: 1.05 Mbits/s
• wireguard, tcp: 248 Kbits/s
• wireguard, udp: 1.05 Mbits/s

So, I get the “full” udp speed, but I get some errors / warnings about ‘connection refused’ and ‘did not receive ack’. Obviously not correctly configured, when it is 10 times slower than tcp.