OK, this was easier than I thought. Just added another section between the first and second location block:
location ~ (.*)\/refs {
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-Proto http;
proxy_set_header X-Forwarded-Port 80;
if ($arg_service = "git-receive-pack") {
proxy_pass http://#push-ip#:8080;
break;
}
proxy_pass http://#pull-ip#:8080;
}