How do you securely set up management interfaces on VLAN on the other side of a layer 3 switch

I have four servers connected to our organization's network. I've obtained a layer 3 switch (Cisco SG300). I've separately connected each server's management interface NIC to this switch. (The management interfaces are Dell iDRAC, in case it matters.) Now I want to isolate the management network on this switch for security reasons, connect the switch to our organization's network, and only allow outside connections from specific hosts such as my laptop.

                                     ,- server 1 management interface
                           ,-------. +- server 2 management interface
external (open) network ---+ SG300 +-+- server 3 management interface
                           `-------' `- server 4 management interface

I think I can work out VLAN configuration for the management network on the right-hand side of the SG300, and if I understand ACLs on the Cisco switch correctly, I should be able to create an ACL that allows only a specific MAC address from the external network to connect through the SG300 to the VLAN on the right-hand side.

My problem is this: how can a connection from the outside network specify which destination (1-4) to connnect to? Suppose the management NICs have IP addresses 192.1.1.1 through 192.1.1.4, and say I'm on the external network, and I want to connect to machine 3's management interface. How do I do that? The servers' management interfaces will not have IP addresses on the external network, so I can't connect to a specific IP address. How do I indicate the desired destination?

This is probably a basic networking question, and obviously I lack clue, but after beating my head against Google for quite some time now, I can't figure this out. What is the basic approach to achieving this configuration, and are there resources that explain how to make it happen?