Two static IPs resolving to the same Apache directory

I currently have a dedicated server that has two static IPs pointing to it, and two domain names pointing to their respective static IPs. Both of those domains have virtual hosts, and both of those domains resolve to the correct Apache directory. Let's call them domain1 (Associated to IP1 through DNS) and domain2(Associated to IP2 through DNS). The hostname is set to the fully qualified domain name (www.domain1.com)

The problem is, if I attempt to navigate through either IPs on a browser, it always refers me to domain1. I would like this to not be the case. I would like direct IPs to either resolve to an error, or to resolve to their specific domains. What am I doing wrong?

EDIT: As asked by a commenter, here are both of the virtual host files. 000-default.conf is empty.

domain1.com.conf, note that this one is a Symfony installation and runs without www:

<VirtualHost *:80>
    ServerName domain1.com
    ServerAlias domain1.com

    DocumentRoot /var/www/html/domain1/web
    <Directory /var/www/html/domain1/web>
        AllowOverride All
        Order Allow,Deny
        Allow from All
    </Directory>

    # uncomment the following lines if you install assets as symlinks
    # or run into problems when compiling LESS/Sass/CoffeeScript assets
    # <Directory /var/www/html/domain1>
    #     Options FollowSymlinks
    # </Directory>


    ErrorLog /var/log/apache2/project_error.log
    CustomLog /var/log/apache2/project_access.log combined
</VirtualHost>

<VirtualHost *:443>
    ServerName domain1.com
    ServerAlias domain1.com

    SSLEngine on

    SSLCertificateFile "/etc/ssl/certs/domain1_com.pem"

    SSLCertificateKeyFile "/etc/ssl/private/domain1.key"

    SSLCACertificateFile "/etc/ssl/certs/domain1_cert.pem"

    DocumentRoot /var/www/html/domain1/web
    <Directory /var/www/html/domain1/web>
        AllowOverride All
        Order Allow,Deny
        Allow from All
    </Directory>

    # uncomment the following lines if you install assets as symlinks
    # or run into problems when compiling LESS/Sass/CoffeeScript assets
    # <Directory /var/www/html/domain1>
    #     Options FollowSymlinks
    # </Directory>


    ErrorLog /var/log/apache2/project_error.log
    CustomLog /var/log/apache2/project_access.log combined
</VirtualHost>

www.domain2.conf, this one doesn't have a ssl certificate yet, so I only put in the virtual host for port 80;

<VirtualHost *:80>
    ServerName domain2.com
    ServerAlias www.domain2.com

    DocumentRoot /var/www/html/domain2
    <Directory /var/www/html/domain2>
        AllowOverride All
        Order Allow,Deny
        Allow from All
    </Directory>

    # uncomment the following lines if you install assets as symlinks
    # or run into problems when compiling LESS/Sass/CoffeeScript assets
    # <Directory /var/www/html/domain2>
    #     Options FollowSymlinks
    # </Directory>


    ErrorLog /var/log/apache2/domain2.log
    CustomLog /var/log/apache2/domain2.log combined
</VirtualHost>