The OVPN client has a functionality that minimizes the client to the system tray. This means that if you minimize the client, it will keep running and be available as a small icon in the system tray.
The Minimize to tray functionality can be activated/inactivated from the Settings tab in the client.