To delete a connection click on the arrow and select the Remove Object(s) on top bar or press the delete key.
