I think it's about a relationship that is past the point of return of being healthy. One that has gone too far to go back. As much as they try and care so much about each other, it just doesn't work. A relationship where they are very comfortable with each other and somewhat codependent. They can't imagine ending it completely and not being in each other's life even as friends. They have a deep connection, a soul tie. As much as they try, they both know that the reality is they have to let each other go at some point. They are holding on to this fantasy of it working and what use to be before it sets in that how much they care about each other isn't enough to hold on. So many times in life, the people you want so much for it to work and could be your soulmate, it doesn't always mean you are going to be with that person. Very sad and leaves an emptiness inside that can never be filled by yourself or anyone else. The way a soulmate makes you feel is unlike anything else and when you can't be together you are forced to walk through the rest of your life feeling like something is always missing.
I like your intepretation of the song, except that this is only James Bay’s feeling toward the relationship. James Bay does not want to continue on with a relationship that he thinks no longer works. Your interpretation has the word “they” and this could be understood that they (James Bay and his special someone) both believe the relationship can no longer work. The other person may probably try to hold onto it though. Thank you!