Kohen Who Married Convert

In the 1960s, my grandfather who apparently was a kohen fell in love with a convert to Judaism. He knew he was not supposed to marry her, but they were madly in love, and he married her under the auspices of a Conservative rabbi, who was open to such a marriage. He remained orthodox otherwise, and he and his wife moved to a new community and kept this secret, not admitting the truth. His surname was not one associated with kohanim and he didn’t tell anyone he was a kohen. He went on to have five children and many grandchildren, myself included, and many great grandchildren, with more likely in the future.
Earlier this year, my grandfather died at a ripe old age. My uncle went through some of his belongings and discovered the truth. I always knew my grandmother was a convert, but I didn’t know until recently that my grandfather was a kohen who violated Jewish law. What impact does this have on me and my family?