As a White guy, I'll never know what it is like to be someone's only friend of color. I've had a few experiences of being someone's only White friend (a couple Asian immigrants, a Mexican-American, and a few African exchange students), but I recognize how different those experiences were from what a member of a racial or ethnic minority group might experience. I imagine it is probably different in some ways I can't even appreciate without having their perspective.
One experience I have had several times involves being Christians' only atheist friend. And despite some minor annoyances associated with how some of them behaved in this context, I can't say I minded it all that much. I often suspected that they had other atheist friends but that I was just the only one who was willing to openly discuss it. And as long as that was okay with them, it was okay with me. In many cases, it even turned out to be a good experience for both of us.
The biggest annoyance would be when they would introduce me to someone I was meeting for the first time and announce that I was an atheist within the first few moments of doing so. There were a few times when it would play out like this: "This is my friend Jack...and he's an atheist." A couple times, my hand would be outstretched for a polite handshake when the "and he's an atheist" part dropped. That was always awkward. The part I often found puzzling was that some of these Christians seemed to take so much pride in the idea that they had an atheist friend. Maybe they thought it signaled their open-mindedness or something. And who knows? Maybe it did.
This annoyance was usually easy to fix. I'd just ask them afterward if they could maybe delay outing me as an atheist to a complete stranger for a few minutes. They were almost always willing to agree to that, especially if I included something about how not all of their fellow Christians were as tolerant or accepting as they were.
Are there any perks associated with being someone's only atheist friend? I suppose so. There have been many times in my life where I would have had no friends at all if I had been unwilling to be someone's only atheist friend. In fact, this has been the case for most of my life. It was the case throughout high school, most of graduate school, and it is very much the case now. College brought a pleasant reprieve, but those years were atypical in many ways. Besides that, the Christians who have been able to get past my atheism have generally been good people. Instead of seeking every opportunity to push their faith, they've been genuinely interested in learning about what others believe. That's certainly been a positive experience for me. And of course, I've enjoyed learning more from them too.
I think the benefits can go in the other direction too. There have been a handful of times when one of my Christian friends expressed anti-atheist bigotry in front of me. While some of these occasions have been more challenging than others, they have also been decent learning opportunities. When I am able to keep a lid on my anger and calmly explain how what they said impacted me and how it is likely to be perceived by other atheists, the result is usually positive.
Are there any downsides to being someone's only atheist friend? If so, I'm not sure what they'd be, and I'd be surprised if they weren't outweighed by the positives. I suppose there are some atheists out there who would refuse to spend time with any religious believers no matter how they behaved, but I'm happy to say that I've never counted myself among them.