2137: Mrs. Hennessy Returns

[Rosa]

Originally Posted by robotparker

Don't mind if I do!

I approached the problem with the mind of a photographer. I got my BFA in Photography and Media Arts and have worked as a freelance photographer for almost a decade, in addition to teaching photography on occasion so I'm no stranger to looking at bad images. When I first saw the original viral image, I immediately pegged it as a poorly exposed, mis-white-balanced image of a white and gold dress. And despite all the explanations I've seen to the contrary (including yours), I've never been able to see it otherwise.

Now, there's no denying that in the original image there is, in fact, blue present in the dress. But I submit that the actual dress isn't blue and only appears to be because of the problems inherent in the low quality sensors and image processors you find in the average cell phone. When I'm touching up and correcting images in a professional context, the first step is always to make sure the white balance is correct. Every light source in a setting affects the overall color temperature of the image, so unless you're working with direct sunlight or controlled studio lighting then your camera is going to have to rely on its best guess and often it's not quite right. Here's that phenomenon in action:

Were people honestly convinced that these were the same dresses? Why can't my mind accept it? I thought that perhaps maybe it was just a matter of complicated lighting and I tried to conceive of what that scenario would look like in practice. Because even if something's black, it would still have some capacity for refection, especially if it were a shiny piece of fabric. If you had a bright enough amber colored light right next to the dress, then it could feasibly result in the black part appearing to be gold in a badly exposed image. And then one night while I was driving, I saw the phenomenon in action. A local bar's sign, comprised of sparse white text and a big black square was being illuminated by the gross orange/amber glow of a street light, and sure enough the highlighted portions of the black did appear to be golden. But the problem was, the white letters also appeared to be golden.

There's simply no scenario where a light powerful enough to make a solid black piece of fabric look golden would also make an adjacent blue piece of fabric appear white. It totally flies in the face of all the practical knowledge I have of photography and studio lighting practices. As for the actual parties involved insisting that the dress is black and blue, I dunno what to tell ya. Maybe there was some sort of miscommunication somewhere that snowballed out of control and/or maybe there's some trolling aspect to it. All I can say is that there's no way a dress can melt steel beams.

SWEEEET. You touch on a bunch of the really interesting nuances in this web.

I dont' even know where to begin, but this is great.

you are totally right, the exposure in this image is whack. thats why this image is 1 in a trillion(?). and why it breaks our brains. it is not realistic, and it is full of ambiguities/conflicting information. it shouldn't be able to happen, given how cameras work. and also as you mention, the material properties of fabric are important. I'm going to punt fabric for a moment, and will come back to it. First, we must address the elephant in the room. we both agree this image should not exist. it should be impossible. May I propose one way that this weird image may have come about (in terms of camera behavior)? your critical feedback would be seriously appreciated on this one...

I wonder if what happened was there was (a) a cool light coming in from behind the dress (that big bright white strip along the side of the image--maybe its a window with bright mid-day daylight, which is blue-biased, pouring in, and (b) there was a bright warm (yellow-biased) incandescent light hanging above the dress, directly lighting it (common in department stores). Then, when the woman pointed her camera at the dress, the camera detected the cool blue light in the background and white balanced for a cool illuminant, even though it should have been calibrating to the warm light that was illuminating the dress from above. If this is the case, it would enhance the amount of warmth left in the image, and would also serve to remove extra blue from the image. So if you had a blue and black dress, the blue would get washed out (pushed in the white direction) and the black would get pushed in the yellow/orange direction. I am an amatuer photographer, so please correct me if I'm wrong on this one, but it feels sensible. One additional reason to believe this could be the case is that the most golden black region of the dress is near the top (neckline), where you could imagine getting that bias from an overhanging light.

OK. Fabric and material properties. I didn't bother touching on this in my earlier post, as it adds a whole additional dimension to the problem. But when I talked about our internal model for the world and the kind of inferences we make and how we weight our various priors, one of the important features that goes into this model is the inferred material properties of the object and our priors about how light interacts with certain material types (which is why it is important to know whether a fabric is matte or shiny). Whatever you infer, this will influence how you interpret the reflectance and the illumination. The appearance of matte and shiny materials should be predictable for shadow vs direct illumination. For this reason, we actually collected data on people's fabric judgments in our study. We indicated four different regions of the dress and had subjects indicate whether they thought the material was matte, sheer, semi-gloss, or high-gloss. We chose a "black/gold" region and a "blue/white" region at the top, and then again a "black/gold" region and a "blue/white" region at the bottom. This is because the illumination cues vary along the vertical axis of the dress... as if the white balance is not consistent across the image. Just to make things extra ambiguous.

The most interesting region to me is the trim around the neckline. This is where the most golden pixels lie. For me, it is this region that conveys the greatest ambiguity about the illumination conditions. When I want to push the white-gold percept, this is where I focus. It is also the region where people’s fabric judgements best predicted if they were Blue-Black (BB) viewers or White-Gold (WG) viewers. In our data, BB folks judged that region to be made of a matte material more often than WG folks, while WG folks judged the fabric there to be more semi-gloss (shinier). If you think the material is matte, then perhaps you attribute the golden glean to being the result of a black matte fabric that is *very* washed out due to being lit by a *very* bright yellow direct illuminant. If so, your brain removes a bunch of the yellow bias in the image giving you the blue/black percept. But if you think the material is glossy/shiny, you think the gold indicates the true color, but that the dress is in shadow (otherwise a shiny material would give off more of a white shine/highlight). If the dress is in shadow, you need to remove some blue (shadows are blue-biased), which enhances the white gold percept.

Here's the kicker: The "black" trim along the neckline is actually sheer, and has some interesting reflective properties that would be worth measuring with a device (see below, or if that pic isn't any good, just take my word for it, because I have the dress in front of me... because I'm such a dork that I ordered it for my Vision Science Conf. in florida this summer... and because I want to take spectral readings under diff. illuminants to see how it behaves). It's the only sheer part of the dress. But most people judge the material there to be the same as the trim everywhere else. But it will necessarily behave differently, so we are probably not modeling that fabric correctly. not sure if this is something that would impact the camera's too... I need to think more about this. if you have thoughts, I'd love to hear them.




To come back to the main point you raised: the exposure in this image is whack. and this is why it breaks our brains. it is not realistic. it shouldn't be able to happen, given how cameras work. But this brings up another interesting feature of the whole phenomenon: because this is a photograph, our brain insists on interpreting it as if it were realistic. and given how weird it is, this ends up being super fucking difficult to do, yielding a very unstable decision threshold that people actually fall on different sides of. this is incredibly rare in color vision. there is no other bi-stable color illusion this compelling in existence (the colored Mach Card is the only one, and that requires actual 3D cues....like a folded corner). SO, given that our brain has to make some decision, it weights very heavily our internal model's priors, which includes the fabric stuff discussed above, but most importantly, information about the typicality of daylight and how it behaves: The impact of our prior over daylight is huge in this illusion. The distribution of pixel values in this particular image (mostly blue and orange/brown.... *brown is dark orange) perfectly aligns with the daylight axis distribution (sunlight measured over the course of the day... you lay a sheet of white paper outside and take spectral readings every hour, then plot the distribution). And the field has good fMRI, ephys, and behavioral data that suggest our neural illumination discounting mechanism is specially tuned to the daylight axis (Lafer-Sousa et al 2012; Pearce et al 2014); we think this is actually very important to understanding why the image is bi-stable and gives two such salient distinct percepts; essentially, you strongly discount one end of the daylight axis or the other; this is the first panel in our figure in the paper, so I can't reproduce it here; but suffice it to say, the two tightly overlap and I think this is something that ends up contributing to how strong the discounting we do is.... I say all of this just so that you fully appreciate that your percept is a) a highly internally biased interpretation of a very ambiguous situation, b) is really quite different from the pixel values (you are doing a lot of illumination discounting) and c) is really quite different from mine, because mine has the same features, but is pushed in the opposite direction along the daylight axis. oh god. i hope any of that makes sense and is useful to you....

I find these figures useful too (made by color vision expert Qasim Zaidi for the NYTimes last week):

[Rosa]

Originally Posted by robotparker

Here's another example that might be clearer:



We all know snow is white, but if you make an image of snow not being hit by direct sunlight, it's probably gonna appear to be a little blue. The original image of the dress faces similar challenges: the actual dress appears to not be directly lit, but the background is very bright which further exacerbates the poor white balance and exposure (quick side lesson: try photographing someone standing in front of a bright window. you can either correctly expose your model, or correctly expose the outside of the window, but not both).

I also like your snow example, since our brain does the same thing. A friend took the girls from my illusion and put them into two snowy boston scenes to make a version of my illusion using natural images, its nice because as you say, we know snow is supposed to be white. But snow that is illuminated by shadow (which is ambiently lit by scattered refracted diffuse blue skylight; one end of the daylight spectrum) will have a blue bias (as you say), and snow illuminated by direct sunlight (especially early and late in the day) will have an orange bias (other end of the daylight axis). Just as you white balance for this, Our brain has an internal model to white balance for this. the result is that if I put the dress on a girl in these two conditions, in one case your brain will remove what it expects to be a blue bias present (gives WG dress), and in the other expects to be an orange bias (gives BB dress):




Does this illusion work for you? do you ever see a Blue-Black dress? how about in my other illusion?

[ghauck]

Originally Posted by Rosa

Does this illusion work for you? do you ever see a Blue-Black dress? how about in my other illusion?

Thank you! Best side by side example I've seen - I clearly see WG and BB between each.

Finally, my brain can put this to rest and stop worrying I'd become Jfod crazy.

[Rosa]

Originally Posted by ghauck

Thank you! Best side by side example I've seen - I clearly see WG and BB between each.

Finally, my brain can put this to rest and stop worrying I'd become Jfod crazy.

SWEET. Did this version not work for you?

[Rosa]

Originally Posted by robotparker

Don't mind if I do!

I approached the problem with the mind of a photographer. I got my BFA in Photography and Media Arts and have worked as a freelance photographer for almost a decade, in addition to teaching photography on occasion so I'm no stranger to looking at bad images. When I first saw the original viral image, I immediately pegged it as a poorly exposed, mis-white-balanced image of a white and gold dress. And despite all the explanations I've seen to the contrary (including yours), I've never been able to see it otherwise.

I should also say: your immense familiarity with over exposed images suggests you have very strong priors for a certain kind of interpretation. Need to think about if it actually predicts you'd see white gold. But this is interesting. Another data point we collected was we asked people if they had professional or hobby-level experience in the visual arts. We had specific category flags including photography. Because we wanted to know if that kind of experience predicts which camp you will fall into. We have yet to look at that data, but the intuition was that it could be telling.

I see now that you said my illusion doesn't work for you. We did find in our data that WG viewers tend to be stickier in their percept. They are significantly less likely to flip their percept with my illusion. But the vast majority still do switch.

Ahhhh this is too much fun but I really should get back to work. I put a different project on hold for the last two weeks while we hunkered down on this dress stuff and now that the prelim paper is submitted I need to switch gears back to the other project. Once that's done I'll come back to the dress stuff to work in a paper that tries to understand where the individual difference come from. At this point the holy grail is figuring out how to predict who will be WG and who will be BB. This will probably involve some hardcore Bayesian modeling. Thanks for going down this rabbit hole with me!

[Scumhook]

Thank fuck my mouse has a unlockable scroll wheel.

[ghauck]

Originally Posted by Rosa

SWEET. Did this version not work for you?

Originally Posted by Scumhook

Thank fuck my mouse has a unlockable scroll wheel.

It works too but my mouse was unlocked and I scrolled by it earlier.

We should stop this thread before we prove solipsism and inadvertently implode everything outside of the KATG world.

[Rosa]

Originally Posted by ghauck

It works too but my mouse was unlocked and I scrolled by it earlier.

We should stop this thread before we prove solipsism and inadvertently implode everything outside of the KATG world.

agreed. but can we all agree that it's all a simulation? or do I have to go find that poll...

[Scumhook]

Originally Posted by ghauck

solipsism

I hope that's a made up word.

[Enunciated Piffle]

Wowza. I think I learned more in the hour it took to read and comprehend Rosa's posts then I did in the last decade. I also have been taking pics of snow for years now and always wondered why it looks so damn blue. So RobotParker, thank you for solving that particular mystery.

The brain controls how you experience the world. As a depressed person, I'm always fascinated by people who can interpret the world in any other way. I had no idea that this extends to color as well. But it makes sense now. Kinda.

Rosa. Your findings are incredible. WIRED is a big time publication and cheers to you and your collaborator on all attention/ success. This is America though. Science needs to be marketed. Sold. How can this info be translated to dollars?

Take my old, favorite Ween band tour shirt, for instance. It's old as fuck. It used to be brown, but is now faded and whitish in appearance. You say there's no such thing as light, but then this shirt was brown; now it's white. Is this from it's inability to reflect light? Do colors ever fade if they never existed in the first place?!

We need to create a band T-shirt that never loses it's color. That's were the money is