Indeed; I believe casual dress in general-purpose, public adult leisure contexts was initially associated strongly with the upper/upper-middle classes and was seen as a signal of higher status, even in cases where the dress item in question was an appropriation from the working class (e.g., jeans).